feat(metadata)!: let clients follow metadata changes without re-listing (#4625) Anything that mirrors Iggy metadata had to list every stream, then the topics of every stream, then the consumer groups of every topic, on every refresh. The metadata journal could not replace that: a checkpoint drains it, it holds requests rather than results, it carries password, token and wrapped stream key material, and it hides side effects such as the grants a delete prunes. Nothing recorded who changed what. Keep a bounded log of the public results of committed metadata operations on every node. Handlers describe each result in the apply reply, both apply paths append the records before the applied frontier moves, and the log rides in the metadata snapshot, so it survives restarts and moves with state transfer. Records are encoded once through the option projection the read routes use, so stream keys, wrapped keys and every secret or internal option stay on the server. Topics now keep the op that created them, as streams do, so a recreated id is never mistaken for its predecessor. Serve the log as metadata.changes (code 17) on TCP, QUIC and WebSocket and as GET /metadata/changes, answered by the receiving node and gated by manage_servers. A caller pages records after a cursor, filtered by groups and a prepare timestamp range. When its cursor expires it pages the redacted current state, anchored to a cursor taken before the scan, and replays the real changes after it. Every SDK supports the command on each of its existing client transports, the CLI follows it, and the web UI lists it for accounts that may read it. Records come from the handlers rather than from diffing state, because only a handler knows the assigned id, the previous values and the deleted resource. Only committed and applied operations are recorded, so a cursor from one replica is valid on another, and a lagging node answers behind instead of skipping records. BREAKING CHANGE: data directories need snapshot format 14 and the new storage tag, topics persist their creating op, and the Rust SystemClient trait gains get_metadata_changes.
diff --git a/.github/actions/php/pre-merge/action.yml b/.github/actions/php/pre-merge/action.yml index 3b8ba14..604084e 100644 --- a/.github/actions/php/pre-merge/action.yml +++ b/.github/actions/php/pre-merge/action.yml
@@ -109,7 +109,7 @@ printf '<?php\n' awk 'NF { print "// " $0; next } { print "//" }' ASF_LICENSE.txt printf '\n' - tail -n +2 "$raw_stubs" | sed -E 's/(public readonly \?string \$[A-Za-z_][A-Za-z0-9_]*) = null;/\1;/' + tail -n +2 "$raw_stubs" | sed -E 's/(public readonly [^$]*\$[A-Za-z_][A-Za-z0-9_]*) = null;/\1;/' } > foreign/php/iggy-php.stubs.php rm -f "$raw_stubs" git diff --exit-code -- foreign/php/iggy-php.stubs.php
diff --git a/.typos.toml b/.typos.toml index 4203157..ce0fe9b 100644 --- a/.typos.toml +++ b/.typos.toml
@@ -164,5 +164,7 @@ "foreign/swift/Tests/IggyTests/Fixtures/golden.json", # Resource options fixture copies in every SDK test tree (hex payloads) "**/resource_options.json", + # Metadata change fixture copies in every SDK test tree (hex and base64url payloads) + "**/metadata_changes.json", "**/.python-version", ]
diff --git a/Cargo.lock b/Cargo.lock index b2155b6..ad0b607 100644 --- a/Cargo.lock +++ b/Cargo.lock
@@ -7089,7 +7089,7 @@ [[package]] name = "iggy" -version = "0.11.1-edge.2" +version = "0.11.1-edge.3" dependencies = [ "async-broadcast", "async-dropper", @@ -7127,7 +7127,7 @@ [[package]] name = "iggy-bench" -version = "0.6.1-edge.2" +version = "0.6.1-edge.3" dependencies = [ "async-trait", "bench-report", @@ -7185,7 +7185,7 @@ [[package]] name = "iggy-cli" -version = "0.14.1-edge.2" +version = "0.14.1-edge.3" dependencies = [ "anyhow", "apple-native-keyring-store", @@ -7219,7 +7219,7 @@ [[package]] name = "iggy-connectors" -version = "0.5.1-edge.2" +version = "0.5.1-edge.3" dependencies = [ "apache-avro 0.22.0", "async-trait", @@ -7319,7 +7319,7 @@ [[package]] name = "iggy-mcp" -version = "0.5.1-edge.2" +version = "0.5.1-edge.3" dependencies = [ "axum", "axum-server", @@ -7354,7 +7354,7 @@ [[package]] name = "iggy_binary_protocol" -version = "0.11.1-edge.2" +version = "0.11.1-edge.3" dependencies = [ "aligned-vec", "bytemuck", @@ -7369,7 +7369,7 @@ [[package]] name = "iggy_common" -version = "0.11.1-edge.2" +version = "0.11.1-edge.3" dependencies = [ "aes-gcm 0.11.1", "aes-gcm-siv", @@ -12982,7 +12982,7 @@ [[package]] name = "server" -version = "0.9.1-edge.2" +version = "0.9.1-edge.3" dependencies = [ "ahash", "arc-swap",
diff --git a/Cargo.toml b/Cargo.toml index d93a388..f726d0c 100644 --- a/Cargo.toml +++ b/Cargo.toml
@@ -213,11 +213,11 @@ iceberg = "0.10.1" iceberg-catalog-rest = "0.10.1" iceberg-storage-opendal = "0.10.1" -iggy = { path = "core/sdk", version = "0.11.1-edge.2" } -iggy-cli = { path = "core/cli", version = "0.14.1-edge.2" } +iggy = { path = "core/sdk", version = "0.11.1-edge.3" } +iggy-cli = { path = "core/cli", version = "0.14.1-edge.3" } iggy-gateway-kafka = { path = "gateways/kafka" } -iggy_binary_protocol = { path = "core/binary_protocol", version = "0.11.1-edge.2" } -iggy_common = { path = "core/common", version = "0.11.1-edge.2" } +iggy_binary_protocol = { path = "core/binary_protocol", version = "0.11.1-edge.3" } +iggy_common = { path = "core/common", version = "0.11.1-edge.3" } iggy_connector_doris_sink = { path = "core/connectors/sinks/doris_sink" } iggy_connector_sdk = { path = "core/connectors/sdk", version = "0.5.1-edge.2" } indexmap = "2.14.2"
diff --git a/README.md b/README.md index 4e4f848..871cd18 100644 --- a/README.md +++ b/README.md
@@ -146,7 +146,7 @@ We also publish edge releases (versions with an `-edge.N` suffix, and `apache/iggy:edge`) for the SDKs and the Docker images. They track the latest changes, but are not guaranteed to be stable and are not recommended for production use. -This checkout moves the binary protocol to 0.12.0 and the metadata snapshot format to 13. Both are breaking: older clients and servers do not interoperate with it, and data directories written by older releases are refused. See [Upgrade recovery](core/server/README.md#upgrade-recovery). +This checkout moves the binary protocol to 0.12.0 and the metadata snapshot format to 14. Both are breaking: older clients and servers do not interoperate with it, and data directories written by older releases are refused. See [Upgrade recovery](core/server/README.md#upgrade-recovery). --- @@ -309,6 +309,8 @@ Streams, topics and users carry options from one catalog, listed by `iggy options <scope>` and changed by one sparse update. Each encrypted stream has its own data key, wrapped under the server's `[encryption] key`, and a stream created without encryption stays plaintext for life. Rotating the server key is a two-phase change through `previous_keys` and `retired_kek_fingerprints`. See [Resource options](core/server/README.md#resource-options) and [Stream encryption](core/server/README.md#stream-encryption). +Each node keeps a bounded log of the committed metadata changes, such as a stream created, a topic renamed, a user's permissions changed or a session closed. A client with `manage_servers` follows it with `metadata.changes` on every transport, `GET /metadata/changes` over HTTP or `iggy metadata changes`, and rebuilds from current state pages when its position falls out of the log. See [Metadata changes](core/server/README.md#metadata-changes). + For the detailed documentation of the configuration file, please refer to the [configuration](https://iggy.apache.org/docs/server/configuration) section. ---
diff --git a/bdd/cpp/features/step_definitions/metadata_changes_steps.cpp b/bdd/cpp/features/step_definitions/metadata_changes_steps.cpp new file mode 100644 index 0000000..967c41d --- /dev/null +++ b/bdd/cpp/features/step_definitions/metadata_changes_steps.cpp
@@ -0,0 +1,333 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// Unique step-class prefix so the generated CukeObject<n> symbols do not clash with the other +// step files when linked together (see background_steps.cpp for details). +#define CUKE_OBJECT_PREFIX IggyBddMetadataChanges + +#include <gtest/gtest.h> + +#include <cucumber-cpp/autodetect.hpp> + +#include <algorithm> +#include <chrono> +#include <cstddef> +#include <cstdint> +#include <map> +#include <optional> +#include <sstream> +#include <stdexcept> +#include <string> +#include <thread> +#include <variant> +#include <vector> + +#include "world.hpp" + +namespace { + +constexpr std::uint32_t kRootUserId = 0; +constexpr std::uint32_t kUnauthorized = 41; +constexpr int kFollowAttempts = 100; +constexpr auto kFollowInterval = std::chrono::milliseconds(100); +constexpr const char *kHiddenOptions[] = {"encryption_key", "encryption_key_wrapped", "critical_keys"}; + +// Walks the wire codes until to_string names the value, so no second name table is needed. +template <typename Code> +Code CodeNamed(const std::string &name) { + for (unsigned value = 0; value <= 255; ++value) { + try { + if (iggy::to_string(static_cast<Code>(value)) == name) { + return static_cast<Code>(value); + } + } catch (const std::invalid_argument &) { + } + } + throw std::invalid_argument("Unknown name " + name); +} + +template <typename Code> +std::vector<Code> CodesNamed(const std::string &names) { + std::vector<Code> codes; + std::stringstream stream(names); + std::string name; + while (std::getline(stream, name, ',')) { + codes.push_back(CodeNamed<Code>(name)); + } + return codes; +} + +iggy::MetadataChanges ReadChanges(bdd::GlobalContext &context, iggy::MetadataChangesQuery query) { + return bdd::sdk_client(context).GetMetadataChanges( + query.SetGroups(context.last_change_groups).SetLimit(iggy::MetadataChangesQuery::kMaxLimit)); +} + +std::optional<std::uint32_t> ChangeStreamId(const iggy::MetadataChangeRecord &record) { + if (const auto *stream = std::get_if<iggy::MetadataStreamChange>(&record.Change())) { + return stream->StreamId(); + } + if (const auto *stream = std::get_if<iggy::MetadataStreamDeleted>(&record.Change())) { + return stream->StreamId(); + } + if (const auto *topic = std::get_if<iggy::MetadataTopicChange>(&record.Change())) { + return topic->StreamId(); + } + if (const auto *topic = std::get_if<iggy::MetadataTopicDeleted>(&record.Change())) { + return topic->StreamId(); + } + if (const auto *partitions = std::get_if<iggy::MetadataPartitionsChanged>(&record.Change())) { + return partitions->StreamId(); + } + return std::nullopt; +} + +const iggy::MetadataChangeRecord &Followed(const bdd::GlobalContext &context, const std::string &kind) { + const auto wanted = CodeNamed<iggy::MetadataChangeKind>(kind); + for (const auto &record : context.last_changes) { + if (record.Kind() == wanted) { + return record; + } + } + throw std::runtime_error("No followed change of kind " + kind); +} + +const iggy::ResourceOptions &FollowedStreamOptions(const bdd::GlobalContext &context, const std::string &kind) { + const auto &stream = std::get<iggy::MetadataStreamChange>(Followed(context, kind).Change()); + if (!stream.Options()) { + throw std::runtime_error("Stream options must stay readable"); + } + return *stream.Options(); +} + +std::optional<std::string> OptionText(const iggy::ResourceOptions &options, const std::string &key) { + for (const auto *entries : {&options.Explicit(), &options.Derived()}) { + const auto entry = entries->find(key); + if (entry != entries->end()) { + const auto &bytes = entry->second.Value(); + return std::string(bytes.begin(), bytes.end()); + } + } + return std::nullopt; +} + +} // namespace + +GIVEN("^I hold the latest metadata change cursor for the groups \"([^\"]+)\"$") { + REGEX_PARAM(std::string, groups); + cucumber::ScenarioScope<bdd::GlobalContext> context; + + context->last_change_groups = CodesNamed<iggy::MetadataChangeGroup>(groups); + const auto page = ReadChanges(*context, iggy::MetadataChangesQuery::Latest()); + ASSERT_EQ(page.Status(), iggy::MetadataChangeStatus::Ok); + ASSERT_TRUE(page.Next().has_value()); + context->last_change_cursor = page.Next(); +} + +WHEN("^I delete the stream$") { + cucumber::ScenarioScope<bdd::GlobalContext> context; + + bdd::sdk_client(*context).DeleteStream(iggy::Identifier::Numeric(context->last_stream_id.value())); +} + +WHEN("^I create and delete ([0-9]+) streams named \"([^\"]+)\"$") { + REGEX_PARAM(int, count); + REGEX_PARAM(std::string, prefix); + cucumber::ScenarioScope<bdd::GlobalContext> context; + + auto &client = bdd::sdk_client(*context); + for (int index = 0; index < count; ++index) { + const auto stream = client.CreateStream(prefix + "-" + std::to_string(index)); + client.DeleteStream(iggy::Identifier::Numeric(stream.Id())); + } +} + +THEN("^following the metadata changes should return the kinds \"([^\"]+)\" for the stream$") { + REGEX_PARAM(std::string, kinds); + cucumber::ScenarioScope<bdd::GlobalContext> context; + + const auto expected = CodesNamed<iggy::MetadataChangeKind>(kinds); + const auto stream_id = context->last_stream_id.value(); + auto cursor = context->last_change_cursor.value(); + std::vector<iggy::MetadataChangeRecord> records; + // A two phase delete commits its record after the call returns. + for (int attempt = 0; attempt < kFollowAttempts; ++attempt) { + const auto page = ReadChanges(*context, iggy::MetadataChangesQuery::After(cursor)); + ASSERT_EQ(page.Status(), iggy::MetadataChangeStatus::Ok); + cursor = page.Next().value(); + records.insert(records.end(), page.Records().begin(), page.Records().end()); + if (page.HasMore()) { + continue; + } + const auto mine = std::count_if(records.begin(), records.end(), [stream_id](const auto &record) { + return ChangeStreamId(record) == stream_id; + }); + if (static_cast<std::size_t>(mine) >= expected.size()) { + break; + } + std::this_thread::sleep_for(kFollowInterval); + } + + context->last_changes.clear(); + std::vector<iggy::MetadataChangeKind> followed; + for (const auto &record : records) { + const auto group = record.Group(); + ASSERT_TRUE(group.has_value()); + EXPECT_NE(std::find(context->last_change_groups.begin(), context->last_change_groups.end(), *group), + context->last_change_groups.end()) + << "record of op " << record.Op() << " is outside the requested groups"; + if (ChangeStreamId(record) == stream_id) { + followed.push_back(record.Kind().value()); + context->last_changes.push_back(record); + } + } + EXPECT_EQ(followed, expected); +} + +THEN("^each followed change should name the root user as its actor$") { + cucumber::ScenarioScope<bdd::GlobalContext> context; + + for (const auto &record : context->last_changes) { + EXPECT_EQ(record.ActorUserId(), kRootUserId) << "record of op " << record.Op(); + } +} + +THEN("^the followed \"([^\"]+)\" change should carry a requested op before its own op$") { + REGEX_PARAM(std::string, kind); + cucumber::ScenarioScope<bdd::GlobalContext> context; + + const auto &record = Followed(*context, kind); + const auto &deleted = std::get<iggy::MetadataStreamDeleted>(record.Change()); + ASSERT_TRUE(deleted.RequestedOp().has_value()); + EXPECT_LT(*deleted.RequestedOp(), record.Op()); +} + +THEN("^the followed \"([^\"]+)\" change should report encryption key source \"([^\"]+)\"$") { + REGEX_PARAM(std::string, kind); + REGEX_PARAM(std::string, source); + cucumber::ScenarioScope<bdd::GlobalContext> context; + + EXPECT_EQ(OptionText(FollowedStreamOptions(*context, kind), "encryption_key_source"), source); +} + +THEN("^the followed \"([^\"]+)\" change should carry the stream key fingerprint$") { + REGEX_PARAM(std::string, kind); + cucumber::ScenarioScope<bdd::GlobalContext> context; + + ASSERT_TRUE(context->last_stream_key.has_value()); + EXPECT_EQ(OptionText(FollowedStreamOptions(*context, kind), "encryption_key_fingerprint"), + iggy::EncryptionKeyFingerprint(*context->last_stream_key)); +} + +THEN("^the followed changes should expose no encryption key material$") { + cucumber::ScenarioScope<bdd::GlobalContext> context; + + for (const auto &record : context->last_changes) { + const auto *stream = std::get_if<iggy::MetadataStreamChange>(&record.Change()); + if (stream == nullptr) { + continue; + } + ASSERT_TRUE(stream->Options().has_value()) << "stream options must stay readable"; + for (const auto *hidden : kHiddenOptions) { + EXPECT_EQ(stream->Options()->Explicit().count(hidden), 0u) << hidden << " must never be returned"; + EXPECT_EQ(stream->Options()->Derived().count(hidden), 0u) << hidden << " must never be returned"; + } + } +} + +THEN("^following the metadata changes should answer \"([^\"]+)\"$") { + REGEX_PARAM(std::string, status); + cucumber::ScenarioScope<bdd::GlobalContext> context; + + const auto expected = CodeNamed<iggy::MetadataChangeStatus>(status); + const auto page = ReadChanges(*context, iggy::MetadataChangesQuery::After(context->last_change_cursor.value())); + EXPECT_EQ(page.Status(), expected); + if (expected != iggy::MetadataChangeStatus::Ok) { + EXPECT_TRUE(page.Records().empty()); + EXPECT_FALSE(page.Next().has_value()); + } +} + +WHEN("^I read the current state for the groups \"([^\"]+)\"$") { + REGEX_PARAM(std::string, groups); + cucumber::ScenarioScope<bdd::GlobalContext> context; + + context->last_change_groups = CodesNamed<iggy::MetadataChangeGroup>(groups); + context->last_current_items.clear(); + auto query = iggy::MetadataChangesQuery::Current(); + while (true) { + const auto page = ReadChanges(*context, query); + ASSERT_EQ(page.Status(), iggy::MetadataChangeStatus::Ok); + context->last_current_items.insert(context->last_current_items.end(), page.Items().begin(), page.Items().end()); + if (!page.CurrentPageToken()) { + ASSERT_TRUE(page.Next().has_value()); + context->last_change_cursor = page.Next(); + break; + } + query = iggy::MetadataChangesQuery::Current(*page.CurrentPageToken()); + } +} + +THEN("^the current state should list the same streams as the stream list$") { + cucumber::ScenarioScope<bdd::GlobalContext> context; + + std::map<std::uint32_t, std::string> listed; + for (const auto &stream : bdd::sdk_client(*context).GetStreams()) { + listed.emplace(stream.Id(), stream.Name()); + } + std::map<std::uint32_t, std::string> projected; + for (const auto &item : context->last_current_items) { + if (const auto *stream = std::get_if<iggy::MetadataStreamChange>(&item.Value())) { + projected.emplace(stream->StreamId(), stream->Name()); + } + } + EXPECT_EQ(projected, listed); +} + +GIVEN("^a user named \"([^\"]+)\" with password \"([^\"]+)\" and no server permissions exists$") { + REGEX_PARAM(std::string, username); + REGEX_PARAM(std::string, password); + cucumber::ScenarioScope<bdd::GlobalContext> context; + + const auto user = bdd::sdk_client(*context).CreateUser( + username, password, iggy::UserStatus::Active, + iggy::Permissions().SetGlobal(iggy::GlobalPermissions().SetReadServers(true)), iggy::UserCreateOptions()); + context->last_user_id = user.Id(); +} + +WHEN("^I authenticate as \"([^\"]+)\" with password \"([^\"]+)\"$") { + REGEX_PARAM(std::string, username); + REGEX_PARAM(std::string, password); + cucumber::ScenarioScope<bdd::GlobalContext> context; + + auto &client = bdd::sdk_client(*context); + client.Logout(); + client.Login(username, password); +} + +THEN("^reading the metadata changes should be refused as unauthorized$") { + cucumber::ScenarioScope<bdd::GlobalContext> context; + + context->last_change_groups.clear(); + try { + ReadChanges(*context, iggy::MetadataChangesQuery::Earliest()); + FAIL() << "manage_servers is required"; + } catch (const iggy::IggyException &error) { + EXPECT_EQ(error.Code(), kUnauthorized) << error.what(); + } +}
diff --git a/bdd/cpp/features/step_definitions/resource_options_steps.cpp b/bdd/cpp/features/step_definitions/resource_options_steps.cpp index 598fab9..81ef226 100644 --- a/bdd/cpp/features/step_definitions/resource_options_steps.cpp +++ b/bdd/cpp/features/step_definitions/resource_options_steps.cpp
@@ -64,19 +64,6 @@ iggy::TopicOptionKey::PreallocateSegments, }; -iggy::IggyBlockingClient &Sdk(bdd::GlobalContext &context) { - if (!context.sdk) { - auto client = - iggy::IggyBlockingClient::Builder() - .WithServerAddress(bdd::required_env("IGGY_TCP_ADDRESS")) - .WithAutoLogin(bdd::required_env("IGGY_ROOT_USERNAME"), bdd::required_env("IGGY_ROOT_PASSWORD")) - .Build(); - client.Connect(); - context.sdk.emplace(std::move(client)); - } - return *context.sdk; -} - template <typename Key, std::size_t Count> Key KeyFromName(const Key (&keys)[Count], const std::string &name) { for (const auto key : keys) { @@ -106,15 +93,15 @@ } iggy::StreamDetails FetchStream(bdd::GlobalContext &context) { - return Sdk(context).GetStream(StreamIdentifier(context)); + return bdd::sdk_client(context).GetStream(StreamIdentifier(context)); } iggy::TopicDetails FetchTopic(bdd::GlobalContext &context) { - return Sdk(context).GetTopic(StreamIdentifier(context), TopicIdentifier(context)); + return bdd::sdk_client(context).GetTopic(StreamIdentifier(context), TopicIdentifier(context)); } iggy::UserInfoDetails FetchUser(bdd::GlobalContext &context) { - return Sdk(context).GetUser(iggy::Identifier::Numeric(context.last_user_id.value())); + return bdd::sdk_client(context).GetUser(iggy::Identifier::Numeric(context.last_user_id.value())); } void CreateStream(bdd::GlobalContext &context, @@ -124,13 +111,13 @@ if (key) { options.SetEncryptionKey(*key); } - const auto stream = Sdk(context).CreateStream(name, options); + const auto stream = bdd::sdk_client(context).CreateStream(name, options); context.last_stream_id = stream.Id(); context.last_stream_key = key; } void CreateTopic(bdd::GlobalContext &context, const std::string &name, iggy::TopicCreateOptions options) { - const auto topic = Sdk(context).CreateTopic(StreamIdentifier(context), name, options); + const auto topic = bdd::sdk_client(context).CreateTopic(StreamIdentifier(context), name, options); context.last_topic_id = topic.Id(); } @@ -172,7 +159,7 @@ } void UpdateTopic(bdd::GlobalContext &context, const iggy::TopicUpdateOptions &options) { - Sdk(context).UpdateTopic(StreamIdentifier(context), TopicIdentifier(context), options); + bdd::sdk_client(context).UpdateTopic(StreamIdentifier(context), TopicIdentifier(context), options); } } // namespace @@ -311,7 +298,7 @@ REGEX_PARAM(std::string, name); cucumber::ScenarioScope<bdd::GlobalContext> context; - Sdk(*context).UpdateStream(StreamIdentifier(*context), iggy::StreamUpdateOptions().SetName(name)); + bdd::sdk_client(*context).UpdateStream(StreamIdentifier(*context), iggy::StreamUpdateOptions().SetName(name)); } THEN("^the stream name should be \"([^\"]+)\"$") { @@ -327,7 +314,7 @@ const auto options = iggy::StreamUpdateOptions().Reset(KeyFromName(kStreamOptionKeys, key)); try { - Sdk(*context).UpdateStream(StreamIdentifier(*context), options); + bdd::sdk_client(*context).UpdateStream(StreamIdentifier(*context), options); context->last_update_rejected = false; } catch (const iggy::IggyException &) { context->last_update_rejected = true; @@ -343,7 +330,7 @@ WHEN("^I describe the stream options$") { cucumber::ScenarioScope<bdd::GlobalContext> context; - context->last_option_specs = Sdk(*context).DescribeOptions(iggy::OptionsScope::Stream); + context->last_option_specs = bdd::sdk_client(*context).DescribeOptions(iggy::OptionsScope::Stream); } THEN("^the stream option \"([^\"]+)\" should be updatable and not secret$") { @@ -446,8 +433,8 @@ REGEX_PARAM(std::string, password); cucumber::ScenarioScope<bdd::GlobalContext> context; - const auto user = - Sdk(*context).CreateUser(username, password, iggy::UserStatus::Active, std::nullopt, iggy::UserCreateOptions()); + const auto user = bdd::sdk_client(*context).CreateUser(username, password, iggy::UserStatus::Active, std::nullopt, + iggy::UserCreateOptions()); context->last_user_id = user.Id(); } @@ -456,8 +443,9 @@ REGEX_PARAM(std::string, status); cucumber::ScenarioScope<bdd::GlobalContext> context; - Sdk(*context).UpdateUser(iggy::Identifier::Numeric(context->last_user_id.value()), - iggy::UserUpdateOptions().SetUsername(username).SetStatus(UserStatusFromName(status))); + bdd::sdk_client(*context).UpdateUser( + iggy::Identifier::Numeric(context->last_user_id.value()), + iggy::UserUpdateOptions().SetUsername(username).SetStatus(UserStatusFromName(status))); } THEN("^the user should have username \"([^\"]+)\"$") {
diff --git a/bdd/cpp/features/step_definitions/world.hpp b/bdd/cpp/features/step_definitions/world.hpp index a102147..9ddd576 100644 --- a/bdd/cpp/features/step_definitions/world.hpp +++ b/bdd/cpp/features/step_definitions/world.hpp
@@ -24,6 +24,7 @@ #include <optional> #include <stdexcept> #include <string> +#include <utility> #include <vector> #include "iggy.hpp" @@ -51,7 +52,7 @@ PolledData polled; std::vector<std::uint8_t> raw_response; std::string raw_error; - // Public client for the option steps, connected on first use. + // Public client for the option and metadata change steps, see sdk_client(). std::optional<iggy::IggyBlockingClient> sdk; std::optional<std::uint32_t> last_stream_id; std::optional<std::uint32_t> last_topic_id; @@ -60,6 +61,10 @@ bool last_update_rejected = false; std::vector<iggy::OptionSpec> last_option_specs; std::string call_error; + std::vector<iggy::MetadataChangeGroup> last_change_groups; + std::optional<iggy::MetadataChangeCursor> last_change_cursor; + std::vector<iggy::MetadataChangeRecord> last_changes; + std::vector<iggy::MetadataCurrentItem> last_current_items; GlobalContext() = default; GlobalContext(const GlobalContext &) = delete; @@ -84,6 +89,19 @@ return std::string(value); } +// Public client for the option and metadata change steps, connected as root on first use. +inline iggy::IggyBlockingClient &sdk_client(GlobalContext &context) { + if (!context.sdk) { + auto client = iggy::IggyBlockingClient::Builder() + .WithServerAddress(required_env("IGGY_TCP_ADDRESS")) + .WithAutoLogin(required_env("IGGY_ROOT_USERNAME"), required_env("IGGY_ROOT_PASSWORD")) + .Build(); + client.Connect(); + context.sdk.emplace(std::move(client)); + } + return *context.sdk; +} + inline iggy::ffi::Identifier make_string_identifier(const std::string &value) { iggy::ffi::Identifier identifier; identifier.set_string(value);
diff --git a/bdd/cpp/scripts/entrypoint.sh b/bdd/cpp/scripts/entrypoint.sh index f04fcfd..0be4d59 100755 --- a/bdd/cpp/scripts/entrypoint.sh +++ b/bdd/cpp/scripts/entrypoint.sh
@@ -60,6 +60,7 @@ raw_command) tags=(--tags @raw-command) ;; stream_options) tags=(--tags @stream-options) ;; resource_options_patch) tags=(--tags @resource-options-patch) ;; + metadata_changes) tags=(--tags @metadata-changes) ;; session_contract) tags=(--tags @session-contract) ;; esac
diff --git a/bdd/docker-compose.coverage.yml b/bdd/docker-compose.coverage.yml index 1bdf761..68421a4 100644 --- a/bdd/docker-compose.coverage.yml +++ b/bdd/docker-compose.coverage.yml
@@ -39,7 +39,7 @@ trap 'cargo llvm-cov report --lcov --output-path /reports/rust-bdd-coverage.lcov' EXIT TEST_ARGS=() case "$$BDD_FEATURE" in - basic_messaging|leader_redirection|raw_command|session_contract|stream_crud|stream_options|resource_options_patch) TEST_ARGS=(--test "$$BDD_FEATURE") ;; + basic_messaging|leader_redirection|raw_command|session_contract|stream_crud|stream_options|resource_options_patch|metadata_changes) TEST_ARGS=(--test "$$BDD_FEATURE") ;; esac cargo test --locked -p bdd --features "$$BDD_RUST_FEATURES" "$${TEST_ARGS[@]}" @@ -105,6 +105,7 @@ leader_redirection) FILTER='--filter-trait Category=requires-leader-awareness' ;; stream_options) FILTER='--filter-trait Category=stream-options' ;; resource_options_patch) FILTER='--filter-trait Category=resource-options-patch' ;; + metadata_changes) FILTER='--filter-trait Category=metadata-changes' ;; session_contract) FILTER='--filter-trait Category=session-contract' ;; esac dotnet test $$FILTER --coverage --coverage-output-format cobertura --coverage-output csharp-bdd-coverage.cobertura.xml --results-directory /reports @@ -124,5 +125,6 @@ stream_crud) export CUCUMBER_FILTER_TAGS='@stream-crud' ;; stream_options) export CUCUMBER_FILTER_TAGS='@stream-options' ;; resource_options_patch) export CUCUMBER_FILTER_TAGS='@resource-options-patch' ;; + metadata_changes) export CUCUMBER_FILTER_TAGS='@metadata-changes' ;; esac gradle --no-daemon test jacocoTestReport && cp build/reports/jacoco/test/jacocoTestReport.xml /reports/java-bdd-coverage.xml
diff --git a/bdd/docker-compose.server.yml b/bdd/docker-compose.server.yml index 0f4c300..b9a8733 100644 --- a/bdd/docker-compose.server.yml +++ b/bdd/docker-compose.server.yml
@@ -73,6 +73,8 @@ # Every feature shares this server, and stream_options asserts server-generated stream keys. - IGGY_ENCRYPTION_ENABLED=true - IGGY_ENCRYPTION_KEY=/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE= + # Small enough for metadata_changes to expire a cursor with a few dozen streams. + - IGGY_METADATA_CHANGE_LOG_RESOURCE_CAPACITY=64 volumes: - iggy_data:/app/local_data - ../target:/profiles
diff --git a/bdd/docker-compose.yml b/bdd/docker-compose.yml index 681f689..3e3e582 100644 --- a/bdd/docker-compose.yml +++ b/bdd/docker-compose.yml
@@ -38,6 +38,7 @@ - ./scenarios/stream_crud.feature:/app/features/stream_crud.feature - ./scenarios/stream_options.feature:/app/features/stream_options.feature - ./scenarios/resource_options_patch.feature:/app/features/resource_options_patch.feature + - ./scenarios/metadata_changes.feature:/app/features/metadata_changes.feature command: - sh - -c @@ -50,6 +51,7 @@ stream_crud) cargo test -p bdd --features "$$BDD_RUST_FEATURES" --test stream_crud ;; stream_options) cargo test -p bdd --features "$$BDD_RUST_FEATURES" --test stream_options ;; resource_options_patch) cargo test -p bdd --features "$$BDD_RUST_FEATURES" --test resource_options_patch ;; + metadata_changes) cargo test -p bdd --features "$$BDD_RUST_FEATURES" --test metadata_changes ;; *) cargo test -p bdd --features "$$BDD_RUST_FEATURES" ;; esac networks: @@ -69,6 +71,7 @@ - ./scenarios/raw_command.feature:/app/features/raw_command.feature - ./scenarios/stream_options.feature:/app/features/stream_options.feature - ./scenarios/resource_options_patch.feature:/app/features/resource_options_patch.feature + - ./scenarios/metadata_changes.feature:/app/features/metadata_changes.feature - ./scenarios/session_contract.feature:/app/features/session_contract.feature working_dir: /app command: @@ -81,6 +84,7 @@ session_contract) TEST_PATH='tests/test_session_contract.py' ;; stream_options) TEST_PATH='tests/test_stream_options.py' ;; resource_options_patch) TEST_PATH='tests/test_resource_options_patch.py' ;; + metadata_changes) TEST_PATH='tests/test_metadata_changes.py' ;; *) TEST_PATH='tests/' ;; esac uv run --project /workspace/bdd/python pytest $$TEST_PATH -v @@ -100,6 +104,7 @@ - ./scenarios/raw_command.feature:/app/features/raw_command.feature - ./scenarios/stream_options.feature:/app/features/stream_options.feature - ./scenarios/session_contract.feature:/app/features/session_contract.feature + - ./scenarios/metadata_changes.feature:/app/features/metadata_changes.feature command: - sh - -c @@ -109,6 +114,7 @@ raw_command) GROUP='--group raw-command' ;; session_contract) GROUP='--group session-contract' ;; stream_options) GROUP='--group stream-options' ;; + metadata_changes) GROUP='--group metadata-changes' ;; *) GROUP='' ;; esac ./scripts/test.sh --configuration /workspace/bdd/php/phpunit.xml.dist $$GROUP @@ -148,6 +154,7 @@ session_contract) TAGS='--tags @session-contract' ;; stream_options) TAGS='--tags @stream-options' ;; resource_options_patch) TAGS='--tags @resource-options-patch' ;; + metadata_changes) TAGS='--tags @metadata-changes' ;; *) TAGS='' ;; esac npm run test:bdd -- $$TAGS @@ -174,6 +181,7 @@ raw_command) FILTER='--filter-trait Category=raw-command' ;; stream_options) FILTER='--filter-trait Category=stream-options' ;; resource_options_patch) FILTER='--filter-trait Category=resource-options-patch' ;; + metadata_changes) FILTER='--filter-trait Category=metadata-changes' ;; session_contract) FILTER='--filter-trait Category=session-contract' ;; esac dotnet test $$FILTER @@ -197,6 +205,7 @@ - ./scenarios/stream_crud.feature:/app/features/stream_crud.feature - ./scenarios/stream_options.feature:/app/features/stream_options.feature - ./scenarios/resource_options_patch.feature:/app/features/resource_options_patch.feature + - ./scenarios/metadata_changes.feature:/app/features/metadata_changes.feature command: - sh - -c @@ -209,6 +218,7 @@ stream_crud) export CUCUMBER_FILTER_TAGS='@stream-crud' ;; stream_options) export CUCUMBER_FILTER_TAGS='@stream-options' ;; resource_options_patch) export CUCUMBER_FILTER_TAGS='@resource-options-patch' ;; + metadata_changes) export CUCUMBER_FILTER_TAGS='@metadata-changes' ;; esac gradle --no-daemon test networks: @@ -228,6 +238,7 @@ - ./scenarios/raw_command.feature:/app/features/raw_command.feature - ./scenarios/stream_options.feature:/app/features/stream_options.feature - ./scenarios/resource_options_patch.feature:/app/features/resource_options_patch.feature + - ./scenarios/metadata_changes.feature:/app/features/metadata_changes.feature - ./scenarios/session_contract.feature:/app/features/session_contract.feature networks: - iggy-bdd-network
diff --git a/bdd/go/tests/metadata_changes.go b/bdd/go/tests/metadata_changes.go new file mode 100644 index 0000000..7a35c6d --- /dev/null +++ b/bdd/go/tests/metadata_changes.go
@@ -0,0 +1,471 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package tests + +import ( + "bytes" + "context" + "encoding/base64" + "encoding/hex" + "errors" + "fmt" + "maps" + "slices" + "strings" + "time" + + "github.com/apache/iggy/bdd/go/tests/env" + iggcon "github.com/apache/iggy/foreign/go/contracts" + ierror "github.com/apache/iggy/foreign/go/errors" + "github.com/cucumber/godog" +) + +const ( + // A two phase delete commits its record after the call returns. + metadataChangesFollowAttempts = 100 + metadataChangesFollowInterval = 100 * time.Millisecond + rootUserID = 0 +) + +type metadataChangesCtxKey struct{} + +type metadataChangesCtx struct { + groups []iggcon.MetadataChangeGroup + cursor *iggcon.MetadataChangeCursor + followed []iggcon.MetadataChangeRecord + currentItems []iggcon.MetadataCurrentItem + signedInAsNew bool +} + +func getMetadataChangesCtx(ctx context.Context) *metadataChangesCtx { + return ctx.Value(metadataChangesCtxKey{}).(*metadataChangesCtx) +} + +type metadataChangesSteps struct{} + +func (s metadataChangesSteps) givenLatestCursor(ctx context.Context, groups string) error { + state := getMetadataChangesCtx(ctx) + parsed, err := parseMetadataChangeGroups(groups) + if err != nil { + return err + } + state.groups = parsed + changes, err := s.read(ctx, iggcon.LatestMetadataChangeStart()) + if err != nil { + return fmt.Errorf("read the latest cursor: %w", err) + } + if changes.Status != iggcon.MetadataChangeStatusOk || changes.Next == nil { + return fmt.Errorf("latest answered %s with cursor %v", changes.Status, changes.Next) + } + state.cursor = changes.Next + return nil +} + +func (metadataChangesSteps) whenDeleteStream(ctx context.Context) error { + options := getResourceOptionsCtx(ctx) + if err := options.client.DeleteStream(ctx, streamIdentifier(options)); err != nil { + return fmt.Errorf("delete stream: %w", err) + } + options.createdStreams = slices.DeleteFunc(options.createdStreams, func(id uint32) bool { return id == *options.streamId }) + return nil +} + +func (metadataChangesSteps) whenChurnStreams(ctx context.Context, count int, prefix string) error { + client := getResourceOptionsCtx(ctx).client + for index := range count { + stream, err := client.CreateStream(ctx, fmt.Sprintf("%s-%d", prefix, index), nil) + if err != nil { + return fmt.Errorf("create stream %d: %w", index, err) + } + streamId, _ := iggcon.NewIdentifier(stream.Id) + if err := client.DeleteStream(ctx, streamId); err != nil { + return fmt.Errorf("delete stream %d: %w", index, err) + } + } + return nil +} + +func (s metadataChangesSteps) thenFollowedKinds(ctx context.Context, kinds string) error { + state := getMetadataChangesCtx(ctx) + streamKey, err := s.createdStreamKey(ctx) + if err != nil { + return err + } + var want []iggcon.MetadataChangeKind + for _, name := range strings.Split(kinds, ",") { + kind, err := iggcon.ParseMetadataChangeKind(strings.TrimSpace(name)) + if err != nil { + return err + } + want = append(want, kind) + } + cursor := *state.cursor + var records []iggcon.MetadataChangeRecord + var mine []iggcon.MetadataChangeRecord + for range metadataChangesFollowAttempts { + changes, err := s.read(ctx, iggcon.AfterMetadataChangeStart(cursor)) + if err != nil { + return fmt.Errorf("follow changes: %w", err) + } + if changes.Status != iggcon.MetadataChangeStatusOk || changes.Next == nil { + return fmt.Errorf("follow answered %s with cursor %v", changes.Status, changes.Next) + } + cursor = *changes.Next + records = append(records, changes.Records...) + if changes.HasMore { + continue + } + mine = slices.DeleteFunc(slices.Clone(records), func(record iggcon.MetadataChangeRecord) bool { + key, ok := changeStreamKey(record) + return !ok || key != streamKey + }) + if len(mine) >= len(want) { + break + } + time.Sleep(metadataChangesFollowInterval) + } + for _, record := range records { + if len(state.groups) > 0 && !slices.Contains(state.groups, record.Group) { + return fmt.Errorf("record of group %s outside the requested %v", record.Group, state.groups) + } + } + got := make([]iggcon.MetadataChangeKind, 0, len(mine)) + for _, record := range mine { + got = append(got, record.Kind) + } + if !slices.Equal(got, want) { + return fmt.Errorf("followed kinds %v, want %v", got, want) + } + state.followed = mine + return nil +} + +func (metadataChangesSteps) thenRootActor(ctx context.Context) error { + for _, record := range getMetadataChangesCtx(ctx).followed { + if record.ActorUserID == nil || *record.ActorUserID != rootUserID { + return fmt.Errorf("op %d names actor %v, want the root user", record.Op, record.ActorUserID) + } + } + return nil +} + +func (s metadataChangesSteps) thenRequestedOpBeforeOwnOp(ctx context.Context, kind string) error { + record, err := s.followedRecord(ctx, kind) + if err != nil { + return err + } + deleted, ok := record.Body.(*iggcon.MetadataStreamDeleted) + if !ok { + return fmt.Errorf("expected a stream deletion, got %T", record.Body) + } + if deleted.RequestedOp == nil || *deleted.RequestedOp >= record.Op { + return fmt.Errorf("requested op %v is not before op %d", deleted.RequestedOp, record.Op) + } + return nil +} + +func (s metadataChangesSteps) thenChangeKeySource(ctx context.Context, kind, source string) error { + stream, err := s.followedStreamOptions(ctx, kind) + if err != nil { + return err + } + if got, ok := stream.EncryptionKeySource(); !ok || got != iggcon.StreamEncryptionSource(source) { + return fmt.Errorf("encryption key source is %q (reported %t), want %q", got, ok, source) + } + return nil +} + +func (s metadataChangesSteps) thenChangeFingerprint(ctx context.Context, kind string) error { + stream, err := s.followedStreamOptions(ctx, kind) + if err != nil { + return err + } + want, err := iggcon.EncryptionKeyFingerprint(getResourceOptionsCtx(ctx).streamKey) + if err != nil { + return fmt.Errorf("fingerprint the stream key: %w", err) + } + if got, ok := stream.EncryptionKeyFingerprint(); !ok || got != want { + return fmt.Errorf("encryption key fingerprint is %q (reported %t), want %q", got, ok, want) + } + return nil +} + +func (metadataChangesSteps) thenChangesHideKeys(ctx context.Context) error { + key := getResourceOptionsCtx(ctx).streamKey + for _, record := range getMetadataChangesCtx(ctx).followed { + stream, ok := record.Body.(*iggcon.MetadataStreamChange) + if !ok { + continue + } + if stream.Options.Unreadable { + return fmt.Errorf("op %d options must stay readable", record.Op) + } + for _, options := range []map[string]iggcon.HeaderValue{stream.Options.Options, stream.Options.DerivedOptions} { + for _, hidden := range []string{string(iggcon.StreamOptionEncryptionKey), "encryption_key_wrapped", "critical_keys"} { + if _, found := options[hidden]; found { + return fmt.Errorf("op %d returns %s", record.Op, hidden) + } + } + for name, value := range options { + if len(key) > 0 && (bytes.Contains(value.Value, key) || + bytes.Contains(value.Value, []byte(base64.StdEncoding.EncodeToString(key))) || + bytes.Contains(value.Value, []byte(hex.EncodeToString(key)))) { + return fmt.Errorf("op %d option %s carries the stream key", record.Op, name) + } + } + } + } + return nil +} + +func (s metadataChangesSteps) thenFollowStatus(ctx context.Context, status string) error { + state := getMetadataChangesCtx(ctx) + changes, err := s.read(ctx, iggcon.AfterMetadataChangeStart(*state.cursor)) + if err != nil { + return fmt.Errorf("follow changes: %w", err) + } + if changes.Status.String() != status { + return fmt.Errorf("follow answered %s, want %s", changes.Status, status) + } + if changes.Status != iggcon.MetadataChangeStatusOk && (len(changes.Records) != 0 || changes.Next != nil) { + return fmt.Errorf("a %s answer carries %d records and cursor %v", changes.Status, len(changes.Records), changes.Next) + } + return nil +} + +func (s metadataChangesSteps) whenReadCurrentState(ctx context.Context, groups string) error { + state := getMetadataChangesCtx(ctx) + parsed, err := parseMetadataChangeGroups(groups) + if err != nil { + return err + } + state.groups = parsed + state.currentItems = nil + var token *iggcon.MetadataCurrentPageToken + for { + changes, err := s.read(ctx, iggcon.CurrentMetadataChangeStart(token)) + if err != nil { + return fmt.Errorf("read current state: %w", err) + } + if changes.Status != iggcon.MetadataChangeStatusOk { + return fmt.Errorf("current state answered %s", changes.Status) + } + state.currentItems = append(state.currentItems, changes.Items...) + if changes.CurrentPageToken == nil { + if changes.Next == nil { + return errors.New("the last current state page returned no anchor") + } + state.cursor = changes.Next + return nil + } + token = changes.CurrentPageToken + } +} + +func (metadataChangesSteps) thenCurrentMatchesList(ctx context.Context) error { + listed, err := getResourceOptionsCtx(ctx).client.GetStreams(ctx) + if err != nil { + return fmt.Errorf("list streams: %w", err) + } + want := make(map[uint32]string, len(listed)) + for _, stream := range listed { + want[stream.Id] = stream.Name + } + got := make(map[uint32]string) + for _, item := range getMetadataChangesCtx(ctx).currentItems { + if stream, ok := item.Body.(*iggcon.MetadataStreamChange); ok { + got[stream.StreamID] = stream.Name + } + } + if !maps.Equal(got, want) { + return fmt.Errorf("current state lists %v, the stream list %v", got, want) + } + return nil +} + +func (metadataChangesSteps) givenUserWithoutServerPermissions(ctx context.Context, username, password string) error { + options := getResourceOptionsCtx(ctx) + permissions := &iggcon.Permissions{Global: iggcon.GlobalPermissions{ReadServers: true}} + user, err := options.client.CreateUser(ctx, username, password, iggcon.Active, permissions, nil) + if err != nil { + return fmt.Errorf("create user %s: %w", username, err) + } + options.userId = &user.Id + options.createdUsers = append(options.createdUsers, user.Id) + return nil +} + +func (metadataChangesSteps) whenAuthenticateAs(ctx context.Context, username, password string) error { + client := getResourceOptionsCtx(ctx).client + if err := client.LogoutUser(ctx); err != nil { + return fmt.Errorf("logout: %w", err) + } + getMetadataChangesCtx(ctx).signedInAsNew = true + if _, err := client.LoginUser(ctx, username, password); err != nil { + return fmt.Errorf("login as %s: %w", username, err) + } + return nil +} + +func (s metadataChangesSteps) thenChangesRefused(ctx context.Context) error { + getMetadataChangesCtx(ctx).groups = nil + _, err := s.read(ctx, iggcon.EarliestMetadataChangeStart()) + if !errors.Is(err, ierror.ErrUnauthorized) { + return fmt.Errorf("reading changes returned %v, want unauthorized", err) + } + return nil +} + +func (metadataChangesSteps) read(ctx context.Context, start iggcon.MetadataChangeStart) (*iggcon.MetadataChanges, error) { + query := iggcon.NewMetadataChangesQuery(start, getMetadataChangesCtx(ctx).groups...) + query.Limit = iggcon.MaxMetadataChangesLimit + return getResourceOptionsCtx(ctx).client.GetMetadataChanges(ctx, query) +} + +// createdStreamKey reads the key of the scenario's stream from its creation +// record. Ids are reused after a delete, so a stream id alone can match a +// stream another test created and deleted meanwhile. +func (metadataChangesSteps) createdStreamKey(ctx context.Context) (iggcon.MetadataStreamKey, error) { + options := getResourceOptionsCtx(ctx) + query := iggcon.NewMetadataChangesQuery( + iggcon.AfterMetadataChangeStart(*getMetadataChangesCtx(ctx).cursor), + iggcon.MetadataChangeGroupStreams, + ) + query.Limit = iggcon.MaxMetadataChangesLimit + for { + changes, err := options.client.GetMetadataChanges(ctx, query) + if err != nil { + return iggcon.MetadataStreamKey{}, fmt.Errorf("read stream changes: %w", err) + } + if changes.Status != iggcon.MetadataChangeStatusOk || changes.Next == nil { + return iggcon.MetadataStreamKey{}, fmt.Errorf("stream changes answered %s with cursor %v", changes.Status, changes.Next) + } + for _, record := range changes.Records { + created, ok := record.Body.(*iggcon.MetadataStreamChange) + if ok && record.Kind == iggcon.MetadataChangeKindStreamCreated && + created.StreamID == *options.streamId && created.Name == options.streamName { + return created.MetadataStreamKey, nil + } + } + if !changes.HasMore { + return iggcon.MetadataStreamKey{}, fmt.Errorf("no stream_created change for stream %d %q", *options.streamId, options.streamName) + } + query.Start = iggcon.AfterMetadataChangeStart(*changes.Next) + } +} + +func (metadataChangesSteps) followedRecord(ctx context.Context, kind string) (iggcon.MetadataChangeRecord, error) { + want, err := iggcon.ParseMetadataChangeKind(kind) + if err != nil { + return iggcon.MetadataChangeRecord{}, err + } + for _, record := range getMetadataChangesCtx(ctx).followed { + if record.Kind == want { + return record, nil + } + } + return iggcon.MetadataChangeRecord{}, fmt.Errorf("no followed %s change", kind) +} + +// followedStreamOptions views a followed stream change's options as a stream, +// so the stream's projection helpers read them. +func (s metadataChangesSteps) followedStreamOptions(ctx context.Context, kind string) (iggcon.Stream, error) { + record, err := s.followedRecord(ctx, kind) + if err != nil { + return iggcon.Stream{}, err + } + stream, ok := record.Body.(*iggcon.MetadataStreamChange) + if !ok { + return iggcon.Stream{}, fmt.Errorf("expected a stream change, got %T", record.Body) + } + if stream.Options.Unreadable { + return iggcon.Stream{}, errors.New("stream options must stay readable") + } + return iggcon.Stream{Options: stream.Options.Options, DerivedOptions: stream.Options.DerivedOptions}, nil +} + +func initMetadataChangesScenario(sc *godog.ScenarioContext) { + sc.Before(func(context.Context, *godog.Scenario) (context.Context, error) { + ctx := context.WithValue(context.Background(), resourceOptionsCtxKey{}, &resourceOptionsCtx{}) + return context.WithValue(ctx, metadataChangesCtxKey{}, &metadataChangesCtx{}), nil + }) + options := resourceOptionsSteps{} + s := metadataChangesSteps{} + sc.Step(`^I have a running Iggy server$`, options.givenRunningServer) + sc.Step(`^I am authenticated as the root user$`, options.givenAuthenticationAsRoot) + sc.Step(`^I create a stream named "([^"]+)" without an encryption key$`, options.whenCreateKeylessStream) + sc.Step(`^I create a stream named "([^"]+)" with an encryption key of 32 bytes valued (\d+)$`, options.whenCreateKeyedStream) + sc.Step(`^I rename the stream to "([^"]+)"$`, options.whenRenameStream) + sc.Step(`^I create a topic named "([^"]+)" with (\d+) partitions? in the stream$`, options.whenCreateTopic) + sc.Step(`^I hold the latest metadata change cursor for the groups "([^"]+)"$`, s.givenLatestCursor) + sc.Step(`^I delete the stream$`, s.whenDeleteStream) + sc.Step(`^I create and delete (\d+) streams named "([^"]+)"$`, s.whenChurnStreams) + sc.Step(`^following the metadata changes should return the kinds "([^"]+)" for the stream$`, s.thenFollowedKinds) + sc.Step(`^each followed change should name the root user as its actor$`, s.thenRootActor) + sc.Step(`^the followed "([^"]+)" change should carry a requested op before its own op$`, s.thenRequestedOpBeforeOwnOp) + sc.Step(`^the followed "([^"]+)" change should report encryption key source "([^"]+)"$`, s.thenChangeKeySource) + sc.Step(`^the followed "([^"]+)" change should carry the stream key fingerprint$`, s.thenChangeFingerprint) + sc.Step(`^the followed changes should expose no encryption key material$`, s.thenChangesHideKeys) + sc.Step(`^following the metadata changes should answer "([^"]+)"$`, s.thenFollowStatus) + sc.Step(`^I read the current state for the groups "([^"]+)"$`, s.whenReadCurrentState) + sc.Step(`^the current state should list the same streams as the stream list$`, s.thenCurrentMatchesList) + sc.Step(`^a user named "([^"]+)" with password "([^"]+)" and no server permissions exists$`, s.givenUserWithoutServerPermissions) + sc.Step(`^I authenticate as "([^"]+)" with password "([^"]+)"$`, s.whenAuthenticateAs) + sc.Step(`^reading the metadata changes should be refused as unauthorized$`, s.thenChangesRefused) + sc.After(func(ctx context.Context, scenario *godog.Scenario, scenarioError error) (context.Context, error) { + options := getResourceOptionsCtx(ctx) + if options.client != nil && getMetadataChangesCtx(ctx).signedInAsNew { + username, password := env.RootCredentials() + logoutError := options.client.LogoutUser(ctx) + _, loginError := options.client.LoginUser(ctx, username, password) + scenarioError = errors.Join(scenarioError, logoutError, loginError) + } + return resourceOptionsCleanup(ctx, scenario, scenarioError) + }) +} + +func parseMetadataChangeGroups(names string) ([]iggcon.MetadataChangeGroup, error) { + var groups []iggcon.MetadataChangeGroup + for _, name := range strings.Split(names, ",") { + group, err := iggcon.ParseMetadataChangeGroup(strings.TrimSpace(name)) + if err != nil { + return nil, err + } + groups = append(groups, group) + } + return groups, nil +} + +// changeStreamKey names the stream a stream, topic or partitions record +// belongs to. +func changeStreamKey(record iggcon.MetadataChangeRecord) (iggcon.MetadataStreamKey, bool) { + switch body := record.Body.(type) { + case *iggcon.MetadataStreamChange: + return body.MetadataStreamKey, true + case *iggcon.MetadataStreamDeleted: + return body.MetadataStreamKey, true + case *iggcon.MetadataTopicChange: + return body.MetadataStreamKey, true + case *iggcon.MetadataTopicDeleted: + return body.MetadataStreamKey, true + case *iggcon.MetadataPartitionsChanged: + return body.MetadataStreamKey, true + default: + return iggcon.MetadataStreamKey{}, false + } +}
diff --git a/bdd/go/tests/resource_options.go b/bdd/go/tests/resource_options.go index 9a250ba..db0c62f 100644 --- a/bdd/go/tests/resource_options.go +++ b/bdd/go/tests/resource_options.go
@@ -42,6 +42,7 @@ type resourceOptionsCtx struct { client iggcon.Client streamId *uint32 + streamName string streamKey []byte topicId *uint32 userId *uint32 @@ -100,6 +101,7 @@ return fmt.Errorf("create stream %s: %w", name, err) } state.streamId = &stream.Id + state.streamName = name state.createdStreams = append(state.createdStreams, stream.Id) return nil } @@ -563,21 +565,25 @@ sc.Step(`^I update the user username to "([^"]+)" and status to "([^"]+)"$`, s.whenUpdateUser) sc.Step(`^the user should have username "([^"]+)"$`, s.thenUserUsername) sc.Step(`^the user should have status "([^"]+)"$`, s.thenUserStatus) - sc.After(func(ctx context.Context, _ *godog.Scenario, scenarioError error) (context.Context, error) { - state := getResourceOptionsCtx(ctx) - if state.client == nil { - return ctx, scenarioError - } - for _, id := range state.createdStreams { - streamId, err := iggcon.NewIdentifier(id) - scenarioError = errors.Join(scenarioError, err, state.client.DeleteStream(ctx, streamId)) - } - for _, id := range state.createdUsers { - userId, err := iggcon.NewIdentifier(id) - scenarioError = errors.Join(scenarioError, err, state.client.DeleteUser(ctx, userId)) - } - return ctx, errors.Join(scenarioError, state.client.Close()) - }) + sc.After(resourceOptionsCleanup) +} + +// resourceOptionsCleanup deletes the streams and users a scenario created and +// closes its client. +func resourceOptionsCleanup(ctx context.Context, _ *godog.Scenario, scenarioError error) (context.Context, error) { + state := getResourceOptionsCtx(ctx) + if state.client == nil { + return ctx, scenarioError + } + for _, id := range state.createdStreams { + streamId, err := iggcon.NewIdentifier(id) + scenarioError = errors.Join(scenarioError, err, state.client.DeleteStream(ctx, streamId)) + } + for _, id := range state.createdUsers { + userId, err := iggcon.NewIdentifier(id) + scenarioError = errors.Join(scenarioError, err, state.client.DeleteUser(ctx, userId)) + } + return ctx, errors.Join(scenarioError, state.client.Close()) } func streamIdentifier(state *resourceOptionsCtx) iggcon.Identifier {
diff --git a/bdd/go/tests/suite_test.go b/bdd/go/tests/suite_test.go index 3adf9c1..ddbff00 100644 --- a/bdd/go/tests/suite_test.go +++ b/bdd/go/tests/suite_test.go
@@ -99,6 +99,18 @@ }) } + if feature == "all" || feature == "metadata_changes" { + suites = append(suites, godog.TestSuite{ + ScenarioInitializer: initMetadataChangesScenario, + Options: &godog.Options{ + Format: "pretty", + Paths: []string{"../../scenarios/metadata_changes.feature"}, + Strict: true, + TestingT: t, + }, + }) + } + if len(suites) == 0 { t.Fatalf("unknown BDD_FEATURE=%q", feature) }
diff --git a/bdd/java/src/test/java/org/apache/iggy/bdd/MetadataChangesSteps.java b/bdd/java/src/test/java/org/apache/iggy/bdd/MetadataChangesSteps.java new file mode 100644 index 0000000..6c2db11 --- /dev/null +++ b/bdd/java/src/test/java/org/apache/iggy/bdd/MetadataChangesSteps.java
@@ -0,0 +1,302 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.bdd; + +import io.cucumber.java.After; +import io.cucumber.java.en.Given; +import io.cucumber.java.en.Then; +import io.cucumber.java.en.When; +import org.apache.iggy.client.blocking.tcp.IggyTcpClient; +import org.apache.iggy.exception.IggyErrorCode; +import org.apache.iggy.exception.IggyServerException; +import org.apache.iggy.stream.EncryptionKeySource; +import org.apache.iggy.stream.EncryptionKeys; +import org.apache.iggy.stream.StreamBase; +import org.apache.iggy.system.MetadataChangeBody; +import org.apache.iggy.system.MetadataChangeCursor; +import org.apache.iggy.system.MetadataChangeGroup; +import org.apache.iggy.system.MetadataChangeKind; +import org.apache.iggy.system.MetadataChangeRecord; +import org.apache.iggy.system.MetadataChangeStart; +import org.apache.iggy.system.MetadataChangeStatus; +import org.apache.iggy.system.MetadataChanges; +import org.apache.iggy.system.MetadataChangesQuery; +import org.apache.iggy.system.MetadataCurrentItem; +import org.apache.iggy.user.GlobalPermissions; +import org.apache.iggy.user.Permissions; +import org.apache.iggy.user.UserOptions; +import org.apache.iggy.user.UserStatus; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.stream.Collectors; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +public final class MetadataChangesSteps { + + private static final long ROOT_USER_ID = 0; + private static final int FOLLOW_ATTEMPTS = 100; + private static final long FOLLOW_INTERVAL_MILLIS = 100; + private static final List<String> HIDDEN_OPTIONS = + List.of("encryption_key", "encryption_key_wrapped", "critical_keys"); + + private IggyTcpClient client; + private List<MetadataChangeGroup> groups = List.of(); + private MetadataChangeCursor cursor; + private List<MetadataChangeRecord> followed = List.of(); + private List<MetadataCurrentItem> items = List.of(); + private Long userId; + + // The client may be signed in as a user without manage users by now, so root signs in again to remove it. + @After + public void close() { + if (client == null) { + return; + } + if (userId != null) { + client.users().logout(); + client.users().login(TestEnvironment.rootUsername(), TestEnvironment.rootPassword()); + client.users().deleteUser(userId); + } + client.close(); + } + + @Given("I hold the latest metadata change cursor for the groups {string}") + public void holdLatestCursor(String names) { + groups = parseGroups(names); + var page = read(MetadataChangeStart.latest()); + assertEquals(MetadataChangeStatus.OK, page.status()); + cursor = page.next().orElseThrow(); + } + + @When("I create and delete {int} streams named {string}") + public void churnStreams(int count, String prefix) { + for (int index = 0; index < count; index++) { + var stream = client().streams().createStream(prefix + "-" + index); + client().streams().deleteStream(stream.id()); + } + } + + @Then("following the metadata changes should return the kinds {string} for the stream") + public void followedKinds(String kinds) throws InterruptedException { + List<MetadataChangeKind> expected = Arrays.stream(kinds.split(",")) + .map(name -> MetadataChangeKind.fromValue(name.trim()).orElseThrow()) + .toList(); + long streamId = ResourceOptionsSteps.current().createdStreamId(); + List<MetadataChangeRecord> records = new ArrayList<>(); + var position = cursor; + // A two phase delete commits its record after the call returns. + for (int attempt = 0; attempt < FOLLOW_ATTEMPTS; attempt++) { + var page = read(MetadataChangeStart.after(position)); + assertEquals(MetadataChangeStatus.OK, page.status()); + position = page.next().orElseThrow(); + records.addAll(page.records()); + if (page.hasMore()) { + continue; + } + if (records.stream().filter(record -> namesStream(record, streamId)).count() >= expected.size()) { + break; + } + Thread.sleep(FOLLOW_INTERVAL_MILLIS); + } + for (MetadataChangeRecord record : records) { + assertTrue(groups.contains(record.group().orElseThrow()), record.toString()); + } + followed = + records.stream().filter(record -> namesStream(record, streamId)).toList(); + assertEquals( + expected, + followed.stream().map(record -> record.kind().orElseThrow()).toList()); + } + + @Then("each followed change should name the root user as its actor") + public void rootActor() { + for (MetadataChangeRecord record : followed) { + assertEquals(Optional.of(ROOT_USER_ID), record.actorUserId(), record.toString()); + } + } + + @Then("the followed {string} change should carry a requested op before its own op") + public void requestedOpBeforeOwnOp(String kind) { + var record = followed(kind); + var deleted = assertInstanceOf(MetadataChangeBody.StreamDeleted.class, record.body()); + assertTrue(deleted.requestedOp().orElseThrow().compareTo(record.op()) < 0, record.toString()); + } + + @Then("the followed {string} change should report encryption key source {string}") + public void keySource(String kind, String source) { + assertEquals( + EncryptionKeySource.fromValue(source).orElseThrow(), + followedStream(kind).encryptionKeySource().orElseThrow()); + } + + @Then("the followed {string} change should carry the stream key fingerprint") + public void keyFingerprint(String kind) { + assertEquals( + EncryptionKeys.fingerprint(ResourceOptionsSteps.current().streamKey()), + followedStream(kind).encryptionKeyFingerprint().orElseThrow()); + } + + @Then("the followed changes should expose no encryption key material") + public void noKeyMaterial() { + for (MetadataChangeRecord record : followed) { + if (record.body() instanceof MetadataChangeBody.StreamChange stream) { + assertFalse(stream.optionsUnreadable(), record.toString()); + for (String hidden : HIDDEN_OPTIONS) { + assertFalse(stream.options().containsKey(hidden), hidden); + assertFalse(stream.derivedOptions().containsKey(hidden), hidden); + } + } + } + } + + @Then("following the metadata changes should answer {string}") + public void followStatus(String status) { + var expected = MetadataChangeStatus.fromValue(status).orElseThrow(); + var page = read(MetadataChangeStart.after(cursor)); + assertEquals(expected, page.status()); + if (expected != MetadataChangeStatus.OK) { + assertTrue(page.records().isEmpty()); + assertTrue(page.next().isEmpty()); + } + } + + @When("I read the current state for the groups {string}") + public void readCurrentState(String names) { + groups = parseGroups(names); + List<MetadataCurrentItem> scanned = new ArrayList<>(); + var start = MetadataChangeStart.current(); + while (true) { + var page = read(start); + assertEquals(MetadataChangeStatus.OK, page.status()); + scanned.addAll(page.items()); + if (page.currentPageToken().isEmpty()) { + cursor = page.next().orElseThrow(); + break; + } + start = MetadataChangeStart.current(page.currentPageToken().orElseThrow()); + } + items = scanned; + } + + @Then("the current state should list the same streams as the stream list") + public void currentStateMatchesList() { + Map<Long, String> listed = + client().streams().getStreams().stream().collect(Collectors.toMap(StreamBase::id, StreamBase::name)); + Map<Long, String> projected = items.stream() + .map(MetadataCurrentItem::body) + .filter(MetadataChangeBody.StreamChange.class::isInstance) + .map(MetadataChangeBody.StreamChange.class::cast) + .collect(Collectors.toMap( + MetadataChangeBody.StreamChange::streamId, MetadataChangeBody.StreamChange::name)); + assertEquals(listed, projected); + } + + @Given("a user named {string} with password {string} and no server permissions exists") + public void userWithoutServerPermissions(String username, String password) { + var permissions = new Permissions( + new GlobalPermissions(false, true, false, false, false, false, false, false, false, false), Map.of()); + var user = client().users() + .createUser(username, password, UserStatus.Active, Optional.of(permissions), UserOptions.empty()); + userId = user.id(); + } + + @When("I authenticate as {string} with password {string}") + public void authenticateAs(String username, String password) { + client().users().logout(); + assertNotNull(client().users().login(username, password)); + } + + @Then("reading the metadata changes should be refused as unauthorized") + public void readingRefused() { + groups = List.of(); + var refusal = assertThrows(IggyServerException.class, () -> read(MetadataChangeStart.earliest())); + assertEquals(IggyErrorCode.UNAUTHORIZED, refusal.getErrorCode()); + } + + private MetadataChanges read(MetadataChangeStart start) { + return client().system() + .getMetadataChanges(MetadataChangesQuery.builder() + .start(start) + .groups(groups) + .limit(MetadataChangesQuery.MAX_LIMIT) + .build()); + } + + private MetadataChangeRecord followed(String kind) { + var parsed = MetadataChangeKind.fromValue(kind).orElseThrow(); + return followed.stream() + .filter(record -> record.kind().equals(Optional.of(parsed))) + .findFirst() + .orElseThrow(); + } + + private MetadataChangeBody.StreamChange followedStream(String kind) { + var stream = assertInstanceOf( + MetadataChangeBody.StreamChange.class, followed(kind).body()); + assertFalse(stream.optionsUnreadable()); + return stream; + } + + private IggyTcpClient client() { + if (client == null) { + String[] address = TestEnvironment.serverAddress().split(":", 2); + client = IggyTcpClient.builder() + .host(address[0]) + .port(Integer.parseInt(address[1])) + .build(); + client.connect(); + client.users().login(TestEnvironment.rootUsername(), TestEnvironment.rootPassword()); + } + return client; + } + + private static List<MetadataChangeGroup> parseGroups(String names) { + return Arrays.stream(names.split(",")) + .map(name -> MetadataChangeGroup.fromValue(name.trim()).orElseThrow()) + .toList(); + } + + private static boolean namesStream(MetadataChangeRecord record, long streamId) { + var body = record.body(); + if (body instanceof MetadataChangeBody.StreamChange stream) { + return stream.streamId() == streamId; + } + if (body instanceof MetadataChangeBody.StreamDeleted stream) { + return stream.streamId() == streamId; + } + if (body instanceof MetadataChangeBody.TopicChange topic) { + return topic.streamId() == streamId; + } + if (body instanceof MetadataChangeBody.TopicDeleted topic) { + return topic.streamId() == streamId; + } + return body instanceof MetadataChangeBody.PartitionsChanged partitions && partitions.streamId() == streamId; + } +}
diff --git a/bdd/java/src/test/java/org/apache/iggy/bdd/ResourceOptionsSteps.java b/bdd/java/src/test/java/org/apache/iggy/bdd/ResourceOptionsSteps.java index c7501d9..de9e70e 100644 --- a/bdd/java/src/test/java/org/apache/iggy/bdd/ResourceOptionsSteps.java +++ b/bdd/java/src/test/java/org/apache/iggy/bdd/ResourceOptionsSteps.java
@@ -20,6 +20,7 @@ package org.apache.iggy.bdd; import io.cucumber.java.After; +import io.cucumber.java.Before; import io.cucumber.java.en.Given; import io.cucumber.java.en.Then; import io.cucumber.java.en.When; @@ -63,14 +64,23 @@ public final class ResourceOptionsSteps { + // Cucumber builds glue classes without injection, so other steps read this scenario's stream through it. + private static ResourceOptionsSteps current; + private IggyTcpClient client; private Long streamId; + private Long createdStreamId; private Long topicId; private Long userId; private String streamName; private byte[] streamKey; private boolean updateRejected; + @Before + public void begin() { + current = this; + } + @After public void close() { if (client == null) { @@ -96,6 +106,7 @@ .createStream( name, StreamOptions.builder().encryptionKey(streamKey).build()); streamId = created.id(); + createdStreamId = streamId; streamName = name; } @@ -108,6 +119,7 @@ public void createKeylessStream(String name) { var created = client().streams().createStream(name); streamId = created.id(); + createdStreamId = streamId; streamName = name; } @@ -195,6 +207,12 @@ HeaderValue.fromString(value), messages.get(0).userHeaders().get(HeaderKey.fromString(key))); } + @When("I delete the stream") + public void deleteStream() { + client().streams().deleteStream(requireStreamId()); + streamId = null; + } + @When("I rename the stream to {string}") public void renameStream(String name) { client().streams() @@ -355,6 +373,21 @@ .toLowerCase()); } + static ResourceOptionsSteps current() { + return current; + } + + /** The stream this scenario created, still known after a step deleted it. */ + Long createdStreamId() { + assertNotNull(createdStreamId); + return createdStreamId; + } + + byte[] streamKey() { + assertNotNull(streamKey); + return streamKey; + } + private IggyTcpClient client() { if (client == null) { String[] address = TestEnvironment.serverAddress().split(":", 2);
diff --git a/bdd/node/Dockerfile b/bdd/node/Dockerfile index a76c12b..7f209a7 100644 --- a/bdd/node/Dockerfile +++ b/bdd/node/Dockerfile
@@ -23,6 +23,7 @@ COPY ./bdd/scenarios/stream_options.feature ./bdd/ COPY ./bdd/scenarios/resource_options_patch.feature ./bdd/ COPY ./bdd/scenarios/session_contract.feature ./bdd/ +COPY ./bdd/scenarios/metadata_changes.feature ./bdd/ RUN npm ci
diff --git a/bdd/php/tests/MetadataChangesFeatureTest.php b/bdd/php/tests/MetadataChangesFeatureTest.php new file mode 100644 index 0000000..6c34389 --- /dev/null +++ b/bdd/php/tests/MetadataChangesFeatureTest.php
@@ -0,0 +1,329 @@ +<?php +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +declare(strict_types=1); + +require_once __DIR__ . '/SharedFeatureParser.php'; +require_once __DIR__ . '/StreamSteps.php'; +require_once __DIR__ . '/TestEnvironment.php'; + +use Iggy\Client as IggyClient; +use Iggy\Exception\AuthenticationException; +use Iggy\MetadataChangeCursor; +use Iggy\MetadataChangeGroup; +use Iggy\MetadataChangeKind; +use Iggy\MetadataChangeRecord; +use Iggy\MetadataChanges; +use Iggy\MetadataChangeStart; +use Iggy\MetadataChangeStatus; +use Iggy\MetadataStreamChange; +use Iggy\MetadataStreamDeleted; +use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\Group; +use PHPUnit\Framework\TestCase; + +final class MetadataChangesFeatureTest extends TestCase +{ + use StreamSteps; + + private const ROOT_USER_ID = 0; + private const UNAUTHORIZED = 41; + private const PAGE_LIMIT = 1000; + // A two phase delete commits its record after the call returns. + private const FOLLOW_ATTEMPTS = 100; + private const FOLLOW_INTERVAL_MICROSECONDS = 100_000; + private const NEVER_RETURNED_KEYS = ['encryption_key', 'encryption_key_wrapped', 'critical_keys']; + + /** @var list<MetadataChangeGroup> */ + private array $groups = []; + private ?MetadataChangeCursor $cursor = null; + /** @var list<MetadataChangeRecord> */ + private array $followed = []; + /** @var array<int, string> */ + private array $currentStreams = []; + private ?string $username = null; + + public static function scenarioCases(): array + { + return SharedFeatureParser::load(__DIR__ . '/../../scenarios/metadata_changes.feature'); + } + + #[DataProvider('scenarioCases')] + #[Group('metadata-changes')] + public function testMetadataChangesScenario(string $scenarioName, array $steps): void + { + assert_true($scenarioName !== '', 'scenario name must not be empty'); + foreach ($steps as $step) { + $this->runStep($step); + } + } + + protected function tearDown(): void + { + if ($this->streamName !== null || $this->username !== null) { + $root = new IggyClient(TestEnvironment::serverAddress()); + $root->connect(); + $root->loginUser(TestEnvironment::rootUsername(), TestEnvironment::rootPassword()); + if ($this->streamName !== null) { + cleanup_stream($root, $this->streamName); + } + if ($this->username !== null) { + delete_user_by_name($root, $this->username); + } + } + + parent::tearDown(); + } + + private function runStep(string $step): void + { + if ($this->runStreamStep($step)) { + return; + } + + if (preg_match('/^I hold the latest metadata change cursor for the groups "([^"]+)"$/', $step, $matches) === 1) { + $this->groups = self::parseGroups($matches[1]); + $page = $this->read(MetadataChangeStart::latest()); + assert_same(MetadataChangeStatus::Ok, $page->status); + $this->cursor = $page->next; + assert_not_null($this->cursor); + return; + } + + if ($step === 'I delete the stream') { + $this->requireClient()->deleteStream($this->requireStreamId()); + return; + } + + if (preg_match('/^I create and delete (\d+) streams named "([^"]+)"$/', $step, $matches) === 1) { + $client = $this->requireClient(); + for ($index = 0; $index < (int) $matches[1]; $index++) { + $name = "{$matches[2]}-{$index}"; + $client->createStream($name); + $client->deleteStream($name); + } + return; + } + + if (preg_match('/^following the metadata changes should return the kinds "([^"]+)" for the stream$/', $step, $matches) === 1) { + $expected = array_map( + static fn (string $name): MetadataChangeKind => MetadataChangeKind::from(trim($name)), + explode(',', $matches[1]), + ); + $records = $this->follow(count($expected)); + foreach ($records as $record) { + assert_true(in_array($record->group, $this->groups, true), 'a record of an unrequested group'); + } + $this->followed = $this->streamRecords($records); + assert_same( + $expected, + array_map(static fn (MetadataChangeRecord $record): ?MetadataChangeKind => $record->kind, $this->followed), + ); + return; + } + + if ($step === 'each followed change should name the root user as its actor') { + assert_true($this->followed !== [], 'changes were followed'); + foreach ($this->followed as $record) { + assert_same(self::ROOT_USER_ID, $record->actor_user_id); + } + return; + } + + if (preg_match('/^the followed "([^"]+)" change should carry a requested op before its own op$/', $step, $matches) === 1) { + $record = $this->followedRecord($matches[1]); + $deleted = $record->change; + assert_instance_of(MetadataStreamDeleted::class, $deleted); + assert_not_null($deleted->requested_op); + assert_true($deleted->requested_op < $record->op, 'the request precedes the completion'); + return; + } + + if (preg_match('/^the followed "([^"]+)" change should report encryption key source "([^"]+)"$/', $step, $matches) === 1) { + assert_same($matches[2], $this->followedStream($matches[1])->derived_options['encryption_key_source'] ?? null); + return; + } + + if (preg_match('/^the followed "([^"]+)" change should carry the stream key fingerprint$/', $step, $matches) === 1) { + assert_not_null($this->keyBase64); + assert_same( + IggyClient::encryptionKeyFingerprint($this->keyBase64), + $this->followedStream($matches[1])->derived_options['encryption_key_fingerprint'] ?? null, + ); + return; + } + + if ($step === 'the followed changes should expose no encryption key material') { + assert_true($this->followed !== [], 'changes were followed'); + foreach ($this->followed as $record) { + $stream = $record->change; + if (!$stream instanceof MetadataStreamChange) { + continue; + } + $explicit = $stream->options; + $derived = $stream->derived_options; + assert_not_null($explicit); + assert_not_null($derived); + foreach (self::NEVER_RETURNED_KEYS as $secret) { + self::assertArrayNotHasKey($secret, $explicit); + self::assertArrayNotHasKey($secret, $derived); + } + if ($this->keyBase64 !== null) { + $values = [...array_values($explicit), ...array_values($derived)]; + self::assertNotContains($this->keyBase64, $values); + self::assertNotContains(base64_decode($this->keyBase64, true), $values); + } + } + return; + } + + if (preg_match('/^following the metadata changes should answer "([^"]+)"$/', $step, $matches) === 1) { + assert_not_null($this->cursor); + $page = $this->read(MetadataChangeStart::after($this->cursor)); + assert_same(MetadataChangeStatus::from($matches[1]), $page->status); + if ($page->status !== MetadataChangeStatus::Ok) { + assert_same([], $page->records); + assert_null($page->next); + } + return; + } + + if (preg_match('/^I read the current state for the groups "([^"]+)"$/', $step, $matches) === 1) { + $this->groups = self::parseGroups($matches[1]); + $this->currentStreams = []; + $token = null; + do { + $page = $this->read(MetadataChangeStart::current($token)); + assert_same(MetadataChangeStatus::Ok, $page->status); + foreach ($page->items as $item) { + $body = $item->body; + if ($body instanceof MetadataStreamChange) { + $this->currentStreams[$body->stream_id] = $body->name; + } + } + $token = $page->current_page_token; + } while ($token !== null); + $this->cursor = $page->next; + assert_not_null($this->cursor); + ksort($this->currentStreams); + return; + } + + if ($step === 'the current state should list the same streams as the stream list') { + assert_same(stream_list($this->requireClient()), $this->currentStreams); + return; + } + + if (preg_match('/^a user named "([^"]+)" with password "([^"]+)" and no server permissions exists$/', $step, $matches) === 1) { + create_user_without_permissions($this->requireClient(), $matches[1], $matches[2]); + $this->username = $matches[1]; + return; + } + + if (preg_match('/^I authenticate as "([^"]+)" with password "([^"]+)"$/', $step, $matches) === 1) { + $this->client = new IggyClient(TestEnvironment::serverAddress()); + $this->client->connect(); + $this->client->loginUser($matches[1], $matches[2]); + return; + } + + if ($step === 'reading the metadata changes should be refused as unauthorized') { + $this->groups = []; + $error = assert_throws(fn () => $this->read(MetadataChangeStart::earliest())); + assert_instance_of(AuthenticationException::class, $error); + assert_same(self::UNAUTHORIZED, $error->getCode()); + return; + } + + self::fail("Unsupported BDD step: {$step}"); + } + + private function read(MetadataChangeStart $start): MetadataChanges + { + return $this->requireClient()->getMetadataChanges($start, $this->groups, limit: self::PAGE_LIMIT); + } + + /** @return list<MetadataChangeRecord> */ + private function follow(int $count): array + { + assert_not_null($this->cursor); + $cursor = $this->cursor; + $records = []; + for ($attempt = 0; $attempt < self::FOLLOW_ATTEMPTS; $attempt++) { + $page = $this->read(MetadataChangeStart::after($cursor)); + assert_same(MetadataChangeStatus::Ok, $page->status); + $cursor = $page->next; + assert_not_null($cursor); + array_push($records, ...$page->records); + if ($page->has_more) { + continue; + } + if (count($this->streamRecords($records)) >= $count) { + break; + } + usleep(self::FOLLOW_INTERVAL_MICROSECONDS); + } + + return $records; + } + + /** @return list<MetadataChangeRecord> */ + private function streamRecords(array $records): array + { + $streamId = $this->requireStreamId(); + + return array_values(array_filter( + $records, + static function (MetadataChangeRecord $record) use ($streamId): bool { + $change = $record->change; + + return property_exists($change, 'stream_id') && $change->stream_id === $streamId; + }, + )); + } + + private function followedRecord(string $kind): MetadataChangeRecord + { + $wanted = MetadataChangeKind::from($kind); + foreach ($this->followed as $record) { + if ($record->kind === $wanted) { + return $record; + } + } + + self::fail("no followed {$kind} change"); + } + + private function followedStream(string $kind): MetadataStreamChange + { + $stream = $this->followedRecord($kind)->change; + assert_instance_of(MetadataStreamChange::class, $stream); + assert_true(!$stream->options_unreadable, 'stream options stay readable'); + + return $stream; + } + + /** @return list<MetadataChangeGroup> */ + private static function parseGroups(string $names): array + { + return array_map( + static fn (string $name): MetadataChangeGroup => MetadataChangeGroup::from(trim($name)), + explode(',', $names), + ); + } +}
diff --git a/bdd/php/tests/StreamOptionsFeatureTest.php b/bdd/php/tests/StreamOptionsFeatureTest.php index 52d1e2d..fe0335c 100644 --- a/bdd/php/tests/StreamOptionsFeatureTest.php +++ b/bdd/php/tests/StreamOptionsFeatureTest.php
@@ -19,6 +19,7 @@ declare(strict_types=1); require_once __DIR__ . '/SharedFeatureParser.php'; +require_once __DIR__ . '/StreamSteps.php'; require_once __DIR__ . '/TestEnvironment.php'; use Iggy\Client as IggyClient; @@ -34,11 +35,8 @@ final class StreamOptionsFeatureTest extends TestCase { - private ?IggyClient $client = null; - private ?int $streamId = null; - private ?string $streamName = null; - private ?string $topicName = null; - private ?string $keyBase64 = null; + use StreamSteps; + private bool $updateRejected = false; /** @var array<string, OptionSpec> */ private array $streamOptionSpecs = []; @@ -76,27 +74,7 @@ private function runStep(string $step): void { - if ($step === 'I have a running Iggy server') { - $this->client = new IggyClient(TestEnvironment::serverAddress()); - $this->client->connect(); - $this->client->ping(); - return; - } - - if ($step === 'I am authenticated as the root user') { - $this->requireClient()->loginUser(TestEnvironment::rootUsername(), TestEnvironment::rootPassword()); - return; - } - - if (preg_match('/^I create a stream named "([^"]+)" with an encryption key of 32 bytes valued (\d+)$/', $step, $matches) === 1 - || preg_match('/^a stream named "([^"]+)" with an encryption key of 32 bytes valued (\d+) exists$/', $step, $matches) === 1) { - $this->keyBase64 = base64_encode(str_repeat(chr((int) $matches[2]), 32)); - $this->createStream($matches[1], $this->keyBase64); - return; - } - - if (preg_match('/^I create a stream named "([^"]+)" without an encryption key$/', $step, $matches) === 1) { - $this->createStream($matches[1], null); + if ($this->runStreamStep($step)) { return; } @@ -137,12 +115,6 @@ return; } - if (preg_match('/^I create a topic named "([^"]+)" with (\d+) partition in the stream$/', $step, $matches) === 1) { - $this->topicName = $matches[1]; - $this->requireClient()->createTopic($this->requireStreamId(), $this->topicName, (int) $matches[2], null, null, null, null); - return; - } - if (preg_match('/^I send a message with payload "([^"]+)" to the topic$/', $step, $matches) === 1) { assert_not_null($this->topicName); $this->requireClient()->sendMessages($this->requireStreamId(), $this->topicName, 0, [new SendMessage($matches[1])]); @@ -158,12 +130,6 @@ return; } - if (preg_match('/^I rename the stream to "([^"]+)"$/', $step, $matches) === 1) { - $this->requireClient()->updateStream($this->requireStreamId(), name: $matches[1]); - $this->streamName = $matches[1]; - return; - } - if (preg_match('/^the stream name should be "([^"]+)"$/', $step, $matches) === 1) { assert_same($matches[1], $this->requireStream()->name); return; @@ -219,27 +185,6 @@ return $this->streamOptionSpecs[$key]; } - private function createStream(string $name, ?string $key): void - { - $this->requireClient()->createStream($name, $key); - $stream = $this->requireClient()->getStream($name); - assert_not_null($stream); - $this->streamId = $stream->id; - $this->streamName = $name; - } - - private function requireClient(): IggyClient - { - assert_not_null($this->client); - return $this->client; - } - - private function requireStreamId(): int - { - assert_not_null($this->streamId); - return $this->streamId; - } - private function requireStream(): StreamDetails { $stream = $this->requireClient()->getStream($this->requireStreamId());
diff --git a/bdd/php/tests/StreamSteps.php b/bdd/php/tests/StreamSteps.php new file mode 100644 index 0000000..c5b8e93 --- /dev/null +++ b/bdd/php/tests/StreamSteps.php
@@ -0,0 +1,99 @@ +<?php +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +declare(strict_types=1); + +require_once __DIR__ . '/TestEnvironment.php'; + +use Iggy\Client as IggyClient; + +/** + * The server, authentication, stream, topic and rename steps the stream options and metadata + * changes features share. + */ +trait StreamSteps +{ + private ?IggyClient $client = null; + private ?int $streamId = null; + private ?string $streamName = null; + private ?string $topicName = null; + private ?string $keyBase64 = null; + + /** Runs a shared step, and tells whether the step was one of them. */ + private function runStreamStep(string $step): bool + { + if ($step === 'I have a running Iggy server') { + $this->client = new IggyClient(TestEnvironment::serverAddress()); + $this->client->connect(); + $this->client->ping(); + return true; + } + + if ($step === 'I am authenticated as the root user') { + $this->requireClient()->loginUser(TestEnvironment::rootUsername(), TestEnvironment::rootPassword()); + return true; + } + + if (preg_match('/^I create a stream named "([^"]+)" with an encryption key of 32 bytes valued (\d+)$/', $step, $matches) === 1 + || preg_match('/^a stream named "([^"]+)" with an encryption key of 32 bytes valued (\d+) exists$/', $step, $matches) === 1) { + $this->keyBase64 = base64_encode(str_repeat(chr((int) $matches[2]), 32)); + $this->createStream($matches[1], $this->keyBase64); + return true; + } + + if (preg_match('/^I create a stream named "([^"]+)" without an encryption key$/', $step, $matches) === 1) { + $this->createStream($matches[1], null); + return true; + } + + if (preg_match('/^I create a topic named "([^"]+)" with (\d+) partition in the stream$/', $step, $matches) === 1) { + $this->topicName = $matches[1]; + $this->requireClient()->createTopic($this->requireStreamId(), $this->topicName, (int) $matches[2], null, null, null, null); + return true; + } + + if (preg_match('/^I rename the stream to "([^"]+)"$/', $step, $matches) === 1) { + $this->requireClient()->updateStream($this->requireStreamId(), name: $matches[1]); + $this->streamName = $matches[1]; + return true; + } + + return false; + } + + private function createStream(string $name, ?string $key): void + { + $this->requireClient()->createStream($name, $key); + $stream = $this->requireClient()->getStream($name); + assert_not_null($stream); + $this->streamId = $stream->id; + $this->streamName = $name; + } + + private function requireClient(): IggyClient + { + assert_not_null($this->client); + return $this->client; + } + + private function requireStreamId(): int + { + assert_not_null($this->streamId); + return $this->streamId; + } +}
diff --git a/bdd/python/tests/conftest.py b/bdd/python/tests/conftest.py index 9930e66..4924f23 100644 --- a/bdd/python/tests/conftest.py +++ b/bdd/python/tests/conftest.py
@@ -21,10 +21,25 @@ import asyncio import os +import socket from dataclasses import dataclass import pytest -from apache_iggy import IggyClient, OptionSpec, ReceiveMessage +from apache_iggy import ( + IggyClient, + IggyExpiry, + MetadataChangeCursor, + MetadataChangeGroup, + MetadataChangeRecord, + MetadataCurrentItem, + OptionSpec, + ReceiveMessage, + StreamOptions, + TopicOptions, +) +from pytest_bdd import given, parsers, when + +STREAM_KEY_LENGTH = 32 @dataclass @@ -47,6 +62,10 @@ last_option_specs: list[OptionSpec] | None = None last_user_id: int | None = None last_call_error: RuntimeError | None = None + last_change_groups: list[MetadataChangeGroup] | None = None + last_change_cursor: MetadataChangeCursor | None = None + last_changes: list[MetadataChangeRecord] | None = None + last_current_items: list[MetadataCurrentItem] | None = None def pytest_configure(config: pytest.Config) -> None: @@ -61,6 +80,8 @@ "session-contract: shared session contract scenarios", "stream-options: shared stream option scenarios", "resource-options-patch: shared resource patch scenarios", + "metadata-changes: shared metadata change log scenarios", + "serial: scenarios that must not share the server with other runs", "message-headers: scenarios that assert message headers", "describe-options: scenarios that assert the option catalog", ): @@ -110,3 +131,114 @@ ctx.server_addr = required_env("IGGY_TCP_ADDRESS") yield ctx + + +# Steps shared by the stream options and metadata changes features. A test +# module that defines the same step text keeps its own definition. + + +@given("I have a running Iggy server") +def running_server(context): + async def connect(): + host, port = context.server_addr.split(":") + try: + address = f"{socket.gethostbyname(host)}:{port}" + except socket.gaierror: + address = context.server_addr + + context.client = IggyClient(address) + await context.client.connect() + await context.client.ping() + + asyncio.run(connect()) + + +@given("I am authenticated as the root user") +def authenticated_root_user(context, root_credentials): + async def login(): + await context.client.login_user(*root_credentials) + + asyncio.run(login()) + + +@when( + parsers.re( + r'I create a stream named "(?P<name>[^"]+)" with an encryption key ' + r"of 32 bytes valued (?P<byte>\d+)" + ), + converters={"byte": int}, +) +def create_stream_with_key(context, name, byte): + create_keyed_stream(context, name, byte) + + +@given( + parsers.re( + r'a stream named "(?P<name>[^"]+)" with an encryption key ' + r"of 32 bytes valued (?P<byte>\d+) exists" + ), + converters={"byte": int}, +) +def keyed_stream_exists(context, name, byte): + create_keyed_stream(context, name, byte) + + +@when( + parsers.re(r'I create a stream named "(?P<name>[^"]+)" without an encryption key') +) +def create_stream_without_key(context, name): + async def create(): + stream = await context.client.create_stream(name) + context.last_stream_id = stream.id + context.last_stream_name = stream.name + context.last_stream_key = None + + asyncio.run(create()) + + +@when( + parsers.re( + r'I create a topic named "(?P<name>[^"]+)" with (?P<partitions>\d+) ' + r"partitions? in the stream" + ), + converters={"partitions": int}, +) +def create_topic_in_stream(context, name, partitions): + async def create(): + await context.client.create_topic( + context.last_stream_id, + name, + TopicOptions( + partitions_count=partitions, + message_expiry=IggyExpiry.NeverExpire(), + ), + ) + topic = await context.client.get_topic(context.last_stream_id, name) + assert topic is not None + context.last_topic_id = topic.id + context.last_topic_name = topic.name + context.last_topic_partitions = partitions + + asyncio.run(create()) + + +@when(parsers.re(r'I rename the stream to "(?P<name>[^"]+)"')) +def rename_stream(context, name): + async def rename(): + await context.client.update_stream(context.last_stream_id, name=name) + + asyncio.run(rename()) + + +def create_keyed_stream(context, name: str, byte: int) -> None: + key = bytes([byte]) * STREAM_KEY_LENGTH + + async def create(): + stream = await context.client.create_stream( + name, StreamOptions(encryption_key=key) + ) + context.last_stream_id = stream.id + context.last_stream_name = stream.name + context.last_stream_key = key + + asyncio.run(create())
diff --git a/bdd/python/tests/test_metadata_changes.py b/bdd/python/tests/test_metadata_changes.py new file mode 100644 index 0000000..a880ae5 --- /dev/null +++ b/bdd/python/tests/test_metadata_changes.py
@@ -0,0 +1,315 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Metadata changes BDD steps for the Python SDK. + +The server, authentication, stream, topic and rename steps are shared through +conftest.py. +""" + +import asyncio +import base64 +from pathlib import Path + +from apache_iggy import ( + GlobalPermissions, + IggyError, + MetadataChangeGroup, + MetadataChangeKind, + MetadataChangeRecord, + MetadataChanges, + MetadataChangeStart, + MetadataChangeStatus, + MetadataStreamChange, + MetadataStreamDeleted, + Permissions, + encryption_key_fingerprint, +) +from pytest_bdd import given, parsers, scenarios, then, when + +FEATURE_FILE = Path("/app/features/metadata_changes.feature") +if not FEATURE_FILE.exists(): + FEATURE_FILE = ( + Path(__file__).resolve().parents[2] / "scenarios/metadata_changes.feature" + ) +scenarios(str(FEATURE_FILE)) + +ROOT_USER_ID = 0 +UNAUTHORIZED = 41 +PAGE_LIMIT = 1000 +# A two phase delete commits its record after the call returns. +FOLLOW_ATTEMPTS = 100 +FOLLOW_INTERVAL_SECONDS = 0.1 +NEVER_RETURNED_KEYS = {"encryption_key", "encryption_key_wrapped", "critical_keys"} + + +@given( + parsers.re( + r'I hold the latest metadata change cursor for the groups "(?P<groups>[^"]+)"' + ) +) +def hold_latest_cursor(context, groups): + context.last_change_groups = parse_groups(groups) + page = asyncio.run(read_changes(context, MetadataChangeStart.Latest())) + assert page.status == MetadataChangeStatus.Ok + assert page.next is not None + context.last_change_cursor = page.next + + +@when("I delete the stream") +def delete_stream(context): + async def delete(): + await context.client.delete_stream(context.last_stream_id) + + asyncio.run(delete()) + + +@when( + parsers.re(r'I create and delete (?P<count>\d+) streams named "(?P<prefix>[^"]+)"'), + converters={"count": int}, +) +def churn_streams(context, count, prefix): + async def churn(): + for index in range(count): + stream = await context.client.create_stream(f"{prefix}-{index}") + await context.client.delete_stream(stream.id) + + asyncio.run(churn()) + + +@then( + parsers.re( + r"following the metadata changes should return the kinds " + r'"(?P<kinds>[^"]+)" for the stream' + ) +) +def followed_kinds(context, kinds): + expected = [parse_member(MetadataChangeKind, name) for name in kinds.split(",")] + + async def follow() -> list[MetadataChangeRecord]: + records: list[MetadataChangeRecord] = [] + cursor = context.last_change_cursor + for _ in range(FOLLOW_ATTEMPTS): + page = await read_changes(context, MetadataChangeStart.After(cursor)) + assert page.status == MetadataChangeStatus.Ok + assert page.next is not None + cursor = page.next + records.extend(page.records) + if page.has_more: + continue + if len(stream_records(context, records)) >= len(expected): + break + await asyncio.sleep(FOLLOW_INTERVAL_SECONDS) + return records + + records = asyncio.run(follow()) + assert all(record.group in context.last_change_groups for record in records) + mine = stream_records(context, records) + assert [record.kind for record in mine] == expected + context.last_changes = mine + + +@then("each followed change should name the root user as its actor") +def root_actor(context): + assert context.last_changes + for record in context.last_changes: + assert record.actor_user_id == ROOT_USER_ID, record + + +@then( + parsers.re( + r'the followed "(?P<kind>[^"]+)" change should carry a requested op ' + r"before its own op" + ) +) +def requested_op_before_own(context, kind): + record = followed(context, kind) + deleted = record.change + assert isinstance(deleted, MetadataStreamDeleted) + assert deleted.requested_op is not None + assert deleted.requested_op < record.op + + +@then( + parsers.re( + r'the followed "(?P<kind>[^"]+)" change should report encryption key ' + r'source "(?P<source>[^"]+)"' + ) +) +def change_key_source(context, kind, source): + derived = followed_stream(context, kind).derived_options + assert derived is not None + assert derived.to_scalar_dict()["encryption_key_source"] == source + + +@then( + parsers.re( + r'the followed "(?P<kind>[^"]+)" change should carry the stream key ' + r"fingerprint" + ) +) +def change_key_fingerprint(context, kind): + assert context.last_stream_key is not None + derived = followed_stream(context, kind).derived_options + assert derived is not None + assert derived.to_scalar_dict()[ + "encryption_key_fingerprint" + ] == encryption_key_fingerprint(context.last_stream_key) + + +@then("the followed changes should expose no encryption key material") +def changes_hide_keys(context): + assert context.last_changes + for record in context.last_changes: + stream = record.change + if not isinstance(stream, MetadataStreamChange): + continue + assert stream.options is not None + assert stream.derived_options is not None + explicit = stream.options.to_scalar_dict() + derived = stream.derived_options.to_scalar_dict() + assert not NEVER_RETURNED_KEYS & (set(explicit) | set(derived)) + if context.last_stream_key is not None: + values = [*explicit.values(), *derived.values()] + assert context.last_stream_key not in values + assert base64.b64encode(context.last_stream_key).decode() not in values + + +@then(parsers.re(r'following the metadata changes should answer "(?P<status>[^"]+)"')) +def follow_status(context, status): + page = asyncio.run( + read_changes(context, MetadataChangeStart.After(context.last_change_cursor)) + ) + assert page.status == parse_member(MetadataChangeStatus, status) + if page.status != MetadataChangeStatus.Ok: + assert page.records == [] + assert page.next is None + + +@when(parsers.re(r'I read the current state for the groups "(?P<groups>[^"]+)"')) +def read_current_state(context, groups): + context.last_change_groups = parse_groups(groups) + + async def scan(): + start = MetadataChangeStart.Current() + items = [] + while True: + page = await read_changes(context, start) + assert page.status == MetadataChangeStatus.Ok + items.extend(page.items) + if page.current_page_token is None: + assert page.next is not None + context.last_change_cursor = page.next + return items + start = MetadataChangeStart.Current(page.current_page_token) + + context.last_current_items = asyncio.run(scan()) + + +@then("the current state should list the same streams as the stream list") +def current_matches_list(context): + async def list_streams(): + return await context.client.get_streams() + + listed = {(stream.id, stream.name) for stream in asyncio.run(list_streams())} + projected = { + (item.body.stream_id, item.body.name) + for item in context.last_current_items + if isinstance(item.body, MetadataStreamChange) + } + assert projected == listed + + +@given( + parsers.re( + r'a user named "(?P<username>[^"]+)" with password "(?P<password>[^"]+)" ' + r"and no server permissions exists" + ) +) +def user_without_server_permissions(context, username, password): + async def create(): + user = await context.client.create_user( + username, + password, + permissions=Permissions( + global_permissions=GlobalPermissions(read_servers=True) + ), + ) + context.last_user_id = user.id + + asyncio.run(create()) + + +@when( + parsers.re( + r'I authenticate as "(?P<username>[^"]+)" with password "(?P<password>[^"]+)"' + ) +) +def authenticate_as(context, username, password): + async def login(): + await context.client.logout_user() + await context.client.login_user(username, password) + + asyncio.run(login()) + + +@then("reading the metadata changes should be refused as unauthorized") +def changes_refused(context): + context.last_change_groups = [] + try: + asyncio.run(read_changes(context, MetadataChangeStart.Earliest())) + except IggyError as error: + assert error.code == UNAUTHORIZED + else: + raise AssertionError("manage_servers is required") + + +async def read_changes(context, start: MetadataChangeStart) -> MetadataChanges: + return await context.client.get_metadata_changes( + start, groups=context.last_change_groups, limit=PAGE_LIMIT + ) + + +def stream_records( + context, records: list[MetadataChangeRecord] +) -> list[MetadataChangeRecord]: + return [ + record + for record in records + if getattr(record.change, "stream_id", None) == context.last_stream_id + ] + + +def followed(context, kind: str) -> MetadataChangeRecord: + wanted = parse_member(MetadataChangeKind, kind) + return next(record for record in context.last_changes if record.kind == wanted) + + +def followed_stream(context, kind: str) -> MetadataStreamChange: + stream = followed(context, kind).change + assert isinstance(stream, MetadataStreamChange) + assert not stream.options_unreadable + return stream + + +def parse_groups(names: str) -> list[MetadataChangeGroup]: + return [parse_member(MetadataChangeGroup, name) for name in names.split(",")] + + +def parse_member(enum_type, name: str): + return getattr(enum_type, "".join(part.title() for part in name.strip().split("_")))
diff --git a/bdd/python/tests/test_stream_options.py b/bdd/python/tests/test_stream_options.py index 2a4b9e0..f83e5ce 100644 --- a/bdd/python/tests/test_stream_options.py +++ b/bdd/python/tests/test_stream_options.py
@@ -21,22 +21,17 @@ import asyncio import base64 -import socket from pathlib import Path from apache_iggy import ( Consumer, - IggyClient, - IggyExpiry, OptionKey, PollingStrategy, SendMessage, StreamDetails, - StreamOptions, - TopicOptions, encryption_key_fingerprint, ) -from pytest_bdd import given, parsers, scenarios, then, when +from pytest_bdd import parsers, scenarios, then, when FEATURE_FILE = Path("/app/features/stream_options.feature") if not FEATURE_FILE.exists(): @@ -45,71 +40,11 @@ ) scenarios(str(FEATURE_FILE)) -KEY_LENGTH = 32 FINGERPRINT_LENGTH = 16 PARTITION_ID = 0 NEVER_RETURNED_KEYS = {"encryption_key", "encryption_key_wrapped", "critical_keys"} -@given("I have a running Iggy server") -def running_server(context): - async def connect(): - host, port = context.server_addr.split(":") - try: - address = f"{socket.gethostbyname(host)}:{port}" - except socket.gaierror: - address = context.server_addr - - context.client = IggyClient(address) - await context.client.connect() - await context.client.ping() - - asyncio.run(connect()) - - -@given("I am authenticated as the root user") -def authenticated_root_user(context, root_credentials): - async def login(): - await context.client.login_user(*root_credentials) - - asyncio.run(login()) - - -@when( - parsers.re( - r'I create a stream named "(?P<name>[^"]+)" with an encryption key ' - r"of 32 bytes valued (?P<byte>\d+)" - ), - converters={"byte": int}, -) -def create_stream_with_key(context, name, byte): - create_keyed_stream(context, name, byte) - - -@given( - parsers.re( - r'a stream named "(?P<name>[^"]+)" with an encryption key ' - r"of 32 bytes valued (?P<byte>\d+) exists" - ), - converters={"byte": int}, -) -def keyed_stream_exists(context, name, byte): - create_keyed_stream(context, name, byte) - - -@when( - parsers.re(r'I create a stream named "(?P<name>[^"]+)" without an encryption key') -) -def create_stream_without_key(context, name): - async def create(): - stream = await context.client.create_stream(name) - context.last_stream_id = stream.id - context.last_stream_name = stream.name - context.last_stream_key = None - - asyncio.run(create()) - - @then(parsers.re(r'the stream should report encryption key source "(?P<source>[^"]+)"')) def stream_reports_source(context, source): assert fetch_stream(context).encryption_key_source == source @@ -146,32 +81,6 @@ assert base64.b64encode(context.last_stream_key).decode() not in values -@when( - parsers.re( - r'I create a topic named "(?P<name>[^"]+)" with (?P<partitions>\d+) ' - r"partitions? in the stream" - ), - converters={"partitions": int}, -) -def create_topic_in_stream(context, name, partitions): - async def create(): - await context.client.create_topic( - context.last_stream_id, - name, - TopicOptions( - partitions_count=partitions, - message_expiry=IggyExpiry.NeverExpire(), - ), - ) - topic = await context.client.get_topic(context.last_stream_id, name) - assert topic is not None - context.last_topic_id = topic.id - context.last_topic_name = topic.name - context.last_topic_partitions = partitions - - asyncio.run(create()) - - @when(parsers.re(r'I send a message with payload "(?P<payload>[^"]+)" to the topic')) def send_payload(context, payload): send_message(context, SendMessage(payload)) @@ -207,14 +116,6 @@ assert headers.to_scalar_dict()[header] == value -@when(parsers.re(r'I rename the stream to "(?P<name>[^"]+)"')) -def rename_stream(context, name): - async def rename(): - await context.client.update_stream(context.last_stream_id, name=name) - - asyncio.run(rename()) - - @then(parsers.re(r'the stream name should be "(?P<name>[^"]+)"')) def stream_name_is(context, name): assert fetch_stream(context).name == name @@ -270,20 +171,6 @@ assert spec.read_only and not spec.updatable and not spec.secret -def create_keyed_stream(context, name: str, byte: int) -> None: - key = bytes([byte]) * KEY_LENGTH - - async def create(): - stream = await context.client.create_stream( - name, StreamOptions(encryption_key=key) - ) - context.last_stream_id = stream.id - context.last_stream_name = stream.name - context.last_stream_key = key - - asyncio.run(create()) - - def fetch_stream(context) -> StreamDetails: async def fetch(): return await context.client.get_stream(context.last_stream_id)
diff --git a/bdd/python/uv.lock b/bdd/python/uv.lock index 8de7e91..3f1a79c 100644 --- a/bdd/python/uv.lock +++ b/bdd/python/uv.lock
@@ -8,7 +8,7 @@ [[package]] name = "apache-iggy" -version = "0.9.1.dev2" +version = "0.9.1.dev3" source = { directory = "../../foreign/python" } [package.metadata]
diff --git a/bdd/rust/Cargo.toml b/bdd/rust/Cargo.toml index cb7249d..c6b83c9 100644 --- a/bdd/rust/Cargo.toml +++ b/bdd/rust/Cargo.toml
@@ -70,5 +70,10 @@ harness = false required-features = ["bdd"] +[[test]] +name = "metadata_changes" +harness = false +required-features = ["bdd"] + [lints] workspace = true
diff --git a/bdd/rust/tests/common/global_context.rs b/bdd/rust/tests/common/global_context.rs index b7d36b8..d0565a0 100644 --- a/bdd/rust/tests/common/global_context.rs +++ b/bdd/rust/tests/common/global_context.rs
@@ -18,7 +18,10 @@ use bytes::Bytes; use cucumber::World; use iggy::clients::client::IggyClient; -use iggy::prelude::{IggyError, IggyMessage, OptionSpec, PolledMessages, StreamEncryptionKey}; +use iggy::prelude::{ + IggyError, IggyMessage, MetadataChangeCursor, MetadataChangeGroup, MetadataChangeRecord, + MetadataCurrentItem, OptionSpec, PolledMessages, StreamEncryptionKey, +}; #[derive(Debug, World, Default)] pub struct GlobalContext { @@ -38,4 +41,8 @@ pub last_option_specs: Vec<OptionSpec>, pub last_user_id: Option<u32>, pub last_call_result: Option<Result<(), IggyError>>, + pub last_change_cursor: Option<MetadataChangeCursor>, + pub last_change_groups: Vec<MetadataChangeGroup>, + pub last_changes: Vec<MetadataChangeRecord>, + pub last_current_items: Vec<MetadataCurrentItem>, }
diff --git a/bdd/rust/tests/metadata_changes.rs b/bdd/rust/tests/metadata_changes.rs new file mode 100644 index 0000000..e3a979a --- /dev/null +++ b/bdd/rust/tests/metadata_changes.rs
@@ -0,0 +1,31 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +pub(crate) mod common; +pub(crate) mod helpers; +pub(crate) mod steps; + +use crate::common::global_context::GlobalContext; +use cucumber::World; + +#[tokio::main] +async fn main() { + GlobalContext::cucumber() + .fail_on_skipped() + .run_and_exit("../../bdd/scenarios/metadata_changes.feature") + .await; +}
diff --git a/bdd/rust/tests/steps/metadata_changes.rs b/bdd/rust/tests/steps/metadata_changes.rs new file mode 100644 index 0000000..9b1604f --- /dev/null +++ b/bdd/rust/tests/steps/metadata_changes.rs
@@ -0,0 +1,339 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use crate::common::global_context::GlobalContext; +use cucumber::{given, then, when}; +use iggy::prelude::{ + GlobalPermissions, HeaderKey, Identifier, IggyError, MAX_METADATA_CHANGES_LIMIT, + MetadataChange, MetadataChangeGroup, MetadataChangeKind, MetadataChangeRecord, + MetadataChangeStart, MetadataChangeStatus, MetadataChanges, MetadataChangesQuery, + MetadataCurrentItem, MetadataOptions, OptionKey, Permissions, ResourceOptions, StreamClient, + StreamCreateOptions, StreamEncryptionSource, SystemClient, UserClient, UserCreateOptions, + UserStatus, +}; +use std::collections::BTreeMap; +use std::str::FromStr; +use std::time::Duration; +use tokio::time::sleep; + +const ROOT_USER_ID: u32 = 0; +const FOLLOW_ATTEMPTS: usize = 100; +const FOLLOW_INTERVAL: Duration = Duration::from_millis(100); +const HIDDEN_OPTIONS: [OptionKey; 3] = [ + OptionKey::EncryptionKey, + OptionKey::EncryptionKeyWrapped, + OptionKey::CriticalKeys, +]; + +#[given(regex = r#"^I hold the latest metadata change cursor for the groups "([^"]+)"$"#)] +pub async fn given_latest_cursor(world: &mut GlobalContext, groups: String) { + world.last_change_groups = groups + .split(',') + .map(|name| MetadataChangeGroup::from_str(name.trim()).expect("known group")) + .collect(); + let page = read_changes(world, MetadataChangeStart::Latest) + .await + .expect("latest"); + assert_eq!(page.status, MetadataChangeStatus::Ok); + world.last_change_cursor = Some(page.next.expect("latest cursor")); +} + +#[when("I delete the stream")] +pub async fn when_delete_stream(world: &mut GlobalContext) { + let client = world.client.as_ref().expect("client"); + let stream_id = world.last_stream_id.expect("stream created"); + client + .delete_stream(&Identifier::numeric(stream_id).expect("stream id")) + .await + .expect("delete stream"); +} + +#[when(regex = r#"^I create and delete (\d+) streams named "([^"]+)"$"#)] +pub async fn when_churn_streams(world: &mut GlobalContext, count: u32, prefix: String) { + let client = world.client.as_ref().expect("client"); + for index in 0..count { + let stream = client + .create_stream( + &format!("{prefix}-{index}"), + &StreamCreateOptions::default(), + ) + .await + .expect("create stream"); + client + .delete_stream(&Identifier::numeric(stream.id).expect("stream id")) + .await + .expect("delete stream"); + } +} + +#[then( + regex = r#"^following the metadata changes should return the kinds "([^"]+)" for the stream$"# +)] +pub async fn then_followed_kinds(world: &mut GlobalContext, kinds: String) { + let expected: Vec<MetadataChangeKind> = kinds + .split(',') + .map(|name| MetadataChangeKind::from_str(name.trim()).expect("known kind")) + .collect(); + let stream_id = world.last_stream_id.expect("stream created"); + let mut records = Vec::new(); + let mut cursor = world.last_change_cursor.expect("held cursor"); + // A two phase delete commits its record after the call returns. + for _ in 0..FOLLOW_ATTEMPTS { + let page = read_changes(world, MetadataChangeStart::After(cursor)) + .await + .expect("change page"); + assert_eq!(page.status, MetadataChangeStatus::Ok); + cursor = page.next.expect("continuation"); + records.extend(page.records); + if page.has_more { + continue; + } + let names_stream = + |record: &&MetadataChangeRecord| change_stream_id(record) == Some(stream_id); + if records.iter().filter(names_stream).count() >= expected.len() { + break; + } + sleep(FOLLOW_INTERVAL).await; + } + for record in &records { + let group = record.change.group().expect("known group"); + assert!(world.last_change_groups.contains(&group), "{record:?}"); + } + let mine: Vec<MetadataChangeRecord> = records + .into_iter() + .filter(|record| change_stream_id(record) == Some(stream_id)) + .collect(); + let kinds: Vec<MetadataChangeKind> = mine + .iter() + .filter_map(|record| record.change.kind()) + .collect(); + assert_eq!(kinds, expected); + world.last_changes = mine; +} + +#[then("each followed change should name the root user as its actor")] +pub async fn then_root_actor(world: &mut GlobalContext) { + for record in &world.last_changes { + assert_eq!(record.actor_user_id, Some(ROOT_USER_ID), "{record:?}"); + } +} + +#[then(regex = r#"^the followed "([^"]+)" change should carry a requested op before its own op$"#)] +pub async fn then_requested_op(world: &mut GlobalContext, kind: String) { + let record = followed(world, &kind); + let MetadataChange::StreamDeleted(deleted) = &record.change else { + panic!("expected a stream deletion, got {record:?}"); + }; + let requested = deleted.requested_op.expect("requested op"); + assert!(requested < record.op, "{record:?}"); +} + +#[then(regex = r#"^the followed "([^"]+)" change should report encryption key source "([^"]+)"$"#)] +pub async fn then_change_key_source(world: &mut GlobalContext, kind: String, source: String) { + let options = followed_stream_options(world, &kind); + let expected: &'static str = StreamEncryptionSource::from_str(&source) + .expect("known source") + .into(); + let reported = options + .get(&HeaderKey::from(OptionKey::EncryptionKeySource)) + .and_then(|option| option.value.as_str().ok().map(str::to_owned)); + assert_eq!(reported.as_deref(), Some(expected)); +} + +#[then(regex = r#"^the followed "([^"]+)" change should carry the stream key fingerprint$"#)] +pub async fn then_change_fingerprint(world: &mut GlobalContext, kind: String) { + let options = followed_stream_options(world, &kind); + let key = world.last_stream_key.as_ref().expect("stream key"); + let reported = options + .get(&HeaderKey::from(OptionKey::EncryptionKeyFingerprint)) + .and_then(|option| option.value.as_str().ok().map(str::to_owned)); + assert_eq!(reported, Some(key.fingerprint())); +} + +#[then("the followed changes should expose no encryption key material")] +pub async fn then_changes_hide_keys(world: &mut GlobalContext) { + for record in &world.last_changes { + let (MetadataChange::StreamCreated(stream) | MetadataChange::StreamUpdated(stream)) = + &record.change + else { + continue; + }; + let MetadataOptions::Readable(options) = &stream.options else { + panic!("stream options must stay readable: {record:?}"); + }; + for hidden in HIDDEN_OPTIONS { + assert!(!options.contains_key(&HeaderKey::from(hidden)), "{hidden}"); + } + } +} + +#[then(regex = r#"^following the metadata changes should answer "([^"]+)"$"#)] +pub async fn then_follow_status(world: &mut GlobalContext, status: String) { + let expected = MetadataChangeStatus::from_str(&status).expect("known status"); + let cursor = world.last_change_cursor.expect("held cursor"); + let page = read_changes(world, MetadataChangeStart::After(cursor)) + .await + .expect("change page"); + assert_eq!(page.status, expected); + if expected != MetadataChangeStatus::Ok { + assert!(page.records.is_empty()); + assert_eq!(page.next, None); + } +} + +#[when(regex = r#"^I read the current state for the groups "([^"]+)"$"#)] +pub async fn when_read_current(world: &mut GlobalContext, groups: String) { + world.last_change_groups = groups + .split(',') + .map(|name| MetadataChangeGroup::from_str(name.trim()).expect("known group")) + .collect(); + let mut start = MetadataChangeStart::Current(None); + let mut items = Vec::new(); + loop { + let page = read_changes(world, start).await.expect("current page"); + assert_eq!(page.status, MetadataChangeStatus::Ok); + items.extend(page.items); + match page.current_page_token { + Some(token) => start = MetadataChangeStart::Current(Some(token)), + None => { + world.last_change_cursor = Some(page.next.expect("anchor")); + break; + } + } + } + world.last_current_items = items; +} + +#[then("the current state should list the same streams as the stream list")] +pub async fn then_current_matches_list(world: &mut GlobalContext) { + let client = world.client.as_ref().expect("client"); + let listed: BTreeMap<u32, String> = client + .get_streams() + .await + .expect("streams") + .into_iter() + .map(|stream| (stream.id, stream.name)) + .collect(); + let projected: BTreeMap<u32, String> = world + .last_current_items + .iter() + .filter_map(|item| match item { + MetadataCurrentItem::Stream(stream) => Some((stream.stream_id, stream.name.clone())), + _ => None, + }) + .collect(); + assert_eq!(projected, listed); +} + +#[given( + regex = r#"^a user named "([^"]+)" with password "([^"]+)" and no server permissions exists$"# +)] +pub async fn given_user_without_server_permissions( + world: &mut GlobalContext, + username: String, + password: String, +) { + let client = world.client.as_ref().expect("client"); + let user = client + .create_user( + &username, + &password, + UserStatus::Active, + Some(Permissions { + global: GlobalPermissions { + read_servers: true, + ..GlobalPermissions::default() + }, + streams: None, + }), + &UserCreateOptions::default(), + ) + .await + .expect("create user"); + world.last_user_id = Some(user.id); +} + +#[when(regex = r#"^I authenticate as "([^"]+)" with password "([^"]+)"$"#)] +pub async fn when_authenticate_as(world: &mut GlobalContext, username: String, password: String) { + let client = world.client.as_ref().expect("client"); + client.logout_user().await.expect("logout"); + client + .login_user(&username, &password) + .await + .expect("login"); +} + +#[then("reading the metadata changes should be refused as unauthorized")] +pub async fn then_changes_refused(world: &mut GlobalContext) { + world.last_change_groups = Vec::new(); + let error = read_changes(world, MetadataChangeStart::Earliest) + .await + .expect_err("manage_servers is required"); + assert_eq!(error.as_code(), IggyError::Unauthorized.as_code()); +} + +async fn read_changes( + world: &GlobalContext, + start: MetadataChangeStart, +) -> Result<MetadataChanges, IggyError> { + let client = world.client.as_ref().expect("client"); + client + .get_metadata_changes(&MetadataChangesQuery { + start, + groups: world.last_change_groups.clone(), + limit: MAX_METADATA_CHANGES_LIMIT, + ..MetadataChangesQuery::default() + }) + .await +} + +fn followed<'a>(world: &'a GlobalContext, kind: &str) -> &'a MetadataChangeRecord { + let kind = MetadataChangeKind::from_str(kind).expect("known kind"); + world + .last_changes + .iter() + .find(|record| record.change.kind() == Some(kind)) + .expect("followed change of that kind") +} + +fn followed_stream_options(world: &GlobalContext, kind: &str) -> ResourceOptions { + let record = followed(world, kind); + let (MetadataChange::StreamCreated(stream) | MetadataChange::StreamUpdated(stream)) = + &record.change + else { + panic!("expected a stream change, got {record:?}"); + }; + let MetadataOptions::Readable(options) = &stream.options else { + panic!("stream options must stay readable: {record:?}"); + }; + options.clone() +} + +const fn change_stream_id(record: &MetadataChangeRecord) -> Option<u32> { + match &record.change { + MetadataChange::StreamCreated(stream) | MetadataChange::StreamUpdated(stream) => { + Some(stream.stream_id) + } + MetadataChange::StreamDeleted(stream) => Some(stream.stream_id), + MetadataChange::TopicCreated(topic) | MetadataChange::TopicUpdated(topic) => { + Some(topic.stream_id) + } + MetadataChange::TopicDeleted(topic) => Some(topic.stream_id), + MetadataChange::PartitionsChanged(partitions) => Some(partitions.stream_id), + _ => None, + } +}
diff --git a/bdd/rust/tests/steps/mod.rs b/bdd/rust/tests/steps/mod.rs index a091ca2..96355c8 100644 --- a/bdd/rust/tests/steps/mod.rs +++ b/bdd/rust/tests/steps/mod.rs
@@ -18,6 +18,7 @@ pub mod auth; pub mod leader_redirection; pub mod messages; +pub mod metadata_changes; pub mod raw_command; pub mod resource_options; pub mod server;
diff --git a/bdd/scenarios/metadata_changes.feature b/bdd/scenarios/metadata_changes.feature new file mode 100644 index 0000000..7fe270b --- /dev/null +++ b/bdd/scenarios/metadata_changes.feature
@@ -0,0 +1,70 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +@metadata-changes +Feature: Metadata changes + As an operator of Apache Iggy + I want to read the metadata changes the server committed + So that a mirror of streams, topics, users and sessions stays current without listing everything again + + Background: + Given I have a running Iggy server + And I am authenticated as the root user + + Scenario: Follow stream changes from the latest position + Given I hold the latest metadata change cursor for the groups "streams" + When I create a stream named "changes-follow" without an encryption key + And I rename the stream to "changes-follow-renamed" + And I delete the stream + Then following the metadata changes should return the kinds "stream_created,stream_updated,stream_deleted" for the stream + And each followed change should name the root user as its actor + + Scenario: Filter changes by group + Given I hold the latest metadata change cursor for the groups "topics" + When I create a stream named "changes-filter" without an encryption key + And I create a topic named "events" with 1 partition in the stream + Then following the metadata changes should return the kinds "topic_created" for the stream + + Scenario: Record a two phase delete at its completion + Given I hold the latest metadata change cursor for the groups "streams" + When I create a stream named "changes-two-phase" without an encryption key + And I create a topic named "events" with 1 partition in the stream + And I delete the stream + Then following the metadata changes should return the kinds "stream_created,stream_deleted" for the stream + And the followed "stream_deleted" change should carry a requested op before its own op + + Scenario: Keyed stream changes carry fingerprints and no key + Given I hold the latest metadata change cursor for the groups "streams" + When I create a stream named "changes-keyed" with an encryption key of 32 bytes valued 21 + Then following the metadata changes should return the kinds "stream_created" for the stream + And the followed "stream_created" change should report encryption key source "stream" + And the followed "stream_created" change should carry the stream key fingerprint + And the followed changes should expose no encryption key material + + @serial + Scenario: Rebuild from current state after the cursor expired + Given I hold the latest metadata change cursor for the groups "streams" + When I create and delete 40 streams named "changes-churn" + Then following the metadata changes should answer "expired" + When I read the current state for the groups "streams" + Then the current state should list the same streams as the stream list + And following the metadata changes should answer "ok" + + Scenario: Reading changes needs the manage servers permission + Given a user named "changes-reader" with password "changes-secret" and no server permissions exists + When I authenticate as "changes-reader" with password "changes-secret" + Then reading the metadata changes should be refused as unauthorized
diff --git a/core/ai/mcp/Cargo.toml b/core/ai/mcp/Cargo.toml index 1fc411c..763484a 100644 --- a/core/ai/mcp/Cargo.toml +++ b/core/ai/mcp/Cargo.toml
@@ -17,7 +17,7 @@ [package] name = "iggy-mcp" -version = "0.5.1-edge.2" +version = "0.5.1-edge.3" description = "MCP Server for Iggy message streaming platform" edition = "2024" license = "Apache-2.0"
diff --git a/core/bench/Cargo.toml b/core/bench/Cargo.toml index f05e136..577084e 100644 --- a/core/bench/Cargo.toml +++ b/core/bench/Cargo.toml
@@ -17,7 +17,7 @@ [package] name = "iggy-bench" -version = "0.6.1-edge.2" +version = "0.6.1-edge.3" edition = "2024" license = "Apache-2.0" repository = "https://github.com/apache/iggy"
diff --git a/core/binary_protocol/Cargo.toml b/core/binary_protocol/Cargo.toml index 531d478..ee64d47 100644 --- a/core/binary_protocol/Cargo.toml +++ b/core/binary_protocol/Cargo.toml
@@ -17,7 +17,7 @@ [package] name = "iggy_binary_protocol" -version = "0.11.1-edge.2" +version = "0.11.1-edge.3" description = "Wire protocol types and codec for the Iggy binary protocol. Shared between server and SDK." edition = "2024" rust-version.workspace = true
diff --git a/core/binary_protocol/src/codes.rs b/core/binary_protocol/src/codes.rs index ab8e8f9..ac84481 100644 --- a/core/binary_protocol/src/codes.rs +++ b/core/binary_protocol/src/codes.rs
@@ -29,6 +29,7 @@ // Code 14 is retired. pub const BIND_SESSION_CODE: u32 = 15; pub const UPDATE_RESOURCE_CODE: u32 = 16; +pub const GET_METADATA_CHANGES_CODE: u32 = 17; pub const GET_ME_CODE: u32 = 20; pub const GET_CLIENT_CODE: u32 = 21; pub const GET_CLIENTS_CODE: u32 = 22; @@ -123,6 +124,7 @@ DESCRIBE_OPTIONS_CODE, BIND_SESSION_CODE, UPDATE_RESOURCE_CODE, + GET_METADATA_CHANGES_CODE, GET_ME_CODE, GET_CLIENT_CODE, GET_CLIENTS_CODE,
diff --git a/core/binary_protocol/src/dispatch.rs b/core/binary_protocol/src/dispatch.rs index 0e809b4..b5106cb 100644 --- a/core/binary_protocol/src/dispatch.rs +++ b/core/binary_protocol/src/dispatch.rs
@@ -187,6 +187,8 @@ "resource.update", Operation::UpdateResource, ), + // Change log of committed metadata operations + CommandMeta::non_replicated(GET_METADATA_CHANGES_CODE, "metadata.changes"), ]; /// Lookup command metadata by command code. @@ -246,6 +248,7 @@ BIND_SESSION_CODE => 45, GET_PARTITION_ROUTE_CODE => 46, UPDATE_RESOURCE_CODE => 47, + GET_METADATA_CHANGES_CODE => 48, _ => return None, }; Some(&COMMAND_TABLE[idx])
diff --git a/core/binary_protocol/src/primitives/metadata_changes/body.rs b/core/binary_protocol/src/primitives/metadata_changes/body.rs new file mode 100644 index 0000000..ba55e95 --- /dev/null +++ b/core/binary_protocol/src/primitives/metadata_changes/body.rs
@@ -0,0 +1,1771 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use super::{ + WireMemberLeftReason, WireMetadataChangeGroup, WireMetadataChangeKind, + WirePermissionsUpdatedReason, WireSessionClosedReason, WireTokenDeletedReason, + optional_u32_size, optional_u64_size, partition_ids_size, put_flag, put_optional_u32, + put_optional_u64, put_partition_ids, read_flag, read_optional_u32, read_optional_u64, + read_partition_ids, read_wire_name, +}; +use crate::WireError; +use crate::codec::{WireDecode, WireEncode, read_u8, read_u32_le, read_u64_le, read_u128_le}; +use crate::primitives::identifier::WireName; +use crate::primitives::options::{ + WireOptions, decode_options_prefixed, encode_options_prefixed, options_prefixed_size, +}; +use crate::primitives::permissions::WirePermissions; +use bytes::{BufMut, Bytes, BytesMut}; +use std::borrow::Cow; + +/// A change record body, typed by its kind. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum WireMetadataChange { + StreamCreated(WireStreamBody), + StreamUpdated(WireStreamBody), + StreamDeleted(WireStreamDeletedBody), + TopicCreated(WireTopicBody), + TopicUpdated(WireTopicBody), + TopicDeleted(WireTopicDeletedBody), + PartitionsChanged(WirePartitionsChangedBody), + ConsumerGroupCreated(WireConsumerGroupCreatedBody), + ConsumerGroupDeleted(WireConsumerGroupDeletedBody), + ConsumerGroupMemberJoined(WireMemberJoinedBody), + ConsumerGroupMemberLeft(WireMemberLeftBody), + ConsumerGroupPartitionHandedOver(WirePartitionHandedOverBody), + UserCreated(WireUserBody), + UserUpdated(WireUserBody), + UserDeleted(WireUserIdentityBody), + UserPasswordChanged(WireUserIdentityBody), + UserPermissionsUpdated(WireUserPermissionsBody), + PersonalAccessTokenCreated(WireTokenBody), + PersonalAccessTokenDeleted(WireTokenDeletedBody), + SessionRegistered(WireSessionBody), + SessionClosed(WireSessionClosedBody), +} + +/// A current state item body, typed by its group. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum WireMetadataCurrentItem { + Stream(WireStreamBody), + Topic(WireTopicItemBody), + Partitions(WirePartitionsItemBody), + ConsumerGroup(WireConsumerGroupItemBody), + ConsumerGroupMember(WireMemberItemBody), + User(WireUserBody), + PersonalAccessToken(WireTokenBody), + Session(WireSessionBody), +} + +/// Stream identity: ids are reused, the creating op is not. +/// +/// Wire format: `[stream_id:u32][stream_created_op:u64]` +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub struct WireStreamKey { + pub stream_id: u32, + pub stream_created_op: u64, +} + +/// Topic identity: the stream key plus the topic id and creating op. +/// +/// Wire format: `[stream key:12][topic_id:u32][topic_created_op:u64]` +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub struct WireTopicKey { + pub stream: WireStreamKey, + pub topic_id: u32, + pub topic_created_op: u64, +} + +/// User identity, the one sessions and tokens bind to. +/// +/// Wire format: `[user_id:u32][user_created_at:u64]` +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub struct WireUserKey { + pub user_id: u32, + pub user_created_at: u64, +} + +/// Public options of a resource, or the marker for stored critical state this +/// build cannot read. +/// +/// Wire format: `[state:u8]`, then for state 0 the explicit and derived blocks +/// as the read responses encode them, nothing for state 1. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum WireChangeOptions { + Readable { + explicit: WireOptions, + derived: WireOptions, + }, + Unreadable, +} + +/// Body of `stream_created`, `stream_updated` and of a `streams` item. +/// +/// Wire format: `[stream key][created_at:u64][name][options]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireStreamBody { + pub key: WireStreamKey, + pub created_at: u64, + pub name: WireName, + pub options: WireChangeOptions, +} + +/// Body of `stream_deleted`. `requested_op` is present on a deferred +/// completion. +/// +/// Wire format: `[stream key][created_at:u64][name][has:u8][requested_op:u64]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireStreamDeletedBody { + pub key: WireStreamKey, + pub created_at: u64, + pub name: WireName, + pub requested_op: Option<u64>, +} + +/// Body of `topic_created` and `topic_updated`. +/// +/// Wire format: `[topic key][created_at:u64][name][partitions_count:u32][options]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireTopicBody { + pub key: WireTopicKey, + pub created_at: u64, + pub name: WireName, + pub partitions_count: u32, + pub options: WireChangeOptions, +} + +/// Body of a `topics` item. Partition counts travel as `partitions` items. +/// +/// Wire format: `[topic key][created_at:u64][name][options]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireTopicItemBody { + pub key: WireTopicKey, + pub created_at: u64, + pub name: WireName, + pub options: WireChangeOptions, +} + +/// Body of `topic_deleted`. +/// +/// Wire format: `[topic key][created_at:u64][name][has:u8][requested_op:u64]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireTopicDeletedBody { + pub key: WireTopicKey, + pub created_at: u64, + pub name: WireName, + pub requested_op: Option<u64>, +} + +/// Body of `partitions_changed`. Removed ids are the actual deleted tail, +/// ascending, empty on create. +/// +/// Wire format: `[topic key][partitions_count:u32][ids][has:u8][requested_op:u64]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WirePartitionsChangedBody { + pub topic: WireTopicKey, + pub partitions_count: u32, + pub removed_partition_ids: Vec<u32>, + pub requested_op: Option<u64>, +} + +/// Body of a `partitions` item. +/// +/// Wire format: `[topic key][partitions_count:u32]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WirePartitionsItemBody { + pub topic: WireTopicKey, + pub partitions_count: u32, +} + +/// Body of `consumer_group_created`. +/// +/// Wire format: `[topic key][group_id:u32][name][consumer_groups_count:u32]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireConsumerGroupCreatedBody { + pub topic: WireTopicKey, + pub group_id: u32, + pub name: WireName, + pub consumer_groups_count: u32, +} + +/// Body of `consumer_group_deleted`. +/// +/// Wire format: `[topic key][group_id:u32][name][consumer_groups_count:u32][has:u8][requested_op:u64]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireConsumerGroupDeletedBody { + pub topic: WireTopicKey, + pub group_id: u32, + pub name: WireName, + pub consumer_groups_count: u32, + pub requested_op: Option<u64>, +} + +/// Body of a `consumer_groups` item. +/// +/// Wire format: `[topic key][group_id:u32][name]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireConsumerGroupItemBody { + pub topic: WireTopicKey, + pub group_id: u32, + pub name: WireName, +} + +/// Body of `consumer_group_member_joined`. +/// +/// Wire format: `[topic key][group_id:u32][member_id:u32][client_id:u128][members_count:u32][ids]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireMemberJoinedBody { + pub topic: WireTopicKey, + pub group_id: u32, + pub member_id: u32, + pub client_id: u128, + pub members_count: u32, + pub partition_ids: Vec<u32>, +} + +/// Body of `consumer_group_member_left`. +/// +/// Wire format: `[topic key][group_id:u32][member_id:u32][client_id:u128][members_count:u32][reason:u8]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireMemberLeftBody { + pub topic: WireTopicKey, + pub group_id: u32, + pub member_id: u32, + pub client_id: u128, + pub members_count: u32, + pub reason: WireMemberLeftReason, +} + +/// Body of `consumer_group_partition_handed_over`. The owner is absent when +/// the last member left. +/// +/// Wire format: `[topic key][group_id:u32][partition_id:u32][has:u8][member_id:u32]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WirePartitionHandedOverBody { + pub topic: WireTopicKey, + pub group_id: u32, + pub partition_id: u32, + pub member_id: Option<u32>, +} + +/// Body of a `consumer_group_members` item. +/// +/// Wire format: `[topic key][group_id:u32][member_id:u32][client_id:u128][ids]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireMemberItemBody { + pub topic: WireTopicKey, + pub group_id: u32, + pub member_id: u32, + pub client_id: u128, + pub partition_ids: Vec<u32>, +} + +/// Body of `user_created`, `user_updated` and of a `users` item. +/// +/// Wire format: `[user key][username][status:u8][has:u8][permissions][options]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireUserBody { + pub key: WireUserKey, + pub username: WireName, + pub status: u8, + pub permissions: Option<WirePermissions>, + pub options: WireChangeOptions, +} + +/// Body of `user_deleted` and `user_password_changed`. +/// +/// Wire format: `[user key][username]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireUserIdentityBody { + pub key: WireUserKey, + pub username: WireName, +} + +/// Body of `user_permissions_updated`. +/// +/// Wire format: `[user key][username][has:u8][permissions][reason:u8]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireUserPermissionsBody { + pub key: WireUserKey, + pub username: WireName, + pub permissions: Option<WirePermissions>, + pub reason: WirePermissionsUpdatedReason, +} + +/// Body of `personal_access_token_created` and of a `personal_access_tokens` +/// item. Expiry 0 means the token never expires. +/// +/// Wire format: `[user key][name][expiry:u64]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireTokenBody { + pub user: WireUserKey, + pub name: WireName, + pub expiry: u64, +} + +/// Body of `personal_access_token_deleted`. +/// +/// Wire format: `[user key][name][reason:u8]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireTokenDeletedBody { + pub user: WireUserKey, + pub name: WireName, + pub reason: WireTokenDeletedReason, +} + +/// Body of `session_registered` and of a `sessions` item. +/// +/// Wire format: `[user key][client_id:u128][session:u64]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireSessionBody { + pub user: WireUserKey, + pub client_id: u128, + pub session: u64, +} + +/// Body of `session_closed`. +/// +/// Wire format: `[user key][client_id:u128][session:u64][reason:u8]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireSessionClosedBody { + pub user: WireUserKey, + pub client_id: u128, + pub session: u64, + pub reason: WireSessionClosedReason, +} + +impl WireMetadataChange { + #[must_use] + pub const fn kind(&self) -> WireMetadataChangeKind { + match self { + Self::StreamCreated(_) => WireMetadataChangeKind::StreamCreated, + Self::StreamUpdated(_) => WireMetadataChangeKind::StreamUpdated, + Self::StreamDeleted(_) => WireMetadataChangeKind::StreamDeleted, + Self::TopicCreated(_) => WireMetadataChangeKind::TopicCreated, + Self::TopicUpdated(_) => WireMetadataChangeKind::TopicUpdated, + Self::TopicDeleted(_) => WireMetadataChangeKind::TopicDeleted, + Self::PartitionsChanged(_) => WireMetadataChangeKind::PartitionsChanged, + Self::ConsumerGroupCreated(_) => WireMetadataChangeKind::ConsumerGroupCreated, + Self::ConsumerGroupDeleted(_) => WireMetadataChangeKind::ConsumerGroupDeleted, + Self::ConsumerGroupMemberJoined(_) => WireMetadataChangeKind::ConsumerGroupMemberJoined, + Self::ConsumerGroupMemberLeft(_) => WireMetadataChangeKind::ConsumerGroupMemberLeft, + Self::ConsumerGroupPartitionHandedOver(_) => { + WireMetadataChangeKind::ConsumerGroupPartitionHandedOver + } + Self::UserCreated(_) => WireMetadataChangeKind::UserCreated, + Self::UserUpdated(_) => WireMetadataChangeKind::UserUpdated, + Self::UserDeleted(_) => WireMetadataChangeKind::UserDeleted, + Self::UserPasswordChanged(_) => WireMetadataChangeKind::UserPasswordChanged, + Self::UserPermissionsUpdated(_) => WireMetadataChangeKind::UserPermissionsUpdated, + Self::PersonalAccessTokenCreated(_) => { + WireMetadataChangeKind::PersonalAccessTokenCreated + } + Self::PersonalAccessTokenDeleted(_) => { + WireMetadataChangeKind::PersonalAccessTokenDeleted + } + Self::SessionRegistered(_) => WireMetadataChangeKind::SessionRegistered, + Self::SessionClosed(_) => WireMetadataChangeKind::SessionClosed, + } + } + + #[must_use] + pub const fn group(&self) -> WireMetadataChangeGroup { + self.kind().group() + } + + /// Encoded body bytes, without the record envelope. + #[must_use] + pub fn body_bytes(&self) -> Bytes { + match self { + Self::StreamCreated(body) | Self::StreamUpdated(body) => body.to_bytes(), + Self::StreamDeleted(body) => body.to_bytes(), + Self::TopicCreated(body) | Self::TopicUpdated(body) => body.to_bytes(), + Self::TopicDeleted(body) => body.to_bytes(), + Self::PartitionsChanged(body) => body.to_bytes(), + Self::ConsumerGroupCreated(body) => body.to_bytes(), + Self::ConsumerGroupDeleted(body) => body.to_bytes(), + Self::ConsumerGroupMemberJoined(body) => body.to_bytes(), + Self::ConsumerGroupMemberLeft(body) => body.to_bytes(), + Self::ConsumerGroupPartitionHandedOver(body) => body.to_bytes(), + Self::UserCreated(body) | Self::UserUpdated(body) => body.to_bytes(), + Self::UserDeleted(body) | Self::UserPasswordChanged(body) => body.to_bytes(), + Self::UserPermissionsUpdated(body) => body.to_bytes(), + Self::PersonalAccessTokenCreated(body) => body.to_bytes(), + Self::PersonalAccessTokenDeleted(body) => body.to_bytes(), + Self::SessionRegistered(body) => body.to_bytes(), + Self::SessionClosed(body) => body.to_bytes(), + } + } + + /// Decode a record body by its group and kind codes. + /// + /// Returns `Ok(None)` for a group or kind this build does not know, so a + /// caller can keep the record with its raw body. + /// + /// # Errors + /// Returns `WireError` for a known kind under another group, a malformed + /// body, an unknown reason code or trailing bytes. + pub fn decode(group: u8, kind: u8, body: &[u8]) -> Result<Option<Self>, WireError> { + let Ok(group) = WireMetadataChangeGroup::from_code(group) else { + return Ok(None); + }; + let Ok(kind) = WireMetadataChangeKind::from_code(kind) else { + return Ok(None); + }; + if kind.group() != group { + return Err(WireError::Validation(Cow::Owned(format!( + "change kind {} does not belong to group {}", + kind.as_u8(), + group.as_u8() + )))); + } + let change = match kind { + WireMetadataChangeKind::StreamCreated => Self::StreamCreated(decode_exact(body)?), + WireMetadataChangeKind::StreamUpdated => Self::StreamUpdated(decode_exact(body)?), + WireMetadataChangeKind::StreamDeleted => Self::StreamDeleted(decode_exact(body)?), + WireMetadataChangeKind::TopicCreated => Self::TopicCreated(decode_exact(body)?), + WireMetadataChangeKind::TopicUpdated => Self::TopicUpdated(decode_exact(body)?), + WireMetadataChangeKind::TopicDeleted => Self::TopicDeleted(decode_exact(body)?), + WireMetadataChangeKind::PartitionsChanged => { + Self::PartitionsChanged(decode_exact(body)?) + } + WireMetadataChangeKind::ConsumerGroupCreated => { + Self::ConsumerGroupCreated(decode_exact(body)?) + } + WireMetadataChangeKind::ConsumerGroupDeleted => { + Self::ConsumerGroupDeleted(decode_exact(body)?) + } + WireMetadataChangeKind::ConsumerGroupMemberJoined => { + Self::ConsumerGroupMemberJoined(decode_exact(body)?) + } + WireMetadataChangeKind::ConsumerGroupMemberLeft => { + Self::ConsumerGroupMemberLeft(decode_exact(body)?) + } + WireMetadataChangeKind::ConsumerGroupPartitionHandedOver => { + Self::ConsumerGroupPartitionHandedOver(decode_exact(body)?) + } + WireMetadataChangeKind::UserCreated => Self::UserCreated(decode_exact(body)?), + WireMetadataChangeKind::UserUpdated => Self::UserUpdated(decode_exact(body)?), + WireMetadataChangeKind::UserDeleted => Self::UserDeleted(decode_exact(body)?), + WireMetadataChangeKind::UserPasswordChanged => { + Self::UserPasswordChanged(decode_exact(body)?) + } + WireMetadataChangeKind::UserPermissionsUpdated => { + Self::UserPermissionsUpdated(decode_exact(body)?) + } + WireMetadataChangeKind::PersonalAccessTokenCreated => { + Self::PersonalAccessTokenCreated(decode_exact(body)?) + } + WireMetadataChangeKind::PersonalAccessTokenDeleted => { + Self::PersonalAccessTokenDeleted(decode_exact(body)?) + } + WireMetadataChangeKind::SessionRegistered => { + Self::SessionRegistered(decode_exact(body)?) + } + WireMetadataChangeKind::SessionClosed => Self::SessionClosed(decode_exact(body)?), + }; + Ok(Some(change)) + } +} + +impl WireMetadataCurrentItem { + #[must_use] + pub const fn group(&self) -> WireMetadataChangeGroup { + match self { + Self::Stream(_) => WireMetadataChangeGroup::Streams, + Self::Topic(_) => WireMetadataChangeGroup::Topics, + Self::Partitions(_) => WireMetadataChangeGroup::Partitions, + Self::ConsumerGroup(_) => WireMetadataChangeGroup::ConsumerGroups, + Self::ConsumerGroupMember(_) => WireMetadataChangeGroup::ConsumerGroupMembers, + Self::User(_) => WireMetadataChangeGroup::Users, + Self::PersonalAccessToken(_) => WireMetadataChangeGroup::PersonalAccessTokens, + Self::Session(_) => WireMetadataChangeGroup::Sessions, + } + } + + /// Encoded body bytes, without the item envelope. + #[must_use] + pub fn body_bytes(&self) -> Bytes { + match self { + Self::Stream(body) => body.to_bytes(), + Self::Topic(body) => body.to_bytes(), + Self::Partitions(body) => body.to_bytes(), + Self::ConsumerGroup(body) => body.to_bytes(), + Self::ConsumerGroupMember(body) => body.to_bytes(), + Self::User(body) => body.to_bytes(), + Self::PersonalAccessToken(body) => body.to_bytes(), + Self::Session(body) => body.to_bytes(), + } + } + + /// Decode an item body by its group code. + /// + /// Returns `Ok(None)` for a group this build does not know. + /// + /// # Errors + /// Returns `WireError` for a malformed body or trailing bytes. + pub fn decode(group: u8, body: &[u8]) -> Result<Option<Self>, WireError> { + let Ok(group) = WireMetadataChangeGroup::from_code(group) else { + return Ok(None); + }; + let item = match group { + WireMetadataChangeGroup::Streams => Self::Stream(decode_exact(body)?), + WireMetadataChangeGroup::Topics => Self::Topic(decode_exact(body)?), + WireMetadataChangeGroup::Partitions => Self::Partitions(decode_exact(body)?), + WireMetadataChangeGroup::ConsumerGroups => Self::ConsumerGroup(decode_exact(body)?), + WireMetadataChangeGroup::ConsumerGroupMembers => { + Self::ConsumerGroupMember(decode_exact(body)?) + } + WireMetadataChangeGroup::Users => Self::User(decode_exact(body)?), + WireMetadataChangeGroup::PersonalAccessTokens => { + Self::PersonalAccessToken(decode_exact(body)?) + } + WireMetadataChangeGroup::Sessions => Self::Session(decode_exact(body)?), + }; + Ok(Some(item)) + } +} + +/// Decode a body that must fill `body` exactly. +fn decode_exact<T: WireDecode>(body: &[u8]) -> Result<T, WireError> { + let (value, consumed) = T::decode(body)?; + if consumed != body.len() { + return Err(WireError::Validation(Cow::Owned(format!( + "{} trailing bytes after a change body", + body.len() - consumed + )))); + } + Ok(value) +} + +/// Sequential reader over one body. Every read checks bounds. +struct BodyReader<'a> { + buf: &'a [u8], + position: usize, +} + +impl<'a> BodyReader<'a> { + const fn new(buf: &'a [u8]) -> Self { + Self { buf, position: 0 } + } + + const fn consumed(&self) -> usize { + self.position + } + + fn u8(&mut self) -> Result<u8, WireError> { + let value = read_u8(self.buf, self.position)?; + self.position += 1; + Ok(value) + } + + fn u32(&mut self) -> Result<u32, WireError> { + let value = read_u32_le(self.buf, self.position)?; + self.position += 4; + Ok(value) + } + + fn u64(&mut self) -> Result<u64, WireError> { + let value = read_u64_le(self.buf, self.position)?; + self.position += 8; + Ok(value) + } + + fn u128(&mut self) -> Result<u128, WireError> { + let value = read_u128_le(self.buf, self.position)?; + self.position += 16; + Ok(value) + } + + fn flag(&mut self) -> Result<bool, WireError> { + let value = read_flag(self.buf, self.position)?; + self.position += 1; + Ok(value) + } + + fn name(&mut self) -> Result<WireName, WireError> { + let (name, consumed) = read_wire_name(self.buf, self.position)?; + self.position += consumed; + Ok(name) + } + + fn optional_u64(&mut self) -> Result<Option<u64>, WireError> { + let (value, consumed) = read_optional_u64(self.buf, self.position)?; + self.position += consumed; + Ok(value) + } + + fn optional_u32(&mut self) -> Result<Option<u32>, WireError> { + let (value, consumed) = read_optional_u32(self.buf, self.position)?; + self.position += consumed; + Ok(value) + } + + fn partition_ids(&mut self) -> Result<Vec<u32>, WireError> { + let (ids, consumed) = read_partition_ids(self.buf, self.position)?; + self.position += consumed; + Ok(ids) + } + + fn decode<T: WireDecode>(&mut self) -> Result<T, WireError> { + let rest = self + .buf + .get(self.position..) + .ok_or(WireError::UnexpectedEof { + offset: self.position, + need: 1, + have: 0, + })?; + let (value, consumed) = T::decode(rest)?; + self.position += consumed; + Ok(value) + } + + fn options(&mut self) -> Result<WireChangeOptions, WireError> { + let offset = self.position; + match self.u8()? { + 0 => { + let (explicit, explicit_consumed) = + decode_options_prefixed(self.buf, self.position)?; + self.position += explicit_consumed; + let (derived, derived_consumed) = decode_options_prefixed(self.buf, self.position)?; + self.position += derived_consumed; + Ok(WireChangeOptions::Readable { explicit, derived }) + } + 1 => Ok(WireChangeOptions::Unreadable), + value => Err(WireError::UnknownDiscriminant { + type_name: "WireChangeOptions", + value, + offset, + }), + } + } + + fn permissions(&mut self) -> Result<Option<WirePermissions>, WireError> { + if self.flag()? { + Ok(Some(self.decode()?)) + } else { + Ok(None) + } + } + + fn reason<T>(&mut self, from_code: fn(u8) -> Result<T, WireError>) -> Result<T, WireError> { + let offset = self.position; + let code = self.u8()?; + from_code(code).map_err(|error| match error { + WireError::UnknownDiscriminant { + type_name, value, .. + } => WireError::UnknownDiscriminant { + type_name, + value, + offset, + }, + other => other, + }) + } +} + +fn options_size(options: &WireChangeOptions) -> usize { + match options { + WireChangeOptions::Readable { explicit, derived } => { + 1 + options_prefixed_size(explicit) + options_prefixed_size(derived) + } + WireChangeOptions::Unreadable => 1, + } +} + +fn put_options(buf: &mut BytesMut, options: &WireChangeOptions) { + match options { + WireChangeOptions::Readable { explicit, derived } => { + buf.put_u8(0); + encode_options_prefixed(explicit, buf); + encode_options_prefixed(derived, buf); + } + WireChangeOptions::Unreadable => buf.put_u8(1), + } +} + +fn permissions_size(permissions: Option<&WirePermissions>) -> usize { + 1 + permissions.map_or(0, WireEncode::encoded_size) +} + +fn put_permissions(buf: &mut BytesMut, permissions: Option<&WirePermissions>) { + put_flag(buf, permissions.is_some()); + if let Some(permissions) = permissions { + permissions.encode(buf); + } +} + +impl WireStreamKey { + const SIZE: usize = 12; +} + +impl WireEncode for WireStreamKey { + fn encoded_size(&self) -> usize { + Self::SIZE + } + + fn encode(&self, buf: &mut BytesMut) { + buf.put_u32_le(self.stream_id); + buf.put_u64_le(self.stream_created_op); + } +} + +impl WireDecode for WireStreamKey { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let key = Self { + stream_id: reader.u32()?, + stream_created_op: reader.u64()?, + }; + Ok((key, reader.consumed())) + } +} + +impl WireTopicKey { + const SIZE: usize = WireStreamKey::SIZE + 12; +} + +impl WireEncode for WireTopicKey { + fn encoded_size(&self) -> usize { + Self::SIZE + } + + fn encode(&self, buf: &mut BytesMut) { + self.stream.encode(buf); + buf.put_u32_le(self.topic_id); + buf.put_u64_le(self.topic_created_op); + } +} + +impl WireDecode for WireTopicKey { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let key = Self { + stream: reader.decode()?, + topic_id: reader.u32()?, + topic_created_op: reader.u64()?, + }; + Ok((key, reader.consumed())) + } +} + +impl WireUserKey { + const SIZE: usize = 12; +} + +impl WireEncode for WireUserKey { + fn encoded_size(&self) -> usize { + Self::SIZE + } + + fn encode(&self, buf: &mut BytesMut) { + buf.put_u32_le(self.user_id); + buf.put_u64_le(self.user_created_at); + } +} + +impl WireDecode for WireUserKey { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let key = Self { + user_id: reader.u32()?, + user_created_at: reader.u64()?, + }; + Ok((key, reader.consumed())) + } +} + +impl WireEncode for WireStreamBody { + fn encoded_size(&self) -> usize { + WireStreamKey::SIZE + 8 + self.name.encoded_size() + options_size(&self.options) + } + + fn encode(&self, buf: &mut BytesMut) { + self.key.encode(buf); + buf.put_u64_le(self.created_at); + self.name.encode(buf); + put_options(buf, &self.options); + } +} + +impl WireDecode for WireStreamBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + key: reader.decode()?, + created_at: reader.u64()?, + name: reader.name()?, + options: reader.options()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireStreamDeletedBody { + fn encoded_size(&self) -> usize { + WireStreamKey::SIZE + 8 + self.name.encoded_size() + optional_u64_size(self.requested_op) + } + + fn encode(&self, buf: &mut BytesMut) { + self.key.encode(buf); + buf.put_u64_le(self.created_at); + self.name.encode(buf); + put_optional_u64(buf, self.requested_op); + } +} + +impl WireDecode for WireStreamDeletedBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + key: reader.decode()?, + created_at: reader.u64()?, + name: reader.name()?, + requested_op: reader.optional_u64()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireTopicBody { + fn encoded_size(&self) -> usize { + WireTopicKey::SIZE + 8 + self.name.encoded_size() + 4 + options_size(&self.options) + } + + fn encode(&self, buf: &mut BytesMut) { + self.key.encode(buf); + buf.put_u64_le(self.created_at); + self.name.encode(buf); + buf.put_u32_le(self.partitions_count); + put_options(buf, &self.options); + } +} + +impl WireDecode for WireTopicBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + key: reader.decode()?, + created_at: reader.u64()?, + name: reader.name()?, + partitions_count: reader.u32()?, + options: reader.options()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireTopicItemBody { + fn encoded_size(&self) -> usize { + WireTopicKey::SIZE + 8 + self.name.encoded_size() + options_size(&self.options) + } + + fn encode(&self, buf: &mut BytesMut) { + self.key.encode(buf); + buf.put_u64_le(self.created_at); + self.name.encode(buf); + put_options(buf, &self.options); + } +} + +impl WireDecode for WireTopicItemBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + key: reader.decode()?, + created_at: reader.u64()?, + name: reader.name()?, + options: reader.options()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireTopicDeletedBody { + fn encoded_size(&self) -> usize { + WireTopicKey::SIZE + 8 + self.name.encoded_size() + optional_u64_size(self.requested_op) + } + + fn encode(&self, buf: &mut BytesMut) { + self.key.encode(buf); + buf.put_u64_le(self.created_at); + self.name.encode(buf); + put_optional_u64(buf, self.requested_op); + } +} + +impl WireDecode for WireTopicDeletedBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + key: reader.decode()?, + created_at: reader.u64()?, + name: reader.name()?, + requested_op: reader.optional_u64()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WirePartitionsChangedBody { + fn encoded_size(&self) -> usize { + WireTopicKey::SIZE + + 4 + + partition_ids_size(&self.removed_partition_ids) + + optional_u64_size(self.requested_op) + } + + fn encode(&self, buf: &mut BytesMut) { + self.topic.encode(buf); + buf.put_u32_le(self.partitions_count); + put_partition_ids(buf, &self.removed_partition_ids); + put_optional_u64(buf, self.requested_op); + } +} + +impl WireDecode for WirePartitionsChangedBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + topic: reader.decode()?, + partitions_count: reader.u32()?, + removed_partition_ids: reader.partition_ids()?, + requested_op: reader.optional_u64()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WirePartitionsItemBody { + fn encoded_size(&self) -> usize { + WireTopicKey::SIZE + 4 + } + + fn encode(&self, buf: &mut BytesMut) { + self.topic.encode(buf); + buf.put_u32_le(self.partitions_count); + } +} + +impl WireDecode for WirePartitionsItemBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + topic: reader.decode()?, + partitions_count: reader.u32()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireConsumerGroupCreatedBody { + fn encoded_size(&self) -> usize { + WireTopicKey::SIZE + 4 + self.name.encoded_size() + 4 + } + + fn encode(&self, buf: &mut BytesMut) { + self.topic.encode(buf); + buf.put_u32_le(self.group_id); + self.name.encode(buf); + buf.put_u32_le(self.consumer_groups_count); + } +} + +impl WireDecode for WireConsumerGroupCreatedBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + topic: reader.decode()?, + group_id: reader.u32()?, + name: reader.name()?, + consumer_groups_count: reader.u32()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireConsumerGroupDeletedBody { + fn encoded_size(&self) -> usize { + WireTopicKey::SIZE + 4 + self.name.encoded_size() + 4 + optional_u64_size(self.requested_op) + } + + fn encode(&self, buf: &mut BytesMut) { + self.topic.encode(buf); + buf.put_u32_le(self.group_id); + self.name.encode(buf); + buf.put_u32_le(self.consumer_groups_count); + put_optional_u64(buf, self.requested_op); + } +} + +impl WireDecode for WireConsumerGroupDeletedBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + topic: reader.decode()?, + group_id: reader.u32()?, + name: reader.name()?, + consumer_groups_count: reader.u32()?, + requested_op: reader.optional_u64()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireConsumerGroupItemBody { + fn encoded_size(&self) -> usize { + WireTopicKey::SIZE + 4 + self.name.encoded_size() + } + + fn encode(&self, buf: &mut BytesMut) { + self.topic.encode(buf); + buf.put_u32_le(self.group_id); + self.name.encode(buf); + } +} + +impl WireDecode for WireConsumerGroupItemBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + topic: reader.decode()?, + group_id: reader.u32()?, + name: reader.name()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireMemberJoinedBody { + fn encoded_size(&self) -> usize { + WireTopicKey::SIZE + 4 + 4 + 16 + 4 + partition_ids_size(&self.partition_ids) + } + + fn encode(&self, buf: &mut BytesMut) { + self.topic.encode(buf); + buf.put_u32_le(self.group_id); + buf.put_u32_le(self.member_id); + buf.put_u128_le(self.client_id); + buf.put_u32_le(self.members_count); + put_partition_ids(buf, &self.partition_ids); + } +} + +impl WireDecode for WireMemberJoinedBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + topic: reader.decode()?, + group_id: reader.u32()?, + member_id: reader.u32()?, + client_id: reader.u128()?, + members_count: reader.u32()?, + partition_ids: reader.partition_ids()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireMemberLeftBody { + fn encoded_size(&self) -> usize { + WireTopicKey::SIZE + 4 + 4 + 16 + 4 + 1 + } + + fn encode(&self, buf: &mut BytesMut) { + self.topic.encode(buf); + buf.put_u32_le(self.group_id); + buf.put_u32_le(self.member_id); + buf.put_u128_le(self.client_id); + buf.put_u32_le(self.members_count); + buf.put_u8(self.reason.as_u8()); + } +} + +impl WireDecode for WireMemberLeftBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + topic: reader.decode()?, + group_id: reader.u32()?, + member_id: reader.u32()?, + client_id: reader.u128()?, + members_count: reader.u32()?, + reason: reader.reason(WireMemberLeftReason::from_code)?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WirePartitionHandedOverBody { + fn encoded_size(&self) -> usize { + WireTopicKey::SIZE + 4 + 4 + optional_u32_size(self.member_id) + } + + fn encode(&self, buf: &mut BytesMut) { + self.topic.encode(buf); + buf.put_u32_le(self.group_id); + buf.put_u32_le(self.partition_id); + put_optional_u32(buf, self.member_id); + } +} + +impl WireDecode for WirePartitionHandedOverBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + topic: reader.decode()?, + group_id: reader.u32()?, + partition_id: reader.u32()?, + member_id: reader.optional_u32()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireMemberItemBody { + fn encoded_size(&self) -> usize { + WireTopicKey::SIZE + 4 + 4 + 16 + partition_ids_size(&self.partition_ids) + } + + fn encode(&self, buf: &mut BytesMut) { + self.topic.encode(buf); + buf.put_u32_le(self.group_id); + buf.put_u32_le(self.member_id); + buf.put_u128_le(self.client_id); + put_partition_ids(buf, &self.partition_ids); + } +} + +impl WireDecode for WireMemberItemBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + topic: reader.decode()?, + group_id: reader.u32()?, + member_id: reader.u32()?, + client_id: reader.u128()?, + partition_ids: reader.partition_ids()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireUserBody { + fn encoded_size(&self) -> usize { + WireUserKey::SIZE + + self.username.encoded_size() + + 1 + + permissions_size(self.permissions.as_ref()) + + options_size(&self.options) + } + + fn encode(&self, buf: &mut BytesMut) { + self.key.encode(buf); + self.username.encode(buf); + buf.put_u8(self.status); + put_permissions(buf, self.permissions.as_ref()); + put_options(buf, &self.options); + } +} + +impl WireDecode for WireUserBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + key: reader.decode()?, + username: reader.name()?, + status: reader.u8()?, + permissions: reader.permissions()?, + options: reader.options()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireUserIdentityBody { + fn encoded_size(&self) -> usize { + WireUserKey::SIZE + self.username.encoded_size() + } + + fn encode(&self, buf: &mut BytesMut) { + self.key.encode(buf); + self.username.encode(buf); + } +} + +impl WireDecode for WireUserIdentityBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + key: reader.decode()?, + username: reader.name()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireUserPermissionsBody { + fn encoded_size(&self) -> usize { + WireUserKey::SIZE + + self.username.encoded_size() + + permissions_size(self.permissions.as_ref()) + + 1 + } + + fn encode(&self, buf: &mut BytesMut) { + self.key.encode(buf); + self.username.encode(buf); + put_permissions(buf, self.permissions.as_ref()); + buf.put_u8(self.reason.as_u8()); + } +} + +impl WireDecode for WireUserPermissionsBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + key: reader.decode()?, + username: reader.name()?, + permissions: reader.permissions()?, + reason: reader.reason(WirePermissionsUpdatedReason::from_code)?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireTokenBody { + fn encoded_size(&self) -> usize { + WireUserKey::SIZE + self.name.encoded_size() + 8 + } + + fn encode(&self, buf: &mut BytesMut) { + self.user.encode(buf); + self.name.encode(buf); + buf.put_u64_le(self.expiry); + } +} + +impl WireDecode for WireTokenBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + user: reader.decode()?, + name: reader.name()?, + expiry: reader.u64()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireTokenDeletedBody { + fn encoded_size(&self) -> usize { + WireUserKey::SIZE + self.name.encoded_size() + 1 + } + + fn encode(&self, buf: &mut BytesMut) { + self.user.encode(buf); + self.name.encode(buf); + buf.put_u8(self.reason.as_u8()); + } +} + +impl WireDecode for WireTokenDeletedBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + user: reader.decode()?, + name: reader.name()?, + reason: reader.reason(WireTokenDeletedReason::from_code)?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireSessionBody { + fn encoded_size(&self) -> usize { + WireUserKey::SIZE + 16 + 8 + } + + fn encode(&self, buf: &mut BytesMut) { + self.user.encode(buf); + buf.put_u128_le(self.client_id); + buf.put_u64_le(self.session); + } +} + +impl WireDecode for WireSessionBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + user: reader.decode()?, + client_id: reader.u128()?, + session: reader.u64()?, + }; + Ok((body, reader.consumed())) + } +} + +impl WireEncode for WireSessionClosedBody { + fn encoded_size(&self) -> usize { + WireUserKey::SIZE + 16 + 8 + 1 + } + + fn encode(&self, buf: &mut BytesMut) { + self.user.encode(buf); + buf.put_u128_le(self.client_id); + buf.put_u64_le(self.session); + buf.put_u8(self.reason.as_u8()); + } +} + +impl WireDecode for WireSessionClosedBody { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mut reader = BodyReader::new(buf); + let body = Self { + user: reader.decode()?, + client_id: reader.u128()?, + session: reader.u64()?, + reason: reader.reason(WireSessionClosedReason::from_code)?, + }; + Ok((body, reader.consumed())) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::primitives::permissions::{ + WireGlobalPermissions, WireStreamPermissions, WireTopicPermissions, + }; + + #[test] + fn given_every_change_kind_when_round_tripping_should_preserve_every_field() { + for change in sample_changes() { + let body = change.body_bytes(); + let decoded = + WireMetadataChange::decode(change.group().as_u8(), change.kind().as_u8(), &body) + .unwrap() + .unwrap(); + assert_eq!(decoded, change); + } + } + + #[test] + fn given_every_item_group_when_round_tripping_should_preserve_every_field() { + for item in sample_items() { + let body = item.body_bytes(); + let decoded = WireMetadataCurrentItem::decode(item.group().as_u8(), &body) + .unwrap() + .unwrap(); + assert_eq!(decoded, item); + } + } + + #[test] + fn given_every_change_kind_when_truncating_the_body_should_refuse_every_prefix() { + for change in sample_changes() { + let body = change.body_bytes(); + for length in 0..body.len() { + assert!( + WireMetadataChange::decode( + change.group().as_u8(), + change.kind().as_u8(), + &body[..length] + ) + .is_err(), + "{:?} accepted a body truncated to {length} bytes", + change.kind() + ); + } + } + } + + #[test] + fn given_trailing_bytes_when_decoding_a_change_body_should_refuse() { + for change in sample_changes() { + let mut body = change.body_bytes().to_vec(); + body.push(0); + assert!( + WireMetadataChange::decode(change.group().as_u8(), change.kind().as_u8(), &body) + .is_err(), + "{:?} accepted a trailing byte", + change.kind() + ); + } + for item in sample_items() { + let mut body = item.body_bytes().to_vec(); + body.push(0); + assert!(WireMetadataCurrentItem::decode(item.group().as_u8(), &body).is_err()); + } + } + + #[test] + fn given_known_kind_under_wrong_group_when_decoding_should_refuse() { + let change = WireMetadataChange::StreamCreated(stream_body()); + let body = change.body_bytes(); + let error = WireMetadataChange::decode( + WireMetadataChangeGroup::Topics.as_u8(), + WireMetadataChangeKind::StreamCreated.as_u8(), + &body, + ); + assert!(error.is_err()); + } + + #[test] + fn given_unknown_kind_or_group_when_decoding_should_return_none() { + assert_eq!(WireMetadataChange::decode(1, 9, &[1, 2, 3]), Ok(None)); + assert_eq!(WireMetadataChange::decode(200, 1, &[]), Ok(None)); + assert_eq!(WireMetadataChange::decode(0, 1, &[]), Ok(None)); + assert_eq!(WireMetadataCurrentItem::decode(9, &[7]), Ok(None)); + } + + #[test] + fn given_unknown_reason_code_when_decoding_member_left_should_refuse() { + let change = WireMetadataChange::ConsumerGroupMemberLeft(WireMemberLeftBody { + topic: topic_key(), + group_id: 1, + member_id: 2, + client_id: 3, + members_count: 0, + reason: WireMemberLeftReason::Left, + }); + let mut body = change.body_bytes().to_vec(); + *body.last_mut().unwrap() = 2; + assert!( + WireMetadataChange::decode(change.group().as_u8(), change.kind().as_u8(), &body) + .is_err() + ); + } + + #[test] + fn given_unknown_reasons_when_decoding_token_session_and_permission_bodies_should_refuse() { + let changes = [ + WireMetadataChange::PersonalAccessTokenDeleted(WireTokenDeletedBody { + user: user_key(), + name: WireName::new("token").unwrap(), + reason: WireTokenDeletedReason::Expired, + }), + WireMetadataChange::SessionClosed(WireSessionClosedBody { + user: user_key(), + client_id: 9, + session: 4, + reason: WireSessionClosedReason::Expired, + }), + WireMetadataChange::UserPermissionsUpdated(WireUserPermissionsBody { + key: user_key(), + username: WireName::new("alice").unwrap(), + permissions: None, + reason: WirePermissionsUpdatedReason::ResourceDeleted, + }), + ]; + for change in changes { + let mut body = change.body_bytes().to_vec(); + *body.last_mut().unwrap() = 7; + assert!( + WireMetadataChange::decode(change.group().as_u8(), change.kind().as_u8(), &body) + .is_err(), + "{:?} accepted reason code 7", + change.kind() + ); + } + } + + #[test] + fn given_options_state_byte_above_one_when_decoding_should_refuse() { + let change = WireMetadataChange::StreamCreated(stream_body()); + let mut body = change.body_bytes().to_vec(); + let state_offset = WireStreamKey::SIZE + 8 + 1 + "orders".len(); + body[state_offset] = 2; + assert!( + WireMetadataChange::decode(change.group().as_u8(), change.kind().as_u8(), &body) + .is_err() + ); + } + + #[test] + fn given_presence_byte_above_one_when_decoding_requested_op_should_refuse() { + let change = WireMetadataChange::StreamDeleted(WireStreamDeletedBody { + key: stream_key(), + created_at: 1, + name: WireName::new("orders").unwrap(), + requested_op: None, + }); + let mut body = change.body_bytes().to_vec(); + *body.last_mut().unwrap() = 2; + assert!( + WireMetadataChange::decode(change.group().as_u8(), change.kind().as_u8(), &body) + .is_err() + ); + } + + #[test] + fn given_unreadable_options_when_round_tripping_should_keep_the_marker() { + let change = WireMetadataChange::TopicUpdated(WireTopicBody { + options: WireChangeOptions::Unreadable, + ..topic_body() + }); + let body = change.body_bytes(); + let decoded = + WireMetadataChange::decode(change.group().as_u8(), change.kind().as_u8(), &body) + .unwrap() + .unwrap(); + assert_eq!(decoded, change); + } + + fn stream_key() -> WireStreamKey { + WireStreamKey { + stream_id: 7, + stream_created_op: 4121, + } + } + + fn topic_key() -> WireTopicKey { + WireTopicKey { + stream: stream_key(), + topic_id: 3, + topic_created_op: 4125, + } + } + + fn user_key() -> WireUserKey { + WireUserKey { + user_id: 9, + user_created_at: 1_760_300_000_000_000, + } + } + + fn options() -> WireChangeOptions { + let mut buf = BytesMut::new(); + let value = 1_000u64.to_le_bytes(); + crate::primitives::user_headers::encode_user_headers( + &[(2u8, b"message_expiry".as_slice(), 8u8, value.as_slice())], + &mut buf, + ); + WireChangeOptions::Readable { + explicit: WireOptions::from_slice(&buf).unwrap(), + derived: WireOptions::empty(), + } + } + + fn permissions() -> WirePermissions { + WirePermissions { + global: WireGlobalPermissions { + manage_servers: false, + read_servers: true, + manage_users: false, + read_users: true, + manage_streams: false, + read_streams: true, + manage_topics: false, + read_topics: true, + poll_messages: true, + send_messages: false, + }, + streams: vec![WireStreamPermissions { + stream_id: 7, + manage_stream: false, + read_stream: true, + manage_topics: false, + read_topics: true, + poll_messages: true, + send_messages: false, + topics: vec![WireTopicPermissions { + topic_id: 3, + manage_topic: false, + read_topic: true, + poll_messages: true, + send_messages: false, + }], + }], + } + } + + fn stream_body() -> WireStreamBody { + WireStreamBody { + key: stream_key(), + created_at: 1_760_356_800_123_456, + name: WireName::new("orders").unwrap(), + options: options(), + } + } + + fn topic_body() -> WireTopicBody { + WireTopicBody { + key: topic_key(), + created_at: 1_760_356_801_000_000, + name: WireName::new("events").unwrap(), + partitions_count: 8, + options: options(), + } + } + + fn user_body(permissions: Option<WirePermissions>) -> WireUserBody { + WireUserBody { + key: user_key(), + username: WireName::new("alice").unwrap(), + status: 1, + permissions, + options: WireChangeOptions::Readable { + explicit: WireOptions::empty(), + derived: WireOptions::empty(), + }, + } + } + + fn sample_changes() -> Vec<WireMetadataChange> { + let name = || WireName::new("audit").unwrap(); + vec![ + WireMetadataChange::StreamCreated(stream_body()), + WireMetadataChange::StreamUpdated(WireStreamBody { + options: WireChangeOptions::Unreadable, + ..stream_body() + }), + WireMetadataChange::StreamDeleted(WireStreamDeletedBody { + key: stream_key(), + created_at: 1, + name: name(), + requested_op: Some(4140), + }), + WireMetadataChange::TopicCreated(topic_body()), + WireMetadataChange::TopicUpdated(topic_body()), + WireMetadataChange::TopicDeleted(WireTopicDeletedBody { + key: topic_key(), + created_at: 2, + name: name(), + requested_op: None, + }), + WireMetadataChange::PartitionsChanged(WirePartitionsChangedBody { + topic: topic_key(), + partitions_count: 5, + removed_partition_ids: vec![5, 6, 7], + requested_op: Some(4200), + }), + WireMetadataChange::ConsumerGroupCreated(WireConsumerGroupCreatedBody { + topic: topic_key(), + group_id: 0, + name: name(), + consumer_groups_count: 1, + }), + WireMetadataChange::ConsumerGroupDeleted(WireConsumerGroupDeletedBody { + topic: topic_key(), + group_id: 0, + name: name(), + consumer_groups_count: 0, + requested_op: Some(4300), + }), + WireMetadataChange::ConsumerGroupMemberJoined(WireMemberJoinedBody { + topic: topic_key(), + group_id: 1, + member_id: 2, + client_id: u128::MAX - 1, + members_count: 2, + partition_ids: vec![], + }), + WireMetadataChange::ConsumerGroupMemberLeft(WireMemberLeftBody { + topic: topic_key(), + group_id: 1, + member_id: 2, + client_id: 77, + members_count: 1, + reason: WireMemberLeftReason::SessionEnded, + }), + WireMetadataChange::ConsumerGroupPartitionHandedOver(WirePartitionHandedOverBody { + topic: topic_key(), + group_id: 1, + partition_id: 4, + member_id: None, + }), + WireMetadataChange::UserCreated(user_body(Some(permissions()))), + WireMetadataChange::UserUpdated(user_body(None)), + WireMetadataChange::UserDeleted(WireUserIdentityBody { + key: user_key(), + username: WireName::new("alice").unwrap(), + }), + WireMetadataChange::UserPasswordChanged(WireUserIdentityBody { + key: user_key(), + username: WireName::new("alice").unwrap(), + }), + WireMetadataChange::UserPermissionsUpdated(WireUserPermissionsBody { + key: user_key(), + username: WireName::new("alice").unwrap(), + permissions: Some(permissions()), + reason: WirePermissionsUpdatedReason::Updated, + }), + WireMetadataChange::PersonalAccessTokenCreated(WireTokenBody { + user: user_key(), + name: WireName::new("ci-token").unwrap(), + expiry: 0, + }), + WireMetadataChange::PersonalAccessTokenDeleted(WireTokenDeletedBody { + user: user_key(), + name: WireName::new("ci-token").unwrap(), + reason: WireTokenDeletedReason::Deleted, + }), + WireMetadataChange::SessionRegistered(WireSessionBody { + user: user_key(), + client_id: 1 << 100, + session: 4, + }), + WireMetadataChange::SessionClosed(WireSessionClosedBody { + user: user_key(), + client_id: 1 << 100, + session: 4, + reason: WireSessionClosedReason::Logout, + }), + ] + } + + fn sample_items() -> Vec<WireMetadataCurrentItem> { + vec![ + WireMetadataCurrentItem::Stream(stream_body()), + WireMetadataCurrentItem::Topic(WireTopicItemBody { + key: topic_key(), + created_at: 3, + name: WireName::new("events").unwrap(), + options: options(), + }), + WireMetadataCurrentItem::Partitions(WirePartitionsItemBody { + topic: topic_key(), + partitions_count: 12, + }), + WireMetadataCurrentItem::ConsumerGroup(WireConsumerGroupItemBody { + topic: topic_key(), + group_id: 4, + name: WireName::new("readers").unwrap(), + }), + WireMetadataCurrentItem::ConsumerGroupMember(WireMemberItemBody { + topic: topic_key(), + group_id: 4, + member_id: 1, + client_id: 12, + partition_ids: vec![0, 2], + }), + WireMetadataCurrentItem::User(user_body(Some(permissions()))), + WireMetadataCurrentItem::PersonalAccessToken(WireTokenBody { + user: user_key(), + name: WireName::new("ci-token").unwrap(), + expiry: 1_760_400_000_000_000, + }), + WireMetadataCurrentItem::Session(WireSessionBody { + user: user_key(), + client_id: 5, + session: 6, + }), + ] + } +}
diff --git a/core/binary_protocol/src/primitives/metadata_changes/mod.rs b/core/binary_protocol/src/primitives/metadata_changes/mod.rs new file mode 100644 index 0000000..ce508f1 --- /dev/null +++ b/core/binary_protocol/src/primitives/metadata_changes/mod.rs
@@ -0,0 +1,454 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Wire codes, record and item bodies, the change cursor and the current +//! state page token of the metadata change log. +//! +//! A record body is encoded once on the server and copied verbatim into every +//! answer, so decoding is strict: unknown reason codes, a known kind under the +//! wrong group, presence bytes other than 0 and 1 and trailing bytes are all +//! refused. A kind or group this build does not know is not an error, the +//! caller keeps it with its raw body. + +mod body; +mod position; + +pub use body::{ + WireChangeOptions, WireConsumerGroupCreatedBody, WireConsumerGroupDeletedBody, + WireConsumerGroupItemBody, WireMemberItemBody, WireMemberJoinedBody, WireMemberLeftBody, + WireMetadataChange, WireMetadataCurrentItem, WirePartitionHandedOverBody, + WirePartitionsChangedBody, WirePartitionsItemBody, WireSessionBody, WireSessionClosedBody, + WireStreamBody, WireStreamDeletedBody, WireStreamKey, WireTokenBody, WireTokenDeletedBody, + WireTopicBody, WireTopicDeletedBody, WireTopicItemBody, WireTopicKey, WireUserBody, + WireUserIdentityBody, WireUserKey, WireUserPermissionsBody, +}; +pub use position::{ + MAX_PAGE_TOKEN_KEY_SIZE, MAX_PAGE_TOKEN_SIZE, METADATA_CHANGE_CURSOR_SIZE, + WireMetadataChangeCursor, WireMetadataCurrentPageToken, +}; + +use crate::WireDecode; +use crate::WireError; +use crate::codec::{bounded_capacity, read_u8, read_u32_le, read_u64_le}; +use crate::primitives::identifier::WireName; +use bytes::{BufMut, BytesMut}; +use std::borrow::Cow; + +macro_rules! wire_code_enum { + ( + $(#[$meta:meta])* + $name:ident { + $($(#[$variant_meta:meta])* $variant:ident = $code:literal),+ $(,)? + } + ) => { + $(#[$meta])* + #[repr(u8)] + #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] + pub enum $name { + $($(#[$variant_meta])* $variant = $code),+ + } + + impl $name { + /// Decode from the wire code. + /// + /// # Errors + /// Returns `WireError::UnknownDiscriminant` for a code this build + /// does not know. + pub const fn from_code(code: u8) -> Result<Self, WireError> { + match code { + $($code => Ok(Self::$variant),)+ + value => Err(WireError::UnknownDiscriminant { + type_name: stringify!($name), + value, + offset: 0, + }), + } + } + + /// The wire code. + #[must_use] + pub const fn as_u8(self) -> u8 { + self as u8 + } + } + }; +} + +wire_code_enum! { + /// Family of change records and the unit of filtering. Codes 9 to 127 are + /// reserved for future groups, 128 to 255 for downstream extensions. + /// + /// A page token holds the groups in an 8 bit mask, so a ninth group needs + /// a new token format. + WireMetadataChangeGroup { + Streams = 1, + Topics = 2, + Partitions = 3, + ConsumerGroups = 4, + ConsumerGroupMembers = 5, + Users = 6, + PersonalAccessTokens = 7, + Sessions = 8, + } +} + +impl WireMetadataChangeGroup { + /// Bit of this group in a page token's group mask. + #[must_use] + pub const fn mask_bit(self) -> u8 { + 1 << (self as u8 - 1) + } +} + +wire_code_enum! { + /// Kind of a change record. Existing kind bodies never change in place, a + /// new layout is a new kind. + WireMetadataChangeKind { + StreamCreated = 1, + StreamUpdated = 2, + StreamDeleted = 3, + TopicCreated = 10, + TopicUpdated = 11, + TopicDeleted = 12, + PartitionsChanged = 20, + ConsumerGroupCreated = 30, + ConsumerGroupDeleted = 31, + ConsumerGroupMemberJoined = 40, + ConsumerGroupMemberLeft = 41, + ConsumerGroupPartitionHandedOver = 42, + UserCreated = 50, + UserUpdated = 51, + UserDeleted = 52, + UserPasswordChanged = 53, + UserPermissionsUpdated = 54, + PersonalAccessTokenCreated = 60, + PersonalAccessTokenDeleted = 61, + SessionRegistered = 70, + SessionClosed = 71, + } +} + +impl WireMetadataChangeKind { + /// The group every record of this kind belongs to. + #[must_use] + pub const fn group(self) -> WireMetadataChangeGroup { + match self { + Self::StreamCreated | Self::StreamUpdated | Self::StreamDeleted => { + WireMetadataChangeGroup::Streams + } + Self::TopicCreated | Self::TopicUpdated | Self::TopicDeleted => { + WireMetadataChangeGroup::Topics + } + Self::PartitionsChanged => WireMetadataChangeGroup::Partitions, + Self::ConsumerGroupCreated | Self::ConsumerGroupDeleted => { + WireMetadataChangeGroup::ConsumerGroups + } + Self::ConsumerGroupMemberJoined + | Self::ConsumerGroupMemberLeft + | Self::ConsumerGroupPartitionHandedOver => { + WireMetadataChangeGroup::ConsumerGroupMembers + } + Self::UserCreated + | Self::UserUpdated + | Self::UserDeleted + | Self::UserPasswordChanged + | Self::UserPermissionsUpdated => WireMetadataChangeGroup::Users, + Self::PersonalAccessTokenCreated | Self::PersonalAccessTokenDeleted => { + WireMetadataChangeGroup::PersonalAccessTokens + } + Self::SessionRegistered | Self::SessionClosed => WireMetadataChangeGroup::Sessions, + } + } +} + +wire_code_enum! { + /// Why a consumer group member was removed. + WireMemberLeftReason { + Left = 0, + SessionEnded = 1, + } +} + +wire_code_enum! { + /// Why a personal access token was removed. + WireTokenDeletedReason { + Deleted = 0, + Expired = 1, + } +} + +wire_code_enum! { + /// Why a session was closed. + WireSessionClosedReason { + Logout = 0, + Expired = 1, + /// The user was deactivated or deleted. + Revoked = 2, + } +} + +wire_code_enum! { + /// Why a user's public permission set changed. + WirePermissionsUpdatedReason { + Updated = 0, + ResourceDeleted = 1, + } +} + +wire_code_enum! { + /// Outcome of a change log query. + WireMetadataChangeStatus { + Ok = 0, + Behind = 1, + Expired = 2, + Reset = 3, + } +} + +wire_code_enum! { + /// Whether an answer holds change records or current state items. + WireMetadataChangePhase { + Changes = 0, + Current = 1, + } +} + +wire_code_enum! { + /// Where a change log query starts. + WireMetadataChangeStartMode { + Earliest = 0, + Latest = 1, + After = 2, + Current = 3, + } +} + +/// Read a presence or boolean byte, refusing anything but 0 and 1. +pub(crate) fn read_flag(buf: &[u8], offset: usize) -> Result<bool, WireError> { + match read_u8(buf, offset)? { + 0 => Ok(false), + 1 => Ok(true), + value => Err(WireError::UnknownDiscriminant { + type_name: "flag", + value, + offset, + }), + } +} + +pub(crate) fn put_flag(buf: &mut BytesMut, value: bool) { + buf.put_u8(u8::from(value)); +} + +pub(crate) fn read_wire_name(buf: &[u8], offset: usize) -> Result<(WireName, usize), WireError> { + let rest = buf.get(offset..).ok_or(WireError::UnexpectedEof { + offset, + need: 1, + have: 0, + })?; + WireName::decode(rest) +} + +pub(crate) const fn optional_u64_size(value: Option<u64>) -> usize { + if value.is_some() { 9 } else { 1 } +} + +pub(crate) fn put_optional_u64(buf: &mut BytesMut, value: Option<u64>) { + put_flag(buf, value.is_some()); + if let Some(value) = value { + buf.put_u64_le(value); + } +} + +pub(crate) fn read_optional_u64( + buf: &[u8], + offset: usize, +) -> Result<(Option<u64>, usize), WireError> { + if read_flag(buf, offset)? { + Ok((Some(read_u64_le(buf, offset + 1)?), 9)) + } else { + Ok((None, 1)) + } +} + +pub(crate) const fn optional_u32_size(value: Option<u32>) -> usize { + if value.is_some() { 5 } else { 1 } +} + +pub(crate) fn put_optional_u32(buf: &mut BytesMut, value: Option<u32>) { + put_flag(buf, value.is_some()); + if let Some(value) = value { + buf.put_u32_le(value); + } +} + +pub(crate) fn read_optional_u32( + buf: &[u8], + offset: usize, +) -> Result<(Option<u32>, usize), WireError> { + if read_flag(buf, offset)? { + Ok((Some(read_u32_le(buf, offset + 1)?), 5)) + } else { + Ok((None, 1)) + } +} + +pub(crate) const fn partition_ids_size(ids: &[u32]) -> usize { + 4 + ids.len() * 4 +} + +/// Encode `[count:u32][id:u32]*`. Callers hold ascending distinct ids. +pub(crate) fn put_partition_ids(buf: &mut BytesMut, ids: &[u32]) { + // A topic holds at most `u32::MAX` partitions, so the count fits. + #[allow(clippy::cast_possible_truncation)] + buf.put_u32_le(ids.len() as u32); + for &id in ids { + buf.put_u32_le(id); + } +} + +/// Decode `[count:u32][id:u32]*`, refusing ids that are not strictly +/// ascending. +pub(crate) fn read_partition_ids( + buf: &[u8], + offset: usize, +) -> Result<(Vec<u32>, usize), WireError> { + let count = read_u32_le(buf, offset)? as usize; + let mut position = offset + 4; + let remaining = buf.len().saturating_sub(position); + let mut ids = Vec::with_capacity(bounded_capacity(count, remaining, 4)); + for _ in 0..count { + let id = read_u32_le(buf, position)?; + if ids.last().is_some_and(|&previous| previous >= id) { + return Err(WireError::Validation(Cow::Borrowed( + "partition ids must be strictly ascending", + ))); + } + ids.push(id); + position += 4; + } + Ok((ids, position - offset)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn given_every_group_when_decoding_its_code_should_round_trip_and_set_one_mask_bit() { + let mut mask = 0u8; + for code in 1..=8u8 { + let group = WireMetadataChangeGroup::from_code(code).unwrap(); + assert_eq!(group.as_u8(), code); + assert_eq!(mask & group.mask_bit(), 0); + mask |= group.mask_bit(); + } + assert_eq!(mask, u8::MAX); + } + + #[test] + fn given_reserved_group_codes_when_decoding_should_refuse() { + for code in [0u8, 9, 127, 128, 255] { + assert!(WireMetadataChangeGroup::from_code(code).is_err()); + } + } + + #[test] + fn given_every_kind_when_decoding_its_code_should_round_trip() { + let codes = [ + 1u8, 2, 3, 10, 11, 12, 20, 30, 31, 40, 41, 42, 50, 51, 52, 53, 54, 60, 61, 70, 71, + ]; + for code in codes { + let kind = WireMetadataChangeKind::from_code(code).unwrap(); + assert_eq!(kind.as_u8(), code); + } + for code in 0..=u8::MAX { + assert_eq!( + WireMetadataChangeKind::from_code(code).is_ok(), + codes.contains(&code) + ); + } + } + + #[test] + fn given_kind_codes_when_mapping_groups_should_follow_the_catalog_decades() { + for code in 0..=u8::MAX { + if let Ok(kind) = WireMetadataChangeKind::from_code(code) { + let expected = match code { + 1..=3 => 1, + 10..=12 => 2, + 20 => 3, + 30 | 31 => 4, + 40..=42 => 5, + 50..=54 => 6, + 60 | 61 => 7, + _ => 8, + }; + assert_eq!(kind.group().as_u8(), expected, "kind {code}"); + } + } + } + + #[test] + fn given_unknown_reason_codes_when_decoding_should_refuse() { + assert!(WireMemberLeftReason::from_code(2).is_err()); + assert!(WireTokenDeletedReason::from_code(2).is_err()); + assert!(WireSessionClosedReason::from_code(3).is_err()); + assert!(WirePermissionsUpdatedReason::from_code(2).is_err()); + assert!(WireMetadataChangeStatus::from_code(4).is_err()); + assert!(WireMetadataChangePhase::from_code(2).is_err()); + assert!(WireMetadataChangeStartMode::from_code(4).is_err()); + } + + #[test] + fn given_flag_byte_above_one_when_reading_should_refuse() { + assert!(!read_flag(&[0], 0).unwrap()); + assert!(read_flag(&[1], 0).unwrap()); + assert!(read_flag(&[2], 0).is_err()); + assert!(read_flag(&[], 0).is_err()); + } + + #[test] + fn given_partition_ids_when_round_tripping_should_preserve_order() { + let ids = [0u32, 3, 7, u32::MAX]; + let mut buf = BytesMut::new(); + put_partition_ids(&mut buf, &ids); + assert_eq!(buf.len(), partition_ids_size(&ids)); + let (decoded, consumed) = read_partition_ids(&buf, 0).unwrap(); + assert_eq!(decoded, ids); + assert_eq!(consumed, buf.len()); + } + + #[test] + fn given_unsorted_or_duplicate_partition_ids_when_decoding_should_refuse() { + for ids in [[3u32, 1], [2, 2]] { + let mut buf = BytesMut::new(); + buf.put_u32_le(2); + for id in ids { + buf.put_u32_le(id); + } + assert!(read_partition_ids(&buf, 0).is_err()); + } + } + + #[test] + fn given_huge_partition_count_when_decoding_should_fail_without_allocating_it() { + let mut buf = BytesMut::new(); + buf.put_u32_le(u32::MAX); + assert!(read_partition_ids(&buf, 0).is_err()); + } +}
diff --git a/core/binary_protocol/src/primitives/metadata_changes/position.rs b/core/binary_protocol/src/primitives/metadata_changes/position.rs new file mode 100644 index 0000000..879007f --- /dev/null +++ b/core/binary_protocol/src/primitives/metadata_changes/position.rs
@@ -0,0 +1,384 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use super::{WireMetadataChangeGroup, read_wire_name}; +use crate::WireError; +use crate::codec::{WireDecode, WireEncode, read_bytes, read_u8, read_u16_le, read_u32_le}; +use crate::codec::{read_u64_le, read_u128_le}; +use bytes::{BufMut, Bytes, BytesMut}; +use std::borrow::Cow; + +/// Encoded size of a change cursor. +pub const METADATA_CHANGE_CURSOR_SIZE: usize = 44; + +/// Longest page token key: a user id and a personal access token name. +pub const MAX_PAGE_TOKEN_KEY_SIZE: usize = 4 + 1 + 255; + +/// Longest decoded page token payload. +pub const MAX_PAGE_TOKEN_SIZE: usize = + METADATA_CHANGE_CURSOR_SIZE + 8 + 1 + 1 + 2 + MAX_PAGE_TOKEN_KEY_SIZE; + +/// Position of the last record a caller consumed or scanned. +/// +/// `(op, checksum, ordinal) = (0, 0, 0)` names the beginning of the history. +/// +/// Wire format: `[log_id:u128][op:u64][checksum:u128][ordinal:u32]` +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)] +pub struct WireMetadataChangeCursor { + pub log_id: u128, + pub op: u64, + pub checksum: u128, + pub ordinal: u32, +} + +impl WireMetadataChangeCursor { + /// The cursor before the first record of `log_id`. + #[must_use] + pub const fn zero(log_id: u128) -> Self { + Self { + log_id, + op: 0, + checksum: 0, + ordinal: 0, + } + } + + /// Whether the cursor names the beginning of its history. + #[must_use] + pub const fn is_zero(&self) -> bool { + self.op == 0 + } + + /// Op 0 carries no checksum and no ordinal. + #[must_use] + pub const fn is_canonical(&self) -> bool { + self.op != 0 || (self.checksum == 0 && self.ordinal == 0) + } +} + +impl WireEncode for WireMetadataChangeCursor { + fn encoded_size(&self) -> usize { + METADATA_CHANGE_CURSOR_SIZE + } + + fn encode(&self, buf: &mut BytesMut) { + buf.put_u128_le(self.log_id); + buf.put_u64_le(self.op); + buf.put_u128_le(self.checksum); + buf.put_u32_le(self.ordinal); + } +} + +impl WireDecode for WireMetadataChangeCursor { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let cursor = Self { + log_id: read_u128_le(buf, 0)?, + op: read_u64_le(buf, 16)?, + checksum: read_u128_le(buf, 24)?, + ordinal: read_u32_le(buf, 40)?, + }; + if !cursor.is_canonical() { + return Err(WireError::Validation(Cow::Borrowed( + "a cursor at op 0 carries no checksum and no ordinal", + ))); + } + Ok((cursor, METADATA_CHANGE_CURSOR_SIZE)) + } +} + +/// Continuation of a current state scan. +/// +/// The page token names the anchor captured before the first page, the +/// highest head any previous page reported, the requested groups and the key +/// of the last item returned, so any node that covers `required_head` can +/// serve the next page. +/// +/// Wire format: +/// ```text +/// [anchor:44][required_head:u64][groups_mask:u8][group:u8][key_len:u16][last_key] +/// ``` +/// +/// Keys are little-endian ids: streams `[stream_id:u32]`, topics and +/// partitions `[stream_id:u32][topic_id:u32]`, consumer groups add +/// `[group_id:u32]`, members add `[member_id:u32]`, users `[user_id:u32]`, +/// personal access tokens `[user_id:u32][name]`, sessions `[epoch:u64]`. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireMetadataCurrentPageToken { + pub anchor: WireMetadataChangeCursor, + pub required_head: u64, + pub groups_mask: u8, + pub group: WireMetadataChangeGroup, + pub last_key: Bytes, +} + +impl WireMetadataCurrentPageToken { + const FIXED_SIZE: usize = METADATA_CHANGE_CURSOR_SIZE + 8 + 1 + 1 + 2; + + /// Check the token's own invariants: the group is selected, the key has + /// the exact width of that group and the head covers the anchor. + /// + /// # Errors + /// Returns `WireError::Validation` naming the first broken invariant. + pub fn validate(&self) -> Result<(), WireError> { + if self.groups_mask & self.group.mask_bit() == 0 { + return Err(WireError::Validation(Cow::Borrowed( + "page token group is not in its group mask", + ))); + } + if self.required_head < self.anchor.op { + return Err(WireError::Validation(Cow::Borrowed( + "page token head is below its anchor", + ))); + } + if !self.anchor.is_canonical() { + return Err(WireError::Validation(Cow::Borrowed( + "page token anchor is not canonical", + ))); + } + if self.last_key.len() > MAX_PAGE_TOKEN_KEY_SIZE { + return Err(WireError::Validation(Cow::Borrowed( + "page token key is too long", + ))); + } + if !key_fits_group(self.group, &self.last_key) { + return Err(WireError::Validation(Cow::Owned(format!( + "page token key of {} bytes does not fit group {}", + self.last_key.len(), + self.group.as_u8() + )))); + } + Ok(()) + } +} + +impl WireEncode for WireMetadataCurrentPageToken { + fn encoded_size(&self) -> usize { + Self::FIXED_SIZE + self.last_key.len() + } + + fn encode(&self, buf: &mut BytesMut) { + self.anchor.encode(buf); + buf.put_u64_le(self.required_head); + buf.put_u8(self.groups_mask); + buf.put_u8(self.group.as_u8()); + // Keys are at most `MAX_PAGE_TOKEN_KEY_SIZE` bytes, see `validate`. + #[allow(clippy::cast_possible_truncation)] + buf.put_u16_le(self.last_key.len() as u16); + buf.put_slice(&self.last_key); + } +} + +impl WireDecode for WireMetadataCurrentPageToken { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let (anchor, mut position) = WireMetadataChangeCursor::decode(buf)?; + let required_head = read_u64_le(buf, position)?; + position += 8; + let groups_mask = read_u8(buf, position)?; + position += 1; + let group_offset = position; + let group = WireMetadataChangeGroup::from_code(read_u8(buf, position)?).map_err(|_| { + WireError::Validation(Cow::Borrowed("page token names an unknown group")) + })?; + position += 1; + let key_length = usize::from(read_u16_le(buf, position)?); + position += 2; + if key_length > MAX_PAGE_TOKEN_KEY_SIZE { + return Err(WireError::Validation(Cow::Owned(format!( + "page token key of {key_length} bytes at offset {group_offset} is too long" + )))); + } + let last_key = Bytes::copy_from_slice(read_bytes(buf, position, key_length)?); + position += key_length; + let token = Self { + anchor, + required_head, + groups_mask, + group, + last_key, + }; + token.validate()?; + Ok((token, position)) + } +} + +fn key_fits_group(group: WireMetadataChangeGroup, key: &[u8]) -> bool { + match group { + WireMetadataChangeGroup::Streams | WireMetadataChangeGroup::Users => key.len() == 4, + WireMetadataChangeGroup::Topics + | WireMetadataChangeGroup::Partitions + | WireMetadataChangeGroup::Sessions => key.len() == 8, + WireMetadataChangeGroup::ConsumerGroups => key.len() == 12, + WireMetadataChangeGroup::ConsumerGroupMembers => key.len() == 16, + WireMetadataChangeGroup::PersonalAccessTokens => { + key.len() > 4 + && read_wire_name(key, 4).is_ok_and(|(_, consumed)| 4 + consumed == key.len()) + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn cursor() -> WireMetadataChangeCursor { + WireMetadataChangeCursor { + log_id: 0x0d4b_8f2a_6e1c_93d7, + op: 4187, + checksum: 0xf2a4_c6e8_b0d1_f359, + ordinal: 1, + } + } + + fn token(group: WireMetadataChangeGroup, last_key: &[u8]) -> WireMetadataCurrentPageToken { + WireMetadataCurrentPageToken { + anchor: cursor(), + required_head: 4200, + groups_mask: u8::MAX, + group, + last_key: Bytes::copy_from_slice(last_key), + } + } + + fn token_key(user_id: u32, name: &str) -> Vec<u8> { + let mut key = user_id.to_le_bytes().to_vec(); + key.push(u8::try_from(name.len()).unwrap()); + key.extend_from_slice(name.as_bytes()); + key + } + + #[test] + fn given_cursor_when_round_tripping_should_use_44_little_endian_bytes() { + let bytes = cursor().to_bytes(); + assert_eq!(bytes.len(), METADATA_CHANGE_CURSOR_SIZE); + assert_eq!(&bytes[16..24], &4187u64.to_le_bytes()); + let (decoded, consumed) = WireMetadataChangeCursor::decode(&bytes).unwrap(); + assert_eq!(decoded, cursor()); + assert_eq!(consumed, METADATA_CHANGE_CURSOR_SIZE); + } + + #[test] + fn given_zero_cursor_when_round_tripping_should_keep_its_log_id() { + let zero = WireMetadataChangeCursor::zero(42); + assert!(zero.is_zero()); + let (decoded, _) = WireMetadataChangeCursor::decode(&zero.to_bytes()).unwrap(); + assert_eq!(decoded, zero); + } + + #[test] + fn given_op_zero_with_checksum_or_ordinal_when_decoding_should_refuse() { + for (checksum, ordinal) in [(1u128, 0u32), (0, 1)] { + let cursor = WireMetadataChangeCursor { + log_id: 1, + op: 0, + checksum, + ordinal, + }; + assert!(!cursor.is_canonical()); + assert!(WireMetadataChangeCursor::decode(&cursor.to_bytes()).is_err()); + } + } + + #[test] + fn given_truncated_cursor_when_decoding_should_refuse() { + let bytes = cursor().to_bytes(); + assert!(WireMetadataChangeCursor::decode(&bytes[..43]).is_err()); + } + + #[test] + fn given_every_group_key_width_when_round_tripping_page_tokens_should_preserve_them() { + let cases = [ + (WireMetadataChangeGroup::Streams, vec![0u8; 4]), + (WireMetadataChangeGroup::Topics, vec![1u8; 8]), + (WireMetadataChangeGroup::Partitions, vec![2u8; 8]), + (WireMetadataChangeGroup::ConsumerGroups, vec![3u8; 12]), + (WireMetadataChangeGroup::ConsumerGroupMembers, vec![4u8; 16]), + (WireMetadataChangeGroup::Users, vec![5u8; 4]), + ( + WireMetadataChangeGroup::PersonalAccessTokens, + token_key(9, "ci"), + ), + (WireMetadataChangeGroup::Sessions, vec![6u8; 8]), + ]; + for (group, key) in cases { + let original = token(group, &key); + let bytes = original.to_bytes(); + let (decoded, consumed) = WireMetadataCurrentPageToken::decode(&bytes).unwrap(); + assert_eq!(decoded, original); + assert_eq!(consumed, bytes.len()); + } + } + + #[test] + fn given_longest_token_name_when_encoding_should_fit_the_maximum_payload() { + let longest = "n".repeat(255); + let original = token( + WireMetadataChangeGroup::PersonalAccessTokens, + &token_key(1, &longest), + ); + assert_eq!(original.to_bytes().len(), MAX_PAGE_TOKEN_SIZE); + assert!(WireMetadataCurrentPageToken::decode(&original.to_bytes()).is_ok()); + } + + #[test] + fn given_wrong_key_width_when_decoding_page_token_should_refuse() { + for (group, width) in [ + (WireMetadataChangeGroup::Streams, 8), + (WireMetadataChangeGroup::Topics, 4), + (WireMetadataChangeGroup::ConsumerGroups, 16), + (WireMetadataChangeGroup::Sessions, 4), + (WireMetadataChangeGroup::PersonalAccessTokens, 4), + ] { + let bytes = token(group, &vec![0u8; width]).to_bytes(); + assert!(WireMetadataCurrentPageToken::decode(&bytes).is_err()); + } + let mut bad_name = token_key(1, "ci"); + bad_name.push(0); + let bytes = token(WireMetadataChangeGroup::PersonalAccessTokens, &bad_name).to_bytes(); + assert!(WireMetadataCurrentPageToken::decode(&bytes).is_err()); + } + + #[test] + fn given_group_outside_mask_when_decoding_page_token_should_refuse() { + let mut original = token(WireMetadataChangeGroup::Topics, &[0u8; 8]); + original.groups_mask = WireMetadataChangeGroup::Streams.mask_bit(); + assert!(WireMetadataCurrentPageToken::decode(&original.to_bytes()).is_err()); + } + + #[test] + fn given_head_below_anchor_when_decoding_page_token_should_refuse() { + let mut original = token(WireMetadataChangeGroup::Streams, &[0u8; 4]); + original.required_head = original.anchor.op - 1; + assert!(WireMetadataCurrentPageToken::decode(&original.to_bytes()).is_err()); + } + + #[test] + fn given_unknown_group_code_when_decoding_page_token_should_refuse() { + let mut bytes = token(WireMetadataChangeGroup::Streams, &[0u8; 4]) + .to_bytes() + .to_vec(); + bytes[METADATA_CHANGE_CURSOR_SIZE + 9] = 9; + assert!(WireMetadataCurrentPageToken::decode(&bytes).is_err()); + } + + #[test] + fn given_truncated_page_token_when_decoding_should_refuse_every_prefix() { + let bytes = token(WireMetadataChangeGroup::Users, &[1u8; 4]).to_bytes(); + for length in 0..bytes.len() { + assert!(WireMetadataCurrentPageToken::decode(&bytes[..length]).is_err()); + } + } +}
diff --git a/core/binary_protocol/src/primitives/mod.rs b/core/binary_protocol/src/primitives/mod.rs index 07f1075..7a031d9 100644 --- a/core/binary_protocol/src/primitives/mod.rs +++ b/core/binary_protocol/src/primitives/mod.rs
@@ -20,6 +20,7 @@ pub mod ack_level; pub mod consumer; pub mod identifier; +pub mod metadata_changes; pub mod options; pub mod partition_assignment; pub mod partition_history;
diff --git a/core/binary_protocol/src/requests/system/get_metadata_changes.rs b/core/binary_protocol/src/requests/system/get_metadata_changes.rs new file mode 100644 index 0000000..c353602 --- /dev/null +++ b/core/binary_protocol/src/requests/system/get_metadata_changes.rs
@@ -0,0 +1,322 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use crate::WireError; +use crate::codec::{WireDecode, WireEncode, read_bytes, read_u8, read_u16_le, read_u32_le}; +use crate::primitives::metadata_changes::{ + METADATA_CHANGE_CURSOR_SIZE, WireMetadataChangeCursor, WireMetadataChangeStartMode, + WireMetadataCurrentPageToken, +}; +use crate::primitives::metadata_changes::{put_optional_u64, read_optional_u64}; +use bytes::{BufMut, BytesMut}; +use std::borrow::Cow; + +/// Where a change log query starts. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum WireMetadataChangeStart { + /// The oldest operation the requested classes still hold completely. + Earliest, + /// No records, a cursor at the newest retained record. + Latest, + /// Records after a cursor. + After(WireMetadataChangeCursor), + /// Current state pages, the first one without a token. + Current(Option<WireMetadataCurrentPageToken>), +} + +/// `GetMetadataChanges` request. +/// +/// Wire format: +/// ```text +/// [start:u8] +/// [log_id:u128][op:u64][checksum:u128][ordinal:u32] cursor, all zero unless start = 2 +/// [has_since:u8][since:u64 if 1] +/// [has_until:u8][until:u64 if 1] +/// [limit:u32] +/// [groups_count:u8][group:u8]* 0 entries = all groups +/// [token_len:u16][token] 0 unless start = 3 with a token +/// ``` +/// +/// Structural rules are enforced here. Which groups exist, the limit range +/// and the time bounds are checked where the request is served. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GetMetadataChangesRequest { + pub start: WireMetadataChangeStart, + pub since: Option<u64>, + pub until: Option<u64>, + pub limit: u32, + pub groups: Vec<u8>, +} + +impl GetMetadataChangesRequest { + const fn start_mode(&self) -> WireMetadataChangeStartMode { + match self.start { + WireMetadataChangeStart::Earliest => WireMetadataChangeStartMode::Earliest, + WireMetadataChangeStart::Latest => WireMetadataChangeStartMode::Latest, + WireMetadataChangeStart::After(_) => WireMetadataChangeStartMode::After, + WireMetadataChangeStart::Current(_) => WireMetadataChangeStartMode::Current, + } + } + + /// Group codes the request can carry, at most `u8::MAX` of them. + fn encoded_groups(&self) -> &[u8] { + &self.groups[..self.groups.len().min(usize::from(u8::MAX))] + } + + const fn token(&self) -> Option<&WireMetadataCurrentPageToken> { + match &self.start { + WireMetadataChangeStart::Current(Some(token)) => Some(token), + _ => None, + } + } +} + +impl WireEncode for GetMetadataChangesRequest { + fn encoded_size(&self) -> usize { + 1 + METADATA_CHANGE_CURSOR_SIZE + + if self.since.is_some() { 9 } else { 1 } + + if self.until.is_some() { 9 } else { 1 } + + 4 + + 1 + + self.encoded_groups().len() + + 2 + + self.token().map_or(0, WireEncode::encoded_size) + } + + fn encode(&self, buf: &mut BytesMut) { + buf.put_u8(self.start_mode().as_u8()); + let cursor = match &self.start { + WireMetadataChangeStart::After(cursor) => *cursor, + _ => WireMetadataChangeCursor::default(), + }; + cursor.encode(buf); + put_optional_u64(buf, self.since); + put_optional_u64(buf, self.until); + buf.put_u32_le(self.limit); + let groups = self.encoded_groups(); + // `encoded_groups` holds at most `u8::MAX` entries. + #[allow(clippy::cast_possible_truncation)] + buf.put_u8(groups.len() as u8); + buf.put_slice(groups); + match self.token() { + Some(token) => { + // A valid token is at most `MAX_PAGE_TOKEN_SIZE` bytes. + #[allow(clippy::cast_possible_truncation)] + buf.put_u16_le(token.encoded_size() as u16); + token.encode(buf); + } + None => buf.put_u16_le(0), + } + } +} + +impl WireDecode for GetMetadataChangesRequest { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let mode = WireMetadataChangeStartMode::from_code(read_u8(buf, 0)?)?; + let cursor_bytes = read_bytes(buf, 1, METADATA_CHANGE_CURSOR_SIZE)?; + let mut position = 1 + METADATA_CHANGE_CURSOR_SIZE; + let (since, consumed) = read_optional_u64(buf, position)?; + position += consumed; + let (until, consumed) = read_optional_u64(buf, position)?; + position += consumed; + let limit = read_u32_le(buf, position)?; + position += 4; + let groups_count = usize::from(read_u8(buf, position)?); + position += 1; + let groups = read_bytes(buf, position, groups_count)?.to_vec(); + position += groups_count; + let token_length = usize::from(read_u16_le(buf, position)?); + position += 2; + let token_bytes = read_bytes(buf, position, token_length)?; + position += token_length; + if position != buf.len() { + return Err(WireError::Validation(Cow::Owned(format!( + "{} trailing bytes after a metadata changes request", + buf.len() - position + )))); + } + + if mode != WireMetadataChangeStartMode::After && cursor_bytes.iter().any(|&byte| byte != 0) + { + return Err(WireError::Validation(Cow::Borrowed( + "a cursor is allowed only when starting after it", + ))); + } + if mode != WireMetadataChangeStartMode::Current && token_length != 0 { + return Err(WireError::Validation(Cow::Borrowed( + "a page token is allowed only in current mode", + ))); + } + let start = match mode { + WireMetadataChangeStartMode::Earliest => WireMetadataChangeStart::Earliest, + WireMetadataChangeStartMode::Latest => WireMetadataChangeStart::Latest, + WireMetadataChangeStartMode::After => { + WireMetadataChangeStart::After(WireMetadataChangeCursor::decode(cursor_bytes)?.0) + } + WireMetadataChangeStartMode::Current if token_length == 0 => { + WireMetadataChangeStart::Current(None) + } + WireMetadataChangeStartMode::Current => { + let (token, consumed) = WireMetadataCurrentPageToken::decode(token_bytes)?; + if consumed != token_length { + return Err(WireError::Validation(Cow::Borrowed( + "trailing bytes after a page token", + ))); + } + WireMetadataChangeStart::Current(Some(token)) + } + }; + Ok(( + Self { + start, + since, + until, + limit, + groups, + }, + position, + )) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::primitives::metadata_changes::WireMetadataChangeGroup; + use bytes::Bytes; + + fn cursor() -> WireMetadataChangeCursor { + WireMetadataChangeCursor { + log_id: 7, + op: 9, + checksum: 11, + ordinal: 2, + } + } + + fn token() -> WireMetadataCurrentPageToken { + WireMetadataCurrentPageToken { + anchor: cursor(), + required_head: 12, + groups_mask: 0b11, + group: WireMetadataChangeGroup::Topics, + last_key: Bytes::from_static(&[1, 0, 0, 0, 2, 0, 0, 0]), + } + } + + fn request(start: WireMetadataChangeStart) -> GetMetadataChangesRequest { + GetMetadataChangesRequest { + start, + since: Some(1_760_353_200_000_000), + until: None, + limit: 500, + groups: vec![1, 2], + } + } + + #[test] + fn given_every_start_mode_when_round_tripping_should_preserve_every_field() { + for start in [ + WireMetadataChangeStart::Earliest, + WireMetadataChangeStart::Latest, + WireMetadataChangeStart::After(cursor()), + WireMetadataChangeStart::Current(None), + WireMetadataChangeStart::Current(Some(token())), + ] { + let original = request(start); + let bytes = original.to_bytes(); + assert_eq!(bytes.len(), original.encoded_size()); + let (decoded, consumed) = GetMetadataChangesRequest::decode(&bytes).unwrap(); + assert_eq!(decoded, original); + assert_eq!(consumed, bytes.len()); + } + } + + #[test] + fn given_no_groups_and_no_bounds_when_round_tripping_should_preserve_them() { + let original = GetMetadataChangesRequest { + start: WireMetadataChangeStart::Latest, + since: None, + until: None, + limit: 1, + groups: vec![], + }; + let bytes = original.to_bytes(); + assert_eq!(bytes.len(), 1 + 44 + 1 + 1 + 4 + 1 + 2); + assert_eq!( + GetMetadataChangesRequest::decode(&bytes).unwrap().0, + original + ); + } + + #[test] + fn given_trailing_bytes_when_decoding_request_should_refuse() { + let mut bytes = request(WireMetadataChangeStart::Earliest) + .to_bytes() + .to_vec(); + bytes.push(0); + assert!(GetMetadataChangesRequest::decode(&bytes).is_err()); + } + + #[test] + fn given_cursor_bytes_without_after_when_decoding_request_should_refuse() { + let mut bytes = request(WireMetadataChangeStart::Latest).to_bytes().to_vec(); + bytes[5] = 1; + assert!(GetMetadataChangesRequest::decode(&bytes).is_err()); + } + + #[test] + fn given_non_canonical_cursor_when_decoding_request_should_refuse() { + let original = request(WireMetadataChangeStart::After(WireMetadataChangeCursor { + log_id: 1, + op: 0, + checksum: 3, + ordinal: 0, + })); + assert!(GetMetadataChangesRequest::decode(&original.to_bytes()).is_err()); + } + + #[test] + fn given_page_token_outside_current_mode_when_decoding_request_should_refuse() { + let mut bytes = request(WireMetadataChangeStart::Current(Some(token()))) + .to_bytes() + .to_vec(); + bytes[0] = WireMetadataChangeStartMode::Earliest.as_u8(); + assert!(GetMetadataChangesRequest::decode(&bytes).is_err()); + } + + #[test] + fn given_unknown_start_or_presence_byte_when_decoding_request_should_refuse() { + let bytes = request(WireMetadataChangeStart::Earliest) + .to_bytes() + .to_vec(); + let mut unknown_start = bytes.clone(); + unknown_start[0] = 4; + assert!(GetMetadataChangesRequest::decode(&unknown_start).is_err()); + let mut bad_presence = bytes; + bad_presence[1 + 44] = 2; + assert!(GetMetadataChangesRequest::decode(&bad_presence).is_err()); + } + + #[test] + fn given_truncated_request_when_decoding_should_refuse_every_prefix() { + let bytes = request(WireMetadataChangeStart::Current(Some(token()))).to_bytes(); + for length in 0..bytes.len() { + assert!(GetMetadataChangesRequest::decode(&bytes[..length]).is_err()); + } + } +}
diff --git a/core/binary_protocol/src/requests/system/mod.rs b/core/binary_protocol/src/requests/system/mod.rs index 9d5c568..e59a75d 100644 --- a/core/binary_protocol/src/requests/system/mod.rs +++ b/core/binary_protocol/src/requests/system/mod.rs
@@ -22,6 +22,7 @@ pub mod get_clients; pub mod get_cluster_metadata; pub mod get_me; +pub mod get_metadata_changes; pub mod get_snapshot; pub mod get_stats; pub mod ping; @@ -37,6 +38,7 @@ pub use get_clients::GetClientsRequest; pub use get_cluster_metadata::GetClusterMetadataRequest; pub use get_me::GetMeRequest; +pub use get_metadata_changes::{GetMetadataChangesRequest, WireMetadataChangeStart}; pub use get_snapshot::GetSnapshotRequest; pub use get_stats::GetStatsRequest; pub use ping::PingRequest;
diff --git a/core/binary_protocol/src/responses/system/get_metadata_changes.rs b/core/binary_protocol/src/responses/system/get_metadata_changes.rs new file mode 100644 index 0000000..b5a12ec --- /dev/null +++ b/core/binary_protocol/src/responses/system/get_metadata_changes.rs
@@ -0,0 +1,488 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use crate::WireError; +use crate::codec::{ + WireDecode, WireEncode, bounded_capacity, read_bytes, read_u8, read_u16_le, read_u32_le, + read_u64_le, read_u128_le, +}; +use crate::primitives::metadata_changes::{ + METADATA_CHANGE_CURSOR_SIZE, WireMetadataChangeCursor, WireMetadataChangePhase, + WireMetadataChangeStatus, WireMetadataCurrentPageToken, +}; +use crate::primitives::metadata_changes::{ + put_flag, put_optional_u32, read_flag, read_optional_u32, +}; +use bytes::{BufMut, Bytes, BytesMut}; +use std::borrow::Cow; + +/// One change record: the envelope stamped from the prepare header and the +/// body the handler encoded at apply, kept as raw bytes so a client can hold a +/// kind it does not know. +/// +/// Wire format: +/// ```text +/// [op:u64][ordinal:u32][checksum:u128][timestamp:u64] +/// [has_actor:u8][actor_user_id:u32 if 1] +/// [group:u8][kind:u8][body_len:u32][body] +/// ``` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireMetadataChangeRecord { + pub op: u64, + pub ordinal: u32, + pub checksum: u128, + pub timestamp: u64, + pub actor_user_id: Option<u32>, + pub group: u8, + pub kind: u8, + pub body: Bytes, +} + +/// One current state item. +/// +/// Wire format: `[group:u8][body_len:u32][body]` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WireMetadataItem { + pub group: u8, + pub body: Bytes, +} + +/// `GetMetadataChanges` response. +/// +/// Wire format: +/// ```text +/// [status:u8][phase:u8] +/// [has_log_id:u8][log_id:u128 if 1] +/// [oldest_op:u64][retained_since:u64][head_op:u64] +/// [has_next:u8][cursor:44 if 1] +/// [has_more:u8] +/// [records_count:u32][record]* +/// [items_count:u32][item]* +/// [token_len:u16][token] +/// ``` +/// +/// A changes page carries records and no items or token, a current page +/// carries items and a token and no records. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GetMetadataChangesResponse { + pub status: WireMetadataChangeStatus, + pub phase: WireMetadataChangePhase, + pub log_id: Option<u128>, + pub oldest_op: u64, + pub retained_since: u64, + pub head_op: u64, + pub next: Option<WireMetadataChangeCursor>, + pub has_more: bool, + pub records: Vec<WireMetadataChangeRecord>, + pub items: Vec<WireMetadataItem>, + pub current_page_token: Option<WireMetadataCurrentPageToken>, +} + +impl WireMetadataChangeRecord { + const MIN_SIZE: usize = 8 + 4 + 16 + 8 + 1 + 1 + 1 + 4; +} + +impl WireEncode for WireMetadataChangeRecord { + fn encoded_size(&self) -> usize { + Self::MIN_SIZE + if self.actor_user_id.is_some() { 4 } else { 0 } + self.body.len() + } + + fn encode(&self, buf: &mut BytesMut) { + buf.put_u64_le(self.op); + buf.put_u32_le(self.ordinal); + buf.put_u128_le(self.checksum); + buf.put_u64_le(self.timestamp); + put_optional_u32(buf, self.actor_user_id); + buf.put_u8(self.group); + buf.put_u8(self.kind); + // Bodies are bounded by the options budget, far below `u32::MAX`. + #[allow(clippy::cast_possible_truncation)] + buf.put_u32_le(self.body.len() as u32); + buf.put_slice(&self.body); + } +} + +impl WireDecode for WireMetadataChangeRecord { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let op = read_u64_le(buf, 0)?; + let ordinal = read_u32_le(buf, 8)?; + let checksum = read_u128_le(buf, 12)?; + let timestamp = read_u64_le(buf, 28)?; + let (actor_user_id, consumed) = read_optional_u32(buf, 36)?; + let mut position = 36 + consumed; + let group = read_u8(buf, position)?; + let kind = read_u8(buf, position + 1)?; + let body_length = read_u32_le(buf, position + 2)? as usize; + position += 6; + let body = Bytes::copy_from_slice(read_bytes(buf, position, body_length)?); + position += body_length; + Ok(( + Self { + op, + ordinal, + checksum, + timestamp, + actor_user_id, + group, + kind, + body, + }, + position, + )) + } +} + +impl WireEncode for WireMetadataItem { + fn encoded_size(&self) -> usize { + 1 + 4 + self.body.len() + } + + fn encode(&self, buf: &mut BytesMut) { + buf.put_u8(self.group); + // Item bodies are bounded by the options budget. + #[allow(clippy::cast_possible_truncation)] + buf.put_u32_le(self.body.len() as u32); + buf.put_slice(&self.body); + } +} + +impl WireDecode for WireMetadataItem { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let group = read_u8(buf, 0)?; + let body_length = read_u32_le(buf, 1)? as usize; + let body = Bytes::copy_from_slice(read_bytes(buf, 5, body_length)?); + Ok((Self { group, body }, 5 + body_length)) + } +} + +impl WireEncode for GetMetadataChangesResponse { + fn encoded_size(&self) -> usize { + 2 + if self.log_id.is_some() { 17 } else { 1 } + + 24 + + if self.next.is_some() { + 1 + METADATA_CHANGE_CURSOR_SIZE + } else { + 1 + } + + 1 + + 4 + + self + .records + .iter() + .map(WireEncode::encoded_size) + .sum::<usize>() + + 4 + + self + .items + .iter() + .map(WireEncode::encoded_size) + .sum::<usize>() + + 2 + + self + .current_page_token + .as_ref() + .map_or(0, WireEncode::encoded_size) + } + + fn encode(&self, buf: &mut BytesMut) { + buf.put_u8(self.status.as_u8()); + buf.put_u8(self.phase.as_u8()); + put_flag(buf, self.log_id.is_some()); + if let Some(log_id) = self.log_id { + buf.put_u128_le(log_id); + } + buf.put_u64_le(self.oldest_op); + buf.put_u64_le(self.retained_since); + buf.put_u64_le(self.head_op); + put_flag(buf, self.next.is_some()); + if let Some(next) = &self.next { + next.encode(buf); + } + put_flag(buf, self.has_more); + // A page holds at most 1000 records or items. + #[allow(clippy::cast_possible_truncation)] + buf.put_u32_le(self.records.len() as u32); + for record in &self.records { + record.encode(buf); + } + #[allow(clippy::cast_possible_truncation)] + buf.put_u32_le(self.items.len() as u32); + for item in &self.items { + item.encode(buf); + } + match &self.current_page_token { + Some(token) => { + // A valid token is at most `MAX_PAGE_TOKEN_SIZE` bytes. + #[allow(clippy::cast_possible_truncation)] + buf.put_u16_le(token.encoded_size() as u16); + token.encode(buf); + } + None => buf.put_u16_le(0), + } + } +} + +impl WireDecode for GetMetadataChangesResponse { + fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> { + let status = WireMetadataChangeStatus::from_code(read_u8(buf, 0)?)?; + let phase = WireMetadataChangePhase::from_code(read_u8(buf, 1)?)?; + let mut position = 2; + let log_id = if read_flag(buf, position)? { + position += 1; + let log_id = read_u128_le(buf, position)?; + position += 16; + Some(log_id) + } else { + position += 1; + None + }; + let oldest_op = read_u64_le(buf, position)?; + let retained_since = read_u64_le(buf, position + 8)?; + let head_op = read_u64_le(buf, position + 16)?; + position += 24; + let next = if read_flag(buf, position)? { + position += 1; + let rest = buf.get(position..).unwrap_or_default(); + let (cursor, consumed) = WireMetadataChangeCursor::decode(rest)?; + position += consumed; + Some(cursor) + } else { + position += 1; + None + }; + let has_more = read_flag(buf, position)?; + position += 1; + + let records_count = read_u32_le(buf, position)? as usize; + position += 4; + let mut records = Vec::with_capacity(bounded_capacity( + records_count, + buf.len().saturating_sub(position), + WireMetadataChangeRecord::MIN_SIZE, + )); + for _ in 0..records_count { + let rest = buf.get(position..).unwrap_or_default(); + let (record, consumed) = WireMetadataChangeRecord::decode(rest)?; + records.push(record); + position += consumed; + } + + let items_count = read_u32_le(buf, position)? as usize; + position += 4; + let mut items = Vec::with_capacity(bounded_capacity( + items_count, + buf.len().saturating_sub(position), + 5, + )); + for _ in 0..items_count { + let rest = buf.get(position..).unwrap_or_default(); + let (item, consumed) = WireMetadataItem::decode(rest)?; + items.push(item); + position += consumed; + } + + let token_length = usize::from(read_u16_le(buf, position)?); + position += 2; + let current_page_token = if token_length == 0 { + None + } else { + let token_bytes = read_bytes(buf, position, token_length)?; + let (token, consumed) = WireMetadataCurrentPageToken::decode(token_bytes)?; + if consumed != token_length { + return Err(WireError::Validation(Cow::Borrowed( + "trailing bytes after a page token", + ))); + } + position += token_length; + Some(token) + }; + + let phase_holds_other_content = match phase { + WireMetadataChangePhase::Changes => !items.is_empty() || current_page_token.is_some(), + WireMetadataChangePhase::Current => !records.is_empty(), + }; + if phase_holds_other_content { + return Err(WireError::Validation(Cow::Borrowed( + "a changes page carries only records and a current page only items", + ))); + } + Ok(( + Self { + status, + phase, + log_id, + oldest_op, + retained_since, + head_op, + next, + has_more, + records, + items, + current_page_token, + }, + position, + )) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::primitives::metadata_changes::WireMetadataChangeGroup; + + fn cursor(op: u64, ordinal: u32) -> WireMetadataChangeCursor { + WireMetadataChangeCursor { + log_id: 0x0d4b_8f2a_6e1c_93d7, + op, + checksum: 0xa5c1_f703_8b9e_2d4c, + ordinal, + } + } + + fn record(actor_user_id: Option<u32>) -> WireMetadataChangeRecord { + WireMetadataChangeRecord { + op: 4121, + ordinal: 0, + checksum: 0xa5c1_f703_8b9e_2d4c, + timestamp: 1_760_356_800_123_456, + actor_user_id, + group: 1, + kind: 1, + body: Bytes::from_static(&[1, 2, 3, 4]), + } + } + + fn changes_page() -> GetMetadataChangesResponse { + GetMetadataChangesResponse { + status: WireMetadataChangeStatus::Ok, + phase: WireMetadataChangePhase::Changes, + log_id: Some(0x0d4b_8f2a_6e1c_93d7), + oldest_op: 3871, + retained_since: 1_760_352_000_000_000, + head_op: 4187, + next: Some(cursor(4187, 1)), + has_more: false, + records: vec![record(Some(4)), record(None)], + items: vec![], + current_page_token: None, + } + } + + fn current_page() -> GetMetadataChangesResponse { + GetMetadataChangesResponse { + status: WireMetadataChangeStatus::Ok, + phase: WireMetadataChangePhase::Current, + log_id: Some(1), + oldest_op: 1, + retained_since: 0, + head_op: 40, + next: Some(cursor(30, 0)), + has_more: true, + records: vec![], + items: vec![WireMetadataItem { + group: 1, + body: Bytes::from_static(&[9; 12]), + }], + current_page_token: Some(WireMetadataCurrentPageToken { + anchor: cursor(30, 0), + required_head: 40, + groups_mask: WireMetadataChangeGroup::Streams.mask_bit(), + group: WireMetadataChangeGroup::Streams, + last_key: Bytes::from_static(&[3, 0, 0, 0]), + }), + } + } + + #[test] + fn given_changes_page_when_round_tripping_should_preserve_every_field() { + let original = changes_page(); + let bytes = original.to_bytes(); + assert_eq!(bytes.len(), original.encoded_size()); + let (decoded, consumed) = GetMetadataChangesResponse::decode(&bytes).unwrap(); + assert_eq!(decoded, original); + assert_eq!(consumed, bytes.len()); + } + + #[test] + fn given_current_page_when_round_tripping_should_preserve_every_field() { + let original = current_page(); + let bytes = original.to_bytes(); + let (decoded, consumed) = GetMetadataChangesResponse::decode(&bytes).unwrap(); + assert_eq!(decoded, original); + assert_eq!(consumed, bytes.len()); + } + + #[test] + fn given_behind_without_log_id_or_next_when_round_tripping_should_preserve_absence() { + let original = GetMetadataChangesResponse { + status: WireMetadataChangeStatus::Behind, + log_id: None, + next: None, + records: vec![], + ..changes_page() + }; + let bytes = original.to_bytes(); + assert_eq!( + GetMetadataChangesResponse::decode(&bytes).unwrap().0, + original + ); + } + + #[test] + fn given_items_in_changes_phase_when_decoding_should_refuse() { + let mut page = current_page(); + page.phase = WireMetadataChangePhase::Changes; + assert!(GetMetadataChangesResponse::decode(&page.to_bytes()).is_err()); + } + + #[test] + fn given_records_in_current_phase_when_decoding_should_refuse() { + let mut page = changes_page(); + page.phase = WireMetadataChangePhase::Current; + assert!(GetMetadataChangesResponse::decode(&page.to_bytes()).is_err()); + } + + #[test] + fn given_unknown_status_when_decoding_should_refuse() { + let mut bytes = changes_page().to_bytes().to_vec(); + bytes[0] = 4; + assert!(GetMetadataChangesResponse::decode(&bytes).is_err()); + } + + #[test] + fn given_truncated_response_when_decoding_should_refuse_every_prefix() { + for page in [changes_page(), current_page()] { + let bytes = page.to_bytes(); + for length in 0..bytes.len() { + assert!( + GetMetadataChangesResponse::decode(&bytes[..length]).is_err(), + "accepted a response truncated to {length} bytes" + ); + } + } + } + + #[test] + fn given_record_count_beyond_the_frame_when_decoding_should_fail_without_allocating_it() { + let mut page = changes_page(); + page.records.clear(); + let mut bytes = page.to_bytes().to_vec(); + let count_offset = 2 + 17 + 24 + 1 + METADATA_CHANGE_CURSOR_SIZE + 1; + bytes[count_offset..count_offset + 4].copy_from_slice(&u32::MAX.to_le_bytes()); + assert!(GetMetadataChangesResponse::decode(&bytes).is_err()); + } +}
diff --git a/core/binary_protocol/src/responses/system/mod.rs b/core/binary_protocol/src/responses/system/mod.rs index a8e21d3..a43a53c 100644 --- a/core/binary_protocol/src/responses/system/mod.rs +++ b/core/binary_protocol/src/responses/system/mod.rs
@@ -18,6 +18,7 @@ pub mod describe_options; pub mod get_cluster_metadata; pub mod get_me; +pub mod get_metadata_changes; pub mod get_snapshot; pub mod get_stats; mod ping; @@ -27,6 +28,9 @@ pub use describe_options::{DescribeOptionsResponse, OptionDescriptor}; pub use get_cluster_metadata::{ClusterMetadataResponse, ClusterNodeResponse}; pub use get_me::GetMeResponse; +pub use get_metadata_changes::{ + GetMetadataChangesResponse, WireMetadataChangeRecord, WireMetadataItem, +}; pub use get_snapshot::GetSnapshotResponse; pub use get_stats::StatsResponse; pub use ping::PingResponse;
diff --git a/core/cli/Cargo.toml b/core/cli/Cargo.toml index f8d4337..c7d365c 100644 --- a/core/cli/Cargo.toml +++ b/core/cli/Cargo.toml
@@ -17,7 +17,7 @@ [package] name = "iggy-cli" -version = "0.14.1-edge.2" +version = "0.14.1-edge.3" edition = "2024" rust-version.workspace = true authors = ["bartosz.ciesla@gmail.com"]
diff --git a/core/cli/src/args/metadata.rs b/core/cli/src/args/metadata.rs new file mode 100644 index 0000000..3f9ddf9 --- /dev/null +++ b/core/cli/src/args/metadata.rs
@@ -0,0 +1,117 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use clap::{Args, Subcommand, ValueEnum}; +use iggy::prelude::{ + IggyDuration, IggyTimestamp, MetadataChangeCursor, MetadataChangeGroup, MetadataChangeStart, + MetadataChangesQuery, +}; + +#[derive(Debug, Clone, Subcommand)] +pub(crate) enum MetadataAction { + /// read the metadata change log + /// + /// Prints every answered page as one JSON object per line: the change + /// records or current state items, the status and the cursor to continue + /// from. Requires root or the global manage_servers permission. + /// + /// Examples + /// iggy metadata changes + /// iggy metadata changes --from latest + /// iggy metadata changes --after mc1_... --groups streams,topics + /// iggy metadata changes --from current --groups streams,topics --follow + #[clap(verbatim_doc_comment, visible_alias = "c")] + Changes(MetadataChangesArgs), +} + +#[derive(Debug, Clone, Args)] +pub(crate) struct MetadataChangesArgs { + /// Where to start reading + /// + /// earliest: the oldest operation the log still holds completely, + /// latest: no records, only the cursor at the newest record, + /// current: the current state, page by page, then its catch-up cursor. + #[arg(verbatim_doc_comment, long, value_enum, default_value_t = MetadataChangesFrom::Earliest, conflicts_with = "after")] + pub(crate) from: MetadataChangesFrom, + + /// Continue after this cursor, the `next` value of an earlier page + #[arg(long)] + pub(crate) after: Option<MetadataChangeCursor>, + + /// Groups to read, comma separated, every group when omitted + /// + /// streams, topics, partitions, consumer_groups, consumer_group_members, + /// users, personal_access_tokens, sessions + #[arg(verbatim_doc_comment, long, value_delimiter = ',')] + pub(crate) groups: Vec<MetadataChangeGroup>, + + /// Oldest prepare timestamp to include, in microseconds since the epoch + #[arg(long, value_parser = parse_micros)] + pub(crate) since: Option<IggyTimestamp>, + + /// Newest prepare timestamp to include, in microseconds since the epoch + #[arg(long, value_parser = parse_micros)] + pub(crate) until: Option<IggyTimestamp>, + + /// Records or items per page, 1 to 1000 + #[arg(short, long, default_value_t = 100)] + pub(crate) limit: u32, + + /// Keep reading new changes once the log has no more + /// + /// When the cursor's history is no longer retained, the command rebuilds + /// from the current state and goes on following. + #[arg(verbatim_doc_comment, short, long, default_value_t = false)] + pub(crate) follow: bool, + + /// Pause between pages while following + #[arg(long, default_value = "1s")] + pub(crate) interval: IggyDuration, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, ValueEnum)] +pub(crate) enum MetadataChangesFrom { + Earliest, + Latest, + Current, +} + +impl From<&MetadataChangesArgs> for MetadataChangesQuery { + fn from(args: &MetadataChangesArgs) -> Self { + let start = match (args.after, args.from) { + (Some(cursor), _) => MetadataChangeStart::After(cursor), + (None, MetadataChangesFrom::Earliest) => MetadataChangeStart::Earliest, + (None, MetadataChangesFrom::Latest) => MetadataChangeStart::Latest, + (None, MetadataChangesFrom::Current) => MetadataChangeStart::Current(None), + }; + Self { + start, + groups: args.groups.clone(), + since: args.since, + until: args.until, + limit: args.limit, + } + } +} + +fn parse_micros(raw: &str) -> Result<IggyTimestamp, String> { + let micros: u64 = raw + .parse() + .map_err(|error| format!("expected microseconds since the epoch, got {raw}: {error}"))?; + IggyTimestamp::checked_from_micros(micros) + .ok_or_else(|| format!("{micros} microseconds is past what this platform's clock holds")) +}
diff --git a/core/cli/src/args/mod.rs b/core/cli/src/args/mod.rs index ffe1145..24a3ca6 100644 --- a/core/cli/src/args/mod.rs +++ b/core/cli/src/args/mod.rs
@@ -34,6 +34,7 @@ consumer_offset::ConsumerOffsetAction, context::ContextAction, message::MessageAction, + metadata::MetadataAction, partition::PartitionAction, personal_access_token::PersonalAccessTokenAction, stream::StreamAction, @@ -56,6 +57,7 @@ pub(crate) mod consumer_offset; pub(crate) mod context; pub(crate) mod message; +pub(crate) mod metadata; pub(crate) mod partition; pub(crate) mod permissions; pub(crate) mod personal_access_token; @@ -191,6 +193,9 @@ /// collect iggy server troubleshooting data #[clap(verbatim_doc_comment)] Snapshot(SnapshotArgs), + /// metadata change log operations + #[command(subcommand, visible_alias = "md")] + Metadata(MetadataAction), /// personal access token operations #[command(subcommand)] Pat(PersonalAccessTokenAction),
diff --git a/core/cli/src/commands/binary_system/metadata_changes.rs b/core/cli/src/commands/binary_system/metadata_changes.rs new file mode 100644 index 0000000..64156fa --- /dev/null +++ b/core/cli/src/commands/binary_system/metadata_changes.rs
@@ -0,0 +1,337 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use crate::commands::cli_command::CliCommand; +use anyhow::{Context, bail}; +use async_trait::async_trait; +use iggy_common::{ + Client, IggyDuration, IggyError, MetadataChangeGroup, MetadataChangeStart, + MetadataChangeStatus, MetadataChanges, MetadataChangesQuery, +}; +use std::future::Future; +use std::io::{Write, stdout}; + +/// Reads metadata change pages and prints each as one JSON line. +pub struct GetMetadataChangesCmd { + query: MetadataChangesQuery, + follow: bool, + interval: IggyDuration, +} + +impl GetMetadataChangesCmd { + pub fn new(query: MetadataChangesQuery, follow: bool, interval: IggyDuration) -> Self { + Self { + query, + follow, + interval, + } + } + + /// Reads pages from `read` and writes each to `output` until the query ends. + async fn read_pages<R, F>( + &self, + mut read: R, + output: &mut (impl Write + Send), + ) -> anyhow::Result<()> + where + R: FnMut(MetadataChangesQuery) -> F + Send, + F: Future<Output = Result<MetadataChanges, IggyError>> + Send, + { + let mut query = self.first_query(); + loop { + let page = match read(query.clone()).await { + Ok(page) => page, + Err(IggyError::TransientNotAccepted) if self.follow => { + tokio::time::sleep(self.interval.get_duration()).await; + continue; + } + Err(error) => { + return Err(error) + .with_context(|| "Problem reading metadata changes".to_owned()); + } + }; + let quiet_follow_page = self.follow + && page.status == MetadataChangeStatus::Ok + && page.records.is_empty() + && page.items.is_empty(); + if !quiet_follow_page { + writeln!(output, "{}", serde_json::to_string(&page)?) + .and_then(|()| output.flush()) + .with_context(|| "Problem writing metadata changes page".to_owned())?; + } + query = match self.step(query, page)? { + Step::Read(next) => next, + Step::Wait(next) => { + tokio::time::sleep(self.interval.get_duration()).await; + next + } + Step::Stop => return Ok(()), + }; + } + } + + /// The query that starts the read, with the parent groups a current scan needs. + fn first_query(&self) -> MetadataChangesQuery { + match self.query.start { + MetadataChangeStart::Current(_) => MetadataChangesQuery { + groups: self.scan_groups(), + ..self.query.clone() + }, + _ => self.query.clone(), + } + } + + /// What to read after `page` answered `query`. + fn step(&self, mut query: MetadataChangesQuery, page: MetadataChanges) -> anyhow::Result<Step> { + match page.status { + MetadataChangeStatus::Ok => {} + MetadataChangeStatus::Behind if self.follow => return Ok(Step::Wait(query)), + MetadataChangeStatus::Behind => return Ok(Step::Stop), + MetadataChangeStatus::Expired | MetadataChangeStatus::Reset if self.follow => { + return Ok(Step::Read(self.current_state_query())); + } + MetadataChangeStatus::Expired | MetadataChangeStatus::Reset => { + bail!( + "The history after the cursor is not available ({}), read it again with --from current", + serde_json::to_string(&page.status)? + ); + } + } + if let MetadataChangeStart::Current(_) = query.start { + return Ok(match (page.current_page_token, page.next) { + (Some(token), _) => { + query.start = MetadataChangeStart::Current(Some(token)); + Step::Read(query) + } + (None, Some(anchor)) if self.follow => Step::Read(MetadataChangesQuery { + start: MetadataChangeStart::After(anchor), + ..self.query.clone() + }), + _ => Step::Stop, + }); + } + if let Some(next) = page.next { + query.start = MetadataChangeStart::After(next); + } + Ok(if page.has_more { + Step::Read(query) + } else if self.follow { + Step::Wait(query) + } else { + Step::Stop + }) + } + + /// The rebuild scan, with the parent groups and no time bounds the server requires. + fn current_state_query(&self) -> MetadataChangesQuery { + MetadataChangesQuery { + start: MetadataChangeStart::Current(None), + groups: self.scan_groups(), + since: None, + until: None, + limit: self.query.limit, + } + } + + /// The requested groups and their parents. + fn scan_groups(&self) -> Vec<MetadataChangeGroup> { + let mut groups = self.query.groups.clone(); + groups.extend(self.query.groups.iter().flat_map(|group| group.parents())); + groups + } +} + +#[async_trait] +impl CliCommand for GetMetadataChangesCmd { + fn explain(&self) -> String { + "metadata changes command".to_owned() + } + + fn stdout_is_data(&self) -> bool { + true + } + + async fn execute_cmd(&mut self, client: &dyn Client) -> anyhow::Result<(), anyhow::Error> { + self.read_pages( + |query| async move { client.get_metadata_changes(&query).await }, + &mut stdout(), + ) + .await + } +} + +#[derive(Debug, PartialEq)] +enum Step { + Read(MetadataChangesQuery), + Wait(MetadataChangesQuery), + Stop, +} + +#[cfg(test)] +mod tests { + use super::*; + use iggy_common::{ + IggyTimestamp, MetadataChangeCursor, MetadataChangeGroup, MetadataChangeGroups, + MetadataChangePhase, + }; + use std::collections::VecDeque; + use std::future::ready; + + #[test] + fn given_current_start_for_topics_when_building_first_query_should_add_their_parent_groups() { + let command = GetMetadataChangesCmd::new( + MetadataChangesQuery { + start: MetadataChangeStart::Current(None), + groups: vec![MetadataChangeGroup::Topics], + ..MetadataChangesQuery::default() + }, + false, + IggyDuration::ONE_SECOND, + ); + + assert_eq!( + command.first_query().validate(), + Ok(MetadataChangeGroups::requested(&[ + MetadataChangeGroup::Streams, + MetadataChangeGroup::Topics + ])) + ); + } + + #[tokio::test] + async fn given_transient_refusal_when_following_should_retry_the_same_query() { + let command = GetMetadataChangesCmd::new( + MetadataChangesQuery::default(), + true, + IggyDuration::from(1), + ); + let mut answers = VecDeque::from([ + Err(IggyError::TransientNotAccepted), + Ok(page(MetadataChangeStatus::Behind, None)), + Err(IggyError::Unauthenticated), + ]); + let mut queries = Vec::new(); + let mut output = Vec::new(); + + let result = command + .read_pages( + |query| { + queries.push(query); + ready(answers.pop_front().expect("scripted answer")) + }, + &mut output, + ) + .await; + + assert!(result.is_err(), "the scripted refusal ends the read"); + assert_eq!(queries.len(), 3); + assert!( + queries + .iter() + .all(|query| *query == MetadataChangesQuery::default()) + ); + assert_eq!(String::from_utf8(output).expect("utf-8").lines().count(), 1); + } + + #[test] + fn given_expired_cursor_when_following_topics_should_rebuild_with_their_parent_groups() { + let command = following(vec![MetadataChangeGroup::Topics]); + let step = command + .step( + command.query.clone(), + page(MetadataChangeStatus::Expired, None), + ) + .expect("recovery step"); + let Step::Read(recovery) = step else { + panic!("expected an immediate read, got {step:?}"); + }; + assert_eq!(recovery.start, MetadataChangeStart::Current(None)); + assert_eq!( + recovery.validate(), + Ok(MetadataChangeGroups::requested(&[ + MetadataChangeGroup::Streams, + MetadataChangeGroup::Topics + ])) + ); + } + + #[test] + fn given_finished_recovery_scan_when_following_should_restore_requested_groups_and_bounds() { + let command = following(vec![MetadataChangeGroup::Topics]); + let Step::Read(recovery) = command + .step( + command.query.clone(), + page(MetadataChangeStatus::Reset, None), + ) + .expect("recovery step") + else { + panic!("expected an immediate read"); + }; + let anchor = MetadataChangeCursor { + log_id: 7, + op: 42, + checksum: 9, + ordinal: 0, + }; + let step = command + .step( + recovery, + MetadataChanges { + phase: MetadataChangePhase::Current, + ..page(MetadataChangeStatus::Ok, Some(anchor)) + }, + ) + .expect("hand over step"); + assert_eq!( + step, + Step::Read(MetadataChangesQuery { + start: MetadataChangeStart::After(anchor), + ..command.query.clone() + }) + ); + } + + fn following(groups: Vec<MetadataChangeGroup>) -> GetMetadataChangesCmd { + GetMetadataChangesCmd::new( + MetadataChangesQuery { + start: MetadataChangeStart::Earliest, + groups, + since: Some(IggyTimestamp::from(1_000)), + until: Some(IggyTimestamp::from(9_000)), + limit: 50, + }, + true, + IggyDuration::ONE_SECOND, + ) + } + + fn page(status: MetadataChangeStatus, next: Option<MetadataChangeCursor>) -> MetadataChanges { + MetadataChanges { + status, + phase: MetadataChangePhase::Changes, + log_id: Some(7), + oldest_op: 1, + retained_since: 0, + head_op: 42, + next, + has_more: false, + records: vec![], + items: vec![], + current_page_token: None, + } + } +}
diff --git a/core/cli/src/commands/binary_system/mod.rs b/core/cli/src/commands/binary_system/mod.rs index 2bbc45a..69223a4 100644 --- a/core/cli/src/commands/binary_system/mod.rs +++ b/core/cli/src/commands/binary_system/mod.rs
@@ -18,6 +18,7 @@ pub mod login; pub mod logout; pub mod me; +pub mod metadata_changes; pub mod options; pub mod ping; pub mod session;
diff --git a/core/cli/src/commands/cli_command.rs b/core/cli/src/commands/cli_command.rs index eb2520a..02bef7b 100644 --- a/core/cli/src/commands/cli_command.rs +++ b/core/cli/src/commands/cli_command.rs
@@ -33,6 +33,10 @@ fn use_tracing(&self) -> bool { true } + /// Whether stdout carries only the command's data, so the banner goes to stderr. + fn stdout_is_data(&self) -> bool { + false + } fn login_required(&self) -> bool { true }
diff --git a/core/cli/src/main.rs b/core/cli/src/main.rs index a3ccad4..b5ee3eb 100644 --- a/core/cli/src/main.rs +++ b/core/cli/src/main.rs
@@ -31,6 +31,7 @@ use crate::logging::Logging; use args::context::ContextAction; use args::message::MessageAction; +use args::metadata::MetadataAction; use args::partition::PartitionAction; use args::segment::SegmentAction; use args::user::UserAction; @@ -39,8 +40,8 @@ use iggy::client_provider::{self, ClientProviderConfig}; use iggy::clients::client::IggyClient; use iggy::prelude::{ - Aes256GcmEncryptor, ClientError, EncryptorKind, PersonalAccessTokenExpiry, StreamUpdateOptions, - TopicUpdateOptions, + Aes256GcmEncryptor, ClientError, EncryptorKind, MetadataChangesQuery, + PersonalAccessTokenExpiry, StreamUpdateOptions, TopicUpdateOptions, }; use iggy_cli::commands::binary_context::common::{ContextManager, validate_transport}; use iggy_cli::commands::binary_context::create_context::CreateContextCmd; @@ -48,6 +49,7 @@ use iggy_cli::commands::binary_context::show_context::ShowContextCmd; use iggy_cli::commands::binary_context::use_context::UseContextCmd; use iggy_cli::commands::binary_segments::delete_segments::DeleteSegmentsCmd; +use iggy_cli::commands::binary_system::metadata_changes::GetMetadataChangesCmd; use iggy_cli::commands::binary_system::snapshot::GetSnapshotCmd; #[cfg(feature = "login-session")] use iggy_cli::commands::cli_command::DIAGNOSTIC_TARGET; @@ -187,6 +189,13 @@ args.snapshot_types, args.out_dir, )), + Command::Metadata(command) => match command { + MetadataAction::Changes(args) => Box::new(GetMetadataChangesCmd::new( + MetadataChangesQuery::from(&args), + args.follow, + args.interval, + )), + }, Command::Pat(command) => match command { PersonalAccessTokenAction::Create(pat_create_args) => { Box::new(CreatePersonalAccessTokenCmd::new( @@ -439,7 +448,11 @@ credentials.set_iggy_client(&client); credentials.login_user().await?; - if command.use_tracing() { + if command.stdout_is_data() { + if !cli_options.quiet { + eprintln!("Executing {}", command.explain()); + } + } else if command.use_tracing() { event!(target: PRINT_TARGET, Level::INFO, "Executing {}", command.explain()); } else { println!("Executing {}", command.explain());
diff --git a/core/common/Cargo.toml b/core/common/Cargo.toml index a48d622..09c5008 100644 --- a/core/common/Cargo.toml +++ b/core/common/Cargo.toml
@@ -17,7 +17,7 @@ [package] name = "iggy_common" -version = "0.11.1-edge.2" +version = "0.11.1-edge.3" description = "Iggy is the persistent message streaming platform written in Rust, supporting QUIC, TCP and HTTP transport protocols, capable of processing millions of messages per second." edition = "2024" rust-version.workspace = true @@ -49,7 +49,7 @@ ring = { workspace = true } secrecy = { workspace = true } serde = { workspace = true } -serde_json = { workspace = true } +serde_json = { workspace = true, features = ["raw_value"] } serde_with = { workspace = true, features = ["base64", "json"] } strum = { workspace = true } thiserror = { workspace = true }
diff --git a/core/common/examples/metadata_changes_fixtures.rs b/core/common/examples/metadata_changes_fixtures.rs new file mode 100644 index 0000000..8f0aaaa --- /dev/null +++ b/core/common/examples/metadata_changes_fixtures.rs
@@ -0,0 +1,37 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#[path = "metadata_changes_fixtures/schema.rs"] +mod schema; + +use schema::{CANONICAL_PATH, FixtureResult, LANGUAGE_COPIES, render}; +use std::fs; +use std::iter; +use std::path::Path; + +fn main() -> FixtureResult<()> { + let json = render()?; + let manifest = Path::new(env!("CARGO_MANIFEST_DIR")); + for relative in iter::once(CANONICAL_PATH).chain(LANGUAGE_COPIES) { + let path = manifest.join(relative); + if let Some(parent) = path.parent() { + fs::create_dir_all(parent)?; + } + fs::write(&path, &json)?; + } + Ok(()) +}
diff --git a/core/common/examples/metadata_changes_fixtures/schema.rs b/core/common/examples/metadata_changes_fixtures/schema.rs new file mode 100644 index 0000000..6197655 --- /dev/null +++ b/core/common/examples/metadata_changes_fixtures/schema.rs
@@ -0,0 +1,1692 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Shape and builder of the metadata change log golden vectors. +//! +//! Shared by the generator example and the drift test, so the test rebuilds +//! the fixture in process and compares it byte for byte with the canonical +//! file and every SDK copy. + +use base64::Engine; +use base64::engine::general_purpose::URL_SAFE_NO_PAD; +use bytes::Bytes; +use iggy_binary_protocol::codes::GET_METADATA_CHANGES_CODE; +use iggy_binary_protocol::lookup_command; +use iggy_binary_protocol::primitives::metadata_changes::{ + MAX_PAGE_TOKEN_KEY_SIZE, MAX_PAGE_TOKEN_SIZE, METADATA_CHANGE_CURSOR_SIZE, WireChangeOptions, + WireConsumerGroupCreatedBody, WireConsumerGroupDeletedBody, WireConsumerGroupItemBody, + WireMemberItemBody, WireMemberJoinedBody, WireMemberLeftBody, WireMemberLeftReason, + WireMetadataChange, WireMetadataChangeCursor, WireMetadataChangeGroup, WireMetadataChangeKind, + WireMetadataChangePhase, WireMetadataChangeStartMode, WireMetadataChangeStatus, + WireMetadataCurrentItem, WireMetadataCurrentPageToken, WirePartitionHandedOverBody, + WirePartitionsChangedBody, WirePartitionsItemBody, WirePermissionsUpdatedReason, + WireSessionBody, WireSessionClosedBody, WireSessionClosedReason, WireStreamBody, + WireStreamDeletedBody, WireStreamKey, WireTokenBody, WireTokenDeletedBody, + WireTokenDeletedReason, WireTopicBody, WireTopicDeletedBody, WireTopicItemBody, WireTopicKey, + WireUserBody, WireUserIdentityBody, WireUserKey, WireUserPermissionsBody, +}; +use iggy_binary_protocol::requests::system::GetMetadataChangesRequest; +use iggy_binary_protocol::responses::system::{ + GetMetadataChangesResponse, WireMetadataChangeRecord, WireMetadataItem, +}; +use iggy_binary_protocol::{WireEncode, WireName, WireOptions}; +use iggy_common::wire_conversions::{permissions_to_wire, resource_options_to_wire_split}; +use iggy_common::{ + DEFAULT_METADATA_CHANGES_LIMIT, GlobalPermissions, HeaderKey, HeaderValue, IggyError, + IggyTimestamp, MAX_METADATA_CHANGES_LIMIT, METADATA_CHANGE_CURSOR_PREFIX, + METADATA_CURRENT_PAGE_TOKEN_PREFIX, MetadataChangeClass, MetadataChangeCursor, + MetadataChangeGroup, MetadataChangeKind, MetadataChangePhase, MetadataChangeRecord, + MetadataChangeStart, MetadataChangeStatus, MetadataChanges, MetadataChangesQuery, + MetadataCurrentItem, MetadataCurrentPageToken, MetadataMemberLeftReason, + MetadataPermissionsUpdatedReason, MetadataSessionClosedReason, MetadataTokenDeletedReason, + OptionKey, OptionValue, Permissions, ResourceOptions, StreamPermissions, TopicPermissions, +}; +use serde::Serialize; +use std::collections::BTreeMap; +use std::error::Error; +use std::str::FromStr; +use strum::IntoEnumIterator; + +pub const SCHEMA_VERSION: u32 = 1; +pub const GENERATOR: &str = "core/common/examples/metadata_changes_fixtures.rs"; +/// Relative to the `iggy_common` manifest directory. +pub const CANONICAL_PATH: &str = "fixtures/metadata_changes.json"; +/// Byte-identical copies each foreign SDK loads from its own test resources. +pub const LANGUAGE_COPIES: [&str; 8] = [ + "../../foreign/python/tests/fixtures/metadata_changes.json", + "../../foreign/cpp/tests/fixtures/metadata_changes.json", + "../../foreign/php/tests/fixtures/metadata_changes.json", + "../../foreign/node/src/wire/fixtures/metadata_changes.json", + "../../foreign/go/contracts/testdata/metadata_changes.json", + "../../foreign/java/java-sdk/src/test/resources/metadata_changes.json", + "../../foreign/csharp/Iggy_SDK_Tests/Fixtures/metadata_changes.json", + "../../foreign/swift/Tests/IggyTests/Fixtures/metadata_changes.json", +]; + +const LOG_ID: u128 = 0x0d4b_8f2a_6e1c_93d7; +const ABOVE_2_POW_53: u64 = (1 << 53) + 1; + +pub type FixtureResult<T> = Result<T, Box<dyn Error>>; + +#[derive(Debug, Serialize)] +pub struct MetadataChangesFixture { + pub schema_version: u32, + pub generated_by: String, + pub conventions: BTreeMap<String, String>, + pub constants: Constants, + pub records: Vec<RecordVector>, + pub items: Vec<ItemVector>, + pub cursors: Vec<CursorVector>, + pub page_tokens: Vec<PageTokenVector>, + pub requests: Vec<RequestVector>, + pub responses: Vec<ResponseVector>, + pub large_values: LargeValuesVector, + pub refused: RefusedVectors, +} + +#[derive(Debug, Serialize)] +pub struct Constants { + pub command_code: u32, + pub command_name: String, + pub cursor_prefix: String, + pub page_token_prefix: String, + pub cursor_size: usize, + pub cursor_text_length: usize, + pub max_page_token_size: usize, + pub max_page_token_key_size: usize, + pub default_limit: u32, + pub max_limit: u32, + pub groups: BTreeMap<String, GroupConstant>, + pub kinds: BTreeMap<String, KindConstant>, + pub statuses: BTreeMap<String, u8>, + pub phases: BTreeMap<String, u8>, + pub start_modes: BTreeMap<String, u8>, + pub member_left_reasons: BTreeMap<String, u8>, + pub token_deleted_reasons: BTreeMap<String, u8>, + pub session_closed_reasons: BTreeMap<String, u8>, + pub permissions_updated_reasons: BTreeMap<String, u8>, +} + +#[derive(Debug, Serialize)] +pub struct GroupConstant { + pub code: u8, + pub class: String, + pub parents: Vec<String>, + pub page_token_key: String, +} + +#[derive(Debug, Serialize)] +pub struct KindConstant { + pub code: u8, + pub group: String, +} + +/// A record's binary forms and the JSON the HTTP route renders for it. +#[derive(Debug, Serialize)] +pub struct RecordVector { + pub case: String, + pub group: u8, + pub kind: u8, + pub body_hex: String, + pub record_hex: String, + pub expected: MetadataChangeRecord, +} + +#[derive(Debug, Serialize)] +pub struct ItemVector { + pub case: String, + pub group: u8, + pub body_hex: String, + pub item_hex: String, + pub expected: MetadataCurrentItem, +} + +#[derive(Debug, Serialize)] +pub struct CursorVector { + pub case: String, + pub text: String, + pub hex: String, + pub log_id: String, + pub op: u64, + pub checksum: String, + pub ordinal: u32, +} + +#[derive(Debug, Serialize)] +pub struct PageTokenVector { + pub case: String, + pub text: String, + pub hex: String, + pub anchor: String, + pub required_head: u64, + pub groups_mask: u8, + pub group: String, + pub last_key_hex: String, +} + +/// A query in the shape every SDK can build: start mode, positions as text. +#[derive(Debug, Clone, Serialize)] +pub struct QueryInput { + pub start: String, + pub cursor: Option<String>, + pub page_token: Option<String>, + pub groups: Vec<String>, + pub since: Option<u64>, + pub until: Option<u64>, + pub limit: u32, +} + +#[derive(Debug, Serialize)] +pub struct RequestVector { + pub case: String, + pub input: QueryInput, + pub request_hex: String, + pub http_query: String, +} + +#[derive(Debug, Serialize)] +pub struct ResponseVector { + pub case: String, + pub response_hex: String, + pub expected: MetadataChanges, +} + +/// Unsigned values above 2^53 as decimal strings, so SDKs can prove they +/// decode them losslessly from the binary form. +#[derive(Debug, Serialize)] +pub struct LargeValuesVector { + pub record_hex: String, + pub op: String, + pub ordinal: u32, + pub timestamp: String, + pub checksum: String, + pub user_created_at: String, + pub session: String, + pub client_id: String, +} + +#[derive(Debug, Serialize)] +pub struct RefusedVectors { + pub cursors: Vec<RefusedText>, + pub page_tokens: Vec<RefusedText>, + pub record_bodies: Vec<RefusedBody>, + pub requests: Vec<RefusedHex>, + pub responses: Vec<RefusedHex>, + pub queries: Vec<RefusedQuery>, + pub http_queries: Vec<RefusedHttpQuery>, +} + +#[derive(Debug, Serialize)] +pub struct RefusedText { + pub case: String, + pub text: String, +} + +#[derive(Debug, Serialize)] +pub struct RefusedBody { + pub case: String, + pub group: u8, + pub kind: u8, + pub body_hex: String, +} + +#[derive(Debug, Serialize)] +pub struct RefusedHex { + pub case: String, + pub hex: String, +} + +#[derive(Debug, Serialize)] +pub struct RefusedQuery { + pub case: String, + pub input: QueryInput, +} + +#[derive(Debug, Serialize)] +pub struct RefusedHttpQuery { + pub case: String, + pub query: String, +} + +impl QueryInput { + fn new(start: &str) -> Self { + Self { + start: start.to_owned(), + cursor: None, + page_token: None, + groups: Vec::new(), + since: None, + until: None, + limit: DEFAULT_METADATA_CHANGES_LIMIT, + } + } + + /// The domain query this input describes. + /// + /// # Errors + /// + /// `InvalidCommand` for an unknown start mode, group or position text. + pub fn query(&self) -> Result<MetadataChangesQuery, IggyError> { + let start = match (self.start.as_str(), &self.cursor, &self.page_token) { + ("earliest", None, None) => MetadataChangeStart::Earliest, + ("latest", None, None) => MetadataChangeStart::Latest, + ("after", Some(cursor), None) => { + MetadataChangeStart::After(MetadataChangeCursor::from_str(cursor)?) + } + ("current", None, token) => MetadataChangeStart::Current( + token + .as_deref() + .map(MetadataCurrentPageToken::from_str) + .transpose()?, + ), + _ => return Err(IggyError::InvalidCommand), + }; + let groups = self + .groups + .iter() + .map(|name| MetadataChangeGroup::from_str(name).map_err(|_| IggyError::InvalidCommand)) + .collect::<Result<_, _>>()?; + Ok(MetadataChangesQuery { + start, + groups, + since: self.since.map(IggyTimestamp::from), + until: self.until.map(IggyTimestamp::from), + limit: self.limit, + }) + } +} + +/// Build the whole fixture from the live codecs. +/// +/// # Errors +/// +/// Propagates any codec or conversion failure, which means the fixture cannot +/// describe the current code. +pub fn build() -> FixtureResult<MetadataChangesFixture> { + let records = record_vectors()?; + let items = item_vectors()?; + let cursors = cursor_vectors(); + let page_tokens = page_token_vectors(); + let requests = request_vectors()?; + let responses = response_vectors(&records, &items)?; + Ok(MetadataChangesFixture { + schema_version: SCHEMA_VERSION, + generated_by: GENERATOR.to_owned(), + conventions: conventions(), + constants: constants()?, + records, + items, + cursors, + page_tokens, + requests, + responses, + large_values: large_values_vector()?, + refused: refused_vectors()?, + }) +} + +/// The fixture as the canonical file text. +/// +/// # Errors +/// +/// Propagates build and serialization failures. +pub fn render() -> FixtureResult<String> { + let mut json = serde_json::to_string_pretty(&build()?)?; + json.push('\n'); + Ok(json) +} + +fn conventions() -> BTreeMap<String, String> { + [ + ( + "byte_strings", + "Every *_hex field is lowercase hex without separators.", + ), + ( + "u128_values", + "128-bit values (log ids, checksums, client ids) are 32 lowercase hex characters in JSON.", + ), + ( + "u64_values", + "Ops and timestamps are JSON numbers below 2^53 in every vector except large_values, which carries them as decimal strings.", + ), + ( + "record_hex", + "record_hex is one record envelope as a response carries it. body_hex is the body alone.", + ), + ( + "expected", + "expected is the JSON the HTTP route renders. Binary decoders must reach the same values.", + ), + ( + "request_hex", + "request_hex is the command payload without framing. http_query is the query string of GET /metadata/changes.", + ), + ( + "refused", + "Every refused vector must fail to decode or validate. Query and HTTP refusals are invalid_command.", + ), + ( + "unknown_kinds", + "A record of an unknown group or kind is kept with its codes and raw body, never refused.", + ), + ] + .into_iter() + .map(|(name, text)| (name.to_owned(), text.to_owned())) + .collect() +} + +fn constants() -> FixtureResult<Constants> { + let command = lookup_command(GET_METADATA_CHANGES_CODE).ok_or("code 17 is not registered")?; + let groups = MetadataChangeGroup::iter() + .map(|group| { + let class = match group.class() { + MetadataChangeClass::Resource => "resource", + MetadataChangeClass::Activity => "activity", + }; + let key = match group { + MetadataChangeGroup::Streams => "[stream_id:u32]", + MetadataChangeGroup::Topics | MetadataChangeGroup::Partitions => { + "[stream_id:u32][topic_id:u32]" + } + MetadataChangeGroup::ConsumerGroups => { + "[stream_id:u32][topic_id:u32][group_id:u32]" + } + MetadataChangeGroup::ConsumerGroupMembers => { + "[stream_id:u32][topic_id:u32][group_id:u32][member_id:u32]" + } + MetadataChangeGroup::Users => "[user_id:u32]", + MetadataChangeGroup::PersonalAccessTokens => "[user_id:u32][name_len:u8][name]", + MetadataChangeGroup::Sessions => "[epoch:u64]", + }; + ( + group.to_string(), + GroupConstant { + code: WireMetadataChangeGroup::from(group).as_u8(), + class: class.to_owned(), + parents: group.parents().iter().map(ToString::to_string).collect(), + page_token_key: key.to_owned(), + }, + ) + }) + .collect(); + let kinds = (0..=u8::MAX) + .filter_map(|code| WireMetadataChangeKind::from_code(code).ok()) + .map(|kind| { + let domain = MetadataChangeKind::from(kind); + ( + domain.to_string(), + KindConstant { + code: kind.as_u8(), + group: domain.group().to_string(), + }, + ) + }) + .collect(); + Ok(Constants { + command_code: GET_METADATA_CHANGES_CODE, + command_name: command.name.to_owned(), + cursor_prefix: METADATA_CHANGE_CURSOR_PREFIX.to_owned(), + page_token_prefix: METADATA_CURRENT_PAGE_TOKEN_PREFIX.to_owned(), + cursor_size: METADATA_CHANGE_CURSOR_SIZE, + cursor_text_length: MetadataChangeCursor::default().to_string().len(), + max_page_token_size: MAX_PAGE_TOKEN_SIZE, + max_page_token_key_size: MAX_PAGE_TOKEN_KEY_SIZE, + default_limit: DEFAULT_METADATA_CHANGES_LIMIT, + max_limit: MAX_METADATA_CHANGES_LIMIT, + groups, + kinds, + statuses: codes( + (0..=u8::MAX).filter_map(|code| WireMetadataChangeStatus::from_code(code).ok()), + |status| MetadataChangeStatus::from(status).to_string(), + WireMetadataChangeStatus::as_u8, + ), + phases: codes( + (0..=u8::MAX).filter_map(|code| WireMetadataChangePhase::from_code(code).ok()), + |phase| MetadataChangePhase::from(phase).to_string(), + WireMetadataChangePhase::as_u8, + ), + start_modes: codes( + (0..=u8::MAX).filter_map(|code| WireMetadataChangeStartMode::from_code(code).ok()), + |mode| format!("{mode:?}").to_lowercase(), + WireMetadataChangeStartMode::as_u8, + ), + member_left_reasons: codes( + (0..=u8::MAX).filter_map(|code| WireMemberLeftReason::from_code(code).ok()), + |reason| MetadataMemberLeftReason::from(reason).to_string(), + WireMemberLeftReason::as_u8, + ), + token_deleted_reasons: codes( + (0..=u8::MAX).filter_map(|code| WireTokenDeletedReason::from_code(code).ok()), + |reason| MetadataTokenDeletedReason::from(reason).to_string(), + WireTokenDeletedReason::as_u8, + ), + session_closed_reasons: codes( + (0..=u8::MAX).filter_map(|code| WireSessionClosedReason::from_code(code).ok()), + |reason| MetadataSessionClosedReason::from(reason).to_string(), + WireSessionClosedReason::as_u8, + ), + permissions_updated_reasons: codes( + (0..=u8::MAX).filter_map(|code| WirePermissionsUpdatedReason::from_code(code).ok()), + |reason| MetadataPermissionsUpdatedReason::from(reason).to_string(), + WirePermissionsUpdatedReason::as_u8, + ), + }) +} + +fn codes<T: Copy>( + values: impl Iterator<Item = T>, + name: impl Fn(T) -> String, + code: impl Fn(T) -> u8, +) -> BTreeMap<String, u8> { + values.map(|value| (name(value), code(value))).collect() +} + +fn stream_key() -> WireStreamKey { + WireStreamKey { + stream_id: 3, + stream_created_op: 4121, + } +} + +fn topic_key() -> WireTopicKey { + WireTopicKey { + stream: stream_key(), + topic_id: 0, + topic_created_op: 4125, + } +} + +fn user_key() -> WireUserKey { + WireUserKey { + user_id: 7, + user_created_at: 1_760_300_000_000_000, + } +} + +fn name(text: &str) -> FixtureResult<WireName> { + Ok(WireName::new(text)?) +} + +fn readable(options: &ResourceOptions) -> FixtureResult<WireChangeOptions> { + let (explicit, derived) = resource_options_to_wire_split(options)?; + Ok(WireChangeOptions::Readable { explicit, derived }) +} + +fn stream_options() -> FixtureResult<WireChangeOptions> { + readable(&ResourceOptions::from([ + ( + HeaderKey::from(OptionKey::EncryptionKeySource), + OptionValue::derived(HeaderValue::try_from("generated")?), + ), + ( + HeaderKey::from(OptionKey::EncryptionKeyFingerprint), + OptionValue::derived(HeaderValue::try_from("9f2c1a7e44b0d816")?), + ), + ( + HeaderKey::from(OptionKey::EncryptionKekFingerprint), + OptionValue::derived(HeaderValue::try_from("51d0e3a9c27b6f48")?), + ), + ])) +} + +fn topic_options() -> FixtureResult<WireChangeOptions> { + readable(&ResourceOptions::from([ + ( + HeaderKey::from(OptionKey::MessageExpiry), + OptionValue::explicit(HeaderValue::from(604_800_000_000u64)), + ), + ( + HeaderKey::from(OptionKey::SegmentSize), + OptionValue::derived(HeaderValue::from(1_073_741_824u64)), + ), + ])) +} + +fn empty_options() -> WireChangeOptions { + WireChangeOptions::Readable { + explicit: WireOptions::empty(), + derived: WireOptions::empty(), + } +} + +fn permissions() -> Permissions { + Permissions { + global: GlobalPermissions { + read_streams: true, + ..GlobalPermissions::default() + }, + streams: Some(BTreeMap::from([( + 3, + StreamPermissions { + read_stream: true, + topics: Some(BTreeMap::from([( + 0, + TopicPermissions { + poll_messages: true, + ..TopicPermissions::default() + }, + )])), + ..StreamPermissions::default() + }, + )])), + } +} + +fn record_hex(record: &WireMetadataChangeRecord) -> String { + hex::encode(record.to_bytes()) +} + +fn wire_record( + change: &WireMetadataChange, + op: u64, + ordinal: u32, + actor_user_id: Option<u32>, +) -> WireMetadataChangeRecord { + WireMetadataChangeRecord { + op, + ordinal, + checksum: 0x0000_0000_0000_0000_a5c1_f703_8b9e_2d4c + u128::from(op), + timestamp: 1_760_356_800_000_000 + op, + actor_user_id, + group: change.group().as_u8(), + kind: change.kind().as_u8(), + body: change.body_bytes(), + } +} + +fn change_cases() -> FixtureResult<Vec<(&'static str, WireMetadataChange, Option<u32>)>> { + let token_name = || name("ci-token"); + Ok(vec![ + ( + "stream_created", + WireMetadataChange::StreamCreated(WireStreamBody { + key: stream_key(), + created_at: 1_760_356_800_123_456, + name: name("telemetry")?, + options: stream_options()?, + }), + Some(4), + ), + ( + "stream_updated_unreadable_options", + WireMetadataChange::StreamUpdated(WireStreamBody { + key: stream_key(), + created_at: 1_760_356_800_123_456, + name: name("telemetry-v2")?, + options: WireChangeOptions::Unreadable, + }), + None, + ), + ( + "stream_deleted_deferred", + WireMetadataChange::StreamDeleted(WireStreamDeletedBody { + key: stream_key(), + created_at: 1_760_356_800_123_456, + name: name("telemetry")?, + requested_op: Some(4140), + }), + Some(4), + ), + ( + "topic_created", + WireMetadataChange::TopicCreated(WireTopicBody { + key: topic_key(), + created_at: 1_760_356_801_000_000, + name: name("node-metrics")?, + partitions_count: 8, + options: topic_options()?, + }), + Some(4), + ), + ( + "topic_updated", + WireMetadataChange::TopicUpdated(WireTopicBody { + key: topic_key(), + created_at: 1_760_356_801_000_000, + name: name("node-metrics-v2")?, + partitions_count: 8, + options: empty_options(), + }), + Some(4), + ), + ( + "topic_deleted_immediate", + WireMetadataChange::TopicDeleted(WireTopicDeletedBody { + key: topic_key(), + created_at: 1_760_356_801_000_000, + name: name("node-metrics")?, + requested_op: None, + }), + Some(4), + ), + ( + "partitions_changed_deleted_tail", + WireMetadataChange::PartitionsChanged(WirePartitionsChangedBody { + topic: topic_key(), + partitions_count: 5, + removed_partition_ids: vec![5, 6, 7], + requested_op: Some(4200), + }), + Some(4), + ), + ( + "partitions_changed_created", + WireMetadataChange::PartitionsChanged(WirePartitionsChangedBody { + topic: topic_key(), + partitions_count: 10, + removed_partition_ids: vec![], + requested_op: None, + }), + Some(4), + ), + ( + "consumer_group_created", + WireMetadataChange::ConsumerGroupCreated(WireConsumerGroupCreatedBody { + topic: topic_key(), + group_id: 0, + name: name("readers")?, + consumer_groups_count: 1, + }), + Some(4), + ), + ( + "consumer_group_deleted", + WireMetadataChange::ConsumerGroupDeleted(WireConsumerGroupDeletedBody { + topic: topic_key(), + group_id: 0, + name: name("readers")?, + consumer_groups_count: 0, + requested_op: Some(4300), + }), + Some(4), + ), + ( + "consumer_group_member_joined", + WireMetadataChange::ConsumerGroupMemberJoined(WireMemberJoinedBody { + topic: topic_key(), + group_id: 0, + member_id: 1, + client_id: 0x0102_0304_0506_0708_090a_0b0c_0d0e_0f10, + members_count: 2, + partition_ids: vec![], + }), + Some(9), + ), + ( + "consumer_group_member_left_session_ended", + WireMetadataChange::ConsumerGroupMemberLeft(WireMemberLeftBody { + topic: topic_key(), + group_id: 0, + member_id: 1, + client_id: 0x0102_0304_0506_0708_090a_0b0c_0d0e_0f10, + members_count: 1, + reason: WireMemberLeftReason::SessionEnded, + }), + Some(9), + ), + ( + "consumer_group_partition_handed_over", + WireMetadataChange::ConsumerGroupPartitionHandedOver(WirePartitionHandedOverBody { + topic: topic_key(), + group_id: 0, + partition_id: 2, + member_id: Some(1), + }), + None, + ), + ( + "consumer_group_partition_released", + WireMetadataChange::ConsumerGroupPartitionHandedOver(WirePartitionHandedOverBody { + topic: topic_key(), + group_id: 0, + partition_id: 3, + member_id: None, + }), + None, + ), + ( + "user_created_with_permissions", + WireMetadataChange::UserCreated(WireUserBody { + key: user_key(), + username: name("telemetry-writer")?, + status: 1, + permissions: Some(permissions_to_wire(&permissions())), + options: empty_options(), + }), + Some(0), + ), + ( + "user_updated_inactive", + WireMetadataChange::UserUpdated(WireUserBody { + key: user_key(), + username: name("telemetry-writer")?, + status: 2, + permissions: None, + options: empty_options(), + }), + Some(0), + ), + ( + "user_deleted", + WireMetadataChange::UserDeleted(WireUserIdentityBody { + key: user_key(), + username: name("telemetry-writer")?, + }), + Some(0), + ), + ( + "user_password_changed", + WireMetadataChange::UserPasswordChanged(WireUserIdentityBody { + key: user_key(), + username: name("telemetry-writer")?, + }), + Some(7), + ), + ( + "user_permissions_pruned", + WireMetadataChange::UserPermissionsUpdated(WireUserPermissionsBody { + key: user_key(), + username: name("telemetry-writer")?, + permissions: Some(permissions_to_wire(&Permissions::default())), + reason: WirePermissionsUpdatedReason::ResourceDeleted, + }), + Some(4), + ), + ( + "personal_access_token_created", + WireMetadataChange::PersonalAccessTokenCreated(WireTokenBody { + user: user_key(), + name: token_name()?, + expiry: 1_760_400_000_000_000, + }), + Some(7), + ), + ( + "personal_access_token_deleted_expired", + WireMetadataChange::PersonalAccessTokenDeleted(WireTokenDeletedBody { + user: user_key(), + name: token_name()?, + reason: WireTokenDeletedReason::Expired, + }), + None, + ), + ( + "session_registered", + WireMetadataChange::SessionRegistered(WireSessionBody { + user: user_key(), + client_id: 0x0102_0304_0506_0708_090a_0b0c_0d0e_0f10, + session: 4500, + }), + Some(7), + ), + ( + "session_closed_expired", + WireMetadataChange::SessionClosed(WireSessionClosedBody { + user: user_key(), + client_id: 0x0102_0304_0506_0708_090a_0b0c_0d0e_0f10, + session: 4500, + reason: WireSessionClosedReason::Expired, + }), + None, + ), + ( + "session_closed_revoked", + WireMetadataChange::SessionClosed(WireSessionClosedBody { + user: user_key(), + client_id: 0x0102_0304_0506_0708_090a_0b0c_0d0e_0f10, + session: 4500, + reason: WireSessionClosedReason::Revoked, + }), + Some(7), + ), + ]) +} + +fn record_vectors() -> FixtureResult<Vec<RecordVector>> { + change_cases()? + .into_iter() + .enumerate() + .map(|(index, (case, change, actor))| { + let op = 4121 + u64::try_from(index)?; + let record = wire_record(&change, op, 0, actor); + Ok(RecordVector { + case: case.to_owned(), + group: record.group, + kind: record.kind, + body_hex: hex::encode(&record.body), + record_hex: record_hex(&record), + expected: MetadataChangeRecord::try_from(record)?, + }) + }) + .collect() +} + +fn item_vectors() -> FixtureResult<Vec<ItemVector>> { + let items = vec![ + ( + "stream", + WireMetadataCurrentItem::Stream(WireStreamBody { + key: stream_key(), + created_at: 1_760_356_800_123_456, + name: name("telemetry")?, + options: stream_options()?, + }), + ), + ( + "topic", + WireMetadataCurrentItem::Topic(WireTopicItemBody { + key: topic_key(), + created_at: 1_760_356_801_000_000, + name: name("node-metrics")?, + options: topic_options()?, + }), + ), + ( + "partitions", + WireMetadataCurrentItem::Partitions(WirePartitionsItemBody { + topic: topic_key(), + partitions_count: 8, + }), + ), + ( + "consumer_group", + WireMetadataCurrentItem::ConsumerGroup(WireConsumerGroupItemBody { + topic: topic_key(), + group_id: 0, + name: name("readers")?, + }), + ), + ( + "consumer_group_member", + WireMetadataCurrentItem::ConsumerGroupMember(WireMemberItemBody { + topic: topic_key(), + group_id: 0, + member_id: 1, + client_id: 0x0102_0304_0506_0708_090a_0b0c_0d0e_0f10, + partition_ids: vec![0, 2, 4], + }), + ), + ( + "user", + WireMetadataCurrentItem::User(WireUserBody { + key: user_key(), + username: name("telemetry-writer")?, + status: 1, + permissions: Some(permissions_to_wire(&permissions())), + options: empty_options(), + }), + ), + ( + "personal_access_token_without_expiry", + WireMetadataCurrentItem::PersonalAccessToken(WireTokenBody { + user: user_key(), + name: name("ci-token")?, + expiry: 0, + }), + ), + ( + "session", + WireMetadataCurrentItem::Session(WireSessionBody { + user: user_key(), + client_id: 0x0102_0304_0506_0708_090a_0b0c_0d0e_0f10, + session: 4500, + }), + ), + ]; + items + .into_iter() + .map(|(case, item)| { + let envelope = WireMetadataItem { + group: item.group().as_u8(), + body: item.body_bytes(), + }; + Ok(ItemVector { + case: case.to_owned(), + group: envelope.group, + body_hex: hex::encode(&envelope.body), + item_hex: hex::encode(envelope.to_bytes()), + expected: MetadataCurrentItem::try_from(envelope)?, + }) + }) + .collect() +} + +fn cursor_vector(case: &str, cursor: MetadataChangeCursor) -> CursorVector { + CursorVector { + case: case.to_owned(), + text: cursor.to_string(), + hex: hex::encode(WireMetadataChangeCursor::from(cursor).to_bytes()), + log_id: format!("{:032x}", cursor.log_id), + op: cursor.op, + checksum: format!("{:032x}", cursor.checksum), + ordinal: cursor.ordinal, + } +} + +fn cursor_vectors() -> Vec<CursorVector> { + vec![ + cursor_vector("zero", MetadataChangeCursor::zero(LOG_ID)), + cursor_vector( + "first_record_of_op", + MetadataChangeCursor { + log_id: LOG_ID, + op: 4120, + checksum: 0x4c2d_9e8b_03f7_c1a5, + ordinal: 0, + }, + ), + cursor_vector( + "inside_op", + MetadataChangeCursor { + log_id: LOG_ID, + op: 4187, + checksum: 0xf2a4_c6e8_b0d1_f359, + ordinal: 1, + }, + ), + cursor_vector( + "every_field_at_maximum", + MetadataChangeCursor { + log_id: u128::MAX, + op: u64::MAX, + checksum: u128::MAX, + ordinal: u32::MAX, + }, + ), + ] +} + +fn page_token( + group: WireMetadataChangeGroup, + groups_mask: u8, + last_key: Vec<u8>, +) -> MetadataCurrentPageToken { + MetadataCurrentPageToken::from(WireMetadataCurrentPageToken { + anchor: MetadataChangeCursor { + log_id: LOG_ID, + op: 4187, + checksum: 0xf2a4_c6e8_b0d1_f359, + ordinal: 1, + } + .into(), + required_head: 4190, + groups_mask, + group, + last_key: Bytes::from(last_key), + }) +} + +fn token_name_key(user_id: u32, token_name: &str) -> Vec<u8> { + let mut key = user_id.to_le_bytes().to_vec(); + key.extend( + WireName::new(token_name) + .map(|name| name.to_bytes().to_vec()) + .unwrap_or_default(), + ); + key +} + +fn page_token_vectors() -> Vec<PageTokenVector> { + let all = u8::MAX; + let ids = |values: &[u32]| { + values + .iter() + .flat_map(|value| value.to_le_bytes()) + .collect::<Vec<u8>>() + }; + let cases = vec![ + ("streams", WireMetadataChangeGroup::Streams, 0b1, ids(&[3])), + ( + "topics", + WireMetadataChangeGroup::Topics, + 0b11, + ids(&[3, 0]), + ), + ( + "partitions", + WireMetadataChangeGroup::Partitions, + all, + ids(&[3, 0]), + ), + ( + "consumer_groups", + WireMetadataChangeGroup::ConsumerGroups, + all, + ids(&[3, 0, 2]), + ), + ( + "consumer_group_members", + WireMetadataChangeGroup::ConsumerGroupMembers, + all, + ids(&[3, 0, 2, 1]), + ), + ( + "users", + WireMetadataChangeGroup::Users, + 0b10_0000, + ids(&[7]), + ), + ( + "personal_access_tokens", + WireMetadataChangeGroup::PersonalAccessTokens, + 0b110_0000, + token_name_key(7, "ci-token"), + ), + ( + "sessions", + WireMetadataChangeGroup::Sessions, + 0b1010_0000, + 4500u64.to_le_bytes().to_vec(), + ), + ( + "longest_token_name", + WireMetadataChangeGroup::PersonalAccessTokens, + all, + token_name_key(u32::MAX, &"n".repeat(255)), + ), + ]; + cases + .into_iter() + .map(|(case, group, mask, key)| { + let token = page_token(group, mask, key); + let wire: &WireMetadataCurrentPageToken = token.as_ref(); + PageTokenVector { + case: case.to_owned(), + text: token.to_string(), + hex: hex::encode(wire.to_bytes()), + anchor: token.anchor().to_string(), + required_head: token.required_head(), + groups_mask: token.groups().mask(), + group: token.last_group().to_string(), + last_key_hex: hex::encode(token.last_key()), + } + }) + .collect() +} + +fn request_vectors() -> FixtureResult<Vec<RequestVector>> { + let cursor = MetadataChangeCursor { + log_id: LOG_ID, + op: 4187, + checksum: 0xf2a4_c6e8_b0d1_f359, + ordinal: 1, + } + .to_string(); + let topics_token = page_token( + WireMetadataChangeGroup::Topics, + 0b11, + [3u32, 0] + .iter() + .flat_map(|value| value.to_le_bytes()) + .collect(), + ) + .to_string(); + let inputs = vec![ + ("earliest_every_group", QueryInput::new("earliest")), + ( + "earliest_many_duplicate_groups", + QueryInput { + groups: std::iter::repeat_n("streams".to_owned(), 255) + .chain(std::iter::once("users".to_owned())) + .collect(), + ..QueryInput::new("earliest") + }, + ), + ( + "earliest_since_users_and_tokens", + QueryInput { + groups: vec!["users".to_owned(), "personal_access_tokens".to_owned()], + since: Some(1_760_353_200_000_000), + ..QueryInput::new("earliest") + }, + ), + ( + "latest_streams", + QueryInput { + groups: vec!["streams".to_owned()], + limit: 1, + ..QueryInput::new("latest") + }, + ), + ( + "after_cursor_with_time_range", + QueryInput { + cursor: Some(cursor), + groups: vec!["topics".to_owned(), "streams".to_owned()], + since: Some(1_760_356_800_000_000), + until: Some(1_760_360_400_000_000), + limit: 500, + ..QueryInput::new("after") + }, + ), + ( + "current_first_page", + QueryInput { + groups: vec!["streams".to_owned(), "topics".to_owned()], + limit: 1000, + ..QueryInput::new("current") + }, + ), + ( + "current_continuation", + QueryInput { + page_token: Some(topics_token), + groups: vec!["streams".to_owned(), "topics".to_owned()], + limit: 1000, + ..QueryInput::new("current") + }, + ), + ]; + inputs + .into_iter() + .map(|(case, input)| { + let query = input.query()?; + query.validate()?; + let request = GetMetadataChangesRequest::from(&query); + let http_query = query + .http_query_pairs() + .into_iter() + .map(|(key, value)| format!("{key}={value}")) + .collect::<Vec<_>>() + .join("&"); + Ok(RequestVector { + case: case.to_owned(), + input, + request_hex: hex::encode(request.to_bytes()), + http_query, + }) + }) + .collect() +} + +fn response_vector( + case: &str, + response: GetMetadataChangesResponse, +) -> FixtureResult<ResponseVector> { + let response_hex = hex::encode(response.to_bytes()); + Ok(ResponseVector { + case: case.to_owned(), + response_hex, + expected: MetadataChanges::try_from(response)?, + }) +} + +fn wire_cursor(op: u64, ordinal: u32) -> WireMetadataChangeCursor { + WireMetadataChangeCursor { + log_id: LOG_ID, + op, + checksum: 0x0000_0000_0000_0000_a5c1_f703_8b9e_2d4c + u128::from(op), + ordinal, + } +} + +fn response_vectors( + records: &[RecordVector], + items: &[ItemVector], +) -> FixtureResult<Vec<ResponseVector>> { + let decode_record = |vector: &RecordVector| -> FixtureResult<WireMetadataChangeRecord> { + let bytes = hex::decode(&vector.record_hex)?; + Ok(iggy_binary_protocol::WireDecode::decode_from(&bytes)?) + }; + let decode_item = |vector: &ItemVector| -> FixtureResult<WireMetadataItem> { + let bytes = hex::decode(&vector.item_hex)?; + Ok(iggy_binary_protocol::WireDecode::decode_from(&bytes)?) + }; + let page_records = records + .iter() + .take(3) + .map(decode_record) + .collect::<FixtureResult<Vec<_>>>()?; + let last = page_records.last().ok_or("no records")?; + let changes = GetMetadataChangesResponse { + status: WireMetadataChangeStatus::Ok, + phase: WireMetadataChangePhase::Changes, + log_id: Some(LOG_ID), + oldest_op: 3871, + retained_since: 1_760_352_000_000_000, + head_op: 4190, + next: Some(wire_cursor(last.op, last.ordinal)), + has_more: true, + records: page_records, + items: vec![], + current_page_token: None, + }; + let mut unknown_record = decode_record(records.first().ok_or("no records")?)?; + unknown_record.kind = 9; + unknown_record.body = Bytes::from_static(&[0xde, 0xad, 0xbe, 0xef]); + let mut future_group = unknown_record.clone(); + future_group.group = 200; + future_group.kind = 1; + future_group.ordinal = 1; + let unknown = GetMetadataChangesResponse { + next: Some(wire_cursor(future_group.op, future_group.ordinal)), + has_more: false, + records: vec![unknown_record, future_group], + ..changes.clone() + }; + let current_items = items + .iter() + .take(2) + .map(decode_item) + .collect::<FixtureResult<Vec<_>>>()?; + let current_token = page_token( + WireMetadataChangeGroup::Topics, + 0b11, + [3u32, 0] + .iter() + .flat_map(|value| value.to_le_bytes()) + .collect(), + ); + let current = GetMetadataChangesResponse { + status: WireMetadataChangeStatus::Ok, + phase: WireMetadataChangePhase::Current, + log_id: Some(LOG_ID), + oldest_op: 1, + retained_since: 0, + head_op: 4190, + next: Some(wire_cursor(4187, 1)), + has_more: true, + records: vec![], + items: current_items, + current_page_token: Some(current_token.into()), + }; + let behind = GetMetadataChangesResponse { + status: WireMetadataChangeStatus::Behind, + phase: WireMetadataChangePhase::Changes, + log_id: None, + oldest_op: 0, + retained_since: 0, + head_op: 0, + next: None, + has_more: false, + records: vec![], + items: vec![], + current_page_token: None, + }; + let expired = GetMetadataChangesResponse { + status: WireMetadataChangeStatus::Expired, + log_id: Some(LOG_ID), + oldest_op: 5000, + retained_since: 1_760_360_000_000_000, + head_op: 9000, + ..behind.clone() + }; + let reset = GetMetadataChangesResponse { + status: WireMetadataChangeStatus::Reset, + ..expired.clone() + }; + Ok(vec![ + response_vector("changes_page", changes)?, + response_vector("unknown_kind_and_group", unknown)?, + response_vector("current_page", current)?, + response_vector("behind_before_op_1", behind)?, + response_vector("expired", expired)?, + response_vector("reset", reset)?, + ]) +} + +fn large_values_vector() -> FixtureResult<LargeValuesVector> { + let client_id = u128::MAX - 1; + let change = WireMetadataChange::SessionRegistered(WireSessionBody { + user: WireUserKey { + user_id: u32::MAX, + user_created_at: ABOVE_2_POW_53, + }, + client_id, + session: ABOVE_2_POW_53 + 1, + }); + let mut record = wire_record(&change, ABOVE_2_POW_53 + 2, u32::MAX, Some(u32::MAX)); + record.checksum = u128::MAX; + record.timestamp = u64::MAX; + Ok(LargeValuesVector { + record_hex: record_hex(&record), + op: record.op.to_string(), + ordinal: record.ordinal, + timestamp: record.timestamp.to_string(), + checksum: format!("{:032x}", record.checksum), + user_created_at: ABOVE_2_POW_53.to_string(), + session: (ABOVE_2_POW_53 + 1).to_string(), + client_id: format!("{client_id:032x}"), + }) +} + +fn refused_vectors() -> FixtureResult<RefusedVectors> { + let valid_cursor = MetadataChangeCursor { + log_id: LOG_ID, + op: 4187, + checksum: 0xf2a4_c6e8_b0d1_f359, + ordinal: 1, + } + .to_string(); + let encoded = &valid_cursor[4..]; + let non_canonical_cursor = WireMetadataChangeCursor { + log_id: LOG_ID, + op: 0, + checksum: 1, + ordinal: 0, + }; + let cursors = vec![ + ("wrong_prefix", format!("mc2_{encoded}")), + ("missing_prefix", encoded.to_owned()), + ("too_long", format!("{valid_cursor}A")), + ("too_short", valid_cursor[..62].to_owned()), + ("padded", format!("mc1_{}=", &encoded[..58])), + ( + "standard_alphabet_character", + format!("mc1_+{}", &encoded[1..]), + ), + ( + "op_zero_with_checksum", + format!( + "mc1_{}", + base64_url(&WireMetadataChangeCursor::to_bytes(&non_canonical_cursor)) + ), + ), + ]; + let valid_token = page_token( + WireMetadataChangeGroup::Streams, + 0b1, + 3u32.to_le_bytes().to_vec(), + ); + let token_text = valid_token.to_string(); + let mut wrong_width = WireMetadataCurrentPageToken::from(valid_token.clone()); + wrong_width.last_key = Bytes::from_static(&[3, 0, 0, 0, 0, 0, 0, 0]); + let mut outside_mask = WireMetadataCurrentPageToken::from(valid_token.clone()); + outside_mask.groups_mask = 0b10; + let mut head_below_anchor = WireMetadataCurrentPageToken::from(valid_token); + head_below_anchor.required_head = 1; + let page_tokens = vec![ + ("wrong_prefix", token_text.replacen("cs1_", "mc1_", 1)), + ( + "trailing_byte", + format!( + "cs1_{}", + base64_url(&[unvalidated_token_bytes(&token_text)?, vec![0]].concat()) + ), + ), + ( + "wrong_key_width", + format!("cs1_{}", base64_url(&wrong_width.to_bytes())), + ), + ( + "group_outside_mask", + format!("cs1_{}", base64_url(&outside_mask.to_bytes())), + ), + ( + "head_below_anchor", + format!("cs1_{}", base64_url(&head_below_anchor.to_bytes())), + ), + ("empty", "cs1_".to_owned()), + ]; + + let records = change_cases()?; + let body = |case: &str| -> FixtureResult<(WireMetadataChange, Vec<u8>)> { + let (_, change, _) = records + .iter() + .find(|(name, _, _)| *name == case) + .ok_or("missing case")?; + Ok((change.clone(), change.body_bytes().to_vec())) + }; + let (member_left, mut bad_reason) = body("consumer_group_member_left_session_ended")?; + if let Some(last) = bad_reason.last_mut() { + *last = 2; + } + let (stream_deleted, mut bad_presence) = body("stream_deleted_deferred")?; + let presence_offset = bad_presence.len() - 9; + bad_presence[presence_offset] = 2; + let (partitions, mut unsorted_ids) = body("partitions_changed_deleted_tail")?; + let first_id = 24 + 4 + 4; + unsorted_ids[first_id..first_id + 4].copy_from_slice(&9u32.to_le_bytes()); + let (stream_created, stream_body) = body("stream_created")?; + let (session, session_body) = body("session_registered")?; + let record_bodies = vec![ + RefusedBody { + case: "unknown_member_left_reason".to_owned(), + group: member_left.group().as_u8(), + kind: member_left.kind().as_u8(), + body_hex: hex::encode(&bad_reason), + }, + RefusedBody { + case: "presence_byte_two".to_owned(), + group: stream_deleted.group().as_u8(), + kind: stream_deleted.kind().as_u8(), + body_hex: hex::encode(&bad_presence), + }, + RefusedBody { + case: "unsorted_removed_partition_ids".to_owned(), + group: partitions.group().as_u8(), + kind: partitions.kind().as_u8(), + body_hex: hex::encode(&unsorted_ids), + }, + RefusedBody { + case: "known_kind_under_wrong_group".to_owned(), + group: WireMetadataChangeGroup::Topics.as_u8(), + kind: stream_created.kind().as_u8(), + body_hex: hex::encode(&stream_body), + }, + RefusedBody { + case: "trailing_byte".to_owned(), + group: session.group().as_u8(), + kind: session.kind().as_u8(), + body_hex: hex::encode([session_body.as_slice(), &[0]].concat()), + }, + RefusedBody { + case: "truncated".to_owned(), + group: session.group().as_u8(), + kind: session.kind().as_u8(), + body_hex: hex::encode(&session_body[..session_body.len() - 1]), + }, + ]; + + let earliest = GetMetadataChangesRequest::from(&MetadataChangesQuery::default()) + .to_bytes() + .to_vec(); + let mut cursor_without_after = earliest.clone(); + cursor_without_after[5] = 1; + let mut trailing = earliest.clone(); + trailing.push(0); + let mut unknown_start = earliest; + unknown_start[0] = 4; + let mut non_canonical_after = vec![WireMetadataChangeStartMode::After.as_u8()]; + non_canonical_after.extend(WireMetadataChangeCursor::to_bytes(&non_canonical_cursor)); + non_canonical_after.extend([0, 0]); + non_canonical_after.extend(DEFAULT_METADATA_CHANGES_LIMIT.to_le_bytes()); + non_canonical_after.extend([0, 0, 0]); + let requests = vec![ + RefusedHex { + case: "cursor_bytes_without_after".to_owned(), + hex: hex::encode(&cursor_without_after), + }, + RefusedHex { + case: "trailing_byte".to_owned(), + hex: hex::encode(&trailing), + }, + RefusedHex { + case: "unknown_start_mode".to_owned(), + hex: hex::encode(&unknown_start), + }, + RefusedHex { + case: "non_canonical_cursor".to_owned(), + hex: hex::encode(&non_canonical_after), + }, + ]; + + let items_in_changes_phase = GetMetadataChangesResponse { + status: WireMetadataChangeStatus::Ok, + phase: WireMetadataChangePhase::Changes, + log_id: Some(LOG_ID), + oldest_op: 1, + retained_since: 0, + head_op: 2, + next: Some(wire_cursor(2, 0)), + has_more: false, + records: vec![], + items: vec![WireMetadataItem { + group: 1, + body: item_body_for_refusal()?, + }], + current_page_token: None, + } + .to_bytes() + .to_vec(); + let mut unknown_status = items_in_changes_phase.clone(); + unknown_status[0] = 4; + let responses = vec![ + RefusedHex { + case: "items_in_changes_phase".to_owned(), + hex: hex::encode(&items_in_changes_phase), + }, + RefusedHex { + case: "unknown_status".to_owned(), + hex: hex::encode(&unknown_status), + }, + ]; + + let queries = vec![ + ( + "limit_zero", + QueryInput { + limit: 0, + ..QueryInput::new("earliest") + }, + ), + ( + "limit_above_maximum", + QueryInput { + limit: MAX_METADATA_CHANGES_LIMIT + 1, + ..QueryInput::new("earliest") + }, + ), + ( + "since_above_until", + QueryInput { + since: Some(2), + until: Some(1), + ..QueryInput::new("earliest") + }, + ), + ( + "time_bound_with_latest", + QueryInput { + since: Some(1), + ..QueryInput::new("latest") + }, + ), + ( + "time_bound_with_current", + QueryInput { + until: Some(1), + ..QueryInput::new("current") + }, + ), + ( + "current_missing_parent_group", + QueryInput { + groups: vec!["topics".to_owned()], + ..QueryInput::new("current") + }, + ), + ( + "page_token_for_other_groups", + QueryInput { + page_token: Some( + page_token( + WireMetadataChangeGroup::Streams, + 0b1, + 3u32.to_le_bytes().to_vec(), + ) + .to_string(), + ), + groups: vec!["streams".to_owned(), "topics".to_owned()], + ..QueryInput::new("current") + }, + ), + ]; + let http_queries = vec![ + ("no_start", "groups=streams"), + ("from_and_after", "from=earliest&after=mc1_x"), + ("unknown_from", "from=newest"), + ("duplicate_key", "from=earliest&from=latest"), + ("unknown_key", "from=earliest&verbose=1"), + ( + "empty_group_element", + "from=earliest&groups=streams,,topics", + ), + ("trailing_group_comma", "from=earliest&groups=streams,"), + ("unknown_group", "from=earliest&groups=everything"), + ("negative_since", "from=earliest&since=-1"), + ("leading_zero_until", "from=earliest&until=01"), + ("non_numeric_limit", "from=earliest&limit=ten"), + ( + "token_without_current", + "from=latest¤t_after=cs1_AAAA", + ), + ]; + Ok(RefusedVectors { + cursors: cursors + .into_iter() + .map(|(case, text)| RefusedText { + case: case.to_owned(), + text, + }) + .collect(), + page_tokens: page_tokens + .into_iter() + .map(|(case, text)| RefusedText { + case: case.to_owned(), + text, + }) + .collect(), + record_bodies, + requests, + responses, + queries: queries + .into_iter() + .map(|(case, input)| RefusedQuery { + case: case.to_owned(), + input, + }) + .collect(), + http_queries: http_queries + .into_iter() + .map(|(case, query)| RefusedHttpQuery { + case: case.to_owned(), + query: query.to_owned(), + }) + .collect(), + }) +} + +fn item_body_for_refusal() -> FixtureResult<Bytes> { + Ok(WireMetadataCurrentItem::Stream(WireStreamBody { + key: stream_key(), + created_at: 1, + name: name("orders")?, + options: empty_options(), + }) + .body_bytes()) +} + +fn unvalidated_token_bytes(text: &str) -> FixtureResult<Vec<u8>> { + let encoded = text.strip_prefix("cs1_").ok_or("missing prefix")?; + Ok(URL_SAFE_NO_PAD.decode(encoded)?) +} + +fn base64_url(bytes: &[u8]) -> String { + URL_SAFE_NO_PAD.encode(bytes) +}
diff --git a/core/common/fixtures/metadata_changes.json b/core/common/fixtures/metadata_changes.json new file mode 100644 index 0000000..26e7faf --- /dev/null +++ b/core/common/fixtures/metadata_changes.json
@@ -0,0 +1,1997 @@ +{ + "schema_version": 1, + "generated_by": "core/common/examples/metadata_changes_fixtures.rs", + "conventions": { + "byte_strings": "Every *_hex field is lowercase hex without separators.", + "expected": "expected is the JSON the HTTP route renders. Binary decoders must reach the same values.", + "record_hex": "record_hex is one record envelope as a response carries it. body_hex is the body alone.", + "refused": "Every refused vector must fail to decode or validate. Query and HTTP refusals are invalid_command.", + "request_hex": "request_hex is the command payload without framing. http_query is the query string of GET /metadata/changes.", + "u128_values": "128-bit values (log ids, checksums, client ids) are 32 lowercase hex characters in JSON.", + "u64_values": "Ops and timestamps are JSON numbers below 2^53 in every vector except large_values, which carries them as decimal strings.", + "unknown_kinds": "A record of an unknown group or kind is kept with its codes and raw body, never refused." + }, + "constants": { + "command_code": 17, + "command_name": "metadata.changes", + "cursor_prefix": "mc1_", + "page_token_prefix": "cs1_", + "cursor_size": 44, + "cursor_text_length": 63, + "max_page_token_size": 316, + "max_page_token_key_size": 260, + "default_limit": 100, + "max_limit": 1000, + "groups": { + "consumer_group_members": { + "code": 5, + "class": "activity", + "parents": [ + "consumer_groups", + "partitions", + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32][member_id:u32]" + }, + "consumer_groups": { + "code": 4, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32]" + }, + "partitions": { + "code": 3, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "personal_access_tokens": { + "code": 7, + "class": "resource", + "parents": [ + "users" + ], + "page_token_key": "[user_id:u32][name_len:u8][name]" + }, + "sessions": { + "code": 8, + "class": "activity", + "parents": [ + "users" + ], + "page_token_key": "[epoch:u64]" + }, + "streams": { + "code": 1, + "class": "resource", + "parents": [], + "page_token_key": "[stream_id:u32]" + }, + "topics": { + "code": 2, + "class": "resource", + "parents": [ + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "users": { + "code": 6, + "class": "resource", + "parents": [], + "page_token_key": "[user_id:u32]" + } + }, + "kinds": { + "consumer_group_created": { + "code": 30, + "group": "consumer_groups" + }, + "consumer_group_deleted": { + "code": 31, + "group": "consumer_groups" + }, + "consumer_group_member_joined": { + "code": 40, + "group": "consumer_group_members" + }, + "consumer_group_member_left": { + "code": 41, + "group": "consumer_group_members" + }, + "consumer_group_partition_handed_over": { + "code": 42, + "group": "consumer_group_members" + }, + "partitions_changed": { + "code": 20, + "group": "partitions" + }, + "personal_access_token_created": { + "code": 60, + "group": "personal_access_tokens" + }, + "personal_access_token_deleted": { + "code": 61, + "group": "personal_access_tokens" + }, + "session_closed": { + "code": 71, + "group": "sessions" + }, + "session_registered": { + "code": 70, + "group": "sessions" + }, + "stream_created": { + "code": 1, + "group": "streams" + }, + "stream_deleted": { + "code": 3, + "group": "streams" + }, + "stream_updated": { + "code": 2, + "group": "streams" + }, + "topic_created": { + "code": 10, + "group": "topics" + }, + "topic_deleted": { + "code": 12, + "group": "topics" + }, + "topic_updated": { + "code": 11, + "group": "topics" + }, + "user_created": { + "code": 50, + "group": "users" + }, + "user_deleted": { + "code": 52, + "group": "users" + }, + "user_password_changed": { + "code": 53, + "group": "users" + }, + "user_permissions_updated": { + "code": 54, + "group": "users" + }, + "user_updated": { + "code": 51, + "group": "users" + } + }, + "statuses": { + "behind": 1, + "expired": 2, + "ok": 0, + "reset": 3 + }, + "phases": { + "changes": 0, + "current": 1 + }, + "start_modes": { + "after": 2, + "current": 3, + "earliest": 0, + "latest": 1 + }, + "member_left_reasons": { + "left": 0, + "session_ended": 1 + }, + "token_deleted_reasons": { + "deleted": 0, + "expired": 1 + }, + "session_closed_reasons": { + "expired": 1, + "logout": 0, + "revoked": 2 + }, + "permissions_updated_reasons": { + "resource_deleted": 1, + "updated": 0 + } + }, + "records": [ + { + "case": "stream_created", + "group": 1, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "record_hex": "191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "stream_updated_unreadable_options", + "group": 1, + "kind": 2, + "body_hex": "03000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "record_hex": "1a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "expected": { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + } + }, + { + "case": "stream_deleted_deferred", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "record_hex": "1b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "expected": { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + }, + { + "case": "topic_created", + "group": 2, + "kind": 10, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "record_hex": "1c1000000000000000000000683d9e8b03f7c1a500000000000000001c00be01094106000104000000020a78000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "op": 4124, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d68", + "timestamp": 1760356800004124, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "partitions_count": 8, + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "topic_updated", + "group": 2, + "kind": 11, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "record_hex": "1d1000000000000000000000693d9e8b03f7c1a500000000000000001d00be01094106000104000000020b3d000000030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "expected": { + "op": 4125, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d69", + "timestamp": 1760356800004125, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_updated", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics-v2", + "partitions_count": 8, + "options": {} + } + }, + { + "case": "topic_deleted_immediate", + "group": 2, + "kind": 12, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "record_hex": "1e10000000000000000000006a3d9e8b03f7c1a500000000000000001e00be01094106000104000000020c2e000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "expected": { + "op": 4126, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6a", + "timestamp": 1760356800004126, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics" + } + }, + { + "case": "partitions_changed_deleted_tail", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "record_hex": "1f10000000000000000000006b3d9e8b03f7c1a500000000000000001f00be01094106000104000000031435000000030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "expected": { + "op": 4127, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6b", + "timestamp": 1760356800004127, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 5, + "removed_partition_ids": [ + 5, + 6, + 7 + ], + "requested_op": 4200 + } + }, + { + "case": "partitions_changed_created", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000a0000000000000000", + "record_hex": "2010000000000000000000006c3d9e8b03f7c1a500000000000000002000be01094106000104000000031421000000030000001910000000000000000000001d100000000000000a0000000000000000", + "expected": { + "op": 4128, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6c", + "timestamp": 1760356800004128, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 10, + "removed_partition_ids": [] + } + }, + { + "case": "consumer_group_created", + "group": 4, + "kind": 30, + "body_hex": "030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "record_hex": "2110000000000000000000006d3d9e8b03f7c1a500000000000000002100be01094106000104000000041e28000000030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "expected": { + "op": 4129, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6d", + "timestamp": 1760356800004129, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 1 + } + }, + { + "case": "consumer_group_deleted", + "group": 4, + "kind": 31, + "body_hex": "030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "record_hex": "2210000000000000000000006e3d9e8b03f7c1a500000000000000002200be01094106000104000000041f31000000030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "expected": { + "op": 4130, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6e", + "timestamp": 1760356800004130, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 0, + "requested_op": 4300 + } + }, + { + "case": "consumer_group_member_joined", + "group": 5, + "kind": 40, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "record_hex": "2310000000000000000000006f3d9e8b03f7c1a500000000000000002300be01094106000109000000052838000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "expected": { + "op": 4131, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6f", + "timestamp": 1760356800004131, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_joined", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 2, + "partition_ids": [] + } + }, + { + "case": "consumer_group_member_left_session_ended", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "record_hex": "241000000000000000000000703d9e8b03f7c1a500000000000000002400be01094106000109000000052935000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "expected": { + "op": 4132, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d70", + "timestamp": 1760356800004132, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_left", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 1, + "reason": "session_ended" + } + }, + { + "case": "consumer_group_partition_handed_over", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d1000000000000000000000020000000101000000", + "record_hex": "251000000000000000000000713d9e8b03f7c1a500000000000000002500be010941060000052a25000000030000001910000000000000000000001d1000000000000000000000020000000101000000", + "expected": { + "op": 4133, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d71", + "timestamp": 1760356800004133, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 2, + "member_id": 1 + } + }, + { + "case": "consumer_group_partition_released", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d10000000000000000000000300000000", + "record_hex": "261000000000000000000000723d9e8b03f7c1a500000000000000002600be010941060000052a21000000030000001910000000000000000000001d10000000000000000000000300000000", + "expected": { + "op": 4134, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d72", + "timestamp": 1760356800004134, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 3 + } + }, + { + "case": "user_created_with_permissions", + "group": 6, + "kind": 50, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "record_hex": "271000000000000000000000733d9e8b03f7c1a500000000000000002700be010941060001000000000632480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "op": 4135, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d73", + "timestamp": 1760356800004135, + "actor_user_id": 0, + "group": "users", + "kind": "user_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "user_updated_inactive", + "group": 6, + "kind": 51, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "record_hex": "281000000000000000000000743d9e8b03f7c1a500000000000000002800be010941060001000000000633280000000700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "expected": { + "op": 4136, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d74", + "timestamp": 1760356800004136, + "actor_user_id": 0, + "group": "users", + "kind": "user_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "inactive", + "permissions": null, + "options": {} + } + }, + { + "case": "user_deleted", + "group": 6, + "kind": 52, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "291000000000000000000000753d9e8b03f7c1a500000000000000002900be0109410600010000000006341d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4137, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d75", + "timestamp": 1760356800004137, + "actor_user_id": 0, + "group": "users", + "kind": "user_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_password_changed", + "group": 6, + "kind": 53, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "2a1000000000000000000000763d9e8b03f7c1a500000000000000002a00be0109410600010700000006351d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4138, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d76", + "timestamp": 1760356800004138, + "actor_user_id": 7, + "group": "users", + "kind": "user_password_changed", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_permissions_pruned", + "group": 6, + "kind": 54, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "record_hex": "2b1000000000000000000000773d9e8b03f7c1a500000000000000002b00be0109410600010400000006362a0000000700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "expected": { + "op": 4139, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d77", + "timestamp": 1760356800004139, + "actor_user_id": 4, + "group": "users", + "kind": "user_permissions_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": false, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": null + }, + "reason": "resource_deleted" + } + }, + { + "case": "personal_access_token_created", + "group": 7, + "kind": 60, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "record_hex": "2c1000000000000000000000783d9e8b03f7c1a500000000000000002c00be01094106000107000000073c1d0000000700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "expected": { + "op": 4140, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d78", + "timestamp": 1760356800004140, + "actor_user_id": 7, + "group": "personal_access_tokens", + "kind": "personal_access_token_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "expiry": 1760400000000000 + } + }, + { + "case": "personal_access_token_deleted_expired", + "group": 7, + "kind": 61, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e01", + "record_hex": "2d1000000000000000000000793d9e8b03f7c1a500000000000000002d00be010941060000073d160000000700000000b832c8fb4006000863692d746f6b656e01", + "expected": { + "op": 4141, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d79", + "timestamp": 1760356800004141, + "group": "personal_access_tokens", + "kind": "personal_access_token_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "reason": "expired" + } + }, + { + "case": "session_registered", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "record_hex": "2e10000000000000000000007a3d9e8b03f7c1a500000000000000002e00be010941060001070000000846240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "op": 4142, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7a", + "timestamp": 1760356800004142, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_registered", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + }, + { + "case": "session_closed_expired", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "record_hex": "2f10000000000000000000007b3d9e8b03f7c1a500000000000000002f00be0109410600000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "expected": { + "op": 4143, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7b", + "timestamp": 1760356800004143, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "expired" + } + }, + { + "case": "session_closed_revoked", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "record_hex": "3010000000000000000000007c3d9e8b03f7c1a500000000000000003000be010941060001070000000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "expected": { + "op": 4144, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7c", + "timestamp": 1760356800004144, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "revoked" + } + } + ], + "items": [ + { + "case": "stream", + "group": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "item_hex": "01b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "topic", + "group": 2, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "item_hex": "0274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "partitions", + "group": 3, + "body_hex": "030000001910000000000000000000001d1000000000000008000000", + "item_hex": "031c000000030000001910000000000000000000001d1000000000000008000000", + "expected": { + "group": "partitions", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 8 + } + }, + { + "case": "consumer_group", + "group": 4, + "body_hex": "030000001910000000000000000000001d10000000000000000000000772656164657273", + "item_hex": "0424000000030000001910000000000000000000001d10000000000000000000000772656164657273", + "expected": { + "group": "consumer_groups", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers" + } + }, + { + "case": "consumer_group_member", + "group": 5, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "item_hex": "0540000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "expected": { + "group": "consumer_group_members", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "partition_ids": [ + 0, + 2, + 4 + ] + } + }, + { + "case": "user", + "group": 6, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "item_hex": "06480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "group": "users", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "personal_access_token_without_expiry", + "group": 7, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "item_hex": "071d0000000700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "expected": { + "group": "personal_access_tokens", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token" + } + }, + { + "case": "session", + "group": 8, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "item_hex": "08240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "group": "sessions", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + } + ], + "cursors": [ + { + "case": "zero", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d000000000000000000000000000000000000000000000000000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 0, + "checksum": "00000000000000000000000000000000", + "ordinal": 0 + }, + { + "case": "first_record_of_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAABgQAAAAAAAApcH3A4ueLUwAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000001810000000000000a5c1f7038b9e2d4c000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4120, + "checksum": "00000000000000004c2d9e8b03f7c1a5", + "ordinal": 0 + }, + { + "case": "inside_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f2000000000000000001000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4187, + "checksum": "0000000000000000f2a4c6e8b0d1f359", + "ordinal": 1 + }, + { + "case": "every_field_at_maximum", + "text": "mc1___________________________________________________________8", + "hex": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + "log_id": "ffffffffffffffffffffffffffffffff", + "op": 18446744073709551615, + "checksum": "ffffffffffffffffffffffffffffffff", + "ordinal": 4294967295 + } + ], + "page_tokens": [ + { + "case": "streams", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000000101040003000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 1, + "group": "streams", + "last_key_hex": "03000000" + }, + { + "case": "topics", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 3, + "group": "topics", + "last_key_hex": "0300000000000000" + }, + { + "case": "partitions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8DCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff0308000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "partitions", + "last_key_hex": "0300000000000000" + }, + { + "case": "consumer_groups", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8EDAADAAAAAAAAAAIAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff040c00030000000000000002000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_groups", + "last_key_hex": "030000000000000002000000" + }, + { + "case": "consumer_group_members", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8FEAADAAAAAAAAAAIAAAABAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff05100003000000000000000200000001000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_group_members", + "last_key_hex": "03000000000000000200000001000000" + }, + { + "case": "users", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAACAGBAAHAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000002006040007000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 32, + "group": "users", + "last_key_hex": "07000000" + }, + { + "case": "personal_access_tokens", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAGAHDQAHAAAACGNpLXRva2Vu", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e1000000000000060070d00070000000863692d746f6b656e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 96, + "group": "personal_access_tokens", + "last_key_hex": "070000000863692d746f6b656e" + }, + { + "case": "sessions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAKAICACUEQAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000a00808009411000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 160, + "group": "sessions", + "last_key_hex": "9411000000000000" + }, + { + "case": "longest_token_name", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8HBAH______25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubg", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff070401ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "personal_access_tokens", + "last_key_hex": "ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e" + } + ], + "requests": [ + { + "case": "earliest_every_group", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000", + "http_query": "from=earliest&limit=100" + }, + { + "case": "earliest_many_duplicate_groups", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "users" + ], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000640000000201060000", + "http_query": "from=earliest&groups=streams,users&limit=100" + }, + { + "case": "earliest_since_users_and_tokens", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "users", + "personal_access_tokens" + ], + "since": 1760353200000000, + "until": null, + "limit": 100 + }, + "request_hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001004c2a2b0841060000640000000206070000", + "http_query": "from=earliest&groups=users,personal_access_tokens&since=1760353200000000&limit=100" + }, + { + "case": "latest_streams", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [ + "streams" + ], + "since": null, + "until": null, + "limit": 1 + }, + "request_hex": "01000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000001010000", + "http_query": "from=latest&groups=streams&limit=1" + }, + { + "case": "after_cursor_with_time_range", + "input": { + "start": "after", + "cursor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "page_token": null, + "groups": [ + "topics", + "streams" + ], + "since": 1760356800000000, + "until": 1760360400000000, + "limit": 500 + }, + "request_hex": "02d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000000100f0bd010941060001009451d809410600f40100000202010000", + "http_query": "after=mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA&groups=streams,topics&since=1760356800000000&until=1760360400000000&limit=500" + }, + { + "case": "current_first_page", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201020000", + "http_query": "from=current&groups=streams,topics&limit=1000" + }, + { + "case": "current_continuation", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201024000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "http_query": "from=current¤t_after=cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA&groups=streams,topics&limit=1000" + } + ], + "responses": [ + { + "case": "changes_page", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001b10000000000000673d9e8b03f7c1a50000000000000000000000000103000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465641a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d7632011b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABsQAAAAAAAAZz2eiwP3waUAAAAAAAAAAAAAAAA", + "has_more": true, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + }, + { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "unknown_kind_and_group", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001910000000000000653d9e8b03f7c1a50000000000000000010000000002000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be01094106000104000000010904000000deadbeef191000000000000001000000653d9e8b03f7c1a500000000000000001900be01094106000104000000c80104000000deadbeef000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABkQAAAAAAAAZT2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": false, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "1", + "kind": "9", + "body": "3q2+7w==" + }, + { + "op": 4121, + "ordinal": 1, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "200", + "kind": "1", + "body": "3q2+7w==" + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "current_page", + "response_hex": "000101d7931c6e2a8f4b0d0000000000000000010000000000000000000000000000005e1000000000000001d7931c6e2a8f4b0d00000000000000005b10000000000000a73d9e8b03f7c1a500000000000000000100000001000000000200000001b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465640274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c0800000000000040000000004000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "expected": { + "status": "ok", + "phase": "current", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 1, + "retained_since": 0, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAApz2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": true, + "records": [], + "items": [ + { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + ], + "current_page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA" + } + }, + { + "case": "behind_before_op_1", + "response_hex": "010000000000000000000000000000000000000000000000000000000000000000000000000000", + "expected": { + "status": "behind", + "phase": "changes", + "log_id": null, + "oldest_op": 0, + "retained_since": 0, + "head_op": 0, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "expired", + "response_hex": "020001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "expired", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "reset", + "response_hex": "030001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "reset", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + } + ], + "large_values": { + "record_hex": "0300000000002000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff01ffffffff084624000000ffffffff0100000000002000feffffffffffffffffffffffffffffff0200000000002000", + "op": "9007199254740995", + "ordinal": 4294967295, + "timestamp": "18446744073709551615", + "checksum": "ffffffffffffffffffffffffffffffff", + "user_created_at": "9007199254740993", + "session": "9007199254740994", + "client_id": "fffffffffffffffffffffffffffffffe" + }, + "refused": { + "cursors": [ + { + "case": "wrong_prefix", + "text": "mc2_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "missing_prefix", + "text": "15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "too_long", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAAA" + }, + { + "case": "too_short", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA" + }, + { + "case": "padded", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA=" + }, + { + "case": "standard_alphabet_character", + "text": "mc1_+5McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "op_zero_with_checksum", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAA" + } + ], + "page_tokens": [ + { + "case": "wrong_prefix", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA" + }, + { + "case": "trailing_byte", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAAAA" + }, + { + "case": "wrong_key_width", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBCAADAAAAAAAAAA" + }, + { + "case": "group_outside_mask", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAIBBAADAAAA" + }, + { + "case": "head_below_anchor", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAABAAAAAAAAAAEBBAADAAAA" + }, + { + "case": "empty", + "text": "cs1_" + } + ], + "record_bodies": [ + { + "case": "unknown_member_left_reason", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000002" + }, + { + "case": "presence_byte_two", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279022c10000000000000" + }, + { + "case": "unsorted_removed_partition_ids", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000090000000600000007000000016810000000000000" + }, + { + "case": "known_kind_under_wrong_group", + "group": 2, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564" + }, + { + "case": "trailing_byte", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000000" + }, + { + "case": "truncated", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a09080706050403020194110000000000" + } + ], + "requests": [ + { + "case": "cursor_bytes_without_after", + "hex": "000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "trailing_byte", + "hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006400000000000000" + }, + { + "case": "unknown_start_mode", + "hex": "040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "non_canonical_cursor", + "hex": "02d7931c6e2a8f4b0d000000000000000000000000000000000100000000000000000000000000000000000000000064000000000000" + } + ], + "responses": [ + { + "case": "items_in_changes_phase", + "hex": "000001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + }, + { + "case": "unknown_status", + "hex": "040001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + } + ], + "queries": [ + { + "case": "limit_zero", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 0 + } + }, + { + "case": "limit_above_maximum", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 1001 + } + }, + { + "case": "since_above_until", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 2, + "until": 1, + "limit": 100 + } + }, + { + "case": "time_bound_with_latest", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 1, + "until": null, + "limit": 100 + } + }, + { + "case": "time_bound_with_current", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": 1, + "limit": 100 + } + }, + { + "case": "current_missing_parent_group", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + }, + { + "case": "page_token_for_other_groups", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + } + ], + "http_queries": [ + { + "case": "no_start", + "query": "groups=streams" + }, + { + "case": "from_and_after", + "query": "from=earliest&after=mc1_x" + }, + { + "case": "unknown_from", + "query": "from=newest" + }, + { + "case": "duplicate_key", + "query": "from=earliest&from=latest" + }, + { + "case": "unknown_key", + "query": "from=earliest&verbose=1" + }, + { + "case": "empty_group_element", + "query": "from=earliest&groups=streams,,topics" + }, + { + "case": "trailing_group_comma", + "query": "from=earliest&groups=streams," + }, + { + "case": "unknown_group", + "query": "from=earliest&groups=everything" + }, + { + "case": "negative_since", + "query": "from=earliest&since=-1" + }, + { + "case": "leading_zero_until", + "query": "from=earliest&until=01" + }, + { + "case": "non_numeric_limit", + "query": "from=earliest&limit=ten" + }, + { + "case": "token_without_current", + "query": "from=latest¤t_after=cs1_AAAA" + } + ] + } +}
diff --git a/core/common/src/lib.rs b/core/common/src/lib.rs index 4568ebb..9bc952d 100644 --- a/core/common/src/lib.rs +++ b/core/common/src/lib.rs
@@ -145,6 +145,7 @@ pub use types::http::HttpMethod; pub use types::identifier::*; pub use types::message::*; +pub use types::metadata_changes::*; pub use types::options::*; pub use types::partition::*; pub use types::permissions::permissions_global::*;
diff --git a/core/common/src/traits/binary_impls/system.rs b/core/common/src/traits/binary_impls/system.rs index e3dc21d..2eda1bd 100644 --- a/core/common/src/traits/binary_impls/system.rs +++ b/core/common/src/traits/binary_impls/system.rs
@@ -18,21 +18,23 @@ use crate::traits::binary_auth::fail_if_not_authenticated; use crate::wire_conversions::clients_from_wire; use crate::{ - BinaryClient, ClientInfo, ClientInfoDetails, IggyError, NonZeroIggyDuration, OptionSpec, - OptionsScope, Snapshot, SnapshotCompression, Stats, SystemClient, SystemSnapshotType, + BinaryClient, ClientInfo, ClientInfoDetails, IggyError, MetadataChanges, MetadataChangesQuery, + NonZeroIggyDuration, OptionSpec, OptionsScope, Snapshot, SnapshotCompression, Stats, + SystemClient, SystemSnapshotType, }; use iggy_binary_protocol::codec::WireEncode; use iggy_binary_protocol::codes::{ - DESCRIBE_OPTIONS_CODE, GET_CLIENT_CODE, GET_CLIENTS_CODE, GET_ME_CODE, GET_SNAPSHOT_FILE_CODE, - GET_STATS_CODE, PING_CODE, + DESCRIBE_OPTIONS_CODE, GET_CLIENT_CODE, GET_CLIENTS_CODE, GET_ME_CODE, + GET_METADATA_CHANGES_CODE, GET_SNAPSHOT_FILE_CODE, GET_STATS_CODE, PING_CODE, }; use iggy_binary_protocol::requests::system::{ - DescribeOptionsRequest, GetClientRequest, GetClientsRequest, GetMeRequest, GetSnapshotRequest, - GetStatsRequest, PingRequest, + DescribeOptionsRequest, GetClientRequest, GetClientsRequest, GetMeRequest, + GetMetadataChangesRequest, GetSnapshotRequest, GetStatsRequest, PingRequest, }; use iggy_binary_protocol::responses::clients::get_client::ClientDetailsResponse; use iggy_binary_protocol::responses::clients::get_clients::GetClientsResponse; use iggy_binary_protocol::responses::system::DescribeOptionsResponse; +use iggy_binary_protocol::responses::system::GetMetadataChangesResponse; use iggy_binary_protocol::responses::system::get_stats::StatsResponse; #[async_trait::async_trait] @@ -129,4 +131,229 @@ let snapshot = Snapshot::new(response.to_vec()); Ok(snapshot) } + + async fn get_metadata_changes( + &self, + query: &MetadataChangesQuery, + ) -> Result<MetadataChanges, IggyError> { + fail_if_not_authenticated(self).await?; + query.validate()?; + let response = self + .send_raw_with_response( + GET_METADATA_CHANGES_CODE, + GetMetadataChangesRequest::from(query).to_bytes(), + ) + .await?; + let wire_resp = super::decode_response::<GetMetadataChangesResponse>(&response)?; + MetadataChanges::try_from(wire_resp) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::{ + BinaryTransport, Client, ClientState, ConsumerGroupClientState, DiagnosticEvent, + MetadataChangeStart, MetadataChangeStatus, PartitionContext, VsrSessionControl, + VsrSessionSealed, + }; + use async_broadcast::Receiver; + use bytes::Bytes; + use iggy_binary_protocol::WireName; + use iggy_binary_protocol::primitives::metadata_changes::{ + WireChangeOptions, WireMetadataChange, WireMetadataChangeCursor, WireMetadataChangePhase, + WireMetadataChangeStatus, WireStreamBody, WireStreamKey, + }; + use iggy_binary_protocol::requests::messages::PollMessagesRequest; + use iggy_binary_protocol::requests::system::SessionIdentity; + use iggy_binary_protocol::requests::users::login_register::BindSecret; + use iggy_binary_protocol::responses::system::WireMetadataChangeRecord; + use iggy_binary_protocol::{WireDecode, WireOptions}; + use std::sync::{Arc, Mutex}; + + #[tokio::test] + async fn given_query_the_rules_refuse_when_reading_changes_should_not_send_it() { + let client = ReplyingClient::default(); + + let refused = client + .get_metadata_changes(&MetadataChangesQuery { + limit: 0, + ..MetadataChangesQuery::default() + }) + .await; + + assert_eq!(refused, Err(IggyError::InvalidCommand)); + assert!(client.sent.lock().expect("lock").is_empty()); + } + + #[tokio::test] + async fn given_change_page_when_reading_changes_should_send_code_17_and_decode_the_page() { + let body = WireMetadataChange::StreamCreated(WireStreamBody { + key: WireStreamKey { + stream_id: 3, + stream_created_op: 41, + }, + created_at: 7, + name: WireName::new("orders").expect("valid name"), + options: WireChangeOptions::Readable { + explicit: WireOptions::empty(), + derived: WireOptions::empty(), + }, + }) + .body_bytes(); + let page = GetMetadataChangesResponse { + status: WireMetadataChangeStatus::Ok, + phase: WireMetadataChangePhase::Changes, + log_id: Some(0x0d4b), + oldest_op: 1, + retained_since: 0, + head_op: 41, + next: Some(WireMetadataChangeCursor { + log_id: 0x0d4b, + op: 41, + checksum: 0x41, + ordinal: 0, + }), + has_more: false, + records: vec![WireMetadataChangeRecord { + op: 41, + ordinal: 0, + checksum: 0x41, + timestamp: 7, + actor_user_id: Some(0), + group: 1, + kind: 1, + body, + }], + items: Vec::new(), + current_page_token: None, + }; + let client = ReplyingClient { + reply: page.to_bytes(), + ..ReplyingClient::default() + }; + let query = MetadataChangesQuery { + start: MetadataChangeStart::Latest, + ..MetadataChangesQuery::default() + }; + + let changes = client + .get_metadata_changes(&query) + .await + .expect("a decoded page"); + + let sent = client.sent.lock().expect("lock").clone(); + assert_eq!(sent.len(), 1); + assert_eq!(sent[0].0, GET_METADATA_CHANGES_CODE); + assert_eq!( + GetMetadataChangesRequest::decode_from(&sent[0].1).expect("a request"), + GetMetadataChangesRequest::from(&query) + ); + assert_eq!(changes.status, MetadataChangeStatus::Ok); + assert_eq!( + changes, + MetadataChanges::try_from(page).expect("valid page") + ); + } + + /// An authenticated binary client that records each request and answers + /// with `reply`. + #[derive(Debug, Default)] + struct ReplyingClient { + sent: Mutex<Vec<(u32, Bytes)>>, + reply: Bytes, + consumer_groups: Arc<ConsumerGroupClientState>, + } + + #[async_trait::async_trait] + impl BinaryTransport for ReplyingClient { + async fn get_state(&self) -> ClientState { + ClientState::Authenticated + } + + async fn set_state(&self, _state: ClientState) {} + + async fn publish_event(&self, _event: DiagnosticEvent) {} + + async fn send_raw_with_context( + &self, + code: u32, + payload: Bytes, + _context: PartitionContext, + ) -> Result<Bytes, IggyError> { + self.sent.lock().expect("lock").push((code, payload)); + Ok(self.reply.clone()) + } + + async fn send_offset_write_with_response( + &self, + _code: u32, + _payload: Bytes, + _context: Option<PartitionContext>, + ) -> Result<Bytes, IggyError> { + Err(IggyError::FeatureUnavailable) + } + + async fn send_poll_with_response( + &self, + _request: &PollMessagesRequest, + _context: Option<PartitionContext>, + ) -> Result<Bytes, IggyError> { + Err(IggyError::FeatureUnavailable) + } + + fn get_heartbeat_interval(&self) -> NonZeroIggyDuration { + NonZeroIggyDuration::ONE_SECOND + } + + fn consumer_group_state(&self) -> Arc<ConsumerGroupClientState> { + Arc::clone(&self.consumer_groups) + } + } + + #[async_trait::async_trait] + impl Client for ReplyingClient { + async fn connect(&self) -> Result<(), IggyError> { + Ok(()) + } + + async fn disconnect(&self) -> Result<(), IggyError> { + Ok(()) + } + + async fn shutdown(&self) -> Result<(), IggyError> { + Ok(()) + } + + async fn subscribe_events(&self) -> Receiver<DiagnosticEvent> { + async_broadcast::broadcast(1).1 + } + } + + impl VsrSessionSealed for ReplyingClient {} + + #[async_trait::async_trait] + impl VsrSessionControl for ReplyingClient { + async fn session_bind_secret(&self) -> Result<BindSecret, IggyError> { + Err(IggyError::FeatureUnavailable) + } + + async fn session_identity(&self) -> Result<SessionIdentity, IggyError> { + Err(IggyError::FeatureUnavailable) + } + + async fn bind_vsr_session(&self, _session: u64) -> Result<(), IggyError> { + Ok(()) + } + + async fn reset_vsr_session(&self) -> Result<(), IggyError> { + Ok(()) + } + + fn sdk_version(&self) -> &'static str { + env!("CARGO_PKG_VERSION") + } + } + + impl BinaryClient for ReplyingClient {} }
diff --git a/core/common/src/traits/system_client.rs b/core/common/src/traits/system_client.rs index a418a5e..2d88357 100644 --- a/core/common/src/traits/system_client.rs +++ b/core/common/src/traits/system_client.rs
@@ -16,8 +16,9 @@ // under the License. use crate::{ - ClientInfo, ClientInfoDetails, IggyError, NonZeroIggyDuration, OptionSpec, OptionsScope, - Snapshot, SnapshotCompression, Stats, SystemSnapshotType, + ClientInfo, ClientInfoDetails, IggyError, MetadataChanges, MetadataChangesQuery, + NonZeroIggyDuration, OptionSpec, OptionsScope, Snapshot, SnapshotCompression, Stats, + SystemSnapshotType, }; use async_trait::async_trait; @@ -65,4 +66,12 @@ compression: SnapshotCompression, snapshot_types: Vec<SystemSnapshotType>, ) -> Result<Snapshot, IggyError>; + /// One page of the node's metadata change log, or of the current state + /// to rebuild from. Requires root or the global `manage_servers` + /// permission. Keep the returned `next` cursor and send it back to + /// continue. + async fn get_metadata_changes( + &self, + query: &MetadataChangesQuery, + ) -> Result<MetadataChanges, IggyError>; }
diff --git a/core/common/src/types/metadata_changes/answer.rs b/core/common/src/types/metadata_changes/answer.rs new file mode 100644 index 0000000..3f5c6ce --- /dev/null +++ b/core/common/src/types/metadata_changes/answer.rs
@@ -0,0 +1,114 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use super::json::hex_u128_option; +use super::{ + MetadataChangeCursor, MetadataChangePhase, MetadataChangeRecord, MetadataChangeStatus, + MetadataCurrentItem, MetadataCurrentPageToken, +}; +use serde::{Deserialize, Serialize}; + +/// One page of the change log, or of the current state. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct MetadataChanges { + pub status: MetadataChangeStatus, + pub phase: MetadataChangePhase, + /// This node's log id, absent before it applied op 1. + #[serde(with = "hex_u128_option")] + pub log_id: Option<u128>, + /// Lowest whole operation from which the requested classes are complete. + pub oldest_op: u64, + /// The requested classes are complete for every timestamp above this + /// value, 0 when nothing was evicted. + pub retained_since: u64, + /// Highest metadata operation this node applied. + pub head_op: u64, + /// The cursor to send back. Present on `ok`. On `behind` it repeats the + /// position the request carried, so the same request can be retried. + pub next: Option<MetadataChangeCursor>, + /// More entries remain after `next`, or another state page exists. + pub has_more: bool, + pub records: Vec<MetadataChangeRecord>, + pub items: Vec<MetadataCurrentItem>, + /// Continuation of a current state scan, absent once it is complete. On + /// `behind` it echoes the token of the page to retry, with `has_more` + /// false. + pub current_page_token: Option<MetadataCurrentPageToken>, +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn given_behind_answer_when_serializing_should_render_nulls_for_absent_positions() { + let answer = MetadataChanges { + status: MetadataChangeStatus::Behind, + phase: MetadataChangePhase::Changes, + log_id: None, + oldest_op: 0, + retained_since: 0, + head_op: 0, + next: None, + has_more: false, + records: vec![], + items: vec![], + current_page_token: None, + }; + let json = serde_json::to_string(&answer).unwrap(); + assert_eq!( + json, + concat!( + r#"{"status":"behind","phase":"changes","log_id":null,"oldest_op":0,"#, + r#""retained_since":0,"head_op":0,"next":null,"has_more":false,"records":[],"#, + r#""items":[],"current_page_token":null}"# + ) + ); + assert_eq!( + serde_json::from_str::<MetadataChanges>(&json).unwrap(), + answer + ); + } + + #[test] + fn given_ok_answer_with_cursor_when_round_tripping_json_should_preserve_it() { + let answer = MetadataChanges { + status: MetadataChangeStatus::Ok, + phase: MetadataChangePhase::Changes, + log_id: Some(0x0d4b_8f2a_6e1c_93d7), + oldest_op: 3871, + retained_since: 1_760_352_000_000_000, + head_op: 4187, + next: Some(MetadataChangeCursor { + log_id: 0x0d4b_8f2a_6e1c_93d7, + op: 4187, + checksum: 0xf2a4_c6e8_b0d1_f359, + ordinal: 1, + }), + has_more: false, + records: vec![], + items: vec![], + current_page_token: None, + }; + let json = serde_json::to_string(&answer).unwrap(); + assert!(json.contains(r#""log_id":"00000000000000000d4b8f2a6e1c93d7""#)); + assert_eq!( + serde_json::from_str::<MetadataChanges>(&json).unwrap(), + answer + ); + } +}
diff --git a/core/common/src/types/metadata_changes/json.rs b/core/common/src/types/metadata_changes/json.rs new file mode 100644 index 0000000..757d281 --- /dev/null +++ b/core/common/src/types/metadata_changes/json.rs
@@ -0,0 +1,117 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! JSON forms shared by change records, items and answers. + +use serde::{Deserialize, Deserializer, Serializer}; + +/// Length of a 128-bit value as lowercase hex. +const HEX_U128_LENGTH: usize = 32; + +/// A 128-bit value as 32 lowercase hex characters. +pub(crate) mod hex_u128 { + use super::{Deserialize, Deserializer, Serializer, parse}; + + pub(crate) fn serialize<S: Serializer>(value: &u128, serializer: S) -> Result<S::Ok, S::Error> { + serializer.serialize_str(&format!("{value:032x}")) + } + + pub(crate) fn deserialize<'de, D: Deserializer<'de>>( + deserializer: D, + ) -> Result<u128, D::Error> { + let text = String::deserialize(deserializer)?; + parse(&text) + } +} + +/// An optional 128-bit value as 32 lowercase hex characters or null. +pub(crate) mod hex_u128_option { + use super::{Deserialize, Deserializer, Serializer, parse}; + + // serde's `with` hands the field by reference. + #[allow(clippy::ref_option)] + pub(crate) fn serialize<S: Serializer>( + value: &Option<u128>, + serializer: S, + ) -> Result<S::Ok, S::Error> { + match value { + Some(value) => serializer.serialize_str(&format!("{value:032x}")), + None => serializer.serialize_none(), + } + } + + pub(crate) fn deserialize<'de, D: Deserializer<'de>>( + deserializer: D, + ) -> Result<Option<u128>, D::Error> { + Option::<String>::deserialize(deserializer)? + .map(|text| parse(&text)) + .transpose() + } +} + +fn parse<E: serde::de::Error>(text: &str) -> Result<u128, E> { + let canonical = text.len() == HEX_U128_LENGTH + && text + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)); + if !canonical { + return Err(E::custom(format!( + "expected {HEX_U128_LENGTH} lowercase hex characters, got {text:?}" + ))); + } + u128::from_str_radix(text, 16).map_err(E::custom) +} + +#[cfg(test)] +mod tests { + use super::*; + use serde::Serialize; + + #[derive(Debug, PartialEq, Serialize, Deserialize)] + struct Holder { + #[serde(with = "hex_u128")] + value: u128, + #[serde(with = "hex_u128_option")] + optional: Option<u128>, + } + + #[test] + fn given_128_bit_values_when_serializing_should_use_32_lowercase_hex_characters() { + let holder = Holder { + value: 0xd4b_8f2a_6e1c_93d7, + optional: None, + }; + let text = serde_json::to_string(&holder).unwrap(); + assert_eq!( + text, + r#"{"value":"00000000000000000d4b8f2a6e1c93d7","optional":null}"# + ); + assert_eq!(serde_json::from_str::<Holder>(&text).unwrap(), holder); + } + + #[test] + fn given_short_uppercase_or_prefixed_hex_when_deserializing_should_refuse() { + for value in [ + "d4b", + "00000000000000000D4B8F2A6E1C93D7", + "0x000000000000000d4b8f2a6e1c93d7", + ] { + let text = format!(r#"{{"value":"{value}","optional":null}}"#); + assert!(serde_json::from_str::<Holder>(&text).is_err(), "{value}"); + } + } +}
diff --git a/core/common/src/types/metadata_changes/mod.rs b/core/common/src/types/metadata_changes/mod.rs new file mode 100644 index 0000000..8a4fd6c --- /dev/null +++ b/core/common/src/types/metadata_changes/mod.rs
@@ -0,0 +1,586 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +pub(crate) mod answer; +pub(crate) mod json; +pub(crate) mod position; +pub(crate) mod query; +pub(crate) mod records; +pub(crate) mod wire; + +pub use answer::MetadataChanges; +pub use position::{ + METADATA_CHANGE_CURSOR_PREFIX, METADATA_CURRENT_PAGE_TOKEN_PREFIX, MetadataChangeCursor, + MetadataCurrentPageToken, +}; +pub use query::{ + DEFAULT_METADATA_CHANGES_LIMIT, MAX_METADATA_CHANGES_LIMIT, MetadataChangeStart, + MetadataChangesQuery, +}; +pub use records::{ + MetadataChange, MetadataChangeRecord, MetadataConsumerGroupCreated, + MetadataConsumerGroupDeleted, MetadataConsumerGroupItem, MetadataCurrentItem, + MetadataMemberItem, MetadataMemberJoined, MetadataMemberLeft, MetadataOptions, + MetadataPartitionHandedOver, MetadataPartitionsChanged, MetadataPartitionsItem, + MetadataRawJson, MetadataSession, MetadataSessionClosed, MetadataStreamChange, + MetadataStreamDeleted, MetadataTokenChange, MetadataTokenDeleted, MetadataTopicChange, + MetadataTopicDeleted, MetadataTopicItem, MetadataUserChange, MetadataUserIdentity, + MetadataUserPermissionsUpdated, +}; + +use iggy_binary_protocol::primitives::metadata_changes::{ + WireMemberLeftReason, WireMetadataChangeGroup, WireMetadataChangeKind, WireMetadataChangePhase, + WireMetadataChangeStatus, WirePermissionsUpdatedReason, WireSessionClosedReason, + WireTokenDeletedReason, +}; +use serde::{Deserialize, Serialize}; +use strum::{Display, EnumIter, EnumString, IntoEnumIterator, IntoStaticStr}; + +/// Family of change records and the unit of filtering. An empty selection +/// means every group. +#[derive( + Debug, + Clone, + Copy, + PartialEq, + Eq, + Hash, + PartialOrd, + Ord, + Serialize, + Deserialize, + Display, + EnumString, + IntoStaticStr, + EnumIter, +)] +#[serde(rename_all = "snake_case")] +#[strum(serialize_all = "snake_case")] +pub enum MetadataChangeGroup { + Streams, + Topics, + Partitions, + ConsumerGroups, + ConsumerGroupMembers, + Users, + PersonalAccessTokens, + Sessions, +} + +/// Bounded sequence a group's records are retained in. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub enum MetadataChangeClass { + /// Streams, topics, partitions, consumer groups, users and tokens. + Resource, + /// Consumer group membership and sessions. + Activity, +} + +/// Kind of a change record. +#[derive( + Debug, + Clone, + Copy, + PartialEq, + Eq, + Hash, + Serialize, + Deserialize, + Display, + EnumString, + IntoStaticStr, +)] +#[serde(rename_all = "snake_case")] +#[strum(serialize_all = "snake_case")] +pub enum MetadataChangeKind { + StreamCreated, + StreamUpdated, + StreamDeleted, + TopicCreated, + TopicUpdated, + TopicDeleted, + PartitionsChanged, + ConsumerGroupCreated, + ConsumerGroupDeleted, + ConsumerGroupMemberJoined, + ConsumerGroupMemberLeft, + ConsumerGroupPartitionHandedOver, + UserCreated, + UserUpdated, + UserDeleted, + UserPasswordChanged, + UserPermissionsUpdated, + PersonalAccessTokenCreated, + PersonalAccessTokenDeleted, + SessionRegistered, + SessionClosed, +} + +/// Outcome of a change log query. +#[derive( + Debug, + Clone, + Copy, + PartialEq, + Eq, + Hash, + Serialize, + Deserialize, + Display, + EnumString, + IntoStaticStr, +)] +#[serde(rename_all = "snake_case")] +#[strum(serialize_all = "snake_case")] +pub enum MetadataChangeStatus { + /// Records or items after the requested position. + Ok, + /// The answering node has not applied the requested position yet. + Behind, + /// The node no longer holds every record after the position. + Expired, + /// The position belongs to another history. + Reset, +} + +/// Whether an answer pages change records or current state items. +#[derive( + Debug, + Clone, + Copy, + PartialEq, + Eq, + Hash, + Serialize, + Deserialize, + Display, + EnumString, + IntoStaticStr, +)] +#[serde(rename_all = "snake_case")] +#[strum(serialize_all = "snake_case")] +pub enum MetadataChangePhase { + Changes, + Current, +} + +/// Why a consumer group member was removed. +#[derive( + Debug, + Clone, + Copy, + PartialEq, + Eq, + Hash, + Serialize, + Deserialize, + Display, + EnumString, + IntoStaticStr, +)] +#[serde(rename_all = "snake_case")] +#[strum(serialize_all = "snake_case")] +pub enum MetadataMemberLeftReason { + Left, + SessionEnded, +} + +/// Why a personal access token was removed. +#[derive( + Debug, + Clone, + Copy, + PartialEq, + Eq, + Hash, + Serialize, + Deserialize, + Display, + EnumString, + IntoStaticStr, +)] +#[serde(rename_all = "snake_case")] +#[strum(serialize_all = "snake_case")] +pub enum MetadataTokenDeletedReason { + Deleted, + Expired, +} + +/// Why a session was closed. +#[derive( + Debug, + Clone, + Copy, + PartialEq, + Eq, + Hash, + Serialize, + Deserialize, + Display, + EnumString, + IntoStaticStr, +)] +#[serde(rename_all = "snake_case")] +#[strum(serialize_all = "snake_case")] +pub enum MetadataSessionClosedReason { + Logout, + Expired, + /// The user was deactivated or deleted. + Revoked, +} + +/// Why a user's public permission set changed. +#[derive( + Debug, + Clone, + Copy, + PartialEq, + Eq, + Hash, + Serialize, + Deserialize, + Display, + EnumString, + IntoStaticStr, +)] +#[serde(rename_all = "snake_case")] +#[strum(serialize_all = "snake_case")] +pub enum MetadataPermissionsUpdatedReason { + /// An `UpdatePermissions` replaced the set. + Updated, + /// A stream or topic delete pruned the user's grants on it. + ResourceDeleted, +} + +/// A set of groups, one bit per group. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)] +pub struct MetadataChangeGroups(u8); + +impl MetadataChangeGroup { + /// Retention class of this group's records. + #[must_use] + pub const fn class(self) -> MetadataChangeClass { + match self { + Self::ConsumerGroupMembers | Self::Sessions => MetadataChangeClass::Activity, + Self::Streams + | Self::Topics + | Self::Partitions + | Self::ConsumerGroups + | Self::Users + | Self::PersonalAccessTokens => MetadataChangeClass::Resource, + } + } + + /// Groups a current state scan of this group must include, so a mirror + /// can apply the cascades of their deletes. + #[must_use] + pub const fn parents(self) -> &'static [Self] { + match self { + Self::Streams | Self::Users => &[], + Self::Topics => &[Self::Streams], + Self::Partitions | Self::ConsumerGroups => &[Self::Topics, Self::Streams], + Self::ConsumerGroupMembers => &[ + Self::ConsumerGroups, + Self::Partitions, + Self::Topics, + Self::Streams, + ], + Self::PersonalAccessTokens | Self::Sessions => &[Self::Users], + } + } + + /// Bit `code - 1` of a group mask. Variants follow the code order. + const fn bit(self) -> u8 { + 1 << self as u8 + } +} + +impl MetadataChangeKind { + /// The group every record of this kind belongs to. + #[must_use] + pub fn group(self) -> MetadataChangeGroup { + WireMetadataChangeKind::from(self).group().into() + } +} + +impl MetadataChangeGroups { + /// Every group. + #[must_use] + pub const fn all() -> Self { + Self(u8::MAX) + } + + /// The requested groups, with duplicates collapsed and an empty request + /// meaning every group. + #[must_use] + pub fn requested(groups: &[MetadataChangeGroup]) -> Self { + if groups.is_empty() { + return Self::all(); + } + Self(groups.iter().fold(0, |mask, group| mask | group.bit())) + } + + /// The set a page token's group mask names. + #[must_use] + pub const fn from_mask(mask: u8) -> Self { + Self(mask) + } + + #[must_use] + pub const fn mask(self) -> u8 { + self.0 + } + + #[must_use] + pub const fn contains(self, group: MetadataChangeGroup) -> bool { + self.0 & group.bit() != 0 + } + + #[must_use] + pub const fn contains_class(self, class: MetadataChangeClass) -> bool { + let activity = + MetadataChangeGroup::ConsumerGroupMembers.bit() | MetadataChangeGroup::Sessions.bit(); + match class { + MetadataChangeClass::Activity => self.0 & activity != 0, + MetadataChangeClass::Resource => self.0 & !activity != 0, + } + } + + /// The groups in code order. + pub fn iter(self) -> impl Iterator<Item = MetadataChangeGroup> { + MetadataChangeGroup::iter().filter(move |group| self.contains(*group)) + } + + /// The first selected group whose parent is missing, with that parent. + #[must_use] + pub fn missing_parent(self) -> Option<(MetadataChangeGroup, MetadataChangeGroup)> { + self.iter().find_map(|group| { + group + .parents() + .iter() + .find(|parent| !self.contains(**parent)) + .map(|parent| (group, *parent)) + }) + } +} + +macro_rules! mirror_wire_enum { + ($domain:ident, $wire:ident, $($variant:ident),+) => { + impl From<$domain> for $wire { + fn from(value: $domain) -> Self { + match value { + $($domain::$variant => Self::$variant,)+ + } + } + } + + impl From<$wire> for $domain { + fn from(value: $wire) -> Self { + match value { + $($wire::$variant => Self::$variant,)+ + } + } + } + }; +} + +mirror_wire_enum!( + MetadataChangeGroup, + WireMetadataChangeGroup, + Streams, + Topics, + Partitions, + ConsumerGroups, + ConsumerGroupMembers, + Users, + PersonalAccessTokens, + Sessions +); +mirror_wire_enum!( + MetadataChangeKind, + WireMetadataChangeKind, + StreamCreated, + StreamUpdated, + StreamDeleted, + TopicCreated, + TopicUpdated, + TopicDeleted, + PartitionsChanged, + ConsumerGroupCreated, + ConsumerGroupDeleted, + ConsumerGroupMemberJoined, + ConsumerGroupMemberLeft, + ConsumerGroupPartitionHandedOver, + UserCreated, + UserUpdated, + UserDeleted, + UserPasswordChanged, + UserPermissionsUpdated, + PersonalAccessTokenCreated, + PersonalAccessTokenDeleted, + SessionRegistered, + SessionClosed +); +mirror_wire_enum!( + MetadataChangeStatus, + WireMetadataChangeStatus, + Ok, + Behind, + Expired, + Reset +); +mirror_wire_enum!( + MetadataChangePhase, + WireMetadataChangePhase, + Changes, + Current +); +mirror_wire_enum!( + MetadataMemberLeftReason, + WireMemberLeftReason, + Left, + SessionEnded +); +mirror_wire_enum!( + MetadataTokenDeletedReason, + WireTokenDeletedReason, + Deleted, + Expired +); +mirror_wire_enum!( + MetadataSessionClosedReason, + WireSessionClosedReason, + Logout, + Expired, + Revoked +); +mirror_wire_enum!( + MetadataPermissionsUpdatedReason, + WirePermissionsUpdatedReason, + Updated, + ResourceDeleted +); + +#[cfg(test)] +mod tests { + use super::*; + use std::str::FromStr; + + #[test] + fn given_every_group_when_mapping_to_wire_should_keep_the_catalog_code_order() { + let codes: Vec<u8> = MetadataChangeGroup::iter() + .map(|group| WireMetadataChangeGroup::from(group).as_u8()) + .collect(); + assert_eq!(codes, (1..=8).collect::<Vec<u8>>()); + for group in MetadataChangeGroup::iter() { + assert_eq!( + MetadataChangeGroup::from(WireMetadataChangeGroup::from(group)), + group + ); + } + } + + #[test] + fn given_group_names_when_parsing_should_use_snake_case() { + assert_eq!( + MetadataChangeGroup::from_str("consumer_group_members").unwrap(), + MetadataChangeGroup::ConsumerGroupMembers + ); + assert_eq!( + MetadataChangeGroup::PersonalAccessTokens.to_string(), + "personal_access_tokens" + ); + assert!(MetadataChangeGroup::from_str("ConsumerGroups").is_err()); + } + + #[test] + fn given_every_group_when_taking_its_bit_should_match_the_wire_mask_bit() { + for group in MetadataChangeGroup::iter() { + assert_eq!(group.bit(), WireMetadataChangeGroup::from(group).mask_bit()); + } + } + + #[test] + fn given_empty_or_duplicate_selection_when_building_group_set_should_normalize_it() { + assert_eq!( + MetadataChangeGroups::requested(&[]), + MetadataChangeGroups::all() + ); + let set = MetadataChangeGroups::requested(&[ + MetadataChangeGroup::Topics, + MetadataChangeGroup::Topics, + MetadataChangeGroup::Streams, + ]); + assert_eq!(set.mask(), 0b11); + assert_eq!( + set.iter().collect::<Vec<_>>(), + vec![MetadataChangeGroup::Streams, MetadataChangeGroup::Topics] + ); + } + + #[test] + fn given_group_without_its_parents_when_checking_closure_should_name_the_missing_parent() { + let members = MetadataChangeGroups::requested(&[ + MetadataChangeGroup::ConsumerGroupMembers, + MetadataChangeGroup::ConsumerGroups, + MetadataChangeGroup::Topics, + MetadataChangeGroup::Streams, + ]); + assert_eq!( + members.missing_parent(), + Some(( + MetadataChangeGroup::ConsumerGroupMembers, + MetadataChangeGroup::Partitions + )) + ); + assert_eq!(MetadataChangeGroups::all().missing_parent(), None); + assert_eq!( + MetadataChangeGroups::requested(&[ + MetadataChangeGroup::Sessions, + MetadataChangeGroup::Users + ]) + .missing_parent(), + None + ); + } + + #[test] + fn given_group_sets_when_checking_classes_should_split_activity_from_resources() { + let activity = MetadataChangeGroups::requested(&[MetadataChangeGroup::Sessions]); + assert!(activity.contains_class(MetadataChangeClass::Activity)); + assert!(!activity.contains_class(MetadataChangeClass::Resource)); + let resource = MetadataChangeGroups::requested(&[MetadataChangeGroup::Users]); + assert!(resource.contains_class(MetadataChangeClass::Resource)); + assert!(!resource.contains_class(MetadataChangeClass::Activity)); + } + + #[test] + fn given_every_kind_when_mapping_its_group_should_match_the_wire_catalog() { + assert_eq!( + MetadataChangeKind::ConsumerGroupPartitionHandedOver.group(), + MetadataChangeGroup::ConsumerGroupMembers + ); + assert_eq!( + MetadataChangeKind::SessionClosed.group(), + MetadataChangeGroup::Sessions + ); + assert_eq!( + MetadataChangeKind::from_str("user_permissions_updated").unwrap(), + MetadataChangeKind::UserPermissionsUpdated + ); + } +}
diff --git a/core/common/src/types/metadata_changes/position.rs b/core/common/src/types/metadata_changes/position.rs new file mode 100644 index 0000000..e03c4f5 --- /dev/null +++ b/core/common/src/types/metadata_changes/position.rs
@@ -0,0 +1,381 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use super::{MetadataChangeGroup, MetadataChangeGroups}; +use crate::IggyError; +use base64::Engine; +use base64::engine::general_purpose::URL_SAFE_NO_PAD; +use iggy_binary_protocol::primitives::metadata_changes::{ + MAX_PAGE_TOKEN_SIZE, METADATA_CHANGE_CURSOR_SIZE, WireMetadataChangeCursor, + WireMetadataCurrentPageToken, +}; +use iggy_binary_protocol::{WireDecode, WireEncode}; +use serde::de::Error as _; +use serde::{Deserialize, Deserializer, Serialize, Serializer}; +use std::fmt::{Display, Formatter}; +use std::str::FromStr; + +/// Text prefix of a change cursor. +pub const METADATA_CHANGE_CURSOR_PREFIX: &str = "mc1_"; + +/// Text prefix of a current state page token. +pub const METADATA_CURRENT_PAGE_TOKEN_PREFIX: &str = "cs1_"; + +/// Text length of a change cursor: the prefix and 44 bytes of unpadded +/// Base64url. +const CURSOR_TEXT_LENGTH: usize = METADATA_CHANGE_CURSOR_PREFIX.len() + 59; + +/// Position of the last change record a caller consumed or scanned. +/// +/// Its text form is `mc1_` followed by unpadded Base64url of +/// `[log_id:u128 LE][op:u64 LE][checksum:u128 LE][ordinal:u32 LE]`. +/// `op = 0` with a zero checksum and ordinal names the beginning of the +/// history `log_id` identifies. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)] +pub struct MetadataChangeCursor { + /// Prepare checksum of op 1, the root bootstrap of this history. + pub log_id: u128, + pub op: u64, + /// Prepare checksum of `op`. + pub checksum: u128, + /// Position of the record within `op`. + pub ordinal: u32, +} + +/// Opaque continuation of a current state scan. +/// +/// Its text form is `cs1_` followed by unpadded Base64url of the page token +/// payload. Send it back unchanged with the same groups. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct MetadataCurrentPageToken(WireMetadataCurrentPageToken); + +impl MetadataChangeCursor { + /// The cursor before the first record of `log_id`. + #[must_use] + pub const fn zero(log_id: u128) -> Self { + Self { + log_id, + op: 0, + checksum: 0, + ordinal: 0, + } + } + + #[must_use] + pub const fn is_zero(&self) -> bool { + self.op == 0 + } + + /// Op 0 carries no checksum and no ordinal. + #[must_use] + pub const fn is_canonical(&self) -> bool { + self.op != 0 || (self.checksum == 0 && self.ordinal == 0) + } +} + +impl MetadataCurrentPageToken { + /// The groups the scan was started with. + #[must_use] + pub const fn groups(&self) -> MetadataChangeGroups { + MetadataChangeGroups::from_mask(self.0.groups_mask) + } + + /// The anchor captured before the scan's first page. + #[must_use] + pub fn anchor(&self) -> MetadataChangeCursor { + self.0.anchor.into() + } + + /// The highest head any page of the scan reported so far. + #[must_use] + pub const fn required_head(&self) -> u64 { + self.0.required_head + } + + /// The group of the last item returned. + #[must_use] + pub fn last_group(&self) -> MetadataChangeGroup { + self.0.group.into() + } + + /// The key of the last item returned, in the page token key layout. + #[must_use] + pub fn last_key(&self) -> &[u8] { + &self.0.last_key + } +} + +impl Display for MetadataChangeCursor { + fn fmt(&self, formatter: &mut Formatter<'_>) -> std::fmt::Result { + let bytes = WireMetadataChangeCursor::from(*self).to_bytes(); + write!( + formatter, + "{METADATA_CHANGE_CURSOR_PREFIX}{}", + URL_SAFE_NO_PAD.encode(bytes) + ) + } +} + +impl FromStr for MetadataChangeCursor { + type Err = IggyError; + + fn from_str(text: &str) -> Result<Self, Self::Err> { + if text.len() != CURSOR_TEXT_LENGTH { + return Err(IggyError::InvalidCommand); + } + let bytes = decode_canonical(text, METADATA_CHANGE_CURSOR_PREFIX)?; + if bytes.len() != METADATA_CHANGE_CURSOR_SIZE { + return Err(IggyError::InvalidCommand); + } + let (cursor, _) = + WireMetadataChangeCursor::decode(&bytes).map_err(|_| IggyError::InvalidCommand)?; + Ok(cursor.into()) + } +} + +impl Display for MetadataCurrentPageToken { + fn fmt(&self, formatter: &mut Formatter<'_>) -> std::fmt::Result { + write!( + formatter, + "{METADATA_CURRENT_PAGE_TOKEN_PREFIX}{}", + URL_SAFE_NO_PAD.encode(self.0.to_bytes()) + ) + } +} + +impl FromStr for MetadataCurrentPageToken { + type Err = IggyError; + + fn from_str(text: &str) -> Result<Self, Self::Err> { + let longest = + METADATA_CURRENT_PAGE_TOKEN_PREFIX.len() + MAX_PAGE_TOKEN_SIZE.div_ceil(3) * 4; + if text.len() > longest { + return Err(IggyError::InvalidCommand); + } + let bytes = decode_canonical(text, METADATA_CURRENT_PAGE_TOKEN_PREFIX)?; + let (token, consumed) = + WireMetadataCurrentPageToken::decode(&bytes).map_err(|_| IggyError::InvalidCommand)?; + if consumed != bytes.len() { + return Err(IggyError::InvalidCommand); + } + Ok(Self(token)) + } +} + +/// Strip `prefix` and decode canonical unpadded Base64url. +fn decode_canonical(text: &str, prefix: &str) -> Result<Vec<u8>, IggyError> { + let encoded = text.strip_prefix(prefix).ok_or(IggyError::InvalidCommand)?; + let bytes = URL_SAFE_NO_PAD + .decode(encoded) + .map_err(|_| IggyError::InvalidCommand)?; + if URL_SAFE_NO_PAD.encode(&bytes) != encoded { + return Err(IggyError::InvalidCommand); + } + Ok(bytes) +} + +impl From<WireMetadataChangeCursor> for MetadataChangeCursor { + fn from(cursor: WireMetadataChangeCursor) -> Self { + Self { + log_id: cursor.log_id, + op: cursor.op, + checksum: cursor.checksum, + ordinal: cursor.ordinal, + } + } +} + +impl From<MetadataChangeCursor> for WireMetadataChangeCursor { + fn from(cursor: MetadataChangeCursor) -> Self { + Self { + log_id: cursor.log_id, + op: cursor.op, + checksum: cursor.checksum, + ordinal: cursor.ordinal, + } + } +} + +impl From<WireMetadataCurrentPageToken> for MetadataCurrentPageToken { + fn from(token: WireMetadataCurrentPageToken) -> Self { + Self(token) + } +} + +impl From<MetadataCurrentPageToken> for WireMetadataCurrentPageToken { + fn from(token: MetadataCurrentPageToken) -> Self { + token.0 + } +} + +impl AsRef<WireMetadataCurrentPageToken> for MetadataCurrentPageToken { + fn as_ref(&self) -> &WireMetadataCurrentPageToken { + &self.0 + } +} + +impl Serialize for MetadataChangeCursor { + fn serialize<S: Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> { + serializer.collect_str(self) + } +} + +impl<'de> Deserialize<'de> for MetadataChangeCursor { + fn deserialize<D: Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> { + let text = String::deserialize(deserializer)?; + Self::from_str(&text) + .map_err(|_| D::Error::custom(format!("invalid change cursor {text:?}"))) + } +} + +impl Serialize for MetadataCurrentPageToken { + fn serialize<S: Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> { + serializer.collect_str(self) + } +} + +impl<'de> Deserialize<'de> for MetadataCurrentPageToken { + fn deserialize<D: Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> { + let text = String::deserialize(deserializer)?; + Self::from_str(&text) + .map_err(|_| D::Error::custom(format!("invalid current page token {text:?}"))) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use bytes::Bytes; + use iggy_binary_protocol::primitives::metadata_changes::WireMetadataChangeGroup; + + fn cursor() -> MetadataChangeCursor { + MetadataChangeCursor { + log_id: 0x0d4b_8f2a_6e1c_93d7, + op: 4187, + checksum: 0xf2a4_c6e8_b0d1_f359, + ordinal: 1, + } + } + + fn token() -> MetadataCurrentPageToken { + MetadataCurrentPageToken(WireMetadataCurrentPageToken { + anchor: cursor().into(), + required_head: 4200, + groups_mask: 0b11, + group: WireMetadataChangeGroup::Topics, + last_key: Bytes::from_static(&[3, 0, 0, 0, 1, 0, 0, 0]), + }) + } + + #[test] + fn given_cursor_when_printing_should_use_the_63_character_mc1_form() { + let text = cursor().to_string(); + assert!(text.starts_with("mc1_")); + assert_eq!(text.len(), 63); + assert_eq!(MetadataChangeCursor::from_str(&text).unwrap(), cursor()); + } + + #[test] + fn given_rfc_sample_cursor_when_parsing_should_name_op_4187_ordinal_1() { + let parsed = MetadataChangeCursor::from_str( + "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + ) + .unwrap(); + assert_eq!(parsed.log_id, 0x0d4b_8f2a_6e1c_93d7); + assert_eq!(parsed.op, 4187); + assert_eq!(parsed.ordinal, 1); + } + + #[test] + fn given_malformed_cursor_text_when_parsing_should_refuse() { + let valid = cursor().to_string(); + let encoded = &valid[4..]; + let cases = [ + format!("mc2_{encoded}"), + encoded.to_owned(), + format!("{valid}A"), + valid[..62].to_owned(), + format!("mc1_{}=", &encoded[..58]), + valid.replace('_', "/"), + ]; + for case in cases { + assert!(MetadataChangeCursor::from_str(&case).is_err(), "{case}"); + } + } + + #[test] + fn given_non_canonical_trailing_bits_when_parsing_cursor_should_refuse() { + let valid = cursor().to_string(); + let last = valid.chars().last().unwrap(); + let alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"; + let index = alphabet.find(last).unwrap(); + let sibling = alphabet.as_bytes()[index ^ 1] as char; + let tampered = format!("{}{sibling}", &valid[..62]); + assert!(MetadataChangeCursor::from_str(&tampered).is_err()); + } + + #[test] + fn given_zero_cursor_with_checksum_when_parsing_should_refuse() { + let non_canonical = WireMetadataChangeCursor { + log_id: 1, + op: 0, + checksum: 2, + ordinal: 0, + }; + let text = format!("mc1_{}", URL_SAFE_NO_PAD.encode(non_canonical.to_bytes())); + assert!(MetadataChangeCursor::from_str(&text).is_err()); + let zero = MetadataChangeCursor::zero(1).to_string(); + assert!(MetadataChangeCursor::from_str(&zero).unwrap().is_zero()); + } + + #[test] + fn given_page_token_when_printing_should_round_trip_through_cs1_text() { + let text = token().to_string(); + assert!(text.starts_with("cs1_")); + let parsed = MetadataCurrentPageToken::from_str(&text).unwrap(); + assert_eq!(parsed, token()); + assert_eq!(parsed.groups().mask(), 0b11); + assert_eq!(parsed.last_group(), MetadataChangeGroup::Topics); + assert_eq!(parsed.anchor(), cursor()); + assert_eq!(parsed.required_head(), 4200); + } + + #[test] + fn given_malformed_page_token_text_when_parsing_should_refuse() { + let valid = token().to_string(); + let cases = [ + valid.replacen("cs1_", "mc1_", 1), + format!("{valid}AA"), + format!("{valid}="), + "cs1_".to_owned(), + format!("cs1_{}", "A".repeat(500)), + ]; + for case in cases { + assert!(MetadataCurrentPageToken::from_str(&case).is_err(), "{case}"); + } + } + + #[test] + fn given_cursor_and_token_when_serializing_json_should_use_their_text_forms() { + let json = serde_json::to_string(&(cursor(), token())).unwrap(); + let (decoded_cursor, decoded_token): (MetadataChangeCursor, MetadataCurrentPageToken) = + serde_json::from_str(&json).unwrap(); + assert_eq!(decoded_cursor, cursor()); + assert_eq!(decoded_token, token()); + assert!(json.contains("mc1_") && json.contains("cs1_")); + } +}
diff --git a/core/common/src/types/metadata_changes/query.rs b/core/common/src/types/metadata_changes/query.rs new file mode 100644 index 0000000..f327327 --- /dev/null +++ b/core/common/src/types/metadata_changes/query.rs
@@ -0,0 +1,429 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use super::{ + MetadataChangeCursor, MetadataChangeGroup, MetadataChangeGroups, MetadataCurrentPageToken, +}; +use crate::{IggyError, IggyTimestamp}; +use std::collections::BTreeSet; +use std::str::FromStr; + +/// Records or items per page when the caller does not say. +pub const DEFAULT_METADATA_CHANGES_LIMIT: u32 = 100; + +/// Most records or items one page returns. +pub const MAX_METADATA_CHANGES_LIMIT: u32 = 1000; + +const FROM_KEY: &str = "from"; +const AFTER_KEY: &str = "after"; +const CURRENT_AFTER_KEY: &str = "current_after"; +const GROUPS_KEY: &str = "groups"; +const SINCE_KEY: &str = "since"; +const UNTIL_KEY: &str = "until"; +const LIMIT_KEY: &str = "limit"; +const FROM_EARLIEST: &str = "earliest"; +const FROM_LATEST: &str = "latest"; +const FROM_CURRENT: &str = "current"; + +/// Where a change log query starts. +#[derive(Debug, Clone, PartialEq, Eq, Default)] +pub enum MetadataChangeStart { + /// The oldest operation the requested classes still hold completely. + #[default] + Earliest, + /// No records, only a cursor at the newest retained record. + Latest, + /// Records after a cursor. + After(MetadataChangeCursor), + /// Current state pages: `None` starts a scan, a token continues it. + Current(Option<MetadataCurrentPageToken>), +} + +/// A change log query. +/// +/// `groups` empty means every group. `since` and `until` keep records whose +/// prepare timestamp is in the inclusive range, and only apply to `Earliest` +/// and `After`. +#[derive(Debug, Clone, PartialEq)] +pub struct MetadataChangesQuery { + pub start: MetadataChangeStart, + pub groups: Vec<MetadataChangeGroup>, + pub since: Option<IggyTimestamp>, + pub until: Option<IggyTimestamp>, + pub limit: u32, +} + +impl Default for MetadataChangesQuery { + fn default() -> Self { + Self { + start: MetadataChangeStart::Earliest, + groups: Vec::new(), + since: None, + until: None, + limit: DEFAULT_METADATA_CHANGES_LIMIT, + } + } +} + +impl MetadataChangesQuery { + /// Check the request rules every transport shares and return the + /// normalized group set. + /// + /// # Errors + /// + /// `InvalidCommand` when the limit is outside 1 to 1000, `since` is above + /// `until`, a time bound accompanies `Latest` or `Current`, a cursor is not + /// canonical, a current scan misses a parent group, or a page token was + /// issued for other groups. + pub fn validate(&self) -> Result<MetadataChangeGroups, IggyError> { + if !(1..=MAX_METADATA_CHANGES_LIMIT).contains(&self.limit) { + return Err(IggyError::InvalidCommand); + } + if let (Some(since), Some(until)) = (self.since, self.until) + && since.as_micros() > until.as_micros() + { + return Err(IggyError::InvalidCommand); + } + let groups = MetadataChangeGroups::requested(&self.groups); + let has_time_bound = self.since.is_some() || self.until.is_some(); + match &self.start { + MetadataChangeStart::Earliest => {} + MetadataChangeStart::After(cursor) => { + if !cursor.is_canonical() { + return Err(IggyError::InvalidCommand); + } + } + MetadataChangeStart::Latest => { + if has_time_bound { + return Err(IggyError::InvalidCommand); + } + } + MetadataChangeStart::Current(token) => { + if has_time_bound || groups.missing_parent().is_some() { + return Err(IggyError::InvalidCommand); + } + if token.as_ref().is_some_and(|token| token.groups() != groups) { + return Err(IggyError::InvalidCommand); + } + } + } + Ok(groups) + } + + /// The HTTP query parameters for this query, in a stable order. + #[must_use] + pub fn http_query_pairs(&self) -> Vec<(&'static str, String)> { + let mut pairs = Vec::with_capacity(5); + match &self.start { + MetadataChangeStart::Earliest => pairs.push((FROM_KEY, FROM_EARLIEST.to_owned())), + MetadataChangeStart::Latest => pairs.push((FROM_KEY, FROM_LATEST.to_owned())), + MetadataChangeStart::After(cursor) => pairs.push((AFTER_KEY, cursor.to_string())), + MetadataChangeStart::Current(token) => { + pairs.push((FROM_KEY, FROM_CURRENT.to_owned())); + if let Some(token) = token { + pairs.push((CURRENT_AFTER_KEY, token.to_string())); + } + } + } + if !self.groups.is_empty() { + let names: Vec<&'static str> = MetadataChangeGroups::requested(&self.groups) + .iter() + .map(Into::into) + .collect(); + pairs.push((GROUPS_KEY, names.join(","))); + } + if let Some(since) = self.since { + pairs.push((SINCE_KEY, since.as_micros().to_string())); + } + if let Some(until) = self.until { + pairs.push((UNTIL_KEY, until.as_micros().to_string())); + } + pairs.push((LIMIT_KEY, self.limit.to_string())); + pairs + } + + /// Parse the HTTP query parameters of `GET /metadata/changes`. + /// + /// Exactly one of `from` and `after` is required. Unknown or repeated + /// keys, an empty group element and malformed values are refused. The + /// result is not yet validated, call [`Self::validate`]. + /// + /// # Errors + /// + /// `InvalidCommand` for any refused parameter. + pub fn from_http_query<'a>( + pairs: impl IntoIterator<Item = (&'a str, &'a str)>, + ) -> Result<Self, IggyError> { + let mut seen = BTreeSet::new(); + let mut from = None; + let mut after = None; + let mut current_after = None; + let mut query = Self::default(); + for (key, value) in pairs { + if !seen.insert(key) { + return Err(IggyError::InvalidCommand); + } + match key { + FROM_KEY => from = Some(value), + AFTER_KEY => after = Some(MetadataChangeCursor::from_str(value)?), + CURRENT_AFTER_KEY => { + current_after = Some(MetadataCurrentPageToken::from_str(value)?) + } + GROUPS_KEY => query.groups = parse_groups(value)?, + SINCE_KEY => query.since = Some(parse_micros(value)?), + UNTIL_KEY => query.until = Some(parse_micros(value)?), + LIMIT_KEY => query.limit = parse_canonical(value)?, + _ => return Err(IggyError::InvalidCommand), + } + } + query.start = match (from, after, current_after) { + (Some(FROM_EARLIEST), None, None) => MetadataChangeStart::Earliest, + (Some(FROM_LATEST), None, None) => MetadataChangeStart::Latest, + (Some(FROM_CURRENT), None, token) => MetadataChangeStart::Current(token), + (None, Some(cursor), None) => MetadataChangeStart::After(cursor), + _ => return Err(IggyError::InvalidCommand), + }; + Ok(query) + } +} + +fn parse_groups(value: &str) -> Result<Vec<MetadataChangeGroup>, IggyError> { + if value.is_empty() { + return Ok(Vec::new()); + } + value + .split(',') + .map(|name| MetadataChangeGroup::from_str(name).map_err(|_| IggyError::InvalidCommand)) + .collect() +} + +fn parse_micros(value: &str) -> Result<IggyTimestamp, IggyError> { + IggyTimestamp::checked_from_micros(parse_canonical(value)?).ok_or(IggyError::InvalidCommand) +} + +/// A decimal integer without a sign, padding or leading zeros. +fn parse_canonical<T: FromStr>(value: &str) -> Result<T, IggyError> { + let canonical = !value.is_empty() + && value.bytes().all(|byte| byte.is_ascii_digit()) + && (value == "0" || !value.starts_with('0')); + if !canonical { + return Err(IggyError::InvalidCommand); + } + value.parse().map_err(|_| IggyError::InvalidCommand) +} + +#[cfg(test)] +mod tests { + use super::*; + use iggy_binary_protocol::primitives::metadata_changes::{ + WireMetadataChangeGroup, WireMetadataCurrentPageToken, + }; + + fn cursor() -> MetadataChangeCursor { + MetadataChangeCursor { + log_id: 9, + op: 12, + checksum: 34, + ordinal: 0, + } + } + + fn token_for(groups: MetadataChangeGroups) -> MetadataCurrentPageToken { + WireMetadataCurrentPageToken { + anchor: cursor().into(), + required_head: 20, + groups_mask: groups.mask(), + group: WireMetadataChangeGroup::Streams, + last_key: bytes::Bytes::from_static(&[1, 0, 0, 0]), + } + .into() + } + + fn query_from(start: MetadataChangeStart) -> MetadataChangesQuery { + MetadataChangesQuery { + start, + ..MetadataChangesQuery::default() + } + } + + #[test] + fn given_default_query_when_validating_should_select_every_group_from_earliest() { + let query = MetadataChangesQuery::default(); + assert_eq!(query.limit, DEFAULT_METADATA_CHANGES_LIMIT); + assert_eq!(query.start, MetadataChangeStart::Earliest); + assert_eq!(query.validate().unwrap(), MetadataChangeGroups::all()); + } + + #[test] + fn given_limit_outside_one_to_thousand_when_validating_should_refuse() { + for limit in [0, 1001, u32::MAX] { + let query = MetadataChangesQuery { + limit, + ..MetadataChangesQuery::default() + }; + assert_eq!(query.validate(), Err(IggyError::InvalidCommand)); + } + for limit in [1, 1000] { + let query = MetadataChangesQuery { + limit, + ..MetadataChangesQuery::default() + }; + assert!(query.validate().is_ok()); + } + } + + #[test] + fn given_since_above_until_when_validating_should_refuse() { + let query = MetadataChangesQuery { + since: Some(IggyTimestamp::from(5)), + until: Some(IggyTimestamp::from(4)), + ..MetadataChangesQuery::default() + }; + assert_eq!(query.validate(), Err(IggyError::InvalidCommand)); + } + + #[test] + fn given_time_bound_with_latest_or_current_when_validating_should_refuse() { + for start in [ + MetadataChangeStart::Latest, + MetadataChangeStart::Current(None), + ] { + let query = MetadataChangesQuery { + since: Some(IggyTimestamp::from(1)), + ..query_from(start) + }; + assert_eq!(query.validate(), Err(IggyError::InvalidCommand)); + } + let after = MetadataChangesQuery { + until: Some(IggyTimestamp::from(1)), + ..query_from(MetadataChangeStart::After(cursor())) + }; + assert!(after.validate().is_ok()); + } + + #[test] + fn given_current_scan_without_parent_group_when_validating_should_refuse() { + let query = MetadataChangesQuery { + groups: vec![MetadataChangeGroup::Topics], + ..query_from(MetadataChangeStart::Current(None)) + }; + assert_eq!(query.validate(), Err(IggyError::InvalidCommand)); + let changes = MetadataChangesQuery { + groups: vec![MetadataChangeGroup::Topics], + ..query_from(MetadataChangeStart::Earliest) + }; + assert!(changes.validate().is_ok()); + } + + #[test] + fn given_page_token_for_other_groups_when_validating_should_refuse() { + let groups = vec![MetadataChangeGroup::Streams, MetadataChangeGroup::Topics]; + let matching = MetadataChangesQuery { + groups: groups.clone(), + ..query_from(MetadataChangeStart::Current(Some(token_for( + MetadataChangeGroups::requested(&groups), + )))) + }; + assert!(matching.validate().is_ok()); + let other = MetadataChangesQuery { + groups, + ..query_from(MetadataChangeStart::Current(Some(token_for( + MetadataChangeGroups::all(), + )))) + }; + assert_eq!(other.validate(), Err(IggyError::InvalidCommand)); + } + + #[test] + fn given_non_canonical_cursor_when_validating_should_refuse() { + let query = query_from(MetadataChangeStart::After(MetadataChangeCursor { + log_id: 1, + op: 0, + checksum: 0, + ordinal: 3, + })); + assert_eq!(query.validate(), Err(IggyError::InvalidCommand)); + } + + #[test] + fn given_every_start_mode_when_round_tripping_http_query_should_preserve_it() { + let groups = vec![MetadataChangeGroup::Topics, MetadataChangeGroup::Streams]; + for start in [ + MetadataChangeStart::Earliest, + MetadataChangeStart::Latest, + MetadataChangeStart::After(cursor()), + MetadataChangeStart::Current(None), + MetadataChangeStart::Current(Some(token_for(MetadataChangeGroups::requested(&groups)))), + ] { + let original = MetadataChangesQuery { + start, + groups: groups.clone(), + since: None, + until: None, + limit: 500, + }; + let pairs = original.http_query_pairs(); + let parsed = MetadataChangesQuery::from_http_query( + pairs.iter().map(|(key, value)| (*key, value.as_str())), + ) + .unwrap(); + assert_eq!(parsed.start, original.start); + assert_eq!(parsed.limit, 500); + assert_eq!( + parsed.groups, + vec![MetadataChangeGroup::Streams, MetadataChangeGroup::Topics] + ); + } + } + + #[test] + fn given_refused_http_parameters_when_parsing_should_refuse_each() { + let after = cursor().to_string(); + let cases: Vec<Vec<(&str, &str)>> = vec![ + vec![], + vec![("from", "earliest"), ("after", &after)], + vec![("from", "newest")], + vec![("from", "earliest"), ("from", "latest")], + vec![("from", "earliest"), ("unknown", "1")], + vec![("from", "earliest"), ("groups", "streams,,topics")], + vec![("from", "earliest"), ("groups", "streams,")], + vec![("from", "earliest"), ("groups", "everything")], + vec![("from", "earliest"), ("since", "-1")], + vec![("from", "earliest"), ("since", "01")], + vec![("from", "earliest"), ("limit", "ten")], + vec![("from", "earliest"), ("limit", "01")], + vec![("from", "earliest"), ("limit", "+5")], + vec![("from", "latest"), ("current_after", "cs1_AAAA")], + vec![("after", "mc1_short")], + ]; + for case in cases { + assert_eq!( + MetadataChangesQuery::from_http_query(case.clone()), + Err(IggyError::InvalidCommand), + "{case:?}" + ); + } + } + + #[test] + fn given_empty_groups_parameter_when_parsing_should_mean_every_group() { + let parsed = + MetadataChangesQuery::from_http_query([("from", "latest"), ("groups", "")]).unwrap(); + assert!(parsed.groups.is_empty()); + assert_eq!(parsed.limit, DEFAULT_METADATA_CHANGES_LIMIT); + } +}
diff --git a/core/common/src/types/metadata_changes/records.rs b/core/common/src/types/metadata_changes/records.rs new file mode 100644 index 0000000..92a41cf --- /dev/null +++ b/core/common/src/types/metadata_changes/records.rs
@@ -0,0 +1,1205 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use super::json::hex_u128; +use super::{ + MetadataChangeGroup, MetadataChangeKind, MetadataMemberLeftReason, + MetadataPermissionsUpdatedReason, MetadataSessionClosedReason, MetadataTokenDeletedReason, +}; +use crate::types::options::resource_options_json; +use crate::{Permissions, ResourceOptions, UserStatus}; +use base64::Engine; +use base64::engine::general_purpose::STANDARD; +use bytes::Bytes; +use serde::de::Error as _; +use serde::{Deserialize, Deserializer, Serialize, Serializer}; +use serde_json::value::RawValue; +use std::str::FromStr; + +/// Public options of a resource after a change, as the read routes return +/// them, or a marker for stored critical state this build cannot read. A +/// mirror keeps its previous values for an unreadable resource. +#[derive(Debug, Clone, PartialEq)] +pub enum MetadataOptions { + Readable(ResourceOptions), + Unreadable, +} + +/// A JSON object kept verbatim: a record or item of a group or kind this +/// build does not know. +#[derive(Debug, Clone)] +pub struct MetadataRawJson(Box<RawValue>); + +/// One change record: the committed operation that produced it, its position +/// within that operation, and what is true after it. +/// +/// Its JSON form is one flat object with the envelope fields, `group`, `kind` +/// and the body fields. +#[derive(Debug, Clone, PartialEq)] +pub struct MetadataChangeRecord { + pub op: u64, + pub ordinal: u32, + /// Prepare checksum of `op`. + pub checksum: u128, + /// Prepare timestamp of `op` in microseconds. + pub timestamp: u64, + /// The acting user, absent for server actions. + pub actor_user_id: Option<u32>, + pub change: MetadataChange, +} + +/// The typed body of a change record. +#[derive(Debug, Clone, PartialEq)] +pub enum MetadataChange { + StreamCreated(MetadataStreamChange), + StreamUpdated(MetadataStreamChange), + StreamDeleted(MetadataStreamDeleted), + TopicCreated(MetadataTopicChange), + TopicUpdated(MetadataTopicChange), + TopicDeleted(MetadataTopicDeleted), + PartitionsChanged(MetadataPartitionsChanged), + ConsumerGroupCreated(MetadataConsumerGroupCreated), + ConsumerGroupDeleted(MetadataConsumerGroupDeleted), + ConsumerGroupMemberJoined(MetadataMemberJoined), + ConsumerGroupMemberLeft(MetadataMemberLeft), + ConsumerGroupPartitionHandedOver(MetadataPartitionHandedOver), + UserCreated(MetadataUserChange), + UserUpdated(MetadataUserChange), + UserDeleted(MetadataUserIdentity), + UserPasswordChanged(MetadataUserIdentity), + UserPermissionsUpdated(MetadataUserPermissionsUpdated), + PersonalAccessTokenCreated(MetadataTokenChange), + PersonalAccessTokenDeleted(MetadataTokenDeleted), + SessionRegistered(MetadataSession), + SessionClosed(MetadataSessionClosed), + /// A binary record of a group or kind this build does not know. + Unknown { + group: u8, + kind: u8, + body: Bytes, + }, + /// A JSON record of a group or kind this build does not know, kept whole. + UnknownJson { + group: String, + kind: String, + body: MetadataRawJson, + }, +} + +/// One current state item. +#[derive(Debug, Clone, PartialEq)] +pub enum MetadataCurrentItem { + Stream(MetadataStreamChange), + Topic(MetadataTopicItem), + Partitions(MetadataPartitionsItem), + ConsumerGroup(MetadataConsumerGroupItem), + ConsumerGroupMember(MetadataMemberItem), + User(MetadataUserChange), + PersonalAccessToken(MetadataTokenChange), + Session(MetadataSession), + /// A binary item of a group this build does not know. + Unknown { + group: u8, + body: Bytes, + }, + /// A JSON item of a group this build does not know, kept whole. + UnknownJson { + group: String, + body: MetadataRawJson, + }, +} + +/// A stream after a create or update, and a `streams` item. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(into = "StreamChangeJson", try_from = "StreamChangeJson")] +pub struct MetadataStreamChange { + pub stream_id: u32, + pub stream_created_op: u64, + pub created_at: u64, + pub name: String, + pub options: MetadataOptions, +} + +/// A removed stream. `requested_op` is the op of a deferred delete's request. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MetadataStreamDeleted { + pub stream_id: u32, + pub stream_created_op: u64, + pub created_at: u64, + pub name: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub requested_op: Option<u64>, +} + +/// A topic after a create or update. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(into = "TopicChangeJson", try_from = "TopicChangeJson")] +pub struct MetadataTopicChange { + pub stream_id: u32, + pub stream_created_op: u64, + pub topic_id: u32, + pub topic_created_op: u64, + pub created_at: u64, + pub name: String, + pub partitions_count: u32, + pub options: MetadataOptions, +} + +/// A `topics` item. Partition counts travel as `partitions` items. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(into = "TopicItemJson", try_from = "TopicItemJson")] +pub struct MetadataTopicItem { + pub stream_id: u32, + pub stream_created_op: u64, + pub topic_id: u32, + pub topic_created_op: u64, + pub created_at: u64, + pub name: String, + pub options: MetadataOptions, +} + +/// A removed topic. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MetadataTopicDeleted { + pub stream_id: u32, + pub stream_created_op: u64, + pub topic_id: u32, + pub topic_created_op: u64, + pub created_at: u64, + pub name: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub requested_op: Option<u64>, +} + +/// A partition count change. `removed_partition_ids` lists the deleted ids, +/// ascending, and is empty on create. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MetadataPartitionsChanged { + pub stream_id: u32, + pub stream_created_op: u64, + pub topic_id: u32, + pub topic_created_op: u64, + pub partitions_count: u32, + pub removed_partition_ids: Vec<u32>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub requested_op: Option<u64>, +} + +/// A `partitions` item. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MetadataPartitionsItem { + pub stream_id: u32, + pub stream_created_op: u64, + pub topic_id: u32, + pub topic_created_op: u64, + pub partitions_count: u32, +} + +/// A created consumer group with the topic's resulting group count. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MetadataConsumerGroupCreated { + pub stream_id: u32, + pub stream_created_op: u64, + pub topic_id: u32, + pub topic_created_op: u64, + pub group_id: u32, + pub name: String, + pub consumer_groups_count: u32, +} + +/// A removed consumer group with the topic's resulting group count. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MetadataConsumerGroupDeleted { + pub stream_id: u32, + pub stream_created_op: u64, + pub topic_id: u32, + pub topic_created_op: u64, + pub group_id: u32, + pub name: String, + pub consumer_groups_count: u32, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub requested_op: Option<u64>, +} + +/// A `consumer_groups` item. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MetadataConsumerGroupItem { + pub stream_id: u32, + pub stream_created_op: u64, + pub topic_id: u32, + pub topic_created_op: u64, + pub group_id: u32, + pub name: String, +} + +/// A member that joined, with its installed partitions (normally none, a +/// handover installs them). +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MetadataMemberJoined { + pub stream_id: u32, + pub stream_created_op: u64, + pub topic_id: u32, + pub topic_created_op: u64, + pub group_id: u32, + pub member_id: u32, + #[serde(with = "hex_u128")] + pub client_id: u128, + pub members_count: u32, + pub partition_ids: Vec<u32>, +} + +/// A member that left the group. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MetadataMemberLeft { + pub stream_id: u32, + pub stream_created_op: u64, + pub topic_id: u32, + pub topic_created_op: u64, + pub group_id: u32, + pub member_id: u32, + #[serde(with = "hex_u128")] + pub client_id: u128, + pub members_count: u32, + pub reason: MetadataMemberLeftReason, +} + +/// A completed partition handover. `member_id` is absent when the last +/// member left. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MetadataPartitionHandedOver { + pub stream_id: u32, + pub stream_created_op: u64, + pub topic_id: u32, + pub topic_created_op: u64, + pub group_id: u32, + pub partition_id: u32, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub member_id: Option<u32>, +} + +/// A `consumer_group_members` item. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MetadataMemberItem { + pub stream_id: u32, + pub stream_created_op: u64, + pub topic_id: u32, + pub topic_created_op: u64, + pub group_id: u32, + pub member_id: u32, + #[serde(with = "hex_u128")] + pub client_id: u128, + pub partition_ids: Vec<u32>, +} + +/// A user after a create or update, and a `users` item. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(into = "UserChangeJson", try_from = "UserChangeJson")] +pub struct MetadataUserChange { + pub user_id: u32, + pub user_created_at: u64, + pub username: String, + pub status: UserStatus, + pub permissions: Option<Permissions>, + pub options: MetadataOptions, +} + +/// A deleted user, or one whose password changed. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MetadataUserIdentity { + pub user_id: u32, + pub user_created_at: u64, + pub username: String, +} + +/// A user's public permission set after a change. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct MetadataUserPermissionsUpdated { + pub user_id: u32, + pub user_created_at: u64, + pub username: String, + pub permissions: Option<Permissions>, + pub reason: MetadataPermissionsUpdatedReason, +} + +/// A created personal access token, and a `personal_access_tokens` item. +/// `expiry` in microseconds is absent for a token that never expires. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MetadataTokenChange { + pub user_id: u32, + pub user_created_at: u64, + pub name: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub expiry: Option<u64>, +} + +/// A removed personal access token. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MetadataTokenDeleted { + pub user_id: u32, + pub user_created_at: u64, + pub name: String, + pub reason: MetadataTokenDeletedReason, +} + +/// A registered session, and a `sessions` item. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MetadataSession { + pub user_id: u32, + pub user_created_at: u64, + #[serde(with = "hex_u128")] + pub client_id: u128, + pub session: u64, +} + +/// A closed session. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct MetadataSessionClosed { + pub user_id: u32, + pub user_created_at: u64, + #[serde(with = "hex_u128")] + pub client_id: u128, + pub session: u64, + pub reason: MetadataSessionClosedReason, +} + +impl MetadataRawJson { + /// The verbatim JSON text. + #[must_use] + pub fn get(&self) -> &str { + self.0.get() + } +} + +impl PartialEq for MetadataRawJson { + fn eq(&self, other: &Self) -> bool { + self.0.get() == other.0.get() + } +} + +impl From<Box<RawValue>> for MetadataRawJson { + fn from(raw: Box<RawValue>) -> Self { + Self(raw) + } +} + +impl MetadataChange { + /// The kind, absent for a record this build does not know. + #[must_use] + pub const fn kind(&self) -> Option<MetadataChangeKind> { + Some(match self { + Self::StreamCreated(_) => MetadataChangeKind::StreamCreated, + Self::StreamUpdated(_) => MetadataChangeKind::StreamUpdated, + Self::StreamDeleted(_) => MetadataChangeKind::StreamDeleted, + Self::TopicCreated(_) => MetadataChangeKind::TopicCreated, + Self::TopicUpdated(_) => MetadataChangeKind::TopicUpdated, + Self::TopicDeleted(_) => MetadataChangeKind::TopicDeleted, + Self::PartitionsChanged(_) => MetadataChangeKind::PartitionsChanged, + Self::ConsumerGroupCreated(_) => MetadataChangeKind::ConsumerGroupCreated, + Self::ConsumerGroupDeleted(_) => MetadataChangeKind::ConsumerGroupDeleted, + Self::ConsumerGroupMemberJoined(_) => MetadataChangeKind::ConsumerGroupMemberJoined, + Self::ConsumerGroupMemberLeft(_) => MetadataChangeKind::ConsumerGroupMemberLeft, + Self::ConsumerGroupPartitionHandedOver(_) => { + MetadataChangeKind::ConsumerGroupPartitionHandedOver + } + Self::UserCreated(_) => MetadataChangeKind::UserCreated, + Self::UserUpdated(_) => MetadataChangeKind::UserUpdated, + Self::UserDeleted(_) => MetadataChangeKind::UserDeleted, + Self::UserPasswordChanged(_) => MetadataChangeKind::UserPasswordChanged, + Self::UserPermissionsUpdated(_) => MetadataChangeKind::UserPermissionsUpdated, + Self::PersonalAccessTokenCreated(_) => MetadataChangeKind::PersonalAccessTokenCreated, + Self::PersonalAccessTokenDeleted(_) => MetadataChangeKind::PersonalAccessTokenDeleted, + Self::SessionRegistered(_) => MetadataChangeKind::SessionRegistered, + Self::SessionClosed(_) => MetadataChangeKind::SessionClosed, + Self::Unknown { .. } | Self::UnknownJson { .. } => return None, + }) + } + + /// The group, absent for a record this build does not know. + #[must_use] + pub fn group(&self) -> Option<MetadataChangeGroup> { + self.kind().map(MetadataChangeKind::group) + } + + fn from_json(kind: MetadataChangeKind, text: &str) -> serde_json::Result<Self> { + Ok(match kind { + MetadataChangeKind::StreamCreated => Self::StreamCreated(serde_json::from_str(text)?), + MetadataChangeKind::StreamUpdated => Self::StreamUpdated(serde_json::from_str(text)?), + MetadataChangeKind::StreamDeleted => Self::StreamDeleted(serde_json::from_str(text)?), + MetadataChangeKind::TopicCreated => Self::TopicCreated(serde_json::from_str(text)?), + MetadataChangeKind::TopicUpdated => Self::TopicUpdated(serde_json::from_str(text)?), + MetadataChangeKind::TopicDeleted => Self::TopicDeleted(serde_json::from_str(text)?), + MetadataChangeKind::PartitionsChanged => { + Self::PartitionsChanged(serde_json::from_str(text)?) + } + MetadataChangeKind::ConsumerGroupCreated => { + Self::ConsumerGroupCreated(serde_json::from_str(text)?) + } + MetadataChangeKind::ConsumerGroupDeleted => { + Self::ConsumerGroupDeleted(serde_json::from_str(text)?) + } + MetadataChangeKind::ConsumerGroupMemberJoined => { + Self::ConsumerGroupMemberJoined(serde_json::from_str(text)?) + } + MetadataChangeKind::ConsumerGroupMemberLeft => { + Self::ConsumerGroupMemberLeft(serde_json::from_str(text)?) + } + MetadataChangeKind::ConsumerGroupPartitionHandedOver => { + Self::ConsumerGroupPartitionHandedOver(serde_json::from_str(text)?) + } + MetadataChangeKind::UserCreated => Self::UserCreated(serde_json::from_str(text)?), + MetadataChangeKind::UserUpdated => Self::UserUpdated(serde_json::from_str(text)?), + MetadataChangeKind::UserDeleted => Self::UserDeleted(serde_json::from_str(text)?), + MetadataChangeKind::UserPasswordChanged => { + Self::UserPasswordChanged(serde_json::from_str(text)?) + } + MetadataChangeKind::UserPermissionsUpdated => { + Self::UserPermissionsUpdated(serde_json::from_str(text)?) + } + MetadataChangeKind::PersonalAccessTokenCreated => { + Self::PersonalAccessTokenCreated(serde_json::from_str(text)?) + } + MetadataChangeKind::PersonalAccessTokenDeleted => { + Self::PersonalAccessTokenDeleted(serde_json::from_str(text)?) + } + MetadataChangeKind::SessionRegistered => { + Self::SessionRegistered(serde_json::from_str(text)?) + } + MetadataChangeKind::SessionClosed => Self::SessionClosed(serde_json::from_str(text)?), + }) + } +} + +impl MetadataCurrentItem { + /// The group, absent for an item this build does not know. + #[must_use] + pub const fn group(&self) -> Option<MetadataChangeGroup> { + Some(match self { + Self::Stream(_) => MetadataChangeGroup::Streams, + Self::Topic(_) => MetadataChangeGroup::Topics, + Self::Partitions(_) => MetadataChangeGroup::Partitions, + Self::ConsumerGroup(_) => MetadataChangeGroup::ConsumerGroups, + Self::ConsumerGroupMember(_) => MetadataChangeGroup::ConsumerGroupMembers, + Self::User(_) => MetadataChangeGroup::Users, + Self::PersonalAccessToken(_) => MetadataChangeGroup::PersonalAccessTokens, + Self::Session(_) => MetadataChangeGroup::Sessions, + Self::Unknown { .. } | Self::UnknownJson { .. } => return None, + }) + } + + fn from_json(group: MetadataChangeGroup, text: &str) -> serde_json::Result<Self> { + Ok(match group { + MetadataChangeGroup::Streams => Self::Stream(serde_json::from_str(text)?), + MetadataChangeGroup::Topics => Self::Topic(serde_json::from_str(text)?), + MetadataChangeGroup::Partitions => Self::Partitions(serde_json::from_str(text)?), + MetadataChangeGroup::ConsumerGroups => Self::ConsumerGroup(serde_json::from_str(text)?), + MetadataChangeGroup::ConsumerGroupMembers => { + Self::ConsumerGroupMember(serde_json::from_str(text)?) + } + MetadataChangeGroup::Users => Self::User(serde_json::from_str(text)?), + MetadataChangeGroup::PersonalAccessTokens => { + Self::PersonalAccessToken(serde_json::from_str(text)?) + } + MetadataChangeGroup::Sessions => Self::Session(serde_json::from_str(text)?), + }) + } +} + +/// The envelope fields of a record object, with the body fields left out. +#[derive(Deserialize)] +struct RecordEnvelopeJson { + op: u64, + ordinal: u32, + #[serde(with = "hex_u128")] + checksum: u128, + timestamp: u64, + #[serde(default)] + actor_user_id: Option<u32>, + group: String, + kind: String, +} + +#[derive(Serialize)] +struct RecordJson<'a, B: Serialize> { + op: u64, + ordinal: u32, + #[serde(with = "hex_u128")] + checksum: u128, + timestamp: u64, + #[serde(skip_serializing_if = "Option::is_none")] + actor_user_id: Option<u32>, + group: &'a str, + kind: &'a str, + #[serde(flatten)] + body: &'a B, +} + +/// A binary record of an unknown kind rendered as JSON: its codes as decimal +/// text and its body in base64, so a JSON reader keeps it as unknown too. +#[derive(Serialize)] +struct UnknownRecordJson { + op: u64, + ordinal: u32, + #[serde(with = "hex_u128")] + checksum: u128, + timestamp: u64, + #[serde(skip_serializing_if = "Option::is_none")] + actor_user_id: Option<u32>, + group: String, + kind: String, + body: String, +} + +#[derive(Deserialize)] +struct ItemEnvelopeJson { + group: String, +} + +#[derive(Serialize)] +struct ItemJson<'a, B: Serialize> { + group: &'a str, + #[serde(flatten)] + body: &'a B, +} + +#[derive(Serialize)] +struct UnknownItemJson { + group: String, + body: String, +} + +impl MetadataChangeRecord { + fn envelope<'a, B: Serialize>( + &self, + group: &'a str, + kind: &'a str, + body: &'a B, + ) -> RecordJson<'a, B> { + RecordJson { + op: self.op, + ordinal: self.ordinal, + checksum: self.checksum, + timestamp: self.timestamp, + actor_user_id: self.actor_user_id, + group, + kind, + body, + } + } + + fn serialize_body<S: Serializer, B: Serialize>( + &self, + serializer: S, + kind: MetadataChangeKind, + body: &B, + ) -> Result<S::Ok, S::Error> { + self.envelope(kind.group().into(), kind.into(), body) + .serialize(serializer) + } +} + +impl Serialize for MetadataChangeRecord { + fn serialize<S: Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> { + let kind = self.change.kind(); + match (&self.change, kind) { + (MetadataChange::UnknownJson { body, .. }, _) => body.0.serialize(serializer), + (MetadataChange::Unknown { group, kind, body }, _) => UnknownRecordJson { + op: self.op, + ordinal: self.ordinal, + checksum: self.checksum, + timestamp: self.timestamp, + actor_user_id: self.actor_user_id, + group: group.to_string(), + kind: kind.to_string(), + body: STANDARD.encode(body), + } + .serialize(serializer), + ( + MetadataChange::StreamCreated(body) | MetadataChange::StreamUpdated(body), + Some(kind), + ) => self.serialize_body(serializer, kind, body), + (MetadataChange::StreamDeleted(body), Some(kind)) => { + self.serialize_body(serializer, kind, body) + } + ( + MetadataChange::TopicCreated(body) | MetadataChange::TopicUpdated(body), + Some(kind), + ) => self.serialize_body(serializer, kind, body), + (MetadataChange::TopicDeleted(body), Some(kind)) => { + self.serialize_body(serializer, kind, body) + } + (MetadataChange::PartitionsChanged(body), Some(kind)) => { + self.serialize_body(serializer, kind, body) + } + (MetadataChange::ConsumerGroupCreated(body), Some(kind)) => { + self.serialize_body(serializer, kind, body) + } + (MetadataChange::ConsumerGroupDeleted(body), Some(kind)) => { + self.serialize_body(serializer, kind, body) + } + (MetadataChange::ConsumerGroupMemberJoined(body), Some(kind)) => { + self.serialize_body(serializer, kind, body) + } + (MetadataChange::ConsumerGroupMemberLeft(body), Some(kind)) => { + self.serialize_body(serializer, kind, body) + } + (MetadataChange::ConsumerGroupPartitionHandedOver(body), Some(kind)) => { + self.serialize_body(serializer, kind, body) + } + (MetadataChange::UserCreated(body) | MetadataChange::UserUpdated(body), Some(kind)) => { + self.serialize_body(serializer, kind, body) + } + ( + MetadataChange::UserDeleted(body) | MetadataChange::UserPasswordChanged(body), + Some(kind), + ) => self.serialize_body(serializer, kind, body), + (MetadataChange::UserPermissionsUpdated(body), Some(kind)) => { + self.serialize_body(serializer, kind, body) + } + (MetadataChange::PersonalAccessTokenCreated(body), Some(kind)) => { + self.serialize_body(serializer, kind, body) + } + (MetadataChange::PersonalAccessTokenDeleted(body), Some(kind)) => { + self.serialize_body(serializer, kind, body) + } + (MetadataChange::SessionRegistered(body), Some(kind)) => { + self.serialize_body(serializer, kind, body) + } + (MetadataChange::SessionClosed(body), Some(kind)) => { + self.serialize_body(serializer, kind, body) + } + (_, None) => Err(serde::ser::Error::custom("a known change has a kind")), + } + } +} + +impl<'de> Deserialize<'de> for MetadataChangeRecord { + fn deserialize<D: Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> { + let raw = Box::<RawValue>::deserialize(deserializer)?; + let envelope: RecordEnvelopeJson = + serde_json::from_str(raw.get()).map_err(D::Error::custom)?; + let group = MetadataChangeGroup::from_str(&envelope.group); + let kind = MetadataChangeKind::from_str(&envelope.kind); + let change = match (group, kind) { + (Ok(group), Ok(kind)) => { + if kind.group() != group { + return Err(D::Error::custom(format!( + "change kind {kind} does not belong to group {group}" + ))); + } + MetadataChange::from_json(kind, raw.get()).map_err(D::Error::custom)? + } + _ => MetadataChange::UnknownJson { + group: envelope.group, + kind: envelope.kind, + body: MetadataRawJson(raw), + }, + }; + Ok(Self { + op: envelope.op, + ordinal: envelope.ordinal, + checksum: envelope.checksum, + timestamp: envelope.timestamp, + actor_user_id: envelope.actor_user_id, + change, + }) + } +} + +impl Serialize for MetadataCurrentItem { + fn serialize<S: Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> { + fn item<S: Serializer, B: Serialize>( + serializer: S, + group: MetadataChangeGroup, + body: &B, + ) -> Result<S::Ok, S::Error> { + ItemJson { + group: group.into(), + body, + } + .serialize(serializer) + } + match self { + Self::Stream(body) => item(serializer, MetadataChangeGroup::Streams, body), + Self::Topic(body) => item(serializer, MetadataChangeGroup::Topics, body), + Self::Partitions(body) => item(serializer, MetadataChangeGroup::Partitions, body), + Self::ConsumerGroup(body) => { + item(serializer, MetadataChangeGroup::ConsumerGroups, body) + } + Self::ConsumerGroupMember(body) => { + item(serializer, MetadataChangeGroup::ConsumerGroupMembers, body) + } + Self::User(body) => item(serializer, MetadataChangeGroup::Users, body), + Self::PersonalAccessToken(body) => { + item(serializer, MetadataChangeGroup::PersonalAccessTokens, body) + } + Self::Session(body) => item(serializer, MetadataChangeGroup::Sessions, body), + Self::Unknown { group, body } => UnknownItemJson { + group: group.to_string(), + body: STANDARD.encode(body), + } + .serialize(serializer), + Self::UnknownJson { body, .. } => body.0.serialize(serializer), + } + } +} + +impl<'de> Deserialize<'de> for MetadataCurrentItem { + fn deserialize<D: Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> { + let raw = Box::<RawValue>::deserialize(deserializer)?; + let envelope: ItemEnvelopeJson = + serde_json::from_str(raw.get()).map_err(D::Error::custom)?; + match MetadataChangeGroup::from_str(&envelope.group) { + Ok(group) => Self::from_json(group, raw.get()).map_err(D::Error::custom), + Err(_) => Ok(Self::UnknownJson { + group: envelope.group, + body: MetadataRawJson(raw), + }), + } + } +} + +/// The readable options map of a body's JSON form. +#[derive(Clone, Serialize, Deserialize)] +struct OptionsJson(#[serde(with = "resource_options_json")] ResourceOptions); + +fn options_to_json(options: MetadataOptions) -> (Option<OptionsJson>, bool) { + match options { + MetadataOptions::Readable(options) => (Some(OptionsJson(options)), false), + MetadataOptions::Unreadable => (None, true), + } +} + +fn options_from_json( + options: Option<OptionsJson>, + unreadable: bool, +) -> Result<MetadataOptions, String> { + match (options, unreadable) { + (Some(OptionsJson(options)), false) => Ok(MetadataOptions::Readable(options)), + (None, true) => Ok(MetadataOptions::Unreadable), + _ => Err("options must be an object, or null with options_unreadable true".to_owned()), + } +} + +#[derive(Clone, Serialize, Deserialize)] +struct StreamChangeJson { + stream_id: u32, + stream_created_op: u64, + created_at: u64, + name: String, + options: Option<OptionsJson>, + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + options_unreadable: bool, +} + +impl From<MetadataStreamChange> for StreamChangeJson { + fn from(body: MetadataStreamChange) -> Self { + let (options, options_unreadable) = options_to_json(body.options); + Self { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + created_at: body.created_at, + name: body.name, + options, + options_unreadable, + } + } +} + +impl TryFrom<StreamChangeJson> for MetadataStreamChange { + type Error = String; + + fn try_from(json: StreamChangeJson) -> Result<Self, Self::Error> { + Ok(Self { + stream_id: json.stream_id, + stream_created_op: json.stream_created_op, + created_at: json.created_at, + name: json.name, + options: options_from_json(json.options, json.options_unreadable)?, + }) + } +} + +#[derive(Clone, Serialize, Deserialize)] +struct TopicChangeJson { + stream_id: u32, + stream_created_op: u64, + topic_id: u32, + topic_created_op: u64, + created_at: u64, + name: String, + partitions_count: u32, + options: Option<OptionsJson>, + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + options_unreadable: bool, +} + +impl From<MetadataTopicChange> for TopicChangeJson { + fn from(body: MetadataTopicChange) -> Self { + let (options, options_unreadable) = options_to_json(body.options); + Self { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + created_at: body.created_at, + name: body.name, + partitions_count: body.partitions_count, + options, + options_unreadable, + } + } +} + +impl TryFrom<TopicChangeJson> for MetadataTopicChange { + type Error = String; + + fn try_from(json: TopicChangeJson) -> Result<Self, Self::Error> { + Ok(Self { + stream_id: json.stream_id, + stream_created_op: json.stream_created_op, + topic_id: json.topic_id, + topic_created_op: json.topic_created_op, + created_at: json.created_at, + name: json.name, + partitions_count: json.partitions_count, + options: options_from_json(json.options, json.options_unreadable)?, + }) + } +} + +#[derive(Clone, Serialize, Deserialize)] +struct TopicItemJson { + stream_id: u32, + stream_created_op: u64, + topic_id: u32, + topic_created_op: u64, + created_at: u64, + name: String, + options: Option<OptionsJson>, + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + options_unreadable: bool, +} + +impl From<MetadataTopicItem> for TopicItemJson { + fn from(body: MetadataTopicItem) -> Self { + let (options, options_unreadable) = options_to_json(body.options); + Self { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + created_at: body.created_at, + name: body.name, + options, + options_unreadable, + } + } +} + +impl TryFrom<TopicItemJson> for MetadataTopicItem { + type Error = String; + + fn try_from(json: TopicItemJson) -> Result<Self, Self::Error> { + Ok(Self { + stream_id: json.stream_id, + stream_created_op: json.stream_created_op, + topic_id: json.topic_id, + topic_created_op: json.topic_created_op, + created_at: json.created_at, + name: json.name, + options: options_from_json(json.options, json.options_unreadable)?, + }) + } +} + +#[derive(Clone, Serialize, Deserialize)] +struct UserChangeJson { + user_id: u32, + user_created_at: u64, + username: String, + status: UserStatus, + permissions: Option<Permissions>, + options: Option<OptionsJson>, + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + options_unreadable: bool, +} + +impl From<MetadataUserChange> for UserChangeJson { + fn from(body: MetadataUserChange) -> Self { + let (options, options_unreadable) = options_to_json(body.options); + Self { + user_id: body.user_id, + user_created_at: body.user_created_at, + username: body.username, + status: body.status, + permissions: body.permissions, + options, + options_unreadable, + } + } +} + +impl TryFrom<UserChangeJson> for MetadataUserChange { + type Error = String; + + fn try_from(json: UserChangeJson) -> Result<Self, Self::Error> { + Ok(Self { + user_id: json.user_id, + user_created_at: json.user_created_at, + username: json.username, + status: json.status, + permissions: json.permissions, + options: options_from_json(json.options, json.options_unreadable)?, + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::{GlobalPermissions, HeaderKey, HeaderValue, OptionValue, StreamPermissions}; + use std::collections::BTreeMap; + + fn stream_change() -> MetadataStreamChange { + let mut options = ResourceOptions::new(); + options.insert( + HeaderKey::from_str("encryption_key_source").unwrap(), + OptionValue::derived(HeaderValue::try_from("generated").unwrap()), + ); + MetadataStreamChange { + stream_id: 3, + stream_created_op: 4121, + created_at: 1_760_356_800_123_456, + name: "telemetry".to_owned(), + options: MetadataOptions::Readable(options), + } + } + + fn permissions() -> Permissions { + Permissions { + global: GlobalPermissions::default(), + streams: Some(BTreeMap::from([( + 7, + StreamPermissions { + read_stream: true, + ..StreamPermissions::default() + }, + )])), + } + } + + fn record(change: MetadataChange, actor_user_id: Option<u32>) -> MetadataChangeRecord { + MetadataChangeRecord { + op: 4121, + ordinal: 0, + checksum: 0xa5c1_f703_8b9e_2d4c, + timestamp: 1_760_356_800_123_456, + actor_user_id, + change, + } + } + + #[test] + fn given_stream_created_record_when_serializing_should_render_one_flat_object() { + let json = serde_json::to_string(&record( + MetadataChange::StreamCreated(stream_change()), + Some(4), + )) + .unwrap(); + assert_eq!( + json, + concat!( + r#"{"op":4121,"ordinal":0,"checksum":"0000000000000000a5c1f7038b9e2d4c","#, + r#""timestamp":1760356800123456,"actor_user_id":4,"group":"streams","#, + r#""kind":"stream_created","stream_id":3,"stream_created_op":4121,"#, + r#""created_at":1760356800123456,"name":"telemetry","options":"#, + r#"{"encryption_key_source":{"value":"generated","kind":"string","explicit":false}}}"# + ) + ); + } + + #[test] + fn given_server_action_when_serializing_should_omit_the_actor() { + let json = serde_json::to_string(&record( + MetadataChange::StreamUpdated(stream_change()), + None, + )) + .unwrap(); + assert!(!json.contains("actor_user_id")); + let decoded: MetadataChangeRecord = serde_json::from_str(&json).unwrap(); + assert_eq!(decoded.actor_user_id, None); + } + + #[test] + fn given_every_body_shape_when_round_tripping_json_should_preserve_it() { + let changes = vec![ + MetadataChange::StreamCreated(stream_change()), + MetadataChange::StreamUpdated(MetadataStreamChange { + options: MetadataOptions::Unreadable, + ..stream_change() + }), + MetadataChange::StreamDeleted(MetadataStreamDeleted { + stream_id: 3, + stream_created_op: 4121, + created_at: 1, + name: "telemetry".to_owned(), + requested_op: Some(4140), + }), + MetadataChange::PartitionsChanged(MetadataPartitionsChanged { + stream_id: 3, + stream_created_op: 4121, + topic_id: 0, + topic_created_op: 4125, + partitions_count: 4, + removed_partition_ids: vec![4, 5], + requested_op: None, + }), + MetadataChange::ConsumerGroupMemberLeft(MetadataMemberLeft { + stream_id: 3, + stream_created_op: 4121, + topic_id: 0, + topic_created_op: 4125, + group_id: 1, + member_id: 2, + client_id: u128::MAX, + members_count: 0, + reason: MetadataMemberLeftReason::SessionEnded, + }), + MetadataChange::UserCreated(MetadataUserChange { + user_id: 7, + user_created_at: 1_760_300_000_000_000, + username: "writer".to_owned(), + status: UserStatus::Inactive, + permissions: Some(permissions()), + options: MetadataOptions::Readable(ResourceOptions::new()), + }), + MetadataChange::UserPermissionsUpdated(MetadataUserPermissionsUpdated { + user_id: 7, + user_created_at: 1_760_300_000_000_000, + username: "writer".to_owned(), + permissions: None, + reason: MetadataPermissionsUpdatedReason::ResourceDeleted, + }), + MetadataChange::PersonalAccessTokenCreated(MetadataTokenChange { + user_id: 7, + user_created_at: 1, + name: "ci".to_owned(), + expiry: None, + }), + MetadataChange::SessionClosed(MetadataSessionClosed { + user_id: 7, + user_created_at: 1, + client_id: 1 << 100, + session: 9, + reason: MetadataSessionClosedReason::Expired, + }), + ]; + for change in changes { + let original = record(change, Some(1)); + let json = serde_json::to_string(&original).unwrap(); + let decoded: MetadataChangeRecord = serde_json::from_str(&json).unwrap(); + assert_eq!(decoded, original, "{json}"); + } + } + + #[test] + fn given_unknown_kind_or_group_when_deserializing_should_keep_the_raw_object() { + let text = concat!( + r#"{"op":5,"ordinal":1,"checksum":"00000000000000000000000000000007","#, + r#""timestamp":9,"group":"streams","kind":"stream_archived","stream_id":3,"extra":[1,2]}"# + ); + let decoded: MetadataChangeRecord = serde_json::from_str(text).unwrap(); + let MetadataChange::UnknownJson { group, kind, body } = &decoded.change else { + panic!("expected an unknown record"); + }; + assert_eq!( + (group.as_str(), kind.as_str()), + ("streams", "stream_archived") + ); + assert_eq!(body.get(), text); + assert_eq!(serde_json::to_string(&decoded).unwrap(), text); + let future_group = text.replace(r#""group":"streams""#, r#""group":"audit""#); + let decoded: MetadataChangeRecord = serde_json::from_str(&future_group).unwrap(); + assert!(decoded.change.kind().is_none()); + } + + #[test] + fn given_known_kind_under_wrong_group_when_deserializing_should_refuse() { + let json = serde_json::to_string(&record( + MetadataChange::StreamCreated(stream_change()), + None, + )) + .unwrap() + .replace(r#""group":"streams""#, r#""group":"topics""#); + assert!(serde_json::from_str::<MetadataChangeRecord>(&json).is_err()); + } + + #[test] + fn given_options_and_unreadable_marker_both_set_when_deserializing_should_refuse() { + let json = serde_json::to_string(&MetadataStreamChange { + options: MetadataOptions::Unreadable, + ..stream_change() + }) + .unwrap(); + assert!(json.contains(r#""options":null,"options_unreadable":true"#)); + let both = json.replace(r#""options":null"#, r#""options":{}"#); + assert!(serde_json::from_str::<MetadataStreamChange>(&both).is_err()); + let neither = json.replace(r#","options_unreadable":true"#, ""); + assert!(serde_json::from_str::<MetadataStreamChange>(&neither).is_err()); + } + + #[test] + fn given_every_item_group_when_round_tripping_json_should_preserve_it() { + let items = vec![ + MetadataCurrentItem::Stream(stream_change()), + MetadataCurrentItem::Partitions(MetadataPartitionsItem { + stream_id: 1, + stream_created_op: 2, + topic_id: 3, + topic_created_op: 4, + partitions_count: 5, + }), + MetadataCurrentItem::ConsumerGroupMember(MetadataMemberItem { + stream_id: 1, + stream_created_op: 2, + topic_id: 3, + topic_created_op: 4, + group_id: 5, + member_id: 6, + client_id: 7, + partition_ids: vec![0, 1], + }), + MetadataCurrentItem::Session(MetadataSession { + user_id: 1, + user_created_at: 2, + client_id: 3, + session: 4, + }), + ]; + for item in items { + let json = serde_json::to_string(&item).unwrap(); + assert!(json.starts_with(r#"{"group":""#), "{json}"); + assert_eq!( + serde_json::from_str::<MetadataCurrentItem>(&json).unwrap(), + item + ); + } + let future: MetadataCurrentItem = + serde_json::from_str(r#"{"group":"audit","id":1}"#).unwrap(); + assert_eq!(future.group(), None); + } +}
diff --git a/core/common/src/types/metadata_changes/wire.rs b/core/common/src/types/metadata_changes/wire.rs new file mode 100644 index 0000000..fd1740e --- /dev/null +++ b/core/common/src/types/metadata_changes/wire.rs
@@ -0,0 +1,731 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Conversions between the wire and domain forms of the change log. + +use super::{ + MetadataChange, MetadataChangeGroup, MetadataChangeRecord, MetadataChangeStart, + MetadataChanges, MetadataChangesQuery, MetadataConsumerGroupCreated, + MetadataConsumerGroupDeleted, MetadataConsumerGroupItem, MetadataCurrentItem, + MetadataMemberItem, MetadataMemberJoined, MetadataMemberLeft, MetadataOptions, + MetadataPartitionHandedOver, MetadataPartitionsChanged, MetadataPartitionsItem, + MetadataSession, MetadataSessionClosed, MetadataStreamChange, MetadataStreamDeleted, + MetadataTokenChange, MetadataTokenDeleted, MetadataTopicChange, MetadataTopicDeleted, + MetadataTopicItem, MetadataUserChange, MetadataUserIdentity, MetadataUserPermissionsUpdated, +}; +use crate::wire_conversions::resource_options_from_wire_split; +use crate::{IggyError, IggyTimestamp, Permissions, UserStatus}; +use iggy_binary_protocol::primitives::metadata_changes::{ + WireChangeOptions, WireConsumerGroupCreatedBody, WireConsumerGroupDeletedBody, + WireConsumerGroupItemBody, WireMemberItemBody, WireMemberJoinedBody, WireMemberLeftBody, + WireMetadataChange, WireMetadataChangeGroup, WireMetadataCurrentItem, + WirePartitionHandedOverBody, WirePartitionsChangedBody, WirePartitionsItemBody, + WireSessionBody, WireSessionClosedBody, WireStreamBody, WireStreamDeletedBody, WireTokenBody, + WireTokenDeletedBody, WireTopicBody, WireTopicDeletedBody, WireTopicItemBody, WireUserBody, + WireUserIdentityBody, WireUserPermissionsBody, +}; +use iggy_binary_protocol::requests::system::{GetMetadataChangesRequest, WireMetadataChangeStart}; +use iggy_binary_protocol::responses::system::{ + GetMetadataChangesResponse, WireMetadataChangeRecord, WireMetadataItem, +}; + +impl TryFrom<WireChangeOptions> for MetadataOptions { + type Error = IggyError; + + fn try_from(options: WireChangeOptions) -> Result<Self, Self::Error> { + match options { + WireChangeOptions::Readable { explicit, derived } => { + resource_options_from_wire_split(&explicit, &derived).map(Self::Readable) + } + WireChangeOptions::Unreadable => Ok(Self::Unreadable), + } + } +} + +impl TryFrom<WireStreamBody> for MetadataStreamChange { + type Error = IggyError; + + fn try_from(body: WireStreamBody) -> Result<Self, Self::Error> { + Ok(Self { + stream_id: body.key.stream_id, + stream_created_op: body.key.stream_created_op, + created_at: body.created_at, + name: body.name.as_str().to_owned(), + options: body.options.try_into()?, + }) + } +} + +impl From<WireStreamDeletedBody> for MetadataStreamDeleted { + fn from(body: WireStreamDeletedBody) -> Self { + Self { + stream_id: body.key.stream_id, + stream_created_op: body.key.stream_created_op, + created_at: body.created_at, + name: body.name.as_str().to_owned(), + requested_op: body.requested_op, + } + } +} + +impl TryFrom<WireTopicBody> for MetadataTopicChange { + type Error = IggyError; + + fn try_from(body: WireTopicBody) -> Result<Self, Self::Error> { + Ok(Self { + stream_id: body.key.stream.stream_id, + stream_created_op: body.key.stream.stream_created_op, + topic_id: body.key.topic_id, + topic_created_op: body.key.topic_created_op, + created_at: body.created_at, + name: body.name.as_str().to_owned(), + partitions_count: body.partitions_count, + options: body.options.try_into()?, + }) + } +} + +impl TryFrom<WireTopicItemBody> for MetadataTopicItem { + type Error = IggyError; + + fn try_from(body: WireTopicItemBody) -> Result<Self, Self::Error> { + Ok(Self { + stream_id: body.key.stream.stream_id, + stream_created_op: body.key.stream.stream_created_op, + topic_id: body.key.topic_id, + topic_created_op: body.key.topic_created_op, + created_at: body.created_at, + name: body.name.as_str().to_owned(), + options: body.options.try_into()?, + }) + } +} + +impl From<WireTopicDeletedBody> for MetadataTopicDeleted { + fn from(body: WireTopicDeletedBody) -> Self { + Self { + stream_id: body.key.stream.stream_id, + stream_created_op: body.key.stream.stream_created_op, + topic_id: body.key.topic_id, + topic_created_op: body.key.topic_created_op, + created_at: body.created_at, + name: body.name.as_str().to_owned(), + requested_op: body.requested_op, + } + } +} + +impl From<WirePartitionsChangedBody> for MetadataPartitionsChanged { + fn from(body: WirePartitionsChangedBody) -> Self { + Self { + stream_id: body.topic.stream.stream_id, + stream_created_op: body.topic.stream.stream_created_op, + topic_id: body.topic.topic_id, + topic_created_op: body.topic.topic_created_op, + partitions_count: body.partitions_count, + removed_partition_ids: body.removed_partition_ids, + requested_op: body.requested_op, + } + } +} + +impl From<WirePartitionsItemBody> for MetadataPartitionsItem { + fn from(body: WirePartitionsItemBody) -> Self { + Self { + stream_id: body.topic.stream.stream_id, + stream_created_op: body.topic.stream.stream_created_op, + topic_id: body.topic.topic_id, + topic_created_op: body.topic.topic_created_op, + partitions_count: body.partitions_count, + } + } +} + +impl From<WireConsumerGroupCreatedBody> for MetadataConsumerGroupCreated { + fn from(body: WireConsumerGroupCreatedBody) -> Self { + Self { + stream_id: body.topic.stream.stream_id, + stream_created_op: body.topic.stream.stream_created_op, + topic_id: body.topic.topic_id, + topic_created_op: body.topic.topic_created_op, + group_id: body.group_id, + name: body.name.as_str().to_owned(), + consumer_groups_count: body.consumer_groups_count, + } + } +} + +impl From<WireConsumerGroupDeletedBody> for MetadataConsumerGroupDeleted { + fn from(body: WireConsumerGroupDeletedBody) -> Self { + Self { + stream_id: body.topic.stream.stream_id, + stream_created_op: body.topic.stream.stream_created_op, + topic_id: body.topic.topic_id, + topic_created_op: body.topic.topic_created_op, + group_id: body.group_id, + name: body.name.as_str().to_owned(), + consumer_groups_count: body.consumer_groups_count, + requested_op: body.requested_op, + } + } +} + +impl From<WireConsumerGroupItemBody> for MetadataConsumerGroupItem { + fn from(body: WireConsumerGroupItemBody) -> Self { + Self { + stream_id: body.topic.stream.stream_id, + stream_created_op: body.topic.stream.stream_created_op, + topic_id: body.topic.topic_id, + topic_created_op: body.topic.topic_created_op, + group_id: body.group_id, + name: body.name.as_str().to_owned(), + } + } +} + +impl From<WireMemberJoinedBody> for MetadataMemberJoined { + fn from(body: WireMemberJoinedBody) -> Self { + Self { + stream_id: body.topic.stream.stream_id, + stream_created_op: body.topic.stream.stream_created_op, + topic_id: body.topic.topic_id, + topic_created_op: body.topic.topic_created_op, + group_id: body.group_id, + member_id: body.member_id, + client_id: body.client_id, + members_count: body.members_count, + partition_ids: body.partition_ids, + } + } +} + +impl From<WireMemberLeftBody> for MetadataMemberLeft { + fn from(body: WireMemberLeftBody) -> Self { + Self { + stream_id: body.topic.stream.stream_id, + stream_created_op: body.topic.stream.stream_created_op, + topic_id: body.topic.topic_id, + topic_created_op: body.topic.topic_created_op, + group_id: body.group_id, + member_id: body.member_id, + client_id: body.client_id, + members_count: body.members_count, + reason: body.reason.into(), + } + } +} + +impl From<WirePartitionHandedOverBody> for MetadataPartitionHandedOver { + fn from(body: WirePartitionHandedOverBody) -> Self { + Self { + stream_id: body.topic.stream.stream_id, + stream_created_op: body.topic.stream.stream_created_op, + topic_id: body.topic.topic_id, + topic_created_op: body.topic.topic_created_op, + group_id: body.group_id, + partition_id: body.partition_id, + member_id: body.member_id, + } + } +} + +impl From<WireMemberItemBody> for MetadataMemberItem { + fn from(body: WireMemberItemBody) -> Self { + Self { + stream_id: body.topic.stream.stream_id, + stream_created_op: body.topic.stream.stream_created_op, + topic_id: body.topic.topic_id, + topic_created_op: body.topic.topic_created_op, + group_id: body.group_id, + member_id: body.member_id, + client_id: body.client_id, + partition_ids: body.partition_ids, + } + } +} + +impl TryFrom<WireUserBody> for MetadataUserChange { + type Error = IggyError; + + fn try_from(body: WireUserBody) -> Result<Self, Self::Error> { + Ok(Self { + user_id: body.key.user_id, + user_created_at: body.key.user_created_at, + username: body.username.as_str().to_owned(), + status: UserStatus::from_code(body.status)?, + permissions: body.permissions.map(Permissions::from), + options: body.options.try_into()?, + }) + } +} + +impl From<WireUserIdentityBody> for MetadataUserIdentity { + fn from(body: WireUserIdentityBody) -> Self { + Self { + user_id: body.key.user_id, + user_created_at: body.key.user_created_at, + username: body.username.as_str().to_owned(), + } + } +} + +impl From<WireUserPermissionsBody> for MetadataUserPermissionsUpdated { + fn from(body: WireUserPermissionsBody) -> Self { + Self { + user_id: body.key.user_id, + user_created_at: body.key.user_created_at, + username: body.username.as_str().to_owned(), + permissions: body.permissions.map(Permissions::from), + reason: body.reason.into(), + } + } +} + +impl From<WireTokenBody> for MetadataTokenChange { + fn from(body: WireTokenBody) -> Self { + Self { + user_id: body.user.user_id, + user_created_at: body.user.user_created_at, + name: body.name.as_str().to_owned(), + expiry: (body.expiry != 0).then_some(body.expiry), + } + } +} + +impl From<WireTokenDeletedBody> for MetadataTokenDeleted { + fn from(body: WireTokenDeletedBody) -> Self { + Self { + user_id: body.user.user_id, + user_created_at: body.user.user_created_at, + name: body.name.as_str().to_owned(), + reason: body.reason.into(), + } + } +} + +impl From<WireSessionBody> for MetadataSession { + fn from(body: WireSessionBody) -> Self { + Self { + user_id: body.user.user_id, + user_created_at: body.user.user_created_at, + client_id: body.client_id, + session: body.session, + } + } +} + +impl From<WireSessionClosedBody> for MetadataSessionClosed { + fn from(body: WireSessionClosedBody) -> Self { + Self { + user_id: body.user.user_id, + user_created_at: body.user.user_created_at, + client_id: body.client_id, + session: body.session, + reason: body.reason.into(), + } + } +} + +impl TryFrom<WireMetadataChange> for MetadataChange { + type Error = IggyError; + + fn try_from(change: WireMetadataChange) -> Result<Self, Self::Error> { + Ok(match change { + WireMetadataChange::StreamCreated(body) => Self::StreamCreated(body.try_into()?), + WireMetadataChange::StreamUpdated(body) => Self::StreamUpdated(body.try_into()?), + WireMetadataChange::StreamDeleted(body) => Self::StreamDeleted(body.into()), + WireMetadataChange::TopicCreated(body) => Self::TopicCreated(body.try_into()?), + WireMetadataChange::TopicUpdated(body) => Self::TopicUpdated(body.try_into()?), + WireMetadataChange::TopicDeleted(body) => Self::TopicDeleted(body.into()), + WireMetadataChange::PartitionsChanged(body) => Self::PartitionsChanged(body.into()), + WireMetadataChange::ConsumerGroupCreated(body) => { + Self::ConsumerGroupCreated(body.into()) + } + WireMetadataChange::ConsumerGroupDeleted(body) => { + Self::ConsumerGroupDeleted(body.into()) + } + WireMetadataChange::ConsumerGroupMemberJoined(body) => { + Self::ConsumerGroupMemberJoined(body.into()) + } + WireMetadataChange::ConsumerGroupMemberLeft(body) => { + Self::ConsumerGroupMemberLeft(body.into()) + } + WireMetadataChange::ConsumerGroupPartitionHandedOver(body) => { + Self::ConsumerGroupPartitionHandedOver(body.into()) + } + WireMetadataChange::UserCreated(body) => Self::UserCreated(body.try_into()?), + WireMetadataChange::UserUpdated(body) => Self::UserUpdated(body.try_into()?), + WireMetadataChange::UserDeleted(body) => Self::UserDeleted(body.into()), + WireMetadataChange::UserPasswordChanged(body) => Self::UserPasswordChanged(body.into()), + WireMetadataChange::UserPermissionsUpdated(body) => { + Self::UserPermissionsUpdated(body.into()) + } + WireMetadataChange::PersonalAccessTokenCreated(body) => { + Self::PersonalAccessTokenCreated(body.into()) + } + WireMetadataChange::PersonalAccessTokenDeleted(body) => { + Self::PersonalAccessTokenDeleted(body.into()) + } + WireMetadataChange::SessionRegistered(body) => Self::SessionRegistered(body.into()), + WireMetadataChange::SessionClosed(body) => Self::SessionClosed(body.into()), + }) + } +} + +impl TryFrom<WireMetadataCurrentItem> for MetadataCurrentItem { + type Error = IggyError; + + fn try_from(item: WireMetadataCurrentItem) -> Result<Self, Self::Error> { + Ok(match item { + WireMetadataCurrentItem::Stream(body) => Self::Stream(body.try_into()?), + WireMetadataCurrentItem::Topic(body) => Self::Topic(body.try_into()?), + WireMetadataCurrentItem::Partitions(body) => Self::Partitions(body.into()), + WireMetadataCurrentItem::ConsumerGroup(body) => Self::ConsumerGroup(body.into()), + WireMetadataCurrentItem::ConsumerGroupMember(body) => { + Self::ConsumerGroupMember(body.into()) + } + WireMetadataCurrentItem::User(body) => Self::User(body.try_into()?), + WireMetadataCurrentItem::PersonalAccessToken(body) => { + Self::PersonalAccessToken(body.into()) + } + WireMetadataCurrentItem::Session(body) => Self::Session(body.into()), + }) + } +} + +impl TryFrom<WireMetadataChangeRecord> for MetadataChangeRecord { + type Error = IggyError; + + fn try_from(record: WireMetadataChangeRecord) -> Result<Self, Self::Error> { + let change = match WireMetadataChange::decode(record.group, record.kind, &record.body) + .map_err(|_| IggyError::InvalidFormat)? + { + Some(change) => change.try_into()?, + None => MetadataChange::Unknown { + group: record.group, + kind: record.kind, + body: record.body, + }, + }; + Ok(Self { + op: record.op, + ordinal: record.ordinal, + checksum: record.checksum, + timestamp: record.timestamp, + actor_user_id: record.actor_user_id, + change, + }) + } +} + +impl TryFrom<WireMetadataItem> for MetadataCurrentItem { + type Error = IggyError; + + fn try_from(item: WireMetadataItem) -> Result<Self, Self::Error> { + match WireMetadataCurrentItem::decode(item.group, &item.body) + .map_err(|_| IggyError::InvalidFormat)? + { + Some(decoded) => decoded.try_into(), + None => Ok(Self::Unknown { + group: item.group, + body: item.body, + }), + } + } +} + +impl TryFrom<GetMetadataChangesResponse> for MetadataChanges { + type Error = IggyError; + + fn try_from(response: GetMetadataChangesResponse) -> Result<Self, Self::Error> { + Ok(Self { + status: response.status.into(), + phase: response.phase.into(), + log_id: response.log_id, + oldest_op: response.oldest_op, + retained_since: response.retained_since, + head_op: response.head_op, + next: response.next.map(Into::into), + has_more: response.has_more, + records: response + .records + .into_iter() + .map(MetadataChangeRecord::try_from) + .collect::<Result<_, _>>()?, + items: response + .items + .into_iter() + .map(MetadataCurrentItem::try_from) + .collect::<Result<_, _>>()?, + current_page_token: response.current_page_token.map(Into::into), + }) + } +} + +impl From<&MetadataChangesQuery> for GetMetadataChangesRequest { + fn from(query: &MetadataChangesQuery) -> Self { + let start = match &query.start { + MetadataChangeStart::Earliest => WireMetadataChangeStart::Earliest, + MetadataChangeStart::Latest => WireMetadataChangeStart::Latest, + MetadataChangeStart::After(cursor) => WireMetadataChangeStart::After((*cursor).into()), + MetadataChangeStart::Current(token) => { + WireMetadataChangeStart::Current(token.clone().map(Into::into)) + } + }; + let mut groups: Vec<u8> = query + .groups + .iter() + .map(|group| WireMetadataChangeGroup::from(*group).as_u8()) + .collect(); + if groups.len() > usize::from(u8::MAX) { + let mut seen = [false; 256]; + groups.retain(|group| !std::mem::replace(&mut seen[usize::from(*group)], true)); + } + Self { + start, + since: query.since.map(|since| since.as_micros()), + until: query.until.map(|until| until.as_micros()), + limit: query.limit, + groups, + } + } +} + +impl TryFrom<GetMetadataChangesRequest> for MetadataChangesQuery { + type Error = IggyError; + + fn try_from(request: GetMetadataChangesRequest) -> Result<Self, Self::Error> { + let timestamp = + |micros| IggyTimestamp::checked_from_micros(micros).ok_or(IggyError::InvalidCommand); + let start = match request.start { + WireMetadataChangeStart::Earliest => MetadataChangeStart::Earliest, + WireMetadataChangeStart::Latest => MetadataChangeStart::Latest, + WireMetadataChangeStart::After(cursor) => MetadataChangeStart::After(cursor.into()), + WireMetadataChangeStart::Current(token) => { + MetadataChangeStart::Current(token.map(Into::into)) + } + }; + let groups = request + .groups + .iter() + .map(|&code| { + WireMetadataChangeGroup::from_code(code) + .map(MetadataChangeGroup::from) + .map_err(|_| IggyError::InvalidCommand) + }) + .collect::<Result<_, _>>()?; + Ok(Self { + start, + groups, + since: request.since.map(timestamp).transpose()?, + until: request.until.map(timestamp).transpose()?, + limit: request.limit, + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::{MetadataChangeCursor, MetadataChangeStatus}; + use iggy_binary_protocol::primitives::metadata_changes::{ + WireMetadataChangePhase, WireMetadataChangeStatus, WireStreamKey, WireTopicKey, WireUserKey, + }; + use iggy_binary_protocol::{WireDecode, WireName}; + use iggy_binary_protocol::{WireEncode, WireOptions}; + + #[test] + fn given_many_duplicate_groups_when_encoding_should_preserve_every_requested_group() { + let mut groups = vec![crate::MetadataChangeGroup::Streams; 255]; + groups.push(crate::MetadataChangeGroup::Users); + let query = MetadataChangesQuery { + groups, + ..MetadataChangesQuery::default() + }; + let bytes = GetMetadataChangesRequest::from(&query).to_bytes(); + let decoded = + MetadataChangesQuery::try_from(GetMetadataChangesRequest::decode_from(&bytes).unwrap()) + .unwrap(); + + assert_eq!(decoded.validate().unwrap(), query.validate().unwrap()); + } + + fn topic_key() -> WireTopicKey { + WireTopicKey { + stream: WireStreamKey { + stream_id: 3, + stream_created_op: 40, + }, + topic_id: 1, + topic_created_op: 41, + } + } + + fn record(change: &WireMetadataChange) -> WireMetadataChangeRecord { + WireMetadataChangeRecord { + op: 41, + ordinal: 0, + checksum: 99, + timestamp: 7, + actor_user_id: Some(4), + group: change.group().as_u8(), + kind: change.kind().as_u8(), + body: change.body_bytes(), + } + } + + #[test] + fn given_topic_created_record_when_converting_from_wire_should_flatten_the_identity() { + let change = WireMetadataChange::TopicCreated(WireTopicBody { + key: topic_key(), + created_at: 5, + name: WireName::new("events").unwrap(), + partitions_count: 8, + options: WireChangeOptions::Readable { + explicit: WireOptions::empty(), + derived: WireOptions::empty(), + }, + }); + let converted = MetadataChangeRecord::try_from(record(&change)).unwrap(); + let MetadataChange::TopicCreated(body) = converted.change else { + panic!("expected a topic record"); + }; + assert_eq!( + ( + body.stream_id, + body.stream_created_op, + body.topic_id, + body.topic_created_op + ), + (3, 40, 1, 41) + ); + assert_eq!(body.partitions_count, 8); + assert_eq!(converted.actor_user_id, Some(4)); + } + + #[test] + fn given_unknown_kind_when_converting_from_wire_should_keep_the_raw_body() { + let wire = WireMetadataChangeRecord { + group: 2, + kind: 19, + body: bytes::Bytes::from_static(&[1, 2, 3]), + ..record(&WireMetadataChange::SessionRegistered(WireSessionBody { + user: WireUserKey { + user_id: 1, + user_created_at: 2, + }, + client_id: 3, + session: 4, + })) + }; + let converted = MetadataChangeRecord::try_from(wire).unwrap(); + assert_eq!( + converted.change, + MetadataChange::Unknown { + group: 2, + kind: 19, + body: bytes::Bytes::from_static(&[1, 2, 3]) + } + ); + } + + #[test] + fn given_malformed_known_body_when_converting_from_wire_should_refuse() { + let mut wire = record(&WireMetadataChange::SessionRegistered(WireSessionBody { + user: WireUserKey { + user_id: 1, + user_created_at: 2, + }, + client_id: 3, + session: 4, + })); + wire.body = wire.body.slice(..wire.body.len() - 1); + assert_eq!( + MetadataChangeRecord::try_from(wire), + Err(IggyError::InvalidFormat) + ); + } + + #[test] + fn given_token_without_expiry_when_converting_from_wire_should_report_no_expiry() { + let token = MetadataTokenChange::from(WireTokenBody { + user: WireUserKey { + user_id: 1, + user_created_at: 2, + }, + name: WireName::new("ci").unwrap(), + expiry: 0, + }); + assert_eq!(token.expiry, None); + } + + #[test] + fn given_query_when_round_tripping_through_the_wire_request_should_preserve_it() { + let query = MetadataChangesQuery { + start: MetadataChangeStart::After(MetadataChangeCursor { + log_id: 1, + op: 2, + checksum: 3, + ordinal: 4, + }), + groups: vec![MetadataChangeGroup::Users, MetadataChangeGroup::Sessions], + since: Some(IggyTimestamp::from(10)), + until: Some(IggyTimestamp::from(20)), + limit: 50, + }; + let request = GetMetadataChangesRequest::from(&query); + assert_eq!(request.groups, vec![6, 8]); + let decoded = GetMetadataChangesRequest::decode_from(&request.to_bytes()).unwrap(); + assert_eq!(MetadataChangesQuery::try_from(decoded).unwrap(), query); + } + + #[test] + fn given_unserved_group_code_when_converting_request_should_refuse() { + let request = GetMetadataChangesRequest { + start: WireMetadataChangeStart::Earliest, + since: None, + until: None, + limit: 10, + groups: vec![1, 200], + }; + assert_eq!( + MetadataChangesQuery::try_from(request), + Err(IggyError::InvalidCommand) + ); + } + + #[test] + fn given_wire_response_when_converting_should_map_status_phase_and_positions() { + let response = GetMetadataChangesResponse { + status: WireMetadataChangeStatus::Expired, + phase: WireMetadataChangePhase::Changes, + log_id: Some(5), + oldest_op: 9, + retained_since: 3, + head_op: 12, + next: Some(MetadataChangeCursor::zero(5).into()), + has_more: false, + records: vec![], + items: vec![], + current_page_token: None, + }; + let answer = MetadataChanges::try_from(response).unwrap(); + assert_eq!(answer.status, MetadataChangeStatus::Expired); + assert_eq!(answer.next, Some(MetadataChangeCursor::zero(5))); + assert_eq!(answer.head_op, 12); + } +}
diff --git a/core/common/src/types/mod.rs b/core/common/src/types/mod.rs index feb5b4b..9d4dfd3 100644 --- a/core/common/src/types/mod.rs +++ b/core/common/src/types/mod.rs
@@ -27,6 +27,7 @@ pub(crate) mod http; pub(crate) mod identifier; pub(crate) mod message; +pub(crate) mod metadata_changes; pub(crate) mod options; pub(crate) mod partition; pub(crate) mod permissions;
diff --git a/core/common/src/utils/timestamp.rs b/core/common/src/utils/timestamp.rs index b96f719..e56e2a2 100644 --- a/core/common/src/utils/timestamp.rs +++ b/core/common/src/utils/timestamp.rs
@@ -55,6 +55,14 @@ IggyTimestamp(UNIX_EPOCH) } + /// `micros` after the Unix epoch, `None` past what this platform's clock + /// holds, for example above year 30828 on Windows. + pub fn checked_from_micros(micros: u64) -> Option<Self> { + UNIX_EPOCH + .checked_add(Duration::from_micros(micros)) + .map(IggyTimestamp) + } + pub fn to_secs(&self) -> u64 { self.0.duration_since(UNIX_EPOCH).unwrap().as_secs() }
diff --git a/core/common/tests/metadata_changes_fixtures.rs b/core/common/tests/metadata_changes_fixtures.rs new file mode 100644 index 0000000..7f29b4f --- /dev/null +++ b/core/common/tests/metadata_changes_fixtures.rs
@@ -0,0 +1,267 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#[path = "../examples/metadata_changes_fixtures/schema.rs"] +mod schema; + +use iggy_binary_protocol::WireDecode; +use iggy_binary_protocol::primitives::metadata_changes::{ + WireMetadataChange, WireMetadataChangeCursor, WireMetadataCurrentPageToken, +}; +use iggy_binary_protocol::requests::system::GetMetadataChangesRequest; +use iggy_binary_protocol::responses::system::{ + GetMetadataChangesResponse, WireMetadataChangeRecord, WireMetadataItem, +}; +use iggy_common::{ + MetadataChange, MetadataChangeCursor, MetadataChangeRecord, MetadataChanges, + MetadataChangesQuery, MetadataCurrentItem, MetadataCurrentPageToken, +}; +use schema::{CANONICAL_PATH, LANGUAGE_COPIES, build, render}; +use std::fs; +use std::path::Path; +use std::str::FromStr; + +#[test] +fn given_live_codecs_when_rendering_fixture_should_match_the_canonical_file() { + let rendered = render().expect("fixture builds"); + assert!( + read_manifest_file(CANONICAL_PATH) == rendered.as_bytes(), + "{CANONICAL_PATH} is stale, rerun: cargo run -p iggy_common --example metadata_changes_fixtures" + ); +} + +#[test] +fn given_language_copies_when_comparing_should_match_canonical_bytes() { + let canonical = read_manifest_file(CANONICAL_PATH); + for copy in LANGUAGE_COPIES { + assert!( + read_manifest_file(copy) == canonical, + "{copy} differs from {CANONICAL_PATH}, rerun the metadata_changes_fixtures example" + ); + } +} + +#[test] +fn given_record_and_item_vectors_when_decoding_should_reach_the_expected_json() { + let fixture = build().expect("fixture builds"); + for vector in &fixture.records { + let bytes = hex::decode(&vector.record_hex).expect("hex"); + let wire = WireMetadataChangeRecord::decode_from(&bytes).expect("record decodes"); + assert_eq!(hex::encode(&wire.body), vector.body_hex, "{}", vector.case); + let record = MetadataChangeRecord::try_from(wire).expect("record converts"); + assert_eq!(record, vector.expected, "{}", vector.case); + let json = serde_json::to_string(&record).expect("record renders"); + let parsed: MetadataChangeRecord = serde_json::from_str(&json).expect("record parses"); + assert_eq!(parsed, vector.expected, "{}", vector.case); + } + for vector in &fixture.items { + let bytes = hex::decode(&vector.item_hex).expect("hex"); + let wire = WireMetadataItem::decode_from(&bytes).expect("item decodes"); + let item = MetadataCurrentItem::try_from(wire).expect("item converts"); + assert_eq!(item, vector.expected, "{}", vector.case); + } +} + +#[test] +fn given_every_kind_when_counting_record_vectors_should_cover_the_whole_catalog() { + let fixture = build().expect("fixture builds"); + let kinds: std::collections::BTreeSet<u8> = + fixture.records.iter().map(|vector| vector.kind).collect(); + let catalog: std::collections::BTreeSet<u8> = fixture + .constants + .kinds + .values() + .map(|kind| kind.code) + .collect(); + assert_eq!(kinds, catalog); + let groups: std::collections::BTreeSet<u8> = + fixture.items.iter().map(|vector| vector.group).collect(); + assert_eq!(groups.len(), fixture.constants.groups.len()); +} + +#[test] +fn given_cursor_and_token_vectors_when_parsing_text_should_match_their_bytes() { + let fixture = build().expect("fixture builds"); + for vector in &fixture.cursors { + let cursor = MetadataChangeCursor::from_str(&vector.text).expect("cursor parses"); + let bytes = hex::decode(&vector.hex).expect("hex"); + let (wire, _) = WireMetadataChangeCursor::decode(&bytes).expect("cursor decodes"); + assert_eq!(MetadataChangeCursor::from(wire), cursor, "{}", vector.case); + assert_eq!(cursor.op, vector.op, "{}", vector.case); + } + for vector in &fixture.page_tokens { + let token = MetadataCurrentPageToken::from_str(&vector.text).expect("token parses"); + let bytes = hex::decode(&vector.hex).expect("hex"); + let (wire, _) = WireMetadataCurrentPageToken::decode(&bytes).expect("token decodes"); + assert_eq!( + MetadataCurrentPageToken::from(wire), + token, + "{}", + vector.case + ); + } +} + +#[test] +fn given_request_and_response_vectors_when_decoding_should_match_inputs_and_answers() { + let fixture = build().expect("fixture builds"); + for vector in &fixture.requests { + let bytes = hex::decode(&vector.request_hex).expect("hex"); + let request = GetMetadataChangesRequest::decode_from(&bytes).expect("request decodes"); + let mut decoded = MetadataChangesQuery::try_from(request).expect("request converts"); + decoded.groups = decoded.validate().expect("valid request").iter().collect(); + let mut expected = vector.input.query().expect("input builds"); + expected.groups = expected.validate().expect("valid input").iter().collect(); + assert_eq!(decoded, expected, "{}", vector.case); + let mut from_http = MetadataChangesQuery::from_http_query( + vector + .http_query + .split('&') + .map(|pair| pair.split_once('=').expect("key and value")), + ) + .expect("http query parses"); + from_http.groups = from_http + .validate() + .expect("valid HTTP query") + .iter() + .collect(); + assert_eq!(from_http, decoded, "{}", vector.case); + } + for vector in &fixture.responses { + let bytes = hex::decode(&vector.response_hex).expect("hex"); + let response = GetMetadataChangesResponse::decode_from(&bytes).expect("response decodes"); + let answer = MetadataChanges::try_from(response).expect("response converts"); + assert_eq!(answer, vector.expected, "{}", vector.case); + let json = serde_json::to_string(&answer).expect("answer renders"); + let parsed: MetadataChanges = serde_json::from_str(&json).expect("answer parses"); + let known = answer + .records + .iter() + .all(|record| record.change.kind().is_some()); + if known { + assert_eq!(parsed, vector.expected, "{}", vector.case); + } else { + assert!( + parsed + .records + .iter() + .all(|record| record.change.kind().is_none()) + ); + } + } + let unknown = fixture + .responses + .iter() + .find(|vector| vector.case == "unknown_kind_and_group") + .expect("unknown vector"); + assert!( + unknown + .expected + .records + .iter() + .all(|record| matches!(record.change, MetadataChange::Unknown { .. })) + ); +} + +#[test] +fn given_large_values_when_decoding_should_keep_every_bit() { + let fixture = build().expect("fixture builds"); + let vector = &fixture.large_values; + let bytes = hex::decode(&vector.record_hex).expect("hex"); + let wire = WireMetadataChangeRecord::decode_from(&bytes).expect("record decodes"); + let record = MetadataChangeRecord::try_from(wire).expect("record converts"); + assert_eq!(record.op.to_string(), vector.op); + assert_eq!(record.timestamp.to_string(), vector.timestamp); + assert_eq!(format!("{:032x}", record.checksum), vector.checksum); + let MetadataChange::SessionRegistered(session) = record.change else { + panic!("expected a session record"); + }; + assert_eq!(session.user_created_at.to_string(), vector.user_created_at); + assert_eq!(session.session.to_string(), vector.session); + assert_eq!(format!("{:032x}", session.client_id), vector.client_id); +} + +#[test] +fn given_refused_vectors_when_decoding_or_validating_should_refuse_each() { + let fixture = build().expect("fixture builds"); + let refused = &fixture.refused; + for vector in &refused.cursors { + assert!( + MetadataChangeCursor::from_str(&vector.text).is_err(), + "{}", + vector.case + ); + } + for vector in &refused.page_tokens { + assert!( + MetadataCurrentPageToken::from_str(&vector.text).is_err(), + "{}", + vector.case + ); + } + for vector in &refused.record_bodies { + let body = hex::decode(&vector.body_hex).expect("hex"); + assert!( + WireMetadataChange::decode(vector.group, vector.kind, &body).is_err(), + "{}", + vector.case + ); + } + for vector in &refused.requests { + let bytes = hex::decode(&vector.hex).expect("hex"); + assert!( + GetMetadataChangesRequest::decode_from(&bytes).is_err(), + "{}", + vector.case + ); + } + for vector in &refused.responses { + let bytes = hex::decode(&vector.hex).expect("hex"); + assert!( + GetMetadataChangesResponse::decode_from(&bytes).is_err(), + "{}", + vector.case + ); + } + for vector in &refused.queries { + let refused_query = vector + .input + .query() + .and_then(|query| query.validate().map(|_| query)); + assert!(refused_query.is_err(), "{}", vector.case); + } + for vector in &refused.http_queries { + let parsed = MetadataChangesQuery::from_http_query( + vector + .query + .split('&') + .map(|pair| pair.split_once('=').unwrap_or((pair, ""))), + ) + .and_then(|query| query.validate().map(|_| query)); + assert!(parsed.is_err(), "{}", vector.case); + } +} + +fn read_manifest_file(relative: &str) -> Vec<u8> { + let path = Path::new(env!("CARGO_MANIFEST_DIR")).join(relative); + fs::read(&path).unwrap_or_else(|error| { + panic!( + "cannot read {}: {error}. Generate it with: cargo run -p iggy_common --example metadata_changes_fixtures", + path.display() + ) + }) +}
diff --git a/core/configs/src/server_config/defaults.rs b/core/configs/src/server_config/defaults.rs index de4f1cd..35b9dc8 100644 --- a/core/configs/src/server_config/defaults.rs +++ b/core/configs/src/server_config/defaults.rs
@@ -175,6 +175,8 @@ journal_slots: metadata.journal_slots as usize, clients_table_max: metadata.clients_table_max as usize, partitions_max: metadata.partitions_max as u32, + change_log_resource_capacity: metadata.change_log_resource_capacity as usize, + change_log_activity_capacity: metadata.change_log_activity_capacity as usize, } } }
diff --git a/core/configs/src/server_config/displays.rs b/core/configs/src/server_config/displays.rs index 413b21c..04c03e5 100644 --- a/core/configs/src/server_config/displays.rs +++ b/core/configs/src/server_config/displays.rs
@@ -86,11 +86,13 @@ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { write!( f, - "{{ prepare_queue_depth: {}, journal_slots: {}, clients_table_max: {}, partitions_max: {} }}", + "{{ prepare_queue_depth: {}, journal_slots: {}, clients_table_max: {}, partitions_max: {}, change_log_resource_capacity: {}, change_log_activity_capacity: {} }}", self.prepare_queue_depth, self.journal_slots, self.clients_table_max, self.partitions_max, + self.change_log_resource_capacity, + self.change_log_activity_capacity, ) } }
diff --git a/core/configs/src/server_config/metadata.rs b/core/configs/src/server_config/metadata.rs index 651e4fd..c77b99f 100644 --- a/core/configs/src/server_config/metadata.rs +++ b/core/configs/src/server_config/metadata.rs
@@ -18,7 +18,7 @@ //! On-disk schema for the metadata consensus plane (shard 0's VSR //! replica: users, streams, topics, sessions). //! -//! Four capacity knobs. The first three were hardcoded in the runtime +//! Six capacity knobs. The first three were hardcoded in the runtime //! crates before: //! //! - `prepare_queue_depth` -> `consensus::PIPELINE_PREPARE_QUEUE_MAX` @@ -32,6 +32,8 @@ //! HTTP session cap tracks it at half. //! - `partitions_max` (the partitions of all streams and topics, checked //! by the metadata primary when it admits a create; zero is no cap) +//! - `change_log_resource_capacity` and `change_log_activity_capacity` +//! (records the metadata change log retains per retention class) //! //! The first two interlock through the forced-checkpoint margin //! (`max(64, prepare_queue_depth)` at bootstrap): while a checkpoint @@ -47,6 +49,7 @@ //! literals against the runtime constants with static asserts. use super::COMPONENT; +use super::defaults::SERVER_CONFIG; use crate::ConfigurationError; use configs::ConfigEnv; use iggy_common::Validatable; @@ -92,6 +95,18 @@ /// population and a likely unit typo. pub const MAX_METADATA_CLIENTS_TABLE_MAX: usize = 1 << 16; +/// Mirrors `metadata::change_log::DEFAULT_RESOURCE_CAPACITY`, the records of +/// streams, topics, partitions, consumer groups, users and tokens retained. +pub const DEFAULT_METADATA_CHANGE_LOG_RESOURCE_CAPACITY: usize = 65_536; + +/// Mirrors `metadata::change_log::DEFAULT_ACTIVITY_CAPACITY`, the records of +/// sessions and consumer group membership retained. +pub const DEFAULT_METADATA_CHANGE_LOG_ACTIVITY_CAPACITY: usize = 16_384; + +/// Mirrors `metadata::change_log::MAX_CAPACITY`. A snapshot holding a larger +/// ring is refused, so a capacity past it could never be restored. +pub const MAX_METADATA_CHANGE_LOG_CAPACITY: usize = 1 << 20; + /// Capacity tunables for the metadata consensus plane. #[derive(Debug, Deserialize, Serialize, Clone, ConfigEnv)] pub struct MetadataConfig { @@ -126,6 +141,17 @@ /// the creates still in flight, so creates never take the count past it. #[serde(default)] pub partitions_max: u32, + + /// Change log records kept for streams, topics, partitions, consumer + /// groups, users and personal access tokens. The oldest record goes once + /// the count passes it. Node-local: replicas may keep different depths. + #[serde(default = "default_change_log_resource_capacity")] + pub change_log_resource_capacity: usize, + + /// Change log records kept for sessions and consumer group membership, + /// separate so a login or membership storm cannot evict resource history. + #[serde(default = "default_change_log_activity_capacity")] + pub change_log_activity_capacity: usize, } impl MetadataConfig { @@ -191,13 +217,41 @@ ); return Err(ConfigurationError::InvalidConfigurationValue); } + for (name, capacity) in [ + ( + "change_log_resource_capacity", + self.change_log_resource_capacity, + ), + ( + "change_log_activity_capacity", + self.change_log_activity_capacity, + ), + ] { + if !(1..=MAX_METADATA_CHANGE_LOG_CAPACITY).contains(&capacity) { + eprintln!( + "{COMPONENT} metadata.{name} ({capacity}) must be between 1 and {MAX_METADATA_CHANGE_LOG_CAPACITY}" + ); + return Err(ConfigurationError::InvalidConfigurationValue); + } + } Ok(()) } } +/// Zero fails validation, so the value comes from the embedded `config.toml`. +fn default_change_log_resource_capacity() -> usize { + SERVER_CONFIG.metadata.change_log_resource_capacity as usize +} + +fn default_change_log_activity_capacity() -> usize { + SERVER_CONFIG.metadata.change_log_activity_capacity as usize +} + #[cfg(test)] mod tests { use super::*; + use figment::Figment; + use figment::providers::{Format, Toml}; #[test] fn defaults_are_valid() { @@ -206,6 +260,8 @@ journal_slots: DEFAULT_METADATA_JOURNAL_SLOTS, clients_table_max: DEFAULT_METADATA_CLIENTS_TABLE_MAX, partitions_max: 0, + change_log_resource_capacity: DEFAULT_METADATA_CHANGE_LOG_RESOURCE_CAPACITY, + change_log_activity_capacity: DEFAULT_METADATA_CHANGE_LOG_ACTIVITY_CAPACITY, }; assert!(config.validate().is_ok()); assert_eq!(config.checkpoint_margin(), METADATA_CHECKPOINT_MARGIN_FLOOR); @@ -218,6 +274,8 @@ journal_slots: 4096, clients_table_max: DEFAULT_METADATA_CLIENTS_TABLE_MAX, partitions_max: 0, + change_log_resource_capacity: DEFAULT_METADATA_CHANGE_LOG_RESOURCE_CAPACITY, + change_log_activity_capacity: DEFAULT_METADATA_CHANGE_LOG_ACTIVITY_CAPACITY, }; assert!(config.validate().is_ok()); assert_eq!(config.checkpoint_margin(), MAX_METADATA_PREPARE_QUEUE_DEPTH); @@ -233,6 +291,8 @@ journal_slots: min_slots, clients_table_max: DEFAULT_METADATA_CLIENTS_TABLE_MAX, partitions_max: 0, + change_log_resource_capacity: DEFAULT_METADATA_CHANGE_LOG_RESOURCE_CAPACITY, + change_log_activity_capacity: DEFAULT_METADATA_CHANGE_LOG_ACTIVITY_CAPACITY, }; assert!(boundary.validate().is_ok()); // ...one slot fewer is refused. @@ -241,6 +301,8 @@ journal_slots: min_slots - 1, clients_table_max: DEFAULT_METADATA_CLIENTS_TABLE_MAX, partitions_max: 0, + change_log_resource_capacity: DEFAULT_METADATA_CHANGE_LOG_RESOURCE_CAPACITY, + change_log_activity_capacity: DEFAULT_METADATA_CHANGE_LOG_ACTIVITY_CAPACITY, }; assert!(starved.validate().is_err()); } @@ -255,6 +317,8 @@ journal_slots: MAX_METADATA_JOURNAL_SLOTS, clients_table_max: DEFAULT_METADATA_CLIENTS_TABLE_MAX, partitions_max: 0, + change_log_resource_capacity: DEFAULT_METADATA_CHANGE_LOG_RESOURCE_CAPACITY, + change_log_activity_capacity: DEFAULT_METADATA_CHANGE_LOG_ACTIVITY_CAPACITY, }; assert!(over.validate().is_err()); assert_eq!( @@ -271,6 +335,8 @@ journal_slots: DEFAULT_METADATA_JOURNAL_SLOTS, clients_table_max: DEFAULT_METADATA_CLIENTS_TABLE_MAX, partitions_max: 0, + change_log_resource_capacity: DEFAULT_METADATA_CHANGE_LOG_RESOURCE_CAPACITY, + change_log_activity_capacity: DEFAULT_METADATA_CHANGE_LOG_ACTIVITY_CAPACITY, }; assert!(config.validate().is_err()); } @@ -292,6 +358,8 @@ journal_slots: DEFAULT_METADATA_JOURNAL_SLOTS, clients_table_max: MIN_METADATA_CLIENTS_TABLE_MAX - 1, partitions_max: 0, + change_log_resource_capacity: DEFAULT_METADATA_CHANGE_LOG_RESOURCE_CAPACITY, + change_log_activity_capacity: DEFAULT_METADATA_CHANGE_LOG_ACTIVITY_CAPACITY, }; assert!(config.validate().is_err()); } @@ -303,17 +371,79 @@ journal_slots: DEFAULT_METADATA_JOURNAL_SLOTS, clients_table_max: MIN_METADATA_CLIENTS_TABLE_MAX, partitions_max: 0, + change_log_resource_capacity: DEFAULT_METADATA_CHANGE_LOG_RESOURCE_CAPACITY, + change_log_activity_capacity: DEFAULT_METADATA_CHANGE_LOG_ACTIVITY_CAPACITY, }; assert!(config.validate().is_ok()); } #[test] + fn embedded_defaults_match_canonical_change_log_capacities() { + let config = MetadataConfig::default(); + assert_eq!( + config.change_log_resource_capacity, + DEFAULT_METADATA_CHANGE_LOG_RESOURCE_CAPACITY + ); + assert_eq!( + config.change_log_activity_capacity, + DEFAULT_METADATA_CHANGE_LOG_ACTIVITY_CAPACITY + ); + } + + #[test] + fn change_log_capacities_absent_from_toml_take_the_defaults() { + let config: MetadataConfig = Figment::new() + .merge(Toml::string( + "prepare_queue_depth = 32\njournal_slots = 1024\nclients_table_max = 8192\n", + )) + .extract() + .expect("config without change log capacities deserializes"); + assert_eq!( + config.change_log_resource_capacity, + DEFAULT_METADATA_CHANGE_LOG_RESOURCE_CAPACITY + ); + assert_eq!( + config.change_log_activity_capacity, + DEFAULT_METADATA_CHANGE_LOG_ACTIVITY_CAPACITY + ); + } + + #[test] + fn change_log_capacity_out_of_bounds_is_refused() { + for capacity in [0, MAX_METADATA_CHANGE_LOG_CAPACITY + 1] { + for resource in [true, false] { + let mut config = MetadataConfig::default(); + if resource { + config.change_log_resource_capacity = capacity; + } else { + config.change_log_activity_capacity = capacity; + } + assert!(config.validate().is_err(), "{capacity} must be refused"); + } + } + } + + #[test] + fn change_log_capacity_at_bounds_is_accepted() { + for capacity in [1, MAX_METADATA_CHANGE_LOG_CAPACITY] { + let config = MetadataConfig { + change_log_resource_capacity: capacity, + change_log_activity_capacity: capacity, + ..MetadataConfig::default() + }; + assert!(config.validate().is_ok(), "{capacity} must be accepted"); + } + } + + #[test] fn clients_table_max_above_ceiling_is_refused() { let config = MetadataConfig { prepare_queue_depth: DEFAULT_METADATA_PREPARE_QUEUE_DEPTH, journal_slots: DEFAULT_METADATA_JOURNAL_SLOTS, clients_table_max: MAX_METADATA_CLIENTS_TABLE_MAX + 1, partitions_max: 0, + change_log_resource_capacity: DEFAULT_METADATA_CHANGE_LOG_RESOURCE_CAPACITY, + change_log_activity_capacity: DEFAULT_METADATA_CHANGE_LOG_ACTIVITY_CAPACITY, }; assert!(config.validate().is_err()); }
diff --git a/core/configs/src/server_config/server.rs b/core/configs/src/server_config/server.rs index 56f23a4..8b981e2 100644 --- a/core/configs/src/server_config/server.rs +++ b/core/configs/src/server_config/server.rs
@@ -461,6 +461,25 @@ } #[test] + #[serial_test::serial] + fn given_change_log_capacity_env_var_when_loading_should_override_the_default() { + const NAME: &str = "IGGY_METADATA_CHANGE_LOG_ACTIVITY_CAPACITY"; + assert!(ServerConfig::all_env_var_names().contains(&NAME)); + // SAFETY: `serial_test::serial` keeps other tests off the environment. + unsafe { env::set_var(NAME, "4096") }; + + let cfg: Result<ServerConfig, _> = Figment::new() + .merge(Toml::string(include_str!("../../../server/config.toml"))) + .merge(ServerConfigEnvProvider::default()) + .extract(); + + // SAFETY: paired with the set above. + unsafe { env::remove_var(NAME) }; + let cfg = cfg.expect("config with the env override deserializes"); + assert_eq!(cfg.metadata.change_log_activity_capacity, 4096); + } + + #[test] fn default_impl_validates() { let cfg = ServerConfig::default(); cfg.validate().expect("Default impl validates");
diff --git a/core/connectors/runtime/Cargo.toml b/core/connectors/runtime/Cargo.toml index 2e8644a..c568ef2 100644 --- a/core/connectors/runtime/Cargo.toml +++ b/core/connectors/runtime/Cargo.toml
@@ -17,7 +17,7 @@ [package] name = "iggy-connectors" -version = "0.5.1-edge.2" +version = "0.5.1-edge.3" description = "Connectors runtime for Iggy message streaming platform" edition = "2024" license = "Apache-2.0"
diff --git a/core/consensus/src/client_table.rs b/core/consensus/src/client_table.rs index 7fb462b..38c1311 100644 --- a/core/consensus/src/client_table.rs +++ b/core/consensus/src/client_table.rs
@@ -1527,6 +1527,16 @@ pub fn count(&self) -> usize { self.index.len() } + + /// Sessions not yet ended, as `(client_id, user_id, epoch)` in slot order. + /// Never the bind verifier or a cached reply. + pub fn active_sessions(&self) -> impl Iterator<Item = (u128, u32, u64)> + '_ { + self.slots + .iter() + .flatten() + .filter(|entry| entry.ended_op.is_none()) + .map(|entry| (entry.client_id, entry.user_id, entry.epoch)) + } } /// Failure decoding the state-transfer WIRE encoding of a client table @@ -2034,6 +2044,22 @@ } #[test] + fn given_ended_and_live_sessions_when_listing_active_should_yield_only_live_identities() { + let mut table = ClientTable::new(8); + for (client, user, epoch) in [(1, 11, 10), (2, 22, 20), (3, 11, 30)] { + table + .commit_register(client, user, [0x5a; 32], make_register_reply(client, epoch)) + .unwrap(); + } + table.end_user_sessions(22, 40); + + let mut active: Vec<_> = table.active_sessions().collect(); + active.sort_unstable(); + + assert_eq!(active, [(1, 11, 10), (3, 11, 30)]); + } + + #[test] fn to_from_snapshot_round_trips_epochs_and_watermarks() { let mut table = ClientTable::new(8); table
diff --git a/core/integration/tests/cli/general/test_help_command.rs b/core/integration/tests/cli/general/test_help_command.rs index 110245c..fa68f20 100644 --- a/core/integration/tests/cli/general/test_help_command.rs +++ b/core/integration/tests/cli/general/test_help_command.rs
@@ -41,6 +41,7 @@ options list the option catalog of a resource scope stats get iggy server statistics snapshot collect iggy server troubleshooting data + metadata metadata change log operations [alias: md] pat personal access token operations user user operations [alias: u] client client operations [alias: c]
diff --git a/core/integration/tests/cli/general/test_overview_command.rs b/core/integration/tests/cli/general/test_overview_command.rs index 20c63fc..0538fc4 100644 --- a/core/integration/tests/cli/general/test_overview_command.rs +++ b/core/integration/tests/cli/general/test_overview_command.rs
@@ -52,6 +52,7 @@ options list the option catalog of a resource scope stats get iggy server statistics snapshot collect iggy server troubleshooting data + metadata metadata change log operations [alias: md] pat personal access token operations user user operations [alias: u] client client operations [alias: c]
diff --git a/core/integration/tests/cli/metadata/mod.rs b/core/integration/tests/cli/metadata/mod.rs new file mode 100644 index 0000000..4722985 --- /dev/null +++ b/core/integration/tests/cli/metadata/mod.rs
@@ -0,0 +1,18 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +mod test_metadata_changes_command;
diff --git a/core/integration/tests/cli/metadata/test_metadata_changes_command.rs b/core/integration/tests/cli/metadata/test_metadata_changes_command.rs new file mode 100644 index 0000000..d1ecfc8 --- /dev/null +++ b/core/integration/tests/cli/metadata/test_metadata_changes_command.rs
@@ -0,0 +1,219 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use crate::cli::common::{ + CLAP_INDENT, IggyCmdCommand, IggyCmdTest, IggyCmdTestCase, TestHelpCmd, USAGE_PREFIX, +}; +use assert_cmd::assert::Assert; +use async_trait::async_trait; +use iggy::prelude::{Client, Identifier, StreamCreateOptions}; +use predicates::str::{contains, starts_with}; +use serial_test::parallel; + +const STREAM: &str = "metadata-cli"; + +#[derive(Debug, Clone, Copy, Eq, PartialEq)] +enum Read { + Changes, + CurrentState, + CurrentStateOfChildGroup, +} + +struct TestMetadataChangesCmd { + read: Read, +} + +#[async_trait] +impl IggyCmdTestCase for TestMetadataChangesCmd { + async fn prepare_server_state(&mut self, client: &dyn Client) { + client + .create_stream(STREAM, &StreamCreateOptions::default()) + .await + .expect("stream created"); + } + + fn get_command(&self) -> IggyCmdCommand { + let command = IggyCmdCommand::new() + .arg("metadata") + .arg("changes") + .with_env_credentials(); + match self.read { + Read::Changes => command.arg("--groups").arg("streams"), + Read::CurrentState => command + .arg("--from") + .arg("current") + .arg("--groups") + .arg("streams"), + Read::CurrentStateOfChildGroup => command + .arg("--from") + .arg("current") + .arg("--groups") + .arg("topics"), + } + } + + fn verify_command(&self, command_state: Assert) { + match self.read { + Read::Changes => { + command_state + .success() + .stderr(starts_with("Executing metadata changes command\n")) + .stdout(starts_with(r#"{"status":"ok""#)) + .stdout(contains(r#""kind":"stream_created""#)) + .stdout(contains(format!(r#""name":"{STREAM}""#))); + } + Read::CurrentState => { + command_state + .success() + .stdout(contains(r#""phase":"current""#)) + .stdout(contains(format!(r#""name":"{STREAM}""#))) + .stdout(contains(r#""current_page_token":null"#)); + } + Read::CurrentStateOfChildGroup => { + command_state + .success() + .stdout(contains(r#""phase":"current""#)) + .stdout(contains(format!(r#""name":"{STREAM}""#))); + } + } + } + + async fn verify_server_state(&self, client: &dyn Client) { + client + .delete_stream(&Identifier::named(STREAM).expect("valid name")) + .await + .expect("stream deleted"); + } +} + +#[tokio::test] +#[parallel] +pub async fn should_be_successful() { + for read in [ + Read::Changes, + Read::CurrentState, + Read::CurrentStateOfChildGroup, + ] { + let mut iggy_cmd_test = IggyCmdTest::default(); + iggy_cmd_test.setup().await; + iggy_cmd_test + .execute_test(TestMetadataChangesCmd { read }) + .await; + } +} + +#[tokio::test] +#[parallel] +pub async fn should_help_match() { + let mut iggy_cmd_test = IggyCmdTest::help_message(); + + iggy_cmd_test + .execute_test_for_help_command(TestHelpCmd::new( + vec!["metadata", "changes", "--help"], + format!( + r#"read the metadata change log + +Prints every answered page as one JSON object per line: the change +records or current state items, the status and the cursor to continue +from. Requires root or the global manage_servers permission. + +Examples + iggy metadata changes + iggy metadata changes --from latest + iggy metadata changes --after mc1_... --groups streams,topics + iggy metadata changes --from current --groups streams,topics --follow + +{USAGE_PREFIX} metadata changes [OPTIONS] + +Options: + --from <FROM> + Where to start reading +{CLAP_INDENT} + earliest: the oldest operation the log still holds completely, + latest: no records, only the cursor at the newest record, + current: the current state, page by page, then its catch-up cursor. +{CLAP_INDENT} + [default: earliest] + [possible values: earliest, latest, current] + + --after <AFTER> + Continue after this cursor, the `next` value of an earlier page + + --groups <GROUPS> + Groups to read, comma separated, every group when omitted +{CLAP_INDENT} + streams, topics, partitions, consumer_groups, consumer_group_members, + users, personal_access_tokens, sessions + + --since <SINCE> + Oldest prepare timestamp to include, in microseconds since the epoch + + --until <UNTIL> + Newest prepare timestamp to include, in microseconds since the epoch + + -l, --limit <LIMIT> + Records or items per page, 1 to 1000 +{CLAP_INDENT} + [default: 100] + + -f, --follow + Keep reading new changes once the log has no more +{CLAP_INDENT} + When the cursor's history is no longer retained, the command rebuilds + from the current state and goes on following. + + --interval <INTERVAL> + Pause between pages while following +{CLAP_INDENT} + [default: 1s] + + -h, --help + Print help (see a summary with '-h') +"#, + ), + )) + .await; +} + +#[tokio::test] +#[parallel] +pub async fn should_short_help_match() { + let mut iggy_cmd_test = IggyCmdTest::help_message(); + + iggy_cmd_test + .execute_test_for_help_command(TestHelpCmd::new( + vec!["metadata", "changes", "-h"], + format!( + r#"read the metadata change log + +{USAGE_PREFIX} metadata changes [OPTIONS] + +Options: + --from <FROM> Where to start reading [default: earliest] [possible values: earliest, latest, current] + --after <AFTER> Continue after this cursor, the `next` value of an earlier page + --groups <GROUPS> Groups to read, comma separated, every group when omitted + --since <SINCE> Oldest prepare timestamp to include, in microseconds since the epoch + --until <UNTIL> Newest prepare timestamp to include, in microseconds since the epoch + -l, --limit <LIMIT> Records or items per page, 1 to 1000 [default: 100] + -f, --follow Keep reading new changes once the log has no more + --interval <INTERVAL> Pause between pages while following [default: 1s] + -h, --help Print help (see more with '--help') +"#, + ), + )) + .await; +}
diff --git a/core/integration/tests/cli/mod.rs b/core/integration/tests/cli/mod.rs index 751edfc..df71228 100644 --- a/core/integration/tests/cli/mod.rs +++ b/core/integration/tests/cli/mod.rs
@@ -22,6 +22,7 @@ mod context; mod general; mod message; +mod metadata; mod partition; mod personal_access_token; mod stream;
diff --git a/core/integration/tests/cluster/metadata_changes.rs b/core/integration/tests/cluster/metadata_changes.rs new file mode 100644 index 0000000..07da838 --- /dev/null +++ b/core/integration/tests/cluster/metadata_changes.rs
@@ -0,0 +1,492 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use std::collections::BTreeMap; +use std::time::{Duration, Instant}; + +use futures::future::join_all; +use iggy::prelude::*; +use iggy_common::MAX_METADATA_CHANGES_LIMIT; +use integration::harness::{TestHarness, disk}; +use integration::iggy_harness; +use tokio::time::sleep; + +const STREAM_PREFIX: &str = "cluster-changes"; +const WRITERS: usize = 3; +const STREAMS_PER_WRITER: usize = 5; +const STREAMS: [MetadataChangeGroup; 1] = [MetadataChangeGroup::Streams]; +const SESSIONS: [MetadataChangeGroup; 1] = [MetadataChangeGroup::Sessions]; +// With `journal_slots = 256` and the checkpoint margin, every node +// checkpoints and drains its journal once 192 operations committed. +const STREAMS_PAST_CHECKPOINT: usize = 220; +const CHECKPOINT_MARKER: &str = "forced checkpoint completed"; +const INSTALL_MARKER: &str = "metadata state transfer installed"; +const CONVERGE_TIMEOUT: Duration = Duration::from_secs(60); +const POLL_INTERVAL: Duration = Duration::from_millis(100); + +#[iggy_harness(cluster_nodes = 3)] +async fn given_followers_on_every_node_when_changes_commit_concurrently_should_read_one_history( + harness: &TestHarness, +) { + let readers = node_readers(harness).await; + let (baseline, anchor) = current_streams(&readers[0]).await; + let writers = harness.tcp_root_clients(WRITERS).await.expect("writers"); + join_all( + writers + .iter() + .enumerate() + .map(|(index, writer)| write_streams(writer, index)), + ) + .await; + let expected = listed_streams(&writers[0]).await; + + let mut histories = Vec::with_capacity(readers.len()); + for reader in &readers { + let history = follow(reader, &anchor, &STREAMS, |records| { + mirror(&baseline, records) == expected + }) + .await; + let (fresh, _) = current_streams(reader).await; + assert_eq!(fresh, expected); + histories.push(history); + } + assert!(histories.windows(2).all(|pair| pair[0] == pair[1])); +} + +#[iggy_harness(cluster_nodes = 3)] +async fn given_cursor_and_page_token_when_continued_on_other_nodes_should_resume_without_gaps( + harness: &TestHarness, +) { + let writer = harness.tcp_root_client().await.expect("writer"); + create_streams(&writer, "seed", 9).await; + let readers = node_readers(harness).await; + let (expected, anchor) = current_streams(&readers[0]).await; + + let mut query = MetadataChangesQuery { + start: MetadataChangeStart::Current(None), + groups: STREAMS.to_vec(), + limit: 2, + ..MetadataChangesQuery::default() + }; + let mut items = BTreeMap::new(); + for reader in readers.iter().cycle() { + let page = until_ok(reader, &query).await; + assert_eq!(page.next, Some(anchor)); + items.extend(stream_names(&page.items)); + match page.current_page_token { + Some(token) => query.start = MetadataChangeStart::Current(Some(token)), + None => break, + } + } + assert_eq!(items, expected); + + create_streams(&writer, "continued", 6).await; + let reference = follow(&readers[0], &anchor, &STREAMS, |records| records.len() == 6).await; + let mut cursor = anchor; + let mut records = Vec::new(); + let deadline = Instant::now() + CONVERGE_TIMEOUT; + for reader in readers.iter().cycle() { + let page = until_ok( + reader, + &MetadataChangesQuery { + limit: 1, + ..after(cursor, &STREAMS) + }, + ) + .await; + cursor = page.next.expect("continuation"); + records.extend(page.records); + if records.len() == reference.len() { + break; + } + assert!(Instant::now() < deadline, "{records:?}"); + } + assert_eq!(records, reference); +} + +#[iggy_harness(cluster_nodes = 3)] +async fn given_primary_failover_and_restart_when_following_should_keep_one_history( + harness: &mut TestHarness, +) { + let writer = harness.tcp_root_client().await.expect("writer"); + create_streams(&writer, "before", 3).await; + drop(writer); + let readers = node_readers(harness).await; + let start = latest(&readers[0], &STREAMS).await; + + let leader = disk::leader_node_index(harness).await; + harness + .kill_node(leader) + .expect("kill the metadata primary"); + let survivors: Vec<usize> = (0..harness.cluster_size()) + .filter(|node| *node != leader) + .collect(); + for index in 0..4 { + create_after_failover( + harness, + survivors[0], + &format!("{STREAM_PREFIX}-after-{index}"), + ) + .await; + } + let done = |records: &[MetadataChangeRecord]| { + records + .iter() + .filter(|record| matches!(record.change, MetadataChange::StreamCreated(_))) + .count() + == 4 + }; + let mut histories = Vec::with_capacity(harness.cluster_size()); + for node in &survivors { + histories.push(follow(&readers[*node], &start, &STREAMS, done).await); + } + + harness.start_node(leader).expect("restart the old primary"); + let restarted = node_reader(harness, leader).await; + histories.push(follow(&restarted, &start, &STREAMS, done).await); + assert!(histories.windows(2).all(|pair| pair[0] == pair[1])); + let page = until_ok(&restarted, &after(start, &STREAMS)).await; + assert_eq!(page.log_id, Some(start.log_id)); +} + +#[iggy_harness(cluster_nodes = 3, server(metadata.journal_slots = "256"))] +async fn given_checkpoints_and_state_transfer_when_following_should_serve_records_below_the_drained_journal( + harness: &mut TestHarness, +) { + let reader = node_reader(harness, 0).await; + let start = latest(&reader, &[]).await; + let writer = harness.tcp_root_client().await.expect("writer"); + create_streams(&writer, "checkpoint", STREAMS_PAST_CHECKPOINT).await; + drop(writer); + for node in 0..harness.cluster_size() { + await_marker(harness, node, CHECKPOINT_MARKER).await; + } + let reference = follow(&reader, &start, &[], |records| { + records + .iter() + .filter(|record| matches!(record.change, MetadataChange::StreamCreated(_))) + .count() + == STREAMS_PAST_CHECKPOINT + }) + .await; + + harness + .restart_node(1) + .expect("restart over the drained journal"); + let restarted = node_reader(harness, 1).await; + let replayed = follow(&restarted, &start, &[], |records| { + records.len() >= reference.len() + }) + .await; + assert_eq!(replayed[..reference.len()], reference[..]); + + harness + .restart_node_from_clean_slate(2) + .expect("restart from a clean slate"); + await_marker(harness, 2, INSTALL_MARKER).await; + let transferred = node_reader(harness, 2).await; + let installed = follow(&transferred, &start, &[], |records| { + records.len() >= reference.len() + }) + .await; + assert_eq!(installed[..reference.len()], reference[..]); +} + +#[iggy_harness( + cluster_nodes = 3, + server( + metadata.change_log_resource_capacity = "8", + metadata.change_log_activity_capacity = "4" + ) +)] +async fn given_small_capacities_when_each_class_overflows_should_expire_only_its_readers( + harness: &TestHarness, +) { + let readers = node_readers(harness).await; + let writer = harness.tcp_root_client().await.expect("writer"); + let resource_start = latest(&readers[0], &STREAMS).await; + let activity_start = latest(&readers[0], &SESSIONS).await; + + create_streams(&writer, "overflow", 10).await; + for reader in &readers { + assert_expires(reader, &resource_start, &STREAMS).await; + let activity = until_ok(reader, &after(activity_start, &SESSIONS)).await; + assert!(activity.records.is_empty(), "{activity:?}"); + } + + for _ in 0..3 { + let client = harness.tcp_root_client().await.expect("short session"); + client.logout_user().await.expect("logout"); + } + for reader in &readers { + assert_expires(reader, &activity_start, &SESSIONS).await; + let (rebuilt, _) = current_streams(reader).await; + assert_eq!(rebuilt, listed_streams(&writer).await); + } +} + +async fn node_readers(harness: &TestHarness) -> Vec<IggyClient> { + let mut readers = Vec::with_capacity(harness.cluster_size()); + for node in 0..harness.cluster_size() { + readers.push(node_reader(harness, node).await); + } + readers +} + +/// An HTTP client of one node. HTTP reads answer from the node they reach, +/// so each reader sees that node's own log. +async fn node_reader(harness: &TestHarness, node: usize) -> IggyClient { + let deadline = Instant::now() + CONVERGE_TIMEOUT; + loop { + let builder = harness.node(node).http_client().expect("HTTP endpoint"); + if let Ok(client) = builder.with_root_login().connect().await { + return client; + } + assert!( + Instant::now() < deadline, + "node {node} never accepted an HTTP login" + ); + sleep(POLL_INTERVAL).await; + } +} + +async fn write_streams(writer: &IggyClient, index: usize) { + let names: Vec<String> = (0..STREAMS_PER_WRITER) + .map(|stream| format!("{STREAM_PREFIX}-{index}-{stream}")) + .collect(); + for name in &names { + writer + .create_stream(name, &StreamCreateOptions::default()) + .await + .expect("create stream"); + } + writer + .update_stream( + &Identifier::named(&names[0]).expect("stream id"), + &StreamUpdateOptions { + name: Some(format!("{}-renamed", names[0])), + ..StreamUpdateOptions::default() + }, + ) + .await + .expect("rename stream"); + writer + .delete_stream(&Identifier::named(&names[1]).expect("stream id")) + .await + .expect("delete stream"); +} + +async fn create_streams(writer: &IggyClient, label: &str, count: usize) { + for index in 0..count { + writer + .create_stream( + &format!("{STREAM_PREFIX}-{label}-{index}"), + &StreamCreateOptions::default(), + ) + .await + .expect("create stream"); + } +} + +/// Create a stream through `node` while the cluster elects a new primary. An +/// attempt whose reply was lost may have committed, so an existing name +/// counts as done. +async fn create_after_failover(harness: &TestHarness, node: usize, name: &str) { + let deadline = Instant::now() + CONVERGE_TIMEOUT; + loop { + if let Ok(client) = harness.root_client_for_node(node).await { + match client + .create_stream(name, &StreamCreateOptions::default()) + .await + { + Ok(_) | Err(IggyError::StreamNameAlreadyExists(_)) => return, + Err(_) => {} + } + } + assert!( + Instant::now() < deadline, + "no node accepted {name} after the failover" + ); + sleep(POLL_INTERVAL).await; + } +} + +async fn latest(client: &IggyClient, groups: &[MetadataChangeGroup]) -> MetadataChangeCursor { + let page = until_ok( + client, + &MetadataChangesQuery { + start: MetadataChangeStart::Latest, + groups: groups.to_vec(), + ..MetadataChangesQuery::default() + }, + ) + .await; + page.next.expect("latest cursor") +} + +fn after(cursor: MetadataChangeCursor, groups: &[MetadataChangeGroup]) -> MetadataChangesQuery { + MetadataChangesQuery { + start: MetadataChangeStart::After(cursor), + groups: groups.to_vec(), + limit: MAX_METADATA_CHANGES_LIMIT, + ..MetadataChangesQuery::default() + } +} + +/// One page, asked again while the node answers `behind`. Any other status +/// fails the test: a node catching up must never skip or reset. +async fn until_ok(client: &IggyClient, query: &MetadataChangesQuery) -> MetadataChanges { + let deadline = Instant::now() + CONVERGE_TIMEOUT; + loop { + match client.get_metadata_changes(query).await { + Ok(page) if page.status == MetadataChangeStatus::Ok => return page, + Ok(page) => assert_eq!(page.status, MetadataChangeStatus::Behind, "{page:?}"), + Err(error) => assert_eq!( + error.as_code(), + IggyError::TransientNotAccepted.as_code(), + "{error}" + ), + } + assert!(Instant::now() < deadline, "the node stayed behind"); + sleep(POLL_INTERVAL).await; + } +} + +async fn follow( + client: &IggyClient, + start: &MetadataChangeCursor, + groups: &[MetadataChangeGroup], + done: impl Fn(&[MetadataChangeRecord]) -> bool, +) -> Vec<MetadataChangeRecord> { + let deadline = Instant::now() + CONVERGE_TIMEOUT; + let mut cursor = *start; + let mut records = Vec::new(); + loop { + let page = until_ok(client, &after(cursor, groups)).await; + cursor = page.next.expect("continuation"); + records.extend(page.records); + if page.has_more { + continue; + } + if done(&records) { + return records; + } + assert!( + Instant::now() < deadline, + "the followed records never reached the expected state: {records:?}" + ); + sleep(POLL_INTERVAL).await; + } +} + +async fn assert_expires( + client: &IggyClient, + start: &MetadataChangeCursor, + groups: &[MetadataChangeGroup], +) { + let deadline = Instant::now() + CONVERGE_TIMEOUT; + loop { + let page = client + .get_metadata_changes(&after(*start, groups)) + .await + .expect("change page"); + if page.status == MetadataChangeStatus::Expired { + assert!(page.records.is_empty()); + assert_eq!(page.next, None); + return; + } + assert!( + matches!( + page.status, + MetadataChangeStatus::Ok | MetadataChangeStatus::Behind + ), + "{page:?}" + ); + assert!(Instant::now() < deadline, "the cursor never expired"); + sleep(POLL_INTERVAL).await; + } +} + +/// Every stream of a fresh current scan, and the scan's anchor. +async fn current_streams(client: &IggyClient) -> (BTreeMap<u32, String>, MetadataChangeCursor) { + let mut query = MetadataChangesQuery { + start: MetadataChangeStart::Current(None), + groups: STREAMS.to_vec(), + limit: MAX_METADATA_CHANGES_LIMIT, + ..MetadataChangesQuery::default() + }; + let mut streams = BTreeMap::new(); + loop { + let page = until_ok(client, &query).await; + streams.extend(stream_names(&page.items)); + match page.current_page_token { + Some(token) => query.start = MetadataChangeStart::Current(Some(token)), + None => return (streams, page.next.expect("anchor")), + } + } +} + +fn stream_names(items: &[MetadataCurrentItem]) -> BTreeMap<u32, String> { + items + .iter() + .filter_map(|item| match item { + MetadataCurrentItem::Stream(stream) => Some((stream.stream_id, stream.name.clone())), + _ => None, + }) + .collect() +} + +async fn listed_streams(client: &IggyClient) -> BTreeMap<u32, String> { + client + .get_streams() + .await + .expect("streams") + .into_iter() + .map(|stream| (stream.id, stream.name)) + .collect() +} + +/// `baseline` with every stream record applied in order. +fn mirror( + baseline: &BTreeMap<u32, String>, + records: &[MetadataChangeRecord], +) -> BTreeMap<u32, String> { + let mut streams = baseline.clone(); + for record in records { + match &record.change { + MetadataChange::StreamCreated(stream) | MetadataChange::StreamUpdated(stream) => { + streams.insert(stream.stream_id, stream.name.clone()); + } + MetadataChange::StreamDeleted(stream) => { + streams.remove(&stream.stream_id); + } + _ => {} + } + } + streams +} + +async fn await_marker(harness: &TestHarness, node: usize, marker: &str) { + let deadline = Instant::now() + CONVERGE_TIMEOUT; + while harness.node(node).stdout_occurrences(marker) == 0 { + assert!( + Instant::now() < deadline, + "node {node} never logged {marker:?}" + ); + sleep(POLL_INTERVAL).await; + } +}
diff --git a/core/integration/tests/cluster/mod.rs b/core/integration/tests/cluster/mod.rs index 180afda..58ca9b9 100644 --- a/core/integration/tests/cluster/mod.rs +++ b/core/integration/tests/cluster/mod.rs
@@ -24,6 +24,7 @@ mod crash_recovery_corruption; mod failover_client_continuity; mod fast_primary_rejoin; +mod metadata_changes; mod metadata_checkpoint_restart; mod metadata_state_transfer; mod multi_shard_partition_convergence;
diff --git a/core/integration/tests/sdk/metadata_changes.rs b/core/integration/tests/sdk/metadata_changes.rs new file mode 100644 index 0000000..47560ad --- /dev/null +++ b/core/integration/tests/sdk/metadata_changes.rs
@@ -0,0 +1,1467 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use std::collections::{BTreeMap, BTreeSet, HashSet}; +use std::time::{Duration, Instant}; + +use iggy::prelude::locking::IggyRwLockFn; +use iggy::prelude::*; +use iggy_binary_protocol::WireEncode; +use iggy_binary_protocol::codes::GET_METADATA_CHANGES_CODE; +use iggy_binary_protocol::primitives::metadata_changes::WireMetadataChangeGroup; +use iggy_binary_protocol::requests::system::{GetMetadataChangesRequest, SessionIdentity}; +use iggy_common::{ + MAX_METADATA_CHANGES_LIMIT, MetadataChangeGroups, MetadataMemberLeftReason, MetadataOptions, + MetadataPermissionsUpdatedReason, MetadataSessionClosedReason, MetadataTokenDeletedReason, + OptionKey, ServerKeyContext, StreamEncryptionKey, VsrSessionControl, text, +}; +use integration::harness::{TestHarness, wait_for_consumer_group_assignment}; +use integration::iggy_harness; +use tokio::time::sleep; + +const STREAM_NAME: &str = "changes-stream"; +const RENAMED_STREAM_NAME: &str = "changes-stream-renamed"; +const TOPIC_NAME: &str = "changes-topic"; +const DROPPED_TOPIC_NAME: &str = "changes-dropped-topic"; +const GROUP_NAME: &str = "changes-group"; +const USERNAME: &str = "changes-user"; +const RENAMED_USERNAME: &str = "changes-user-renamed"; +const PASSWORD: &str = "changes-secret"; +const NEW_PASSWORD: &str = "changes-secret-2"; +const TOKEN_NAME: &str = "changes-token"; +const OWN_KEY_STREAM: &str = "changes-own-key"; +const GENERATED_KEY_STREAM: &str = "changes-generated-key"; +const SERVER_KEY: &str = "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="; +const ROTATED_SERVER_KEY: &str = "AQIDBAUGBwgJCgsMDQ4PEBESExQVFhcYGRobHB0eHyA="; +const ROOT_USER_ID: u32 = 0; +const FOLLOW_TIMEOUT: Duration = Duration::from_secs(60); +const POLL_INTERVAL: Duration = Duration::from_millis(100); +const EVERY_KIND: [MetadataChangeKind; 21] = [ + MetadataChangeKind::StreamCreated, + MetadataChangeKind::StreamUpdated, + MetadataChangeKind::StreamDeleted, + MetadataChangeKind::TopicCreated, + MetadataChangeKind::TopicUpdated, + MetadataChangeKind::TopicDeleted, + MetadataChangeKind::PartitionsChanged, + MetadataChangeKind::ConsumerGroupCreated, + MetadataChangeKind::ConsumerGroupDeleted, + MetadataChangeKind::ConsumerGroupMemberJoined, + MetadataChangeKind::ConsumerGroupMemberLeft, + MetadataChangeKind::ConsumerGroupPartitionHandedOver, + MetadataChangeKind::UserCreated, + MetadataChangeKind::UserUpdated, + MetadataChangeKind::UserDeleted, + MetadataChangeKind::UserPasswordChanged, + MetadataChangeKind::UserPermissionsUpdated, + MetadataChangeKind::PersonalAccessTokenCreated, + MetadataChangeKind::PersonalAccessTokenDeleted, + MetadataChangeKind::SessionRegistered, + MetadataChangeKind::SessionClosed, +]; + +// Harness clients never ping, so a short heartbeat would end the writer's +// session mid-test and add an expiry record the assertions do not expect. +#[iggy_harness( + test_client_transport = [Tcp, Http, Quic, WebSocket], + server(heartbeat.enabled = true, heartbeat.interval = "60s") +)] +async fn given_every_kind_of_write_when_following_should_record_each_kind_with_its_actor( + harness: &TestHarness, +) { + let reader = harness.root_client().await.expect("reader"); + let start = latest(&reader, &[]).await; + let writer = harness.tcp_root_client().await.expect("writer"); + let writer_session = session_identity(&writer).await; + let stream_id = Identifier::named(STREAM_NAME).expect("stream id"); + let topic_id = Identifier::named(TOPIC_NAME).expect("topic id"); + let group_id = Identifier::named(GROUP_NAME).expect("group id"); + + let stream = writer + .create_stream(STREAM_NAME, &StreamCreateOptions::default()) + .await + .expect("create stream"); + let topic = writer + .create_topic( + &stream_id, + TOPIC_NAME, + &TopicCreateOptions { + partitions_count: Some(2), + ..TopicCreateOptions::default() + }, + ) + .await + .expect("create topic"); + writer + .update_topic( + &stream_id, + &topic_id, + &TopicUpdateOptions { + message_expiry: Some(IggyExpiry::ExpireDuration(IggyDuration::from( + Duration::from_secs(3600), + ))), + ..TopicUpdateOptions::default() + }, + ) + .await + .expect("update topic"); + writer + .create_topic( + &stream_id, + DROPPED_TOPIC_NAME, + &TopicCreateOptions::default(), + ) + .await + .expect("create dropped topic"); + writer + .delete_topic( + &stream_id, + &Identifier::named(DROPPED_TOPIC_NAME).expect("dropped topic id"), + ) + .await + .expect("delete topic"); + writer + .create_partitions(&stream_id, &topic_id, 1) + .await + .expect("create partitions"); + writer + .delete_partitions(&stream_id, &topic_id, 1) + .await + .expect("delete partitions"); + writer + .create_consumer_group(&stream_id, &topic_id, GROUP_NAME) + .await + .expect("create group"); + writer + .join_consumer_group(&stream_id, &topic_id, &group_id) + .await + .expect("join group"); + let assigned = wait_for_consumer_group_assignment( + &writer, + &stream_id, + &topic_id, + &group_id, + 1, + FOLLOW_TIMEOUT, + ) + .await; + writer + .leave_consumer_group(&stream_id, &topic_id, &group_id) + .await + .expect("leave group"); + writer + .delete_consumer_group(&stream_id, &topic_id, &group_id) + .await + .expect("delete group"); + let user = writer + .create_user( + USERNAME, + PASSWORD, + UserStatus::Active, + Some(stream_grant(stream.id, false)), + &UserCreateOptions::default(), + ) + .await + .expect("create user"); + let user_id = Identifier::numeric(user.id).expect("user id"); + writer + .update_user( + &user_id, + &UserUpdateOptions { + username: Some(RENAMED_USERNAME.to_owned()), + ..UserUpdateOptions::default() + }, + ) + .await + .expect("update user"); + writer + .change_password(&user_id, PASSWORD, NEW_PASSWORD) + .await + .expect("change password"); + writer + .update_permissions(&user_id, Some(stream_grant(stream.id, true))) + .await + .expect("update permissions"); + writer + .create_personal_access_token(TOKEN_NAME, PersonalAccessTokenExpiry::NeverExpire) + .await + .expect("create token"); + writer + .delete_personal_access_token(TOKEN_NAME) + .await + .expect("delete token"); + writer + .update_stream( + &stream_id, + &StreamUpdateOptions { + name: Some(RENAMED_STREAM_NAME.to_owned()), + ..StreamUpdateOptions::default() + }, + ) + .await + .expect("rename stream"); + writer + .delete_stream(&Identifier::named(RENAMED_STREAM_NAME).expect("renamed id")) + .await + .expect("delete stream"); + let pruned = wait_for_user(&writer, &user_id, |details| { + details + .permissions + .as_ref() + .is_none_or(|permissions| permissions.streams.is_none()) + }) + .await; + writer.delete_user(&user_id).await.expect("delete user"); + writer.logout_user().await.expect("logout"); + + let records = follow_until(&reader, &start, &[], |records| { + records.iter().any(|record| { + matches!( + &record.change, + MetadataChange::SessionClosed(closed) if closed.client_id == writer_session.client_id + ) + }) && records + .iter() + .any(|record| matches!(record.change, MetadataChange::StreamDeleted(_))) + }) + .await; + + let seen: HashSet<MetadataChangeKind> = records + .iter() + .filter_map(|record| record.change.kind()) + .collect(); + assert_eq!(seen, EVERY_KIND.into_iter().collect::<HashSet<_>>()); + assert_positions_increase(&start, &records); + for record in &records { + let expected_actor = match &record.change { + MetadataChange::ConsumerGroupPartitionHandedOver(_) => None, + MetadataChange::SessionRegistered(session) + if session.client_id != writer_session.client_id => + { + continue; + } + MetadataChange::SessionClosed(closed) + if closed.client_id != writer_session.client_id => + { + continue; + } + _ => Some(ROOT_USER_ID), + }; + assert_eq!(record.actor_user_id, expected_actor, "{record:?}"); + } + + let stream_created = only(&records, |change| match change { + MetadataChange::StreamCreated(body) => Some(body.clone()), + _ => None, + }); + assert_eq!(stream_created.stream_id, stream.id); + assert_eq!(stream_created.name, STREAM_NAME); + assert_eq!(stream_created.created_at, stream.created_at.as_micros()); + assert_eq!( + stream_created.options, + MetadataOptions::Readable(stream.options.clone()) + ); + let stream_key = (stream_created.stream_id, stream_created.stream_created_op); + let creation_op = records + .iter() + .find(|record| matches!(record.change, MetadataChange::StreamCreated(_))) + .map(|record| record.op); + assert_eq!(Some(stream_created.stream_created_op), creation_op); + + let renamed = only(&records, |change| match change { + MetadataChange::StreamUpdated(body) => Some(body.clone()), + _ => None, + }); + assert_eq!((renamed.stream_id, renamed.stream_created_op), stream_key); + assert_eq!(renamed.name, RENAMED_STREAM_NAME); + let deleted = only(&records, |change| match change { + MetadataChange::StreamDeleted(body) => Some(body.clone()), + _ => None, + }); + assert_eq!((deleted.stream_id, deleted.stream_created_op), stream_key); + assert_eq!(deleted.name, RENAMED_STREAM_NAME); + let deletion_op = records + .iter() + .find(|record| matches!(record.change, MetadataChange::StreamDeleted(_))) + .map(|record| record.op) + .expect("deletion record"); + // The stream held partitions, so the delete finished at a later completion. + assert!( + deleted + .requested_op + .is_some_and(|requested| requested < deletion_op), + "{deleted:?}" + ); + + let topics_created: Vec<_> = records + .iter() + .filter_map(|record| match &record.change { + MetadataChange::TopicCreated(body) => Some(body), + _ => None, + }) + .collect(); + assert_eq!(topics_created.len(), 2); + assert_eq!(topics_created[0].topic_id, topic.id); + assert_eq!(topics_created[0].name, TOPIC_NAME); + assert_eq!(topics_created[0].partitions_count, 2); + assert_eq!( + ( + topics_created[0].stream_id, + topics_created[0].stream_created_op + ), + stream_key + ); + let topic_updated = only(&records, |change| match change { + MetadataChange::TopicUpdated(body) => Some(body.clone()), + _ => None, + }); + assert_eq!(topic_updated.topic_id, topic.id); + assert_eq!( + topic_updated.topic_created_op, + topics_created[0].topic_created_op + ); + + let partitions: Vec<_> = records + .iter() + .filter_map(|record| match &record.change { + MetadataChange::PartitionsChanged(body) => { + Some((body.partitions_count, body.removed_partition_ids.clone())) + } + _ => None, + }) + .collect(); + assert_eq!(partitions, vec![(3, vec![]), (2, vec![2])]); + + let member = &assigned.members[0]; + let joined = only(&records, |change| match change { + MetadataChange::ConsumerGroupMemberJoined(body) => Some(body.clone()), + _ => None, + }); + assert_eq!(joined.member_id, member.id); + assert_eq!(joined.client_id, writer_session.client_id); + assert_eq!(joined.members_count, 1); + let mut installed: BTreeSet<u32> = joined.partition_ids.iter().copied().collect(); + installed.extend(records.iter().filter_map(|record| match &record.change { + MetadataChange::ConsumerGroupPartitionHandedOver(body) + if body.member_id == Some(member.id) => + { + Some(body.partition_id) + } + _ => None, + })); + assert_eq!(installed, member.partitions.iter().copied().collect()); + let left = only(&records, |change| match change { + MetadataChange::ConsumerGroupMemberLeft(body) => Some(body.clone()), + _ => None, + }); + assert_eq!(left.member_id, member.id); + assert_eq!(left.members_count, 0); + assert_eq!(left.reason, MetadataMemberLeftReason::Left); + + let user_created = only(&records, |change| match change { + MetadataChange::UserCreated(body) => Some(body.clone()), + _ => None, + }); + assert_eq!(user_created.user_id, user.id); + assert_eq!(user_created.username, USERNAME); + assert_eq!(user_created.user_created_at, user.created_at.as_micros()); + assert_eq!(user_created.permissions, user.permissions); + let user_updated = only(&records, |change| match change { + MetadataChange::UserUpdated(body) => Some(body.clone()), + _ => None, + }); + assert_eq!(user_updated.username, RENAMED_USERNAME); + let permission_updates: Vec<_> = records + .iter() + .filter_map(|record| match &record.change { + MetadataChange::UserPermissionsUpdated(body) => { + Some((body.reason, body.permissions.clone())) + } + _ => None, + }) + .collect(); + assert_eq!( + permission_updates, + vec![ + ( + MetadataPermissionsUpdatedReason::Updated, + Some(stream_grant(stream.id, true)) + ), + ( + MetadataPermissionsUpdatedReason::ResourceDeleted, + pruned.permissions + ), + ] + ); + let token_deleted = only(&records, |change| match change { + MetadataChange::PersonalAccessTokenDeleted(body) => Some(body.clone()), + _ => None, + }); + assert_eq!(token_deleted.user_id, ROOT_USER_ID); + assert_eq!(token_deleted.name, TOKEN_NAME); + assert_eq!(token_deleted.reason, MetadataTokenDeletedReason::Deleted); + + let registered = only(&records, |change| match change { + MetadataChange::SessionRegistered(body) if body.client_id == writer_session.client_id => { + Some(body.clone()) + } + _ => None, + }); + assert_eq!(registered.user_id, ROOT_USER_ID); + assert_eq!(registered.session, writer_session.session); + let closed = only(&records, |change| match change { + MetadataChange::SessionClosed(body) if body.client_id == writer_session.client_id => { + Some(body.clone()) + } + _ => None, + }); + assert_eq!(closed.session, writer_session.session); + assert_eq!(closed.reason, MetadataSessionClosedReason::Logout); + + let next = records + .last() + .map(|record| cursor_of(start.log_id, record)) + .expect("records"); + let caught_up = reader + .get_metadata_changes(&after(next, &[])) + .await + .expect("caught up page"); + assert_eq!(caught_up.status, MetadataChangeStatus::Ok); + assert!(caught_up.records.is_empty()); + assert!(!caught_up.has_more); + assert_eq!(caught_up.next, Some(next)); +} + +#[iggy_harness( + test_client_transport = [Tcp, Http, Quic, WebSocket], + server(heartbeat.enabled = true, heartbeat.interval = "60s") +)] +async fn given_mixed_changes_when_filtering_should_return_exactly_the_matching_records( + harness: &TestHarness, +) { + let reader = harness.root_client().await.expect("reader"); + let writer = harness.tcp_root_client().await.expect("writer"); + seed_catalog(&writer).await; + let everything = earliest(&reader, &[], None, None, MAX_METADATA_CHANGES_LIMIT).await; + let present: BTreeSet<MetadataChangeGroup> = everything + .iter() + .filter_map(|record| record.change.group()) + .collect(); + assert_eq!( + present, + BTreeSet::from([ + MetadataChangeGroup::Streams, + MetadataChangeGroup::Topics, + MetadataChangeGroup::ConsumerGroups, + MetadataChangeGroup::Users, + MetadataChangeGroup::Sessions, + ]) + ); + + for group in MetadataChangeGroups::all().iter() { + let filtered = earliest(&reader, &[group], None, None, MAX_METADATA_CHANGES_LIMIT).await; + assert_eq!(filtered, matching(&everything, &[group]), "{group}"); + } + let across_classes = [MetadataChangeGroup::Topics, MetadataChangeGroup::Sessions]; + let filtered = earliest( + &reader, + &across_classes, + None, + None, + MAX_METADATA_CHANGES_LIMIT, + ) + .await; + assert!(!filtered.is_empty()); + assert_eq!(filtered, matching(&everything, &across_classes)); + let duplicated = earliest( + &reader, + &[MetadataChangeGroup::Users, MetadataChangeGroup::Users], + None, + None, + MAX_METADATA_CHANGES_LIMIT, + ) + .await; + assert_eq!( + duplicated, + matching(&everything, &[MetadataChangeGroup::Users]) + ); + + let one_by_one = earliest(&reader, &[], None, None, 1).await; + assert_eq!(one_by_one, everything); + + let middle = everything.len() / 2; + let since = everything[middle / 2].timestamp; + let until = everything[middle].timestamp; + let ranged = earliest( + &reader, + &[], + Some(since.into()), + Some(until.into()), + MAX_METADATA_CHANGES_LIMIT, + ) + .await; + let expected: Vec<_> = everything + .iter() + .filter(|record| (since..=until).contains(&record.timestamp)) + .cloned() + .collect(); + assert!(!expected.is_empty()); + assert_eq!(ranged, expected); + + let future = everything.last().expect("records").timestamp + 3_600_000_000; + let empty = reader + .get_metadata_changes(&MetadataChangesQuery { + since: Some(future.into()), + ..MetadataChangesQuery::default() + }) + .await + .expect("empty filtered page"); + assert_eq!(empty.status, MetadataChangeStatus::Ok); + assert!(empty.records.is_empty()); + let continued = reader + .get_metadata_changes(&after(empty.next.expect("continuation"), &[])) + .await + .expect("continue after an empty page"); + assert_eq!(continued.status, MetadataChangeStatus::Ok); + + for query in refused_queries() { + let error = reader + .get_metadata_changes(&query) + .await + .expect_err("the SDK refuses before sending"); + assert_eq!( + error.as_code(), + IggyError::InvalidCommand.as_code(), + "{query:?}" + ); + assert_refused_by_server(&reader, &query).await; + } + assert_raw_refusals(&reader).await; +} + +#[iggy_harness(test_client_transport = [Tcp, Http, Quic, WebSocket])] +async fn given_foreign_future_or_mismatched_cursor_when_reading_should_answer_reset_or_behind( + harness: &TestHarness, +) { + let reader = harness.root_client().await.expect("reader"); + let writer = harness.tcp_root_client().await.expect("writer"); + seed_catalog(&writer).await; + let head = reader + .get_metadata_changes(&MetadataChangesQuery { + start: MetadataChangeStart::Latest, + ..MetadataChangesQuery::default() + }) + .await + .expect("latest"); + let cursor = head.next.expect("latest cursor"); + assert_ne!(cursor.op, 0); + + for (cursor, expected) in [ + ( + MetadataChangeCursor { + log_id: cursor.log_id ^ 1, + ..cursor + }, + MetadataChangeStatus::Reset, + ), + ( + MetadataChangeCursor { + checksum: cursor.checksum ^ 1, + ..cursor + }, + MetadataChangeStatus::Reset, + ), + ( + MetadataChangeCursor { + op: head.head_op + 1_000, + ..cursor + }, + MetadataChangeStatus::Behind, + ), + ] { + let answer = reader + .get_metadata_changes(&after(cursor, &[])) + .await + .expect("position answer"); + assert_eq!(answer.status, expected, "{cursor:?}"); + assert!(answer.records.is_empty()); + let echoed = (expected == MetadataChangeStatus::Behind).then_some(cursor); + assert_eq!(answer.next, echoed, "{cursor:?}"); + assert_eq!(answer.log_id, head.log_id); + } + + let from_zero = reader + .get_metadata_changes(&after(MetadataChangeCursor::zero(cursor.log_id), &[])) + .await + .expect("zero cursor"); + let from_earliest = reader + .get_metadata_changes(&MetadataChangesQuery::default()) + .await + .expect("earliest"); + assert_eq!(from_zero.status, MetadataChangeStatus::Ok); + assert_eq!(from_zero.records, from_earliest.records); + assert_eq!(from_zero.records[0].op, 1); +} + +#[iggy_harness( + test_client_transport = [Tcp, Http, Quic, WebSocket], + server(metadata.change_log_resource_capacity = "4") +)] +async fn given_evicted_records_when_following_should_expire_and_rebuild_from_current_state( + harness: &TestHarness, +) { + let groups = [MetadataChangeGroup::Streams]; + let reader = harness.root_client().await.expect("reader"); + let writer = harness.tcp_root_client().await.expect("writer"); + let start = latest(&reader, &groups).await; + for index in 0..6 { + writer + .create_stream( + &format!("{STREAM_NAME}-{index}"), + &StreamCreateOptions::default(), + ) + .await + .expect("create stream"); + } + + let expired = reader + .get_metadata_changes(&after(start, &groups)) + .await + .expect("expired answer"); + assert_eq!(expired.status, MetadataChangeStatus::Expired); + assert!(expired.records.is_empty()); + assert!(expired.oldest_op > start.op); + let retained = earliest(&reader, &groups, None, None, MAX_METADATA_CHANGES_LIMIT).await; + assert!(!retained.is_empty() && retained.len() <= 4, "{retained:?}"); + assert!(retained.iter().all(|record| record.op >= expired.oldest_op)); + + let (items, anchor) = current_state(&reader, &groups, 1).await; + assert_eq!(stream_items(&items), listed_streams(&reader).await); + + writer + .create_stream(STREAM_NAME, &StreamCreateOptions::default()) + .await + .expect("create stream after the rebuild"); + let caught_up = follow_until(&reader, &anchor, &groups, |records| !records.is_empty()).await; + assert_eq!(caught_up.len(), 1); + let MetadataChange::StreamCreated(created) = &caught_up[0].change else { + panic!("expected a stream record, got {caught_up:?}"); + }; + assert_eq!(created.name, STREAM_NAME); +} + +#[iggy_harness( + test_client_transport = [Tcp, Http, Quic, WebSocket], + server(heartbeat.enabled = true, heartbeat.interval = "60s") +)] +async fn given_catalog_when_paging_current_state_should_match_list_routes_for_every_closed_subset( + harness: &TestHarness, +) { + let reader = harness.root_client().await.expect("reader"); + let writer = harness.tcp_root_client().await.expect("writer"); + let writer_session = session_identity(&writer).await; + let stream_id = seed_catalog(&writer).await; + let topic_id = Identifier::named(TOPIC_NAME).expect("topic id"); + let group_id = Identifier::named(GROUP_NAME).expect("group id"); + writer + .join_consumer_group(&stream_id, &topic_id, &group_id) + .await + .expect("join group"); + let assigned = wait_for_consumer_group_assignment( + &writer, + &stream_id, + &topic_id, + &group_id, + 1, + FOLLOW_TIMEOUT, + ) + .await; + writer + .create_personal_access_token(TOKEN_NAME, PersonalAccessTokenExpiry::NeverExpire) + .await + .expect("create token"); + + let (items, anchor) = current_state(&reader, &[], 1).await; + let (bulk, bulk_anchor) = current_state(&reader, &[], MAX_METADATA_CHANGES_LIMIT).await; + assert_eq!(items, bulk); + assert_eq!(anchor, bulk_anchor); + + assert_eq!(stream_items(&items), listed_streams(&reader).await); + let topics: BTreeMap<u32, (String, u32)> = writer + .get_topics(&stream_id) + .await + .expect("topics") + .into_iter() + .map(|topic| (topic.id, (topic.name, topic.partitions_count))) + .collect(); + let projected_topics: BTreeMap<u32, (String, u32)> = items + .iter() + .filter_map(|item| match item { + MetadataCurrentItem::Topic(topic) => Some((topic.topic_id, topic.name.clone())), + _ => None, + }) + .map(|(id, name)| { + let count = items + .iter() + .find_map(|item| match item { + MetadataCurrentItem::Partitions(partitions) if partitions.topic_id == id => { + Some(partitions.partitions_count) + } + _ => None, + }) + .expect("partitions item"); + (id, (name, count)) + }) + .collect(); + assert_eq!(projected_topics, topics); + let groups: Vec<(u32, String)> = items + .iter() + .filter_map(|item| match item { + MetadataCurrentItem::ConsumerGroup(group) => Some((group.group_id, group.name.clone())), + _ => None, + }) + .collect(); + assert_eq!(groups, vec![(assigned.id, assigned.name.clone())]); + let members: Vec<(u32, u128, Vec<u32>)> = items + .iter() + .filter_map(|item| match item { + MetadataCurrentItem::ConsumerGroupMember(member) => Some(( + member.member_id, + member.client_id, + member.partition_ids.clone(), + )), + _ => None, + }) + .collect(); + let mut partitions = assigned.members[0].partitions.clone(); + partitions.sort_unstable(); + assert_eq!( + members, + vec![(assigned.members[0].id, writer_session.client_id, partitions)] + ); + let users: BTreeMap<u32, (String, UserStatus)> = reader + .get_users() + .await + .expect("users") + .into_iter() + .map(|user| (user.id, (user.username, user.status))) + .collect(); + let projected_users: BTreeMap<u32, (String, UserStatus)> = items + .iter() + .filter_map(|item| match item { + MetadataCurrentItem::User(user) => { + Some((user.user_id, (user.username.clone(), user.status))) + } + _ => None, + }) + .collect(); + assert_eq!(projected_users, users); + let tokens: Vec<(u32, String)> = items + .iter() + .filter_map(|item| match item { + MetadataCurrentItem::PersonalAccessToken(token) => { + Some((token.user_id, token.name.clone())) + } + _ => None, + }) + .collect(); + assert_eq!(tokens, vec![(ROOT_USER_ID, TOKEN_NAME.to_owned())]); + assert!(items.iter().any(|item| matches!( + item, + MetadataCurrentItem::Session(session) + if session.client_id == writer_session.client_id + && session.session == writer_session.session + ))); + + for mask in 1..=u8::MAX { + let requested: Vec<MetadataChangeGroup> = MetadataChangeGroups::all() + .iter() + .filter(|group| mask & WireMetadataChangeGroup::from(*group).mask_bit() != 0) + .collect(); + let query = MetadataChangesQuery { + start: MetadataChangeStart::Current(None), + groups: requested.clone(), + limit: MAX_METADATA_CHANGES_LIMIT, + ..MetadataChangesQuery::default() + }; + if MetadataChangeGroups::requested(&requested) + .missing_parent() + .is_some() + { + assert_refused_by_server(&reader, &query).await; + continue; + } + let (subset, subset_anchor) = + current_state(&reader, &requested, MAX_METADATA_CHANGES_LIMIT).await; + let expected: Vec<_> = items + .iter() + .filter(|item| item.group().is_some_and(|group| requested.contains(&group))) + .cloned() + .collect(); + assert_eq!(subset, expected, "{requested:?}"); + assert_eq!(subset_anchor.log_id, anchor.log_id); + } + + let first = reader + .get_metadata_changes(&MetadataChangesQuery { + start: MetadataChangeStart::Current(None), + limit: 1, + ..MetadataChangesQuery::default() + }) + .await + .expect("first current page"); + let token = first.current_page_token.expect("more pages"); + let mismatched = MetadataChangesQuery { + start: MetadataChangeStart::Current(Some(token)), + groups: vec![MetadataChangeGroup::Users], + limit: 1, + ..MetadataChangesQuery::default() + }; + let error = reader + .get_metadata_changes(&mismatched) + .await + .expect_err("a token is bound to its groups"); + assert_eq!(error.as_code(), IggyError::InvalidCommand.as_code()); + assert_refused_by_server(&reader, &mismatched).await; +} + +#[iggy_harness( + test_client_transport = [Tcp, Http, Quic, WebSocket], + server(encryption = "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE=") +)] +async fn given_keyed_streams_when_rotating_server_key_should_publish_fingerprints_without_key_material( + harness: &mut TestHarness, +) { + let groups = [MetadataChangeGroup::Streams]; + let own_key = StreamEncryptionKey::try_from([7u8; 32].as_slice()).expect("stream key"); + let start = latest(&harness.root_client().await.expect("reader"), &groups).await; + let writer = harness.tcp_root_client().await.expect("writer"); + let created = [ + writer + .create_stream( + OWN_KEY_STREAM, + &StreamCreateOptions { + encryption_key: Some(own_key.clone()), + ..StreamCreateOptions::default() + }, + ) + .await + .expect("create own key stream"), + writer + .create_stream(GENERATED_KEY_STREAM, &StreamCreateOptions::default()) + .await + .expect("create generated key stream"), + ]; + drop(writer); + + rotate_server_key(harness).await; + let reader = harness.root_client().await.expect("reader after restart"); + let mut rotated = vec![ + kek_fingerprint(SERVER_KEY), + kek_fingerprint(ROTATED_SERVER_KEY), + ]; + rotated.sort(); + let fetched = wait_for_kek_fingerprints(&reader, &rotated).await; + + let rotated_options: Vec<MetadataOptions> = fetched + .iter() + .map(|details| MetadataOptions::Readable(details.options.clone())) + .collect(); + let records = follow_until(&reader, &start, &groups, |records| { + rotated_options.iter().all(|options| { + records.iter().any(|record| { + matches!(&record.change, MetadataChange::StreamUpdated(body) if &body.options == options) + }) + }) + }) + .await; + let (items, _) = current_state(&reader, &groups, MAX_METADATA_CHANGES_LIMIT).await; + for record in &records { + match &record.change { + MetadataChange::StreamCreated(_) => { + assert_eq!(record.actor_user_id, Some(ROOT_USER_ID)); + } + MetadataChange::StreamUpdated(_) => assert_eq!(record.actor_user_id, None), + other => panic!("unexpected record {other:?}"), + } + } + + let mut published: Vec<&ResourceOptions> = + created.iter().map(|stream| &stream.options).collect(); + published.extend(fetched.iter().map(|stream| &stream.options)); + for change in records.iter().map(|record| &record.change) { + if let MetadataChange::StreamCreated(body) | MetadataChange::StreamUpdated(body) = change { + let MetadataOptions::Readable(options) = &body.options else { + panic!("stream options must stay readable: {body:?}"); + }; + published.push(options); + } + } + for item in &items { + let MetadataCurrentItem::Stream(stream) = item else { + panic!("unexpected item {item:?}"); + }; + let MetadataOptions::Readable(options) = &stream.options else { + panic!("stream options must stay readable: {stream:?}"); + }; + let listed = fetched + .iter() + .find(|details| details.id == stream.stream_id) + .expect("listed stream"); + assert_eq!(options, &listed.options); + published.push(options); + } + let own_key_text = [text::as_base64(own_key.as_bytes()), hex(own_key.as_bytes())]; + for options in published { + for hidden in [ + OptionKey::EncryptionKey, + OptionKey::EncryptionKeyWrapped, + OptionKey::CriticalKeys, + ] { + assert!(!options.contains_key(&HeaderKey::from(hidden)), "{hidden}"); + } + let rendered = format!("{options:?}"); + for key_text in &own_key_text { + assert!(!rendered.contains(key_text.as_str())); + } + } + + let http = harness.http_root_client().await.expect("HTTP client"); + for path in [ + "/streams".to_owned(), + format!("/streams/{OWN_KEY_STREAM}"), + format!("/streams/{GENERATED_KEY_STREAM}"), + "/metadata/changes?from=earliest&groups=streams".to_owned(), + "/metadata/changes?from=current&groups=streams".to_owned(), + ] { + let body = http + .send_http_request(HttpMethod::Get, &path, None) + .await + .expect("HTTP read"); + let body = String::from_utf8_lossy(&body); + for key_text in &own_key_text { + assert!(!body.contains(key_text.as_str()), "{path}"); + } + for hidden in [ + OptionKey::EncryptionKey, + OptionKey::EncryptionKeyWrapped, + OptionKey::CriticalKeys, + ] { + let name: &'static str = hidden.into(); + assert!(!body.contains(&format!("\"{name}\"")), "{path}: {name}"); + } + } + let own = fetched + .iter() + .find(|details| details.name == OWN_KEY_STREAM) + .expect("own key stream"); + assert_eq!( + own.encryption_key_fingerprint(), + Some(own_key.fingerprint()) + ); + assert_eq!( + own.encryption_key_source(), + Some(StreamEncryptionSource::Stream) + ); + let generated = fetched + .iter() + .find(|details| details.name == GENERATED_KEY_STREAM) + .expect("generated key stream"); + assert_eq!( + generated.encryption_key_source(), + Some(StreamEncryptionSource::Server) + ); +} + +/// Restart every node in turn under the rotated key, waiting for each to serve +/// before the next goes down. +async fn rotate_server_key(harness: &mut TestHarness) { + let stream_id = Identifier::named(OWN_KEY_STREAM).expect("stream id"); + for node in 0..harness.cluster_size() { + let server = harness.node_mut(node); + server.add_env("IGGY_ENCRYPTION_KEY", ROTATED_SERVER_KEY); + server.add_env( + "IGGY_ENCRYPTION_PREVIOUS_KEYS", + format!("[\"{SERVER_KEY}\"]"), + ); + harness.restart_node(node).expect("rolling restart"); + let deadline = Instant::now() + FOLLOW_TIMEOUT; + loop { + if let Ok(client) = harness.root_client_for_node(node).await + && matches!(client.get_stream(&stream_id).await, Ok(Some(_))) + { + break; + } + assert!( + Instant::now() < deadline, + "node {node} never served after its restart" + ); + sleep(POLL_INTERVAL).await; + } + } +} + +/// Streams, topics, a group and a user: records in every resource group. +async fn seed_catalog(writer: &IggyClient) -> Identifier { + let stream = writer + .create_stream(STREAM_NAME, &StreamCreateOptions::default()) + .await + .expect("create stream"); + let stream_id = Identifier::numeric(stream.id).expect("stream id"); + writer + .create_topic( + &stream_id, + TOPIC_NAME, + &TopicCreateOptions { + partitions_count: Some(3), + ..TopicCreateOptions::default() + }, + ) + .await + .expect("create topic"); + writer + .create_topic( + &stream_id, + DROPPED_TOPIC_NAME, + &TopicCreateOptions::default(), + ) + .await + .expect("create second topic"); + writer + .create_consumer_group( + &stream_id, + &Identifier::named(TOPIC_NAME).expect("topic id"), + GROUP_NAME, + ) + .await + .expect("create group"); + writer + .create_stream(RENAMED_STREAM_NAME, &StreamCreateOptions::default()) + .await + .expect("create second stream"); + writer + .create_user( + USERNAME, + PASSWORD, + UserStatus::Active, + Some(stream_grant(stream.id, false)), + &UserCreateOptions::default(), + ) + .await + .expect("create user"); + stream_id +} + +async fn latest(client: &IggyClient, groups: &[MetadataChangeGroup]) -> MetadataChangeCursor { + let answer = client + .get_metadata_changes(&MetadataChangesQuery { + start: MetadataChangeStart::Latest, + groups: groups.to_vec(), + ..MetadataChangesQuery::default() + }) + .await + .expect("latest"); + assert_eq!(answer.status, MetadataChangeStatus::Ok); + assert!(answer.records.is_empty()); + answer.next.expect("latest cursor") +} + +fn after(cursor: MetadataChangeCursor, groups: &[MetadataChangeGroup]) -> MetadataChangesQuery { + MetadataChangesQuery { + start: MetadataChangeStart::After(cursor), + groups: groups.to_vec(), + limit: MAX_METADATA_CHANGES_LIMIT, + ..MetadataChangesQuery::default() + } +} + +/// Every retained record of `groups` in the time range, paged by `limit`. +async fn earliest( + client: &IggyClient, + groups: &[MetadataChangeGroup], + since: Option<IggyTimestamp>, + until: Option<IggyTimestamp>, + limit: u32, +) -> Vec<MetadataChangeRecord> { + let mut query = MetadataChangesQuery { + start: MetadataChangeStart::Earliest, + groups: groups.to_vec(), + since, + until, + limit, + }; + let mut records = Vec::new(); + loop { + let page = client + .get_metadata_changes(&query) + .await + .expect("change page"); + assert_eq!(page.status, MetadataChangeStatus::Ok); + assert!(page.records.len() <= limit as usize); + records.extend(page.records); + if !page.has_more { + return records; + } + query.start = MetadataChangeStart::After(page.next.expect("continuation")); + } +} + +/// Follow from `start` until `done` holds for everything read so far. +async fn follow_until( + client: &IggyClient, + start: &MetadataChangeCursor, + groups: &[MetadataChangeGroup], + done: impl Fn(&[MetadataChangeRecord]) -> bool, +) -> Vec<MetadataChangeRecord> { + let deadline = Instant::now() + FOLLOW_TIMEOUT; + let mut cursor = *start; + let mut records = Vec::new(); + loop { + let page = client + .get_metadata_changes(&after(cursor, groups)) + .await + .expect("change page"); + assert_eq!(page.status, MetadataChangeStatus::Ok, "{page:?}"); + cursor = page.next.expect("continuation"); + records.extend(page.records); + if page.has_more { + continue; + } + if done(&records) { + return records; + } + assert!( + Instant::now() < deadline, + "the followed records never reached the expected state: {records:?}" + ); + sleep(POLL_INTERVAL).await; + } +} + +/// Every current state page of `groups`, checking the anchor stays fixed. +async fn current_state( + client: &IggyClient, + groups: &[MetadataChangeGroup], + limit: u32, +) -> (Vec<MetadataCurrentItem>, MetadataChangeCursor) { + let mut query = MetadataChangesQuery { + start: MetadataChangeStart::Current(None), + groups: groups.to_vec(), + limit, + ..MetadataChangesQuery::default() + }; + let mut items = Vec::new(); + let mut anchor = None; + loop { + let page = client + .get_metadata_changes(&query) + .await + .expect("current page"); + assert_eq!(page.status, MetadataChangeStatus::Ok, "{page:?}"); + assert_eq!(page.phase, MetadataChangePhase::Current); + assert!(page.records.is_empty()); + assert!(page.items.len() <= limit as usize); + let next = page.next.expect("anchor"); + assert_eq!(*anchor.get_or_insert(next), next); + items.extend(page.items); + match page.current_page_token { + Some(token) => query.start = MetadataChangeStart::Current(Some(token)), + None => return (items, next), + } + } +} + +fn matching( + records: &[MetadataChangeRecord], + groups: &[MetadataChangeGroup], +) -> Vec<MetadataChangeRecord> { + records + .iter() + .filter(|record| { + record + .change + .group() + .is_some_and(|group| groups.contains(&group)) + }) + .cloned() + .collect() +} + +fn only<T>(records: &[MetadataChangeRecord], select: impl Fn(&MetadataChange) -> Option<T>) -> T { + let mut found: Vec<T> = records + .iter() + .filter_map(|record| select(&record.change)) + .collect(); + assert_eq!(found.len(), 1, "expected exactly one matching record"); + found.remove(0) +} + +const fn cursor_of(log_id: u128, record: &MetadataChangeRecord) -> MetadataChangeCursor { + MetadataChangeCursor { + log_id, + op: record.op, + checksum: record.checksum, + ordinal: record.ordinal, + } +} + +fn assert_positions_increase(start: &MetadataChangeCursor, records: &[MetadataChangeRecord]) { + let mut previous = (start.op, start.ordinal); + for record in records { + let position = (record.op, record.ordinal); + assert!(position > previous, "{previous:?} then {position:?}"); + previous = position; + } +} + +fn stream_items(items: &[MetadataCurrentItem]) -> BTreeMap<u32, (String, u64, ResourceOptions)> { + items + .iter() + .filter_map(|item| match item { + MetadataCurrentItem::Stream(stream) => match &stream.options { + MetadataOptions::Readable(options) => Some(( + stream.stream_id, + (stream.name.clone(), stream.created_at, options.clone()), + )), + MetadataOptions::Unreadable => panic!("unreadable stream {stream:?}"), + }, + _ => None, + }) + .collect() +} + +async fn listed_streams(client: &IggyClient) -> BTreeMap<u32, (String, u64, ResourceOptions)> { + client + .get_streams() + .await + .expect("streams") + .into_iter() + .map(|stream| { + ( + stream.id, + (stream.name, stream.created_at.as_micros(), stream.options), + ) + }) + .collect() +} + +fn refused_queries() -> Vec<MetadataChangesQuery> { + let cursor = MetadataChangeCursor { + log_id: 7, + op: 0, + checksum: 1, + ordinal: 0, + }; + vec![ + MetadataChangesQuery { + limit: 0, + ..MetadataChangesQuery::default() + }, + MetadataChangesQuery { + limit: MAX_METADATA_CHANGES_LIMIT + 1, + ..MetadataChangesQuery::default() + }, + MetadataChangesQuery { + since: Some(IggyTimestamp::from(5)), + until: Some(IggyTimestamp::from(4)), + ..MetadataChangesQuery::default() + }, + MetadataChangesQuery { + start: MetadataChangeStart::Latest, + since: Some(IggyTimestamp::from(1)), + ..MetadataChangesQuery::default() + }, + MetadataChangesQuery { + start: MetadataChangeStart::Current(None), + until: Some(IggyTimestamp::from(1)), + ..MetadataChangesQuery::default() + }, + MetadataChangesQuery { + start: MetadataChangeStart::Current(None), + groups: vec![MetadataChangeGroup::Topics], + ..MetadataChangesQuery::default() + }, + MetadataChangesQuery { + start: MetadataChangeStart::After(cursor), + ..MetadataChangesQuery::default() + }, + ] +} + +/// Send `query` past the SDK's own checks and expect the server to refuse it. +async fn assert_refused_by_server(client: &IggyClient, query: &MetadataChangesQuery) { + match client.get_connection_info().await.protocol { + TransportProtocol::Http => { + let pairs: Vec<String> = query + .http_query_pairs() + .into_iter() + .map(|(key, value)| format!("{key}={value}")) + .collect(); + assert_http_refused(client, &pairs.join("&")).await; + } + _ => { + assert_binary_refused(client, GetMetadataChangesRequest::from(query)).await; + } + } +} + +/// Refusals only one transport can express: unknown group codes on the +/// binary request, malformed text on HTTP. +async fn assert_raw_refusals(client: &IggyClient) { + match client.get_connection_info().await.protocol { + TransportProtocol::Http => { + for query in [ + "", + "from=earliest&from=latest", + "from=earliest&after=mc1_AAAA", + "from=sometime", + "from=earliest&unknown=1", + "from=earliest&groups=streams,", + "from=earliest&groups=queues", + "from=earliest&since=01", + "from=earliest&limit=ten", + "after=mc1_not-a-cursor", + "from=current¤t_after=cs1_not-a-token", + ] { + assert_http_refused(client, query).await; + } + } + _ => { + let mut unknown_group = + GetMetadataChangesRequest::from(&MetadataChangesQuery::default()); + unknown_group.groups.push(99); + assert_binary_refused(client, unknown_group).await; + let encoded = + GetMetadataChangesRequest::from(&MetadataChangesQuery::default()).to_bytes(); + let error = client + .send_binary_request( + GET_METADATA_CHANGES_CODE, + encoded.slice(..encoded.len() - 1), + ) + .await + .expect_err("a truncated body is refused"); + assert_eq!(error.as_code(), IggyError::InvalidCommand.as_code()); + } + } +} + +async fn assert_http_refused(client: &IggyClient, query: &str) { + let error = client + .send_http_request(HttpMethod::Get, &format!("/metadata/changes?{query}"), None) + .await + .expect_err(query); + assert!( + matches!(error, IggyError::HttpResponseError(400, _)), + "{query}: {error:?}" + ); +} + +async fn assert_binary_refused(client: &IggyClient, request: GetMetadataChangesRequest) { + let error = client + .send_binary_request(GET_METADATA_CHANGES_CODE, request.to_bytes()) + .await + .expect_err("the server refuses the request"); + assert_eq!( + error.as_code(), + IggyError::InvalidCommand.as_code(), + "{request:?}: {error:?}" + ); +} + +async fn session_identity(client: &IggyClient) -> SessionIdentity { + let handle = client.client(); + let transport = handle.read().await; + let ClientWrapper::Tcp(tcp) = &*transport else { + panic!("the writer uses the TCP transport"); + }; + tcp.session_identity().await.expect("bound session") +} + +async fn wait_for_user( + client: &IggyClient, + user_id: &Identifier, + done: impl Fn(&UserInfoDetails) -> bool, +) -> UserInfoDetails { + let deadline = Instant::now() + FOLLOW_TIMEOUT; + loop { + let details = client + .get_user(user_id) + .await + .expect("read user") + .expect("user exists"); + if done(&details) { + return details; + } + assert!( + Instant::now() < deadline, + "user never converged: {details:?}" + ); + sleep(POLL_INTERVAL).await; + } +} + +async fn wait_for_kek_fingerprints(client: &IggyClient, expected: &[String]) -> Vec<StreamDetails> { + let deadline = Instant::now() + FOLLOW_TIMEOUT; + loop { + let mut fetched = Vec::with_capacity(2); + for name in [OWN_KEY_STREAM, GENERATED_KEY_STREAM] { + if let Ok(Some(details)) = client + .get_stream(&Identifier::named(name).expect("stream id")) + .await + { + fetched.push(details); + } + } + if fetched.len() == 2 + && fetched + .iter() + .all(|details| details.encryption_kek_fingerprints() == expected) + { + return fetched; + } + assert!( + Instant::now() < deadline, + "streams never reported KEK fingerprints {expected:?}" + ); + sleep(POLL_INTERVAL).await; + } +} + +fn stream_grant(stream_id: u32, manage_stream: bool) -> Permissions { + Permissions { + global: GlobalPermissions::default(), + streams: Some(BTreeMap::from([( + stream_id as usize, + StreamPermissions { + manage_stream, + read_stream: true, + ..StreamPermissions::default() + }, + )])), + } +} + +fn kek_fingerprint(key: &str) -> String { + ServerKeyContext::from_config(key, &[], true) + .expect("server key") + .kek_fingerprint() + .expect("fingerprint") +} + +fn hex(bytes: &[u8]) -> String { + bytes.iter().map(|byte| format!("{byte:02x}")).collect() +}
diff --git a/core/integration/tests/sdk/mod.rs b/core/integration/tests/sdk/mod.rs index 483aff0..817d09b 100644 --- a/core/integration/tests/sdk/mod.rs +++ b/core/integration/tests/sdk/mod.rs
@@ -23,6 +23,7 @@ mod disconnect_relogin; mod hello_world; mod http_refresh; +mod metadata_changes; mod options; mod producer; mod protocol_version;
diff --git a/core/integration/tests/server/scenarios/authentication_scenario.rs b/core/integration/tests/server/scenarios/authentication_scenario.rs index 9d433a3..02f7d42 100644 --- a/core/integration/tests/server/scenarios/authentication_scenario.rs +++ b/core/integration/tests/server/scenarios/authentication_scenario.rs
@@ -36,13 +36,16 @@ use iggy_binary_protocol::requests::system::{BindSessionRequest, SessionIdentity}; use iggy_binary_protocol::requests::users::login_register::BindSecret; use iggy_binary_protocol::{ClientVersionInfo, IGGY_PROTOCOL_VERSION, WireName}; -use iggy_common::VsrSessionControl; use iggy_common::wire_conversions::identifier_to_wire; +use iggy_common::{MetadataChangeGroups, VsrSessionControl}; use integration::harness::{TestHarness, login_root}; const STREAM_NAME: &str = "auth-test-stream"; const TOPIC_NAME: &str = "auth-test-topic"; const INVALID_BIND_SECRET: [u8; 32] = [0; 32]; +const METADATA_READER_USERNAME: &str = "auth-test-metadata-reader"; +const METADATA_SERVER_MANAGER_USERNAME: &str = "auth-test-metadata-manager"; +const METADATA_USER_PASSWORD: &str = "auth-test-secret"; /// Shared test context with identifiers used across command tests. struct TestContext { @@ -89,6 +92,7 @@ assert_eq!(identity.user_id, 0, "root user should have id 0"); setup_test_resources(&client).await; verify_auth_works(&client).await; + verify_metadata_changes_need_manage_servers(harness, &client).await; // Phase 4: Logout and verify commands fail again client.logout_user().await.expect("logout should succeed"); @@ -177,6 +181,10 @@ .describe_options(OptionsScope::Topic) .await .map(|_| ()), + GET_METADATA_CHANGES_CODE => client + .get_metadata_changes(&MetadataChangesQuery::default()) + .await + .map(|_| ()), BIND_SESSION_CODE => client .send_binary_request( code, @@ -427,6 +435,78 @@ client.get_personal_access_tokens().await.expect("get_pats"); } +/// Reading metadata changes needs `manage_servers`, whatever groups are asked +/// for and however the scan starts. +async fn verify_metadata_changes_need_manage_servers(harness: &TestHarness, root: &IggyClient) { + let mut queries = vec![ + MetadataChangesQuery::default(), + MetadataChangesQuery { + start: MetadataChangeStart::Latest, + ..MetadataChangesQuery::default() + }, + MetadataChangesQuery { + start: MetadataChangeStart::Current(None), + ..MetadataChangesQuery::default() + }, + ]; + queries.extend( + MetadataChangeGroups::all() + .iter() + .map(|group| MetadataChangesQuery { + groups: vec![group], + ..MetadataChangesQuery::default() + }), + ); + for (username, manage_servers) in [ + (METADATA_READER_USERNAME, false), + (METADATA_SERVER_MANAGER_USERNAME, true), + ] { + root.create_user( + username, + METADATA_USER_PASSWORD, + UserStatus::Active, + Some(Permissions { + global: GlobalPermissions { + manage_servers, + read_servers: true, + read_streams: true, + read_users: true, + ..GlobalPermissions::default() + }, + streams: None, + }), + &UserCreateOptions::default(), + ) + .await + .expect("create metadata user"); + let client = create_client(harness).await; + client + .login_user(username, METADATA_USER_PASSWORD) + .await + .expect("metadata user login"); + for query in &queries { + let result = client.get_metadata_changes(query).await; + if manage_servers { + result.expect("manage_servers reads metadata changes"); + } else { + let error = result.expect_err("reading metadata changes needs manage_servers"); + assert_eq!( + error.as_code(), + IggyError::Unauthorized.as_code(), + "{query:?}" + ); + } + } + client.logout_user().await.expect("metadata user logout"); + root.delete_user(&Identifier::named(username).unwrap()) + .await + .expect("delete metadata user"); + } + root.get_metadata_changes(&MetadataChangesQuery::default()) + .await + .expect("root reads metadata changes"); +} + async fn setup_test_resources(client: &IggyClient) { client .create_stream(STREAM_NAME, &StreamCreateOptions::default())
diff --git a/core/metadata/Cargo.toml b/core/metadata/Cargo.toml index 3caae88..0accf01 100644 --- a/core/metadata/Cargo.toml +++ b/core/metadata/Cargo.toml
@@ -60,6 +60,12 @@ bytes = { workspace = true } tempfile = { workspace = true } +# Seeds root the way the simulator does, so it needs that feature. +[[bench]] +name = "change_log" +harness = false +required-features = ["simulator"] + [lints.clippy] enum_glob_use = "deny" pedantic = { level = "deny", priority = -1 }
diff --git a/core/metadata/benches/change_log.rs b/core/metadata/benches/change_log.rs new file mode 100644 index 0000000..9a65834 --- /dev/null +++ b/core/metadata/benches/change_log.rs
@@ -0,0 +1,415 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +// Timings and sizes are printed as floating point means and percentiles. +#![allow( + clippy::cast_precision_loss, + clippy::cast_possible_truncation, + clippy::cast_sign_loss +)] + +use consensus::{CLIENTS_TABLE_MAX, ClientTable}; +use iggy_binary_protocol::primitives::metadata_changes::{ + WireMetadataChange, WireSessionBody, WireUserKey, +}; +use iggy_binary_protocol::primitives::partition_assignment::CreatedPartitionAssignment; +use iggy_binary_protocol::requests::streams::CreateStreamRequest; +use iggy_binary_protocol::requests::topics::{ + CreateTopicRequest, CreateTopicWithAssignmentsRequest, +}; +use iggy_binary_protocol::requests::users::CreateUserRequest; +use iggy_binary_protocol::requests::users::login_register::RegisterRequest; +use iggy_binary_protocol::{ + Command, Operation, PrepareHeader, WireEncode, WireIdentifier, WireName, WireOptions, +}; +use iggy_common::{ + MetadataChangeGroup, MetadataChangeGroups, MetadataChangeStart, MetadataChangesQuery, + UserStatus, variadic, +}; +use metadata::change_log::{ + AppliedChange, ChangeLogCapacities, ChangeLogSnapshot, CurrentState, MetadataChangeLog, + change_page, current_page, +}; +use metadata::impls::metadata::StreamsFrontend; +use metadata::stm::stream::Streams; +use metadata::stm::user::{ROOT_USER_CREATED_AT_MICROS, Users}; +use metadata::{MuxStateMachine, apply_committed_prepare}; +use server_common::Message; +use server_common::iobuf::Owned; +use std::cell::RefCell; +use std::env; +use std::hint::black_box; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::thread; +use std::time::{Duration, Instant}; + +const HEADER_SIZE: usize = size_of::<PrepareHeader>(); +const ROOT_USER_ID: u32 = 0; +const ROOT_CLIENT: u128 = 7; +const DEFAULT_STREAMS: usize = 1_000_000; +const PAGE_LIMIT: u32 = 1000; +const READER_THREADS: usize = 3; +const APPENDS_UNDER_READERS: u64 = 20_000; +const SNAPSHOT_ROUNDS: u32 = 5; + +/// The metadata state machine of one replica, driven the way the commit path +/// drives it. +struct Replica { + mux: MuxStateMachine<variadic!(Users, Streams)>, + client_table: RefCell<ClientTable>, + change_log: MetadataChangeLog, + op: u64, +} + +impl Replica { + fn new() -> Self { + let users = Users::default(); + users.ensure_root_user("iggy", "hash"); + let streams = Streams::default(); + Self { + mux: MuxStateMachine::new(variadic!(users, streams)), + client_table: RefCell::new(ClientTable::new(CLIENTS_TABLE_MAX)), + change_log: MetadataChangeLog::new(ChangeLogCapacities::default()), + op: 0, + } + } + + fn commit( + &mut self, + client: u128, + operation: Operation, + request: &impl WireEncode, + ) -> Vec<AppliedChange> { + self.op += 1; + let body = request.to_bytes(); + let total = HEADER_SIZE + body.len(); + let mut buffer = Owned::<4096>::zeroed(total); + { + let header = bytemuck::checked::from_bytes_mut::<PrepareHeader>( + &mut buffer.as_mut_slice()[..HEADER_SIZE], + ); + header.command = Command::Prepare; + header.operation = operation; + header.op = self.op; + header.commit = self.op - 1; + header.timestamp = self.op * 1_000; + header.checksum = u128::from(self.op) << 64 | 1; + header.client = client; + header.user_id = ROOT_USER_ID; + header.request = if operation == Operation::Register { + 0 + } else { + self.op + }; + header.retry_capacity = u32::try_from(CLIENTS_TABLE_MAX).expect("capacity"); + header.size = u32::try_from(total).expect("size"); + } + buffer.as_mut_slice()[HEADER_SIZE..].copy_from_slice(&body); + let prepare: Message<PrepareHeader> = Message::try_from(buffer).expect("valid prepare"); + let header = *prepare.header(); + let changes = + apply_committed_prepare(&self.mux, &self.client_table, true, |_| {}, prepare).changes; + self.change_log + .append(&header, changes.clone()) + .expect("committed ops append in order"); + changes + } +} + +fn main() { + let streams = env::args() + .nth(1) + .and_then(|value| value.parse().ok()) + .unwrap_or(DEFAULT_STREAMS); + let mut replica = Replica::new(); + record_sizes(&mut replica); + build_catalog(&mut replica, streams); + current_paging(&replica, streams); + fill_activity_ring(&mut replica); + snapshot_round_trip(&replica); + publication_under_readers(&mut replica); +} + +fn record_sizes(replica: &mut Replica) { + println!("measurement, kind, value, unit"); + let register = replica.commit( + ROOT_CLIENT, + Operation::Register, + &RegisterRequest { + user_created_at: ROOT_USER_CREATED_AT_MICROS, + bind_verifier: [0x5a; 32], + }, + ); + let stream = replica.commit(ROOT_CLIENT, Operation::CreateStream, &create_stream(0)); + let topic = replica.commit( + ROOT_CLIENT, + Operation::CreateTopicWithAssignments, + &CreateTopicWithAssignmentsRequest { + request: CreateTopicRequest { + stream_id: WireIdentifier::numeric(0), + partitions_count: 3, + name: WireName::new("events").expect("name"), + options: WireOptions::empty(), + }, + derived_options: WireOptions::empty(), + partitions: (0..3) + .map(|partition_id| CreatedPartitionAssignment { + partition_id, + consensus_group_id: 1 + u64::from(partition_id), + initial_view: 0, + }) + .collect(), + }, + ); + let user = replica.commit( + ROOT_CLIENT, + Operation::CreateUser, + &CreateUserRequest { + username: WireName::new("operator").expect("name"), + password: "hash".to_owned(), + status: UserStatus::Active.as_code(), + permissions: None, + options: WireOptions::empty(), + }, + ); + for change in register.iter().chain(&stream).chain(&topic).chain(&user) { + println!( + "record_body, {:?}, {}, bytes", + change.kind, + change.body.len() + ); + } +} + +fn build_catalog(replica: &mut Replica, streams: usize) { + let start = Instant::now(); + for index in 1..streams { + black_box(replica.commit(ROOT_CLIENT, Operation::CreateStream, &create_stream(index))); + } + let elapsed = start.elapsed(); + println!("catalog, streams, {streams}, count"); + println!( + "apply_and_append, create_stream, {:.0}, ns_per_op", + elapsed.as_nanos() as f64 / streams as f64 + ); +} + +fn current_paging(replica: &Replica, streams: usize) { + let groups = MetadataChangeGroups::requested(&[MetadataChangeGroup::Streams]); + let head_op = replica.op; + let table = replica.client_table.borrow(); + let mut token = None; + let mut pages = 0u32; + let mut items = 0usize; + let mut slowest = Duration::ZERO; + let start = Instant::now(); + loop { + let page_start = Instant::now(); + let page = replica.mux.streams().read(|streams_inner| { + replica.mux.users().read(|users_inner| { + replica + .change_log + .read(|log| { + current_page( + log, + head_op, + groups, + token.as_ref(), + PAGE_LIMIT, + &CurrentState { + streams: streams_inner, + users: users_inner, + sessions: Some(&table), + }, + ) + }) + .expect("writer present") + .expect("current page") + }) + }); + slowest = slowest.max(page_start.elapsed()); + pages += 1; + items += page.items.len(); + match page.current_page_token { + Some(next) => token = Some(next), + None => break, + } + } + let elapsed = start.elapsed(); + assert_eq!(items, streams); + println!("current_paging, pages_of_{PAGE_LIMIT}, {pages}, count"); + println!( + "current_paging, page_mean, {:.1}, us", + elapsed.as_secs_f64() * 1e6 / f64::from(pages) + ); + println!( + "current_paging, page_max, {:.1}, us", + slowest.as_secs_f64() * 1e6 + ); + println!("current_paging, full_scan, {:.3}, s", elapsed.as_secs_f64()); +} + +/// Session records until the activity ring holds as many as it keeps. +fn fill_activity_ring(replica: &mut Replica) { + let capacity = ChangeLogCapacities::default().activity; + for session in 0..u64::try_from(capacity).expect("capacity") { + replica.op += 1; + let header = PrepareHeader { + op: replica.op, + checksum: u128::from(replica.op) << 64 | 1, + timestamp: replica.op * 1_000, + ..PrepareHeader::default() + }; + let change = AppliedChange::new( + &WireMetadataChange::SessionRegistered(WireSessionBody { + user: WireUserKey { + user_id: ROOT_USER_ID, + user_created_at: 0, + }, + client_id: u128::from(session) + 1_000, + session, + }), + Some(ROOT_USER_ID), + ); + replica + .change_log + .append(&header, vec![change]) + .expect("monotone append"); + } +} + +fn snapshot_round_trip(replica: &Replica) { + let mut capture = Duration::ZERO; + let mut encode = Duration::ZERO; + let mut decode = Duration::ZERO; + let mut restore = Duration::ZERO; + let mut bytes = 0; + for _ in 0..SNAPSHOT_ROUNDS { + let start = Instant::now(); + let snapshot = replica.change_log.snapshot(); + capture += start.elapsed(); + + let start = Instant::now(); + let encoded = rmp_serde::to_vec(&snapshot).expect("encodes"); + encode += start.elapsed(); + bytes = encoded.len(); + + let start = Instant::now(); + let decoded: ChangeLogSnapshot = rmp_serde::from_slice(&encoded).expect("decodes"); + decode += start.elapsed(); + + let start = Instant::now(); + decoded.validate(replica.op).expect("valid snapshot"); + black_box( + decoded + .restore(ChangeLogCapacities::default()) + .expect("restores"), + ); + restore += start.elapsed(); + } + let mean = |total: Duration| total.as_secs_f64() * 1e3 / f64::from(SNAPSHOT_ROUNDS); + let capacities = ChangeLogCapacities::default(); + println!( + "full_ring_snapshot, records, {}, count", + capacities.resource + capacities.activity + ); + println!("full_ring_snapshot, encoded, {bytes}, bytes"); + println!("full_ring_snapshot, capture, {:.2}, ms", mean(capture)); + println!("full_ring_snapshot, encode, {:.2}, ms", mean(encode)); + println!("full_ring_snapshot, decode, {:.2}, ms", mean(decode)); + println!( + "full_ring_snapshot, validate_and_restore, {:.2}, ms", + mean(restore) + ); +} + +fn publication_under_readers(replica: &mut Replica) { + let stop = Arc::new(AtomicBool::new(false)); + let readers: Vec<_> = (0..READER_THREADS) + .map(|_| { + let factory = replica.change_log.factory(); + let stop = Arc::clone(&stop); + thread::spawn(move || { + let log = MetadataChangeLog::from_factory(factory); + let query = MetadataChangesQuery { + start: MetadataChangeStart::Earliest, + limit: PAGE_LIMIT, + ..MetadataChangesQuery::default() + }; + let groups = MetadataChangeGroups::requested(&[]); + let mut pages = 0u64; + while !stop.load(Ordering::Relaxed) { + black_box(log.read(|log| { + change_page(log, log.last_record_op(), &query, groups).expect("change page") + })); + pages += 1; + } + pages + }) + }) + .collect(); + let mut latencies = Vec::with_capacity(usize::try_from(APPENDS_UNDER_READERS).expect("count")); + for index in 0..APPENDS_UNDER_READERS { + replica.op += 1; + let header = PrepareHeader { + op: replica.op, + checksum: u128::from(replica.op) << 64 | 1, + timestamp: replica.op * 1_000, + ..PrepareHeader::default() + }; + let change = AppliedChange::new( + &WireMetadataChange::SessionRegistered(WireSessionBody { + user: WireUserKey { + user_id: ROOT_USER_ID, + user_created_at: 0, + }, + client_id: u128::from(index) + 1_000_000, + session: index, + }), + Some(ROOT_USER_ID), + ); + let start = Instant::now(); + replica + .change_log + .append(&header, vec![change]) + .expect("monotone append"); + latencies.push(start.elapsed()); + } + stop.store(true, Ordering::Relaxed); + let reader_pages: u64 = readers + .into_iter() + .map(|reader| reader.join().expect("reader thread")) + .sum(); + latencies.sort_unstable(); + let percentile = |fraction: f64| { + let index = ((latencies.len() - 1) as f64 * fraction) as usize; + latencies[index].as_secs_f64() * 1e6 + }; + println!("publication, readers, {READER_THREADS}, threads"); + println!("publication, reader_pages, {reader_pages}, count"); + println!("publication, append_p50, {:.1}, us", percentile(0.50)); + println!("publication, append_p99, {:.1}, us", percentile(0.99)); + println!("publication, append_max, {:.1}, us", percentile(1.0)); +} + +fn create_stream(index: usize) -> CreateStreamRequest { + CreateStreamRequest { + name: WireName::new(format!("stream-{index}")).expect("name"), + options: WireOptions::empty(), + } +}
diff --git a/core/metadata/src/change_log/current.rs b/core/metadata/src/change_log/current.rs new file mode 100644 index 0000000..7942ae0 --- /dev/null +++ b/core/metadata/src/change_log/current.rs
@@ -0,0 +1,618 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use crate::change_log::query::{CursorCheck, LogView, MAX_PAGE_BYTES, MAX_SCANNED_ENTRIES}; +use crate::change_log::store::ChangeLog; +use crate::change_log::{change_options, stream_body, user_body}; +use crate::stm::stream::StreamsInner; +use crate::stm::user::UsersInner; +use bytes::{BufMut, Bytes, BytesMut}; +use consensus::ClientTable; +use iggy_binary_protocol::WireName; +use iggy_binary_protocol::codec::{WireDecode, WireEncode, read_u32_le, read_u64_le}; +use iggy_binary_protocol::primitives::metadata_changes::{ + WireConsumerGroupItemBody, WireMemberItemBody, WireMetadataChangeGroup, + WireMetadataChangePhase, WireMetadataChangeStatus, WireMetadataCurrentItem, + WireMetadataCurrentPageToken, WirePartitionsItemBody, WireSessionBody, WireTokenBody, + WireTopicItemBody, WireUserKey, +}; +use iggy_binary_protocol::responses::system::{GetMetadataChangesResponse, WireMetadataItem}; +use iggy_common::{IggyError, MetadataChangeGroup, MetadataChangeGroups, OptionsScope}; +use std::cmp::Reverse; +use std::collections::BinaryHeap; +use std::ops::Bound; +use std::sync::Arc; + +/// A resume key id past every id of its level, since wire ids are `u32`. +const PAST_EVERY_ID: usize = u32::MAX as usize; + +/// The applied state a current page projects, all of one image. +pub struct CurrentState<'a> { + pub streams: &'a StreamsInner, + pub users: &'a UsersInner, + /// The client table aligned with the applied state, `None` while the + /// protection table runs ahead of it. + pub sessions: Option<&'a ClientTable>, +} + +/// One current state page: the projected items after the token's key, in +/// group order then key order. +/// +/// # Errors +/// `InvalidCommand` for a last key that does not fit its group, which the +/// token decode refuses first. +pub fn current_page( + log: &ChangeLog, + head_op: u64, + groups: MetadataChangeGroups, + token: Option<&WireMetadataCurrentPageToken>, + limit: u32, + state: &CurrentState<'_>, +) -> Result<GetMetadataChangesResponse, IggyError> { + current_page_within( + log, + head_op, + groups, + token, + limit, + state, + MAX_SCANNED_ENTRIES, + ) +} + +/// [`current_page`] that walks at most `scan_budget` streams and topics, so a +/// catalog with few items to emit still ends the page. +fn current_page_within( + log: &ChangeLog, + head_op: u64, + groups: MetadataChangeGroups, + token: Option<&WireMetadataCurrentPageToken>, + limit: u32, + state: &CurrentState<'_>, + scan_budget: usize, +) -> Result<GetMetadataChangesResponse, IggyError> { + let view = LogView::new(log, groups, head_op); + let answer = |status, next| GetMetadataChangesResponse { + phase: WireMetadataChangePhase::Current, + ..view.answer(status, next) + }; + let behind = || GetMetadataChangesResponse { + current_page_token: token.cloned(), + ..answer( + WireMetadataChangeStatus::Behind, + token.map(|token| token.anchor), + ) + }; + let Some(log_id) = view.log().log_id() else { + return Ok(behind()); + }; + let anchor = match token { + None => view.latest_cursor(log_id), + Some(token) if token.anchor.log_id != log_id => { + return Ok(answer(WireMetadataChangeStatus::Reset, None)); + } + Some(token) if view.head_op() < token.required_head => return Ok(behind()), + Some(token) => match view.check(token.anchor) { + CursorCheck::Valid(_) => token.anchor, + CursorCheck::Behind => return Ok(behind()), + CursorCheck::Expired => { + return Ok(answer(WireMetadataChangeStatus::Expired, None)); + } + CursorCheck::Reset => { + return Ok(answer(WireMetadataChangeStatus::Reset, None)); + } + }, + }; + if groups.contains(MetadataChangeGroup::Sessions) && state.sessions.is_none() { + return Ok(behind()); + } + + let mut projection = Projection { + state, + limit: limit as usize, + items: Vec::new(), + bytes: 0, + examined: 0, + scan_budget, + last: None, + more: false, + }; + let resume = token.map(|token| (MetadataChangeGroup::from(token.group), &token.last_key[..])); + for group in groups.iter() { + let after = match resume { + Some((resume_group, _)) if group < resume_group => continue, + Some((resume_group, last_key)) if group == resume_group => Some(last_key), + _ => None, + }; + if !projection.project(group, after)? { + break; + } + } + let current_page_token = + projection + .more + .then_some(projection.last) + .flatten() + .map(|(group, last_key)| WireMetadataCurrentPageToken { + anchor, + required_head: view.head_op(), + groups_mask: groups.mask(), + group, + last_key, + }); + Ok(GetMetadataChangesResponse { + has_more: current_page_token.is_some(), + items: projection.items, + current_page_token, + ..answer(WireMetadataChangeStatus::Ok, Some(anchor)) + }) +} + +struct Projection<'a> { + state: &'a CurrentState<'a>, + limit: usize, + items: Vec<WireMetadataItem>, + bytes: usize, + examined: usize, + scan_budget: usize, + last: Option<(WireMetadataChangeGroup, Bytes)>, + more: bool, +} + +impl Projection<'_> { + /// Projects `group` after `after`, `false` once the page is full. + #[allow(clippy::too_many_lines, clippy::cast_possible_truncation)] + fn project( + &mut self, + group: MetadataChangeGroup, + after: Option<&[u8]>, + ) -> Result<bool, IggyError> { + let streams = self.state.streams; + let users = self.state.users; + let key = |index: usize| -> Result<Option<usize>, IggyError> { + after + .map(|key| { + read_u32_le(key, index * 4) + .map(|id| id as usize) + .map_err(|_| IggyError::InvalidCommand) + }) + .transpose() + }; + match group { + MetadataChangeGroup::Streams => { + let from = key(0)?.map_or(0, |stream_id| stream_id + 1); + for (stream_id, stream) in streams.items.iter_from(from) { + let Some(body) = stream_body(stream) else { + continue; + }; + if !self.push(group, &[stream_id], &WireMetadataCurrentItem::Stream(body)) { + return Ok(false); + } + } + } + MetadataChangeGroup::Topics | MetadataChangeGroup::Partitions => { + let (after_stream, after_topic) = (key(0)?, key(1)?); + for (stream_id, stream) in streams.items.iter_from(after_stream.unwrap_or(0)) { + let from = after_topic + .filter(|_| after_stream == Some(stream_id)) + .map_or(0, |topic_id| topic_id + 1); + for (topic_id, topic) in stream.topics.iter_from(from) { + let key = stream.topic_key(topic); + let item = if group == MetadataChangeGroup::Topics { + let Ok(name) = WireName::new(topic.name.as_ref()) else { + continue; + }; + WireMetadataCurrentItem::Topic(WireTopicItemBody { + key, + created_at: topic.created_at.as_micros(), + name, + options: change_options(OptionsScope::Topic, &topic.options), + }) + } else { + WireMetadataCurrentItem::Partitions(WirePartitionsItemBody { + topic: key, + partitions_count: topic.partitions.len() as u32, + }) + }; + if !self.push(group, &[stream_id, topic_id], &item) { + return Ok(false); + } + } + if !self.examine(group, &[stream_id, PAST_EVERY_ID]) { + return Ok(false); + } + } + } + MetadataChangeGroup::ConsumerGroups | MetadataChangeGroup::ConsumerGroupMembers => { + let (after_stream, after_topic, after_group) = (key(0)?, key(1)?, key(2)?); + let (after_member, width) = if group == MetadataChangeGroup::ConsumerGroupMembers { + (key(3)?, 4) + } else { + (None, 3) + }; + for (stream_id, stream) in streams.items.iter_from(after_stream.unwrap_or(0)) { + let same_stream = after_stream == Some(stream_id); + let topic_from = after_topic.filter(|_| same_stream).unwrap_or(0); + for (topic_id, topic) in stream.topics.iter_from(topic_from) { + let same_topic = same_stream && after_topic == Some(topic_id); + // Group ids are monotone and bounded per topic. + let mut group_ids: Vec<u64> = topic + .consumer_groups + .keys() + .copied() + .filter(|group_id| { + after_group.filter(|_| same_topic).is_none_or(|after| { + let after = after as u64; + *group_id > after + || (group == MetadataChangeGroup::ConsumerGroupMembers + && *group_id == after) + }) + }) + .collect(); + group_ids.sort_unstable(); + for group_id in group_ids { + let Some(consumer_group) = topic.consumer_groups.get(&group_id) else { + continue; + }; + let wire_group_id = group_id as u32; + if group == MetadataChangeGroup::ConsumerGroups { + let Ok(name) = WireName::new(consumer_group.name.as_ref()) else { + continue; + }; + let item = WireMetadataCurrentItem::ConsumerGroup( + WireConsumerGroupItemBody { + topic: stream.topic_key(topic), + group_id: wire_group_id, + name, + }, + ); + if !self.push( + group, + &[stream_id, topic_id, group_id as usize], + &item, + ) { + return Ok(false); + } + continue; + } + let member_from = after_member + .filter(|_| same_topic && after_group == Some(group_id as usize)) + .map_or(0, |member_id| member_id + 1); + for (member_id, member) in consumer_group.members.iter_from(member_from) + { + let mut partition_ids: Vec<u32> = member + .partitions + .iter() + .map(|&partition_id| partition_id as u32) + .collect(); + partition_ids.sort_unstable(); + let item = WireMetadataCurrentItem::ConsumerGroupMember( + WireMemberItemBody { + topic: stream.topic_key(topic), + group_id: wire_group_id, + member_id: member_id as u32, + client_id: member.client_id, + partition_ids, + }, + ); + if !self.push( + group, + &[stream_id, topic_id, group_id as usize, member_id], + &item, + ) { + return Ok(false); + } + } + } + let past_topic = [stream_id, topic_id, PAST_EVERY_ID, PAST_EVERY_ID]; + if !self.examine(group, &past_topic[..width]) { + return Ok(false); + } + } + let past_stream = [stream_id, PAST_EVERY_ID, PAST_EVERY_ID, PAST_EVERY_ID]; + if !self.examine(group, &past_stream[..width]) { + return Ok(false); + } + } + } + MetadataChangeGroup::Users => { + let from = key(0)?.map_or(0, |user_id| user_id + 1); + for (user_id, user) in users.items.iter_from(from) { + let Some(body) = user_body(user) else { + continue; + }; + if !self.push(group, &[user_id], &WireMetadataCurrentItem::User(body)) { + return Ok(false); + } + } + } + MetadataChangeGroup::PersonalAccessTokens => { + let lower = match after { + Some(key) => { + let user_id = read_u32_le(key, 0).map_err(|_| IggyError::InvalidCommand)?; + let (name, _) = + WireName::decode(&key[4..]).map_err(|_| IggyError::InvalidCommand)?; + Bound::Excluded((user_id, Arc::from(name.as_str()))) + } + None => Bound::Unbounded, + }; + for (user_id, name) in users + .personal_access_token_name_index + .range((lower, Bound::Unbounded)) + { + let (Some(user), Some(token), Ok(wire_name)) = ( + users.items.get(*user_id as usize), + users + .personal_access_tokens + .get(user_id) + .and_then(|tokens| tokens.get(name)), + WireName::new(name.as_ref()), + ) else { + continue; + }; + let mut key = BytesMut::with_capacity(4 + wire_name.encoded_size()); + key.put_u32_le(*user_id); + wire_name.encode(&mut key); + let item = WireMetadataCurrentItem::PersonalAccessToken(WireTokenBody { + user: WireUserKey::from(user), + name: wire_name, + expiry: token.expiry_at.map_or(0, |expiry_at| expiry_at.as_micros()), + }); + if !self.push_keyed(group, key.freeze(), &item) { + return Ok(false); + } + } + } + MetadataChangeGroup::Sessions => { + let Some(table) = self.state.sessions else { + return Ok(true); + }; + let after = after + .map(|key| read_u64_le(key, 0).map_err(|_| IggyError::InvalidCommand)) + .transpose()?; + // A heap pops only what the page takes, a full sort would order every session. + let mut sessions: BinaryHeap<Reverse<(u64, u128, u32)>> = table + .active_sessions() + .filter(|(_, _, epoch)| after.is_none_or(|after| *epoch > after)) + .map(|(client_id, user_id, epoch)| Reverse((epoch, client_id, user_id))) + .collect(); + while let Some(Reverse((epoch, client_id, user_id))) = sessions.pop() { + let Some(user) = users.items.get(user_id as usize) else { + continue; + }; + let item = WireMetadataCurrentItem::Session(WireSessionBody { + user: WireUserKey::from(user), + client_id, + session: epoch, + }); + if !self.push_keyed(group, Bytes::copy_from_slice(&epoch.to_le_bytes()), &item) + { + return Ok(false); + } + } + } + } + Ok(true) + } + + /// Adds an item keyed by little-endian `u32` ids, `false` when the page + /// was already full. + fn push( + &mut self, + group: MetadataChangeGroup, + ids: &[usize], + item: &WireMetadataCurrentItem, + ) -> bool { + self.push_keyed(group, id_key(ids), item) + } + + /// Counts a walked stream or topic, `false` once the page walked its budget. The + /// next page then continues after `resume`. + fn examine(&mut self, group: MetadataChangeGroup, resume: &[usize]) -> bool { + self.examined += 1; + if self.examined < self.scan_budget { + return true; + } + self.more = true; + self.last = Some((WireMetadataChangeGroup::from(group), id_key(resume))); + false + } + + fn push_keyed( + &mut self, + group: MetadataChangeGroup, + key: Bytes, + item: &WireMetadataCurrentItem, + ) -> bool { + let group = WireMetadataChangeGroup::from(group); + let item = WireMetadataItem { + group: group.as_u8(), + body: item.body_bytes(), + }; + let bytes = self.bytes + item.encoded_size(); + if self.items.len() == self.limit || (!self.items.is_empty() && bytes > MAX_PAGE_BYTES) { + self.more = true; + return false; + } + self.bytes = bytes; + self.items.push(item); + self.last = Some((group, key)); + true + } +} + +/// A key of little-endian `u32` ids. +#[allow(clippy::cast_possible_truncation)] +fn id_key(ids: &[usize]) -> Bytes { + let mut key = BytesMut::with_capacity(ids.len() * 4); + for id in ids { + key.put_u32_le(*id as u32); + } + key.freeze() +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::change_log::{ChangeLogCapacities, MetadataChangeLog}; + use crate::stm::StateHandler; + use crate::stm::stream::StreamsInner; + use crate::stm::user::UsersInner; + use iggy_binary_protocol::primitives::metadata_changes::{WireStreamKey, WireTopicKey}; + use iggy_binary_protocol::primitives::partition_assignment::CreatedPartitionAssignment; + use iggy_binary_protocol::requests::consumer_groups::CreateConsumerGroupRequest; + use iggy_binary_protocol::requests::streams::CreateStreamRequest; + use iggy_binary_protocol::requests::topics::{ + CreateTopicRequest, CreateTopicWithAssignmentsRequest, + }; + use iggy_binary_protocol::{PrepareHeader, WireIdentifier, WireOptions}; + use iggy_common::IggyTimestamp; + + #[test] + fn given_sparse_catalog_when_paging_with_small_scan_budget_should_return_every_item_once() { + let mut streams = StreamsInner::new(); + let now = IggyTimestamp::now(); + for name in ["empty", "orders", "audit"] { + let request = CreateStreamRequest { + name: WireName::new(name).expect("name"), + options: WireOptions::empty(), + }; + assert_eq!(StateHandler::apply(&request, &mut streams, now).code, 0); + } + for (partition_group, name) in [(1, "events"), (2, "alerts")] { + let request = CreateTopicWithAssignmentsRequest { + request: CreateTopicRequest { + stream_id: WireIdentifier::numeric(1), + partitions_count: 1, + name: WireName::new(name).expect("name"), + options: WireOptions::empty(), + }, + derived_options: WireOptions::empty(), + partitions: vec![CreatedPartitionAssignment { + partition_id: 0, + consensus_group_id: partition_group, + initial_view: 0, + }], + }; + assert_eq!(StateHandler::apply(&request, &mut streams, now).code, 0); + } + let group = CreateConsumerGroupRequest { + stream_id: WireIdentifier::numeric(1), + topic_id: WireIdentifier::numeric(1), + name: WireName::new("shipping").expect("name"), + }; + assert_eq!(StateHandler::apply(&group, &mut streams, now).code, 0); + let users = UsersInner::new(); + let state = CurrentState { + streams: &streams, + users: &users, + sessions: None, + }; + let log = MetadataChangeLog::new(ChangeLogCapacities::default()); + let first_op = PrepareHeader { + op: 1, + checksum: 7, + ..PrepareHeader::default() + }; + log.append(&first_op, Vec::new()).expect("append"); + let groups = MetadataChangeGroups::requested(&[ + MetadataChangeGroup::Streams, + MetadataChangeGroup::Topics, + MetadataChangeGroup::Partitions, + MetadataChangeGroup::ConsumerGroups, + MetadataChangeGroup::ConsumerGroupMembers, + ]); + let page = |token: Option<&WireMetadataCurrentPageToken>, scan_budget| { + log.read(|log| current_page_within(log, 1, groups, token, 1000, &state, scan_budget)) + .expect("readable") + .expect("page") + }; + + let whole = page(None, usize::MAX); + let mut items = Vec::new(); + let mut token = None; + let mut pages = 0; + loop { + let answer = page(token.as_ref(), 2); + pages += 1; + items.extend(answer.items); + match answer.current_page_token { + Some(next) => token = Some(next), + None => break, + } + } + + assert_eq!(whole.current_page_token, None); + assert_eq!(whole.items.len(), 8); + assert!(pages > 1, "a scan budget of 2 ends a page early"); + assert_eq!(items, whole.items); + } + + #[test] + fn given_items_past_the_byte_budget_when_projecting_should_stop_before_the_overflow() { + let streams = StreamsInner::new(); + let users = UsersInner::new(); + let state = CurrentState { + streams: &streams, + users: &users, + sessions: None, + }; + let mut projection = Projection { + state: &state, + limit: 1000, + items: Vec::new(), + bytes: 0, + examined: 0, + scan_budget: MAX_SCANNED_ENTRIES, + last: None, + more: false, + }; + let member = |member_id: u32, partitions: usize| { + WireMetadataCurrentItem::ConsumerGroupMember(WireMemberItemBody { + topic: WireTopicKey { + stream: WireStreamKey { + stream_id: 0, + stream_created_op: 1, + }, + topic_id: 0, + topic_created_op: 2, + }, + group_id: 0, + member_id, + client_id: 7, + partition_ids: vec![0; partitions], + }) + }; + let group = MetadataChangeGroup::ConsumerGroupMembers; + + assert!( + projection.push(group, &[0, 0, 0, 0], &member(0, MAX_PAGE_BYTES / 4)), + "an item larger than the budget still makes progress" + ); + assert!(!projection.push(group, &[0, 0, 0, 1], &member(1, 0))); + + assert_eq!(projection.items.len(), 1); + assert!(projection.more); + assert_eq!( + projection.last, + Some(( + WireMetadataChangeGroup::ConsumerGroupMembers, + Bytes::from_static(&[0; 16]) + )) + ); + } +}
diff --git a/core/metadata/src/change_log/mod.rs b/core/metadata/src/change_log/mod.rs new file mode 100644 index 0000000..3a5a248 --- /dev/null +++ b/core/metadata/src/change_log/mod.rs
@@ -0,0 +1,304 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +mod current; +mod query; +mod snapshot; +mod store; + +pub use current::{CurrentState, current_page}; +pub use query::{MAX_SCANNED_ENTRIES, change_page}; + +pub use snapshot::{ChangeEntrySnapshot, ChangeLogSnapshot, RingSnapshot}; +pub use store::{ + ChangeEntry, ChangeLog, ChangeLogCapacities, ChangeLogCommand, ChangeLogError, + DEFAULT_ACTIVITY_CAPACITY, DEFAULT_RESOURCE_CAPACITY, MAX_CAPACITY, MetadataChangeLog, Ring, +}; + +use crate::stm::ApplyContext; +use crate::stm::stream::{Stream, StreamsInner, Topic}; +use crate::stm::user::User; +use bytes::Bytes; +use consensus::EXPIRED_SESSION_REQUEST_ID; +use iggy_binary_protocol::primitives::metadata_changes::{ + WireChangeOptions, WireMemberLeftBody, WireMetadataChange, WireMetadataChangeGroup, + WireMetadataChangeKind, WirePartitionHandedOverBody, WirePartitionsChangedBody, + WireSessionBody, WireSessionClosedBody, WireSessionClosedReason, WireStreamBody, WireStreamKey, + WireTopicBody, WireTopicKey, WireUserBody, WireUserKey, +}; +use iggy_binary_protocol::{PrepareHeader, WireName}; +use iggy_common::wire_conversions::{permissions_to_wire, resource_options_to_wire_split}; +use iggy_common::{OptionsScope, ResourceOptions, public_options}; + +/// One change a committed operation made to public metadata. +/// +/// The body is encoded once, at apply, from the public projection, so the log +/// never holds more than a reader may see. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct AppliedChange { + pub kind: WireMetadataChangeKind, + /// The requesting user, absent for a server action. + pub actor_user_id: Option<u32>, + pub body: Bytes, +} + +impl AppliedChange { + #[must_use] + pub fn new(change: &WireMetadataChange, actor_user_id: Option<u32>) -> Self { + Self { + kind: change.kind(), + actor_user_id, + body: change.body_bytes(), + } + } + + #[must_use] + pub const fn group(&self) -> WireMetadataChangeGroup { + self.kind.group() + } +} + +impl From<&Stream> for WireStreamKey { + #[allow(clippy::cast_possible_truncation)] + fn from(stream: &Stream) -> Self { + Self { + stream_id: stream.id as u32, + stream_created_op: stream.created_op, + } + } +} + +impl From<&User> for WireUserKey { + fn from(user: &User) -> Self { + Self { + user_id: user.id, + user_created_at: user.created_at.as_micros(), + } + } +} + +impl Stream { + /// The identity a record gives `topic`, one of this stream's topics. + #[allow(clippy::cast_possible_truncation)] + pub(crate) fn topic_key(&self, topic: &Topic) -> WireTopicKey { + WireTopicKey { + stream: WireStreamKey::from(self), + topic_id: topic.id as u32, + topic_created_op: topic.created_op, + } + } +} + +impl StreamsInner { + /// The identity a record gives the topic at these slab ids, if it exists. + pub(crate) fn topic_key(&self, stream_id: usize, topic_id: usize) -> Option<WireTopicKey> { + let stream = self.items.get(stream_id)?; + Some(stream.topic_key(stream.topics.get(topic_id)?)) + } + + /// The topic's partition count after this op, with the ids it removed. + pub(crate) fn partitions_changed( + &self, + stream_id: usize, + topic_id: usize, + removed_partition_ids: Vec<u32>, + requested_op: Option<u64>, + ) -> Option<WirePartitionsChangedBody> { + let stream = self.items.get(stream_id)?; + let topic = stream.topics.get(topic_id)?; + Some(WirePartitionsChangedBody { + topic: stream.topic_key(topic), + partitions_count: u32::try_from(topic.partitions.len()).ok()?, + removed_partition_ids, + requested_op, + }) + } + + /// `requested_op` and actor of a delete record: those of the intent when + /// this apply finishes a deferred delete, this operation's otherwise. + pub(crate) fn deletion_origin(&self) -> (Option<u64>, Option<u32>) { + self.finalizing_lifecycle.as_ref().map_or_else( + || (None, change_actor(&self.apply_context)), + |intent| (Some(intent.metadata_op), change_actor(intent)), + ) + } +} + +/// The user a record names as its actor. Absent for the root bootstrap, the +/// expiry cleaner, rewraps and partition completions, which commit under +/// client 0 or a server-internal client. +pub(crate) fn change_actor(context: &ApplyContext) -> Option<u32> { + (context.client_id != 0 && !message_bus::is_server_internal_client(context.client_id)) + .then_some(context.user_id) +} + +/// The record of a Register that inserted its client, with the epoch the +/// table gave it. +pub(crate) fn session_registered( + header: &PrepareHeader, + user_created_at: u64, + epoch: u64, +) -> AppliedChange { + AppliedChange::new( + &WireMetadataChange::SessionRegistered(WireSessionBody { + user: WireUserKey { + user_id: header.user_id, + user_created_at, + }, + client_id: header.client, + session: epoch, + }), + change_actor(&ApplyContext::from(header)), + ) +} + +/// The record of a Logout that ended its session. A lease expiry is a server +/// action, so it names no actor. +pub(crate) fn session_closed(header: &PrepareHeader, user: WireUserKey) -> AppliedChange { + let (reason, actor) = if header.request == EXPIRED_SESSION_REQUEST_ID { + (WireSessionClosedReason::Expired, None) + } else { + ( + WireSessionClosedReason::Logout, + change_actor(&ApplyContext::from(header)), + ) + }; + AppliedChange::new( + &WireMetadataChange::SessionClosed(WireSessionClosedBody { + user, + client_id: header.client, + session: header.session, + reason, + }), + actor, + ) +} + +/// The record of a session that a user's deactivation or deletion ended. +pub(crate) fn session_revoked( + user: WireUserKey, + client_id: u128, + session: u64, + actor_user_id: Option<u32>, +) -> AppliedChange { + AppliedChange::new( + &WireMetadataChange::SessionClosed(WireSessionClosedBody { + user, + client_id, + session, + reason: WireSessionClosedReason::Revoked, + }), + actor_user_id, + ) +} + +/// The public projection of `stored`, or the unreadable marker for critical +/// state this build cannot interpret. Every replica reaches the same verdict. +pub(crate) fn change_options(scope: OptionsScope, stored: &ResourceOptions) -> WireChangeOptions { + public_options(scope, stored) + .and_then(|public| resource_options_to_wire_split(&public)) + .map_or(WireChangeOptions::Unreadable, |(explicit, derived)| { + WireChangeOptions::Readable { explicit, derived } + }) +} + +/// `None` only for a stored name no wire name can carry, which create and +/// rename refuse, so every replica skips the same record. +pub(crate) fn stream_body(stream: &Stream) -> Option<WireStreamBody> { + Some(WireStreamBody { + key: WireStreamKey::from(stream), + created_at: stream.created_at.as_micros(), + name: WireName::new(stream.name.as_ref()).ok()?, + options: change_options(OptionsScope::Stream, &stream.options), + }) +} + +pub(crate) fn topic_body(stream: &Stream, topic: &Topic) -> Option<WireTopicBody> { + Some(WireTopicBody { + key: stream.topic_key(topic), + created_at: topic.created_at.as_micros(), + name: WireName::new(topic.name.as_ref()).ok()?, + partitions_count: u32::try_from(topic.partitions.len()).ok()?, + options: change_options(OptionsScope::Topic, &topic.options), + }) +} + +pub(crate) fn user_body(user: &User) -> Option<WireUserBody> { + Some(WireUserBody { + key: WireUserKey::from(user), + username: WireName::new(user.username.as_ref()).ok()?, + status: user.status.as_code(), + permissions: user.permissions.as_deref().map(permissions_to_wire), + options: change_options(OptionsScope::User, &user.options), + }) +} + +/// The handovers a member's session change makes: its installed partitions +/// no longer match it, so each one it lost has no installed member owner, and +/// each one it gained has it. Ascending by partition id. +#[allow(clippy::cast_possible_truncation)] +pub(crate) fn installed_partition_changes( + topic: WireTopicKey, + group_id: u32, + member_id: u32, + before: &[usize], + after: &[usize], +) -> Vec<WirePartitionHandedOverBody> { + let mut changes: Vec<WirePartitionHandedOverBody> = before + .iter() + .filter(|partition_id| !after.contains(partition_id)) + .map(|&partition_id| (partition_id, None)) + .chain( + after + .iter() + .filter(|partition_id| !before.contains(partition_id)) + .map(|&partition_id| (partition_id, Some(member_id))), + ) + .map(|(partition_id, member_id)| WirePartitionHandedOverBody { + topic, + group_id, + partition_id: partition_id as u32, + member_id, + }) + .collect(); + changes.sort_unstable_by_key(|change| change.partition_id); + changes +} + +/// Membership records of one operation by stream, topic, group, member and +/// client. Slot and hash map order differ per replica, the records may not. +pub(crate) fn sort_member_records(records: &mut [WireMemberLeftBody]) { + let key = |record: &WireMemberLeftBody| { + ( + record.topic.stream.stream_id, + record.topic.topic_id, + record.group_id, + record.member_id, + record.client_id, + ) + }; + records.sort_unstable_by_key(key); + debug_assert!( + records + .windows(2) + .all(|pair| key(&pair[0]) != key(&pair[1])), + "one operation removed the same member twice" + ); +} + +#[cfg(test)] +mod tests;
diff --git a/core/metadata/src/change_log/query.rs b/core/metadata/src/change_log/query.rs new file mode 100644 index 0000000..f6dc1f3 --- /dev/null +++ b/core/metadata/src/change_log/query.rs
@@ -0,0 +1,682 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use crate::change_log::store::{ChangeEntry, ChangeLog, Ring}; +use iggy_binary_protocol::WireEncode; +use iggy_binary_protocol::primitives::metadata_changes::{ + WireMetadataChangeCursor, WireMetadataChangePhase, WireMetadataChangeStatus, +}; +use iggy_binary_protocol::responses::system::{ + GetMetadataChangesResponse, WireMetadataChangeRecord, +}; +use iggy_common::{ + IggyError, MetadataChangeClass, MetadataChangeGroup, MetadataChangeGroups, MetadataChangeStart, + MetadataChangesQuery, +}; + +/// Entries one change page examines at most, returned or skipped. +pub const MAX_SCANNED_ENTRIES: usize = 65_536; + +/// Encoded record or item bytes one page carries at most, unless its first alone is larger. +pub const MAX_PAGE_BYTES: usize = 1024 * 1024; + +const CLASSES: [MetadataChangeClass; 2] = + [MetadataChangeClass::Resource, MetadataChangeClass::Activity]; + +/// One change page for a validated query, from a log image that holds every +/// record of the ops up to `head_op` and none above. +/// +/// # Errors +/// `InvalidCommand` for a current state start, which [`current_page`](super::current_page) serves. +pub fn change_page( + log: &ChangeLog, + head_op: u64, + query: &MetadataChangesQuery, + groups: MetadataChangeGroups, +) -> Result<GetMetadataChangesResponse, IggyError> { + let view = LogView::new(log, groups, head_op); + let supplied = match &query.start { + MetadataChangeStart::After(cursor) => Some(WireMetadataChangeCursor::from(*cursor)), + MetadataChangeStart::Earliest | MetadataChangeStart::Latest => None, + MetadataChangeStart::Current(_) => return Err(IggyError::InvalidCommand), + }; + let Some(log_id) = log.log_id() else { + return Ok(view.answer(WireMetadataChangeStatus::Behind, supplied)); + }; + let latest = view.latest_cursor(log_id); + let start = match &query.start { + MetadataChangeStart::Latest => { + return Ok(view.answer(WireMetadataChangeStatus::Ok, Some(latest))); + } + // A partly retained floor operation is skipped whole. + MetadataChangeStart::Earliest => (view.floor().0, u32::MAX), + MetadataChangeStart::After(cursor) => { + match view.check(WireMetadataChangeCursor::from(*cursor)) { + CursorCheck::Valid(position) => position, + CursorCheck::Behind => { + return Ok(view.answer(WireMetadataChangeStatus::Behind, supplied)); + } + CursorCheck::Expired => { + return Ok(view.answer(WireMetadataChangeStatus::Expired, None)); + } + CursorCheck::Reset => { + return Ok(view.answer(WireMetadataChangeStatus::Reset, None)); + } + } + } + MetadataChangeStart::Current(_) => return Err(IggyError::InvalidCommand), + }; + let page = view.scan( + start, + query.since.map(|since| since.as_micros()), + query.until.map(|until| until.as_micros()), + query.limit as usize, + ); + let next = page + .last_examined + .map_or_else(|| supplied.unwrap_or(latest), |entry| entry.cursor(log_id)); + Ok(GetMetadataChangesResponse { + next: Some(next), + has_more: page.has_more, + records: page.records, + ..view.answer(WireMetadataChangeStatus::Ok, None) + }) +} + +/// A log image read for one selection of groups. +pub struct LogView<'a> { + log: &'a ChangeLog, + groups: MetadataChangeGroups, + head_op: u64, +} + +/// What the cursor rules decide for a supplied position. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum CursorCheck { + /// Serve the entries after this position. + Valid((u64, u32)), + Behind, + Expired, + Reset, +} + +struct ScannedPage<'a> { + records: Vec<WireMetadataChangeRecord>, + last_examined: Option<&'a ChangeEntry>, + has_more: bool, +} + +impl<'a> LogView<'a> { + pub const fn new(log: &'a ChangeLog, groups: MetadataChangeGroups, head_op: u64) -> Self { + Self { + log, + groups, + head_op, + } + } + + pub const fn head_op(&self) -> u64 { + self.head_op + } + + pub const fn log(&self) -> &'a ChangeLog { + self.log + } + + /// The rings of the requested classes. + fn rings(&self) -> impl Iterator<Item = &'a Ring> + '_ { + CLASSES + .into_iter() + .filter(|class| self.groups.contains_class(*class)) + .map(|class| self.log.ring(class)) + } + + /// The newest evicted position of the requested classes. + pub fn floor(&self) -> (u64, u32) { + self.rings() + .map(Ring::evicted_through) + .max() + .unwrap_or((0, 0)) + } + + fn retained_since(&self) -> u64 { + self.rings() + .map(Ring::evicted_through_timestamp) + .max() + .unwrap_or(0) + } + + /// Cursor at the newest record of the requested classes, or the zero + /// cursor while they hold none. + pub fn latest_cursor(&self, log_id: u128) -> WireMetadataChangeCursor { + self.rings() + .filter_map(|ring| ring.entries().back()) + .max_by_key(|entry| entry.position()) + .map_or_else( + || WireMetadataChangeCursor::zero(log_id), + |entry| entry.cursor(log_id), + ) + } + + /// Cursor rules 2 to 6. Rule 1, an unknown log id, is the caller's. + pub fn check(&self, cursor: WireMetadataChangeCursor) -> CursorCheck { + if self.log.log_id() != Some(cursor.log_id) { + return CursorCheck::Reset; + } + let floor = self.floor(); + if cursor.op == 0 { + return if floor == (0, 0) { + CursorCheck::Valid((0, 0)) + } else { + CursorCheck::Expired + }; + } + if cursor.op > self.head_op { + return CursorCheck::Behind; + } + let position = (cursor.op, cursor.ordinal); + let found = CLASSES.into_iter().find_map(|class| { + let entries = self.log.ring(class).entries(); + let index = entries.partition_point(|entry| entry.position() < position); + entries + .get(index) + .filter(|entry| entry.position() == position) + }); + match found { + Some(entry) if entry.checksum != cursor.checksum => CursorCheck::Reset, + Some(_) if position <= floor => CursorCheck::Expired, + Some(_) => CursorCheck::Valid(position), + None => { + let highest_floor = CLASSES + .into_iter() + .map(|class| self.log.ring(class).evicted_through()) + .max() + .unwrap_or((0, 0)); + if position <= highest_floor { + CursorCheck::Expired + } else { + CursorCheck::Reset + } + } + } + } + + /// An answer without records or items, `next` as given. + pub fn answer( + &self, + status: WireMetadataChangeStatus, + next: Option<WireMetadataChangeCursor>, + ) -> GetMetadataChangesResponse { + GetMetadataChangesResponse { + status, + phase: WireMetadataChangePhase::Changes, + log_id: self.log.log_id(), + oldest_op: self.floor().0.saturating_add(1), + retained_since: self.retained_since(), + head_op: self.head_op, + next, + has_more: false, + records: Vec::new(), + items: Vec::new(), + current_page_token: None, + } + } + + /// Merges the requested rings after `start` by position. The time bounds + /// work as op bounds, since timestamps never decrease with the op. + fn scan( + &self, + start: (u64, u32), + since: Option<u64>, + until: Option<u64>, + limit: usize, + ) -> ScannedPage<'a> { + let mut heads: Vec<(&'a Ring, usize)> = self + .rings() + .map(|ring| { + let entries = ring.entries(); + let after = entries.partition_point(|entry| entry.position() <= start); + let since = since.map_or(0, |since| { + entries.partition_point(|entry| entry.timestamp < since) + }); + (ring, after.max(since)) + }) + .collect(); + let in_range = |entry: &ChangeEntry| until.is_none_or(|until| entry.timestamp <= until); + let next_entry = |heads: &[(&'a Ring, usize)]| { + heads + .iter() + .enumerate() + .filter_map(|(head, (ring, index))| { + ring.entries().get(*index).map(|entry| (head, &**entry)) + }) + .min_by_key(|(_, entry)| entry.position()) + .filter(|(_, entry)| in_range(entry)) + }; + let mut page = ScannedPage { + records: Vec::new(), + last_examined: None, + has_more: false, + }; + let mut examined = 0; + let mut bytes = 0; + while let Some((head, entry)) = next_entry(&heads) { + let record = self + .groups + .contains(MetadataChangeGroup::from(entry.group())) + .then(|| WireMetadataChangeRecord::from(entry)); + let size = record.as_ref().map_or(0, WireEncode::encoded_size); + if examined == MAX_SCANNED_ENTRIES + || page.records.len() == limit + || (!page.records.is_empty() && bytes + size > MAX_PAGE_BYTES) + { + page.has_more = true; + break; + } + examined += 1; + heads[head].1 += 1; + page.last_examined = Some(entry); + if let Some(record) = record { + bytes += size; + page.records.push(record); + } + } + page + } +} + +impl From<&ChangeEntry> for WireMetadataChangeRecord { + fn from(entry: &ChangeEntry) -> Self { + Self { + op: entry.op, + ordinal: entry.ordinal, + checksum: entry.checksum, + timestamp: entry.timestamp, + actor_user_id: entry.actor_user_id, + group: entry.group().as_u8(), + kind: entry.kind.as_u8(), + body: entry.body.clone(), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::change_log::AppliedChange; + use crate::change_log::store::{ChangeLogCapacities, MetadataChangeLog}; + use bytes::Bytes; + use iggy_binary_protocol::PrepareHeader; + use iggy_binary_protocol::primitives::metadata_changes::WireMetadataChangeKind as Kind; + use iggy_common::{IggyTimestamp, MetadataChangeCursor}; + + const LOG_ID: u128 = 0x0d4b; + const NO_RECORDS: [WireMetadataChangeRecord; 0] = []; + + #[test] + fn given_unknown_log_id_when_reading_any_change_start_should_answer_behind() { + let log = MetadataChangeLog::new(ChangeLogCapacities::default()); + let cursor = MetadataChangeCursor { + log_id: LOG_ID, + op: 3, + checksum: 3, + ordinal: 0, + }; + + for (start, next) in [ + (MetadataChangeStart::Earliest, None), + (MetadataChangeStart::Latest, None), + ( + MetadataChangeStart::After(cursor), + Some(WireMetadataChangeCursor::from(cursor)), + ), + ] { + let page = page(&log, 0, start, &[], 10); + assert_eq!(page.status, WireMetadataChangeStatus::Behind); + assert_eq!(page.log_id, None); + assert_eq!(page.next, next); + } + } + + #[test] + fn given_activity_only_reader_when_resource_ring_evicts_should_keep_zero_cursor_valid_until_activity_evicts() + { + let log = MetadataChangeLog::new(ChangeLogCapacities { + resource: 1, + activity: 1, + }); + append(&log, 1, &[Kind::UserCreated]); + append(&log, 2, &[Kind::StreamCreated]); + let sessions = [MetadataChangeGroup::Sessions]; + + let latest = page(&log, 2, MetadataChangeStart::Latest, &sessions, 10); + assert_eq!(latest.status, WireMetadataChangeStatus::Ok); + assert_eq!(latest.next, Some(WireMetadataChangeCursor::zero(LOG_ID))); + let zero = MetadataChangeStart::After(MetadataChangeCursor::zero(LOG_ID)); + assert_eq!( + page(&log, 2, zero.clone(), &sessions, 10).status, + WireMetadataChangeStatus::Ok, + "the resource eviction is outside the requested classes" + ); + + append(&log, 3, &[Kind::SessionRegistered]); + let first = page(&log, 3, zero.clone(), &sessions, 10); + assert_eq!(positions(&first), [(3, 0)]); + assert_eq!(first.next.map(|next| (next.op, next.ordinal)), Some((3, 0))); + + append(&log, 4, &[Kind::SessionClosed]); + assert_eq!( + page(&log, 4, zero, &sessions, 10).status, + WireMetadataChangeStatus::Expired + ); + } + + #[test] + fn given_each_cursor_rule_when_checking_should_answer_its_status() { + let log = MetadataChangeLog::new(ChangeLogCapacities { + resource: 2, + activity: 4, + }); + for op in 1..=4 { + append(&log, op, &[Kind::StreamCreated]); + } + let cursor = |log_id, op: u64, checksum: u128, ordinal| { + MetadataChangeStart::After(MetadataChangeCursor { + log_id, + op, + checksum, + ordinal, + }) + }; + + for (start, status) in [ + (cursor(LOG_ID + 1, 4, 4, 0), WireMetadataChangeStatus::Reset), + (cursor(LOG_ID, 0, 0, 0), WireMetadataChangeStatus::Expired), + (cursor(LOG_ID, 9, 9, 0), WireMetadataChangeStatus::Behind), + (cursor(LOG_ID, 4, 5, 0), WireMetadataChangeStatus::Reset), + (cursor(LOG_ID, 2, 2, 0), WireMetadataChangeStatus::Expired), + (cursor(LOG_ID, 1, 1, 0), WireMetadataChangeStatus::Expired), + (cursor(LOG_ID, 3, 3, 7), WireMetadataChangeStatus::Reset), + (cursor(LOG_ID, 3, 3, 0), WireMetadataChangeStatus::Ok), + ] { + let answer = page(&log, 4, start.clone(), &[], 10); + assert_eq!(answer.status, status, "{start:?}"); + match (status, &start) { + (WireMetadataChangeStatus::Ok, _) => {} + (WireMetadataChangeStatus::Behind, MetadataChangeStart::After(supplied)) => { + assert_eq!(answer.next, Some(WireMetadataChangeCursor::from(*supplied))); + assert_eq!(answer.records, NO_RECORDS); + } + _ => { + assert_eq!(answer.next, None, "{start:?}"); + assert_eq!(answer.records, NO_RECORDS); + } + } + } + } + + #[test] + fn given_limit_inside_one_operation_when_paging_should_continue_at_the_next_ordinal() { + let log = MetadataChangeLog::new(ChangeLogCapacities::default()); + append( + &log, + 1, + &[ + Kind::StreamDeleted, + Kind::UserPermissionsUpdated, + Kind::UserPermissionsUpdated, + ], + ); + + let first = page(&log, 1, MetadataChangeStart::Earliest, &[], 2); + assert_eq!(positions(&first), [(1, 0), (1, 1)]); + assert!(first.has_more); + let next = first.next.expect("ok pages carry a cursor"); + let second = page( + &log, + 1, + MetadataChangeStart::After(MetadataChangeCursor::from(next)), + &[], + 2, + ); + + assert_eq!(positions(&second), [(1, 2)]); + assert!(!second.has_more); + } + + #[test] + fn given_unrequested_groups_when_paging_should_skip_them_and_still_advance() { + let log = MetadataChangeLog::new(ChangeLogCapacities::default()); + append(&log, 1, &[Kind::StreamCreated, Kind::UserCreated]); + append(&log, 2, &[Kind::StreamCreated]); + + let users = page( + &log, + 2, + MetadataChangeStart::Earliest, + &[MetadataChangeGroup::Users], + 10, + ); + + assert_eq!(positions(&users), [(1, 1)]); + assert_eq!( + users.next.map(|next| (next.op, next.ordinal)), + Some((2, 0)), + "next names the last examined entry, filtered or not" + ); + assert!(!users.has_more); + } + + #[test] + fn given_more_entries_than_the_scan_budget_when_paging_should_stop_with_more_to_examine() { + let log = MetadataChangeLog::new(ChangeLogCapacities { + resource: MAX_SCANNED_ENTRIES + 10, + activity: 1, + }); + for op in 1..=(MAX_SCANNED_ENTRIES as u64 + 2) { + append(&log, op, &[Kind::StreamCreated]); + } + + let users = page( + &log, + MAX_SCANNED_ENTRIES as u64 + 2, + MetadataChangeStart::Earliest, + &[MetadataChangeGroup::Users], + 1000, + ); + + assert_eq!(users.records, NO_RECORDS); + assert!(users.has_more); + assert_eq!( + users.next.map(|next| next.op), + Some(MAX_SCANNED_ENTRIES as u64) + ); + } + + #[test] + fn given_records_past_the_byte_budget_when_paging_should_stop_before_the_overflow() { + let log = MetadataChangeLog::new(ChangeLogCapacities::default()); + append_body(&log, 1, Bytes::from(vec![0; MAX_PAGE_BYTES])); + append_body(&log, 2, Bytes::from(vec![0; MAX_PAGE_BYTES / 2])); + append_body(&log, 3, Bytes::from(vec![0; MAX_PAGE_BYTES / 2])); + + let mut pages = Vec::new(); + let mut start = MetadataChangeStart::Earliest; + loop { + let answer = page(&log, 3, start, &[], 10); + let next = answer.next.expect("ok pages carry a cursor"); + start = MetadataChangeStart::After(MetadataChangeCursor::from(next)); + pages.push(positions(&answer)); + if !answer.has_more { + break; + } + } + + assert_eq!( + pages, + [vec![(1, 0)], vec![(2, 0)], vec![(3, 0)]], + "a record larger than the budget still makes progress" + ); + } + + #[test] + fn given_time_bounds_when_paging_should_keep_the_inclusive_range_with_saturated_timestamps() { + let log = MetadataChangeLog::new(ChangeLogCapacities::default()); + // A runaway clock saturates: ops 3 to 5 share one timestamp. + for (op, timestamp) in [(1, 10), (2, 20), (3, 30), (4, 30), (5, 30), (6, 60)] { + append_at(&log, op, timestamp, &[Kind::StreamCreated]); + } + let bounded = |start, since: Option<u64>, until: Option<u64>| { + let query = MetadataChangesQuery { + start, + since: since.map(IggyTimestamp::from), + until: until.map(IggyTimestamp::from), + limit: 10, + ..MetadataChangesQuery::default() + }; + log.read(|log| change_page(log, 6, &query, MetadataChangeGroups::all())) + .expect("readable") + .expect("change start") + }; + let after_two = MetadataChangeStart::After(MetadataChangeCursor { + log_id: LOG_ID, + op: 2, + checksum: 2, + ordinal: 0, + }); + + let ops = |answer: &GetMetadataChangesResponse| { + answer + .records + .iter() + .map(|record| record.op) + .collect::<Vec<_>>() + }; + assert_eq!( + ops(&bounded(MetadataChangeStart::Earliest, Some(30), None)), + [3, 4, 5, 6] + ); + assert_eq!( + ops(&bounded(MetadataChangeStart::Earliest, None, Some(30))), + [1, 2, 3, 4, 5] + ); + assert_eq!( + ops(&bounded(MetadataChangeStart::Earliest, Some(20), Some(30))), + [2, 3, 4, 5] + ); + let after = bounded(after_two.clone(), Some(10), Some(30)); + assert_eq!(ops(&after), [3, 4, 5]); + assert!(!after.has_more, "op 6 is outside the range"); + assert_eq!(ops(&bounded(after_two, Some(60), None)), [6]); + } + + #[test] + fn given_evictions_when_answering_should_report_oldest_op_and_retained_since() { + let log = MetadataChangeLog::new(ChangeLogCapacities { + resource: 2, + activity: 1, + }); + append_at(&log, 1, 100, &[Kind::StreamCreated, Kind::StreamCreated]); + append_at( + &log, + 2, + 200, + &[Kind::StreamCreated, Kind::SessionRegistered], + ); + + let resource = page( + &log, + 2, + MetadataChangeStart::Earliest, + &[MetadataChangeGroup::Streams], + 10, + ); + let all = page(&log, 2, MetadataChangeStart::Earliest, &[], 10); + + assert_eq!((resource.oldest_op, resource.retained_since), (2, 100)); + assert_eq!( + positions(&resource), + [(2, 0)], + "earliest skips the partly retained floor operation" + ); + assert_eq!((all.oldest_op, all.retained_since), (2, 100)); + assert_eq!(positions(&all), [(2, 0), (2, 1)]); + } + + fn page( + log: &MetadataChangeLog, + head_op: u64, + start: MetadataChangeStart, + groups: &[MetadataChangeGroup], + limit: u32, + ) -> GetMetadataChangesResponse { + let query = MetadataChangesQuery { + start, + groups: groups.to_vec(), + limit, + ..MetadataChangesQuery::default() + }; + let groups = query.validate().expect("valid query"); + log.read(|log| change_page(log, head_op, &query, groups).expect("change start")) + .expect("readable") + } + + fn positions(answer: &GetMetadataChangesResponse) -> Vec<(u64, u32)> { + answer + .records + .iter() + .map(|record| (record.op, record.ordinal)) + .collect() + } + + fn append(log: &MetadataChangeLog, op: u64, kinds: &[Kind]) { + append_at(log, op, op, kinds); + } + + fn append_body(log: &MetadataChangeLog, op: u64, body: Bytes) { + let header = PrepareHeader { + op, + checksum: if op == 1 { LOG_ID } else { u128::from(op) }, + timestamp: op, + ..PrepareHeader::default() + }; + let change = AppliedChange { + kind: Kind::StreamCreated, + actor_user_id: None, + body, + }; + log.append(&header, vec![change]).expect("append"); + } + + fn append_at(log: &MetadataChangeLog, op: u64, timestamp: u64, kinds: &[Kind]) { + let header = PrepareHeader { + op, + checksum: if op == 1 { LOG_ID } else { u128::from(op) }, + timestamp, + ..PrepareHeader::default() + }; + let changes = kinds + .iter() + .map(|&kind| AppliedChange { + kind, + actor_user_id: None, + body: Bytes::new(), + }) + .collect(); + log.append(&header, changes).expect("append"); + } +}
diff --git a/core/metadata/src/change_log/snapshot.rs b/core/metadata/src/change_log/snapshot.rs new file mode 100644 index 0000000..ae8f516 --- /dev/null +++ b/core/metadata/src/change_log/snapshot.rs
@@ -0,0 +1,471 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use crate::change_log::store::{ChangeEntry, ChangeLog, ChangeLogCapacities, MAX_CAPACITY, Ring}; +use crate::stm::snapshot::SnapshotError; +use bytes::Bytes; +use iggy_binary_protocol::primitives::metadata_changes::WireMetadataChangeKind; +use iggy_common::{MetadataChangeClass, MetadataChangeGroup}; +use serde::{Deserialize, Serialize}; +use std::collections::{BTreeMap, VecDeque}; +use std::sync::Arc; + +/// The change log as the metadata snapshot stores it. Entries keep their +/// encoded bodies and ring order, so identical logs encode identically. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct ChangeLogSnapshot { + pub log_id: Option<u128>, + pub last_record_op: u64, + pub resource: RingSnapshot, + pub activity: RingSnapshot, +} + +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct RingSnapshot { + pub evicted_through_op: u64, + pub evicted_through_ordinal: u32, + pub evicted_through_timestamp: u64, + pub entries: Vec<ChangeEntrySnapshot>, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct ChangeEntrySnapshot { + pub op: u64, + pub ordinal: u32, + pub checksum: u128, + pub timestamp: u64, + pub actor_user_id: Option<u32>, + pub group: u8, + pub kind: u8, + /// A msgpack blob: the default sequence form spends two bytes on every + /// byte at or above 0x80. + #[serde(with = "body_bytes")] + pub body: Bytes, +} + +impl ChangeLogSnapshot { + /// Checks the log against the snapshot it rides in. + /// + /// # Errors + /// [`SnapshotError::InvalidState`] naming the first rule the log breaks. + pub fn validate(&self, sequence_number: u64) -> Result<(), SnapshotError> { + if self.last_record_op > sequence_number { + return Err(SnapshotError::InvalidState( + "change log holds a record above the snapshot", + )); + } + let mut operations: BTreeMap<u64, (u128, u64)> = BTreeMap::new(); + let mut positions = Vec::new(); + for (ring, class) in [ + (&self.resource, MetadataChangeClass::Resource), + (&self.activity, MetadataChangeClass::Activity), + ] { + if ring.entries.len() > MAX_CAPACITY { + return Err(SnapshotError::InvalidState( + "change log ring is larger than any capacity", + )); + } + let floor = (ring.evicted_through_op, ring.evicted_through_ordinal); + if floor.0 > self.last_record_op { + return Err(SnapshotError::InvalidState( + "change log floor is above its newest record", + )); + } + let mut previous = floor; + for entry in &ring.entries { + let kind = WireMetadataChangeKind::from_code(entry.kind).map_err(|_| { + SnapshotError::InvalidState("change log holds an unknown record kind") + })?; + if kind.group().as_u8() != entry.group + || MetadataChangeGroup::from(kind.group()).class() != class + { + return Err(SnapshotError::InvalidState( + "change log record sits in the wrong group or class", + )); + } + let position = (entry.op, entry.ordinal); + if position <= previous { + return Err(SnapshotError::InvalidState( + "change log ring is not ascending above its floor", + )); + } + if entry.op > self.last_record_op { + return Err(SnapshotError::InvalidState( + "change log record is above its newest record op", + )); + } + let stamp = (entry.checksum, entry.timestamp); + if *operations.entry(entry.op).or_insert(stamp) != stamp { + return Err(SnapshotError::InvalidState( + "change log records of one operation disagree on its checksum", + )); + } + positions.push(position); + previous = position; + } + } + positions.sort_unstable(); + if positions.windows(2).any(|pair| pair[0] == pair[1]) { + return Err(SnapshotError::InvalidState( + "change log rings repeat an operation ordinal", + )); + } + let holds_history = !positions.is_empty() + || self.resource.evicted_through_op != 0 + || self.activity.evicted_through_op != 0; + if holds_history && self.log_id.is_none() { + return Err(SnapshotError::InvalidState( + "change log holds history without a log id", + )); + } + Ok(()) + } + + /// The log this snapshot describes, each ring trimmed from the front to + /// its local capacity. + /// + /// # Errors + /// [`SnapshotError::InvalidState`] for a record kind this build does not + /// know, which [`Self::validate`] refuses first. + pub fn restore(&self, capacities: ChangeLogCapacities) -> Result<ChangeLog, SnapshotError> { + Ok(ChangeLog::from_parts( + self.log_id, + self.last_record_op, + self.resource.restore(capacities.resource)?, + self.activity.restore(capacities.activity)?, + )) + } +} + +impl RingSnapshot { + fn restore(&self, capacity: usize) -> Result<Ring, SnapshotError> { + let entries = self + .entries + .iter() + .map(|entry| { + Ok(Arc::new(ChangeEntry { + op: entry.op, + ordinal: entry.ordinal, + checksum: entry.checksum, + timestamp: entry.timestamp, + actor_user_id: entry.actor_user_id, + kind: WireMetadataChangeKind::from_code(entry.kind).map_err(|_| { + SnapshotError::InvalidState("change log holds an unknown record kind") + })?, + body: entry.body.clone(), + })) + }) + .collect::<Result<VecDeque<_>, SnapshotError>>()?; + Ok(Ring::restored( + entries, + (self.evicted_through_op, self.evicted_through_ordinal), + self.evicted_through_timestamp, + capacity, + )) + } +} + +impl From<&ChangeLog> for ChangeLogSnapshot { + fn from(log: &ChangeLog) -> Self { + Self { + log_id: log.log_id(), + last_record_op: log.last_record_op(), + resource: RingSnapshot::from(log.ring(MetadataChangeClass::Resource)), + activity: RingSnapshot::from(log.ring(MetadataChangeClass::Activity)), + } + } +} + +impl From<&Ring> for RingSnapshot { + fn from(ring: &Ring) -> Self { + let (evicted_through_op, evicted_through_ordinal) = ring.evicted_through(); + Self { + evicted_through_op, + evicted_through_ordinal, + evicted_through_timestamp: ring.evicted_through_timestamp(), + entries: ring + .entries() + .iter() + .map(|entry| ChangeEntrySnapshot { + op: entry.op, + ordinal: entry.ordinal, + checksum: entry.checksum, + timestamp: entry.timestamp, + actor_user_id: entry.actor_user_id, + group: entry.group().as_u8(), + kind: entry.kind.as_u8(), + body: entry.body.clone(), + }) + .collect(), + } + } +} + +mod body_bytes { + use bytes::Bytes; + use serde::de::{Error, Visitor}; + use serde::{Deserializer, Serializer}; + use std::fmt; + + pub fn serialize<S: Serializer>(bytes: &Bytes, serializer: S) -> Result<S::Ok, S::Error> { + serializer.serialize_bytes(bytes) + } + + pub fn deserialize<'de, D: Deserializer<'de>>(deserializer: D) -> Result<Bytes, D::Error> { + deserializer.deserialize_byte_buf(BodyVisitor) + } + + struct BodyVisitor; + + impl Visitor<'_> for BodyVisitor { + type Value = Bytes; + + fn expecting(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str("change record body bytes") + } + + fn visit_bytes<E: Error>(self, bytes: &[u8]) -> Result<Self::Value, E> { + Ok(Bytes::copy_from_slice(bytes)) + } + + fn visit_byte_buf<E: Error>(self, bytes: Vec<u8>) -> Result<Self::Value, E> { + Ok(Bytes::from(bytes)) + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::change_log::AppliedChange; + use crate::change_log::store::MetadataChangeLog; + use iggy_binary_protocol::PrepareHeader; + use iggy_binary_protocol::primitives::metadata_changes::WireMetadataChangeKind as Kind; + + type Corruption = Box<dyn Fn(&mut ChangeLogSnapshot)>; + + #[test] + fn given_log_when_round_tripping_snapshot_should_restore_the_same_log_and_bytes() { + let log = populated_log(); + let snapshot = log + .read(|log| ChangeLogSnapshot::from(log)) + .expect("readable"); + let encoded = rmp_serde::to_vec(&snapshot).expect("encode"); + + let decoded: ChangeLogSnapshot = rmp_serde::from_slice(&encoded).expect("decode"); + decoded.validate(10).expect("valid"); + let restored = MetadataChangeLog::new(ChangeLogCapacities::default()); + restored + .restore( + decoded + .restore(ChangeLogCapacities::default()) + .expect("restore"), + ) + .expect("writer"); + + let reencoded = restored + .read(|log| rmp_serde::to_vec(&ChangeLogSnapshot::from(log)).expect("encode")) + .expect("readable"); + assert_eq!(decoded, snapshot); + assert_eq!(reencoded, encoded, "identical logs must encode identically"); + } + + #[test] + fn given_known_zero_and_unknown_log_ids_when_round_tripping_should_keep_them_apart() { + for log_id in [None, Some(0)] { + let snapshot = ChangeLogSnapshot { + log_id, + ..ChangeLogSnapshot::default() + }; + let encoded = rmp_serde::to_vec(&snapshot).expect("encode"); + let decoded: ChangeLogSnapshot = rmp_serde::from_slice(&encoded).expect("decode"); + assert_eq!(decoded.log_id, log_id); + decoded + .validate(5) + .expect("an empty log is valid at any sequence"); + } + } + + #[test] + fn given_ring_above_local_capacity_when_restoring_should_trim_front_and_advance_floor() { + let snapshot = populated_log() + .read(|log| ChangeLogSnapshot::from(log)) + .expect("readable"); + + let restored = snapshot + .restore(ChangeLogCapacities { + resource: 1, + activity: 1, + }) + .expect("restore"); + + let resource = restored.ring(MetadataChangeClass::Resource); + assert_eq!( + resource + .entries() + .iter() + .map(|entry| entry.position()) + .collect::<Vec<_>>(), + [(10, 0)] + ); + assert_eq!(resource.evicted_through(), (4, 1)); + assert_eq!(resource.evicted_through_timestamp(), 400); + assert_eq!(restored.last_record_op(), 10); + } + + #[test] + fn given_broken_log_when_validating_should_refuse_each_rule() { + let valid = populated_log() + .read(|log| ChangeLogSnapshot::from(log)) + .expect("readable"); + let cases: Vec<(&str, Corruption, u64)> = vec![ + ("record above the snapshot", Box::new(|_| {}), 9), + ( + "unsorted ring", + Box::new(|log| log.resource.entries.swap(0, 1)), + 10, + ), + ( + "checksum differs within an operation", + Box::new(|log| log.resource.entries[1].checksum += 1), + 10, + ), + ( + "ordinal repeated across rings", + Box::new(|log| log.activity.entries[0].ordinal = 0), + 10, + ), + ( + "record at or below its floor", + Box::new(|log| { + log.resource.evicted_through_op = 4; + log.resource.evicted_through_ordinal = 0; + }), + 10, + ), + ( + "floor above the newest record", + Box::new(|log| log.activity.evicted_through_op = 11), + 11, + ), + ( + "record above the newest record op", + Box::new(|log| log.last_record_op = 4), + 10, + ), + ( + "history without a log id", + Box::new(|log| log.log_id = None), + 10, + ), + ( + "unknown kind", + Box::new(|log| log.resource.entries[0].kind = 99), + 10, + ), + ( + "kind under another group", + Box::new(|log| log.resource.entries[0].group = 2), + 10, + ), + ( + "activity kind in the resource ring", + Box::new(|log| { + let mut entry = log.activity.entries.remove(0); + entry.ordinal = 5; + entry.op = 10; + log.resource.entries.push(entry); + }), + 10, + ), + ]; + + for (rule, corrupt, sequence_number) in cases { + let mut snapshot = valid.clone(); + corrupt(&mut snapshot); + assert!( + matches!( + snapshot.validate(sequence_number), + Err(SnapshotError::InvalidState(_)) + ), + "{rule} must be refused" + ); + } + valid.validate(10).expect("the untouched log is valid"); + } + + #[test] + fn given_ring_above_the_largest_capacity_when_validating_should_refuse_it() { + let entry = ChangeEntrySnapshot { + op: 1, + ordinal: 0, + checksum: 1, + timestamp: 1, + actor_user_id: None, + group: Kind::StreamCreated.group().as_u8(), + kind: Kind::StreamCreated.as_u8(), + body: Bytes::new(), + }; + let snapshot = ChangeLogSnapshot { + log_id: Some(1), + last_record_op: 1, + resource: RingSnapshot { + entries: vec![entry; MAX_CAPACITY + 1], + ..RingSnapshot::default() + }, + activity: RingSnapshot::default(), + }; + + assert!(matches!( + snapshot.validate(1), + Err(SnapshotError::InvalidState(_)) + )); + } + + /// Resource records at ops 4 and 10, an activity record at op 4. + fn populated_log() -> MetadataChangeLog { + let log = MetadataChangeLog::new(ChangeLogCapacities::default()); + let change = |kind| AppliedChange { + kind, + actor_user_id: Some(1), + body: Bytes::from_static(&[0x80, 0xff, 1]), + }; + log.append(&header(1, 0x0d4b, 100), Vec::new()) + .expect("append"); + log.append( + &header(4, 0x44, 400), + vec![ + change(Kind::UserDeleted), + change(Kind::UserPermissionsUpdated), + change(Kind::ConsumerGroupMemberLeft), + ], + ) + .expect("append"); + log.append(&header(10, 0x10, 1_000), vec![change(Kind::StreamCreated)]) + .expect("append"); + log + } + + fn header(op: u64, checksum: u128, timestamp: u64) -> PrepareHeader { + PrepareHeader { + op, + checksum, + timestamp, + ..PrepareHeader::default() + } + } +}
diff --git a/core/metadata/src/change_log/store.rs b/core/metadata/src/change_log/store.rs new file mode 100644 index 0000000..7a76089 --- /dev/null +++ b/core/metadata/src/change_log/store.rs
@@ -0,0 +1,717 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use crate::applied_frontier::AppliedFrontier; +use crate::change_log::{AppliedChange, ChangeLogSnapshot}; +use crate::stm::WriteCell; +use bytes::Bytes; +use iggy_binary_protocol::PrepareHeader; +use iggy_binary_protocol::primitives::metadata_changes::{ + WireMetadataChangeCursor, WireMetadataChangeGroup, WireMetadataChangeKind, +}; +use iggy_common::{MetadataChangeClass, MetadataChangeGroup}; +use left_right::{Absorb, ReadHandle, ReadHandleFactory}; +use std::collections::VecDeque; +use std::fmt; +use std::hint; +use std::sync::Arc; + +/// Records kept for streams, topics, partitions, consumer groups, users and +/// personal access tokens by default. +pub const DEFAULT_RESOURCE_CAPACITY: usize = 65_536; + +/// Records kept for sessions and consumer group membership by default. +pub const DEFAULT_ACTIVITY_CAPACITY: usize = 16_384; + +/// Largest ring a snapshot may hold, and the largest configurable capacity. +pub const MAX_CAPACITY: usize = 1 << 20; + +/// Record body bytes one ring keeps, so large permission sets cannot grow the +/// snapshot past what state transfer carries. +pub const MAX_RING_BYTES: usize = 128 << 20; + +/// Captures a coherent image tries before a read gives up as retryable. +const COHERENT_READ_ATTEMPTS: usize = 64; + +/// The change log of one node: the public results of the committed metadata +/// operations it applied, newest last, in two bounded retention classes. +#[derive(Debug, Clone)] +pub struct ChangeLog { + /// Checksum of op 1, `None` until this node applied or restored it. + log_id: Option<u128>, + /// Op of the newest record appended in either class. + last_record_op: u64, + resource: Ring, + activity: Ring, +} + +impl ChangeLog { + #[must_use] + pub const fn new(capacities: ChangeLogCapacities) -> Self { + Self { + log_id: None, + last_record_op: 0, + resource: Ring::new(capacities.resource), + activity: Ring::new(capacities.activity), + } + } + + #[must_use] + pub const fn log_id(&self) -> Option<u128> { + self.log_id + } + + #[must_use] + pub const fn last_record_op(&self) -> u64 { + self.last_record_op + } + + #[must_use] + pub const fn ring(&self, class: MetadataChangeClass) -> &Ring { + match class { + MetadataChangeClass::Resource => &self.resource, + MetadataChangeClass::Activity => &self.activity, + } + } + + pub(crate) const fn from_parts( + log_id: Option<u128>, + last_record_op: u64, + resource: Ring, + activity: Ring, + ) -> Self { + Self { + log_id, + last_record_op, + resource, + activity, + } + } + + const fn ring_mut(&mut self, class: MetadataChangeClass) -> &mut Ring { + match class { + MetadataChangeClass::Resource => &mut self.resource, + MetadataChangeClass::Activity => &mut self.activity, + } + } + + fn apply(&mut self, command: &ChangeLogCommand) { + match command { + ChangeLogCommand::Append { + log_id, + op, + entries, + } => { + if let Some(log_id) = log_id { + self.log_id = Some(*log_id); + } + if entries.is_empty() { + return; + } + for entry in entries { + self.ring_mut(entry.class()).push(Arc::clone(entry)); + } + self.last_record_op = *op; + } + ChangeLogCommand::Restore(log) => *self = (**log).clone(), + } + } +} + +impl Absorb<ChangeLogCommand> for ChangeLog { + fn absorb_first(&mut self, command: &mut ChangeLogCommand, _other: &Self) { + self.apply(command); + } + + fn absorb_second(&mut self, command: ChangeLogCommand, _other: &Self) { + self.apply(&command); + } + + fn sync_with(&mut self, first: &Self) { + *self = first.clone(); + } +} + +/// One retention class: retained records ascending by `(op, ordinal)`, and +/// the position of the newest record it evicted. +#[derive(Debug, Clone)] +pub struct Ring { + entries: VecDeque<Arc<ChangeEntry>>, + /// Body bytes of the retained records. + bytes: usize, + evicted_through: (u64, u32), + evicted_through_timestamp: u64, + capacity: usize, +} + +impl Ring { + #[must_use] + pub const fn new(capacity: usize) -> Self { + Self { + entries: VecDeque::new(), + bytes: 0, + evicted_through: (0, 0), + evicted_through_timestamp: 0, + capacity, + } + } + + #[must_use] + pub const fn entries(&self) -> &VecDeque<Arc<ChangeEntry>> { + &self.entries + } + + /// Body bytes of the retained records. + #[must_use] + pub const fn bytes(&self) -> usize { + self.bytes + } + + /// `(op, ordinal)` of the newest evicted record, `(0, 0)` when none was. + #[must_use] + pub const fn evicted_through(&self) -> (u64, u32) { + self.evicted_through + } + + /// Timestamp of the newest evicted record, 0 when none was. + #[must_use] + pub const fn evicted_through_timestamp(&self) -> u64 { + self.evicted_through_timestamp + } + + /// A restored ring, trimmed from the front to `capacity` and + /// [`MAX_RING_BYTES`]. + pub(crate) fn restored( + entries: VecDeque<Arc<ChangeEntry>>, + evicted_through: (u64, u32), + evicted_through_timestamp: u64, + capacity: usize, + ) -> Self { + let bytes = entries.iter().map(|entry| entry.body.len()).sum(); + let mut ring = Self { + entries, + bytes, + evicted_through, + evicted_through_timestamp, + capacity, + }; + ring.trim(); + ring + } + + fn push(&mut self, entry: Arc<ChangeEntry>) { + self.bytes += entry.body.len(); + self.entries.push_back(entry); + self.trim(); + } + + /// Evicts record by record, so one large operation can stay partly + /// retained. + fn trim(&mut self) { + while self.entries.len() > self.capacity || self.bytes > MAX_RING_BYTES { + let Some(evicted) = self.entries.pop_front() else { + return; + }; + self.bytes -= evicted.body.len(); + self.evicted_through = evicted.position(); + self.evicted_through_timestamp = evicted.timestamp; + } + } +} + +/// One retained record, stamped with the position of its operation. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ChangeEntry { + pub op: u64, + pub ordinal: u32, + pub checksum: u128, + pub timestamp: u64, + pub actor_user_id: Option<u32>, + pub kind: WireMetadataChangeKind, + pub body: Bytes, +} + +impl ChangeEntry { + #[must_use] + pub const fn position(&self) -> (u64, u32) { + (self.op, self.ordinal) + } + + #[must_use] + pub const fn group(&self) -> WireMetadataChangeGroup { + self.kind.group() + } + + #[must_use] + pub fn class(&self) -> MetadataChangeClass { + MetadataChangeGroup::from(self.group()).class() + } + + /// The cursor naming this record in the log `log_id`. + #[must_use] + pub const fn cursor(&self, log_id: u128) -> WireMetadataChangeCursor { + WireMetadataChangeCursor { + log_id, + op: self.op, + checksum: self.checksum, + ordinal: self.ordinal, + } + } +} + +/// Record budgets of the two retention classes. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ChangeLogCapacities { + pub resource: usize, + pub activity: usize, +} + +impl Default for ChangeLogCapacities { + fn default() -> Self { + Self { + resource: DEFAULT_RESOURCE_CAPACITY, + activity: DEFAULT_ACTIVITY_CAPACITY, + } + } +} + +#[derive(Debug, Clone)] +pub enum ChangeLogCommand { + /// The records of one operation, in canonical order. Op 1 also carries + /// the log id. + Append { + log_id: Option<u128>, + op: u64, + entries: Vec<Arc<ChangeEntry>>, + }, + /// Replace the whole log, as a restored snapshot does. + Restore(Box<ChangeLog>), +} + +/// The change log behind a left-right pair: the write side on the metadata +/// shard, read handles on every shard. +pub struct MetadataChangeLog { + write: Option<Writer>, + read: ReadHandle<ChangeLog>, +} + +impl MetadataChangeLog { + /// An empty log with its write side, for the metadata shard. + #[must_use] + pub fn new(capacities: ChangeLogCapacities) -> Self { + let (write, read) = left_right::new_from_empty(ChangeLog::new(capacities)); + Self { + write: Some(Writer { + cell: WriteCell::new(write), + capacities, + }), + read, + } + } + + /// A read-only view of the log another shard writes. + #[must_use] + #[allow(clippy::needless_pass_by_value)] + pub fn from_factory(factory: ReadHandleFactory<ChangeLog>) -> Self { + Self { + write: None, + read: factory.handle(), + } + } + + /// A `Send + Sync` factory other shards build their read handles from. + #[must_use] + pub fn factory(&self) -> ReadHandleFactory<ChangeLog> { + self.read.factory() + } + + /// The capacities the write side was built with. + /// + /// # Errors + /// [`ChangeLogError::ReaderOnly`] on a read-only view, which does not know them. + pub fn capacities(&self) -> Result<ChangeLogCapacities, ChangeLogError> { + self.write + .as_ref() + .map(|write| write.capacities) + .ok_or(ChangeLogError::ReaderOnly) + } + + /// Runs `read` on the published log, `None` once the write side is gone. + pub fn read<R>(&self, read: impl FnOnce(&ChangeLog) -> R) -> Option<R> { + self.read.enter().map(|log| read(&log)) + } + + /// Runs `read` on an image of the log holding every record of the ops up + /// to the frontier it is given and none above. `None` once the write side + /// is gone or publications kept racing the capture. + /// + /// A frontier read is a lower bound: the frontier is read before and + /// after entering the log, and a log ahead of it is retried, since + /// appends publish before the frontier advances. + pub fn read_coherent<R>( + &self, + frontier: &AppliedFrontier, + read: impl Fn(&ChangeLog, u64) -> R, + ) -> Option<R> { + for _ in 0..COHERENT_READ_ATTEMPTS { + let before = frontier.get(); + let log = self.read.enter()?; + let head_op = frontier.get(); + if before == head_op && log.last_record_op() <= head_op { + return Some(read(&log, head_op)); + } + drop(log); + hint::spin_loop(); + } + None + } + + /// The persisted form of the published log. Empty only on a read-only + /// view whose write side is gone, and only the write side checkpoints. + #[must_use] + pub fn snapshot(&self) -> ChangeLogSnapshot { + self.read(|log| ChangeLogSnapshot::from(log)) + .unwrap_or_default() + } + + /// Appends the records of one committed operation and publishes them. + /// + /// # Errors + /// [`ChangeLogError::ReaderOnly`] on a read-only view, and + /// [`ChangeLogError::NotMonotone`] for records of an operation at or below + /// the newest appended one, which only a bug can produce. + pub fn append( + &self, + header: &PrepareHeader, + changes: Vec<AppliedChange>, + ) -> Result<(), ChangeLogError> { + let write = self.write.as_ref().ok_or(ChangeLogError::ReaderOnly)?; + let log_id = (header.op == 1).then_some(header.checksum); + if changes.is_empty() && log_id.is_none() { + return Ok(()); + } + if !changes.is_empty() { + let last_record_op = self.read(ChangeLog::last_record_op).unwrap_or_default(); + if header.op <= last_record_op { + return Err(ChangeLogError::NotMonotone { + op: header.op, + last_record_op, + }); + } + } + let entries = changes + .into_iter() + .enumerate() + .map(|(ordinal, change)| { + Ok(Arc::new(ChangeEntry { + op: header.op, + ordinal: u32::try_from(ordinal) + .map_err(|_| ChangeLogError::TooManyRecords { op: header.op })?, + checksum: header.checksum, + timestamp: header.timestamp, + actor_user_id: change.actor_user_id, + kind: change.kind, + body: change.body, + })) + }) + .collect::<Result<Vec<_>, ChangeLogError>>()?; + write.cell.apply(ChangeLogCommand::Append { + log_id, + op: header.op, + entries, + }); + Ok(()) + } + + /// Replaces the log with `log` and publishes it. + /// + /// # Errors + /// [`ChangeLogError::ReaderOnly`] on a read-only view. + pub fn restore(&self, log: ChangeLog) -> Result<(), ChangeLogError> { + let write = self.write.as_ref().ok_or(ChangeLogError::ReaderOnly)?; + write.cell.apply(ChangeLogCommand::Restore(Box::new(log))); + Ok(()) + } +} + +impl fmt::Debug for MetadataChangeLog { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("MetadataChangeLog") + .field("writer", &self.write.is_some()) + .field("capacities", &self.capacities().ok()) + .finish_non_exhaustive() + } +} + +/// The write side of the log and the capacities it was built with. +struct Writer { + cell: WriteCell<ChangeLog, ChangeLogCommand>, + capacities: ChangeLogCapacities, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ChangeLogError { + /// A write reached a read-only view of the log. + ReaderOnly, + /// Records of an operation at or below the newest appended one. + NotMonotone { op: u64, last_record_op: u64 }, + /// One operation produced more records than an ordinal can number. + TooManyRecords { op: u64 }, +} + +impl fmt::Display for ChangeLogError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::ReaderOnly => f.write_str("change log write on a read-only view"), + Self::NotMonotone { op, last_record_op } => write!( + f, + "change log append at op {op} is not above the newest record op {last_record_op}" + ), + Self::TooManyRecords { op } => { + write!( + f, + "op {op} produced more change records than fit an ordinal" + ) + } + } + } +} + +impl std::error::Error for ChangeLogError {} + +#[cfg(test)] +mod tests { + use super::*; + use iggy_binary_protocol::primitives::metadata_changes::WireMetadataChangeKind as Kind; + + #[test] + fn given_records_of_one_op_when_appending_should_stamp_header_and_consecutive_ordinals() { + let log = MetadataChangeLog::new(ChangeLogCapacities::default()); + + log.append( + &header(7, 0xabc, 7_000), + vec![ + change(Kind::UserDeleted, Some(3)), + change(Kind::ConsumerGroupMemberLeft, Some(3)), + ], + ) + .expect("append"); + + let (resource, activity) = log + .read(|log| { + ( + entries(log.ring(MetadataChangeClass::Resource)), + entries(log.ring(MetadataChangeClass::Activity)), + ) + }) + .expect("readable"); + assert_eq!( + resource, + [ChangeEntry { + op: 7, + ordinal: 0, + checksum: 0xabc, + timestamp: 7_000, + actor_user_id: Some(3), + kind: Kind::UserDeleted, + body: Bytes::from_static(b"body"), + }] + ); + assert_eq!(activity[0].position(), (7, 1)); + assert_eq!(log.read(ChangeLog::last_record_op), Some(7)); + } + + #[test] + fn given_op_one_when_appending_should_set_log_id_even_without_records() { + let log = MetadataChangeLog::new(ChangeLogCapacities::default()); + assert_eq!(log.read(ChangeLog::log_id), Some(None)); + + log.append(&header(1, 0, 1), Vec::new()).expect("append"); + log.append(&header(2, 0x77, 2), Vec::new()).expect("append"); + + assert_eq!(log.read(ChangeLog::log_id), Some(Some(0))); + assert_eq!(log.read(ChangeLog::last_record_op), Some(0)); + } + + #[test] + fn given_full_ring_when_appending_should_evict_by_record_and_keep_a_partial_operation() { + let log = MetadataChangeLog::new(ChangeLogCapacities { + resource: 3, + activity: 2, + }); + log.append( + &header(4, 4, 400), + vec![change(Kind::StreamCreated, None); 2], + ) + .expect("append"); + + log.append( + &header(5, 5, 500), + vec![change(Kind::StreamCreated, None); 2], + ) + .expect("append"); + + let ring = log + .read(|log| log.ring(MetadataChangeClass::Resource).clone()) + .expect("readable"); + assert_eq!( + ring.entries() + .iter() + .map(|entry| entry.position()) + .collect::<Vec<_>>(), + [(4, 1), (5, 0), (5, 1)] + ); + assert_eq!(ring.evicted_through(), (4, 0)); + assert_eq!(ring.evicted_through_timestamp(), 400); + let activity = log + .read(|log| log.ring(MetadataChangeClass::Activity).evicted_through()) + .expect("readable"); + assert_eq!(activity, (0, 0), "the other class evicts on its own budget"); + } + + #[test] + fn given_op_at_or_below_newest_record_when_appending_should_refuse_it() { + let log = MetadataChangeLog::new(ChangeLogCapacities::default()); + log.append(&header(9, 9, 9), vec![change(Kind::StreamCreated, None)]) + .expect("append"); + + for op in [9, 8] { + assert_eq!( + log.append( + &header(op, op.into(), op), + vec![change(Kind::StreamCreated, None)] + ), + Err(ChangeLogError::NotMonotone { + op, + last_record_op: 9 + }) + ); + } + assert_eq!( + log.read(|log| log.ring(MetadataChangeClass::Resource).entries().len()), + Some(1) + ); + } + + #[test] + fn given_bodies_past_the_byte_budget_when_appending_should_evict_the_oldest_records() { + let log = MetadataChangeLog::new(ChangeLogCapacities::default()); + for op in 1..=3 { + log.append( + &header(op, u128::from(op), op), + vec![sized_change(MAX_RING_BYTES / 2 + 1)], + ) + .expect("append"); + } + + let ring = log + .read(|log| log.ring(MetadataChangeClass::Resource).clone()) + .expect("readable"); + + assert_eq!(ring.entries().len(), 1); + assert_eq!(ring.evicted_through(), (2, 0)); + assert_eq!(ring.bytes(), MAX_RING_BYTES / 2 + 1); + } + + #[test] + fn given_snapshot_past_the_byte_budget_when_restoring_should_trim_the_oldest_records() { + let entries = (1..=3) + .map(|op| { + Arc::new(ChangeEntry { + op, + ordinal: 0, + checksum: u128::from(op), + timestamp: op, + actor_user_id: None, + kind: Kind::UserPermissionsUpdated, + body: Bytes::from(vec![0; MAX_RING_BYTES / 2 + 1]), + }) + }) + .collect(); + + let ring = Ring::restored(entries, (0, 0), 0, DEFAULT_RESOURCE_CAPACITY); + + assert_eq!(ring.entries().len(), 1); + assert_eq!(ring.evicted_through(), (2, 0)); + } + + #[test] + fn given_reader_view_when_asking_capacities_should_refuse_instead_of_guessing() { + let capacities = ChangeLogCapacities { + resource: 7, + activity: 3, + }; + let writer = MetadataChangeLog::new(capacities); + let reader = MetadataChangeLog::from_factory(writer.factory()); + + assert_eq!(writer.capacities(), Ok(capacities)); + assert_eq!(reader.capacities(), Err(ChangeLogError::ReaderOnly)); + } + + #[test] + fn given_reader_view_when_writing_should_refuse_and_see_the_writer_state() { + let writer = MetadataChangeLog::new(ChangeLogCapacities::default()); + let reader = MetadataChangeLog::from_factory(writer.factory()); + + writer + .append(&header(1, 0x11, 1), vec![change(Kind::UserCreated, None)]) + .expect("append"); + + assert_eq!( + reader.append(&header(2, 2, 2), vec![change(Kind::UserCreated, None)]), + Err(ChangeLogError::ReaderOnly) + ); + assert_eq!( + reader.restore(ChangeLog::new(ChangeLogCapacities::default())), + Err(ChangeLogError::ReaderOnly) + ); + assert_eq!(reader.read(ChangeLog::log_id), Some(Some(0x11))); + assert_eq!(reader.read(ChangeLog::last_record_op), Some(1)); + } + + fn header(op: u64, checksum: u128, timestamp: u64) -> PrepareHeader { + PrepareHeader { + op, + checksum, + timestamp, + ..PrepareHeader::default() + } + } + + fn change(kind: Kind, actor_user_id: Option<u32>) -> AppliedChange { + AppliedChange { + kind, + actor_user_id, + body: Bytes::from_static(b"body"), + } + } + + fn sized_change(body_bytes: usize) -> AppliedChange { + AppliedChange { + kind: Kind::UserPermissionsUpdated, + actor_user_id: None, + body: Bytes::from(vec![0; body_bytes]), + } + } + + fn entries(ring: &Ring) -> Vec<ChangeEntry> { + ring.entries() + .iter() + .map(|entry| (**entry).clone()) + .collect() + } +}
diff --git a/core/metadata/src/change_log/tests.rs b/core/metadata/src/change_log/tests.rs new file mode 100644 index 0000000..7c935b0 --- /dev/null +++ b/core/metadata/src/change_log/tests.rs
@@ -0,0 +1,2217 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use super::*; +use crate::MuxStateMachine; +use crate::change_log::{ChangeLogCapacities, CurrentState, MetadataChangeLog, current_page}; +use crate::impls::metadata::{StreamsFrontend, apply_committed_prepare}; +use crate::stm::consumer_group::{ + CompleteConsumerGroupRevocationRequest, JoinConsumerGroupRequest, LeaveConsumerGroupRequest, +}; +use crate::stm::lifecycle::{CompleteLifecycleRequest, LifecycleCompletion}; +use crate::stm::stream::Streams; +use crate::stm::user::{ + CreatePersonalAccessTokenRequest, DeletePersonalAccessTokenRequest, PAT_TOKEN_HASH_BYTES, + ROOT_USER_CREATED_AT_MICROS, Users, +}; +use consensus::{CLIENTS_TABLE_MAX, ClientTable, EXPIRED_SESSION_REQUEST_ID}; +use iggy_binary_protocol::primitives::metadata_changes::{ + WireConsumerGroupCreatedBody, WireConsumerGroupDeletedBody, WireMemberJoinedBody, + WireMemberLeftReason, WireMetadataChangeStatus, WireMetadataCurrentItem, + WireMetadataCurrentPageToken, WirePartitionHandedOverBody, WirePartitionsChangedBody, + WirePermissionsUpdatedReason, WireSessionBody, WireSessionClosedBody, WireSessionClosedReason, + WireStreamDeletedBody, WireTokenBody, WireTokenDeletedBody, WireTokenDeletedReason, + WireTopicDeletedBody, WireUserIdentityBody, WireUserPermissionsBody, +}; +use iggy_binary_protocol::primitives::partition_assignment::CreatedPartitionAssignment; +use iggy_binary_protocol::primitives::permissions::{ + WireGlobalPermissions, WirePermissions, WireStreamPermissions, WireTopicPermissions, +}; +use iggy_binary_protocol::requests::consumer_groups::{ + CreateConsumerGroupRequest, DeleteConsumerGroupRequest, +}; +use iggy_binary_protocol::requests::partitions::{ + CreatePartitionsRequest, CreatePartitionsWithAssignmentsRequest, DeletePartitionsRequest, +}; +use iggy_binary_protocol::requests::streams::{ + CreateStreamRequest, DeleteStreamRequest, RewrapStreamKeyEntry, RewrapStreamKeysRequest, + UpdateStreamResolvedRequest, +}; +use iggy_binary_protocol::requests::topics::{ + CreateTopicRequest, CreateTopicWithAssignmentsRequest, DeleteTopicRequest, + UpdateTopicResolvedRequest, +}; +use iggy_binary_protocol::requests::users::login_register::RegisterRequest; +use iggy_binary_protocol::requests::users::{ + ChangePasswordRequest, CreateUserRequest, DeleteUserRequest, UpdatePermissionsRequest, + UpdateUserResolvedRequest, +}; +use iggy_binary_protocol::responses::system::{GetMetadataChangesResponse, WireMetadataItem}; +use iggy_binary_protocol::{Command, Operation, PrepareHeader, WireEncode, WireIdentifier}; +use iggy_binary_protocol::{WireName, WireOptions}; +use iggy_common::wire_conversions::resource_options_to_wire; +use iggy_common::{ + HeaderKey, HeaderValue, MetadataChangeGroup, MetadataChangeGroups, OptionKey, OptionStorage, + OptionValue, OptionVisibility, OptionsProvenance, ServerKeyContext, StreamEncryptionKey, + StreamKeyOrigin, UserStatus, WrapSet, descriptors_for, variadic, +}; +use server_common::Message; +use server_common::iobuf::Owned; +use std::cell::{Cell, RefCell}; + +const HEADER_SIZE: usize = size_of::<PrepareHeader>(); +const ROOT: u32 = 0; +const ROOT_CLIENT: u128 = 7; +const ALICE: u32 = 1; +const BOB: u32 = 2; +const CLIENT_A: u128 = 101; +const CLIENT_B: u128 = 102; +const OLD_CONFIG_KEY: &str = "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="; +const NEW_CONFIG_KEY: &str = "AQIDBAUGBwgJCgsMDQ4PEBESExQVFhcYGRobHB0eHyA="; +const SENTINEL: &str = "change-log-sentinel-6f1d"; +const NO_CHANGES: [AppliedChange; 0] = []; +const NO_ITEMS: [WireMetadataItem; 0] = []; + +#[test] +fn given_created_stream_when_applying_should_record_its_identity_and_public_options() { + let harness = Harness::new(); + let committed = harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("orders"), + ); + let op = harness.last_op(); + + let stream = harness.stream(0); + assert_eq!(actors(&committed), [Some(ROOT)]); + assert_eq!( + decoded(&committed), + [WireMetadataChange::StreamCreated(WireStreamBody { + key: WireStreamKey { + stream_id: 0, + stream_created_op: op, + }, + created_at: stream.created_at.as_micros(), + name: name("orders"), + options: read_projection(OptionsScope::Stream, &stream.options), + })] + ); +} + +#[test] +fn given_equal_stream_patch_when_applying_should_record_nothing() { + let harness = Harness::new(); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("orders"), + ); + + let committed = harness.commit( + ROOT_CALLER, + Operation::UpdateStreamResolved, + &UpdateStreamResolvedRequest { + stream_id: WireIdentifier::numeric(0), + name: Some(name("orders")), + explicit_options: WireOptions::empty(), + derived_options: WireOptions::empty(), + }, + ); + + assert_eq!(committed, NO_CHANGES); +} + +#[test] +fn given_stream_rename_when_applying_should_record_stream_updated_with_values_after_it() { + let harness = Harness::new(); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("orders"), + ); + + let committed = harness.commit( + ROOT_CALLER, + Operation::UpdateStreamResolved, + &UpdateStreamResolvedRequest { + stream_id: WireIdentifier::numeric(0), + name: Some(name("invoices")), + explicit_options: WireOptions::empty(), + derived_options: WireOptions::empty(), + }, + ); + + let stream = harness.stream(0); + assert_eq!( + decoded(&committed), + [WireMetadataChange::StreamUpdated( + stream_body(&stream).expect("valid stream") + )] + ); + assert_eq!(stream.name.as_ref(), "invoices"); +} + +#[test] +fn given_rewrap_when_kek_fingerprint_changes_should_record_stream_updated_without_actor() { + let harness = Harness::new(); + let old = key_context(OLD_CONFIG_KEY, &[]); + let rotated = key_context(NEW_CONFIG_KEY, &[OLD_CONFIG_KEY]); + let stored = WrapSet::from( + old.wrap( + &StreamEncryptionKey::try_from([3u8; 32].as_slice()).expect("valid key"), + StreamKeyOrigin::Server, + ) + .expect("wrap"), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &encrypted_stream("vault", &stored), + ); + let before = stream_body(&harness.stream(0)).expect("valid stream"); + let rewrapped = rotated + .target_set(&stored) + .expect("rewrap") + .expect("rotation changes the set"); + + let committed = harness.commit( + SERVER, + Operation::RewrapStreamKeys, + &RewrapStreamKeysRequest { + entries: vec![RewrapStreamKeyEntry { + stream_id: 0, + created_op: harness.stream(0).created_op, + key_fingerprint: stored + .key_fingerprint_bytes() + .try_into() + .expect("fingerprint width"), + wrap_set: Bytes::copy_from_slice(rewrapped.as_ref()), + }], + }, + ); + + let after = stream_body(&harness.stream(0)).expect("valid stream"); + assert_ne!(before.options, after.options, "the KEK fingerprint moved"); + assert_eq!(actors(&committed), [None]); + assert_eq!( + decoded(&committed), + [WireMetadataChange::StreamUpdated(after)] + ); + for body in committed.iter().map(|change| &change.body) { + assert!(!contains(body, stored.as_ref())); + assert!(!contains(body, rewrapped.as_ref())); + } +} + +#[test] +fn given_rewrap_entry_that_changes_nothing_when_applying_should_record_nothing() { + let harness = Harness::new(); + let old = key_context(OLD_CONFIG_KEY, &[]); + let stored = WrapSet::from( + old.wrap( + &StreamEncryptionKey::try_from([3u8; 32].as_slice()).expect("valid key"), + StreamKeyOrigin::Server, + ) + .expect("wrap"), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &encrypted_stream("vault", &stored), + ); + + let committed = harness.commit( + SERVER, + Operation::RewrapStreamKeys, + &RewrapStreamKeysRequest { + entries: vec![RewrapStreamKeyEntry { + stream_id: 0, + created_op: harness.stream(0).created_op, + key_fingerprint: stored + .key_fingerprint_bytes() + .try_into() + .expect("fingerprint width"), + wrap_set: Bytes::copy_from_slice(stored.as_ref()), + }], + }, + ); + + assert_eq!(committed, NO_CHANGES); +} + +#[test] +fn given_stream_without_partitions_when_deleting_should_record_stream_deleted_at_once() { + let harness = Harness::new(); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("orders"), + ); + let stream = harness.stream(0); + + let committed = harness.commit( + ROOT_CALLER, + Operation::DeleteStream, + &DeleteStreamRequest { + stream_id: WireIdentifier::numeric(0), + }, + ); + + assert_eq!( + decoded(&committed), + [WireMetadataChange::StreamDeleted(WireStreamDeletedBody { + key: WireStreamKey::from(&stream), + created_at: stream.created_at.as_micros(), + name: name("orders"), + requested_op: None, + })] + ); +} + +#[test] +fn given_stream_delete_with_grants_when_applying_should_record_permission_updates_in_user_order() { + let harness = Harness::new(); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("orders"), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("audit"), + ); + for (username, streams) in [("alice", vec![0, 1]), ("bob", vec![0]), ("carol", vec![1])] { + harness.commit( + ROOT_CALLER, + Operation::CreateUser, + &create_user(username, Some(stream_grants(&streams))), + ); + } + + let committed = harness.commit( + ROOT_CALLER, + Operation::DeleteStream, + &DeleteStreamRequest { + stream_id: WireIdentifier::numeric(0), + }, + ); + + let records = decoded(&committed); + assert_eq!(records.len(), 3); + assert!(matches!(records[0], WireMetadataChange::StreamDeleted(_))); + for (record, user_id) in records[1..].iter().zip([ALICE, BOB]) { + let user = harness.user(user_id); + assert_eq!( + *record, + WireMetadataChange::UserPermissionsUpdated(WireUserPermissionsBody { + key: WireUserKey::from(&user), + username: name(&user.username), + permissions: user.permissions.as_deref().map(permissions_to_wire), + reason: WirePermissionsUpdatedReason::ResourceDeleted, + }) + ); + } + assert_eq!(actors(&committed), [Some(ROOT); 3]); +} + +#[test] +fn given_created_topic_when_applying_should_record_its_partitions_and_created_op() { + let harness = Harness::new(); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("orders"), + ); + + let committed = harness.commit( + ROOT_CALLER, + Operation::CreateTopicWithAssignments, + &create_topic("events", 3), + ); + let op = harness.last_op(); + + let stream = harness.stream(0); + let topic = &stream.topics[0]; + assert_eq!(topic.created_op, op); + assert_eq!( + decoded(&committed), + [WireMetadataChange::TopicCreated(WireTopicBody { + key: WireTopicKey { + stream: WireStreamKey::from(&stream), + topic_id: 0, + topic_created_op: op, + }, + created_at: topic.created_at.as_micros(), + name: name("events"), + partitions_count: 3, + options: read_projection(OptionsScope::Topic, &topic.options), + })] + ); +} + +#[test] +fn given_reused_topic_id_when_recreating_should_have_distinct_created_op() { + let harness = Harness::new(); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("orders"), + ); + let first = harness.commit( + ROOT_CALLER, + Operation::CreateTopicWithAssignments, + &create_topic("events", 0), + ); + harness.commit( + ROOT_CALLER, + Operation::DeleteTopic, + &DeleteTopicRequest { + stream_id: WireIdentifier::numeric(0), + topic_id: WireIdentifier::numeric(0), + }, + ); + + let second = harness.commit( + ROOT_CALLER, + Operation::CreateTopicWithAssignments, + &create_topic("events", 0), + ); + + let (first, second) = (decoded(&first), decoded(&second)); + let ([WireMetadataChange::TopicCreated(first)], [WireMetadataChange::TopicCreated(second)]) = + (first.as_slice(), second.as_slice()) + else { + panic!("each create records one topic"); + }; + assert_eq!(first.key.topic_id, second.key.topic_id); + assert_ne!(first.key.topic_created_op, second.key.topic_created_op); +} + +#[test] +fn given_topic_patch_when_values_change_should_record_topic_updated_and_nothing_when_equal() { + let harness = Harness::new(); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("orders"), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateTopicWithAssignments, + &create_topic("events", 1), + ); + let patch = |name: &str| UpdateTopicResolvedRequest { + stream_id: WireIdentifier::numeric(0), + topic_id: WireIdentifier::numeric(0), + name: Some(WireName::new(name).expect("valid name")), + explicit_options: WireOptions::empty(), + derived_options: WireOptions::empty(), + }; + + let equal = harness.commit( + ROOT_CALLER, + Operation::UpdateTopicResolved, + &patch("events"), + ); + let renamed = harness.commit(ROOT_CALLER, Operation::UpdateTopicResolved, &patch("audit")); + + let stream = harness.stream(0); + assert_eq!(equal, NO_CHANGES); + assert_eq!( + decoded(&renamed), + [WireMetadataChange::TopicUpdated( + topic_body(&stream, &stream.topics[0]).expect("valid topic") + )] + ); +} + +#[test] +fn given_two_phase_topic_delete_when_completing_should_record_requested_op_and_requester() { + let harness = Harness::new(); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("orders"), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateTopicWithAssignments, + &create_topic("events", 2), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateUser, + &create_user("alice", Some(topic_grant(0, 0))), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateUser, + &create_user("bob", Some(managing_topics())), + ); + let stream = harness.stream(0); + let topic_key = stream.topic_key(&stream.topics[0]); + let topic_created_at = stream.topics[0].created_at.as_micros(); + let bob = Caller { + client: CLIENT_B, + user_id: BOB, + session: harness.register(CLIENT_B, BOB), + }; + + let intent = harness.commit( + bob, + Operation::DeleteTopic, + &DeleteTopicRequest { + stream_id: WireIdentifier::numeric(0), + topic_id: WireIdentifier::numeric(0), + }, + ); + let intent_op = harness.last_op(); + let partial = harness.complete_lifecycle(intent_op, &[0]); + let completed = harness.complete_lifecycle(intent_op, &[1]); + + assert_eq!(intent, NO_CHANGES, "the topic still exists at the intent"); + assert_eq!(partial, NO_CHANGES, "a partial batch finishes nothing"); + let alice = harness.user(ALICE); + assert_eq!( + decoded(&completed), + [ + WireMetadataChange::TopicDeleted(WireTopicDeletedBody { + key: topic_key, + created_at: topic_created_at, + name: name("events"), + requested_op: Some(intent_op), + }), + WireMetadataChange::UserPermissionsUpdated(WireUserPermissionsBody { + key: WireUserKey::from(&alice), + username: name("alice"), + permissions: alice.permissions.as_deref().map(permissions_to_wire), + reason: WirePermissionsUpdatedReason::ResourceDeleted, + }), + ] + ); + assert_eq!(actors(&completed), [Some(BOB), Some(BOB)]); +} + +#[test] +fn given_partitions_added_and_removed_when_applying_should_record_counts_and_removed_tail() { + let harness = Harness::new(); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("orders"), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateTopicWithAssignments, + &create_topic("events", 1), + ); + let topic_key = { + let stream = harness.stream(0); + stream.topic_key(&stream.topics[0]) + }; + + let added = harness.commit( + ROOT_CALLER, + Operation::CreatePartitionsWithAssignments, + &CreatePartitionsWithAssignmentsRequest { + request: CreatePartitionsRequest { + stream_id: WireIdentifier::numeric(0), + topic_id: WireIdentifier::numeric(0), + partitions_count: 2, + }, + partitions: assignments(2, 10), + }, + ); + let intent = harness.commit( + ROOT_CALLER, + Operation::DeletePartitions, + &DeletePartitionsRequest { + stream_id: WireIdentifier::numeric(0), + topic_id: WireIdentifier::numeric(0), + partitions_count: 2, + }, + ); + let intent_op = harness.last_op(); + let removed = harness.complete_lifecycle(intent_op, &[1, 2]); + + assert_eq!( + decoded(&added), + [WireMetadataChange::PartitionsChanged( + WirePartitionsChangedBody { + topic: topic_key, + partitions_count: 3, + removed_partition_ids: vec![], + requested_op: None, + } + )] + ); + assert_eq!(intent, NO_CHANGES); + assert_eq!( + decoded(&removed), + [WireMetadataChange::PartitionsChanged( + WirePartitionsChangedBody { + topic: topic_key, + partitions_count: 1, + removed_partition_ids: vec![1, 2], + requested_op: Some(intent_op), + } + )] + ); +} + +#[test] +fn given_consumer_group_lifecycle_when_applying_should_record_counts_after_each_change() { + let harness = Harness::new(); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("orders"), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateTopicWithAssignments, + &create_topic("events", 0), + ); + let topic_key = { + let stream = harness.stream(0); + stream.topic_key(&stream.topics[0]) + }; + + let first = harness.commit( + ROOT_CALLER, + Operation::CreateConsumerGroup, + &create_group("billing"), + ); + let second = harness.commit( + ROOT_CALLER, + Operation::CreateConsumerGroup, + &create_group("shipping"), + ); + let deleted = harness.commit( + ROOT_CALLER, + Operation::DeleteConsumerGroup, + &DeleteConsumerGroupRequest { + stream_id: WireIdentifier::numeric(0), + topic_id: WireIdentifier::numeric(0), + group_id: WireIdentifier::numeric(0), + }, + ); + + assert_eq!( + [decoded(&first), decoded(&second), decoded(&deleted)], + [ + vec![WireMetadataChange::ConsumerGroupCreated( + WireConsumerGroupCreatedBody { + topic: topic_key, + group_id: 0, + name: name("billing"), + consumer_groups_count: 1, + } + )], + vec![WireMetadataChange::ConsumerGroupCreated( + WireConsumerGroupCreatedBody { + topic: topic_key, + group_id: 1, + name: name("shipping"), + consumer_groups_count: 2, + } + )], + vec![WireMetadataChange::ConsumerGroupDeleted( + WireConsumerGroupDeletedBody { + topic: topic_key, + group_id: 0, + name: name("billing"), + consumer_groups_count: 1, + requested_op: None, + } + )], + ] + ); +} + +#[test] +fn given_group_on_partitions_when_delete_completes_should_record_requested_op() { + let harness = Harness::new(); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("orders"), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateTopicWithAssignments, + &create_topic("events", 1), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateConsumerGroup, + &create_group("billing"), + ); + + let intent = harness.commit( + ROOT_CALLER, + Operation::DeleteConsumerGroup, + &DeleteConsumerGroupRequest { + stream_id: WireIdentifier::numeric(0), + topic_id: WireIdentifier::numeric(0), + group_id: WireIdentifier::numeric(0), + }, + ); + let intent_op = harness.last_op(); + let completed = harness.complete_lifecycle(intent_op, &[0]); + + assert_eq!(intent, NO_CHANGES); + let records = decoded(&completed); + let [WireMetadataChange::ConsumerGroupDeleted(body)] = records.as_slice() else { + panic!("the completion records the group delete"); + }; + assert_eq!(body.requested_op, Some(intent_op)); + assert_eq!(body.consumer_groups_count, 0); + assert_eq!(actors(&completed), [Some(ROOT)]); +} + +#[test] +fn given_new_member_when_joining_should_record_it_once_and_nothing_on_rejoin() { + let harness = Harness::with_group(2); + let session = harness.register(CLIENT_A, ROOT); + let topic_key = harness.topic_key(); + + let joined = harness.join(CLIENT_A, session); + let rejoined = harness.join(CLIENT_A, session); + + assert_eq!( + decoded(&joined), + [WireMetadataChange::ConsumerGroupMemberJoined( + WireMemberJoinedBody { + topic: topic_key, + group_id: 0, + member_id: 0, + client_id: CLIENT_A, + members_count: 1, + partition_ids: vec![], + } + )] + ); + assert_eq!(rejoined, NO_CHANGES); +} + +#[test] +fn given_member_when_leaving_should_record_left_with_resulting_count() { + let harness = Harness::with_group(2); + let session = harness.register(CLIENT_A, ROOT); + harness.join(CLIENT_A, session); + let topic_key = harness.topic_key(); + + let left = harness.commit( + ROOT_CALLER, + Operation::LeaveConsumerGroup, + &LeaveConsumerGroupRequest { + stream_id: WireIdentifier::numeric(0), + topic_id: WireIdentifier::numeric(0), + group_id: WireIdentifier::numeric(0), + client_id: CLIENT_A, + }, + ); + + assert_eq!( + decoded(&left), + [WireMetadataChange::ConsumerGroupMemberLeft( + WireMemberLeftBody { + topic: topic_key, + group_id: 0, + member_id: 0, + client_id: CLIENT_A, + members_count: 0, + reason: WireMemberLeftReason::Left, + } + )] + ); +} + +#[test] +fn given_completed_handover_when_applying_should_record_owner_matching_group_details() { + let harness = Harness::with_group(2); + let session = harness.register(CLIENT_A, ROOT); + harness.join(CLIENT_A, session); + let topic_key = harness.topic_key(); + + let mut records = Vec::new(); + for transition in harness.mux.streams().consumer_group_pending_revocations() { + records.extend(decoded(&harness.commit( + SERVER, + Operation::CompleteConsumerGroupRevocation, + &CompleteConsumerGroupRevocationRequest { + stream_id: WireIdentifier::numeric(transition.stream_id), + topic_id: WireIdentifier::numeric(transition.topic_id), + partition_id: transition.partition_id, + installation: transition.installation, + partition_op: 1, + }, + ))); + } + + let details = harness + .mux + .streams() + .consumer_group_details( + &WireIdentifier::numeric(0), + &WireIdentifier::numeric(0), + &WireIdentifier::numeric(0), + ) + .expect("group exists"); + let expected: Vec<_> = details.members[0] + .partitions + .iter() + .map(|&partition_id| { + WireMetadataChange::ConsumerGroupPartitionHandedOver(WirePartitionHandedOverBody { + topic: topic_key, + group_id: 0, + partition_id, + member_id: Some(details.members[0].id), + }) + }) + .collect(); + assert_eq!(expected.len(), 2); + assert_eq!(records, expected); +} + +#[test] +fn given_owner_rejoining_with_new_session_when_applying_should_record_partitions_left_without_owner() + { + let harness = Harness::with_group(2); + let session = harness.register(CLIENT_A, ROOT); + harness.join(CLIENT_A, session); + for transition in harness.mux.streams().consumer_group_pending_revocations() { + harness.commit( + SERVER, + Operation::CompleteConsumerGroupRevocation, + &CompleteConsumerGroupRevocationRequest { + stream_id: WireIdentifier::numeric(transition.stream_id), + topic_id: WireIdentifier::numeric(transition.topic_id), + partition_id: transition.partition_id, + installation: transition.installation, + partition_op: 1, + }, + ); + } + let topic_key = harness.topic_key(); + + let rejoined = harness.join(CLIENT_A, session + 100); + + assert_eq!( + decoded(&rejoined), + [0, 1].map(|partition_id| { + WireMetadataChange::ConsumerGroupPartitionHandedOver(WirePartitionHandedOverBody { + topic: topic_key, + group_id: 0, + partition_id, + member_id: None, + }) + }) + ); + let details = harness + .mux + .streams() + .consumer_group_details( + &WireIdentifier::numeric(0), + &WireIdentifier::numeric(0), + &WireIdentifier::numeric(0), + ) + .expect("group exists"); + assert_eq!(details.members[0].partitions, [] as [u32; 0]); +} + +#[test] +fn given_revocation_of_clients_in_overlapping_groups_when_permuting_slots_should_emit_identical_records() + { + let registered_in_order = |clients: [u128; 2]| { + let harness = Harness::with_group(1); + harness.commit( + ROOT_CALLER, + Operation::CreateConsumerGroup, + &create_group("shipping"), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateUser, + &create_user("alice", None), + ); + for client in clients { + harness.register(client, ALICE); + } + for group_id in [0, 1] { + for client in [CLIENT_A, CLIENT_B] { + let session = harness + .client_table + .borrow() + .get_epoch(client) + .expect("registered"); + harness.join_group(client, session, group_id); + } + } + harness.commit( + ROOT_CALLER, + Operation::DeleteUser, + &DeleteUserRequest { + user_id: WireIdentifier::numeric(ALICE), + }, + ) + }; + + let first = decoded(®istered_in_order([CLIENT_A, CLIENT_B])); + let second = decoded(®istered_in_order([CLIENT_B, CLIENT_A])); + + // Session epochs follow the registration order, nothing else may. + let closed_clients = |records: &[WireMetadataChange]| { + records + .iter() + .filter_map(|record| match record { + WireMetadataChange::SessionClosed(body) => Some(body.client_id), + _ => None, + }) + .collect::<Vec<_>>() + }; + assert_eq!(closed_clients(&first), [CLIENT_A, CLIENT_B]); + assert_eq!(closed_clients(&second), [CLIENT_A, CLIENT_B]); + assert_eq!(first[3..], second[3..]); + assert!(matches!(first[0], WireMetadataChange::UserDeleted(_))); + let removed: Vec<_> = first[3..] + .iter() + .map(|record| match record { + WireMetadataChange::ConsumerGroupMemberLeft(body) => ( + body.group_id, + body.member_id, + body.client_id, + body.members_count, + body.reason, + ), + other => panic!("expected a member removal, got {other:?}"), + }) + .collect(); + assert_eq!( + removed, + [ + (0, 0, CLIENT_A, 0, WireMemberLeftReason::SessionEnded), + (0, 1, CLIENT_B, 0, WireMemberLeftReason::SessionEnded), + (1, 0, CLIENT_A, 0, WireMemberLeftReason::SessionEnded), + (1, 1, CLIENT_B, 0, WireMemberLeftReason::SessionEnded), + ] + ); +} + +#[test] +fn given_created_user_when_applying_should_record_permissions_matching_get_user() { + let harness = Harness::new(); + for stream_name in ["orders", "audit", "billing"] { + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream(stream_name), + ); + } + + let committed = harness.commit( + ROOT_CALLER, + Operation::CreateUser, + &create_user("alice", Some(stream_grants(&[2, 1]))), + ); + + let user = harness.user(ALICE); + assert_eq!( + decoded(&committed), + [WireMetadataChange::UserCreated(WireUserBody { + key: WireUserKey::from(&user), + username: name("alice"), + status: UserStatus::Active.as_code(), + permissions: user.permissions.as_deref().map(permissions_to_wire), + options: read_projection(OptionsScope::User, &user.options), + })] + ); +} + +#[test] +fn given_user_patches_when_applying_should_record_only_changed_values() { + let harness = Harness::new(); + harness.commit( + ROOT_CALLER, + Operation::CreateUser, + &create_user("alice", None), + ); + let patch = |username: &str, status: UserStatus| UpdateUserResolvedRequest { + user_id: WireIdentifier::numeric(ALICE), + username: Some(WireName::new(username).expect("valid name")), + status: Some(status.as_code()), + explicit_options: WireOptions::empty(), + derived_options: WireOptions::empty(), + }; + + let equal = harness.commit( + ROOT_CALLER, + Operation::UpdateUserResolved, + &patch("alice", UserStatus::Active), + ); + let renamed = harness.commit( + ROOT_CALLER, + Operation::UpdateUserResolved, + &patch("alicia", UserStatus::Active), + ); + + assert_eq!(equal, NO_CHANGES); + assert_eq!( + decoded(&renamed), + [WireMetadataChange::UserUpdated( + user_body(&harness.user(ALICE)).expect("valid user") + )] + ); +} + +#[test] +fn given_deactivated_user_with_member_when_applying_should_record_update_session_closed_then_member_left() + { + let harness = Harness::with_group(1); + harness.commit( + ROOT_CALLER, + Operation::CreateUser, + &create_user("alice", None), + ); + let session = harness.register(CLIENT_A, ALICE); + harness.join(CLIENT_A, session); + let alice = WireUserKey::from(&harness.user(ALICE)); + + let committed = harness.commit( + ROOT_CALLER, + Operation::UpdateUserResolved, + &UpdateUserResolvedRequest { + user_id: WireIdentifier::numeric(ALICE), + username: None, + status: Some(UserStatus::Inactive.as_code()), + explicit_options: WireOptions::empty(), + derived_options: WireOptions::empty(), + }, + ); + + let records = decoded(&committed); + assert_eq!(records.len(), 3); + assert!(matches!(&records[0], WireMetadataChange::UserUpdated(body) + if body.status == UserStatus::Inactive.as_code())); + assert_eq!( + records[1], + WireMetadataChange::SessionClosed(WireSessionClosedBody { + user: alice, + client_id: CLIENT_A, + session, + reason: WireSessionClosedReason::Revoked, + }) + ); + assert!( + matches!(&records[2], WireMetadataChange::ConsumerGroupMemberLeft(body) + if body.client_id == CLIENT_A && body.reason == WireMemberLeftReason::SessionEnded) + ); + assert_eq!(actors(&committed), [Some(ROOT), Some(ROOT), Some(ROOT)]); +} + +#[test] +fn given_deleted_user_with_sessions_when_applying_should_close_each_session_in_client_order() { + let harness = Harness::new(); + harness.commit( + ROOT_CALLER, + Operation::CreateUser, + &create_user("alice", None), + ); + let second = harness.register(CLIENT_B, ALICE); + let first = harness.register(CLIENT_A, ALICE); + let alice = WireUserKey::from(&harness.user(ALICE)); + + let committed = harness.commit( + ROOT_CALLER, + Operation::DeleteUser, + &DeleteUserRequest { + user_id: WireIdentifier::numeric(ALICE), + }, + ); + + let closed = |client_id, session| { + WireMetadataChange::SessionClosed(WireSessionClosedBody { + user: alice, + client_id, + session, + reason: WireSessionClosedReason::Revoked, + }) + }; + let records = decoded(&committed); + assert!(matches!(&records[0], WireMetadataChange::UserDeleted(_))); + assert_eq!( + records[1..], + [closed(CLIENT_A, first), closed(CLIENT_B, second)] + ); + assert_eq!(actors(&committed), [Some(ROOT); 3]); +} + +#[test] +fn given_deleted_user_and_changed_password_when_applying_should_record_identity_only() { + let harness = Harness::new(); + harness.commit( + ROOT_CALLER, + Operation::CreateUser, + &create_user("alice", None), + ); + let alice = harness.user(ALICE); + let identity = WireUserIdentityBody { + key: WireUserKey::from(&alice), + username: name("alice"), + }; + let change_password = |new_password: &str| ChangePasswordRequest { + user_id: WireIdentifier::numeric(ALICE), + current_password: String::new(), + new_password: new_password.to_owned(), + }; + + let refused = harness.commit(ROOT_CALLER, Operation::ChangePassword, &change_password("")); + let changed = harness.commit( + ROOT_CALLER, + Operation::ChangePassword, + &change_password("new-hash"), + ); + let deleted = harness.commit( + ROOT_CALLER, + Operation::DeleteUser, + &DeleteUserRequest { + user_id: WireIdentifier::numeric(ALICE), + }, + ); + + assert_eq!(refused, NO_CHANGES); + assert_eq!( + decoded(&changed), + [WireMetadataChange::UserPasswordChanged(identity.clone())] + ); + assert_eq!( + decoded(&deleted), + [WireMetadataChange::UserDeleted(identity)] + ); + assert!(!contains(&changed[0].body, b"new-hash")); +} + +#[test] +fn given_permission_replacements_when_applying_should_record_only_a_different_set() { + let harness = Harness::new(); + for stream_name in ["orders", "audit", "billing"] { + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream(stream_name), + ); + } + harness.commit( + ROOT_CALLER, + Operation::CreateUser, + &create_user("alice", Some(stream_grants(&[1]))), + ); + let replace = |permissions: Option<WirePermissions>| UpdatePermissionsRequest { + user_id: WireIdentifier::numeric(ALICE), + permissions, + }; + + let equal = harness.commit( + ROOT_CALLER, + Operation::UpdatePermissions, + &replace(Some(stream_grants(&[1]))), + ); + let changed = harness.commit( + ROOT_CALLER, + Operation::UpdatePermissions, + &replace(Some(stream_grants(&[1, 2]))), + ); + + let alice = harness.user(ALICE); + assert_eq!(equal, NO_CHANGES); + assert_eq!( + decoded(&changed), + [WireMetadataChange::UserPermissionsUpdated( + WireUserPermissionsBody { + key: WireUserKey::from(&alice), + username: name("alice"), + permissions: alice.permissions.as_deref().map(permissions_to_wire), + reason: WirePermissionsUpdatedReason::Updated, + } + )] + ); +} + +#[test] +fn given_personal_access_tokens_when_created_and_deleted_should_record_names_without_secrets() { + let harness = Harness::new(); + let root = harness.user(ROOT); + let token_hash = [b'f'; PAT_TOKEN_HASH_BYTES]; + let create = |name: &str, expiry: u64| CreatePersonalAccessTokenRequest { + user_id: ROOT, + name: WireName::new(name).expect("valid name"), + expiry, + token_hash, + }; + let delete = |name: &str, only_if_expired: bool| DeletePersonalAccessTokenRequest { + user_id: ROOT, + name: WireName::new(name).expect("valid name"), + only_if_expired, + }; + + let created = harness.commit( + ROOT_CALLER, + Operation::CreatePersonalAccessToken, + &create("ci", 0), + ); + let renewed_skip = harness.commit( + SERVER, + Operation::DeletePersonalAccessToken, + &delete("ci", true), + ); + let deleted = harness.commit( + ROOT_CALLER, + Operation::DeletePersonalAccessToken, + &delete("ci", false), + ); + harness.commit( + ROOT_CALLER, + Operation::CreatePersonalAccessToken, + &create("short", 1), + ); + let expired = harness.commit_bytes( + SERVER, + Operation::DeletePersonalAccessToken, + &delete("short", true).to_bytes(), + |header| header.timestamp += 10_000_000_000, + ); + + assert_eq!( + decoded(&created), + [WireMetadataChange::PersonalAccessTokenCreated( + WireTokenBody { + user: WireUserKey::from(&root), + name: name("ci"), + expiry: 0, + } + )] + ); + assert!(!contains(&created[0].body, &token_hash)); + assert_eq!(renewed_skip, NO_CHANGES, "a token that never expires stays"); + assert_eq!( + decoded(&deleted), + [WireMetadataChange::PersonalAccessTokenDeleted( + WireTokenDeletedBody { + user: WireUserKey::from(&root), + name: name("ci"), + reason: WireTokenDeletedReason::Deleted, + } + )] + ); + assert_eq!( + decoded(&expired), + [WireMetadataChange::PersonalAccessTokenDeleted( + WireTokenDeletedBody { + user: WireUserKey::from(&root), + name: name("short"), + reason: WireTokenDeletedReason::Expired, + } + )] + ); + assert_eq!(actors(&expired), [None]); +} + +#[test] +fn given_first_registration_when_applying_should_record_session_with_its_epoch() { + let harness = Harness::new(); + + let committed = harness.commit_register(CLIENT_A, ROOT, ROOT_USER_CREATED_AT_MICROS); + let op = harness.last_op(); + + assert_eq!(harness.client_table.borrow().get_epoch(CLIENT_A), Some(op)); + assert_eq!( + decoded(&committed), + [WireMetadataChange::SessionRegistered(WireSessionBody { + user: WireUserKey::from(&harness.user(ROOT)), + client_id: CLIENT_A, + session: op, + })] + ); + assert_eq!(actors(&committed), [Some(ROOT)]); +} + +#[test] +fn given_unauthenticated_register_when_applying_should_record_nothing() { + let harness = Harness::new(); + + let committed = harness.commit_register(CLIENT_A, ROOT, ROOT_USER_CREATED_AT_MICROS + 1); + + assert_eq!(committed, NO_CHANGES); + assert!(!harness.client_table.borrow().contains(CLIENT_A)); +} + +#[test] +fn given_retried_register_when_applying_should_record_nothing_and_keep_epoch() { + let harness = Harness::new(); + let epoch = harness.register(CLIENT_A, ROOT); + + let retried = harness.commit_register(CLIENT_A, ROOT, ROOT_USER_CREATED_AT_MICROS); + + assert_eq!(retried, NO_CHANGES); + assert_eq!( + harness.client_table.borrow().get_epoch(CLIENT_A), + Some(epoch) + ); +} + +#[test] +fn given_logout_of_member_when_applying_should_record_session_closed_before_member_left() { + let harness = Harness::with_group(1); + let session = harness.register(CLIENT_A, ROOT); + harness.join(CLIENT_A, session); + let caller = Caller { + client: CLIENT_A, + user_id: ROOT, + session, + }; + + let committed = harness.commit_bytes(caller, Operation::Logout, &[], |_| {}); + let repeated = harness.commit_bytes(caller, Operation::Logout, &[], |_| {}); + + let records = decoded(&committed); + assert_eq!( + records[0], + WireMetadataChange::SessionClosed(WireSessionClosedBody { + user: WireUserKey::from(&harness.user(ROOT)), + client_id: CLIENT_A, + session, + reason: WireSessionClosedReason::Logout, + }) + ); + assert!( + matches!(&records[1], WireMetadataChange::ConsumerGroupMemberLeft(body) + if body.client_id == CLIENT_A && body.reason == WireMemberLeftReason::SessionEnded) + ); + assert_eq!(records.len(), 2); + assert_eq!(actors(&committed), [Some(ROOT), Some(ROOT)]); + assert!( + repeated.is_empty(), + "only the logout that ends the session records" + ); +} + +#[test] +fn given_lease_expiry_logout_when_applying_should_record_expired_close_without_actor() { + let harness = Harness::new(); + let session = harness.register(CLIENT_A, ROOT); + + let committed = harness.commit_bytes( + Caller { + client: CLIENT_A, + user_id: ROOT, + session, + }, + Operation::Logout, + &[], + |header| header.request = EXPIRED_SESSION_REQUEST_ID, + ); + + assert_eq!( + decoded(&committed), + [WireMetadataChange::SessionClosed(WireSessionClosedBody { + user: WireUserKey::from(&harness.user(ROOT)), + client_id: CLIENT_A, + session, + reason: WireSessionClosedReason::Expired, + })] + ); + assert_eq!(actors(&committed), [None]); +} + +#[test] +fn given_refused_denied_or_stale_operations_when_applying_should_record_nothing() { + let harness = Harness::new(); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("orders"), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateUser, + &create_user("alice", None), + ); + let session = harness.register(CLIENT_A, ALICE); + + let duplicate = harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("orders"), + ); + let denied = harness.commit( + Caller { + client: CLIENT_A, + user_id: ALICE, + session, + }, + Operation::CreateStream, + &create_stream("audit"), + ); + let stale = harness.commit( + Caller { + client: CLIENT_A, + user_id: ALICE, + session: session + 1, + }, + Operation::CreateStream, + &create_stream("audit"), + ); + + assert_eq!(duplicate, NO_CHANGES); + assert_eq!(denied, NO_CHANGES); + assert_eq!(stale, NO_CHANGES); + assert!( + harness + .mux + .streams() + .read(|streams| streams.items.len() == 1) + ); +} + +#[test] +fn given_sentinel_in_every_hidden_option_when_recording_should_never_encode_it() { + for scope in [ + OptionsScope::Stream, + OptionsScope::Topic, + OptionsScope::User, + ] { + for descriptor in descriptors_for(scope).filter(|descriptor| { + descriptor.visibility != OptionVisibility::Public + || descriptor.storage != OptionStorage::Map + }) { + let mut options = ResourceOptions::new(); + options.insert( + HeaderKey::from(descriptor.key), + OptionValue::explicit(HeaderValue::try_from(SENTINEL).expect("valid value")), + ); + let body = match scope { + OptionsScope::User => WireMetadataChange::UserCreated( + user_body(&User { + username: "alice".into(), + options, + ..User::default() + }) + .expect("valid user"), + ), + OptionsScope::Topic => { + let stream = Stream { + name: "orders".into(), + ..Stream::default() + }; + let topic = Topic { + name: "events".into(), + options, + ..Topic::default() + }; + WireMetadataChange::TopicCreated(topic_body(&stream, &topic).expect("valid")) + } + OptionsScope::Stream => WireMetadataChange::StreamCreated( + stream_body(&Stream { + name: "orders".into(), + options, + ..Stream::default() + }) + .expect("valid stream"), + ), + }; + assert!( + !contains(&body.body_bytes(), SENTINEL.as_bytes()), + "{scope:?} record echoed {:?}", + descriptor.key + ); + } + } +} + +#[test] +fn given_unreadable_critical_state_when_recording_should_mark_options_unreadable() { + let mut options = ResourceOptions::new(); + options.insert( + HeaderKey::from(OptionKey::EncryptionKeyWrapped), + OptionValue::explicit(HeaderValue::try_from(SENTINEL).expect("valid value")), + ); + + let body = stream_body(&Stream { + name: "orders".into(), + options, + ..Stream::default() + }) + .expect("valid stream"); + + assert_eq!(body.options, WireChangeOptions::Unreadable); +} + +#[test] +fn given_state_when_paging_one_item_at_a_time_should_equal_one_page_in_group_and_key_order() { + let harness = Harness::with_group(2); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("audit"), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateConsumerGroup, + &create_group("shipping"), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateUser, + &create_user("alice", None), + ); + let session = harness.register(CLIENT_A, ALICE); + harness.join(CLIENT_A, session); + harness.commit( + ROOT_CALLER, + Operation::CreatePersonalAccessToken, + &CreatePersonalAccessTokenRequest { + user_id: ALICE, + name: name("ci"), + expiry: 0, + token_hash: [b'f'; PAT_TOKEN_HASH_BYTES], + }, + ); + + let whole = harness.scan(&[], 1000); + let one_by_one = harness.scan(&[], 1); + + assert_eq!(one_by_one, whole); + let groups: Vec<_> = whole.iter().map(WireMetadataCurrentItem::group).collect(); + let mut sorted = groups.clone(); + sorted.sort(); + assert_eq!(groups, sorted, "items come in group order"); + let count = |group| groups.iter().filter(|item| **item == group).count(); + assert_eq!(count(WireMetadataChangeGroup::Streams), 2); + assert_eq!(count(WireMetadataChangeGroup::Topics), 1); + assert_eq!(count(WireMetadataChangeGroup::Partitions), 1); + assert_eq!(count(WireMetadataChangeGroup::ConsumerGroups), 2); + assert_eq!(count(WireMetadataChangeGroup::ConsumerGroupMembers), 1); + assert_eq!(count(WireMetadataChangeGroup::Users), 2); + assert_eq!(count(WireMetadataChangeGroup::PersonalAccessTokens), 1); + assert_eq!(count(WireMetadataChangeGroup::Sessions), 2); +} + +#[test] +fn given_delete_and_rename_between_pages_when_continuing_should_seek_past_the_last_key() { + let harness = Harness::new(); + for stream_name in ["orders", "audit", "billing"] { + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream(stream_name), + ); + } + let streams = [MetadataChangeGroup::Streams]; + let first = harness.current(&streams, None, 1, true); + let token = first + .current_page_token + .clone() + .expect("more streams follow"); + + harness.commit( + ROOT_CALLER, + Operation::DeleteStream, + &DeleteStreamRequest { + stream_id: WireIdentifier::numeric(0), + }, + ); + harness.commit( + ROOT_CALLER, + Operation::UpdateStreamResolved, + &UpdateStreamResolvedRequest { + stream_id: WireIdentifier::numeric(1), + name: Some(name("invoices")), + explicit_options: WireOptions::empty(), + derived_options: WireOptions::empty(), + }, + ); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("orders"), + ); + let second = harness.current(&streams, Some(&token), 10, true); + + let names: Vec<_> = second + .items + .iter() + .map(|item| match decoded_item(item) { + WireMetadataCurrentItem::Stream(body) => { + (body.key.stream_id, body.name.as_str().to_owned()) + } + other => panic!("expected a stream, got {other:?}"), + }) + .collect(); + assert_eq!( + names, + [(1, "invoices".to_owned()), (2, "billing".to_owned())], + "the deleted key still seeks, and the recreated id below it is a change after the anchor" + ); + assert_eq!(second.next, first.next, "every page carries the anchor"); + assert_eq!(second.current_page_token, None); +} + +#[test] +fn given_continuation_above_the_applied_head_when_paging_should_answer_behind_with_the_token() { + let harness = Harness::new(); + for stream_name in ["orders", "audit"] { + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream(stream_name), + ); + } + let streams = [MetadataChangeGroup::Streams]; + let mut token = harness + .current(&streams, None, 1, true) + .current_page_token + .expect("more streams follow"); + token.required_head = harness.last_op() + 1; + + let page = harness.current(&streams, Some(&token), 10, true); + + assert_eq!(page.status, WireMetadataChangeStatus::Behind); + assert_eq!(page.current_page_token, Some(token)); + assert_eq!(page.items, NO_ITEMS); +} + +#[test] +fn given_anchor_evicted_between_pages_when_continuing_should_expire_without_items() { + let harness = Harness::with_capacities(ChangeLogCapacities { + resource: 2, + activity: 16, + }); + for stream_name in ["orders", "audit"] { + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream(stream_name), + ); + } + let streams = [MetadataChangeGroup::Streams]; + let token = harness + .current(&streams, None, 1, true) + .current_page_token + .expect("more streams follow"); + for stream_name in ["billing", "shipping"] { + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream(stream_name), + ); + } + + let page = harness.current(&streams, Some(&token), 10, true); + + assert_eq!(page.status, WireMetadataChangeStatus::Expired); + assert_eq!(page.next, None); + assert_eq!(page.items, NO_ITEMS); + assert_eq!(page.current_page_token, None); +} + +#[test] +fn given_mismatched_current_anchor_when_continuing_should_reset_without_a_cursor() { + let harness = Harness::new(); + for stream_name in ["orders", "audit"] { + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream(stream_name), + ); + } + let streams = [MetadataChangeGroup::Streams]; + let token = harness + .current(&streams, None, 1, true) + .current_page_token + .expect("more streams follow"); + + for different_history in [true, false] { + let mut mismatched = token.clone(); + if different_history { + mismatched.anchor.log_id ^= 1; + } else { + mismatched.anchor.checksum ^= 1; + } + let page = harness.current(&streams, Some(&mismatched), 10, true); + + assert_eq!(page.status, WireMetadataChangeStatus::Reset); + assert_eq!(page.next, None); + assert_eq!(page.items, NO_ITEMS); + assert_eq!(page.current_page_token, None); + } +} + +#[test] +fn given_protection_table_ahead_when_paging_sessions_should_answer_behind() { + let harness = Harness::new(); + + let sessions = harness.current( + &[MetadataChangeGroup::Users, MetadataChangeGroup::Sessions], + None, + 10, + false, + ); + let users = harness.current(&[MetadataChangeGroup::Users], None, 10, false); + + assert_eq!(sessions.status, WireMetadataChangeStatus::Behind); + assert_eq!(users.status, WireMetadataChangeStatus::Ok); +} + +#[test] +fn given_sessions_and_tokens_when_paging_should_carry_identities_only() { + let harness = Harness::new(); + let token_hash = [b'f'; PAT_TOKEN_HASH_BYTES]; + harness.commit( + ROOT_CALLER, + Operation::CreatePersonalAccessToken, + &CreatePersonalAccessTokenRequest { + user_id: ROOT, + name: name("ci"), + expiry: 0, + token_hash, + }, + ); + + let page = harness.current( + &[ + MetadataChangeGroup::Users, + MetadataChangeGroup::PersonalAccessTokens, + MetadataChangeGroup::Sessions, + ], + None, + 100, + true, + ); + + let root = WireUserKey::from(&harness.user(ROOT)); + let items: Vec<_> = page.items.iter().map(decoded_item).collect(); + assert!( + items.contains(&WireMetadataCurrentItem::Session(WireSessionBody { + user: root, + client_id: ROOT_CLIENT, + session: 1, + })) + ); + assert!( + items.contains(&WireMetadataCurrentItem::PersonalAccessToken( + WireTokenBody { + user: root, + name: name("ci"), + expiry: 0, + } + )) + ); + for item in &page.items { + assert!(!contains(&item.body, &[0x5a; 32]), "no bind verifier"); + assert!(!contains(&item.body, &token_hash), "no token hash"); + assert!(!contains(&item.body, b"hash"), "no password hash"); + } +} + +#[test] +fn given_users_and_members_when_paging_should_match_their_read_projections() { + let harness = Harness::with_group(2); + harness.commit( + ROOT_CALLER, + Operation::CreateUser, + &create_user("alice", Some(stream_grants(&[0]))), + ); + let session = harness.register(CLIENT_A, ALICE); + harness.join(CLIENT_A, session); + for transition in harness.mux.streams().consumer_group_pending_revocations() { + harness.commit( + SERVER, + Operation::CompleteConsumerGroupRevocation, + &CompleteConsumerGroupRevocationRequest { + stream_id: WireIdentifier::numeric(transition.stream_id), + topic_id: WireIdentifier::numeric(transition.topic_id), + partition_id: transition.partition_id, + installation: transition.installation, + partition_op: 1, + }, + ); + } + + let items = harness.scan(&[], 100); + + let alice = harness.user(ALICE); + assert!(items.contains(&WireMetadataCurrentItem::User( + user_body(&alice).expect("valid user") + ))); + let details = harness + .mux + .streams() + .consumer_group_details( + &WireIdentifier::numeric(0), + &WireIdentifier::numeric(0), + &WireIdentifier::numeric(0), + ) + .expect("group exists"); + let members: Vec<_> = items + .iter() + .filter_map(|item| match item { + WireMetadataCurrentItem::ConsumerGroupMember(member) => { + Some((member.member_id, member.partition_ids.clone())) + } + _ => None, + }) + .collect(); + assert_eq!( + members, + [(details.members[0].id, details.members[0].partitions.clone())] + ); + assert_eq!(members[0].1, [0, 1]); +} + +#[derive(Clone, Copy)] +struct Caller { + client: u128, + user_id: u32, + session: u64, +} + +const ROOT_CALLER: Caller = Caller { + client: ROOT_CLIENT, + user_id: ROOT, + session: 0, +}; + +const SERVER: Caller = Caller { + client: 0, + user_id: ROOT, + session: 0, +}; + +struct Harness { + mux: MuxStateMachine<variadic!(Users, Streams)>, + client_table: RefCell<ClientTable>, + change_log: MetadataChangeLog, + op: Cell<u64>, +} + +impl Harness { + fn new() -> Self { + Self::with_capacities(ChangeLogCapacities::default()) + } + + fn with_capacities(capacities: ChangeLogCapacities) -> Self { + let users = Users::default(); + users.ensure_root_user("iggy", "hash"); + let streams = Streams::default(); + let harness = Self { + mux: MuxStateMachine::new(variadic!(users, streams)), + client_table: RefCell::new(ClientTable::new(CLIENTS_TABLE_MAX)), + change_log: MetadataChangeLog::new(capacities), + op: Cell::new(0), + }; + // A client commit without a session is a fatal receipt error. + harness.register(ROOT_CLIENT, ROOT); + harness + } + + /// A stream with a topic of `partitions` partitions and one group on it. + fn with_group(partitions: u32) -> Self { + let harness = Self::new(); + harness.commit( + ROOT_CALLER, + Operation::CreateStream, + &create_stream("orders"), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateTopicWithAssignments, + &create_topic("events", partitions), + ); + harness.commit( + ROOT_CALLER, + Operation::CreateConsumerGroup, + &create_group("billing"), + ); + harness + } + + fn last_op(&self) -> u64 { + self.op.get() + } + + fn commit( + &self, + caller: Caller, + operation: Operation, + request: &impl WireEncode, + ) -> Vec<AppliedChange> { + self.commit_bytes(caller, operation, &request.to_bytes(), |_| {}) + } + + fn commit_bytes( + &self, + caller: Caller, + operation: Operation, + body: &[u8], + adjust: impl FnOnce(&mut PrepareHeader), + ) -> Vec<AppliedChange> { + let op = self.op.get() + 1; + self.op.set(op); + let total = HEADER_SIZE + body.len(); + let mut buffer = Owned::<4096>::zeroed(total); + { + let header = bytemuck::checked::from_bytes_mut::<PrepareHeader>( + &mut buffer.as_mut_slice()[..HEADER_SIZE], + ); + header.command = Command::Prepare; + header.operation = operation; + header.op = op; + header.commit = op - 1; + header.timestamp = op * 1_000; + header.client = caller.client; + header.user_id = caller.user_id; + header.session = caller.session; + header.request = if operation == Operation::Register { + 0 + } else { + op + }; + header.retry_capacity = u32::try_from(CLIENTS_TABLE_MAX).expect("capacity"); + header.size = u32::try_from(total).expect("size"); + adjust(header); + } + buffer.as_mut_slice()[HEADER_SIZE..].copy_from_slice(body); + let prepare: Message<PrepareHeader> = Message::try_from(buffer).expect("valid prepare"); + let header = *prepare.header(); + let changes = + apply_committed_prepare(&self.mux, &self.client_table, true, |_| {}, prepare).changes; + self.change_log + .append(&header, changes.clone()) + .expect("committed ops append in order"); + changes + } + + /// One current state page over the applied state of this harness. + fn current( + &self, + groups: &[MetadataChangeGroup], + token: Option<&WireMetadataCurrentPageToken>, + limit: u32, + sessions_aligned: bool, + ) -> GetMetadataChangesResponse { + let table = self.client_table.borrow(); + self.mux.streams().read(|streams| { + self.mux.users().read(|users| { + self.change_log + .read(|log| { + current_page( + log, + self.last_op(), + MetadataChangeGroups::requested(groups), + token, + limit, + &CurrentState { + streams, + users, + sessions: sessions_aligned.then_some(&*table), + }, + ) + }) + .expect("readable") + .expect("valid page") + }) + }) + } + + /// Every item of a scan, page by page. + fn scan(&self, groups: &[MetadataChangeGroup], limit: u32) -> Vec<WireMetadataCurrentItem> { + let mut items = Vec::new(); + let mut token = None; + loop { + let page = self.current(groups, token.as_ref(), limit, true); + assert_eq!(page.status, WireMetadataChangeStatus::Ok); + assert!(page.items.len() <= limit as usize); + items.extend(page.items.iter().map(decoded_item)); + match page.current_page_token { + Some(next) => token = Some(next), + None => return items, + } + } + } + + fn commit_register( + &self, + client: u128, + user_id: u32, + user_created_at: u64, + ) -> Vec<AppliedChange> { + self.commit( + Caller { + client, + user_id, + session: 0, + }, + Operation::Register, + &RegisterRequest { + user_created_at, + bind_verifier: [0x5a; 32], + }, + ) + } + + /// Registers `client` for `user_id` and returns its session epoch. + fn register(&self, client: u128, user_id: u32) -> u64 { + let user_created_at = self.user(user_id).created_at.as_micros(); + self.commit_register(client, user_id, user_created_at); + self.client_table + .borrow() + .get_epoch(client) + .expect("registration commits") + } + + fn join(&self, client: u128, session: u64) -> Vec<AppliedChange> { + self.join_group(client, session, 0) + } + + fn join_group(&self, client: u128, session: u64, group_id: u32) -> Vec<AppliedChange> { + self.commit( + ROOT_CALLER, + Operation::JoinConsumerGroup, + &JoinConsumerGroupRequest { + stream_id: WireIdentifier::numeric(0), + topic_id: WireIdentifier::numeric(0), + group_id: WireIdentifier::numeric(group_id), + client_id: client, + session, + }, + ) + } + + /// Reports the fences of `partitions` of the intent at `intent_op` as + /// installed, from a server-internal client. + fn complete_lifecycle(&self, intent_op: u64, partitions: &[u32]) -> Vec<AppliedChange> { + let completions = partitions + .iter() + .map(|&partition_id| LifecycleCompletion { + topic_id: 0, + partition_id, + partition_op: 1, + }) + .collect(); + self.commit( + Caller { + client: message_bus::server_internal_client_id(intent_op), + user_id: ROOT, + session: 0, + }, + Operation::CompleteLifecycle, + &CompleteLifecycleRequest::new(intent_op, 0, completions).expect("valid batch"), + ) + } + + fn stream(&self, stream_id: usize) -> Stream { + self.mux + .streams() + .read(|streams| streams.items.get(stream_id).cloned()) + .expect("stream exists") + } + + fn user(&self, user_id: u32) -> User { + self.mux + .users() + .read(|users| users.items.get(user_id as usize).cloned()) + .expect("user exists") + } + + fn topic_key(&self) -> WireTopicKey { + let stream = self.stream(0); + stream.topic_key(&stream.topics[0]) + } +} + +fn decoded(changes: &[AppliedChange]) -> Vec<WireMetadataChange> { + changes + .iter() + .map(|change| { + WireMetadataChange::decode(change.group().as_u8(), change.kind.as_u8(), &change.body) + .expect("valid body") + .expect("known kind") + }) + .collect() +} + +fn decoded_item(item: &WireMetadataItem) -> WireMetadataCurrentItem { + WireMetadataCurrentItem::decode(item.group, &item.body) + .expect("valid item") + .expect("known group") +} + +fn actors(changes: &[AppliedChange]) -> Vec<Option<u32>> { + changes.iter().map(|change| change.actor_user_id).collect() +} + +fn contains(haystack: &[u8], needle: &[u8]) -> bool { + haystack + .windows(needle.len()) + .any(|window| window == needle) +} + +fn name(value: &str) -> WireName { + WireName::new(value).expect("valid name") +} + +/// The option blocks the read routes return for `stored`. +fn read_projection(scope: OptionsScope, stored: &ResourceOptions) -> WireChangeOptions { + let (explicit, derived) = + resource_options_to_wire_split(&public_options(scope, stored).expect("readable")) + .expect("encodable"); + WireChangeOptions::Readable { explicit, derived } +} + +fn key_context(key: &str, previous: &[&str]) -> ServerKeyContext { + let previous: Vec<String> = previous.iter().map(|key| (*key).to_owned()).collect(); + ServerKeyContext::from_config(key, &previous, true).expect("valid key") +} + +fn create_stream(stream_name: &str) -> CreateStreamRequest { + CreateStreamRequest { + name: name(stream_name), + options: WireOptions::empty(), + } +} + +fn encrypted_stream(stream_name: &str, set: &WrapSet) -> CreateStreamRequest { + let mut options = ResourceOptions::new(); + options.insert( + HeaderKey::from(OptionKey::EncryptionKeyWrapped), + OptionValue::explicit(HeaderValue::try_from(set.as_ref()).expect("valid set")), + ); + CreateStreamRequest { + name: name(stream_name), + options: resource_options_to_wire(&options, OptionsProvenance::All).expect("encodable"), + } +} + +fn create_topic(topic_name: &str, partitions: u32) -> CreateTopicWithAssignmentsRequest { + CreateTopicWithAssignmentsRequest { + request: CreateTopicRequest { + stream_id: WireIdentifier::numeric(0), + partitions_count: partitions, + name: name(topic_name), + options: WireOptions::empty(), + }, + derived_options: WireOptions::empty(), + partitions: assignments(partitions, 1), + } +} + +fn assignments(count: u32, first_group: u64) -> Vec<CreatedPartitionAssignment> { + (0..count) + .map(|partition_id| CreatedPartitionAssignment { + partition_id, + consensus_group_id: first_group + u64::from(partition_id), + initial_view: 0, + }) + .collect() +} + +fn create_group(group_name: &str) -> CreateConsumerGroupRequest { + CreateConsumerGroupRequest { + stream_id: WireIdentifier::numeric(0), + topic_id: WireIdentifier::numeric(0), + name: name(group_name), + } +} + +fn create_user(username: &str, permissions: Option<WirePermissions>) -> CreateUserRequest { + CreateUserRequest { + username: name(username), + password: "hash".to_owned(), + status: UserStatus::Active.as_code(), + permissions, + options: WireOptions::empty(), + } +} + +fn no_global() -> WireGlobalPermissions { + WireGlobalPermissions { + manage_servers: false, + read_servers: false, + manage_users: false, + read_users: false, + manage_streams: false, + read_streams: false, + manage_topics: false, + read_topics: false, + poll_messages: false, + send_messages: false, + } +} + +fn managing_topics() -> WirePermissions { + WirePermissions { + global: WireGlobalPermissions { + manage_topics: true, + read_topics: true, + ..no_global() + }, + streams: Vec::new(), + } +} + +fn stream_grants(stream_ids: &[u32]) -> WirePermissions { + WirePermissions { + global: no_global(), + streams: stream_ids + .iter() + .map(|&stream_id| WireStreamPermissions { + stream_id, + manage_stream: false, + read_stream: true, + manage_topics: false, + read_topics: false, + poll_messages: true, + send_messages: false, + topics: Vec::new(), + }) + .collect(), + } +} + +fn topic_grant(stream_id: u32, topic_id: u32) -> WirePermissions { + WirePermissions { + global: no_global(), + streams: vec![WireStreamPermissions { + stream_id, + manage_stream: false, + read_stream: true, + manage_topics: false, + read_topics: false, + poll_messages: false, + send_messages: false, + topics: vec![WireTopicPermissions { + topic_id, + manage_topic: false, + read_topic: true, + poll_messages: true, + send_messages: false, + }], + }], + } +}
diff --git a/core/metadata/src/impls/metadata.rs b/core/metadata/src/impls/metadata.rs index 372111e..edb7f38 100644 --- a/core/metadata/src/impls/metadata.rs +++ b/core/metadata/src/impls/metadata.rs
@@ -17,6 +17,10 @@ use crate::MuxStateMachine; use crate::applied_frontier::AppliedFrontier; +use crate::change_log::{ + AppliedChange, ChangeLogCapacities, ChangeLogSnapshot, CurrentState, MetadataChangeLog, + change_actor, change_page, current_page, session_closed, session_registered, session_revoked, +}; use crate::impls::placement::PlacementState; use crate::stm::authz::{ PartitionsCreate, admits_partitions_create, gated_apply, reject_stale_session, @@ -27,7 +31,7 @@ }; use crate::stm::stream::{Streams, TruncatePartitionRequest}; use crate::stm::user::{CreatePersonalAccessTokenRequest, DeletePersonalAccessTokenRequest, Users}; -use crate::stm::{ConsensusGroupAllocator, StateMachine}; +use crate::stm::{ApplyContext, ConsensusGroupAllocator, StateMachine}; use bytes::Bytes; use compio::io::AsyncWriteAtExt; use consensus::{ @@ -47,6 +51,7 @@ use futures::lock::Mutex; use iggy_binary_protocol::WireIdentifier; use iggy_binary_protocol::consensus::CHECKSUM_UNSEALED; +use iggy_binary_protocol::primitives::metadata_changes::WireUserKey; use iggy_binary_protocol::primitives::partition_assignment::CreatedPartitionAssignment; use iggy_binary_protocol::requests::partitions::CreatePartitionsRequest as WireCreatePartitionsRequest; use iggy_binary_protocol::requests::partitions::CreatePartitionsWithAssignmentsRequest as PersistedCreatePartitionsRequest; @@ -62,6 +67,7 @@ use iggy_binary_protocol::requests::topics::UpdateTopicResolvedRequest; use iggy_binary_protocol::requests::users::login_register::RegisterRequest; use iggy_binary_protocol::requests::users::{CreateUserRequest, UpdateUserResolvedRequest}; +use iggy_binary_protocol::responses::system::GetMetadataChangesResponse; use iggy_binary_protocol::{ Command, ConsensusHeader, EvictionReason, GenericHeader, Operation, PrepareHeader, PrepareOkHeader, ProtocolVersion, ReplyHeader, RoutedRequestHeader, WireDecode, WireEncode, @@ -71,10 +77,11 @@ use iggy_common::variadic; use iggy_common::wire_conversions::{resource_options_from_wire, resource_options_to_wire}; use iggy_common::{ - HeaderKey, IggyError, OptionAdmissionContext, OptionKey, OptionOperation, OptionValue, - OptionVisibility, OptionsProvenance, OptionsScope, ResourceOptions, ServerKeyContext, - TopicCreateOptions, TopicResolvedOptions, UserId, admit, critical_keys_marker, - derived_defaults, stream_wrap_set, topic_option_keys, validate_topic_segment_size, + HeaderKey, IggyError, MetadataChangeStart, MetadataChangesQuery, OptionAdmissionContext, + OptionKey, OptionOperation, OptionValue, OptionVisibility, OptionsProvenance, OptionsScope, + ResourceOptions, ServerKeyContext, TopicCreateOptions, TopicResolvedOptions, UserId, admit, + critical_keys_marker, derived_defaults, stream_wrap_set, topic_option_keys, + validate_topic_segment_size, }; use journal::local_gate::LocalGate; use journal::superblock::{ @@ -95,6 +102,7 @@ use std::path::Path; use std::rc::Rc; use std::sync::Arc; +use std::thread; use tracing::{debug, error, info, warn}; // The writer retains this gate after caller cancellation, until the rename is durable. @@ -484,6 +492,29 @@ u128::from(calculate_checksum(encoded)) } +/// The encoded snapshot and the checksum over exactly those bytes. +fn encode_with_checksum(snapshot: &IggySnapshot) -> Result<(Vec<u8>, u128), SnapshotError> { + let encoded = snapshot.encode()?; + let checksum = checkpoint_checksum(&encoded); + Ok((encoded, checksum)) +} + +/// [`encode_with_checksum`] on a dedicated thread. The runtime has no blocking +/// pool, and a full change log alone keeps a core busy for milliseconds. +async fn encode_on_own_thread(snapshot: IggySnapshot) -> Result<(Vec<u8>, u128), SnapshotError> { + let (sender, receiver) = oneshot::channel(); + thread::Builder::new() + .name("iggy-checkpoint-encode".to_owned()) + .spawn(move || { + // The receiver is gone only with its checkpoint, which then needs no result. + let _ = sender.send(encode_with_checksum(&snapshot)); + }) + .map_err(SnapshotError::Io)?; + receiver.await.map_err(|_| { + SnapshotError::InvalidState("checkpoint encode thread ended without a result") + })? +} + /// The highest resident WAL entry at or above the snapshot floor and below /// `bound`, or the snapshot floor with no checksum when no such entry remains. #[must_use] @@ -644,8 +675,10 @@ /// Create and durably persist a snapshot at `commit_op`, record the pairing, and /// return its checksum. Does NOT drain the WAL: the caller must durably record /// the pairing in the superblock first, so a crash between persist and drain - /// recovers a consistent checkpoint with the WAL intact. Snapshot creation and - /// encoding finish before the first await, preserving the captured frontier. + /// recovers a consistent checkpoint with the WAL intact. Snapshot creation + /// finishes before the first await, preserving the captured frontier. The owned + /// capture is then encoded on its own thread, so a large catalog or change log + /// does not stall this core. #[allow(clippy::future_not_send)] async fn persist_snapshot( &self, @@ -653,17 +686,21 @@ commit_op: u64, created_at: u64, client_table: Option<ClientTableSnapshot>, + change_log: ChangeLogSnapshot, ) -> Result<u128, SnapshotError> { let mut snapshot = (self.create_snapshot)(stm, commit_op, created_at)?; - // Fold in the client table, which `create_snapshot` does not see since it is - // not a state machine. Recovery restores it as the reconstruction floor for - // the drained WAL prefix. + // Fold in the client table and the change log, which `create_snapshot` + // does not see since neither is a state machine. Recovery restores the + // table as the reconstruction floor for the drained WAL prefix. snapshot.snapshot.client_table = client_table; + snapshot.snapshot.change_log = change_log; // Encode once: the checksum and the on-disk record share one buffer, so the - // snapshot is not serialized twice while the checkpoint lock freezes this - // core, and the pairing is provably over the bytes that reach the file. - let encoded = snapshot.encode()?; - let checksum = checkpoint_checksum(&encoded); + // pairing is provably over the bytes that reach the file. + let (encoded, checksum) = match self.io { + SnapshotIo::Compio => encode_on_own_thread(snapshot).await?, + #[cfg(feature = "simulator")] + SnapshotIo::Synchronous => encode_with_checksum(&snapshot)?, + }; self.write_snapshot(encoded).await?; self.last_checkpoint.set((commit_op, checksum)); Ok(checksum) @@ -852,6 +889,14 @@ slot } +/// What one committed prepare produced: the reply to cache and send, and the +/// public changes the change log records for it, in canonical order. +#[derive(Debug)] +pub struct CommittedApply { + pub reply: Message<ReplyHeader>, + pub changes: Vec<AppliedChange>, +} + /// Apply one committed prepare to the state machine and client table. /// /// The backup commit walk's per-op logic, shared so the simulator's WAL @@ -879,7 +924,7 @@ table_mutations_allowed: bool, fire_notifier: impl Fn(Operation), prepare: Message<PrepareHeader>, -) -> Message<ReplyHeader> +) -> CommittedApply where M: StreamsFrontend + StateMachine< @@ -914,10 +959,16 @@ .is_active_account(header.user_id, registration.user_created_at) { let denied = crate::stm::result::ApplyReply::err(IggyError::Unauthenticated.as_code()); - return build_reply_message_with(&header, denied.reply_body_len(), |dst| { - denied.write_reply_body(dst); - }); + return CommittedApply { + reply: build_reply_message_with(&header, denied.reply_body_len(), |dst| { + denied.write_reply_body(dst); + }), + changes: Vec::new(), + }; } + // A retry resolves to the existing entry and keeps its epoch, so only + // an inserted client is a new session. + let inserted = table_mutations_allowed && !client_table.borrow().contains(header.client); if table_mutations_allowed { let reply = build_reply_message(&header, &bytes::Bytes::new()); client_table @@ -939,13 +990,23 @@ .borrow() .get_epoch(header.client) .unwrap_or(header.op); - mux_stm.streams().refresh_consumer_group_session( + // The session record comes first, then the handovers its memberships + // took from the session change. + let mut changes: Vec<AppliedChange> = inserted + .then(|| session_registered(&header, registration.user_created_at, epoch)) + .into_iter() + .collect(); + changes.extend(mux_stm.streams().refresh_consumer_group_session( header.client, epoch, header.op, iggy_common::IggyTimestamp::from(header.timestamp), - ); - return build_reply_message(&header, &bytes::Bytes::new()); + change_actor(&ApplyContext::from(&header)), + )); + return CommittedApply { + reply: build_reply_message(&header, &bytes::Bytes::new()), + changes, + }; } if header.operation == Operation::Logout { let ended = if table_mutations_allowed { @@ -966,14 +1027,29 @@ } else { mux_stm.streams().consumer_group_session(header.client) == Some(header.session) }; + let mut changes = Vec::new(); if ended { - mux_stm.streams().remove_consumer_group_member( + // The session record comes first, then the memberships it ended. + let user = mux_stm.users().read(|users| { + users + .items + .get(header.user_id as usize) + .map(WireUserKey::from) + }); + let closed = user.map(|user| session_closed(&header, user)); + let actor = closed.as_ref().and_then(|closed| closed.actor_user_id); + changes.extend(closed); + changes.extend(mux_stm.streams().remove_consumer_group_member( header.client, iggy_common::IggyTimestamp::from(header.timestamp), header.op, - ); + actor, + )); } - return build_reply_message(&header, &bytes::Bytes::new()); + return CommittedApply { + reply: build_reply_message(&header, &bytes::Bytes::new()), + changes, + }; } if header.operation == Operation::FinalizeSession { let request = FinalizeSessionRequest::decode_from(prepare.body()).unwrap_or_else(|_| { @@ -988,12 +1064,15 @@ { client_table.borrow_mut().finalize_session(request.identity); } - return build_reply_message(&header, &bytes::Bytes::new()); + return CommittedApply { + reply: build_reply_message(&header, &bytes::Bytes::new()), + changes: Vec::new(), + }; } // Normal op: apply, build the reply. `Err` is decode/corruption only; a // business rejection commits as a deterministic no-op whose code rides // the reply body, replayed on retry. - let apply = reject_stale_session(&header, &client_table.borrow()) + let mut apply = reject_stale_session(&header, &client_table.borrow()) .map_or_else(|| gated_apply(mux_stm, prepare), Ok) .unwrap_or_else(|err| { panic!( @@ -1001,14 +1080,15 @@ header.op ); }); - if let Some(user_id) = apply.revoked_user { - apply_user_revocation( + if let Some(user) = apply.revoked_user { + let ended = apply_user_revocation( mux_stm, client_table, table_mutations_allowed, - user_id, + user, &header, ); + apply.changes.extend(ended); } fire_notifier(header.operation); let reply = build_reply_message_with(&header, apply.reply_body_len(), |dst| { @@ -1028,39 +1108,53 @@ .commit_reply(header.client, header.user_id, reply.clone()); log_commit_reply_outcome(outcome, header.client, header.op); } - reply + CommittedApply { + reply, + changes: apply.changes, + } } +/// Ends the user's sessions and returns their close records, then the records +/// of the memberships that ended with them. fn apply_user_revocation<M: StreamsFrontend>( mux_stm: &M, client_table: &RefCell<ClientTable>, table_mutations_allowed: bool, - user_id: u32, + user: WireUserKey, header: &PrepareHeader, -) { +) -> Vec<AppliedChange> { if table_mutations_allowed { client_table .borrow_mut() - .end_user_sessions(user_id, header.op); + .end_user_sessions(user.user_id, header.op); } // A transferred table may already contain this end marker while the // streams snapshot still needs the matching membership release. - let client_ids: Vec<u128> = { + let mut sessions: Vec<SessionIdentity> = { let table = client_table.borrow(); table .ended_sessions() .filter(|identity| { identity.metadata_watermark == header.op - && table.get_user_id(identity.client_id) == Some(user_id) + && table.get_user_id(identity.client_id) == Some(user.user_id) }) - .map(|identity| identity.client_id) .collect() }; - mux_stm.streams().remove_consumer_group_members( + // Slot order may differ after a restore, so every apply path orders by client. + sessions.sort_by_key(|identity| identity.client_id); + let actor = change_actor(&ApplyContext::from(header)); + let client_ids: Vec<u128> = sessions.iter().map(|identity| identity.client_id).collect(); + let mut changes: Vec<AppliedChange> = sessions + .iter() + .map(|identity| session_revoked(user, identity.client_id, identity.session, actor)) + .collect(); + changes.extend(mux_stm.streams().remove_consumer_group_members( &client_ids, iggy_common::IggyTimestamp::from(header.timestamp), header.op, - ); + actor, + )); + changes } /// Late-bound callback invoked after every committed op on shard 0's metadata @@ -1199,6 +1293,9 @@ /// This node's server keys. The primary refuses to prepare a /// `CreateStream` whose wrap set none of them opens. stream_key_context: Arc<ServerKeyContext>, + /// The public results of applied operations. The write side lives on + /// shard 0 with the state machines, every other shard reads it. + change_log: Rc<MetadataChangeLog>, } impl<B, J, S, M, SB> IggyMetadata<VsrConsensus<B>, J, S, M, SB> @@ -1267,10 +1364,79 @@ max_tokens_per_user: Cell::new(0), applied_frontier: Arc::default(), stream_key_context: Arc::default(), + change_log: Rc::new(MetadataChangeLog::new(ChangeLogCapacities::default())), } } } +impl<C, J, S, M, SB> IggyMetadata<C, J, S, M, SB> +where + M: StreamsFrontend, +{ + /// One change page of this node's log. Every shard answers it from its + /// read view, on an image coherent with the applied frontier. + /// + /// # Errors + /// `InvalidCommand` for a request the rules refuse or a current state + /// start, `TransientNotAccepted` when no coherent image was captured. + pub fn metadata_changes( + &self, + query: &MetadataChangesQuery, + ) -> Result<GetMetadataChangesResponse, IggyError> { + let groups = query.validate()?; + self.change_log + .read_coherent(&self.applied_frontier, |log, head_op| { + change_page(log, head_op, query, groups) + }) + .ok_or(IggyError::TransientNotAccepted)? + } + + /// One current state page, assembled in one synchronous pass over the + /// applied state. Only the metadata shard holds the client table and + /// writes the state, so only it answers. + /// + /// # Errors + /// `InvalidCommand` for a request the rules refuse or a change start, + /// `InvalidConfiguration` on any other shard, `TransientNotAccepted` once + /// the log's write side is gone. + pub fn metadata_current_state( + &self, + query: &MetadataChangesQuery, + ) -> Result<GetMetadataChangesResponse, IggyError> { + let groups = query.validate()?; + let MetadataChangeStart::Current(token) = &query.start else { + return Err(IggyError::InvalidCommand); + }; + if self.consensus.is_none() { + return Err(IggyError::InvalidConfiguration); + } + let head_op = self.applied_frontier.get(); + let client_table = self.client_table.borrow(); + // A transferred protection table can run ahead of the applied state. + let sessions = (self.client_table_frontier.get() <= head_op).then_some(&*client_table); + self.mux_stm.streams().read(|streams| { + self.mux_stm.users().read(|users| { + self.change_log + .read(|log| { + current_page( + log, + head_op, + groups, + token.as_ref().map(AsRef::as_ref), + query.limit, + &CurrentState { + streams, + users, + sessions, + }, + ) + }) + .ok_or(IggyError::TransientNotAccepted)? + }) + }) + } +} + impl<C, J, S, M, SB> IggyMetadata<C, J, S, M, SB> { /// Take and clear the gap-drop count (`prepare_gap_drops`). #[must_use = "dropping the count loses the only record those prepares existed"] @@ -1290,6 +1456,19 @@ self } + /// Install the change log: the one recovery rebuilt on shard 0, a read + /// view of it on every other shard. + #[must_use] + pub fn with_change_log(mut self, change_log: impl Into<Rc<MetadataChangeLog>>) -> Self { + self.change_log = change_log.into(); + self + } + + #[must_use] + pub const fn change_log(&self) -> &Rc<MetadataChangeLog> { + &self.change_log + } + /// Admit `CreateStream` wrap sets against this node's server keys, the /// ones its partitions open stream keys with. Without them every wrapped /// create is refused. @@ -1412,32 +1591,40 @@ Error = IggyError, >, { - let op = prepare.header().op; - if self.client_table_mutation_allowed(op) { - return apply_committed_prepare( + let header = *prepare.header(); + let op = header.op; + let committed = if self.client_table_mutation_allowed(op) { + apply_committed_prepare( &*self.mux_stm, &self.client_table, true, |operation| self.fire_commit_notifier(operation), prepare, + ) + } else { + let history = self.replay_client_table.borrow(); + let table = history + .as_ref() + .expect("transferred history must retain its replay table"); + let committed = apply_committed_prepare( + &*self.mux_stm, + table, + true, + |operation| self.fire_commit_notifier(operation), + prepare, ); + drop(history); + if op == self.client_table_frontier.get() { + self.replay_client_table.borrow_mut().take(); + } + committed + }; + // Published before the caller advances the applied frontier and + // `commit_min`, so no reader sees a record above them. + if let Err(error) = self.change_log.append(&header, committed.changes) { + panic!("apply_prepare: change log refused committed metadata op={op}: {error}"); } - let history = self.replay_client_table.borrow(); - let table = history - .as_ref() - .expect("transferred history must retain its replay table"); - let reply = apply_committed_prepare( - &*self.mux_stm, - table, - true, - |operation| self.fire_commit_notifier(operation), - prepare, - ); - drop(history); - if op == self.client_table_frontier.get() { - self.replay_client_table.borrow_mut().take(); - } - reply + committed.reply } /// Raise the forced-checkpoint margin to cover a configured @@ -2599,8 +2786,17 @@ local_applied = consensus.commit_min(); snapshot_ahead = snapshot_seq > local_applied; if snapshot_ahead { + let capacities = self.change_log.capacities().map_err(|_| { + SnapshotError::InvalidState("change log restore reached a read-only view") + })?; + let change_log = snapshot.snapshot().change_log.restore(capacities)?; self.mux_stm .restore_snapshot_in_place(snapshot.snapshot())?; + // Replaced with the state machines and before the frontier + // jumps. The kept WAL suffix appends through the commit walk. + self.change_log.restore(change_log).map_err(|_| { + SnapshotError::InvalidState("change log restore reached a read-only view") + })?; } } else { tracing::info!( @@ -4218,6 +4414,7 @@ /// to its append regardless (see the phantom-op comment at the call /// site). The next prepare over the boundary simply retries. #[allow(clippy::future_not_send)] + #[allow(clippy::too_many_lines)] async fn checkpoint_if_needed(&self, consensus: &VsrConsensus<B, P>, journal: &J) { let Some(coordinator) = &self.coordinator else { return; @@ -4271,13 +4468,20 @@ // then-current commit_min. The prepare being replicated appends regardless // (see the phantom-op comment at the call site). let client_table = self.client_table.borrow().to_snapshot(); + let change_log = self.change_log.snapshot(); #[cfg(test)] if checkpoint_boundary(CheckpointBoundary::TableCaptured).is_err() { return; } self.clear_state_transfer_offer_cache(); let checksum = match coordinator - .persist_snapshot(&*self.mux_stm, snap_op, created_at, Some(client_table)) + .persist_snapshot( + &*self.mux_stm, + snap_op, + created_at, + Some(client_table), + change_log, + ) .await { Ok(checksum) => checksum, @@ -5650,6 +5854,9 @@ use crate::stm::user::{PAT_TOKEN_HASH_BYTES, Users, UsersInner}; use consensus::{HistoryRecovery, LocalPipeline}; use iggy_binary_protocol::codes::CREATE_USER_CODE; + use iggy_binary_protocol::primitives::metadata_changes::{ + WireMetadataChangeKind, WireMetadataChangeStatus, + }; use iggy_binary_protocol::primitives::permissions::{ WireGlobalPermissions, WirePermissions, WireStreamPermissions, }; @@ -5658,6 +5865,7 @@ use iggy_binary_protocol::requests::topics::{CreateTopicRequest, DeleteTopicRequest}; use iggy_binary_protocol::requests::users::{CreateUserRequest, DeleteUserRequest}; use iggy_binary_protocol::{RequestHeader, StartViewHeader, WireOptions}; + use iggy_common::MetadataChangeGroup; use iggy_common::{ HeaderValue, IggyExpiry, IggyTimestamp, OptionKey, OptionStorage, OptionValueCodec, ServerKeyContext, StreamEncryptionKey, StreamKeyOrigin, TopicUpdateOptions, UserStatus, @@ -5944,6 +6152,154 @@ )); } + #[test] + fn given_log_and_frontier_when_reading_changes_should_answer_only_a_coherent_image() { + let metadata = metadata_plane(); + let latest = MetadataChangesQuery { + start: MetadataChangeStart::Latest, + ..MetadataChangesQuery::default() + }; + let header = |op| PrepareHeader { + op, + checksum: 0x5eed, + timestamp: op, + ..PrepareHeader::default() + }; + assert_eq!( + metadata.metadata_changes(&latest).unwrap().status, + WireMetadataChangeStatus::Behind, + "no op 1 applied yet" + ); + + metadata + .change_log() + .append( + &header(1), + vec![AppliedChange { + kind: WireMetadataChangeKind::UserCreated, + actor_user_id: None, + body: Bytes::new(), + }], + ) + .unwrap(); + assert_eq!( + metadata.metadata_changes(&latest), + Err(IggyError::TransientNotAccepted), + "a log ahead of the frontier is not a coherent image" + ); + metadata.advance_applied_frontier(1); + let page = metadata.metadata_changes(&latest).unwrap(); + assert_eq!(page.status, WireMetadataChangeStatus::Ok); + assert_eq!(page.head_op, 1); + assert_eq!(page.next.map(|next| (next.op, next.ordinal)), Some((1, 0))); + + let current = MetadataChangesQuery { + start: MetadataChangeStart::Current(None), + groups: vec![MetadataChangeGroup::Streams], + ..MetadataChangesQuery::default() + }; + assert_eq!( + metadata.metadata_changes(¤t), + Err(IggyError::InvalidCommand) + ); + assert_eq!( + metadata.metadata_current_state(¤t).unwrap().status, + WireMetadataChangeStatus::Ok + ); + assert_eq!( + peer_metadata().metadata_current_state(¤t), + Err(IggyError::InvalidConfiguration), + "only the metadata shard holds the client table" + ); + } + + #[compio::test] + async fn given_state_transfer_when_installing_should_replace_the_change_log_only_with_the_state_machines() + { + const TRANSFER_OP: u64 = 2; + let metadata = metadata_plane(); + let local = |op, checksum| PrepareHeader { + op, + checksum, + timestamp: op, + ..PrepareHeader::default() + }; + metadata + .change_log() + .append(&local(1, 0xaa), Vec::new()) + .unwrap(); + let before = metadata.change_log().snapshot(); + let transferred = |sequence_number, log_id| { + let mut snapshot = + IggySnapshot::create(&TestMux::default(), sequence_number, 1).unwrap(); + snapshot.snapshot_mut().client_table = + Some(ClientTable::new(CLIENTS_TABLE_MAX).to_snapshot()); + let log = MetadataChangeLog::new(ChangeLogCapacities::default()); + log.append(&local(1, log_id), Vec::new()).unwrap(); + log.append( + &local(sequence_number, log_id), + vec![AppliedChange { + kind: WireMetadataChangeKind::StreamCreated, + actor_user_id: None, + body: Bytes::from_static(b"body"), + }], + ) + .unwrap(); + snapshot.snapshot_mut().change_log = log.snapshot(); + snapshot + }; + + let incoherent = transferred(TRANSFER_OP, 0xbb).encode().unwrap(); + assert!(matches!( + metadata + .install_state_transfer( + &incoherent, + ClientTable::new(CLIENTS_TABLE_MAX), + TRANSFER_OP, + TRANSFER_OP - 1, + ) + .await, + Err(SnapshotError::IncoherentManifest { .. }) + )); + assert_eq!( + metadata.change_log().snapshot(), + before, + "a refused transfer leaves the log untouched" + ); + + let ahead = transferred(TRANSFER_OP, 0xbb); + metadata + .install_state_transfer( + &ahead.encode().unwrap(), + ClientTable::new(CLIENTS_TABLE_MAX), + TRANSFER_OP, + TRANSFER_OP, + ) + .await + .unwrap(); + assert_eq!( + metadata.change_log().snapshot(), + ahead.snapshot().change_log, + "the transfer replaces the log with the state machines" + ); + + let behind = transferred(TRANSFER_OP, 0xcc); + metadata + .install_state_transfer( + &behind.encode().unwrap(), + ClientTable::new(CLIENTS_TABLE_MAX), + TRANSFER_OP, + TRANSFER_OP, + ) + .await + .unwrap(); + assert_eq!( + metadata.change_log().snapshot(), + ahead.snapshot().change_log, + "a transfer at the applied frontier keeps the local state and log" + ); + } + /// The offer ships state at or below `commit_min`, which a recovered suffix /// above `commit_max` cannot change, so the barrier waiting on that suffix must /// not withhold it. @@ -9426,7 +9782,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap_or_else(|error| panic!("{boundary:?}: {error}")); @@ -10587,6 +10943,17 @@ .unwrap() .0; assert_eq!(checkpoint.sequence_number(), pipelined_op); + let live_log = metadata.change_log().snapshot(); + assert_eq!( + checkpoint.snapshot().change_log, + live_log, + "{revocation:?}: the checkpoint must carry the applied change log" + ); + assert_eq!( + live_log.activity.entries.len(), + 3, + "{revocation:?}: both registrations and the revoked session are recorded" + ); let receiver = metadata_plane(); receiver @@ -10602,6 +10969,7 @@ receiver.client_table.borrow().encode() == table, "{revocation:?}: the receiver must hold the transferred table" ); + assert_eq!(receiver.change_log().snapshot(), live_log); drop(metadata); let recovered = crate::impls::recovery::recover::<TestMux>( @@ -10614,7 +10982,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .expect("restart must accept the checkpointed table"); @@ -10622,6 +10990,7 @@ recovered.client_table.ended_sessions().collect::<Vec<_>>(), ended ); + assert_eq!(recovered.change_log.snapshot(), live_log); } } @@ -10696,7 +11065,10 @@ &*metadata.mux_stm, &metadata.client_table, true, - GROUP_MEMBER_USER, + WireUserKey { + user_id: GROUP_MEMBER_USER, + user_created_at: 0, + }, &header, ); let seen = observer.join().unwrap(); @@ -11577,7 +11949,7 @@ JOURNAL_SLOTS, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -11884,7 +12256,7 @@ DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -11972,7 +12344,7 @@ DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -12296,7 +12668,7 @@ DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -12414,7 +12786,7 @@ DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -12447,7 +12819,7 @@ DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -12505,7 +12877,7 @@ DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -12579,7 +12951,7 @@ DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap();
diff --git a/core/metadata/src/impls/recovery.rs b/core/metadata/src/impls/recovery.rs index 13aeb1e..63b4312 100644 --- a/core/metadata/src/impls/recovery.rs +++ b/core/metadata/src/impls/recovery.rs
@@ -15,10 +15,14 @@ // specific language governing permissions and limitations // under the License. +use crate::change_log::{ + ChangeLogCapacities, ChangeLogError, MetadataChangeLog, change_actor, session_closed, + session_registered, session_revoked, +}; use crate::impls::metadata::IggySnapshot; -use crate::stm::StateMachine; use crate::stm::authz::{GatedApply, reject_stale_session}; use crate::stm::snapshot::{MetadataSnapshot, RestoreSnapshot, Snapshot, SnapshotError}; +use crate::stm::{ApplyContext, StateMachine}; use consensus::{ ClientTable, ClientTableDecodeError, CommitReply, HistoryRecovery, VsrState, VsrStateError, build_reply_message, build_reply_message_with, @@ -43,6 +47,9 @@ Snapshot(SnapshotError), Journal(JournalError), StateMachine(IggyError), + /// Replay produced change records the log cannot take, which only a bug + /// in record production can cause. + ChangeLog(ChangeLogError), Io(std::io::Error), DisconnectedSoloJournal { path: PathBuf, @@ -152,6 +159,7 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { match self { Self::Snapshot(e) => write!(f, "recovery snapshot error: {e}"), + Self::ChangeLog(e) => write!(f, "recovery change log error: {e}"), Self::Journal(e) => write!(f, "recovery journal error: {e}"), Self::StateMachine(e) => write!(f, "recovery state machine error: {e}"), Self::Io(e) => write!(f, "recovery I/O error: {e}"), @@ -239,6 +247,7 @@ Self::Snapshot(e) => Some(e), Self::Journal(e) => Some(e), Self::StateMachine(e) => Some(e), + Self::ChangeLog(e) => Some(e), Self::Io(e) | Self::Superblock { source: e, .. } => Some(e), Self::ClientTableDecode(e) => Some(e), Self::RetryProtection(e) => Some(e), @@ -282,6 +291,12 @@ } } +impl From<ChangeLogError> for RecoveryError { + fn from(e: ChangeLogError) -> Self { + Self::ChangeLog(e) + } +} + /// Result of a successful metadata recovery. pub struct RecoveredMetadata<M> { pub journal: PrepareJournal, @@ -304,6 +319,9 @@ /// post-boot view-change superblock write records the real pairing. pub snapshot_checkpoint: (u64, u128), pub mux_stm: M, + /// Change log restored from the checkpoint and advanced by the replayed + /// committed suffix, so it equals the log the node held before it stopped. + pub change_log: MetadataChangeLog, /// Client table restored from the checkpoint and advanced by the replayed /// committed suffix: registers carry their own commit op as the epoch, so /// replay reads fences out of the log rather than deriving them from replay @@ -418,7 +436,7 @@ journal_slots: usize, clients_table_max: usize, seed_baseline: impl FnOnce(&M), - on_replayed_session: impl Fn(&M, &PrepareHeader), + change_log_capacities: ChangeLogCapacities, ) -> Result<RecoveredMetadata<M>, RecoveryError> where M: StateMachine<Input = Message<PrepareHeader>, Error = IggyError> @@ -547,6 +565,16 @@ seed_baseline(&mux_stm); mux_stm }; + // Decode already checked the log against the snapshot's sequence number. + let change_log = MetadataChangeLog::new(change_log_capacities); + if let Some(snapshot) = snapshot.as_ref() { + change_log.restore( + snapshot + .snapshot() + .change_log + .restore(change_log_capacities)?, + )?; + } let journal_path = metadata_dir.join("journal.wal"); let watermark = snapshot.as_ref().map_or(0, IggySnapshot::sequence_number); @@ -681,9 +709,11 @@ RecoveryError::RetryProtection(consensus::ClientTableWireError::InvalidReply) })?; if !mux_stm.is_active_account(header.user_id, registration.user_created_at) { + change_log.append(header, Vec::new())?; last_applied_op = Some(header.op); continue; } + let inserted = !client_table.contains(header.client); let reply = build_reply_message(header, &bytes::Bytes::new()); client_table .commit_register( @@ -693,11 +723,22 @@ reply, ) .map_err(RecoveryError::RetryProtection)?; - on_replayed_session(&mux_stm, header); + let epoch = client_table.get_epoch(header.client).unwrap_or(header.op); + let mut changes: Vec<_> = inserted + .then(|| session_registered(header, registration.user_created_at, epoch)) + .into_iter() + .collect(); + changes.extend(mux_stm.refresh_session( + header, + epoch, + change_actor(&ApplyContext::from(header)), + )); + change_log.append(header, changes)?; last_applied_op = Some(header.op); continue; } if header.operation == Operation::Logout { + let mut changes = Vec::new(); if client_table .commit_logout( header.client, @@ -707,8 +748,16 @@ ) .map_err(RecoveryError::RetryProtection)? { - on_replayed_session(&mux_stm, header); + // The order the commit walk records: the session, then the + // memberships it ended. + let closed = mux_stm + .user_key(header.user_id) + .map(|user| session_closed(header, user)); + let actor = closed.as_ref().and_then(|closed| closed.actor_user_id); + changes.extend(closed); + changes.extend(mux_stm.end_memberships(&[header.client], header, actor)); } + change_log.append(header, changes)?; last_applied_op = Some(header.op); continue; } @@ -725,6 +774,7 @@ if mux_stm.namespace_revision() == request.namespace_revision { client_table.finalize_session(request.identity); } + change_log.append(header, Vec::new())?; last_applied_op = Some(header.op); continue; } @@ -737,19 +787,20 @@ })?; // WAL replay must recompute authorization denials identically to the // primary/backup commit paths, so it goes through the same gate. - let reply = reject_stale_session(header, &client_table) + let mut reply = reject_stale_session(header, &client_table) .map_or_else(|| mux_stm.gated_update(entry), Ok)?; - if let Some(user_id) = reply.revoked_user { - for identity in client_table.end_user_sessions(user_id, header.op) { - on_replayed_session( - &mux_stm, - &PrepareHeader { - client: identity.client_id, - session: identity.session, - ..*header - }, - ); - } + if let Some(user) = reply.revoked_user { + let mut sessions = client_table.end_user_sessions(user.user_id, header.op); + sessions.sort_by_key(|identity| identity.client_id); + let actor = change_actor(&ApplyContext::from(header)); + let client_ids: Vec<u128> = + sessions.iter().map(|identity| identity.client_id).collect(); + reply.changes.extend(sessions.iter().map(|identity| { + session_revoked(user, identity.client_id, identity.session, actor) + })); + reply + .changes + .extend(mux_stm.end_memberships(&client_ids, header, actor)); } // Every client commit must restore its original receipt before admission. if header.client != consensus::client_table::RESERVED_CLIENT_ID @@ -780,6 +831,7 @@ user_id = header.user_id, "recovery replayed op" ); + change_log.append(header, reply.changes)?; last_applied_op = Some(header.op); } @@ -790,6 +842,7 @@ recovered_state, snapshot_checkpoint: (snapshot_op, snapshot_checksum), mux_stm, + change_log, client_table, last_applied_op, last_journaled_op, @@ -903,19 +956,26 @@ mod tests { use super::*; use crate::impls::metadata::{StreamsFrontend, apply_committed_prepare, checkpoint_checksum}; + use crate::stm::consumer_group::JoinConsumerGroupRequest; use crate::stm::lifecycle::{CompleteLifecycleRequest, LifecycleCompletion}; use crate::stm::snapshot::{SNAPSHOT_FORMAT_VERSION, Snapshotable}; use crate::stm::stream::{Streams, StreamsSnapshot}; use crate::stm::user::{CreatePersonalAccessTokenRequest, PAT_TOKEN_HASH_BYTES, Users}; use consensus::{CLIENTS_TABLE_MAX, HistoryCertificate}; use iggy_binary_protocol::consensus::{Command, Operation}; + use iggy_binary_protocol::primitives::metadata_changes::WireMetadataChangeKind; + use iggy_binary_protocol::primitives::partition_assignment::CreatedPartitionAssignment; use iggy_binary_protocol::primitives::permissions::{WireGlobalPermissions, WirePermissions}; + use iggy_binary_protocol::requests::consumer_groups::CreateConsumerGroupRequest; use iggy_binary_protocol::requests::streams::CreateStreamRequest; + use iggy_binary_protocol::requests::topics::{ + CreateTopicRequest, CreateTopicWithAssignmentsRequest, + }; use iggy_binary_protocol::requests::users::{ CreateUserRequest, DeleteUserRequest, UpdateUserResolvedRequest, }; use iggy_binary_protocol::{WireEncode, WireIdentifier, WireName, WireOptions, result_code}; - use iggy_common::UserStatus; + use iggy_common::{MetadataChangeClass, UserStatus}; use journal::Journal; use server_common::iobuf::Owned; use std::cell::RefCell; @@ -1099,7 +1159,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -1126,7 +1186,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -1162,7 +1222,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -1225,7 +1285,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -1264,7 +1324,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -1316,7 +1376,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -1370,7 +1430,7 @@ assert!(matches!( recover::<TestStm>(directory.path(), SOLO, journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, - |_| {}, |_, _| {}).await, + |_| {}, ChangeLogCapacities::default()).await, Err(RecoveryError::DisconnectedSoloJournal { op, .. }) if op == if forked { 101 } else { 179 } )); @@ -1411,7 +1471,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -1524,7 +1584,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -1605,7 +1665,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -1652,7 +1712,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await; @@ -1703,7 +1763,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -1742,7 +1802,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -1794,7 +1854,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await; assert!(matches!( @@ -1875,7 +1935,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -1970,7 +2030,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -2026,7 +2086,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -2078,18 +2138,13 @@ journal.storage_ref().fsync().await.unwrap(); } - let replayed_logouts = std::cell::RefCell::new(Vec::new()); let recovered = recover::<TestStm>( dir.path(), SOLO, journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, header| { - if header.operation == Operation::Logout { - replayed_logouts.borrow_mut().push(header.client); - } - }, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -2111,10 +2166,23 @@ .check_request(CLIENT, 1, 2, Operation::CreateStream), consensus::client_table::RequestStatus::NoSession )); + let sessions = recovered + .change_log + .read(|log| { + log.ring(MetadataChangeClass::Activity) + .entries() + .iter() + .map(|entry| (entry.op, entry.kind)) + .collect::<Vec<_>>() + }) + .expect("readable"); assert_eq!( - replayed_logouts.into_inner(), - vec![CLIENT], - "replay must hand the logged-out client to the group-removal hook" + sessions, + [ + (1, WireMetadataChangeKind::SessionRegistered), + (2, WireMetadataChangeKind::SessionClosed) + ], + "replay must record the session the logout ended" ); assert_eq!(recovered.last_applied_op, Some(2)); } @@ -2309,7 +2377,7 @@ )); parent = prepare.header().checksum; let reply = - apply_committed_prepare(&stm, &table, true, |_| {}, prepare.clone()); + apply_committed_prepare(&stm, &table, true, |_| {}, prepare.clone()).reply; let status = if operation == Operation::Register { result_code(reply.body()).unwrap_or(0) } else { @@ -2339,7 +2407,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |stm| stm.users().ensure_root_user("root", "hash"), - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -2453,6 +2521,7 @@ let stm = UserStm::default(); stm.users().ensure_root_user("root", "hash"); let table = RefCell::new(ClientTable::new(CLIENTS_TABLE_MAX)); + let live_log = MetadataChangeLog::new(ChangeLogCapacities::default()); let mut last_reply = None; let mut parent = 0; { @@ -2484,13 +2553,12 @@ }, )); parent = prepare.header().checksum; - last_reply = Some(apply_committed_prepare( - &stm, - &table, - true, - |_| {}, - prepare.clone(), - )); + let committed = + apply_committed_prepare(&stm, &table, true, |_| {}, prepare.clone()); + live_log + .append(prepare.header(), committed.changes) + .expect("live ops append in order"); + last_reply = Some(committed.reply); journal.append(prepare).await.unwrap(); } journal.storage_ref().fsync().await.unwrap(); @@ -2503,7 +2571,6 @@ } let live = table.borrow().encode(); ClientTable::decode(&live).expect("the state transfer form must decode"); - let revoked = RefCell::new(Vec::new()); let recovered = recover::<UserStm>( directory.path(), SOLO, @@ -2512,13 +2579,7 @@ |stm| { stm.users().ensure_root_user("root", "hash"); }, - |_, header| { - if header.operation == operation { - revoked - .borrow_mut() - .push((header.client, header.session, header.op)); - } - }, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -2532,9 +2593,13 @@ recovered.client_table.get_epoch(ROOT_CLIENT), Some(ROOT_SESSION) ); + assert_eq!( + recovered.change_log.snapshot(), + live_log.snapshot(), + "{operation:?}: WAL replay must rebuild the live change log" + ); let ended: Vec<_> = recovered.client_table.ended_sessions().collect(); if register_first { - assert_eq!(*revoked.borrow(), vec![(CLIENT, SESSION, REVOCATION_OP)]); assert_eq!( ended, vec![iggy_binary_protocol::requests::system::SessionIdentity { @@ -2544,7 +2609,6 @@ }] ); } else { - assert!(revoked.borrow().is_empty()); assert_eq!( ended, [] as [iggy_binary_protocol::requests::system::SessionIdentity; 0] @@ -2564,7 +2628,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |stm| stm.users().ensure_root_user("root", "hash"), - |_, _| {}, + ChangeLogCapacities::default(), ) .await .expect("restart must accept the checkpointed table"); @@ -2576,6 +2640,216 @@ } #[compio::test] + #[allow(clippy::too_many_lines)] + async fn given_memberships_ended_by_logout_and_revocation_when_replaying_wal_should_rebuild_the_live_change_log() + { + const ROOT_CLIENT: u128 = 0x10; + const CLIENT_A: u128 = 0x11; + const CLIENT_B: u128 = 0x12; + const USER: u32 = 1; + // Epochs are the ops of the registrations below. + const ROOT_SESSION: u64 = 1; + const SESSION_A: u64 = 6; + const SESSION_B: u64 = 7; + type UserStm = MuxStateMachine<iggy_common::variadic!(Users, Streams)>; + let register = |user_created_at| { + RegisterRequest { + user_created_at, + bind_verifier: [0x5a; 32], + } + .to_bytes() + }; + let join = |client_id, session| { + JoinConsumerGroupRequest { + stream_id: WireIdentifier::numeric(0), + topic_id: WireIdentifier::numeric(0), + group_id: WireIdentifier::numeric(0), + client_id, + session, + } + .to_bytes() + }; + let entries = [ + (Operation::Register, register(1), ROOT_CLIENT, 0, 0), + ( + Operation::CreateUser, + CreateUserRequest { + username: WireName::new("user").unwrap(), + password: "hash".to_owned(), + status: UserStatus::Active.as_code(), + permissions: None, + options: WireOptions::empty(), + } + .to_bytes(), + ROOT_CLIENT, + 0, + ROOT_SESSION, + ), + ( + Operation::CreateStream, + CreateStreamRequest { + name: WireName::new("orders").unwrap(), + options: WireOptions::empty(), + } + .to_bytes(), + ROOT_CLIENT, + 0, + ROOT_SESSION, + ), + ( + Operation::CreateTopicWithAssignments, + CreateTopicWithAssignmentsRequest { + request: CreateTopicRequest { + stream_id: WireIdentifier::numeric(0), + partitions_count: 2, + name: WireName::new("events").unwrap(), + options: WireOptions::empty(), + }, + derived_options: WireOptions::empty(), + partitions: (0..2) + .map(|partition_id| CreatedPartitionAssignment { + partition_id, + consensus_group_id: u64::from(partition_id) + 1, + initial_view: 0, + }) + .collect(), + } + .to_bytes(), + ROOT_CLIENT, + 0, + ROOT_SESSION, + ), + ( + Operation::CreateConsumerGroup, + CreateConsumerGroupRequest { + stream_id: WireIdentifier::numeric(0), + topic_id: WireIdentifier::numeric(0), + name: WireName::new("billing").unwrap(), + } + .to_bytes(), + ROOT_CLIENT, + 0, + ROOT_SESSION, + ), + // The user was created at op 2, whose timestamp is 2. + (Operation::Register, register(2), CLIENT_A, USER, 0), + (Operation::Register, register(2), CLIENT_B, USER, 0), + ( + Operation::JoinConsumerGroup, + join(CLIENT_A, SESSION_A), + ROOT_CLIENT, + 0, + ROOT_SESSION, + ), + ( + Operation::JoinConsumerGroup, + join(CLIENT_B, SESSION_B), + ROOT_CLIENT, + 0, + ROOT_SESSION, + ), + ( + Operation::Logout, + bytes::Bytes::new(), + CLIENT_A, + USER, + SESSION_A, + ), + ( + Operation::DeleteUser, + DeleteUserRequest { + user_id: WireIdentifier::numeric(USER), + } + .to_bytes(), + ROOT_CLIENT, + 0, + ROOT_SESSION, + ), + ]; + let directory = tempdir().unwrap(); + let metadata_dir = directory.path().join("metadata"); + std::fs::create_dir_all(&metadata_dir).unwrap(); + let stm = UserStm::default(); + stm.users().ensure_root_user("root", "hash"); + let table = RefCell::new(ClientTable::new(CLIENTS_TABLE_MAX)); + let live_log = MetadataChangeLog::new(ChangeLogCapacities::default()); + { + let journal = PrepareJournal::open(&metadata_dir.join("journal.wal"), 0) + .await + .unwrap(); + let mut parent = 0; + for (index, (operation, body, client, user_id, session)) in + entries.into_iter().enumerate() + { + let op = index as u64 + 1; + let mut prepare = make_prepare(op, body.len()); + prepare.as_mut_slice()[HEADER_SIZE..].copy_from_slice(&body); + let prepare = consensus::seal_prepare_checksum(prepare.transmute_header( + |old, header: &mut PrepareHeader| { + *header = old; + header.operation = operation; + header.timestamp = op; + header.client = client; + header.user_id = user_id; + header.session = session; + header.request = if operation == Operation::Register { + 0 + } else { + op + }; + header.parent = parent; + header.checksum_body = u128::from(iggy_common::calculate_checksum(&body)); + }, + )); + parent = prepare.header().checksum; + let committed = + apply_committed_prepare(&stm, &table, true, |_| {}, prepare.clone()); + assert_eq!(result_code(committed.reply.body()).unwrap_or(0), 0); + live_log + .append(prepare.header(), committed.changes) + .expect("live ops append in order"); + journal.append(prepare).await.unwrap(); + } + journal.storage_ref().fsync().await.unwrap(); + } + + let recovered = recover::<UserStm>( + directory.path(), + SOLO, + journal::prepare_journal::DEFAULT_SLOT_COUNT, + CLIENTS_TABLE_MAX, + |stm| stm.users().ensure_root_user("root", "hash"), + ChangeLogCapacities::default(), + ) + .await + .unwrap(); + + let live = live_log.snapshot(); + assert_eq!(recovered.change_log.snapshot(), live); + let kinds: Vec<_> = live + .activity + .entries + .iter() + .map(|entry| (entry.op, entry.kind)) + .collect(); + assert_eq!( + kinds, + [ + (1, WireMetadataChangeKind::SessionRegistered.as_u8()), + (6, WireMetadataChangeKind::SessionRegistered.as_u8()), + (7, WireMetadataChangeKind::SessionRegistered.as_u8()), + (8, WireMetadataChangeKind::ConsumerGroupMemberJoined.as_u8()), + (9, WireMetadataChangeKind::ConsumerGroupMemberJoined.as_u8()), + (10, WireMetadataChangeKind::SessionClosed.as_u8()), + (10, WireMetadataChangeKind::ConsumerGroupMemberLeft.as_u8()), + (11, WireMetadataChangeKind::SessionClosed.as_u8()), + (11, WireMetadataChangeKind::ConsumerGroupMemberLeft.as_u8()), + ], + "the replayed history must hold every session and membership change" + ); + } + + #[compio::test] async fn snapshot_persist_load_roundtrip() { let dir = tempdir().unwrap(); let path = dir.path().join("snapshot.bin"); @@ -2656,7 +2930,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await; assert!(matches!( @@ -2689,7 +2963,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await; assert!(matches!( @@ -2721,7 +2995,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -2774,7 +3048,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await; match result { @@ -2817,7 +3091,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await { @@ -2885,7 +3159,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap(); @@ -2937,7 +3211,7 @@ journal::prepare_journal::DEFAULT_SLOT_COUNT, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap();
diff --git a/core/metadata/src/lib.rs b/core/metadata/src/lib.rs index c475cc7..af44a48 100644 --- a/core/metadata/src/lib.rs +++ b/core/metadata/src/lib.rs
@@ -18,6 +18,7 @@ //! Iggy metadata module pub mod applied_frontier; +pub mod change_log; pub mod impls; pub mod permissioner; pub mod stm; @@ -28,8 +29,8 @@ // Re-export IggyMetadata for use in other modules pub use impls::metadata::{ - BoundSession, CommitNotifier, CommitWalkStop, IggyMetadata, MetadataSubmitError, - StateTransferOffer, apply_committed_prepare, + BoundSession, CommitNotifier, CommitWalkStop, CommittedApply, IggyMetadata, + MetadataSubmitError, StateTransferOffer, apply_committed_prepare, }; // Recovery vocabulary other crates name in their own signatures and error
diff --git a/core/metadata/src/permissioner/permissioner_rules/system.rs b/core/metadata/src/permissioner/permissioner_rules/system.rs index 4884d6c..62d3143 100644 --- a/core/metadata/src/permissioner/permissioner_rules/system.rs +++ b/core/metadata/src/permissioner/permissioner_rules/system.rs
@@ -38,6 +38,12 @@ self.manage_servers(user_id) } + /// The feed exposes deployment-wide state, sessions included, so it takes + /// the gate of the snapshot whatever groups the caller selects. + pub fn get_metadata_changes(&self, user_id: u32) -> Result<(), IggyError> { + self.manage_servers(user_id) + } + fn get_server_info(&self, user_id: u32) -> Result<(), IggyError> { if let Some(global_permissions) = self.users_permissions.get(&user_id) && (global_permissions.manage_servers || global_permissions.read_servers) @@ -107,6 +113,37 @@ )); } + #[test] + fn given_read_grants_without_manage_servers_when_reading_metadata_changes_should_be_unauthorized() + { + let permissioner = permissioner_with(GlobalPermissions { + read_servers: true, + read_streams: true, + read_topics: true, + read_users: true, + manage_streams: true, + manage_users: true, + ..GlobalPermissions::default() + }); + + assert!(matches!( + permissioner.get_metadata_changes(USER_ID), + Err(IggyError::Unauthorized) + )); + } + + #[test] + fn given_manage_servers_when_reading_metadata_changes_should_be_allowed() { + let permissioner = permissioner_with(GlobalPermissions { + manage_servers: true, + ..GlobalPermissions::default() + }); + + permissioner + .get_metadata_changes(USER_ID) + .expect("manage_servers grants the feed"); + } + fn permissioner_with(global: GlobalPermissions) -> Permissioner { let mut permissioner = Permissioner::new(); permissioner.init_permissions_for_user(
diff --git a/core/metadata/src/stm/authz.rs b/core/metadata/src/stm/authz.rs index ec2ada6..ac41cc3 100644 --- a/core/metadata/src/stm/authz.rs +++ b/core/metadata/src/stm/authz.rs
@@ -34,6 +34,7 @@ //! table through `reject_stale_session` before this permission gate. use crate::MuxStateMachine; +use crate::change_log::{AppliedChange, change_actor}; use crate::impls::metadata::StreamsFrontend; use crate::permissioner::Permissioner; use crate::stm::consumer_group::{JoinConsumerGroupRequest, LeaveConsumerGroupRequest}; @@ -42,6 +43,7 @@ use crate::stm::user::Users; use crate::stm::{ApplyContext, StateMachine}; use consensus::ClientTable; +use iggy_binary_protocol::primitives::metadata_changes::WireUserKey; use iggy_binary_protocol::requests::consumer_groups::{ CreateConsumerGroupRequest, DeleteConsumerGroupRequest, }; @@ -93,11 +95,20 @@ let header = prepare.header(); let timestamp = IggyTimestamp::from(header.timestamp); let context = ApplyContext::from(header); - let reply = mux.update(prepare)?; + let mut reply = mux.update(prepare)?; if let Some(deleted) = reply.deleted_resource { + // Pruning is part of the delete, so its records name the delete's + // actor: the requester of a two-phase delete, not the completing server. + let actor = reply + .changes + .first() + .map_or_else(|| change_actor(&context), |deletion| deletion.actor_user_id); // Publish the pruned grants before a later op can reuse the resource ID. // Peer shards may see the deletion first, but cannot see its replacement yet. - mux.users().remove_grants(deleted, timestamp, context); + let pruned = mux + .users() + .remove_grants(deleted, timestamp, context, actor); + reply.changes.extend(pruned); } Ok(reply) } @@ -138,6 +149,27 @@ fn namespace_revision(&self) -> u64; fn is_active_account(&self, user_id: u32, created_at: u64) -> bool; + + /// Rebinds the consumer group memberships of the client a replayed + /// Register committed to `session`, returning the handover records. + fn refresh_session( + &self, + header: &PrepareHeader, + session: u64, + actor_user_id: Option<u32>, + ) -> Vec<AppliedChange>; + + /// Ends the consumer group memberships of `client_ids` as the replayed + /// op did, returning the records of the removals in canonical order. + fn end_memberships( + &self, + client_ids: &[u128], + header: &PrepareHeader, + actor_user_id: Option<u32>, + ) -> Vec<AppliedChange>; + + /// The identity a session record gives `user_id`, `None` for no such user. + fn user_key(&self, user_id: u32) -> Option<WireUserKey>; } impl GatedApply for MuxStateMachine<variadic!(Users, Streams)> { @@ -145,6 +177,40 @@ self.users().is_active_account(user_id, created_at) } + fn refresh_session( + &self, + header: &PrepareHeader, + session: u64, + actor_user_id: Option<u32>, + ) -> Vec<AppliedChange> { + self.streams().refresh_consumer_group_session( + header.client, + session, + header.op, + IggyTimestamp::from(header.timestamp), + actor_user_id, + ) + } + + fn end_memberships( + &self, + client_ids: &[u128], + header: &PrepareHeader, + actor_user_id: Option<u32>, + ) -> Vec<AppliedChange> { + self.streams().remove_consumer_group_members( + client_ids, + IggyTimestamp::from(header.timestamp), + header.op, + actor_user_id, + ) + } + + fn user_key(&self, user_id: u32) -> Option<WireUserKey> { + self.users() + .read(|users| users.items.get(user_id as usize).map(WireUserKey::from)) + } + fn gated_update(&self, prepare: Message<PrepareHeader>) -> Result<ApplyReply, IggyError> { gated_apply(self, prepare) } @@ -532,6 +598,32 @@ true } + fn refresh_session( + &self, + _header: &PrepareHeader, + _session: u64, + _actor_user_id: Option<u32>, + ) -> Vec<AppliedChange> { + Vec::new() + } + + fn end_memberships( + &self, + _client_ids: &[u128], + _header: &PrepareHeader, + _actor_user_id: Option<u32>, + ) -> Vec<AppliedChange> { + Vec::new() + } + + /// Every account exists here, as [`Self::is_active_account`] says. + fn user_key(&self, user_id: u32) -> Option<WireUserKey> { + Some(WireUserKey { + user_id, + user_created_at: 0, + }) + } + fn gated_update(&self, prepare: Message<PrepareHeader>) -> Result<ApplyReply, IggyError> { self.update(prepare) }
diff --git a/core/metadata/src/stm/consumer_group.rs b/core/metadata/src/stm/consumer_group.rs index 1cb39bc..cca236f 100644 --- a/core/metadata/src/stm/consumer_group.rs +++ b/core/metadata/src/stm/consumer_group.rs
@@ -26,6 +26,9 @@ //! never-reused id guarantees a new group can never inherit a deleted group's //! offset -- making a metadata->offset purge unnecessary for correctness. +use crate::change_log::{ + AppliedChange, change_actor, installed_partition_changes, sort_member_records, +}; use crate::stm::StateHandler; use crate::stm::id_slab::IdSlab; use crate::stm::lifecycle::LifecycleAction; @@ -37,8 +40,11 @@ use bytes::Bytes; use bytes::{BufMut, BytesMut}; -use iggy_binary_protocol::WireIdentifier; use iggy_binary_protocol::codec::{WireDecode, WireEncode, read_u32_le, read_u64_le, read_u128_le}; +use iggy_binary_protocol::primitives::metadata_changes::{ + WireConsumerGroupCreatedBody, WireConsumerGroupDeletedBody, WireMemberJoinedBody, + WireMemberLeftBody, WireMemberLeftReason, WireMetadataChange, WirePartitionHandedOverBody, +}; use iggy_binary_protocol::primitives::partition_history::ConsumerGroupOwner; use iggy_binary_protocol::requests::consumer_groups::{ CreateConsumerGroupRequest, DeleteConsumerGroupRequest, @@ -47,6 +53,7 @@ use iggy_binary_protocol::requests::partitions::retire_consumer_group_owners::CONSUMER_GROUP_OWNERS_MAX; use iggy_binary_protocol::responses::consumer_groups::consumer_group_response::ConsumerGroupResponse; use iggy_binary_protocol::responses::consumer_groups::get_consumer_group::ConsumerGroupDetailsResponse; +use iggy_binary_protocol::{WireIdentifier, WireName}; use iggy_common::IggyTimestamp; use serde::{Deserialize, Serialize}; use std::collections::BTreeMap; @@ -551,19 +558,38 @@ .insert(id, ConsumerGroup::new(id, name.clone())); topic.consumer_group_index.insert(name, id); let partitions_count = topic.partitions.len() as u32; + let consumer_groups_count = topic.consumer_groups.len() as u32; + let changes = state + .topic_key(stream_id, topic_id) + .map(|topic| { + AppliedChange::new( + &WireMetadataChange::ConsumerGroupCreated(WireConsumerGroupCreatedBody { + topic, + group_id: id as u32, + name: self.name.clone(), + consumer_groups_count, + }), + change_actor(&state.apply_context), + ) + }) + .into_iter() + .collect(); - ApplyReply::ok( - ConsumerGroupDetailsResponse { - group: ConsumerGroupResponse { - id: id as u32, - partitions_count, - members_count: 0, - name: self.name.clone(), - }, - members: Vec::new(), - } - .to_bytes(), - ) + ApplyReply { + changes, + ..ApplyReply::ok( + ConsumerGroupDetailsResponse { + group: ConsumerGroupResponse { + id: id as u32, + partitions_count, + members_count: 0, + name: self.name.clone(), + }, + members: Vec::new(), + } + .to_bytes(), + ) + } } } @@ -616,6 +642,7 @@ return ApplyReply::err(DeleteConsumerGroupResult::ConsumerGroupNotFound); }; topic.consumer_group_index.remove(&group.name); + let consumer_groups_count = u32::try_from(topic.consumer_groups.len()).ok(); // A finalized deletion applies at its completion op. A sibling group // created after the intent can hold a later op than the intent. topic.consumer_group_catalog_op = state.apply_context.metadata_op; @@ -625,12 +652,35 @@ state.revision = state.revision.wrapping_add(1); // The dropped group may have held pending revocations. state.recompute_consumer_group_metadata(); - ApplyReply::ok(Bytes::new()) + let (requested_op, actor) = state.deletion_origin(); + let deleted = state + .topic_key(stream_id, topic_id) + .zip(consumer_groups_count) + .zip(WireName::new(group.name.as_ref()).ok()) + .map( + |((topic, consumer_groups_count), name)| WireConsumerGroupDeletedBody { + topic, + group_id: wire_group_id, + name, + consumer_groups_count, + requested_op, + }, + ); + ApplyReply { + changes: deleted + .map(|body| { + AppliedChange::new(&WireMetadataChange::ConsumerGroupDeleted(body), actor) + }) + .into_iter() + .collect(), + ..ApplyReply::ok(Bytes::new()) + } } } impl StateHandler for JoinConsumerGroupRequest { type State = StreamsInner; + #[allow(clippy::too_many_lines)] fn apply(&self, state: &mut StreamsInner, timestamp: iggy_common::IggyTimestamp) -> ApplyReply { // Resolve level by level so the committed rejection names the level that // missed, mirroring the legacy `resolve_consumer_group` ladder instead of @@ -669,18 +719,41 @@ .members .iter_mut() .find(|(_, member)| member.client_id == self.client_id); - if let Some((_, member)) = already { + if let Some((member_key, member)) = already { let session = Some(self.session); - if member.session != session { - member.session = session; - group.rebalance_members( - &topic.partitions, - state.apply_context.metadata_op, - timestamp.as_micros(), - ); - state.recompute_consumer_group_metadata(); + if member.session == session { + return ApplyReply::ok(Bytes::new()); } - return ApplyReply::ok(Bytes::new()); + let before = member.partitions.clone(); + member.session = session; + group.rebalance_members( + &topic.partitions, + state.apply_context.metadata_op, + timestamp.as_micros(), + ); + let after = group.members[member_key].partitions.clone(); + state.recompute_consumer_group_metadata(); + let actor = change_actor(&state.apply_context); + let changes = state + .topic_key(stream_id, topic_id) + .zip(u32::try_from(group_id).ok()) + .zip(u32::try_from(member_key).ok()) + .map(|((topic, group_id), member_id)| { + installed_partition_changes(topic, group_id, member_id, &before, &after) + }) + .unwrap_or_default() + .into_iter() + .map(|body| { + AppliedChange::new( + &WireMetadataChange::ConsumerGroupPartitionHandedOver(body), + actor, + ) + }) + .collect(); + return ApplyReply { + changes, + ..ApplyReply::ok(Bytes::new()) + }; } let member_key = group .members @@ -692,8 +765,37 @@ state.apply_context.metadata_op, timestamp.as_micros(), ); + // Ownership is installed only by a completed handover, so these ids + // are normally empty. + let partition_ids: Vec<u32> = group.members[member_key] + .partitions + .iter() + .filter_map(|&partition_id| u32::try_from(partition_id).ok()) + .collect(); + let members_count = group.members.len(); state.recompute_consumer_group_metadata(); - ApplyReply::ok(Bytes::new()) + let joined = state.topic_key(stream_id, topic_id).and_then(|topic| { + Some(WireMemberJoinedBody { + topic, + group_id: u32::try_from(group_id).ok()?, + member_id: u32::try_from(member_key).ok()?, + client_id: self.client_id, + members_count: u32::try_from(members_count).ok()?, + partition_ids, + }) + }); + ApplyReply { + changes: joined + .map(|body| { + AppliedChange::new( + &WireMetadataChange::ConsumerGroupMemberJoined(body), + change_actor(&state.apply_context), + ) + }) + .into_iter() + .collect(), + ..ApplyReply::ok(Bytes::new()) + } } } @@ -749,8 +851,30 @@ state.apply_context.metadata_op, timestamp.as_micros(), ); + let members_count = group.members.len(); state.recompute_consumer_group_metadata(); - ApplyReply::ok(Bytes::new()) + let left = state.topic_key(stream_id, topic_id).and_then(|topic| { + Some(WireMemberLeftBody { + topic, + group_id: u32::try_from(group_id).ok()?, + member_id: u32::try_from(key).ok()?, + client_id: self.client_id, + members_count: u32::try_from(members_count).ok()?, + reason: WireMemberLeftReason::Left, + }) + }); + ApplyReply { + changes: left + .map(|body| { + AppliedChange::new( + &WireMetadataChange::ConsumerGroupMemberLeft(body), + change_actor(&state.apply_context), + ) + }) + .into_iter() + .collect(), + ..ApplyReply::ok(Bytes::new()) + } } } @@ -784,33 +908,93 @@ impl StateHandler for RemoveConsumerGroupMemberRequest { type State = StreamsInner; fn apply(&self, state: &mut StreamsInner, timestamp: iggy_common::IggyTimestamp) -> ApplyReply { - let mut any_removed = false; - for (_, stream) in &mut state.items { - for (_, topic) in &mut stream.topics { - for group in topic.consumer_groups.values_mut() { - let member_key = group - .members - .iter() - .find(|(_, m)| m.client_id == self.client_id) - .map(|(key, _)| key); - if let Some(key) = member_key { - group.members.remove(key); - group.rebalance_members( - &topic.partitions, - state.apply_context.metadata_op, - timestamp.as_micros(), - ); - any_removed = true; + RemoveConsumerGroupMembersRequest { + client_ids: vec![self.client_id], + actor_user_id: change_actor(&state.apply_context), + } + .apply(state, timestamp) + } +} + +/// The disconnect cleanups of one committed op, applied in order with one +/// publish. Never decoded from the wire. +#[derive(Debug, Clone)] +pub struct RemoveConsumerGroupMembersRequest { + pub client_ids: Vec<u128>, + /// The actor of the op that ended the sessions. + pub actor_user_id: Option<u32>, +} + +impl StateHandler for RemoveConsumerGroupMembersRequest { + type State = StreamsInner; + fn apply(&self, state: &mut StreamsInner, timestamp: iggy_common::IggyTimestamp) -> ApplyReply { + let mut removed = Vec::new(); + for &client_id in &self.client_ids { + let mut any_removed = false; + for (stream_id, stream) in &mut state.items { + for (topic_id, topic) in &mut stream.topics { + for group in topic.consumer_groups.values_mut() { + let member_key = group + .members + .iter() + .find(|(_, m)| m.client_id == client_id) + .map(|(key, _)| key); + if let Some(key) = member_key { + group.members.remove(key); + group.rebalance_members( + &topic.partitions, + state.apply_context.metadata_op, + timestamp.as_micros(), + ); + removed.push((stream_id, topic_id, group.id, key, client_id)); + any_removed = true; + } } } } + // Only perturb the reconciler when a membership actually changed. + if any_removed { + state.revision = state.revision.wrapping_add(1); + state.recompute_consumer_group_metadata(); + } } - // Only perturb the reconciler when a membership actually changed. - if any_removed { - state.revision = state.revision.wrapping_add(1); - state.recompute_consumer_group_metadata(); + // Counts are the groups' final ones, so the records do not depend on + // the order the sessions ended in. + let mut records: Vec<WireMemberLeftBody> = removed + .into_iter() + .filter_map(|(stream_id, topic_id, group_id, member_id, client_id)| { + let members_count = state + .items + .get(stream_id)? + .topics + .get(topic_id)? + .consumer_groups + .get(&group_id)? + .members + .len(); + Some(WireMemberLeftBody { + topic: state.topic_key(stream_id, topic_id)?, + group_id: u32::try_from(group_id).ok()?, + member_id: u32::try_from(member_id).ok()?, + client_id, + members_count: u32::try_from(members_count).ok()?, + reason: WireMemberLeftReason::SessionEnded, + }) + }) + .collect(); + sort_member_records(&mut records); + ApplyReply { + changes: records + .into_iter() + .map(|body| { + AppliedChange::new( + &WireMetadataChange::ConsumerGroupMemberLeft(body), + self.actor_user_id, + ) + }) + .collect(), + ..ApplyReply::ok(Bytes::new()) } - ApplyReply::ok(Bytes::new()) } } @@ -895,26 +1079,60 @@ else { return ApplyReply::ok(Bytes::new()); }; - let completed = topic + let owner = self.installation.owner; + // `Some` once the handover completed, holding the new owner's member. + let handed_over = topic .consumer_groups .get_mut(&self.installation.group_id) - .is_some_and(|group| { - group.complete_revocation( - self.partition_id as usize, - &self.installation, - self.partition_op, - ) + .and_then(|group| { + group + .complete_revocation( + self.partition_id as usize, + &self.installation, + self.partition_op, + ) + .then(|| { + group + .members + .iter() + .find(|(_, member)| { + !owner.is_unassigned() + && member.client_id == owner.client_id + && member.session == Some(owner.session) + }) + .map(|(member_id, _)| member_id) + }) }); - if completed { - state.revision = state.revision.wrapping_add(1); - state.pending_revocations.remove(&( - stream_id, - topic_id, - self.installation.group_id, - self.partition_id as usize, - )); + let Some(owner_member) = handed_over else { + return ApplyReply::ok(Bytes::new()); + }; + state.revision = state.revision.wrapping_add(1); + state.pending_revocations.remove(&( + stream_id, + topic_id, + self.installation.group_id, + self.partition_id as usize, + )); + let record = state.topic_key(stream_id, topic_id).and_then(|topic| { + Some(WirePartitionHandedOverBody { + topic, + group_id: u32::try_from(self.installation.group_id).ok()?, + partition_id: self.partition_id, + member_id: owner_member.and_then(|member_id| u32::try_from(member_id).ok()), + }) + }); + ApplyReply { + changes: record + .map(|body| { + AppliedChange::new( + &WireMetadataChange::ConsumerGroupPartitionHandedOver(body), + change_actor(&state.apply_context), + ) + }) + .into_iter() + .collect(), + ..ApplyReply::ok(Bytes::new()) } - ApplyReply::ok(Bytes::new()) } } @@ -923,12 +1141,15 @@ pub struct RefreshConsumerGroupSessionRequest { pub client_id: u128, pub session: u64, + /// The actor of the Register, which the handover records repeat. + pub actor_user_id: Option<u32>, } impl StateHandler for RefreshConsumerGroupSessionRequest { type State = StreamsInner; fn apply(&self, state: &mut StreamsInner, timestamp: IggyTimestamp) -> ApplyReply { + let mut handovers = Vec::new(); if let Some(memberships) = state.consumer_group_members.get(&self.client_id) { for &(stream_id, topic_id, group_id, member_id) in memberships { let Some(topic) = state @@ -945,6 +1166,7 @@ continue; }; if member.session != Some(self.session) { + let before = member.partitions.clone(); member.session = Some(self.session); group.rebalance_members( &topic.partitions, @@ -956,10 +1178,51 @@ topic_id, &mut state.pending_revocations, ); + let after = group + .members + .get(member_id) + .map(|member| member.partitions.clone()) + .unwrap_or_default(); + handovers.push((stream_id, topic_id, group_id, member_id, before, after)); } } } - ApplyReply::ok(Bytes::new()) + let mut records: Vec<WirePartitionHandedOverBody> = handovers + .into_iter() + .filter_map( + |(stream_id, topic_id, group_id, member_id, before, after)| { + Some(installed_partition_changes( + state.topic_key(stream_id, topic_id)?, + u32::try_from(group_id).ok()?, + u32::try_from(member_id).ok()?, + &before, + &after, + )) + }, + ) + .flatten() + .collect(); + // The membership index is in slot order, which differs per replica. + records.sort_unstable_by_key(|record| { + ( + record.topic.stream.stream_id, + record.topic.topic_id, + record.group_id, + record.partition_id, + ) + }); + ApplyReply { + changes: records + .into_iter() + .map(|body| { + AppliedChange::new( + &WireMetadataChange::ConsumerGroupPartitionHandedOver(body), + self.actor_user_id, + ) + }) + .collect(), + ..ApplyReply::ok(Bytes::new()) + } } } @@ -1443,10 +1706,10 @@ for streams in [&sequential, &batched] { // A left-right absorbs its first command in place, so the // removals below go through the op log of both copies. - streams.remove_consumer_group_member(ABSENT, timestamp, REVOCATION_OP - 1); + streams.remove_consumer_group_member(ABSENT, timestamp, REVOCATION_OP - 1, None); } - sequential.remove_consumer_group_member(REVOKED[0], timestamp, REVOCATION_OP); + sequential.remove_consumer_group_member(REVOKED[0], timestamp, REVOCATION_OP, None); assert!( sequential .consumer_group_pending_revocations() @@ -1454,8 +1717,8 @@ .any(|transition| transition.installation.owner.client_id == REVOKED[1]), "the first removal alone decides an owner that the second one replaces" ); - sequential.remove_consumer_group_member(REVOKED[1], timestamp, REVOCATION_OP); - batched.remove_consumer_group_members(&REVOKED, timestamp, REVOCATION_OP); + sequential.remove_consumer_group_member(REVOKED[1], timestamp, REVOCATION_OP, None); + batched.remove_consumer_group_members(&REVOKED, timestamp, REVOCATION_OP, None); let decisions: Vec<_> = batched .consumer_group_pending_revocations() @@ -1475,7 +1738,7 @@ assert_eq!(observe(&batched), observe(&sequential)); for streams in [&sequential, &batched] { // The next publish exposes the copy that replayed the removals. - streams.remove_consumer_group_member(ABSENT, timestamp, REVOCATION_OP + 1); + streams.remove_consumer_group_member(ABSENT, timestamp, REVOCATION_OP + 1, None); } assert_eq!(observe(&batched), observe(&sequential)); } @@ -1494,6 +1757,7 @@ let refresh = RefreshConsumerGroupSessionRequest { client_id: CLIENT, session: SESSION, + actor_user_id: None, }; assert_eq!( StateHandler::apply(&refresh, &mut state, IggyTimestamp::now()).code, @@ -1552,6 +1816,7 @@ 11, 11, iggy_common::IggyTimestamp::default(), + None, ); assert_eq!(streams.consumer_group_session(CLIENT), Some(11)); assert_eq!(streams.consumer_group_session(OTHER), None); @@ -1651,12 +1916,14 @@ epoch, epoch, iggy_common::IggyTimestamp::default(), + None, ); streams.refresh_consumer_group_session( u128::MAX, epoch, epoch, iggy_common::IggyTimestamp::default(), + None, ); assert!(streams.read(|inner| { let topic = &inner.items[0].topics[0];
diff --git a/core/metadata/src/stm/id_slab.rs b/core/metadata/src/stm/id_slab.rs index e7f4c1a..d47a331 100644 --- a/core/metadata/src/stm/id_slab.rs +++ b/core/metadata/src/stm/id_slab.rs
@@ -126,6 +126,16 @@ } } + /// Ascending by key, from `key` on. The seek costs no scan. + #[must_use] + pub fn iter_from(&self, key: usize) -> Iter<'_, T> { + let mut inner = self.entries.iter().enumerate(); + if let Some(skipped) = key.checked_sub(1) { + inner.nth(skipped); + } + Iter { inner } + } + /// Ascending by key. pub fn iter_mut(&mut self) -> IterMut<'_, T> { IterMut { @@ -265,6 +275,23 @@ mod tests { use super::IdSlab; + #[test] + fn given_gaps_when_iterating_from_a_key_should_yield_occupied_keys_from_it() { + let mut slab: IdSlab<char> = IdSlab::new(); + for value in ['a', 'b', 'c', 'd'] { + slab.insert(value); + } + slab.remove(2); + + let keys = |from| slab.iter_from(from).map(|(key, _)| key).collect::<Vec<_>>(); + + assert_eq!(keys(0), [0, 1, 3]); + assert_eq!(keys(2), [3]); + assert_eq!(keys(3), [3]); + assert_eq!(keys(4), [] as [usize; 0]); + assert_eq!(keys(usize::MAX), [] as [usize; 0]); + } + /// The property the type exists for. `Slab` fails it: LIFO reuse makes the /// head depend on delete order while a rebuild scans ascending, so /// descending deletes diverge.
diff --git a/core/metadata/src/stm/lifecycle.rs b/core/metadata/src/stm/lifecycle.rs index fe87e86..f1bfef5 100644 --- a/core/metadata/src/stm/lifecycle.rs +++ b/core/metadata/src/stm/lifecycle.rs
@@ -363,7 +363,7 @@ topic_id: Option<usize>, action: LifecycleAction, ) -> Option<ApplyReply> { - if self.finalizing_lifecycle { + if self.finalizing_lifecycle.is_some() { return None; } if self.lifecycle_blocks(stream_id, topic_id) { @@ -492,7 +492,7 @@ let topic_id = WireIdentifier::numeric(intent.topic_id.unwrap_or_default()); // The deferred request applies at this op, not at the intent op: an // owner its rebalance decides must outrank every owner decided since. - state.finalizing_lifecycle = true; + state.finalizing_lifecycle = Some(intent.context); let reply = match intent.action { LifecycleAction::DeleteStream => { DeleteStreamRequest { stream_id }.apply(state, timestamp) @@ -515,7 +515,7 @@ } .apply(state, timestamp), }; - state.finalizing_lifecycle = false; + state.finalizing_lifecycle = None; if reply.code != 0 { // Keep the final report pending so reconciliation can retry finalization. for &(index, _) in &recorded { @@ -984,6 +984,7 @@ NEW_SESSION, REGISTER_OP, IggyTimestamp::default(), + None, ); let decided = streams.consumer_group_pending_revocations(); assert_eq!(decided.len(), 2); @@ -1348,7 +1349,7 @@ ); assert!(streams.lifecycle_pending(INTENT_OP)); - streams.remove_consumer_group_member(18, IggyTimestamp::default(), LOGOUT_OP); + streams.remove_consumer_group_member(18, IggyTimestamp::default(), LOGOUT_OP, None); let moved = streams.consumer_group_pending_revocations(); assert_eq!(moved.len(), 1); assert_eq!(moved[0].partition_id, 1);
diff --git a/core/metadata/src/stm/mod.rs b/core/metadata/src/stm/mod.rs index 68f55b5..7b88ea1 100644 --- a/core/metadata/src/stm/mod.rs +++ b/core/metadata/src/stm/mod.rs
@@ -413,6 +413,17 @@ inner: $crate::stm::LeftRight::from_factory(factory), } } + + /// The change records of the command applied last. + pub(crate) fn last_changes(&self) -> Vec<$crate::change_log::AppliedChange> { + self.inner.read(|state| { + state + .last_result + .as_ref() + .map(|reply| reply.changes.clone()) + .unwrap_or_default() + }) + } } impl $crate::stm::mux::WithFactory for $state {
diff --git a/core/metadata/src/stm/result.rs b/core/metadata/src/stm/result.rs index 9c999ad..86b14e4 100644 --- a/core/metadata/src/stm/result.rs +++ b/core/metadata/src/stm/result.rs
@@ -27,11 +27,13 @@ //! the section gains `{index, result}` entries and these enums survive unchanged, //! so no variant may encode a carrier assumption (no index, no framing). +use crate::change_log::AppliedChange; use bytes::{Bytes, BytesMut}; use iggy_binary_protocol::Operation; use iggy_binary_protocol::consensus::{ REJECTION_SECTION_LEN, RESULT_COUNT_LEN, write_rejection_section, }; +use iggy_binary_protocol::primitives::metadata_changes::WireUserKey; /// Decode side of the result section, shared with every client (SDK, simulator) /// via `binary_protocol`. Re-exported so server-side callers keep one import. @@ -58,11 +60,15 @@ pub struct ApplyReply { pub code: u32, pub body: Bytes, - /// Local apply effect, recomputed from the committed operation on every replay. - /// It is not part of the client reply or persisted receipt. - pub(crate) revoked_user: Option<u32>, + /// The user whose sessions end. A local apply effect, recomputed from the + /// committed operation on every replay and never part of the client reply + /// or persisted receipt. + pub(crate) revoked_user: Option<WireUserKey>, /// Local apply effect, like `revoked_user`: the grants on this resource must go. pub(crate) deleted_resource: Option<DeletedResource>, + /// Public results for the change log, in canonical order. Like + /// `revoked_user`, never part of the client reply or persisted receipt. + pub(crate) changes: Vec<AppliedChange>, } impl ApplyReply { @@ -74,6 +80,7 @@ body, revoked_user: None, deleted_resource: None, + changes: Vec::new(), } } @@ -86,6 +93,7 @@ body: Bytes::new(), revoked_user: None, deleted_resource: None, + changes: Vec::new(), } }
diff --git a/core/metadata/src/stm/snapshot.rs b/core/metadata/src/stm/snapshot.rs index 8d24c23..fa42390 100644 --- a/core/metadata/src/stm/snapshot.rs +++ b/core/metadata/src/stm/snapshot.rs
@@ -19,6 +19,7 @@ use serde::{Deserialize, Serialize, de::DeserializeOwned}; use std::fmt; +use crate::change_log::{ChangeLogSnapshot, RingSnapshot}; use crate::stm::stream::StreamsSnapshot; use crate::stm::user::UsersSnapshot; @@ -63,7 +64,10 @@ /// Version 13: `TopicSnapshot` dropped its typed expiry, compression and size /// fields, which the options map already holds, and `StreamSnapshot` gained /// `created_op`, the identity of a stream incarnation. -pub const SNAPSHOT_FORMAT_VERSION: u32 = 13; +/// +/// Version 14: `TopicSnapshot` gained `created_op`, the identity of a topic +/// incarnation, and the snapshot carries the metadata change log. +pub const SNAPSHOT_FORMAT_VERSION: u32 = 14; /// Oldest format version [`MetadataSnapshot::decode`] still reads. /// @@ -278,6 +282,10 @@ /// op. Not a state machine, but folded in so a restart that drained the WAL /// prefix still recovers pre-checkpoint sessions. pub client_table: Option<consensus::ClientTableSnapshot>, + /// The change log at the checkpoint op. Not a state machine either, it + /// is folded in so a restart or a transfer keeps the history readers + /// resume from. + pub change_log: ChangeLogSnapshot, /// The release that wrote this snapshot, as a packed `iggy_binary_protocol` /// semver. Provenance only, never a gate. See [`SNAPSHOT_WRITER_RELEASE`]. pub writer_release: u32, @@ -305,6 +313,22 @@ users: None, streams: None, client_table: None, + change_log: ChangeLogSnapshot { + log_id: None, + last_record_op: 0, + resource: RingSnapshot { + evicted_through_op: 0, + evicted_through_ordinal: 0, + evicted_through_timestamp: 0, + entries: Vec::new(), + }, + activity: RingSnapshot { + evicted_through_op: 0, + evicted_through_ordinal: 0, + evicted_through_timestamp: 0, + entries: Vec::new(), + }, + }, writer_release: SNAPSHOT_WRITER_RELEASE, } } @@ -348,7 +372,7 @@ if let Some(streams) = &self.streams { streams.validate_ownership_history(self.sequence_number)?; } - Ok(()) + self.change_log.validate(self.sequence_number) } } @@ -627,8 +651,8 @@ // turns "shape changed, version did not" into a failing test rather than an // operator's boot reading one field's bytes as another's. Changing either // number is the reminder to change the other. - const FIELD_COUNT: u32 = 7; - const PINNED_VERSION: u32 = 13; + const FIELD_COUNT: u32 = 8; + const PINNED_VERSION: u32 = 14; let encoded = MetadataSnapshot::new(0).encode().unwrap(); let mut cursor = encoded.as_slice(); @@ -650,7 +674,7 @@ // the same element count: a version 2 file then decoded field-by-field // one position out of place instead of failing. The types the STM // actually grows need their own pins. - const TOPIC_FIELD_COUNT: u32 = 9; + const TOPIC_FIELD_COUNT: u32 = 10; const STREAM_FIELD_COUNT: u32 = 7; const STREAMS_FIELD_COUNT: u32 = 4; const USER_FIELD_COUNT: u32 = 7; @@ -688,6 +712,7 @@ next_consumer_group_id: 0, options: ResourceOptions::new(), consumer_group_catalog_op: 0, + created_op: 0, }; let encoded = rmp_serde::to_vec(&topic).unwrap(); assert_eq!( @@ -809,6 +834,22 @@ } #[test] + fn given_change_log_above_the_snapshot_when_decoding_should_refuse_it() { + let mut snapshot = MetadataSnapshot::new(9); + snapshot.change_log.log_id = Some(1); + snapshot.change_log.last_record_op = 10; + let encoded = snapshot.encode().unwrap(); + + assert!(matches!( + MetadataSnapshot::decode(&encoded), + Err(SnapshotError::InvalidState(_)) + )); + snapshot.change_log.last_record_op = 9; + let decoded = MetadataSnapshot::decode(&snapshot.encode().unwrap()).unwrap(); + assert_eq!(decoded.change_log, snapshot.change_log); + } + + #[test] fn decode_refuses_snapshots_without_durable_session_protection() { for version in 0..SNAPSHOT_FORMAT_VERSION { let mut snapshot = MetadataSnapshot::new(9); @@ -833,8 +874,8 @@ } #[test] - fn given_format_thirteen_snapshot_when_restoring_should_preserve_options_and_created_op() { - const PREVIOUS_FORMAT: u32 = 12; + fn given_format_fourteen_snapshot_when_restoring_should_preserve_options_and_created_op() { + const PREVIOUS_FORMAT: u32 = 13; let mut options = ResourceOptions::new(); options.insert( HeaderKey::from(OptionKey::MessageExpiry), @@ -973,6 +1014,7 @@ next_consumer_group_id: 5, options: ResourceOptions::default(), consumer_group_catalog_op: 7, + created_op: 8, }, )], options: ResourceOptions::default(), @@ -1005,6 +1047,7 @@ // the field were dropped from the wire format, so pin its survival. assert_eq!(topic.next_consumer_group_id, 5); assert_eq!(topic.consumer_group_catalog_op, 7); + assert_eq!(topic.created_op, 8); } fn user_snapshot_fixture(id: u32, username: &str, password_hash: &str) -> UserSnapshot {
diff --git a/core/metadata/src/stm/stream.rs b/core/metadata/src/stm/stream.rs index cd4c442..dc2a1d5 100644 --- a/core/metadata/src/stm/stream.rs +++ b/core/metadata/src/stm/stream.rs
@@ -15,10 +15,12 @@ // specific language governing permissions and limitations // under the License. +use crate::change_log::{AppliedChange, change_actor, stream_body, topic_body}; use crate::stm::consumer_group::{ CompleteConsumerGroupRevocationRequest, ConsumerGroup, ConsumerGroupOwnershipTransition, ConsumerGroupSnapshot, JoinConsumerGroupRequest, LeaveConsumerGroupRequest, RefreshConsumerGroupSessionRequest, RemoveConsumerGroupMemberRequest, + RemoveConsumerGroupMembersRequest, }; use crate::stm::id_slab::IdSlab; use crate::stm::lifecycle::{ @@ -35,6 +37,9 @@ use ahash::{AHashMap, AHashSet}; use bytes::{BufMut, Bytes, BytesMut}; use iggy_binary_protocol::codec::{WireDecode, WireEncode}; +use iggy_binary_protocol::primitives::metadata_changes::{ + WireMetadataChange, WireStreamBody, WireStreamDeletedBody, WireStreamKey, WireTopicDeletedBody, +}; use iggy_binary_protocol::primitives::partition_assignment::CreatedPartitionAssignment; use iggy_binary_protocol::requests::consumer_groups::{ CreateConsumerGroupRequest, DeleteConsumerGroupRequest, @@ -177,6 +182,9 @@ pub options: ResourceOptions, #[serde(default)] pub consumer_group_catalog_op: u64, + /// Metadata op that created this topic incarnation. A reused topic id + /// gets a new value. + pub created_op: u64, } #[derive(Debug, Clone)] @@ -215,6 +223,8 @@ /// set. A partition expects one allocation boundary per op, so a reader /// takes the catalog and this op from the same published state. pub consumer_group_catalog_op: u64, + /// Metadata op that created this topic incarnation. + pub created_op: u64, } impl Default for Topic { @@ -232,6 +242,7 @@ consumer_group_index: AHashMap::default(), next_consumer_group_id: 0, consumer_group_catalog_op: 0, + created_op: 0, } } } @@ -251,6 +262,7 @@ consumer_group_index: AHashMap::default(), next_consumer_group_id: 0, consumer_group_catalog_op: 0, + created_op: 0, } } @@ -783,7 +795,9 @@ // The keys of `lifecycle_intents` ordered by stream and topic, so a // blocker query visits only its own scope. Rebuilt on restore. pub(crate) lifecycle_scopes: BTreeSet<LifecycleScope>, - pub(crate) finalizing_lifecycle: bool, + // The context of the intent a `CompleteLifecycle` is finishing, so its + // delete records name the request and the requester. + pub(crate) finalizing_lifecycle: Option<ApplyContext>, // Rebuilt after membership changes and restore; ticks visit only open transitions. pub(crate) pending_revocations: BTreeMap<(usize, usize, u64, usize), ConsumerGroupOwnershipTransition>, // Locations are (stream, topic, group, member); never persisted. @@ -1007,7 +1021,7 @@ CompleteLifecycle, RewrapStreamKeys, } - internal { RefreshConsumerGroupSession } + internal { RefreshConsumerGroupSession, RemoveConsumerGroupMembers } } impl StreamsInner { @@ -1659,16 +1673,21 @@ session: u64, metadata_op: u64, timestamp: IggyTimestamp, - ) { + actor_user_id: Option<u32>, + ) -> Vec<AppliedChange> { if session == 0 || !self .inner .read(|inner| inner.consumer_group_members.contains_key(&client_id)) { - return; + return Vec::new(); } let cmd = StreamsCommand::RefreshConsumerGroupSession( - RefreshConsumerGroupSessionRequest { client_id, session }, + RefreshConsumerGroupSessionRequest { + client_id, + session, + actor_user_id, + }, timestamp, ApplyContext { metadata_op, @@ -1677,53 +1696,61 @@ ); if let Err(error) = self.inner.try_apply(cmd) { tracing::error!(client_id, %error, "consumer session refresh dispatched to reader-only Streams STM"); + return Vec::new(); } + self.last_changes() } /// Drop a disconnected client from every consumer group it joined and /// rebalance. Applied through the left-right writer as a deterministic /// side-effect of the `Logout` commit on each replica (not a separate /// replicated op). A no-op on the reader-mode peers, where commits aren't - /// applied. + /// applied. Returns the removals' change records. pub fn remove_consumer_group_member( &self, client_id: u128, timestamp: IggyTimestamp, metadata_op: u64, - ) { - self.remove_consumer_group_members(&[client_id], timestamp, metadata_op); + actor_user_id: Option<u32>, + ) -> Vec<AppliedChange> { + self.remove_consumer_group_members(&[client_id], timestamp, metadata_op, actor_user_id) } /// Remove `client_ids` in order, as [`Self::remove_consumer_group_member`] /// would, with one publish. The removals of one op decide owners at the /// same generation, so a reader that installed an owner between two of - /// them could never install the final one. + /// them could never install the final one. The records come back in + /// canonical order, whatever the order of `client_ids`. pub fn remove_consumer_group_members( &self, client_ids: &[u128], timestamp: IggyTimestamp, metadata_op: u64, - ) { + actor_user_id: Option<u32>, + ) -> Vec<AppliedChange> { if client_ids.is_empty() { - return; + return Vec::new(); } - let cmds = client_ids.iter().map(|&client_id| { - StreamsCommand::RemoveConsumerGroupMember( - RemoveConsumerGroupMemberRequest { client_id }, - timestamp, - ApplyContext { - metadata_op, - ..ApplyContext::default() - }, - ) - }); - if let Err(error) = self.inner.try_apply_all(cmds) { + let command = StreamsCommand::RemoveConsumerGroupMembers( + RemoveConsumerGroupMembersRequest { + client_ids: client_ids.to_vec(), + actor_user_id, + }, + timestamp, + ApplyContext { + metadata_op, + ..ApplyContext::default() + }, + ); + if let Err(error) = self.inner.try_apply(command) { tracing::error!( ?client_ids, %error, "remove_consumer_group_members dispatched to reader-only Streams STM" ); + return Vec::new(); } + self.last_changes() } /// Total consumer-group count across all topics (for stats). @@ -2233,11 +2260,23 @@ let inserted = state.items.insert(stream); debug_assert_eq!(inserted, id, "vacant_key must match the insert slot"); state.index.insert(name_arc, id); + let changes = state + .items + .get(id) + .and_then(stream_body) + .map(|body| { + AppliedChange::new( + &WireMetadataChange::StreamCreated(body), + change_actor(&state.apply_context), + ) + }) + .into_iter() + .collect(); // Reply body: a freshly created stream has no topics. The SDK // `create_stream` decodes a `GetStreamResponse`. Serialization is // local to this state machine (it owns the committed shape). - ApplyReply::ok( + let reply = ApplyReply::ok( GetStreamResponse { stream: StreamResponse { id: id as u32, @@ -2252,7 +2291,8 @@ topics: Vec::new(), } .to_bytes(), - ) + ); + ApplyReply { changes, ..reply } } } @@ -2290,6 +2330,7 @@ return ApplyReply::err(UpdateStreamResult::OptionsBlockTooLarge); } }; + let before = stream_body(stream); let Some(stream) = state.items.get_mut(stream_id) else { return ApplyReply::err(UpdateStreamResult::StreamNotFound); @@ -2300,7 +2341,22 @@ state.index.insert(name, stream_id); } stream.options = options; - ApplyReply::ok(Bytes::new()) + // An equal patch commits as ok but changes nothing public. + let after = stream_body(stream); + let changes = after + .filter(|after| before.as_ref() != Some(after)) + .map(|body| { + AppliedChange::new( + &WireMetadataChange::StreamUpdated(body), + change_actor(&state.apply_context), + ) + }) + .into_iter() + .collect(); + ApplyReply { + changes, + ..ApplyReply::ok(Bytes::new()) + } } } @@ -2318,6 +2374,15 @@ return ApplyReply::err(DeleteStreamResult::StreamNotFound); }; let name = stream.name.clone(); + let (requested_op, actor) = state.deletion_origin(); + let deleted = WireName::new(name.as_ref()) + .ok() + .map(|wire_name| WireStreamDeletedBody { + key: WireStreamKey::from(stream), + created_at: stream.created_at.as_micros(), + name: wire_name, + requested_op, + }); // Evict registry entries so a reused slab id starts with fresh stats. // Before the removal: the committed tree is what names the entries. state.stats_registry.remove_stream(stream_id, stream); @@ -2332,6 +2397,10 @@ state.recompute_consumer_group_metadata(); ApplyReply { deleted_resource: Some(DeletedResource::Stream(stream_id)), + changes: deleted + .map(|body| AppliedChange::new(&WireMetadataChange::StreamDeleted(body), actor)) + .into_iter() + .collect(), ..ApplyReply::ok(Bytes::new()) } } @@ -2443,6 +2512,7 @@ consumer_group_index: AHashMap::default(), next_consumer_group_id: 0, consumer_group_catalog_op: 0, + created_op: state.apply_context.metadata_op, }; let inserted = stream.topics.insert(topic); @@ -2470,12 +2540,24 @@ let Some(topic) = stream.topics.get(topic_id) else { return ApplyReply::err(CreateTopicResult::StreamNotFound); }; - ApplyReply::ok(encode_create_topic_reply( - &self.request.name, - topic_id, - topic, - state.apply_context.metadata_op, - )) + let changes = topic_body(stream, topic) + .map(|body| { + AppliedChange::new( + &WireMetadataChange::TopicCreated(body), + change_actor(&state.apply_context), + ) + }) + .into_iter() + .collect(); + ApplyReply { + changes, + ..ApplyReply::ok(encode_create_topic_reply( + &self.request.name, + topic_id, + topic, + state.apply_context.metadata_op, + )) + } } } @@ -2609,6 +2691,7 @@ { return ApplyReply::err(UpdateTopicResult::InvalidTopicSize); } + let before = topic_body(stream, topic); let Some(topic) = stream.topics.get_mut(topic_id) else { return ApplyReply::err(UpdateTopicResult::TopicNotFound); @@ -2619,7 +2702,25 @@ stream.topic_index.insert(name, topic_id); } topic.set_options(options); - ApplyReply::ok(Bytes::new()) + // An equal patch commits as ok but changes nothing public. + let after = stream + .topics + .get(topic_id) + .and_then(|topic| topic_body(stream, topic)); + let changes = after + .filter(|after| before.as_ref() != Some(after)) + .map(|body| { + AppliedChange::new( + &WireMetadataChange::TopicUpdated(body), + change_actor(&state.apply_context), + ) + }) + .into_iter() + .collect(); + ApplyReply { + changes, + ..ApplyReply::ok(Bytes::new()) + } } } @@ -2631,6 +2732,9 @@ /// Reply body: the number of sets replaced, `u32` LE. fn apply(&self, state: &mut StreamsInner, _timestamp: IggyTimestamp) -> ApplyReply { let mut replaced = 0u32; + // The record of each rewrapped stream compares its first and last + // state in this op, ascending by stream id. + let mut before: BTreeMap<usize, Option<WireStreamBody>> = BTreeMap::new(); for entry in &self.entries { let Some(stream) = state.items.get_mut(entry.stream_id as usize) else { continue; @@ -2653,13 +2757,28 @@ let Ok(value) = HeaderValue::try_from(set.as_ref()) else { continue; }; + before + .entry(entry.stream_id as usize) + .or_insert_with(|| stream_body(stream)); stream.options.insert( HeaderKey::from(OptionKey::EncryptionKeyWrapped), OptionValue::explicit(value), ); replaced += 1; } - ApplyReply::ok(Bytes::copy_from_slice(&replaced.to_le_bytes())) + let actor = change_actor(&state.apply_context); + let changes = before + .into_iter() + .filter_map(|(stream_id, before)| { + let after = state.items.get(stream_id).and_then(stream_body)?; + (before.as_ref() != Some(&after)) + .then(|| AppliedChange::new(&WireMetadataChange::StreamUpdated(after), actor)) + }) + .collect(); + ApplyReply { + changes, + ..ApplyReply::ok(Bytes::copy_from_slice(&replaced.to_le_bytes())) + } } } @@ -2731,6 +2850,7 @@ { return reply; } + let (requested_op, actor) = state.deletion_origin(); let Some(stream) = state.items.get_mut(stream_id) else { return ApplyReply::err(DeleteTopicResult::StreamNotFound); }; @@ -2739,6 +2859,14 @@ return ApplyReply::err(DeleteTopicResult::TopicNotFound); }; let name = topic.name.clone(); + let deleted = WireName::new(name.as_ref()) + .ok() + .map(|wire_name| WireTopicDeletedBody { + key: stream.topic_key(topic), + created_at: topic.created_at.as_micros(), + name: wire_name, + requested_op, + }); // Read the ids off the committed topic before it goes: they are what // names its registry entries. let partition_ids: Vec<usize> = topic @@ -2763,6 +2891,10 @@ stream_id, topic_id, }), + changes: deleted + .map(|body| AppliedChange::new(&WireMetadataChange::TopicDeleted(body), actor)) + .into_iter() + .collect(), ..ApplyReply::ok(Bytes::new()) } } @@ -2862,11 +2994,28 @@ // Added partitions are unassigned until the groups rebalance. topic.rebalance_consumer_groups(state.apply_context.metadata_op, timestamp); state.recompute_consumer_group_metadata(); + let changes = if self.partitions.is_empty() { + Vec::new() + } else { + state + .partitions_changed(stream_id, topic_id, Vec::new(), None) + .map(|body| { + AppliedChange::new( + &WireMetadataChange::PartitionsChanged(body), + change_actor(&state.apply_context), + ) + }) + .into_iter() + .collect() + }; // Matches legacy CreatePartitions wire contract: empty-ok body on // success. SDK discards the reply payload (resolved ids are derivable // from the request's base + count). - ApplyReply::ok(Bytes::new()) + ApplyReply { + changes, + ..ApplyReply::ok(Bytes::new()) + } } } @@ -2912,6 +3061,7 @@ }; // Zero count is rejected pre-consensus; a replayed legacy entry still // applies as the historical ok no-op. + let mut changes = Vec::new(); if count_to_delete > 0 { let retained = topic.partitions.len() - count_to_delete; // Read the ids off the removed tail: `retained` is a count, and @@ -2930,8 +3080,24 @@ state.revision = state.revision.wrapping_add(1); state.namespace_revision = state.revision; state.retire_partitions(stream_id, topic_id, &removed); + let (requested_op, actor) = state.deletion_origin(); + let mut removed_ids: Vec<u32> = removed_ids + .iter() + .filter_map(|&id| u32::try_from(id).ok()) + .collect(); + removed_ids.sort_unstable(); + changes.extend( + state + .partitions_changed(stream_id, topic_id, removed_ids, requested_op) + .map(|body| { + AppliedChange::new(&WireMetadataChange::PartitionsChanged(body), actor) + }), + ); } - ApplyReply::ok(Bytes::new()) + ApplyReply { + changes, + ..ApplyReply::ok(Bytes::new()) + } } } @@ -3001,6 +3167,7 @@ .collect(), next_consumer_group_id: topic.next_consumer_group_id, consumer_group_catalog_op: topic.consumer_group_catalog_op, + created_op: topic.created_op, }, ) }) @@ -3139,6 +3306,7 @@ .unwrap_or(0), ), consumer_group_catalog_op: topic_snap.consumer_group_catalog_op, + created_op: topic_snap.created_op, consumer_groups: topic_snap .consumer_groups .into_iter() @@ -3184,7 +3352,7 @@ namespace_revision: snapshot.namespace_revision, lifecycle_intents, lifecycle_scopes, - finalizing_lifecycle: false, + finalizing_lifecycle: None, // Recomputed from the restored groups just below. pending_revocations: BTreeMap::new(), consumer_group_members: AHashMap::new(),
diff --git a/core/metadata/src/stm/user.rs b/core/metadata/src/stm/user.rs index b4f9767..5a773ce 100644 --- a/core/metadata/src/stm/user.rs +++ b/core/metadata/src/stm/user.rs
@@ -15,6 +15,7 @@ // specific language governing permissions and limitations // under the License. +use crate::change_log::{AppliedChange, change_actor, user_body}; use crate::permissioner::Permissioner; use crate::stm::id_slab::IdSlab; use crate::stm::result::{ @@ -32,6 +33,10 @@ #[cfg(any(test, feature = "simulator"))] use iggy_binary_protocol::WireOptions; use iggy_binary_protocol::codec::{WireDecode, WireEncode, read_u32_le, read_u64_le}; +use iggy_binary_protocol::primitives::metadata_changes::{ + WireMetadataChange, WirePermissionsUpdatedReason, WireTokenBody, WireTokenDeletedBody, + WireTokenDeletedReason, WireUserIdentityBody, WireUserKey, WireUserPermissionsBody, +}; #[cfg(any(test, feature = "simulator"))] use iggy_binary_protocol::primitives::permissions::{WireGlobalPermissions, WirePermissions}; use iggy_binary_protocol::requests::users::{ @@ -42,7 +47,7 @@ use iggy_binary_protocol::responses::users::user_response::UserResponse; use iggy_binary_protocol::{WireIdentifier, WireName}; use iggy_common::defaults::{DEFAULT_ROOT_USER_ID, MAX_USERNAME_LENGTH, MIN_USERNAME_LENGTH}; -use iggy_common::wire_conversions::resource_options_from_wire; +use iggy_common::wire_conversions::{permissions_to_wire, resource_options_from_wire}; use iggy_common::{ GlobalPermissions, IggyError, IggyExpiry, IggyTimestamp, OptionsScope, Permissions, PersonalAccessToken, ResourceOptions, StreamPermissions, UserId, UserStatus, @@ -121,6 +126,9 @@ // is deterministic across replicas), which lets the PAT cleaner find // expired tokens in O(log n) per tick instead of scanning every token. pub(crate) personal_access_token_expiry_index: BTreeSet<(u64, UserId, Arc<str>)>, + // `(user_id, name)` of every token, ordered, so current state pages + // seek by key instead of sorting the per-user maps. Never persisted. + pub(crate) personal_access_token_name_index: BTreeSet<(UserId, Arc<str>)>, pub(crate) permissioner: Permissioner, } } @@ -314,6 +322,8 @@ // cannot make replicas diverge. for pat in tokens.values() { self.personal_access_token_index.remove(&pat.token); + self.personal_access_token_name_index + .remove(&(user_id, Arc::clone(&pat.name))); if let Some(expiry_at) = pat.expiry_at { self.personal_access_token_expiry_index.remove(&( expiry_at.as_micros(), @@ -357,16 +367,27 @@ /// Drop every grant on a stream or topic that a committed delete removed. /// Applied through the writer as a deterministic side effect of that delete /// on each replica (not a separate replicated op), so WAL replay repeats it. + /// Returns a record per pruned user, ascending by user id. pub(crate) fn remove_grants( &self, deleted: DeletedResource, timestamp: IggyTimestamp, context: ApplyContext, - ) { - let cmd = UsersCommand::RemoveGrants(RemoveGrantsRequest { deleted }, timestamp, context); + actor_user_id: Option<u32>, + ) -> Vec<AppliedChange> { + let cmd = UsersCommand::RemoveGrants( + RemoveGrantsRequest { + deleted, + actor_user_id, + }, + timestamp, + context, + ); if let Err(error) = self.inner.try_apply(cmd) { tracing::error!(?deleted, %error, "remove_grants dispatched to reader-only Users STM"); + return Vec::new(); } + self.last_changes() } /// Insert a root fixture into an empty user set for tests and the simulator. @@ -577,10 +598,22 @@ state .permissioner .init_permissions_for_user(id as UserId, user_permissions); + let changes = state + .items + .get(id) + .and_then(user_body) + .map(|body| { + AppliedChange::new( + &WireMetadataChange::UserCreated(body), + change_actor(&state.apply_context), + ) + }) + .into_iter() + .collect(); // Reply body: the SDK `create_user` decodes a `UserDetailsResponse`. // Serialization local to this state machine. - ApplyReply::ok( + let reply = ApplyReply::ok( UserDetailsResponse { user: UserResponse { id: id as u32, @@ -593,7 +626,8 @@ permissions: self.permissions.clone(), } .to_bytes(), - ) + ); + ApplyReply { changes, ..reply } } } @@ -645,6 +679,7 @@ return ApplyReply::err(UpdateUserResult::OptionsBlockTooLarge); } }; + let before = user_body(user); let Some(user) = state.items.get_mut(user_id) else { return ApplyReply::err(UpdateUserResult::UserNotFound); @@ -658,8 +693,20 @@ user.status = new_status; } user.options = options; + // An equal patch commits as ok but changes nothing public. + let changes = user_body(user) + .filter(|after| before.as_ref() != Some(after)) + .map(|body| { + AppliedChange::new( + &WireMetadataChange::UserUpdated(body), + change_actor(&state.apply_context), + ) + }) + .into_iter() + .collect(); ApplyReply { - revoked_user: (user.status != UserStatus::Active).then_some(user_id as UserId), + revoked_user: (user.status != UserStatus::Active).then(|| WireUserKey::from(&*user)), + changes, ..ApplyReply::ok(Bytes::new()) } } @@ -682,8 +729,20 @@ return ApplyReply::err(DeleteUserResult::CannotDeleteUser); } + let mut changes = Vec::new(); + let mut revoked_user = None; if let Some(user) = state.items.get(user_id) { + revoked_user = Some(WireUserKey::from(user)); let username = user.username.clone(); + changes.extend(WireName::new(username.as_ref()).ok().map(|wire_name| { + AppliedChange::new( + &WireMetadataChange::UserDeleted(WireUserIdentityBody { + key: WireUserKey::from(user), + username: wire_name, + }), + change_actor(&state.apply_context), + ) + })); state.items.remove(user_id); state.index.remove(&username); state.remove_personal_access_tokens(user_id as UserId); @@ -692,7 +751,8 @@ .delete_permissions_for_user(user_id as UserId); } ApplyReply { - revoked_user: Some(user_id as UserId), + revoked_user, + changes, ..ApplyReply::ok(Bytes::new()) } } @@ -716,10 +776,23 @@ return ApplyReply::err(ChangePasswordResult::InvalidCredentials); } + let mut changes = Vec::new(); if let Some(user) = state.items.get_mut(user_id) { user.password_hash = Arc::from(self.new_password.as_str()); + changes.extend(WireName::new(user.username.as_ref()).ok().map(|username| { + AppliedChange::new( + &WireMetadataChange::UserPasswordChanged(WireUserIdentityBody { + key: WireUserKey::from(&*user), + username, + }), + change_actor(&state.apply_context), + ) + })); } - ApplyReply::ok(Bytes::new()) + ApplyReply { + changes, + ..ApplyReply::ok(Bytes::new()) + } } } @@ -732,17 +805,37 @@ Err(result) => return ApplyReply::err(result), }; + let mut changes = Vec::new(); if let Some(user) = state.items.get_mut(user_id) { + let before = user.permissions.as_deref().map(permissions_to_wire); user.permissions = self .permissions .as_ref() .map(|p| Arc::new(Permissions::from(p.clone()))); + // An equal replacement changes nothing public. + let after = user.permissions.as_deref().map(permissions_to_wire); + if after != before + && let Ok(username) = WireName::new(user.username.as_ref()) + { + changes.push(AppliedChange::new( + &WireMetadataChange::UserPermissionsUpdated(WireUserPermissionsBody { + key: WireUserKey::from(&*user), + username, + permissions: after, + reason: WirePermissionsUpdatedReason::Updated, + }), + change_actor(&state.apply_context), + )); + } } let user_permissions = state.permissions_of(user_id); state .permissioner .update_permissions_for_user(user_id as UserId, user_permissions); - ApplyReply::ok(Bytes::new()) + ApplyReply { + changes, + ..ApplyReply::ok(Bytes::new()) + } } } @@ -753,11 +846,14 @@ #[derive(Debug, Clone)] pub struct RemoveGrantsRequest { pub deleted: DeletedResource, + /// The actor of the delete, which the pruning records repeat. + pub actor_user_id: Option<u32>, } impl StateHandler for RemoveGrantsRequest { type State = UsersInner; fn apply(&self, state: &mut UsersInner, _timestamp: IggyTimestamp) -> ApplyReply { + let mut changes = Vec::new(); for (_, user) in &mut state.items { let Some(pruned) = user .permissions @@ -795,8 +891,22 @@ } } user.permissions = Some(Arc::new(pruned)); + if let Ok(username) = WireName::new(user.username.as_ref()) { + changes.push(AppliedChange::new( + &WireMetadataChange::UserPermissionsUpdated(WireUserPermissionsBody { + key: WireUserKey::from(&*user), + username, + permissions: user.permissions.as_deref().map(permissions_to_wire), + reason: WirePermissionsUpdatedReason::ResourceDeleted, + }), + self.actor_user_id, + )); + } } - ApplyReply::ok(Bytes::new()) + ApplyReply { + changes, + ..ApplyReply::ok(Bytes::new()) + } } } @@ -904,9 +1014,30 @@ )); } state + .personal_access_token_name_index + .insert((self.user_id, Arc::clone(&name))); + state .personal_access_token_index .insert(token_hash, (self.user_id, name)); - ApplyReply::ok(Bytes::new()) + let changes = state + .items + .get(self.user_id as usize) + .map(|user| { + AppliedChange::new( + &WireMetadataChange::PersonalAccessTokenCreated(WireTokenBody { + user: WireUserKey::from(user), + name: self.name.clone(), + expiry: expiry_at.map_or(0, |expiry_at| expiry_at.as_micros()), + }), + change_actor(&state.apply_context), + ) + }) + .into_iter() + .collect(); + ApplyReply { + changes, + ..ApplyReply::ok(Bytes::new()) + } } } @@ -938,6 +1069,9 @@ match user_tokens.remove(&name_arc) { Some(pat) => { state.personal_access_token_index.remove(&pat.token); + state + .personal_access_token_name_index + .remove(&(self.user_id, Arc::clone(&name_arc))); if let Some(expiry_at) = pat.expiry_at { state.personal_access_token_expiry_index.remove(&( expiry_at.as_micros(), @@ -945,7 +1079,30 @@ name_arc, )); } - ApplyReply::ok(Bytes::new()) + let reason = if self.only_if_expired { + WireTokenDeletedReason::Expired + } else { + WireTokenDeletedReason::Deleted + }; + let changes = state + .items + .get(self.user_id as usize) + .map(|user| { + AppliedChange::new( + &WireMetadataChange::PersonalAccessTokenDeleted(WireTokenDeletedBody { + user: WireUserKey::from(user), + name: self.name.clone(), + reason, + }), + change_actor(&state.apply_context), + ) + }) + .into_iter() + .collect(); + ApplyReply { + changes, + ..ApplyReply::ok(Bytes::new()) + } } None => ApplyReply::err(DeletePersonalAccessTokenResult::NotFound), } @@ -1117,6 +1274,7 @@ AHashMap::new(); let mut personal_access_token_expiry_index: BTreeSet<(u64, UserId, Arc<str>)> = BTreeSet::new(); + let mut personal_access_token_name_index: BTreeSet<(UserId, Arc<str>)> = BTreeSet::new(); for (user_id, tokens) in snapshot.personal_access_tokens { let mut token_map: AHashMap<Arc<str>, PersonalAccessToken> = AHashMap::new(); for (name, pat_snap) in tokens { @@ -1136,6 +1294,7 @@ } personal_access_token_index .insert(Arc::clone(&pat.token), (user_id, Arc::clone(&name))); + personal_access_token_name_index.insert((user_id, Arc::clone(&name))); token_map.insert(name, pat); } personal_access_tokens.insert(user_id, token_map); @@ -1151,6 +1310,7 @@ personal_access_tokens, personal_access_token_index, personal_access_token_expiry_index, + personal_access_token_name_index, permissioner, last_result: None, apply_context: ApplyContext::default(), @@ -1304,6 +1464,41 @@ } #[test] + fn given_token_lifecycle_when_indexing_names_should_follow_every_change_and_restore() { + let mut users = UsersInner::new(); + create_user(&mut users, "alice"); + let token = |name: &str| CreatePersonalAccessTokenRequest { + user_id: 0, + name: WireName::new(name).unwrap(), + expiry: 0, + token_hash: [b'a'; PAT_TOKEN_HASH_BYTES], + }; + let names = |users: &UsersInner| { + users + .personal_access_token_name_index + .iter() + .map(|(user_id, name)| (*user_id, name.to_string())) + .collect::<Vec<_>>() + }; + for name in ["ci", "build"] { + token(name).apply(&mut users, IggyTimestamp::zero()); + } + DeletePersonalAccessTokenRequest { + user_id: 0, + name: WireName::new("ci").unwrap(), + only_if_expired: false, + } + .apply(&mut users, IggyTimestamp::zero()); + assert_eq!(names(&users), [(0, "build".to_owned())]); + + let restored = UsersInner::inner_from_snapshot(Users::from(users.clone()).to_snapshot()); + assert_eq!(names(&restored), names(&users)); + + users.remove_personal_access_tokens(0); + assert_eq!(names(&users), [] as [(UserId, String); 0]); + } + + #[test] fn expiry_gated_delete_skips_token_until_expired() { let mut users = UsersInner::new(); // Created at the epoch with a 1-unit expiry, so it lands in the expiry @@ -1868,7 +2063,11 @@ let stream_only_before = stream_only_record(&inner); let keeps_stream = matches!(deleted, DeletedResource::Topic { .. }); - let reply = RemoveGrantsRequest { deleted }.apply(&mut inner, IggyTimestamp::now()); + let reply = RemoveGrantsRequest { + deleted, + actor_user_id: None, + } + .apply(&mut inner, IggyTimestamp::now()); assert_eq!(reply.code, 0); assert_eq!( Arc::ptr_eq(&stream_only_before, &stream_only_record(&inner)), @@ -2118,7 +2317,10 @@ IggyTimestamp::now(), ); assert_eq!(deactivate.code, 0); - assert_eq!(deactivate.revoked_user, Some(alice_id)); + assert_eq!( + deactivate.revoked_user.map(|user| user.user_id), + Some(alice_id) + ); assert_eq!( users.items.get(alice_id as usize).map(|user| user.status), Some(UserStatus::Inactive)
diff --git a/core/sdk/Cargo.toml b/core/sdk/Cargo.toml index f5553e0..aed6019 100644 --- a/core/sdk/Cargo.toml +++ b/core/sdk/Cargo.toml
@@ -17,7 +17,7 @@ [package] name = "iggy" -version = "0.11.1-edge.2" +version = "0.11.1-edge.3" description = "Iggy is the persistent message streaming platform written in Rust, supporting QUIC, TCP and HTTP transport protocols, capable of processing millions of messages per second." edition = "2024" rust-version.workspace = true
diff --git a/core/sdk/README.md b/core/sdk/README.md index 0838b49..ebb5239 100644 --- a/core/sdk/README.md +++ b/core/sdk/README.md
@@ -251,6 +251,82 @@ For `IggyConsumerConfig`, `partitions_count` controls topic creation only. An ordinary consumer uses partition `0` unless the builder's `partition_id` or the config's `with_partition_id` selects another partition. Code that previously used `partitions_count` to select an existing partition must set `partition_id` explicitly. Consumer-group assignment ignores `partition_id`. +## Metadata changes + +`get_metadata_changes` reads the node's change log: one record per public result of a committed metadata operation, such as a stream created, a topic renamed, a user's permissions changed or a session closed. A mirror keeps a cursor, asks for the records after it, and stores the returned `next` cursor. The call needs root or the global `manage_servers` permission. + +```rust +use iggy::prelude::*; +use std::time::Duration; + +async fn follow(client: &IggyClient, groups: Vec<MetadataChangeGroup>) -> Result<(), IggyError> { + let mut query = MetadataChangesQuery { + groups, + ..MetadataChangesQuery::default() + }; + loop { + let page = client.get_metadata_changes(&query).await?; + match page.status { + MetadataChangeStatus::Ok => { + for record in &page.records { + println!("op {} by {:?}: {:?}", record.op, record.actor_user_id, record.change); + } + if let Some(next) = page.next { + query.start = MetadataChangeStart::After(next); + } + if !page.has_more { + tokio::time::sleep(Duration::from_secs(1)).await; + } + } + MetadataChangeStatus::Behind => tokio::time::sleep(Duration::from_millis(100)).await, + MetadataChangeStatus::Expired | MetadataChangeStatus::Reset => { + query.start = rebuild(client, &query.groups).await?; + } + } + } +} + +/// Pages the current state, then returns the position to follow from. +async fn rebuild( + client: &IggyClient, + groups: &[MetadataChangeGroup], +) -> Result<MetadataChangeStart, IggyError> { + let mut scan_groups = groups.to_vec(); + scan_groups.extend(groups.iter().flat_map(|group| group.parents())); + let mut query = MetadataChangesQuery { + start: MetadataChangeStart::Current(None), + groups: scan_groups, + ..MetadataChangesQuery::default() + }; + loop { + let page = client.get_metadata_changes(&query).await?; + match page.status { + MetadataChangeStatus::Ok => {} + MetadataChangeStatus::Behind => { + tokio::time::sleep(Duration::from_millis(100)).await; + continue; + } + MetadataChangeStatus::Expired | MetadataChangeStatus::Reset => { + query.start = MetadataChangeStart::Current(None); + continue; + } + } + for item in &page.items { + println!("{item:?}"); + } + match (page.current_page_token, page.next) { + (Some(token), _) => query.start = MetadataChangeStart::Current(Some(token)), + (None, Some(anchor)) => return Ok(MetadataChangeStart::After(anchor)), + (None, None) => query.start = MetadataChangeStart::Current(None), + } + } +} +``` + +`Expired` means the log no longer holds the history after the cursor, and `Reset` means the cursor belongs to another history, for example after the data directory was rebuilt. Either way the mirror rebuilds from current state pages and then follows the changes after the anchor they return. + +Current pages need the parent groups of every selected group, for example `topics` with `streams`, so `rebuild` adds them through `MetadataChangeGroup::parents()`. Keep a rebuilt state private until a change page answers `Ok` with `has_more == false` and a `head_op` at or above the last current page's. + ## Versioning Stable releases follow semver (`x.y.z`). Edge releases (`x.y.z-edge.N`) are cut from `master` between stable versions and may include unreleased fixes; pin to a stable version for production.
diff --git a/core/sdk/src/client_wrappers/binary_system_client.rs b/core/sdk/src/client_wrappers/binary_system_client.rs index 19c89ca..9f8e7d4 100644 --- a/core/sdk/src/client_wrappers/binary_system_client.rs +++ b/core/sdk/src/client_wrappers/binary_system_client.rs
@@ -19,8 +19,9 @@ use async_trait::async_trait; use iggy_common::SystemClient; use iggy_common::{ - ClientInfo, ClientInfoDetails, IggyError, NonZeroIggyDuration, OptionSpec, OptionsScope, - Snapshot, SnapshotCompression, Stats, SystemSnapshotType, + ClientInfo, ClientInfoDetails, IggyError, MetadataChanges, MetadataChangesQuery, + NonZeroIggyDuration, OptionSpec, OptionsScope, Snapshot, SnapshotCompression, Stats, + SystemSnapshotType, }; #[async_trait] @@ -120,4 +121,17 @@ ClientWrapper::WebSocket(client) => client.snapshot(compression, snapshot_types).await, } } + + async fn get_metadata_changes( + &self, + query: &MetadataChangesQuery, + ) -> Result<MetadataChanges, IggyError> { + match self { + ClientWrapper::Iggy(client) => client.get_metadata_changes(query).await, + ClientWrapper::Http(client) => client.get_metadata_changes(query).await, + ClientWrapper::Tcp(client) => client.get_metadata_changes(query).await, + ClientWrapper::Quic(client) => client.get_metadata_changes(query).await, + ClientWrapper::WebSocket(client) => client.get_metadata_changes(query).await, + } + } }
diff --git a/core/sdk/src/clients/binary_system.rs b/core/sdk/src/clients/binary_system.rs index ab885a9..e317327 100644 --- a/core/sdk/src/clients/binary_system.rs +++ b/core/sdk/src/clients/binary_system.rs
@@ -20,8 +20,9 @@ use iggy_common::SystemClient; use iggy_common::locking::IggyRwLockFn; use iggy_common::{ - ClientInfo, ClientInfoDetails, IggyError, NonZeroIggyDuration, OptionSpec, OptionsScope, - Snapshot, SnapshotCompression, Stats, SystemSnapshotType, + ClientInfo, ClientInfoDetails, IggyError, MetadataChanges, MetadataChangesQuery, + NonZeroIggyDuration, OptionSpec, OptionsScope, Snapshot, SnapshotCompression, Stats, + SystemSnapshotType, }; #[async_trait] @@ -61,4 +62,11 @@ ) -> Result<Snapshot, IggyError> { super::with_client!(self, client => client.snapshot(compression, snapshot_types).await) } + + async fn get_metadata_changes( + &self, + query: &MetadataChangesQuery, + ) -> Result<MetadataChanges, IggyError> { + super::with_client!(self, client => client.get_metadata_changes(query).await) + } }
diff --git a/core/sdk/src/http/system.rs b/core/sdk/src/http/system.rs index 09d2e9f..2cc47e8 100644 --- a/core/sdk/src/http/system.rs +++ b/core/sdk/src/http/system.rs
@@ -24,6 +24,7 @@ use iggy_common::SystemClient; use iggy_common::get_snapshot::GetSnapshot; use iggy_common::{ClientInfo, ClientInfoDetails}; +use iggy_common::{MetadataChanges, MetadataChangesQuery}; use iggy_common::{OptionSpec, OptionsScope}; use iggy_common::{SnapshotCompression, SystemSnapshotType}; @@ -31,6 +32,7 @@ const CLIENTS: &str = "/clients"; const STATS: &str = "/stats"; const SNAPSHOT: &str = "/snapshot"; +const METADATA_CHANGES: &str = "/metadata/changes"; #[async_trait] impl SystemClient for HttpClient { @@ -111,4 +113,18 @@ let snapshot = Snapshot::new(file.to_vec()); Ok(snapshot) } + + async fn get_metadata_changes( + &self, + query: &MetadataChangesQuery, + ) -> Result<MetadataChanges, IggyError> { + query.validate()?; + let response = self + .get_with_query(METADATA_CHANGES, &query.http_query_pairs()) + .await?; + response + .json() + .await + .map_err(|_| IggyError::InvalidJsonResponse) + } }
diff --git a/core/sdk/src/prelude.rs b/core/sdk/src/prelude.rs index 2863c90..f55176d 100644 --- a/core/sdk/src/prelude.rs +++ b/core/sdk/src/prelude.rs
@@ -86,3 +86,16 @@ RESYNC_REQUIRED_PARTITION_SENTINEL, SEC_IN_MICRO, defaults::{DEFAULT_ROOT_PASSWORD, DEFAULT_ROOT_USER_ID, DEFAULT_ROOT_USERNAME}, }; +pub use iggy_common::{ + MAX_METADATA_CHANGES_LIMIT, MetadataChange, MetadataChangeCursor, MetadataChangeGroup, + MetadataChangeKind, MetadataChangePhase, MetadataChangeRecord, MetadataChangeStart, + MetadataChangeStatus, MetadataChanges, MetadataChangesQuery, MetadataConsumerGroupCreated, + MetadataConsumerGroupDeleted, MetadataConsumerGroupItem, MetadataCurrentItem, + MetadataCurrentPageToken, MetadataMemberItem, MetadataMemberJoined, MetadataMemberLeft, + MetadataMemberLeftReason, MetadataOptions, MetadataPartitionHandedOver, + MetadataPartitionsChanged, MetadataPartitionsItem, MetadataPermissionsUpdatedReason, + MetadataRawJson, MetadataSession, MetadataSessionClosed, MetadataSessionClosedReason, + MetadataStreamChange, MetadataStreamDeleted, MetadataTokenChange, MetadataTokenDeleted, + MetadataTokenDeletedReason, MetadataTopicChange, MetadataTopicDeleted, MetadataTopicItem, + MetadataUserChange, MetadataUserIdentity, MetadataUserPermissionsUpdated, +};
diff --git a/core/server/Cargo.toml b/core/server/Cargo.toml index 7c47bdf..cdddfde 100644 --- a/core/server/Cargo.toml +++ b/core/server/Cargo.toml
@@ -17,7 +17,7 @@ [package] name = "server" -version = "0.9.1-edge.2" +version = "0.9.1-edge.3" edition = "2024" license = "Apache-2.0" publish = false
diff --git a/core/server/README.md b/core/server/README.md index 1dd20ed..f2c3f03 100644 --- a/core/server/README.md +++ b/core/server/README.md
@@ -47,8 +47,10 @@ Deploy matching server and SDK releases together. Rolling upgrades and mixed server versions are unsupported. Partition control frames from older servers carry incarnation zero and are rejected, so a mixed-version cluster cannot -complete partition view changes. The `IGGY-RESOURCE-OPTIONS-1` storage gate +complete partition view changes. The `IGGY-METADATA-CHANGE-LOG-1` storage gate requires fresh storage and rejects older data directories without modifying them. +Metadata snapshots now carry the metadata change log and topic creation ops, so +`IGGY-RESOURCE-OPTIONS-1` data directories must be wiped before the upgrade. Cached replies now preserve a committed-result marker, which older receipts lack. Partition creation records carry an initial view in each 16-byte assignment, replacing the 12-byte assignments and single request-wide view in older metadata @@ -358,6 +360,26 @@ A stream created during phase 1 also carries a wrap under the old key until phase 2 drops it. Rotation changes only the wraps, so whoever already holds a stream's data key, or an old wrap together with the old server key, can still read that stream. Changing a stream's own data key is not supported. +## Metadata changes + +Each node keeps a bounded log of the public results of committed metadata operations: streams, topics, partition counts, consumer groups and their members, users, personal access tokens and sessions. One operation can produce several records, for example a stream delete followed by the permission updates that pruned grants on it. Refused, denied and no-op operations produce none. Records are produced by the state machine on every replica alike, so a cursor taken on one node continues on another. + +The binary `GetMetadataChanges` command (17) serves it over TCP, QUIC and WebSocket, `GET /metadata/changes` over HTTP and `iggy metadata changes` on the command line. Reading it needs root or the global `manage_servers` permission. Other callers get `Unauthorized` (41), 403 over HTTP, whatever groups they ask for. + +A request starts at `earliest`, at `latest` (a cursor and no records), after a cursor, or at `current`, which pages the present state instead of history. It may name groups (`streams`, `topics`, `partitions`, `consumer_groups`, `consumer_group_members`, `users`, `personal_access_tokens`, `sessions`), a time range in microseconds and a limit from 1 to 1000. Cursors print as `mc1_` tokens and state page tokens as `cs1_` tokens. Send them back unchanged. + +A time range only goes with `earliest` or a cursor. A `current` request must also name the parent groups of every selected group, for example `streams` with `topics`. A request that breaks either rule is refused with `InvalidCommand` (3). + +A page also ends before its encoded records or items pass 1 MiB, so it can hold fewer than the limit with `has_more` set. A single larger record or item still comes alone. A current page also ends after it walks 65536 streams and topics, so it can carry a page token and no items. + +Each answer has a status. `ok` carries records and the `next` cursor. `behind` means this node has not applied the cursor's operation yet, so ask again or ask another node. `expired` means the node no longer holds every record after the cursor for those groups, and `reset` means the cursor belongs to another history, for example a rebuilt data directory. Neither carries a cursor: page `current` again and follow from the anchor its pages return. + +Records name streams and topics by id and creation op, because ids are reused after a delete. Each record carries the acting user, absent for server actions such as key rotation, partition handovers and expiry. Stream options appear only as the read routes return them: key source and fingerprints, never a stream key or its wraps. + +Two rings bound the log. `[metadata] change_log_resource_capacity` (default 65536) holds records of streams, topics, partitions, groups, users and tokens, and `change_log_activity_capacity` (default 16384) holds sessions and group membership, so busy sessions never evict resource history. Both accept 1 to 1048576 records. The log rides the metadata snapshot, survives restarts and moves with state transfer. + +Each ring also keeps at most 128 MiB of record bodies and evicts its oldest records past that, so large permission sets cannot grow the snapshot past what state transfer carries. + ## Upgrade recovery This release changes protocol and storage formats. Servers and compatible @@ -384,7 +406,7 @@ procedure above for an upgrade. A recovery from a backup must use a separately verified, compatible backup; copying a format marker is not a migration. -Within compatible `IGGY-RESOURCE-OPTIONS-1` storage, this release reads superblock +Within compatible `IGGY-METADATA-CHANGE-LOG-1` storage, this release reads superblock envelope versions 1 and 2 and writes version 2. Legacy 66- and 74-byte consensus records remain readable. New records atomically retain an adopted view, its advertised head and exact prepare headers alongside the existing consensus state. @@ -457,7 +479,7 @@ `partition-initialization/<namespace>/created.revision` under the system data directory. This atomic record remains outside the partition directory, including after that directory is lost. Preserve it with metadata and partition backups. -Storage format `IGGY-RESOURCE-OPTIONS-1` requires this initialization contract; +Storage format `IGGY-METADATA-CHANGE-LOG-1` requires this initialization contract; older data directories are refused before mutation. Retirement also writes `retirement.fence` in that external namespace directory
diff --git a/core/server/config.toml b/core/server/config.toml index ac9f94f..fc63036 100644 --- a/core/server/config.toml +++ b/core/server/config.toml
@@ -921,6 +921,20 @@ # instead, and the client retries it until its deadline. 0 means no cap. partitions_max = 0 +# Records the metadata change log keeps for streams, topics, partitions, +# consumer groups, users and personal access tokens. Once the count passes +# it, the oldest records go, and a reader whose position they held must +# rebuild from current state. The log is part of the metadata snapshot, so a +# larger value costs memory and checkpoint size. Node-local: replicas may keep +# different depths. Must be between 1 and 1048576. Each of the two record +# budgets also keeps at most 128 MiB of record bodies. +change_log_resource_capacity = 65536 + +# Records the metadata change log keeps for sessions and consumer group +# membership. A separate budget, so a login or membership storm cannot evict +# resource history. Must be between 1 and 1048576. +change_log_activity_capacity = 16384 + # Per-partition consensus plane tunables. Unlike [metadata] (one shard-0 # plane), a pipeline exists per partition, so raising this multiplies pinned # request-buffer memory by the partition count. Keep it modest.
diff --git a/core/server/server.http b/core/server/server.http index 803263d..cbf4ea0 100644 --- a/core/server/server.http +++ b/core/server/server.http
@@ -18,6 +18,8 @@ @url = http://localhost:3000 @stream_id = 0 @topic_id = 0 +@metadata_cursor = mc1_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA +@current_page_token = REPLACE_WITH_THE_PREVIOUS_CURRENT_PAGE_TOKEN @partition_id = 0 @consumer_group_id = 1 @consumer_id = 1 @@ -73,6 +75,26 @@ "snapshot_types": ["All"] } +### Metadata changes from the oldest retained operation +GET {{url}}/metadata/changes?from=earliest&limit=100 +Authorization: Bearer {{access_token}} + +### Metadata changes: a cursor at the newest record, no records +GET {{url}}/metadata/changes?from=latest&groups=streams,topics +Authorization: Bearer {{access_token}} + +### Metadata changes after a cursor, within a time range in microseconds +GET {{url}}/metadata/changes?after={{metadata_cursor}}&since=1760352000000000&limit=100 +Authorization: Bearer {{access_token}} + +### Current state, first page, then continue with current_after +GET {{url}}/metadata/changes?from=current&groups=streams,topics&limit=100 +Authorization: Bearer {{access_token}} + +### Current state, next page +GET {{url}}/metadata/changes?from=current&groups=streams,topics¤t_after={{current_page_token}} +Authorization: Bearer {{access_token}} + ### GET {{url}}/cluster/metadata Authorization: Bearer {{access_token}}
diff --git a/core/server/src/boot/handoff.rs b/core/server/src/boot/handoff.rs index df0b44e..3c848a3 100644 --- a/core/server/src/boot/handoff.rs +++ b/core/server/src/boot/handoff.rs
@@ -236,6 +236,8 @@ mod tests { use super::*; use crate::shell::ServerMuxStateMachine; + use iggy_binary_protocol::PrepareHeader; + use metadata::change_log::{ChangeLog, ChangeLogCapacities, MetadataChangeLog}; const TEST_POLL_INTERVAL: Duration = Duration::from_millis(50); @@ -246,10 +248,14 @@ let (bundle_tx, _bundle_rx) = crossfire::mpmc::bounded_async::<ServerMetadataBundle>(0); let flag = Arc::new(AtomicBool::new(false)); let mux = ServerMuxStateMachine::default(); + let change_log = MetadataChangeLog::new(ChangeLogCapacities::default()); broadcast_metadata_bundle( 0, &bundle_tx, - mux.factory_bundle(), + ServerMetadataBundle { + states: mux.factory_bundle(), + change_log: change_log.factory(), + }, 0, &flag, TEST_POLL_INTERVAL, @@ -262,15 +268,19 @@ async fn metadata_bundle_round_trips_through_channel() { // End-to-end: shard 0 mints a bundle, a peer receives it on // another runtime, and `from_factory_bundle` constructs a - // reader-mode mux that observes shard 0's writes via the same - // LeftRight pair. + // reader-mode mux and change log that observe shard 0's writes via + // the same LeftRight pairs. let peers = 1u16; let (bundle_tx, bundle_rx) = crossfire::mpmc::bounded_async::<ServerMetadataBundle>(usize::from(peers)); let flag = Arc::new(AtomicBool::new(false)); let owner = ServerMuxStateMachine::default(); - let bundle = owner.factory_bundle(); + let owner_log = MetadataChangeLog::new(ChangeLogCapacities::default()); + let bundle = ServerMetadataBundle { + states: owner.factory_bundle(), + change_log: owner_log.factory(), + }; broadcast_metadata_bundle(0, &bundle_tx, bundle, peers, &flag, TEST_POLL_INTERVAL) .await .expect("broadcast must succeed with one peer drained"); @@ -278,7 +288,19 @@ let received = await_metadata_bundle(1, &bundle_rx, &flag, TEST_POLL_INTERVAL) .await .expect("peer must receive the broadcast bundle"); - let _peer_mux = ServerMuxStateMachine::from_factory_bundle(received); + let _peer_mux = ServerMuxStateMachine::from_factory_bundle(received.states); + let peer_log = MetadataChangeLog::from_factory(received.change_log); + owner_log + .append( + &PrepareHeader { + op: 1, + checksum: 0x5eed, + ..PrepareHeader::default() + }, + Vec::new(), + ) + .expect("shard 0 owns the write side"); + assert_eq!(peer_log.read(ChangeLog::log_id), Some(Some(0x5eed))); } #[compio::test] @@ -291,11 +313,15 @@ drop(bundle_rx); let flag = Arc::new(AtomicBool::new(false)); let mux = ServerMuxStateMachine::default(); + let change_log = MetadataChangeLog::new(ChangeLogCapacities::default()); let err = broadcast_metadata_bundle( 0, &bundle_tx, - mux.factory_bundle(), + ServerMetadataBundle { + states: mux.factory_bundle(), + change_log: change_log.factory(), + }, 3, &flag, TEST_POLL_INTERVAL,
diff --git a/core/server/src/boot/mod.rs b/core/server/src/boot/mod.rs index aa10cb5..15a1046 100644 --- a/core/server/src/boot/mod.rs +++ b/core/server/src/boot/mod.rs
@@ -66,7 +66,7 @@ use message_bus::replica::handshake::ReplicaHandshakeCtx; use message_bus::transports::tls::install_default_crypto_provider; use message_bus::{IggyMessageBus, ReplicaOwnerTable}; -use metadata::impls::metadata::StreamsFrontend; +use metadata::change_log::{ChangeLogCapacities, MetadataChangeLog}; use metadata::impls::recovery::recover; use metadata::{AppliedFrontier, ReplicaIdentity}; use server_common::bootstrap::create_directories; @@ -87,7 +87,7 @@ const STORAGE_FORMAT_FILE: &str = "storage-format"; const STORAGE_FORMAT_TEMP_FILE: &str = "storage-format.tmp"; -const STORAGE_FORMAT: &[u8] = b"IGGY-RESOURCE-OPTIONS-1\n"; +const STORAGE_FORMAT: &[u8] = b"IGGY-METADATA-CHANGE-LOG-1\n"; /// Load the server configuration from the active config provider. /// @@ -569,73 +569,64 @@ // so the writer must first be parked in a binding that outlives the peer // wait armed below. A `recover()` failure inside the arm is safe for the // same reason -- no peer holds a handle yet. - let (mux_stm, pending_bundle_tx, owner_state, root_credentials) = match metadata_handoff { - MetadataHandoff::Owner { - bundle_tx, - root_credentials, - } => { - let recovered = recover::<ServerMuxStateMachine>( - data_dir, - ReplicaIdentity { - cluster: topology.cluster_id, - replica_id: topology.self_replica_id, - replica_count: topology.replica_count, - }, - config.metadata.journal_slots, - config.metadata.clients_table_max, - |_| {}, - |mux_stm, header| { - if header.operation == iggy_binary_protocol::Operation::Register { - mux_stm.streams().refresh_consumer_group_session( - header.client, - header.op, - header.op, - iggy_common::IggyTimestamp::from(header.timestamp), - ); - } else { - mux_stm.streams().remove_consumer_group_member( - header.client, - iggy_common::IggyTimestamp::from(header.timestamp), - header.op, - ); - } - }, - ) - .await - .map_err(ServerError::MetadataRecovery)?; - ( - Rc::new(recovered.mux_stm), - Some(bundle_tx), - Some(RecoveredOwnerState { - journal: recovered.journal, - snapshot: recovered.snapshot, - last_applied_op: recovered.last_applied_op, - last_journaled_op: recovered.last_journaled_op, - client_table: recovered.client_table, - superblock: recovered.superblock, - recovered_state: recovered.recovered_state, - snapshot_checkpoint: recovered.snapshot_checkpoint, - history: recovered.history, - }), + let (mux_stm, change_log, pending_bundle_tx, owner_state, root_credentials) = + match metadata_handoff { + MetadataHandoff::Owner { + bundle_tx, root_credentials, - ) - } - MetadataHandoff::Waiter { bundle_rx } => { - let bundle = await_metadata_bundle( - shard_id, - &bundle_rx, - &shutdown_flag_for_handoff, - poll_interval, - ) - .await?; - ( - Rc::new(ServerMuxStateMachine::from_factory_bundle(bundle)), - None, - None, - None, - ) - } - }; + } => { + let recovered = recover::<ServerMuxStateMachine>( + data_dir, + ReplicaIdentity { + cluster: topology.cluster_id, + replica_id: topology.self_replica_id, + replica_count: topology.replica_count, + }, + config.metadata.journal_slots, + config.metadata.clients_table_max, + |_| {}, + ChangeLogCapacities { + resource: config.metadata.change_log_resource_capacity, + activity: config.metadata.change_log_activity_capacity, + }, + ) + .await + .map_err(ServerError::MetadataRecovery)?; + ( + Rc::new(recovered.mux_stm), + Rc::new(recovered.change_log), + Some(bundle_tx), + Some(RecoveredOwnerState { + journal: recovered.journal, + snapshot: recovered.snapshot, + last_applied_op: recovered.last_applied_op, + last_journaled_op: recovered.last_journaled_op, + client_table: recovered.client_table, + superblock: recovered.superblock, + recovered_state: recovered.recovered_state, + snapshot_checkpoint: recovered.snapshot_checkpoint, + history: recovered.history, + }), + root_credentials, + ) + } + MetadataHandoff::Waiter { bundle_rx } => { + let bundle = await_metadata_bundle( + shard_id, + &bundle_rx, + &shutdown_flag_for_handoff, + poll_interval, + ) + .await?; + ( + Rc::new(ServerMuxStateMachine::from_factory_bundle(bundle.states)), + Rc::new(MetadataChangeLog::from_factory(bundle.change_log)), + None, + None, + None, + ) + } + }; // Shard 0 owns the metadata state machine's only write handle, and the // peers read through it until their runtimes are gone. Declared after @@ -655,7 +646,10 @@ broadcast_metadata_bundle( shard_id, &bundle_tx, - mux_stm.factory_bundle(), + ServerMetadataBundle { + states: mux_stm.factory_bundle(), + change_log: change_log.factory(), + }, total_shards.saturating_sub(1), &shutdown_flag_for_handoff, poll_interval, @@ -700,7 +694,8 @@ Rc::clone(&mux_stm), Some(PathBuf::from(&config.path)), ) - .with_applied_frontier(metadata_applied_frontier); + .with_applied_frontier(metadata_applied_frontier) + .with_change_log(Rc::clone(&change_log)); metadata.seed_applied_frontier_from_consensus(); // Size the VSR client table before listeners bind and any client registers. // Must precede the recovered-table install below: the setter rebuilds the @@ -1289,6 +1284,7 @@ b"IGGY-BOUND-GRANTS-4\n", b"IGGY-BOUND-GRANTS-5\n", b"IGGY-DURABLE-SESSIONS-3\n", + b"IGGY-RESOURCE-OPTIONS-1\n", &STORAGE_FORMAT[..STORAGE_FORMAT.len() - 1], ] { std::fs::write(&marker, format).unwrap(); @@ -1322,6 +1318,7 @@ b"IGGY-BOUND-GRANTS-3\n".as_slice(), b"IGGY-BOUND-GRANTS-4\n", b"IGGY-BOUND-GRANTS-5\n", + b"IGGY-RESOURCE-OPTIONS-1\n", ] { assert_ne!( message_bus::replica::handshake::binary_identity(crate::VERSION, STORAGE_FORMAT),
diff --git a/core/server/src/boot/recovery.rs b/core/server/src/boot/recovery.rs index a7f33bb..f6cd676 100644 --- a/core/server/src/boot/recovery.rs +++ b/core/server/src/boot/recovery.rs
@@ -462,6 +462,17 @@ ); const _: () = assert!(configs::metadata::DEFAULT_METADATA_CLIENTS_TABLE_MAX == consensus::CLIENTS_TABLE_MAX); +const _: () = assert!( + configs::metadata::DEFAULT_METADATA_CHANGE_LOG_RESOURCE_CAPACITY + == metadata::change_log::DEFAULT_RESOURCE_CAPACITY +); +const _: () = assert!( + configs::metadata::DEFAULT_METADATA_CHANGE_LOG_ACTIVITY_CAPACITY + == metadata::change_log::DEFAULT_ACTIVITY_CAPACITY +); +const _: () = assert!( + configs::metadata::MAX_METADATA_CHANGE_LOG_CAPACITY == metadata::change_log::MAX_CAPACITY +); const _: () = assert!(configs::cluster::DEFAULT_VIEW_PROBE_ATTEMPTS_MAX == consensus::PROBE_ATTEMPTS_MAX); const _: () =
diff --git a/core/server/src/consumer_group/liveness.rs b/core/server/src/consumer_group/liveness.rs index 05c7b7a..d8e9ea0 100644 --- a/core/server/src/consumer_group/liveness.rs +++ b/core/server/src/consumer_group/liveness.rs
@@ -1073,6 +1073,7 @@ SESSION + 1, SESSION + 1, iggy_common::IggyTimestamp::default(), + None, ); let mut table = ClientTable::new(MAX_LOGOUTS_PER_PASS + 1); for index in 0..=MAX_LOGOUTS_PER_PASS { @@ -1265,6 +1266,7 @@ SESSION, SESSION, iggy_common::IggyTimestamp::default(), + None, ); } (dir, Rc::new(shard))
diff --git a/core/server/src/dispatch/authz.rs b/core/server/src/dispatch/authz.rs index a5d05e2..f05e165 100644 --- a/core/server/src/dispatch/authz.rs +++ b/core/server/src/dispatch/authz.rs
@@ -463,8 +463,8 @@ use iggy_binary_protocol::COMMAND_TABLE; use iggy_binary_protocol::codes::{ BIND_SESSION_CODE, CREATE_STREAM_CODE, GET_CLIENT_CODE, GET_CLIENTS_CODE, - GET_CONSUMER_OFFSET_CODE, GET_ME_CODE, GET_PARTITION_ROUTE_CODE, GET_SNAPSHOT_FILE_CODE, - LOGIN_REGISTER_CODE, LOGIN_REGISTER_WITH_PAT_CODE, LOGIN_USER_CODE, + GET_CONSUMER_OFFSET_CODE, GET_ME_CODE, GET_METADATA_CHANGES_CODE, GET_PARTITION_ROUTE_CODE, + GET_SNAPSHOT_FILE_CODE, LOGIN_REGISTER_CODE, LOGIN_REGISTER_WITH_PAT_CODE, LOGIN_USER_CODE, LOGIN_WITH_PERSONAL_ACCESS_TOKEN_CODE, LOGOUT_USER_CODE, PING_CODE, POLL_MESSAGES_CODE, SYNC_CONSUMER_GROUP_CODE, }; @@ -550,6 +550,7 @@ invalid_command, ), (DESCRIBE_OPTIONS_CODE, unauthenticated, allow), + (GET_METADATA_CHANGES_CODE, invalid_command, invalid_command), ] }
diff --git a/core/server/src/dispatch/metadata_changes.rs b/core/server/src/dispatch/metadata_changes.rs new file mode 100644 index 0000000..700b7ae --- /dev/null +++ b/core/server/src/dispatch/metadata_changes.rs
@@ -0,0 +1,127 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use crate::dispatch::authz::authorize_uid; +use crate::shell::{ShellBus, ShellShard}; +use bytes::Bytes; +use consensus::MetadataHandle; +use futures::future::{Either, select}; +use iggy_binary_protocol::requests::system::GetMetadataChangesRequest; +use iggy_binary_protocol::responses::system::GetMetadataChangesResponse; +use iggy_binary_protocol::{PrepareHeader, WireDecode, WireEncode}; +use iggy_common::{IggyError, MetadataChangeStart, MetadataChangesQuery}; +use journal::superblock::SuperblockStore; +use journal::{Journal, JournalHandle}; +use metadata::permissioner::Permissioner; +use server_common::Message; +use shard::MetadataSubmit; +use std::pin::pin; +use std::rc::Rc; + +/// The binary answer to `metadata.changes`: the request decoded under the +/// rules every transport shares, then one page. +/// +/// # Errors +/// As [`metadata_changes_page`], plus `InvalidCommand` for a body that does +/// not decode and `Unauthenticated` without a caller. +#[allow(clippy::future_not_send)] +pub async fn metadata_changes_reply<B, MJ, S, SB>( + shard: &Rc<ShellShard<B, MJ, S, SB>>, + body: &[u8], + user_id: Option<u32>, +) -> Result<Bytes, IggyError> +where + B: ShellBus, + MJ: JournalHandle + 'static, + MJ::Target: Journal<Entry = Message<PrepareHeader>, Header = PrepareHeader>, + S: 'static, + SB: SuperblockStore + 'static, +{ + let user_id = user_id.ok_or(IggyError::Unauthenticated)?; + let query = GetMetadataChangesRequest::decode_from(body) + .map_err(|_| IggyError::InvalidCommand) + .and_then(MetadataChangesQuery::try_from)?; + metadata_changes_page(shard, query, user_id) + .await + .map(|page| page.to_bytes()) +} + +/// One `metadata.changes` page for an authorized caller. +/// +/// A change page answers from this shard's view of the log and is never +/// forwarded. A current state page is assembled on shard 0, which holds the +/// client table and the only write side of the state, and which checks the +/// rule again right before it projects. +/// +/// # Errors +/// `InvalidCommand` for a request the rules refuse, `Unauthorized` when the +/// caller lost the grant, `TransientNotAccepted` when no coherent image was +/// captured or shard 0 did not answer within the read budget. +#[allow(clippy::future_not_send)] +pub async fn metadata_changes_page<B, MJ, S, SB>( + shard: &Rc<ShellShard<B, MJ, S, SB>>, + query: MetadataChangesQuery, + user_id: u32, +) -> Result<GetMetadataChangesResponse, IggyError> +where + B: ShellBus, + MJ: JournalHandle + 'static, + MJ::Target: Journal<Entry = Message<PrepareHeader>, Header = PrepareHeader>, + S: 'static, + SB: SuperblockStore + 'static, +{ + if !matches!(query.start, MetadataChangeStart::Current(_)) { + return shard.plane.metadata().metadata_changes(&query); + } + if shard.id == 0 { + return current_state_page(shard, &query, user_id); + } + let (reply, receiver) = shard::channel(1); + shard.forward_metadata_submit(MetadataSubmit::MetadataCurrentState { + query: Box::new(query), + user_id, + reply, + }); + let outcome = pin!(receiver.recv()); + let deadline = pin!( + shard + .bus + .sleep(shard.plane.metadata().applied_frontier().read_budget()) + ); + match select(outcome, deadline).await { + Either::Left((result, _)) => result.map_err(|_| IggyError::TransientNotAccepted)?, + Either::Right(_) => Err(IggyError::TransientNotAccepted), + } +} + +/// A current state page on shard 0: the rule, then the projection, with no +/// await between them. +pub fn current_state_page<B, MJ, S, SB>( + shard: &Rc<ShellShard<B, MJ, S, SB>>, + query: &MetadataChangesQuery, + user_id: u32, +) -> Result<GetMetadataChangesResponse, IggyError> +where + B: ShellBus, + MJ: JournalHandle + 'static, + MJ::Target: Journal<Entry = Message<PrepareHeader>, Header = PrepareHeader>, + S: 'static, + SB: SuperblockStore + 'static, +{ + authorize_uid(shard, Some(user_id), Permissioner::get_metadata_changes)?; + shard.plane.metadata().metadata_current_state(query) +}
diff --git a/core/server/src/dispatch/mod.rs b/core/server/src/dispatch/mod.rs index f8b75b5..2d5dfc8 100644 --- a/core/server/src/dispatch/mod.rs +++ b/core/server/src/dispatch/mod.rs
@@ -36,6 +36,7 @@ mod failure; pub mod host; pub mod login_error; +pub mod metadata_changes; pub mod partition; pub mod reads; mod segment_delete; @@ -1223,8 +1224,10 @@ use iggy_binary_protocol::Command; use iggy_binary_protocol::codes::{ GET_CLIENT_CODE, GET_CLIENTS_CODE, GET_CLUSTER_METADATA_CODE, GET_CONSUMER_OFFSET_CODE, - GET_ME_CODE, GET_PARTITION_ROUTE_CODE, GET_STATS_CODE, POLL_MESSAGES_CODE, + GET_ME_CODE, GET_METADATA_CHANGES_CODE, GET_PARTITION_ROUTE_CODE, GET_STATS_CODE, + POLL_MESSAGES_CODE, }; + use iggy_binary_protocol::primitives::metadata_changes::WireMetadataChangeStatus; use iggy_binary_protocol::primitives::polling_strategy::WirePollingStrategy; use iggy_binary_protocol::requests::consumer_offsets::{ GetConsumerOffsetRequest, StoreConsumerOffsetRequest, @@ -1235,8 +1238,8 @@ CreateStreamRequest, UpdateStreamResolvedRequest, }; use iggy_binary_protocol::requests::system::{ - BindSessionRequest, GetClientRequest, GetClientsRequest, GetStatsRequest, ResourceTarget, - UpdateResourceRequest, + BindSessionRequest, GetClientRequest, GetClientsRequest, GetMetadataChangesRequest, + GetStatsRequest, ResourceTarget, UpdateResourceRequest, }; use iggy_binary_protocol::requests::topics::{DeleteTopicRequest, UpdateTopicResolvedRequest}; use iggy_binary_protocol::requests::users::login_register::{ @@ -1246,13 +1249,14 @@ use iggy_binary_protocol::requests::users::{ CreateUserRequest, DeleteUserRequest, UpdatePermissionsRequest, UpdateUserResolvedRequest, }; + use iggy_binary_protocol::responses::system::GetMetadataChangesResponse; use iggy_binary_protocol::{ AckLevel, ClientVersionInfo, EvictionHeader, IGGY_PROTOCOL_VERSION, ReplyHeader, WireConsumer, }; - use iggy_binary_protocol::{WireEncode, WireIdentifier, WireName, WireOptions}; + use iggy_binary_protocol::{WireDecode, WireEncode, WireIdentifier, WireName, WireOptions}; use iggy_common::defaults::DEFAULT_ROOT_USER_ID; - use iggy_common::{Identifier, IggyByteSize}; + use iggy_common::{Identifier, IggyByteSize, MetadataChangeStart, MetadataChangesQuery}; use journal::prepare_journal::PrepareJournal; use message_bus::installer::conn_info::ClientTransportKind; use metadata::impls::metadata::IggySnapshot; @@ -1309,6 +1313,7 @@ |_| {}, prepare, ) + .reply }; let create = CreateUserRequest { username: WireName::new("original").unwrap(), @@ -1692,6 +1697,61 @@ } #[compio::test] + async fn given_root_session_when_reading_metadata_changes_should_answer_pages_and_refuse_bad_bodies() + { + let (bus, shard, host) = root_host(ServerConfig::default()); + bind_root_session(&shard, host.sessions()).await; + let metadata = shard.plane.metadata(); + metadata + .change_log() + .append( + &PrepareHeader { + op: 1, + checksum: 0x5eed, + ..PrepareHeader::default() + }, + Vec::new(), + ) + .unwrap(); + let read = |request, body: &[u8]| { + let mut read = wire_request( + Operation::NonReplicated, + ROOT_CLIENT, + ROOT_SESSION, + request, + body, + ); + bytemuck::checked::from_bytes_mut::<RequestHeader>( + &mut read.as_mut_slice()[..size_of::<RequestHeader>()], + ) + .reserved[..4] + .copy_from_slice(&GET_METADATA_CHANGES_CODE.to_le_bytes()); + read.into_generic() + }; + let latest = GetMetadataChangesRequest::from(&MetadataChangesQuery { + start: MetadataChangeStart::Latest, + ..MetadataChangesQuery::default() + }) + .to_bytes(); + + host.on_client_request(ROOT_TRANSPORT, read(1, &latest)); + host.on_client_request(ROOT_TRANSPORT, read(2, &[0xff])); + await_replies(&bus, 2).await; + + let replies = bus.client_replies.borrow(); + assert_eq!(replies.len(), 2); + let (header, body) = replies[0].1.split_at(size_of::<ReplyHeader>()); + assert_eq!(reply_status(header), 0); + let page = GetMetadataChangesResponse::decode_from(body).expect("a change page"); + assert_eq!(page.status, WireMetadataChangeStatus::Ok); + assert_eq!(page.log_id, Some(0x5eed)); + assert_eq!( + reply_status(&replies[1].1), + IggyError::InvalidCommand.as_code() + ); + } + + #[compio::test] async fn given_a_read_held_at_the_frontier_when_the_session_ends_should_refuse_stale_client() { let (bus, shard, host) = root_host(ServerConfig::default()); bind_root_session(&shard, host.sessions()).await; @@ -2348,7 +2408,8 @@ true, |_| {}, consensus::seal_prepare_checksum(revocation), - ); + ) + .reply; assert_eq!(iggy_binary_protocol::result_code(reply.body()), Some(0)); } let old_binding = sessions.borrow().get_session(TRANSPORT); @@ -4030,7 +4091,8 @@ true, |_| {}, prepare, - ); + ) + .reply; assert_eq!(iggy_binary_protocol::result_code(created.body()), Some(0)); let create = |request, name| { let body = WireCreatePersonalAccessTokenRequest { @@ -4123,7 +4185,8 @@ true, |_| {}, prepare, - ); + ) + .reply; assert_eq!(iggy_binary_protocol::result_code(created.body()), Some(0)); let resent = submit_client_request_on_owner(&shard, minted)
diff --git a/core/server/src/dispatch/reads.rs b/core/server/src/dispatch/reads.rs index 6466337..301f42b 100644 --- a/core/server/src/dispatch/reads.rs +++ b/core/server/src/dispatch/reads.rs
@@ -35,6 +35,7 @@ ClientOrigin, FrameChannel, send_host_frame, send_non_replicated_bytes, send_non_replicated_deny, }; +use crate::dispatch::metadata_changes::metadata_changes_reply; use crate::dispatch::partition::{handle_get_consumer_offset, handle_poll_messages}; use crate::namespace::resolve_partition_namespace; use crate::reply_frame::{build_empty_reply, current_metadata_commit}; @@ -54,9 +55,9 @@ use iggy_binary_protocol::PrepareHeader; use iggy_binary_protocol::codes::{ BIND_SESSION_CODE, DESCRIBE_OPTIONS_CODE, GET_CLIENT_CODE, GET_CLIENTS_CODE, - GET_CLUSTER_METADATA_CODE, GET_CONSUMER_OFFSET_CODE, GET_ME_CODE, GET_PARTITION_ROUTE_CODE, - GET_PERSONAL_ACCESS_TOKENS_CODE, GET_SNAPSHOT_FILE_CODE, GET_STATS_CODE, PING_CODE, - POLL_MESSAGES_CODE, SYNC_CONSUMER_GROUP_CODE, + GET_CLUSTER_METADATA_CODE, GET_CONSUMER_OFFSET_CODE, GET_ME_CODE, GET_METADATA_CHANGES_CODE, + GET_PARTITION_ROUTE_CODE, GET_PERSONAL_ACCESS_TOKENS_CODE, GET_SNAPSHOT_FILE_CODE, + GET_STATS_CODE, PING_CODE, POLL_MESSAGES_CODE, SYNC_CONSUMER_GROUP_CODE, }; use iggy_binary_protocol::dispatch::lookup_command; use iggy_binary_protocol::requests::consumer_groups::SyncConsumerGroupRequest; @@ -520,6 +521,36 @@ GET_SNAPSHOT_FILE_CODE => { handle_get_snapshot(shard, server_config, origin, &request, user_id, attachment).await; } + GET_METADATA_CHANGES_CODE => { + let authorize = || authorize_uid(shard, user_id, Permissioner::get_metadata_changes); + if let Err(error) = + authorize_and_hold_read(shard, attachment, code, watermark, &authorize).await + { + send_non_replicated_deny(shard, &request, origin, error.as_code()).await; + return; + } + // A current state page hops to shard 0, and the session can end + // during the hop. + let reply = metadata_changes_reply(shard, request_body(&request), user_id) + .await + .and_then(|bytes| authorize_attached(attachment, authorize).map(|()| bytes)); + match reply { + Ok(bytes) => { + send_non_replicated_bytes( + shard, + &request, + origin, + bytes, + FrameChannel::Reply, + "metadata_changes", + ) + .await; + } + Err(error) => { + send_non_replicated_deny(shard, &request, origin, error.as_code()).await; + } + } + } BIND_SESSION_CODE | GET_PARTITION_ROUTE_CODE => { let result = if code == BIND_SESSION_CODE { bind_session(shard, sessions, origin, &request).await
diff --git a/core/server/src/dispatch/submit.rs b/core/server/src/dispatch/submit.rs index f224043..733dba9 100644 --- a/core/server/src/dispatch/submit.rs +++ b/core/server/src/dispatch/submit.rs
@@ -132,6 +132,14 @@ .bind_session(vsr_client_id, session, secret.expose_secret()); let _ = reply.try_send(attached); } + shard::MetadataSubmit::MetadataCurrentState { + query, + user_id, + reply, + } => { + let page = super::metadata_changes::current_state_page(&shard, &query, user_id); + let _ = reply.try_send(page); + } shard::MetadataSubmit::Register { vsr_client_id, user_id,
diff --git a/core/server/src/http.rs b/core/server/src/http.rs index dc5c31f..370d29b 100644 --- a/core/server/src/http.rs +++ b/core/server/src/http.rs
@@ -72,9 +72,9 @@ change_password, create_cg, create_partitions, create_pat, create_stream, create_topic, create_user, delete_cg, delete_consumer_offset, delete_partitions, delete_pat, delete_segments, delete_stream, delete_topic, delete_user, describe_options, get_cg, get_cgs, get_client, - get_clients, get_cluster_metadata, get_consumer_offset, get_pats, get_snapshot, get_stats, - get_stream, get_streams, get_topic, get_topics, get_user, get_users, login_user, - login_with_personal_access_token, logout_user, ping, poll_messages, refresh_token, + get_clients, get_cluster_metadata, get_consumer_offset, get_metadata_changes, get_pats, + get_snapshot, get_stats, get_stream, get_streams, get_topic, get_topics, get_user, get_users, + login_user, login_with_personal_access_token, logout_user, ping, poll_messages, refresh_token, send_messages, store_consumer_offset, update_permissions, update_stream, update_topic, update_user, }; @@ -412,6 +412,7 @@ .route("/stats", get(get_stats)) .route("/options/{scope}", get(describe_options)) .route("/snapshot", post(get_snapshot)) + .route("/metadata/changes", get(get_metadata_changes)) .route("/cluster/metadata", get(get_cluster_metadata)) .route( forward::METADATA_FLOOR_PATH,
diff --git a/core/server/src/http/handlers.rs b/core/server/src/http/handlers.rs index 76f7059..ccd4673 100644 --- a/core/server/src/http/handlers.rs +++ b/core/server/src/http/handlers.rs
@@ -33,9 +33,10 @@ use consensus::MetadataHandle; use iggy_binary_protocol::codes::{ DESCRIBE_OPTIONS_CODE, GET_CLIENT_CODE, GET_CLIENTS_CODE, GET_CONSUMER_GROUP_CODE, - GET_CONSUMER_GROUPS_CODE, GET_CONSUMER_OFFSET_CODE, GET_PERSONAL_ACCESS_TOKENS_CODE, - GET_SNAPSHOT_FILE_CODE, GET_STATS_CODE, GET_STREAM_CODE, GET_STREAMS_CODE, GET_TOPIC_CODE, - GET_TOPICS_CODE, GET_USER_CODE, GET_USERS_CODE, POLL_MESSAGES_CODE, + GET_CONSUMER_GROUPS_CODE, GET_CONSUMER_OFFSET_CODE, GET_METADATA_CHANGES_CODE, + GET_PERSONAL_ACCESS_TOKENS_CODE, GET_SNAPSHOT_FILE_CODE, GET_STATS_CODE, GET_STREAM_CODE, + GET_STREAMS_CODE, GET_TOPIC_CODE, GET_TOPICS_CODE, GET_USER_CODE, GET_USERS_CODE, + POLL_MESSAGES_CODE, }; use iggy_binary_protocol::primitives::partition_history::PartitionContext; use iggy_binary_protocol::requests::consumer_groups::{ @@ -102,10 +103,10 @@ use iggy_common::{ ClientInfo, ClientInfoDetails, ClusterMetadata, Consumer, ConsumerGroup, ConsumerGroupDetails, ConsumerOffsetInfo, DEFAULT_MAX_TOPIC_SIZE, Identifier, IdentityInfo, IggyError, IggyTimestamp, - OptionSpec, OptionStorage, OptionsScope, Partitioning, PersonalAccessTokenInfo, PollMessages, - PolledMessages, RawPersonalAccessToken, SendMessages, SendMessagesConfirmations, Stats, Stream, - StreamDetails, TokenInfo, Topic, TopicDetails, UserInfo, UserInfoDetails, Validatable, - descriptors_for, erase_buffer, + MetadataChanges, MetadataChangesQuery, OptionSpec, OptionStorage, OptionsScope, Partitioning, + PersonalAccessTokenInfo, PollMessages, PolledMessages, RawPersonalAccessToken, SendMessages, + SendMessagesConfirmations, Stats, Stream, StreamDetails, TokenInfo, Topic, TopicDetails, + UserInfo, UserInfoDetails, Validatable, descriptors_for, erase_buffer, }; use metadata::impls::metadata::StreamsFrontend; use metadata::permissioner::Permissioner; @@ -115,6 +116,7 @@ use serde::Deserialize; use shard::{PartitionRead, PartitionReadReply}; +use crate::dispatch::metadata_changes::metadata_changes_page; use crate::dispatch::partition::{ ConsumerOffsetTarget, fence_poll_context, resolve_consumer_offset_request, resolve_poll_request, resolve_primary_consumer_offset_request, @@ -777,6 +779,37 @@ Ok((headers, Body::from(archive))) } +/// `GET /metadata/changes`: one page of this node's metadata change log, or +/// of the current state. Local like `/snapshot`: committed records are the +/// same on every replica of one history, so the page is never redirected. +/// The query is strict: an unknown, repeated or malformed parameter is a 400. +pub(in crate::http) async fn get_metadata_changes( + State(state): State<HttpState>, + identity: Identity, + Query(pairs): Query<Vec<(String, String)>>, +) -> Result<Json<MetadataChanges>, ReadError> { + SendWrapper::new(gate_local_read( + &state, + &identity, + Consistency::Serializable, + GET_METADATA_CHANGES_CODE, + Permissioner::get_metadata_changes, + )) + .await?; + let query = MetadataChangesQuery::from_http_query( + pairs + .iter() + .map(|(key, value)| (key.as_str(), value.as_str())), + ) + .map_err(ReadError::Rejected)?; + let page = SendWrapper::new(metadata_changes_page(&state.shard, query, identity.user_id)) + .await + .map_err(ReadError::Rejected)?; + MetadataChanges::try_from(page) + .map(Json) + .map_err(ReadError::Rejected) +} + /// `GET /cluster/metadata`: report the live cluster topology as the same /// `ClusterMetadata` JSON the legacy server returns. ///
diff --git a/core/server/src/http/reads.rs b/core/server/src/http/reads.rs index 9a2de98..d8fdb6c 100644 --- a/core/server/src/http/reads.rs +++ b/core/server/src/http/reads.rs
@@ -609,7 +609,11 @@ WireName, WireOptions, }; use iggy_common::create_personal_access_token::CreatePersonalAccessToken; - use iggy_common::{IggyByteSize, IggyError, IggyTimestamp, UserStatus}; + use iggy_common::defaults::DEFAULT_ROOT_USER_ID as ROOT_USER_ID; + use iggy_common::{ + IggyByteSize, IggyError, IggyTimestamp, MetadataChangeCursor, MetadataChangePhase, + MetadataChangeStatus, UserStatus, + }; use message_bus::IggyMessageBus; use message_bus::installer::conn_info::ClientTransportKind; use message_bus::transports::tls::install_default_crypto_provider; @@ -618,6 +622,7 @@ use metadata::stm::StateHandler; use metadata::stm::authz::GatedApply; use metadata::stm::stream::StreamsInner; + use metadata::stm::user::ROOT_USER_CREATED_AT_MICROS; use metadata::stm::user::{CreatePersonalAccessTokenRequest, PAT_TOKEN_HASH_BYTES}; use partitions::{IggyPartitions, PartitionPathLayout, PartitionsConfig}; use send_wrapper::SendWrapper; @@ -643,7 +648,7 @@ use crate::http::error::{ConsistencyQuery, WriteError}; use crate::http::extractor::{Authenticated, Identity}; use crate::http::handlers::{ - create_pat, get_client, get_clients, get_metrics, get_stats, get_user, + create_pat, get_client, get_clients, get_metadata_changes, get_metrics, get_stats, get_user, }; use crate::http::metrics::HttpMetrics; use crate::http::relay_proof::RelayKey; @@ -757,6 +762,93 @@ } } + #[compio::test] + async fn given_metadata_changes_requests_over_http_should_refuse_bad_queries_and_answer_pages() + { + let state = read_state(); + commit_user_request( + &state, + READER_CREATED_AT, + Operation::CreateUser, + &create_reader_request().to_bytes(), + ); + let metadata = state.shard.plane.metadata(); + metadata + .change_log() + .append( + &PrepareHeader { + op: 1, + checksum: 0x5eed, + ..PrepareHeader::default() + }, + Vec::new(), + ) + .unwrap(); + metadata.advance_applied_frontier(1); + let query = |pairs: &[(&str, &str)]| { + Query( + pairs + .iter() + .map(|(key, value)| ((*key).to_owned(), (*value).to_owned())) + .collect::<Vec<_>>(), + ) + }; + + let reader = authenticated_identity(&state, READER_ID, READER_CREATED_AT).await; + let denied = get_metadata_changes(State(state.clone()), reader, query(&[])) + .await + .into_response(); + assert_eq!(denied.status(), StatusCode::FORBIDDEN); + + for refused in [ + &[("from", "latest"), ("from", "latest")][..], + &[("from", "earliest"), ("cursor", "mc1_")], + &[("from", "earliest"), ("since", "5"), ("until", "4")], + &[("from", "latest"), ("since", "1")], + &[("from", "current"), ("until", "1")], + &[("from", "current"), ("groups", "topics")], + &[("after", "mc1_not-a-cursor")], + &[("current_after", "cs1_AAAA")], + &[("from", "earliest"), ("limit", "0")], + &[("from", "earliest"), ("limit", "1001")], + &[("from", "earliest"), ("limit", "01")], + &[("from", "earliest"), ("limit", "+5")], + &[("from", "earliest"), ("groups", "streams,,users")], + &[("from", "earliest"), ("groups", "nope")], + ] { + let root = + authenticated_identity(&state, ROOT_USER_ID, ROOT_USER_CREATED_AT_MICROS).await; + let response = get_metadata_changes(State(state.clone()), root, query(refused)) + .await + .into_response(); + assert_eq!(response.status(), StatusCode::BAD_REQUEST, "{refused:?}"); + } + + let root = authenticated_identity(&state, ROOT_USER_ID, ROOT_USER_CREATED_AT_MICROS).await; + let Ok(Json(latest)) = + get_metadata_changes(State(state.clone()), root, query(&[("from", "latest")])).await + else { + panic!("root reads the feed"); + }; + assert_eq!(latest.status, MetadataChangeStatus::Ok); + assert_eq!(latest.log_id, Some(0x5eed)); + assert_eq!(latest.next, Some(MetadataChangeCursor::zero(0x5eed))); + + let root = authenticated_identity(&state, ROOT_USER_ID, ROOT_USER_CREATED_AT_MICROS).await; + let Ok(Json(users)) = get_metadata_changes( + State(state.clone()), + root, + query(&[("from", "current"), ("groups", "users")]), + ) + .await + else { + panic!("root reads the current state"); + }; + assert_eq!(users.phase, MetadataChangePhase::Current); + assert_eq!(users.items.len(), 2, "root and the reader"); + assert_eq!(users.current_page_token, None); + } + /// The client reads gather from every shard after their gate, and the /// account behind the token can change during the gather. #[compio::test]
diff --git a/core/server/src/partition_reconciler.rs b/core/server/src/partition_reconciler.rs index efe13f0..892cde1 100644 --- a/core/server/src/partition_reconciler.rs +++ b/core/server/src/partition_reconciler.rs
@@ -4867,8 +4867,13 @@ let stm = &shard.plane.metadata().mux_stm; seed_consumer_group_deletion_intent(stm, 5, 0, 0, 0); let now = iggy_common::IggyTimestamp::now(); - stm.streams() - .refresh_consumer_group_session(CLIENT_ID, NEW_SESSION, REGISTER_OP, now); + stm.streams().refresh_consumer_group_session( + CLIENT_ID, + NEW_SESSION, + REGISTER_OP, + now, + None, + ); assert_eq!( stm.streams().consumer_group_pending_revocations().len(), 1, @@ -5747,8 +5752,12 @@ assert_eq!(assigned(200).map(|p| p.len()), Some(1)); // Client 100 disconnects. - mux.streams() - .remove_consumer_group_member(100, iggy_common::IggyTimestamp::default(), 6); + mux.streams().remove_consumer_group_member( + 100, + iggy_common::IggyTimestamp::default(), + 6, + None, + ); assert_eq!(assigned(200), Some(vec![1])); seed_installed_owners(&mux);
diff --git a/core/server/src/responses.rs b/core/server/src/responses.rs index 7a8daca..fe2b1ce 100644 --- a/core/server/src/responses.rs +++ b/core/server/src/responses.rs
@@ -1220,6 +1220,7 @@ consumer_group_index: ahash::AHashMap::default(), next_consumer_group_id: 0, consumer_group_catalog_op: 0, + created_op: 0, }); streams.items.insert(stream);
diff --git a/core/server/src/shell.rs b/core/server/src/shell.rs index 26cc59b..46d0e7a 100644 --- a/core/server/src/shell.rs +++ b/core/server/src/shell.rs
@@ -31,7 +31,9 @@ use message_bus::{ConnectionInstaller, IggyMessageBus, MessageBus}; use metadata::IggyMetadata; use metadata::MuxStateMachine; +use metadata::change_log::ChangeLog; use metadata::impls::metadata::IggySnapshot; +use metadata::stm::LeftRightFactory; use metadata::stm::mux::WithFactory; use metadata::stm::stream::Streams; use metadata::stm::user::Users; @@ -44,10 +46,15 @@ pub(crate) type ServerMuxStateMachine = MuxStateMachine<variadic!(Users, Streams)>; /// Cross-thread bundle carrying one `ReadHandleFactory` per metadata -/// state. Shard 0 mints one after `recover()` and broadcasts a clone to -/// every peer shard; each peer rebuilds a reader-mode -/// [`ServerMuxStateMachine`] on its own runtime, skipping the WAL. -pub(crate) type ServerMetadataBundle = <variadic!(Users, Streams) as WithFactory>::Bundle; +/// state and one for the change log. Shard 0 mints one after `recover()` +/// and broadcasts a clone to every peer shard. Each peer rebuilds a +/// reader-mode [`ServerMuxStateMachine`] and change log on its own runtime, +/// skipping the WAL. +#[derive(Clone, Debug)] +pub(crate) struct ServerMetadataBundle { + pub(crate) states: <variadic!(Users, Streams) as WithFactory>::Bundle, + pub(crate) change_log: LeftRightFactory<ChangeLog>, +} pub(crate) type ServerMetadata = IggyMetadata< VsrConsensus<Rc<IggyMessageBus>>,
diff --git a/core/shard/src/lib.rs b/core/shard/src/lib.rs index d560f66..1c35bb5 100644 --- a/core/shard/src/lib.rs +++ b/core/shard/src/lib.rs
@@ -46,6 +46,7 @@ #[cfg(any(test, feature = "simulator"))] use crossfire::AsyncRxTrait; use futures::FutureExt; +use iggy_binary_protocol::responses::system::GetMetadataChangesResponse; use iggy_binary_protocol::{ CHECKSUM_UNSEALED, Command, CommitHeader, ConsensusHeader, ConsumerSession, ConsumerSessionHeartbeatHeader, DoViewChangeHeader, ForwardCompletionHeader, @@ -59,7 +60,9 @@ #[cfg(feature = "simulator")] use iggy_common::PartitionStats; use iggy_common::variadic; -use iggy_common::{ConsumerKind, EncryptorKind, IggyError, IggyExpiry, IggyTimestamp}; +use iggy_common::{ + ConsumerKind, EncryptorKind, IggyError, IggyExpiry, IggyTimestamp, MetadataChangesQuery, +}; use journal::superblock::{PingPongSuperblock, SuperblockStore}; use journal::{Journal, JournalHandle}; use message_bus::client_listener::RequestHandler; @@ -248,6 +251,16 @@ secret: iggy_binary_protocol::requests::users::login_register::BindSecret, reply: Sender<Result<(u32, u64, consensus::client_table::SessionAttachment), IggyError>>, }, + /// A home shard asks for one current state page of `metadata.changes`: + /// the client table, the token index and the only write side of the + /// state live on shard 0, which checks `user_id` again right before it + /// projects. + MetadataCurrentState { + /// Boxed so the inbox slot every frame pays for stays small. + query: Box<MetadataChangesQuery>, + user_id: u32, + reply: Sender<Result<GetMetadataChangesResponse, IggyError>>, + }, Register { vsr_client_id: u128, user_id: u32, @@ -16345,6 +16358,7 @@ use journal::prepare_journal::PrepareJournal; use message_bus::IggyMessageBus; use metadata::MuxStateMachine; + use metadata::change_log::ChangeLogCapacities; use metadata::impls::metadata::IggySnapshot; use metadata::impls::recovery::{ReplicaIdentity, recover}; use metadata::stm::stream::Streams; @@ -16798,7 +16812,7 @@ JOURNAL_SLOTS, CLIENTS_TABLE_MAX, |_| {}, - |_, _| {}, + ChangeLogCapacities::default(), ) .await .unwrap();
diff --git a/core/shard/src/poll/completion_tests.rs b/core/shard/src/poll/completion_tests.rs index cc0f5ee..cae659d 100644 --- a/core/shard/src/poll/completion_tests.rs +++ b/core/shard/src/poll/completion_tests.rs
@@ -294,13 +294,23 @@ table.end_session(CLIENT, USER, table.get_epoch(CLIENT).unwrap(), 100); } Change::Leave => { - streams.remove_consumer_group_member(CLIENT, IggyTimestamp::default(), 0); + streams.remove_consumer_group_member(CLIENT, IggyTimestamp::default(), 0, None); } Change::OtherLeave => { - streams.remove_consumer_group_member(OTHER_CLIENT, IggyTimestamp::default(), 0); + streams.remove_consumer_group_member( + OTHER_CLIENT, + IggyTimestamp::default(), + 0, + None, + ); } Change::MissingLeave => { - streams.remove_consumer_group_member(OTHER_CLIENT + 1, IggyTimestamp::default(), 0); + streams.remove_consumer_group_member( + OTHER_CLIENT + 1, + IggyTimestamp::default(), + 0, + None, + ); } Change::Rejoin => apply_poll_metadata( &owner, @@ -1357,7 +1367,7 @@ .unwrap(); } Change::Leave => { - streams.remove_consumer_group_member(PARENT, IggyTimestamp::default(), 0); + streams.remove_consumer_group_member(PARENT, IggyTimestamp::default(), 0, None); } Change::Delete => apply_poll_metadata( &owner,
diff --git a/core/simulator/src/bin/workload-fuzz.rs b/core/simulator/src/bin/workload-fuzz.rs index 59cf9ff..40afd42 100644 --- a/core/simulator/src/bin/workload-fuzz.rs +++ b/core/simulator/src/bin/workload-fuzz.rs
@@ -651,7 +651,8 @@ println!( "quiesced and converged (leader-relative; entity oracle: {entity_oracle}; \ evictions={}; ops_compared={} partition_ops_compared={} data_namespaces_compared={} \ - data_records_compared={} deleted_partition_ops_compared={} replicas_compared={} namespaces_checked={} \ + data_records_compared={} deleted_partition_ops_compared={} replicas_compared={} \ + change_records_compared={} projection_items_compared={} namespaces_checked={} \ owners_compared={} offsets_compared_with_shadow={} stale_owner_stores={} owner_fence_refusals={})", workload.evictions(), convergence.ops_compared, @@ -660,6 +661,8 @@ convergence.data_records_compared, workload.deleted_partition_ops_compared(), convergence.replicas_compared, + convergence.change_records_compared, + convergence.projection_items_compared, convergence.namespaces_checked, convergence.owners_compared, convergence.offsets_compared_with_shadow,
diff --git a/core/simulator/src/lib.rs b/core/simulator/src/lib.rs index f847104..1992214 100644 --- a/core/simulator/src/lib.rs +++ b/core/simulator/src/lib.rs
@@ -16172,6 +16172,14 @@ "no committed metadata op was witnessed on two replicas, so this seed \ would pass on a diverged cluster" ); + assert!( + report.change_records_compared > 0, + "no change record was compared between replicas" + ); + assert!( + report.projection_items_compared > 0, + "the leader's change records were never replayed against its current state" + ); } }
diff --git a/core/simulator/src/replica.rs b/core/simulator/src/replica.rs index f036fcb..628919a 100644 --- a/core/simulator/src/replica.rs +++ b/core/simulator/src/replica.rs
@@ -29,8 +29,10 @@ use iggy_common::{IggyByteSize, ServerKeyContext}; use journal::Journal; use message_bus::MessageBus; +use metadata::change_log::{ChangeLog, MetadataChangeLog}; use metadata::impls::metadata::IggySnapshot; use metadata::impls::metadata::StreamsFrontend; +use metadata::stm::LeftRightFactory; use metadata::stm::mux::WithFactory; use metadata::stm::snapshot::RestoreSnapshot; use metadata::stm::stream::{Streams, StreamsInner}; @@ -77,13 +79,17 @@ SimSuperblock, >; -/// Read-side handoff bundle for the metadata STM. +/// Read-side handoff bundle for the metadata STM and the change log. /// /// Shard 0 (the sole writer) mints one via `factory_bundle`; every peer shard /// rebuilds a reader-mode mirror from it with `from_factory_bundle`, exactly as /// the server bootstrap does with its `ServerMetadataBundle`. /// `Clone + Send + Sync`. -pub type SimMetadataBundle = <variadic!(Users, Streams) as WithFactory>::Bundle; +#[derive(Clone, Debug)] +pub struct SimMetadataBundle { + pub states: <variadic!(Users, Streams) as WithFactory>::Bundle, + pub change_log: LeftRightFactory<ChangeLog>, +} /// Capacity of each shard inbox in the simulator. /// @@ -246,7 +252,10 @@ .as_ref() .and_then(|journal| journal.last_op()); - let mux = reader_bundle.map_or_else( + let (reader_states, reader_change_log) = reader_bundle + .map(|bundle| (bundle.states, bundle.change_log)) + .unzip(); + let mux = reader_states.map_or_else( // Most workload tests begin with a root fixture. Bootstrap tests start // empty and run the server initializer through metadata replication. || { @@ -361,6 +370,9 @@ data_dir, ) .with_applied_frontier(applied_frontier); + if let Some(factory) = reader_change_log { + metadata = metadata.with_change_log(MetadataChangeLog::from_factory(factory)); + } if let Some(coordinator) = metadata.coordinator.as_mut() { coordinator.use_synchronous_io(); } @@ -381,6 +393,20 @@ "a freshly built plane holds no sessions, so this must install" ); } + let change_log = snapshot + .snapshot() + .change_log + .restore( + metadata + .change_log() + .capacities() + .expect("shard 0 owns the change log writer"), + ) + .expect("the local checkpoint's change log restores"); + metadata + .change_log() + .restore(change_log) + .expect("shard 0 owns the change log writer"); } // BEFORE the replay, matching a fresh boot: `CreateStream::apply` takes @@ -427,20 +453,28 @@ // `true`: the frontier is volatile, so a rebuilt plane starts at zero and // every op above the floor must mutate the table. A frontier only fences a // LIVE table a state transfer just replaced. - apply_committed_prepare( + let header = *entry.header(); + let committed = apply_committed_prepare( &*metadata.mux_stm, &metadata.client_table, true, |_| {}, entry, ); + metadata + .change_log() + .append(&header, committed.changes) + .expect("replay appends committed ops in order"); } } // Same seed the server bootstrap runs after its own replay. metadata.seed_applied_frontier_from_consensus(); // Mint the peers' read-side bundle AFTER reconstruction so it reflects the // recovered state. Shard 0 only; peers pass it back in as `reader_bundle`. - let metadata_bundle = (shard_idx == 0).then(|| metadata.mux_stm.factory_bundle()); + let metadata_bundle = (shard_idx == 0).then(|| SimMetadataBundle { + states: metadata.mux_stm.factory_bundle(), + change_log: metadata.change_log().factory(), + }); let partitions_config = PartitionsConfig { messages_required_to_save: 1000,
diff --git a/core/simulator/src/workload/change_log_oracle.rs b/core/simulator/src/workload/change_log_oracle.rs new file mode 100644 index 0000000..9481472 --- /dev/null +++ b/core/simulator/src/workload/change_log_oracle.rs
@@ -0,0 +1,434 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use crate::Simulator; +use consensus::MetadataHandle; +use iggy_binary_protocol::primitives::metadata_changes::{ + WireConsumerGroupItemBody, WireMemberItemBody, WireMetadataChange, WireMetadataChangeStatus, + WireMetadataCurrentItem, WirePartitionsItemBody, WireSessionBody, WireStreamBody, + WireTokenBody, WireTopicItemBody, WireTopicKey, WireUserBody, +}; +use iggy_common::{ + MetadataChangeClass, MetadataChangeStart, MetadataChangesQuery, MetadataCurrentPageToken, +}; +use metadata::change_log::{ChangeEntrySnapshot, ChangeLog, ChangeLogSnapshot}; +use metadata::stm::user::ROOT_USER_CREATED_AT_MICROS; +use std::collections::{BTreeMap, BTreeSet, HashMap}; + +/// Per tick, on one live replica: no change record above the applied +/// frontier, and the newest record op never moves back within one +/// incarnation. +pub(crate) fn check_change_log_head( + sim: &Simulator, + replica_idx: u8, + heads: &mut HashMap<u8, (u128, u64)>, + seed: u64, +) { + let replica = &sim.replicas[usize::from(replica_idx)]; + let metadata = replica.shards[0].plane.metadata(); + let Some(last_record_op) = metadata.change_log().read(ChangeLog::last_record_op) else { + return; + }; + let frontier = metadata.applied_frontier().get(); + assert!( + last_record_op <= frontier, + "replica {replica_idx} holds a change record at op {last_record_op} above its applied \ + frontier {frontier} (seed={seed:#x})" + ); + let incarnation = replica.metadata_incarnation; + if let Some((previous_incarnation, previous)) = + heads.insert(replica_idx, (incarnation, last_record_op)) + && previous_incarnation == incarnation + { + assert!( + last_record_op >= previous, + "replica {replica_idx} change log moved back from op {previous} to {last_record_op} \ + (seed={seed:#x})" + ); + } +} + +/// At quiesce: live replicas hold byte-identical records for every operation +/// they all applied and all still retain. Returns the records compared. +pub(crate) fn assert_change_logs_agree(sim: &Simulator, live: &[usize], seed: u64) -> usize { + let logs: Vec<(usize, u64, ChangeLogSnapshot)> = live + .iter() + .filter_map(|&replica_idx| { + let metadata = sim.replicas[replica_idx].shards[0].plane.metadata(); + let applied = metadata.consensus.as_ref()?.commit_min(); + Some((replica_idx, applied, metadata.change_log().snapshot())) + }) + .collect(); + let Some((first_idx, first_applied, first)) = logs.first() else { + return 0; + }; + let mut compared = 0; + for (replica_idx, applied, log) in &logs[1..] { + if first.log_id.is_some() && log.log_id.is_some() { + assert_eq!( + first.log_id, log.log_id, + "replicas {first_idx} and {replica_idx} report different change log ids \ + (seed={seed:#x})" + ); + } + let common_applied = (*first_applied).min(*applied); + for (class, left, right) in [ + ( + MetadataChangeClass::Resource, + &first.resource, + &log.resource, + ), + ( + MetadataChangeClass::Activity, + &first.activity, + &log.activity, + ), + ] { + let floor = (left.evicted_through_op, left.evicted_through_ordinal) + .max((right.evicted_through_op, right.evicted_through_ordinal)); + let window = |entries: &[ChangeEntrySnapshot]| -> Vec<ChangeEntrySnapshot> { + entries + .iter() + .filter(|entry| entry.op <= common_applied && (entry.op, entry.ordinal) > floor) + .cloned() + .collect() + }; + let (left, right) = (window(&left.entries), window(&right.entries)); + assert_eq!( + left, right, + "replicas {first_idx} and {replica_idx} hold different {class:?} change records \ + through op {common_applied} (seed={seed:#x})" + ); + compared += left.len(); + } + } + compared +} + +/// At quiesce: a mirror built from the leader's records alone equals the +/// leader's current state pages, entities the harness seeded without a +/// committed operation aside. Returns the items compared, zero when the log +/// already evicted history the mirror needs. +pub(crate) fn assert_change_log_projection(sim: &Simulator, leader: usize, seed: u64) -> usize { + let metadata = sim.replicas[leader].shards[0].plane.metadata(); + let log = metadata.change_log().snapshot(); + if ( + log.resource.evicted_through_op, + log.activity.evicted_through_op, + ) != (0, 0) + { + return 0; + } + let mut mirror = Mirror::default(); + let mut entries: Vec<&ChangeEntrySnapshot> = log + .resource + .entries + .iter() + .chain(&log.activity.entries) + .collect(); + entries.sort_by_key(|entry| (entry.op, entry.ordinal)); + for entry in entries { + let Ok(Some(change)) = WireMetadataChange::decode(entry.group, entry.kind, &entry.body) + else { + panic!( + "leader holds an undecodable change record at op {} (seed={seed:#x})", + entry.op + ); + }; + mirror.apply(change); + } + + let mut query = MetadataChangesQuery { + start: MetadataChangeStart::Current(None), + limit: 1000, + ..MetadataChangesQuery::default() + }; + let mut current = Vec::new(); + loop { + let Ok(page) = metadata.metadata_current_state(&query) else { + return 0; + }; + if page.status != WireMetadataChangeStatus::Ok { + return 0; + } + current.extend(page.items); + match page.current_page_token { + Some(token) => { + query.start = + MetadataChangeStart::Current(Some(MetadataCurrentPageToken::from(token))); + } + None => break, + } + } + let projected: BTreeSet<Vec<u8>> = current + .iter() + .filter_map(|item| { + let Ok(Some(decoded)) = WireMetadataCurrentItem::decode(item.group, &item.body) else { + panic!("leader served an undecodable current item (seed={seed:#x})"); + }; + (!seeded_by_harness(&decoded)).then(|| encoded(&decoded)) + }) + .collect(); + let mirrored: BTreeSet<Vec<u8>> = mirror + .items() + .iter() + .filter(|item| !seeded_by_harness(item)) + .map(encoded) + .collect(); + assert_eq!( + mirrored, projected, + "a mirror of the leader's change records differs from its current state \ + (leader={leader}, seed={seed:#x})" + ); + projected.len() +} + +/// The model a client builds from change records alone. +#[derive(Debug, Default)] +struct Mirror { + streams: BTreeMap<u32, WireStreamBody>, + topics: BTreeMap<(u32, u32), (WireTopicItemBody, u32)>, + groups: BTreeMap<(u32, u32, u32), WireConsumerGroupItemBody>, + members: BTreeMap<(u32, u32, u32, u32), WireMemberItemBody>, + users: BTreeMap<u32, WireUserBody>, + tokens: BTreeMap<(u32, String), WireTokenBody>, + sessions: BTreeMap<u128, WireSessionBody>, +} + +impl Mirror { + #[allow(clippy::too_many_lines)] + fn apply(&mut self, change: WireMetadataChange) { + match change { + WireMetadataChange::StreamCreated(body) | WireMetadataChange::StreamUpdated(body) => { + self.streams.insert(body.key.stream_id, body); + } + WireMetadataChange::StreamDeleted(deleted) => { + let stream_id = deleted.key.stream_id; + self.streams.remove(&stream_id); + self.topics.retain(|key, _| key.0 != stream_id); + self.groups.retain(|key, _| key.0 != stream_id); + self.members.retain(|key, _| key.0 != stream_id); + } + WireMetadataChange::TopicCreated(body) | WireMetadataChange::TopicUpdated(body) => { + self.topics.insert( + (body.key.stream.stream_id, body.key.topic_id), + ( + WireTopicItemBody { + key: body.key, + created_at: body.created_at, + name: body.name, + options: body.options, + }, + body.partitions_count, + ), + ); + } + WireMetadataChange::TopicDeleted(deleted) => { + let topic = (deleted.key.stream.stream_id, deleted.key.topic_id); + self.topics.remove(&topic); + self.groups.retain(|key, _| (key.0, key.1) != topic); + self.members.retain(|key, _| (key.0, key.1) != topic); + } + WireMetadataChange::PartitionsChanged(changed) => { + let topic = (changed.topic.stream.stream_id, changed.topic.topic_id); + if let Some((_, partitions_count)) = self.topics.get_mut(&topic) { + *partitions_count = changed.partitions_count; + } + for (key, member) in &mut self.members { + if (key.0, key.1) == topic { + member.partition_ids.retain(|partition_id| { + !changed.removed_partition_ids.contains(partition_id) + }); + } + } + } + WireMetadataChange::ConsumerGroupCreated(created) => { + self.groups.insert( + ( + created.topic.stream.stream_id, + created.topic.topic_id, + created.group_id, + ), + WireConsumerGroupItemBody { + topic: created.topic, + group_id: created.group_id, + name: created.name, + }, + ); + } + WireMetadataChange::ConsumerGroupDeleted(deleted) => { + let group = ( + deleted.topic.stream.stream_id, + deleted.topic.topic_id, + deleted.group_id, + ); + self.groups.remove(&group); + self.members.retain(|key, _| (key.0, key.1, key.2) != group); + } + WireMetadataChange::ConsumerGroupMemberJoined(joined) => { + self.members.insert( + ( + joined.topic.stream.stream_id, + joined.topic.topic_id, + joined.group_id, + joined.member_id, + ), + WireMemberItemBody { + topic: joined.topic, + group_id: joined.group_id, + member_id: joined.member_id, + client_id: joined.client_id, + partition_ids: joined.partition_ids, + }, + ); + } + WireMetadataChange::ConsumerGroupMemberLeft(left) => { + self.members.remove(&( + left.topic.stream.stream_id, + left.topic.topic_id, + left.group_id, + left.member_id, + )); + } + WireMetadataChange::ConsumerGroupPartitionHandedOver(handed) => { + let group = ( + handed.topic.stream.stream_id, + handed.topic.topic_id, + handed.group_id, + ); + for (key, member) in &mut self.members { + if (key.0, key.1, key.2) != group { + continue; + } + member + .partition_ids + .retain(|partition_id| *partition_id != handed.partition_id); + if Some(key.3) == handed.member_id { + member.partition_ids.push(handed.partition_id); + member.partition_ids.sort_unstable(); + } + } + } + WireMetadataChange::UserCreated(body) | WireMetadataChange::UserUpdated(body) => { + self.users.insert(body.key.user_id, body); + } + WireMetadataChange::UserDeleted(deleted) => { + let user_id = deleted.key.user_id; + self.users.remove(&user_id); + self.tokens.retain(|key, _| key.0 != user_id); + } + WireMetadataChange::UserPasswordChanged(_) => {} + WireMetadataChange::UserPermissionsUpdated(updated) => { + if let Some(user) = self.users.get_mut(&updated.key.user_id) { + user.permissions = updated.permissions; + } + } + WireMetadataChange::PersonalAccessTokenCreated(token) => { + self.tokens + .insert((token.user.user_id, token.name.as_str().to_owned()), token); + } + WireMetadataChange::PersonalAccessTokenDeleted(deleted) => { + self.tokens + .remove(&(deleted.user.user_id, deleted.name.as_str().to_owned())); + } + WireMetadataChange::SessionRegistered(session) => { + self.sessions.insert(session.client_id, session); + } + WireMetadataChange::SessionClosed(closed) => { + self.sessions.remove(&closed.client_id); + } + } + } + + fn items(&self) -> Vec<WireMetadataCurrentItem> { + let streams = self + .streams + .values() + .cloned() + .map(WireMetadataCurrentItem::Stream); + let topics = self + .topics + .values() + .map(|(topic, _)| WireMetadataCurrentItem::Topic(topic.clone())); + let partitions = self.topics.values().map(|(topic, partitions_count)| { + WireMetadataCurrentItem::Partitions(WirePartitionsItemBody { + topic: topic.key, + partitions_count: *partitions_count, + }) + }); + let groups = self + .groups + .values() + .cloned() + .map(WireMetadataCurrentItem::ConsumerGroup); + let members = self + .members + .values() + .cloned() + .map(WireMetadataCurrentItem::ConsumerGroupMember); + let users = self + .users + .values() + .cloned() + .map(WireMetadataCurrentItem::User); + let tokens = self + .tokens + .values() + .cloned() + .map(WireMetadataCurrentItem::PersonalAccessToken); + let sessions = self + .sessions + .values() + .cloned() + .map(WireMetadataCurrentItem::Session); + streams + .chain(topics) + .chain(partitions) + .chain(groups) + .chain(members) + .chain(users) + .chain(tokens) + .chain(sessions) + .collect() + } +} + +/// Whether the harness seeded this entity straight into the state machines, +/// without the committed operation a record would come from. +fn seeded_by_harness(item: &WireMetadataCurrentItem) -> bool { + let topic_seeded = + |topic: &WireTopicKey| topic.stream.stream_created_op == 0 || topic.topic_created_op == 0; + match item { + WireMetadataCurrentItem::Stream(stream) => stream.key.stream_created_op == 0, + WireMetadataCurrentItem::Topic(topic) => topic_seeded(&topic.key), + WireMetadataCurrentItem::Partitions(partitions) => topic_seeded(&partitions.topic), + WireMetadataCurrentItem::ConsumerGroup(group) => topic_seeded(&group.topic), + WireMetadataCurrentItem::ConsumerGroupMember(member) => topic_seeded(&member.topic), + WireMetadataCurrentItem::User(user) => { + user.key.user_id == 0 && user.key.user_created_at == ROOT_USER_CREATED_AT_MICROS + } + WireMetadataCurrentItem::PersonalAccessToken(_) | WireMetadataCurrentItem::Session(_) => { + false + } + } +} + +fn encoded(item: &WireMetadataCurrentItem) -> Vec<u8> { + let mut bytes = vec![item.group().as_u8()]; + bytes.extend_from_slice(&item.body_bytes()); + bytes +}
diff --git a/core/simulator/src/workload/invariants.rs b/core/simulator/src/workload/invariants.rs index 3b7f304..129dcbd 100644 --- a/core/simulator/src/workload/invariants.rs +++ b/core/simulator/src/workload/invariants.rs
@@ -23,6 +23,7 @@ //! trace and the determinism baseline (`workload_replay_is_deterministic`) //! unchanged. +use crate::workload::change_log_oracle::check_change_log_head; use crate::workload::state_checker::StateChecker; use crate::workload::{CLIENT_REQUEST_QUEUE_MAX, Workload}; use crate::{CommitHoldKind, CommitPrefixHole, Simulator}; @@ -83,6 +84,9 @@ /// Cross-replica committed-log agreement. Runs every tick like the rest, so a /// divergence is reported where it appears rather than at the next quiesce. state_checker: StateChecker, + /// Metadata incarnation and newest change record op each replica held at + /// the last check. + change_log_heads: HashMap<u8, (u128, u64)>, } impl Invariants { @@ -133,6 +137,7 @@ } self.check_recovery_barrier(sim, seed, replica_idx); check_applied_log_contradictions(sim, seed, replica_idx); + check_change_log_head(sim, replica_idx, &mut self.change_log_heads, seed); let incarnation = sim.replicas[usize::from(replica_idx)].metadata_incarnation; if self .replica_incarnation
diff --git a/core/simulator/src/workload/mod.rs b/core/simulator/src/workload/mod.rs index 58c2a78..ad993ec 100644 --- a/core/simulator/src/workload/mod.rs +++ b/core/simulator/src/workload/mod.rs
@@ -26,6 +26,7 @@ pub mod actions; pub mod auditor; +pub mod change_log_oracle; pub mod effect; pub mod ids; pub mod invariants;
diff --git a/core/simulator/src/workload/oracle.rs b/core/simulator/src/workload/oracle.rs index 1ef4242..fc68a7f 100644 --- a/core/simulator/src/workload/oracle.rs +++ b/core/simulator/src/workload/oracle.rs
@@ -41,6 +41,7 @@ use crate::client::SimClient; use crate::replica::Replica; +use crate::workload::change_log_oracle::{assert_change_log_projection, assert_change_logs_agree}; use crate::workload::invariants::Invariants; use crate::workload::shadow::Shadow; use crate::workload::{ @@ -521,6 +522,7 @@ /// If a replica is ahead of the leader, two replicas disagree on a committed op or /// on the message data it applied, or the shadow mismatches the leader. The /// workload seed is in the message so the failing run replays deterministically. +#[allow(clippy::too_many_lines)] pub fn assert_converged(sim: &Simulator, workload: &mut Workload) -> ConvergenceReport { let seed = workload.options.seed; let live: Vec<usize> = (0..sim.replica_count) @@ -604,6 +606,8 @@ workload.readback.assert_served_committed(sim, seed); let replicas_compared = assert_committed_metadata_agrees(sim, &live, seed); + let change_records_compared = assert_change_logs_agree(sim, &live, seed); + let projection_items_compared = assert_change_log_projection(sim, leader, seed); let owners_compared = assert_settled_owners_installed(sim, workload, leader, seed); let retirements_compared = assert_retired_owners_removed(sim, workload, leader, &live, seed); @@ -616,6 +620,8 @@ data_namespaces_compared: data.namespaces, data_records_compared: data.records, replicas_compared, + change_records_compared, + projection_items_compared, namespaces_checked, owners_compared, offsets_compared_with_shadow, @@ -683,6 +689,12 @@ /// Live replicas whose committed metadata CONTENT was compared against a peer /// sharing its commit point. Zero on a solo cluster. pub replicas_compared: usize, + /// Change records compared between live replicas for the operations they + /// all applied and retain. + pub change_records_compared: usize, + /// Current state items of the leader a mirror of its change records had to + /// reproduce. Zero once the log evicted history the mirror needs. + pub projection_items_compared: usize, /// Namespaces still present in committed metadata that were required to have a /// host and exactly one primary. pub namespaces_checked: usize,
diff --git a/examples/node/package-lock.json b/examples/node/package-lock.json index b6ac323..ae09d22 100644 --- a/examples/node/package-lock.json +++ b/examples/node/package-lock.json
@@ -23,7 +23,7 @@ }, "../../foreign/node": { "name": "apache-iggy", - "version": "0.10.1-edge.2", + "version": "0.10.1-edge.3", "license": "Apache-2.0", "dependencies": { "@node-rs/xxhash": "1.7.8", @@ -36,7 +36,7 @@ "@cucumber/cucumber": "13.3.0", "@swc-node/register": "1.12.1", "@types/debug": "4.1.13", - "@types/node": "26.6.4", + "@types/node": "26.6.5", "c8": "^12.0.0", "husky": "9.1.7", "typescript": "6.0.3",
diff --git a/examples/python/uv.lock b/examples/python/uv.lock index 0b63d77..a986d97 100644 --- a/examples/python/uv.lock +++ b/examples/python/uv.lock
@@ -8,7 +8,7 @@ [[package]] name = "apache-iggy" -version = "0.9.1.dev2" +version = "0.9.1.dev3" source = { directory = "../../foreign/python" } [package.metadata]
diff --git a/foreign/cpp/BUILD.bazel b/foreign/cpp/BUILD.bazel index b1ca2a1..ec6ce33 100644 --- a/foreign/cpp/BUILD.bazel +++ b/foreign/cpp/BUILD.bazel
@@ -195,6 +195,7 @@ srcs = [ "tests/bdd/background_steps.cpp", "tests/bdd/messaging_steps.cpp", + "tests/bdd/metadata_changes_steps.cpp", "tests/bdd/raw_command_steps.cpp", "tests/bdd/resource_options_steps.cpp", "tests/bdd/session_contract_steps.cpp", @@ -241,6 +242,7 @@ ], }), data = [ + "tests/fixtures/metadata_changes.json", "tests/fixtures/resource_options.json", ], deps = [ @@ -275,6 +277,8 @@ tags = [ "e2e", ], + # The suite already takes about four of the default five minutes. + timeout = "long", deps = [ ":iggy-cpp", "@googletest//:gtest_main",
diff --git a/foreign/cpp/Cargo.toml b/foreign/cpp/Cargo.toml index c213672..71dc6b3 100644 --- a/foreign/cpp/Cargo.toml +++ b/foreign/cpp/Cargo.toml
@@ -17,7 +17,7 @@ [package] name = "iggy-cpp" -version = "0.1.4" +version = "0.1.5" edition = "2024" license = "Apache-2.0"
diff --git a/foreign/cpp/MODULE.bazel b/foreign/cpp/MODULE.bazel index 20faa4d..8830498 100644 --- a/foreign/cpp/MODULE.bazel +++ b/foreign/cpp/MODULE.bazel
@@ -17,7 +17,7 @@ module( name = "iggy_cpp", - version = "0.1.4", + version = "0.1.5", ) bazel_dep(name = "rules_cc", version = "0.2.26")
diff --git a/foreign/cpp/README.md b/foreign/cpp/README.md index 5c89d98..88cb30a 100644 --- a/foreign/cpp/README.md +++ b/foreign/cpp/README.md
@@ -25,7 +25,7 @@ bazel test //:e2e ``` -The E2E suite expects a local server on the default ports with encryption enabled, because the resource options cases create streams with their own keys and expect a server-generated key for keyless streams. Start it with `IGGY_ENCRYPTION_ENABLED=true` and `IGGY_ENCRYPTION_KEY` set to a base64 encoded 32-byte key, for example `IGGY_ENCRYPTION_ENABLED=true IGGY_ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64) cargo run --bin iggy-server -- --with-default-root-credentials`. +The E2E suite expects a local server on the default ports with encryption enabled, because the resource options cases and the keyed metadata change cases create streams with their own keys and expect a server-generated key for keyless streams. Start it with `IGGY_ENCRYPTION_ENABLED=true` and `IGGY_ENCRYPTION_KEY` set to a base64 encoded 32-byte key, for example `IGGY_ENCRYPTION_ENABLED=true IGGY_ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64) cargo run --bin iggy-server -- --with-default-root-credentials`. Receipt access follows the server's [session and revocation policy](../../core/server/README.md#sessions-and-retry-receipts). An unresolved operation cannot recover its receipt after its original session @@ -106,22 +106,15 @@ - `CreatePartitions`, `DeletePartitions`, and `DeleteSegments`. - `CreateConsumerGroup`, `DeleteConsumerGroup`, `JoinConsumerGroup`, and `LeaveConsumerGroup`. - `StoreConsumerOffset`, `StoreConsumerPosition`, and `DeleteConsumerOffset`. +- `GetMetadataChanges`. -Other methods, including reads and `Connect`, `Disconnect`, and `Shutdown`, report -errors through the exception message only. C++ and bridge-local argument errors -also leave `Code()` and `OperationId()` empty and `Kind()` empty, including for the -methods listed above. +Other methods, including reads and `Connect`, `Disconnect`, and `Shutdown`, report errors through the exception message only. C++ and bridge-local argument errors also leave `Code()` and `OperationId()` empty and `Kind()` empty, including for the methods listed above. `GetMetadataChanges` is the exception: it reports every invalid query as `invalid_command`, including one the bridge refuses. -Direct generated-bridge callers pass a local `iggy::ffi::ErrorInfo` as the last -argument to login/logout, mutations, polls, raw commands, and recovery calls. -It is reset at entry and populated only when the Rust SDK returns an `IggyError`. -Bridge-local argument errors leave it empty. Inspect `kind`, `code`, and, when -`has_operation_id` is true, `operation_id`. Use separate values for concurrent -calls; error state is never stored globally or on the client. +Direct generated-bridge callers pass a local `iggy::ffi::ErrorInfo` as the last argument to login/logout, mutations, polls, raw commands, recovery calls, and the metadata change functions. It is reset at entry and populated only when the Rust SDK returns an `IggyError`. Bridge-local argument errors leave it empty, except in the metadata change functions, which record invalid input as `invalid_command`. -The `IggyException(const std::string &, const ffi::ErrorInfo &)` constructor is -private. Only `IggyBlockingClient` builds an exception with a code. Other code -builds an `IggyException` from its message. +Inspect `kind`, `code`, and, when `has_operation_id` is true, `operation_id`. Use separate values for concurrent calls. Error state is never stored globally or on the client. + +The `IggyException(const std::string &, const ffi::ErrorInfo &)` constructor is private. Only the SDK builds an exception with a code. Other code builds an `IggyException` from its message. Concurrent calls on one client are safe. Moving, assigning, or destroying the client while any call is in progress is undefined behavior. @@ -138,3 +131,70 @@ `max_pending_operation_bytes` query parameters for TCP, QUIC, and WebSocket. HTTP continues to use its configured transport behavior and has no retained operation recovery. + +## Metadata changes + +`GetMetadataChanges()` reads the node's change log: one record per public result of a committed metadata operation, such as a stream created, a topic renamed, a user's permissions changed or a session closed. A mirror keeps a cursor, asks for the records after it, and stores the returned `Next()` cursor. The call needs root or the global `manage_servers` permission and works on every transport. + +```cpp +// Pages the current state into a private staged copy, then returns the query that follows the changes after it. +iggy::MetadataChangesQuery Rebuild(iggy::IggyBlockingClient &client, const std::vector<iggy::MetadataChangeGroup> &groups) { + ClearStaged(); + auto query = iggy::MetadataChangesQuery::Current().SetGroups(groups); + while (true) { + const auto page = client.GetMetadataChanges(query); + switch (page.Status()) { + case iggy::MetadataChangeStatus::Ok: + break; + case iggy::MetadataChangeStatus::Behind: + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + continue; + case iggy::MetadataChangeStatus::Expired: + case iggy::MetadataChangeStatus::Reset: + ClearStaged(); + query = iggy::MetadataChangesQuery::Current().SetGroups(groups); + continue; + } + for (const auto &item : page.Items()) { + StageItem(item); + } + if (!page.CurrentPageToken()) { + return iggy::MetadataChangesQuery::After(*page.Next()).SetGroups(groups); + } + query = iggy::MetadataChangesQuery::Current(*page.CurrentPageToken()).SetGroups(groups); + } +} + +void Follow(iggy::IggyBlockingClient &client, const std::vector<iggy::MetadataChangeGroup> &groups) { + auto query = Rebuild(client, groups); + while (true) { + const auto page = client.GetMetadataChanges(query); + switch (page.Status()) { + case iggy::MetadataChangeStatus::Ok: + for (const auto &record : page.Records()) { + if (const auto *stream = std::get_if<iggy::MetadataStreamChange>(&record.Change())) { + ApplyStream(*record.Kind(), *stream); + } + } + query = iggy::MetadataChangesQuery::After(*page.Next()).SetGroups(groups); + if (!page.HasMore()) { + std::this_thread::sleep_for(std::chrono::seconds(1)); + } + break; + case iggy::MetadataChangeStatus::Behind: + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + break; + case iggy::MetadataChangeStatus::Expired: + case iggy::MetadataChangeStatus::Reset: + query = Rebuild(client, groups); + break; + } + } +} +``` + +`Expired` means the log no longer holds the history after the cursor, and `Reset` means the cursor belongs to another history, for example after the data directory was rebuilt. Either way the mirror rebuilds from current state pages, as it does on start, and then follows the changes after the anchor they return. Each scan begins from an empty staged state, so nothing an abandoned scan staged survives. Persist a cursor with `MetadataChangeCursor::ToString()` and restore it with `MetadataChangeCursor::Parse()`. + +Current pages need the parent groups of every selected group, for example `topics` with `streams`. Keep a rebuilt state private until a change page answers `Ok` with `HasMore()` false and a `HeadOp()` at or above the last current page's. An invalid query is refused with `invalid_command` before anything is sent. + +A record or item this build does not know arrives as `MetadataUnknownChange` or `MetadataUnknownItem` with its codes and raw body on TCP, QUIC and WebSocket, and as `MetadataUnknownJsonChange` or `MetadataUnknownJsonItem` with its names and whole JSON object on HTTP. Records carry option values as the read routes return them, never key material.
diff --git a/foreign/cpp/include/iggy.hpp b/foreign/cpp/include/iggy.hpp index a8f0d11..5aea1ca 100644 --- a/foreign/cpp/include/iggy.hpp +++ b/foreign/cpp/include/iggy.hpp
@@ -78,6 +78,12 @@ class ConsumerGroupMember; class IggyMessagePolled; class IggyMessageToSend; +class MetadataChangeCursor; +class MetadataChangeRecord; +class MetadataChanges; +class MetadataChangesQuery; +class MetadataCurrentItem; +class MetadataCurrentPageToken; namespace detail { /** @brief Internal base for string-backed option types. */ @@ -97,6 +103,10 @@ std::string value_; }; +/** @brief Runs a bridge call and rethrows its failure as IggyException with the call's ErrorInfo. */ +template <typename Operation> +decltype(auto) RethrowAsIggyException(Operation &&operation); + } // namespace detail /** @@ -104,7 +114,9 @@ * * Code() and Kind() are available only when a call forwards a typed Rust SDK * error. IggyBlockingClient documents the covered methods. Other calls and - * C++ or bridge-local argument errors leave Code() empty and Kind() empty. + * C++ or bridge-local argument errors leave Code() empty and Kind() empty, + * except the metadata change calls, which report invalid input with the kind + * each one documents. * OperationId() is present only for operation_uncertain, lane_blocked, and * operation_not_retained errors and preserves the full unsigned 64-bit ID. */ @@ -131,9 +143,8 @@ } } - friend class IggyBlockingClient; - friend std::string EncryptionKeyFingerprint(const std::vector<std::uint8_t> &key); - friend std::string EncryptionKeyFingerprint(const std::string &key_base64); + template <typename Operation> + friend decltype(auto) detail::RethrowAsIggyException(Operation &&operation); std::optional<std::uint32_t> code_; // Shared, so that copying the exception cannot throw. @@ -141,6 +152,26 @@ std::optional<std::uint64_t> operation_id_; }; +namespace detail { + +template <typename Operation> +decltype(auto) RethrowAsIggyException(Operation &&operation) { + ffi::ErrorInfo error_info{}; + try { + if constexpr (std::is_invocable_v<Operation, ffi::ErrorInfo &>) { + return std::forward<Operation>(operation)(error_info); + } else { + return std::forward<Operation>(operation)(); + } + } catch (const IggyException &) { + throw; + } catch (const std::exception &error) { + throw IggyException(error.what(), error_info); + } +} + +} // namespace detail + /** * @brief Details returned after a successful login. * @@ -728,6 +759,8 @@ friend class IggyBlockingClient; friend class UserInfoDetails; + friend class MetadataUserChange; + friend class MetadataUserPermissionsUpdated; GlobalPermissions global_; std::map<std::uint32_t, StreamPermissions> streams_; @@ -1147,6 +1180,10 @@ friend class StreamDetails; friend class UserInfo; friend class UserInfoDetails; + friend class MetadataStreamChange; + friend class MetadataTopicChange; + friend class MetadataTopicItem; + friend class MetadataUserChange; std::map<std::string, HeaderField> explicit_; std::map<std::string, HeaderField> derived_; @@ -3970,6 +4007,1621 @@ std::string EncryptionKeyFingerprint(const std::string &key_base64); /** + * @brief Family of metadata change records and the unit of filtering. + * + * The values are the wire codes. A query without groups selects every group. + */ +enum class MetadataChangeGroup : std::uint8_t { + Streams = 1, + Topics = 2, + Partitions = 3, + ConsumerGroups = 4, + ConsumerGroupMembers = 5, + Users = 6, + PersonalAccessTokens = 7, + Sessions = 8, +}; + +constexpr std::string_view to_string(const MetadataChangeGroup group) { + switch (group) { + case MetadataChangeGroup::Streams: + return "streams"; + case MetadataChangeGroup::Topics: + return "topics"; + case MetadataChangeGroup::Partitions: + return "partitions"; + case MetadataChangeGroup::ConsumerGroups: + return "consumer_groups"; + case MetadataChangeGroup::ConsumerGroupMembers: + return "consumer_group_members"; + case MetadataChangeGroup::Users: + return "users"; + case MetadataChangeGroup::PersonalAccessTokens: + return "personal_access_tokens"; + case MetadataChangeGroup::Sessions: + return "sessions"; + } + throw std::invalid_argument("Unknown metadata change group"); +} + +/** + * @brief Kind of a metadata change record. The values are the wire codes. + */ +enum class MetadataChangeKind : std::uint8_t { + StreamCreated = 1, + StreamUpdated = 2, + StreamDeleted = 3, + TopicCreated = 10, + TopicUpdated = 11, + TopicDeleted = 12, + PartitionsChanged = 20, + ConsumerGroupCreated = 30, + ConsumerGroupDeleted = 31, + ConsumerGroupMemberJoined = 40, + ConsumerGroupMemberLeft = 41, + ConsumerGroupPartitionHandedOver = 42, + UserCreated = 50, + UserUpdated = 51, + UserDeleted = 52, + UserPasswordChanged = 53, + UserPermissionsUpdated = 54, + PersonalAccessTokenCreated = 60, + PersonalAccessTokenDeleted = 61, + SessionRegistered = 70, + SessionClosed = 71, +}; + +constexpr std::string_view to_string(const MetadataChangeKind kind) { + switch (kind) { + case MetadataChangeKind::StreamCreated: + return "stream_created"; + case MetadataChangeKind::StreamUpdated: + return "stream_updated"; + case MetadataChangeKind::StreamDeleted: + return "stream_deleted"; + case MetadataChangeKind::TopicCreated: + return "topic_created"; + case MetadataChangeKind::TopicUpdated: + return "topic_updated"; + case MetadataChangeKind::TopicDeleted: + return "topic_deleted"; + case MetadataChangeKind::PartitionsChanged: + return "partitions_changed"; + case MetadataChangeKind::ConsumerGroupCreated: + return "consumer_group_created"; + case MetadataChangeKind::ConsumerGroupDeleted: + return "consumer_group_deleted"; + case MetadataChangeKind::ConsumerGroupMemberJoined: + return "consumer_group_member_joined"; + case MetadataChangeKind::ConsumerGroupMemberLeft: + return "consumer_group_member_left"; + case MetadataChangeKind::ConsumerGroupPartitionHandedOver: + return "consumer_group_partition_handed_over"; + case MetadataChangeKind::UserCreated: + return "user_created"; + case MetadataChangeKind::UserUpdated: + return "user_updated"; + case MetadataChangeKind::UserDeleted: + return "user_deleted"; + case MetadataChangeKind::UserPasswordChanged: + return "user_password_changed"; + case MetadataChangeKind::UserPermissionsUpdated: + return "user_permissions_updated"; + case MetadataChangeKind::PersonalAccessTokenCreated: + return "personal_access_token_created"; + case MetadataChangeKind::PersonalAccessTokenDeleted: + return "personal_access_token_deleted"; + case MetadataChangeKind::SessionRegistered: + return "session_registered"; + case MetadataChangeKind::SessionClosed: + return "session_closed"; + } + throw std::invalid_argument("Unknown metadata change kind"); +} + +/** + * @brief Outcome of a metadata change query. + */ +enum class MetadataChangeStatus : std::uint8_t { + Ok = 0, ///< Records or items after the requested position. + Behind = 1, ///< The answering node has not applied the requested position yet. Retry later. + Expired = 2, ///< The node no longer holds every record after the position. Rebuild from current state. + Reset = 3, ///< The position belongs to another history. Rebuild from current state. +}; + +constexpr std::string_view to_string(const MetadataChangeStatus status) { + switch (status) { + case MetadataChangeStatus::Ok: + return "ok"; + case MetadataChangeStatus::Behind: + return "behind"; + case MetadataChangeStatus::Expired: + return "expired"; + case MetadataChangeStatus::Reset: + return "reset"; + } + throw std::invalid_argument("Unknown metadata change status"); +} + +/** + * @brief Whether an answer pages change records or current state items. + */ +enum class MetadataChangePhase : std::uint8_t { + Changes = 0, + Current = 1, +}; + +constexpr std::string_view to_string(const MetadataChangePhase phase) { + switch (phase) { + case MetadataChangePhase::Changes: + return "changes"; + case MetadataChangePhase::Current: + return "current"; + } + throw std::invalid_argument("Unknown metadata change phase"); +} + +/** @brief Why a consumer group member was removed. */ +enum class MetadataMemberLeftReason : std::uint8_t { + Left = 0, + SessionEnded = 1, +}; + +constexpr std::string_view to_string(const MetadataMemberLeftReason reason) { + switch (reason) { + case MetadataMemberLeftReason::Left: + return "left"; + case MetadataMemberLeftReason::SessionEnded: + return "session_ended"; + } + throw std::invalid_argument("Unknown member left reason"); +} + +/** @brief Why a personal access token was removed. */ +enum class MetadataTokenDeletedReason : std::uint8_t { + Deleted = 0, + Expired = 1, +}; + +constexpr std::string_view to_string(const MetadataTokenDeletedReason reason) { + switch (reason) { + case MetadataTokenDeletedReason::Deleted: + return "deleted"; + case MetadataTokenDeletedReason::Expired: + return "expired"; + } + throw std::invalid_argument("Unknown token deleted reason"); +} + +/** @brief Why a session was closed. */ +enum class MetadataSessionClosedReason : std::uint8_t { + Logout = 0, + Expired = 1, + Revoked = 2, ///< The user was deactivated or deleted. +}; + +constexpr std::string_view to_string(const MetadataSessionClosedReason reason) { + switch (reason) { + case MetadataSessionClosedReason::Logout: + return "logout"; + case MetadataSessionClosedReason::Expired: + return "expired"; + case MetadataSessionClosedReason::Revoked: + return "revoked"; + } + throw std::invalid_argument("Unknown session closed reason"); +} + +/** @brief Why a user's public permission set changed. */ +enum class MetadataPermissionsUpdatedReason : std::uint8_t { + Updated = 0, ///< UpdatePermissions() replaced the set. + ResourceDeleted = 1, ///< A stream or topic delete pruned the user's grants on it. +}; + +constexpr std::string_view to_string(const MetadataPermissionsUpdatedReason reason) { + switch (reason) { + case MetadataPermissionsUpdatedReason::Updated: + return "updated"; + case MetadataPermissionsUpdatedReason::ResourceDeleted: + return "resource_deleted"; + } + throw std::invalid_argument("Unknown permissions updated reason"); +} + +/** + * @brief Position of the last metadata change record a caller consumed. + * + * Persist ToString() and pass the cursor back through + * MetadataChangesQuery::After() to read the records after it. Op() zero names + * the beginning of the history LogId() identifies. + */ +class MetadataChangeCursor final { + public: + /** + * @brief Parses the `mc1_` text form ToString() returns. + * @param text Cursor text. + * @return Parsed cursor. + * @throws IggyException with Kind() invalid_command when @p text is not a + * canonical cursor. The message never contains the text. + */ + static MetadataChangeCursor Parse(const std::string &text); + + /** @brief Returns the prepare checksum of op 1 of this history. */ + [[nodiscard]] absl::uint128 LogId() const noexcept { return log_id_; } + + /** @brief Returns the metadata operation of the consumed record. */ + [[nodiscard]] std::uint64_t Op() const noexcept { return op_; } + + /** @brief Returns the prepare checksum of Op(). */ + [[nodiscard]] absl::uint128 Checksum() const noexcept { return checksum_; } + + /** @brief Returns the position of the consumed record within Op(). */ + [[nodiscard]] std::uint32_t Ordinal() const noexcept { return ordinal_; } + + /** @brief Returns whether the cursor names the beginning of its history. */ + [[nodiscard]] bool IsZero() const noexcept { return op_ == 0; } + + /** @brief Returns the `mc1_` text form, `mc1_` and 59 characters of unpadded base64url. */ + [[nodiscard]] const std::string &ToString() const noexcept { return text_; } + + friend bool operator==(const MetadataChangeCursor &left, const MetadataChangeCursor &right) noexcept { + return left.text_ == right.text_; + } + + friend bool operator!=(const MetadataChangeCursor &left, const MetadataChangeCursor &right) noexcept { + return !(left == right); + } + + private: + MetadataChangeCursor() = default; + + static MetadataChangeCursor FromFfi(ffi::MetadataChangeCursor cursor); + + friend class MetadataChanges; + friend class MetadataCurrentPageToken; + + absl::uint128 log_id_{}; + std::uint64_t op_{}; + absl::uint128 checksum_{}; + std::uint32_t ordinal_{}; + std::string text_; +}; + +/** + * @brief Continuation of a current state scan. + * + * Pass it back unchanged through MetadataChangesQuery::Current() with the + * same groups to read the next page. + */ +class MetadataCurrentPageToken final { + public: + /** + * @brief Parses the `cs1_` text form ToString() returns. + * @param text Page token text. + * @return Parsed page token. + * @throws IggyException with Kind() invalid_command when @p text is not a + * canonical page token. The message never contains the text. + */ + static MetadataCurrentPageToken Parse(const std::string &text); + + /** @brief Returns the `cs1_` text form. */ + [[nodiscard]] const std::string &ToString() const noexcept { return text_; } + + /** @brief Returns the position captured before the scan's first page. */ + [[nodiscard]] const MetadataChangeCursor &Anchor() const noexcept { return anchor_; } + + /** @brief Returns the highest head any page of the scan reported so far. */ + [[nodiscard]] std::uint64_t RequiredHead() const noexcept { return required_head_; } + + /** @brief Returns the groups the scan was started with, in code order. */ + [[nodiscard]] std::vector<MetadataChangeGroup> Groups() const { + std::vector<MetadataChangeGroup> groups; + for (std::uint8_t code = 1; code <= 8; ++code) { + if ((groups_mask_ & (1U << (code - 1))) != 0) { + groups.push_back(static_cast<MetadataChangeGroup>(code)); + } + } + return groups; + } + + /** @brief Returns the group of the last item returned. */ + [[nodiscard]] MetadataChangeGroup LastGroup() const noexcept { return last_group_; } + + /** @brief Returns the key of the last item returned, in the page token key layout. */ + [[nodiscard]] const std::vector<std::uint8_t> &LastKey() const noexcept { return last_key_; } + + private: + MetadataCurrentPageToken() = default; + + static MetadataCurrentPageToken FromFfi(ffi::MetadataCurrentPageToken token); + + friend class MetadataChanges; + + std::string text_; + MetadataChangeCursor anchor_; + std::uint64_t required_head_{}; + std::uint8_t groups_mask_{}; + MetadataChangeGroup last_group_{MetadataChangeGroup::Streams}; + std::vector<std::uint8_t> last_key_; +}; + +/** + * @brief One read of the metadata change log or of the current state. + * + * Start with Earliest(), Latest(), After(), or Current(), then narrow the + * read with the setters. IggyBlockingClient::GetMetadataChanges() refuses an + * invalid query with Kind() invalid_command before anything is sent. + */ +class MetadataChangesQuery final { + public: + /** @brief Where the read starts. The values are the wire codes. */ + enum class Start : std::uint8_t { + Earliest = 0, ///< The oldest operation the requested groups still hold completely. + Latest = 1, ///< No records, only a cursor at the newest retained record. + After = 2, ///< The records after Cursor(). + Current = 3, ///< Current state pages, continuing after PageToken() when set. + }; + + static constexpr std::uint32_t kDefaultLimit = 100; + static constexpr std::uint32_t kMaxLimit = 1000; + + /** @brief Reads from the oldest operation the requested groups still hold completely. */ + static MetadataChangesQuery Earliest() { return MetadataChangesQuery(Start::Earliest); } + + /** @brief Reads no records and returns a cursor at the newest retained record to follow from. */ + static MetadataChangesQuery Latest() { return MetadataChangesQuery(Start::Latest); } + + /** @brief Reads the records after @p cursor. */ + static MetadataChangesQuery After(MetadataChangeCursor cursor) { + MetadataChangesQuery query(Start::After); + query.cursor_ = std::move(cursor); + return query; + } + + /** @brief Reads the first page of a current state scan. */ + static MetadataChangesQuery Current() { return MetadataChangesQuery(Start::Current); } + + /** @brief Reads the page of a current state scan after @p token. Keep the scan's groups. */ + static MetadataChangesQuery Current(MetadataCurrentPageToken token) { + MetadataChangesQuery query(Start::Current); + query.page_token_ = std::move(token); + return query; + } + + /** @brief Returns where the read starts. */ + [[nodiscard]] Start StartMode() const noexcept { return start_; } + + /** @brief Returns the cursor of an After() read. */ + [[nodiscard]] const std::optional<MetadataChangeCursor> &Cursor() const noexcept { return cursor_; } + + /** @brief Returns the page token of a continued Current() read. */ + [[nodiscard]] const std::optional<MetadataCurrentPageToken> &PageToken() const noexcept { return page_token_; } + + /** @brief Returns the requested groups. Empty means every group. */ + [[nodiscard]] const std::vector<MetadataChangeGroup> &Groups() const noexcept { return groups_; } + + /** @brief Returns the inclusive lower bound on record timestamps, in microseconds. */ + [[nodiscard]] std::optional<std::uint64_t> Since() const noexcept { return since_; } + + /** @brief Returns the inclusive upper bound on record timestamps, in microseconds. */ + [[nodiscard]] std::optional<std::uint64_t> Until() const noexcept { return until_; } + + /** @brief Returns the most records or items one page returns. */ + [[nodiscard]] std::uint32_t Limit() const noexcept { return limit_; } + + /** + * @brief Selects the groups to read. + * + * A current state scan must include each selected group's parents, so a + * mirror can apply the cascades of their deletes. + * @param groups Requested groups. Empty selects every group. + * @return Reference to this query. + */ + MetadataChangesQuery &SetGroups(std::vector<MetadataChangeGroup> groups) { + groups_ = std::move(groups); + return *this; + } + + /** + * @brief Keeps records with a timestamp at or after @p micros. + * + * Time bounds apply to Earliest() and After() reads only. + * @param micros Microseconds since the Unix epoch. + * @return Reference to this query. + */ + MetadataChangesQuery &SetSince(std::uint64_t micros) { + since_ = micros; + return *this; + } + + /** + * @brief Keeps records with a timestamp at or before @p micros. + * + * Time bounds apply to Earliest() and After() reads only. + * @param micros Microseconds since the Unix epoch. + * @return Reference to this query. + */ + MetadataChangesQuery &SetUntil(std::uint64_t micros) { + until_ = micros; + return *this; + } + + /** + * @brief Sets the most records or items one page returns. + * @param limit From 1 to kMaxLimit. The default is kDefaultLimit. + * @return Reference to this query. + */ + MetadataChangesQuery &SetLimit(std::uint32_t limit) { + limit_ = limit; + return *this; + } + + /** + * @brief Applies the checks GetMetadataChanges() runs before sending. + * @throws IggyException with Kind() invalid_command when the limit is + * outside 1 to kMaxLimit, Since() is above Until(), a time bound + * accompanies Latest() or Current() or lies past what the + * platform clock holds, a current state scan misses a parent + * group, or the page token was issued for other groups. + */ + void Validate() const; + + /** + * @brief Encodes the query as the binary transports send it, without + * validating it. + * @return Request payload without framing. + * @throws IggyException with Kind() invalid_command for a group this build + * does not know or a time bound past what the platform clock holds. + */ + [[nodiscard]] std::vector<std::uint8_t> ToBytes() const; + + private: + explicit MetadataChangesQuery(Start start) : start_(start) {} + + [[nodiscard]] ffi::MetadataChangesQuery ToFfi() const; + + friend class IggyBlockingClient; + + Start start_; + std::optional<MetadataChangeCursor> cursor_; + std::optional<MetadataCurrentPageToken> page_token_; + std::vector<MetadataChangeGroup> groups_; + std::optional<std::uint64_t> since_; + std::optional<std::uint64_t> until_; + std::uint32_t limit_{kDefaultLimit}; +}; + +/** + * @brief A stream after a create or update, and a current state streams item. + */ +class MetadataStreamChange final { + public: + /** @brief Returns the numeric stream ID. */ + [[nodiscard]] std::uint32_t StreamId() const noexcept { return stream_id_; } + + /** @brief Returns the op that created the stream. A recreated stream with a reused ID has another. */ + [[nodiscard]] std::uint64_t StreamCreatedOp() const noexcept { return stream_created_op_; } + + /** @brief Returns the creation timestamp in microseconds. */ + [[nodiscard]] std::uint64_t CreatedAt() const noexcept { return created_at_; } + + /** @brief Returns the stream name after the change. */ + [[nodiscard]] const std::string &Name() const noexcept { return name_; } + + /** + * @brief Returns the public options after the change, as the read routes return them. + * @return `std::nullopt` when the server holds option state this build + * cannot read. Keep the previous values then. + */ + [[nodiscard]] const std::optional<ResourceOptions> &Options() const noexcept { return options_; } + + private: + MetadataStreamChange() = default; + + static MetadataStreamChange FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + friend class MetadataCurrentItem; + + std::uint32_t stream_id_{}; + std::uint64_t stream_created_op_{}; + std::uint64_t created_at_{}; + std::string name_; + std::optional<ResourceOptions> options_; +}; + +/** + * @brief A removed stream. + */ +class MetadataStreamDeleted final { + public: + /** @brief Returns the numeric stream ID. */ + [[nodiscard]] std::uint32_t StreamId() const noexcept { return stream_id_; } + + /** @brief Returns the op that created the stream. */ + [[nodiscard]] std::uint64_t StreamCreatedOp() const noexcept { return stream_created_op_; } + + /** @brief Returns the creation timestamp in microseconds. */ + [[nodiscard]] std::uint64_t CreatedAt() const noexcept { return created_at_; } + + /** @brief Returns the stream name. */ + [[nodiscard]] const std::string &Name() const noexcept { return name_; } + + /** @brief Returns the op of a deferred delete's request, absent for an immediate delete. */ + [[nodiscard]] std::optional<std::uint64_t> RequestedOp() const noexcept { return requested_op_; } + + private: + MetadataStreamDeleted() = default; + + static MetadataStreamDeleted FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + + std::uint32_t stream_id_{}; + std::uint64_t stream_created_op_{}; + std::uint64_t created_at_{}; + std::string name_; + std::optional<std::uint64_t> requested_op_; +}; + +/** + * @brief A topic after a create or update. + */ +class MetadataTopicChange final { + public: + /** @brief Returns the numeric stream ID. */ + [[nodiscard]] std::uint32_t StreamId() const noexcept { return stream_id_; } + + /** @brief Returns the op that created the stream. */ + [[nodiscard]] std::uint64_t StreamCreatedOp() const noexcept { return stream_created_op_; } + + /** @brief Returns the numeric topic ID. */ + [[nodiscard]] std::uint32_t TopicId() const noexcept { return topic_id_; } + + /** @brief Returns the op that created the topic. */ + [[nodiscard]] std::uint64_t TopicCreatedOp() const noexcept { return topic_created_op_; } + + /** @brief Returns the creation timestamp in microseconds. */ + [[nodiscard]] std::uint64_t CreatedAt() const noexcept { return created_at_; } + + /** @brief Returns the topic name after the change. */ + [[nodiscard]] const std::string &Name() const noexcept { return name_; } + + /** @brief Returns the partition count after the change. */ + [[nodiscard]] std::uint32_t PartitionsCount() const noexcept { return partitions_count_; } + + /** @brief Returns the public options, `std::nullopt` when this build cannot read them. */ + [[nodiscard]] const std::optional<ResourceOptions> &Options() const noexcept { return options_; } + + private: + MetadataTopicChange() = default; + + static MetadataTopicChange FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + + std::uint32_t stream_id_{}; + std::uint64_t stream_created_op_{}; + std::uint32_t topic_id_{}; + std::uint64_t topic_created_op_{}; + std::uint64_t created_at_{}; + std::string name_; + std::uint32_t partitions_count_{}; + std::optional<ResourceOptions> options_; +}; + +/** + * @brief A current state topics item. Partition counts travel as partitions items. + */ +class MetadataTopicItem final { + public: + /** @brief Returns the numeric stream ID. */ + [[nodiscard]] std::uint32_t StreamId() const noexcept { return stream_id_; } + + /** @brief Returns the op that created the stream. */ + [[nodiscard]] std::uint64_t StreamCreatedOp() const noexcept { return stream_created_op_; } + + /** @brief Returns the numeric topic ID. */ + [[nodiscard]] std::uint32_t TopicId() const noexcept { return topic_id_; } + + /** @brief Returns the op that created the topic. */ + [[nodiscard]] std::uint64_t TopicCreatedOp() const noexcept { return topic_created_op_; } + + /** @brief Returns the creation timestamp in microseconds. */ + [[nodiscard]] std::uint64_t CreatedAt() const noexcept { return created_at_; } + + /** @brief Returns the topic name. */ + [[nodiscard]] const std::string &Name() const noexcept { return name_; } + + /** @brief Returns the public options, `std::nullopt` when this build cannot read them. */ + [[nodiscard]] const std::optional<ResourceOptions> &Options() const noexcept { return options_; } + + private: + MetadataTopicItem() = default; + + static MetadataTopicItem FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataCurrentItem; + + std::uint32_t stream_id_{}; + std::uint64_t stream_created_op_{}; + std::uint32_t topic_id_{}; + std::uint64_t topic_created_op_{}; + std::uint64_t created_at_{}; + std::string name_; + std::optional<ResourceOptions> options_; +}; + +/** + * @brief A removed topic. + */ +class MetadataTopicDeleted final { + public: + /** @brief Returns the numeric stream ID. */ + [[nodiscard]] std::uint32_t StreamId() const noexcept { return stream_id_; } + + /** @brief Returns the op that created the stream. */ + [[nodiscard]] std::uint64_t StreamCreatedOp() const noexcept { return stream_created_op_; } + + /** @brief Returns the numeric topic ID. */ + [[nodiscard]] std::uint32_t TopicId() const noexcept { return topic_id_; } + + /** @brief Returns the op that created the topic. */ + [[nodiscard]] std::uint64_t TopicCreatedOp() const noexcept { return topic_created_op_; } + + /** @brief Returns the creation timestamp in microseconds. */ + [[nodiscard]] std::uint64_t CreatedAt() const noexcept { return created_at_; } + + /** @brief Returns the topic name. */ + [[nodiscard]] const std::string &Name() const noexcept { return name_; } + + /** @brief Returns the op of a deferred delete's request, absent for an immediate delete. */ + [[nodiscard]] std::optional<std::uint64_t> RequestedOp() const noexcept { return requested_op_; } + + private: + MetadataTopicDeleted() = default; + + static MetadataTopicDeleted FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + + std::uint32_t stream_id_{}; + std::uint64_t stream_created_op_{}; + std::uint32_t topic_id_{}; + std::uint64_t topic_created_op_{}; + std::uint64_t created_at_{}; + std::string name_; + std::optional<std::uint64_t> requested_op_; +}; + +/** + * @brief A partition count change. + */ +class MetadataPartitionsChanged final { + public: + /** @brief Returns the numeric stream ID. */ + [[nodiscard]] std::uint32_t StreamId() const noexcept { return stream_id_; } + + /** @brief Returns the op that created the stream. */ + [[nodiscard]] std::uint64_t StreamCreatedOp() const noexcept { return stream_created_op_; } + + /** @brief Returns the numeric topic ID. */ + [[nodiscard]] std::uint32_t TopicId() const noexcept { return topic_id_; } + + /** @brief Returns the op that created the topic. */ + [[nodiscard]] std::uint64_t TopicCreatedOp() const noexcept { return topic_created_op_; } + + /** @brief Returns the partition count after the change. */ + [[nodiscard]] std::uint32_t PartitionsCount() const noexcept { return partitions_count_; } + + /** @brief Returns the deleted partition IDs in ascending order, empty on create. */ + [[nodiscard]] const std::vector<std::uint32_t> &RemovedPartitionIds() const noexcept { + return removed_partition_ids_; + } + + /** @brief Returns the op of a deferred delete's request, absent for an immediate change. */ + [[nodiscard]] std::optional<std::uint64_t> RequestedOp() const noexcept { return requested_op_; } + + private: + MetadataPartitionsChanged() = default; + + static MetadataPartitionsChanged FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + + std::uint32_t stream_id_{}; + std::uint64_t stream_created_op_{}; + std::uint32_t topic_id_{}; + std::uint64_t topic_created_op_{}; + std::uint32_t partitions_count_{}; + std::vector<std::uint32_t> removed_partition_ids_; + std::optional<std::uint64_t> requested_op_; +}; + +/** + * @brief A current state partitions item. + */ +class MetadataPartitionsItem final { + public: + /** @brief Returns the numeric stream ID. */ + [[nodiscard]] std::uint32_t StreamId() const noexcept { return stream_id_; } + + /** @brief Returns the op that created the stream. */ + [[nodiscard]] std::uint64_t StreamCreatedOp() const noexcept { return stream_created_op_; } + + /** @brief Returns the numeric topic ID. */ + [[nodiscard]] std::uint32_t TopicId() const noexcept { return topic_id_; } + + /** @brief Returns the op that created the topic. */ + [[nodiscard]] std::uint64_t TopicCreatedOp() const noexcept { return topic_created_op_; } + + /** @brief Returns the partition count. */ + [[nodiscard]] std::uint32_t PartitionsCount() const noexcept { return partitions_count_; } + + private: + MetadataPartitionsItem() = default; + + static MetadataPartitionsItem FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataCurrentItem; + + std::uint32_t stream_id_{}; + std::uint64_t stream_created_op_{}; + std::uint32_t topic_id_{}; + std::uint64_t topic_created_op_{}; + std::uint32_t partitions_count_{}; +}; + +/** + * @brief A created consumer group with the topic's resulting group count. + */ +class MetadataConsumerGroupCreated final { + public: + /** @brief Returns the numeric stream ID. */ + [[nodiscard]] std::uint32_t StreamId() const noexcept { return stream_id_; } + + /** @brief Returns the op that created the stream. */ + [[nodiscard]] std::uint64_t StreamCreatedOp() const noexcept { return stream_created_op_; } + + /** @brief Returns the numeric topic ID. */ + [[nodiscard]] std::uint32_t TopicId() const noexcept { return topic_id_; } + + /** @brief Returns the op that created the topic. */ + [[nodiscard]] std::uint64_t TopicCreatedOp() const noexcept { return topic_created_op_; } + + /** @brief Returns the numeric consumer group ID. */ + [[nodiscard]] std::uint32_t GroupId() const noexcept { return group_id_; } + + /** @brief Returns the consumer group name. */ + [[nodiscard]] const std::string &Name() const noexcept { return name_; } + + /** @brief Returns the topic's consumer group count after the change. */ + [[nodiscard]] std::uint32_t ConsumerGroupsCount() const noexcept { return consumer_groups_count_; } + + private: + MetadataConsumerGroupCreated() = default; + + static MetadataConsumerGroupCreated FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + + std::uint32_t stream_id_{}; + std::uint64_t stream_created_op_{}; + std::uint32_t topic_id_{}; + std::uint64_t topic_created_op_{}; + std::uint32_t group_id_{}; + std::string name_; + std::uint32_t consumer_groups_count_{}; +}; + +/** + * @brief A removed consumer group with the topic's resulting group count. + */ +class MetadataConsumerGroupDeleted final { + public: + /** @brief Returns the numeric stream ID. */ + [[nodiscard]] std::uint32_t StreamId() const noexcept { return stream_id_; } + + /** @brief Returns the op that created the stream. */ + [[nodiscard]] std::uint64_t StreamCreatedOp() const noexcept { return stream_created_op_; } + + /** @brief Returns the numeric topic ID. */ + [[nodiscard]] std::uint32_t TopicId() const noexcept { return topic_id_; } + + /** @brief Returns the op that created the topic. */ + [[nodiscard]] std::uint64_t TopicCreatedOp() const noexcept { return topic_created_op_; } + + /** @brief Returns the numeric consumer group ID. */ + [[nodiscard]] std::uint32_t GroupId() const noexcept { return group_id_; } + + /** @brief Returns the consumer group name. */ + [[nodiscard]] const std::string &Name() const noexcept { return name_; } + + /** @brief Returns the topic's consumer group count after the change. */ + [[nodiscard]] std::uint32_t ConsumerGroupsCount() const noexcept { return consumer_groups_count_; } + + /** @brief Returns the op of a deferred delete's request, absent for an immediate delete. */ + [[nodiscard]] std::optional<std::uint64_t> RequestedOp() const noexcept { return requested_op_; } + + private: + MetadataConsumerGroupDeleted() = default; + + static MetadataConsumerGroupDeleted FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + + std::uint32_t stream_id_{}; + std::uint64_t stream_created_op_{}; + std::uint32_t topic_id_{}; + std::uint64_t topic_created_op_{}; + std::uint32_t group_id_{}; + std::string name_; + std::uint32_t consumer_groups_count_{}; + std::optional<std::uint64_t> requested_op_; +}; + +/** + * @brief A current state consumer groups item. + */ +class MetadataConsumerGroupItem final { + public: + /** @brief Returns the numeric stream ID. */ + [[nodiscard]] std::uint32_t StreamId() const noexcept { return stream_id_; } + + /** @brief Returns the op that created the stream. */ + [[nodiscard]] std::uint64_t StreamCreatedOp() const noexcept { return stream_created_op_; } + + /** @brief Returns the numeric topic ID. */ + [[nodiscard]] std::uint32_t TopicId() const noexcept { return topic_id_; } + + /** @brief Returns the op that created the topic. */ + [[nodiscard]] std::uint64_t TopicCreatedOp() const noexcept { return topic_created_op_; } + + /** @brief Returns the numeric consumer group ID. */ + [[nodiscard]] std::uint32_t GroupId() const noexcept { return group_id_; } + + /** @brief Returns the consumer group name. */ + [[nodiscard]] const std::string &Name() const noexcept { return name_; } + + private: + MetadataConsumerGroupItem() = default; + + static MetadataConsumerGroupItem FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataCurrentItem; + + std::uint32_t stream_id_{}; + std::uint64_t stream_created_op_{}; + std::uint32_t topic_id_{}; + std::uint64_t topic_created_op_{}; + std::uint32_t group_id_{}; + std::string name_; +}; + +/** + * @brief A member that joined a consumer group. + */ +class MetadataMemberJoined final { + public: + /** @brief Returns the numeric stream ID. */ + [[nodiscard]] std::uint32_t StreamId() const noexcept { return stream_id_; } + + /** @brief Returns the op that created the stream. */ + [[nodiscard]] std::uint64_t StreamCreatedOp() const noexcept { return stream_created_op_; } + + /** @brief Returns the numeric topic ID. */ + [[nodiscard]] std::uint32_t TopicId() const noexcept { return topic_id_; } + + /** @brief Returns the op that created the topic. */ + [[nodiscard]] std::uint64_t TopicCreatedOp() const noexcept { return topic_created_op_; } + + /** @brief Returns the numeric consumer group ID. */ + [[nodiscard]] std::uint32_t GroupId() const noexcept { return group_id_; } + + /** @brief Returns the member ID within the group. */ + [[nodiscard]] std::uint32_t MemberId() const noexcept { return member_id_; } + + /** @brief Returns the session's client ID. */ + [[nodiscard]] absl::uint128 ClientId() const noexcept { return client_id_; } + + /** @brief Returns the group's member count after the change. */ + [[nodiscard]] std::uint32_t MembersCount() const noexcept { return members_count_; } + + /** @brief Returns the installed partition IDs, normally empty because a handover installs them. */ + [[nodiscard]] const std::vector<std::uint32_t> &PartitionIds() const noexcept { return partition_ids_; } + + private: + MetadataMemberJoined() = default; + + static MetadataMemberJoined FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + + std::uint32_t stream_id_{}; + std::uint64_t stream_created_op_{}; + std::uint32_t topic_id_{}; + std::uint64_t topic_created_op_{}; + std::uint32_t group_id_{}; + std::uint32_t member_id_{}; + absl::uint128 client_id_{}; + std::uint32_t members_count_{}; + std::vector<std::uint32_t> partition_ids_; +}; + +/** + * @brief A member that left a consumer group. + */ +class MetadataMemberLeft final { + public: + /** @brief Returns the numeric stream ID. */ + [[nodiscard]] std::uint32_t StreamId() const noexcept { return stream_id_; } + + /** @brief Returns the op that created the stream. */ + [[nodiscard]] std::uint64_t StreamCreatedOp() const noexcept { return stream_created_op_; } + + /** @brief Returns the numeric topic ID. */ + [[nodiscard]] std::uint32_t TopicId() const noexcept { return topic_id_; } + + /** @brief Returns the op that created the topic. */ + [[nodiscard]] std::uint64_t TopicCreatedOp() const noexcept { return topic_created_op_; } + + /** @brief Returns the numeric consumer group ID. */ + [[nodiscard]] std::uint32_t GroupId() const noexcept { return group_id_; } + + /** @brief Returns the member ID within the group. */ + [[nodiscard]] std::uint32_t MemberId() const noexcept { return member_id_; } + + /** @brief Returns the session's client ID. */ + [[nodiscard]] absl::uint128 ClientId() const noexcept { return client_id_; } + + /** @brief Returns the group's member count after the change. */ + [[nodiscard]] std::uint32_t MembersCount() const noexcept { return members_count_; } + + /** @brief Returns why the member was removed. */ + [[nodiscard]] MetadataMemberLeftReason Reason() const noexcept { return reason_; } + + private: + MetadataMemberLeft() = default; + + static MetadataMemberLeft FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + + std::uint32_t stream_id_{}; + std::uint64_t stream_created_op_{}; + std::uint32_t topic_id_{}; + std::uint64_t topic_created_op_{}; + std::uint32_t group_id_{}; + std::uint32_t member_id_{}; + absl::uint128 client_id_{}; + std::uint32_t members_count_{}; + MetadataMemberLeftReason reason_{MetadataMemberLeftReason::Left}; +}; + +/** + * @brief A completed partition handover within a consumer group. + */ +class MetadataPartitionHandedOver final { + public: + /** @brief Returns the numeric stream ID. */ + [[nodiscard]] std::uint32_t StreamId() const noexcept { return stream_id_; } + + /** @brief Returns the op that created the stream. */ + [[nodiscard]] std::uint64_t StreamCreatedOp() const noexcept { return stream_created_op_; } + + /** @brief Returns the numeric topic ID. */ + [[nodiscard]] std::uint32_t TopicId() const noexcept { return topic_id_; } + + /** @brief Returns the op that created the topic. */ + [[nodiscard]] std::uint64_t TopicCreatedOp() const noexcept { return topic_created_op_; } + + /** @brief Returns the numeric consumer group ID. */ + [[nodiscard]] std::uint32_t GroupId() const noexcept { return group_id_; } + + /** @brief Returns the partition that changed owner. */ + [[nodiscard]] std::uint32_t PartitionId() const noexcept { return partition_id_; } + + /** @brief Returns the new owner, absent when the last member left. */ + [[nodiscard]] std::optional<std::uint32_t> MemberId() const noexcept { return member_id_; } + + private: + MetadataPartitionHandedOver() = default; + + static MetadataPartitionHandedOver FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + + std::uint32_t stream_id_{}; + std::uint64_t stream_created_op_{}; + std::uint32_t topic_id_{}; + std::uint64_t topic_created_op_{}; + std::uint32_t group_id_{}; + std::uint32_t partition_id_{}; + std::optional<std::uint32_t> member_id_; +}; + +/** + * @brief A current state consumer group members item. + */ +class MetadataMemberItem final { + public: + /** @brief Returns the numeric stream ID. */ + [[nodiscard]] std::uint32_t StreamId() const noexcept { return stream_id_; } + + /** @brief Returns the op that created the stream. */ + [[nodiscard]] std::uint64_t StreamCreatedOp() const noexcept { return stream_created_op_; } + + /** @brief Returns the numeric topic ID. */ + [[nodiscard]] std::uint32_t TopicId() const noexcept { return topic_id_; } + + /** @brief Returns the op that created the topic. */ + [[nodiscard]] std::uint64_t TopicCreatedOp() const noexcept { return topic_created_op_; } + + /** @brief Returns the numeric consumer group ID. */ + [[nodiscard]] std::uint32_t GroupId() const noexcept { return group_id_; } + + /** @brief Returns the member ID within the group. */ + [[nodiscard]] std::uint32_t MemberId() const noexcept { return member_id_; } + + /** @brief Returns the session's client ID. */ + [[nodiscard]] absl::uint128 ClientId() const noexcept { return client_id_; } + + /** @brief Returns the installed partition IDs in ascending order. */ + [[nodiscard]] const std::vector<std::uint32_t> &PartitionIds() const noexcept { return partition_ids_; } + + private: + MetadataMemberItem() = default; + + static MetadataMemberItem FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataCurrentItem; + + std::uint32_t stream_id_{}; + std::uint64_t stream_created_op_{}; + std::uint32_t topic_id_{}; + std::uint64_t topic_created_op_{}; + std::uint32_t group_id_{}; + std::uint32_t member_id_{}; + absl::uint128 client_id_{}; + std::vector<std::uint32_t> partition_ids_; +}; + +/** + * @brief A user after a create or update, and a current state users item. + */ +class MetadataUserChange final { + public: + /** @brief Returns the numeric user ID. */ + [[nodiscard]] std::uint32_t UserId() const noexcept { return user_id_; } + + /** @brief Returns the creation timestamp in microseconds. A recreated user with a reused ID has another. */ + [[nodiscard]] std::uint64_t UserCreatedAt() const noexcept { return user_created_at_; } + + /** @brief Returns the username after the change. */ + [[nodiscard]] const std::string &Username() const noexcept { return username_; } + + /** @brief Returns the user status after the change. */ + [[nodiscard]] UserStatus Status() const noexcept { return status_; } + + /** @brief Returns the complete public permission set, empty when the user has none. */ + [[nodiscard]] const std::optional<::iggy::Permissions> &Permissions() const noexcept { return permissions_; } + + /** @brief Returns the public options, `std::nullopt` when this build cannot read them. */ + [[nodiscard]] const std::optional<ResourceOptions> &Options() const noexcept { return options_; } + + private: + MetadataUserChange() = default; + + static MetadataUserChange FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + friend class MetadataCurrentItem; + + std::uint32_t user_id_{}; + std::uint64_t user_created_at_{}; + std::string username_; + UserStatus status_{UserStatus::Active}; + std::optional<::iggy::Permissions> permissions_; + std::optional<ResourceOptions> options_; +}; + +/** + * @brief A deleted user, or one whose password changed. + */ +class MetadataUserIdentity final { + public: + /** @brief Returns the numeric user ID. */ + [[nodiscard]] std::uint32_t UserId() const noexcept { return user_id_; } + + /** @brief Returns the creation timestamp in microseconds. */ + [[nodiscard]] std::uint64_t UserCreatedAt() const noexcept { return user_created_at_; } + + /** @brief Returns the username. */ + [[nodiscard]] const std::string &Username() const noexcept { return username_; } + + private: + MetadataUserIdentity() = default; + + static MetadataUserIdentity FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + + std::uint32_t user_id_{}; + std::uint64_t user_created_at_{}; + std::string username_; +}; + +/** + * @brief A user's public permission set after a change. + */ +class MetadataUserPermissionsUpdated final { + public: + /** @brief Returns the numeric user ID. */ + [[nodiscard]] std::uint32_t UserId() const noexcept { return user_id_; } + + /** @brief Returns the creation timestamp in microseconds. */ + [[nodiscard]] std::uint64_t UserCreatedAt() const noexcept { return user_created_at_; } + + /** @brief Returns the username. */ + [[nodiscard]] const std::string &Username() const noexcept { return username_; } + + /** @brief Returns the complete permission set after the change, empty when the user has none. */ + [[nodiscard]] const std::optional<::iggy::Permissions> &Permissions() const noexcept { return permissions_; } + + /** @brief Returns why the set changed. */ + [[nodiscard]] MetadataPermissionsUpdatedReason Reason() const noexcept { return reason_; } + + private: + MetadataUserPermissionsUpdated() = default; + + static MetadataUserPermissionsUpdated FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + + std::uint32_t user_id_{}; + std::uint64_t user_created_at_{}; + std::string username_; + std::optional<::iggy::Permissions> permissions_; + MetadataPermissionsUpdatedReason reason_{MetadataPermissionsUpdatedReason::Updated}; +}; + +/** + * @brief A created personal access token, and a current state personal access tokens item. + */ +class MetadataTokenChange final { + public: + /** @brief Returns the owner's numeric user ID. */ + [[nodiscard]] std::uint32_t UserId() const noexcept { return user_id_; } + + /** @brief Returns the owner's creation timestamp in microseconds. */ + [[nodiscard]] std::uint64_t UserCreatedAt() const noexcept { return user_created_at_; } + + /** @brief Returns the token name. */ + [[nodiscard]] const std::string &Name() const noexcept { return name_; } + + /** @brief Returns the expiry in microseconds, absent for a token that never expires. */ + [[nodiscard]] std::optional<std::uint64_t> Expiry() const noexcept { return expiry_; } + + private: + MetadataTokenChange() = default; + + static MetadataTokenChange FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + friend class MetadataCurrentItem; + + std::uint32_t user_id_{}; + std::uint64_t user_created_at_{}; + std::string name_; + std::optional<std::uint64_t> expiry_; +}; + +/** + * @brief A removed personal access token. + */ +class MetadataTokenDeleted final { + public: + /** @brief Returns the owner's numeric user ID. */ + [[nodiscard]] std::uint32_t UserId() const noexcept { return user_id_; } + + /** @brief Returns the owner's creation timestamp in microseconds. */ + [[nodiscard]] std::uint64_t UserCreatedAt() const noexcept { return user_created_at_; } + + /** @brief Returns the token name. */ + [[nodiscard]] const std::string &Name() const noexcept { return name_; } + + /** @brief Returns why the token was removed. */ + [[nodiscard]] MetadataTokenDeletedReason Reason() const noexcept { return reason_; } + + private: + MetadataTokenDeleted() = default; + + static MetadataTokenDeleted FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + + std::uint32_t user_id_{}; + std::uint64_t user_created_at_{}; + std::string name_; + MetadataTokenDeletedReason reason_{MetadataTokenDeletedReason::Deleted}; +}; + +/** + * @brief A registered session, and a current state sessions item. + */ +class MetadataSession final { + public: + /** @brief Returns the session user's numeric ID. */ + [[nodiscard]] std::uint32_t UserId() const noexcept { return user_id_; } + + /** @brief Returns the session user's creation timestamp in microseconds. */ + [[nodiscard]] std::uint64_t UserCreatedAt() const noexcept { return user_created_at_; } + + /** @brief Returns the session's client ID. */ + [[nodiscard]] absl::uint128 ClientId() const noexcept { return client_id_; } + + /** @brief Returns the session epoch. */ + [[nodiscard]] std::uint64_t Session() const noexcept { return session_; } + + private: + MetadataSession() = default; + + static MetadataSession FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + friend class MetadataCurrentItem; + + std::uint32_t user_id_{}; + std::uint64_t user_created_at_{}; + absl::uint128 client_id_{}; + std::uint64_t session_{}; +}; + +/** + * @brief A closed session. + */ +class MetadataSessionClosed final { + public: + /** @brief Returns the session user's numeric ID. */ + [[nodiscard]] std::uint32_t UserId() const noexcept { return user_id_; } + + /** @brief Returns the session user's creation timestamp in microseconds. */ + [[nodiscard]] std::uint64_t UserCreatedAt() const noexcept { return user_created_at_; } + + /** @brief Returns the session's client ID. */ + [[nodiscard]] absl::uint128 ClientId() const noexcept { return client_id_; } + + /** @brief Returns the session epoch. */ + [[nodiscard]] std::uint64_t Session() const noexcept { return session_; } + + /** @brief Returns why the session was closed. */ + [[nodiscard]] MetadataSessionClosedReason Reason() const noexcept { return reason_; } + + private: + MetadataSessionClosed() = default; + + static MetadataSessionClosed FromFfi(ffi::MetadataChangeBody &body); + + friend class MetadataChangeRecord; + + std::uint32_t user_id_{}; + std::uint64_t user_created_at_{}; + absl::uint128 client_id_{}; + std::uint64_t session_{}; + MetadataSessionClosedReason reason_{MetadataSessionClosedReason::Logout}; +}; + +/** + * @brief A binary record of a group or kind this build does not know, kept + * with its codes and raw body. + */ +class MetadataUnknownChange final { + public: + /** @brief Returns the group's wire code. */ + [[nodiscard]] std::uint8_t GroupCode() const noexcept { return group_code_; } + + /** @brief Returns the kind's wire code. */ + [[nodiscard]] std::uint8_t KindCode() const noexcept { return kind_code_; } + + /** @brief Returns the raw record body. */ + [[nodiscard]] const std::vector<std::uint8_t> &Body() const noexcept { return body_; } + + private: + MetadataUnknownChange() = default; + + friend class MetadataChangeRecord; + + std::uint8_t group_code_{}; + std::uint8_t kind_code_{}; + std::vector<std::uint8_t> body_; +}; + +/** + * @brief A JSON record of a group or kind this build does not know, kept whole. + */ +class MetadataUnknownJsonChange final { + public: + /** @brief Returns the group name the server sent. */ + [[nodiscard]] const std::string &Group() const noexcept { return group_; } + + /** @brief Returns the kind name the server sent. */ + [[nodiscard]] const std::string &Kind() const noexcept { return kind_; } + + /** @brief Returns the whole record object as JSON text. */ + [[nodiscard]] const std::string &Json() const noexcept { return json_; } + + private: + MetadataUnknownJsonChange() = default; + + friend class MetadataChangeRecord; + + std::string group_; + std::string kind_; + std::string json_; +}; + +/** + * @brief A binary current state item of a group this build does not know. + */ +class MetadataUnknownItem final { + public: + /** @brief Returns the group's wire code. */ + [[nodiscard]] std::uint8_t GroupCode() const noexcept { return group_code_; } + + /** @brief Returns the raw item body. */ + [[nodiscard]] const std::vector<std::uint8_t> &Body() const noexcept { return body_; } + + private: + MetadataUnknownItem() = default; + + friend class MetadataCurrentItem; + + std::uint8_t group_code_{}; + std::vector<std::uint8_t> body_; +}; + +/** + * @brief A JSON current state item of a group this build does not know, kept whole. + */ +class MetadataUnknownJsonItem final { + public: + /** @brief Returns the group name the server sent. */ + [[nodiscard]] const std::string &Group() const noexcept { return group_; } + + /** @brief Returns the whole item object as JSON text. */ + [[nodiscard]] const std::string &Json() const noexcept { return json_; } + + private: + MetadataUnknownJsonItem() = default; + + friend class MetadataCurrentItem; + + std::string group_; + std::string json_; +}; + +/** + * @brief One metadata change record: the committed operation that produced + * it, its position within that operation, and what is true after it. + * + * Change() holds the typed body. Kind() tells apart kinds that share a body + * type, such as stream_created and stream_updated. + */ +class MetadataChangeRecord final { + public: + /** @brief The typed body of a record. */ + using Body = std::variant<MetadataStreamChange, + MetadataStreamDeleted, + MetadataTopicChange, + MetadataTopicDeleted, + MetadataPartitionsChanged, + MetadataConsumerGroupCreated, + MetadataConsumerGroupDeleted, + MetadataMemberJoined, + MetadataMemberLeft, + MetadataPartitionHandedOver, + MetadataUserChange, + MetadataUserIdentity, + MetadataUserPermissionsUpdated, + MetadataTokenChange, + MetadataTokenDeleted, + MetadataSession, + MetadataSessionClosed, + MetadataUnknownChange, + MetadataUnknownJsonChange>; + + /** + * @brief Decodes one record envelope as a binary response carries it. + * @param bytes Record envelope and body. + * @return Decoded record. A group or kind this build does not know is + * kept as MetadataUnknownChange. + * @throws IggyException with Kind() invalid_format when @p bytes are malformed. + */ + static MetadataChangeRecord FromBytes(const std::vector<std::uint8_t> &bytes); + + /** @brief Returns the metadata operation that produced the record. */ + [[nodiscard]] std::uint64_t Op() const noexcept { return op_; } + + /** @brief Returns the record's position within Op(). */ + [[nodiscard]] std::uint32_t Ordinal() const noexcept { return ordinal_; } + + /** @brief Returns the prepare checksum of Op(). */ + [[nodiscard]] absl::uint128 Checksum() const noexcept { return checksum_; } + + /** @brief Returns the prepare timestamp of Op() in microseconds. */ + [[nodiscard]] std::uint64_t Timestamp() const noexcept { return timestamp_; } + + /** @brief Returns the acting user, absent for server actions. */ + [[nodiscard]] std::optional<std::uint32_t> ActorUserId() const noexcept { return actor_user_id_; } + + /** @brief Returns the kind, absent for a record this build does not know. */ + [[nodiscard]] std::optional<MetadataChangeKind> Kind() const noexcept { return kind_; } + + /** @brief Returns the group, absent for a record this build does not know. */ + [[nodiscard]] std::optional<MetadataChangeGroup> Group() const noexcept { return group_; } + + /** @brief Returns the typed body. */ + [[nodiscard]] const Body &Change() const noexcept { return change_; } + + private: + MetadataChangeRecord(std::uint64_t op, + std::uint32_t ordinal, + absl::uint128 checksum, + std::uint64_t timestamp, + std::optional<std::uint32_t> actor_user_id, + std::optional<MetadataChangeKind> kind, + std::optional<MetadataChangeGroup> group, + Body change) + : op_(op), + ordinal_(ordinal), + checksum_(checksum), + timestamp_(timestamp), + actor_user_id_(actor_user_id), + kind_(kind), + group_(group), + change_(std::move(change)) {} + + static MetadataChangeRecord FromFfi(ffi::MetadataChangeRecord record); + static Body BodyFromFfi(MetadataChangeKind kind, ffi::MetadataChangeBody &body); + + friend class MetadataChanges; + + std::uint64_t op_; + std::uint32_t ordinal_; + absl::uint128 checksum_; + std::uint64_t timestamp_; + std::optional<std::uint32_t> actor_user_id_; + std::optional<MetadataChangeKind> kind_; + std::optional<MetadataChangeGroup> group_; + Body change_; +}; + +/** + * @brief One current state item: a resource or session as it is now. + */ +class MetadataCurrentItem final { + public: + /** @brief The typed body of an item. Each group has its own type. */ + using Body = std::variant<MetadataStreamChange, + MetadataTopicItem, + MetadataPartitionsItem, + MetadataConsumerGroupItem, + MetadataMemberItem, + MetadataUserChange, + MetadataTokenChange, + MetadataSession, + MetadataUnknownItem, + MetadataUnknownJsonItem>; + + /** + * @brief Decodes one item as a binary response carries it. + * @param bytes Item envelope and body. + * @return Decoded item. A group this build does not know is kept as + * MetadataUnknownItem. + * @throws IggyException with Kind() invalid_format when @p bytes are malformed. + */ + static MetadataCurrentItem FromBytes(const std::vector<std::uint8_t> &bytes); + + /** @brief Returns the group, absent for an item this build does not know. */ + [[nodiscard]] std::optional<MetadataChangeGroup> Group() const noexcept { return group_; } + + /** @brief Returns the typed body. */ + [[nodiscard]] const Body &Value() const noexcept { return value_; } + + private: + MetadataCurrentItem(std::optional<MetadataChangeGroup> group, Body value) + : group_(group), value_(std::move(value)) {} + + static MetadataCurrentItem FromFfi(ffi::MetadataCurrentItem item); + static Body BodyFromFfi(MetadataChangeGroup group, ffi::MetadataChangeBody &body); + + friend class MetadataChanges; + + std::optional<MetadataChangeGroup> group_; + Body value_; +}; + +/** + * @brief One page of the metadata change log, or of the current state. + * + * IggyBlockingClient::GetMetadataChanges() returns this value. On + * MetadataChangeStatus::Ok keep Next() and pass it to the following read. On + * Behind retry later, on Expired or Reset rebuild from the current state. + */ +class MetadataChanges final { + public: + /** + * @brief Decodes a response payload as the binary transports carry it. + * @param payload Response payload without framing. + * @return Decoded page. + * @throws IggyException with Kind() invalid_format when @p payload is malformed. + */ + static MetadataChanges FromBytes(const std::vector<std::uint8_t> &payload); + + /** @brief Returns the outcome of the query. */ + [[nodiscard]] MetadataChangeStatus Status() const noexcept { return status_; } + + /** @brief Returns whether the page carries records or current state items. */ + [[nodiscard]] MetadataChangePhase Phase() const noexcept { return phase_; } + + /** @brief Returns the node's log ID, absent before it applied op 1. */ + [[nodiscard]] std::optional<absl::uint128> LogId() const noexcept { return log_id_; } + + /** @brief Returns the lowest whole operation from which the requested groups are complete. */ + [[nodiscard]] std::uint64_t OldestOp() const noexcept { return oldest_op_; } + + /** + * @brief Returns the timestamp above which the requested groups are complete. + * @return Microseconds, zero when nothing was evicted. + */ + [[nodiscard]] std::uint64_t RetainedSince() const noexcept { return retained_since_; } + + /** @brief Returns the highest metadata operation the node applied. */ + [[nodiscard]] std::uint64_t HeadOp() const noexcept { return head_op_; } + + /** + * @brief Returns the cursor to send back. + * @return Present on MetadataChangeStatus::Ok, echoed on Behind, absent on + * Expired and Reset. The last current state page carries the + * anchor to follow the change log from. + */ + [[nodiscard]] const std::optional<MetadataChangeCursor> &Next() const noexcept { return next_; } + + /** @brief Returns whether more records remain after Next(), or another state page exists. */ + [[nodiscard]] bool HasMore() const noexcept { return has_more_; } + + /** @brief Returns the change records of a changes page. */ + [[nodiscard]] const std::vector<MetadataChangeRecord> &Records() const noexcept { return records_; } + + /** @brief Returns the items of a current state page. */ + [[nodiscard]] const std::vector<MetadataCurrentItem> &Items() const noexcept { return items_; } + + /** @brief Returns the continuation of a current state scan, absent once it is complete. */ + [[nodiscard]] const std::optional<MetadataCurrentPageToken> &CurrentPageToken() const noexcept { + return current_page_token_; + } + + private: + MetadataChanges() = default; + + static MetadataChanges FromFfi(ffi::MetadataChanges changes); + + friend class IggyBlockingClient; + + MetadataChangeStatus status_{MetadataChangeStatus::Ok}; + MetadataChangePhase phase_{MetadataChangePhase::Changes}; + std::optional<absl::uint128> log_id_; + std::uint64_t oldest_op_{}; + std::uint64_t retained_since_{}; + std::uint64_t head_op_{}; + std::optional<MetadataChangeCursor> next_; + bool has_more_{}; + std::vector<MetadataChangeRecord> records_; + std::vector<MetadataCurrentItem> items_; + std::optional<MetadataCurrentPageToken> current_page_token_; +}; + +/** * @brief Starting position for polling messages. * * @note The strategy kind and value are passed across the Rust FFI as a pair. @@ -4048,10 +5700,13 @@ * - CreateConsumerGroup(), DeleteConsumerGroup(), JoinConsumerGroup(), and * LeaveConsumerGroup(). * - StoreConsumerOffset(), StoreConsumerPosition(), and DeleteConsumerOffset(). + * - GetMetadataChanges(). * * Other methods report errors through the exception message only. C++ and * bridge-local argument validation failures also have no structured error - * details, even for the methods listed above. + * details, even for the methods listed above. GetMetadataChanges() is the + * exception: it reports every invalid query as invalid_command, including one + * the bridge refuses. * * @code{.cpp} * auto client{iggy::IggyBlockingClient::Builder() @@ -4900,6 +6555,25 @@ std::vector<OptionSpec> DescribeOptions(OptionsScope scope); /** + * @brief Reads one page of the metadata change log or of the current state. + * + * The node records one change per public result of a committed metadata + * operation, such as a stream created, a topic renamed, a user's + * permissions changed, or a session closed. A mirror keeps a cursor, reads + * the records after it, and stores the returned MetadataChanges::Next(). + * Records carry option values as the read routes return them and never + * key material. + * + * @param query Start position, groups, time bounds, and page size. + * @return One page of records or current state items. + * @throws IggyException with Kind() invalid_command when @p query is + * invalid, before anything is sent; with Kind() unauthorized when + * the caller is neither root nor holds the global manage_servers + * permission; or when the request fails. + */ + MetadataChanges GetMetadataChanges(const MetadataChangesQuery &query); + + /** * @brief Retrieves server process, storage, and resource statistics. * * The returned fields are an observed snapshot and can change immediately. @@ -4952,22 +6626,6 @@ private: explicit IggyBlockingClient(ffi::Client *client); - template <typename Operation> - static decltype(auto) RethrowAsIggyException(Operation &&operation) { - ffi::ErrorInfo error_info{}; - try { - if constexpr (std::is_invocable_v<Operation, ffi::ErrorInfo &>) { - return std::forward<Operation>(operation)(error_info); - } else { - return std::forward<Operation>(operation)(); - } - } catch (const IggyException &) { - throw; - } catch (const std::exception &error) { - throw IggyException(error.what(), error_info); - } - } - [[nodiscard]] ffi::Client *Handle() const; void Reset() noexcept;
diff --git a/foreign/cpp/src/client.cpp b/foreign/cpp/src/client.cpp index 7be218a..15ea8b8 100644 --- a/foreign/cpp/src/client.cpp +++ b/foreign/cpp/src/client.cpp
@@ -21,6 +21,8 @@ namespace iggy { +using detail::RethrowAsIggyException; + namespace { rust::Vec<ffi::HeaderEntry> ToFfiRawOptions(const std::map<std::string, std::string> &raw_options) { @@ -520,6 +522,12 @@ }); } +MetadataChanges IggyBlockingClient::GetMetadataChanges(const MetadataChangesQuery &query) { + return RethrowAsIggyException([this, &query](ffi::ErrorInfo &error_info) { + return MetadataChanges::FromFfi(Handle()->get_metadata_changes(query.ToFfi(), error_info)); + }); +} + Stats IggyBlockingClient::GetStats() { return RethrowAsIggyException([this] { return Stats::FromFfi(Handle()->get_stats()); }); } @@ -547,25 +555,68 @@ } std::string EncryptionKeyFingerprint(const std::vector<std::uint8_t> &key) { - ffi::ErrorInfo error_info{}; - try { + return RethrowAsIggyException([&key](ffi::ErrorInfo &error_info) { const auto fingerprint = ffi::encryption_key_fingerprint(rust::Slice<const std::uint8_t>(key.data(), key.size()), error_info); return std::string(fingerprint.data(), fingerprint.size()); - } catch (const std::exception &error) { - throw IggyException(error.what(), error_info); - } + }); } std::string EncryptionKeyFingerprint(const std::string &key_base64) { - ffi::ErrorInfo error_info{}; - try { + return RethrowAsIggyException([&key_base64](ffi::ErrorInfo &error_info) { const auto fingerprint = ffi::encryption_key_fingerprint_base64(rust::Str(key_base64.data(), key_base64.size()), error_info); return std::string(fingerprint.data(), fingerprint.size()); - } catch (const std::exception &error) { - throw IggyException(error.what(), error_info); - } + }); +} + +MetadataChangeCursor MetadataChangeCursor::Parse(const std::string &text) { + return RethrowAsIggyException([&text](ffi::ErrorInfo &error_info) { + return FromFfi(ffi::parse_metadata_change_cursor( + rust::Slice<const std::uint8_t>(reinterpret_cast<const std::uint8_t *>(text.data()), text.size()), + error_info)); + }); +} + +MetadataCurrentPageToken MetadataCurrentPageToken::Parse(const std::string &text) { + return RethrowAsIggyException([&text](ffi::ErrorInfo &error_info) { + return FromFfi(ffi::parse_metadata_current_page_token( + rust::Slice<const std::uint8_t>(reinterpret_cast<const std::uint8_t *>(text.data()), text.size()), + error_info)); + }); +} + +void MetadataChangesQuery::Validate() const { + RethrowAsIggyException( + [this](ffi::ErrorInfo &error_info) { ffi::validate_metadata_changes_query(ToFfi(), error_info); }); +} + +std::vector<std::uint8_t> MetadataChangesQuery::ToBytes() const { + return RethrowAsIggyException([this](ffi::ErrorInfo &error_info) { + const auto bytes = ffi::encode_metadata_changes_query(ToFfi(), error_info); + return std::vector<std::uint8_t>(bytes.begin(), bytes.end()); + }); +} + +MetadataChanges MetadataChanges::FromBytes(const std::vector<std::uint8_t> &payload) { + return RethrowAsIggyException([&payload](ffi::ErrorInfo &error_info) { + return FromFfi( + ffi::decode_metadata_changes(rust::Slice<const std::uint8_t>(payload.data(), payload.size()), error_info)); + }); +} + +MetadataChangeRecord MetadataChangeRecord::FromBytes(const std::vector<std::uint8_t> &bytes) { + return RethrowAsIggyException([&bytes](ffi::ErrorInfo &error_info) { + return FromFfi(ffi::decode_metadata_change_record(rust::Slice<const std::uint8_t>(bytes.data(), bytes.size()), + error_info)); + }); +} + +MetadataCurrentItem MetadataCurrentItem::FromBytes(const std::vector<std::uint8_t> &bytes) { + return RethrowAsIggyException([&bytes](ffi::ErrorInfo &error_info) { + return FromFfi( + ffi::decode_metadata_current_item(rust::Slice<const std::uint8_t>(bytes.data(), bytes.size()), error_info)); + }); } IggyBlockingClient::Builder::Builder() = default; @@ -669,7 +720,7 @@ } IggyBlockingClient IggyBlockingClient::Builder::Build() const { - return IggyBlockingClient::RethrowAsIggyException([this] { + return RethrowAsIggyException([this] { ffi::IggyClientConfig config{}; config.server_address = server_address_; config.auto_login_kind = auto_login_kind_;
diff --git a/foreign/cpp/src/client.rs b/foreign/cpp/src/client.rs index b9a62e9..0b96925 100644 --- a/foreign/cpp/src/client.rs +++ b/foreign/cpp/src/client.rs
@@ -32,8 +32,15 @@ UserClient, UserCreateOptions as RustUserCreateOptions, UserStatus as RustUserStatus, UserUpdateOptions as RustUserUpdateOptions, }; +use iggy_binary_protocol::requests::system::GetMetadataChangesRequest; +use iggy_binary_protocol::responses::system::{ + GetMetadataChangesResponse, WireMetadataChangeRecord, WireMetadataItem, +}; +use iggy_binary_protocol::{WireDecode, WireEncode}; use iggy_common::{ - Credentials as RustCredentials, IggyError, PendingOperationsConfig, stream_key_fingerprint, + Credentials as RustCredentials, IggyError, MetadataChangeCursor, MetadataChangeRecord, + MetadataChanges, MetadataChangesQuery, MetadataCurrentItem, MetadataCurrentPageToken, + PendingOperationsConfig, stream_key_fingerprint, }; use serde::Deserialize; use std::collections::HashSet; @@ -1184,6 +1191,31 @@ }) } + /// Reads one page of the metadata change log or of the current state. + /// + /// The Rust SDK validates the query before anything is sent, so a refused + /// query and a refusal by the server both carry their error code. + pub fn get_metadata_changes( + &self, + query: ffi::MetadataChangesQuery, + error_info: &mut ffi::ErrorInfo, + ) -> Result<ffi::MetadataChanges, String> { + *error_info = ffi::ErrorInfo::default(); + let query = MetadataChangesQuery::try_from(query) + .inspect_err(|error| error_info.record(error)) + .map_err(|error| format!("Could not get metadata changes: {error}"))?; + + RUNTIME.block_on(async { + let changes = self + .inner + .get_metadata_changes(&query) + .await + .inspect_err(|error| error_info.record(error)) + .map_err(|error| format!("Could not get metadata changes: {error}"))?; + Ok(ffi::MetadataChanges::from(changes)) + }) + } + pub fn ping(&self) -> Result<(), String> { RUNTIME.block_on(async { self.inner @@ -1488,6 +1520,95 @@ .map_err(|error| format!("Could not compute the stream key fingerprint: {error}")) } +/// Parses the `mc1_` text form. The error never echoes the text. +pub fn parse_metadata_change_cursor( + text: &[u8], + error_info: &mut ffi::ErrorInfo, +) -> Result<ffi::MetadataChangeCursor, String> { + *error_info = ffi::ErrorInfo::default(); + str::from_utf8(text) + .map_err(|_| IggyError::InvalidCommand) + .and_then(MetadataChangeCursor::from_str) + .map(ffi::MetadataChangeCursor::from) + .inspect_err(|error| error_info.record(error)) + .map_err(|error| format!("Could not parse the metadata change cursor: {error}")) +} + +/// Parses the `cs1_` text form. The error never echoes the text. +pub fn parse_metadata_current_page_token( + text: &[u8], + error_info: &mut ffi::ErrorInfo, +) -> Result<ffi::MetadataCurrentPageToken, String> { + *error_info = ffi::ErrorInfo::default(); + str::from_utf8(text) + .map_err(|_| IggyError::InvalidCommand) + .and_then(MetadataCurrentPageToken::from_str) + .map(ffi::MetadataCurrentPageToken::from) + .inspect_err(|error| error_info.record(error)) + .map_err(|error| format!("Could not parse the metadata current page token: {error}")) +} + +pub fn validate_metadata_changes_query( + query: ffi::MetadataChangesQuery, + error_info: &mut ffi::ErrorInfo, +) -> Result<(), String> { + *error_info = ffi::ErrorInfo::default(); + MetadataChangesQuery::try_from(query) + .and_then(|query| query.validate().map(|_| ())) + .inspect_err(|error| error_info.record(error)) + .map_err(|error| format!("Invalid metadata changes query: {error}")) +} + +pub fn encode_metadata_changes_query( + query: ffi::MetadataChangesQuery, + error_info: &mut ffi::ErrorInfo, +) -> Result<Vec<u8>, String> { + *error_info = ffi::ErrorInfo::default(); + MetadataChangesQuery::try_from(query) + .map(|query| GetMetadataChangesRequest::from(&query).to_bytes().to_vec()) + .inspect_err(|error| error_info.record(error)) + .map_err(|error| format!("Could not encode the metadata changes query: {error}")) +} + +pub fn decode_metadata_changes( + payload: &[u8], + error_info: &mut ffi::ErrorInfo, +) -> Result<ffi::MetadataChanges, String> { + *error_info = ffi::ErrorInfo::default(); + GetMetadataChangesResponse::decode_from(payload) + .map_err(|_| IggyError::InvalidFormat) + .and_then(MetadataChanges::try_from) + .map(ffi::MetadataChanges::from) + .inspect_err(|error| error_info.record(error)) + .map_err(|error| format!("Could not decode the metadata changes: {error}")) +} + +pub fn decode_metadata_change_record( + bytes: &[u8], + error_info: &mut ffi::ErrorInfo, +) -> Result<ffi::MetadataChangeRecord, String> { + *error_info = ffi::ErrorInfo::default(); + WireMetadataChangeRecord::decode_from(bytes) + .map_err(|_| IggyError::InvalidFormat) + .and_then(MetadataChangeRecord::try_from) + .map(ffi::MetadataChangeRecord::from) + .inspect_err(|error| error_info.record(error)) + .map_err(|error| format!("Could not decode the metadata change record: {error}")) +} + +pub fn decode_metadata_current_item( + bytes: &[u8], + error_info: &mut ffi::ErrorInfo, +) -> Result<ffi::MetadataCurrentItem, String> { + *error_info = ffi::ErrorInfo::default(); + WireMetadataItem::decode_from(bytes) + .map_err(|_| IggyError::InvalidFormat) + .and_then(MetadataCurrentItem::try_from) + .map(ffi::MetadataCurrentItem::from) + .inspect_err(|error| error_info.record(error)) + .map_err(|error| format!("Could not decode the metadata current item: {error}")) +} + fn is_option_error(error: &IggyError) -> bool { matches!( error,
diff --git a/foreign/cpp/src/lib.rs b/foreign/cpp/src/lib.rs index 055a8cc..185c059 100644 --- a/foreign/cpp/src/lib.rs +++ b/foreign/cpp/src/lib.rs
@@ -23,8 +23,11 @@ mod type_conversion; use client::{ - Client, delete_connection as delete_client, encryption_key_fingerprint, + Client, decode_metadata_change_record, decode_metadata_changes, decode_metadata_current_item, + delete_connection as delete_client, encode_metadata_changes_query, encryption_key_fingerprint, encryption_key_fingerprint_base64, from_connection_string, new_connection, + parse_metadata_change_cursor, parse_metadata_current_page_token, + validate_metadata_changes_query, }; use consumer::Consumer; use messages::make_message; @@ -577,6 +580,150 @@ raw_options: Vec<HeaderEntry>, } + /// Where a metadata change query starts. The values are the wire codes. + #[repr(u8)] + enum MetadataChangeStart { + Earliest = 0, + Latest = 1, + After = 2, + Current = 3, + } + + /// Position of the last change record a caller consumed, with its `mc1_` + /// text form. + #[derive(Default)] + struct MetadataChangeCursor { + log_id_lo: u64, + log_id_hi: u64, + op: u64, + checksum_lo: u64, + checksum_hi: u64, + ordinal: u32, + text: String, + } + + /// Continuation of a current state scan, with its `cs1_` text form. + #[derive(Default)] + struct MetadataCurrentPageToken { + text: String, + anchor: MetadataChangeCursor, + required_head: u64, + groups_mask: u8, + last_group: u8, + last_key: Vec<u8>, + } + + /// A metadata change query. Cursor and page token travel as their text + /// forms, an empty page token starts a current state scan. + struct MetadataChangesQuery { + start: MetadataChangeStart, + cursor: String, + page_token: String, + groups: Vec<u8>, + has_since: bool, + since: u64, + has_until: bool, + until: u64, + limit: u32, + } + + /// Whether this build knows a record's or item's group and kind. + #[repr(u8)] + enum MetadataChangeForm { + Known = 0, + /// A binary body this build does not know, kept in `unknown_body`. + Unknown = 1, + /// A JSON object this build does not know, kept whole in `unknown_body`. + UnknownJson = 2, + } + + /// Every field a record or item body can carry. A body fills the fields + /// its kind carries and leaves the rest at zero. + struct MetadataChangeBody { + stream_id: u32, + stream_created_op: u64, + topic_id: u32, + topic_created_op: u64, + group_id: u32, + has_member_id: bool, + member_id: u32, + partition_id: u32, + created_at: u64, + /// The stream, topic, consumer group or token name, or the username. + name: String, + partitions_count: u32, + consumer_groups_count: u32, + members_count: u32, + client_id_lo: u64, + client_id_hi: u64, + /// Removed ids of a partition count change, installed ids of a member. + partition_ids: Vec<u32>, + has_requested_op: bool, + requested_op: u64, + /// Wire code of a removal or permission change reason. + reason: u8, + user_id: u32, + user_created_at: u64, + status: UserStatus, + has_permissions: bool, + permissions: Permissions, + /// False when the server holds option state this build cannot read. + options_readable: bool, + options: Vec<HeaderEntry>, + derived_options: Vec<HeaderEntry>, + has_expiry: bool, + expiry: u64, + session: u64, + unknown_body: Vec<u8>, + } + + /// One change record. `group` and `kind` are wire codes, the names of a + /// JSON record this build does not know are in `group_name` and + /// `kind_name`. + struct MetadataChangeRecord { + op: u64, + ordinal: u32, + checksum_lo: u64, + checksum_hi: u64, + timestamp: u64, + has_actor_user_id: bool, + actor_user_id: u32, + form: MetadataChangeForm, + group: u8, + kind: u8, + group_name: String, + kind_name: String, + body: MetadataChangeBody, + } + + /// One current state item. See [`MetadataChangeRecord`] for the codes. + struct MetadataCurrentItem { + form: MetadataChangeForm, + group: u8, + group_name: String, + body: MetadataChangeBody, + } + + /// One page of the change log or of the current state. `status` and + /// `phase` are wire codes. + struct MetadataChanges { + status: u8, + phase: u8, + has_log_id: bool, + log_id_lo: u64, + log_id_hi: u64, + oldest_op: u64, + retained_since: u64, + head_op: u64, + has_next: bool, + next: MetadataChangeCursor, + has_more: bool, + records: Vec<MetadataChangeRecord>, + items: Vec<MetadataCurrentItem>, + has_current_page_token: bool, + current_page_token: MetadataCurrentPageToken, + } + extern "Rust" { type Client; type Consumer; @@ -800,6 +947,14 @@ /// "user". A scope with no keys yet answers with an empty vector, which /// is an empty catalog rather than a failure. fn describe_options(self: &Client, scope: String) -> Result<Vec<OptionSpec>>; + /// Reads one page of the metadata change log or of the current state. + /// Needs root or the global manage_servers permission. An invalid + /// query fails with InvalidCommand (3) before anything is sent. + fn get_metadata_changes( + self: &Client, + query: MetadataChangesQuery, + error_info: &mut ErrorInfo, + ) -> Result<MetadataChanges>; /// Fingerprint of a 32-byte stream key, computed locally. fn encryption_key_fingerprint(key: &[u8], error_info: &mut ErrorInfo) -> Result<String>; /// Fingerprint of a stream key given as canonical padded base64. @@ -807,6 +962,36 @@ key: &str, error_info: &mut ErrorInfo, ) -> Result<String>; + fn parse_metadata_change_cursor( + text: &[u8], + error_info: &mut ErrorInfo, + ) -> Result<MetadataChangeCursor>; + fn parse_metadata_current_page_token( + text: &[u8], + error_info: &mut ErrorInfo, + ) -> Result<MetadataCurrentPageToken>; + fn validate_metadata_changes_query( + query: MetadataChangesQuery, + error_info: &mut ErrorInfo, + ) -> Result<()>; + /// The binary request payload of a query, without validating it. + fn encode_metadata_changes_query( + query: MetadataChangesQuery, + error_info: &mut ErrorInfo, + ) -> Result<Vec<u8>>; + /// Decodes a response payload as the binary transports carry it. + fn decode_metadata_changes( + payload: &[u8], + error_info: &mut ErrorInfo, + ) -> Result<MetadataChanges>; + fn decode_metadata_change_record( + bytes: &[u8], + error_info: &mut ErrorInfo, + ) -> Result<MetadataChangeRecord>; + fn decode_metadata_current_item( + bytes: &[u8], + error_info: &mut ErrorInfo, + ) -> Result<MetadataCurrentItem>; fn ping(self: &Client) -> Result<()>; fn heartbeat_interval(self: &Client) -> u64; fn snapshot(
diff --git a/foreign/cpp/src/type_conversion.rs b/foreign/cpp/src/type_conversion.rs index e712eb3..ba5dbf2 100644 --- a/foreign/cpp/src/type_conversion.rs +++ b/foreign/cpp/src/type_conversion.rs
@@ -39,6 +39,11 @@ UserUpdateOptions as RustUserUpdateOptions, Validatable, }; use iggy_binary_protocol::WireUserHeaders; +use iggy_binary_protocol::primitives::metadata_changes::{ + WireMemberLeftReason, WireMetadataChangeGroup, WireMetadataChangeKind, WireMetadataChangePhase, + WireMetadataChangeStatus, WirePermissionsUpdatedReason, WireSessionClosedReason, + WireTokenDeletedReason, +}; use iggy_common::{ CacheMetrics as RustCacheMetrics, CacheMetricsKey as RustCacheMetricsKey, ClientInfo as RustClientInfo, ClientInfoDetails as RustClientInfoDetails, @@ -47,7 +52,15 @@ ConsumerGroupMember as RustConsumerGroupMember, ConsumerOffsetInfo as RustConsumerOffsetInfo, GlobalPermissions as RustGlobalPermissions, HeaderEntry as RustHeaderEntry, HeaderField as RustHeaderField, HeaderKind as RustHeaderKind, IdentityInfo as RustIdentityInfo, - IggyError, Permissions as RustPermissions, Stats as RustStats, + IggyError, IggyTimestamp, MetadataChange, MetadataChangeCursor, MetadataChangeGroup, + MetadataChangeRecord, MetadataChangeStart, MetadataChanges, MetadataChangesQuery, + MetadataConsumerGroupCreated, MetadataConsumerGroupDeleted, MetadataConsumerGroupItem, + MetadataCurrentItem, MetadataCurrentPageToken, MetadataMemberItem, MetadataMemberJoined, + MetadataMemberLeft, MetadataOptions, MetadataPartitionHandedOver, MetadataPartitionsChanged, + MetadataPartitionsItem, MetadataSession, MetadataSessionClosed, MetadataStreamChange, + MetadataStreamDeleted, MetadataTokenChange, MetadataTokenDeleted, MetadataTopicChange, + MetadataTopicDeleted, MetadataTopicItem, MetadataUserChange, MetadataUserIdentity, + MetadataUserPermissionsUpdated, Permissions as RustPermissions, Stats as RustStats, StreamPermissions as RustStreamPermissions, TopicPermissions as RustTopicPermissions, TransportEndpoints as RustTransportEndpoints, UserInfo as RustUserInfo, UserInfoDetails as RustUserInfoDetails, UserStatus as RustUserStatus, @@ -1187,6 +1200,581 @@ } } +impl TryFrom<ffi::MetadataChangesQuery> for MetadataChangesQuery { + type Error = IggyError; + + fn try_from(query: ffi::MetadataChangesQuery) -> Result<Self, Self::Error> { + let timestamp = + |micros| IggyTimestamp::checked_from_micros(micros).ok_or(IggyError::InvalidCommand); + let start = match query.start { + ffi::MetadataChangeStart::Earliest => MetadataChangeStart::Earliest, + ffi::MetadataChangeStart::Latest => MetadataChangeStart::Latest, + ffi::MetadataChangeStart::After => { + MetadataChangeStart::After(MetadataChangeCursor::from_str(&query.cursor)?) + } + ffi::MetadataChangeStart::Current if query.page_token.is_empty() => { + MetadataChangeStart::Current(None) + } + ffi::MetadataChangeStart::Current => MetadataChangeStart::Current(Some( + MetadataCurrentPageToken::from_str(&query.page_token)?, + )), + _ => return Err(IggyError::InvalidCommand), + }; + let groups = query + .groups + .iter() + .map(|&code| { + WireMetadataChangeGroup::from_code(code) + .map(MetadataChangeGroup::from) + .map_err(|_| IggyError::InvalidCommand) + }) + .collect::<Result<_, _>>()?; + Ok(Self { + start, + groups, + since: query + .has_since + .then(|| timestamp(query.since)) + .transpose()?, + until: query + .has_until + .then(|| timestamp(query.until)) + .transpose()?, + limit: query.limit, + }) + } +} + +impl From<MetadataChangeCursor> for ffi::MetadataChangeCursor { + fn from(cursor: MetadataChangeCursor) -> Self { + ffi::MetadataChangeCursor { + log_id_lo: cursor.log_id as u64, + log_id_hi: (cursor.log_id >> 64) as u64, + op: cursor.op, + checksum_lo: cursor.checksum as u64, + checksum_hi: (cursor.checksum >> 64) as u64, + ordinal: cursor.ordinal, + text: cursor.to_string(), + } + } +} + +impl From<MetadataCurrentPageToken> for ffi::MetadataCurrentPageToken { + fn from(token: MetadataCurrentPageToken) -> Self { + ffi::MetadataCurrentPageToken { + text: token.to_string(), + anchor: token.anchor().into(), + required_head: token.required_head(), + groups_mask: token.groups().mask(), + last_group: WireMetadataChangeGroup::from(token.last_group()).as_u8(), + last_key: token.last_key().to_vec(), + } + } +} + +impl From<MetadataChanges> for ffi::MetadataChanges { + fn from(changes: MetadataChanges) -> Self { + let log_id = changes.log_id.unwrap_or_default(); + ffi::MetadataChanges { + status: WireMetadataChangeStatus::from(changes.status).as_u8(), + phase: WireMetadataChangePhase::from(changes.phase).as_u8(), + has_log_id: changes.log_id.is_some(), + log_id_lo: log_id as u64, + log_id_hi: (log_id >> 64) as u64, + oldest_op: changes.oldest_op, + retained_since: changes.retained_since, + head_op: changes.head_op, + has_next: changes.next.is_some(), + next: changes.next.map(Into::into).unwrap_or_default(), + has_more: changes.has_more, + records: changes + .records + .into_iter() + .map(ffi::MetadataChangeRecord::from) + .collect(), + items: changes + .items + .into_iter() + .map(ffi::MetadataCurrentItem::from) + .collect(), + has_current_page_token: changes.current_page_token.is_some(), + current_page_token: changes + .current_page_token + .map(Into::into) + .unwrap_or_default(), + } + } +} + +impl From<MetadataChangeRecord> for ffi::MetadataChangeRecord { + fn from(record: MetadataChangeRecord) -> Self { + let mut converted = ffi::MetadataChangeRecord { + op: record.op, + ordinal: record.ordinal, + checksum_lo: record.checksum as u64, + checksum_hi: (record.checksum >> 64) as u64, + timestamp: record.timestamp, + has_actor_user_id: record.actor_user_id.is_some(), + actor_user_id: record.actor_user_id.unwrap_or_default(), + form: ffi::MetadataChangeForm::Known, + group: 0, + kind: 0, + group_name: String::new(), + kind_name: String::new(), + body: ffi::MetadataChangeBody::default(), + }; + if let Some(kind) = record.change.kind() { + converted.kind = WireMetadataChangeKind::from(kind).as_u8(); + converted.group = WireMetadataChangeGroup::from(kind.group()).as_u8(); + } + converted.body = match record.change { + MetadataChange::StreamCreated(body) | MetadataChange::StreamUpdated(body) => { + body.into() + } + MetadataChange::StreamDeleted(body) => body.into(), + MetadataChange::TopicCreated(body) | MetadataChange::TopicUpdated(body) => body.into(), + MetadataChange::TopicDeleted(body) => body.into(), + MetadataChange::PartitionsChanged(body) => body.into(), + MetadataChange::ConsumerGroupCreated(body) => body.into(), + MetadataChange::ConsumerGroupDeleted(body) => body.into(), + MetadataChange::ConsumerGroupMemberJoined(body) => body.into(), + MetadataChange::ConsumerGroupMemberLeft(body) => body.into(), + MetadataChange::ConsumerGroupPartitionHandedOver(body) => body.into(), + MetadataChange::UserCreated(body) | MetadataChange::UserUpdated(body) => body.into(), + MetadataChange::UserDeleted(body) | MetadataChange::UserPasswordChanged(body) => { + body.into() + } + MetadataChange::UserPermissionsUpdated(body) => body.into(), + MetadataChange::PersonalAccessTokenCreated(body) => body.into(), + MetadataChange::PersonalAccessTokenDeleted(body) => body.into(), + MetadataChange::SessionRegistered(body) => body.into(), + MetadataChange::SessionClosed(body) => body.into(), + MetadataChange::Unknown { group, kind, body } => { + converted.form = ffi::MetadataChangeForm::Unknown; + converted.group = group; + converted.kind = kind; + ffi::MetadataChangeBody { + unknown_body: body.to_vec(), + ..ffi::MetadataChangeBody::default() + } + } + MetadataChange::UnknownJson { group, kind, body } => { + converted.form = ffi::MetadataChangeForm::UnknownJson; + converted.group_name = group; + converted.kind_name = kind; + ffi::MetadataChangeBody { + unknown_body: body.get().as_bytes().to_vec(), + ..ffi::MetadataChangeBody::default() + } + } + }; + converted + } +} + +impl From<MetadataCurrentItem> for ffi::MetadataCurrentItem { + fn from(item: MetadataCurrentItem) -> Self { + let group = item + .group() + .map_or(0, |group| WireMetadataChangeGroup::from(group).as_u8()); + let known = |body: ffi::MetadataChangeBody| ffi::MetadataCurrentItem { + form: ffi::MetadataChangeForm::Known, + group, + group_name: String::new(), + body, + }; + match item { + MetadataCurrentItem::Stream(body) => known(body.into()), + MetadataCurrentItem::Topic(body) => known(body.into()), + MetadataCurrentItem::Partitions(body) => known(body.into()), + MetadataCurrentItem::ConsumerGroup(body) => known(body.into()), + MetadataCurrentItem::ConsumerGroupMember(body) => known(body.into()), + MetadataCurrentItem::User(body) => known(body.into()), + MetadataCurrentItem::PersonalAccessToken(body) => known(body.into()), + MetadataCurrentItem::Session(body) => known(body.into()), + MetadataCurrentItem::Unknown { group, body } => ffi::MetadataCurrentItem { + form: ffi::MetadataChangeForm::Unknown, + group, + group_name: String::new(), + body: ffi::MetadataChangeBody { + unknown_body: body.to_vec(), + ..ffi::MetadataChangeBody::default() + }, + }, + MetadataCurrentItem::UnknownJson { group, body } => ffi::MetadataCurrentItem { + form: ffi::MetadataChangeForm::UnknownJson, + group: 0, + group_name: group, + body: ffi::MetadataChangeBody { + unknown_body: body.get().as_bytes().to_vec(), + ..ffi::MetadataChangeBody::default() + }, + }, + } + } +} + +impl Default for ffi::MetadataChangeBody { + fn default() -> Self { + ffi::MetadataChangeBody { + stream_id: 0, + stream_created_op: 0, + topic_id: 0, + topic_created_op: 0, + group_id: 0, + has_member_id: false, + member_id: 0, + partition_id: 0, + created_at: 0, + name: String::new(), + partitions_count: 0, + consumer_groups_count: 0, + members_count: 0, + client_id_lo: 0, + client_id_hi: 0, + partition_ids: Vec::new(), + has_requested_op: false, + requested_op: 0, + reason: 0, + user_id: 0, + user_created_at: 0, + status: ffi::UserStatus::Active, + has_permissions: false, + permissions: ffi::Permissions::from(RustPermissions::default()), + options_readable: false, + options: Vec::new(), + derived_options: Vec::new(), + has_expiry: false, + expiry: 0, + session: 0, + unknown_body: Vec::new(), + } + } +} + +impl From<MetadataOptions> for ffi::MetadataChangeBody { + fn from(options: MetadataOptions) -> Self { + match options { + MetadataOptions::Readable(options) => ffi::MetadataChangeBody { + options_readable: true, + options: resource_options_to_ffi(&options, true), + derived_options: resource_options_to_ffi(&options, false), + ..ffi::MetadataChangeBody::default() + }, + MetadataOptions::Unreadable => ffi::MetadataChangeBody::default(), + } + } +} + +impl From<MetadataStreamChange> for ffi::MetadataChangeBody { + fn from(body: MetadataStreamChange) -> Self { + ffi::MetadataChangeBody { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + created_at: body.created_at, + name: body.name, + ..body.options.into() + } + } +} + +impl From<MetadataStreamDeleted> for ffi::MetadataChangeBody { + fn from(body: MetadataStreamDeleted) -> Self { + ffi::MetadataChangeBody { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + created_at: body.created_at, + name: body.name, + has_requested_op: body.requested_op.is_some(), + requested_op: body.requested_op.unwrap_or_default(), + ..ffi::MetadataChangeBody::default() + } + } +} + +impl From<MetadataTopicChange> for ffi::MetadataChangeBody { + fn from(body: MetadataTopicChange) -> Self { + ffi::MetadataChangeBody { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + created_at: body.created_at, + name: body.name, + partitions_count: body.partitions_count, + ..body.options.into() + } + } +} + +impl From<MetadataTopicItem> for ffi::MetadataChangeBody { + fn from(body: MetadataTopicItem) -> Self { + ffi::MetadataChangeBody { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + created_at: body.created_at, + name: body.name, + ..body.options.into() + } + } +} + +impl From<MetadataTopicDeleted> for ffi::MetadataChangeBody { + fn from(body: MetadataTopicDeleted) -> Self { + ffi::MetadataChangeBody { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + created_at: body.created_at, + name: body.name, + has_requested_op: body.requested_op.is_some(), + requested_op: body.requested_op.unwrap_or_default(), + ..ffi::MetadataChangeBody::default() + } + } +} + +impl From<MetadataPartitionsChanged> for ffi::MetadataChangeBody { + fn from(body: MetadataPartitionsChanged) -> Self { + ffi::MetadataChangeBody { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + partitions_count: body.partitions_count, + partition_ids: body.removed_partition_ids, + has_requested_op: body.requested_op.is_some(), + requested_op: body.requested_op.unwrap_or_default(), + ..ffi::MetadataChangeBody::default() + } + } +} + +impl From<MetadataPartitionsItem> for ffi::MetadataChangeBody { + fn from(body: MetadataPartitionsItem) -> Self { + ffi::MetadataChangeBody { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + partitions_count: body.partitions_count, + ..ffi::MetadataChangeBody::default() + } + } +} + +impl From<MetadataConsumerGroupCreated> for ffi::MetadataChangeBody { + fn from(body: MetadataConsumerGroupCreated) -> Self { + ffi::MetadataChangeBody { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + group_id: body.group_id, + name: body.name, + consumer_groups_count: body.consumer_groups_count, + ..ffi::MetadataChangeBody::default() + } + } +} + +impl From<MetadataConsumerGroupDeleted> for ffi::MetadataChangeBody { + fn from(body: MetadataConsumerGroupDeleted) -> Self { + ffi::MetadataChangeBody { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + group_id: body.group_id, + name: body.name, + consumer_groups_count: body.consumer_groups_count, + has_requested_op: body.requested_op.is_some(), + requested_op: body.requested_op.unwrap_or_default(), + ..ffi::MetadataChangeBody::default() + } + } +} + +impl From<MetadataConsumerGroupItem> for ffi::MetadataChangeBody { + fn from(body: MetadataConsumerGroupItem) -> Self { + ffi::MetadataChangeBody { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + group_id: body.group_id, + name: body.name, + ..ffi::MetadataChangeBody::default() + } + } +} + +impl From<MetadataMemberJoined> for ffi::MetadataChangeBody { + fn from(body: MetadataMemberJoined) -> Self { + ffi::MetadataChangeBody { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + group_id: body.group_id, + has_member_id: true, + member_id: body.member_id, + client_id_lo: body.client_id as u64, + client_id_hi: (body.client_id >> 64) as u64, + members_count: body.members_count, + partition_ids: body.partition_ids, + ..ffi::MetadataChangeBody::default() + } + } +} + +impl From<MetadataMemberLeft> for ffi::MetadataChangeBody { + fn from(body: MetadataMemberLeft) -> Self { + ffi::MetadataChangeBody { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + group_id: body.group_id, + has_member_id: true, + member_id: body.member_id, + client_id_lo: body.client_id as u64, + client_id_hi: (body.client_id >> 64) as u64, + members_count: body.members_count, + reason: WireMemberLeftReason::from(body.reason).as_u8(), + ..ffi::MetadataChangeBody::default() + } + } +} + +impl From<MetadataPartitionHandedOver> for ffi::MetadataChangeBody { + fn from(body: MetadataPartitionHandedOver) -> Self { + ffi::MetadataChangeBody { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + group_id: body.group_id, + partition_id: body.partition_id, + has_member_id: body.member_id.is_some(), + member_id: body.member_id.unwrap_or_default(), + ..ffi::MetadataChangeBody::default() + } + } +} + +impl From<MetadataMemberItem> for ffi::MetadataChangeBody { + fn from(body: MetadataMemberItem) -> Self { + ffi::MetadataChangeBody { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + group_id: body.group_id, + has_member_id: true, + member_id: body.member_id, + client_id_lo: body.client_id as u64, + client_id_hi: (body.client_id >> 64) as u64, + partition_ids: body.partition_ids, + ..ffi::MetadataChangeBody::default() + } + } +} + +impl From<MetadataUserChange> for ffi::MetadataChangeBody { + fn from(body: MetadataUserChange) -> Self { + ffi::MetadataChangeBody { + user_id: body.user_id, + user_created_at: body.user_created_at, + name: body.username, + status: body.status.into(), + has_permissions: body.permissions.is_some(), + permissions: ffi::Permissions::from(body.permissions.unwrap_or_default()), + ..body.options.into() + } + } +} + +impl From<MetadataUserIdentity> for ffi::MetadataChangeBody { + fn from(body: MetadataUserIdentity) -> Self { + ffi::MetadataChangeBody { + user_id: body.user_id, + user_created_at: body.user_created_at, + name: body.username, + ..ffi::MetadataChangeBody::default() + } + } +} + +impl From<MetadataUserPermissionsUpdated> for ffi::MetadataChangeBody { + fn from(body: MetadataUserPermissionsUpdated) -> Self { + ffi::MetadataChangeBody { + user_id: body.user_id, + user_created_at: body.user_created_at, + name: body.username, + has_permissions: body.permissions.is_some(), + permissions: ffi::Permissions::from(body.permissions.unwrap_or_default()), + reason: WirePermissionsUpdatedReason::from(body.reason).as_u8(), + ..ffi::MetadataChangeBody::default() + } + } +} + +impl From<MetadataTokenChange> for ffi::MetadataChangeBody { + fn from(body: MetadataTokenChange) -> Self { + ffi::MetadataChangeBody { + user_id: body.user_id, + user_created_at: body.user_created_at, + name: body.name, + has_expiry: body.expiry.is_some(), + expiry: body.expiry.unwrap_or_default(), + ..ffi::MetadataChangeBody::default() + } + } +} + +impl From<MetadataTokenDeleted> for ffi::MetadataChangeBody { + fn from(body: MetadataTokenDeleted) -> Self { + ffi::MetadataChangeBody { + user_id: body.user_id, + user_created_at: body.user_created_at, + name: body.name, + reason: WireTokenDeletedReason::from(body.reason).as_u8(), + ..ffi::MetadataChangeBody::default() + } + } +} + +impl From<MetadataSession> for ffi::MetadataChangeBody { + fn from(body: MetadataSession) -> Self { + ffi::MetadataChangeBody { + user_id: body.user_id, + user_created_at: body.user_created_at, + client_id_lo: body.client_id as u64, + client_id_hi: (body.client_id >> 64) as u64, + session: body.session, + ..ffi::MetadataChangeBody::default() + } + } +} + +impl From<MetadataSessionClosed> for ffi::MetadataChangeBody { + fn from(body: MetadataSessionClosed) -> Self { + ffi::MetadataChangeBody { + user_id: body.user_id, + user_created_at: body.user_created_at, + client_id_lo: body.client_id as u64, + client_id_hi: (body.client_id >> 64) as u64, + session: body.session, + reason: WireSessionClosedReason::from(body.reason).as_u8(), + ..ffi::MetadataChangeBody::default() + } + } +} + #[cfg(test)] mod tests { use super::*; @@ -1442,4 +2030,75 @@ (3, 5, 7) ); } + + #[test] + fn given_json_record_of_unknown_kind_when_converting_should_keep_names_and_the_whole_object() { + let json = r#"{"op":5,"ordinal":1,"checksum":"0000000000000000000000000000002a","timestamp":7,"group":"future_group","kind":"future_kind","extra":true}"#; + let record: MetadataChangeRecord = serde_json::from_str(json).expect("record parses"); + + let converted = ffi::MetadataChangeRecord::from(record); + + assert!(converted.form == ffi::MetadataChangeForm::UnknownJson); + assert_eq!( + (converted.group_name.as_str(), converted.kind_name.as_str()), + ("future_group", "future_kind") + ); + assert_eq!((converted.op, converted.ordinal), (5, 1)); + assert_eq!((converted.checksum_lo, converted.checksum_hi), (42, 0)); + assert!(!converted.has_actor_user_id); + assert_eq!(converted.body.unknown_body, json.as_bytes()); + } + + #[test] + fn given_json_item_of_unknown_group_when_converting_should_keep_the_whole_object() { + let json = r#"{"group":"future_group","count":3}"#; + let item: MetadataCurrentItem = serde_json::from_str(json).expect("item parses"); + + let converted = ffi::MetadataCurrentItem::from(item); + + assert!(converted.form == ffi::MetadataChangeForm::UnknownJson); + assert_eq!(converted.group_name, "future_group"); + assert_eq!(converted.body.unknown_body, json.as_bytes()); + } + + #[test] + fn given_bridge_query_when_converting_should_map_every_start_mode_and_refuse_unknown_codes() { + let query = |start: ffi::MetadataChangeStart, groups: Vec<u8>| ffi::MetadataChangesQuery { + start, + cursor: String::new(), + page_token: String::new(), + groups, + has_since: false, + since: 9, + has_until: true, + until: 11, + limit: 7, + }; + + let current = + MetadataChangesQuery::try_from(query(ffi::MetadataChangeStart::Current, vec![1])) + .expect("current query"); + assert_eq!( + current, + MetadataChangesQuery { + start: MetadataChangeStart::Current(None), + groups: vec![MetadataChangeGroup::Streams], + since: None, + until: Some(IggyTimestamp::from(11)), + limit: 7, + } + ); + assert_eq!( + MetadataChangesQuery::try_from(query(ffi::MetadataChangeStart { repr: 4 }, Vec::new())), + Err(IggyError::InvalidCommand) + ); + assert_eq!( + MetadataChangesQuery::try_from(query(ffi::MetadataChangeStart::Earliest, vec![9])), + Err(IggyError::InvalidCommand) + ); + assert_eq!( + MetadataChangesQuery::try_from(query(ffi::MetadataChangeStart::After, Vec::new())), + Err(IggyError::InvalidCommand) + ); + } }
diff --git a/foreign/cpp/src/type_conversions.cpp b/foreign/cpp/src/type_conversions.cpp index 20e8a21..523af49 100644 --- a/foreign/cpp/src/type_conversions.cpp +++ b/foreign/cpp/src/type_conversions.cpp
@@ -407,4 +407,452 @@ stats.open_files_count, stats.open_files_limit); } +MetadataChangeCursor MetadataChangeCursor::FromFfi(ffi::MetadataChangeCursor cursor) { + MetadataChangeCursor result; + result.log_id_ = absl::MakeUint128(cursor.log_id_hi, cursor.log_id_lo); + result.op_ = cursor.op; + result.checksum_ = absl::MakeUint128(cursor.checksum_hi, cursor.checksum_lo); + result.ordinal_ = cursor.ordinal; + result.text_ = std::string(cursor.text.data(), cursor.text.size()); + return result; +} + +MetadataCurrentPageToken MetadataCurrentPageToken::FromFfi(ffi::MetadataCurrentPageToken token) { + MetadataCurrentPageToken result; + result.text_ = std::string(token.text.data(), token.text.size()); + result.anchor_ = MetadataChangeCursor::FromFfi(std::move(token.anchor)); + result.required_head_ = token.required_head; + result.groups_mask_ = token.groups_mask; + result.last_group_ = static_cast<MetadataChangeGroup>(token.last_group); + result.last_key_.assign(token.last_key.begin(), token.last_key.end()); + return result; +} + +ffi::MetadataChangesQuery MetadataChangesQuery::ToFfi() const { + ffi::MetadataChangesQuery query{}; + query.start = static_cast<ffi::MetadataChangeStart>(start_); + if (cursor_) { + query.cursor = cursor_->ToString(); + } + if (page_token_) { + query.page_token = page_token_->ToString(); + } + query.groups.reserve(groups_.size()); + for (const auto group : groups_) { + query.groups.push_back(static_cast<std::uint8_t>(group)); + } + query.has_since = since_.has_value(); + query.since = since_.value_or(0); + query.has_until = until_.has_value(); + query.until = until_.value_or(0); + query.limit = limit_; + return query; +} + +MetadataStreamChange MetadataStreamChange::FromFfi(ffi::MetadataChangeBody &body) { + MetadataStreamChange change; + change.stream_id_ = body.stream_id; + change.stream_created_op_ = body.stream_created_op; + change.created_at_ = body.created_at; + change.name_ = std::string(body.name.data(), body.name.size()); + if (body.options_readable) { + change.options_ = ResourceOptions::FromFfi(std::move(body.options), std::move(body.derived_options)); + } + return change; +} + +MetadataStreamDeleted MetadataStreamDeleted::FromFfi(ffi::MetadataChangeBody &body) { + MetadataStreamDeleted change; + change.stream_id_ = body.stream_id; + change.stream_created_op_ = body.stream_created_op; + change.created_at_ = body.created_at; + change.name_ = std::string(body.name.data(), body.name.size()); + if (body.has_requested_op) { + change.requested_op_ = body.requested_op; + } + return change; +} + +MetadataTopicChange MetadataTopicChange::FromFfi(ffi::MetadataChangeBody &body) { + MetadataTopicChange change; + change.stream_id_ = body.stream_id; + change.stream_created_op_ = body.stream_created_op; + change.topic_id_ = body.topic_id; + change.topic_created_op_ = body.topic_created_op; + change.created_at_ = body.created_at; + change.name_ = std::string(body.name.data(), body.name.size()); + change.partitions_count_ = body.partitions_count; + if (body.options_readable) { + change.options_ = ResourceOptions::FromFfi(std::move(body.options), std::move(body.derived_options)); + } + return change; +} + +MetadataTopicItem MetadataTopicItem::FromFfi(ffi::MetadataChangeBody &body) { + MetadataTopicItem item; + item.stream_id_ = body.stream_id; + item.stream_created_op_ = body.stream_created_op; + item.topic_id_ = body.topic_id; + item.topic_created_op_ = body.topic_created_op; + item.created_at_ = body.created_at; + item.name_ = std::string(body.name.data(), body.name.size()); + if (body.options_readable) { + item.options_ = ResourceOptions::FromFfi(std::move(body.options), std::move(body.derived_options)); + } + return item; +} + +MetadataTopicDeleted MetadataTopicDeleted::FromFfi(ffi::MetadataChangeBody &body) { + MetadataTopicDeleted change; + change.stream_id_ = body.stream_id; + change.stream_created_op_ = body.stream_created_op; + change.topic_id_ = body.topic_id; + change.topic_created_op_ = body.topic_created_op; + change.created_at_ = body.created_at; + change.name_ = std::string(body.name.data(), body.name.size()); + if (body.has_requested_op) { + change.requested_op_ = body.requested_op; + } + return change; +} + +MetadataPartitionsChanged MetadataPartitionsChanged::FromFfi(ffi::MetadataChangeBody &body) { + MetadataPartitionsChanged change; + change.stream_id_ = body.stream_id; + change.stream_created_op_ = body.stream_created_op; + change.topic_id_ = body.topic_id; + change.topic_created_op_ = body.topic_created_op; + change.partitions_count_ = body.partitions_count; + change.removed_partition_ids_.assign(body.partition_ids.begin(), body.partition_ids.end()); + if (body.has_requested_op) { + change.requested_op_ = body.requested_op; + } + return change; +} + +MetadataPartitionsItem MetadataPartitionsItem::FromFfi(ffi::MetadataChangeBody &body) { + MetadataPartitionsItem item; + item.stream_id_ = body.stream_id; + item.stream_created_op_ = body.stream_created_op; + item.topic_id_ = body.topic_id; + item.topic_created_op_ = body.topic_created_op; + item.partitions_count_ = body.partitions_count; + return item; +} + +MetadataConsumerGroupCreated MetadataConsumerGroupCreated::FromFfi(ffi::MetadataChangeBody &body) { + MetadataConsumerGroupCreated change; + change.stream_id_ = body.stream_id; + change.stream_created_op_ = body.stream_created_op; + change.topic_id_ = body.topic_id; + change.topic_created_op_ = body.topic_created_op; + change.group_id_ = body.group_id; + change.name_ = std::string(body.name.data(), body.name.size()); + change.consumer_groups_count_ = body.consumer_groups_count; + return change; +} + +MetadataConsumerGroupDeleted MetadataConsumerGroupDeleted::FromFfi(ffi::MetadataChangeBody &body) { + MetadataConsumerGroupDeleted change; + change.stream_id_ = body.stream_id; + change.stream_created_op_ = body.stream_created_op; + change.topic_id_ = body.topic_id; + change.topic_created_op_ = body.topic_created_op; + change.group_id_ = body.group_id; + change.name_ = std::string(body.name.data(), body.name.size()); + change.consumer_groups_count_ = body.consumer_groups_count; + if (body.has_requested_op) { + change.requested_op_ = body.requested_op; + } + return change; +} + +MetadataConsumerGroupItem MetadataConsumerGroupItem::FromFfi(ffi::MetadataChangeBody &body) { + MetadataConsumerGroupItem item; + item.stream_id_ = body.stream_id; + item.stream_created_op_ = body.stream_created_op; + item.topic_id_ = body.topic_id; + item.topic_created_op_ = body.topic_created_op; + item.group_id_ = body.group_id; + item.name_ = std::string(body.name.data(), body.name.size()); + return item; +} + +MetadataMemberJoined MetadataMemberJoined::FromFfi(ffi::MetadataChangeBody &body) { + MetadataMemberJoined change; + change.stream_id_ = body.stream_id; + change.stream_created_op_ = body.stream_created_op; + change.topic_id_ = body.topic_id; + change.topic_created_op_ = body.topic_created_op; + change.group_id_ = body.group_id; + change.member_id_ = body.member_id; + change.client_id_ = absl::MakeUint128(body.client_id_hi, body.client_id_lo); + change.members_count_ = body.members_count; + change.partition_ids_.assign(body.partition_ids.begin(), body.partition_ids.end()); + return change; +} + +MetadataMemberLeft MetadataMemberLeft::FromFfi(ffi::MetadataChangeBody &body) { + MetadataMemberLeft change; + change.stream_id_ = body.stream_id; + change.stream_created_op_ = body.stream_created_op; + change.topic_id_ = body.topic_id; + change.topic_created_op_ = body.topic_created_op; + change.group_id_ = body.group_id; + change.member_id_ = body.member_id; + change.client_id_ = absl::MakeUint128(body.client_id_hi, body.client_id_lo); + change.members_count_ = body.members_count; + change.reason_ = static_cast<MetadataMemberLeftReason>(body.reason); + return change; +} + +MetadataPartitionHandedOver MetadataPartitionHandedOver::FromFfi(ffi::MetadataChangeBody &body) { + MetadataPartitionHandedOver change; + change.stream_id_ = body.stream_id; + change.stream_created_op_ = body.stream_created_op; + change.topic_id_ = body.topic_id; + change.topic_created_op_ = body.topic_created_op; + change.group_id_ = body.group_id; + change.partition_id_ = body.partition_id; + if (body.has_member_id) { + change.member_id_ = body.member_id; + } + return change; +} + +MetadataMemberItem MetadataMemberItem::FromFfi(ffi::MetadataChangeBody &body) { + MetadataMemberItem item; + item.stream_id_ = body.stream_id; + item.stream_created_op_ = body.stream_created_op; + item.topic_id_ = body.topic_id; + item.topic_created_op_ = body.topic_created_op; + item.group_id_ = body.group_id; + item.member_id_ = body.member_id; + item.client_id_ = absl::MakeUint128(body.client_id_hi, body.client_id_lo); + item.partition_ids_.assign(body.partition_ids.begin(), body.partition_ids.end()); + return item; +} + +MetadataUserChange MetadataUserChange::FromFfi(ffi::MetadataChangeBody &body) { + MetadataUserChange change; + change.user_id_ = body.user_id; + change.user_created_at_ = body.user_created_at; + change.username_ = std::string(body.name.data(), body.name.size()); + change.status_ = static_cast<UserStatus>(body.status); + if (body.has_permissions) { + change.permissions_ = ::iggy::Permissions::FromFfi(body.permissions); + } + if (body.options_readable) { + change.options_ = ResourceOptions::FromFfi(std::move(body.options), std::move(body.derived_options)); + } + return change; +} + +MetadataUserIdentity MetadataUserIdentity::FromFfi(ffi::MetadataChangeBody &body) { + MetadataUserIdentity change; + change.user_id_ = body.user_id; + change.user_created_at_ = body.user_created_at; + change.username_ = std::string(body.name.data(), body.name.size()); + return change; +} + +MetadataUserPermissionsUpdated MetadataUserPermissionsUpdated::FromFfi(ffi::MetadataChangeBody &body) { + MetadataUserPermissionsUpdated change; + change.user_id_ = body.user_id; + change.user_created_at_ = body.user_created_at; + change.username_ = std::string(body.name.data(), body.name.size()); + if (body.has_permissions) { + change.permissions_ = ::iggy::Permissions::FromFfi(body.permissions); + } + change.reason_ = static_cast<MetadataPermissionsUpdatedReason>(body.reason); + return change; +} + +MetadataTokenChange MetadataTokenChange::FromFfi(ffi::MetadataChangeBody &body) { + MetadataTokenChange change; + change.user_id_ = body.user_id; + change.user_created_at_ = body.user_created_at; + change.name_ = std::string(body.name.data(), body.name.size()); + if (body.has_expiry) { + change.expiry_ = body.expiry; + } + return change; +} + +MetadataTokenDeleted MetadataTokenDeleted::FromFfi(ffi::MetadataChangeBody &body) { + MetadataTokenDeleted change; + change.user_id_ = body.user_id; + change.user_created_at_ = body.user_created_at; + change.name_ = std::string(body.name.data(), body.name.size()); + change.reason_ = static_cast<MetadataTokenDeletedReason>(body.reason); + return change; +} + +MetadataSession MetadataSession::FromFfi(ffi::MetadataChangeBody &body) { + MetadataSession change; + change.user_id_ = body.user_id; + change.user_created_at_ = body.user_created_at; + change.client_id_ = absl::MakeUint128(body.client_id_hi, body.client_id_lo); + change.session_ = body.session; + return change; +} + +MetadataSessionClosed MetadataSessionClosed::FromFfi(ffi::MetadataChangeBody &body) { + MetadataSessionClosed change; + change.user_id_ = body.user_id; + change.user_created_at_ = body.user_created_at; + change.client_id_ = absl::MakeUint128(body.client_id_hi, body.client_id_lo); + change.session_ = body.session; + change.reason_ = static_cast<MetadataSessionClosedReason>(body.reason); + return change; +} + +MetadataChangeRecord MetadataChangeRecord::FromFfi(ffi::MetadataChangeRecord record) { + const auto checksum = absl::MakeUint128(record.checksum_hi, record.checksum_lo); + std::optional<std::uint32_t> actor_user_id; + if (record.has_actor_user_id) { + actor_user_id = record.actor_user_id; + } + switch (record.form) { + case ffi::MetadataChangeForm::Known: { + const auto kind = static_cast<MetadataChangeKind>(record.kind); + return MetadataChangeRecord(record.op, record.ordinal, checksum, record.timestamp, actor_user_id, kind, + static_cast<MetadataChangeGroup>(record.group), BodyFromFfi(kind, record.body)); + } + case ffi::MetadataChangeForm::Unknown: { + MetadataUnknownChange change; + change.group_code_ = record.group; + change.kind_code_ = record.kind; + change.body_.assign(record.body.unknown_body.begin(), record.body.unknown_body.end()); + return MetadataChangeRecord(record.op, record.ordinal, checksum, record.timestamp, actor_user_id, + std::nullopt, std::nullopt, std::move(change)); + } + case ffi::MetadataChangeForm::UnknownJson: { + MetadataUnknownJsonChange change; + change.group_ = std::string(record.group_name.data(), record.group_name.size()); + change.kind_ = std::string(record.kind_name.data(), record.kind_name.size()); + change.json_.assign(record.body.unknown_body.begin(), record.body.unknown_body.end()); + return MetadataChangeRecord(record.op, record.ordinal, checksum, record.timestamp, actor_user_id, + std::nullopt, std::nullopt, std::move(change)); + } + } + throw IggyException("Unknown metadata change record form"); +} + +MetadataChangeRecord::Body MetadataChangeRecord::BodyFromFfi(MetadataChangeKind kind, ffi::MetadataChangeBody &body) { + switch (kind) { + case MetadataChangeKind::StreamCreated: + case MetadataChangeKind::StreamUpdated: + return MetadataStreamChange::FromFfi(body); + case MetadataChangeKind::StreamDeleted: + return MetadataStreamDeleted::FromFfi(body); + case MetadataChangeKind::TopicCreated: + case MetadataChangeKind::TopicUpdated: + return MetadataTopicChange::FromFfi(body); + case MetadataChangeKind::TopicDeleted: + return MetadataTopicDeleted::FromFfi(body); + case MetadataChangeKind::PartitionsChanged: + return MetadataPartitionsChanged::FromFfi(body); + case MetadataChangeKind::ConsumerGroupCreated: + return MetadataConsumerGroupCreated::FromFfi(body); + case MetadataChangeKind::ConsumerGroupDeleted: + return MetadataConsumerGroupDeleted::FromFfi(body); + case MetadataChangeKind::ConsumerGroupMemberJoined: + return MetadataMemberJoined::FromFfi(body); + case MetadataChangeKind::ConsumerGroupMemberLeft: + return MetadataMemberLeft::FromFfi(body); + case MetadataChangeKind::ConsumerGroupPartitionHandedOver: + return MetadataPartitionHandedOver::FromFfi(body); + case MetadataChangeKind::UserCreated: + case MetadataChangeKind::UserUpdated: + return MetadataUserChange::FromFfi(body); + case MetadataChangeKind::UserDeleted: + case MetadataChangeKind::UserPasswordChanged: + return MetadataUserIdentity::FromFfi(body); + case MetadataChangeKind::UserPermissionsUpdated: + return MetadataUserPermissionsUpdated::FromFfi(body); + case MetadataChangeKind::PersonalAccessTokenCreated: + return MetadataTokenChange::FromFfi(body); + case MetadataChangeKind::PersonalAccessTokenDeleted: + return MetadataTokenDeleted::FromFfi(body); + case MetadataChangeKind::SessionRegistered: + return MetadataSession::FromFfi(body); + case MetadataChangeKind::SessionClosed: + return MetadataSessionClosed::FromFfi(body); + } + throw IggyException("Unknown metadata change kind"); +} + +MetadataCurrentItem MetadataCurrentItem::FromFfi(ffi::MetadataCurrentItem item) { + switch (item.form) { + case ffi::MetadataChangeForm::Known: { + const auto group = static_cast<MetadataChangeGroup>(item.group); + return MetadataCurrentItem(group, BodyFromFfi(group, item.body)); + } + case ffi::MetadataChangeForm::Unknown: { + MetadataUnknownItem value; + value.group_code_ = item.group; + value.body_.assign(item.body.unknown_body.begin(), item.body.unknown_body.end()); + return MetadataCurrentItem(std::nullopt, std::move(value)); + } + case ffi::MetadataChangeForm::UnknownJson: { + MetadataUnknownJsonItem value; + value.group_ = std::string(item.group_name.data(), item.group_name.size()); + value.json_.assign(item.body.unknown_body.begin(), item.body.unknown_body.end()); + return MetadataCurrentItem(std::nullopt, std::move(value)); + } + } + throw IggyException("Unknown metadata current item form"); +} + +MetadataCurrentItem::Body MetadataCurrentItem::BodyFromFfi(MetadataChangeGroup group, ffi::MetadataChangeBody &body) { + switch (group) { + case MetadataChangeGroup::Streams: + return MetadataStreamChange::FromFfi(body); + case MetadataChangeGroup::Topics: + return MetadataTopicItem::FromFfi(body); + case MetadataChangeGroup::Partitions: + return MetadataPartitionsItem::FromFfi(body); + case MetadataChangeGroup::ConsumerGroups: + return MetadataConsumerGroupItem::FromFfi(body); + case MetadataChangeGroup::ConsumerGroupMembers: + return MetadataMemberItem::FromFfi(body); + case MetadataChangeGroup::Users: + return MetadataUserChange::FromFfi(body); + case MetadataChangeGroup::PersonalAccessTokens: + return MetadataTokenChange::FromFfi(body); + case MetadataChangeGroup::Sessions: + return MetadataSession::FromFfi(body); + } + throw IggyException("Unknown metadata change group"); +} + +MetadataChanges MetadataChanges::FromFfi(ffi::MetadataChanges changes) { + MetadataChanges result; + result.status_ = static_cast<MetadataChangeStatus>(changes.status); + result.phase_ = static_cast<MetadataChangePhase>(changes.phase); + if (changes.has_log_id) { + result.log_id_ = absl::MakeUint128(changes.log_id_hi, changes.log_id_lo); + } + result.oldest_op_ = changes.oldest_op; + result.retained_since_ = changes.retained_since; + result.head_op_ = changes.head_op; + if (changes.has_next) { + result.next_ = MetadataChangeCursor::FromFfi(std::move(changes.next)); + } + result.has_more_ = changes.has_more; + result.records_.reserve(changes.records.size()); + for (auto &record : changes.records) { + result.records_.push_back(MetadataChangeRecord::FromFfi(std::move(record))); + } + result.items_.reserve(changes.items.size()); + for (auto &item : changes.items) { + result.items_.push_back(MetadataCurrentItem::FromFfi(std::move(item))); + } + if (changes.has_current_page_token) { + result.current_page_token_ = MetadataCurrentPageToken::FromFfi(std::move(changes.current_page_token)); + } + return result; +} + } // namespace iggy
diff --git a/foreign/cpp/tests/e2e/metadata_changes.cpp b/foreign/cpp/tests/e2e/metadata_changes.cpp new file mode 100644 index 0000000..3276e88 --- /dev/null +++ b/foreign/cpp/tests/e2e/metadata_changes.cpp
@@ -0,0 +1,423 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +#include <algorithm> +#include <chrono> +#include <cstddef> +#include <cstdint> +#include <functional> +#include <map> +#include <optional> +#include <set> +#include <string> +#include <thread> +#include <type_traits> +#include <utility> +#include <variant> +#include <vector> + +#include <gtest/gtest.h> + +#include "tests/e2e/test_helpers.hpp" + +struct MetadataChangesTransport { + std::string name; + // Empty selects the TCP builder. + std::string scheme; + std::string address; + + [[nodiscard]] std::string ConnectionString(const std::string &username, const std::string &password) const { + return scheme + "://" + username + ":" + password + "@" + address; + } +}; + +namespace { + +constexpr std::uint32_t kRootUserId = 0; +constexpr std::uint32_t kInvalidCommand = 3; +constexpr std::uint32_t kUnauthorized = 41; +constexpr int kFollowAttempts = 100; +constexpr auto kFollowInterval = std::chrono::milliseconds(100); +constexpr std::size_t kKeyLength = 32; +constexpr std::uint8_t kKeyByte = 21; +constexpr std::size_t kFingerprintHex = 16; +constexpr const char *kHiddenOptions[] = {"encryption_key", "encryption_key_wrapped", "critical_keys"}; + +const std::vector<iggy::MetadataChangeGroup> kStreamGroups{iggy::MetadataChangeGroup::Streams}; + +std::string PrintTransport(const ::testing::TestParamInfo<MetadataChangesTransport> &info) { + return info.param.name; +} + +// A record's stream ID and creation op, which tells a stream from an earlier one with the same ID. +std::optional<std::pair<std::uint32_t, std::uint64_t>> ChangeStream(const iggy::MetadataChangeRecord &record) { + return std::visit( + [](const auto &body) -> std::optional<std::pair<std::uint32_t, std::uint64_t>> { + using Body = std::decay_t<decltype(body)>; + if constexpr (std::is_same_v<Body, iggy::MetadataStreamChange> || + std::is_same_v<Body, iggy::MetadataStreamDeleted> || + std::is_same_v<Body, iggy::MetadataTopicChange> || + std::is_same_v<Body, iggy::MetadataTopicDeleted> || + std::is_same_v<Body, iggy::MetadataPartitionsChanged>) { + return std::make_pair(body.StreamId(), body.StreamCreatedOp()); + } else { + return std::nullopt; + } + }, + record.Change()); +} + +std::vector<iggy::MetadataChangeKind> KindsOf(const std::vector<iggy::MetadataChangeRecord> &records) { + std::vector<iggy::MetadataChangeKind> kinds; + kinds.reserve(records.size()); + for (const auto &record : records) { + kinds.push_back(record.Kind().value()); + } + return kinds; +} + +std::optional<std::string> OptionText(const iggy::ResourceOptions &options, const std::string &key) { + for (const auto *entries : {&options.Explicit(), &options.Derived()}) { + const auto entry = entries->find(key); + if (entry != entries->end()) { + const auto &bytes = entry->second.Value(); + return std::string(bytes.begin(), bytes.end()); + } + } + return std::nullopt; +} + +::testing::AssertionResult ThrowsIggyError(const std::function<void()> &operation, + std::uint32_t code, + const std::string &kind) { + try { + operation(); + } catch (const iggy::IggyException &exception) { + if (exception.Code() == code && exception.Kind() == kind) { + return ::testing::AssertionSuccess(); + } + return ::testing::AssertionFailure() + << "expected " << kind << " (" << code << "), got '" << exception.what() << "' with code " + << exception.Code().value_or(0) << " and kind '" << exception.Kind() << "'"; + } + return ::testing::AssertionFailure() << "expected " << kind << " (" << code << "), nothing was thrown"; +} + +} // namespace + +class MetadataChangesTransportFixture : public E2ETestFixture, + public ::testing::WithParamInterface<MetadataChangesTransport> { + protected: + iggy::IggyBlockingClient GetTransportClient() { return GetTransportClient("iggy", "iggy"); } + + iggy::IggyBlockingClient GetTransportClient(const std::string &username, const std::string &password) { + const auto &transport = GetParam(); + auto client = + transport.scheme.empty() + ? iggy::IggyBlockingClient::Builder().WithAutoLogin(username, password).Build() + : iggy::IggyBlockingClient::FromConnectionString(transport.ConnectionString(username, password)); + client.Connect(); + if (transport.name == "http") { + client.Login(username, password); + } + return client; + } + + static iggy::MetadataChangeCursor LatestCursor(iggy::IggyBlockingClient &client, + const std::vector<iggy::MetadataChangeGroup> &groups) { + const auto latest = client.GetMetadataChanges(iggy::MetadataChangesQuery::Latest().SetGroups(groups)); + EXPECT_EQ(latest.Status(), iggy::MetadataChangeStatus::Ok); + EXPECT_EQ(latest.Phase(), iggy::MetadataChangePhase::Changes); + EXPECT_TRUE(latest.Records().empty()); + EXPECT_TRUE(latest.LogId().has_value()); + return latest.Next().value(); + } + + // A two phase delete commits its record after the call returns, so an incomplete read polls + // again. The deferred delete of an earlier stream with the same ID is skipped. + static std::vector<iggy::MetadataChangeRecord> Follow(iggy::IggyBlockingClient &client, + iggy::MetadataChangeCursor cursor, + const std::vector<iggy::MetadataChangeGroup> &groups, + std::uint32_t stream_id, + std::size_t count, + std::uint32_t limit = iggy::MetadataChangesQuery::kMaxLimit) { + std::vector<iggy::MetadataChangeRecord> mine; + std::optional<std::uint64_t> stream_created_op; + for (int attempt = 0; attempt < kFollowAttempts; ++attempt) { + const auto page = + client.GetMetadataChanges(iggy::MetadataChangesQuery::After(cursor).SetGroups(groups).SetLimit(limit)); + EXPECT_EQ(page.Status(), iggy::MetadataChangeStatus::Ok); + EXPECT_LE(page.Records().size(), limit); + cursor = page.Next().value(); + for (const auto &record : page.Records()) { + const auto group = record.Group(); + EXPECT_TRUE(group.has_value() && std::find(groups.begin(), groups.end(), *group) != groups.end()) + << "record of op " << record.Op() << " is outside the requested groups"; + const auto stream = ChangeStream(record); + if (!stream || stream->first != stream_id) { + continue; + } + if (!stream_created_op && (record.Kind() == iggy::MetadataChangeKind::StreamCreated || + record.Kind() == iggy::MetadataChangeKind::TopicCreated)) { + stream_created_op = stream->second; + } + if (stream->second == stream_created_op) { + mine.push_back(record); + } + } + if (page.HasMore()) { + continue; + } + if (mine.size() >= count) { + break; + } + std::this_thread::sleep_for(kFollowInterval); + } + return mine; + } +}; + +class E2E_MetadataChangesTransport : public MetadataChangesTransportFixture {}; + +// Needs a server encryption key. The shipped default lane skips every *ResourceOptions* suite. +class E2E_MetadataChangesResourceOptionsTransport : public MetadataChangesTransportFixture {}; + +INSTANTIATE_TEST_SUITE_P(Transports, + E2E_MetadataChangesTransport, + ::testing::Values(MetadataChangesTransport{"tcp", "", ""}, + MetadataChangesTransport{"quic", "iggy+quic", "127.0.0.1:8080"}, + MetadataChangesTransport{"http", "iggy+http", "127.0.0.1:3000"}, + MetadataChangesTransport{"websocket", "iggy+ws", "127.0.0.1:8092"}), + PrintTransport); + +INSTANTIATE_TEST_SUITE_P(Transports, + E2E_MetadataChangesResourceOptionsTransport, + ::testing::Values(MetadataChangesTransport{"tcp", "", ""}, + MetadataChangesTransport{"quic", "iggy+quic", "127.0.0.1:8080"}, + MetadataChangesTransport{"http", "iggy+http", "127.0.0.1:3000"}, + MetadataChangesTransport{"websocket", "iggy+ws", "127.0.0.1:8092"}), + PrintTransport); + +TEST_P(E2E_MetadataChangesTransport, GivenLatestCursorWhenCreatingRenamingAndDeletingStreamShouldFollowEachChange) { + RecordProperty("description", + "Holds the latest cursor for the streams group, creates, renames and deletes a stream, and follows " + "one record per change in order, each naming the root user as its actor."); + auto client = GetTransportClient(); + const auto cursor = LatestCursor(client, kStreamGroups); + const auto name = GetRandomName(40); + const auto renamed = name + "-renamed"; + + const auto stream = client.CreateStream(name); + const auto stream_id = iggy::Identifier::Numeric(stream.Id()); + TrackStream(stream.Id()); + client.UpdateStream(stream_id, iggy::StreamUpdateOptions().SetName(renamed)); + client.DeleteStream(stream_id); + ForgetTrackedStream(stream.Id()); + + const auto records = Follow(client, cursor, kStreamGroups, stream.Id(), 3); + ASSERT_EQ(KindsOf(records), (std::vector<iggy::MetadataChangeKind>{iggy::MetadataChangeKind::StreamCreated, + iggy::MetadataChangeKind::StreamUpdated, + iggy::MetadataChangeKind::StreamDeleted})); + EXPECT_EQ(std::get<iggy::MetadataStreamChange>(records[0].Change()).Name(), name); + EXPECT_EQ(std::get<iggy::MetadataStreamChange>(records[1].Change()).Name(), renamed); + const auto &deleted = std::get<iggy::MetadataStreamDeleted>(records[2].Change()); + EXPECT_EQ(deleted.Name(), renamed); + EXPECT_EQ(deleted.StreamCreatedOp(), records[0].Op()); + for (std::size_t index = 0; index < records.size(); ++index) { + EXPECT_EQ(records[index].ActorUserId(), kRootUserId) << index; + EXPECT_EQ(records[index].Group(), iggy::MetadataChangeGroup::Streams) << index; + if (index > 0) { + EXPECT_GT(records[index].Op(), records[index - 1].Op()) << index; + } + } +} + +TEST_P(E2E_MetadataChangesTransport, GivenTopicsGroupWhenCreatingStreamAndTopicShouldFollowOnlyTheTopic) { + RecordProperty("description", + "Holds the latest cursor for the topics group, creates a stream and a topic, and follows only the " + "topic record."); + auto client = GetTransportClient(); + const std::vector<iggy::MetadataChangeGroup> groups = {iggy::MetadataChangeGroup::Topics}; + const auto cursor = LatestCursor(client, groups); + const auto name = GetRandomName(40); + + const auto stream = client.CreateStream(name); + TrackStream(name); + client.CreateTopic(iggy::Identifier::Numeric(stream.Id()), "events", + iggy::TopicCreateOptions().SetPartitionsCount(1)); + + const auto records = Follow(client, cursor, groups, stream.Id(), 1); + ASSERT_EQ(KindsOf(records), (std::vector<iggy::MetadataChangeKind>{iggy::MetadataChangeKind::TopicCreated})); + const auto &topic = std::get<iggy::MetadataTopicChange>(records[0].Change()); + EXPECT_EQ(topic.Name(), "events"); + EXPECT_EQ(topic.PartitionsCount(), 1u); + EXPECT_GT(topic.TopicCreatedOp(), topic.StreamCreatedOp()); + EXPECT_TRUE(topic.Options().has_value()); +} + +TEST_P(E2E_MetadataChangesTransport, GivenLimitOfOneWhenFollowingShouldPageOneRecordAtATimeInOrder) { + RecordProperty("description", + "Creates three streams after the latest cursor and pages the records one at a time, reaching the " + "same records a single large page returns."); + auto client = GetTransportClient(); + const auto cursor = LatestCursor(client, kStreamGroups); + std::set<std::uint32_t> stream_ids; + for (int index = 0; index < 3; ++index) { + const auto name = GetRandomName(40); + stream_ids.insert(client.CreateStream(name).Id()); + TrackStream(name); + } + // A deferred delete of an earlier test can commit after the cursor, so only these creates count. + const auto created_here = [&stream_ids](const iggy::MetadataChangeRecord &record) { + const auto stream = ChangeStream(record); + return record.Kind() == iggy::MetadataChangeKind::StreamCreated && stream && + stream_ids.count(stream->first) == 1; + }; + + std::vector<std::uint64_t> paged; + auto position = cursor; + while (true) { + const auto page = + client.GetMetadataChanges(iggy::MetadataChangesQuery::After(position).SetGroups(kStreamGroups).SetLimit(1)); + ASSERT_EQ(page.Status(), iggy::MetadataChangeStatus::Ok); + ASSERT_LE(page.Records().size(), 1u); + for (const auto &record : page.Records()) { + if (created_here(record)) { + paged.push_back(record.Op()); + } + } + position = page.Next().value(); + if (!page.HasMore()) { + break; + } + } + + const auto whole = client.GetMetadataChanges(iggy::MetadataChangesQuery::After(cursor).SetGroups(kStreamGroups)); + std::vector<std::uint64_t> expected; + for (const auto &record : whole.Records()) { + if (created_here(record)) { + expected.push_back(record.Op()); + } + } + EXPECT_EQ(expected.size(), stream_ids.size()); + EXPECT_EQ(paged, expected); + EXPECT_TRUE(std::is_sorted(paged.begin(), paged.end())); +} + +TEST_P(E2E_MetadataChangesTransport, GivenStreamsWhenPagingCurrentStateShouldListTheSameStreamsAsTheStreamList) { + RecordProperty("description", + "Pages the current state of the streams group one item at a time, compares it with GetStreams(), " + "and follows the change log from the anchor the last page returns."); + auto client = GetTransportClient(); + for (int index = 0; index < 2; ++index) { + const auto name = GetRandomName(40); + client.CreateStream(name); + TrackStream(name); + } + + std::map<std::uint32_t, std::string> projected; + auto query = iggy::MetadataChangesQuery::Current().SetGroups(kStreamGroups).SetLimit(1); + std::optional<iggy::MetadataChangeCursor> anchor; + while (!anchor) { + const auto page = client.GetMetadataChanges(query); + ASSERT_EQ(page.Status(), iggy::MetadataChangeStatus::Ok); + ASSERT_EQ(page.Phase(), iggy::MetadataChangePhase::Current); + ASSERT_TRUE(page.Records().empty()); + ASSERT_LE(page.Items().size(), 1u); + for (const auto &item : page.Items()) { + const auto &stream = std::get<iggy::MetadataStreamChange>(item.Value()); + projected.emplace(stream.StreamId(), stream.Name()); + } + if (page.CurrentPageToken()) { + query = iggy::MetadataChangesQuery::Current(*page.CurrentPageToken()).SetGroups(kStreamGroups).SetLimit(1); + } else { + anchor = page.Next(); + ASSERT_TRUE(anchor.has_value()); + } + } + + std::map<std::uint32_t, std::string> listed; + for (const auto &stream : client.GetStreams()) { + listed.emplace(stream.Id(), stream.Name()); + } + EXPECT_EQ(projected, listed); + const auto follow = client.GetMetadataChanges(iggy::MetadataChangesQuery::After(*anchor).SetGroups(kStreamGroups)); + EXPECT_EQ(follow.Status(), iggy::MetadataChangeStatus::Ok); +} + +TEST_P(E2E_MetadataChangesTransport, GivenInvalidQueriesWhenReadingShouldRefuseEachWithInvalidCommand) { + RecordProperty("description", + "Refuses limits outside 1 to 1000, since above until, time bounds with latest or current, and a " + "current scan without its parent group before anything is sent."); + auto client = GetTransportClient(); + const auto queries = { + iggy::MetadataChangesQuery::Earliest().SetLimit(0), + iggy::MetadataChangesQuery::Earliest().SetLimit(iggy::MetadataChangesQuery::kMaxLimit + 1), + iggy::MetadataChangesQuery::Earliest().SetSince(2).SetUntil(1), + iggy::MetadataChangesQuery::Latest().SetSince(1), + iggy::MetadataChangesQuery::Current().SetUntil(1), + iggy::MetadataChangesQuery::Current().SetGroups({iggy::MetadataChangeGroup::Topics}), + }; + for (const auto &query : queries) { + EXPECT_TRUE(ThrowsIggyError([&client, &query] { client.GetMetadataChanges(query); }, kInvalidCommand, + "invalid_command")); + } +} + +TEST_P(E2E_MetadataChangesTransport, GivenUserWithoutManageServersWhenReadingShouldBeRefusedAsUnauthorized) { + RecordProperty("description", "A user with read_servers but not manage_servers is refused with Unauthorized (41)."); + auto root = GetTransportClient(); + const auto username = GetRandomName(30); + const std::string password{"changes-secret"}; + CreateUser(root, username, password, iggy::UserStatus::Active, + iggy::Permissions().SetGlobal(iggy::GlobalPermissions().SetReadServers(true))); + + auto reader = GetTransportClient(username, password); + EXPECT_TRUE(ThrowsIggyError([&reader] { reader.GetMetadataChanges(iggy::MetadataChangesQuery::Earliest()); }, + kUnauthorized, "unauthorized")); +} + +TEST_P(E2E_MetadataChangesResourceOptionsTransport, + GivenKeyedStreamWhenFollowingShouldExposeFingerprintsAndNoKeyMaterial) { + RecordProperty("description", + "Creates a stream with its own key and follows its record, which reports the key source and " + "fingerprints and carries no key material."); + auto client = GetTransportClient(); + const auto cursor = LatestCursor(client, kStreamGroups); + const auto name = GetRandomName(40); + const std::vector<std::uint8_t> key(kKeyLength, kKeyByte); + + const auto stream = client.CreateStream(name, iggy::StreamCreateOptions().SetEncryptionKey(key)); + TrackStream(name); + + const auto records = Follow(client, cursor, kStreamGroups, stream.Id(), 1); + ASSERT_EQ(KindsOf(records), (std::vector<iggy::MetadataChangeKind>{iggy::MetadataChangeKind::StreamCreated})); + const auto &options = std::get<iggy::MetadataStreamChange>(records[0].Change()).Options(); + ASSERT_TRUE(options.has_value()); + EXPECT_EQ(OptionText(*options, "encryption_key_source"), + std::string(iggy::to_string(iggy::StreamEncryptionSource::Stream))); + EXPECT_EQ(OptionText(*options, "encryption_key_fingerprint"), iggy::EncryptionKeyFingerprint(key)); + const auto kek_fingerprint = OptionText(*options, "encryption_kek_fingerprint"); + ASSERT_TRUE(kek_fingerprint.has_value()); + EXPECT_EQ(kek_fingerprint->size(), kFingerprintHex); + for (const auto *hidden : kHiddenOptions) { + EXPECT_EQ(options->Explicit().count(hidden), 0u) << hidden; + EXPECT_EQ(options->Derived().count(hidden), 0u) << hidden; + } +}
diff --git a/foreign/cpp/tests/fixtures/metadata_changes.json b/foreign/cpp/tests/fixtures/metadata_changes.json new file mode 100644 index 0000000..26e7faf --- /dev/null +++ b/foreign/cpp/tests/fixtures/metadata_changes.json
@@ -0,0 +1,1997 @@ +{ + "schema_version": 1, + "generated_by": "core/common/examples/metadata_changes_fixtures.rs", + "conventions": { + "byte_strings": "Every *_hex field is lowercase hex without separators.", + "expected": "expected is the JSON the HTTP route renders. Binary decoders must reach the same values.", + "record_hex": "record_hex is one record envelope as a response carries it. body_hex is the body alone.", + "refused": "Every refused vector must fail to decode or validate. Query and HTTP refusals are invalid_command.", + "request_hex": "request_hex is the command payload without framing. http_query is the query string of GET /metadata/changes.", + "u128_values": "128-bit values (log ids, checksums, client ids) are 32 lowercase hex characters in JSON.", + "u64_values": "Ops and timestamps are JSON numbers below 2^53 in every vector except large_values, which carries them as decimal strings.", + "unknown_kinds": "A record of an unknown group or kind is kept with its codes and raw body, never refused." + }, + "constants": { + "command_code": 17, + "command_name": "metadata.changes", + "cursor_prefix": "mc1_", + "page_token_prefix": "cs1_", + "cursor_size": 44, + "cursor_text_length": 63, + "max_page_token_size": 316, + "max_page_token_key_size": 260, + "default_limit": 100, + "max_limit": 1000, + "groups": { + "consumer_group_members": { + "code": 5, + "class": "activity", + "parents": [ + "consumer_groups", + "partitions", + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32][member_id:u32]" + }, + "consumer_groups": { + "code": 4, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32]" + }, + "partitions": { + "code": 3, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "personal_access_tokens": { + "code": 7, + "class": "resource", + "parents": [ + "users" + ], + "page_token_key": "[user_id:u32][name_len:u8][name]" + }, + "sessions": { + "code": 8, + "class": "activity", + "parents": [ + "users" + ], + "page_token_key": "[epoch:u64]" + }, + "streams": { + "code": 1, + "class": "resource", + "parents": [], + "page_token_key": "[stream_id:u32]" + }, + "topics": { + "code": 2, + "class": "resource", + "parents": [ + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "users": { + "code": 6, + "class": "resource", + "parents": [], + "page_token_key": "[user_id:u32]" + } + }, + "kinds": { + "consumer_group_created": { + "code": 30, + "group": "consumer_groups" + }, + "consumer_group_deleted": { + "code": 31, + "group": "consumer_groups" + }, + "consumer_group_member_joined": { + "code": 40, + "group": "consumer_group_members" + }, + "consumer_group_member_left": { + "code": 41, + "group": "consumer_group_members" + }, + "consumer_group_partition_handed_over": { + "code": 42, + "group": "consumer_group_members" + }, + "partitions_changed": { + "code": 20, + "group": "partitions" + }, + "personal_access_token_created": { + "code": 60, + "group": "personal_access_tokens" + }, + "personal_access_token_deleted": { + "code": 61, + "group": "personal_access_tokens" + }, + "session_closed": { + "code": 71, + "group": "sessions" + }, + "session_registered": { + "code": 70, + "group": "sessions" + }, + "stream_created": { + "code": 1, + "group": "streams" + }, + "stream_deleted": { + "code": 3, + "group": "streams" + }, + "stream_updated": { + "code": 2, + "group": "streams" + }, + "topic_created": { + "code": 10, + "group": "topics" + }, + "topic_deleted": { + "code": 12, + "group": "topics" + }, + "topic_updated": { + "code": 11, + "group": "topics" + }, + "user_created": { + "code": 50, + "group": "users" + }, + "user_deleted": { + "code": 52, + "group": "users" + }, + "user_password_changed": { + "code": 53, + "group": "users" + }, + "user_permissions_updated": { + "code": 54, + "group": "users" + }, + "user_updated": { + "code": 51, + "group": "users" + } + }, + "statuses": { + "behind": 1, + "expired": 2, + "ok": 0, + "reset": 3 + }, + "phases": { + "changes": 0, + "current": 1 + }, + "start_modes": { + "after": 2, + "current": 3, + "earliest": 0, + "latest": 1 + }, + "member_left_reasons": { + "left": 0, + "session_ended": 1 + }, + "token_deleted_reasons": { + "deleted": 0, + "expired": 1 + }, + "session_closed_reasons": { + "expired": 1, + "logout": 0, + "revoked": 2 + }, + "permissions_updated_reasons": { + "resource_deleted": 1, + "updated": 0 + } + }, + "records": [ + { + "case": "stream_created", + "group": 1, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "record_hex": "191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "stream_updated_unreadable_options", + "group": 1, + "kind": 2, + "body_hex": "03000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "record_hex": "1a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "expected": { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + } + }, + { + "case": "stream_deleted_deferred", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "record_hex": "1b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "expected": { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + }, + { + "case": "topic_created", + "group": 2, + "kind": 10, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "record_hex": "1c1000000000000000000000683d9e8b03f7c1a500000000000000001c00be01094106000104000000020a78000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "op": 4124, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d68", + "timestamp": 1760356800004124, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "partitions_count": 8, + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "topic_updated", + "group": 2, + "kind": 11, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "record_hex": "1d1000000000000000000000693d9e8b03f7c1a500000000000000001d00be01094106000104000000020b3d000000030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "expected": { + "op": 4125, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d69", + "timestamp": 1760356800004125, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_updated", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics-v2", + "partitions_count": 8, + "options": {} + } + }, + { + "case": "topic_deleted_immediate", + "group": 2, + "kind": 12, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "record_hex": "1e10000000000000000000006a3d9e8b03f7c1a500000000000000001e00be01094106000104000000020c2e000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "expected": { + "op": 4126, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6a", + "timestamp": 1760356800004126, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics" + } + }, + { + "case": "partitions_changed_deleted_tail", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "record_hex": "1f10000000000000000000006b3d9e8b03f7c1a500000000000000001f00be01094106000104000000031435000000030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "expected": { + "op": 4127, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6b", + "timestamp": 1760356800004127, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 5, + "removed_partition_ids": [ + 5, + 6, + 7 + ], + "requested_op": 4200 + } + }, + { + "case": "partitions_changed_created", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000a0000000000000000", + "record_hex": "2010000000000000000000006c3d9e8b03f7c1a500000000000000002000be01094106000104000000031421000000030000001910000000000000000000001d100000000000000a0000000000000000", + "expected": { + "op": 4128, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6c", + "timestamp": 1760356800004128, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 10, + "removed_partition_ids": [] + } + }, + { + "case": "consumer_group_created", + "group": 4, + "kind": 30, + "body_hex": "030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "record_hex": "2110000000000000000000006d3d9e8b03f7c1a500000000000000002100be01094106000104000000041e28000000030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "expected": { + "op": 4129, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6d", + "timestamp": 1760356800004129, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 1 + } + }, + { + "case": "consumer_group_deleted", + "group": 4, + "kind": 31, + "body_hex": "030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "record_hex": "2210000000000000000000006e3d9e8b03f7c1a500000000000000002200be01094106000104000000041f31000000030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "expected": { + "op": 4130, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6e", + "timestamp": 1760356800004130, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 0, + "requested_op": 4300 + } + }, + { + "case": "consumer_group_member_joined", + "group": 5, + "kind": 40, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "record_hex": "2310000000000000000000006f3d9e8b03f7c1a500000000000000002300be01094106000109000000052838000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "expected": { + "op": 4131, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6f", + "timestamp": 1760356800004131, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_joined", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 2, + "partition_ids": [] + } + }, + { + "case": "consumer_group_member_left_session_ended", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "record_hex": "241000000000000000000000703d9e8b03f7c1a500000000000000002400be01094106000109000000052935000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "expected": { + "op": 4132, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d70", + "timestamp": 1760356800004132, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_left", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 1, + "reason": "session_ended" + } + }, + { + "case": "consumer_group_partition_handed_over", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d1000000000000000000000020000000101000000", + "record_hex": "251000000000000000000000713d9e8b03f7c1a500000000000000002500be010941060000052a25000000030000001910000000000000000000001d1000000000000000000000020000000101000000", + "expected": { + "op": 4133, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d71", + "timestamp": 1760356800004133, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 2, + "member_id": 1 + } + }, + { + "case": "consumer_group_partition_released", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d10000000000000000000000300000000", + "record_hex": "261000000000000000000000723d9e8b03f7c1a500000000000000002600be010941060000052a21000000030000001910000000000000000000001d10000000000000000000000300000000", + "expected": { + "op": 4134, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d72", + "timestamp": 1760356800004134, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 3 + } + }, + { + "case": "user_created_with_permissions", + "group": 6, + "kind": 50, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "record_hex": "271000000000000000000000733d9e8b03f7c1a500000000000000002700be010941060001000000000632480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "op": 4135, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d73", + "timestamp": 1760356800004135, + "actor_user_id": 0, + "group": "users", + "kind": "user_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "user_updated_inactive", + "group": 6, + "kind": 51, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "record_hex": "281000000000000000000000743d9e8b03f7c1a500000000000000002800be010941060001000000000633280000000700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "expected": { + "op": 4136, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d74", + "timestamp": 1760356800004136, + "actor_user_id": 0, + "group": "users", + "kind": "user_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "inactive", + "permissions": null, + "options": {} + } + }, + { + "case": "user_deleted", + "group": 6, + "kind": 52, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "291000000000000000000000753d9e8b03f7c1a500000000000000002900be0109410600010000000006341d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4137, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d75", + "timestamp": 1760356800004137, + "actor_user_id": 0, + "group": "users", + "kind": "user_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_password_changed", + "group": 6, + "kind": 53, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "2a1000000000000000000000763d9e8b03f7c1a500000000000000002a00be0109410600010700000006351d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4138, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d76", + "timestamp": 1760356800004138, + "actor_user_id": 7, + "group": "users", + "kind": "user_password_changed", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_permissions_pruned", + "group": 6, + "kind": 54, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "record_hex": "2b1000000000000000000000773d9e8b03f7c1a500000000000000002b00be0109410600010400000006362a0000000700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "expected": { + "op": 4139, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d77", + "timestamp": 1760356800004139, + "actor_user_id": 4, + "group": "users", + "kind": "user_permissions_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": false, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": null + }, + "reason": "resource_deleted" + } + }, + { + "case": "personal_access_token_created", + "group": 7, + "kind": 60, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "record_hex": "2c1000000000000000000000783d9e8b03f7c1a500000000000000002c00be01094106000107000000073c1d0000000700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "expected": { + "op": 4140, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d78", + "timestamp": 1760356800004140, + "actor_user_id": 7, + "group": "personal_access_tokens", + "kind": "personal_access_token_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "expiry": 1760400000000000 + } + }, + { + "case": "personal_access_token_deleted_expired", + "group": 7, + "kind": 61, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e01", + "record_hex": "2d1000000000000000000000793d9e8b03f7c1a500000000000000002d00be010941060000073d160000000700000000b832c8fb4006000863692d746f6b656e01", + "expected": { + "op": 4141, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d79", + "timestamp": 1760356800004141, + "group": "personal_access_tokens", + "kind": "personal_access_token_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "reason": "expired" + } + }, + { + "case": "session_registered", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "record_hex": "2e10000000000000000000007a3d9e8b03f7c1a500000000000000002e00be010941060001070000000846240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "op": 4142, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7a", + "timestamp": 1760356800004142, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_registered", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + }, + { + "case": "session_closed_expired", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "record_hex": "2f10000000000000000000007b3d9e8b03f7c1a500000000000000002f00be0109410600000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "expected": { + "op": 4143, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7b", + "timestamp": 1760356800004143, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "expired" + } + }, + { + "case": "session_closed_revoked", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "record_hex": "3010000000000000000000007c3d9e8b03f7c1a500000000000000003000be010941060001070000000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "expected": { + "op": 4144, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7c", + "timestamp": 1760356800004144, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "revoked" + } + } + ], + "items": [ + { + "case": "stream", + "group": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "item_hex": "01b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "topic", + "group": 2, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "item_hex": "0274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "partitions", + "group": 3, + "body_hex": "030000001910000000000000000000001d1000000000000008000000", + "item_hex": "031c000000030000001910000000000000000000001d1000000000000008000000", + "expected": { + "group": "partitions", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 8 + } + }, + { + "case": "consumer_group", + "group": 4, + "body_hex": "030000001910000000000000000000001d10000000000000000000000772656164657273", + "item_hex": "0424000000030000001910000000000000000000001d10000000000000000000000772656164657273", + "expected": { + "group": "consumer_groups", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers" + } + }, + { + "case": "consumer_group_member", + "group": 5, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "item_hex": "0540000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "expected": { + "group": "consumer_group_members", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "partition_ids": [ + 0, + 2, + 4 + ] + } + }, + { + "case": "user", + "group": 6, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "item_hex": "06480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "group": "users", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "personal_access_token_without_expiry", + "group": 7, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "item_hex": "071d0000000700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "expected": { + "group": "personal_access_tokens", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token" + } + }, + { + "case": "session", + "group": 8, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "item_hex": "08240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "group": "sessions", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + } + ], + "cursors": [ + { + "case": "zero", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d000000000000000000000000000000000000000000000000000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 0, + "checksum": "00000000000000000000000000000000", + "ordinal": 0 + }, + { + "case": "first_record_of_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAABgQAAAAAAAApcH3A4ueLUwAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000001810000000000000a5c1f7038b9e2d4c000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4120, + "checksum": "00000000000000004c2d9e8b03f7c1a5", + "ordinal": 0 + }, + { + "case": "inside_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f2000000000000000001000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4187, + "checksum": "0000000000000000f2a4c6e8b0d1f359", + "ordinal": 1 + }, + { + "case": "every_field_at_maximum", + "text": "mc1___________________________________________________________8", + "hex": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + "log_id": "ffffffffffffffffffffffffffffffff", + "op": 18446744073709551615, + "checksum": "ffffffffffffffffffffffffffffffff", + "ordinal": 4294967295 + } + ], + "page_tokens": [ + { + "case": "streams", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000000101040003000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 1, + "group": "streams", + "last_key_hex": "03000000" + }, + { + "case": "topics", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 3, + "group": "topics", + "last_key_hex": "0300000000000000" + }, + { + "case": "partitions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8DCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff0308000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "partitions", + "last_key_hex": "0300000000000000" + }, + { + "case": "consumer_groups", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8EDAADAAAAAAAAAAIAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff040c00030000000000000002000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_groups", + "last_key_hex": "030000000000000002000000" + }, + { + "case": "consumer_group_members", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8FEAADAAAAAAAAAAIAAAABAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff05100003000000000000000200000001000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_group_members", + "last_key_hex": "03000000000000000200000001000000" + }, + { + "case": "users", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAACAGBAAHAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000002006040007000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 32, + "group": "users", + "last_key_hex": "07000000" + }, + { + "case": "personal_access_tokens", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAGAHDQAHAAAACGNpLXRva2Vu", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e1000000000000060070d00070000000863692d746f6b656e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 96, + "group": "personal_access_tokens", + "last_key_hex": "070000000863692d746f6b656e" + }, + { + "case": "sessions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAKAICACUEQAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000a00808009411000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 160, + "group": "sessions", + "last_key_hex": "9411000000000000" + }, + { + "case": "longest_token_name", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8HBAH______25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubg", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff070401ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "personal_access_tokens", + "last_key_hex": "ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e" + } + ], + "requests": [ + { + "case": "earliest_every_group", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000", + "http_query": "from=earliest&limit=100" + }, + { + "case": "earliest_many_duplicate_groups", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "users" + ], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000640000000201060000", + "http_query": "from=earliest&groups=streams,users&limit=100" + }, + { + "case": "earliest_since_users_and_tokens", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "users", + "personal_access_tokens" + ], + "since": 1760353200000000, + "until": null, + "limit": 100 + }, + "request_hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001004c2a2b0841060000640000000206070000", + "http_query": "from=earliest&groups=users,personal_access_tokens&since=1760353200000000&limit=100" + }, + { + "case": "latest_streams", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [ + "streams" + ], + "since": null, + "until": null, + "limit": 1 + }, + "request_hex": "01000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000001010000", + "http_query": "from=latest&groups=streams&limit=1" + }, + { + "case": "after_cursor_with_time_range", + "input": { + "start": "after", + "cursor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "page_token": null, + "groups": [ + "topics", + "streams" + ], + "since": 1760356800000000, + "until": 1760360400000000, + "limit": 500 + }, + "request_hex": "02d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000000100f0bd010941060001009451d809410600f40100000202010000", + "http_query": "after=mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA&groups=streams,topics&since=1760356800000000&until=1760360400000000&limit=500" + }, + { + "case": "current_first_page", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201020000", + "http_query": "from=current&groups=streams,topics&limit=1000" + }, + { + "case": "current_continuation", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201024000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "http_query": "from=current¤t_after=cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA&groups=streams,topics&limit=1000" + } + ], + "responses": [ + { + "case": "changes_page", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001b10000000000000673d9e8b03f7c1a50000000000000000000000000103000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465641a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d7632011b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABsQAAAAAAAAZz2eiwP3waUAAAAAAAAAAAAAAAA", + "has_more": true, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + }, + { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "unknown_kind_and_group", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001910000000000000653d9e8b03f7c1a50000000000000000010000000002000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be01094106000104000000010904000000deadbeef191000000000000001000000653d9e8b03f7c1a500000000000000001900be01094106000104000000c80104000000deadbeef000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABkQAAAAAAAAZT2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": false, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "1", + "kind": "9", + "body": "3q2+7w==" + }, + { + "op": 4121, + "ordinal": 1, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "200", + "kind": "1", + "body": "3q2+7w==" + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "current_page", + "response_hex": "000101d7931c6e2a8f4b0d0000000000000000010000000000000000000000000000005e1000000000000001d7931c6e2a8f4b0d00000000000000005b10000000000000a73d9e8b03f7c1a500000000000000000100000001000000000200000001b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465640274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c0800000000000040000000004000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "expected": { + "status": "ok", + "phase": "current", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 1, + "retained_since": 0, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAApz2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": true, + "records": [], + "items": [ + { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + ], + "current_page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA" + } + }, + { + "case": "behind_before_op_1", + "response_hex": "010000000000000000000000000000000000000000000000000000000000000000000000000000", + "expected": { + "status": "behind", + "phase": "changes", + "log_id": null, + "oldest_op": 0, + "retained_since": 0, + "head_op": 0, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "expired", + "response_hex": "020001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "expired", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "reset", + "response_hex": "030001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "reset", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + } + ], + "large_values": { + "record_hex": "0300000000002000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff01ffffffff084624000000ffffffff0100000000002000feffffffffffffffffffffffffffffff0200000000002000", + "op": "9007199254740995", + "ordinal": 4294967295, + "timestamp": "18446744073709551615", + "checksum": "ffffffffffffffffffffffffffffffff", + "user_created_at": "9007199254740993", + "session": "9007199254740994", + "client_id": "fffffffffffffffffffffffffffffffe" + }, + "refused": { + "cursors": [ + { + "case": "wrong_prefix", + "text": "mc2_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "missing_prefix", + "text": "15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "too_long", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAAA" + }, + { + "case": "too_short", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA" + }, + { + "case": "padded", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA=" + }, + { + "case": "standard_alphabet_character", + "text": "mc1_+5McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "op_zero_with_checksum", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAA" + } + ], + "page_tokens": [ + { + "case": "wrong_prefix", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA" + }, + { + "case": "trailing_byte", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAAAA" + }, + { + "case": "wrong_key_width", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBCAADAAAAAAAAAA" + }, + { + "case": "group_outside_mask", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAIBBAADAAAA" + }, + { + "case": "head_below_anchor", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAABAAAAAAAAAAEBBAADAAAA" + }, + { + "case": "empty", + "text": "cs1_" + } + ], + "record_bodies": [ + { + "case": "unknown_member_left_reason", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000002" + }, + { + "case": "presence_byte_two", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279022c10000000000000" + }, + { + "case": "unsorted_removed_partition_ids", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000090000000600000007000000016810000000000000" + }, + { + "case": "known_kind_under_wrong_group", + "group": 2, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564" + }, + { + "case": "trailing_byte", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000000" + }, + { + "case": "truncated", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a09080706050403020194110000000000" + } + ], + "requests": [ + { + "case": "cursor_bytes_without_after", + "hex": "000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "trailing_byte", + "hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006400000000000000" + }, + { + "case": "unknown_start_mode", + "hex": "040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "non_canonical_cursor", + "hex": "02d7931c6e2a8f4b0d000000000000000000000000000000000100000000000000000000000000000000000000000064000000000000" + } + ], + "responses": [ + { + "case": "items_in_changes_phase", + "hex": "000001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + }, + { + "case": "unknown_status", + "hex": "040001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + } + ], + "queries": [ + { + "case": "limit_zero", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 0 + } + }, + { + "case": "limit_above_maximum", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 1001 + } + }, + { + "case": "since_above_until", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 2, + "until": 1, + "limit": 100 + } + }, + { + "case": "time_bound_with_latest", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 1, + "until": null, + "limit": 100 + } + }, + { + "case": "time_bound_with_current", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": 1, + "limit": 100 + } + }, + { + "case": "current_missing_parent_group", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + }, + { + "case": "page_token_for_other_groups", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + } + ], + "http_queries": [ + { + "case": "no_start", + "query": "groups=streams" + }, + { + "case": "from_and_after", + "query": "from=earliest&after=mc1_x" + }, + { + "case": "unknown_from", + "query": "from=newest" + }, + { + "case": "duplicate_key", + "query": "from=earliest&from=latest" + }, + { + "case": "unknown_key", + "query": "from=earliest&verbose=1" + }, + { + "case": "empty_group_element", + "query": "from=earliest&groups=streams,,topics" + }, + { + "case": "trailing_group_comma", + "query": "from=earliest&groups=streams," + }, + { + "case": "unknown_group", + "query": "from=earliest&groups=everything" + }, + { + "case": "negative_since", + "query": "from=earliest&since=-1" + }, + { + "case": "leading_zero_until", + "query": "from=earliest&until=01" + }, + { + "case": "non_numeric_limit", + "query": "from=earliest&limit=ten" + }, + { + "case": "token_without_current", + "query": "from=latest¤t_after=cs1_AAAA" + } + ] + } +}
diff --git a/foreign/cpp/tests/unit/metadata_changes.cpp b/foreign/cpp/tests/unit/metadata_changes.cpp new file mode 100644 index 0000000..c37a01e --- /dev/null +++ b/foreign/cpp/tests/unit/metadata_changes.cpp
@@ -0,0 +1,820 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +#include <chrono> +#include <cstddef> +#include <cstdint> +#include <fstream> +#include <functional> +#include <map> +#include <optional> +#include <set> +#include <stdexcept> +#include <string> +#include <thread> +#include <utility> +#include <variant> +#include <vector> + +#include <gtest/gtest.h> +#include <nlohmann/json.hpp> + +#include "iggy.hpp" + +namespace { + +constexpr char kFixturePath[] = "tests/fixtures/metadata_changes.json"; +constexpr std::uint32_t kInvalidCommand = 3; +constexpr std::uint32_t kInvalidFormat = 4; + +const nlohmann::json &Fixture() { + static const nlohmann::json fixture = [] { + std::ifstream file(kFixturePath); + if (!file) { + throw std::runtime_error(std::string("Cannot open ") + kFixturePath); + } + return nlohmann::json::parse(file); + }(); + return fixture; +} + +std::vector<std::uint8_t> FromHex(const std::string &hex) { + std::vector<std::uint8_t> bytes; + bytes.reserve(hex.size() / 2); + for (std::size_t index = 0; index + 1 < hex.size(); index += 2) { + bytes.push_back(static_cast<std::uint8_t>(std::stoul(hex.substr(index, 2), nullptr, 16))); + } + return bytes; +} + +std::string ToHex(const std::vector<std::uint8_t> &bytes) { + static constexpr char kDigits[] = "0123456789abcdef"; + std::string hex; + hex.reserve(bytes.size() * 2); + for (const auto byte : bytes) { + hex.push_back(kDigits[byte >> 4]); + hex.push_back(kDigits[byte & 0x0F]); + } + return hex; +} + +std::string ToHex(absl::uint128 value) { + std::vector<std::uint8_t> bytes(16); + for (std::size_t index = 16; index > 0; --index) { + bytes[index - 1] = static_cast<std::uint8_t>(absl::Uint128Low64(value) & 0xFF); + value >>= 8; + } + return ToHex(bytes); +} + +std::string ToBase64(const std::vector<std::uint8_t> &bytes) { + static constexpr char kAlphabet[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + std::string text; + text.reserve((bytes.size() + 2) / 3 * 4); + for (std::size_t index = 0; index < bytes.size(); index += 3) { + const std::size_t remaining = bytes.size() - index; + std::uint32_t chunk = static_cast<std::uint32_t>(bytes[index]) << 16; + if (remaining > 1) { + chunk |= static_cast<std::uint32_t>(bytes[index + 1]) << 8; + } + if (remaining > 2) { + chunk |= bytes[index + 2]; + } + text.push_back(kAlphabet[(chunk >> 18) & 0x3F]); + text.push_back(kAlphabet[(chunk >> 12) & 0x3F]); + text.push_back(remaining > 1 ? kAlphabet[(chunk >> 6) & 0x3F] : '='); + text.push_back(remaining > 2 ? kAlphabet[chunk & 0x3F] : '='); + } + return text; +} + +std::uint64_t ReadLittleEndian(const std::vector<std::uint8_t> &bytes) { + std::uint64_t value = 0; + for (std::size_t index = bytes.size(); index > 0; --index) { + value = (value << 8) | bytes[index - 1]; + } + return value; +} + +std::string HeaderKindName(iggy::HeaderKind kind) { + static const char *const kNames[] = {"raw", "string", "bool", "int8", "int16", + "int32", "int64", "int128", "uint8", "uint16", + "uint32", "uint64", "uint128", "float32", "float64"}; + return kNames[static_cast<std::size_t>(kind) - 1]; +} + +std::string HeaderValueText(const iggy::HeaderField &field) { + const auto &bytes = field.Value(); + switch (field.Kind()) { + case iggy::HeaderKind::String: + return std::string(bytes.begin(), bytes.end()); + case iggy::HeaderKind::Bool: + return bytes.front() == 0 ? "false" : "true"; + case iggy::HeaderKind::Uint8: + case iggy::HeaderKind::Uint16: + case iggy::HeaderKind::Uint32: + case iggy::HeaderKind::Uint64: + return std::to_string(ReadLittleEndian(bytes)); + default: + throw std::invalid_argument("No text form for option kind " + HeaderKindName(field.Kind())); + } +} + +nlohmann::json OptionsJson(const std::optional<iggy::ResourceOptions> &options) { + if (!options) { + return nullptr; + } + auto json = nlohmann::json::object(); + const auto render = [&json](const std::map<std::string, iggy::HeaderField> &entries, bool is_explicit) { + for (const auto &[key, field] : entries) { + json[key] = { + {"value", HeaderValueText(field)}, {"kind", HeaderKindName(field.Kind())}, {"explicit", is_explicit}}; + } + }; + render(options->Explicit(), true); + render(options->Derived(), false); + return json; +} + +nlohmann::json PermissionsJson(const std::optional<iggy::Permissions> &permissions) { + if (!permissions) { + return nullptr; + } + const auto &global = permissions->Global(); + nlohmann::json json{{"global", + {{"manage_servers", global.ManageServers()}, + {"read_servers", global.ReadServers()}, + {"manage_users", global.ManageUsers()}, + {"read_users", global.ReadUsers()}, + {"manage_streams", global.ManageStreams()}, + {"read_streams", global.ReadStreams()}, + {"manage_topics", global.ManageTopics()}, + {"read_topics", global.ReadTopics()}, + {"poll_messages", global.PollMessages()}, + {"send_messages", global.SendMessages()}}}, + {"streams", nullptr}}; + for (const auto &[stream_id, stream] : permissions->Streams()) { + nlohmann::json stream_json{{"manage_stream", stream.ManageStream()}, + {"read_stream", stream.ReadStream()}, + {"manage_topics", stream.ManageTopics()}, + {"read_topics", stream.ReadTopics()}, + {"poll_messages", stream.PollMessages()}, + {"send_messages", stream.SendMessages()}, + {"topics", nullptr}}; + for (const auto &[topic_id, topic] : stream.Topics()) { + stream_json["topics"][std::to_string(topic_id)] = {{"manage_topic", topic.ManageTopic()}, + {"read_topic", topic.ReadTopic()}, + {"poll_messages", topic.PollMessages()}, + {"send_messages", topic.SendMessages()}}; + } + json["streams"][std::to_string(stream_id)] = std::move(stream_json); + } + return json; +} + +template <typename Body> +nlohmann::json StreamKeyJson(const Body &body) { + return {{"stream_id", body.StreamId()}, {"stream_created_op", body.StreamCreatedOp()}}; +} + +template <typename Body> +nlohmann::json TopicKeyJson(const Body &body) { + auto json = StreamKeyJson(body); + json["topic_id"] = body.TopicId(); + json["topic_created_op"] = body.TopicCreatedOp(); + return json; +} + +template <typename Body> +nlohmann::json UserKeyJson(const Body &body) { + return {{"user_id", body.UserId()}, {"user_created_at", body.UserCreatedAt()}}; +} + +void SetOptional(nlohmann::json &json, const char *key, const std::optional<std::uint64_t> &value) { + if (value) { + json[key] = *value; + } +} + +void SetOptions(nlohmann::json &json, const std::optional<iggy::ResourceOptions> &options) { + json["options"] = OptionsJson(options); + if (!options) { + json["options_unreadable"] = true; + } +} + +// The JSON the HTTP route renders for each body, so every vector compares whole. +struct BodyJson { + nlohmann::json operator()(const iggy::MetadataStreamChange &body) const { + auto json = StreamKeyJson(body); + json["created_at"] = body.CreatedAt(); + json["name"] = body.Name(); + SetOptions(json, body.Options()); + return json; + } + + nlohmann::json operator()(const iggy::MetadataStreamDeleted &body) const { + auto json = StreamKeyJson(body); + json["created_at"] = body.CreatedAt(); + json["name"] = body.Name(); + SetOptional(json, "requested_op", body.RequestedOp()); + return json; + } + + nlohmann::json operator()(const iggy::MetadataTopicChange &body) const { + auto json = TopicKeyJson(body); + json["created_at"] = body.CreatedAt(); + json["name"] = body.Name(); + json["partitions_count"] = body.PartitionsCount(); + SetOptions(json, body.Options()); + return json; + } + + nlohmann::json operator()(const iggy::MetadataTopicItem &body) const { + auto json = TopicKeyJson(body); + json["created_at"] = body.CreatedAt(); + json["name"] = body.Name(); + SetOptions(json, body.Options()); + return json; + } + + nlohmann::json operator()(const iggy::MetadataTopicDeleted &body) const { + auto json = TopicKeyJson(body); + json["created_at"] = body.CreatedAt(); + json["name"] = body.Name(); + SetOptional(json, "requested_op", body.RequestedOp()); + return json; + } + + nlohmann::json operator()(const iggy::MetadataPartitionsChanged &body) const { + auto json = TopicKeyJson(body); + json["partitions_count"] = body.PartitionsCount(); + json["removed_partition_ids"] = body.RemovedPartitionIds(); + SetOptional(json, "requested_op", body.RequestedOp()); + return json; + } + + nlohmann::json operator()(const iggy::MetadataPartitionsItem &body) const { + auto json = TopicKeyJson(body); + json["partitions_count"] = body.PartitionsCount(); + return json; + } + + nlohmann::json operator()(const iggy::MetadataConsumerGroupCreated &body) const { + auto json = TopicKeyJson(body); + json["group_id"] = body.GroupId(); + json["name"] = body.Name(); + json["consumer_groups_count"] = body.ConsumerGroupsCount(); + return json; + } + + nlohmann::json operator()(const iggy::MetadataConsumerGroupDeleted &body) const { + auto json = TopicKeyJson(body); + json["group_id"] = body.GroupId(); + json["name"] = body.Name(); + json["consumer_groups_count"] = body.ConsumerGroupsCount(); + SetOptional(json, "requested_op", body.RequestedOp()); + return json; + } + + nlohmann::json operator()(const iggy::MetadataConsumerGroupItem &body) const { + auto json = TopicKeyJson(body); + json["group_id"] = body.GroupId(); + json["name"] = body.Name(); + return json; + } + + nlohmann::json operator()(const iggy::MetadataMemberJoined &body) const { + auto json = TopicKeyJson(body); + json["group_id"] = body.GroupId(); + json["member_id"] = body.MemberId(); + json["client_id"] = ToHex(body.ClientId()); + json["members_count"] = body.MembersCount(); + json["partition_ids"] = body.PartitionIds(); + return json; + } + + nlohmann::json operator()(const iggy::MetadataMemberLeft &body) const { + auto json = TopicKeyJson(body); + json["group_id"] = body.GroupId(); + json["member_id"] = body.MemberId(); + json["client_id"] = ToHex(body.ClientId()); + json["members_count"] = body.MembersCount(); + json["reason"] = iggy::to_string(body.Reason()); + return json; + } + + nlohmann::json operator()(const iggy::MetadataPartitionHandedOver &body) const { + auto json = TopicKeyJson(body); + json["group_id"] = body.GroupId(); + json["partition_id"] = body.PartitionId(); + if (body.MemberId()) { + json["member_id"] = *body.MemberId(); + } + return json; + } + + nlohmann::json operator()(const iggy::MetadataMemberItem &body) const { + auto json = TopicKeyJson(body); + json["group_id"] = body.GroupId(); + json["member_id"] = body.MemberId(); + json["client_id"] = ToHex(body.ClientId()); + json["partition_ids"] = body.PartitionIds(); + return json; + } + + nlohmann::json operator()(const iggy::MetadataUserChange &body) const { + auto json = UserKeyJson(body); + json["username"] = body.Username(); + json["status"] = body.Status() == iggy::UserStatus::Active ? "active" : "inactive"; + json["permissions"] = PermissionsJson(body.Permissions()); + SetOptions(json, body.Options()); + return json; + } + + nlohmann::json operator()(const iggy::MetadataUserIdentity &body) const { + auto json = UserKeyJson(body); + json["username"] = body.Username(); + return json; + } + + nlohmann::json operator()(const iggy::MetadataUserPermissionsUpdated &body) const { + auto json = UserKeyJson(body); + json["username"] = body.Username(); + json["permissions"] = PermissionsJson(body.Permissions()); + json["reason"] = iggy::to_string(body.Reason()); + return json; + } + + nlohmann::json operator()(const iggy::MetadataTokenChange &body) const { + auto json = UserKeyJson(body); + json["name"] = body.Name(); + SetOptional(json, "expiry", body.Expiry()); + return json; + } + + nlohmann::json operator()(const iggy::MetadataTokenDeleted &body) const { + auto json = UserKeyJson(body); + json["name"] = body.Name(); + json["reason"] = iggy::to_string(body.Reason()); + return json; + } + + nlohmann::json operator()(const iggy::MetadataSession &body) const { + auto json = UserKeyJson(body); + json["client_id"] = ToHex(body.ClientId()); + json["session"] = body.Session(); + return json; + } + + nlohmann::json operator()(const iggy::MetadataSessionClosed &body) const { + auto json = UserKeyJson(body); + json["client_id"] = ToHex(body.ClientId()); + json["session"] = body.Session(); + json["reason"] = iggy::to_string(body.Reason()); + return json; + } + + nlohmann::json operator()(const iggy::MetadataUnknownChange &body) const { + return {{"group", std::to_string(body.GroupCode())}, + {"kind", std::to_string(body.KindCode())}, + {"body", ToBase64(body.Body())}}; + } + + nlohmann::json operator()(const iggy::MetadataUnknownJsonChange &body) const { + return nlohmann::json::parse(body.Json()); + } + + nlohmann::json operator()(const iggy::MetadataUnknownItem &body) const { + return {{"group", std::to_string(body.GroupCode())}, {"body", ToBase64(body.Body())}}; + } + + nlohmann::json operator()(const iggy::MetadataUnknownJsonItem &body) const { + return nlohmann::json::parse(body.Json()); + } +}; + +nlohmann::json RecordJson(const iggy::MetadataChangeRecord &record) { + nlohmann::json json{{"op", record.Op()}, + {"ordinal", record.Ordinal()}, + {"checksum", ToHex(record.Checksum())}, + {"timestamp", record.Timestamp()}}; + if (record.ActorUserId()) { + json["actor_user_id"] = *record.ActorUserId(); + } + if (record.Kind()) { + json["group"] = iggy::to_string(*record.Group()); + json["kind"] = iggy::to_string(*record.Kind()); + } + json.update(std::visit(BodyJson{}, record.Change())); + return json; +} + +nlohmann::json ItemJson(const iggy::MetadataCurrentItem &item) { + auto json = std::visit(BodyJson{}, item.Value()); + if (item.Group()) { + json["group"] = iggy::to_string(*item.Group()); + } + return json; +} + +nlohmann::json AnswerJson(const iggy::MetadataChanges &answer) { + nlohmann::json json{{"status", iggy::to_string(answer.Status())}, + {"phase", iggy::to_string(answer.Phase())}, + {"log_id", nullptr}, + {"oldest_op", answer.OldestOp()}, + {"retained_since", answer.RetainedSince()}, + {"head_op", answer.HeadOp()}, + {"next", nullptr}, + {"has_more", answer.HasMore()}, + {"records", nlohmann::json::array()}, + {"items", nlohmann::json::array()}, + {"current_page_token", nullptr}}; + if (answer.LogId()) { + json["log_id"] = ToHex(*answer.LogId()); + } + if (answer.Next()) { + json["next"] = answer.Next()->ToString(); + } + for (const auto &record : answer.Records()) { + json["records"].push_back(RecordJson(record)); + } + for (const auto &item : answer.Items()) { + json["items"].push_back(ItemJson(item)); + } + if (answer.CurrentPageToken()) { + json["current_page_token"] = answer.CurrentPageToken()->ToString(); + } + return json; +} + +// Walks the codes until to_string refuses one, so a value added to a public +// enum is compared with the fixture without a second list here. +template <typename Code> +std::map<std::string, std::uint32_t> PublicCodes() { + std::map<std::string, std::uint32_t> codes; + for (unsigned value = 0; value <= 255; ++value) { + try { + codes.emplace(iggy::to_string(static_cast<Code>(value)), value); + } catch (const std::invalid_argument &) { + } + } + return codes; +} + +std::map<std::string, std::uint32_t> FixtureCodes(const nlohmann::json &table) { + std::map<std::string, std::uint32_t> codes; + for (const auto &[name, value] : table.items()) { + codes.emplace(name, value.is_object() ? value.at("code").get<std::uint32_t>() : value.get<std::uint32_t>()); + } + return codes; +} + +iggy::MetadataChangeGroup GroupNamed(const std::string &name) { + for (const auto &[group_name, code] : PublicCodes<iggy::MetadataChangeGroup>()) { + if (group_name == name) { + return static_cast<iggy::MetadataChangeGroup>(code); + } + } + throw std::invalid_argument("Unknown group " + name); +} + +iggy::MetadataChangesQuery QueryFrom(const nlohmann::json &input) { + const auto start = input.at("start").get<std::string>(); + auto query = iggy::MetadataChangesQuery::Earliest(); + if (start == "latest") { + query = iggy::MetadataChangesQuery::Latest(); + } else if (start == "after") { + query = iggy::MetadataChangesQuery::After(iggy::MetadataChangeCursor::Parse(input.at("cursor"))); + } else if (start == "current") { + query = + input.at("page_token").is_null() + ? iggy::MetadataChangesQuery::Current() + : iggy::MetadataChangesQuery::Current(iggy::MetadataCurrentPageToken::Parse(input.at("page_token"))); + } + std::vector<iggy::MetadataChangeGroup> groups; + for (const auto &name : input.at("groups")) { + groups.push_back(GroupNamed(name)); + } + query.SetGroups(std::move(groups)).SetLimit(input.at("limit").get<std::uint32_t>()); + if (!input.at("since").is_null()) { + query.SetSince(input.at("since").get<std::uint64_t>()); + } + if (!input.at("until").is_null()) { + query.SetUntil(input.at("until").get<std::uint64_t>()); + } + return query; +} + +// A record envelope as the fixture's record_hex convention lays it out, with no actor. +std::vector<std::uint8_t> RecordEnvelope(std::uint8_t group, std::uint8_t kind, const std::vector<std::uint8_t> &body) { + std::vector<std::uint8_t> bytes(8 + 4 + 16 + 8 + 1, 0); + bytes.push_back(group); + bytes.push_back(kind); + const auto length = static_cast<std::uint32_t>(body.size()); + for (unsigned shift = 0; shift < 32; shift += 8) { + bytes.push_back(static_cast<std::uint8_t>((length >> shift) & 0xFF)); + } + bytes.insert(bytes.end(), body.begin(), body.end()); + return bytes; +} + +::testing::AssertionResult ThrowsIggyError(const std::function<void()> &operation, + std::uint32_t code, + const std::string &kind) { + try { + operation(); + } catch (const iggy::IggyException &exception) { + if (exception.Code() == code && exception.Kind() == kind) { + return ::testing::AssertionSuccess(); + } + return ::testing::AssertionFailure() << "expected " << kind << " (" << code << "), got '" << exception.what() + << "' with kind '" << exception.Kind() << "'"; + } + return ::testing::AssertionFailure() << "expected " << kind << " (" << code << "), nothing was thrown"; +} + +} // namespace + +TEST(MetadataChangesFixtureTest, GivenFixtureConstantsWhenComparingPublicEnumsShouldMatchCodesAndNames) { + const auto &constants = Fixture().at("constants"); + EXPECT_EQ(PublicCodes<iggy::MetadataChangeGroup>(), FixtureCodes(constants.at("groups"))); + EXPECT_EQ(PublicCodes<iggy::MetadataChangeKind>(), FixtureCodes(constants.at("kinds"))); + EXPECT_EQ(PublicCodes<iggy::MetadataChangeStatus>(), FixtureCodes(constants.at("statuses"))); + EXPECT_EQ(PublicCodes<iggy::MetadataChangePhase>(), FixtureCodes(constants.at("phases"))); + const auto &start_modes = constants.at("start_modes"); + EXPECT_EQ(static_cast<std::uint32_t>(iggy::MetadataChangesQuery::Start::Earliest), start_modes.at("earliest")); + EXPECT_EQ(static_cast<std::uint32_t>(iggy::MetadataChangesQuery::Start::Latest), start_modes.at("latest")); + EXPECT_EQ(static_cast<std::uint32_t>(iggy::MetadataChangesQuery::Start::After), start_modes.at("after")); + EXPECT_EQ(static_cast<std::uint32_t>(iggy::MetadataChangesQuery::Start::Current), start_modes.at("current")); + EXPECT_EQ(PublicCodes<iggy::MetadataMemberLeftReason>(), FixtureCodes(constants.at("member_left_reasons"))); + EXPECT_EQ(PublicCodes<iggy::MetadataTokenDeletedReason>(), FixtureCodes(constants.at("token_deleted_reasons"))); + EXPECT_EQ(PublicCodes<iggy::MetadataSessionClosedReason>(), FixtureCodes(constants.at("session_closed_reasons"))); + EXPECT_EQ(PublicCodes<iggy::MetadataPermissionsUpdatedReason>(), + FixtureCodes(constants.at("permissions_updated_reasons"))); + EXPECT_EQ(iggy::MetadataChangesQuery::kDefaultLimit, constants.at("default_limit").get<std::uint32_t>()); + EXPECT_EQ(iggy::MetadataChangesQuery::kMaxLimit, constants.at("max_limit").get<std::uint32_t>()); + EXPECT_EQ(iggy::MetadataChangesQuery::Earliest().Limit(), iggy::MetadataChangesQuery::kDefaultLimit); +} + +TEST(MetadataChangesFixtureTest, GivenRecordVectorsWhenDecodingShouldReachTheExpectedJsonForEveryKind) { + std::set<std::uint32_t> kinds; + for (const auto &vector : Fixture().at("records")) { + SCOPED_TRACE(vector.at("case").get<std::string>()); + const auto record = iggy::MetadataChangeRecord::FromBytes(FromHex(vector.at("record_hex"))); + ASSERT_TRUE(record.Kind().has_value()); + EXPECT_EQ(static_cast<std::uint32_t>(*record.Kind()), vector.at("kind").get<std::uint32_t>()); + EXPECT_EQ(static_cast<std::uint32_t>(*record.Group()), vector.at("group").get<std::uint32_t>()); + EXPECT_EQ(RecordJson(record), vector.at("expected")); + kinds.insert(static_cast<std::uint32_t>(*record.Kind())); + } + std::set<std::uint32_t> catalog; + for (const auto &[name, code] : PublicCodes<iggy::MetadataChangeKind>()) { + catalog.insert(code); + } + EXPECT_EQ(kinds, catalog); +} + +TEST(MetadataChangesFixtureTest, GivenItemVectorsWhenDecodingShouldReachTheExpectedJsonForEveryGroup) { + std::set<std::uint32_t> groups; + for (const auto &vector : Fixture().at("items")) { + SCOPED_TRACE(vector.at("case").get<std::string>()); + const auto item = iggy::MetadataCurrentItem::FromBytes(FromHex(vector.at("item_hex"))); + ASSERT_TRUE(item.Group().has_value()); + EXPECT_EQ(static_cast<std::uint32_t>(*item.Group()), vector.at("group").get<std::uint32_t>()); + EXPECT_EQ(ItemJson(item), vector.at("expected")); + groups.insert(static_cast<std::uint32_t>(*item.Group())); + } + EXPECT_EQ(groups.size(), PublicCodes<iggy::MetadataChangeGroup>().size()); +} + +TEST(MetadataChangesFixtureTest, GivenResponseVectorsWhenDecodingShouldReachTheExpectedAnswers) { + for (const auto &vector : Fixture().at("responses")) { + SCOPED_TRACE(vector.at("case").get<std::string>()); + const auto answer = iggy::MetadataChanges::FromBytes(FromHex(vector.at("response_hex"))); + EXPECT_EQ(AnswerJson(answer), vector.at("expected")); + } +} + +TEST(MetadataChangesFixtureTest, GivenUnknownKindAndGroupWhenDecodingShouldKeepCodesAndRawBody) { + for (const auto &vector : Fixture().at("responses")) { + if (vector.at("case") != "unknown_kind_and_group") { + continue; + } + const auto answer = iggy::MetadataChanges::FromBytes(FromHex(vector.at("response_hex"))); + ASSERT_EQ(answer.Records().size(), 2u); + for (const auto &record : answer.Records()) { + EXPECT_FALSE(record.Kind().has_value()); + EXPECT_FALSE(record.Group().has_value()); + const auto *unknown = std::get_if<iggy::MetadataUnknownChange>(&record.Change()); + ASSERT_NE(unknown, nullptr); + EXPECT_EQ(unknown->Body(), (std::vector<std::uint8_t>{0xde, 0xad, 0xbe, 0xef})); + } + return; + } + FAIL() << "No unknown_kind_and_group vector"; +} + +TEST(MetadataChangesFixtureTest, GivenLargeValuesWhenDecodingShouldKeepEveryBit) { + const auto &vector = Fixture().at("large_values"); + const auto record = iggy::MetadataChangeRecord::FromBytes(FromHex(vector.at("record_hex"))); + EXPECT_EQ(std::to_string(record.Op()), vector.at("op").get<std::string>()); + EXPECT_EQ(record.Ordinal(), vector.at("ordinal").get<std::uint32_t>()); + EXPECT_EQ(std::to_string(record.Timestamp()), vector.at("timestamp").get<std::string>()); + EXPECT_EQ(ToHex(record.Checksum()), vector.at("checksum").get<std::string>()); + const auto *session = std::get_if<iggy::MetadataSession>(&record.Change()); + ASSERT_NE(session, nullptr); + EXPECT_EQ(std::to_string(session->UserCreatedAt()), vector.at("user_created_at").get<std::string>()); + EXPECT_EQ(std::to_string(session->Session()), vector.at("session").get<std::string>()); + EXPECT_EQ(ToHex(session->ClientId()), vector.at("client_id").get<std::string>()); +} + +TEST(MetadataChangesFixtureTest, GivenCursorVectorsWhenParsingShouldMatchTheirFieldsAndText) { + const auto &constants = Fixture().at("constants"); + for (const auto &vector : Fixture().at("cursors")) { + SCOPED_TRACE(vector.at("case").get<std::string>()); + const auto text = vector.at("text").get<std::string>(); + const auto cursor = iggy::MetadataChangeCursor::Parse(text); + EXPECT_EQ(ToHex(cursor.LogId()), vector.at("log_id").get<std::string>()); + EXPECT_EQ(cursor.Op(), vector.at("op").get<std::uint64_t>()); + EXPECT_EQ(ToHex(cursor.Checksum()), vector.at("checksum").get<std::string>()); + EXPECT_EQ(cursor.Ordinal(), vector.at("ordinal").get<std::uint32_t>()); + EXPECT_EQ(cursor.IsZero(), cursor.Op() == 0); + EXPECT_EQ(cursor.ToString(), text); + EXPECT_EQ(cursor.ToString().size(), constants.at("cursor_text_length").get<std::size_t>()); + EXPECT_EQ(cursor.ToString().rfind(constants.at("cursor_prefix").get<std::string>(), 0), 0u); + EXPECT_EQ(cursor, iggy::MetadataChangeCursor::Parse(cursor.ToString())); + } +} + +TEST(MetadataChangesFixtureTest, GivenPageTokenVectorsWhenParsingShouldMatchTheirFieldsAndText) { + for (const auto &vector : Fixture().at("page_tokens")) { + SCOPED_TRACE(vector.at("case").get<std::string>()); + const auto text = vector.at("text").get<std::string>(); + const auto token = iggy::MetadataCurrentPageToken::Parse(text); + EXPECT_EQ(token.ToString(), text); + EXPECT_EQ(token.Anchor().ToString(), vector.at("anchor").get<std::string>()); + EXPECT_EQ(token.RequiredHead(), vector.at("required_head").get<std::uint64_t>()); + std::uint32_t mask = 0; + for (const auto group : token.Groups()) { + mask |= 1U << (static_cast<std::uint32_t>(group) - 1); + } + EXPECT_EQ(mask, vector.at("groups_mask").get<std::uint32_t>()); + EXPECT_EQ(iggy::to_string(token.LastGroup()), vector.at("group").get<std::string>()); + EXPECT_EQ(ToHex(token.LastKey()), vector.at("last_key_hex").get<std::string>()); + } +} + +TEST(MetadataChangesFixtureTest, GivenRequestVectorsWhenEncodingShouldValidateAndMatchTheRequestBytes) { + for (const auto &vector : Fixture().at("requests")) { + SCOPED_TRACE(vector.at("case").get<std::string>()); + const auto query = QueryFrom(vector.at("input")); + EXPECT_NO_THROW(query.Validate()); + EXPECT_EQ(ToHex(query.ToBytes()), vector.at("request_hex").get<std::string>()); + } +} + +TEST(MetadataChangesFixtureTest, GivenRefusedCursorAndTokenTextWhenParsingShouldThrowInvalidCommandWithoutTheText) { + const auto expect_refused = [](const std::string &text, const std::function<void()> &parse) { + EXPECT_TRUE(ThrowsIggyError(parse, kInvalidCommand, "invalid_command")); + try { + parse(); + } catch (const iggy::IggyException &error) { + EXPECT_EQ(std::string(error.what()).find(text), std::string::npos); + } + }; + const auto &refused = Fixture().at("refused"); + for (const auto &vector : refused.at("cursors")) { + SCOPED_TRACE(vector.at("case").get<std::string>()); + const auto text = vector.at("text").get<std::string>(); + expect_refused(text, [&text] { iggy::MetadataChangeCursor::Parse(text); }); + } + for (const auto &vector : refused.at("page_tokens")) { + SCOPED_TRACE(vector.at("case").get<std::string>()); + const auto text = vector.at("text").get<std::string>(); + expect_refused(text, [&text] { iggy::MetadataCurrentPageToken::Parse(text); }); + } +} + +TEST(MetadataChangesFixtureTest, GivenNonUtf8CursorAndTokenTextWhenParsingShouldThrowInvalidCommand) { + const std::string cursor_text = "mc1_" + std::string(58, 'A') + "\xff"; + const std::string token_text = "cs1_\xff"; + EXPECT_TRUE(ThrowsIggyError([&cursor_text] { iggy::MetadataChangeCursor::Parse(cursor_text); }, kInvalidCommand, + "invalid_command")); + EXPECT_TRUE(ThrowsIggyError([&token_text] { iggy::MetadataCurrentPageToken::Parse(token_text); }, kInvalidCommand, + "invalid_command")); +} + +TEST(MetadataChangesFixtureTest, GivenRefusedQueriesWhenValidatingShouldThrowInvalidCommand) { + for (const auto &vector : Fixture().at("refused").at("queries")) { + SCOPED_TRACE(vector.at("case").get<std::string>()); + const auto query = QueryFrom(vector.at("input")); + EXPECT_TRUE(ThrowsIggyError([&query] { query.Validate(); }, kInvalidCommand, "invalid_command")); + } +} + +TEST(MetadataChangesFixtureTest, GivenRefusedRecordBodiesWhenDecodingShouldThrowInvalidFormat) { + for (const auto &vector : Fixture().at("refused").at("record_bodies")) { + SCOPED_TRACE(vector.at("case").get<std::string>()); + const auto bytes = RecordEnvelope(vector.at("group").get<std::uint8_t>(), vector.at("kind").get<std::uint8_t>(), + FromHex(vector.at("body_hex"))); + EXPECT_TRUE(ThrowsIggyError([&bytes] { iggy::MetadataChangeRecord::FromBytes(bytes); }, kInvalidFormat, + "invalid_format")); + } +} + +// Refused request bytes and HTTP queries are left out because the Rust SDK under this one parses them. +TEST(MetadataChangesFixtureTest, GivenRefusedResponsesWhenDecodingShouldThrowInvalidFormat) { + for (const auto &vector : Fixture().at("refused").at("responses")) { + SCOPED_TRACE(vector.at("case").get<std::string>()); + const auto bytes = FromHex(vector.at("hex")); + EXPECT_TRUE( + ThrowsIggyError([&bytes] { iggy::MetadataChanges::FromBytes(bytes); }, kInvalidFormat, "invalid_format")); + } +} + +TEST(MetadataChangesFixtureTest, GivenGroupCodeThisBuildDoesNotKnowWhenEncodingShouldThrowInvalidCommand) { + const auto query = iggy::MetadataChangesQuery::Earliest().SetGroups({static_cast<iggy::MetadataChangeGroup>(9)}); + EXPECT_TRUE(ThrowsIggyError([&query] { static_cast<void>(query.ToBytes()); }, kInvalidCommand, "invalid_command")); + EXPECT_TRUE(ThrowsIggyError([&query] { query.Validate(); }, kInvalidCommand, "invalid_command")); +} + +TEST(MetadataChangesFixtureTest, GivenReadmeSampleWhenCompilingShouldMatchThePublicApi) { + using Groups = std::vector<iggy::MetadataChangeGroup>; + using ApplyStreamFunction = std::function<void(iggy::MetadataChangeKind, const iggy::MetadataStreamChange &)>; + using StageItemFunction = std::function<void(const iggy::MetadataCurrentItem &)>; + using ClearStagedFunction = std::function<void()>; + // Never invoked, they only have to compile, so the README sample cannot drift from the API. + const auto rebuild = [](iggy::IggyBlockingClient &client, const Groups &groups, + const ClearStagedFunction &ClearStaged, const StageItemFunction &StageItem) { + ClearStaged(); + auto query = iggy::MetadataChangesQuery::Current().SetGroups(groups); + while (true) { + const auto page = client.GetMetadataChanges(query); + switch (page.Status()) { + case iggy::MetadataChangeStatus::Ok: + break; + case iggy::MetadataChangeStatus::Behind: + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + continue; + case iggy::MetadataChangeStatus::Expired: + case iggy::MetadataChangeStatus::Reset: + ClearStaged(); + query = iggy::MetadataChangesQuery::Current().SetGroups(groups); + continue; + } + for (const auto &item : page.Items()) { + StageItem(item); + } + if (!page.CurrentPageToken()) { + return iggy::MetadataChangesQuery::After(*page.Next()).SetGroups(groups); + } + query = iggy::MetadataChangesQuery::Current(*page.CurrentPageToken()).SetGroups(groups); + } + }; + const auto follow = [&rebuild](iggy::IggyBlockingClient &client, const Groups &groups, + const ApplyStreamFunction &ApplyStream, const ClearStagedFunction &ClearStaged, + const StageItemFunction &StageItem) { + auto query = rebuild(client, groups, ClearStaged, StageItem); + while (true) { + const auto page = client.GetMetadataChanges(query); + switch (page.Status()) { + case iggy::MetadataChangeStatus::Ok: + for (const auto &record : page.Records()) { + if (const auto *stream = std::get_if<iggy::MetadataStreamChange>(&record.Change())) { + ApplyStream(*record.Kind(), *stream); + } + } + query = iggy::MetadataChangesQuery::After(*page.Next()).SetGroups(groups); + if (!page.HasMore()) { + std::this_thread::sleep_for(std::chrono::seconds(1)); + } + break; + case iggy::MetadataChangeStatus::Behind: + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + break; + case iggy::MetadataChangeStatus::Expired: + case iggy::MetadataChangeStatus::Reset: + query = rebuild(client, groups, ClearStaged, StageItem); + break; + } + } + }; + static_cast<void>(follow); +}
diff --git a/foreign/csharp/Iggy_SDK.Tests.BDD/Context/TestContext.cs b/foreign/csharp/Iggy_SDK.Tests.BDD/Context/TestContext.cs index 91de7b6..1203827 100644 --- a/foreign/csharp/Iggy_SDK.Tests.BDD/Context/TestContext.cs +++ b/foreign/csharp/Iggy_SDK.Tests.BDD/Context/TestContext.cs
@@ -35,6 +35,7 @@ public Message? LastSendMessage { get; set; } public bool RedirectionOccurred { get; set; } public uint? LastStreamId { get; set; } + public byte[]? LastStreamKey { get; set; } public byte[]? LastRawResponse { get; set; } public Exception? LastRawError { get; set; } public Exception? LastCallError { get; set; }
diff --git a/foreign/csharp/Iggy_SDK.Tests.BDD/Context/TestHooks.cs b/foreign/csharp/Iggy_SDK.Tests.BDD/Context/TestHooks.cs index 23b8e44..7369160 100644 --- a/foreign/csharp/Iggy_SDK.Tests.BDD/Context/TestHooks.cs +++ b/foreign/csharp/Iggy_SDK.Tests.BDD/Context/TestHooks.cs
@@ -36,6 +36,7 @@ _context.CreatedStream = null; _context.RedirectionOccurred = false; _context.LastStreamId = null; + _context.LastStreamKey = null; _context.LastRawResponse = null; _context.LastRawError = null; _context.LastCallError = null;
diff --git a/foreign/csharp/Iggy_SDK.Tests.BDD/Iggy_SDK.Tests.BDD.csproj b/foreign/csharp/Iggy_SDK.Tests.BDD/Iggy_SDK.Tests.BDD.csproj index aa5a38b..6784190 100644 --- a/foreign/csharp/Iggy_SDK.Tests.BDD/Iggy_SDK.Tests.BDD.csproj +++ b/foreign/csharp/Iggy_SDK.Tests.BDD/Iggy_SDK.Tests.BDD.csproj
@@ -73,6 +73,9 @@ <ReqnrollFeatureFile Include="..\..\..\bdd\scenarios\resource_options_patch.feature"> <Link>Features\resource_options_patch.feature</Link> </ReqnrollFeatureFile> + <ReqnrollFeatureFile Include="..\..\..\bdd\scenarios\metadata_changes.feature"> + <Link>Features\metadata_changes.feature</Link> + </ReqnrollFeatureFile> <ReqnrollFeatureFile Include="..\..\..\bdd\scenarios\session_contract.feature"> <Link>Features\session_contract.feature</Link> </ReqnrollFeatureFile>
diff --git a/foreign/csharp/Iggy_SDK.Tests.BDD/StepDefinitions/MetadataChangesSteps.cs b/foreign/csharp/Iggy_SDK.Tests.BDD/StepDefinitions/MetadataChangesSteps.cs new file mode 100644 index 0000000..c488ae7 --- /dev/null +++ b/foreign/csharp/Iggy_SDK.Tests.BDD/StepDefinitions/MetadataChangesSteps.cs
@@ -0,0 +1,306 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +using Apache.Iggy.Configuration; +using Apache.Iggy.Contracts; +using Apache.Iggy.Contracts.Auth; +using Apache.Iggy.Encryption; +using Apache.Iggy.Enums; +using Apache.Iggy.Exceptions; +using Apache.Iggy.Factory; +using Apache.Iggy.Headers; +using Reqnroll; +using Shouldly; +using TestContext = Apache.Iggy.Tests.BDD.Context.TestContext; +using TestEnvironment = Apache.Iggy.Tests.BDD.Context.TestEnvironment; + +namespace Apache.Iggy.Tests.BDD.StepDefinitions; + +[Binding] +public sealed class MetadataChangesSteps +{ + private const uint RootUserId = 0; + private const int FollowAttempts = 100; + private const int Unauthorized = 41; + + private static readonly TimeSpan FollowInterval = TimeSpan.FromMilliseconds(100); + private static readonly string[] HiddenOptions = ["encryption_key", "encryption_key_wrapped", "critical_keys"]; + + private readonly TestContext _context; + private IReadOnlyList<MetadataChangeGroup> _groups = []; + private MetadataChangeCursor? _cursor; + private List<MetadataChangeRecord> _followed = []; + private List<MetadataCurrentItem> _items = []; + private uint? _userId; + + public MetadataChangesSteps(TestContext context) + { + _context = context; + } + + // The client may be signed in as a user without manage users by now, so a root client removes the user. + [AfterScenario(Order = 0)] + public async Task DeleteCreatedUser() + { + if (_userId is not { } userId) + { + return; + } + + using var root = IggyClientFactory.CreateClient(new IggyClientConfigurator + { + BaseAddress = _context.TcpUrl, + Protocol = Protocol.Tcp + }); + await root.ConnectAsync(); + await root.LoginUserAsync(TestEnvironment.RootUsername, TestEnvironment.RootPassword); + await root.DeleteUserAsync(Identifier.Numeric(userId)); + } + + [Given("I hold the latest metadata change cursor for the groups {string}")] + public async Task HoldLatestCursor(string groups) + { + _groups = ParseGroups(groups); + var page = await Read(MetadataChangeStart.Latest); + page.Status.ShouldBe(MetadataChangeStatus.Ok); + _cursor = page.Next.ShouldNotBeNull(); + } + + [When("I create and delete {int} streams named {string}")] + public async Task ChurnStreams(int count, string prefix) + { + for (var index = 0; index < count; index++) + { + var stream = await _context.IggyClient.CreateStreamAsync($"{prefix}-{index}"); + stream.ShouldNotBeNull(); + await _context.IggyClient.DeleteStreamAsync(Identifier.Numeric(stream.Id)); + } + } + + [Then("following the metadata changes should return the kinds {string} for the stream")] + public async Task FollowedKinds(string kinds) + { + var expected = kinds.Split(',').Select(name => Enum.Parse<MetadataChangeKind>(name.Replace("_", ""), true)) + .ToList(); + var streamId = StreamId(); + var records = new List<MetadataChangeRecord>(); + var cursor = _cursor.ShouldNotBeNull(); + // A two phase delete commits its record after the call returns. + for (var attempt = 0; attempt < FollowAttempts; attempt++) + { + var page = await Read(MetadataChangeStart.After(cursor)); + page.Status.ShouldBe(MetadataChangeStatus.Ok); + cursor = page.Next.ShouldNotBeNull(); + records.AddRange(page.Records); + if (page.HasMore) + { + continue; + } + + if (records.Count(record => ChangeStreamId(record) == streamId) >= expected.Count) + { + break; + } + + await Task.Delay(FollowInterval); + } + + foreach (var record in records) + { + record.Group.ShouldNotBeNull(); + _groups.ShouldContain(record.Group.Value); + } + + _followed = records.Where(record => ChangeStreamId(record) == streamId).ToList(); + _followed.Select(record => record.Kind).ShouldBe(expected.Select(kind => (MetadataChangeKind?)kind)); + } + + [Then("each followed change should name the root user as its actor")] + public void RootActor() + { + _followed.ShouldAllBe(record => record.ActorUserId == RootUserId); + } + + [Then("the followed {string} change should carry a requested op before its own op")] + public void RequestedOpBeforeOwnOp(string kind) + { + var record = Followed(kind); + var deleted = record.Body.ShouldBeOfType<MetadataStreamDeleted>(); + deleted.RequestedOp.ShouldNotBeNull(); + deleted.RequestedOp.Value.ShouldBeLessThan(record.Op); + } + + [Then("the followed {string} change should report encryption key source {string}")] + public void KeySource(string kind, string source) + { + FollowedStream(kind).EncryptionKeySource.ShouldBe(Enum.Parse<StreamEncryptionSource>(source, true)); + } + + [Then("the followed {string} change should carry the stream key fingerprint")] + public void KeyFingerprint(string kind) + { + var key = _context.LastStreamKey.ShouldNotBeNull(); + FollowedStream(kind).EncryptionKeyFingerprint.ShouldBe(EncryptionKeys.Fingerprint(key)); + } + + [Then("the followed changes should expose no encryption key material")] + public void NoKeyMaterial() + { + foreach (var stream in _followed.Select(record => record.Body).OfType<MetadataStreamChange>()) + { + stream.OptionsUnreadable.ShouldBeFalse(); + foreach (var options in new[] { stream.Options, stream.DerivedOptions }) + { + foreach (var name in HiddenOptions) + { + options.ShouldNotBeNull().ContainsKey(HeaderKey.FromString(name)).ShouldBeFalse(); + } + } + } + } + + [Then("following the metadata changes should answer {string}")] + public async Task FollowStatus(string status) + { + var expected = Enum.Parse<MetadataChangeStatus>(status, true); + var page = await Read(MetadataChangeStart.After(_cursor.ShouldNotBeNull())); + page.Status.ShouldBe(expected); + if (expected != MetadataChangeStatus.Ok) + { + page.Records.ShouldBeEmpty(); + page.Next.ShouldBeNull(); + } + } + + [When("I read the current state for the groups {string}")] + public async Task ReadCurrentState(string groups) + { + _groups = ParseGroups(groups); + var start = MetadataChangeStart.Current(); + var items = new List<MetadataCurrentItem>(); + while (true) + { + var page = await Read(start); + page.Status.ShouldBe(MetadataChangeStatus.Ok); + items.AddRange(page.Items); + if (page.CurrentPageToken is not { } token) + { + _cursor = page.Next.ShouldNotBeNull(); + break; + } + + start = MetadataChangeStart.Current(token); + } + + _items = items; + } + + [Then("the current state should list the same streams as the stream list")] + public async Task CurrentStateMatchesList() + { + var listed = (await _context.IggyClient.GetStreamsAsync()) + .ToDictionary(stream => stream.Id, stream => stream.Name); + var projected = _items.Select(item => item.Body).OfType<MetadataStreamChange>() + .ToDictionary(stream => stream.StreamId, stream => stream.Name); + projected.ShouldBe(listed, ignoreOrder: true); + } + + [Given("a user named {string} with password {string} and no server permissions exists")] + public async Task UserWithoutServerPermissions(string username, string password) + { + var user = await _context.IggyClient.CreateUserAsync(username, password, UserStatus.Active, new Permissions + { + Global = new GlobalPermissions + { + ManageServers = false, + ReadServers = true, + ManageUsers = false, + ReadUsers = false, + ManageStreams = false, + ReadStreams = false, + ManageTopics = false, + ReadTopics = false, + PollMessages = false, + SendMessages = false + } + }); + user.ShouldNotBeNull(); + _userId = user.Id; + } + + [When("I authenticate as {string} with password {string}")] + public async Task AuthenticateAs(string username, string password) + { + await _context.IggyClient.LogoutUserAsync(); + (await _context.IggyClient.LoginUserAsync(username, password)).ShouldNotBeNull(); + } + + [Then("reading the metadata changes should be refused as unauthorized")] + public async Task ReadingRefused() + { + _groups = []; + var error = await Should.ThrowAsync<IggyInvalidStatusCodeException>(Read(MetadataChangeStart.Earliest)); + error.StatusCode.ShouldBe(Unauthorized); + } + + private Task<MetadataChanges> Read(MetadataChangeStart start) + { + return _context.IggyClient.GetMetadataChangesAsync(new MetadataChangesQuery + { + Start = start, + Groups = _groups, + Limit = MetadataChangesQuery.MAX_LIMIT + }); + } + + private MetadataChangeRecord Followed(string kind) + { + var parsed = Enum.Parse<MetadataChangeKind>(kind.Replace("_", ""), true); + return _followed.First(record => record.Kind == parsed); + } + + private MetadataStreamChange FollowedStream(string kind) + { + var stream = Followed(kind).Body.ShouldBeOfType<MetadataStreamChange>(); + stream.OptionsUnreadable.ShouldBeFalse(); + return stream; + } + + private uint StreamId() + { + return _context.CreatedStream.ShouldNotBeNull().Id; + } + + private static IReadOnlyList<MetadataChangeGroup> ParseGroups(string groups) + { + return groups.Split(',').Select(name => Enum.Parse<MetadataChangeGroup>(name.Trim().Replace("_", ""), true)) + .ToArray(); + } + + private static uint? ChangeStreamId(MetadataChangeRecord record) + { + return record.Body switch + { + MetadataStreamChange stream => stream.StreamId, + MetadataStreamDeleted stream => stream.StreamId, + MetadataTopicChange topic => topic.StreamId, + MetadataTopicDeleted topic => topic.StreamId, + MetadataPartitionsChanged partitions => partitions.StreamId, + _ => null + }; + } +}
diff --git a/foreign/csharp/Iggy_SDK.Tests.BDD/StepDefinitions/ResourceOptionsSteps.cs b/foreign/csharp/Iggy_SDK.Tests.BDD/StepDefinitions/ResourceOptionsSteps.cs index 2eabc4d..c465fc8 100644 --- a/foreign/csharp/Iggy_SDK.Tests.BDD/StepDefinitions/ResourceOptionsSteps.cs +++ b/foreign/csharp/Iggy_SDK.Tests.BDD/StepDefinitions/ResourceOptionsSteps.cs
@@ -64,6 +64,7 @@ public async Task CreateKeyedStream(string name, int value) { _streamKey = Enumerable.Repeat((byte)value, 32).ToArray(); + _context.LastStreamKey = _streamKey; _context.CreatedStream = await _context.IggyClient.CreateStreamAsync(name, new StreamOptions { EncryptionKey = _streamKey }); _context.CreatedStream.ShouldNotBeNull(); @@ -171,6 +172,13 @@ headers![HeaderKey.FromString(name)].ToString().ShouldBe(value); } + [When("I delete the stream")] + public async Task DeleteStream() + { + await _context.IggyClient.DeleteStreamAsync(StreamId()); + _streamId = null; + } + [When("I rename the stream to {string}")] public Task RenameStream(string name) => _context.IggyClient.UpdateStreamAsync(StreamId(), new StreamUpdate { Name = name });
diff --git a/foreign/csharp/Iggy_SDK.Tests.Integration/IggyTlsConnectionTests.cs b/foreign/csharp/Iggy_SDK.Tests.Integration/IggyTlsConnectionTests.cs index 03bfa37..fd8146a 100644 --- a/foreign/csharp/Iggy_SDK.Tests.Integration/IggyTlsConnectionTests.cs +++ b/foreign/csharp/Iggy_SDK.Tests.Integration/IggyTlsConnectionTests.cs
@@ -16,6 +16,7 @@ // under the License. using Apache.Iggy.Configuration; +using Apache.Iggy.Contracts; using Apache.Iggy.Enums; using Apache.Iggy.Exceptions; using Apache.Iggy.Factory; @@ -104,6 +105,44 @@ } [Test] + public async Task given_tls_connection_when_reading_metadata_changes_should_follow_a_created_stream() + { + using var client = IggyClientFactory.CreateClient(new IggyClientConfigurator + { + BaseAddress = await Fixture.GetIggyAddressAsync(Protocol.Tcp), + Protocol = Protocol.Tcp, + ReconnectionSettings = new ReconnectionSettings { Enabled = false }, + TlsSettings = new TlsSettings + { + Enabled = true, + Hostname = "localhost", + CertificatePath = "Certs/iggy_ca_cert.pem" + } + }); + await client.ConnectAsync(); + await client.LoginUserAsync("iggy", "iggy"); + MetadataChangeGroup[] groups = [MetadataChangeGroup.Streams]; + var latest = await client.GetMetadataChangesAsync(new MetadataChangesQuery + { + Start = MetadataChangeStart.Latest, + Groups = groups + }); + var name = $"changes-tls-{Guid.NewGuid():N}"; + var stream = await client.CreateStreamAsync(name); + + var page = await client.GetMetadataChangesAsync(new MetadataChangesQuery + { + Start = MetadataChangeStart.After(latest.Next.ShouldNotBeNull()), + Groups = groups + }); + + page.Status.ShouldBe(MetadataChangeStatus.Ok); + page.Records.Select(record => record.Body).OfType<MetadataStreamChange>() + .ShouldContain(created => created.StreamId == stream!.Id && created.Name == name); + await client.DeleteStreamAsync(Identifier.Numeric(stream!.Id)); + } + + [Test] public async Task Connect_WithTlsConnectionString_Should_SignIn() { var address = await Fixture.GetIggyAddressAsync(Protocol.Tcp);
diff --git a/foreign/csharp/Iggy_SDK.Tests.Integration/MetadataChangesTests.cs b/foreign/csharp/Iggy_SDK.Tests.Integration/MetadataChangesTests.cs new file mode 100644 index 0000000..8582f87 --- /dev/null +++ b/foreign/csharp/Iggy_SDK.Tests.Integration/MetadataChangesTests.cs
@@ -0,0 +1,483 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +using System.Security.Cryptography; +using Apache.Iggy.Contracts; +using Apache.Iggy.Contracts.Auth; +using Apache.Iggy.Encryption; +using Apache.Iggy.Enums; +using Apache.Iggy.Exceptions; +using Apache.Iggy.IggyClient; +using Apache.Iggy.Tests.Integrations.Fixtures; +using Shouldly; + +namespace Apache.Iggy.Tests.Integrations; + +public class MetadataChangesTests +{ + private const int InvalidCommand = 3; + private const int Unauthorized = 41; + private const uint RootUserId = 0; + private const int FollowAttempts = 100; + + private static readonly TimeSpan FollowInterval = TimeSpan.FromMilliseconds(100); + + private static readonly string[] HiddenOptions = ["encryption_key", "encryption_key_wrapped", "critical_keys"]; + + [ClassDataSource<IggyServerFixture>(Shared = SharedType.PerAssembly)] + public required IggyServerFixture Fixture { get; init; } + + [Test] + [MethodDataSource<IggyServerFixture>(nameof(IggyServerFixture.ProtocolData))] + public async Task given_stream_lifecycle_when_following_from_latest_should_return_its_records(Protocol protocol) + { + var client = await Fixture.CreateAuthenticatedClient(protocol); + var latest = await Read(client, MetadataChangeStart.Latest, [MetadataChangeGroup.Streams]); + latest.Status.ShouldBe(MetadataChangeStatus.Ok); + latest.Phase.ShouldBe(MetadataChangePhase.Changes); + latest.Records.ShouldBeEmpty(); + latest.Next.ShouldNotBeNull(); + latest.LogId.ShouldBe(latest.Next.Value.LogId); + var name = $"changes-{Guid.NewGuid():N}"; + var renamed = $"{name}-renamed"; + + var created = await client.CreateStreamAsync(name); + created.ShouldNotBeNull(); + await client.UpdateStreamAsync(Identifier.Numeric(created.Id), new StreamUpdate { Name = renamed }); + await client.DeleteStreamAsync(Identifier.Numeric(created.Id)); + + var records = await Follow(client, latest.Next.Value, [MetadataChangeGroup.Streams], + followed => OfStream(followed, created.Id, name).Count >= 3); + records.ShouldAllBe(record => record.Group == MetadataChangeGroup.Streams); + records.Select(record => (record.Op, record.Ordinal)).ShouldBeInOrder(); + var mine = OfStream(records, created.Id, name); + mine.Select(record => record.Kind).ShouldBe([ + MetadataChangeKind.StreamCreated, MetadataChangeKind.StreamUpdated, MetadataChangeKind.StreamDeleted + ]); + mine.ShouldAllBe(record => record.ActorUserId == RootUserId); + mine.ShouldAllBe(record => record.Checksum != UInt128.Zero && record.Timestamp > 0); + var createdBody = mine[0].Body.ShouldBeOfType<MetadataStreamChange>(); + createdBody.Name.ShouldBe(name); + createdBody.StreamCreatedOp.ShouldBe(mine[0].Op); + createdBody.OptionsUnreadable.ShouldBeFalse(); + mine[1].Body.ShouldBeOfType<MetadataStreamChange>().Name.ShouldBe(renamed); + var deleted = mine[2].Body.ShouldBeOfType<MetadataStreamDeleted>(); + deleted.Name.ShouldBe(renamed); + deleted.CreatedAt.ShouldBe(createdBody.CreatedAt); + deleted.RequestedOp.ShouldBeNull(); + } + + [Test] + [MethodDataSource<IggyServerFixture>(nameof(IggyServerFixture.ProtocolData))] + public async Task given_topics_group_when_following_should_return_only_topic_records(Protocol protocol) + { + var client = await Fixture.CreateAuthenticatedClient(protocol); + var cursor = await LatestCursor(client, [MetadataChangeGroup.Topics]); + var created = await client.CreateStreamAsync($"changes-filter-{Guid.NewGuid():N}"); + created.ShouldNotBeNull(); + var streamId = Identifier.Numeric(created.Id); + await client.CreateTopicAsync(streamId, "events", 1); + + try + { + var records = await Follow(client, cursor, [MetadataChangeGroup.Topics], + followed => followed.Any(record => IsTopicOf(record, created.Id))); + + records.ShouldAllBe(record => record.Group == MetadataChangeGroup.Topics); + var topic = records.Single(record => IsTopicOf(record, created.Id)).Body + .ShouldBeOfType<MetadataTopicChange>(); + topic.Name.ShouldBe("events"); + topic.PartitionsCount.ShouldBe(1u); + topic.Options.ShouldNotBeNull(); + topic.DerivedOptions.ShouldNotBeNull(); + topic.DerivedOptions.Keys.Select(key => key.AsString()).ShouldContain("segment_size"); + } + finally + { + await client.DeleteStreamAsync(streamId); + } + } + + [Test] + [MethodDataSource<IggyServerFixture>(nameof(IggyServerFixture.ProtocolData))] + public async Task given_limit_of_one_when_following_should_page_one_record_at_a_time(Protocol protocol) + { + var client = await Fixture.CreateAuthenticatedClient(protocol); + var cursor = await LatestCursor(client, [MetadataChangeGroup.Streams]); + var names = Enumerable.Range(0, 3).Select(index => $"changes-page-{index}-{Guid.NewGuid():N}").ToArray(); + var created = new List<uint>(); + foreach (var name in names) + { + created.Add((await client.CreateStreamAsync(name))!.Id); + } + + try + { + var seen = new List<string>(); + for (var attempt = 0; attempt < FollowAttempts && seen.Count < names.Length; attempt++) + { + var page = await Read(client, MetadataChangeStart.After(cursor), [MetadataChangeGroup.Streams], 1); + page.Status.ShouldBe(MetadataChangeStatus.Ok); + page.Records.Count.ShouldBeLessThanOrEqualTo(1); + var next = page.Next.ShouldNotBeNull(); + if (page.Records.Count == 1) + { + (next.Op, next.Ordinal).ShouldBe((page.Records[0].Op, page.Records[0].Ordinal)); + page.Records[0].Op.ShouldBeGreaterThan(cursor.Op); + if (page.Records[0].Body is MetadataStreamChange stream && names.Contains(stream.Name)) + { + seen.Add(stream.Name); + } + } + else if (!page.HasMore) + { + await Task.Delay(FollowInterval); + } + + cursor = next; + } + + seen.ShouldBe(names); + } + finally + { + foreach (var id in created) + { + await client.DeleteStreamAsync(Identifier.Numeric(id)); + } + } + } + + [Test] + [NotInParallel] + [MethodDataSource<IggyServerFixture>(nameof(IggyServerFixture.ProtocolData))] + public async Task given_current_state_scan_when_paging_should_list_the_streams_the_list_returns(Protocol protocol) + { + var client = await Fixture.CreateAuthenticatedClient(protocol); + var created = new List<uint>(); + for (var index = 0; index < 3; index++) + { + created.Add((await client.CreateStreamAsync($"changes-current-{index}-{Guid.NewGuid():N}"))!.Id); + } + + var withTopic = Identifier.Numeric(created[0]); + await client.CreateTopicAsync(withTopic, "events", 2); + MetadataChangeGroup[] groups = [MetadataChangeGroup.Streams, MetadataChangeGroup.Topics, MetadataChangeGroup.Partitions]; + + try + { + // A deferred delete another test started can complete at any time, so the scan is compared with a + // stream list that did not change around it. + List<MetadataCurrentItem> items = []; + MetadataChangeCursor? anchor = null; + Dictionary<uint, string> listed = []; + for (var attempt = 0; attempt < 3; attempt++) + { + var before = await ListStreams(client); + (items, anchor) = await ScanCurrent(client, groups, 2); + listed = await ListStreams(client); + if (before.Count == listed.Count && !before.Except(listed).Any()) + { + break; + } + } + + var projected = items.Select(item => item.Body).OfType<MetadataStreamChange>() + .ToDictionary(stream => stream.StreamId, stream => stream.Name); + projected.ShouldBe(listed, ignoreOrder: true); + items.ShouldAllBe(item => item.Group != null && groups.Contains(item.Group.Value)); + var topic = items.Select(item => item.Body).OfType<MetadataTopicItem>() + .Single(item => item.StreamId == created[0]); + topic.Name.ShouldBe("events"); + items.Select(item => item.Body).OfType<MetadataPartitionsItem>() + .Single(item => item.StreamId == created[0] && item.TopicId == topic.TopicId) + .PartitionsCount.ShouldBe(2u); + + var followed = await Read(client, MetadataChangeStart.After(anchor.ShouldNotBeNull()), groups); + followed.Status.ShouldBe(MetadataChangeStatus.Ok); + } + finally + { + foreach (var id in created) + { + await client.DeleteStreamAsync(Identifier.Numeric(id)); + } + } + } + + [Test] + [MethodDataSource<IggyServerFixture>(nameof(IggyServerFixture.ProtocolData))] + public async Task given_users_scan_when_reading_current_state_should_list_the_root_user(Protocol protocol) + { + var client = await Fixture.CreateAuthenticatedClient(protocol); + + var page = await Read(client, MetadataChangeStart.Current(), [MetadataChangeGroup.Users]); + + page.Status.ShouldBe(MetadataChangeStatus.Ok); + var root = page.Items.Select(item => item.Body).OfType<MetadataUserChange>().First(); + root.UserId.ShouldBe(RootUserId); + root.Username.ShouldBe("iggy"); + root.Status.ShouldBe(UserStatus.Active); + root.OptionsUnreadable.ShouldBeFalse(); + } + + [Test] + [MethodDataSource<IggyServerFixture>(nameof(IggyServerFixture.ProtocolData))] + public async Task given_cursor_of_another_history_or_beyond_the_head_when_following_should_answer_reset_or_behind( + Protocol protocol) + { + var client = await Fixture.CreateAuthenticatedClient(protocol); + var latest = await LatestCursor(client, [MetadataChangeGroup.Streams]); + + var reset = await Read(client, MetadataChangeStart.After(latest with { LogId = latest.LogId + UInt128.One }), + [MetadataChangeGroup.Streams]); + var behind = await Read(client, MetadataChangeStart.After(latest with { Op = ulong.MaxValue }), + [MetadataChangeGroup.Streams]); + + reset.Status.ShouldBe(MetadataChangeStatus.Reset); + reset.Records.ShouldBeEmpty(); + reset.Next.ShouldBeNull(); + behind.Status.ShouldBe(MetadataChangeStatus.Behind); + behind.Records.ShouldBeEmpty(); + behind.Next.ShouldBe(latest with { Op = ulong.MaxValue }); + } + + [Test] + [MethodDataSource<IggyServerFixture>(nameof(IggyServerFixture.ProtocolData))] + public async Task given_query_the_rules_refuse_when_reading_changes_should_refuse_it_before_sending(Protocol protocol) + { + var client = await Fixture.CreateAuthenticatedClient(protocol); + var first = await client.CreateStreamAsync($"changes-refused-{Guid.NewGuid():N}"); + var second = await client.CreateStreamAsync($"changes-refused-{Guid.NewGuid():N}"); + + try + { + var page = await Read(client, MetadataChangeStart.Current(), [MetadataChangeGroup.Streams], 1); + var token = page.CurrentPageToken.ShouldNotBeNull(); + MetadataChangesQuery[] refused = + [ + new() { Limit = 0 }, + new() { Limit = MetadataChangesQuery.MAX_LIMIT + 1 }, + new() { Since = 2, Until = 1 }, + new() { Start = MetadataChangeStart.Latest, Since = 1 }, + new() { Start = MetadataChangeStart.Current(), Until = 1 }, + new() { Start = MetadataChangeStart.Current(), Groups = [MetadataChangeGroup.Topics] }, + new() + { + Start = MetadataChangeStart.Current(token), + Groups = [MetadataChangeGroup.Streams, MetadataChangeGroup.Topics] + }, + new() { Start = MetadataChangeStart.After(new MetadataChangeCursor(1UL, 0, 2UL, 0)) } + ]; + + foreach (var query in refused) + { + var error = await Should.ThrowAsync<IggyInvalidStatusCodeException>( + client.GetMetadataChangesAsync(query)); + error.StatusCode.ShouldBe(InvalidCommand); + error.FromServer.ShouldBeFalse(); + } + } + finally + { + await client.DeleteStreamAsync(Identifier.Numeric(first!.Id)); + await client.DeleteStreamAsync(Identifier.Numeric(second!.Id)); + } + } + + [Test] + [MethodDataSource<IggyServerFixture>(nameof(IggyServerFixture.ProtocolData))] + public async Task given_user_without_manage_servers_when_reading_changes_should_be_refused_as_unauthorized( + Protocol protocol) + { + var root = await Fixture.CreateAuthenticatedClient(protocol); + var username = $"changes-reader-{Guid.NewGuid():N}"; + const string password = "changes-secret"; + var user = await root.CreateUserAsync(username, password, UserStatus.Active, new Permissions + { + Global = new GlobalPermissions + { + ManageServers = false, + ReadServers = true, + ManageUsers = false, + ReadUsers = false, + ManageStreams = false, + ReadStreams = false, + ManageTopics = false, + ReadTopics = false, + PollMessages = false, + SendMessages = false + } + }); + user.ShouldNotBeNull(); + + try + { + var reader = await Fixture.CreateAuthenticatedClient(protocol, username, password); + + var error = await Should.ThrowAsync<IggyInvalidStatusCodeException>( + reader.GetMetadataChangesAsync(new MetadataChangesQuery())); + + error.StatusCode.ShouldBe(Unauthorized); + error.FromServer.ShouldBeTrue(); + } + finally + { + await root.DeleteUserAsync(Identifier.Numeric(user.Id)); + } + } + + [Test] + [MethodDataSource<IggyServerFixture>(nameof(IggyServerFixture.ProtocolData))] + public async Task given_keyed_and_generated_stream_keys_when_following_should_report_fingerprints_and_no_key( + Protocol protocol) + { + var client = await Fixture.CreateAuthenticatedClient(protocol); + var cursor = await LatestCursor(client, [MetadataChangeGroup.Streams]); + var key = RandomNumberGenerator.GetBytes(EncryptionKeys.KEY_LENGTH); + var keyedName = $"changes-keyed-{Guid.NewGuid():N}"; + var generatedName = $"changes-generated-{Guid.NewGuid():N}"; + var keyed = await client.CreateStreamAsync(keyedName, new StreamOptions { EncryptionKey = key }); + var generated = await client.CreateStreamAsync(generatedName); + + try + { + var records = await Follow(client, cursor, [MetadataChangeGroup.Streams], + followed => OfStream(followed, keyed!.Id, keyedName).Count == 1 + && OfStream(followed, generated!.Id, generatedName).Count == 1); + + var keyedStream = OfStream(records, keyed!.Id, keyedName).Single().Body + .ShouldBeOfType<MetadataStreamChange>(); + keyedStream.EncryptionKeySource.ShouldBe(StreamEncryptionSource.Stream); + keyedStream.EncryptionKeyFingerprint.ShouldBe(EncryptionKeys.Fingerprint(key)); + var generatedStream = OfStream(records, generated!.Id, generatedName).Single().Body + .ShouldBeOfType<MetadataStreamChange>(); + generatedStream.EncryptionKeySource.ShouldBe(StreamEncryptionSource.Server); + generatedStream.EncryptionKeyFingerprint.ShouldNotBeNull(); + foreach (var stream in new[] { keyedStream, generatedStream }) + { + foreach (var options in new[] { stream.Options!, stream.DerivedOptions! }) + { + options.Keys.Select(option => option.AsString()).ShouldNotContain(name => HiddenOptions.Contains(name)); + options.Values.ShouldAllBe(value => !value.Value.SequenceEqual(key)); + } + } + } + finally + { + await client.DeleteStreamAsync(Identifier.Numeric(keyed!.Id)); + await client.DeleteStreamAsync(Identifier.Numeric(generated!.Id)); + } + } + + private static async Task<Dictionary<uint, string>> ListStreams(IIggyClient client) + { + return (await client.GetStreamsAsync()).ToDictionary(stream => stream.Id, stream => stream.Name); + } + + /// <summary>Pages a current state scan to its end and returns the items and the anchor to follow from.</summary> + private static async Task<(List<MetadataCurrentItem> Items, MetadataChangeCursor Anchor)> ScanCurrent( + IIggyClient client, IReadOnlyList<MetadataChangeGroup> groups, uint limit) + { + var items = new List<MetadataCurrentItem>(); + var start = MetadataChangeStart.Current(); + while (true) + { + var page = await Read(client, start, groups, limit); + page.Status.ShouldBe(MetadataChangeStatus.Ok); + page.Phase.ShouldBe(MetadataChangePhase.Current); + page.Records.ShouldBeEmpty(); + page.Items.Count.ShouldBeLessThanOrEqualTo((int)limit); + page.HasMore.ShouldBe(page.CurrentPageToken is not null); + items.AddRange(page.Items); + if (page.CurrentPageToken is not { } token) + { + return (items, page.Next.ShouldNotBeNull()); + } + + token.Groups.ShouldBe(groups); + start = MetadataChangeStart.Current(token); + } + } + + private static Task<MetadataChanges> Read(IIggyClient client, MetadataChangeStart start, + IReadOnlyList<MetadataChangeGroup> groups, uint limit = MetadataChangesQuery.MAX_LIMIT) + { + return client.GetMetadataChangesAsync(new MetadataChangesQuery { Start = start, Groups = groups, Limit = limit }); + } + + private static async Task<MetadataChangeCursor> LatestCursor(IIggyClient client, + IReadOnlyList<MetadataChangeGroup> groups) + { + var page = await Read(client, MetadataChangeStart.Latest, groups); + page.Status.ShouldBe(MetadataChangeStatus.Ok); + return page.Next.ShouldNotBeNull(); + } + + /// <summary> + /// Follows the change log from <paramref name="cursor" /> until <paramref name="done" /> holds. A two phase + /// delete commits its record after the call returns, so an empty tail is polled again. + /// </summary> + private static async Task<List<MetadataChangeRecord>> Follow(IIggyClient client, MetadataChangeCursor cursor, + IReadOnlyList<MetadataChangeGroup> groups, Func<List<MetadataChangeRecord>, bool> done) + { + var records = new List<MetadataChangeRecord>(); + for (var attempt = 0; attempt < FollowAttempts; attempt++) + { + var page = await Read(client, MetadataChangeStart.After(cursor), groups); + page.Status.ShouldBe(MetadataChangeStatus.Ok); + cursor = page.Next.ShouldNotBeNull(); + records.AddRange(page.Records); + if (page.HasMore) + { + continue; + } + + if (done(records)) + { + return records; + } + + await Task.Delay(FollowInterval); + } + + throw new TimeoutException("The followed metadata changes never reached the expected records."); + } + + /// <summary> + /// The records of the stream created under <paramref name="name" />, matched by its full identity because + /// stream ids are reused after a delete. + /// </summary> + private static List<MetadataChangeRecord> OfStream(List<MetadataChangeRecord> records, uint streamId, string name) + { + var createdOp = records.FirstOrDefault(record => record.Body is MetadataStreamChange stream + && record.Kind == MetadataChangeKind.StreamCreated + && stream.StreamId == streamId && stream.Name == name)?.Op; + return records.Where(record => createdOp is { } op && record.Body switch + { + MetadataStreamChange stream => stream.StreamId == streamId && stream.StreamCreatedOp == op, + MetadataStreamDeleted deleted => deleted.StreamId == streamId && deleted.StreamCreatedOp == op, + _ => false + }).ToList(); + } + + private static bool IsTopicOf(MetadataChangeRecord record, uint streamId) + { + return record is { Kind: MetadataChangeKind.TopicCreated, Body: MetadataTopicChange topic } + && topic.StreamId == streamId && topic.TopicCreatedOp == record.Op; + } +}
diff --git a/foreign/csharp/Iggy_SDK/Contracts/MetadataChangeBodies.cs b/foreign/csharp/Iggy_SDK/Contracts/MetadataChangeBodies.cs new file mode 100644 index 0000000..ae6cbbd --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Contracts/MetadataChangeBodies.cs
@@ -0,0 +1,570 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +using Apache.Iggy.Contracts.Auth; +using Apache.Iggy.Enums; +using Apache.Iggy.Headers; + +namespace Apache.Iggy.Contracts; + +/// <summary> +/// The body of a metadata change record or current state item. Streams are named by +/// <c>(StreamId, StreamCreatedOp)</c>, topics by their stream key plus <c>(TopicId, TopicCreatedOp)</c> and +/// users by <c>(UserId, UserCreatedAt)</c>, because ids are reused after a delete. +/// </summary> +public abstract class MetadataChangeBody +{ +} + +/// <summary> +/// A stream after a create or update, and a streams item. Options are the public projection the stream read +/// routes return, null with <see cref="OptionsUnreadable" /> set when the server cannot read the stored state. +/// </summary> +public sealed class MetadataStreamChange : MetadataChangeBody +{ + /// <summary>Stream id, reused after a delete.</summary> + public required uint StreamId { get; init; } + + /// <summary>The op that created the stream.</summary> + public required ulong StreamCreatedOp { get; init; } + + /// <summary>Creation time in microseconds since the Unix epoch.</summary> + public required ulong CreatedAt { get; init; } + + /// <summary>Stream name.</summary> + public required string Name { get; init; } + + /// <summary>Options a client set.</summary> + public Dictionary<HeaderKey, HeaderValue>? Options { get; init; } + + /// <summary>Options the server filled or computes, such as the encryption key source and fingerprints.</summary> + public Dictionary<HeaderKey, HeaderValue>? DerivedOptions { get; init; } + + /// <summary>The server holds stored critical state it cannot read. A mirror keeps its previous values.</summary> + public bool OptionsUnreadable { get; init; } + + /// <summary>Who chose the key that encrypts the stream at rest, or null when the record reports no source.</summary> + public StreamEncryptionSource? EncryptionKeySource => + OptionValues.EncryptionKeySource(Options, DerivedOptions); + + /// <summary>Fingerprint of the stream's key. Null for a plaintext stream.</summary> + public string? EncryptionKeyFingerprint => + OptionValues.Text(Options, DerivedOptions, StreamOptionKey.EncryptionKeyFingerprint); +} + +/// <summary>A removed stream. <see cref="RequestedOp" /> is the op of a deferred delete's request.</summary> +public sealed class MetadataStreamDeleted : MetadataChangeBody +{ + /// <summary>Stream id.</summary> + public required uint StreamId { get; init; } + + /// <summary>The op that created the stream.</summary> + public required ulong StreamCreatedOp { get; init; } + + /// <summary>Creation time in microseconds since the Unix epoch.</summary> + public required ulong CreatedAt { get; init; } + + /// <summary>Stream name.</summary> + public required string Name { get; init; } + + /// <summary>The op that requested a deferred delete, null when the delete completed at once.</summary> + public ulong? RequestedOp { get; init; } +} + +/// <summary>A topic after a create or update.</summary> +public sealed class MetadataTopicChange : MetadataChangeBody +{ + /// <summary>Stream id.</summary> + public required uint StreamId { get; init; } + + /// <summary>The op that created the stream.</summary> + public required ulong StreamCreatedOp { get; init; } + + /// <summary>Topic id, reused after a delete.</summary> + public required uint TopicId { get; init; } + + /// <summary>The op that created the topic.</summary> + public required ulong TopicCreatedOp { get; init; } + + /// <summary>Creation time in microseconds since the Unix epoch.</summary> + public required ulong CreatedAt { get; init; } + + /// <summary>Topic name.</summary> + public required string Name { get; init; } + + /// <summary>Partition count after the change.</summary> + public required uint PartitionsCount { get; init; } + + /// <summary>Options a client set.</summary> + public Dictionary<HeaderKey, HeaderValue>? Options { get; init; } + + /// <summary>Options the server filled from its defaults.</summary> + public Dictionary<HeaderKey, HeaderValue>? DerivedOptions { get; init; } + + /// <summary>The server holds stored critical state it cannot read. A mirror keeps its previous values.</summary> + public bool OptionsUnreadable { get; init; } +} + +/// <summary>A topics item. Partition counts travel as partitions items.</summary> +public sealed class MetadataTopicItem : MetadataChangeBody +{ + /// <summary>Stream id.</summary> + public required uint StreamId { get; init; } + + /// <summary>The op that created the stream.</summary> + public required ulong StreamCreatedOp { get; init; } + + /// <summary>Topic id.</summary> + public required uint TopicId { get; init; } + + /// <summary>The op that created the topic.</summary> + public required ulong TopicCreatedOp { get; init; } + + /// <summary>Creation time in microseconds since the Unix epoch.</summary> + public required ulong CreatedAt { get; init; } + + /// <summary>Topic name.</summary> + public required string Name { get; init; } + + /// <summary>Options a client set.</summary> + public Dictionary<HeaderKey, HeaderValue>? Options { get; init; } + + /// <summary>Options the server filled from its defaults.</summary> + public Dictionary<HeaderKey, HeaderValue>? DerivedOptions { get; init; } + + /// <summary>The server holds stored critical state it cannot read.</summary> + public bool OptionsUnreadable { get; init; } +} + +/// <summary>A removed topic.</summary> +public sealed class MetadataTopicDeleted : MetadataChangeBody +{ + /// <summary>Stream id.</summary> + public required uint StreamId { get; init; } + + /// <summary>The op that created the stream.</summary> + public required ulong StreamCreatedOp { get; init; } + + /// <summary>Topic id.</summary> + public required uint TopicId { get; init; } + + /// <summary>The op that created the topic.</summary> + public required ulong TopicCreatedOp { get; init; } + + /// <summary>Creation time in microseconds since the Unix epoch.</summary> + public required ulong CreatedAt { get; init; } + + /// <summary>Topic name.</summary> + public required string Name { get; init; } + + /// <summary>The op that requested a deferred delete, null when the delete completed at once.</summary> + public ulong? RequestedOp { get; init; } +} + +/// <summary> +/// A partition count change. <see cref="RemovedPartitionIds" /> lists the deleted ids in ascending order and is +/// empty on create. Stored ids can be sparse, so the count is not an id boundary. +/// </summary> +public sealed class MetadataPartitionsChanged : MetadataChangeBody +{ + /// <summary>Stream id.</summary> + public required uint StreamId { get; init; } + + /// <summary>The op that created the stream.</summary> + public required ulong StreamCreatedOp { get; init; } + + /// <summary>Topic id.</summary> + public required uint TopicId { get; init; } + + /// <summary>The op that created the topic.</summary> + public required ulong TopicCreatedOp { get; init; } + + /// <summary>Partition count after the change.</summary> + public required uint PartitionsCount { get; init; } + + /// <summary>Ids of the deleted partitions, ascending.</summary> + public required IReadOnlyList<uint> RemovedPartitionIds { get; init; } + + /// <summary>The op that requested a deferred delete, null otherwise.</summary> + public ulong? RequestedOp { get; init; } +} + +/// <summary>A partitions item.</summary> +public sealed class MetadataPartitionsItem : MetadataChangeBody +{ + /// <summary>Stream id.</summary> + public required uint StreamId { get; init; } + + /// <summary>The op that created the stream.</summary> + public required ulong StreamCreatedOp { get; init; } + + /// <summary>Topic id.</summary> + public required uint TopicId { get; init; } + + /// <summary>The op that created the topic.</summary> + public required ulong TopicCreatedOp { get; init; } + + /// <summary>Partition count.</summary> + public required uint PartitionsCount { get; init; } +} + +/// <summary>A created consumer group with the topic's resulting group count.</summary> +public sealed class MetadataConsumerGroupCreated : MetadataChangeBody +{ + /// <summary>Stream id.</summary> + public required uint StreamId { get; init; } + + /// <summary>The op that created the stream.</summary> + public required ulong StreamCreatedOp { get; init; } + + /// <summary>Topic id.</summary> + public required uint TopicId { get; init; } + + /// <summary>The op that created the topic.</summary> + public required ulong TopicCreatedOp { get; init; } + + /// <summary>Consumer group id.</summary> + public required uint GroupId { get; init; } + + /// <summary>Consumer group name.</summary> + public required string Name { get; init; } + + /// <summary>The topic's consumer group count after the insertion.</summary> + public required uint ConsumerGroupsCount { get; init; } +} + +/// <summary>A removed consumer group with the topic's resulting group count.</summary> +public sealed class MetadataConsumerGroupDeleted : MetadataChangeBody +{ + /// <summary>Stream id.</summary> + public required uint StreamId { get; init; } + + /// <summary>The op that created the stream.</summary> + public required ulong StreamCreatedOp { get; init; } + + /// <summary>Topic id.</summary> + public required uint TopicId { get; init; } + + /// <summary>The op that created the topic.</summary> + public required ulong TopicCreatedOp { get; init; } + + /// <summary>Consumer group id.</summary> + public required uint GroupId { get; init; } + + /// <summary>Consumer group name.</summary> + public required string Name { get; init; } + + /// <summary>The topic's consumer group count after the removal.</summary> + public required uint ConsumerGroupsCount { get; init; } + + /// <summary>The op that requested a deferred delete, null when the delete completed at once.</summary> + public ulong? RequestedOp { get; init; } +} + +/// <summary>A consumer groups item.</summary> +public sealed class MetadataConsumerGroupItem : MetadataChangeBody +{ + /// <summary>Stream id.</summary> + public required uint StreamId { get; init; } + + /// <summary>The op that created the stream.</summary> + public required ulong StreamCreatedOp { get; init; } + + /// <summary>Topic id.</summary> + public required uint TopicId { get; init; } + + /// <summary>The op that created the topic.</summary> + public required ulong TopicCreatedOp { get; init; } + + /// <summary>Consumer group id.</summary> + public required uint GroupId { get; init; } + + /// <summary>Consumer group name.</summary> + public required string Name { get; init; } +} + +/// <summary>A member that joined, with its installed partitions, normally none because a handover installs them.</summary> +public sealed class MetadataMemberJoined : MetadataChangeBody +{ + /// <summary>Stream id.</summary> + public required uint StreamId { get; init; } + + /// <summary>The op that created the stream.</summary> + public required ulong StreamCreatedOp { get; init; } + + /// <summary>Topic id.</summary> + public required uint TopicId { get; init; } + + /// <summary>The op that created the topic.</summary> + public required ulong TopicCreatedOp { get; init; } + + /// <summary>Consumer group id.</summary> + public required uint GroupId { get; init; } + + /// <summary>Member id within the group.</summary> + public required uint MemberId { get; init; } + + /// <summary>The member's client id.</summary> + public required UInt128 ClientId { get; init; } + + /// <summary>The group's member count after the join.</summary> + public required uint MembersCount { get; init; } + + /// <summary>Installed partition ids, ascending.</summary> + public required IReadOnlyList<uint> PartitionIds { get; init; } +} + +/// <summary>A member that left the group.</summary> +public sealed class MetadataMemberLeft : MetadataChangeBody +{ + /// <summary>Stream id.</summary> + public required uint StreamId { get; init; } + + /// <summary>The op that created the stream.</summary> + public required ulong StreamCreatedOp { get; init; } + + /// <summary>Topic id.</summary> + public required uint TopicId { get; init; } + + /// <summary>The op that created the topic.</summary> + public required ulong TopicCreatedOp { get; init; } + + /// <summary>Consumer group id.</summary> + public required uint GroupId { get; init; } + + /// <summary>Member id within the group.</summary> + public required uint MemberId { get; init; } + + /// <summary>The member's client id.</summary> + public required UInt128 ClientId { get; init; } + + /// <summary>The group's member count after the removal.</summary> + public required uint MembersCount { get; init; } + + /// <summary>Why the member was removed.</summary> + public required MetadataMemberLeftReason Reason { get; init; } +} + +/// <summary>A completed partition handover. <see cref="MemberId" /> is null when the last member left.</summary> +public sealed class MetadataPartitionHandedOver : MetadataChangeBody +{ + /// <summary>Stream id.</summary> + public required uint StreamId { get; init; } + + /// <summary>The op that created the stream.</summary> + public required ulong StreamCreatedOp { get; init; } + + /// <summary>Topic id.</summary> + public required uint TopicId { get; init; } + + /// <summary>The op that created the topic.</summary> + public required ulong TopicCreatedOp { get; init; } + + /// <summary>Consumer group id.</summary> + public required uint GroupId { get; init; } + + /// <summary>The partition handed over.</summary> + public required uint PartitionId { get; init; } + + /// <summary>The new owner, null when nobody owns the partition.</summary> + public uint? MemberId { get; init; } +} + +/// <summary>A consumer group members item.</summary> +public sealed class MetadataMemberItem : MetadataChangeBody +{ + /// <summary>Stream id.</summary> + public required uint StreamId { get; init; } + + /// <summary>The op that created the stream.</summary> + public required ulong StreamCreatedOp { get; init; } + + /// <summary>Topic id.</summary> + public required uint TopicId { get; init; } + + /// <summary>The op that created the topic.</summary> + public required ulong TopicCreatedOp { get; init; } + + /// <summary>Consumer group id.</summary> + public required uint GroupId { get; init; } + + /// <summary>Member id within the group.</summary> + public required uint MemberId { get; init; } + + /// <summary>The member's client id.</summary> + public required UInt128 ClientId { get; init; } + + /// <summary>Installed partition ids, ascending.</summary> + public required IReadOnlyList<uint> PartitionIds { get; init; } +} + +/// <summary>A user after a create or update, and a users item.</summary> +public sealed class MetadataUserChange : MetadataChangeBody +{ + /// <summary>User id, reused after a delete.</summary> + public required uint UserId { get; init; } + + /// <summary>Creation time of the user in microseconds since the Unix epoch.</summary> + public required ulong UserCreatedAt { get; init; } + + /// <summary>Username.</summary> + public required string Username { get; init; } + + /// <summary>User status. An inactive user's sessions end.</summary> + public required UserStatus Status { get; init; } + + /// <summary>Public permissions, null when the user has none.</summary> + public Permissions? Permissions { get; init; } + + /// <summary>Options a client set.</summary> + public Dictionary<HeaderKey, HeaderValue>? Options { get; init; } + + /// <summary>Options the server filled from its defaults.</summary> + public Dictionary<HeaderKey, HeaderValue>? DerivedOptions { get; init; } + + /// <summary>The server holds stored critical state it cannot read.</summary> + public bool OptionsUnreadable { get; init; } +} + +/// <summary>A deleted user, or one whose password changed.</summary> +public sealed class MetadataUserIdentity : MetadataChangeBody +{ + /// <summary>User id.</summary> + public required uint UserId { get; init; } + + /// <summary>Creation time of the user in microseconds since the Unix epoch.</summary> + public required ulong UserCreatedAt { get; init; } + + /// <summary>Username.</summary> + public required string Username { get; init; } +} + +/// <summary>A user's public permission set after a change.</summary> +public sealed class MetadataUserPermissionsUpdated : MetadataChangeBody +{ + /// <summary>User id.</summary> + public required uint UserId { get; init; } + + /// <summary>Creation time of the user in microseconds since the Unix epoch.</summary> + public required ulong UserCreatedAt { get; init; } + + /// <summary>Username.</summary> + public required string Username { get; init; } + + /// <summary>Public permissions after the change, null when the user has none.</summary> + public Permissions? Permissions { get; init; } + + /// <summary>Why the set changed.</summary> + public required MetadataPermissionsUpdatedReason Reason { get; init; } +} + +/// <summary>A created personal access token, and a personal access tokens item. Never the token itself.</summary> +public sealed class MetadataTokenChange : MetadataChangeBody +{ + /// <summary>Owner id.</summary> + public required uint UserId { get; init; } + + /// <summary>Creation time of the owner in microseconds since the Unix epoch.</summary> + public required ulong UserCreatedAt { get; init; } + + /// <summary>Token name.</summary> + public required string Name { get; init; } + + /// <summary>Expiry in microseconds since the Unix epoch, null for a token that never expires.</summary> + public ulong? Expiry { get; init; } +} + +/// <summary>A removed personal access token.</summary> +public sealed class MetadataTokenDeleted : MetadataChangeBody +{ + /// <summary>Owner id.</summary> + public required uint UserId { get; init; } + + /// <summary>Creation time of the owner in microseconds since the Unix epoch.</summary> + public required ulong UserCreatedAt { get; init; } + + /// <summary>Token name.</summary> + public required string Name { get; init; } + + /// <summary>Why the token was removed.</summary> + public required MetadataTokenDeletedReason Reason { get; init; } +} + +/// <summary>A registered session, and a sessions item.</summary> +public sealed class MetadataSession : MetadataChangeBody +{ + /// <summary>User id.</summary> + public required uint UserId { get; init; } + + /// <summary>Creation time of the user in microseconds since the Unix epoch.</summary> + public required ulong UserCreatedAt { get; init; } + + /// <summary>The client id the session is bound to.</summary> + public required UInt128 ClientId { get; init; } + + /// <summary>Session epoch.</summary> + public required ulong Session { get; init; } +} + +/// <summary>A closed session.</summary> +public sealed class MetadataSessionClosed : MetadataChangeBody +{ + /// <summary>User id.</summary> + public required uint UserId { get; init; } + + /// <summary>Creation time of the user in microseconds since the Unix epoch.</summary> + public required ulong UserCreatedAt { get; init; } + + /// <summary>The client id the session was bound to.</summary> + public required UInt128 ClientId { get; init; } + + /// <summary>Session epoch.</summary> + public required ulong Session { get; init; } + + /// <summary>Why the session was closed.</summary> + public required MetadataSessionClosedReason Reason { get; init; } +} + +/// <summary> +/// A binary record or item of a group or kind this client does not know, kept with its codes and raw body. +/// </summary> +public sealed class MetadataUnknownBody : MetadataChangeBody +{ + /// <summary>The group code.</summary> + public required byte Group { get; init; } + + /// <summary>The kind code, null for an item.</summary> + public byte? Kind { get; init; } + + /// <summary>The body exactly as the server sent it.</summary> + public required byte[] Body { get; init; } +} + +/// <summary> +/// A JSON record or item of a group or kind this client does not know, kept as the whole object. +/// </summary> +public sealed class MetadataUnknownJsonBody : MetadataChangeBody +{ + /// <summary>The group name.</summary> + public required string Group { get; init; } + + /// <summary>The kind name, null for an item.</summary> + public string? Kind { get; init; } + + /// <summary>The record or item object exactly as the server rendered it.</summary> + public required string Json { get; init; } +}
diff --git a/foreign/csharp/Iggy_SDK/Contracts/MetadataChangeCursor.cs b/foreign/csharp/Iggy_SDK/Contracts/MetadataChangeCursor.cs new file mode 100644 index 0000000..30710bf --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Contracts/MetadataChangeCursor.cs
@@ -0,0 +1,103 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +using System.Buffers.Binary; +using System.Diagnostics.CodeAnalysis; +using Apache.Iggy.Utils; + +namespace Apache.Iggy.Contracts; + +/// <summary> +/// Position of the last metadata change record a caller consumed or scanned. Send the cursor an answer returns +/// in <see cref="MetadataChanges.Next" /> back to read the records after it. +/// </summary> +/// <remarks> +/// The text form is <c>mc1_</c> followed by unpadded Base64url of +/// <c>[log_id:u128][op:u64][checksum:u128][ordinal:u32]</c>, little-endian. Op 0 with a zero checksum and ordinal +/// names the beginning of the history <see cref="LogId" /> identifies. +/// </remarks> +/// <param name="LogId">Prepare checksum of op 1, the root bootstrap of this history.</param> +/// <param name="Op">The metadata operation of the record.</param> +/// <param name="Checksum">Prepare checksum of <paramref name="Op" />.</param> +/// <param name="Ordinal">Position of the record within <paramref name="Op" />.</param> +public readonly record struct MetadataChangeCursor(UInt128 LogId, ulong Op, UInt128 Checksum, uint Ordinal) +{ + /// <summary>Text prefix of a change cursor.</summary> + public const string PREFIX = "mc1_"; + + internal const int SIZE = 44; + + private const int TEXT_LENGTH = 63; + + /// <summary>Whether the cursor names the beginning of its history.</summary> + public bool IsZero => Op == 0; + + /// <summary>Op 0 carries no checksum and no ordinal. A cursor that does is refused.</summary> + public bool IsCanonical => Op != 0 || (Checksum == UInt128.Zero && Ordinal == 0); + + /// <summary> + /// Parses the <c>mc1_</c> text form. + /// </summary> + /// <exception cref="FormatException">The text is not a canonical change cursor.</exception> + public static MetadataChangeCursor Parse(string text) + { + return TryParse(text, out var cursor) + ? cursor + : throw new FormatException("The text is not a canonical metadata change cursor."); + } + + /// <summary> + /// Parses the <c>mc1_</c> text form, refusing a wrong prefix or length, padding, the standard Base64 + /// alphabet, non-canonical encodings and a non-canonical cursor. + /// </summary> + public static bool TryParse([NotNullWhen(true)] string? text, out MetadataChangeCursor cursor) + { + cursor = default; + if (text is not { Length: TEXT_LENGTH } || !text.StartsWith(PREFIX, StringComparison.Ordinal) + || !Base64Url.TryDecode(text.AsSpan(PREFIX.Length), out var bytes) || bytes.Length != SIZE) + { + return false; + } + + cursor = Read(bytes); + return cursor.IsCanonical; + } + + /// <summary>The <c>mc1_</c> text form.</summary> + public override string ToString() + { + Span<byte> bytes = stackalloc byte[SIZE]; + WriteTo(bytes); + return PREFIX + Base64Url.Encode(bytes); + } + + internal void WriteTo(Span<byte> destination) + { + BinaryPrimitives.WriteUInt128LittleEndian(destination[..16], LogId); + BinaryPrimitives.WriteUInt64LittleEndian(destination[16..24], Op); + BinaryPrimitives.WriteUInt128LittleEndian(destination[24..40], Checksum); + BinaryPrimitives.WriteUInt32LittleEndian(destination[40..44], Ordinal); + } + + internal static MetadataChangeCursor Read(ReadOnlySpan<byte> source) + { + return new MetadataChangeCursor(BinaryPrimitives.ReadUInt128LittleEndian(source[..16]), + BinaryPrimitives.ReadUInt64LittleEndian(source[16..24]), + BinaryPrimitives.ReadUInt128LittleEndian(source[24..40]), + BinaryPrimitives.ReadUInt32LittleEndian(source[40..44])); + } +}
diff --git a/foreign/csharp/Iggy_SDK/Contracts/MetadataChangeGroups.cs b/foreign/csharp/Iggy_SDK/Contracts/MetadataChangeGroups.cs new file mode 100644 index 0000000..d0c5305 --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Contracts/MetadataChangeGroups.cs
@@ -0,0 +1,116 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +using Apache.Iggy.Enums; + +namespace Apache.Iggy.Contracts; + +/// <summary> +/// Group and kind codes and the parent rules of a current state scan. +/// </summary> +internal static class MetadataChangeGroups +{ + internal const byte ALL = byte.MaxValue; + + internal static bool IsDefined(MetadataChangeGroup group) + { + return group is >= MetadataChangeGroup.Streams and <= MetadataChangeGroup.Sessions; + } + + internal static bool IsDefined(MetadataChangeKind kind) + { + return Enum.IsDefined(kind); + } + + /// <summary>Bit <c>code - 1</c> of a group mask.</summary> + internal static byte Bit(MetadataChangeGroup group) + { + return (byte)(1 << ((byte)group - 1)); + } + + /// <summary>The mask of the requested groups, every group when none is requested.</summary> + internal static byte Mask(IReadOnlyCollection<MetadataChangeGroup> groups) + { + if (groups.Count == 0) + { + return ALL; + } + + byte mask = 0; + foreach (var group in groups) + { + mask |= Bit(group); + } + + return mask; + } + + /// <summary>The groups a mask selects, in code order.</summary> + internal static IReadOnlyList<MetadataChangeGroup> FromMask(byte mask) + { + return Enum.GetValues<MetadataChangeGroup>().Where(group => (mask & Bit(group)) != 0).ToArray(); + } + + internal static MetadataChangeGroup Of(MetadataChangeKind kind) + { + return kind switch + { + MetadataChangeKind.StreamCreated or MetadataChangeKind.StreamUpdated or MetadataChangeKind.StreamDeleted => + MetadataChangeGroup.Streams, + MetadataChangeKind.TopicCreated or MetadataChangeKind.TopicUpdated or MetadataChangeKind.TopicDeleted => + MetadataChangeGroup.Topics, + MetadataChangeKind.PartitionsChanged => MetadataChangeGroup.Partitions, + MetadataChangeKind.ConsumerGroupCreated or MetadataChangeKind.ConsumerGroupDeleted => + MetadataChangeGroup.ConsumerGroups, + MetadataChangeKind.ConsumerGroupMemberJoined or MetadataChangeKind.ConsumerGroupMemberLeft + or MetadataChangeKind.ConsumerGroupPartitionHandedOver => MetadataChangeGroup.ConsumerGroupMembers, + MetadataChangeKind.UserCreated or MetadataChangeKind.UserUpdated or MetadataChangeKind.UserDeleted + or MetadataChangeKind.UserPasswordChanged or MetadataChangeKind.UserPermissionsUpdated => + MetadataChangeGroup.Users, + MetadataChangeKind.PersonalAccessTokenCreated or MetadataChangeKind.PersonalAccessTokenDeleted => + MetadataChangeGroup.PersonalAccessTokens, + MetadataChangeKind.SessionRegistered or MetadataChangeKind.SessionClosed => MetadataChangeGroup.Sessions, + _ => throw new ArgumentOutOfRangeException(nameof(kind), kind, null) + }; + } + + /// <summary> + /// Groups a current state scan of <paramref name="group" /> must include, so a mirror can apply the + /// cascades of their deletes. + /// </summary> + internal static MetadataChangeGroup[] Parents(MetadataChangeGroup group) + { + return group switch + { + MetadataChangeGroup.Topics => [MetadataChangeGroup.Streams], + MetadataChangeGroup.Partitions or MetadataChangeGroup.ConsumerGroups => + [MetadataChangeGroup.Topics, MetadataChangeGroup.Streams], + MetadataChangeGroup.ConsumerGroupMembers => + [ + MetadataChangeGroup.ConsumerGroups, MetadataChangeGroup.Partitions, MetadataChangeGroup.Topics, + MetadataChangeGroup.Streams + ], + MetadataChangeGroup.PersonalAccessTokens or MetadataChangeGroup.Sessions => [MetadataChangeGroup.Users], + _ => [] + }; + } + + internal static bool MissesParent(byte mask) + { + return FromMask(mask).Any(group => Parents(group).Any(parent => (mask & Bit(parent)) == 0)); + } +}
diff --git a/foreign/csharp/Iggy_SDK/Contracts/MetadataChanges.cs b/foreign/csharp/Iggy_SDK/Contracts/MetadataChanges.cs new file mode 100644 index 0000000..22f0959 --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Contracts/MetadataChanges.cs
@@ -0,0 +1,122 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +using System.Text.Json.Serialization; +using Apache.Iggy.Enums; +using Apache.Iggy.JsonConverters; + +namespace Apache.Iggy.Contracts; + +/// <summary> +/// One page of the metadata change log, or of the current state. +/// </summary> +/// <remarks> +/// JSON support is read only. <see cref="System.Text.Json.JsonSerializer" /> deserializes the HTTP answer, and +/// serializing a page throws <see cref="NotSupportedException" />. +/// </remarks> +[JsonConverter(typeof(MetadataChangesConverter))] +public sealed class MetadataChanges +{ + /// <summary>Outcome of the query. Only <see cref="MetadataChangeStatus.Ok" /> carries records or items.</summary> + public required MetadataChangeStatus Status { get; init; } + + /// <summary>Whether the page holds change records or current state items.</summary> + public required MetadataChangePhase Phase { get; init; } + + /// <summary>The answering node's log id, null before it applied op 1.</summary> + public UInt128? LogId { get; init; } + + /// <summary>Lowest whole operation from which the requested groups are complete.</summary> + public required ulong OldestOp { get; init; } + + /// <summary> + /// The requested groups are complete for every timestamp above this value, in microseconds, 0 when nothing + /// was evicted. + /// </summary> + public required ulong RetainedSince { get; init; } + + /// <summary>Highest metadata operation the answering node applied.</summary> + public required ulong HeadOp { get; init; } + + /// <summary> + /// The cursor to send back. Present on <see cref="MetadataChangeStatus.Ok" />, echoed on + /// <see cref="MetadataChangeStatus.Behind" />, null on expired and reset. On a current state page it is the + /// anchor to follow the changes from once the scan is complete. + /// </summary> + public MetadataChangeCursor? Next { get; init; } + + /// <summary>More entries remain after <see cref="Next" />, or another state page exists.</summary> + public required bool HasMore { get; init; } + + /// <summary>Change records in <c>(op, ordinal)</c> order. Empty on a current state page.</summary> + public required IReadOnlyList<MetadataChangeRecord> Records { get; init; } + + /// <summary>Current state items. Empty on a change page.</summary> + public required IReadOnlyList<MetadataCurrentItem> Items { get; init; } + + /// <summary>Continuation of a current state scan, null once it is complete.</summary> + public MetadataCurrentPageToken? CurrentPageToken { get; init; } +} + +/// <summary> +/// One change record: the committed operation that produced it, its position within that operation, and what +/// is true after it. +/// </summary> +public sealed class MetadataChangeRecord +{ + /// <summary>The metadata operation that produced the record.</summary> + public required ulong Op { get; init; } + + /// <summary>Position of the record within <see cref="Op" />.</summary> + public required uint Ordinal { get; init; } + + /// <summary>Prepare checksum of <see cref="Op" />.</summary> + public required UInt128 Checksum { get; init; } + + /// <summary>Prepare timestamp of <see cref="Op" />, in microseconds since the Unix epoch.</summary> + public required ulong Timestamp { get; init; } + + /// <summary>The acting user, null for server actions.</summary> + public uint? ActorUserId { get; init; } + + /// <summary>The kind, null for a record this client does not know.</summary> + public MetadataChangeKind? Kind { get; init; } + + /// <summary>The group, null for a record this client does not know.</summary> + public MetadataChangeGroup? Group => Kind is { } kind ? MetadataChangeGroups.Of(kind) : null; + + /// <summary> + /// The typed body, or <see cref="MetadataUnknownBody" /> and <see cref="MetadataUnknownJsonBody" /> for a + /// record this client does not know. A mirror that cannot apply a record stops without advancing its cursor. + /// </summary> + public required MetadataChangeBody Body { get; init; } +} + +/// <summary> +/// One current state item. +/// </summary> +public sealed class MetadataCurrentItem +{ + /// <summary>The group, null for an item this client does not know.</summary> + public MetadataChangeGroup? Group { get; init; } + + /// <summary> + /// The typed body, or <see cref="MetadataUnknownBody" /> and <see cref="MetadataUnknownJsonBody" /> for an + /// item this client does not know. + /// </summary> + public required MetadataChangeBody Body { get; init; } +}
diff --git a/foreign/csharp/Iggy_SDK/Contracts/MetadataChangesQuery.cs b/foreign/csharp/Iggy_SDK/Contracts/MetadataChangesQuery.cs new file mode 100644 index 0000000..858d5eb --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Contracts/MetadataChangesQuery.cs
@@ -0,0 +1,187 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +using System.Globalization; +using Apache.Iggy.Enums; +using Apache.Iggy.Exceptions; +using Apache.Iggy.IggyClient; +using Apache.Iggy.Vsr; + +namespace Apache.Iggy.Contracts; + +/// <summary> +/// Where a metadata changes query starts. +/// </summary> +public sealed record MetadataChangeStart +{ + private MetadataChangeStart(MetadataChangeStartMode mode, MetadataChangeCursor? cursor = null, + MetadataCurrentPageToken? pageToken = null) + { + Mode = mode; + Cursor = cursor; + PageToken = pageToken; + } + + /// <summary>The oldest operation the requested groups still hold completely.</summary> + public static MetadataChangeStart Earliest { get; } = new(MetadataChangeStartMode.Earliest); + + /// <summary>No records, only a cursor at the newest retained record of the requested groups.</summary> + public static MetadataChangeStart Latest { get; } = new(MetadataChangeStartMode.Latest); + + /// <summary>The start mode.</summary> + public MetadataChangeStartMode Mode { get; } + + /// <summary>The cursor the records follow, set only for <see cref="MetadataChangeStartMode.After" />.</summary> + public MetadataChangeCursor? Cursor { get; } + + /// <summary>The page token of a continued current state scan, null on its first page.</summary> + public MetadataCurrentPageToken? PageToken { get; } + + /// <summary>Records after <paramref name="cursor" />.</summary> + public static MetadataChangeStart After(MetadataChangeCursor cursor) + { + return new MetadataChangeStart(MetadataChangeStartMode.After, cursor); + } + + /// <summary> + /// Current state pages: without a token the scan starts, with the token a page returned it continues. + /// </summary> + public static MetadataChangeStart Current(MetadataCurrentPageToken? pageToken = null) + { + return new MetadataChangeStart(MetadataChangeStartMode.Current, pageToken: pageToken); + } +} + +/// <summary> +/// A query of <see cref="IIggySystem.GetMetadataChangesAsync" />. +/// </summary> +/// <remarks> +/// <see cref="Since" /> and <see cref="Until" /> keep records whose prepare timestamp is in the inclusive range +/// and apply only to <see cref="MetadataChangeStart.Earliest" /> and <see cref="MetadataChangeStart.After" />. +/// </remarks> +public sealed record MetadataChangesQuery +{ + /// <summary>Records or items per page when the caller does not say.</summary> + public const uint DEFAULT_LIMIT = 100; + + /// <summary>Most records or items one page returns.</summary> + public const uint MAX_LIMIT = 1000; + + /// <summary>Where the query starts.</summary> + public MetadataChangeStart Start { get; init; } = MetadataChangeStart.Earliest; + + /// <summary> + /// The groups to read. Empty means every group, duplicates are ignored. A current state scan must include + /// the parent groups of every requested group, for example <c>Topics</c> needs <c>Streams</c>. + /// </summary> + public IReadOnlyList<MetadataChangeGroup> Groups { get; init; } = []; + + /// <summary>Lowest prepare timestamp to keep, in microseconds since the Unix epoch.</summary> + public ulong? Since { get; init; } + + /// <summary>Highest prepare timestamp to keep, in microseconds since the Unix epoch.</summary> + public ulong? Until { get; init; } + + /// <summary>Records or items per page, 1 to 1000.</summary> + public uint Limit { get; init; } = DEFAULT_LIMIT; + + /// <summary> + /// Applies the rules every transport and the server share, so a refused query is never sent. + /// </summary> + /// <exception cref="IggyInvalidStatusCodeException">Invalid command (code 3) naming the broken rule.</exception> + internal void Validate() + { + ArgumentNullException.ThrowIfNull(Start); + ArgumentNullException.ThrowIfNull(Groups); + if (Limit is 0 or > MAX_LIMIT) + { + throw Refused($"The limit must be 1 to {MAX_LIMIT}, got {Limit}."); + } + + if (Since > Until) + { + throw Refused("Since must not be above until."); + } + + if (Groups.Any(group => !MetadataChangeGroups.IsDefined(group))) + { + throw Refused("The groups name a group this client does not know."); + } + + var mask = MetadataChangeGroups.Mask(Groups); + var hasTimeBound = Since.HasValue || Until.HasValue; + switch (Start.Mode) + { + case MetadataChangeStartMode.After when Start.Cursor is not { IsCanonical: true }: + throw Refused("A cursor at op 0 carries no checksum and no ordinal."); + case MetadataChangeStartMode.Latest or MetadataChangeStartMode.Current when hasTimeBound: + throw Refused("Time bounds apply only when starting at the earliest record or after a cursor."); + case MetadataChangeStartMode.Current when MetadataChangeGroups.MissesParent(mask): + throw Refused("A current state scan must include the parent groups of every requested group."); + case MetadataChangeStartMode.Current when Start.PageToken is { } token && token.GroupsMask != mask: + throw Refused("The page token was issued for other groups."); + } + } + + /// <summary>The query string of <c>GET /metadata/changes</c>, in the order the server documents.</summary> + internal string ToHttpQuery() + { + var pairs = new List<string>(6); + switch (Start.Mode) + { + case MetadataChangeStartMode.After: + pairs.Add($"after={Start.Cursor}"); + break; + case MetadataChangeStartMode.Current: + pairs.Add("from=current"); + if (Start.PageToken is { } token) + { + pairs.Add($"current_after={token}"); + } + + break; + default: + pairs.Add($"from={Start.Mode.ToString().ToLowerInvariant()}"); + break; + } + + if (Groups.Count > 0) + { + var names = MetadataChangeGroups.FromMask(MetadataChangeGroups.Mask(Groups)) + .Select(SnakeCaseNames<MetadataChangeGroup>.Name); + pairs.Add($"groups={string.Join(',', names)}"); + } + + if (Since is { } since) + { + pairs.Add($"since={since.ToString(CultureInfo.InvariantCulture)}"); + } + + if (Until is { } until) + { + pairs.Add($"until={until.ToString(CultureInfo.InvariantCulture)}"); + } + + pairs.Add($"limit={Limit.ToString(CultureInfo.InvariantCulture)}"); + return string.Join('&', pairs); + } + + private static IggyInvalidStatusCodeException Refused(string message) + { + return VsrError.Exception(VsrError.INVALID_COMMAND, $"Metadata changes query refused: {message}"); + } +}
diff --git a/foreign/csharp/Iggy_SDK/Contracts/MetadataCurrentPageToken.cs b/foreign/csharp/Iggy_SDK/Contracts/MetadataCurrentPageToken.cs new file mode 100644 index 0000000..ed6d6f7 --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Contracts/MetadataCurrentPageToken.cs
@@ -0,0 +1,186 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +using System.Buffers.Binary; +using System.Diagnostics.CodeAnalysis; +using System.Text; +using Apache.Iggy.Enums; +using Apache.Iggy.Utils; + +namespace Apache.Iggy.Contracts; + +/// <summary> +/// Opaque continuation of a current state scan. Send it back unchanged with the same groups through +/// <see cref="MetadataChangeStart.Current" />. +/// </summary> +/// <remarks> +/// The text form is <c>cs1_</c> followed by unpadded Base64url of +/// <c>[anchor:44][required_head:u64][groups_mask:u8][group:u8][key_len:u16][last_key]</c>, little-endian. +/// </remarks> +public sealed class MetadataCurrentPageToken : IEquatable<MetadataCurrentPageToken> +{ + /// <summary>Text prefix of a current state page token.</summary> + public const string PREFIX = "cs1_"; + + /// <summary>Longest key: a user id and a personal access token name.</summary> + internal const int MAX_KEY_SIZE = 4 + 1 + WireName.MAX_LENGTH; + + internal const int MAX_SIZE = FIXED_SIZE + MAX_KEY_SIZE; + + private const int FIXED_SIZE = MetadataChangeCursor.SIZE + 8 + 1 + 1 + 2; + private const int MAX_TEXT_LENGTH = 4 + (MAX_SIZE + 2) / 3 * 4; + + private static readonly UTF8Encoding StrictUtf8 = new(false, true); + + private readonly byte[] _payload; + + private MetadataCurrentPageToken(byte[] payload) + { + _payload = payload; + Anchor = MetadataChangeCursor.Read(payload); + RequiredHead = BinaryPrimitives.ReadUInt64LittleEndian(payload.AsSpan(MetadataChangeCursor.SIZE, 8)); + GroupsMask = payload[MetadataChangeCursor.SIZE + 8]; + LastGroup = (MetadataChangeGroup)payload[MetadataChangeCursor.SIZE + 9]; + } + + /// <summary>The anchor captured before the scan's first page.</summary> + public MetadataChangeCursor Anchor { get; } + + /// <summary>The highest head any page of the scan reported so far.</summary> + public ulong RequiredHead { get; } + + /// <summary>The groups the scan was started with, in code order.</summary> + public IReadOnlyList<MetadataChangeGroup> Groups => MetadataChangeGroups.FromMask(GroupsMask); + + /// <summary>The group of the last item returned.</summary> + public MetadataChangeGroup LastGroup { get; } + + /// <summary>The key of the last item returned, in the page token key layout.</summary> + public ReadOnlyMemory<byte> LastKey => _payload.AsMemory(FIXED_SIZE); + + internal byte GroupsMask { get; } + + internal ReadOnlySpan<byte> Payload => _payload; + + /// <summary> + /// Parses the <c>cs1_</c> text form. + /// </summary> + /// <exception cref="FormatException">The text is not a canonical page token.</exception> + public static MetadataCurrentPageToken Parse(string text) + { + return TryParse(text, out var token) + ? token + : throw new FormatException("The text is not a canonical metadata current page token."); + } + + /// <summary> + /// Parses the <c>cs1_</c> text form, refusing a wrong prefix, padding, non-canonical encodings and a payload + /// that breaks the token's own rules. + /// </summary> + public static bool TryParse([NotNullWhen(true)] string? text, [NotNullWhen(true)] out MetadataCurrentPageToken? token) + { + token = null; + return text is { Length: <= MAX_TEXT_LENGTH } && text.StartsWith(PREFIX, StringComparison.Ordinal) + && Base64Url.TryDecode(text.AsSpan(PREFIX.Length), out var bytes) && TryDecode(bytes, out token); + } + + /// <summary> + /// Decodes a token payload, requiring it to be exactly one token: a canonical anchor, a known group selected + /// by the mask, a head at or above the anchor and a key of the exact width that group uses. + /// </summary> + internal static bool TryDecode(ReadOnlySpan<byte> payload, [NotNullWhen(true)] out MetadataCurrentPageToken? token) + { + token = null; + if (payload.Length < FIXED_SIZE) + { + return false; + } + + var keyLength = BinaryPrimitives.ReadUInt16LittleEndian(payload[(FIXED_SIZE - 2)..FIXED_SIZE]); + if (keyLength > MAX_KEY_SIZE || payload.Length != FIXED_SIZE + keyLength) + { + return false; + } + + var anchor = MetadataChangeCursor.Read(payload); + var requiredHead = BinaryPrimitives.ReadUInt64LittleEndian(payload[MetadataChangeCursor.SIZE..]); + var mask = payload[MetadataChangeCursor.SIZE + 8]; + var group = (MetadataChangeGroup)payload[MetadataChangeCursor.SIZE + 9]; + if (!anchor.IsCanonical || requiredHead < anchor.Op || !MetadataChangeGroups.IsDefined(group) + || (mask & MetadataChangeGroups.Bit(group)) == 0 || !KeyFitsGroup(group, payload[FIXED_SIZE..])) + { + return false; + } + + token = new MetadataCurrentPageToken(payload.ToArray()); + return true; + } + + /// <summary>The <c>cs1_</c> text form.</summary> + public override string ToString() + { + return PREFIX + Base64Url.Encode(_payload); + } + + /// <inheritdoc /> + public bool Equals(MetadataCurrentPageToken? other) + { + return other is not null && _payload.AsSpan().SequenceEqual(other._payload); + } + + /// <inheritdoc /> + public override bool Equals(object? obj) + { + return obj is MetadataCurrentPageToken other && Equals(other); + } + + /// <inheritdoc /> + public override int GetHashCode() + { + var hash = new HashCode(); + hash.AddBytes(_payload); + return hash.ToHashCode(); + } + + private static bool KeyFitsGroup(MetadataChangeGroup group, ReadOnlySpan<byte> key) + { + return group switch + { + MetadataChangeGroup.Streams or MetadataChangeGroup.Users => key.Length == 4, + MetadataChangeGroup.Topics or MetadataChangeGroup.Partitions or MetadataChangeGroup.Sessions => + key.Length == 8, + MetadataChangeGroup.ConsumerGroups => key.Length == 12, + MetadataChangeGroup.ConsumerGroupMembers => key.Length == 16, + MetadataChangeGroup.PersonalAccessTokens => + key.Length > 5 && key.Length == 5 + key[4] && IsUtf8(key[5..]), + _ => false + }; + } + + private static bool IsUtf8(ReadOnlySpan<byte> bytes) + { + try + { + StrictUtf8.GetCharCount(bytes); + return true; + } + catch (DecoderFallbackException) + { + return false; + } + } +}
diff --git a/foreign/csharp/Iggy_SDK/Contracts/OptionValues.cs b/foreign/csharp/Iggy_SDK/Contracts/OptionValues.cs index 8d97d33..6d2838e 100644 --- a/foreign/csharp/Iggy_SDK/Contracts/OptionValues.cs +++ b/foreign/csharp/Iggy_SDK/Contracts/OptionValues.cs
@@ -15,6 +15,7 @@ // specific language governing permissions and limitations // under the License. +using Apache.Iggy.Enums; using Apache.Iggy.Headers; namespace Apache.Iggy.Contracts; @@ -39,6 +40,18 @@ : null; } + internal static StreamEncryptionSource? EncryptionKeySource(Dictionary<HeaderKey, HeaderValue>? explicitOptions, + Dictionary<HeaderKey, HeaderValue>? derivedOptions) + { + return Text(explicitOptions, derivedOptions, StreamOptionKey.EncryptionKeySource) switch + { + "none" => StreamEncryptionSource.None, + "stream" => StreamEncryptionSource.Stream, + "server" => StreamEncryptionSource.Server, + _ => null + }; + } + /// <summary>A u64 setting, <see cref="ulong.MaxValue" /> when absent: never expire and unlimited.</summary> internal static ulong UInt64(Dictionary<HeaderKey, HeaderValue>? explicitOptions, Dictionary<HeaderKey, HeaderValue>? derivedOptions, TopicOptionKey key)
diff --git a/foreign/csharp/Iggy_SDK/Contracts/SnakeCaseNames.cs b/foreign/csharp/Iggy_SDK/Contracts/SnakeCaseNames.cs new file mode 100644 index 0000000..fd71a03 --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Contracts/SnakeCaseNames.cs
@@ -0,0 +1,39 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +using System.Text.Json; + +namespace Apache.Iggy.Contracts; + +/// <summary> +/// The snake case names the metadata changes query and JSON answer give the members of +/// <typeparamref name="T" />. +/// </summary> +internal static class SnakeCaseNames<T> where T : struct, Enum +{ + private static readonly Dictionary<string, T> Values = Enum.GetValues<T>().ToDictionary(Name); + + internal static string Name(T value) + { + return JsonNamingPolicy.SnakeCaseLower.ConvertName(value.ToString()); + } + + internal static bool TryParse(string name, out T value) + { + return Values.TryGetValue(name, out value); + } +}
diff --git a/foreign/csharp/Iggy_SDK/Contracts/StreamResponse.cs b/foreign/csharp/Iggy_SDK/Contracts/StreamResponse.cs index 972bcf8..5b9a1de 100644 --- a/foreign/csharp/Iggy_SDK/Contracts/StreamResponse.cs +++ b/foreign/csharp/Iggy_SDK/Contracts/StreamResponse.cs
@@ -81,13 +81,7 @@ /// </summary> [JsonIgnore] public StreamEncryptionSource? EncryptionKeySource => - OptionValues.Text(Options, DerivedOptions, StreamOptionKey.EncryptionKeySource) switch - { - "none" => StreamEncryptionSource.None, - "stream" => StreamEncryptionSource.Stream, - "server" => StreamEncryptionSource.Server, - _ => null - }; + OptionValues.EncryptionKeySource(Options, DerivedOptions); /// <summary> /// Fingerprint of the stream's key, the value <see cref="Encryption.EncryptionKeys.Fingerprint(ReadOnlySpan{byte})" />
diff --git a/foreign/csharp/Iggy_SDK/Contracts/Tcp/TcpContracts.cs b/foreign/csharp/Iggy_SDK/Contracts/Tcp/TcpContracts.cs index 9cb6302..daaa1a4 100644 --- a/foreign/csharp/Iggy_SDK/Contracts/Tcp/TcpContracts.cs +++ b/foreign/csharp/Iggy_SDK/Contracts/Tcp/TcpContracts.cs
@@ -823,6 +823,46 @@ return bytes.ToArray(); } + /// <summary> + /// Encodes a validated metadata changes query: + /// <c>[start:u8][cursor:44][has_since:u8][since:u64]?[has_until:u8][until:u64]?[limit:u32][groups_count:u8][group:u8]*[token_len:u16][token]</c>. + /// The cursor is all zero unless the query starts after one, and groups keep the caller's order. + /// </summary> + internal static byte[] GetMetadataChanges(MetadataChangesQuery query) + { + var groups = query.Groups.Count > byte.MaxValue ? query.Groups.Distinct().ToArray() : query.Groups; + var groupsCount = groups.Count; + var token = query.Start.PageToken is { } pageToken ? pageToken.Payload : ReadOnlySpan<byte>.Empty; + var length = 1 + MetadataChangeCursor.SIZE + (query.Since.HasValue ? 9 : 1) + (query.Until.HasValue ? 9 : 1) + + 4 + 1 + groupsCount + 2 + token.Length; + var bytes = new byte[length]; + bytes[0] = (byte)query.Start.Mode; + query.Start.Cursor?.WriteTo(bytes.AsSpan(1, MetadataChangeCursor.SIZE)); + var position = 1 + MetadataChangeCursor.SIZE; + foreach (var bound in new[] { query.Since, query.Until }) + { + bytes[position++] = bound.HasValue ? (byte)1 : (byte)0; + if (bound is { } value) + { + BinaryPrimitives.WriteUInt64LittleEndian(bytes.AsSpan(position, 8), value); + position += 8; + } + } + + BinaryPrimitives.WriteUInt32LittleEndian(bytes.AsSpan(position, 4), query.Limit); + position += 4; + bytes[position++] = (byte)groupsCount; + for (var index = 0; index < groupsCount; index++) + { + bytes[position++] = (byte)groups[index]; + } + + BinaryPrimitives.WriteUInt16LittleEndian(bytes.AsSpan(position, 2), (ushort)token.Length); + position += 2; + token.CopyTo(bytes.AsSpan(position)); + return bytes; + } + internal static byte[] DeleteOffset(Identifier streamId, Identifier topicId, Consumer consumer, uint? partitionId) { Span<byte> bytes =
diff --git a/foreign/csharp/Iggy_SDK/Enums/MetadataChangeGroup.cs b/foreign/csharp/Iggy_SDK/Enums/MetadataChangeGroup.cs new file mode 100644 index 0000000..2f4f11e --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Enums/MetadataChangeGroup.cs
@@ -0,0 +1,48 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +namespace Apache.Iggy.Enums; + +/// <summary> +/// Family of metadata change records and the unit of filtering. The values are the wire codes. +/// </summary> +public enum MetadataChangeGroup : byte +{ + /// <summary>Stream created, updated and deleted.</summary> + Streams = 1, + + /// <summary>Topic created, updated and deleted.</summary> + Topics = 2, + + /// <summary>Partition count changes.</summary> + Partitions = 3, + + /// <summary>Consumer group created and deleted.</summary> + ConsumerGroups = 4, + + /// <summary>Consumer group members joining, leaving and receiving partitions.</summary> + ConsumerGroupMembers = 5, + + /// <summary>User created, updated, deleted, password and permission changes.</summary> + Users = 6, + + /// <summary>Personal access token created and deleted.</summary> + PersonalAccessTokens = 7, + + /// <summary>Session registered and closed.</summary> + Sessions = 8 +}
diff --git a/foreign/csharp/Iggy_SDK/Enums/MetadataChangeKind.cs b/foreign/csharp/Iggy_SDK/Enums/MetadataChangeKind.cs new file mode 100644 index 0000000..9ade483 --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Enums/MetadataChangeKind.cs
@@ -0,0 +1,88 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +namespace Apache.Iggy.Enums; + +/// <summary> +/// Kind of a metadata change record. The values are the wire codes, and a kind always belongs to one +/// <see cref="MetadataChangeGroup" />. +/// </summary> +public enum MetadataChangeKind : byte +{ + /// <summary>A stream was created.</summary> + StreamCreated = 1, + + /// <summary>A stream's name or options changed.</summary> + StreamUpdated = 2, + + /// <summary>A stream was removed.</summary> + StreamDeleted = 3, + + /// <summary>A topic was created.</summary> + TopicCreated = 10, + + /// <summary>A topic's name or options changed.</summary> + TopicUpdated = 11, + + /// <summary>A topic was removed.</summary> + TopicDeleted = 12, + + /// <summary>A topic's partition count changed.</summary> + PartitionsChanged = 20, + + /// <summary>A consumer group was created.</summary> + ConsumerGroupCreated = 30, + + /// <summary>A consumer group was removed.</summary> + ConsumerGroupDeleted = 31, + + /// <summary>A member joined a consumer group.</summary> + ConsumerGroupMemberJoined = 40, + + /// <summary>A member left a consumer group.</summary> + ConsumerGroupMemberLeft = 41, + + /// <summary>A partition moved to another member, or to nobody.</summary> + ConsumerGroupPartitionHandedOver = 42, + + /// <summary>A user was created.</summary> + UserCreated = 50, + + /// <summary>A user's name, status or options changed.</summary> + UserUpdated = 51, + + /// <summary>A user was removed.</summary> + UserDeleted = 52, + + /// <summary>A user's password was replaced.</summary> + UserPasswordChanged = 53, + + /// <summary>A user's public permission set changed.</summary> + UserPermissionsUpdated = 54, + + /// <summary>A personal access token was created.</summary> + PersonalAccessTokenCreated = 60, + + /// <summary>A personal access token was removed.</summary> + PersonalAccessTokenDeleted = 61, + + /// <summary>A session was registered.</summary> + SessionRegistered = 70, + + /// <summary>A session was closed.</summary> + SessionClosed = 71 +}
diff --git a/foreign/csharp/Iggy_SDK/Enums/MetadataChangePhase.cs b/foreign/csharp/Iggy_SDK/Enums/MetadataChangePhase.cs new file mode 100644 index 0000000..20cf100 --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Enums/MetadataChangePhase.cs
@@ -0,0 +1,30 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +namespace Apache.Iggy.Enums; + +/// <summary> +/// Whether a metadata changes answer pages change records or current state items. +/// </summary> +public enum MetadataChangePhase : byte +{ + /// <summary>Change records, the answer to earliest, latest and after.</summary> + Changes = 0, + + /// <summary>Current state items, the answer to current.</summary> + Current = 1 +}
diff --git a/foreign/csharp/Iggy_SDK/Enums/MetadataChangeStartMode.cs b/foreign/csharp/Iggy_SDK/Enums/MetadataChangeStartMode.cs new file mode 100644 index 0000000..22a04da --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Enums/MetadataChangeStartMode.cs
@@ -0,0 +1,36 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +namespace Apache.Iggy.Enums; + +/// <summary> +/// Where a metadata changes query starts. The values are the wire codes. +/// </summary> +public enum MetadataChangeStartMode : byte +{ + /// <summary>The oldest operation the requested groups still hold completely.</summary> + Earliest = 0, + + /// <summary>No records, only a cursor at the newest retained record.</summary> + Latest = 1, + + /// <summary>Records after a change cursor.</summary> + After = 2, + + /// <summary>Current state pages, the first one without a page token.</summary> + Current = 3 +}
diff --git a/foreign/csharp/Iggy_SDK/Enums/MetadataChangeStatus.cs b/foreign/csharp/Iggy_SDK/Enums/MetadataChangeStatus.cs new file mode 100644 index 0000000..3a63e9d --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Enums/MetadataChangeStatus.cs
@@ -0,0 +1,36 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +namespace Apache.Iggy.Enums; + +/// <summary> +/// Outcome of a metadata changes query. +/// </summary> +public enum MetadataChangeStatus : byte +{ + /// <summary>Records or items after the requested position.</summary> + Ok = 0, + + /// <summary>The answering node has not applied the requested position yet. Ask again later or ask another node.</summary> + Behind = 1, + + /// <summary>The node no longer holds every record after the position. Rebuild from current state.</summary> + Expired = 2, + + /// <summary>The position belongs to another history. Rebuild from current state.</summary> + Reset = 3 +}
diff --git a/foreign/csharp/Iggy_SDK/Enums/MetadataMemberLeftReason.cs b/foreign/csharp/Iggy_SDK/Enums/MetadataMemberLeftReason.cs new file mode 100644 index 0000000..a1f0caf --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Enums/MetadataMemberLeftReason.cs
@@ -0,0 +1,30 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +namespace Apache.Iggy.Enums; + +/// <summary> +/// Why a consumer group member was removed. +/// </summary> +public enum MetadataMemberLeftReason : byte +{ + /// <summary>The member left the group.</summary> + Left = 0, + + /// <summary>The member's session ended.</summary> + SessionEnded = 1 +}
diff --git a/foreign/csharp/Iggy_SDK/Enums/MetadataPermissionsUpdatedReason.cs b/foreign/csharp/Iggy_SDK/Enums/MetadataPermissionsUpdatedReason.cs new file mode 100644 index 0000000..4935a6d --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Enums/MetadataPermissionsUpdatedReason.cs
@@ -0,0 +1,30 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +namespace Apache.Iggy.Enums; + +/// <summary> +/// Why a user's public permission set changed. +/// </summary> +public enum MetadataPermissionsUpdatedReason : byte +{ + /// <summary>A permissions update replaced the set.</summary> + Updated = 0, + + /// <summary>A stream or topic delete pruned the user's grants on it.</summary> + ResourceDeleted = 1 +}
diff --git a/foreign/csharp/Iggy_SDK/Enums/MetadataSessionClosedReason.cs b/foreign/csharp/Iggy_SDK/Enums/MetadataSessionClosedReason.cs new file mode 100644 index 0000000..4afa339 --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Enums/MetadataSessionClosedReason.cs
@@ -0,0 +1,33 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +namespace Apache.Iggy.Enums; + +/// <summary> +/// Why a session was closed. +/// </summary> +public enum MetadataSessionClosedReason : byte +{ + /// <summary>The client logged out.</summary> + Logout = 0, + + /// <summary>The session lease expired.</summary> + Expired = 1, + + /// <summary>The user was deactivated or deleted.</summary> + Revoked = 2 +}
diff --git a/foreign/csharp/Iggy_SDK/Enums/MetadataTokenDeletedReason.cs b/foreign/csharp/Iggy_SDK/Enums/MetadataTokenDeletedReason.cs new file mode 100644 index 0000000..e9b0083 --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Enums/MetadataTokenDeletedReason.cs
@@ -0,0 +1,30 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +namespace Apache.Iggy.Enums; + +/// <summary> +/// Why a personal access token was removed. +/// </summary> +public enum MetadataTokenDeletedReason : byte +{ + /// <summary>A client deleted the token.</summary> + Deleted = 0, + + /// <summary>The expiry cleaner removed the token.</summary> + Expired = 1 +}
diff --git a/foreign/csharp/Iggy_SDK/IggyClient/IIggySystem.cs b/foreign/csharp/Iggy_SDK/IggyClient/IIggySystem.cs index 4640646..5855ab5 100644 --- a/foreign/csharp/Iggy_SDK/IggyClient/IIggySystem.cs +++ b/foreign/csharp/Iggy_SDK/IggyClient/IIggySystem.cs
@@ -101,6 +101,26 @@ Task<ClusterMetadata?> GetClusterMetadataAsync(CancellationToken token = default); /// <summary> + /// Reads one page of the node's metadata change log, or of its current state. + /// </summary> + /// <remarks> + /// Every committed metadata operation that changes public state leaves records such as a stream created, a + /// topic renamed or a session closed. A mirror keeps a cursor, asks for the records after it and stores the + /// returned <see cref="MetadataChanges.Next" />. On <see cref="MetadataChangeStatus.Expired" /> or + /// <see cref="MetadataChangeStatus.Reset" /> it rebuilds from current state pages and follows the changes + /// after the anchor they return. Requires root or the global manage servers permission. The query is checked + /// before anything is sent. Available over TCP and HTTP. + /// </remarks> + /// <param name="query">Where to start, which groups, the time range and the page size.</param> + /// <param name="token">The cancellation token to cancel the operation.</param> + /// <returns>A task that represents the asynchronous operation and returns the page.</returns> + /// <exception cref="Exceptions.IggyInvalidStatusCodeException"> + /// Invalid command (code 3) for a query the rules refuse, unauthorized (code 41) without the manage servers + /// permission. + /// </exception> + Task<MetadataChanges> GetMetadataChangesAsync(MetadataChangesQuery query, CancellationToken token = default); + + /// <summary> /// Sends a ping request to the server to verify connectivity. /// </summary> /// <remarks>
diff --git a/foreign/csharp/Iggy_SDK/IggyClient/Implementations/HttpMessageStream.cs b/foreign/csharp/Iggy_SDK/IggyClient/Implementations/HttpMessageStream.cs index 61e77bb..c027675 100644 --- a/foreign/csharp/Iggy_SDK/IggyClient/Implementations/HttpMessageStream.cs +++ b/foreign/csharp/Iggy_SDK/IggyClient/Implementations/HttpMessageStream.cs
@@ -555,6 +555,23 @@ } /// <inheritdoc /> + public async Task<MetadataChanges> GetMetadataChangesAsync(MetadataChangesQuery query, + CancellationToken token = default) + { + ArgumentNullException.ThrowIfNull(query); + query.Validate(); + var response = await _httpClient.GetAsync($"/metadata/changes?{query.ToHttpQuery()}", token); + if (!response.IsSuccessStatusCode) + { + // A 404 is a server without the route, not an empty answer. + await HandleResponseAsync(response, shouldThrowOnGetNotFound: true); + } + + return await response.Content.ReadFromJsonAsync<MetadataChanges>(_jsonSerializerOptions, token) + ?? throw new InvalidResponseException("The metadata changes answer is empty."); + } + + /// <inheritdoc /> public async Task PingAsync(CancellationToken token = default) { var response = await _httpClient.GetAsync("/ping", token);
diff --git a/foreign/csharp/Iggy_SDK/IggyClient/Implementations/TcpMessageStream.cs b/foreign/csharp/Iggy_SDK/IggyClient/Implementations/TcpMessageStream.cs index 566e3be..fce9166 100644 --- a/foreign/csharp/Iggy_SDK/IggyClient/Implementations/TcpMessageStream.cs +++ b/foreign/csharp/Iggy_SDK/IggyClient/Implementations/TcpMessageStream.cs
@@ -575,6 +575,19 @@ } /// <inheritdoc /> + public async Task<MetadataChanges> GetMetadataChangesAsync(MetadataChangesQuery query, + CancellationToken token = default) + { + ArgumentNullException.ThrowIfNull(query); + query.Validate(); + var message = TcpContracts.GetMetadataChanges(query); + using IMemoryOwner<byte> responseBuffer + = await SendWithResponseAsync(CommandCodes.GET_METADATA_CHANGES_CODE, message, token: token); + + return BinaryMapper.MapMetadataChanges(responseBuffer.Memory.Span); + } + + /// <inheritdoc /> public async Task PingAsync(CancellationToken token = default) { var message = Array.Empty<byte>();
diff --git a/foreign/csharp/Iggy_SDK/Iggy_SDK.csproj b/foreign/csharp/Iggy_SDK/Iggy_SDK.csproj index cc60480..6d01e0f 100644 --- a/foreign/csharp/Iggy_SDK/Iggy_SDK.csproj +++ b/foreign/csharp/Iggy_SDK/Iggy_SDK.csproj
@@ -26,7 +26,7 @@ <TargetFrameworks>net8.0;net10.0</TargetFrameworks> <AssemblyName>Apache.Iggy</AssemblyName> <RootNamespace>Apache.Iggy</RootNamespace> - <Version>0.9.1-edge.2</Version> + <Version>0.9.1-edge.3</Version> <GenerateDocumentationFile>true</GenerateDocumentationFile> </PropertyGroup>
diff --git a/foreign/csharp/Iggy_SDK/JsonConverters/MetadataChangesConverter.cs b/foreign/csharp/Iggy_SDK/JsonConverters/MetadataChangesConverter.cs new file mode 100644 index 0000000..466a41b --- /dev/null +++ b/foreign/csharp/Iggy_SDK/JsonConverters/MetadataChangesConverter.cs
@@ -0,0 +1,499 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +using System.Globalization; +using System.Text.Json; +using System.Text.Json.Serialization; +using Apache.Iggy.Contracts; +using Apache.Iggy.Contracts.Auth; +using Apache.Iggy.Enums; + +namespace Apache.Iggy.JsonConverters; + +/// <summary> +/// Reads the JSON answer of <c>GET /metadata/changes</c>. Records and items are flat objects of the envelope +/// fields and the body fields, 128-bit values are 32 lowercase hex characters, and a record or item of a group +/// or kind this build does not know is kept whole as <see cref="MetadataUnknownJsonBody" />. +/// </summary> +internal sealed class MetadataChangesConverter : JsonConverter<MetadataChanges> +{ + private const int HEX_U128_LENGTH = 32; + + private static readonly JsonSerializerOptions BodyOptions = new() + { + PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower, + Converters = { new ResourceOptionsConverter() } + }; + + public override MetadataChanges Read(ref Utf8JsonReader reader, Type typeToConvert, + JsonSerializerOptions options) + { + using var document = JsonDocument.ParseValue(ref reader); + var answer = document.RootElement; + try + { + return new MetadataChanges + { + Status = Name<MetadataChangeStatus>(answer, "status"), + Phase = Name<MetadataChangePhase>(answer, "phase"), + LogId = IsNull(answer, "log_id") ? null : HexU128(answer, "log_id"), + OldestOp = Property(answer, "oldest_op").GetUInt64(), + RetainedSince = Property(answer, "retained_since").GetUInt64(), + HeadOp = Property(answer, "head_op").GetUInt64(), + Next = IsNull(answer, "next") ? null : Cursor(Property(answer, "next")), + HasMore = Property(answer, "has_more").GetBoolean(), + Records = Property(answer, "records").EnumerateArray().Select(ReadRecord).ToArray(), + Items = Property(answer, "items").EnumerateArray().Select(ReadItem).ToArray(), + CurrentPageToken = IsNull(answer, "current_page_token") ? null : PageToken(answer) + }; + } + catch (Exception error) when (error is InvalidOperationException or FormatException) + { + // JsonElement reports a value of the wrong JSON type or range with these instead of JsonException. + throw new JsonException("The metadata changes answer does not have the documented shape.", error); + } + } + + /// <summary>The answer is only ever read from a response, never written.</summary> + public override void Write(Utf8JsonWriter writer, MetadataChanges value, JsonSerializerOptions options) + { + throw new NotSupportedException("Metadata changes are read from a response, not written to a request."); + } + + private static MetadataChangeRecord ReadRecord(JsonElement record) + { + var groupName = Text(record, "group"); + var kindName = Text(record, "kind"); + MetadataChangeKind? kind = null; + MetadataChangeBody body; + if (SnakeCaseNames<MetadataChangeGroup>.TryParse(groupName, out var group) + && SnakeCaseNames<MetadataChangeKind>.TryParse(kindName, out var knownKind)) + { + if (MetadataChangeGroups.Of(knownKind) != group) + { + throw new JsonException($"Metadata change kind {kindName} does not belong to group {groupName}."); + } + + kind = knownKind; + body = ChangeBody(knownKind, record); + } + else + { + body = new MetadataUnknownJsonBody { Group = groupName, Kind = kindName, Json = record.GetRawText() }; + } + + return new MetadataChangeRecord + { + Op = Property(record, "op").GetUInt64(), + Ordinal = Property(record, "ordinal").GetUInt32(), + Checksum = HexU128(record, "checksum"), + Timestamp = Property(record, "timestamp").GetUInt64(), + ActorUserId = OptionalU32(record, "actor_user_id"), + Kind = kind, + Body = body + }; + } + + private static MetadataCurrentItem ReadItem(JsonElement item) + { + var groupName = Text(item, "group"); + if (!SnakeCaseNames<MetadataChangeGroup>.TryParse(groupName, out var group)) + { + return new MetadataCurrentItem + { + Body = new MetadataUnknownJsonBody { Group = groupName, Json = item.GetRawText() } + }; + } + + MetadataChangeBody body = group switch + { + MetadataChangeGroup.Streams => StreamChange(item), + MetadataChangeGroup.Topics => TopicItemBody(item), + MetadataChangeGroup.Partitions => PartitionsItemBody(item), + MetadataChangeGroup.ConsumerGroups => ConsumerGroupItemBody(item), + MetadataChangeGroup.ConsumerGroupMembers => MemberItemBody(item), + MetadataChangeGroup.Users => UserChange(item), + MetadataChangeGroup.PersonalAccessTokens => TokenChange(item), + MetadataChangeGroup.Sessions => SessionBody(item), + _ => throw new JsonException($"Metadata current item group {groupName} has no decoder.") + }; + return new MetadataCurrentItem { Group = group, Body = body }; + } + + private static MetadataChangeBody ChangeBody(MetadataChangeKind kind, JsonElement record) + { + return kind switch + { + MetadataChangeKind.StreamCreated or MetadataChangeKind.StreamUpdated => StreamChange(record), + MetadataChangeKind.StreamDeleted => new MetadataStreamDeleted + { + StreamId = Property(record, "stream_id").GetUInt32(), + StreamCreatedOp = Property(record, "stream_created_op").GetUInt64(), + CreatedAt = Property(record, "created_at").GetUInt64(), + Name = Text(record, "name"), + RequestedOp = OptionalU64(record, "requested_op") + }, + MetadataChangeKind.TopicCreated or MetadataChangeKind.TopicUpdated => TopicChange(record), + MetadataChangeKind.TopicDeleted => new MetadataTopicDeleted + { + StreamId = Property(record, "stream_id").GetUInt32(), + StreamCreatedOp = Property(record, "stream_created_op").GetUInt64(), + TopicId = Property(record, "topic_id").GetUInt32(), + TopicCreatedOp = Property(record, "topic_created_op").GetUInt64(), + CreatedAt = Property(record, "created_at").GetUInt64(), + Name = Text(record, "name"), + RequestedOp = OptionalU64(record, "requested_op") + }, + MetadataChangeKind.PartitionsChanged => new MetadataPartitionsChanged + { + StreamId = Property(record, "stream_id").GetUInt32(), + StreamCreatedOp = Property(record, "stream_created_op").GetUInt64(), + TopicId = Property(record, "topic_id").GetUInt32(), + TopicCreatedOp = Property(record, "topic_created_op").GetUInt64(), + PartitionsCount = Property(record, "partitions_count").GetUInt32(), + RemovedPartitionIds = Ids(record, "removed_partition_ids"), + RequestedOp = OptionalU64(record, "requested_op") + }, + MetadataChangeKind.ConsumerGroupCreated => new MetadataConsumerGroupCreated + { + StreamId = Property(record, "stream_id").GetUInt32(), + StreamCreatedOp = Property(record, "stream_created_op").GetUInt64(), + TopicId = Property(record, "topic_id").GetUInt32(), + TopicCreatedOp = Property(record, "topic_created_op").GetUInt64(), + GroupId = Property(record, "group_id").GetUInt32(), + Name = Text(record, "name"), + ConsumerGroupsCount = Property(record, "consumer_groups_count").GetUInt32() + }, + MetadataChangeKind.ConsumerGroupDeleted => new MetadataConsumerGroupDeleted + { + StreamId = Property(record, "stream_id").GetUInt32(), + StreamCreatedOp = Property(record, "stream_created_op").GetUInt64(), + TopicId = Property(record, "topic_id").GetUInt32(), + TopicCreatedOp = Property(record, "topic_created_op").GetUInt64(), + GroupId = Property(record, "group_id").GetUInt32(), + Name = Text(record, "name"), + ConsumerGroupsCount = Property(record, "consumer_groups_count").GetUInt32(), + RequestedOp = OptionalU64(record, "requested_op") + }, + MetadataChangeKind.ConsumerGroupMemberJoined => new MetadataMemberJoined + { + StreamId = Property(record, "stream_id").GetUInt32(), + StreamCreatedOp = Property(record, "stream_created_op").GetUInt64(), + TopicId = Property(record, "topic_id").GetUInt32(), + TopicCreatedOp = Property(record, "topic_created_op").GetUInt64(), + GroupId = Property(record, "group_id").GetUInt32(), + MemberId = Property(record, "member_id").GetUInt32(), + ClientId = HexU128(record, "client_id"), + MembersCount = Property(record, "members_count").GetUInt32(), + PartitionIds = Ids(record, "partition_ids") + }, + MetadataChangeKind.ConsumerGroupMemberLeft => new MetadataMemberLeft + { + StreamId = Property(record, "stream_id").GetUInt32(), + StreamCreatedOp = Property(record, "stream_created_op").GetUInt64(), + TopicId = Property(record, "topic_id").GetUInt32(), + TopicCreatedOp = Property(record, "topic_created_op").GetUInt64(), + GroupId = Property(record, "group_id").GetUInt32(), + MemberId = Property(record, "member_id").GetUInt32(), + ClientId = HexU128(record, "client_id"), + MembersCount = Property(record, "members_count").GetUInt32(), + Reason = Name<MetadataMemberLeftReason>(record, "reason") + }, + MetadataChangeKind.ConsumerGroupPartitionHandedOver => new MetadataPartitionHandedOver + { + StreamId = Property(record, "stream_id").GetUInt32(), + StreamCreatedOp = Property(record, "stream_created_op").GetUInt64(), + TopicId = Property(record, "topic_id").GetUInt32(), + TopicCreatedOp = Property(record, "topic_created_op").GetUInt64(), + GroupId = Property(record, "group_id").GetUInt32(), + PartitionId = Property(record, "partition_id").GetUInt32(), + MemberId = OptionalU32(record, "member_id") + }, + MetadataChangeKind.UserCreated or MetadataChangeKind.UserUpdated => UserChange(record), + MetadataChangeKind.UserDeleted or MetadataChangeKind.UserPasswordChanged => new MetadataUserIdentity + { + UserId = Property(record, "user_id").GetUInt32(), + UserCreatedAt = Property(record, "user_created_at").GetUInt64(), + Username = Text(record, "username") + }, + MetadataChangeKind.UserPermissionsUpdated => new MetadataUserPermissionsUpdated + { + UserId = Property(record, "user_id").GetUInt32(), + UserCreatedAt = Property(record, "user_created_at").GetUInt64(), + Username = Text(record, "username"), + Permissions = ReadPermissions(record), + Reason = Name<MetadataPermissionsUpdatedReason>(record, "reason") + }, + MetadataChangeKind.PersonalAccessTokenCreated => TokenChange(record), + MetadataChangeKind.PersonalAccessTokenDeleted => new MetadataTokenDeleted + { + UserId = Property(record, "user_id").GetUInt32(), + UserCreatedAt = Property(record, "user_created_at").GetUInt64(), + Name = Text(record, "name"), + Reason = Name<MetadataTokenDeletedReason>(record, "reason") + }, + MetadataChangeKind.SessionRegistered => SessionBody(record), + MetadataChangeKind.SessionClosed => new MetadataSessionClosed + { + UserId = Property(record, "user_id").GetUInt32(), + UserCreatedAt = Property(record, "user_created_at").GetUInt64(), + ClientId = HexU128(record, "client_id"), + Session = Property(record, "session").GetUInt64(), + Reason = Name<MetadataSessionClosedReason>(record, "reason") + }, + _ => throw new JsonException($"Metadata change kind {kind} has no decoder.") + }; + } + + private static MetadataStreamChange StreamChange(JsonElement body) + { + var unreadable = ReadOptions(body, out var options); + return new MetadataStreamChange + { + StreamId = Property(body, "stream_id").GetUInt32(), + StreamCreatedOp = Property(body, "stream_created_op").GetUInt64(), + CreatedAt = Property(body, "created_at").GetUInt64(), + Name = Text(body, "name"), + Options = options?.Explicit, + DerivedOptions = options?.Derived, + OptionsUnreadable = unreadable + }; + } + + private static MetadataTopicChange TopicChange(JsonElement body) + { + var unreadable = ReadOptions(body, out var options); + return new MetadataTopicChange + { + StreamId = Property(body, "stream_id").GetUInt32(), + StreamCreatedOp = Property(body, "stream_created_op").GetUInt64(), + TopicId = Property(body, "topic_id").GetUInt32(), + TopicCreatedOp = Property(body, "topic_created_op").GetUInt64(), + CreatedAt = Property(body, "created_at").GetUInt64(), + Name = Text(body, "name"), + PartitionsCount = Property(body, "partitions_count").GetUInt32(), + Options = options?.Explicit, + DerivedOptions = options?.Derived, + OptionsUnreadable = unreadable + }; + } + + private static MetadataTopicItem TopicItemBody(JsonElement body) + { + var unreadable = ReadOptions(body, out var options); + return new MetadataTopicItem + { + StreamId = Property(body, "stream_id").GetUInt32(), + StreamCreatedOp = Property(body, "stream_created_op").GetUInt64(), + TopicId = Property(body, "topic_id").GetUInt32(), + TopicCreatedOp = Property(body, "topic_created_op").GetUInt64(), + CreatedAt = Property(body, "created_at").GetUInt64(), + Name = Text(body, "name"), + Options = options?.Explicit, + DerivedOptions = options?.Derived, + OptionsUnreadable = unreadable + }; + } + + private static MetadataPartitionsItem PartitionsItemBody(JsonElement body) + { + return new MetadataPartitionsItem + { + StreamId = Property(body, "stream_id").GetUInt32(), + StreamCreatedOp = Property(body, "stream_created_op").GetUInt64(), + TopicId = Property(body, "topic_id").GetUInt32(), + TopicCreatedOp = Property(body, "topic_created_op").GetUInt64(), + PartitionsCount = Property(body, "partitions_count").GetUInt32() + }; + } + + private static MetadataConsumerGroupItem ConsumerGroupItemBody(JsonElement body) + { + return new MetadataConsumerGroupItem + { + StreamId = Property(body, "stream_id").GetUInt32(), + StreamCreatedOp = Property(body, "stream_created_op").GetUInt64(), + TopicId = Property(body, "topic_id").GetUInt32(), + TopicCreatedOp = Property(body, "topic_created_op").GetUInt64(), + GroupId = Property(body, "group_id").GetUInt32(), + Name = Text(body, "name") + }; + } + + private static MetadataMemberItem MemberItemBody(JsonElement body) + { + return new MetadataMemberItem + { + StreamId = Property(body, "stream_id").GetUInt32(), + StreamCreatedOp = Property(body, "stream_created_op").GetUInt64(), + TopicId = Property(body, "topic_id").GetUInt32(), + TopicCreatedOp = Property(body, "topic_created_op").GetUInt64(), + GroupId = Property(body, "group_id").GetUInt32(), + MemberId = Property(body, "member_id").GetUInt32(), + ClientId = HexU128(body, "client_id"), + PartitionIds = Ids(body, "partition_ids") + }; + } + + private static MetadataUserChange UserChange(JsonElement body) + { + var unreadable = ReadOptions(body, out var options); + return new MetadataUserChange + { + UserId = Property(body, "user_id").GetUInt32(), + UserCreatedAt = Property(body, "user_created_at").GetUInt64(), + Username = Text(body, "username"), + Status = Text(body, "status") switch + { + "active" => UserStatus.Active, + "inactive" => UserStatus.Inactive, + var status => throw new JsonException($"User status {status} is unknown.") + }, + Permissions = ReadPermissions(body), + Options = options?.Explicit, + DerivedOptions = options?.Derived, + OptionsUnreadable = unreadable + }; + } + + private static MetadataTokenChange TokenChange(JsonElement body) + { + return new MetadataTokenChange + { + UserId = Property(body, "user_id").GetUInt32(), + UserCreatedAt = Property(body, "user_created_at").GetUInt64(), + Name = Text(body, "name"), + Expiry = OptionalU64(body, "expiry") + }; + } + + private static MetadataSession SessionBody(JsonElement body) + { + return new MetadataSession + { + UserId = Property(body, "user_id").GetUInt32(), + UserCreatedAt = Property(body, "user_created_at").GetUInt64(), + ClientId = HexU128(body, "client_id"), + Session = Property(body, "session").GetUInt64() + }; + } + + /// <summary> + /// Reads <c>options</c> and <c>options_unreadable</c>: an object with the flag absent or false, or null with + /// the flag true. Returns whether the state was unreadable. + /// </summary> + private static bool ReadOptions(JsonElement body, out ResourceOptions? options) + { + options = null; + var unreadable = body.TryGetProperty("options_unreadable", out var flag) && flag.GetBoolean(); + var hasOptions = body.TryGetProperty("options", out var value) && value.ValueKind != JsonValueKind.Null; + if (hasOptions == unreadable) + { + throw new JsonException("Options must be an object, or null with options_unreadable true."); + } + + if (hasOptions) + { + options = value.Deserialize<ResourceOptions>(BodyOptions); + } + + return unreadable; + } + + private static Permissions? ReadPermissions(JsonElement body) + { + return Property(body, "permissions") is { ValueKind: not JsonValueKind.Null } permissions + ? permissions.Deserialize<Permissions>(BodyOptions) + : null; + } + + private static uint[] Ids(JsonElement body, string name) + { + var ids = Property(body, name); + var result = new uint[ids.GetArrayLength()]; + var index = 0; + foreach (var id in ids.EnumerateArray()) + { + result[index++] = id.GetUInt32(); + } + + return result; + } + + private static T Name<T>(JsonElement body, string name) where T : struct, Enum + { + var text = Text(body, name); + return SnakeCaseNames<T>.TryParse(text, out var value) + ? value + : throw new JsonException($"Metadata changes {name} {text} is unknown."); + } + + private static MetadataChangeCursor Cursor(JsonElement element) + { + return MetadataChangeCursor.TryParse(element.GetString(), out var cursor) + ? cursor + : throw new JsonException("Metadata changes next is not a canonical change cursor."); + } + + private static MetadataCurrentPageToken PageToken(JsonElement answer) + { + return MetadataCurrentPageToken.TryParse(Property(answer, "current_page_token").GetString(), out var token) + ? token + : throw new JsonException("Metadata changes current_page_token is not a canonical page token."); + } + + private static UInt128 HexU128(JsonElement body, string name) + { + var text = Text(body, name); + var canonical = text.Length == HEX_U128_LENGTH && text.All(character => char.IsAsciiDigit(character) + || character is >= 'a' and <= 'f'); + return canonical + ? UInt128.Parse(text, NumberStyles.AllowHexSpecifier, CultureInfo.InvariantCulture) + : throw new JsonException($"Metadata changes {name} must be {HEX_U128_LENGTH} lowercase hex characters."); + } + + private static ulong? OptionalU64(JsonElement body, string name) + { + return body.TryGetProperty(name, out var value) && value.ValueKind != JsonValueKind.Null + ? value.GetUInt64() + : null; + } + + private static uint? OptionalU32(JsonElement body, string name) + { + return body.TryGetProperty(name, out var value) && value.ValueKind != JsonValueKind.Null + ? value.GetUInt32() + : null; + } + + private static string Text(JsonElement body, string name) + { + return Property(body, name).GetString() ?? throw new JsonException($"Metadata changes {name} is null."); + } + + private static bool IsNull(JsonElement body, string name) + { + return Property(body, name).ValueKind == JsonValueKind.Null; + } + + private static JsonElement Property(JsonElement body, string name) + { + return body.TryGetProperty(name, out var value) + ? value + : throw new JsonException($"Metadata changes {name} is missing."); + } +}
diff --git a/foreign/csharp/Iggy_SDK/Mappers/BinaryMapper.MetadataChanges.cs b/foreign/csharp/Iggy_SDK/Mappers/BinaryMapper.MetadataChanges.cs new file mode 100644 index 0000000..065ce16 --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Mappers/BinaryMapper.MetadataChanges.cs
@@ -0,0 +1,773 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +using System.Buffers.Binary; +using System.Text; +using Apache.Iggy.Contracts; +using Apache.Iggy.Contracts.Auth; +using Apache.Iggy.Enums; +using Apache.Iggy.Exceptions; +using Apache.Iggy.Headers; + +namespace Apache.Iggy.Mappers; + +internal static partial class BinaryMapper +{ + private const int METADATA_RECORD_MIN_SIZE = 8 + 4 + 16 + 8 + 1 + 1 + 1 + 4; + private const int METADATA_ITEM_MIN_SIZE = 1 + 4; + + /// <summary> + /// Maps a metadata changes answer. Bodies are decoded strictly: unknown reason codes, a known kind under the + /// wrong group, presence bytes other than 0 and 1 and trailing bytes are refused, while a group or kind this + /// build does not know is kept with its raw body. + /// </summary> + internal static MetadataChanges MapMetadataChanges(ReadOnlySpan<byte> payload) + { + var reader = new WireReader(payload); + var status = (MetadataChangeStatus)reader.CodeAtMost((byte)MetadataChangeStatus.Reset, "status"); + var phase = (MetadataChangePhase)reader.CodeAtMost((byte)MetadataChangePhase.Current, "phase"); + UInt128? logId = reader.Flag("log id presence") ? reader.U128() : null; + var oldestOp = reader.U64(); + var retainedSince = reader.U64(); + var headOp = reader.U64(); + MetadataChangeCursor? next = reader.Flag("next cursor presence") ? reader.Cursor() : null; + var hasMore = reader.Flag("has more"); + + var records = new MetadataChangeRecord[ValidatedCollectionSize(reader.U32(), reader.Remaining, + METADATA_RECORD_MIN_SIZE, "Metadata change records count")]; + for (var index = 0; index < records.Length; index++) + { + records[index] = ReadChangeRecord(ref reader); + } + + var items = new MetadataCurrentItem[ValidatedCollectionSize(reader.U32(), reader.Remaining, + METADATA_ITEM_MIN_SIZE, "Metadata current items count")]; + for (var index = 0; index < items.Length; index++) + { + items[index] = ReadCurrentItem(ref reader); + } + + MetadataCurrentPageToken? pageToken = null; + var tokenLength = reader.U16(); + if (tokenLength > 0 && !MetadataCurrentPageToken.TryDecode(reader.Bytes(tokenLength), out pageToken)) + { + throw new MalformedResponseException("Metadata changes answer carries a malformed page token."); + } + + var holdsOtherPhase = phase == MetadataChangePhase.Changes + ? items.Length > 0 || pageToken is not null + : records.Length > 0; + if (holdsOtherPhase) + { + throw new MalformedResponseException( + "Metadata changes answer mixes phases: a changes page carries only records, a current page only items."); + } + + return new MetadataChanges + { + Status = status, + Phase = phase, + LogId = logId, + OldestOp = oldestOp, + RetainedSince = retainedSince, + HeadOp = headOp, + Next = next, + HasMore = hasMore, + Records = records, + Items = items, + CurrentPageToken = pageToken + }; + } + + /// <summary>Maps exactly one change record envelope with its body.</summary> + internal static MetadataChangeRecord MapMetadataChangeRecord(ReadOnlySpan<byte> payload) + { + var reader = new WireReader(payload); + var record = ReadChangeRecord(ref reader); + reader.EnsureConsumed("change record"); + return record; + } + + /// <summary>Maps exactly one current state item envelope with its body.</summary> + internal static MetadataCurrentItem MapMetadataCurrentItem(ReadOnlySpan<byte> payload) + { + var reader = new WireReader(payload); + var item = ReadCurrentItem(ref reader); + reader.EnsureConsumed("current item"); + return item; + } + + /// <summary> + /// Decodes a change body under its group and kind codes. A group or kind this build does not know yields + /// <see cref="MetadataUnknownBody" /> and a null kind. + /// </summary> + internal static MetadataChangeBody MapMetadataChangeBody(byte group, byte kind, ReadOnlySpan<byte> body, + out MetadataChangeKind? knownKind) + { + knownKind = null; + var changeGroup = (MetadataChangeGroup)group; + var changeKind = (MetadataChangeKind)kind; + if (!MetadataChangeGroups.IsDefined(changeGroup) || !MetadataChangeGroups.IsDefined(changeKind)) + { + return new MetadataUnknownBody { Group = group, Kind = kind, Body = body.ToArray() }; + } + + if (MetadataChangeGroups.Of(changeKind) != changeGroup) + { + throw new MalformedResponseException($"Metadata change kind {kind} does not belong to group {group}."); + } + + knownKind = changeKind; + var reader = new WireReader(body); + MetadataChangeBody change = changeKind switch + { + MetadataChangeKind.StreamCreated or MetadataChangeKind.StreamUpdated => ReadStream(ref reader), + MetadataChangeKind.StreamDeleted => ReadStreamDeleted(ref reader), + MetadataChangeKind.TopicCreated or MetadataChangeKind.TopicUpdated => ReadTopic(ref reader), + MetadataChangeKind.TopicDeleted => ReadTopicDeleted(ref reader), + MetadataChangeKind.PartitionsChanged => ReadPartitionsChanged(ref reader), + MetadataChangeKind.ConsumerGroupCreated => ReadConsumerGroupCreated(ref reader), + MetadataChangeKind.ConsumerGroupDeleted => ReadConsumerGroupDeleted(ref reader), + MetadataChangeKind.ConsumerGroupMemberJoined => ReadMemberJoined(ref reader), + MetadataChangeKind.ConsumerGroupMemberLeft => ReadMemberLeft(ref reader), + MetadataChangeKind.ConsumerGroupPartitionHandedOver => ReadPartitionHandedOver(ref reader), + MetadataChangeKind.UserCreated or MetadataChangeKind.UserUpdated => ReadUser(ref reader), + MetadataChangeKind.UserDeleted or MetadataChangeKind.UserPasswordChanged => ReadUserIdentity(ref reader), + MetadataChangeKind.UserPermissionsUpdated => ReadUserPermissions(ref reader), + MetadataChangeKind.PersonalAccessTokenCreated => ReadToken(ref reader), + MetadataChangeKind.PersonalAccessTokenDeleted => ReadTokenDeleted(ref reader), + MetadataChangeKind.SessionRegistered => ReadSession(ref reader), + MetadataChangeKind.SessionClosed => ReadSessionClosed(ref reader), + _ => throw new MalformedResponseException($"Metadata change kind {kind} has no decoder.") + }; + reader.EnsureConsumed("change body"); + return change; + } + + /// <summary> + /// Decodes a current state item body under its group code. A group this build does not know yields + /// <see cref="MetadataUnknownBody" /> and a null group. + /// </summary> + internal static MetadataChangeBody MapMetadataItemBody(byte group, ReadOnlySpan<byte> body, + out MetadataChangeGroup? knownGroup) + { + knownGroup = null; + var itemGroup = (MetadataChangeGroup)group; + if (!MetadataChangeGroups.IsDefined(itemGroup)) + { + return new MetadataUnknownBody { Group = group, Body = body.ToArray() }; + } + + knownGroup = itemGroup; + var reader = new WireReader(body); + MetadataChangeBody item = itemGroup switch + { + MetadataChangeGroup.Streams => ReadStream(ref reader), + MetadataChangeGroup.Topics => ReadTopicItem(ref reader), + MetadataChangeGroup.Partitions => ReadPartitionsItem(ref reader), + MetadataChangeGroup.ConsumerGroups => ReadConsumerGroupItem(ref reader), + MetadataChangeGroup.ConsumerGroupMembers => ReadMemberItem(ref reader), + MetadataChangeGroup.Users => ReadUser(ref reader), + MetadataChangeGroup.PersonalAccessTokens => ReadToken(ref reader), + MetadataChangeGroup.Sessions => ReadSession(ref reader), + _ => throw new MalformedResponseException($"Metadata current item group {group} has no decoder.") + }; + reader.EnsureConsumed("current item body"); + return item; + } + + private static MetadataChangeRecord ReadChangeRecord(ref WireReader reader) + { + var op = reader.U64(); + var ordinal = reader.U32(); + var checksum = reader.U128(); + var timestamp = reader.U64(); + uint? actorUserId = reader.Flag("actor presence") ? reader.U32() : null; + var group = reader.U8(); + var kind = reader.U8(); + var body = MapMetadataChangeBody(group, kind, reader.Bytes(reader.U32()), out var knownKind); + return new MetadataChangeRecord + { + Op = op, + Ordinal = ordinal, + Checksum = checksum, + Timestamp = timestamp, + ActorUserId = actorUserId, + Kind = knownKind, + Body = body + }; + } + + private static MetadataCurrentItem ReadCurrentItem(ref WireReader reader) + { + var group = reader.U8(); + var body = MapMetadataItemBody(group, reader.Bytes(reader.U32()), out var knownGroup); + return new MetadataCurrentItem { Group = knownGroup, Body = body }; + } + + private static MetadataStreamChange ReadStream(ref WireReader reader) + { + var streamId = reader.U32(); + var streamCreatedOp = reader.U64(); + var createdAt = reader.U64(); + var name = reader.Name(); + var unreadable = ReadChangeOptions(ref reader, out var options, out var derivedOptions); + return new MetadataStreamChange + { + StreamId = streamId, + StreamCreatedOp = streamCreatedOp, + CreatedAt = createdAt, + Name = name, + Options = options, + DerivedOptions = derivedOptions, + OptionsUnreadable = unreadable + }; + } + + private static MetadataStreamDeleted ReadStreamDeleted(ref WireReader reader) + { + return new MetadataStreamDeleted + { + StreamId = reader.U32(), + StreamCreatedOp = reader.U64(), + CreatedAt = reader.U64(), + Name = reader.Name(), + RequestedOp = reader.OptionalU64("requested op presence") + }; + } + + private static MetadataTopicChange ReadTopic(ref WireReader reader) + { + var topic = ReadTopicKey(ref reader); + var createdAt = reader.U64(); + var name = reader.Name(); + var partitionsCount = reader.U32(); + var unreadable = ReadChangeOptions(ref reader, out var options, out var derivedOptions); + return new MetadataTopicChange + { + StreamId = topic.StreamId, + StreamCreatedOp = topic.StreamCreatedOp, + TopicId = topic.TopicId, + TopicCreatedOp = topic.TopicCreatedOp, + CreatedAt = createdAt, + Name = name, + PartitionsCount = partitionsCount, + Options = options, + DerivedOptions = derivedOptions, + OptionsUnreadable = unreadable + }; + } + + private static MetadataTopicItem ReadTopicItem(ref WireReader reader) + { + var topic = ReadTopicKey(ref reader); + var createdAt = reader.U64(); + var name = reader.Name(); + var unreadable = ReadChangeOptions(ref reader, out var options, out var derivedOptions); + return new MetadataTopicItem + { + StreamId = topic.StreamId, + StreamCreatedOp = topic.StreamCreatedOp, + TopicId = topic.TopicId, + TopicCreatedOp = topic.TopicCreatedOp, + CreatedAt = createdAt, + Name = name, + Options = options, + DerivedOptions = derivedOptions, + OptionsUnreadable = unreadable + }; + } + + private static MetadataTopicDeleted ReadTopicDeleted(ref WireReader reader) + { + var topic = ReadTopicKey(ref reader); + return new MetadataTopicDeleted + { + StreamId = topic.StreamId, + StreamCreatedOp = topic.StreamCreatedOp, + TopicId = topic.TopicId, + TopicCreatedOp = topic.TopicCreatedOp, + CreatedAt = reader.U64(), + Name = reader.Name(), + RequestedOp = reader.OptionalU64("requested op presence") + }; + } + + private static MetadataPartitionsChanged ReadPartitionsChanged(ref WireReader reader) + { + var topic = ReadTopicKey(ref reader); + return new MetadataPartitionsChanged + { + StreamId = topic.StreamId, + StreamCreatedOp = topic.StreamCreatedOp, + TopicId = topic.TopicId, + TopicCreatedOp = topic.TopicCreatedOp, + PartitionsCount = reader.U32(), + RemovedPartitionIds = reader.PartitionIds(), + RequestedOp = reader.OptionalU64("requested op presence") + }; + } + + private static MetadataPartitionsItem ReadPartitionsItem(ref WireReader reader) + { + var topic = ReadTopicKey(ref reader); + return new MetadataPartitionsItem + { + StreamId = topic.StreamId, + StreamCreatedOp = topic.StreamCreatedOp, + TopicId = topic.TopicId, + TopicCreatedOp = topic.TopicCreatedOp, + PartitionsCount = reader.U32() + }; + } + + private static MetadataConsumerGroupCreated ReadConsumerGroupCreated(ref WireReader reader) + { + var topic = ReadTopicKey(ref reader); + return new MetadataConsumerGroupCreated + { + StreamId = topic.StreamId, + StreamCreatedOp = topic.StreamCreatedOp, + TopicId = topic.TopicId, + TopicCreatedOp = topic.TopicCreatedOp, + GroupId = reader.U32(), + Name = reader.Name(), + ConsumerGroupsCount = reader.U32() + }; + } + + private static MetadataConsumerGroupDeleted ReadConsumerGroupDeleted(ref WireReader reader) + { + var topic = ReadTopicKey(ref reader); + return new MetadataConsumerGroupDeleted + { + StreamId = topic.StreamId, + StreamCreatedOp = topic.StreamCreatedOp, + TopicId = topic.TopicId, + TopicCreatedOp = topic.TopicCreatedOp, + GroupId = reader.U32(), + Name = reader.Name(), + ConsumerGroupsCount = reader.U32(), + RequestedOp = reader.OptionalU64("requested op presence") + }; + } + + private static MetadataConsumerGroupItem ReadConsumerGroupItem(ref WireReader reader) + { + var topic = ReadTopicKey(ref reader); + return new MetadataConsumerGroupItem + { + StreamId = topic.StreamId, + StreamCreatedOp = topic.StreamCreatedOp, + TopicId = topic.TopicId, + TopicCreatedOp = topic.TopicCreatedOp, + GroupId = reader.U32(), + Name = reader.Name() + }; + } + + private static MetadataMemberJoined ReadMemberJoined(ref WireReader reader) + { + var topic = ReadTopicKey(ref reader); + return new MetadataMemberJoined + { + StreamId = topic.StreamId, + StreamCreatedOp = topic.StreamCreatedOp, + TopicId = topic.TopicId, + TopicCreatedOp = topic.TopicCreatedOp, + GroupId = reader.U32(), + MemberId = reader.U32(), + ClientId = reader.U128(), + MembersCount = reader.U32(), + PartitionIds = reader.PartitionIds() + }; + } + + private static MetadataMemberLeft ReadMemberLeft(ref WireReader reader) + { + var topic = ReadTopicKey(ref reader); + return new MetadataMemberLeft + { + StreamId = topic.StreamId, + StreamCreatedOp = topic.StreamCreatedOp, + TopicId = topic.TopicId, + TopicCreatedOp = topic.TopicCreatedOp, + GroupId = reader.U32(), + MemberId = reader.U32(), + ClientId = reader.U128(), + MembersCount = reader.U32(), + Reason = (MetadataMemberLeftReason)reader.CodeAtMost((byte)MetadataMemberLeftReason.SessionEnded, + "member left reason") + }; + } + + private static MetadataPartitionHandedOver ReadPartitionHandedOver(ref WireReader reader) + { + var topic = ReadTopicKey(ref reader); + return new MetadataPartitionHandedOver + { + StreamId = topic.StreamId, + StreamCreatedOp = topic.StreamCreatedOp, + TopicId = topic.TopicId, + TopicCreatedOp = topic.TopicCreatedOp, + GroupId = reader.U32(), + PartitionId = reader.U32(), + MemberId = reader.Flag("member presence") ? reader.U32() : null + }; + } + + private static MetadataMemberItem ReadMemberItem(ref WireReader reader) + { + var topic = ReadTopicKey(ref reader); + return new MetadataMemberItem + { + StreamId = topic.StreamId, + StreamCreatedOp = topic.StreamCreatedOp, + TopicId = topic.TopicId, + TopicCreatedOp = topic.TopicCreatedOp, + GroupId = reader.U32(), + MemberId = reader.U32(), + ClientId = reader.U128(), + PartitionIds = reader.PartitionIds() + }; + } + + private static MetadataUserChange ReadUser(ref WireReader reader) + { + var userId = reader.U32(); + var userCreatedAt = reader.U64(); + var username = reader.Name(); + var status = reader.U8() switch + { + 1 => UserStatus.Active, + 2 => UserStatus.Inactive, + var code => throw new MalformedResponseException($"Metadata change user status {code} is unknown.") + }; + var permissions = reader.Flag("permissions presence") ? reader.Permissions() : null; + var unreadable = ReadChangeOptions(ref reader, out var options, out var derivedOptions); + return new MetadataUserChange + { + UserId = userId, + UserCreatedAt = userCreatedAt, + Username = username, + Status = status, + Permissions = permissions, + Options = options, + DerivedOptions = derivedOptions, + OptionsUnreadable = unreadable + }; + } + + private static MetadataUserIdentity ReadUserIdentity(ref WireReader reader) + { + return new MetadataUserIdentity + { + UserId = reader.U32(), + UserCreatedAt = reader.U64(), + Username = reader.Name() + }; + } + + private static MetadataUserPermissionsUpdated ReadUserPermissions(ref WireReader reader) + { + return new MetadataUserPermissionsUpdated + { + UserId = reader.U32(), + UserCreatedAt = reader.U64(), + Username = reader.Name(), + Permissions = reader.Flag("permissions presence") ? reader.Permissions() : null, + Reason = (MetadataPermissionsUpdatedReason)reader.CodeAtMost( + (byte)MetadataPermissionsUpdatedReason.ResourceDeleted, "permissions updated reason") + }; + } + + private static MetadataTokenChange ReadToken(ref WireReader reader) + { + var userId = reader.U32(); + var userCreatedAt = reader.U64(); + var name = reader.Name(); + var expiry = reader.U64(); + return new MetadataTokenChange + { + UserId = userId, + UserCreatedAt = userCreatedAt, + Name = name, + Expiry = expiry == 0 ? null : expiry + }; + } + + private static MetadataTokenDeleted ReadTokenDeleted(ref WireReader reader) + { + return new MetadataTokenDeleted + { + UserId = reader.U32(), + UserCreatedAt = reader.U64(), + Name = reader.Name(), + Reason = (MetadataTokenDeletedReason)reader.CodeAtMost((byte)MetadataTokenDeletedReason.Expired, + "token deleted reason") + }; + } + + private static MetadataSession ReadSession(ref WireReader reader) + { + return new MetadataSession + { + UserId = reader.U32(), + UserCreatedAt = reader.U64(), + ClientId = reader.U128(), + Session = reader.U64() + }; + } + + private static MetadataSessionClosed ReadSessionClosed(ref WireReader reader) + { + return new MetadataSessionClosed + { + UserId = reader.U32(), + UserCreatedAt = reader.U64(), + ClientId = reader.U128(), + Session = reader.U64(), + Reason = (MetadataSessionClosedReason)reader.CodeAtMost((byte)MetadataSessionClosedReason.Revoked, + "session closed reason") + }; + } + + private static (uint StreamId, ulong StreamCreatedOp, uint TopicId, ulong TopicCreatedOp) ReadTopicKey( + ref WireReader reader) + { + return (reader.U32(), reader.U64(), reader.U32(), reader.U64()); + } + + /// <summary> + /// Reads <c>[state:u8]</c>: 0 followed by the explicit and derived option blocks exactly as the read + /// responses carry them, or 1 for stored state the server cannot read. Returns whether it was unreadable. + /// </summary> + private static bool ReadChangeOptions(ref WireReader reader, out Dictionary<HeaderKey, HeaderValue>? options, + out Dictionary<HeaderKey, HeaderValue>? derivedOptions) + { + options = null; + derivedOptions = null; + var state = reader.CodeAtMost(1, "options state"); + if (state == 1) + { + return true; + } + + options = MapOptions(reader.Buffer, reader.Position, out var explicitBytes); + reader.Skip(explicitBytes); + derivedOptions = MapOptions(reader.Buffer, reader.Position, out var derivedBytes); + reader.Skip(derivedBytes); + return false; + } + + /// <summary> + /// A bounds-checked little-endian cursor over a metadata changes answer or a user's permission set. Every + /// overrun is a <see cref="MalformedResponseException" /> instead of a slicing exception. + /// </summary> + private ref struct WireReader + { + private static readonly UTF8Encoding StrictUtf8 = new(false, true); + + internal WireReader(ReadOnlySpan<byte> buffer) + { + Buffer = buffer; + } + + internal ReadOnlySpan<byte> Buffer { get; } + + internal int Position { get; private set; } + + internal readonly int Remaining => Buffer.Length - Position; + + internal ReadOnlySpan<byte> Bytes(uint length) + { + if (length > Remaining) + { + throw new MalformedResponseException( + $"Response field of {length} bytes at byte {Position} overruns the {Buffer.Length}-byte payload."); + } + + var bytes = Buffer.Slice(Position, (int)length); + Position += (int)length; + return bytes; + } + + internal void Skip(int length) + { + Bytes((uint)length); + } + + internal byte U8() + { + return Bytes(1)[0]; + } + + internal ushort U16() + { + return BinaryPrimitives.ReadUInt16LittleEndian(Bytes(2)); + } + + internal uint U32() + { + return BinaryPrimitives.ReadUInt32LittleEndian(Bytes(4)); + } + + internal ulong U64() + { + return BinaryPrimitives.ReadUInt64LittleEndian(Bytes(8)); + } + + internal UInt128 U128() + { + return BinaryPrimitives.ReadUInt128LittleEndian(Bytes(16)); + } + + internal byte CodeAtMost(byte maximum, string field) + { + var offset = Position; + var code = U8(); + return code <= maximum + ? code + : throw new MalformedResponseException($"Metadata changes {field} {code} at byte {offset} is unknown."); + } + + internal bool Flag(string field) + { + return CodeAtMost(1, field) == 1; + } + + internal ulong? OptionalU64(string field) + { + return Flag(field) ? U64() : null; + } + + internal string Name() + { + var offset = Position; + var length = U8(); + if (length == 0) + { + throw new MalformedResponseException($"Metadata changes name at byte {offset} is empty."); + } + + try + { + return StrictUtf8.GetString(Bytes(length)); + } + catch (DecoderFallbackException) + { + throw new MalformedResponseException($"Metadata changes name at byte {offset} is not UTF-8."); + } + } + + internal MetadataChangeCursor Cursor() + { + var cursor = MetadataChangeCursor.Read(Bytes(MetadataChangeCursor.SIZE)); + return cursor.IsCanonical + ? cursor + : throw new MalformedResponseException("Metadata changes cursor at op 0 carries a checksum or ordinal."); + } + + /// <summary>Reads <c>[count:u32][id:u32]*</c>, refusing ids that are not strictly ascending.</summary> + internal uint[] PartitionIds() + { + var ids = new uint[ValidatedCollectionSize(U32(), Remaining, 4, "Metadata changes partition ids count")]; + for (var index = 0; index < ids.Length; index++) + { + ids[index] = U32(); + if (index > 0 && ids[index] <= ids[index - 1]) + { + throw new MalformedResponseException("Metadata changes partition ids must be strictly ascending."); + } + } + + return ids; + } + + /// <summary> + /// Reads the public permission set: ten global flags, then streams and their topics, each list + /// continued while its next flag is set. + /// </summary> + internal Permissions Permissions() + { + var global = new GlobalPermissions + { + ManageServers = U8() != 0, + ReadServers = U8() != 0, + ManageUsers = U8() != 0, + ReadUsers = U8() != 0, + ManageStreams = U8() != 0, + ReadStreams = U8() != 0, + ManageTopics = U8() != 0, + ReadTopics = U8() != 0, + PollMessages = U8() != 0, + SendMessages = U8() != 0 + }; + + Dictionary<uint, StreamPermissions>? streams = null; + var hasStream = U8() != 0; + while (hasStream) + { + var streamId = U32(); + var manageStream = U8() != 0; + var readStream = U8() != 0; + var manageTopics = U8() != 0; + var readTopics = U8() != 0; + var pollMessages = U8() != 0; + var sendMessages = U8() != 0; + Dictionary<uint, TopicPermissions>? topics = null; + var hasTopic = U8() != 0; + while (hasTopic) + { + topics ??= new Dictionary<uint, TopicPermissions>(); + topics[U32()] = new TopicPermissions + { + ManageTopic = U8() != 0, + ReadTopic = U8() != 0, + PollMessages = U8() != 0, + SendMessages = U8() != 0 + }; + hasTopic = U8() != 0; + } + + streams ??= new Dictionary<uint, StreamPermissions>(); + streams[streamId] = new StreamPermissions + { + ManageStream = manageStream, + ReadStream = readStream, + ManageTopics = manageTopics, + ReadTopics = readTopics, + PollMessages = pollMessages, + SendMessages = sendMessages, + Topics = topics + }; + hasStream = U8() != 0; + } + + return new Permissions { Global = global, Streams = streams }; + } + + internal readonly void EnsureConsumed(string what) + { + if (Remaining != 0) + { + throw new MalformedResponseException( + $"Metadata changes {what} has {Remaining} trailing bytes after byte {Position}."); + } + } + } +}
diff --git a/foreign/csharp/Iggy_SDK/Mappers/BinaryMapper.cs b/foreign/csharp/Iggy_SDK/Mappers/BinaryMapper.cs index 0d4b814..5027742 100644 --- a/foreign/csharp/Iggy_SDK/Mappers/BinaryMapper.cs +++ b/foreign/csharp/Iggy_SDK/Mappers/BinaryMapper.cs
@@ -31,7 +31,7 @@ namespace Apache.Iggy.Mappers; -internal static class BinaryMapper +internal static partial class BinaryMapper { private const int CONSUMER_GROUP_HEADER_SIZE = 13; private const int MEMBER_HEADER_SIZE = 8; @@ -113,7 +113,7 @@ { var permissionLength = ReadLength(payload, position + 1, "User permissions"); ReadOnlySpan<byte> permissionsPayload = payload[(position + 5)..(position + 5 + permissionLength)]; - var permissions = MapPermissions(permissionsPayload); + var permissions = new WireReader(permissionsPayload).Permissions(); return new UserResponse { Permissions = permissions, @@ -138,94 +138,6 @@ }; } - private static Permissions MapPermissions(ReadOnlySpan<byte> bytes) - { - var streamMap = new Dictionary<uint, StreamPermissions>(); - var index = 0; - - var globalPermissions = new GlobalPermissions - { - ManageServers = bytes[index++] == 1, - ReadServers = bytes[index++] == 1, - ManageUsers = bytes[index++] == 1, - ReadUsers = bytes[index++] == 1, - ManageStreams = bytes[index++] == 1, - ReadStreams = bytes[index++] == 1, - ManageTopics = bytes[index++] == 1, - ReadTopics = bytes[index++] == 1, - PollMessages = bytes[index++] == 1, - SendMessages = bytes[index++] == 1 - }; - - if (bytes[index++] == 1) - { - while (true) - { - var streamId = BinaryPrimitives.ReadUInt32LittleEndian(bytes[index..(index + 4)]); - index += sizeof(uint); - - var manageStream = bytes[index++] == 1; - var readStream = bytes[index++] == 1; - var manageTopics = bytes[index++] == 1; - var readTopics = bytes[index++] == 1; - var pollMessagesStream = bytes[index++] == 1; - var sendMessagesStream = bytes[index++] == 1; - var topicsMap = new Dictionary<uint, TopicPermissions>(); - - if (bytes[index++] == 1) - { - while (true) - { - var topicId = BinaryPrimitives.ReadUInt32LittleEndian(bytes[index..(index + 4)]); - index += sizeof(uint); - - var manageTopic = bytes[index++] == 1; - var readTopic = bytes[index++] == 1; - var pollMessagesTopic = bytes[index++] == 1; - var sendMessagesTopic = bytes[index++] == 1; - - topicsMap.Add(topicId, - new TopicPermissions - { - ManageTopic = manageTopic, - ReadTopic = readTopic, - PollMessages = pollMessagesTopic, - SendMessages = sendMessagesTopic - }); - - if (bytes[index++] == 0) - { - break; - } - } - } - - streamMap.Add(streamId, - new StreamPermissions - { - ManageStream = manageStream, - ReadStream = readStream, - ManageTopics = manageTopics, - ReadTopics = readTopics, - PollMessages = pollMessagesStream, - SendMessages = sendMessagesStream, - Topics = topicsMap.Count > 0 ? topicsMap : null - }); - - if (bytes[index++] == 0) - { - break; - } - } - } - - return new Permissions - { - Global = globalPermissions, - Streams = streamMap.Count > 0 ? streamMap : null - }; - } - private static (UserResponse response, int position) MapToUserResponse(ReadOnlySpan<byte> payload, int position) { var id = BinaryPrimitives.ReadUInt32LittleEndian(payload[position..(position + 4)]);
diff --git a/foreign/csharp/Iggy_SDK/Utils/Base64Url.cs b/foreign/csharp/Iggy_SDK/Utils/Base64Url.cs new file mode 100644 index 0000000..e31dcbf --- /dev/null +++ b/foreign/csharp/Iggy_SDK/Utils/Base64Url.cs
@@ -0,0 +1,78 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +using System.Diagnostics.CodeAnalysis; + +namespace Apache.Iggy.Utils; + +/// <summary> +/// Canonical unpadded Base64url, the text form of metadata change cursors and page tokens. +/// </summary> +internal static class Base64Url +{ + internal static string Encode(ReadOnlySpan<byte> bytes) + { + return Convert.ToBase64String(bytes).TrimEnd('=').Replace('+', '-').Replace('/', '_'); + } + + /// <summary> + /// Decodes <paramref name="text" /> only when it is the canonical encoding of its bytes: the Base64url + /// alphabet, no padding and zero trailing bits, so one value has exactly one text form. + /// </summary> + internal static bool TryDecode(ReadOnlySpan<char> text, [NotNullWhen(true)] out byte[]? bytes) + { + bytes = null; + if (text.Length % 4 == 1) + { + return false; + } + + var padding = (4 - text.Length % 4) % 4; + var standard = new char[text.Length + padding]; + for (var index = 0; index < text.Length; index++) + { + var character = text[index]; + standard[index] = character switch + { + >= 'A' and <= 'Z' or >= 'a' and <= 'z' or >= '0' and <= '9' => character, + '-' => '+', + '_' => '/', + _ => '\0' + }; + if (standard[index] == '\0') + { + return false; + } + } + + standard.AsSpan(text.Length).Fill('='); + var decoded = new byte[standard.Length / 4 * 3]; + if (!Convert.TryFromBase64Chars(standard, decoded, out var written)) + { + return false; + } + + decoded = decoded[..written]; + if (!Encode(decoded).AsSpan().SequenceEqual(text)) + { + return false; + } + + bytes = decoded; + return true; + } +}
diff --git a/foreign/csharp/Iggy_SDK/Utils/CommandCodes.cs b/foreign/csharp/Iggy_SDK/Utils/CommandCodes.cs index f1f2349..1b46305 100644 --- a/foreign/csharp/Iggy_SDK/Utils/CommandCodes.cs +++ b/foreign/csharp/Iggy_SDK/Utils/CommandCodes.cs
@@ -26,6 +26,7 @@ internal const int DESCRIBE_OPTIONS_CODE = 13; internal const int BIND_SESSION_CODE = 15; internal const int UPDATE_RESOURCE_CODE = 16; + internal const int GET_METADATA_CHANGES_CODE = 17; internal const int GET_ME_CODE = 20; internal const int GET_CLIENT_CODE = 21; internal const int GET_CLIENTS_CODE = 22;
diff --git a/foreign/csharp/Iggy_SDK/Vsr/VsrOperation.cs b/foreign/csharp/Iggy_SDK/Vsr/VsrOperation.cs index 05f2c09..7431743 100644 --- a/foreign/csharp/Iggy_SDK/Vsr/VsrOperation.cs +++ b/foreign/csharp/Iggy_SDK/Vsr/VsrOperation.cs
@@ -88,6 +88,7 @@ CommandCodes.GET_STATS_CODE, CommandCodes.GET_SNAPSHOT_CODE, CommandCodes.GET_CLUSTER_METADATA_CODE, + CommandCodes.GET_METADATA_CHANGES_CODE, CommandCodes.GET_ME_CODE, CommandCodes.GET_CLIENT_CODE, CommandCodes.GET_CLIENTS_CODE,
diff --git a/foreign/csharp/Iggy_SDK_Tests/ClientTests/HttpMetadataChangesTests.cs b/foreign/csharp/Iggy_SDK_Tests/ClientTests/HttpMetadataChangesTests.cs new file mode 100644 index 0000000..91947bc --- /dev/null +++ b/foreign/csharp/Iggy_SDK_Tests/ClientTests/HttpMetadataChangesTests.cs
@@ -0,0 +1,194 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +using System.Net; +using System.Text; +using System.Text.Json; +using Apache.Iggy.Contracts; +using Apache.Iggy.Enums; +using Apache.Iggy.Exceptions; +using Apache.Iggy.IggyClient.Implementations; +using Apache.Iggy.Tests.MapperTests; +using Apache.Iggy.Vsr; + +namespace Apache.Iggy.Tests.ClientTests; + +public sealed class HttpMetadataChangesTests +{ + [Fact] + public async Task given_query_when_reading_changes_over_http_should_send_the_documented_query_and_decode_the_page() + { + var request = Vector("requests", "after_cursor_with_time_range"); + var page = Vector("responses", "changes_page").GetProperty("expected"); + var handler = new RecordingHandler(page.GetRawText()); + using var client = Client(handler); + + var answer = await client.GetMetadataChangesAsync(new MetadataChangesQuery + { + Start = MetadataChangeStart.After(MetadataChangeCursor.Parse( + request.GetProperty("input").GetProperty("cursor").GetString()!)), + Groups = [MetadataChangeGroup.Topics, MetadataChangeGroup.Streams], + Since = 1760356800000000, + Until = 1760360400000000, + Limit = 500 + }, TestContext.Current.CancellationToken); + + Assert.Equal("/metadata/changes", handler.Path); + Assert.Equal($"?{request.GetProperty("http_query").GetString()}", handler.Query); + Assert.Equal(MetadataChangeStatus.Ok, answer.Status); + Assert.Equal(page.GetProperty("next").GetString(), answer.Next.ToString()); + Assert.Equal( + new MetadataChangeKind?[] + { + MetadataChangeKind.StreamCreated, MetadataChangeKind.StreamUpdated, MetadataChangeKind.StreamDeleted + }, + answer.Records.Select(record => record.Kind)); + Assert.True(Assert.IsType<MetadataStreamChange>(answer.Records[1].Body).OptionsUnreadable); + } + + [Fact] + public async Task given_refused_query_when_reading_changes_over_http_should_not_send_it() + { + var handler = new RecordingHandler("{}"); + using var client = Client(handler); + + var refusal = await Assert.ThrowsAsync<IggyInvalidStatusCodeException>(() => + client.GetMetadataChangesAsync(new MetadataChangesQuery + { + Start = MetadataChangeStart.Current(), + Groups = [MetadataChangeGroup.Topics] + }, TestContext.Current.CancellationToken)); + + Assert.Equal(VsrError.INVALID_COMMAND, refusal.StatusCode); + Assert.Equal(0, handler.Requests); + } + + [Fact] + public async Task given_forbidden_answer_when_reading_changes_over_http_should_raise_unauthorized() + { + var handler = new RecordingHandler("""{"id":41,"code":"unauthorized","reason":"Unauthorized"}""") + { + StatusCode = HttpStatusCode.Forbidden + }; + using var client = Client(handler); + + var refusal = await Assert.ThrowsAsync<IggyInvalidStatusCodeException>(() => + client.GetMetadataChangesAsync(new MetadataChangesQuery(), TestContext.Current.CancellationToken)); + + Assert.Equal(VsrError.UNAUTHORIZED, refusal.StatusCode); + Assert.True(refusal.FromServer); + } + + [Fact] + public async Task given_server_without_the_route_when_reading_changes_over_http_should_raise_instead_of_an_empty_page() + { + var handler = new RecordingHandler("") { StatusCode = HttpStatusCode.NotFound }; + using var client = Client(handler); + + await Assert.ThrowsAsync<IggyInvalidStatusCodeException>(() => + client.GetMetadataChangesAsync(new MetadataChangesQuery(), TestContext.Current.CancellationToken)); + } + + [Fact] + public async Task given_record_of_a_kind_this_client_does_not_know_when_reading_over_http_should_keep_it_whole() + { + const string unknown = """{"op":9,"ordinal":0,"checksum":"0000000000000000000000000000000a","timestamp":1,"group":"streams","kind":"stream_archived","archive":"cold"}"""; + var handler = new RecordingHandler($$""" + {"status":"ok","phase":"changes","log_id":"0000000000000000000000000000000a","oldest_op":1, + "retained_since":0,"head_op":9,"next":null,"has_more":false,"records":[{{unknown}}],"items":[], + "current_page_token":null} + """); + using var client = Client(handler); + + var answer = await client.GetMetadataChangesAsync(new MetadataChangesQuery(), + TestContext.Current.CancellationToken); + + var record = Assert.Single(answer.Records); + Assert.Null(record.Kind); + Assert.Null(record.Group); + Assert.Equal(9UL, record.Op); + var body = Assert.IsType<MetadataUnknownJsonBody>(record.Body); + Assert.Equal("streams", body.Group); + Assert.Equal("stream_archived", body.Kind); + Assert.Equal(unknown, body.Json); + } + + [Fact] + public async Task given_answer_with_a_value_of_the_wrong_type_when_reading_over_http_should_raise_a_json_error() + { + var handler = new RecordingHandler(""" + {"status":"ok","phase":"changes","log_id":null,"oldest_op":"1","retained_since":0,"head_op":0, + "next":null,"has_more":false,"records":[],"items":[],"current_page_token":null} + """); + using var client = Client(handler); + + await Assert.ThrowsAsync<JsonException>(() => + client.GetMetadataChangesAsync(new MetadataChangesQuery(), TestContext.Current.CancellationToken)); + } + + [Theory] + [InlineData("changes", """[{"op":9,"ordinal":0,"checksum":"0000000000000000000000000000000a","timestamp":1,"group":null,"kind":"stream_created"}]""", "[]")] + [InlineData("changes", """[{"op":9,"ordinal":0,"checksum":"0000000000000000000000000000000a","timestamp":1,"group":"streams","kind":null}]""", "[]")] + [InlineData("current", "[]", """[{"group":null}]""")] + public async Task given_record_or_item_with_a_null_group_or_kind_when_reading_over_http_should_raise_a_json_error( + string phase, string records, string items) + { + var handler = new RecordingHandler($$""" + {"status":"ok","phase":"{{phase}}","log_id":null,"oldest_op":0,"retained_since":0,"head_op":0, + "next":null,"has_more":false,"records":{{records}},"items":{{items}},"current_page_token":null} + """); + using var client = Client(handler); + + await Assert.ThrowsAsync<JsonException>(() => + client.GetMetadataChangesAsync(new MetadataChangesQuery(), TestContext.Current.CancellationToken)); + } + + private static HttpMessageStream Client(RecordingHandler handler) + { + return new HttpMessageStream(new HttpClient(handler) { BaseAddress = new Uri("http://localhost") }); + } + + private static JsonElement Vector(string section, string vectorCase) + { + return MetadataChangesTests.Fixture.GetProperty(section).EnumerateArray() + .Single(entry => entry.GetProperty("case").GetString() == vectorCase); + } + + private sealed class RecordingHandler(string json) : HttpMessageHandler + { + internal HttpStatusCode StatusCode { get; init; } = HttpStatusCode.OK; + + internal int Requests { get; private set; } + + internal string Path { get; private set; } = string.Empty; + + internal string Query { get; private set; } = string.Empty; + + protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, + CancellationToken cancellationToken) + { + Requests++; + Path = request.RequestUri?.AbsolutePath ?? string.Empty; + Query = request.RequestUri?.Query ?? string.Empty; + return Task.FromResult(new HttpResponseMessage(StatusCode) + { + RequestMessage = request, + Content = new StringContent(json, Encoding.UTF8, "application/json") + }); + } + } +}
diff --git a/foreign/csharp/Iggy_SDK_Tests/Fixtures/metadata_changes.json b/foreign/csharp/Iggy_SDK_Tests/Fixtures/metadata_changes.json new file mode 100644 index 0000000..26e7faf --- /dev/null +++ b/foreign/csharp/Iggy_SDK_Tests/Fixtures/metadata_changes.json
@@ -0,0 +1,1997 @@ +{ + "schema_version": 1, + "generated_by": "core/common/examples/metadata_changes_fixtures.rs", + "conventions": { + "byte_strings": "Every *_hex field is lowercase hex without separators.", + "expected": "expected is the JSON the HTTP route renders. Binary decoders must reach the same values.", + "record_hex": "record_hex is one record envelope as a response carries it. body_hex is the body alone.", + "refused": "Every refused vector must fail to decode or validate. Query and HTTP refusals are invalid_command.", + "request_hex": "request_hex is the command payload without framing. http_query is the query string of GET /metadata/changes.", + "u128_values": "128-bit values (log ids, checksums, client ids) are 32 lowercase hex characters in JSON.", + "u64_values": "Ops and timestamps are JSON numbers below 2^53 in every vector except large_values, which carries them as decimal strings.", + "unknown_kinds": "A record of an unknown group or kind is kept with its codes and raw body, never refused." + }, + "constants": { + "command_code": 17, + "command_name": "metadata.changes", + "cursor_prefix": "mc1_", + "page_token_prefix": "cs1_", + "cursor_size": 44, + "cursor_text_length": 63, + "max_page_token_size": 316, + "max_page_token_key_size": 260, + "default_limit": 100, + "max_limit": 1000, + "groups": { + "consumer_group_members": { + "code": 5, + "class": "activity", + "parents": [ + "consumer_groups", + "partitions", + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32][member_id:u32]" + }, + "consumer_groups": { + "code": 4, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32]" + }, + "partitions": { + "code": 3, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "personal_access_tokens": { + "code": 7, + "class": "resource", + "parents": [ + "users" + ], + "page_token_key": "[user_id:u32][name_len:u8][name]" + }, + "sessions": { + "code": 8, + "class": "activity", + "parents": [ + "users" + ], + "page_token_key": "[epoch:u64]" + }, + "streams": { + "code": 1, + "class": "resource", + "parents": [], + "page_token_key": "[stream_id:u32]" + }, + "topics": { + "code": 2, + "class": "resource", + "parents": [ + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "users": { + "code": 6, + "class": "resource", + "parents": [], + "page_token_key": "[user_id:u32]" + } + }, + "kinds": { + "consumer_group_created": { + "code": 30, + "group": "consumer_groups" + }, + "consumer_group_deleted": { + "code": 31, + "group": "consumer_groups" + }, + "consumer_group_member_joined": { + "code": 40, + "group": "consumer_group_members" + }, + "consumer_group_member_left": { + "code": 41, + "group": "consumer_group_members" + }, + "consumer_group_partition_handed_over": { + "code": 42, + "group": "consumer_group_members" + }, + "partitions_changed": { + "code": 20, + "group": "partitions" + }, + "personal_access_token_created": { + "code": 60, + "group": "personal_access_tokens" + }, + "personal_access_token_deleted": { + "code": 61, + "group": "personal_access_tokens" + }, + "session_closed": { + "code": 71, + "group": "sessions" + }, + "session_registered": { + "code": 70, + "group": "sessions" + }, + "stream_created": { + "code": 1, + "group": "streams" + }, + "stream_deleted": { + "code": 3, + "group": "streams" + }, + "stream_updated": { + "code": 2, + "group": "streams" + }, + "topic_created": { + "code": 10, + "group": "topics" + }, + "topic_deleted": { + "code": 12, + "group": "topics" + }, + "topic_updated": { + "code": 11, + "group": "topics" + }, + "user_created": { + "code": 50, + "group": "users" + }, + "user_deleted": { + "code": 52, + "group": "users" + }, + "user_password_changed": { + "code": 53, + "group": "users" + }, + "user_permissions_updated": { + "code": 54, + "group": "users" + }, + "user_updated": { + "code": 51, + "group": "users" + } + }, + "statuses": { + "behind": 1, + "expired": 2, + "ok": 0, + "reset": 3 + }, + "phases": { + "changes": 0, + "current": 1 + }, + "start_modes": { + "after": 2, + "current": 3, + "earliest": 0, + "latest": 1 + }, + "member_left_reasons": { + "left": 0, + "session_ended": 1 + }, + "token_deleted_reasons": { + "deleted": 0, + "expired": 1 + }, + "session_closed_reasons": { + "expired": 1, + "logout": 0, + "revoked": 2 + }, + "permissions_updated_reasons": { + "resource_deleted": 1, + "updated": 0 + } + }, + "records": [ + { + "case": "stream_created", + "group": 1, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "record_hex": "191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "stream_updated_unreadable_options", + "group": 1, + "kind": 2, + "body_hex": "03000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "record_hex": "1a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "expected": { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + } + }, + { + "case": "stream_deleted_deferred", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "record_hex": "1b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "expected": { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + }, + { + "case": "topic_created", + "group": 2, + "kind": 10, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "record_hex": "1c1000000000000000000000683d9e8b03f7c1a500000000000000001c00be01094106000104000000020a78000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "op": 4124, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d68", + "timestamp": 1760356800004124, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "partitions_count": 8, + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "topic_updated", + "group": 2, + "kind": 11, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "record_hex": "1d1000000000000000000000693d9e8b03f7c1a500000000000000001d00be01094106000104000000020b3d000000030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "expected": { + "op": 4125, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d69", + "timestamp": 1760356800004125, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_updated", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics-v2", + "partitions_count": 8, + "options": {} + } + }, + { + "case": "topic_deleted_immediate", + "group": 2, + "kind": 12, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "record_hex": "1e10000000000000000000006a3d9e8b03f7c1a500000000000000001e00be01094106000104000000020c2e000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "expected": { + "op": 4126, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6a", + "timestamp": 1760356800004126, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics" + } + }, + { + "case": "partitions_changed_deleted_tail", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "record_hex": "1f10000000000000000000006b3d9e8b03f7c1a500000000000000001f00be01094106000104000000031435000000030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "expected": { + "op": 4127, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6b", + "timestamp": 1760356800004127, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 5, + "removed_partition_ids": [ + 5, + 6, + 7 + ], + "requested_op": 4200 + } + }, + { + "case": "partitions_changed_created", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000a0000000000000000", + "record_hex": "2010000000000000000000006c3d9e8b03f7c1a500000000000000002000be01094106000104000000031421000000030000001910000000000000000000001d100000000000000a0000000000000000", + "expected": { + "op": 4128, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6c", + "timestamp": 1760356800004128, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 10, + "removed_partition_ids": [] + } + }, + { + "case": "consumer_group_created", + "group": 4, + "kind": 30, + "body_hex": "030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "record_hex": "2110000000000000000000006d3d9e8b03f7c1a500000000000000002100be01094106000104000000041e28000000030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "expected": { + "op": 4129, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6d", + "timestamp": 1760356800004129, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 1 + } + }, + { + "case": "consumer_group_deleted", + "group": 4, + "kind": 31, + "body_hex": "030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "record_hex": "2210000000000000000000006e3d9e8b03f7c1a500000000000000002200be01094106000104000000041f31000000030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "expected": { + "op": 4130, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6e", + "timestamp": 1760356800004130, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 0, + "requested_op": 4300 + } + }, + { + "case": "consumer_group_member_joined", + "group": 5, + "kind": 40, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "record_hex": "2310000000000000000000006f3d9e8b03f7c1a500000000000000002300be01094106000109000000052838000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "expected": { + "op": 4131, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6f", + "timestamp": 1760356800004131, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_joined", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 2, + "partition_ids": [] + } + }, + { + "case": "consumer_group_member_left_session_ended", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "record_hex": "241000000000000000000000703d9e8b03f7c1a500000000000000002400be01094106000109000000052935000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "expected": { + "op": 4132, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d70", + "timestamp": 1760356800004132, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_left", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 1, + "reason": "session_ended" + } + }, + { + "case": "consumer_group_partition_handed_over", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d1000000000000000000000020000000101000000", + "record_hex": "251000000000000000000000713d9e8b03f7c1a500000000000000002500be010941060000052a25000000030000001910000000000000000000001d1000000000000000000000020000000101000000", + "expected": { + "op": 4133, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d71", + "timestamp": 1760356800004133, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 2, + "member_id": 1 + } + }, + { + "case": "consumer_group_partition_released", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d10000000000000000000000300000000", + "record_hex": "261000000000000000000000723d9e8b03f7c1a500000000000000002600be010941060000052a21000000030000001910000000000000000000001d10000000000000000000000300000000", + "expected": { + "op": 4134, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d72", + "timestamp": 1760356800004134, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 3 + } + }, + { + "case": "user_created_with_permissions", + "group": 6, + "kind": 50, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "record_hex": "271000000000000000000000733d9e8b03f7c1a500000000000000002700be010941060001000000000632480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "op": 4135, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d73", + "timestamp": 1760356800004135, + "actor_user_id": 0, + "group": "users", + "kind": "user_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "user_updated_inactive", + "group": 6, + "kind": 51, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "record_hex": "281000000000000000000000743d9e8b03f7c1a500000000000000002800be010941060001000000000633280000000700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "expected": { + "op": 4136, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d74", + "timestamp": 1760356800004136, + "actor_user_id": 0, + "group": "users", + "kind": "user_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "inactive", + "permissions": null, + "options": {} + } + }, + { + "case": "user_deleted", + "group": 6, + "kind": 52, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "291000000000000000000000753d9e8b03f7c1a500000000000000002900be0109410600010000000006341d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4137, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d75", + "timestamp": 1760356800004137, + "actor_user_id": 0, + "group": "users", + "kind": "user_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_password_changed", + "group": 6, + "kind": 53, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "2a1000000000000000000000763d9e8b03f7c1a500000000000000002a00be0109410600010700000006351d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4138, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d76", + "timestamp": 1760356800004138, + "actor_user_id": 7, + "group": "users", + "kind": "user_password_changed", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_permissions_pruned", + "group": 6, + "kind": 54, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "record_hex": "2b1000000000000000000000773d9e8b03f7c1a500000000000000002b00be0109410600010400000006362a0000000700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "expected": { + "op": 4139, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d77", + "timestamp": 1760356800004139, + "actor_user_id": 4, + "group": "users", + "kind": "user_permissions_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": false, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": null + }, + "reason": "resource_deleted" + } + }, + { + "case": "personal_access_token_created", + "group": 7, + "kind": 60, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "record_hex": "2c1000000000000000000000783d9e8b03f7c1a500000000000000002c00be01094106000107000000073c1d0000000700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "expected": { + "op": 4140, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d78", + "timestamp": 1760356800004140, + "actor_user_id": 7, + "group": "personal_access_tokens", + "kind": "personal_access_token_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "expiry": 1760400000000000 + } + }, + { + "case": "personal_access_token_deleted_expired", + "group": 7, + "kind": 61, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e01", + "record_hex": "2d1000000000000000000000793d9e8b03f7c1a500000000000000002d00be010941060000073d160000000700000000b832c8fb4006000863692d746f6b656e01", + "expected": { + "op": 4141, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d79", + "timestamp": 1760356800004141, + "group": "personal_access_tokens", + "kind": "personal_access_token_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "reason": "expired" + } + }, + { + "case": "session_registered", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "record_hex": "2e10000000000000000000007a3d9e8b03f7c1a500000000000000002e00be010941060001070000000846240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "op": 4142, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7a", + "timestamp": 1760356800004142, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_registered", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + }, + { + "case": "session_closed_expired", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "record_hex": "2f10000000000000000000007b3d9e8b03f7c1a500000000000000002f00be0109410600000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "expected": { + "op": 4143, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7b", + "timestamp": 1760356800004143, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "expired" + } + }, + { + "case": "session_closed_revoked", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "record_hex": "3010000000000000000000007c3d9e8b03f7c1a500000000000000003000be010941060001070000000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "expected": { + "op": 4144, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7c", + "timestamp": 1760356800004144, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "revoked" + } + } + ], + "items": [ + { + "case": "stream", + "group": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "item_hex": "01b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "topic", + "group": 2, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "item_hex": "0274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "partitions", + "group": 3, + "body_hex": "030000001910000000000000000000001d1000000000000008000000", + "item_hex": "031c000000030000001910000000000000000000001d1000000000000008000000", + "expected": { + "group": "partitions", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 8 + } + }, + { + "case": "consumer_group", + "group": 4, + "body_hex": "030000001910000000000000000000001d10000000000000000000000772656164657273", + "item_hex": "0424000000030000001910000000000000000000001d10000000000000000000000772656164657273", + "expected": { + "group": "consumer_groups", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers" + } + }, + { + "case": "consumer_group_member", + "group": 5, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "item_hex": "0540000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "expected": { + "group": "consumer_group_members", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "partition_ids": [ + 0, + 2, + 4 + ] + } + }, + { + "case": "user", + "group": 6, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "item_hex": "06480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "group": "users", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "personal_access_token_without_expiry", + "group": 7, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "item_hex": "071d0000000700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "expected": { + "group": "personal_access_tokens", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token" + } + }, + { + "case": "session", + "group": 8, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "item_hex": "08240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "group": "sessions", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + } + ], + "cursors": [ + { + "case": "zero", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d000000000000000000000000000000000000000000000000000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 0, + "checksum": "00000000000000000000000000000000", + "ordinal": 0 + }, + { + "case": "first_record_of_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAABgQAAAAAAAApcH3A4ueLUwAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000001810000000000000a5c1f7038b9e2d4c000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4120, + "checksum": "00000000000000004c2d9e8b03f7c1a5", + "ordinal": 0 + }, + { + "case": "inside_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f2000000000000000001000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4187, + "checksum": "0000000000000000f2a4c6e8b0d1f359", + "ordinal": 1 + }, + { + "case": "every_field_at_maximum", + "text": "mc1___________________________________________________________8", + "hex": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + "log_id": "ffffffffffffffffffffffffffffffff", + "op": 18446744073709551615, + "checksum": "ffffffffffffffffffffffffffffffff", + "ordinal": 4294967295 + } + ], + "page_tokens": [ + { + "case": "streams", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000000101040003000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 1, + "group": "streams", + "last_key_hex": "03000000" + }, + { + "case": "topics", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 3, + "group": "topics", + "last_key_hex": "0300000000000000" + }, + { + "case": "partitions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8DCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff0308000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "partitions", + "last_key_hex": "0300000000000000" + }, + { + "case": "consumer_groups", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8EDAADAAAAAAAAAAIAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff040c00030000000000000002000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_groups", + "last_key_hex": "030000000000000002000000" + }, + { + "case": "consumer_group_members", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8FEAADAAAAAAAAAAIAAAABAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff05100003000000000000000200000001000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_group_members", + "last_key_hex": "03000000000000000200000001000000" + }, + { + "case": "users", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAACAGBAAHAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000002006040007000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 32, + "group": "users", + "last_key_hex": "07000000" + }, + { + "case": "personal_access_tokens", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAGAHDQAHAAAACGNpLXRva2Vu", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e1000000000000060070d00070000000863692d746f6b656e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 96, + "group": "personal_access_tokens", + "last_key_hex": "070000000863692d746f6b656e" + }, + { + "case": "sessions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAKAICACUEQAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000a00808009411000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 160, + "group": "sessions", + "last_key_hex": "9411000000000000" + }, + { + "case": "longest_token_name", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8HBAH______25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubg", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff070401ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "personal_access_tokens", + "last_key_hex": "ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e" + } + ], + "requests": [ + { + "case": "earliest_every_group", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000", + "http_query": "from=earliest&limit=100" + }, + { + "case": "earliest_many_duplicate_groups", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "users" + ], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000640000000201060000", + "http_query": "from=earliest&groups=streams,users&limit=100" + }, + { + "case": "earliest_since_users_and_tokens", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "users", + "personal_access_tokens" + ], + "since": 1760353200000000, + "until": null, + "limit": 100 + }, + "request_hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001004c2a2b0841060000640000000206070000", + "http_query": "from=earliest&groups=users,personal_access_tokens&since=1760353200000000&limit=100" + }, + { + "case": "latest_streams", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [ + "streams" + ], + "since": null, + "until": null, + "limit": 1 + }, + "request_hex": "01000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000001010000", + "http_query": "from=latest&groups=streams&limit=1" + }, + { + "case": "after_cursor_with_time_range", + "input": { + "start": "after", + "cursor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "page_token": null, + "groups": [ + "topics", + "streams" + ], + "since": 1760356800000000, + "until": 1760360400000000, + "limit": 500 + }, + "request_hex": "02d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000000100f0bd010941060001009451d809410600f40100000202010000", + "http_query": "after=mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA&groups=streams,topics&since=1760356800000000&until=1760360400000000&limit=500" + }, + { + "case": "current_first_page", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201020000", + "http_query": "from=current&groups=streams,topics&limit=1000" + }, + { + "case": "current_continuation", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201024000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "http_query": "from=current¤t_after=cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA&groups=streams,topics&limit=1000" + } + ], + "responses": [ + { + "case": "changes_page", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001b10000000000000673d9e8b03f7c1a50000000000000000000000000103000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465641a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d7632011b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABsQAAAAAAAAZz2eiwP3waUAAAAAAAAAAAAAAAA", + "has_more": true, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + }, + { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "unknown_kind_and_group", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001910000000000000653d9e8b03f7c1a50000000000000000010000000002000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be01094106000104000000010904000000deadbeef191000000000000001000000653d9e8b03f7c1a500000000000000001900be01094106000104000000c80104000000deadbeef000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABkQAAAAAAAAZT2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": false, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "1", + "kind": "9", + "body": "3q2+7w==" + }, + { + "op": 4121, + "ordinal": 1, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "200", + "kind": "1", + "body": "3q2+7w==" + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "current_page", + "response_hex": "000101d7931c6e2a8f4b0d0000000000000000010000000000000000000000000000005e1000000000000001d7931c6e2a8f4b0d00000000000000005b10000000000000a73d9e8b03f7c1a500000000000000000100000001000000000200000001b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465640274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c0800000000000040000000004000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "expected": { + "status": "ok", + "phase": "current", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 1, + "retained_since": 0, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAApz2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": true, + "records": [], + "items": [ + { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + ], + "current_page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA" + } + }, + { + "case": "behind_before_op_1", + "response_hex": "010000000000000000000000000000000000000000000000000000000000000000000000000000", + "expected": { + "status": "behind", + "phase": "changes", + "log_id": null, + "oldest_op": 0, + "retained_since": 0, + "head_op": 0, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "expired", + "response_hex": "020001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "expired", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "reset", + "response_hex": "030001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "reset", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + } + ], + "large_values": { + "record_hex": "0300000000002000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff01ffffffff084624000000ffffffff0100000000002000feffffffffffffffffffffffffffffff0200000000002000", + "op": "9007199254740995", + "ordinal": 4294967295, + "timestamp": "18446744073709551615", + "checksum": "ffffffffffffffffffffffffffffffff", + "user_created_at": "9007199254740993", + "session": "9007199254740994", + "client_id": "fffffffffffffffffffffffffffffffe" + }, + "refused": { + "cursors": [ + { + "case": "wrong_prefix", + "text": "mc2_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "missing_prefix", + "text": "15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "too_long", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAAA" + }, + { + "case": "too_short", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA" + }, + { + "case": "padded", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA=" + }, + { + "case": "standard_alphabet_character", + "text": "mc1_+5McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "op_zero_with_checksum", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAA" + } + ], + "page_tokens": [ + { + "case": "wrong_prefix", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA" + }, + { + "case": "trailing_byte", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAAAA" + }, + { + "case": "wrong_key_width", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBCAADAAAAAAAAAA" + }, + { + "case": "group_outside_mask", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAIBBAADAAAA" + }, + { + "case": "head_below_anchor", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAABAAAAAAAAAAEBBAADAAAA" + }, + { + "case": "empty", + "text": "cs1_" + } + ], + "record_bodies": [ + { + "case": "unknown_member_left_reason", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000002" + }, + { + "case": "presence_byte_two", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279022c10000000000000" + }, + { + "case": "unsorted_removed_partition_ids", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000090000000600000007000000016810000000000000" + }, + { + "case": "known_kind_under_wrong_group", + "group": 2, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564" + }, + { + "case": "trailing_byte", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000000" + }, + { + "case": "truncated", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a09080706050403020194110000000000" + } + ], + "requests": [ + { + "case": "cursor_bytes_without_after", + "hex": "000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "trailing_byte", + "hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006400000000000000" + }, + { + "case": "unknown_start_mode", + "hex": "040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "non_canonical_cursor", + "hex": "02d7931c6e2a8f4b0d000000000000000000000000000000000100000000000000000000000000000000000000000064000000000000" + } + ], + "responses": [ + { + "case": "items_in_changes_phase", + "hex": "000001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + }, + { + "case": "unknown_status", + "hex": "040001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + } + ], + "queries": [ + { + "case": "limit_zero", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 0 + } + }, + { + "case": "limit_above_maximum", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 1001 + } + }, + { + "case": "since_above_until", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 2, + "until": 1, + "limit": 100 + } + }, + { + "case": "time_bound_with_latest", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 1, + "until": null, + "limit": 100 + } + }, + { + "case": "time_bound_with_current", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": 1, + "limit": 100 + } + }, + { + "case": "current_missing_parent_group", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + }, + { + "case": "page_token_for_other_groups", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + } + ], + "http_queries": [ + { + "case": "no_start", + "query": "groups=streams" + }, + { + "case": "from_and_after", + "query": "from=earliest&after=mc1_x" + }, + { + "case": "unknown_from", + "query": "from=newest" + }, + { + "case": "duplicate_key", + "query": "from=earliest&from=latest" + }, + { + "case": "unknown_key", + "query": "from=earliest&verbose=1" + }, + { + "case": "empty_group_element", + "query": "from=earliest&groups=streams,,topics" + }, + { + "case": "trailing_group_comma", + "query": "from=earliest&groups=streams," + }, + { + "case": "unknown_group", + "query": "from=earliest&groups=everything" + }, + { + "case": "negative_since", + "query": "from=earliest&since=-1" + }, + { + "case": "leading_zero_until", + "query": "from=earliest&until=01" + }, + { + "case": "non_numeric_limit", + "query": "from=earliest&limit=ten" + }, + { + "case": "token_without_current", + "query": "from=latest¤t_after=cs1_AAAA" + } + ] + } +}
diff --git a/foreign/csharp/Iggy_SDK_Tests/Iggy_SDK_Tests.csproj b/foreign/csharp/Iggy_SDK_Tests/Iggy_SDK_Tests.csproj index f2ca489..6b93152 100644 --- a/foreign/csharp/Iggy_SDK_Tests/Iggy_SDK_Tests.csproj +++ b/foreign/csharp/Iggy_SDK_Tests/Iggy_SDK_Tests.csproj
@@ -52,6 +52,9 @@ <None Include="Fixtures/resource_options.json"> <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory> </None> + <None Include="Fixtures/metadata_changes.json"> + <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory> + </None> </ItemGroup> </Project>
diff --git a/foreign/csharp/Iggy_SDK_Tests/MapperTests/BinaryMapper.cs b/foreign/csharp/Iggy_SDK_Tests/MapperTests/BinaryMapper.cs index 68a0977..d4020ca 100644 --- a/foreign/csharp/Iggy_SDK_Tests/MapperTests/BinaryMapper.cs +++ b/foreign/csharp/Iggy_SDK_Tests/MapperTests/BinaryMapper.cs
@@ -941,6 +941,21 @@ Assert.Throws<MalformedResponseException>(() => Mappers.BinaryMapper.MapClient(payload)); } + [Fact] + public void MapUser_PermissionsOverrunningTheirLength_Throws() + { + var payload = new byte[39]; + BinaryPrimitives.WriteUInt32LittleEndian(payload.AsSpan(0, 4), 1); // user id + payload[12] = 1; // status active + payload[13] = 1; // username length + payload[14] = (byte)'u'; + payload[23] = 1; // permissions present + BinaryPrimitives.WriteUInt32LittleEndian(payload.AsSpan(24, 4), 11); // permissions length + payload[38] = 1; // a stream follows the ten global flags, past the length + + Assert.Throws<MalformedResponseException>(() => Mappers.BinaryMapper.MapUser(payload)); + } + private static byte[] CreateClientPayload(uint userId, byte transport) { var payload = new byte[17];
diff --git a/foreign/csharp/Iggy_SDK_Tests/MapperTests/MetadataChangesTests.cs b/foreign/csharp/Iggy_SDK_Tests/MapperTests/MetadataChangesTests.cs new file mode 100644 index 0000000..5c67995 --- /dev/null +++ b/foreign/csharp/Iggy_SDK_Tests/MapperTests/MetadataChangesTests.cs
@@ -0,0 +1,844 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +using System.Globalization; +using System.Text.Json; +using System.Text.Json.Nodes; +using Apache.Iggy.Contracts; +using Apache.Iggy.Contracts.Tcp; +using Apache.Iggy.Enums; +using Apache.Iggy.Exceptions; +using Apache.Iggy.Headers; +using Apache.Iggy.JsonConverters; +using Apache.Iggy.Utils; +using Apache.Iggy.Vsr; + +namespace Apache.Iggy.Tests.MapperTests; + +/// <summary> +/// The cross-SDK metadata changes contract. Every expectation comes from the shared fixture the Rust encoders +/// generate, and both the binary decoder and the HTTP JSON decoder must reach the JSON the server renders. +/// </summary> +public sealed class MetadataChangesTests +{ + [Fact] + public void given_many_duplicate_groups_when_encoding_should_preserve_every_requested_group() + { + var query = new MetadataChangesQuery + { + Groups = Enumerable.Repeat(MetadataChangeGroup.Streams, 255).Append(MetadataChangeGroup.Users).ToArray() + }; + var expected = new MetadataChangesQuery { Groups = [MetadataChangeGroup.Streams, MetadataChangeGroup.Users] }; + Assert.Equal(TcpContracts.GetMetadataChanges(expected), TcpContracts.GetMetadataChanges(query)); + } + + internal static readonly JsonElement Fixture = LoadFixture(); + + private static readonly JsonSerializerOptions PermissionsJson = new() + { + PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower + }; + + public static TheoryData<string, string> SharedVectors + { + get + { + var vectors = new TheoryData<string, string> + { + { "constants", "constants" }, + { "large_values", "large_values" }, + { "refused_requests", "non_canonical_cursor" } + }; + foreach (var section in new[] { "records", "items", "cursors", "page_tokens", "requests", "responses" }) + { + AddCases(vectors, Fixture.GetProperty(section), section); + } + + foreach (var section in new[] { "cursors", "page_tokens", "record_bodies", "responses", "queries" }) + { + AddCases(vectors, Fixture.GetProperty("refused").GetProperty(section), $"refused_{section}"); + } + + return vectors; + } + } + + public static TheoryData<string> RecordCases => Cases(Fixture.GetProperty("records")); + + public static TheoryData<string> ItemCases => Cases(Fixture.GetProperty("items")); + + [Theory] + [MemberData(nameof(SharedVectors))] + public void given_shared_vectors_when_decoding_should_match_server_contract(string section, string vectorCase) + { + switch (section) + { + case "constants": + AssertConstants(Fixture.GetProperty("constants")); + break; + case "records": + AssertRecord(Find(Fixture.GetProperty(section), vectorCase)); + break; + case "items": + AssertItem(Find(Fixture.GetProperty(section), vectorCase)); + break; + case "cursors": + AssertCursor(Find(Fixture.GetProperty(section), vectorCase)); + break; + case "page_tokens": + AssertPageToken(Find(Fixture.GetProperty(section), vectorCase)); + break; + case "requests": + AssertRequest(Find(Fixture.GetProperty(section), vectorCase)); + break; + case "responses": + AssertResponse(Find(Fixture.GetProperty(section), vectorCase)); + break; + case "large_values": + AssertLargeValues(Fixture.GetProperty(section)); + break; + case "refused_cursors": + var cursorText = Refused("cursors", vectorCase).GetProperty("text").GetString(); + Assert.False(MetadataChangeCursor.TryParse(cursorText, out _)); + Assert.Throws<FormatException>(() => MetadataChangeCursor.Parse(cursorText!)); + break; + case "refused_page_tokens": + var tokenText = Refused("page_tokens", vectorCase).GetProperty("text").GetString(); + Assert.False(MetadataCurrentPageToken.TryParse(tokenText, out _)); + Assert.Throws<FormatException>(() => MetadataCurrentPageToken.Parse(tokenText!)); + break; + case "refused_record_bodies": + var body = Refused("record_bodies", vectorCase); + Assert.Throws<MalformedResponseException>(() => Mappers.BinaryMapper.MapMetadataChangeBody( + (byte)body.GetProperty("group").GetInt32(), (byte)body.GetProperty("kind").GetInt32(), + Hex(body, "body_hex"), out _)); + break; + case "refused_responses": + var response = Hex(Refused("responses", vectorCase), "hex"); + Assert.Throws<MalformedResponseException>(() => Mappers.BinaryMapper.MapMetadataChanges(response)); + break; + case "refused_queries": + AssertRefused(Query(Refused("queries", vectorCase).GetProperty("input"))); + break; + case "refused_requests": + // The SDK cannot express a cursor without after, an unknown start or trailing bytes. A + // non-canonical cursor it can, and validation refuses it before anything is sent. + var request = Hex(Refused("requests", vectorCase), "hex"); + AssertRefused(new MetadataChangesQuery + { + Start = MetadataChangeStart.After(MetadataChangeCursor.Read(request.AsSpan(1))) + }); + break; + default: + Assert.Fail($"Unknown fixture section {section}."); + break; + } + } + + [Theory] + [MemberData(nameof(RecordCases))] + public void given_record_body_when_truncating_it_should_refuse_every_prefix(string vectorCase) + { + var record = Find(Fixture.GetProperty("records"), vectorCase); + var group = (byte)record.GetProperty("group").GetInt32(); + var kind = (byte)record.GetProperty("kind").GetInt32(); + var body = Hex(record, "body_hex"); + for (var length = 0; length < body.Length; length++) + { + var prefix = body[..length]; + Assert.Throws<MalformedResponseException>(() => + Mappers.BinaryMapper.MapMetadataChangeBody(group, kind, prefix, out _)); + } + + Assert.Throws<MalformedResponseException>(() => + Mappers.BinaryMapper.MapMetadataChangeBody(group, kind, [.. body, 0], out _)); + } + + [Theory] + [MemberData(nameof(ItemCases))] + public void given_item_body_when_truncating_it_should_refuse_every_prefix(string vectorCase) + { + var item = Find(Fixture.GetProperty("items"), vectorCase); + var group = (byte)item.GetProperty("group").GetInt32(); + var body = Hex(item, "body_hex"); + for (var length = 0; length < body.Length; length++) + { + var prefix = body[..length]; + Assert.Throws<MalformedResponseException>(() => Mappers.BinaryMapper.MapMetadataItemBody(group, prefix, out _)); + } + + Assert.Throws<MalformedResponseException>(() => Mappers.BinaryMapper.MapMetadataItemBody(group, [.. body, 0], out _)); + } + + [Fact] + public void given_unknown_codes_when_decoding_a_body_should_keep_the_codes_and_raw_bytes() + { + byte[] raw = [0xde, 0xad, 0xbe, 0xef]; + + var change = Mappers.BinaryMapper.MapMetadataChangeBody(9, 1, raw, out var kind); + var item = Mappers.BinaryMapper.MapMetadataItemBody(128, raw, out var group); + + Assert.Null(kind); + Assert.Null(group); + var unknownChange = Assert.IsType<MetadataUnknownBody>(change); + Assert.Equal((byte)9, unknownChange.Group); + Assert.Equal((byte)1, unknownChange.Kind); + Assert.Equal(raw, unknownChange.Body); + var unknownItem = Assert.IsType<MetadataUnknownBody>(item); + Assert.Equal((byte)128, unknownItem.Group); + Assert.Null(unknownItem.Kind); + Assert.Equal(raw, unknownItem.Body); + } + + [Fact] + public void given_cursor_text_with_nonzero_trailing_bits_when_parsing_should_refuse() + { + var maximum = Find(Fixture.GetProperty("cursors"), "every_field_at_maximum").GetProperty("text").GetString()!; + Assert.True(MetadataChangeCursor.TryParse(maximum, out _)); + + Assert.False(MetadataChangeCursor.TryParse(maximum[..^1] + "9", out _)); + Assert.False(MetadataChangeCursor.TryParse(maximum[..^1] + " ", out _)); + Assert.False(MetadataChangeCursor.TryParse(null, out _)); + } + + [Fact] + public void given_page_token_text_above_the_longest_token_when_parsing_should_refuse() + { + var longest = Find(Fixture.GetProperty("page_tokens"), "longest_token_name").GetProperty("text").GetString()!; + Assert.True(MetadataCurrentPageToken.TryParse(longest, out var token)); + Assert.Equal(MetadataCurrentPageToken.MAX_SIZE, token.Payload.Length); + + Assert.False(MetadataCurrentPageToken.TryParse(longest + "AAAA", out _)); + } + + [Fact] + public void given_queries_the_rules_allow_when_validating_should_accept_them() + { + var topicsToken = MetadataCurrentPageToken.Parse( + Find(Fixture.GetProperty("page_tokens"), "topics").GetProperty("text").GetString()!); + MetadataChangesQuery[] accepted = + [ + new(), + new() { Start = MetadataChangeStart.Latest, Limit = 1 }, + new() { Since = 5, Until = 5, Limit = MetadataChangesQuery.MAX_LIMIT }, + new() { Start = MetadataChangeStart.Current(), Groups = [MetadataChangeGroup.Sessions, MetadataChangeGroup.Users] }, + new() + { + Start = MetadataChangeStart.Current(topicsToken), + Groups = [MetadataChangeGroup.Topics, MetadataChangeGroup.Streams, MetadataChangeGroup.Topics] + }, + new() { Start = MetadataChangeStart.After(new MetadataChangeCursor(7UL, 0, UInt128.Zero, 0)), Groups = [MetadataChangeGroup.Topics] } + ]; + + foreach (var query in accepted) + { + query.Validate(); + } + } + + [Fact] + public void given_group_this_client_does_not_know_when_validating_should_refuse() + { + AssertRefused(new MetadataChangesQuery { Groups = [(MetadataChangeGroup)9] }); + AssertRefused(new MetadataChangesQuery { Groups = [(MetadataChangeGroup)0] }); + } + + [Fact] + public void given_groups_in_any_order_when_encoding_should_keep_binary_order_and_sort_the_http_names() + { + var query = new MetadataChangesQuery + { + Groups = [MetadataChangeGroup.Users, MetadataChangeGroup.Streams, MetadataChangeGroup.Users] + }; + + var request = TcpContracts.GetMetadataChanges(query); + + Assert.Equal(new byte[] { 3, 6, 1, 6, 0, 0 }, request[^6..]); + Assert.Equal("from=earliest&groups=streams,users&limit=100", query.ToHttpQuery()); + } + + private static void AssertConstants(JsonElement constants) + { + Assert.Equal(CommandCodes.GET_METADATA_CHANGES_CODE, constants.GetProperty("command_code").GetInt32()); + Assert.Equal(MetadataChangeCursor.PREFIX, constants.GetProperty("cursor_prefix").GetString()); + Assert.Equal(MetadataCurrentPageToken.PREFIX, constants.GetProperty("page_token_prefix").GetString()); + Assert.Equal(MetadataChangeCursor.SIZE, constants.GetProperty("cursor_size").GetInt32()); + Assert.Equal(constants.GetProperty("cursor_text_length").GetInt32(), default(MetadataChangeCursor).ToString().Length); + Assert.Equal(MetadataCurrentPageToken.MAX_SIZE, constants.GetProperty("max_page_token_size").GetInt32()); + Assert.Equal(MetadataCurrentPageToken.MAX_KEY_SIZE, constants.GetProperty("max_page_token_key_size").GetInt32()); + Assert.Equal(MetadataChangesQuery.DEFAULT_LIMIT, constants.GetProperty("default_limit").GetUInt32()); + Assert.Equal(MetadataChangesQuery.MAX_LIMIT, constants.GetProperty("max_limit").GetUInt32()); + Assert.Equal(MetadataChangesQuery.DEFAULT_LIMIT, new MetadataChangesQuery().Limit); + Assert.Equal(MetadataChangeStartMode.Earliest, new MetadataChangesQuery().Start.Mode); + + var groups = constants.GetProperty("groups").EnumerateObject().ToList(); + Assert.Equal(Enum.GetValues<MetadataChangeGroup>().Length, groups.Count); + foreach (var entry in groups) + { + Assert.True(SnakeCaseNames<MetadataChangeGroup>.TryParse(entry.Name, out var group), entry.Name); + Assert.Equal(entry.Value.GetProperty("code").GetInt32(), (int)group); + Assert.Equal(entry.Value.GetProperty("parents").EnumerateArray().Select(parent => parent.GetString()!), + MetadataChangeGroups.Parents(group).Select(SnakeCaseNames<MetadataChangeGroup>.Name)); + } + + var kinds = constants.GetProperty("kinds").EnumerateObject().ToList(); + Assert.Equal(Enum.GetValues<MetadataChangeKind>().Length, kinds.Count); + foreach (var entry in kinds) + { + Assert.True(SnakeCaseNames<MetadataChangeKind>.TryParse(entry.Name, out var kind), entry.Name); + Assert.Equal(entry.Value.GetProperty("code").GetInt32(), (int)kind); + Assert.Equal(entry.Value.GetProperty("group").GetString(), + SnakeCaseNames<MetadataChangeGroup>.Name(MetadataChangeGroups.Of(kind))); + } + + AssertCodes<MetadataChangeStatus>(constants.GetProperty("statuses")); + AssertCodes<MetadataChangePhase>(constants.GetProperty("phases")); + AssertCodes<MetadataChangeStartMode>(constants.GetProperty("start_modes")); + AssertCodes<MetadataMemberLeftReason>(constants.GetProperty("member_left_reasons")); + AssertCodes<MetadataTokenDeletedReason>(constants.GetProperty("token_deleted_reasons")); + AssertCodes<MetadataSessionClosedReason>(constants.GetProperty("session_closed_reasons")); + AssertCodes<MetadataPermissionsUpdatedReason>(constants.GetProperty("permissions_updated_reasons")); + } + + private static void AssertCodes<T>(JsonElement codes) where T : struct, Enum + { + var expected = codes.EnumerateObject() + .ToDictionary(entry => entry.Name, entry => entry.Value.GetInt32()); + var actual = Enum.GetValues<T>() + .ToDictionary(value => SnakeCase(value), value => Convert.ToInt32(value, CultureInfo.InvariantCulture)); + Assert.Equal(expected.OrderBy(entry => entry.Key), actual.OrderBy(entry => entry.Key)); + } + + private static void AssertRecord(JsonElement vector) + { + var expected = vector.GetProperty("expected"); + var group = (byte)vector.GetProperty("group").GetInt32(); + var kind = (byte)vector.GetProperty("kind").GetInt32(); + + var body = Mappers.BinaryMapper.MapMetadataChangeBody(group, kind, Hex(vector, "body_hex"), out var knownKind); + var record = Mappers.BinaryMapper.MapMetadataChangeRecord(Hex(vector, "record_hex")); + + Assert.Equal((MetadataChangeKind)kind, knownKind); + Assert.Equal(body.GetType(), record.Body.GetType()); + AssertJson(expected, RenderRecord(record)); + var answer = ReadJsonAnswer(AnswerJson("changes", records: [expected])); + AssertJson(expected, RenderRecord(Assert.Single(answer.Records))); + } + + private static void AssertItem(JsonElement vector) + { + var expected = vector.GetProperty("expected"); + var group = (byte)vector.GetProperty("group").GetInt32(); + + Mappers.BinaryMapper.MapMetadataItemBody(group, Hex(vector, "body_hex"), out var knownGroup); + var item = Mappers.BinaryMapper.MapMetadataCurrentItem(Hex(vector, "item_hex")); + + Assert.Equal((MetadataChangeGroup)group, knownGroup); + AssertJson(expected, RenderItem(item)); + var answer = ReadJsonAnswer(AnswerJson("current", items: [expected])); + AssertJson(expected, RenderItem(Assert.Single(answer.Items))); + } + + private static void AssertCursor(JsonElement vector) + { + var text = vector.GetProperty("text").GetString()!; + + var cursor = MetadataChangeCursor.Parse(text); + + Assert.Equal(vector.GetProperty("log_id").GetString(), HexText(cursor.LogId)); + Assert.Equal(vector.GetProperty("op").GetUInt64(), cursor.Op); + Assert.Equal(vector.GetProperty("checksum").GetString(), HexText(cursor.Checksum)); + Assert.Equal(vector.GetProperty("ordinal").GetUInt32(), cursor.Ordinal); + Assert.Equal(text, cursor.ToString()); + var bytes = new byte[MetadataChangeCursor.SIZE]; + cursor.WriteTo(bytes); + Assert.Equal(Hex(vector, "hex"), bytes); + } + + private static void AssertPageToken(JsonElement vector) + { + var text = vector.GetProperty("text").GetString()!; + + var token = MetadataCurrentPageToken.Parse(text); + + Assert.Equal(vector.GetProperty("anchor").GetString(), token.Anchor.ToString()); + Assert.Equal(vector.GetProperty("required_head").GetUInt64(), token.RequiredHead); + Assert.Equal(vector.GetProperty("groups_mask").GetInt32(), token.GroupsMask); + Assert.Equal(MetadataChangeGroups.FromMask(token.GroupsMask), token.Groups); + Assert.Equal(vector.GetProperty("group").GetString(), SnakeCaseNames<MetadataChangeGroup>.Name(token.LastGroup)); + Assert.Equal(Hex(vector, "last_key_hex"), token.LastKey.ToArray()); + Assert.Equal(Hex(vector, "hex"), token.Payload.ToArray()); + Assert.Equal(text, token.ToString()); + Assert.Equal(token, MetadataCurrentPageToken.Parse(text)); + } + + private static void AssertRequest(JsonElement vector) + { + var query = Query(vector.GetProperty("input")); + + query.Validate(); + + Assert.Equal(Hex(vector, "request_hex"), TcpContracts.GetMetadataChanges(query)); + Assert.Equal(vector.GetProperty("http_query").GetString(), query.ToHttpQuery()); + } + + private static void AssertResponse(JsonElement vector) + { + var expected = vector.GetProperty("expected"); + + var answer = Mappers.BinaryMapper.MapMetadataChanges(Hex(vector, "response_hex")); + + AssertJson(expected, RenderAnswer(answer)); + AssertJson(expected, RenderAnswer(ReadJsonAnswer(expected.GetRawText()))); + } + + private static void AssertLargeValues(JsonElement vector) + { + var record = Mappers.BinaryMapper.MapMetadataChangeRecord(Hex(vector, "record_hex")); + + Assert.Equal(ParseU64(vector, "op"), record.Op); + Assert.Equal(vector.GetProperty("ordinal").GetUInt32(), record.Ordinal); + Assert.Equal(ParseU64(vector, "timestamp"), record.Timestamp); + Assert.Equal(vector.GetProperty("checksum").GetString(), HexText(record.Checksum)); + var session = Assert.IsType<MetadataSession>(record.Body); + Assert.Equal(ParseU64(vector, "user_created_at"), session.UserCreatedAt); + Assert.Equal(ParseU64(vector, "session"), session.Session); + Assert.Equal(vector.GetProperty("client_id").GetString(), HexText(session.ClientId)); + } + + private static void AssertRefused(MetadataChangesQuery query) + { + var refusal = Assert.Throws<IggyInvalidStatusCodeException>(query.Validate); + Assert.Equal(VsrError.INVALID_COMMAND, refusal.StatusCode); + Assert.False(refusal.FromServer); + } + + private static MetadataChangesQuery Query(JsonElement input) + { + var start = input.GetProperty("start").GetString() switch + { + "earliest" => MetadataChangeStart.Earliest, + "latest" => MetadataChangeStart.Latest, + "after" => MetadataChangeStart.After(MetadataChangeCursor.Parse(input.GetProperty("cursor").GetString()!)), + "current" => MetadataChangeStart.Current(input.GetProperty("page_token").GetString() is { } token + ? MetadataCurrentPageToken.Parse(token) + : null), + var other => throw new InvalidOperationException($"Unknown start {other}.") + }; + return new MetadataChangesQuery + { + Start = start, + Groups = input.GetProperty("groups").EnumerateArray() + .Select(name => SnakeCaseNames<MetadataChangeGroup>.TryParse(name.GetString()!, out var group) + ? group + : throw new InvalidOperationException($"Unknown group {name}.")) + .ToArray(), + Since = OptionalU64(input, "since"), + Until = OptionalU64(input, "until"), + Limit = input.GetProperty("limit").GetUInt32() + }; + } + + private static MetadataChanges ReadJsonAnswer(string json) + { + return JsonSerializer.Deserialize<MetadataChanges>(json)!; + } + + private static string AnswerJson(string phase, JsonElement[]? records = null, JsonElement[]? items = null) + { + var answer = new JsonObject + { + ["status"] = "ok", + ["phase"] = phase, + ["log_id"] = null, + ["oldest_op"] = 0, + ["retained_since"] = 0, + ["head_op"] = 0, + ["next"] = null, + ["has_more"] = false, + ["records"] = new JsonArray((records ?? []).Select(record => JsonNode.Parse(record.GetRawText())).ToArray()), + ["items"] = new JsonArray((items ?? []).Select(item => JsonNode.Parse(item.GetRawText())).ToArray()), + ["current_page_token"] = null + }; + return answer.ToJsonString(); + } + + private static JsonObject RenderAnswer(MetadataChanges answer) + { + return new JsonObject + { + ["status"] = SnakeCase(answer.Status), + ["phase"] = SnakeCase(answer.Phase), + ["log_id"] = answer.LogId is { } logId ? HexText(logId) : null, + ["oldest_op"] = answer.OldestOp, + ["retained_since"] = answer.RetainedSince, + ["head_op"] = answer.HeadOp, + ["next"] = answer.Next?.ToString(), + ["has_more"] = answer.HasMore, + ["records"] = new JsonArray(answer.Records.Select(record => (JsonNode)RenderRecord(record)).ToArray()), + ["items"] = new JsonArray(answer.Items.Select(item => (JsonNode)RenderItem(item)).ToArray()), + ["current_page_token"] = answer.CurrentPageToken?.ToString() + }; + } + + private static JsonObject RenderRecord(MetadataChangeRecord record) + { + if (record.Body is MetadataUnknownJsonBody json) + { + return JsonNode.Parse(json.Json)!.AsObject(); + } + + var node = new JsonObject + { + ["op"] = record.Op, + ["ordinal"] = record.Ordinal, + ["checksum"] = HexText(record.Checksum), + ["timestamp"] = record.Timestamp + }; + if (record.ActorUserId is { } actor) + { + node["actor_user_id"] = actor; + } + + if (record.Body is MetadataUnknownBody unknown) + { + node["group"] = unknown.Group.ToString(CultureInfo.InvariantCulture); + node["kind"] = unknown.Kind!.Value.ToString(CultureInfo.InvariantCulture); + node["body"] = Convert.ToBase64String(unknown.Body); + return node; + } + + node["group"] = SnakeCaseNames<MetadataChangeGroup>.Name(record.Group!.Value); + node["kind"] = SnakeCaseNames<MetadataChangeKind>.Name(record.Kind!.Value); + RenderBody(node, record.Body); + return node; + } + + private static JsonObject RenderItem(MetadataCurrentItem item) + { + switch (item.Body) + { + case MetadataUnknownJsonBody json: + return JsonNode.Parse(json.Json)!.AsObject(); + case MetadataUnknownBody unknown: + return new JsonObject + { + ["group"] = unknown.Group.ToString(CultureInfo.InvariantCulture), + ["body"] = Convert.ToBase64String(unknown.Body) + }; + } + + var node = new JsonObject { ["group"] = SnakeCaseNames<MetadataChangeGroup>.Name(item.Group!.Value) }; + RenderBody(node, item.Body); + return node; + } + + private static void RenderBody(JsonObject node, MetadataChangeBody body) + { + switch (body) + { + case MetadataStreamChange stream: + StreamKey(node, stream.StreamId, stream.StreamCreatedOp); + node["created_at"] = stream.CreatedAt; + node["name"] = stream.Name; + RenderOptions(node, stream.Options, stream.DerivedOptions, stream.OptionsUnreadable); + break; + case MetadataStreamDeleted deleted: + StreamKey(node, deleted.StreamId, deleted.StreamCreatedOp); + node["created_at"] = deleted.CreatedAt; + node["name"] = deleted.Name; + Optional(node, "requested_op", deleted.RequestedOp); + break; + case MetadataTopicChange topic: + TopicKey(node, topic.StreamId, topic.StreamCreatedOp, topic.TopicId, topic.TopicCreatedOp); + node["created_at"] = topic.CreatedAt; + node["name"] = topic.Name; + node["partitions_count"] = topic.PartitionsCount; + RenderOptions(node, topic.Options, topic.DerivedOptions, topic.OptionsUnreadable); + break; + case MetadataTopicItem topic: + TopicKey(node, topic.StreamId, topic.StreamCreatedOp, topic.TopicId, topic.TopicCreatedOp); + node["created_at"] = topic.CreatedAt; + node["name"] = topic.Name; + RenderOptions(node, topic.Options, topic.DerivedOptions, topic.OptionsUnreadable); + break; + case MetadataTopicDeleted topic: + TopicKey(node, topic.StreamId, topic.StreamCreatedOp, topic.TopicId, topic.TopicCreatedOp); + node["created_at"] = topic.CreatedAt; + node["name"] = topic.Name; + Optional(node, "requested_op", topic.RequestedOp); + break; + case MetadataPartitionsChanged partitions: + TopicKey(node, partitions.StreamId, partitions.StreamCreatedOp, partitions.TopicId, + partitions.TopicCreatedOp); + node["partitions_count"] = partitions.PartitionsCount; + node["removed_partition_ids"] = Ids(partitions.RemovedPartitionIds); + Optional(node, "requested_op", partitions.RequestedOp); + break; + case MetadataPartitionsItem partitions: + TopicKey(node, partitions.StreamId, partitions.StreamCreatedOp, partitions.TopicId, + partitions.TopicCreatedOp); + node["partitions_count"] = partitions.PartitionsCount; + break; + case MetadataConsumerGroupCreated group: + TopicKey(node, group.StreamId, group.StreamCreatedOp, group.TopicId, group.TopicCreatedOp); + node["group_id"] = group.GroupId; + node["name"] = group.Name; + node["consumer_groups_count"] = group.ConsumerGroupsCount; + break; + case MetadataConsumerGroupDeleted group: + TopicKey(node, group.StreamId, group.StreamCreatedOp, group.TopicId, group.TopicCreatedOp); + node["group_id"] = group.GroupId; + node["name"] = group.Name; + node["consumer_groups_count"] = group.ConsumerGroupsCount; + Optional(node, "requested_op", group.RequestedOp); + break; + case MetadataConsumerGroupItem group: + TopicKey(node, group.StreamId, group.StreamCreatedOp, group.TopicId, group.TopicCreatedOp); + node["group_id"] = group.GroupId; + node["name"] = group.Name; + break; + case MetadataMemberJoined member: + TopicKey(node, member.StreamId, member.StreamCreatedOp, member.TopicId, member.TopicCreatedOp); + node["group_id"] = member.GroupId; + node["member_id"] = member.MemberId; + node["client_id"] = HexText(member.ClientId); + node["members_count"] = member.MembersCount; + node["partition_ids"] = Ids(member.PartitionIds); + break; + case MetadataMemberLeft member: + TopicKey(node, member.StreamId, member.StreamCreatedOp, member.TopicId, member.TopicCreatedOp); + node["group_id"] = member.GroupId; + node["member_id"] = member.MemberId; + node["client_id"] = HexText(member.ClientId); + node["members_count"] = member.MembersCount; + node["reason"] = SnakeCase(member.Reason); + break; + case MetadataPartitionHandedOver handover: + TopicKey(node, handover.StreamId, handover.StreamCreatedOp, handover.TopicId, handover.TopicCreatedOp); + node["group_id"] = handover.GroupId; + node["partition_id"] = handover.PartitionId; + Optional(node, "member_id", handover.MemberId); + break; + case MetadataMemberItem member: + TopicKey(node, member.StreamId, member.StreamCreatedOp, member.TopicId, member.TopicCreatedOp); + node["group_id"] = member.GroupId; + node["member_id"] = member.MemberId; + node["client_id"] = HexText(member.ClientId); + node["partition_ids"] = Ids(member.PartitionIds); + break; + case MetadataUserChange user: + UserKey(node, user.UserId, user.UserCreatedAt); + node["username"] = user.Username; + node["status"] = SnakeCase(user.Status); + node["permissions"] = JsonSerializer.SerializeToNode(user.Permissions, PermissionsJson); + RenderOptions(node, user.Options, user.DerivedOptions, user.OptionsUnreadable); + break; + case MetadataUserIdentity user: + UserKey(node, user.UserId, user.UserCreatedAt); + node["username"] = user.Username; + break; + case MetadataUserPermissionsUpdated user: + UserKey(node, user.UserId, user.UserCreatedAt); + node["username"] = user.Username; + node["permissions"] = JsonSerializer.SerializeToNode(user.Permissions, PermissionsJson); + node["reason"] = SnakeCase(user.Reason); + break; + case MetadataTokenChange token: + UserKey(node, token.UserId, token.UserCreatedAt); + node["name"] = token.Name; + Optional(node, "expiry", token.Expiry); + break; + case MetadataTokenDeleted token: + UserKey(node, token.UserId, token.UserCreatedAt); + node["name"] = token.Name; + node["reason"] = SnakeCase(token.Reason); + break; + case MetadataSession session: + UserKey(node, session.UserId, session.UserCreatedAt); + node["client_id"] = HexText(session.ClientId); + node["session"] = session.Session; + break; + case MetadataSessionClosed session: + UserKey(node, session.UserId, session.UserCreatedAt); + node["client_id"] = HexText(session.ClientId); + node["session"] = session.Session; + node["reason"] = SnakeCase(session.Reason); + break; + default: + Assert.Fail($"No rendering for {body.GetType().Name}."); + break; + } + } + + private static void RenderOptions(JsonObject node, Dictionary<HeaderKey, HeaderValue>? options, + Dictionary<HeaderKey, HeaderValue>? derivedOptions, bool unreadable) + { + if (unreadable) + { + Assert.Null(options); + Assert.Null(derivedOptions); + node["options"] = null; + node["options_unreadable"] = true; + return; + } + + var rendered = new JsonObject(); + foreach (var (entries, isExplicit) in new[] { (options!, true), (derivedOptions!, false) }) + { + foreach (var (key, value) in entries) + { + rendered[key.AsString()] = new JsonObject + { + ["value"] = ResourceOptionsConverter.EncodeValue(value), + ["kind"] = value.Kind switch + { + HeaderKind.Float => "float32", + HeaderKind.Double => "float64", + var kind => kind.ToString().ToLowerInvariant() + }, + ["explicit"] = isExplicit + }; + } + } + + node["options"] = rendered; + } + + private static void StreamKey(JsonObject node, uint streamId, ulong streamCreatedOp) + { + node["stream_id"] = streamId; + node["stream_created_op"] = streamCreatedOp; + } + + private static void TopicKey(JsonObject node, uint streamId, ulong streamCreatedOp, uint topicId, + ulong topicCreatedOp) + { + StreamKey(node, streamId, streamCreatedOp); + node["topic_id"] = topicId; + node["topic_created_op"] = topicCreatedOp; + } + + private static void UserKey(JsonObject node, uint userId, ulong userCreatedAt) + { + node["user_id"] = userId; + node["user_created_at"] = userCreatedAt; + } + + private static void Optional(JsonObject node, string name, ulong? value) + { + if (value is { } present) + { + node[name] = present; + } + } + + private static void Optional(JsonObject node, string name, uint? value) + { + if (value is { } present) + { + node[name] = present; + } + } + + private static JsonArray Ids(IReadOnlyList<uint> ids) + { + return new JsonArray(ids.Select(id => (JsonNode)id).ToArray()); + } + + /// <summary>Compares JSON documents with object members sorted, so member order never matters.</summary> + private static void AssertJson(JsonElement expected, JsonNode actual) + { + Assert.Equal(Canonical(JsonNode.Parse(expected.GetRawText())), Canonical(actual)); + } + + private static string Canonical(JsonNode? node) + { + return node switch + { + null => "null", + JsonObject members => "{" + string.Join(",", members.OrderBy(member => member.Key, StringComparer.Ordinal) + .Select(member => $"{JsonSerializer.Serialize(member.Key)}:{Canonical(member.Value)}")) + "}", + JsonArray elements => "[" + string.Join(",", elements.Select(Canonical)) + "]", + _ => node.ToJsonString() + }; + } + + private static string SnakeCase<T>(T value) where T : struct, Enum + { + return JsonNamingPolicy.SnakeCaseLower.ConvertName(value.ToString()); + } + + private static string HexText(UInt128 value) + { + return value.ToString("x32", CultureInfo.InvariantCulture); + } + + private static ulong? OptionalU64(JsonElement element, string property) + { + return element.GetProperty(property).ValueKind == JsonValueKind.Null + ? null + : element.GetProperty(property).GetUInt64(); + } + + private static ulong ParseU64(JsonElement element, string property) + { + return ulong.Parse(element.GetProperty(property).GetString()!, CultureInfo.InvariantCulture); + } + + private static byte[] Hex(JsonElement vector, string property) + { + return Convert.FromHexString(vector.GetProperty(property).GetString()!); + } + + private static JsonElement Refused(string section, string vectorCase) + { + return Find(Fixture.GetProperty("refused").GetProperty(section), vectorCase); + } + + private static JsonElement Find(JsonElement section, string vectorCase) + { + return section.EnumerateArray().Single(entry => entry.GetProperty("case").GetString() == vectorCase); + } + + private static void AddCases(TheoryData<string, string> vectors, JsonElement section, string name) + { + foreach (var entry in section.EnumerateArray()) + { + vectors.Add(name, entry.GetProperty("case").GetString()!); + } + } + + private static TheoryData<string> Cases(JsonElement section) + { + var cases = new TheoryData<string>(); + foreach (var entry in section.EnumerateArray()) + { + cases.Add(entry.GetProperty("case").GetString()!); + } + + return cases; + } + + private static JsonElement LoadFixture() + { + var path = Path.Combine(AppContext.BaseDirectory, "Fixtures", "metadata_changes.json"); + using var document = JsonDocument.Parse(File.ReadAllText(path)); + Assert.Equal(1, document.RootElement.GetProperty("schema_version").GetInt32()); + return document.RootElement.Clone(); + } +}
diff --git a/foreign/csharp/README.md b/foreign/csharp/README.md index 4ba424d..9d2f14f 100644 --- a/foreign/csharp/README.md +++ b/foreign/csharp/README.md
@@ -772,6 +772,89 @@ Available snapshot types: `FilesystemOverview`, `ProcessList`, `ResourceUsage`, `Test`, `ServerLogs`, `ServerConfig`, `All`. +### Metadata Changes + +`GetMetadataChangesAsync` reads the node's change log: one record per public result of a committed metadata operation, such as a stream created, a topic renamed, a user's permissions changed or a session closed. A mirror keeps a cursor, asks for the records after it, and stores the returned `Next` cursor. The call needs root or the global `ManageServers` permission and works over TCP and HTTP. + +```c# +var query = new MetadataChangesQuery { Groups = [MetadataChangeGroup.Streams, MetadataChangeGroup.Topics] }; +while (true) +{ + var page = await client.GetMetadataChangesAsync(query); + switch (page.Status) + { + case MetadataChangeStatus.Ok: + foreach (var record in page.Records) + { + Console.WriteLine($"op {record.Op} by {record.ActorUserId}: {record.Kind}"); + } + + if (page.Next is { } next) + { + query = query with { Start = MetadataChangeStart.After(next) }; + } + + if (!page.HasMore) + { + await Task.Delay(TimeSpan.FromSeconds(1)); + } + + break; + case MetadataChangeStatus.Behind: + await Task.Delay(TimeSpan.FromMilliseconds(100)); + break; + default: + query = query with { Start = await RebuildAsync(client, query.Groups) }; + break; + } +} + +// Pages the current state, then returns the position to follow from. +static async Task<MetadataChangeStart> RebuildAsync(IIggyClient client, IReadOnlyList<MetadataChangeGroup> groups) +{ + var query = new MetadataChangesQuery { Start = MetadataChangeStart.Current(), Groups = groups }; + while (true) + { + var page = await client.GetMetadataChangesAsync(query); + if (page.Status == MetadataChangeStatus.Behind) + { + await Task.Delay(TimeSpan.FromMilliseconds(100)); + continue; + } + + if (page.Status != MetadataChangeStatus.Ok) + { + query = query with { Start = MetadataChangeStart.Current() }; + continue; + } + + foreach (var item in page.Items) + { + Console.WriteLine($"{item.Group}: {item.Body.GetType().Name}"); + } + + if (page.CurrentPageToken is { } token) + { + query = query with { Start = MetadataChangeStart.Current(token) }; + } + else if (page.Next is { } anchor) + { + return MetadataChangeStart.After(anchor); + } + else + { + query = query with { Start = MetadataChangeStart.Current() }; + } + } +} +``` + +`Expired` means the log no longer holds the history after the cursor, and `Reset` means the cursor belongs to another history, for example after the data directory was rebuilt. Either way the mirror rebuilds from current state pages and then follows the changes after the anchor they return. + +Current pages need the parent groups of every selected group, for example `Topics` with `Streams`. Keep a rebuilt state private until a change page answers `Ok` with `HasMore` false and a `HeadOp` at or above the last current page's. + +A cursor's `ToString()` is its `mc1_` text, which `MetadataChangeCursor.Parse` reads back, so a mirror can store it. A query the rules refuse fails with code 3 before anything is sent, and a record of a kind this SDK does not know arrives as `MetadataUnknownBody` or `MetadataUnknownJsonBody`. + ### Segment Management Delete up to N of the oldest sealed segments from a partition; the active segment is retained:
diff --git a/foreign/go/README.md b/foreign/go/README.md index 5ac54e5..aa4a884 100644 --- a/foreign/go/README.md +++ b/foreign/go/README.md
@@ -96,6 +96,85 @@ The server stores the key wrapped under its own key and never returns it. Reads report only its source and fingerprints. Without a key, a server with encryption enabled generates one for the stream, and a server without it keeps the stream plaintext and refuses a client key. +## Metadata changes + +`GetMetadataChanges` reads the node's change log: one record per public result of a committed metadata operation, such as a stream created, a topic renamed, a user's permissions changed or a session closed. A mirror keeps a cursor, asks for the records after it, and stores the returned `Next` cursor. The call needs root or the global `manage_servers` permission. + +```go +func followMetadataChanges(ctx context.Context, cli iggcon.Client, groups ...iggcon.MetadataChangeGroup) error { + query := iggcon.NewMetadataChangesQuery(iggcon.EarliestMetadataChangeStart(), groups...) + for { + page, err := cli.GetMetadataChanges(ctx, query) + if err != nil { + return err + } + switch page.Status { + case iggcon.MetadataChangeStatusOk: + for _, record := range page.Records { + log.Printf("op %d %s: %+v", record.Op, record.Kind, record.Body) + } + if page.Next != nil { + query.Start = iggcon.AfterMetadataChangeStart(*page.Next) + } + if !page.HasMore { + time.Sleep(time.Second) + } + case iggcon.MetadataChangeStatusBehind: + time.Sleep(100 * time.Millisecond) + default: + if query.Start, err = rebuildMetadataMirror(ctx, cli, groups); err != nil { + return err + } + } + } +} + +// rebuildMetadataMirror pages the current state, then returns the position to +// follow from. +func rebuildMetadataMirror( + ctx context.Context, + cli iggcon.Client, + groups []iggcon.MetadataChangeGroup, +) (iggcon.MetadataChangeStart, error) { + query := iggcon.NewMetadataChangesQuery(iggcon.CurrentMetadataChangeStart(nil), groups...) + for { + page, err := cli.GetMetadataChanges(ctx, query) + if err != nil { + return iggcon.MetadataChangeStart{}, err + } + switch page.Status { + case iggcon.MetadataChangeStatusBehind: + time.Sleep(100 * time.Millisecond) + continue + case iggcon.MetadataChangeStatusExpired, iggcon.MetadataChangeStatusReset: + query.Start = iggcon.CurrentMetadataChangeStart(nil) + continue + } + for _, item := range page.Items { + log.Printf("%s: %+v", item.Group, item.Body) + } + switch { + case page.CurrentPageToken != nil: + query.Start = iggcon.CurrentMetadataChangeStart(page.CurrentPageToken) + case page.Next != nil: + return iggcon.AfterMetadataChangeStart(*page.Next), nil + default: + query.Start = iggcon.CurrentMetadataChangeStart(nil) + } + } +} +``` + +The sample is in `client/example_metadata_changes_test.go`. + +`MetadataChangeStatusExpired` means the log no longer holds the history after the cursor, and `MetadataChangeStatusReset` means the cursor belongs to another history, for example after the data directory was rebuilt. Either way the mirror rebuilds from current state pages and then follows the changes after the anchor they return. + +Current pages need the parent groups of every selected group, for example `MetadataChangeGroupTopics` with `MetadataChangeGroupStreams`. Keep a rebuilt state private until a change page answers `MetadataChangeStatusOk` with `HasMore` false and a `HeadOp` at or above the last current page's. + +Record and item bodies are typed by kind and group, for example `*iggcon.MetadataStreamChange` or `*iggcon.MetadataUserPermissionsUpdated`. A group or kind newer than this build arrives as `*iggcon.MetadataUnknownBody` with its raw codes on the record, so a mirror that cannot apply it can stop without advancing its cursor. + +A query that breaks the request rules, such as a limit outside 1 to 1000 or a time bound with a latest or current start, is refused with `ierror.ErrInvalidCommand` before anything is sent. Cursors and page tokens print and parse as the `mc1_` and `cs1_` text forms the HTTP API and the CLI use. Page tokens also implement `encoding.TextMarshaler` and `encoding.TextUnmarshaler`, so `encoding/json` stores them as their `cs1_` text. + ## Delivery semantics Receipt access follows the server's [session and revocation policy](../../core/server/README.md#sessions-and-retry-receipts).
diff --git a/foreign/go/binary_serialization/metadata_changes_deserializer.go b/foreign/go/binary_serialization/metadata_changes_deserializer.go new file mode 100644 index 0000000..9c23518 --- /dev/null +++ b/foreign/go/binary_serialization/metadata_changes_deserializer.go
@@ -0,0 +1,636 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package binaryserialization + +import ( + "encoding/binary" + "fmt" + "unicode/utf8" + + iggcon "github.com/apache/iggy/foreign/go/contracts" + ierror "github.com/apache/iggy/foreign/go/errors" +) + +const ( + // metadataChangeRecordMinimumSize is a record with no actor and an empty + // body: op, ordinal, checksum, timestamp, actor flag, group, kind, length. + metadataChangeRecordMinimumSize = 8 + 4 + 16 + 8 + 1 + 1 + 1 + 4 + // metadataCurrentItemMinimumSize is an item with an empty body. + metadataCurrentItemMinimumSize = 1 + 4 +) + +// changeReader reads a response or one body front to back. The first failed +// read sticks: later reads return zero values and err reports the failure. +type changeReader struct { + data []byte + position int + err error +} + +// DeserializeMetadataChanges reads a GetMetadataChanges response. +// +// Wire format: [status:u8][phase:u8][has_log_id:u8][log_id:u128] +// [oldest_op:u64][retained_since:u64][head_op:u64][has_next:u8][cursor:44] +// [has_more:u8][records_count:u32][record]*[items_count:u32][item]* +// [token_len:u16][token]. Bodies are decoded strictly: unknown reason codes, +// presence bytes other than 0 and 1, a known kind under another group and +// trailing body bytes are refused, while a group or kind this build does not +// know is kept with its raw body. Every refusal wraps ierror.ErrInvalidFormat. +func DeserializeMetadataChanges(payload []byte) (*iggcon.MetadataChanges, error) { + reader := &changeReader{data: payload} + changes := &iggcon.MetadataChanges{ + Status: iggcon.MetadataChangeStatus(reader.readCode("status", uint8(iggcon.MetadataChangeStatusReset))), + Phase: iggcon.MetadataChangePhase(reader.readCode("phase", uint8(iggcon.MetadataChangePhaseCurrent))), + } + if reader.readFlag() { + logID := reader.readUnsigned128() + changes.LogID = &logID + } + changes.OldestOp = reader.readUint64() + changes.RetainedSince = reader.readUint64() + changes.HeadOp = reader.readUint64() + if reader.readFlag() { + next := reader.readCursor() + changes.Next = &next + } + changes.HasMore = reader.readFlag() + + recordsCount := reader.readUint32() + changes.Records = make([]iggcon.MetadataChangeRecord, 0, + boundedCapacity(recordsCount, reader.remaining(), metadataChangeRecordMinimumSize)) + for range recordsCount { + record := reader.readRecord() + if reader.err != nil { + break + } + changes.Records = append(changes.Records, record) + } + + itemsCount := reader.readUint32() + changes.Items = make([]iggcon.MetadataCurrentItem, 0, + boundedCapacity(itemsCount, reader.remaining(), metadataCurrentItemMinimumSize)) + for range itemsCount { + item := reader.readItem() + if reader.err != nil { + break + } + changes.Items = append(changes.Items, item) + } + + if tokenLength := int(reader.readUint16()); tokenLength != 0 { + tokenBytes := reader.take(tokenLength) + if reader.err == nil { + token := &iggcon.MetadataCurrentPageToken{} + if err := token.UnmarshalBinary(tokenBytes); err != nil { + return nil, err + } + changes.CurrentPageToken = token + } + } + if reader.err != nil { + return nil, reader.err + } + + holdsOtherContent := len(changes.Items) != 0 || changes.CurrentPageToken != nil + if changes.Phase == iggcon.MetadataChangePhaseCurrent { + holdsOtherContent = len(changes.Records) != 0 + } + if holdsOtherContent { + return nil, fmt.Errorf("%w: a changes page carries only records and a current page only items", + ierror.ErrInvalidFormat) + } + return changes, nil +} + +func (r *changeReader) readRecord() iggcon.MetadataChangeRecord { + record := iggcon.MetadataChangeRecord{ + Op: r.readUint64(), + Ordinal: r.readUint32(), + Checksum: r.readUnsigned128(), + Timestamp: r.readUint64(), + ActorUserID: r.readOptionalUint32(), + Group: iggcon.MetadataChangeGroup(r.readUint8()), + Kind: iggcon.MetadataChangeKind(r.readUint8()), + } + body := r.take(int(r.readUint32())) + if r.err != nil { + return record + } + record.Body, r.err = decodeChangeBody(record.Group, record.Kind, body) + return record +} + +func (r *changeReader) readItem() iggcon.MetadataCurrentItem { + item := iggcon.MetadataCurrentItem{Group: iggcon.MetadataChangeGroup(r.readUint8())} + body := r.take(int(r.readUint32())) + if r.err != nil { + return item + } + item.Body, r.err = decodeItemBody(item.Group, body) + return item +} + +// decodeChangeBody types a record body by its codes. A group or kind this +// build does not know keeps the raw body. +func decodeChangeBody(group iggcon.MetadataChangeGroup, kind iggcon.MetadataChangeKind, body []byte) (iggcon.MetadataChangeBody, error) { + if !group.IsKnown() || !kind.Group().IsKnown() { + return &iggcon.MetadataUnknownBody{Body: append([]byte(nil), body...)}, nil + } + if kind.Group() != group { + return nil, fmt.Errorf("%w: change kind %d does not belong to group %d", + ierror.ErrInvalidFormat, uint8(kind), uint8(group)) + } + reader := &changeReader{data: body} + var decoded iggcon.MetadataChangeBody + switch kind { + case iggcon.MetadataChangeKindStreamCreated, iggcon.MetadataChangeKindStreamUpdated: + decoded = reader.readStreamChange() + case iggcon.MetadataChangeKindStreamDeleted: + decoded = &iggcon.MetadataStreamDeleted{ + MetadataStreamKey: reader.readStreamKey(), + CreatedAt: reader.readUint64(), + Name: reader.readName(), + RequestedOp: reader.readOptionalUint64(), + } + case iggcon.MetadataChangeKindTopicCreated, iggcon.MetadataChangeKindTopicUpdated: + decoded = &iggcon.MetadataTopicChange{ + MetadataTopicKey: reader.readTopicKey(), + CreatedAt: reader.readUint64(), + Name: reader.readName(), + PartitionsCount: reader.readUint32(), + Options: reader.readOptions(), + } + case iggcon.MetadataChangeKindTopicDeleted: + decoded = &iggcon.MetadataTopicDeleted{ + MetadataTopicKey: reader.readTopicKey(), + CreatedAt: reader.readUint64(), + Name: reader.readName(), + RequestedOp: reader.readOptionalUint64(), + } + case iggcon.MetadataChangeKindPartitionsChanged: + decoded = &iggcon.MetadataPartitionsChanged{ + MetadataTopicKey: reader.readTopicKey(), + PartitionsCount: reader.readUint32(), + RemovedPartitionIDs: reader.readPartitionIDs(), + RequestedOp: reader.readOptionalUint64(), + } + case iggcon.MetadataChangeKindConsumerGroupCreated: + decoded = &iggcon.MetadataConsumerGroupCreated{ + MetadataTopicKey: reader.readTopicKey(), + GroupID: reader.readUint32(), + Name: reader.readName(), + ConsumerGroupsCount: reader.readUint32(), + } + case iggcon.MetadataChangeKindConsumerGroupDeleted: + decoded = &iggcon.MetadataConsumerGroupDeleted{ + MetadataTopicKey: reader.readTopicKey(), + GroupID: reader.readUint32(), + Name: reader.readName(), + ConsumerGroupsCount: reader.readUint32(), + RequestedOp: reader.readOptionalUint64(), + } + case iggcon.MetadataChangeKindConsumerGroupMemberJoined: + decoded = &iggcon.MetadataMemberJoined{ + MetadataTopicKey: reader.readTopicKey(), + GroupID: reader.readUint32(), + MemberID: reader.readUint32(), + ClientID: reader.readUnsigned128(), + MembersCount: reader.readUint32(), + PartitionIDs: reader.readPartitionIDs(), + } + case iggcon.MetadataChangeKindConsumerGroupMemberLeft: + decoded = &iggcon.MetadataMemberLeft{ + MetadataTopicKey: reader.readTopicKey(), + GroupID: reader.readUint32(), + MemberID: reader.readUint32(), + ClientID: reader.readUnsigned128(), + MembersCount: reader.readUint32(), + Reason: iggcon.MetadataMemberLeftReason( + reader.readCode("member left reason", uint8(iggcon.MetadataMemberLeftReasonSessionEnded))), + } + case iggcon.MetadataChangeKindConsumerGroupPartitionHandedOver: + decoded = &iggcon.MetadataPartitionHandedOver{ + MetadataTopicKey: reader.readTopicKey(), + GroupID: reader.readUint32(), + PartitionID: reader.readUint32(), + MemberID: reader.readOptionalUint32(), + } + case iggcon.MetadataChangeKindUserCreated, iggcon.MetadataChangeKindUserUpdated: + decoded = reader.readUserChange() + case iggcon.MetadataChangeKindUserDeleted, iggcon.MetadataChangeKindUserPasswordChanged: + decoded = &iggcon.MetadataUserIdentity{ + MetadataUserKey: reader.readUserKey(), + Username: reader.readName(), + } + case iggcon.MetadataChangeKindUserPermissionsUpdated: + decoded = &iggcon.MetadataUserPermissionsUpdated{ + MetadataUserKey: reader.readUserKey(), + Username: reader.readName(), + Permissions: reader.readPermissions(), + Reason: iggcon.MetadataPermissionsUpdatedReason( + reader.readCode("permissions updated reason", uint8(iggcon.MetadataPermissionsUpdatedReasonResourceDeleted))), + } + case iggcon.MetadataChangeKindPersonalAccessTokenCreated: + decoded = reader.readTokenChange() + case iggcon.MetadataChangeKindPersonalAccessTokenDeleted: + decoded = &iggcon.MetadataTokenDeleted{ + MetadataUserKey: reader.readUserKey(), + Name: reader.readName(), + Reason: iggcon.MetadataTokenDeletedReason( + reader.readCode("token deleted reason", uint8(iggcon.MetadataTokenDeletedReasonExpired))), + } + case iggcon.MetadataChangeKindSessionRegistered: + decoded = reader.readSession() + case iggcon.MetadataChangeKindSessionClosed: + decoded = &iggcon.MetadataSessionClosed{ + MetadataUserKey: reader.readUserKey(), + ClientID: reader.readUnsigned128(), + Session: reader.readUint64(), + Reason: iggcon.MetadataSessionClosedReason( + reader.readCode("session closed reason", uint8(iggcon.MetadataSessionClosedReasonRevoked))), + } + } + if err := reader.finish(); err != nil { + return nil, err + } + return decoded, nil +} + +// decodeItemBody types a current state item body by its group. A group this +// build does not know keeps the raw body. +func decodeItemBody(group iggcon.MetadataChangeGroup, body []byte) (iggcon.MetadataChangeBody, error) { + reader := &changeReader{data: body} + var decoded iggcon.MetadataChangeBody + switch group { + case iggcon.MetadataChangeGroupStreams: + decoded = reader.readStreamChange() + case iggcon.MetadataChangeGroupTopics: + decoded = &iggcon.MetadataTopicItem{ + MetadataTopicKey: reader.readTopicKey(), + CreatedAt: reader.readUint64(), + Name: reader.readName(), + Options: reader.readOptions(), + } + case iggcon.MetadataChangeGroupPartitions: + decoded = &iggcon.MetadataPartitionsItem{ + MetadataTopicKey: reader.readTopicKey(), + PartitionsCount: reader.readUint32(), + } + case iggcon.MetadataChangeGroupConsumerGroups: + decoded = &iggcon.MetadataConsumerGroupItem{ + MetadataTopicKey: reader.readTopicKey(), + GroupID: reader.readUint32(), + Name: reader.readName(), + } + case iggcon.MetadataChangeGroupConsumerGroupMembers: + decoded = &iggcon.MetadataMemberItem{ + MetadataTopicKey: reader.readTopicKey(), + GroupID: reader.readUint32(), + MemberID: reader.readUint32(), + ClientID: reader.readUnsigned128(), + PartitionIDs: reader.readPartitionIDs(), + } + case iggcon.MetadataChangeGroupUsers: + decoded = reader.readUserChange() + case iggcon.MetadataChangeGroupPersonalAccessTokens: + decoded = reader.readTokenChange() + case iggcon.MetadataChangeGroupSessions: + decoded = reader.readSession() + default: + return &iggcon.MetadataUnknownBody{Body: append([]byte(nil), body...)}, nil + } + if err := reader.finish(); err != nil { + return nil, err + } + return decoded, nil +} + +func (r *changeReader) readStreamChange() *iggcon.MetadataStreamChange { + return &iggcon.MetadataStreamChange{ + MetadataStreamKey: r.readStreamKey(), + CreatedAt: r.readUint64(), + Name: r.readName(), + Options: r.readOptions(), + } +} + +func (r *changeReader) readUserChange() *iggcon.MetadataUserChange { + return &iggcon.MetadataUserChange{ + MetadataUserKey: r.readUserKey(), + Username: r.readName(), + Status: r.readUserStatus(), + Permissions: r.readPermissions(), + Options: r.readOptions(), + } +} + +// readTokenChange reads a token body, where expiry 0 means never. +func (r *changeReader) readTokenChange() *iggcon.MetadataTokenChange { + token := &iggcon.MetadataTokenChange{ + MetadataUserKey: r.readUserKey(), + Name: r.readName(), + } + if expiry := r.readUint64(); expiry != 0 { + token.Expiry = &expiry + } + return token +} + +func (r *changeReader) readSession() *iggcon.MetadataSession { + return &iggcon.MetadataSession{ + MetadataUserKey: r.readUserKey(), + ClientID: r.readUnsigned128(), + Session: r.readUint64(), + } +} + +func (r *changeReader) readStreamKey() iggcon.MetadataStreamKey { + return iggcon.MetadataStreamKey{ + StreamID: r.readUint32(), + StreamCreatedOp: r.readUint64(), + } +} + +func (r *changeReader) readTopicKey() iggcon.MetadataTopicKey { + return iggcon.MetadataTopicKey{ + MetadataStreamKey: r.readStreamKey(), + TopicID: r.readUint32(), + TopicCreatedOp: r.readUint64(), + } +} + +func (r *changeReader) readUserKey() iggcon.MetadataUserKey { + return iggcon.MetadataUserKey{ + UserID: r.readUint32(), + UserCreatedAt: r.readUint64(), + } +} + +// readOptions reads [state:u8], then for state 0 the explicit and derived +// blocks exactly as the stream, topic and user reads carry them. +func (r *changeReader) readOptions() iggcon.MetadataOptions { + offset := r.position + switch state := r.readUint8(); { + case r.err != nil: + return iggcon.MetadataOptions{} + case state == 1: + return iggcon.MetadataOptions{Unreadable: true} + case state != 0: + r.fail("options state %d at offset %d", state, offset) + return iggcon.MetadataOptions{} + } + explicit, consumed, err := deserializeOptions(r.data, r.position) + if err != nil { + r.fail("options at offset %d: %v", r.position, err) + return iggcon.MetadataOptions{} + } + r.position += consumed + derived, consumed, err := deserializeOptions(r.data, r.position) + if err != nil { + r.fail("derived options at offset %d: %v", r.position, err) + return iggcon.MetadataOptions{} + } + r.position += consumed + return iggcon.MetadataOptions{Options: explicit, DerivedOptions: derived} +} + +// readPermissions reads [has:u8] and the permission set without a length +// prefix. Grant bytes read as booleans the way the server's decoder does. +func (r *changeReader) readPermissions() *iggcon.Permissions { + if !r.readFlag() { + return nil + } + permissions := &iggcon.Permissions{Global: iggcon.GlobalPermissions{ + ManageServers: r.readBool(), + ReadServers: r.readBool(), + ManageUsers: r.readBool(), + ReadUsers: r.readBool(), + ManageStreams: r.readBool(), + ReadStreams: r.readBool(), + ManageTopics: r.readBool(), + ReadTopics: r.readBool(), + PollMessages: r.readBool(), + SendMessages: r.readBool(), + }} + if !r.readBool() { + return permissions + } + permissions.Streams = make(map[int]*iggcon.StreamPermissions) + for r.err == nil { + streamID := int(r.readUint32()) + stream := &iggcon.StreamPermissions{ + ManageStream: r.readBool(), + ReadStream: r.readBool(), + ManageTopics: r.readBool(), + ReadTopics: r.readBool(), + PollMessages: r.readBool(), + SendMessages: r.readBool(), + } + if r.readBool() { + stream.Topics = make(map[int]*iggcon.TopicPermissions) + for r.err == nil { + topicID := int(r.readUint32()) + stream.Topics[topicID] = &iggcon.TopicPermissions{ + ManageTopic: r.readBool(), + ReadTopic: r.readBool(), + PollMessages: r.readBool(), + SendMessages: r.readBool(), + } + if !r.readBool() { + break + } + } + } + permissions.Streams[streamID] = stream + if !r.readBool() { + break + } + } + return permissions +} + +// readPartitionIDs reads [count:u32][id:u32]*, refusing ids that are not +// strictly ascending. +func (r *changeReader) readPartitionIDs() []uint32 { + count := r.readUint32() + ids := make([]uint32, 0, boundedCapacity(count, r.remaining(), 4)) + for range count { + id := r.readUint32() + if r.err != nil { + break + } + if len(ids) > 0 && ids[len(ids)-1] >= id { + r.fail("partition ids must be strictly ascending at offset %d", r.position-4) + break + } + ids = append(ids, id) + } + return ids +} + +func (r *changeReader) readUserStatus() iggcon.UserStatus { + offset := r.position + switch status := r.readUint8(); { + case r.err != nil: + return iggcon.Active + case status == 1: + return iggcon.Active + case status == 2: + return iggcon.Inactive + default: + r.fail("user status %d at offset %d", status, offset) + return iggcon.Active + } +} + +// readName reads a 1 to 255 byte UTF-8 name behind its u8 length. +func (r *changeReader) readName() string { + offset := r.position + length := int(r.readUint8()) + if r.err == nil && length == 0 { + r.fail("empty name at offset %d", offset) + } + name := r.take(length) + if r.err == nil && !utf8.Valid(name) { + r.fail("name at offset %d is not UTF-8", offset) + } + return string(name) +} + +func (r *changeReader) readCursor() iggcon.MetadataChangeCursor { + var cursor iggcon.MetadataChangeCursor + if data := r.take(iggcon.MetadataChangeCursorSize); r.err == nil { + if err := cursor.UnmarshalBinary(data); err != nil { + r.err = err + } + } + return cursor +} + +// readCode reads a one-byte code, refusing any above highest. +func (r *changeReader) readCode(name string, highest uint8) uint8 { + offset := r.position + code := r.readUint8() + if r.err == nil && code > highest { + r.fail("unknown %s %d at offset %d", name, code, offset) + } + return code +} + +func (r *changeReader) readOptionalUint64() *uint64 { + if !r.readFlag() { + return nil + } + value := r.readUint64() + return &value +} + +func (r *changeReader) readOptionalUint32() *uint32 { + if !r.readFlag() { + return nil + } + value := r.readUint32() + return &value +} + +// readFlag reads a presence or boolean byte, refusing anything but 0 and 1. +func (r *changeReader) readFlag() bool { + offset := r.position + value := r.readUint8() + if r.err == nil && value > 1 { + r.fail("flag byte %d at offset %d", value, offset) + } + return value == 1 && r.err == nil +} + +func (r *changeReader) readBool() bool { + return r.readUint8() != 0 +} + +func (r *changeReader) readUnsigned128() iggcon.Unsigned128 { + data := r.take(16) + if data == nil { + return iggcon.Unsigned128{} + } + return iggcon.Unsigned128{ + Low: binary.LittleEndian.Uint64(data[0:8]), + High: binary.LittleEndian.Uint64(data[8:16]), + } +} + +func (r *changeReader) readUint64() uint64 { + if data := r.take(8); data != nil { + return binary.LittleEndian.Uint64(data) + } + return 0 +} + +func (r *changeReader) readUint32() uint32 { + if data := r.take(4); data != nil { + return binary.LittleEndian.Uint32(data) + } + return 0 +} + +func (r *changeReader) readUint16() uint16 { + if data := r.take(2); data != nil { + return binary.LittleEndian.Uint16(data) + } + return 0 +} + +func (r *changeReader) readUint8() uint8 { + if data := r.take(1); data != nil { + return data[0] + } + return 0 +} + +// take returns the next count bytes, aliasing the input. +func (r *changeReader) take(count int) []byte { + if r.err != nil { + return nil + } + // A u32 length above MaxInt32 converts to a negative int where int is 32 + // bits wide. + if count < 0 || r.remaining() < count { + r.fail("need %d bytes at offset %d, have %d", count, r.position, r.remaining()) + return nil + } + data := r.data[r.position : r.position+count] + r.position += count + return data +} + +func (r *changeReader) remaining() int { + return len(r.data) - r.position +} + +// finish refuses bytes left after a body. +func (r *changeReader) finish() error { + if r.err == nil && r.remaining() != 0 { + r.fail("%d trailing bytes after a change body", r.remaining()) + } + return r.err +} + +func (r *changeReader) fail(format string, arguments ...any) { + if r.err == nil { + r.err = fmt.Errorf("%w: %s", ierror.ErrInvalidFormat, fmt.Sprintf(format, arguments...)) + } +}
diff --git a/foreign/go/client/example_metadata_changes_test.go b/foreign/go/client/example_metadata_changes_test.go new file mode 100644 index 0000000..91ea289 --- /dev/null +++ b/foreign/go/client/example_metadata_changes_test.go
@@ -0,0 +1,112 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package client_test + +import ( + "context" + "log" + "time" + + "github.com/apache/iggy/foreign/go/client" + "github.com/apache/iggy/foreign/go/client/tcp" + iggcon "github.com/apache/iggy/foreign/go/contracts" +) + +// The README's metadata changes sample, kept here so it compiles. +func Example_metadataChanges() { + ctx := context.Background() + cli, err := client.NewIggyClient(client.WithTcp(tcp.WithServerAddress("127.0.0.1:8090"))) + if err != nil { + log.Fatal(err) + } + defer func() { _ = cli.Close() }() + if err := cli.Connect(ctx); err != nil { + log.Fatal(err) + } + if _, err := cli.LoginUser(ctx, "iggy", "iggy"); err != nil { + log.Fatal(err) + } + + err = followMetadataChanges(ctx, cli, iggcon.MetadataChangeGroupStreams, iggcon.MetadataChangeGroupTopics) + if err != nil { + log.Fatal(err) + } +} + +func followMetadataChanges(ctx context.Context, cli iggcon.Client, groups ...iggcon.MetadataChangeGroup) error { + query := iggcon.NewMetadataChangesQuery(iggcon.EarliestMetadataChangeStart(), groups...) + for { + page, err := cli.GetMetadataChanges(ctx, query) + if err != nil { + return err + } + switch page.Status { + case iggcon.MetadataChangeStatusOk: + for _, record := range page.Records { + log.Printf("op %d %s: %+v", record.Op, record.Kind, record.Body) + } + if page.Next != nil { + query.Start = iggcon.AfterMetadataChangeStart(*page.Next) + } + if !page.HasMore { + time.Sleep(time.Second) + } + case iggcon.MetadataChangeStatusBehind: + time.Sleep(100 * time.Millisecond) + default: + if query.Start, err = rebuildMetadataMirror(ctx, cli, groups); err != nil { + return err + } + } + } +} + +// rebuildMetadataMirror pages the current state, then returns the position to +// follow from. +func rebuildMetadataMirror( + ctx context.Context, + cli iggcon.Client, + groups []iggcon.MetadataChangeGroup, +) (iggcon.MetadataChangeStart, error) { + query := iggcon.NewMetadataChangesQuery(iggcon.CurrentMetadataChangeStart(nil), groups...) + for { + page, err := cli.GetMetadataChanges(ctx, query) + if err != nil { + return iggcon.MetadataChangeStart{}, err + } + switch page.Status { + case iggcon.MetadataChangeStatusBehind: + time.Sleep(100 * time.Millisecond) + continue + case iggcon.MetadataChangeStatusExpired, iggcon.MetadataChangeStatusReset: + query.Start = iggcon.CurrentMetadataChangeStart(nil) + continue + } + for _, item := range page.Items { + log.Printf("%s: %+v", item.Group, item.Body) + } + switch { + case page.CurrentPageToken != nil: + query.Start = iggcon.CurrentMetadataChangeStart(page.CurrentPageToken) + case page.Next != nil: + return iggcon.AfterMetadataChangeStart(*page.Next), nil + default: + query.Start = iggcon.CurrentMetadataChangeStart(nil) + } + } +}
diff --git a/foreign/go/client/tcp/tcp_utilities.go b/foreign/go/client/tcp/tcp_utilities.go index 76ab245..4db6a7e 100644 --- a/foreign/go/client/tcp/tcp_utilities.go +++ b/foreign/go/client/tcp/tcp_utilities.go
@@ -38,6 +38,17 @@ return binaryserialization.DeserializeOptionSpecs(buffer) } +// GetMetadataChanges reads one page of the server's metadata change log, or +// of its current state. A query that breaks the request rules is refused with +// ierror.ErrInvalidCommand before anything is sent. +func (c *IggyTcpClient) GetMetadataChanges(ctx context.Context, query iggcon.MetadataChangesQuery) (*iggcon.MetadataChanges, error) { + buffer, err := c.do(ctx, &command.GetMetadataChanges{Query: query}) + if err != nil { + return nil, err + } + return binaryserialization.DeserializeMetadataChanges(buffer) +} + func (c *IggyTcpClient) GetStats(ctx context.Context) (*iggcon.Stats, error) { buffer, err := c.do(ctx, &command.GetStats{}) if err != nil {
diff --git a/foreign/go/contracts/client.go b/foreign/go/contracts/client.go index 53f6d71..d7eca07 100644 --- a/foreign/go/contracts/client.go +++ b/foreign/go/contracts/client.go
@@ -86,6 +86,16 @@ // Authentication is required. DescribeOptions(ctx context.Context, scope OptionsScope) ([]OptionSpec, error) + // GetMetadataChanges get one page of the server's metadata change log: the + // public results of committed metadata operations after a cursor, or a + // page of the current state with a current start. Keep the returned Next + // cursor and send it back to continue. Expired and reset answers mean the + // mirror rebuilds from current state pages. + // A query that breaks the request rules is refused with + // ierror.ErrInvalidCommand before anything is sent. + // Authentication is required, and the permission to manage the servers. + GetMetadataChanges(ctx context.Context, query MetadataChangesQuery) (*MetadataChanges, error) + // CreateTopic create a new topic. Nil options select the defaults. // Authentication is required, and the permission to manage the topics. //
diff --git a/foreign/go/contracts/metadata_change_position.go b/foreign/go/contracts/metadata_change_position.go new file mode 100644 index 0000000..c081d1d --- /dev/null +++ b/foreign/go/contracts/metadata_change_position.go
@@ -0,0 +1,323 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package iggcon + +import ( + "encoding/base64" + "encoding/binary" + "fmt" + "strings" + "unicode/utf8" + + ierror "github.com/apache/iggy/foreign/go/errors" +) + +const ( + // MetadataChangeCursorPrefix starts the text form of a change cursor. + MetadataChangeCursorPrefix = "mc1_" + // MetadataCurrentPageTokenPrefix starts the text form of a current state + // page token. + MetadataCurrentPageTokenPrefix = "cs1_" + // MetadataChangeCursorSize is the encoded size of a change cursor. + MetadataChangeCursorSize = 44 +) + +const ( + metadataChangeCursorTextLength = len(MetadataChangeCursorPrefix) + 59 + // maxPageTokenKeySize fits a user id and a personal access token name. + maxPageTokenKeySize = 4 + 1 + 255 + pageTokenFixedSize = MetadataChangeCursorSize + 8 + 1 + 1 + 2 + maxPageTokenSize = pageTokenFixedSize + maxPageTokenKeySize + maxPageTokenTextLength = len(MetadataCurrentPageTokenPrefix) + (maxPageTokenSize+2)/3*4 +) + +// Unsigned128 is an unsigned 128-bit value: a log id, a checksum or a client +// id. String renders the 32 lowercase hex characters the HTTP API uses. +type Unsigned128 struct { + High uint64 + Low uint64 +} + +// MetadataChangeCursor is the position of the last change record a caller +// consumed or scanned. Op 0 with a zero checksum and ordinal names the +// beginning of the history LogID identifies. +// +// Its text form is MetadataChangeCursorPrefix followed by unpadded Base64url +// of the 44 encoded bytes. +type MetadataChangeCursor struct { + // LogID is the prepare checksum of op 1, the root of this history. + LogID Unsigned128 + Op uint64 + // Checksum is the prepare checksum of Op. + Checksum Unsigned128 + // Ordinal is the position of the record within Op. + Ordinal uint32 +} + +// MetadataCurrentPageToken continues a current state scan. Send it back +// unchanged with the same groups. +// +// Its text form is MetadataCurrentPageTokenPrefix followed by unpadded +// Base64url of the encoded token. +type MetadataCurrentPageToken struct { + anchor MetadataChangeCursor + requiredHead uint64 + groupsMask uint8 + group MetadataChangeGroup + lastKey []byte +} + +// ParseMetadataChangeCursor reads the text form of a change cursor. Anything +// but the prefix, the exact length and canonical unpadded Base64url of a +// cursor whose op 0 carries no checksum and ordinal is +// ierror.ErrInvalidCommand. +func ParseMetadataChangeCursor(text string) (MetadataChangeCursor, error) { + if len(text) != metadataChangeCursorTextLength { + return MetadataChangeCursor{}, ierror.ErrInvalidCommand + } + payload, err := decodeCanonicalBase64(text, MetadataChangeCursorPrefix) + if err != nil { + return MetadataChangeCursor{}, err + } + var cursor MetadataChangeCursor + if err := cursor.UnmarshalBinary(payload); err != nil { + return MetadataChangeCursor{}, ierror.ErrInvalidCommand + } + return cursor, nil +} + +// ParseMetadataCurrentPageToken reads the text form of a page token. Anything +// but the prefix, canonical unpadded Base64url and a token that keeps its own +// invariants is ierror.ErrInvalidCommand. +func ParseMetadataCurrentPageToken(text string) (*MetadataCurrentPageToken, error) { + if len(text) > maxPageTokenTextLength { + return nil, ierror.ErrInvalidCommand + } + payload, err := decodeCanonicalBase64(text, MetadataCurrentPageTokenPrefix) + if err != nil { + return nil, err + } + token := &MetadataCurrentPageToken{} + if err := token.UnmarshalBinary(payload); err != nil { + return nil, ierror.ErrInvalidCommand + } + return token, nil +} + +// IsZero reports whether every bit is clear. +func (v Unsigned128) IsZero() bool { + return v.High == 0 && v.Low == 0 +} + +func (v Unsigned128) String() string { + return fmt.Sprintf("%016x%016x", v.High, v.Low) +} + +// String renders the cursor's text form. +func (c MetadataChangeCursor) String() string { + var buffer [MetadataChangeCursorSize]byte + payload, _ := c.AppendBinary(buffer[:0]) + return MetadataChangeCursorPrefix + base64.RawURLEncoding.EncodeToString(payload) +} + +// AppendBinary appends the 44 wire bytes: +// [log_id:u128][op:u64][checksum:u128][ordinal:u32], little-endian. +func (c MetadataChangeCursor) AppendBinary(b []byte) ([]byte, error) { + b = appendUnsigned128(b, c.LogID) + b = binary.LittleEndian.AppendUint64(b, c.Op) + b = appendUnsigned128(b, c.Checksum) + return binary.LittleEndian.AppendUint32(b, c.Ordinal), nil +} + +// UnmarshalBinary reads exactly the 44 wire bytes and refuses a cursor at op +// 0 that carries a checksum or an ordinal. +func (c *MetadataChangeCursor) UnmarshalBinary(data []byte) error { + if len(data) != MetadataChangeCursorSize { + return fmt.Errorf("%w: a change cursor is %d bytes, got %d", + ierror.ErrInvalidFormat, MetadataChangeCursorSize, len(data)) + } + cursor := MetadataChangeCursor{ + LogID: readUnsigned128(data[0:16]), + Op: binary.LittleEndian.Uint64(data[16:24]), + Checksum: readUnsigned128(data[24:40]), + Ordinal: binary.LittleEndian.Uint32(data[40:44]), + } + if !cursor.isCanonical() { + return fmt.Errorf("%w: a cursor at op 0 carries no checksum and no ordinal", ierror.ErrInvalidFormat) + } + *c = cursor + return nil +} + +// String renders the token's text form. +func (t *MetadataCurrentPageToken) String() string { + payload, _ := t.AppendBinary(make([]byte, 0, pageTokenFixedSize+len(t.lastKey))) + return MetadataCurrentPageTokenPrefix + base64.RawURLEncoding.EncodeToString(payload) +} + +// MarshalText renders the token's text form, so text encoders such as +// encoding/json keep the scan position. +func (t *MetadataCurrentPageToken) MarshalText() ([]byte, error) { + return []byte(t.String()), nil +} + +// UnmarshalText reads the text form with the checks of +// ParseMetadataCurrentPageToken. +func (t *MetadataCurrentPageToken) UnmarshalText(text []byte) error { + token, err := ParseMetadataCurrentPageToken(string(text)) + if err != nil { + return err + } + *t = *token + return nil +} + +// Anchor is the cursor captured before the scan's first page. +func (t *MetadataCurrentPageToken) Anchor() MetadataChangeCursor { + return t.anchor +} + +// RequiredHead is the highest head any page of the scan reported so far. +func (t *MetadataCurrentPageToken) RequiredHead() uint64 { + return t.requiredHead +} + +// Groups are the groups the scan was started with, in code order. +func (t *MetadataCurrentPageToken) Groups() []MetadataChangeGroup { + return groupsOfMask(t.groupsMask) +} + +// LastGroup is the group of the last item returned. +func (t *MetadataCurrentPageToken) LastGroup() MetadataChangeGroup { + return t.group +} + +// LastKey is a copy of the key of the last item returned, in the page token +// key layout of its group. +func (t *MetadataCurrentPageToken) LastKey() []byte { + return append([]byte(nil), t.lastKey...) +} + +// AppendBinary appends the token payload: +// [anchor:44][required_head:u64][groups_mask:u8][group:u8][key_len:u16][last_key]. +func (t *MetadataCurrentPageToken) AppendBinary(b []byte) ([]byte, error) { + b, _ = t.anchor.AppendBinary(b) + b = binary.LittleEndian.AppendUint64(b, t.requiredHead) + b = append(b, t.groupsMask, byte(t.group)) + b = binary.LittleEndian.AppendUint16(b, uint16(len(t.lastKey))) + return append(b, t.lastKey...), nil +} + +// UnmarshalBinary reads a token that fills data exactly and keeps its own +// invariants: a known group inside the mask, a head at or above the anchor and +// a key of that group's exact width. +func (t *MetadataCurrentPageToken) UnmarshalBinary(data []byte) error { + if len(data) < pageTokenFixedSize { + return fmt.Errorf("%w: a page token is at least %d bytes, got %d", + ierror.ErrInvalidFormat, pageTokenFixedSize, len(data)) + } + var anchor MetadataChangeCursor + if err := anchor.UnmarshalBinary(data[:MetadataChangeCursorSize]); err != nil { + return err + } + group := MetadataChangeGroup(data[MetadataChangeCursorSize+9]) + if !group.IsKnown() { + return fmt.Errorf("%w: page token names unknown group %d", ierror.ErrInvalidFormat, uint8(group)) + } + keyLength := int(binary.LittleEndian.Uint16(data[MetadataChangeCursorSize+10:])) + if keyLength > maxPageTokenKeySize || len(data) != pageTokenFixedSize+keyLength { + return fmt.Errorf("%w: page token key of %d bytes does not fill a %d byte token", + ierror.ErrInvalidFormat, keyLength, len(data)) + } + token := MetadataCurrentPageToken{ + anchor: anchor, + requiredHead: binary.LittleEndian.Uint64(data[MetadataChangeCursorSize:]), + groupsMask: data[MetadataChangeCursorSize+8], + group: group, + lastKey: append([]byte(nil), data[pageTokenFixedSize:]...), + } + if err := token.validate(); err != nil { + return err + } + *t = token + return nil +} + +func (c MetadataChangeCursor) isCanonical() bool { + return c.Op != 0 || (c.Checksum.IsZero() && c.Ordinal == 0) +} + +func (t *MetadataCurrentPageToken) validate() error { + switch { + case t.groupsMask&t.group.maskBit() == 0: + return fmt.Errorf("%w: page token group is not in its group mask", ierror.ErrInvalidFormat) + case t.requiredHead < t.anchor.Op: + return fmt.Errorf("%w: page token head is below its anchor", ierror.ErrInvalidFormat) + case !keyFitsGroup(t.group, t.lastKey): + return fmt.Errorf("%w: page token key of %d bytes does not fit group %s", + ierror.ErrInvalidFormat, len(t.lastKey), t.group) + } + return nil +} + +func keyFitsGroup(group MetadataChangeGroup, key []byte) bool { + switch group { + case MetadataChangeGroupStreams, MetadataChangeGroupUsers: + return len(key) == 4 + case MetadataChangeGroupTopics, MetadataChangeGroupPartitions, MetadataChangeGroupSessions: + return len(key) == 8 + case MetadataChangeGroupConsumerGroups: + return len(key) == 12 + case MetadataChangeGroupConsumerGroupMembers: + return len(key) == 16 + case MetadataChangeGroupPersonalAccessTokens: + if len(key) <= 5 { + return false + } + nameLength := int(key[4]) + return nameLength > 0 && len(key) == 5+nameLength && utf8.Valid(key[5:]) + default: + return false + } +} + +// decodeCanonicalBase64 strips prefix and decodes unpadded Base64url that +// re-encodes to the same text, so one value has exactly one text form. +func decodeCanonicalBase64(text, prefix string) ([]byte, error) { + encoded, found := strings.CutPrefix(text, prefix) + if !found { + return nil, ierror.ErrInvalidCommand + } + payload, err := base64.RawURLEncoding.Strict().DecodeString(encoded) + if err != nil || base64.RawURLEncoding.EncodeToString(payload) != encoded { + return nil, ierror.ErrInvalidCommand + } + return payload, nil +} + +func appendUnsigned128(b []byte, value Unsigned128) []byte { + b = binary.LittleEndian.AppendUint64(b, value.Low) + return binary.LittleEndian.AppendUint64(b, value.High) +} + +func readUnsigned128(data []byte) Unsigned128 { + return Unsigned128{ + Low: binary.LittleEndian.Uint64(data[0:8]), + High: binary.LittleEndian.Uint64(data[8:16]), + } +}
diff --git a/foreign/go/contracts/metadata_changes.go b/foreign/go/contracts/metadata_changes.go new file mode 100644 index 0000000..14ee9ed --- /dev/null +++ b/foreign/go/contracts/metadata_changes.go
@@ -0,0 +1,762 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package iggcon + +import ( + "fmt" + + ierror "github.com/apache/iggy/foreign/go/errors" +) + +const ( + // DefaultMetadataChangesLimit is the page size NewMetadataChangesQuery + // selects. + DefaultMetadataChangesLimit uint32 = 100 + // MaxMetadataChangesLimit is the most records or items one page returns. + MaxMetadataChangesLimit uint32 = 1000 +) + +// MetadataChangeGroup is the family of change records and the unit of +// filtering. Codes 9 to 127 are reserved for future groups, 128 to 255 for +// downstream extensions. +type MetadataChangeGroup uint8 + +const ( + MetadataChangeGroupStreams MetadataChangeGroup = 1 + MetadataChangeGroupTopics MetadataChangeGroup = 2 + MetadataChangeGroupPartitions MetadataChangeGroup = 3 + MetadataChangeGroupConsumerGroups MetadataChangeGroup = 4 + MetadataChangeGroupConsumerGroupMembers MetadataChangeGroup = 5 + MetadataChangeGroupUsers MetadataChangeGroup = 6 + MetadataChangeGroupPersonalAccessTokens MetadataChangeGroup = 7 + MetadataChangeGroupSessions MetadataChangeGroup = 8 +) + +// MetadataChangeKind is the kind of a change record. Existing kind bodies +// never change in place, a new layout is a new kind. +type MetadataChangeKind uint8 + +const ( + MetadataChangeKindStreamCreated MetadataChangeKind = 1 + MetadataChangeKindStreamUpdated MetadataChangeKind = 2 + MetadataChangeKindStreamDeleted MetadataChangeKind = 3 + MetadataChangeKindTopicCreated MetadataChangeKind = 10 + MetadataChangeKindTopicUpdated MetadataChangeKind = 11 + MetadataChangeKindTopicDeleted MetadataChangeKind = 12 + MetadataChangeKindPartitionsChanged MetadataChangeKind = 20 + MetadataChangeKindConsumerGroupCreated MetadataChangeKind = 30 + MetadataChangeKindConsumerGroupDeleted MetadataChangeKind = 31 + MetadataChangeKindConsumerGroupMemberJoined MetadataChangeKind = 40 + MetadataChangeKindConsumerGroupMemberLeft MetadataChangeKind = 41 + MetadataChangeKindConsumerGroupPartitionHandedOver MetadataChangeKind = 42 + MetadataChangeKindUserCreated MetadataChangeKind = 50 + MetadataChangeKindUserUpdated MetadataChangeKind = 51 + MetadataChangeKindUserDeleted MetadataChangeKind = 52 + MetadataChangeKindUserPasswordChanged MetadataChangeKind = 53 + MetadataChangeKindUserPermissionsUpdated MetadataChangeKind = 54 + MetadataChangeKindPersonalAccessTokenCreated MetadataChangeKind = 60 + MetadataChangeKindPersonalAccessTokenDeleted MetadataChangeKind = 61 + MetadataChangeKindSessionRegistered MetadataChangeKind = 70 + MetadataChangeKindSessionClosed MetadataChangeKind = 71 +) + +// MetadataChangeStatus is the outcome of a change log query. +type MetadataChangeStatus uint8 + +const ( + // MetadataChangeStatusOk carries the records or items after the requested + // position. + MetadataChangeStatusOk MetadataChangeStatus = 0 + // MetadataChangeStatusBehind means the answering node has not applied the + // requested position yet. Ask again later or ask another node. + MetadataChangeStatusBehind MetadataChangeStatus = 1 + // MetadataChangeStatusExpired means the node no longer holds every record + // after the position. Rebuild from current state. + MetadataChangeStatusExpired MetadataChangeStatus = 2 + // MetadataChangeStatusReset means the position belongs to another history. + // Rebuild from current state. + MetadataChangeStatusReset MetadataChangeStatus = 3 +) + +// MetadataChangePhase says whether a page holds change records or current +// state items. +type MetadataChangePhase uint8 + +const ( + MetadataChangePhaseChanges MetadataChangePhase = 0 + MetadataChangePhaseCurrent MetadataChangePhase = 1 +) + +// MetadataMemberLeftReason says why a consumer group member was removed. +type MetadataMemberLeftReason uint8 + +const ( + MetadataMemberLeftReasonLeft MetadataMemberLeftReason = 0 + MetadataMemberLeftReasonSessionEnded MetadataMemberLeftReason = 1 +) + +// MetadataTokenDeletedReason says why a personal access token was removed. +type MetadataTokenDeletedReason uint8 + +const ( + MetadataTokenDeletedReasonDeleted MetadataTokenDeletedReason = 0 + MetadataTokenDeletedReasonExpired MetadataTokenDeletedReason = 1 +) + +// MetadataSessionClosedReason says why a session was closed. +type MetadataSessionClosedReason uint8 + +const ( + MetadataSessionClosedReasonLogout MetadataSessionClosedReason = 0 + MetadataSessionClosedReasonExpired MetadataSessionClosedReason = 1 + // MetadataSessionClosedReasonRevoked is a session ended because its user + // was deactivated or deleted. + MetadataSessionClosedReasonRevoked MetadataSessionClosedReason = 2 +) + +// MetadataPermissionsUpdatedReason says why a user's public permission set +// changed. +type MetadataPermissionsUpdatedReason uint8 + +const ( + // MetadataPermissionsUpdatedReasonUpdated is an UpdatePermissions. + MetadataPermissionsUpdatedReasonUpdated MetadataPermissionsUpdatedReason = 0 + // MetadataPermissionsUpdatedReasonResourceDeleted is a stream or topic + // delete that pruned the user's grants on it. + MetadataPermissionsUpdatedReasonResourceDeleted MetadataPermissionsUpdatedReason = 1 +) + +// MetadataChangeStartMode says where a change log query starts. +type MetadataChangeStartMode uint8 + +const ( + // MetadataChangeStartEarliest starts at the oldest operation the requested + // groups still hold completely. + MetadataChangeStartEarliest MetadataChangeStartMode = 0 + // MetadataChangeStartLatest returns no records, only a cursor at the + // newest retained record. + MetadataChangeStartLatest MetadataChangeStartMode = 1 + // MetadataChangeStartAfter returns the records after a cursor. + MetadataChangeStartAfter MetadataChangeStartMode = 2 + // MetadataChangeStartCurrent pages the current state instead of changes. + MetadataChangeStartCurrent MetadataChangeStartMode = 3 +) + +// MetadataChangeStart is where a change log query starts. Build it with +// EarliestMetadataChangeStart, LatestMetadataChangeStart, +// AfterMetadataChangeStart or CurrentMetadataChangeStart. +type MetadataChangeStart struct { + Mode MetadataChangeStartMode + // Cursor is the position MetadataChangeStartAfter continues from. + Cursor MetadataChangeCursor + // PageToken continues a MetadataChangeStartCurrent scan, nil starts one. + PageToken *MetadataCurrentPageToken +} + +// MetadataChangesQuery is one request for a page of the change log or of the +// current state. NewMetadataChangesQuery fills the default limit: a zero +// Limit is refused. +type MetadataChangesQuery struct { + Start MetadataChangeStart + // Groups selects the record families, empty means every group. + Groups []MetadataChangeGroup + // Since keeps records stamped at or after it, in microseconds since the + // Unix epoch. Only for the earliest and after starts. + Since *uint64 + // Until keeps records stamped at or before it, in microseconds since the + // Unix epoch. Only for the earliest and after starts. + Until *uint64 + // Limit caps the records or items of the page, 1 to + // MaxMetadataChangesLimit. + Limit uint32 +} + +// MetadataChanges is one page of the change log, or of the current state. +type MetadataChanges struct { + Status MetadataChangeStatus + Phase MetadataChangePhase + // LogID is the node's log id, nil before it applied op 1. + LogID *Unsigned128 + // OldestOp is the lowest whole operation from which the requested groups + // are complete. + OldestOp uint64 + // RetainedSince is the timestamp above which the requested groups are + // complete, 0 when nothing was evicted. + RetainedSince uint64 + // HeadOp is the highest metadata operation the node applied. + HeadOp uint64 + // Next is the cursor to send back. Present on ok, echoed on behind, nil + // on expired and reset. + Next *MetadataChangeCursor + // HasMore says more records remain after Next, or another state page + // exists. + HasMore bool + Records []MetadataChangeRecord + Items []MetadataCurrentItem + // CurrentPageToken continues a current state scan, nil once it is + // complete. + CurrentPageToken *MetadataCurrentPageToken +} + +// MetadataChangeRecord is one public result of a committed metadata +// operation: what is true after it. +type MetadataChangeRecord struct { + Op uint64 + Ordinal uint32 + // Checksum is the prepare checksum of Op. + Checksum Unsigned128 + // Timestamp is the prepare timestamp of Op in microseconds. + Timestamp uint64 + // ActorUserID is the acting user, nil for a server action. + ActorUserID *uint32 + // Group and Kind are the wire codes, kept even when this build does not + // know them. + Group MetadataChangeGroup + Kind MetadataChangeKind + // Body is the typed body of Kind, or a *MetadataUnknownBody. + Body MetadataChangeBody +} + +// MetadataCurrentItem is one item of a current state page. +type MetadataCurrentItem struct { + // Group is the wire code, kept even when this build does not know it. + Group MetadataChangeGroup + // Body is the typed body of Group, or a *MetadataUnknownBody. + Body MetadataChangeBody +} + +// MetadataChangeBody is the body of a record or item. Records hold +// *MetadataStreamChange, *MetadataStreamDeleted, *MetadataTopicChange, +// *MetadataTopicDeleted, *MetadataPartitionsChanged, +// *MetadataConsumerGroupCreated, *MetadataConsumerGroupDeleted, +// *MetadataMemberJoined, *MetadataMemberLeft, *MetadataPartitionHandedOver, +// *MetadataUserChange, *MetadataUserIdentity, +// *MetadataUserPermissionsUpdated, *MetadataTokenChange, +// *MetadataTokenDeleted, *MetadataSession or *MetadataSessionClosed. Items +// hold *MetadataStreamChange, *MetadataTopicItem, *MetadataPartitionsItem, +// *MetadataConsumerGroupItem, *MetadataMemberItem, *MetadataUserChange, +// *MetadataTokenChange or *MetadataSession. Both hold *MetadataUnknownBody +// for a group or kind this build does not know. +type MetadataChangeBody interface { + isMetadataChangeBody() +} + +// MetadataStreamKey names a stream. Ids are reused after a delete, the +// creating op is not. +type MetadataStreamKey struct { + StreamID uint32 + StreamCreatedOp uint64 +} + +// MetadataTopicKey names a topic within its stream. +type MetadataTopicKey struct { + MetadataStreamKey + TopicID uint32 + TopicCreatedOp uint64 +} + +// MetadataUserKey names a user, the identity sessions and tokens bind to. +type MetadataUserKey struct { + UserID uint32 + UserCreatedAt uint64 +} + +// MetadataOptions are the public options of a resource after a change, as +// the read commands return them. Unreadable marks stored critical state this +// build cannot read: both maps are then nil and a mirror keeps its previous +// values. +type MetadataOptions struct { + Unreadable bool + // Options are the keys the client set. + Options map[string]HeaderValue + // DerivedOptions are the values the server filled or computes. + DerivedOptions map[string]HeaderValue +} + +// MetadataStreamChange is a stream after a create or update, and a streams +// item. +type MetadataStreamChange struct { + MetadataStreamKey + CreatedAt uint64 + Name string + Options MetadataOptions +} + +// MetadataStreamDeleted is a removed stream. RequestedOp is the op of a +// deferred delete's request. +type MetadataStreamDeleted struct { + MetadataStreamKey + CreatedAt uint64 + Name string + RequestedOp *uint64 +} + +// MetadataTopicChange is a topic after a create or update. +type MetadataTopicChange struct { + MetadataTopicKey + CreatedAt uint64 + Name string + PartitionsCount uint32 + Options MetadataOptions +} + +// MetadataTopicItem is a topics item. Partition counts travel as partitions +// items. +type MetadataTopicItem struct { + MetadataTopicKey + CreatedAt uint64 + Name string + Options MetadataOptions +} + +// MetadataTopicDeleted is a removed topic. +type MetadataTopicDeleted struct { + MetadataTopicKey + CreatedAt uint64 + Name string + RequestedOp *uint64 +} + +// MetadataPartitionsChanged is a partition count change. +// RemovedPartitionIDs lists the deleted ids, ascending, empty on create. +type MetadataPartitionsChanged struct { + MetadataTopicKey + PartitionsCount uint32 + RemovedPartitionIDs []uint32 + RequestedOp *uint64 +} + +// MetadataPartitionsItem is a partitions item. +type MetadataPartitionsItem struct { + MetadataTopicKey + PartitionsCount uint32 +} + +// MetadataConsumerGroupCreated is a created consumer group with the topic's +// resulting group count. +type MetadataConsumerGroupCreated struct { + MetadataTopicKey + GroupID uint32 + Name string + ConsumerGroupsCount uint32 +} + +// MetadataConsumerGroupDeleted is a removed consumer group with the topic's +// resulting group count. +type MetadataConsumerGroupDeleted struct { + MetadataTopicKey + GroupID uint32 + Name string + ConsumerGroupsCount uint32 + RequestedOp *uint64 +} + +// MetadataConsumerGroupItem is a consumer_groups item. +type MetadataConsumerGroupItem struct { + MetadataTopicKey + GroupID uint32 + Name string +} + +// MetadataMemberJoined is a member that joined, with its installed +// partitions. +type MetadataMemberJoined struct { + MetadataTopicKey + GroupID uint32 + MemberID uint32 + ClientID Unsigned128 + MembersCount uint32 + PartitionIDs []uint32 +} + +// MetadataMemberLeft is a member that left the group. +type MetadataMemberLeft struct { + MetadataTopicKey + GroupID uint32 + MemberID uint32 + ClientID Unsigned128 + MembersCount uint32 + Reason MetadataMemberLeftReason +} + +// MetadataPartitionHandedOver is a completed partition handover. MemberID is +// nil when the last member left. +type MetadataPartitionHandedOver struct { + MetadataTopicKey + GroupID uint32 + PartitionID uint32 + MemberID *uint32 +} + +// MetadataMemberItem is a consumer_group_members item. +type MetadataMemberItem struct { + MetadataTopicKey + GroupID uint32 + MemberID uint32 + ClientID Unsigned128 + PartitionIDs []uint32 +} + +// MetadataUserChange is a user after a create or update, and a users item. +type MetadataUserChange struct { + MetadataUserKey + Username string + Status UserStatus + Permissions *Permissions + Options MetadataOptions +} + +// MetadataUserIdentity is a deleted user, or one whose password changed. +type MetadataUserIdentity struct { + MetadataUserKey + Username string +} + +// MetadataUserPermissionsUpdated is a user's public permission set after a +// change. +type MetadataUserPermissionsUpdated struct { + MetadataUserKey + Username string + Permissions *Permissions + Reason MetadataPermissionsUpdatedReason +} + +// MetadataTokenChange is a created personal access token, and a +// personal_access_tokens item. Expiry in microseconds is nil for a token that +// never expires. +type MetadataTokenChange struct { + MetadataUserKey + Name string + Expiry *uint64 +} + +// MetadataTokenDeleted is a removed personal access token. +type MetadataTokenDeleted struct { + MetadataUserKey + Name string + Reason MetadataTokenDeletedReason +} + +// MetadataSession is a registered session, and a sessions item. +type MetadataSession struct { + MetadataUserKey + ClientID Unsigned128 + Session uint64 +} + +// MetadataSessionClosed is a closed session. +type MetadataSessionClosed struct { + MetadataUserKey + ClientID Unsigned128 + Session uint64 + Reason MetadataSessionClosedReason +} + +// MetadataUnknownBody is the raw body of a group or kind this build does not +// know. A mirror that cannot apply it stops without advancing its cursor. +type MetadataUnknownBody struct { + Body []byte +} + +var metadataChangeGroupNames = [...]string{ + MetadataChangeGroupStreams: "streams", + MetadataChangeGroupTopics: "topics", + MetadataChangeGroupPartitions: "partitions", + MetadataChangeGroupConsumerGroups: "consumer_groups", + MetadataChangeGroupConsumerGroupMembers: "consumer_group_members", + MetadataChangeGroupUsers: "users", + MetadataChangeGroupPersonalAccessTokens: "personal_access_tokens", + MetadataChangeGroupSessions: "sessions", +} + +type metadataChangeKindEntry struct { + name string + group MetadataChangeGroup +} + +var metadataChangeKinds = map[MetadataChangeKind]metadataChangeKindEntry{ + MetadataChangeKindStreamCreated: {"stream_created", MetadataChangeGroupStreams}, + MetadataChangeKindStreamUpdated: {"stream_updated", MetadataChangeGroupStreams}, + MetadataChangeKindStreamDeleted: {"stream_deleted", MetadataChangeGroupStreams}, + MetadataChangeKindTopicCreated: {"topic_created", MetadataChangeGroupTopics}, + MetadataChangeKindTopicUpdated: {"topic_updated", MetadataChangeGroupTopics}, + MetadataChangeKindTopicDeleted: {"topic_deleted", MetadataChangeGroupTopics}, + MetadataChangeKindPartitionsChanged: {"partitions_changed", MetadataChangeGroupPartitions}, + MetadataChangeKindConsumerGroupCreated: {"consumer_group_created", MetadataChangeGroupConsumerGroups}, + MetadataChangeKindConsumerGroupDeleted: {"consumer_group_deleted", MetadataChangeGroupConsumerGroups}, + MetadataChangeKindConsumerGroupMemberJoined: {"consumer_group_member_joined", MetadataChangeGroupConsumerGroupMembers}, + MetadataChangeKindConsumerGroupMemberLeft: {"consumer_group_member_left", MetadataChangeGroupConsumerGroupMembers}, + MetadataChangeKindConsumerGroupPartitionHandedOver: {"consumer_group_partition_handed_over", MetadataChangeGroupConsumerGroupMembers}, + MetadataChangeKindUserCreated: {"user_created", MetadataChangeGroupUsers}, + MetadataChangeKindUserUpdated: {"user_updated", MetadataChangeGroupUsers}, + MetadataChangeKindUserDeleted: {"user_deleted", MetadataChangeGroupUsers}, + MetadataChangeKindUserPasswordChanged: {"user_password_changed", MetadataChangeGroupUsers}, + MetadataChangeKindUserPermissionsUpdated: {"user_permissions_updated", MetadataChangeGroupUsers}, + MetadataChangeKindPersonalAccessTokenCreated: {"personal_access_token_created", MetadataChangeGroupPersonalAccessTokens}, + MetadataChangeKindPersonalAccessTokenDeleted: {"personal_access_token_deleted", MetadataChangeGroupPersonalAccessTokens}, + MetadataChangeKindSessionRegistered: {"session_registered", MetadataChangeGroupSessions}, + MetadataChangeKindSessionClosed: {"session_closed", MetadataChangeGroupSessions}, +} + +// metadataChangeGroupParents are the groups a current state scan of a group +// must include, so a mirror can apply the cascades of their deletes. +var metadataChangeGroupParents = [...][]MetadataChangeGroup{ + MetadataChangeGroupTopics: {MetadataChangeGroupStreams}, + MetadataChangeGroupPartitions: {MetadataChangeGroupTopics, MetadataChangeGroupStreams}, + MetadataChangeGroupConsumerGroups: { + MetadataChangeGroupTopics, MetadataChangeGroupStreams, + }, + MetadataChangeGroupConsumerGroupMembers: { + MetadataChangeGroupConsumerGroups, MetadataChangeGroupPartitions, + MetadataChangeGroupTopics, MetadataChangeGroupStreams, + }, + MetadataChangeGroupPersonalAccessTokens: {MetadataChangeGroupUsers}, + MetadataChangeGroupSessions: {MetadataChangeGroupUsers}, +} + +var ( + metadataChangeStatusNames = []string{"ok", "behind", "expired", "reset"} + metadataChangePhaseNames = []string{"changes", "current"} + metadataMemberLeftReasonNames = []string{"left", "session_ended"} + metadataTokenDeletedReasonNames = []string{"deleted", "expired"} + metadataSessionClosedReasonNames = []string{"logout", "expired", "revoked"} + metadataPermissionsUpdatedReasonNames = []string{"updated", "resource_deleted"} +) + +// allMetadataChangeGroups is the group mask of an empty selection. +const allMetadataChangeGroups uint8 = 0xff + +// EarliestMetadataChangeStart starts at the oldest operation the requested +// groups still hold completely. +func EarliestMetadataChangeStart() MetadataChangeStart { + return MetadataChangeStart{Mode: MetadataChangeStartEarliest} +} + +// LatestMetadataChangeStart asks for a cursor at the newest retained record +// and no records. +func LatestMetadataChangeStart() MetadataChangeStart { + return MetadataChangeStart{Mode: MetadataChangeStartLatest} +} + +// AfterMetadataChangeStart asks for the records after cursor. +func AfterMetadataChangeStart(cursor MetadataChangeCursor) MetadataChangeStart { + return MetadataChangeStart{Mode: MetadataChangeStartAfter, Cursor: cursor} +} + +// CurrentMetadataChangeStart pages the current state: a nil token starts a +// scan, the token of the previous page continues it. +func CurrentMetadataChangeStart(token *MetadataCurrentPageToken) MetadataChangeStart { + return MetadataChangeStart{Mode: MetadataChangeStartCurrent, PageToken: token} +} + +// NewMetadataChangesQuery builds a query with the default limit. +func NewMetadataChangesQuery(start MetadataChangeStart, groups ...MetadataChangeGroup) MetadataChangesQuery { + return MetadataChangesQuery{Start: start, Groups: groups, Limit: DefaultMetadataChangesLimit} +} + +// ParseMetadataChangeGroup reads a group name such as "consumer_groups". +func ParseMetadataChangeGroup(name string) (MetadataChangeGroup, error) { + for code, candidate := range metadataChangeGroupNames { + if candidate != "" && candidate == name { + return MetadataChangeGroup(code), nil + } + } + return 0, fmt.Errorf("%w: unknown metadata change group %q", ierror.ErrInvalidCommand, name) +} + +// ParseMetadataChangeKind reads a kind name such as "stream_created". +func ParseMetadataChangeKind(name string) (MetadataChangeKind, error) { + for kind, entry := range metadataChangeKinds { + if entry.name == name { + return kind, nil + } + } + return 0, fmt.Errorf("%w: unknown metadata change kind %q", ierror.ErrInvalidCommand, name) +} + +// Validate applies the request rules the server enforces, so a query that +// breaks one is refused with ierror.ErrInvalidCommand before anything is +// sent: a limit outside 1 to MaxMetadataChangesLimit, Since above Until, a +// time bound with the latest or current start, an unknown group or start, a +// cursor at op 0 that carries a checksum or ordinal, a current scan that +// misses a parent group, and a page token issued for other groups. +func (q MetadataChangesQuery) Validate() error { + if q.Limit < 1 || q.Limit > MaxMetadataChangesLimit { + return ierror.ErrInvalidCommand + } + if q.Since != nil && q.Until != nil && *q.Since > *q.Until { + return ierror.ErrInvalidCommand + } + mask, err := requestedGroupsMask(q.Groups) + if err != nil { + return err + } + hasTimeBound := q.Since != nil || q.Until != nil + switch q.Start.Mode { + case MetadataChangeStartEarliest: + return nil + case MetadataChangeStartAfter: + if !q.Start.Cursor.isCanonical() { + return ierror.ErrInvalidCommand + } + return nil + case MetadataChangeStartLatest: + if hasTimeBound { + return ierror.ErrInvalidCommand + } + return nil + case MetadataChangeStartCurrent: + if hasTimeBound || missesParentGroup(mask) { + return ierror.ErrInvalidCommand + } + if q.Start.PageToken != nil && q.Start.PageToken.groupsMask != mask { + return ierror.ErrInvalidCommand + } + return nil + default: + return ierror.ErrInvalidCommand + } +} + +// IsKnown reports whether this build knows the group code. +func (g MetadataChangeGroup) IsKnown() bool { + return g >= MetadataChangeGroupStreams && g <= MetadataChangeGroupSessions +} + +func (g MetadataChangeGroup) String() string { + if g.IsKnown() { + return metadataChangeGroupNames[g] + } + return unknownName(uint8(g)) +} + +// Group is the group every record of the kind belongs to, zero for a kind +// this build does not know. +func (k MetadataChangeKind) Group() MetadataChangeGroup { + return metadataChangeKinds[k].group +} + +func (k MetadataChangeKind) String() string { + if entry, ok := metadataChangeKinds[k]; ok { + return entry.name + } + return unknownName(uint8(k)) +} + +func (s MetadataChangeStatus) String() string { + return enumName(metadataChangeStatusNames, uint8(s)) +} + +func (p MetadataChangePhase) String() string { + return enumName(metadataChangePhaseNames, uint8(p)) +} + +func (r MetadataMemberLeftReason) String() string { + return enumName(metadataMemberLeftReasonNames, uint8(r)) +} + +func (r MetadataTokenDeletedReason) String() string { + return enumName(metadataTokenDeletedReasonNames, uint8(r)) +} + +func (r MetadataSessionClosedReason) String() string { + return enumName(metadataSessionClosedReasonNames, uint8(r)) +} + +func (r MetadataPermissionsUpdatedReason) String() string { + return enumName(metadataPermissionsUpdatedReasonNames, uint8(r)) +} + +func (*MetadataStreamChange) isMetadataChangeBody() {} +func (*MetadataStreamDeleted) isMetadataChangeBody() {} +func (*MetadataTopicChange) isMetadataChangeBody() {} +func (*MetadataTopicItem) isMetadataChangeBody() {} +func (*MetadataTopicDeleted) isMetadataChangeBody() {} +func (*MetadataPartitionsChanged) isMetadataChangeBody() {} +func (*MetadataPartitionsItem) isMetadataChangeBody() {} +func (*MetadataConsumerGroupCreated) isMetadataChangeBody() {} +func (*MetadataConsumerGroupDeleted) isMetadataChangeBody() {} +func (*MetadataConsumerGroupItem) isMetadataChangeBody() {} +func (*MetadataMemberJoined) isMetadataChangeBody() {} +func (*MetadataMemberLeft) isMetadataChangeBody() {} +func (*MetadataPartitionHandedOver) isMetadataChangeBody() {} +func (*MetadataMemberItem) isMetadataChangeBody() {} +func (*MetadataUserChange) isMetadataChangeBody() {} +func (*MetadataUserIdentity) isMetadataChangeBody() {} +func (*MetadataUserPermissionsUpdated) isMetadataChangeBody() {} +func (*MetadataTokenChange) isMetadataChangeBody() {} +func (*MetadataTokenDeleted) isMetadataChangeBody() {} +func (*MetadataSession) isMetadataChangeBody() {} +func (*MetadataSessionClosed) isMetadataChangeBody() {} +func (*MetadataUnknownBody) isMetadataChangeBody() {} + +// maskBit is bit code - 1 of a page token's group mask. +func (g MetadataChangeGroup) maskBit() uint8 { + if !g.IsKnown() { + return 0 + } + return 1 << (g - 1) +} + +// requestedGroupsMask collapses duplicates, with an empty selection meaning +// every group. +func requestedGroupsMask(groups []MetadataChangeGroup) (uint8, error) { + if len(groups) == 0 { + return allMetadataChangeGroups, nil + } + var mask uint8 + for _, group := range groups { + if !group.IsKnown() { + return 0, ierror.ErrInvalidCommand + } + mask |= group.maskBit() + } + return mask, nil +} + +func missesParentGroup(mask uint8) bool { + for _, group := range groupsOfMask(mask) { + for _, parent := range metadataChangeGroupParents[group] { + if mask&parent.maskBit() == 0 { + return true + } + } + } + return false +} + +func groupsOfMask(mask uint8) []MetadataChangeGroup { + groups := make([]MetadataChangeGroup, 0, 8) + for group := MetadataChangeGroupStreams; group <= MetadataChangeGroupSessions; group++ { + if mask&group.maskBit() != 0 { + groups = append(groups, group) + } + } + return groups +} + +func enumName(names []string, value uint8) string { + if int(value) < len(names) { + return names[value] + } + return unknownName(value) +} + +func unknownName(value uint8) string { + return fmt.Sprintf("unknown(%d)", value) +}
diff --git a/foreign/go/contracts/metadata_changes_test.go b/foreign/go/contracts/metadata_changes_test.go new file mode 100644 index 0000000..ba6c052 --- /dev/null +++ b/foreign/go/contracts/metadata_changes_test.go
@@ -0,0 +1,936 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package iggcon_test + +import ( + "encoding/base64" + "encoding/binary" + "encoding/hex" + "encoding/json" + "os" + "path/filepath" + "slices" + "strconv" + "strings" + "testing" + + binaryserialization "github.com/apache/iggy/foreign/go/binary_serialization" + iggcon "github.com/apache/iggy/foreign/go/contracts" + ierror "github.com/apache/iggy/foreign/go/errors" + "github.com/apache/iggy/foreign/go/internal/command" + "github.com/apache/iggy/foreign/go/internal/vsr" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestMetadataChanges_SharedVectors(t *testing.T) { + fixture := loadMetadataChangesFixture(t) + + t.Run("given_shared_vectors_when_decoding_should_match_server_contract", func(t *testing.T) { + t.Run("constants", func(t *testing.T) { checkMetadataChangesConstants(t, fixture.Constants) }) + + for _, vector := range fixture.Records { + t.Run("records/"+vector.Case, func(t *testing.T) { + require.True(t, strings.HasSuffix(vector.RecordHex, vector.BodyHex)) + page := decodeChangesPage(t, decodeHex(t, vector.RecordHex)) + require.Len(t, page.Records, 1) + record := page.Records[0] + assert.Equal(t, vector.Group, uint8(record.Group)) + assert.Equal(t, vector.Kind, uint8(record.Kind)) + assert.JSONEq(t, string(vector.Expected), marshalView(t, recordView(t, record))) + }) + } + for _, vector := range fixture.Items { + t.Run("items/"+vector.Case, func(t *testing.T) { + require.True(t, strings.HasSuffix(vector.ItemHex, vector.BodyHex)) + page := decodeCurrentPage(t, decodeHex(t, vector.ItemHex)) + require.Len(t, page.Items, 1) + assert.Equal(t, vector.Group, uint8(page.Items[0].Group)) + assert.JSONEq(t, string(vector.Expected), marshalView(t, itemView(t, page.Items[0]))) + }) + } + for _, vector := range fixture.Cursors { + t.Run("cursors/"+vector.Case, func(t *testing.T) { checkCursorVector(t, vector) }) + } + for _, vector := range fixture.PageTokens { + t.Run("page_tokens/"+vector.Case, func(t *testing.T) { checkPageTokenVector(t, vector) }) + } + // The Go SDK has no HTTP transport, so http_query and the refused HTTP + // queries have no encoder to check. + for _, vector := range fixture.Requests { + t.Run("requests/"+vector.Case, func(t *testing.T) { + body, err := (&command.GetMetadataChanges{Query: fixtureQuery(t, vector.Input)}).MarshalBinary() + require.NoError(t, err) + assert.Equal(t, vector.RequestHex, hex.EncodeToString(body)) + }) + } + for _, vector := range fixture.Responses { + t.Run("responses/"+vector.Case, func(t *testing.T) { + changes, err := binaryserialization.DeserializeMetadataChanges(decodeHex(t, vector.ResponseHex)) + require.NoError(t, err) + assert.JSONEq(t, string(vector.Expected), marshalView(t, changesView(t, changes))) + }) + } + t.Run("large_values", func(t *testing.T) { checkLargeValues(t, fixture.LargeValues) }) + }) + + t.Run("given_refused_vectors_when_decoding_or_validating_should_refuse", func(t *testing.T) { + for _, vector := range fixture.Refused.Cursors { + t.Run("cursors/"+vector.Case, func(t *testing.T) { + _, err := iggcon.ParseMetadataChangeCursor(vector.Text) + assert.ErrorIs(t, err, ierror.ErrInvalidCommand) + }) + } + for _, vector := range fixture.Refused.PageTokens { + t.Run("page_tokens/"+vector.Case, func(t *testing.T) { + _, err := iggcon.ParseMetadataCurrentPageToken(vector.Text) + assert.ErrorIs(t, err, ierror.ErrInvalidCommand) + }) + } + for _, vector := range fixture.Refused.RecordBodies { + t.Run("record_bodies/"+vector.Case, func(t *testing.T) { + record := recordEnvelope(vector.Group, vector.Kind, decodeHex(t, vector.BodyHex)) + _, err := binaryserialization.DeserializeMetadataChanges(changesPagePayload(record)) + assert.ErrorIs(t, err, ierror.ErrInvalidFormat) + }) + } + for _, vector := range fixture.Refused.Requests { + t.Run("requests/"+vector.Case, func(t *testing.T) { checkRefusedRequest(t, vector.Case, decodeHex(t, vector.Hex)) }) + } + for _, vector := range fixture.Refused.Responses { + t.Run("responses/"+vector.Case, func(t *testing.T) { + _, err := binaryserialization.DeserializeMetadataChanges(decodeHex(t, vector.Hex)) + assert.ErrorIs(t, err, ierror.ErrInvalidFormat) + }) + } + for _, vector := range fixture.Refused.Queries { + t.Run("queries/"+vector.Case, func(t *testing.T) { + query := fixtureQuery(t, vector.Input) + assert.ErrorIs(t, query.Validate(), ierror.ErrInvalidCommand) + body, err := (&command.GetMetadataChanges{Query: query}).MarshalBinary() + assert.ErrorIs(t, err, ierror.ErrInvalidCommand) + assert.Empty(t, body) + }) + } + }) +} + +func TestMetadataChanges_Positions(t *testing.T) { + fixture := loadMetadataChangesFixture(t) + tokenText := fixture.PageTokens[0].Text + + t.Run("given_line_breaks_inside_a_token_when_parsing_should_refuse", func(t *testing.T) { + broken := tokenText[:20] + "\n" + tokenText[20:] + _, err := iggcon.ParseMetadataCurrentPageToken(broken) + assert.ErrorIs(t, err, ierror.ErrInvalidCommand) + }) + + t.Run("given_token_longer_than_the_longest_valid_when_parsing_should_refuse", func(t *testing.T) { + longest := fixture.PageTokens[len(fixture.PageTokens)-1].Text + _, err := iggcon.ParseMetadataCurrentPageToken(longest) + require.NoError(t, err) + _, err = iggcon.ParseMetadataCurrentPageToken(longest + "AAAA") + assert.ErrorIs(t, err, ierror.ErrInvalidCommand) + }) + + t.Run("given_parsed_token_when_changing_its_last_key_copy_should_keep_the_token", func(t *testing.T) { + token, err := iggcon.ParseMetadataCurrentPageToken(tokenText) + require.NoError(t, err) + key := token.LastKey() + key[0] ^= 0xff + assert.Equal(t, tokenText, token.String()) + }) + + t.Run("given_page_token_when_round_tripping_through_json_should_keep_its_text_form", func(t *testing.T) { + token, err := iggcon.ParseMetadataCurrentPageToken(tokenText) + require.NoError(t, err) + encoded, err := json.Marshal(iggcon.MetadataChanges{CurrentPageToken: token}) + require.NoError(t, err) + assert.Contains(t, string(encoded), `"CurrentPageToken":"`+tokenText+`"`) + var decoded iggcon.MetadataChanges + require.NoError(t, json.Unmarshal(encoded, &decoded)) + assert.Equal(t, token, decoded.CurrentPageToken) + }) + + t.Run("given_non_canonical_page_token_text_when_unmarshaling_json_should_refuse", func(t *testing.T) { + var token iggcon.MetadataCurrentPageToken + err := json.Unmarshal([]byte(`"`+tokenText+`="`), &token) + assert.ErrorIs(t, err, ierror.ErrInvalidCommand) + }) + + t.Run("given_cursor_with_last_character_carrying_extra_bits_when_parsing_should_refuse", func(t *testing.T) { + text := fixture.Cursors[0].Text + last := text[len(text)-1] + require.Equal(t, byte('A'), last) + _, err := iggcon.ParseMetadataChangeCursor(text[:len(text)-1] + "B") + assert.ErrorIs(t, err, ierror.ErrInvalidCommand) + }) + + t.Run("given_wrong_cursor_length_when_unmarshaling_should_refuse", func(t *testing.T) { + var cursor iggcon.MetadataChangeCursor + assert.ErrorIs(t, cursor.UnmarshalBinary(make([]byte, iggcon.MetadataChangeCursorSize-1)), ierror.ErrInvalidFormat) + assert.ErrorIs(t, cursor.UnmarshalBinary(make([]byte, iggcon.MetadataChangeCursorSize+1)), ierror.ErrInvalidFormat) + assert.NoError(t, cursor.UnmarshalBinary(make([]byte, iggcon.MetadataChangeCursorSize))) + }) +} + +func TestMetadataChanges_Validate(t *testing.T) { + fixture := loadMetadataChangesFixture(t) + streamsToken, err := iggcon.ParseMetadataCurrentPageToken(fixture.PageTokens[0].Text) + require.NoError(t, err) + require.Equal(t, []iggcon.MetadataChangeGroup{iggcon.MetadataChangeGroupStreams}, streamsToken.Groups()) + cursor, err := iggcon.ParseMetadataChangeCursor(fixture.Cursors[2].Text) + require.NoError(t, err) + one := uint64(1) + + accepted := map[string]iggcon.MetadataChangesQuery{ + "limit_one": {Start: iggcon.EarliestMetadataChangeStart(), Limit: 1}, + "limit_maximum": {Start: iggcon.EarliestMetadataChangeStart(), Limit: iggcon.MaxMetadataChangesLimit}, + "equal_time_bounds": {Start: iggcon.AfterMetadataChangeStart(cursor), Since: &one, Until: &one, Limit: 1}, + "zero_cursor": iggcon.NewMetadataChangesQuery(iggcon.AfterMetadataChangeStart(iggcon.MetadataChangeCursor{})), + "duplicate_groups": iggcon.NewMetadataChangesQuery(iggcon.LatestMetadataChangeStart(), iggcon.MetadataChangeGroupUsers, iggcon.MetadataChangeGroupUsers), + "current_token_of_same_group": iggcon.NewMetadataChangesQuery(iggcon.CurrentMetadataChangeStart(streamsToken), iggcon.MetadataChangeGroupStreams), + "current_every_group": iggcon.NewMetadataChangesQuery(iggcon.CurrentMetadataChangeStart(nil)), + "current_members_with_parents": iggcon.NewMetadataChangesQuery(iggcon.CurrentMetadataChangeStart(nil), + iggcon.MetadataChangeGroupConsumerGroupMembers, iggcon.MetadataChangeGroupConsumerGroups, + iggcon.MetadataChangeGroupPartitions, iggcon.MetadataChangeGroupTopics, iggcon.MetadataChangeGroupStreams), + } + for name, query := range accepted { + t.Run("given_"+name+"_when_validating_should_accept", func(t *testing.T) { + assert.NoError(t, query.Validate()) + }) + } + + refused := map[string]iggcon.MetadataChangesQuery{ + "zero_value": {}, + "unknown_group": iggcon.NewMetadataChangesQuery(iggcon.EarliestMetadataChangeStart(), iggcon.MetadataChangeGroup(9)), + "group_zero": iggcon.NewMetadataChangesQuery(iggcon.EarliestMetadataChangeStart(), iggcon.MetadataChangeGroup(0)), + "unknown_start": iggcon.NewMetadataChangesQuery(iggcon.MetadataChangeStart{Mode: 4}), + "current_sessions_alone": iggcon.NewMetadataChangesQuery(iggcon.CurrentMetadataChangeStart(nil), iggcon.MetadataChangeGroupSessions), + "current_token_of_all": iggcon.NewMetadataChangesQuery(iggcon.CurrentMetadataChangeStart(streamsToken)), + "current_zero_value_token": iggcon.NewMetadataChangesQuery(iggcon.CurrentMetadataChangeStart(&iggcon.MetadataCurrentPageToken{}), iggcon.MetadataChangeGroupStreams), + "after_op_zero_with_checks": iggcon.NewMetadataChangesQuery(iggcon.AfterMetadataChangeStart(iggcon.MetadataChangeCursor{Ordinal: 1})), + } + for name, query := range refused { + t.Run("given_"+name+"_when_validating_should_refuse", func(t *testing.T) { + assert.ErrorIs(t, query.Validate(), ierror.ErrInvalidCommand) + }) + } + + t.Run("given_every_group_with_its_parents_when_scanning_current_state_should_accept_only_the_full_set", func(t *testing.T) { + for name, entry := range fixture.Constants.Groups { + group := mustGroup(t, name) + alone := iggcon.NewMetadataChangesQuery(iggcon.CurrentMetadataChangeStart(nil), group) + withParents := alone + for _, parent := range entry.Parents { + withParents.Groups = append(slices.Clone(withParents.Groups), mustGroup(t, parent)) + } + if len(entry.Parents) == 0 { + assert.NoError(t, alone.Validate(), name) + continue + } + assert.ErrorIs(t, alone.Validate(), ierror.ErrInvalidCommand, name) + assert.NoError(t, withParents.Validate(), name) + } + }) +} + +func TestMetadataChanges_Decoding(t *testing.T) { + fixture := loadMetadataChangesFixture(t) + + t.Run("given_every_fixture_response_when_truncating_should_refuse_every_prefix", func(t *testing.T) { + for _, vector := range fixture.Responses { + response := decodeHex(t, vector.ResponseHex) + for length := range len(response) { + _, err := binaryserialization.DeserializeMetadataChanges(response[:length]) + require.ErrorIs(t, err, ierror.ErrInvalidFormat, "%s truncated to %d bytes", vector.Case, length) + } + } + }) + + t.Run("given_every_record_body_when_truncating_or_extending_should_refuse", func(t *testing.T) { + for _, vector := range fixture.Records { + body := decodeHex(t, vector.BodyHex) + for length := range len(body) { + payload := changesPagePayload(recordEnvelope(vector.Group, vector.Kind, body[:length])) + _, err := binaryserialization.DeserializeMetadataChanges(payload) + require.ErrorIs(t, err, ierror.ErrInvalidFormat, "%s truncated to %d bytes", vector.Case, length) + } + payload := changesPagePayload(recordEnvelope(vector.Group, vector.Kind, append(slices.Clone(body), 0))) + _, err := binaryserialization.DeserializeMetadataChanges(payload) + require.ErrorIs(t, err, ierror.ErrInvalidFormat, "%s with a trailing byte", vector.Case) + } + }) + + t.Run("given_huge_record_count_when_decoding_should_refuse_without_allocating_it", func(t *testing.T) { + payload := changesPagePayload() + countAt := len(payload) - 4 - 2 - 4 + binary.LittleEndian.PutUint32(payload[countAt:], ^uint32(0)) + _, err := binaryserialization.DeserializeMetadataChanges(payload) + assert.ErrorIs(t, err, ierror.ErrInvalidFormat) + }) + + t.Run("given_unknown_kind_of_a_known_group_when_decoding_should_keep_the_raw_body", func(t *testing.T) { + body := []byte{0xde, 0xad} + page := decodeChangesPage(t, recordEnvelope(uint8(iggcon.MetadataChangeGroupStreams), 9, body)) + require.Len(t, page.Records, 1) + assert.Equal(t, iggcon.MetadataChangeKind(9), page.Records[0].Kind) + assert.Equal(t, &iggcon.MetadataUnknownBody{Body: body}, page.Records[0].Body) + assert.Equal(t, "unknown(9)", page.Records[0].Kind.String()) + }) + + t.Run("given_unknown_item_group_when_decoding_should_keep_the_raw_body", func(t *testing.T) { + item := append([]byte{200}, binary.LittleEndian.AppendUint32(nil, 3)...) + page := decodeCurrentPage(t, append(item, 1, 2, 3)) + require.Len(t, page.Items, 1) + assert.Equal(t, iggcon.MetadataChangeGroup(200), page.Items[0].Group) + assert.Equal(t, &iggcon.MetadataUnknownBody{Body: []byte{1, 2, 3}}, page.Items[0].Body) + }) +} + +type metadataChangesFixture struct { + SchemaVersion int `json:"schema_version"` + Constants metadataChangesConstants `json:"constants"` + Records []struct { + Case string `json:"case"` + Group uint8 `json:"group"` + Kind uint8 `json:"kind"` + BodyHex string `json:"body_hex"` + RecordHex string `json:"record_hex"` + Expected json.RawMessage `json:"expected"` + } `json:"records"` + Items []struct { + Case string `json:"case"` + Group uint8 `json:"group"` + BodyHex string `json:"body_hex"` + ItemHex string `json:"item_hex"` + Expected json.RawMessage `json:"expected"` + } `json:"items"` + Cursors []cursorVector `json:"cursors"` + PageTokens []pageTokenVector `json:"page_tokens"` + Requests []struct { + Case string `json:"case"` + Input metadataChangesQueryInput `json:"input"` + RequestHex string `json:"request_hex"` + } `json:"requests"` + Responses []struct { + Case string `json:"case"` + ResponseHex string `json:"response_hex"` + Expected json.RawMessage `json:"expected"` + } `json:"responses"` + LargeValues largeValuesVector `json:"large_values"` + Refused struct { + Cursors []textVector `json:"cursors"` + PageTokens []textVector `json:"page_tokens"` + RecordBodies []struct { + Case string `json:"case"` + Group uint8 `json:"group"` + Kind uint8 `json:"kind"` + BodyHex string `json:"body_hex"` + } `json:"record_bodies"` + Requests []hexVector `json:"requests"` + Responses []hexVector `json:"responses"` + Queries []struct { + Case string `json:"case"` + Input metadataChangesQueryInput `json:"input"` + } `json:"queries"` + } `json:"refused"` +} + +type metadataChangesConstants struct { + CommandCode uint32 `json:"command_code"` + CursorPrefix string `json:"cursor_prefix"` + PageTokenPrefix string `json:"page_token_prefix"` + CursorSize int `json:"cursor_size"` + CursorTextLength int `json:"cursor_text_length"` + MaxPageTokenSize int `json:"max_page_token_size"` + DefaultLimit uint32 `json:"default_limit"` + MaxLimit uint32 `json:"max_limit"` + Groups map[string]struct { + Code uint8 `json:"code"` + Parents []string `json:"parents"` + } `json:"groups"` + Kinds map[string]struct { + Code uint8 `json:"code"` + Group string `json:"group"` + } `json:"kinds"` + Statuses map[string]uint8 `json:"statuses"` + Phases map[string]uint8 `json:"phases"` + StartModes map[string]uint8 `json:"start_modes"` + MemberLeftReasons map[string]uint8 `json:"member_left_reasons"` + TokenDeletedReasons map[string]uint8 `json:"token_deleted_reasons"` + SessionClosedReasons map[string]uint8 `json:"session_closed_reasons"` + PermissionsUpdatedReasons map[string]uint8 `json:"permissions_updated_reasons"` +} + +type cursorVector struct { + Case string `json:"case"` + Text string `json:"text"` + Hex string `json:"hex"` + LogID string `json:"log_id"` + Op uint64 `json:"op"` + Checksum string `json:"checksum"` + Ordinal uint32 `json:"ordinal"` +} + +type pageTokenVector struct { + Case string `json:"case"` + Text string `json:"text"` + Hex string `json:"hex"` + Anchor string `json:"anchor"` + RequiredHead uint64 `json:"required_head"` + GroupsMask uint8 `json:"groups_mask"` + Group string `json:"group"` + LastKeyHex string `json:"last_key_hex"` +} + +type metadataChangesQueryInput struct { + Start string `json:"start"` + Cursor *string `json:"cursor"` + PageToken *string `json:"page_token"` + Groups []string `json:"groups"` + Since *uint64 `json:"since"` + Until *uint64 `json:"until"` + Limit uint32 `json:"limit"` +} + +type largeValuesVector struct { + RecordHex string `json:"record_hex"` + Op string `json:"op"` + Ordinal uint32 `json:"ordinal"` + Timestamp string `json:"timestamp"` + Checksum string `json:"checksum"` + UserCreatedAt string `json:"user_created_at"` + Session string `json:"session"` + ClientID string `json:"client_id"` +} + +type textVector struct { + Case string `json:"case"` + Text string `json:"text"` +} + +type hexVector struct { + Case string `json:"case"` + Hex string `json:"hex"` +} + +func loadMetadataChangesFixture(t *testing.T) metadataChangesFixture { + t.Helper() + content, err := os.ReadFile(filepath.Join("testdata", "metadata_changes.json")) + require.NoError(t, err) + var fixture metadataChangesFixture + require.NoError(t, json.Unmarshal(content, &fixture)) + require.Equal(t, 1, fixture.SchemaVersion) + return fixture +} + +func checkMetadataChangesConstants(t *testing.T, constants metadataChangesConstants) { + assert.Equal(t, constants.CommandCode, uint32(command.GetMetadataChangesCode)) + assert.Equal(t, vsr.OperationNonReplicated, vsr.OperationForCode(constants.CommandCode)) + assert.Equal(t, constants.CursorPrefix, iggcon.MetadataChangeCursorPrefix) + assert.Equal(t, constants.PageTokenPrefix, iggcon.MetadataCurrentPageTokenPrefix) + assert.Equal(t, constants.CursorSize, iggcon.MetadataChangeCursorSize) + assert.Len(t, iggcon.MetadataChangeCursor{}.String(), constants.CursorTextLength) + assert.Equal(t, constants.DefaultLimit, iggcon.DefaultMetadataChangesLimit) + assert.Equal(t, constants.DefaultLimit, iggcon.NewMetadataChangesQuery(iggcon.EarliestMetadataChangeStart()).Limit) + assert.Equal(t, constants.MaxLimit, iggcon.MaxMetadataChangesLimit) + + for name, entry := range constants.Groups { + group := mustGroup(t, name) + assert.Equal(t, entry.Code, uint8(group), name) + assert.Equal(t, name, group.String()) + assert.True(t, group.IsKnown(), name) + } + for code := range 256 { + group := iggcon.MetadataChangeGroup(code) + _, listed := constants.Groups[group.String()] + assert.Equal(t, listed, group.IsKnown(), "group code %d", code) + } + for name, entry := range constants.Kinds { + kind, err := iggcon.ParseMetadataChangeKind(name) + require.NoError(t, err, name) + assert.Equal(t, entry.Code, uint8(kind), name) + assert.Equal(t, name, kind.String()) + assert.Equal(t, entry.Group, kind.Group().String(), name) + } + known := 0 + for code := range 256 { + if iggcon.MetadataChangeKind(code).Group().IsKnown() { + known++ + } + } + assert.Equal(t, len(constants.Kinds), known, "every kind this build knows is listed") + _, err := iggcon.ParseMetadataChangeKind("stream_renamed") + assert.ErrorIs(t, err, ierror.ErrInvalidCommand) + _, err = iggcon.ParseMetadataChangeGroup("everything") + assert.ErrorIs(t, err, ierror.ErrInvalidCommand) + + checkNames(t, constants.Statuses, func(code uint8) string { return iggcon.MetadataChangeStatus(code).String() }) + checkNames(t, constants.Phases, func(code uint8) string { return iggcon.MetadataChangePhase(code).String() }) + checkNames(t, constants.MemberLeftReasons, func(code uint8) string { return iggcon.MetadataMemberLeftReason(code).String() }) + checkNames(t, constants.TokenDeletedReasons, func(code uint8) string { return iggcon.MetadataTokenDeletedReason(code).String() }) + checkNames(t, constants.SessionClosedReasons, func(code uint8) string { return iggcon.MetadataSessionClosedReason(code).String() }) + checkNames(t, constants.PermissionsUpdatedReasons, func(code uint8) string { + return iggcon.MetadataPermissionsUpdatedReason(code).String() + }) + assert.Equal(t, map[string]uint8{ + "earliest": uint8(iggcon.MetadataChangeStartEarliest), + "latest": uint8(iggcon.MetadataChangeStartLatest), + "after": uint8(iggcon.MetadataChangeStartAfter), + "current": uint8(iggcon.MetadataChangeStartCurrent), + }, constants.StartModes) +} + +func checkNames(t *testing.T, names map[string]uint8, name func(uint8) string) { + t.Helper() + for want, code := range names { + assert.Equal(t, want, name(code), "code %d", code) + } + assert.Equal(t, "unknown("+strconv.Itoa(len(names))+")", name(uint8(len(names)))) +} + +func checkCursorVector(t *testing.T, vector cursorVector) { + cursor, err := iggcon.ParseMetadataChangeCursor(vector.Text) + require.NoError(t, err) + assert.Equal(t, vector.LogID, cursor.LogID.String()) + assert.Equal(t, vector.Op, cursor.Op) + assert.Equal(t, vector.Checksum, cursor.Checksum.String()) + assert.Equal(t, vector.Ordinal, cursor.Ordinal) + assert.Equal(t, vector.Text, cursor.String()) + + encoded, err := cursor.AppendBinary(nil) + require.NoError(t, err) + assert.Equal(t, vector.Hex, hex.EncodeToString(encoded)) + var decoded iggcon.MetadataChangeCursor + require.NoError(t, decoded.UnmarshalBinary(decodeHex(t, vector.Hex))) + assert.Equal(t, cursor, decoded) +} + +func checkPageTokenVector(t *testing.T, vector pageTokenVector) { + token, err := iggcon.ParseMetadataCurrentPageToken(vector.Text) + require.NoError(t, err) + assert.Equal(t, vector.Anchor, token.Anchor().String()) + assert.Equal(t, vector.RequiredHead, token.RequiredHead()) + var mask uint8 + for _, group := range token.Groups() { + mask |= 1 << (uint8(group) - 1) + } + assert.Equal(t, vector.GroupsMask, mask) + assert.Equal(t, vector.Group, token.LastGroup().String()) + assert.Equal(t, vector.LastKeyHex, hex.EncodeToString(token.LastKey())) + assert.Equal(t, vector.Text, token.String()) + + encoded, err := token.AppendBinary(nil) + require.NoError(t, err) + assert.Equal(t, vector.Hex, hex.EncodeToString(encoded)) + decoded := &iggcon.MetadataCurrentPageToken{} + require.NoError(t, decoded.UnmarshalBinary(decodeHex(t, vector.Hex))) + assert.Equal(t, token, decoded) +} + +func checkLargeValues(t *testing.T, vector largeValuesVector) { + page := decodeChangesPage(t, decodeHex(t, vector.RecordHex)) + require.Len(t, page.Records, 1) + record := page.Records[0] + session, ok := record.Body.(*iggcon.MetadataSession) + require.True(t, ok, "%T", record.Body) + assert.Equal(t, vector.Op, strconv.FormatUint(record.Op, 10)) + assert.Equal(t, vector.Ordinal, record.Ordinal) + assert.Equal(t, vector.Timestamp, strconv.FormatUint(record.Timestamp, 10)) + assert.Equal(t, vector.Checksum, record.Checksum.String()) + assert.Equal(t, vector.UserCreatedAt, strconv.FormatUint(session.UserCreatedAt, 10)) + assert.Equal(t, vector.Session, strconv.FormatUint(session.Session, 10)) + assert.Equal(t, vector.ClientID, session.ClientID.String()) +} + +// checkRefusedRequest proves the encoder cannot produce a request the server +// refuses: each vector names a shape the query either never encodes or +// refuses before sending. +func checkRefusedRequest(t *testing.T, name string, request []byte) { + t.Helper() + earliest := iggcon.NewMetadataChangesQuery(iggcon.EarliestMetadataChangeStart()) + switch name { + case "cursor_bytes_without_after": + var cursor iggcon.MetadataChangeCursor + require.NoError(t, cursor.UnmarshalBinary(request[1:1+iggcon.MetadataChangeCursorSize])) + earliest.Start.Cursor = cursor + body, err := (&command.GetMetadataChanges{Query: earliest}).MarshalBinary() + require.NoError(t, err) + assert.NotEqual(t, request, body) + assert.Equal(t, make([]byte, iggcon.MetadataChangeCursorSize), body[1:1+iggcon.MetadataChangeCursorSize]) + case "trailing_byte": + body, err := (&command.GetMetadataChanges{Query: earliest}).MarshalBinary() + require.NoError(t, err) + assert.Equal(t, request[:len(request)-1], body) + case "unknown_start_mode": + query := earliest + query.Start.Mode = iggcon.MetadataChangeStartMode(request[0]) + assert.ErrorIs(t, query.Validate(), ierror.ErrInvalidCommand) + case "non_canonical_cursor": + var cursor iggcon.MetadataChangeCursor + assert.ErrorIs(t, cursor.UnmarshalBinary(request[1:1+iggcon.MetadataChangeCursorSize]), ierror.ErrInvalidFormat) + query := earliest + query.Start = iggcon.AfterMetadataChangeStart(iggcon.MetadataChangeCursor{ + LogID: iggcon.Unsigned128{Low: binary.LittleEndian.Uint64(request[1:9]), High: binary.LittleEndian.Uint64(request[9:17])}, + Op: binary.LittleEndian.Uint64(request[17:25]), + Checksum: iggcon.Unsigned128{Low: binary.LittleEndian.Uint64(request[25:33]), High: binary.LittleEndian.Uint64(request[33:41])}, + Ordinal: binary.LittleEndian.Uint32(request[41:45]), + }) + assert.ErrorIs(t, query.Validate(), ierror.ErrInvalidCommand) + default: + require.FailNow(t, "a refused request vector this test does not cover", name) + } +} + +func fixtureQuery(t *testing.T, input metadataChangesQueryInput) iggcon.MetadataChangesQuery { + t.Helper() + query := iggcon.MetadataChangesQuery{Since: input.Since, Until: input.Until, Limit: input.Limit} + for _, name := range input.Groups { + query.Groups = append(query.Groups, mustGroup(t, name)) + } + switch input.Start { + case "earliest": + query.Start = iggcon.EarliestMetadataChangeStart() + case "latest": + query.Start = iggcon.LatestMetadataChangeStart() + case "after": + require.NotNil(t, input.Cursor) + cursor, err := iggcon.ParseMetadataChangeCursor(*input.Cursor) + require.NoError(t, err) + query.Start = iggcon.AfterMetadataChangeStart(cursor) + case "current": + var token *iggcon.MetadataCurrentPageToken + if input.PageToken != nil { + parsed, err := iggcon.ParseMetadataCurrentPageToken(*input.PageToken) + require.NoError(t, err) + token = parsed + } + query.Start = iggcon.CurrentMetadataChangeStart(token) + default: + require.FailNow(t, "unknown start", input.Start) + } + return query +} + +func mustGroup(t *testing.T, name string) iggcon.MetadataChangeGroup { + t.Helper() + group, err := iggcon.ParseMetadataChangeGroup(name) + require.NoError(t, err) + return group +} + +// recordEnvelope wraps a body as a record of op 1 by root. +func recordEnvelope(group, kind uint8, body []byte) []byte { + record := binary.LittleEndian.AppendUint64(nil, 1) + record = binary.LittleEndian.AppendUint32(record, 0) + record = append(record, make([]byte, 16)...) + record = binary.LittleEndian.AppendUint64(record, 7) + record = append(record, 1, 0, 0, 0, 0, group, kind) + record = binary.LittleEndian.AppendUint32(record, uint32(len(body))) + return append(record, body...) +} + +// changesPagePayload builds an ok changes page holding records. +func changesPagePayload(records ...[]byte) []byte { + payload := pageHeader(iggcon.MetadataChangePhaseChanges) + payload = binary.LittleEndian.AppendUint32(payload, uint32(len(records))) + for _, record := range records { + payload = append(payload, record...) + } + payload = binary.LittleEndian.AppendUint32(payload, 0) + return binary.LittleEndian.AppendUint16(payload, 0) +} + +// pageHeader is an ok page without a log id or cursor. +func pageHeader(phase iggcon.MetadataChangePhase) []byte { + payload := []byte{byte(iggcon.MetadataChangeStatusOk), byte(phase), 0} + payload = append(payload, make([]byte, 24)...) + return append(payload, 0, 0) +} + +func decodeChangesPage(t *testing.T, records ...[]byte) *iggcon.MetadataChanges { + t.Helper() + changes, err := binaryserialization.DeserializeMetadataChanges(changesPagePayload(records...)) + require.NoError(t, err) + return changes +} + +func decodeCurrentPage(t *testing.T, item []byte) *iggcon.MetadataChanges { + t.Helper() + payload := pageHeader(iggcon.MetadataChangePhaseCurrent) + payload = binary.LittleEndian.AppendUint32(payload, 0) + payload = binary.LittleEndian.AppendUint32(payload, 1) + payload = append(payload, item...) + payload = binary.LittleEndian.AppendUint16(payload, 0) + changes, err := binaryserialization.DeserializeMetadataChanges(payload) + require.NoError(t, err) + return changes +} + +func marshalView(t *testing.T, view any) string { + t.Helper() + encoded, err := json.Marshal(view) + require.NoError(t, err) + return string(encoded) +} + +// changesView renders an answer in the HTTP JSON shape the fixture expects. +func changesView(t *testing.T, changes *iggcon.MetadataChanges) map[string]any { + t.Helper() + view := map[string]any{ + "status": changes.Status.String(), + "phase": changes.Phase.String(), + "log_id": nil, + "oldest_op": changes.OldestOp, + "retained_since": changes.RetainedSince, + "head_op": changes.HeadOp, + "next": nil, + "has_more": changes.HasMore, + "current_page_token": nil, + } + if changes.LogID != nil { + view["log_id"] = changes.LogID.String() + } + if changes.Next != nil { + view["next"] = changes.Next.String() + } + if changes.CurrentPageToken != nil { + view["current_page_token"] = changes.CurrentPageToken.String() + } + records := make([]map[string]any, 0, len(changes.Records)) + for _, record := range changes.Records { + records = append(records, recordView(t, record)) + } + items := make([]map[string]any, 0, len(changes.Items)) + for _, item := range changes.Items { + items = append(items, itemView(t, item)) + } + view["records"] = records + view["items"] = items + return view +} + +func recordView(t *testing.T, record iggcon.MetadataChangeRecord) map[string]any { + t.Helper() + view := map[string]any{ + "op": record.Op, + "ordinal": record.Ordinal, + "checksum": record.Checksum.String(), + "timestamp": record.Timestamp, + "group": record.Group.String(), + "kind": record.Kind.String(), + } + if record.ActorUserID != nil { + view["actor_user_id"] = *record.ActorUserID + } + if unknown, ok := record.Body.(*iggcon.MetadataUnknownBody); ok { + view["group"] = strconv.Itoa(int(record.Group)) + view["kind"] = strconv.Itoa(int(record.Kind)) + view["body"] = base64.StdEncoding.EncodeToString(unknown.Body) + return view + } + addBodyView(t, record.Body, view) + return view +} + +func itemView(t *testing.T, item iggcon.MetadataCurrentItem) map[string]any { + t.Helper() + view := map[string]any{"group": item.Group.String()} + if unknown, ok := item.Body.(*iggcon.MetadataUnknownBody); ok { + view["group"] = strconv.Itoa(int(item.Group)) + view["body"] = base64.StdEncoding.EncodeToString(unknown.Body) + return view + } + addBodyView(t, item.Body, view) + return view +} + +func addBodyView(t *testing.T, body iggcon.MetadataChangeBody, view map[string]any) { + t.Helper() + switch body := body.(type) { + case *iggcon.MetadataStreamChange: + addStreamKey(view, body.MetadataStreamKey) + addFields(view, "created_at", body.CreatedAt, "name", body.Name) + addOptions(t, view, body.Options) + case *iggcon.MetadataStreamDeleted: + addStreamKey(view, body.MetadataStreamKey) + addFields(view, "created_at", body.CreatedAt, "name", body.Name) + addOptional(view, "requested_op", body.RequestedOp) + case *iggcon.MetadataTopicChange: + addTopicKey(view, body.MetadataTopicKey) + addFields(view, "created_at", body.CreatedAt, "name", body.Name, "partitions_count", body.PartitionsCount) + addOptions(t, view, body.Options) + case *iggcon.MetadataTopicItem: + addTopicKey(view, body.MetadataTopicKey) + addFields(view, "created_at", body.CreatedAt, "name", body.Name) + addOptions(t, view, body.Options) + case *iggcon.MetadataTopicDeleted: + addTopicKey(view, body.MetadataTopicKey) + addFields(view, "created_at", body.CreatedAt, "name", body.Name) + addOptional(view, "requested_op", body.RequestedOp) + case *iggcon.MetadataPartitionsChanged: + addTopicKey(view, body.MetadataTopicKey) + addFields(view, "partitions_count", body.PartitionsCount, "removed_partition_ids", body.RemovedPartitionIDs) + addOptional(view, "requested_op", body.RequestedOp) + case *iggcon.MetadataPartitionsItem: + addTopicKey(view, body.MetadataTopicKey) + addFields(view, "partitions_count", body.PartitionsCount) + case *iggcon.MetadataConsumerGroupCreated: + addTopicKey(view, body.MetadataTopicKey) + addFields(view, "group_id", body.GroupID, "name", body.Name, "consumer_groups_count", body.ConsumerGroupsCount) + case *iggcon.MetadataConsumerGroupDeleted: + addTopicKey(view, body.MetadataTopicKey) + addFields(view, "group_id", body.GroupID, "name", body.Name, "consumer_groups_count", body.ConsumerGroupsCount) + addOptional(view, "requested_op", body.RequestedOp) + case *iggcon.MetadataConsumerGroupItem: + addTopicKey(view, body.MetadataTopicKey) + addFields(view, "group_id", body.GroupID, "name", body.Name) + case *iggcon.MetadataMemberJoined: + addTopicKey(view, body.MetadataTopicKey) + addFields(view, "group_id", body.GroupID, "member_id", body.MemberID, "client_id", body.ClientID.String(), + "members_count", body.MembersCount, "partition_ids", body.PartitionIDs) + case *iggcon.MetadataMemberLeft: + addTopicKey(view, body.MetadataTopicKey) + addFields(view, "group_id", body.GroupID, "member_id", body.MemberID, "client_id", body.ClientID.String(), + "members_count", body.MembersCount, "reason", body.Reason.String()) + case *iggcon.MetadataPartitionHandedOver: + addTopicKey(view, body.MetadataTopicKey) + addFields(view, "group_id", body.GroupID, "partition_id", body.PartitionID) + addOptional(view, "member_id", body.MemberID) + case *iggcon.MetadataMemberItem: + addTopicKey(view, body.MetadataTopicKey) + addFields(view, "group_id", body.GroupID, "member_id", body.MemberID, "client_id", body.ClientID.String(), + "partition_ids", body.PartitionIDs) + case *iggcon.MetadataUserChange: + addUserKey(view, body.MetadataUserKey) + addFields(view, "username", body.Username, "status", userStatusName(t, body.Status), + "permissions", permissionsView(body.Permissions)) + addOptions(t, view, body.Options) + case *iggcon.MetadataUserIdentity: + addUserKey(view, body.MetadataUserKey) + addFields(view, "username", body.Username) + case *iggcon.MetadataUserPermissionsUpdated: + addUserKey(view, body.MetadataUserKey) + addFields(view, "username", body.Username, "permissions", permissionsView(body.Permissions), + "reason", body.Reason.String()) + case *iggcon.MetadataTokenChange: + addUserKey(view, body.MetadataUserKey) + addFields(view, "name", body.Name) + addOptional(view, "expiry", body.Expiry) + case *iggcon.MetadataTokenDeleted: + addUserKey(view, body.MetadataUserKey) + addFields(view, "name", body.Name, "reason", body.Reason.String()) + case *iggcon.MetadataSession: + addUserKey(view, body.MetadataUserKey) + addFields(view, "client_id", body.ClientID.String(), "session", body.Session) + case *iggcon.MetadataSessionClosed: + addUserKey(view, body.MetadataUserKey) + addFields(view, "client_id", body.ClientID.String(), "session", body.Session, "reason", body.Reason.String()) + default: + require.FailNow(t, "unexpected body", "%T", body) + } +} + +func addStreamKey(view map[string]any, key iggcon.MetadataStreamKey) { + addFields(view, "stream_id", key.StreamID, "stream_created_op", key.StreamCreatedOp) +} + +func addTopicKey(view map[string]any, key iggcon.MetadataTopicKey) { + addStreamKey(view, key.MetadataStreamKey) + addFields(view, "topic_id", key.TopicID, "topic_created_op", key.TopicCreatedOp) +} + +func addUserKey(view map[string]any, key iggcon.MetadataUserKey) { + addFields(view, "user_id", key.UserID, "user_created_at", key.UserCreatedAt) +} + +func addFields(view map[string]any, pairs ...any) { + for index := 0; index < len(pairs); index += 2 { + view[pairs[index].(string)] = pairs[index+1] + } +} + +func addOptional[T any](view map[string]any, name string, value *T) { + if value != nil { + view[name] = *value + } +} + +func addOptions(t *testing.T, view map[string]any, options iggcon.MetadataOptions) { + t.Helper() + if options.Unreadable { + assert.Nil(t, options.Options) + assert.Nil(t, options.DerivedOptions) + view["options"] = nil + view["options_unreadable"] = true + return + } + view["options"] = optionViews(t, options.Options, options.DerivedOptions) +} + +func userStatusName(t *testing.T, status iggcon.UserStatus) string { + t.Helper() + names := map[iggcon.UserStatus]string{iggcon.Active: "active", iggcon.Inactive: "inactive"} + name, ok := names[status] + require.True(t, ok, "unexpected status %d", status) + return name +} + +// permissionsView renders permissions in the get_user JSON shape. +func permissionsView(permissions *iggcon.Permissions) any { + if permissions == nil { + return nil + } + global := permissions.Global + view := map[string]any{ + "global": map[string]bool{ + "manage_servers": global.ManageServers, "read_servers": global.ReadServers, + "manage_users": global.ManageUsers, "read_users": global.ReadUsers, + "manage_streams": global.ManageStreams, "read_streams": global.ReadStreams, + "manage_topics": global.ManageTopics, "read_topics": global.ReadTopics, + "poll_messages": global.PollMessages, "send_messages": global.SendMessages, + }, + "streams": nil, + } + if permissions.Streams == nil { + return view + } + streams := make(map[string]any, len(permissions.Streams)) + for streamID, stream := range permissions.Streams { + streamView := map[string]any{ + "manage_stream": stream.ManageStream, "read_stream": stream.ReadStream, + "manage_topics": stream.ManageTopics, "read_topics": stream.ReadTopics, + "poll_messages": stream.PollMessages, "send_messages": stream.SendMessages, + "topics": nil, + } + if stream.Topics != nil { + topics := make(map[string]any, len(stream.Topics)) + for topicID, topic := range stream.Topics { + topics[strconv.Itoa(topicID)] = map[string]bool{ + "manage_topic": topic.ManageTopic, "read_topic": topic.ReadTopic, + "poll_messages": topic.PollMessages, "send_messages": topic.SendMessages, + } + } + streamView["topics"] = topics + } + streams[strconv.Itoa(streamID)] = streamView + } + view["streams"] = streams + return view +}
diff --git a/foreign/go/contracts/testdata/metadata_changes.json b/foreign/go/contracts/testdata/metadata_changes.json new file mode 100644 index 0000000..26e7faf --- /dev/null +++ b/foreign/go/contracts/testdata/metadata_changes.json
@@ -0,0 +1,1997 @@ +{ + "schema_version": 1, + "generated_by": "core/common/examples/metadata_changes_fixtures.rs", + "conventions": { + "byte_strings": "Every *_hex field is lowercase hex without separators.", + "expected": "expected is the JSON the HTTP route renders. Binary decoders must reach the same values.", + "record_hex": "record_hex is one record envelope as a response carries it. body_hex is the body alone.", + "refused": "Every refused vector must fail to decode or validate. Query and HTTP refusals are invalid_command.", + "request_hex": "request_hex is the command payload without framing. http_query is the query string of GET /metadata/changes.", + "u128_values": "128-bit values (log ids, checksums, client ids) are 32 lowercase hex characters in JSON.", + "u64_values": "Ops and timestamps are JSON numbers below 2^53 in every vector except large_values, which carries them as decimal strings.", + "unknown_kinds": "A record of an unknown group or kind is kept with its codes and raw body, never refused." + }, + "constants": { + "command_code": 17, + "command_name": "metadata.changes", + "cursor_prefix": "mc1_", + "page_token_prefix": "cs1_", + "cursor_size": 44, + "cursor_text_length": 63, + "max_page_token_size": 316, + "max_page_token_key_size": 260, + "default_limit": 100, + "max_limit": 1000, + "groups": { + "consumer_group_members": { + "code": 5, + "class": "activity", + "parents": [ + "consumer_groups", + "partitions", + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32][member_id:u32]" + }, + "consumer_groups": { + "code": 4, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32]" + }, + "partitions": { + "code": 3, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "personal_access_tokens": { + "code": 7, + "class": "resource", + "parents": [ + "users" + ], + "page_token_key": "[user_id:u32][name_len:u8][name]" + }, + "sessions": { + "code": 8, + "class": "activity", + "parents": [ + "users" + ], + "page_token_key": "[epoch:u64]" + }, + "streams": { + "code": 1, + "class": "resource", + "parents": [], + "page_token_key": "[stream_id:u32]" + }, + "topics": { + "code": 2, + "class": "resource", + "parents": [ + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "users": { + "code": 6, + "class": "resource", + "parents": [], + "page_token_key": "[user_id:u32]" + } + }, + "kinds": { + "consumer_group_created": { + "code": 30, + "group": "consumer_groups" + }, + "consumer_group_deleted": { + "code": 31, + "group": "consumer_groups" + }, + "consumer_group_member_joined": { + "code": 40, + "group": "consumer_group_members" + }, + "consumer_group_member_left": { + "code": 41, + "group": "consumer_group_members" + }, + "consumer_group_partition_handed_over": { + "code": 42, + "group": "consumer_group_members" + }, + "partitions_changed": { + "code": 20, + "group": "partitions" + }, + "personal_access_token_created": { + "code": 60, + "group": "personal_access_tokens" + }, + "personal_access_token_deleted": { + "code": 61, + "group": "personal_access_tokens" + }, + "session_closed": { + "code": 71, + "group": "sessions" + }, + "session_registered": { + "code": 70, + "group": "sessions" + }, + "stream_created": { + "code": 1, + "group": "streams" + }, + "stream_deleted": { + "code": 3, + "group": "streams" + }, + "stream_updated": { + "code": 2, + "group": "streams" + }, + "topic_created": { + "code": 10, + "group": "topics" + }, + "topic_deleted": { + "code": 12, + "group": "topics" + }, + "topic_updated": { + "code": 11, + "group": "topics" + }, + "user_created": { + "code": 50, + "group": "users" + }, + "user_deleted": { + "code": 52, + "group": "users" + }, + "user_password_changed": { + "code": 53, + "group": "users" + }, + "user_permissions_updated": { + "code": 54, + "group": "users" + }, + "user_updated": { + "code": 51, + "group": "users" + } + }, + "statuses": { + "behind": 1, + "expired": 2, + "ok": 0, + "reset": 3 + }, + "phases": { + "changes": 0, + "current": 1 + }, + "start_modes": { + "after": 2, + "current": 3, + "earliest": 0, + "latest": 1 + }, + "member_left_reasons": { + "left": 0, + "session_ended": 1 + }, + "token_deleted_reasons": { + "deleted": 0, + "expired": 1 + }, + "session_closed_reasons": { + "expired": 1, + "logout": 0, + "revoked": 2 + }, + "permissions_updated_reasons": { + "resource_deleted": 1, + "updated": 0 + } + }, + "records": [ + { + "case": "stream_created", + "group": 1, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "record_hex": "191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "stream_updated_unreadable_options", + "group": 1, + "kind": 2, + "body_hex": "03000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "record_hex": "1a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "expected": { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + } + }, + { + "case": "stream_deleted_deferred", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "record_hex": "1b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "expected": { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + }, + { + "case": "topic_created", + "group": 2, + "kind": 10, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "record_hex": "1c1000000000000000000000683d9e8b03f7c1a500000000000000001c00be01094106000104000000020a78000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "op": 4124, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d68", + "timestamp": 1760356800004124, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "partitions_count": 8, + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "topic_updated", + "group": 2, + "kind": 11, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "record_hex": "1d1000000000000000000000693d9e8b03f7c1a500000000000000001d00be01094106000104000000020b3d000000030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "expected": { + "op": 4125, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d69", + "timestamp": 1760356800004125, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_updated", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics-v2", + "partitions_count": 8, + "options": {} + } + }, + { + "case": "topic_deleted_immediate", + "group": 2, + "kind": 12, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "record_hex": "1e10000000000000000000006a3d9e8b03f7c1a500000000000000001e00be01094106000104000000020c2e000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "expected": { + "op": 4126, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6a", + "timestamp": 1760356800004126, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics" + } + }, + { + "case": "partitions_changed_deleted_tail", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "record_hex": "1f10000000000000000000006b3d9e8b03f7c1a500000000000000001f00be01094106000104000000031435000000030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "expected": { + "op": 4127, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6b", + "timestamp": 1760356800004127, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 5, + "removed_partition_ids": [ + 5, + 6, + 7 + ], + "requested_op": 4200 + } + }, + { + "case": "partitions_changed_created", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000a0000000000000000", + "record_hex": "2010000000000000000000006c3d9e8b03f7c1a500000000000000002000be01094106000104000000031421000000030000001910000000000000000000001d100000000000000a0000000000000000", + "expected": { + "op": 4128, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6c", + "timestamp": 1760356800004128, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 10, + "removed_partition_ids": [] + } + }, + { + "case": "consumer_group_created", + "group": 4, + "kind": 30, + "body_hex": "030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "record_hex": "2110000000000000000000006d3d9e8b03f7c1a500000000000000002100be01094106000104000000041e28000000030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "expected": { + "op": 4129, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6d", + "timestamp": 1760356800004129, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 1 + } + }, + { + "case": "consumer_group_deleted", + "group": 4, + "kind": 31, + "body_hex": "030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "record_hex": "2210000000000000000000006e3d9e8b03f7c1a500000000000000002200be01094106000104000000041f31000000030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "expected": { + "op": 4130, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6e", + "timestamp": 1760356800004130, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 0, + "requested_op": 4300 + } + }, + { + "case": "consumer_group_member_joined", + "group": 5, + "kind": 40, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "record_hex": "2310000000000000000000006f3d9e8b03f7c1a500000000000000002300be01094106000109000000052838000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "expected": { + "op": 4131, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6f", + "timestamp": 1760356800004131, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_joined", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 2, + "partition_ids": [] + } + }, + { + "case": "consumer_group_member_left_session_ended", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "record_hex": "241000000000000000000000703d9e8b03f7c1a500000000000000002400be01094106000109000000052935000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "expected": { + "op": 4132, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d70", + "timestamp": 1760356800004132, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_left", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 1, + "reason": "session_ended" + } + }, + { + "case": "consumer_group_partition_handed_over", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d1000000000000000000000020000000101000000", + "record_hex": "251000000000000000000000713d9e8b03f7c1a500000000000000002500be010941060000052a25000000030000001910000000000000000000001d1000000000000000000000020000000101000000", + "expected": { + "op": 4133, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d71", + "timestamp": 1760356800004133, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 2, + "member_id": 1 + } + }, + { + "case": "consumer_group_partition_released", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d10000000000000000000000300000000", + "record_hex": "261000000000000000000000723d9e8b03f7c1a500000000000000002600be010941060000052a21000000030000001910000000000000000000001d10000000000000000000000300000000", + "expected": { + "op": 4134, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d72", + "timestamp": 1760356800004134, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 3 + } + }, + { + "case": "user_created_with_permissions", + "group": 6, + "kind": 50, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "record_hex": "271000000000000000000000733d9e8b03f7c1a500000000000000002700be010941060001000000000632480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "op": 4135, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d73", + "timestamp": 1760356800004135, + "actor_user_id": 0, + "group": "users", + "kind": "user_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "user_updated_inactive", + "group": 6, + "kind": 51, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "record_hex": "281000000000000000000000743d9e8b03f7c1a500000000000000002800be010941060001000000000633280000000700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "expected": { + "op": 4136, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d74", + "timestamp": 1760356800004136, + "actor_user_id": 0, + "group": "users", + "kind": "user_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "inactive", + "permissions": null, + "options": {} + } + }, + { + "case": "user_deleted", + "group": 6, + "kind": 52, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "291000000000000000000000753d9e8b03f7c1a500000000000000002900be0109410600010000000006341d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4137, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d75", + "timestamp": 1760356800004137, + "actor_user_id": 0, + "group": "users", + "kind": "user_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_password_changed", + "group": 6, + "kind": 53, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "2a1000000000000000000000763d9e8b03f7c1a500000000000000002a00be0109410600010700000006351d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4138, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d76", + "timestamp": 1760356800004138, + "actor_user_id": 7, + "group": "users", + "kind": "user_password_changed", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_permissions_pruned", + "group": 6, + "kind": 54, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "record_hex": "2b1000000000000000000000773d9e8b03f7c1a500000000000000002b00be0109410600010400000006362a0000000700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "expected": { + "op": 4139, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d77", + "timestamp": 1760356800004139, + "actor_user_id": 4, + "group": "users", + "kind": "user_permissions_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": false, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": null + }, + "reason": "resource_deleted" + } + }, + { + "case": "personal_access_token_created", + "group": 7, + "kind": 60, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "record_hex": "2c1000000000000000000000783d9e8b03f7c1a500000000000000002c00be01094106000107000000073c1d0000000700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "expected": { + "op": 4140, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d78", + "timestamp": 1760356800004140, + "actor_user_id": 7, + "group": "personal_access_tokens", + "kind": "personal_access_token_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "expiry": 1760400000000000 + } + }, + { + "case": "personal_access_token_deleted_expired", + "group": 7, + "kind": 61, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e01", + "record_hex": "2d1000000000000000000000793d9e8b03f7c1a500000000000000002d00be010941060000073d160000000700000000b832c8fb4006000863692d746f6b656e01", + "expected": { + "op": 4141, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d79", + "timestamp": 1760356800004141, + "group": "personal_access_tokens", + "kind": "personal_access_token_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "reason": "expired" + } + }, + { + "case": "session_registered", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "record_hex": "2e10000000000000000000007a3d9e8b03f7c1a500000000000000002e00be010941060001070000000846240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "op": 4142, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7a", + "timestamp": 1760356800004142, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_registered", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + }, + { + "case": "session_closed_expired", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "record_hex": "2f10000000000000000000007b3d9e8b03f7c1a500000000000000002f00be0109410600000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "expected": { + "op": 4143, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7b", + "timestamp": 1760356800004143, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "expired" + } + }, + { + "case": "session_closed_revoked", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "record_hex": "3010000000000000000000007c3d9e8b03f7c1a500000000000000003000be010941060001070000000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "expected": { + "op": 4144, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7c", + "timestamp": 1760356800004144, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "revoked" + } + } + ], + "items": [ + { + "case": "stream", + "group": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "item_hex": "01b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "topic", + "group": 2, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "item_hex": "0274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "partitions", + "group": 3, + "body_hex": "030000001910000000000000000000001d1000000000000008000000", + "item_hex": "031c000000030000001910000000000000000000001d1000000000000008000000", + "expected": { + "group": "partitions", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 8 + } + }, + { + "case": "consumer_group", + "group": 4, + "body_hex": "030000001910000000000000000000001d10000000000000000000000772656164657273", + "item_hex": "0424000000030000001910000000000000000000001d10000000000000000000000772656164657273", + "expected": { + "group": "consumer_groups", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers" + } + }, + { + "case": "consumer_group_member", + "group": 5, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "item_hex": "0540000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "expected": { + "group": "consumer_group_members", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "partition_ids": [ + 0, + 2, + 4 + ] + } + }, + { + "case": "user", + "group": 6, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "item_hex": "06480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "group": "users", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "personal_access_token_without_expiry", + "group": 7, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "item_hex": "071d0000000700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "expected": { + "group": "personal_access_tokens", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token" + } + }, + { + "case": "session", + "group": 8, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "item_hex": "08240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "group": "sessions", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + } + ], + "cursors": [ + { + "case": "zero", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d000000000000000000000000000000000000000000000000000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 0, + "checksum": "00000000000000000000000000000000", + "ordinal": 0 + }, + { + "case": "first_record_of_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAABgQAAAAAAAApcH3A4ueLUwAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000001810000000000000a5c1f7038b9e2d4c000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4120, + "checksum": "00000000000000004c2d9e8b03f7c1a5", + "ordinal": 0 + }, + { + "case": "inside_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f2000000000000000001000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4187, + "checksum": "0000000000000000f2a4c6e8b0d1f359", + "ordinal": 1 + }, + { + "case": "every_field_at_maximum", + "text": "mc1___________________________________________________________8", + "hex": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + "log_id": "ffffffffffffffffffffffffffffffff", + "op": 18446744073709551615, + "checksum": "ffffffffffffffffffffffffffffffff", + "ordinal": 4294967295 + } + ], + "page_tokens": [ + { + "case": "streams", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000000101040003000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 1, + "group": "streams", + "last_key_hex": "03000000" + }, + { + "case": "topics", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 3, + "group": "topics", + "last_key_hex": "0300000000000000" + }, + { + "case": "partitions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8DCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff0308000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "partitions", + "last_key_hex": "0300000000000000" + }, + { + "case": "consumer_groups", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8EDAADAAAAAAAAAAIAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff040c00030000000000000002000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_groups", + "last_key_hex": "030000000000000002000000" + }, + { + "case": "consumer_group_members", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8FEAADAAAAAAAAAAIAAAABAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff05100003000000000000000200000001000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_group_members", + "last_key_hex": "03000000000000000200000001000000" + }, + { + "case": "users", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAACAGBAAHAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000002006040007000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 32, + "group": "users", + "last_key_hex": "07000000" + }, + { + "case": "personal_access_tokens", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAGAHDQAHAAAACGNpLXRva2Vu", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e1000000000000060070d00070000000863692d746f6b656e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 96, + "group": "personal_access_tokens", + "last_key_hex": "070000000863692d746f6b656e" + }, + { + "case": "sessions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAKAICACUEQAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000a00808009411000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 160, + "group": "sessions", + "last_key_hex": "9411000000000000" + }, + { + "case": "longest_token_name", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8HBAH______25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubg", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff070401ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "personal_access_tokens", + "last_key_hex": "ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e" + } + ], + "requests": [ + { + "case": "earliest_every_group", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000", + "http_query": "from=earliest&limit=100" + }, + { + "case": "earliest_many_duplicate_groups", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "users" + ], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000640000000201060000", + "http_query": "from=earliest&groups=streams,users&limit=100" + }, + { + "case": "earliest_since_users_and_tokens", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "users", + "personal_access_tokens" + ], + "since": 1760353200000000, + "until": null, + "limit": 100 + }, + "request_hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001004c2a2b0841060000640000000206070000", + "http_query": "from=earliest&groups=users,personal_access_tokens&since=1760353200000000&limit=100" + }, + { + "case": "latest_streams", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [ + "streams" + ], + "since": null, + "until": null, + "limit": 1 + }, + "request_hex": "01000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000001010000", + "http_query": "from=latest&groups=streams&limit=1" + }, + { + "case": "after_cursor_with_time_range", + "input": { + "start": "after", + "cursor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "page_token": null, + "groups": [ + "topics", + "streams" + ], + "since": 1760356800000000, + "until": 1760360400000000, + "limit": 500 + }, + "request_hex": "02d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000000100f0bd010941060001009451d809410600f40100000202010000", + "http_query": "after=mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA&groups=streams,topics&since=1760356800000000&until=1760360400000000&limit=500" + }, + { + "case": "current_first_page", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201020000", + "http_query": "from=current&groups=streams,topics&limit=1000" + }, + { + "case": "current_continuation", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201024000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "http_query": "from=current¤t_after=cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA&groups=streams,topics&limit=1000" + } + ], + "responses": [ + { + "case": "changes_page", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001b10000000000000673d9e8b03f7c1a50000000000000000000000000103000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465641a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d7632011b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABsQAAAAAAAAZz2eiwP3waUAAAAAAAAAAAAAAAA", + "has_more": true, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + }, + { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "unknown_kind_and_group", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001910000000000000653d9e8b03f7c1a50000000000000000010000000002000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be01094106000104000000010904000000deadbeef191000000000000001000000653d9e8b03f7c1a500000000000000001900be01094106000104000000c80104000000deadbeef000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABkQAAAAAAAAZT2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": false, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "1", + "kind": "9", + "body": "3q2+7w==" + }, + { + "op": 4121, + "ordinal": 1, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "200", + "kind": "1", + "body": "3q2+7w==" + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "current_page", + "response_hex": "000101d7931c6e2a8f4b0d0000000000000000010000000000000000000000000000005e1000000000000001d7931c6e2a8f4b0d00000000000000005b10000000000000a73d9e8b03f7c1a500000000000000000100000001000000000200000001b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465640274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c0800000000000040000000004000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "expected": { + "status": "ok", + "phase": "current", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 1, + "retained_since": 0, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAApz2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": true, + "records": [], + "items": [ + { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + ], + "current_page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA" + } + }, + { + "case": "behind_before_op_1", + "response_hex": "010000000000000000000000000000000000000000000000000000000000000000000000000000", + "expected": { + "status": "behind", + "phase": "changes", + "log_id": null, + "oldest_op": 0, + "retained_since": 0, + "head_op": 0, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "expired", + "response_hex": "020001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "expired", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "reset", + "response_hex": "030001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "reset", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + } + ], + "large_values": { + "record_hex": "0300000000002000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff01ffffffff084624000000ffffffff0100000000002000feffffffffffffffffffffffffffffff0200000000002000", + "op": "9007199254740995", + "ordinal": 4294967295, + "timestamp": "18446744073709551615", + "checksum": "ffffffffffffffffffffffffffffffff", + "user_created_at": "9007199254740993", + "session": "9007199254740994", + "client_id": "fffffffffffffffffffffffffffffffe" + }, + "refused": { + "cursors": [ + { + "case": "wrong_prefix", + "text": "mc2_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "missing_prefix", + "text": "15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "too_long", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAAA" + }, + { + "case": "too_short", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA" + }, + { + "case": "padded", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA=" + }, + { + "case": "standard_alphabet_character", + "text": "mc1_+5McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "op_zero_with_checksum", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAA" + } + ], + "page_tokens": [ + { + "case": "wrong_prefix", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA" + }, + { + "case": "trailing_byte", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAAAA" + }, + { + "case": "wrong_key_width", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBCAADAAAAAAAAAA" + }, + { + "case": "group_outside_mask", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAIBBAADAAAA" + }, + { + "case": "head_below_anchor", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAABAAAAAAAAAAEBBAADAAAA" + }, + { + "case": "empty", + "text": "cs1_" + } + ], + "record_bodies": [ + { + "case": "unknown_member_left_reason", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000002" + }, + { + "case": "presence_byte_two", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279022c10000000000000" + }, + { + "case": "unsorted_removed_partition_ids", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000090000000600000007000000016810000000000000" + }, + { + "case": "known_kind_under_wrong_group", + "group": 2, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564" + }, + { + "case": "trailing_byte", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000000" + }, + { + "case": "truncated", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a09080706050403020194110000000000" + } + ], + "requests": [ + { + "case": "cursor_bytes_without_after", + "hex": "000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "trailing_byte", + "hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006400000000000000" + }, + { + "case": "unknown_start_mode", + "hex": "040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "non_canonical_cursor", + "hex": "02d7931c6e2a8f4b0d000000000000000000000000000000000100000000000000000000000000000000000000000064000000000000" + } + ], + "responses": [ + { + "case": "items_in_changes_phase", + "hex": "000001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + }, + { + "case": "unknown_status", + "hex": "040001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + } + ], + "queries": [ + { + "case": "limit_zero", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 0 + } + }, + { + "case": "limit_above_maximum", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 1001 + } + }, + { + "case": "since_above_until", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 2, + "until": 1, + "limit": 100 + } + }, + { + "case": "time_bound_with_latest", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 1, + "until": null, + "limit": 100 + } + }, + { + "case": "time_bound_with_current", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": 1, + "limit": 100 + } + }, + { + "case": "current_missing_parent_group", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + }, + { + "case": "page_token_for_other_groups", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + } + ], + "http_queries": [ + { + "case": "no_start", + "query": "groups=streams" + }, + { + "case": "from_and_after", + "query": "from=earliest&after=mc1_x" + }, + { + "case": "unknown_from", + "query": "from=newest" + }, + { + "case": "duplicate_key", + "query": "from=earliest&from=latest" + }, + { + "case": "unknown_key", + "query": "from=earliest&verbose=1" + }, + { + "case": "empty_group_element", + "query": "from=earliest&groups=streams,,topics" + }, + { + "case": "trailing_group_comma", + "query": "from=earliest&groups=streams," + }, + { + "case": "unknown_group", + "query": "from=earliest&groups=everything" + }, + { + "case": "negative_since", + "query": "from=earliest&since=-1" + }, + { + "case": "leading_zero_until", + "query": "from=earliest&until=01" + }, + { + "case": "non_numeric_limit", + "query": "from=earliest&limit=ten" + }, + { + "case": "token_without_current", + "query": "from=latest¤t_after=cs1_AAAA" + } + ] + } +}
diff --git a/foreign/go/contracts/version.go b/foreign/go/contracts/version.go index ab6b6ff..703b69f 100644 --- a/foreign/go/contracts/version.go +++ b/foreign/go/contracts/version.go
@@ -17,4 +17,4 @@ package iggcon -const Version = "0.9.1-edge.2" +const Version = "0.9.1-edge.3"
diff --git a/foreign/go/internal/command/code.go b/foreign/go/internal/command/code.go index 45f2611..4ff3aa2 100644 --- a/foreign/go/internal/command/code.go +++ b/foreign/go/internal/command/code.go
@@ -27,6 +27,7 @@ DescribeOptionsCode Code = 13 BindSessionCode Code = 15 UpdateResourceCode Code = 16 + GetMetadataChangesCode Code = 17 GetMeCode Code = 20 GetClientCode Code = 21 GetClientsCode Code = 22
diff --git a/foreign/go/internal/command/system.go b/foreign/go/internal/command/system.go index 103b080..5d3588d 100644 --- a/foreign/go/internal/command/system.go +++ b/foreign/go/internal/command/system.go
@@ -74,6 +74,67 @@ return []byte{byte(d.Scope)}, nil } +// GetMetadataChanges asks for one page of the metadata change log, or of the +// current state. +type GetMetadataChanges struct { + Query iggcon.MetadataChangesQuery +} + +// maxMetadataChangeGroups is the most group codes the one-byte count carries. +const maxMetadataChangeGroups = 255 + +func (g *GetMetadataChanges) Code() Code { + return GetMetadataChangesCode +} + +func (g *GetMetadataChanges) MarshalBinary() ([]byte, error) { + size := 1 + iggcon.MetadataChangeCursorSize + 9 + 9 + 4 + 1 + len(g.Query.Groups) + 2 + return g.AppendBinary(make([]byte, 0, size)) +} + +// AppendBinary validates the query, then encodes [start:u8][cursor:44] +// [has_since:u8][since:u64][has_until:u8][until:u64][limit:u32] +// [groups_count:u8][group:u8]*[token_len:u16][token]. The cursor is all zero +// unless the query starts after one, and the token is empty unless it +// continues a current scan. Groups keep the caller's order. +func (g *GetMetadataChanges) AppendBinary(b []byte) ([]byte, error) { + query := g.Query + if err := query.Validate(); err != nil { + return nil, err + } + b = append(b, byte(query.Start.Mode)) + var cursor iggcon.MetadataChangeCursor + if query.Start.Mode == iggcon.MetadataChangeStartAfter { + cursor = query.Start.Cursor + } + b, _ = cursor.AppendBinary(b) + b = appendOptionalUint64(b, query.Since) + b = appendOptionalUint64(b, query.Until) + b = binary.LittleEndian.AppendUint32(b, query.Limit) + groups := query.Groups + if len(groups) > maxMetadataChangeGroups { + groups = make([]iggcon.MetadataChangeGroup, 0, 8) + var seen [256]bool + for _, group := range query.Groups { + if !seen[byte(group)] { + seen[byte(group)] = true + groups = append(groups, group) + } + } + } + b = append(b, byte(len(groups))) + for _, group := range groups { + b = append(b, byte(group)) + } + tokenLengthAt := len(b) + b = binary.LittleEndian.AppendUint16(b, 0) + if token := query.Start.PageToken; query.Start.Mode == iggcon.MetadataChangeStartCurrent && token != nil { + b, _ = token.AppendBinary(b) + binary.LittleEndian.PutUint16(b[tokenLengthAt:], uint16(len(b)-tokenLengthAt-2)) + } + return b, nil +} + type GetStats struct{} func (c *GetStats) Code() Code { @@ -105,3 +166,10 @@ func (p *Ping) MarshalBinary() ([]byte, error) { return []byte{}, nil } + +func appendOptionalUint64(b []byte, value *uint64) []byte { + if value == nil { + return append(b, 0) + } + return binary.LittleEndian.AppendUint64(append(b, 1), *value) +}
diff --git a/foreign/go/internal/command/system_test.go b/foreign/go/internal/command/system_test.go index b139a16..3b8999b 100644 --- a/foreign/go/internal/command/system_test.go +++ b/foreign/go/internal/command/system_test.go
@@ -19,9 +19,12 @@ import ( "bytes" + "errors" + "fmt" "testing" iggcon "github.com/apache/iggy/foreign/go/contracts" + ierror "github.com/apache/iggy/foreign/go/errors" ) func TestGetClientMarshalBinary(t *testing.T) { @@ -82,6 +85,115 @@ } } +func TestGetMetadataChangesMarshalBinary(t *testing.T) { + earliest := iggcon.NewMetadataChangesQuery(iggcon.EarliestMetadataChangeStart()) + earliestBody := append([]byte{0}, make([]byte, iggcon.MetadataChangeCursorSize)...) + earliestBody = append(earliestBody, 0, 0, 100, 0, 0, 0, 0, 0, 0) + + t.Run("earliest every group", func(t *testing.T) { + got, err := (&GetMetadataChanges{Query: earliest}).MarshalBinary() + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(got, earliestBody) { + t.Fatalf("GetMetadataChanges body = %v, want %v", got, earliestBody) + } + }) + + t.Run("appends after a prefix", func(t *testing.T) { + got, err := (&GetMetadataChanges{Query: earliest}).AppendBinary([]byte{0xaa}) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(got, append([]byte{0xaa}, earliestBody...)) { + t.Fatalf("GetMetadataChanges appended %v", got) + } + }) + + t.Run("cursor and token only where the start uses them", func(t *testing.T) { + query := earliest + query.Start = iggcon.LatestMetadataChangeStart() + query.Start.Cursor = iggcon.MetadataChangeCursor{Op: 7, Ordinal: 1} + query.Start.PageToken = &iggcon.MetadataCurrentPageToken{} + got, err := (&GetMetadataChanges{Query: query}).MarshalBinary() + if err != nil { + t.Fatal(err) + } + want := append([]byte{1}, earliestBody[1:]...) + if !bytes.Equal(got, want) { + t.Fatalf("GetMetadataChanges body = %v, want %v", got, want) + } + }) + + t.Run("oversized groups retain every requested group", func(t *testing.T) { + query := earliest + query.Groups = []iggcon.MetadataChangeGroup{iggcon.MetadataChangeGroupTopics, iggcon.MetadataChangeGroupStreams, iggcon.MetadataChangeGroupTopics} + for len(query.Groups) < 300 { + query.Groups = append(query.Groups, iggcon.MetadataChangeGroupUsers) + } + got, err := (&GetMetadataChanges{Query: query}).MarshalBinary() + if err != nil { + t.Fatal(err) + } + groupsAt := len(earliestBody) - 3 + if got[groupsAt] != 3 || len(got) != len(earliestBody)+3 { + t.Fatalf("GetMetadataChanges carries %d groups in %d bytes", got[groupsAt], len(got)) + } + if !bytes.Equal(got[groupsAt+1:groupsAt+4], []byte{2, 1, 6}) { + t.Fatalf("GetMetadataChanges groups start %v", got[groupsAt+1:groupsAt+4]) + } + }) + + t.Run("time bounds behind presence bytes", func(t *testing.T) { + since, until := uint64(0x0102030405060708), uint64(0x1112131415161718) + query := earliest + query.Since, query.Until = &since, &until + got, err := (&GetMetadataChanges{Query: query}).MarshalBinary() + if err != nil { + t.Fatal(err) + } + bounds := got[1+iggcon.MetadataChangeCursorSize : 1+iggcon.MetadataChangeCursorSize+18] + want := []byte{1, 8, 7, 6, 5, 4, 3, 2, 1, 1, 0x18, 0x17, 0x16, 0x15, 0x14, 0x13, 0x12, 0x11} + if !bytes.Equal(bounds, want) { + t.Fatalf("GetMetadataChanges bounds = %v, want %v", bounds, want) + } + }) + + for _, limit := range []uint32{0, iggcon.MaxMetadataChangesLimit + 1} { + t.Run(fmt.Sprintf("limit %d refused before encoding", limit), func(t *testing.T) { + query := earliest + query.Limit = limit + got, err := (&GetMetadataChanges{Query: query}).AppendBinary([]byte{0xaa}) + if !errors.Is(err, ierror.ErrInvalidCommand) { + t.Fatalf("limit %d error = %v", limit, err) + } + if got != nil { + t.Fatalf("limit %d returned a body on error: %v", limit, got) + } + }) + } +} + +func TestGetMetadataChanges_GivenManyDuplicateGroupsWhenEncodingShouldPreserveEveryRequestedGroup(t *testing.T) { + query := iggcon.NewMetadataChangesQuery(iggcon.EarliestMetadataChangeStart()) + for range 255 { + query.Groups = append(query.Groups, iggcon.MetadataChangeGroupStreams) + } + query.Groups = append(query.Groups, iggcon.MetadataChangeGroupUsers) + actual, err := (&GetMetadataChanges{Query: query}).MarshalBinary() + if err != nil { + t.Fatal(err) + } + query.Groups = []iggcon.MetadataChangeGroup{iggcon.MetadataChangeGroupStreams, iggcon.MetadataChangeGroupUsers} + expected, err := (&GetMetadataChanges{Query: query}).MarshalBinary() + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(actual, expected) { + t.Fatal("encoding lost a requested group") + } +} + func TestEmptySystemCommandBodies(t *testing.T) { for _, test := range []struct { name string
diff --git a/foreign/go/internal/vsr/operation_test.go b/foreign/go/internal/vsr/operation_test.go index 1412884..c491e04 100644 --- a/foreign/go/internal/vsr/operation_test.go +++ b/foreign/go/internal/vsr/operation_test.go
@@ -76,6 +76,7 @@ command.GetStatsCode, command.GetSnapshotFileCode, command.GetClusterMetadataCode, + command.GetMetadataChangesCode, command.PollMessagesCode, command.GetOffsetCode, command.GetStreamCode,
diff --git a/foreign/go/tests/e2e_metadata_changes_test.go b/foreign/go/tests/e2e_metadata_changes_test.go new file mode 100644 index 0000000..b3d3e43 --- /dev/null +++ b/foreign/go/tests/e2e_metadata_changes_test.go
@@ -0,0 +1,399 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package tests_test + +import ( + "bytes" + "context" + "encoding/base64" + "encoding/hex" + "maps" + "testing" + "time" + + iggcon "github.com/apache/iggy/foreign/go/contracts" + ierror "github.com/apache/iggy/foreign/go/errors" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +const ( + // A two phase delete commits its record after the call returns, so a + // follow waits for it within this budget. + changesFollowBudget = 15 * time.Second + changesFollowInterval = 100 * time.Millisecond +) + +func TestE2E_MetadataChangesFollow(t *testing.T) { + t.Run("given_latest_cursor_when_creating_renaming_and_deleting_a_stream_should_follow_its_records", func(t *testing.T) { + connected, actor := connectAs(t, rootUsername, rootPassword) + ctx := context.Background() + cursor := latestChangeCursor(t, connected, iggcon.MetadataChangeGroupStreams) + + name := uniqueName("changes-follow") + created, err := connected.CreateStream(ctx, name, nil) + require.NoError(t, err) + streamId := scratchStream(t, connected, created.Id) + renamed := uniqueName("changes-renamed") + require.NoError(t, connected.UpdateStream(ctx, streamId, iggcon.StreamUpdate{Name: &renamed})) + require.NoError(t, connected.DeleteStream(ctx, streamId)) + + records, _ := followStreamRecords(t, connected, cursor, created.Id, 3, iggcon.MetadataChangeGroupStreams) + assert.Equal(t, []iggcon.MetadataChangeKind{ + iggcon.MetadataChangeKindStreamCreated, + iggcon.MetadataChangeKindStreamUpdated, + iggcon.MetadataChangeKindStreamDeleted, + }, recordKinds(records)) + for _, record := range records { + require.NotNil(t, record.ActorUserID, "%+v", record) + assert.Equal(t, actor, *record.ActorUserID) + assert.Equal(t, iggcon.MetadataChangeGroupStreams, record.Group) + assert.False(t, record.Checksum.IsZero()) + assert.Positive(t, record.Timestamp) + } + + createdBody := records[0].Body.(*iggcon.MetadataStreamChange) + updatedBody := records[1].Body.(*iggcon.MetadataStreamChange) + deletedBody := records[2].Body.(*iggcon.MetadataStreamDeleted) + assert.Equal(t, name, createdBody.Name) + assert.Equal(t, renamed, updatedBody.Name) + assert.Equal(t, renamed, deletedBody.Name) + assert.Equal(t, createdBody.MetadataStreamKey, updatedBody.MetadataStreamKey) + assert.Equal(t, createdBody.MetadataStreamKey, deletedBody.MetadataStreamKey) + assert.Equal(t, records[0].Op, createdBody.StreamCreatedOp) + assert.False(t, createdBody.Options.Unreadable) + assert.Less(t, records[0].Op, records[1].Op) + assert.Less(t, records[1].Op, records[2].Op) + }) +} + +func TestE2E_MetadataChangesGroupFilter(t *testing.T) { + t.Run("given_topics_group_when_creating_a_stream_and_topic_should_return_only_the_topic_record", func(t *testing.T) { + connected := connect(t) + ctx := context.Background() + cursor := latestChangeCursor(t, connected, iggcon.MetadataChangeGroupTopics) + + stream, err := connected.CreateStream(ctx, uniqueName("changes-filter"), nil) + require.NoError(t, err) + streamId := scratchStream(t, connected, stream.Id) + partitions := uint32(2) + topic, err := connected.CreateTopic(ctx, streamId, "events", &iggcon.TopicOptions{PartitionsCount: &partitions}) + require.NoError(t, err) + + records, _ := followStreamRecords(t, connected, cursor, stream.Id, 1, iggcon.MetadataChangeGroupTopics) + require.Equal(t, []iggcon.MetadataChangeKind{iggcon.MetadataChangeKindTopicCreated}, recordKinds(records)) + body := records[0].Body.(*iggcon.MetadataTopicChange) + assert.Equal(t, topic.Id, body.TopicID) + assert.Equal(t, "events", body.Name) + assert.Equal(t, partitions, body.PartitionsCount) + assert.Equal(t, records[0].Op, body.TopicCreatedOp) + }) +} + +func TestE2E_MetadataChangesLimitPaging(t *testing.T) { + t.Run("given_limit_of_one_when_following_three_streams_should_page_one_record_at_a_time", func(t *testing.T) { + connected := connect(t) + ctx := context.Background() + cursor := latestChangeCursor(t, connected, iggcon.MetadataChangeGroupStreams) + + created := make(map[uint32]string) + for range 3 { + name := uniqueName("changes-page") + stream, err := connected.CreateStream(ctx, name, nil) + require.NoError(t, err) + scratchStream(t, connected, stream.Id) + created[stream.Id] = name + } + + seen := make(map[uint32]string) + deadline := time.Now().Add(changesFollowBudget) + for len(seen) < len(created) { + require.False(t, time.Now().After(deadline), "paged %v of %v", seen, created) + query := iggcon.NewMetadataChangesQuery(iggcon.AfterMetadataChangeStart(cursor), iggcon.MetadataChangeGroupStreams) + query.Limit = 1 + changes, err := connected.GetMetadataChanges(ctx, query) + require.NoError(t, err) + require.Equal(t, iggcon.MetadataChangeStatusOk, changes.Status) + require.LessOrEqual(t, len(changes.Records), 1) + require.NotNil(t, changes.Next) + for _, record := range changes.Records { + if body, ok := record.Body.(*iggcon.MetadataStreamChange); ok && created[body.StreamID] == body.Name { + seen[body.StreamID] = body.Name + } + assert.Equal(t, record.Op, changes.Next.Op) + assert.Equal(t, record.Ordinal, changes.Next.Ordinal) + } + assert.GreaterOrEqual(t, changes.Next.Op, cursor.Op) + cursor = *changes.Next + if !changes.HasMore && len(seen) < len(created) { + time.Sleep(changesFollowInterval) + } + } + assert.Equal(t, created, seen) + }) +} + +func TestE2E_MetadataChangesCurrentState(t *testing.T) { + t.Run("given_streams_when_paging_current_state_one_item_at_a_time_should_list_the_same_streams", func(t *testing.T) { + connected := connect(t) + ctx := context.Background() + for range 2 { + stream, err := connected.CreateStream(ctx, uniqueName("changes-current"), nil) + require.NoError(t, err) + scratchStream(t, connected, stream.Id) + } + + // A deferred delete of an earlier test may land between the two reads. + deadline := time.Now().Add(changesFollowBudget) + for { + projected, anchor := currentStreams(t, connected) + listed, err := connected.GetStreams(ctx) + require.NoError(t, err) + names := make(map[uint32]string, len(listed)) + for _, stream := range listed { + names[stream.Id] = stream.Name + } + assert.Positive(t, anchor.Op) + if maps.Equal(projected, names) { + break + } + require.False(t, time.Now().After(deadline), "current state %v, stream list %v", projected, names) + time.Sleep(changesFollowInterval) + } + }) +} + +func TestE2E_MetadataChangesRefusesInvalidQueries(t *testing.T) { + t.Run("given_queries_breaking_the_request_rules_when_reading_should_refuse_before_sending", func(t *testing.T) { + connected := connect(t) + ctx := context.Background() + token := currentPageToken(t, connected) + later, earlier := uint64(2), uint64(1) + + refused := map[string]iggcon.MetadataChangesQuery{ + "limit_zero": {Start: iggcon.EarliestMetadataChangeStart()}, + "limit_above_maximum": {Start: iggcon.EarliestMetadataChangeStart(), Limit: iggcon.MaxMetadataChangesLimit + 1}, + "since_above_until": {Start: iggcon.EarliestMetadataChangeStart(), Since: &later, Until: &earlier, Limit: 1}, + "time_bound_latest": {Start: iggcon.LatestMetadataChangeStart(), Since: &earlier, Limit: 1}, + "time_bound_current": {Start: iggcon.CurrentMetadataChangeStart(nil), Until: &earlier, Limit: 1}, + "missing_parent_group": iggcon.NewMetadataChangesQuery(iggcon.CurrentMetadataChangeStart(nil), iggcon.MetadataChangeGroupTopics), + "token_of_other_groups": iggcon.NewMetadataChangesQuery(iggcon.CurrentMetadataChangeStart(token), + iggcon.MetadataChangeGroupStreams, iggcon.MetadataChangeGroupUsers), + } + for name, query := range refused { + _, err := connected.GetMetadataChanges(ctx, query) + assert.ErrorIs(t, err, ierror.ErrInvalidCommand, name) + } + + changes, err := connected.GetMetadataChanges(ctx, iggcon.NewMetadataChangesQuery(iggcon.LatestMetadataChangeStart())) + require.NoError(t, err, "the connection stays usable after refused queries") + assert.Equal(t, iggcon.MetadataChangeStatusOk, changes.Status) + }) +} + +func TestE2E_MetadataChangesPermission(t *testing.T) { + t.Run("given_user_without_manage_servers_when_reading_changes_should_be_refused_as_unauthorized", func(t *testing.T) { + root := connect(t) + ctx := context.Background() + username := uniqueUsername("changes-reader") + password := "changes-secret" + user, err := root.CreateUser(ctx, username, password, iggcon.Active, + &iggcon.Permissions{Global: iggcon.GlobalPermissions{ReadServers: true, ReadStreams: true, ReadUsers: true}}, nil) + require.NoError(t, err) + userId, err := iggcon.NewIdentifier(user.Id) + require.NoError(t, err) + t.Cleanup(func() { _ = root.DeleteUser(context.Background(), userId) }) + + reader, _ := connectAs(t, username, password) + _, err = reader.GetMetadataChanges(ctx, iggcon.NewMetadataChangesQuery(iggcon.EarliestMetadataChangeStart())) + assert.ErrorIs(t, err, ierror.ErrUnauthorized) + var iggyError ierror.IggyError + require.ErrorAs(t, err, &iggyError) + assert.Equal(t, ierror.Code(41), iggyError.Code()) + }) +} + +// checkKeyedStreamChange runs inside TestE2E_StreamKeyProjection, which the +// shipped-default lane skips because a client key needs a server key. +func checkKeyedStreamChange(t *testing.T) { + connected := connect(t) + ctx := context.Background() + cursor := latestChangeCursor(t, connected, iggcon.MetadataChangeGroupStreams) + key := randomStreamKey(t) + fingerprint, err := iggcon.EncryptionKeyFingerprint(key) + require.NoError(t, err) + + created, err := connected.CreateStream(ctx, uniqueName("changes-keyed"), &iggcon.StreamOptions{EncryptionKey: key}) + require.NoError(t, err) + scratchStream(t, connected, created.Id) + + records, _ := followStreamRecords(t, connected, cursor, created.Id, 1, iggcon.MetadataChangeGroupStreams) + require.Equal(t, []iggcon.MetadataChangeKind{iggcon.MetadataChangeKindStreamCreated}, recordKinds(records)) + options := records[0].Body.(*iggcon.MetadataStreamChange).Options + require.False(t, options.Unreadable) + projected := iggcon.Stream{Options: options.Options, DerivedOptions: options.DerivedOptions} + requireProjection(t, "change", projected, iggcon.StreamEncryptionSourceStream) + got, ok := projected.EncryptionKeyFingerprint() + require.True(t, ok) + assert.Equal(t, fingerprint, got) + assert.Equal(t, created.EncryptionKekFingerprints(), projected.EncryptionKekFingerprints()) + requireNoKeyMaterial(t, "change", projected, key) + for _, values := range []map[string]iggcon.HeaderValue{options.Options, options.DerivedOptions} { + for name, value := range values { + assert.False(t, bytes.Contains(value.Value, []byte(hex.EncodeToString(key))), name) + assert.NotContains(t, string(value.Value), base64.RawURLEncoding.EncodeToString(key), name) + } + } +} + +// connectAs signs a new client in and returns the signed-in user id. +func connectAs(t *testing.T, username, password string) (iggcon.Client, uint32) { + t.Helper() + connected := newClient(t) + identity, err := connected.LoginUser(context.Background(), username, password) + require.NoError(t, err) + return connected, identity.UserId +} + +func latestChangeCursor(t *testing.T, connected iggcon.Client, groups ...iggcon.MetadataChangeGroup) iggcon.MetadataChangeCursor { + t.Helper() + changes, err := connected.GetMetadataChanges(context.Background(), + iggcon.NewMetadataChangesQuery(iggcon.LatestMetadataChangeStart(), groups...)) + require.NoError(t, err) + require.Equal(t, iggcon.MetadataChangeStatusOk, changes.Status) + require.Empty(t, changes.Records) + require.NotNil(t, changes.LogID) + require.NotNil(t, changes.Next) + assert.Equal(t, *changes.LogID, changes.Next.LogID) + return *changes.Next +} + +// followStreamRecords follows the change log after cursor until want records +// name the stream, and checks that no record leaves the requested groups. +func followStreamRecords( + t *testing.T, + connected iggcon.Client, + cursor iggcon.MetadataChangeCursor, + streamID uint32, + want int, + groups ...iggcon.MetadataChangeGroup, +) ([]iggcon.MetadataChangeRecord, iggcon.MetadataChangeCursor) { + t.Helper() + var records []iggcon.MetadataChangeRecord + deadline := time.Now().Add(changesFollowBudget) + for { + query := iggcon.NewMetadataChangesQuery(iggcon.AfterMetadataChangeStart(cursor), groups...) + query.Limit = iggcon.MaxMetadataChangesLimit + changes, err := connected.GetMetadataChanges(context.Background(), query) + require.NoError(t, err) + require.Equal(t, iggcon.MetadataChangeStatusOk, changes.Status) + require.Equal(t, iggcon.MetadataChangePhaseChanges, changes.Phase) + require.Empty(t, changes.Items) + require.Nil(t, changes.CurrentPageToken) + require.NotNil(t, changes.Next) + cursor = *changes.Next + for _, record := range changes.Records { + assert.Contains(t, groups, record.Group, "%+v", record) + if id, ok := recordStreamID(record); ok && id == streamID { + records = append(records, record) + } + } + if changes.HasMore { + continue + } + if len(records) >= want || time.Now().After(deadline) { + break + } + time.Sleep(changesFollowInterval) + } + require.Len(t, records, want) + return records, cursor +} + +// currentStreams pages the streams state one item at a time and returns the +// names by id with the scan's anchor. +func currentStreams(t *testing.T, connected iggcon.Client) (map[uint32]string, iggcon.MetadataChangeCursor) { + t.Helper() + streams := make(map[uint32]string) + var token *iggcon.MetadataCurrentPageToken + for page := 0; ; page++ { + require.Less(t, page, 10_000, "the current state scan does not end") + query := iggcon.NewMetadataChangesQuery(iggcon.CurrentMetadataChangeStart(token), iggcon.MetadataChangeGroupStreams) + query.Limit = 1 + changes, err := connected.GetMetadataChanges(context.Background(), query) + require.NoError(t, err) + require.Equal(t, iggcon.MetadataChangeStatusOk, changes.Status) + require.Equal(t, iggcon.MetadataChangePhaseCurrent, changes.Phase) + require.Empty(t, changes.Records) + require.LessOrEqual(t, len(changes.Items), 1) + for _, item := range changes.Items { + stream, ok := item.Body.(*iggcon.MetadataStreamChange) + require.True(t, ok, "%T", item.Body) + streams[stream.StreamID] = stream.Name + } + if changes.CurrentPageToken == nil { + require.NotNil(t, changes.Next, "the last page returns the anchor") + return streams, *changes.Next + } + assert.True(t, changes.HasMore) + token = changes.CurrentPageToken + } +} + +// currentPageToken returns a token of a streams scan that has a next page. +func currentPageToken(t *testing.T, connected iggcon.Client) *iggcon.MetadataCurrentPageToken { + t.Helper() + for range 2 { + stream, err := connected.CreateStream(context.Background(), uniqueName("changes-token"), nil) + require.NoError(t, err) + scratchStream(t, connected, stream.Id) + } + query := iggcon.NewMetadataChangesQuery(iggcon.CurrentMetadataChangeStart(nil), iggcon.MetadataChangeGroupStreams) + query.Limit = 1 + changes, err := connected.GetMetadataChanges(context.Background(), query) + require.NoError(t, err) + require.NotNil(t, changes.CurrentPageToken) + return changes.CurrentPageToken +} + +func recordKinds(records []iggcon.MetadataChangeRecord) []iggcon.MetadataChangeKind { + kinds := make([]iggcon.MetadataChangeKind, 0, len(records)) + for _, record := range records { + kinds = append(kinds, record.Kind) + } + return kinds +} + +// recordStreamID names the stream a stream, topic or partitions record +// belongs to. +func recordStreamID(record iggcon.MetadataChangeRecord) (uint32, bool) { + switch body := record.Body.(type) { + case *iggcon.MetadataStreamChange: + return body.StreamID, true + case *iggcon.MetadataStreamDeleted: + return body.StreamID, true + case *iggcon.MetadataTopicChange: + return body.StreamID, true + case *iggcon.MetadataTopicDeleted: + return body.StreamID, true + case *iggcon.MetadataPartitionsChanged: + return body.StreamID, true + default: + return 0, false + } +}
diff --git a/foreign/go/tests/e2e_resource_options_test.go b/foreign/go/tests/e2e_resource_options_test.go index fb4f1f1..10f87a0 100644 --- a/foreign/go/tests/e2e_resource_options_test.go +++ b/foreign/go/tests/e2e_resource_options_test.go
@@ -60,6 +60,7 @@ } assert.Equal(t, created.EncryptionKekFingerprints(), fetched.EncryptionKekFingerprints()) }) + t.Run("given_keyed_stream_when_following_its_creation_should_expose_fingerprints_and_no_key", checkKeyedStreamChange) } func TestE2E_KeylessStream(t *testing.T) {
diff --git a/foreign/java/README.md b/foreign/java/README.md index 864fcfc..32993dc 100644 --- a/foreign/java/README.md +++ b/foreign/java/README.md
@@ -38,7 +38,7 @@ ### Snapshot Versions -Development builds of the next release carry the version `0.9.2-SNAPSHOT`. Get them from the ASF snapshot repository: +Development builds of the next release carry the version `0.9.3-SNAPSHOT`. Get them from the ASF snapshot repository: **Gradle:** @@ -51,7 +51,7 @@ } dependencies { - implementation 'org.apache.iggy:iggy:0.9.2-SNAPSHOT' + implementation 'org.apache.iggy:iggy:0.9.3-SNAPSHOT' } ``` @@ -72,7 +72,7 @@ <dependency> <groupId>org.apache.iggy</groupId> <artifactId>iggy</artifactId> - <version>0.9.2-SNAPSHOT</version> + <version>0.9.3-SNAPSHOT</version> </dependency> </dependencies> ``` @@ -344,6 +344,70 @@ The typed builders refuse an update the server would refuse for its shape before anything is sent, with the server's error code. `raw(key, value)` and `rawReset(key)` reach keys this build has no field for, and the server decides whether they are allowed. +### Metadata Changes + +`getMetadataChanges` reads the node's change log: one record per public result of a committed metadata operation, such as a stream created, a topic renamed, a user's permissions changed or a session closed. A mirror keeps a cursor, asks for the records after it, and stores the returned `next` cursor. The call needs root or the global `manageServers` permission and works on blocking TCP, async TCP and HTTP. + +```java +import org.apache.iggy.system.MetadataChangeGroup; +import org.apache.iggy.system.MetadataChangeStart; +import org.apache.iggy.system.MetadataChanges; +import org.apache.iggy.system.MetadataChangesQuery; + +var query = MetadataChangesQuery.builder() + .groups(MetadataChangeGroup.STREAMS, MetadataChangeGroup.TOPICS) + .build(); +while (true) { + MetadataChanges page = client.system().getMetadataChanges(query); + switch (page.status()) { + case OK -> { + page.records().forEach(record -> + System.out.println("op " + record.op() + " by " + record.actorUserId() + ": " + record.kind())); + if (page.next().isPresent()) { + query = query.withStart(MetadataChangeStart.after(page.next().get())); + } + if (!page.hasMore()) { + Thread.sleep(1000); + } + } + case BEHIND -> Thread.sleep(100); + default -> query = query.withStart(rebuild(client, query.groups())); + } +} + +// Pages the current state, then returns the position to follow from. +static MetadataChangeStart rebuild(IggyBaseClient client, List<MetadataChangeGroup> groups) + throws InterruptedException { + var query = MetadataChangesQuery.builder() + .start(MetadataChangeStart.current()) + .groups(groups) + .build(); + while (true) { + MetadataChanges page = client.system().getMetadataChanges(query); + switch (page.status()) { + case OK -> { + page.items().forEach(item -> System.out.println(item.group() + ": " + item.body())); + if (page.currentPageToken().isPresent()) { + query = query.withStart(MetadataChangeStart.current(page.currentPageToken().get())); + } else if (page.next().isPresent()) { + return MetadataChangeStart.after(page.next().get()); + } else { + query = query.withStart(MetadataChangeStart.current()); + } + } + case BEHIND -> Thread.sleep(100); + default -> query = query.withStart(MetadataChangeStart.current()); + } + } +} +``` + +`EXPIRED` means the log no longer holds the history after the cursor, and `RESET` means the cursor belongs to another history, for example after the data directory was rebuilt. Either way the mirror rebuilds from current state pages and then follows the changes after the anchor they return. + +Current pages need the parent groups of every selected group, for example `TOPICS` with `STREAMS`. Keep a rebuilt state private until a change page answers `OK` with `hasMore()` false and a `headOp()` at or above the last current page's. + +A cursor's `toString()` is its `mc1_` text, which `MetadataChangeCursor.parse` reads back, so a mirror can store it. A query the rules refuse fails with an `IggyValidationException` for invalid command before anything is sent, and a record of a kind this SDK does not know arrives as `MetadataChangeBody.Unknown` or `MetadataChangeBody.UnknownJson`. + ## Exception Handling The SDK's custom exception types inherit from `IggyException`. When you join a failed future, the cause can come back inside a `CompletionException`. The HTTP client's `close()` method declares `IOException`. Handle these two boundaries as well as the specific SDK errors.
diff --git a/foreign/java/gradle.properties b/foreign/java/gradle.properties index d36a136..855b42c 100644 --- a/foreign/java/gradle.properties +++ b/foreign/java/gradle.properties
@@ -15,5 +15,5 @@ # specific language governing permissions and limitations # under the License. -version=0.9.2-SNAPSHOT +version=0.9.3-SNAPSHOT group=org.apache.iggy
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/async/SystemClient.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/async/SystemClient.java index 6eab11a..a0b1963 100644 --- a/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/async/SystemClient.java +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/async/SystemClient.java
@@ -22,6 +22,8 @@ import org.apache.iggy.cluster.ClusterMetadata; import org.apache.iggy.system.ClientInfo; import org.apache.iggy.system.ClientInfoDetails; +import org.apache.iggy.system.MetadataChanges; +import org.apache.iggy.system.MetadataChangesQuery; import org.apache.iggy.system.OptionSpec; import org.apache.iggy.system.OptionsScope; import org.apache.iggy.system.Stats; @@ -61,6 +63,19 @@ CompletableFuture<List<OptionSpec>> describeOptions(OptionsScope scope); /** + * Reads one page of the node's metadata change log, or of its current state, asynchronously. + * + * <p>A mirror keeps a cursor, asks for the records after it and stores the returned + * {@link MetadataChanges#next()}. On expired or reset it rebuilds from current state pages and follows the + * changes after the anchor they return. Requires root or the global manage servers permission. A query the rules + * refuse completes exceptionally with invalid command before anything is sent. + * + * @param query where to start, which groups, the time range and the page size + * @return A CompletableFuture containing the page + */ + CompletableFuture<MetadataChanges> getMetadataChanges(MetadataChangesQuery query); + + /** * Gets information about the current client asynchronously. * * @return A CompletableFuture containing current client details
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/async/tcp/SystemTcpClient.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/async/tcp/SystemTcpClient.java index 6ab6d6a..fef5752 100644 --- a/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/async/tcp/SystemTcpClient.java +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/async/tcp/SystemTcpClient.java
@@ -22,10 +22,14 @@ import io.netty.buffer.Unpooled; import org.apache.iggy.client.async.SystemClient; import org.apache.iggy.cluster.ClusterMetadata; +import org.apache.iggy.exception.IggyValidationException; import org.apache.iggy.serde.BytesDeserializer; +import org.apache.iggy.serde.BytesSerializer; import org.apache.iggy.serde.CommandCode; import org.apache.iggy.system.ClientInfo; import org.apache.iggy.system.ClientInfoDetails; +import org.apache.iggy.system.MetadataChanges; +import org.apache.iggy.system.MetadataChangesQuery; import org.apache.iggy.system.OptionSpec; import org.apache.iggy.system.OptionsScope; import org.apache.iggy.system.Stats; @@ -86,6 +90,27 @@ } @Override + public CompletableFuture<MetadataChanges> getMetadataChanges(MetadataChangesQuery query) { + try { + query.validate(); + } catch (IggyValidationException refused) { + return CompletableFuture.failedFuture(refused); + } + var payload = BytesSerializer.toGetMetadataChanges(query); + + log.debug("Reading metadata changes"); + + return AsyncTcpConnection.mapResponse( + connection().send(CommandCode.System.GET_METADATA_CHANGES.getValue(), payload), response -> { + try { + return BytesDeserializer.readMetadataChanges(response); + } finally { + response.release(); + } + }); + } + + @Override public CompletableFuture<ClusterMetadata> getClusterMetadata() { var payload = Unpooled.EMPTY_BUFFER;
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/blocking/SystemClient.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/blocking/SystemClient.java index 612c8c1..97ec047 100644 --- a/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/blocking/SystemClient.java +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/blocking/SystemClient.java
@@ -22,6 +22,8 @@ import org.apache.iggy.cluster.ClusterMetadata; import org.apache.iggy.system.ClientInfo; import org.apache.iggy.system.ClientInfoDetails; +import org.apache.iggy.system.MetadataChanges; +import org.apache.iggy.system.MetadataChangesQuery; import org.apache.iggy.system.OptionSpec; import org.apache.iggy.system.OptionsScope; import org.apache.iggy.system.Stats; @@ -44,6 +46,20 @@ */ List<OptionSpec> describeOptions(OptionsScope scope); + /** + * Reads one page of the node's metadata change log, or of its current state. + * + * <p>Every committed metadata operation that changes public state leaves records such as a stream created, a + * topic renamed or a session closed. A mirror keeps a cursor, asks for the records after it and stores the + * returned {@link MetadataChanges#next()}. On expired or reset it rebuilds from current state pages and follows + * the changes after the anchor they return. Requires root or the global manage servers permission, and the query + * is checked before anything is sent. + * + * @throws org.apache.iggy.exception.IggyValidationException invalid command for a query the rules refuse + * @throws org.apache.iggy.exception.IggyServerException unauthorized without the manage servers permission + */ + MetadataChanges getMetadataChanges(MetadataChangesQuery query); + ClientInfoDetails getMe(); ClientInfoDetails getClient(Long clientId);
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/blocking/http/MetadataChangesJson.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/blocking/http/MetadataChangesJson.java new file mode 100644 index 0000000..4b74cc9 --- /dev/null +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/blocking/http/MetadataChangesJson.java
@@ -0,0 +1,517 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.client.blocking.http; + +import org.apache.iggy.exception.IggyMalformedResponseException; +import org.apache.iggy.exception.IggyValidationException; +import org.apache.iggy.message.HeaderValue; +import org.apache.iggy.system.MetadataChangeBody; +import org.apache.iggy.system.MetadataChangeCursor; +import org.apache.iggy.system.MetadataChangeGroup; +import org.apache.iggy.system.MetadataChangeKind; +import org.apache.iggy.system.MetadataChangePhase; +import org.apache.iggy.system.MetadataChangeRecord; +import org.apache.iggy.system.MetadataChangeStatus; +import org.apache.iggy.system.MetadataChanges; +import org.apache.iggy.system.MetadataCurrentItem; +import org.apache.iggy.system.MetadataCurrentPageToken; +import org.apache.iggy.user.Permissions; +import org.apache.iggy.user.UserStatus; +import tools.jackson.core.JacksonException; +import tools.jackson.core.type.TypeReference; +import tools.jackson.databind.JsonNode; +import tools.jackson.databind.ObjectMapper; + +import java.math.BigInteger; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.function.Function; +import java.util.regex.Pattern; + +/** + * Reads the JSON answer of {@code GET /metadata/changes}. Records and items are flat objects of the envelope fields + * and the body fields, 128-bit values are 32 lowercase hex characters, and a record or item of a group or kind this + * build does not know is kept whole as {@link MetadataChangeBody.UnknownJson}. + */ +final class MetadataChangesJson { + + private static final Pattern HEX_U128 = Pattern.compile("[0-9a-f]{32}"); + private static final TypeReference<Map<String, HttpOptionValue>> OPTIONS = new TypeReference<>() {}; + private static final ObjectMapper OBJECT_MAPPER = ObjectMapperFactory.getInstance(); + + private MetadataChangesJson() {} + + static MetadataChanges read(JsonNode answer) { + Optional<BigInteger> logId = + present(answer, "log_id") ? Optional.of(hexU128(answer, "log_id")) : Optional.empty(); + Optional<MetadataChangeCursor> next = present(answer, "next") + ? Optional.of(parsed(answer, "next", MetadataChangeCursor::parse)) + : Optional.empty(); + Optional<MetadataCurrentPageToken> pageToken = present(answer, "current_page_token") + ? Optional.of(parsed(answer, "current_page_token", MetadataCurrentPageToken::parse)) + : Optional.empty(); + return new MetadataChanges( + MetadataChangeStatus.fromValue(text(answer, "status")) + .orElseThrow(() -> malformed("status is unknown")), + MetadataChangePhase.fromValue(text(answer, "phase")).orElseThrow(() -> malformed("phase is unknown")), + logId, + u64(answer, "oldest_op"), + u64(answer, "retained_since"), + u64(answer, "head_op"), + next, + bool(answer, "has_more"), + list(answer, "records", MetadataChangesJson::record), + list(answer, "items", MetadataChangesJson::item), + pageToken); + } + + static MetadataChangeRecord record(JsonNode record) { + String groupName = text(record, "group"); + String kindName = text(record, "kind"); + Optional<MetadataChangeKind> kind = knownKind(groupName, kindName); + MetadataChangeBody body = kind.map(known -> changeBody(known, record)) + .orElseGet( + () -> new MetadataChangeBody.UnknownJson(groupName, Optional.of(kindName), record.toString())); + Optional<Long> actorUserId = + present(record, "actor_user_id") ? Optional.of(u32(record, "actor_user_id")) : Optional.empty(); + return new MetadataChangeRecord( + u64(record, "op"), + u32(record, "ordinal"), + hexU128(record, "checksum"), + u64(record, "timestamp"), + actorUserId, + kind, + body); + } + + static MetadataCurrentItem item(JsonNode item) { + String groupName = text(item, "group"); + Optional<MetadataChangeGroup> group = MetadataChangeGroup.fromValue(groupName); + MetadataChangeBody body = group.map(known -> itemBody(known, item)) + .orElseGet(() -> new MetadataChangeBody.UnknownJson(groupName, Optional.empty(), item.toString())); + return new MetadataCurrentItem(group, body); + } + + private static Optional<MetadataChangeKind> knownKind(String groupName, String kindName) { + Optional<MetadataChangeGroup> group = MetadataChangeGroup.fromValue(groupName); + Optional<MetadataChangeKind> kind = MetadataChangeKind.fromValue(kindName); + if (group.isEmpty() || kind.isEmpty()) { + return Optional.empty(); + } + if (kind.get().group() != group.get()) { + throw malformed("change kind " + kindName + " does not belong to group " + groupName); + } + return kind; + } + + @SuppressWarnings("checkstyle:CyclomaticComplexity") + private static MetadataChangeBody changeBody(MetadataChangeKind kind, JsonNode body) { + return switch (kind) { + case STREAM_CREATED, STREAM_UPDATED -> stream(body); + case STREAM_DELETED -> streamDeleted(body); + case TOPIC_CREATED, TOPIC_UPDATED -> topic(body); + case TOPIC_DELETED -> topicDeleted(body); + case PARTITIONS_CHANGED -> partitionsChanged(body); + case CONSUMER_GROUP_CREATED -> consumerGroupCreated(body); + case CONSUMER_GROUP_DELETED -> consumerGroupDeleted(body); + case CONSUMER_GROUP_MEMBER_JOINED -> memberJoined(body); + case CONSUMER_GROUP_MEMBER_LEFT -> memberLeft(body); + case CONSUMER_GROUP_PARTITION_HANDED_OVER -> handedOver(body); + case USER_CREATED, USER_UPDATED -> user(body); + case USER_DELETED, USER_PASSWORD_CHANGED -> userIdentity(body); + case USER_PERMISSIONS_UPDATED -> userPermissions(body); + case PERSONAL_ACCESS_TOKEN_CREATED -> token(body); + case PERSONAL_ACCESS_TOKEN_DELETED -> tokenDeleted(body); + case SESSION_REGISTERED -> session(body); + case SESSION_CLOSED -> sessionClosed(body); + }; + } + + private static MetadataChangeBody itemBody(MetadataChangeGroup group, JsonNode body) { + return switch (group) { + case STREAMS -> stream(body); + case TOPICS -> topicItem(body); + case PARTITIONS -> partitionsItem(body); + case CONSUMER_GROUPS -> consumerGroupItem(body); + case CONSUMER_GROUP_MEMBERS -> memberItem(body); + case USERS -> user(body); + case PERSONAL_ACCESS_TOKENS -> token(body); + case SESSIONS -> session(body); + }; + } + + private static MetadataChangeBody stream(JsonNode body) { + var options = options(body); + return new MetadataChangeBody.StreamChange( + u32(body, "stream_id"), + u64(body, "stream_created_op"), + u64(body, "created_at"), + text(body, "name"), + options.explicit(), + options.derived(), + options.unreadable()); + } + + private static MetadataChangeBody streamDeleted(JsonNode body) { + return new MetadataChangeBody.StreamDeleted( + u32(body, "stream_id"), + u64(body, "stream_created_op"), + u64(body, "created_at"), + text(body, "name"), + optionalU64(body, "requested_op")); + } + + private static MetadataChangeBody topic(JsonNode body) { + var options = options(body); + return new MetadataChangeBody.TopicChange( + u32(body, "stream_id"), + u64(body, "stream_created_op"), + u32(body, "topic_id"), + u64(body, "topic_created_op"), + u64(body, "created_at"), + text(body, "name"), + u32(body, "partitions_count"), + options.explicit(), + options.derived(), + options.unreadable()); + } + + private static MetadataChangeBody topicItem(JsonNode body) { + var options = options(body); + return new MetadataChangeBody.TopicItem( + u32(body, "stream_id"), + u64(body, "stream_created_op"), + u32(body, "topic_id"), + u64(body, "topic_created_op"), + u64(body, "created_at"), + text(body, "name"), + options.explicit(), + options.derived(), + options.unreadable()); + } + + private static MetadataChangeBody topicDeleted(JsonNode body) { + return new MetadataChangeBody.TopicDeleted( + u32(body, "stream_id"), + u64(body, "stream_created_op"), + u32(body, "topic_id"), + u64(body, "topic_created_op"), + u64(body, "created_at"), + text(body, "name"), + optionalU64(body, "requested_op")); + } + + private static MetadataChangeBody partitionsChanged(JsonNode body) { + return new MetadataChangeBody.PartitionsChanged( + u32(body, "stream_id"), + u64(body, "stream_created_op"), + u32(body, "topic_id"), + u64(body, "topic_created_op"), + u32(body, "partitions_count"), + ids(body, "removed_partition_ids"), + optionalU64(body, "requested_op")); + } + + private static MetadataChangeBody partitionsItem(JsonNode body) { + return new MetadataChangeBody.PartitionsItem( + u32(body, "stream_id"), + u64(body, "stream_created_op"), + u32(body, "topic_id"), + u64(body, "topic_created_op"), + u32(body, "partitions_count")); + } + + private static MetadataChangeBody consumerGroupCreated(JsonNode body) { + return new MetadataChangeBody.ConsumerGroupCreated( + u32(body, "stream_id"), + u64(body, "stream_created_op"), + u32(body, "topic_id"), + u64(body, "topic_created_op"), + u32(body, "group_id"), + text(body, "name"), + u32(body, "consumer_groups_count")); + } + + private static MetadataChangeBody consumerGroupDeleted(JsonNode body) { + return new MetadataChangeBody.ConsumerGroupDeleted( + u32(body, "stream_id"), + u64(body, "stream_created_op"), + u32(body, "topic_id"), + u64(body, "topic_created_op"), + u32(body, "group_id"), + text(body, "name"), + u32(body, "consumer_groups_count"), + optionalU64(body, "requested_op")); + } + + private static MetadataChangeBody consumerGroupItem(JsonNode body) { + return new MetadataChangeBody.ConsumerGroupItem( + u32(body, "stream_id"), + u64(body, "stream_created_op"), + u32(body, "topic_id"), + u64(body, "topic_created_op"), + u32(body, "group_id"), + text(body, "name")); + } + + private static MetadataChangeBody memberJoined(JsonNode body) { + return new MetadataChangeBody.MemberJoined( + u32(body, "stream_id"), + u64(body, "stream_created_op"), + u32(body, "topic_id"), + u64(body, "topic_created_op"), + u32(body, "group_id"), + u32(body, "member_id"), + hexU128(body, "client_id"), + u32(body, "members_count"), + ids(body, "partition_ids")); + } + + private static MetadataChangeBody memberLeft(JsonNode body) { + return new MetadataChangeBody.MemberLeft( + u32(body, "stream_id"), + u64(body, "stream_created_op"), + u32(body, "topic_id"), + u64(body, "topic_created_op"), + u32(body, "group_id"), + u32(body, "member_id"), + hexU128(body, "client_id"), + u32(body, "members_count"), + reason(body, MetadataChangeBody.MemberLeftReason::fromValue)); + } + + private static MetadataChangeBody handedOver(JsonNode body) { + Optional<Long> memberId = present(body, "member_id") ? Optional.of(u32(body, "member_id")) : Optional.empty(); + return new MetadataChangeBody.PartitionHandedOver( + u32(body, "stream_id"), + u64(body, "stream_created_op"), + u32(body, "topic_id"), + u64(body, "topic_created_op"), + u32(body, "group_id"), + u32(body, "partition_id"), + memberId); + } + + private static MetadataChangeBody memberItem(JsonNode body) { + return new MetadataChangeBody.MemberItem( + u32(body, "stream_id"), + u64(body, "stream_created_op"), + u32(body, "topic_id"), + u64(body, "topic_created_op"), + u32(body, "group_id"), + u32(body, "member_id"), + hexU128(body, "client_id"), + ids(body, "partition_ids")); + } + + private static MetadataChangeBody user(JsonNode body) { + var options = options(body); + UserStatus status = switch (text(body, "status")) { + case "active" -> UserStatus.Active; + case "inactive" -> UserStatus.Inactive; + default -> throw malformed("user status " + text(body, "status") + " is unknown"); + }; + return new MetadataChangeBody.UserChange( + u32(body, "user_id"), + u64(body, "user_created_at"), + text(body, "username"), + status, + permissions(body), + options.explicit(), + options.derived(), + options.unreadable()); + } + + private static MetadataChangeBody userIdentity(JsonNode body) { + return new MetadataChangeBody.UserIdentity( + u32(body, "user_id"), u64(body, "user_created_at"), text(body, "username")); + } + + private static MetadataChangeBody userPermissions(JsonNode body) { + return new MetadataChangeBody.UserPermissionsUpdated( + u32(body, "user_id"), + u64(body, "user_created_at"), + text(body, "username"), + permissions(body), + reason(body, MetadataChangeBody.PermissionsUpdatedReason::fromValue)); + } + + private static MetadataChangeBody token(JsonNode body) { + return new MetadataChangeBody.TokenChange( + u32(body, "user_id"), u64(body, "user_created_at"), text(body, "name"), optionalU64(body, "expiry")); + } + + private static MetadataChangeBody tokenDeleted(JsonNode body) { + return new MetadataChangeBody.TokenDeleted( + u32(body, "user_id"), + u64(body, "user_created_at"), + text(body, "name"), + reason(body, MetadataChangeBody.TokenDeletedReason::fromValue)); + } + + private static MetadataChangeBody session(JsonNode body) { + return new MetadataChangeBody.Session( + u32(body, "user_id"), u64(body, "user_created_at"), hexU128(body, "client_id"), u64(body, "session")); + } + + private static MetadataChangeBody sessionClosed(JsonNode body) { + return new MetadataChangeBody.SessionClosed( + u32(body, "user_id"), + u64(body, "user_created_at"), + hexU128(body, "client_id"), + u64(body, "session"), + reason(body, MetadataChangeBody.SessionClosedReason::fromValue)); + } + + /** + * Reads {@code options} and {@code options_unreadable}: an object of {@code value}, {@code kind} and + * {@code explicit} entries with the flag absent or false, or null with the flag true. + */ + private static ChangeOptions options(JsonNode body) { + boolean unreadable = present(body, "options_unreadable") && bool(body, "options_unreadable"); + boolean readable = present(body, "options"); + if (readable == unreadable) { + throw malformed("options must be an object, or null with options_unreadable true"); + } + if (unreadable) { + return new ChangeOptions(Map.of(), Map.of(), true); + } + Map<String, HttpOptionValue> entries; + try { + entries = OBJECT_MAPPER.convertValue(body.get("options"), OPTIONS); + } catch (JacksonException | IllegalArgumentException invalid) { + throw new IggyMalformedResponseException("Malformed metadata changes answer: options", invalid); + } + return new ChangeOptions(HttpOptionValue.select(entries, true), HttpOptionValue.select(entries, false), false); + } + + private static Optional<Permissions> permissions(JsonNode body) { + if (!present(body, "permissions")) { + return Optional.empty(); + } + try { + return Optional.of(OBJECT_MAPPER.treeToValue(body.get("permissions"), Permissions.class)); + } catch (JacksonException | IllegalArgumentException invalid) { + throw new IggyMalformedResponseException("Malformed metadata changes answer: permissions", invalid); + } + } + + private static <T> T reason(JsonNode body, Function<String, Optional<T>> fromValue) { + String name = text(body, "reason"); + return fromValue.apply(name).orElseThrow(() -> malformed("reason " + name + " is unknown")); + } + + private static <T> List<T> list(JsonNode body, String field, Function<JsonNode, T> reader) { + JsonNode values = required(body, field); + if (!values.isArray()) { + throw malformed(field + " is not an array"); + } + List<T> result = new ArrayList<>(values.size()); + values.forEach(value -> result.add(reader.apply(value))); + return result; + } + + private static List<Long> ids(JsonNode body, String field) { + JsonNode values = required(body, field); + if (!values.isArray()) { + throw malformed(field + " is not an array"); + } + List<Long> result = new ArrayList<>(values.size()); + values.forEach(value -> result.add(unsigned(value, Integer.SIZE, field).longValueExact())); + return List.copyOf(result); + } + + private static BigInteger u64(JsonNode body, String field) { + return unsigned(required(body, field), Long.SIZE, field); + } + + private static long u32(JsonNode body, String field) { + return unsigned(required(body, field), Integer.SIZE, field).longValueExact(); + } + + private static Optional<BigInteger> optionalU64(JsonNode body, String field) { + return present(body, field) ? Optional.of(u64(body, field)) : Optional.empty(); + } + + private static BigInteger unsigned(JsonNode value, int bits, String field) { + if (!value.isIntegralNumber()) { + throw malformed(field + " is not an integer"); + } + BigInteger number = value.bigIntegerValue(); + if (number.signum() < 0 || number.bitLength() > bits) { + throw malformed(field + " is outside an unsigned " + bits + "-bit value"); + } + return number; + } + + private static <T> T parsed(JsonNode body, String field, Function<String, T> parser) { + try { + return parser.apply(text(body, field)); + } catch (IggyValidationException refused) { + throw new IggyMalformedResponseException("Malformed metadata changes answer: " + field, refused); + } + } + + private static BigInteger hexU128(JsonNode body, String field) { + String hex = text(body, field); + if (!HEX_U128.matcher(hex).matches()) { + throw malformed(field + " must be 32 lowercase hex characters"); + } + return new BigInteger(hex, 16); + } + + private static boolean bool(JsonNode body, String field) { + JsonNode value = required(body, field); + if (!value.isBoolean()) { + throw malformed(field + " is not a boolean"); + } + return value.booleanValue(); + } + + private static String text(JsonNode body, String field) { + JsonNode value = required(body, field); + if (!value.isString()) { + throw malformed(field + " is not a string"); + } + return value.stringValue(); + } + + private static boolean present(JsonNode body, String field) { + JsonNode value = body.get(field); + return value != null && !value.isNull(); + } + + private static JsonNode required(JsonNode body, String field) { + JsonNode value = body.get(field); + if (value == null) { + throw malformed(field + " is missing"); + } + return value; + } + + private static IggyMalformedResponseException malformed(String reason) { + return new IggyMalformedResponseException("Malformed metadata changes answer: " + reason); + } + + private record ChangeOptions( + Map<String, HeaderValue> explicit, Map<String, HeaderValue> derived, boolean unreadable) {} +}
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/blocking/http/SystemHttpClient.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/blocking/http/SystemHttpClient.java index 2e88c44..0767db9 100644 --- a/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/blocking/http/SystemHttpClient.java +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/blocking/http/SystemHttpClient.java
@@ -26,10 +26,13 @@ import org.apache.iggy.message.HeaderValue; import org.apache.iggy.system.ClientInfo; import org.apache.iggy.system.ClientInfoDetails; +import org.apache.iggy.system.MetadataChanges; +import org.apache.iggy.system.MetadataChangesQuery; import org.apache.iggy.system.OptionSpec; import org.apache.iggy.system.OptionsScope; import org.apache.iggy.system.Stats; import tools.jackson.core.type.TypeReference; +import tools.jackson.databind.JsonNode; import java.util.List; import java.util.Locale; @@ -41,6 +44,7 @@ private static final String CLIENTS = "/clients"; private static final String PING = "/ping"; private static final String OPTIONS = "/options"; + private static final String METADATA_CHANGES = "/metadata/changes"; private final InternalHttpClient httpClient; public SystemHttpClient(InternalHttpClient httpClient) { @@ -67,6 +71,13 @@ } @Override + public MetadataChanges getMetadataChanges(MetadataChangesQuery query) { + query.validate(); + var request = httpClient.prepareGetRequest(METADATA_CHANGES + "?" + query.httpQuery()); + return MetadataChangesJson.read(httpClient.execute(request, JsonNode.class)); + } + + @Override public ClientInfoDetails getMe() { throw new IggyOperationNotSupportedException("getMe", "HTTP"); }
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/blocking/tcp/SystemTcpClient.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/blocking/tcp/SystemTcpClient.java index 59f4336..a564e1a 100644 --- a/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/blocking/tcp/SystemTcpClient.java +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/client/blocking/tcp/SystemTcpClient.java
@@ -23,6 +23,8 @@ import org.apache.iggy.cluster.ClusterMetadata; import org.apache.iggy.system.ClientInfo; import org.apache.iggy.system.ClientInfoDetails; +import org.apache.iggy.system.MetadataChanges; +import org.apache.iggy.system.MetadataChangesQuery; import org.apache.iggy.system.OptionSpec; import org.apache.iggy.system.OptionsScope; import org.apache.iggy.system.Stats; @@ -53,6 +55,11 @@ } @Override + public MetadataChanges getMetadataChanges(MetadataChangesQuery query) { + return FutureUtil.resolve(delegate.getMetadataChanges(query)); + } + + @Override public ClientInfoDetails getMe() { return FutureUtil.resolve(delegate.getMe()); }
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/serde/BytesDeserializer.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/serde/BytesDeserializer.java index e8c8828..ed543f7 100644 --- a/foreign/java/java-sdk/src/main/java/org/apache/iggy/serde/BytesDeserializer.java +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/serde/BytesDeserializer.java
@@ -53,6 +53,10 @@ import org.apache.iggy.system.ClientInfo; import org.apache.iggy.system.ClientInfoDetails; import org.apache.iggy.system.ConsumerGroupInfo; +import org.apache.iggy.system.MetadataChangeBody; +import org.apache.iggy.system.MetadataChangeRecord; +import org.apache.iggy.system.MetadataChanges; +import org.apache.iggy.system.MetadataCurrentItem; import org.apache.iggy.system.OptionSpec; import org.apache.iggy.system.Stats; import org.apache.iggy.topic.Topic; @@ -73,6 +77,7 @@ import java.util.List; import java.util.Map; import java.util.Optional; +import java.util.function.Supplier; /** * Unified deserializer for both blocking and async clients. @@ -561,6 +566,35 @@ replicaId, name, ip, new TransportEndpoints(tcpPort, quicPort, httpPort, websocketPort), role, status); } + /** + * Reads a metadata changes answer. A group or kind this build does not know is kept with its raw body, while a + * truncated answer, a body with trailing bytes, an unknown reason code and a known kind under the wrong group + * are malformed. + */ + public static MetadataChanges readMetadataChanges(ByteBuf response) { + return readMetadataChangesPart(() -> MetadataChangesReader.readAnswer(response)); + } + + /** Reads one change record envelope with its body. */ + public static MetadataChangeRecord readMetadataChangeRecord(ByteBuf record) { + return readMetadataChangesPart(() -> MetadataChangesReader.readRecord(record)); + } + + /** Reads one current state item envelope with its body. */ + public static MetadataCurrentItem readMetadataCurrentItem(ByteBuf item) { + return readMetadataChangesPart(() -> MetadataChangesReader.readItem(item)); + } + + /** Decodes a change body under its group and kind codes, all of {@code body} and nothing past it. */ + public static MetadataChangeBody readMetadataChangeBody(int group, int kind, ByteBuf body) { + return readMetadataChangesPart(() -> MetadataChangesReader.readChangeBody(group, kind, body)); + } + + /** Decodes a current state item body under its group code, all of {@code body} and nothing past it. */ + public static MetadataChangeBody readMetadataItemBody(int group, ByteBuf body) { + return readMetadataChangesPart(() -> MetadataChangesReader.readItemBody(group, body)); + } + public static UserInfoDetails readUserInfoDetails(ByteBuf response) { var userInfo = readUserInfo(response); @@ -673,7 +707,7 @@ * build has no name for is dropped rather than failing the whole response - the wire * contract forwards unknown value kinds so a mixed-version cluster can round-trip them. */ - private static Map<String, HeaderValue> readOptionsBlock(ByteBuf buffer, String field) { + static Map<String, HeaderValue> readOptionsBlock(ByteBuf buffer, String field) { if (buffer.readableBytes() < Integer.BYTES) { throw new IggyMalformedResponseException("Missing length prefix for " + field); } @@ -752,7 +786,7 @@ return buffer.readCharSequence(toInt(length), StandardCharsets.UTF_8).toString(); } - private static int validatedCollectionSize(long count, int readableBytes, int entryBytes, String field) { + static int validatedCollectionSize(long count, int readableBytes, int entryBytes, String field) { if (count > readableBytes / entryBytes) { throw new IggyMalformedResponseException( field + " " + count + " exceeds remaining payload of " + readableBytes + " bytes"); @@ -760,6 +794,14 @@ return Math.toIntExact(count); } + private static <T> T readMetadataChangesPart(Supplier<T> reader) { + try { + return reader.get(); + } catch (IndexOutOfBoundsException truncated) { + throw new IggyMalformedResponseException("Malformed metadata changes answer: it is truncated", truncated); + } + } + static BigInteger readU64AsBigInteger(ByteBuf buffer) { var bytesArray = new byte[8]; buffer.readBytes(bytesArray, 0, 8);
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/serde/BytesSerializer.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/serde/BytesSerializer.java index b0bf015..7b38d32 100644 --- a/foreign/java/java-sdk/src/main/java/org/apache/iggy/serde/BytesSerializer.java +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/serde/BytesSerializer.java
@@ -38,6 +38,11 @@ import org.apache.iggy.message.Partitioning; import org.apache.iggy.message.PollingStrategy; import org.apache.iggy.stream.StreamUpdate; +import org.apache.iggy.system.MetadataChangeCursor; +import org.apache.iggy.system.MetadataChangeGroup; +import org.apache.iggy.system.MetadataChangeStart; +import org.apache.iggy.system.MetadataChangesQuery; +import org.apache.iggy.system.MetadataCurrentPageToken; import org.apache.iggy.system.OptionsScope; import org.apache.iggy.topic.TopicUpdate; import org.apache.iggy.user.GlobalPermissions; @@ -278,6 +283,36 @@ } /** + * {@code GetMetadataChanges}: {@code [start:u8][cursor:44][has_since:u8][since:u64]?[has_until:u8][until:u64]?} + * {@code [limit:u32][groups_count:u8][group:u8]*[token_len:u16][token]}. The cursor is all zero unless the query + * starts after one, and groups keep the caller's order. Validate the query first. + */ + public static ByteBuf toGetMetadataChanges(MetadataChangesQuery query) { + var start = query.start(); + byte[] cursor = start instanceof MetadataChangeStart.After after + ? after.cursor().toBytes() + : new byte[MetadataChangeCursor.SIZE]; + byte[] token = start instanceof MetadataChangeStart.Current current + ? current.pageToken().map(MetadataCurrentPageToken::payload).orElse(new byte[0]) + : new byte[0]; + List<MetadataChangeGroup> groups = + query.groups().size() > 255 ? query.groups().stream().distinct().toList() : query.groups(); + var payload = Unpooled.buffer(1 + MetadataChangeCursor.SIZE + 9 + 9 + 4 + 1 + groups.size() + 2 + token.length); + payload.writeByte(start.asCode()); + payload.writeBytes(cursor); + for (Optional<BigInteger> bound : List.of(query.since(), query.until())) { + payload.writeByte(bound.isPresent() ? 1 : 0); + bound.ifPresent(value -> writeAndRelease(payload, toBytesAsU64(value))); + } + payload.writeIntLE((int) query.limit()); + payload.writeByte(groups.size()); + groups.forEach(group -> payload.writeByte(group.asCode())); + payload.writeShortLE(token.length); + payload.writeBytes(token); + return payload; + } + + /** * {@code CreateTopic}: {@code [stream_id][partitions_count:u32][name][options to end]}. * {@code partitions_count} is an argument of the command, never an option: admission consumes it to * compute partition assignments.
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/serde/CommandCode.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/serde/CommandCode.java index b4ebe46..36993a0 100644 --- a/foreign/java/java-sdk/src/main/java/org/apache/iggy/serde/CommandCode.java +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/serde/CommandCode.java
@@ -34,6 +34,7 @@ DESCRIBE_OPTIONS(13), BIND_SESSION(15), UPDATE_RESOURCE(16), + GET_METADATA_CHANGES(17), GET_ME(20), GET_CLIENT(21), GET_ALL_CLIENTS(22);
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/serde/MetadataChangesReader.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/serde/MetadataChangesReader.java new file mode 100644 index 0000000..aa204e8 --- /dev/null +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/serde/MetadataChangesReader.java
@@ -0,0 +1,620 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.serde; + +import io.netty.buffer.ByteBuf; +import org.apache.iggy.exception.IggyInvalidArgumentException; +import org.apache.iggy.exception.IggyMalformedResponseException; +import org.apache.iggy.message.HeaderValue; +import org.apache.iggy.system.MetadataChangeBody; +import org.apache.iggy.system.MetadataChangeCursor; +import org.apache.iggy.system.MetadataChangeGroup; +import org.apache.iggy.system.MetadataChangeKind; +import org.apache.iggy.system.MetadataChangePhase; +import org.apache.iggy.system.MetadataChangeRecord; +import org.apache.iggy.system.MetadataChangeStatus; +import org.apache.iggy.system.MetadataChanges; +import org.apache.iggy.system.MetadataCurrentItem; +import org.apache.iggy.system.MetadataCurrentPageToken; +import org.apache.iggy.user.Permissions; +import org.apache.iggy.user.StreamPermissions; +import org.apache.iggy.user.UserStatus; + +import java.math.BigInteger; +import java.nio.charset.CharacterCodingException; +import java.nio.charset.CodingErrorAction; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.function.Function; +import java.util.function.IntFunction; + +/** + * Decodes metadata changes answers. Bodies are decoded strictly: unknown reason codes, a known kind under the + * wrong group, presence bytes other than 0 and 1 and trailing bytes are refused, while a group or kind this build + * does not know is kept with its raw body. + */ +final class MetadataChangesReader { + + private static final int RECORD_MIN_BYTES = 8 + 4 + 16 + 8 + 1 + 1 + 1 + 4; + private static final int ITEM_MIN_BYTES = 1 + 4; + private static final int U128_BYTES = 16; + + private MetadataChangesReader() {} + + static MetadataChanges readAnswer(ByteBuf buffer) { + var status = MetadataChangeStatus.fromCode(buffer.readUnsignedByte()) + .orElseThrow(() -> malformed("the status is unknown")); + var phase = MetadataChangePhase.fromCode(buffer.readUnsignedByte()) + .orElseThrow(() -> malformed("the phase is unknown")); + Optional<BigInteger> logId = flag(buffer, "log id presence") ? Optional.of(u128(buffer)) : Optional.empty(); + var oldestOp = BytesDeserializer.readU64AsBigInteger(buffer); + var retainedSince = BytesDeserializer.readU64AsBigInteger(buffer); + var headOp = BytesDeserializer.readU64AsBigInteger(buffer); + Optional<MetadataChangeCursor> next = + flag(buffer, "next cursor presence") ? Optional.of(cursor(buffer)) : Optional.empty(); + boolean hasMore = flag(buffer, "has more"); + List<MetadataChangeRecord> records = + readList(buffer, RECORD_MIN_BYTES, "records", MetadataChangesReader::readRecord); + List<MetadataCurrentItem> items = readList(buffer, ITEM_MIN_BYTES, "items", MetadataChangesReader::readItem); + Optional<MetadataCurrentPageToken> pageToken = pageToken(buffer); + boolean holdsOtherPhase = + phase == MetadataChangePhase.CHANGES ? !items.isEmpty() || pageToken.isPresent() : !records.isEmpty(); + if (holdsOtherPhase) { + throw malformed("a changes page carries only records and a current page only items"); + } + return new MetadataChanges( + status, phase, logId, oldestOp, retainedSince, headOp, next, hasMore, records, items, pageToken); + } + + static MetadataChangeRecord readRecord(ByteBuf buffer) { + var op = BytesDeserializer.readU64AsBigInteger(buffer); + long ordinal = buffer.readUnsignedIntLE(); + var checksum = u128(buffer); + var timestamp = BytesDeserializer.readU64AsBigInteger(buffer); + Optional<Long> actorUserId = + flag(buffer, "actor presence") ? Optional.of(buffer.readUnsignedIntLE()) : Optional.empty(); + int group = buffer.readUnsignedByte(); + int kind = buffer.readUnsignedByte(); + ByteBuf body = buffer.readSlice(length(buffer)); + return new MetadataChangeRecord( + op, + ordinal, + checksum, + timestamp, + actorUserId, + knownKind(group, kind), + readChangeBody(group, kind, body)); + } + + static MetadataCurrentItem readItem(ByteBuf buffer) { + int group = buffer.readUnsignedByte(); + ByteBuf body = buffer.readSlice(length(buffer)); + return new MetadataCurrentItem(MetadataChangeGroup.fromCode(group), readItemBody(group, body)); + } + + static MetadataChangeBody readChangeBody(int group, int kind, ByteBuf body) { + return knownKind(group, kind) + .map(known -> exact(body, changeBody(known, body))) + .orElseGet(() -> new MetadataChangeBody.Unknown(group, Optional.of(kind), bytes(body))); + } + + static MetadataChangeBody readItemBody(int group, ByteBuf body) { + return MetadataChangeGroup.fromCode(group) + .map(known -> exact(body, itemBody(known, body))) + .orElseGet(() -> new MetadataChangeBody.Unknown(group, Optional.empty(), bytes(body))); + } + + private static Optional<MetadataChangeKind> knownKind(int group, int kind) { + Optional<MetadataChangeGroup> knownGroup = MetadataChangeGroup.fromCode(group); + Optional<MetadataChangeKind> knownKind = MetadataChangeKind.fromCode(kind); + if (knownGroup.isEmpty() || knownKind.isEmpty()) { + return Optional.empty(); + } + if (knownKind.get().group() != knownGroup.get()) { + throw malformed("change kind " + kind + " does not belong to group " + group); + } + return knownKind; + } + + private static MetadataChangeBody exact(ByteBuf body, MetadataChangeBody decoded) { + if (body.isReadable()) { + throw malformed(body.readableBytes() + " trailing bytes after a body"); + } + return decoded; + } + + private static <T> List<T> readList(ByteBuf buffer, int minimumBytes, String field, Function<ByteBuf, T> reader) { + int count = BytesDeserializer.validatedCollectionSize( + buffer.readUnsignedIntLE(), buffer.readableBytes(), minimumBytes, "Metadata changes " + field); + List<T> values = new ArrayList<>(count); + for (int index = 0; index < count; index++) { + values.add(reader.apply(buffer)); + } + return values; + } + + private static Optional<MetadataCurrentPageToken> pageToken(ByteBuf buffer) { + int length = buffer.readUnsignedShortLE(); + if (length == 0) { + return Optional.empty(); + } + byte[] payload = new byte[length]; + buffer.readBytes(payload); + return Optional.of(MetadataCurrentPageToken.fromPayload(payload) + .orElseThrow(() -> malformed("the page token is malformed"))); + } + + @SuppressWarnings("checkstyle:CyclomaticComplexity") + private static MetadataChangeBody changeBody(MetadataChangeKind kind, ByteBuf body) { + return switch (kind) { + case STREAM_CREATED, STREAM_UPDATED -> stream(body); + case STREAM_DELETED -> streamDeleted(body); + case TOPIC_CREATED, TOPIC_UPDATED -> topic(body); + case TOPIC_DELETED -> topicDeleted(body); + case PARTITIONS_CHANGED -> partitionsChanged(body); + case CONSUMER_GROUP_CREATED -> consumerGroupCreated(body); + case CONSUMER_GROUP_DELETED -> consumerGroupDeleted(body); + case CONSUMER_GROUP_MEMBER_JOINED -> memberJoined(body); + case CONSUMER_GROUP_MEMBER_LEFT -> memberLeft(body); + case CONSUMER_GROUP_PARTITION_HANDED_OVER -> handedOver(body); + case USER_CREATED, USER_UPDATED -> user(body); + case USER_DELETED, USER_PASSWORD_CHANGED -> userIdentity(body); + case USER_PERMISSIONS_UPDATED -> userPermissions(body); + case PERSONAL_ACCESS_TOKEN_CREATED -> token(body); + case PERSONAL_ACCESS_TOKEN_DELETED -> tokenDeleted(body); + case SESSION_REGISTERED -> session(body); + case SESSION_CLOSED -> sessionClosed(body); + }; + } + + private static MetadataChangeBody itemBody(MetadataChangeGroup group, ByteBuf body) { + return switch (group) { + case STREAMS -> stream(body); + case TOPICS -> topicItem(body); + case PARTITIONS -> partitionsItem(body); + case CONSUMER_GROUPS -> consumerGroupItem(body); + case CONSUMER_GROUP_MEMBERS -> memberItem(body); + case USERS -> user(body); + case PERSONAL_ACCESS_TOKENS -> token(body); + case SESSIONS -> session(body); + }; + } + + private static MetadataChangeBody stream(ByteBuf body) { + long streamId = body.readUnsignedIntLE(); + var streamCreatedOp = BytesDeserializer.readU64AsBigInteger(body); + var createdAt = BytesDeserializer.readU64AsBigInteger(body); + var name = name(body); + var options = options(body); + return new MetadataChangeBody.StreamChange( + streamId, + streamCreatedOp, + createdAt, + name, + options.explicit(), + options.derived(), + options.unreadable()); + } + + private static MetadataChangeBody streamDeleted(ByteBuf body) { + long streamId = body.readUnsignedIntLE(); + var streamCreatedOp = BytesDeserializer.readU64AsBigInteger(body); + var createdAt = BytesDeserializer.readU64AsBigInteger(body); + var name = name(body); + return new MetadataChangeBody.StreamDeleted(streamId, streamCreatedOp, createdAt, name, optionalU64(body)); + } + + private static MetadataChangeBody topic(ByteBuf body) { + var key = TopicKey.read(body); + var createdAt = BytesDeserializer.readU64AsBigInteger(body); + var name = name(body); + long partitionsCount = body.readUnsignedIntLE(); + var options = options(body); + return new MetadataChangeBody.TopicChange( + key.streamId(), + key.streamCreatedOp(), + key.topicId(), + key.topicCreatedOp(), + createdAt, + name, + partitionsCount, + options.explicit(), + options.derived(), + options.unreadable()); + } + + private static MetadataChangeBody topicItem(ByteBuf body) { + var key = TopicKey.read(body); + var createdAt = BytesDeserializer.readU64AsBigInteger(body); + var name = name(body); + var options = options(body); + return new MetadataChangeBody.TopicItem( + key.streamId(), + key.streamCreatedOp(), + key.topicId(), + key.topicCreatedOp(), + createdAt, + name, + options.explicit(), + options.derived(), + options.unreadable()); + } + + private static MetadataChangeBody topicDeleted(ByteBuf body) { + var key = TopicKey.read(body); + var createdAt = BytesDeserializer.readU64AsBigInteger(body); + var name = name(body); + return new MetadataChangeBody.TopicDeleted( + key.streamId(), + key.streamCreatedOp(), + key.topicId(), + key.topicCreatedOp(), + createdAt, + name, + optionalU64(body)); + } + + private static MetadataChangeBody partitionsChanged(ByteBuf body) { + var key = TopicKey.read(body); + long partitionsCount = body.readUnsignedIntLE(); + var removed = partitionIds(body); + return new MetadataChangeBody.PartitionsChanged( + key.streamId(), + key.streamCreatedOp(), + key.topicId(), + key.topicCreatedOp(), + partitionsCount, + removed, + optionalU64(body)); + } + + private static MetadataChangeBody partitionsItem(ByteBuf body) { + var key = TopicKey.read(body); + return new MetadataChangeBody.PartitionsItem( + key.streamId(), key.streamCreatedOp(), key.topicId(), key.topicCreatedOp(), body.readUnsignedIntLE()); + } + + private static MetadataChangeBody consumerGroupCreated(ByteBuf body) { + var key = TopicKey.read(body); + long groupId = body.readUnsignedIntLE(); + var name = name(body); + return new MetadataChangeBody.ConsumerGroupCreated( + key.streamId(), + key.streamCreatedOp(), + key.topicId(), + key.topicCreatedOp(), + groupId, + name, + body.readUnsignedIntLE()); + } + + private static MetadataChangeBody consumerGroupDeleted(ByteBuf body) { + var key = TopicKey.read(body); + long groupId = body.readUnsignedIntLE(); + var name = name(body); + long consumerGroupsCount = body.readUnsignedIntLE(); + return new MetadataChangeBody.ConsumerGroupDeleted( + key.streamId(), + key.streamCreatedOp(), + key.topicId(), + key.topicCreatedOp(), + groupId, + name, + consumerGroupsCount, + optionalU64(body)); + } + + private static MetadataChangeBody consumerGroupItem(ByteBuf body) { + var key = TopicKey.read(body); + long groupId = body.readUnsignedIntLE(); + return new MetadataChangeBody.ConsumerGroupItem( + key.streamId(), key.streamCreatedOp(), key.topicId(), key.topicCreatedOp(), groupId, name(body)); + } + + private static MetadataChangeBody memberJoined(ByteBuf body) { + var key = TopicKey.read(body); + long groupId = body.readUnsignedIntLE(); + long memberId = body.readUnsignedIntLE(); + var clientId = u128(body); + long membersCount = body.readUnsignedIntLE(); + return new MetadataChangeBody.MemberJoined( + key.streamId(), + key.streamCreatedOp(), + key.topicId(), + key.topicCreatedOp(), + groupId, + memberId, + clientId, + membersCount, + partitionIds(body)); + } + + private static MetadataChangeBody memberLeft(ByteBuf body) { + var key = TopicKey.read(body); + long groupId = body.readUnsignedIntLE(); + long memberId = body.readUnsignedIntLE(); + var clientId = u128(body); + long membersCount = body.readUnsignedIntLE(); + return new MetadataChangeBody.MemberLeft( + key.streamId(), + key.streamCreatedOp(), + key.topicId(), + key.topicCreatedOp(), + groupId, + memberId, + clientId, + membersCount, + reason(body, MetadataChangeBody.MemberLeftReason::fromCode, "member left reason")); + } + + private static MetadataChangeBody handedOver(ByteBuf body) { + var key = TopicKey.read(body); + long groupId = body.readUnsignedIntLE(); + long partitionId = body.readUnsignedIntLE(); + Optional<Long> memberId = + flag(body, "member presence") ? Optional.of(body.readUnsignedIntLE()) : Optional.empty(); + return new MetadataChangeBody.PartitionHandedOver( + key.streamId(), + key.streamCreatedOp(), + key.topicId(), + key.topicCreatedOp(), + groupId, + partitionId, + memberId); + } + + private static MetadataChangeBody memberItem(ByteBuf body) { + var key = TopicKey.read(body); + long groupId = body.readUnsignedIntLE(); + long memberId = body.readUnsignedIntLE(); + var clientId = u128(body); + return new MetadataChangeBody.MemberItem( + key.streamId(), + key.streamCreatedOp(), + key.topicId(), + key.topicCreatedOp(), + groupId, + memberId, + clientId, + partitionIds(body)); + } + + private static MetadataChangeBody user(ByteBuf body) { + long userId = body.readUnsignedIntLE(); + var userCreatedAt = BytesDeserializer.readU64AsBigInteger(body); + var username = name(body); + UserStatus status = userStatus(body.readUnsignedByte()); + var permissions = permissions(body); + var options = options(body); + return new MetadataChangeBody.UserChange( + userId, + userCreatedAt, + username, + status, + permissions, + options.explicit(), + options.derived(), + options.unreadable()); + } + + private static MetadataChangeBody userIdentity(ByteBuf body) { + long userId = body.readUnsignedIntLE(); + var userCreatedAt = BytesDeserializer.readU64AsBigInteger(body); + return new MetadataChangeBody.UserIdentity(userId, userCreatedAt, name(body)); + } + + private static MetadataChangeBody userPermissions(ByteBuf body) { + long userId = body.readUnsignedIntLE(); + var userCreatedAt = BytesDeserializer.readU64AsBigInteger(body); + var username = name(body); + var permissions = permissions(body); + return new MetadataChangeBody.UserPermissionsUpdated( + userId, + userCreatedAt, + username, + permissions, + reason(body, MetadataChangeBody.PermissionsUpdatedReason::fromCode, "permissions updated reason")); + } + + private static MetadataChangeBody token(ByteBuf body) { + long userId = body.readUnsignedIntLE(); + var userCreatedAt = BytesDeserializer.readU64AsBigInteger(body); + var name = name(body); + var expiry = BytesDeserializer.readU64AsBigInteger(body); + return new MetadataChangeBody.TokenChange( + userId, userCreatedAt, name, expiry.signum() == 0 ? Optional.empty() : Optional.of(expiry)); + } + + private static MetadataChangeBody tokenDeleted(ByteBuf body) { + long userId = body.readUnsignedIntLE(); + var userCreatedAt = BytesDeserializer.readU64AsBigInteger(body); + var name = name(body); + return new MetadataChangeBody.TokenDeleted( + userId, + userCreatedAt, + name, + reason(body, MetadataChangeBody.TokenDeletedReason::fromCode, "token deleted reason")); + } + + private static MetadataChangeBody session(ByteBuf body) { + long userId = body.readUnsignedIntLE(); + var userCreatedAt = BytesDeserializer.readU64AsBigInteger(body); + var clientId = u128(body); + return new MetadataChangeBody.Session( + userId, userCreatedAt, clientId, BytesDeserializer.readU64AsBigInteger(body)); + } + + private static MetadataChangeBody sessionClosed(ByteBuf body) { + long userId = body.readUnsignedIntLE(); + var userCreatedAt = BytesDeserializer.readU64AsBigInteger(body); + var clientId = u128(body); + var session = BytesDeserializer.readU64AsBigInteger(body); + return new MetadataChangeBody.SessionClosed( + userId, + userCreatedAt, + clientId, + session, + reason(body, MetadataChangeBody.SessionClosedReason::fromCode, "session closed reason")); + } + + private static UserStatus userStatus(int code) { + try { + return UserStatus.fromCode(code); + } catch (IggyInvalidArgumentException unknown) { + throw malformed("user status " + code + " is unknown"); + } + } + + /** + * Reads {@code [state:u8]}: 0 followed by the explicit and derived option blocks exactly as the read responses + * carry them, or 1 for stored state the server cannot read. + */ + private static ChangeOptions options(ByteBuf body) { + int state = body.readUnsignedByte(); + if (state == 1) { + return new ChangeOptions(Map.of(), Map.of(), true); + } + if (state != 0) { + throw malformed("options state " + state + " is unknown"); + } + var explicit = BytesDeserializer.readOptionsBlock(body, "change explicit options"); + var derived = BytesDeserializer.readOptionsBlock(body, "change derived options"); + return new ChangeOptions(explicit, derived, false); + } + + /** Reads the optional public permission set: ten global flags, then streams and their topics. */ + private static Optional<Permissions> permissions(ByteBuf body) { + if (!flag(body, "permissions presence")) { + return Optional.empty(); + } + var global = BytesDeserializer.readGlobalPermissions(body); + Map<Long, StreamPermissions> streams = new HashMap<>(); + while (body.readBoolean()) { + long streamId = body.readUnsignedIntLE(); + streams.put(streamId, BytesDeserializer.readStreamPermissions(body)); + } + return Optional.of(new Permissions(global, streams)); + } + + /** Reads {@code [count:u32][id:u32]*}, refusing ids that are not strictly ascending. */ + private static List<Long> partitionIds(ByteBuf body) { + int count = BytesDeserializer.validatedCollectionSize( + body.readUnsignedIntLE(), body.readableBytes(), Integer.BYTES, "Metadata change partition ids"); + List<Long> ids = new ArrayList<>(count); + for (int index = 0; index < count; index++) { + long id = body.readUnsignedIntLE(); + if (!ids.isEmpty() && ids.get(ids.size() - 1) >= id) { + throw malformed("partition ids must be strictly ascending"); + } + ids.add(id); + } + return List.copyOf(ids); + } + + private static <T> T reason(ByteBuf body, IntFunction<Optional<T>> fromCode, String field) { + int code = body.readUnsignedByte(); + return fromCode.apply(code).orElseThrow(() -> malformed(field + " " + code + " is unknown")); + } + + private static boolean flag(ByteBuf buffer, String field) { + int value = buffer.readUnsignedByte(); + if (value > 1) { + throw malformed(field + " byte " + value + " is neither 0 nor 1"); + } + return value == 1; + } + + private static Optional<BigInteger> optionalU64(ByteBuf body) { + return flag(body, "requested op presence") + ? Optional.of(BytesDeserializer.readU64AsBigInteger(body)) + : Optional.empty(); + } + + private static String name(ByteBuf body) { + int length = body.readUnsignedByte(); + if (length == 0) { + throw malformed("a name is empty"); + } + try { + return StandardCharsets.UTF_8 + .newDecoder() + .onMalformedInput(CodingErrorAction.REPORT) + .onUnmappableCharacter(CodingErrorAction.REPORT) + .decode(body.readSlice(length).nioBuffer()) + .toString(); + } catch (CharacterCodingException notUtf8) { + throw malformed("a name is not UTF-8"); + } + } + + private static MetadataChangeCursor cursor(ByteBuf buffer) { + byte[] bytes = new byte[MetadataChangeCursor.SIZE]; + buffer.readBytes(bytes); + var cursor = MetadataChangeCursor.fromBytes(bytes, 0); + if (!cursor.isCanonical()) { + throw malformed("a cursor at op 0 carries a checksum or ordinal"); + } + return cursor; + } + + static BigInteger u128(ByteBuf buffer) { + byte[] bytes = new byte[U128_BYTES]; + buffer.readBytes(bytes); + byte[] bigEndian = new byte[U128_BYTES]; + for (int index = 0; index < U128_BYTES; index++) { + bigEndian[U128_BYTES - 1 - index] = bytes[index]; + } + return new BigInteger(1, bigEndian); + } + + private static int length(ByteBuf buffer) { + long length = buffer.readUnsignedIntLE(); + if (length > buffer.readableBytes()) { + throw malformed("a body of " + length + " bytes overruns the remaining " + buffer.readableBytes()); + } + return (int) length; + } + + private static byte[] bytes(ByteBuf body) { + byte[] bytes = new byte[body.readableBytes()]; + body.readBytes(bytes); + return bytes; + } + + static IggyMalformedResponseException malformed(String reason) { + return new IggyMalformedResponseException("Malformed metadata changes answer: " + reason); + } + + private record ChangeOptions( + Map<String, HeaderValue> explicit, Map<String, HeaderValue> derived, boolean unreadable) {} + + private record TopicKey(long streamId, BigInteger streamCreatedOp, long topicId, BigInteger topicCreatedOp) { + static TopicKey read(ByteBuf body) { + long streamId = body.readUnsignedIntLE(); + var streamCreatedOp = BytesDeserializer.readU64AsBigInteger(body); + long topicId = body.readUnsignedIntLE(); + return new TopicKey(streamId, streamCreatedOp, topicId, BytesDeserializer.readU64AsBigInteger(body)); + } + } +}
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/stream/EncryptionProjection.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/stream/EncryptionProjection.java index 885a050..d7879e4 100644 --- a/foreign/java/java-sdk/src/main/java/org/apache/iggy/stream/EncryptionProjection.java +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/stream/EncryptionProjection.java
@@ -28,15 +28,16 @@ import java.util.Optional; /** Reads the encryption projections a server reports in a stream's derived options. */ -final class EncryptionProjection { +public final class EncryptionProjection { private EncryptionProjection() {} - static Optional<EncryptionKeySource> source(Map<String, HeaderValue> options, Map<String, HeaderValue> derived) { + public static Optional<EncryptionKeySource> source( + Map<String, HeaderValue> options, Map<String, HeaderValue> derived) { return text(options, derived, StreamOptionKey.ENCRYPTION_KEY_SOURCE).flatMap(EncryptionKeySource::fromValue); } - static Optional<String> keyFingerprint(Map<String, HeaderValue> options, Map<String, HeaderValue> derived) { + public static Optional<String> keyFingerprint(Map<String, HeaderValue> options, Map<String, HeaderValue> derived) { return text(options, derived, StreamOptionKey.ENCRYPTION_KEY_FINGERPRINT); }
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeBody.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeBody.java new file mode 100644 index 0000000..144f356 --- /dev/null +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeBody.java
@@ -0,0 +1,475 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.system; + +import org.apache.iggy.message.HeaderValue; +import org.apache.iggy.stream.EncryptionKeySource; +import org.apache.iggy.stream.EncryptionProjection; +import org.apache.iggy.user.Permissions; +import org.apache.iggy.user.UserStatus; + +import java.math.BigInteger; +import java.util.Arrays; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; + +/** + * The body of a metadata change record or current state item. Streams are named by + * {@code (streamId, streamCreatedOp)}, topics by their stream key plus {@code (topicId, topicCreatedOp)} and users + * by {@code (userId, userCreatedAt)}, because ids are reused after a delete. + * + * <p>Bodies with options carry the public projection the read routes return, split into what a client set and + * what the server derived. When the server cannot read the stored state both maps are empty and + * {@code optionsUnreadable} is set, and a mirror keeps its previous values. + */ +public sealed interface MetadataChangeBody { + + /** Why a consumer group member was removed. The codes are the wire codes. */ + enum MemberLeftReason { + /** The member left the group. */ + LEFT(0, "left"), + /** The member's session ended. */ + SESSION_ENDED(1, "session_ended"); + + private final int code; + private final String value; + + MemberLeftReason(int code, String value) { + this.code = code; + this.value = value; + } + + /** The reason with wire code {@code code}, or empty for a reason this build does not know. */ + public static Optional<MemberLeftReason> fromCode(int code) { + for (MemberLeftReason reason : values()) { + if (reason.code == code) { + return Optional.of(reason); + } + } + return Optional.empty(); + } + + /** The reason named {@code value}, or empty for a reason this build does not know. */ + public static Optional<MemberLeftReason> fromValue(String value) { + for (MemberLeftReason reason : values()) { + if (reason.value.equals(value)) { + return Optional.of(reason); + } + } + return Optional.empty(); + } + + public int asCode() { + return code; + } + + public String value() { + return value; + } + } + + /** Why a personal access token was removed. The codes are the wire codes. */ + enum TokenDeletedReason { + /** A client deleted the token. */ + DELETED(0, "deleted"), + /** The expiry cleaner removed the token. */ + EXPIRED(1, "expired"); + + private final int code; + private final String value; + + TokenDeletedReason(int code, String value) { + this.code = code; + this.value = value; + } + + /** The reason with wire code {@code code}, or empty for a reason this build does not know. */ + public static Optional<TokenDeletedReason> fromCode(int code) { + for (TokenDeletedReason reason : values()) { + if (reason.code == code) { + return Optional.of(reason); + } + } + return Optional.empty(); + } + + /** The reason named {@code value}, or empty for a reason this build does not know. */ + public static Optional<TokenDeletedReason> fromValue(String value) { + for (TokenDeletedReason reason : values()) { + if (reason.value.equals(value)) { + return Optional.of(reason); + } + } + return Optional.empty(); + } + + public int asCode() { + return code; + } + + public String value() { + return value; + } + } + + /** Why a session was closed. The codes are the wire codes. */ + enum SessionClosedReason { + /** The client logged out. */ + LOGOUT(0, "logout"), + /** The session lease expired. */ + EXPIRED(1, "expired"), + /** The user was deactivated or deleted. */ + REVOKED(2, "revoked"); + + private final int code; + private final String value; + + SessionClosedReason(int code, String value) { + this.code = code; + this.value = value; + } + + /** The reason with wire code {@code code}, or empty for a reason this build does not know. */ + public static Optional<SessionClosedReason> fromCode(int code) { + for (SessionClosedReason reason : values()) { + if (reason.code == code) { + return Optional.of(reason); + } + } + return Optional.empty(); + } + + /** The reason named {@code value}, or empty for a reason this build does not know. */ + public static Optional<SessionClosedReason> fromValue(String value) { + for (SessionClosedReason reason : values()) { + if (reason.value.equals(value)) { + return Optional.of(reason); + } + } + return Optional.empty(); + } + + public int asCode() { + return code; + } + + public String value() { + return value; + } + } + + /** Why a user's public permission set changed. The codes are the wire codes. */ + enum PermissionsUpdatedReason { + /** A permissions update replaced the set. */ + UPDATED(0, "updated"), + /** A stream or topic delete pruned the user's grants on it. */ + RESOURCE_DELETED(1, "resource_deleted"); + + private final int code; + private final String value; + + PermissionsUpdatedReason(int code, String value) { + this.code = code; + this.value = value; + } + + /** The reason with wire code {@code code}, or empty for a reason this build does not know. */ + public static Optional<PermissionsUpdatedReason> fromCode(int code) { + for (PermissionsUpdatedReason reason : values()) { + if (reason.code == code) { + return Optional.of(reason); + } + } + return Optional.empty(); + } + + /** The reason named {@code value}, or empty for a reason this build does not know. */ + public static Optional<PermissionsUpdatedReason> fromValue(String value) { + for (PermissionsUpdatedReason reason : values()) { + if (reason.value.equals(value)) { + return Optional.of(reason); + } + } + return Optional.empty(); + } + + public int asCode() { + return code; + } + + public String value() { + return value; + } + } + + /** A stream after a create or update, and a streams item. */ + record StreamChange( + long streamId, + BigInteger streamCreatedOp, + BigInteger createdAt, + String name, + Map<String, HeaderValue> options, + Map<String, HeaderValue> derivedOptions, + boolean optionsUnreadable) + implements MetadataChangeBody { + + /** Who chose the key that encrypts the stream at rest, empty when the record reports no source. */ + public Optional<EncryptionKeySource> encryptionKeySource() { + return EncryptionProjection.source(options, derivedOptions); + } + + /** Fingerprint of the stream's key, empty for a plaintext stream. */ + public Optional<String> encryptionKeyFingerprint() { + return EncryptionProjection.keyFingerprint(options, derivedOptions); + } + } + + /** A removed stream. {@code requestedOp} is the op of a deferred delete's request. */ + record StreamDeleted( + long streamId, + BigInteger streamCreatedOp, + BigInteger createdAt, + String name, + Optional<BigInteger> requestedOp) + implements MetadataChangeBody {} + + /** A topic after a create or update. */ + record TopicChange( + long streamId, + BigInteger streamCreatedOp, + long topicId, + BigInteger topicCreatedOp, + BigInteger createdAt, + String name, + long partitionsCount, + Map<String, HeaderValue> options, + Map<String, HeaderValue> derivedOptions, + boolean optionsUnreadable) + implements MetadataChangeBody {} + + /** A topics item. Partition counts travel as partitions items. */ + record TopicItem( + long streamId, + BigInteger streamCreatedOp, + long topicId, + BigInteger topicCreatedOp, + BigInteger createdAt, + String name, + Map<String, HeaderValue> options, + Map<String, HeaderValue> derivedOptions, + boolean optionsUnreadable) + implements MetadataChangeBody {} + + /** A removed topic. */ + record TopicDeleted( + long streamId, + BigInteger streamCreatedOp, + long topicId, + BigInteger topicCreatedOp, + BigInteger createdAt, + String name, + Optional<BigInteger> requestedOp) + implements MetadataChangeBody {} + + /** + * A partition count change. {@code removedPartitionIds} lists the deleted ids in ascending order and is empty + * on create. Stored ids can be sparse, so the count is not an id boundary. + */ + record PartitionsChanged( + long streamId, + BigInteger streamCreatedOp, + long topicId, + BigInteger topicCreatedOp, + long partitionsCount, + List<Long> removedPartitionIds, + Optional<BigInteger> requestedOp) + implements MetadataChangeBody {} + + /** A partitions item. */ + record PartitionsItem( + long streamId, BigInteger streamCreatedOp, long topicId, BigInteger topicCreatedOp, long partitionsCount) + implements MetadataChangeBody {} + + /** A created consumer group with the topic's resulting group count. */ + record ConsumerGroupCreated( + long streamId, + BigInteger streamCreatedOp, + long topicId, + BigInteger topicCreatedOp, + long groupId, + String name, + long consumerGroupsCount) + implements MetadataChangeBody {} + + /** A removed consumer group with the topic's resulting group count. */ + record ConsumerGroupDeleted( + long streamId, + BigInteger streamCreatedOp, + long topicId, + BigInteger topicCreatedOp, + long groupId, + String name, + long consumerGroupsCount, + Optional<BigInteger> requestedOp) + implements MetadataChangeBody {} + + /** A consumer groups item. */ + record ConsumerGroupItem( + long streamId, + BigInteger streamCreatedOp, + long topicId, + BigInteger topicCreatedOp, + long groupId, + String name) + implements MetadataChangeBody {} + + /** A member that joined, with its installed partitions, normally none because a handover installs them. */ + record MemberJoined( + long streamId, + BigInteger streamCreatedOp, + long topicId, + BigInteger topicCreatedOp, + long groupId, + long memberId, + BigInteger clientId, + long membersCount, + List<Long> partitionIds) + implements MetadataChangeBody {} + + /** A member that left the group. */ + record MemberLeft( + long streamId, + BigInteger streamCreatedOp, + long topicId, + BigInteger topicCreatedOp, + long groupId, + long memberId, + BigInteger clientId, + long membersCount, + MemberLeftReason reason) + implements MetadataChangeBody {} + + /** A completed partition handover. {@code memberId} is empty when the last member left. */ + record PartitionHandedOver( + long streamId, + BigInteger streamCreatedOp, + long topicId, + BigInteger topicCreatedOp, + long groupId, + long partitionId, + Optional<Long> memberId) + implements MetadataChangeBody {} + + /** A consumer group members item. */ + record MemberItem( + long streamId, + BigInteger streamCreatedOp, + long topicId, + BigInteger topicCreatedOp, + long groupId, + long memberId, + BigInteger clientId, + List<Long> partitionIds) + implements MetadataChangeBody {} + + /** A user after a create or update, and a users item. */ + record UserChange( + long userId, + BigInteger userCreatedAt, + String username, + UserStatus status, + Optional<Permissions> permissions, + Map<String, HeaderValue> options, + Map<String, HeaderValue> derivedOptions, + boolean optionsUnreadable) + implements MetadataChangeBody {} + + /** A deleted user, or one whose password changed. */ + record UserIdentity(long userId, BigInteger userCreatedAt, String username) implements MetadataChangeBody {} + + /** A user's public permission set after a change. */ + record UserPermissionsUpdated( + long userId, + BigInteger userCreatedAt, + String username, + Optional<Permissions> permissions, + PermissionsUpdatedReason reason) + implements MetadataChangeBody {} + + /** A created personal access token, and a personal access tokens item. Never the token itself. */ + record TokenChange(long userId, BigInteger userCreatedAt, String name, Optional<BigInteger> expiry) + implements MetadataChangeBody {} + + /** A removed personal access token. */ + record TokenDeleted(long userId, BigInteger userCreatedAt, String name, TokenDeletedReason reason) + implements MetadataChangeBody {} + + /** A registered session, and a sessions item. */ + record Session(long userId, BigInteger userCreatedAt, BigInteger clientId, BigInteger session) + implements MetadataChangeBody {} + + /** A closed session. */ + record SessionClosed( + long userId, BigInteger userCreatedAt, BigInteger clientId, BigInteger session, SessionClosedReason reason) + implements MetadataChangeBody {} + + /** + * A binary record or item of a group or kind this build does not know, kept with its codes and raw body. + * + * @param group the group code + * @param kind the kind code, empty for an item + * @param body the body exactly as the server sent it + */ + record Unknown(int group, Optional<Integer> kind, byte[] body) implements MetadataChangeBody { + + public Unknown { + body = MetadataChangeEncoding.copy(body); + } + + @Override + public byte[] body() { + return MetadataChangeEncoding.copy(body); + } + + @Override + public boolean equals(Object other) { + return other instanceof Unknown unknown + && group == unknown.group + && Objects.equals(kind, unknown.kind) + && Arrays.equals(body, unknown.body); + } + + @Override + public int hashCode() { + return 31 * Objects.hash(group, kind) + Arrays.hashCode(body); + } + } + + /** + * A JSON record or item of a group or kind this build does not know, kept as the whole object. + * + * @param group the group name + * @param kind the kind name, empty for an item + * @param json the record or item object + */ + record UnknownJson(String group, Optional<String> kind, String json) implements MetadataChangeBody {} +}
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeCursor.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeCursor.java new file mode 100644 index 0000000..b60cc7c --- /dev/null +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeCursor.java
@@ -0,0 +1,108 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.system; + +import java.math.BigInteger; + +/** + * Position of the last metadata change record a caller consumed or scanned. Send the cursor an answer returns in + * {@link MetadataChanges#next()} back to read the records after it. + * + * <p>The text form is {@code mc1_} followed by unpadded Base64url of + * {@code [log_id:u128][op:u64][checksum:u128][ordinal:u32]}, little-endian. Op 0 with a zero checksum and ordinal + * names the beginning of the history {@code logId} identifies. + * + * @param logId prepare checksum of op 1, the root bootstrap of this history + * @param op the metadata operation of the record + * @param checksum prepare checksum of {@code op} + * @param ordinal position of the record within {@code op} + */ +public record MetadataChangeCursor(BigInteger logId, BigInteger op, BigInteger checksum, long ordinal) { + + /** Text prefix of a change cursor. */ + public static final String PREFIX = "mc1_"; + + /** Encoded size of a change cursor. */ + public static final int SIZE = 44; + + private static final int TEXT_LENGTH = 63; + + public MetadataChangeCursor { + MetadataChangeEncoding.requireUnsigned(logId, 128, "Cursor log id"); + MetadataChangeEncoding.requireUnsigned(op, 64, "Cursor op"); + MetadataChangeEncoding.requireUnsigned(checksum, 128, "Cursor checksum"); + MetadataChangeEncoding.requireUnsigned(BigInteger.valueOf(ordinal), 32, "Cursor ordinal"); + } + + /** + * Parses the {@code mc1_} text form, refusing a wrong prefix or length, padding, the standard Base64 alphabet, + * non-canonical encodings and a non-canonical cursor. + * + * @throws org.apache.iggy.exception.IggyValidationException invalid command for any refused text + */ + public static MetadataChangeCursor parse(String text) { + if (text == null || text.length() != TEXT_LENGTH || !text.startsWith(PREFIX)) { + throw MetadataChangeEncoding.refused("A change cursor is mc1_ followed by 59 Base64url characters"); + } + byte[] bytes = MetadataChangeEncoding.decodeCanonical(text.substring(PREFIX.length())) + .filter(decoded -> decoded.length == SIZE) + .orElseThrow(() -> MetadataChangeEncoding.refused("A change cursor is not canonical Base64url")); + var cursor = fromBytes(bytes, 0); + if (!cursor.isCanonical()) { + throw MetadataChangeEncoding.refused("A cursor at op 0 carries no checksum and no ordinal"); + } + return cursor; + } + + /** Reads the 44 bytes at {@code offset} without the canonical check. */ + public static MetadataChangeCursor fromBytes(byte[] bytes, int offset) { + return new MetadataChangeCursor( + MetadataChangeEncoding.readUnsigned(bytes, offset, 16), + MetadataChangeEncoding.readUnsigned(bytes, offset + 16, 8), + MetadataChangeEncoding.readUnsigned(bytes, offset + 24, 16), + MetadataChangeEncoding.readUnsignedInt(bytes, offset + 40)); + } + + /** Whether the cursor names the beginning of its history. */ + public boolean isZero() { + return op.signum() == 0; + } + + /** Op 0 carries no checksum and no ordinal. A cursor that does is refused. */ + public boolean isCanonical() { + return op.signum() != 0 || (checksum.signum() == 0 && ordinal == 0); + } + + /** The 44-byte wire form. */ + public byte[] toBytes() { + byte[] bytes = new byte[SIZE]; + MetadataChangeEncoding.writeUnsigned(bytes, 0, 16, logId); + MetadataChangeEncoding.writeUnsigned(bytes, 16, 8, op); + MetadataChangeEncoding.writeUnsigned(bytes, 24, 16, checksum); + MetadataChangeEncoding.writeUnsigned(bytes, 40, 4, BigInteger.valueOf(ordinal)); + return bytes; + } + + /** The {@code mc1_} text form. */ + @Override + public String toString() { + return PREFIX + MetadataChangeEncoding.encode(toBytes()); + } +}
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeEncoding.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeEncoding.java new file mode 100644 index 0000000..d830418 --- /dev/null +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeEncoding.java
@@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.system; + +import org.apache.iggy.exception.IggyErrorCode; +import org.apache.iggy.exception.IggyInvalidArgumentException; +import org.apache.iggy.exception.IggyValidationException; + +import java.math.BigInteger; +import java.util.Arrays; +import java.util.Base64; +import java.util.Optional; + +/** Little-endian unsigned values and canonical Base64url, the encodings of change cursors and page tokens. */ +final class MetadataChangeEncoding { + + private MetadataChangeEncoding() {} + + static BigInteger readUnsigned(byte[] bytes, int offset, int width) { + byte[] bigEndian = new byte[width]; + for (int index = 0; index < width; index++) { + bigEndian[width - 1 - index] = bytes[offset + index]; + } + return new BigInteger(1, bigEndian); + } + + static long readUnsignedInt(byte[] bytes, int offset) { + return readUnsigned(bytes, offset, Integer.BYTES).longValueExact(); + } + + static void writeUnsigned(byte[] target, int offset, int width, BigInteger value) { + for (int index = 0; index < width; index++) { + target[offset + index] = (byte) value.shiftRight(Byte.SIZE * index).intValue(); + } + } + + /** Checks that {@code value} fits an unsigned integer of {@code bits} bits. */ + static BigInteger requireUnsigned(BigInteger value, int bits, String field) { + if (value == null || value.signum() < 0 || value.bitLength() > bits) { + throw new IggyInvalidArgumentException(field + " must be an unsigned " + bits + "-bit value"); + } + return value; + } + + static String encode(byte[] bytes) { + return Base64.getUrlEncoder().withoutPadding().encodeToString(bytes); + } + + /** + * Decodes {@code text} only when it is the canonical encoding of its bytes: the Base64url alphabet, no + * padding and zero trailing bits, so one value has exactly one text form. + */ + static Optional<byte[]> decodeCanonical(String text) { + try { + byte[] bytes = Base64.getUrlDecoder().decode(text); + return encode(bytes).equals(text) ? Optional.of(bytes) : Optional.empty(); + } catch (IllegalArgumentException malformed) { + return Optional.empty(); + } + } + + /** The refusal the server returns for the same query, raised before anything is sent. */ + static IggyValidationException refused(String reason) { + return new IggyValidationException( + IggyErrorCode.INVALID_COMMAND, + IggyErrorCode.INVALID_COMMAND.getCode(), + reason, + Optional.empty(), + Optional.empty()); + } + + static byte[] copy(byte[] bytes) { + return Arrays.copyOf(bytes, bytes.length); + } +}
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeGroup.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeGroup.java new file mode 100644 index 0000000..6efa097 --- /dev/null +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeGroup.java
@@ -0,0 +1,123 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.system; + +import java.util.EnumSet; +import java.util.List; +import java.util.Optional; +import java.util.Set; + +/** Family of metadata change records and the unit of filtering. The codes are the wire codes. */ +public enum MetadataChangeGroup { + /** Stream created, updated and deleted. */ + STREAMS(1, "streams"), + /** Topic created, updated and deleted. */ + TOPICS(2, "topics"), + /** Partition count changes. */ + PARTITIONS(3, "partitions"), + /** Consumer group created and deleted. */ + CONSUMER_GROUPS(4, "consumer_groups"), + /** Consumer group members joining, leaving and receiving partitions. */ + CONSUMER_GROUP_MEMBERS(5, "consumer_group_members"), + /** User created, updated, deleted, password and permission changes. */ + USERS(6, "users"), + /** Personal access token created and deleted. */ + PERSONAL_ACCESS_TOKENS(7, "personal_access_tokens"), + /** Session registered and closed. */ + SESSIONS(8, "sessions"); + + private final int code; + private final String value; + + MetadataChangeGroup(int code, String value) { + this.code = code; + this.value = value; + } + + /** The group with wire code {@code code}, or empty for a group this build does not know. */ + public static Optional<MetadataChangeGroup> fromCode(int code) { + for (MetadataChangeGroup group : values()) { + if (group.code == code) { + return Optional.of(group); + } + } + return Optional.empty(); + } + + /** The group named {@code value}, or empty for a group this build does not know. */ + public static Optional<MetadataChangeGroup> fromValue(String value) { + for (MetadataChangeGroup group : values()) { + if (group.value.equals(value)) { + return Optional.of(group); + } + } + return Optional.empty(); + } + + /** The groups a page token's mask selects, in code order. */ + public static Set<MetadataChangeGroup> fromMask(int mask) { + Set<MetadataChangeGroup> groups = EnumSet.noneOf(MetadataChangeGroup.class); + for (MetadataChangeGroup group : values()) { + if ((mask & group.bit()) != 0) { + groups.add(group); + } + } + return groups; + } + + /** The mask of {@code groups}, every group when none is requested. */ + public static int mask(List<MetadataChangeGroup> groups) { + if (groups.isEmpty()) { + return 0xFF; + } + int mask = 0; + for (MetadataChangeGroup group : groups) { + mask |= group.bit(); + } + return mask; + } + + public int asCode() { + return code; + } + + public String value() { + return value; + } + + /** Bit {@code code - 1} of a group mask. */ + public int bit() { + return 1 << (code - 1); + } + + /** + * Groups a current state scan of this group must include, so a mirror can apply the cascades of their + * deletes. + */ + public List<MetadataChangeGroup> parents() { + return switch (this) { + case TOPICS -> List.of(STREAMS); + case PARTITIONS, CONSUMER_GROUPS -> List.of(TOPICS, STREAMS); + case CONSUMER_GROUP_MEMBERS -> List.of(CONSUMER_GROUPS, PARTITIONS, TOPICS, STREAMS); + case PERSONAL_ACCESS_TOKENS, SESSIONS -> List.of(USERS); + default -> List.of(); + }; + } +}
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeKind.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeKind.java new file mode 100644 index 0000000..5b12ddb --- /dev/null +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeKind.java
@@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.system; + +import java.util.Optional; + +/** Kind of a metadata change record. The codes are the wire codes, and a kind always belongs to one group. */ +public enum MetadataChangeKind { + STREAM_CREATED(1, "stream_created", MetadataChangeGroup.STREAMS), + STREAM_UPDATED(2, "stream_updated", MetadataChangeGroup.STREAMS), + STREAM_DELETED(3, "stream_deleted", MetadataChangeGroup.STREAMS), + TOPIC_CREATED(10, "topic_created", MetadataChangeGroup.TOPICS), + TOPIC_UPDATED(11, "topic_updated", MetadataChangeGroup.TOPICS), + TOPIC_DELETED(12, "topic_deleted", MetadataChangeGroup.TOPICS), + PARTITIONS_CHANGED(20, "partitions_changed", MetadataChangeGroup.PARTITIONS), + CONSUMER_GROUP_CREATED(30, "consumer_group_created", MetadataChangeGroup.CONSUMER_GROUPS), + CONSUMER_GROUP_DELETED(31, "consumer_group_deleted", MetadataChangeGroup.CONSUMER_GROUPS), + CONSUMER_GROUP_MEMBER_JOINED(40, "consumer_group_member_joined", MetadataChangeGroup.CONSUMER_GROUP_MEMBERS), + CONSUMER_GROUP_MEMBER_LEFT(41, "consumer_group_member_left", MetadataChangeGroup.CONSUMER_GROUP_MEMBERS), + CONSUMER_GROUP_PARTITION_HANDED_OVER( + 42, "consumer_group_partition_handed_over", MetadataChangeGroup.CONSUMER_GROUP_MEMBERS), + USER_CREATED(50, "user_created", MetadataChangeGroup.USERS), + USER_UPDATED(51, "user_updated", MetadataChangeGroup.USERS), + USER_DELETED(52, "user_deleted", MetadataChangeGroup.USERS), + USER_PASSWORD_CHANGED(53, "user_password_changed", MetadataChangeGroup.USERS), + USER_PERMISSIONS_UPDATED(54, "user_permissions_updated", MetadataChangeGroup.USERS), + PERSONAL_ACCESS_TOKEN_CREATED(60, "personal_access_token_created", MetadataChangeGroup.PERSONAL_ACCESS_TOKENS), + PERSONAL_ACCESS_TOKEN_DELETED(61, "personal_access_token_deleted", MetadataChangeGroup.PERSONAL_ACCESS_TOKENS), + SESSION_REGISTERED(70, "session_registered", MetadataChangeGroup.SESSIONS), + SESSION_CLOSED(71, "session_closed", MetadataChangeGroup.SESSIONS); + + private final int code; + private final String value; + private final MetadataChangeGroup group; + + MetadataChangeKind(int code, String value, MetadataChangeGroup group) { + this.code = code; + this.value = value; + this.group = group; + } + + /** The kind with wire code {@code code}, or empty for a kind this build does not know. */ + public static Optional<MetadataChangeKind> fromCode(int code) { + for (MetadataChangeKind kind : values()) { + if (kind.code == code) { + return Optional.of(kind); + } + } + return Optional.empty(); + } + + /** The kind named {@code value}, or empty for a kind this build does not know. */ + public static Optional<MetadataChangeKind> fromValue(String value) { + for (MetadataChangeKind kind : values()) { + if (kind.value.equals(value)) { + return Optional.of(kind); + } + } + return Optional.empty(); + } + + public int asCode() { + return code; + } + + public String value() { + return value; + } + + /** The group every record of this kind belongs to. */ + public MetadataChangeGroup group() { + return group; + } +}
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangePhase.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangePhase.java new file mode 100644 index 0000000..c44a2ad --- /dev/null +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangePhase.java
@@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.system; + +import java.util.Optional; + +/** Whether a metadata changes answer pages change records or current state items. */ +public enum MetadataChangePhase { + /** Change records, the answer to earliest, latest and after. */ + CHANGES(0, "changes"), + /** Current state items, the answer to current. */ + CURRENT(1, "current"); + + private final int code; + private final String value; + + MetadataChangePhase(int code, String value) { + this.code = code; + this.value = value; + } + + /** The phase with wire code {@code code}, or empty for a phase this build does not know. */ + public static Optional<MetadataChangePhase> fromCode(int code) { + for (MetadataChangePhase phase : values()) { + if (phase.code == code) { + return Optional.of(phase); + } + } + return Optional.empty(); + } + + /** The phase named {@code value}, or empty for a phase this build does not know. */ + public static Optional<MetadataChangePhase> fromValue(String value) { + for (MetadataChangePhase phase : values()) { + if (phase.value.equals(value)) { + return Optional.of(phase); + } + } + return Optional.empty(); + } + + public int asCode() { + return code; + } + + public String value() { + return value; + } +}
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeRecord.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeRecord.java new file mode 100644 index 0000000..2270f81 --- /dev/null +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeRecord.java
@@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.system; + +import java.math.BigInteger; +import java.util.Optional; + +/** + * One change record: the committed operation that produced it, its position within that operation, and what is + * true after it. + * + * @param op the metadata operation that produced the record + * @param ordinal position of the record within {@code op} + * @param checksum prepare checksum of {@code op} + * @param timestamp prepare timestamp of {@code op}, in microseconds since the Unix epoch + * @param actorUserId the acting user, empty for server actions + * @param kind the kind, empty for a record this build does not know + * @param body the typed body, or {@link MetadataChangeBody.Unknown} and {@link MetadataChangeBody.UnknownJson} for + * a record this build does not know. A mirror that cannot apply a record stops without advancing its cursor. + */ +public record MetadataChangeRecord( + BigInteger op, + long ordinal, + BigInteger checksum, + BigInteger timestamp, + Optional<Long> actorUserId, + Optional<MetadataChangeKind> kind, + MetadataChangeBody body) { + + /** The group, empty for a record this build does not know. */ + public Optional<MetadataChangeGroup> group() { + return kind.map(MetadataChangeKind::group); + } +}
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeStart.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeStart.java new file mode 100644 index 0000000..dc2e8dd --- /dev/null +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeStart.java
@@ -0,0 +1,95 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.system; + +import java.util.Objects; +import java.util.Optional; + +/** Where a metadata changes query starts. */ +public sealed interface MetadataChangeStart { + + /** The oldest operation the requested groups still hold completely. */ + static MetadataChangeStart earliest() { + return new Earliest(); + } + + /** No records, only a cursor at the newest retained record of the requested groups. */ + static MetadataChangeStart latest() { + return new Latest(); + } + + /** Records after {@code cursor}. */ + static MetadataChangeStart after(MetadataChangeCursor cursor) { + return new After(cursor); + } + + /** The first page of a current state scan. */ + static MetadataChangeStart current() { + return new Current(Optional.empty()); + } + + /** The page of a current state scan that {@code pageToken} continues. */ + static MetadataChangeStart current(MetadataCurrentPageToken pageToken) { + return new Current(Optional.of(pageToken)); + } + + /** The start mode code on the wire. */ + int asCode(); + + /** The oldest operation the requested groups still hold completely. */ + record Earliest() implements MetadataChangeStart { + @Override + public int asCode() { + return 0; + } + } + + /** No records, only a cursor at the newest retained record. */ + record Latest() implements MetadataChangeStart { + @Override + public int asCode() { + return 1; + } + } + + /** Records after a change cursor. */ + record After(MetadataChangeCursor cursor) implements MetadataChangeStart { + public After { + Objects.requireNonNull(cursor, "cursor"); + } + + @Override + public int asCode() { + return 2; + } + } + + /** Current state pages, the first one without a page token. */ + record Current(Optional<MetadataCurrentPageToken> pageToken) implements MetadataChangeStart { + public Current { + Objects.requireNonNull(pageToken, "pageToken"); + } + + @Override + public int asCode() { + return 3; + } + } +}
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeStatus.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeStatus.java new file mode 100644 index 0000000..3c67e15 --- /dev/null +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangeStatus.java
@@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.system; + +import java.util.Optional; + +/** Outcome of a metadata changes query. */ +public enum MetadataChangeStatus { + /** Records or items after the requested position. */ + OK(0, "ok"), + /** The answering node has not applied the requested position yet. Ask again later or ask another node. */ + BEHIND(1, "behind"), + /** The node no longer holds every record after the position. Rebuild from current state. */ + EXPIRED(2, "expired"), + /** The position belongs to another history. Rebuild from current state. */ + RESET(3, "reset"); + + private final int code; + private final String value; + + MetadataChangeStatus(int code, String value) { + this.code = code; + this.value = value; + } + + /** The status with wire code {@code code}, or empty for a status this build does not know. */ + public static Optional<MetadataChangeStatus> fromCode(int code) { + for (MetadataChangeStatus status : values()) { + if (status.code == code) { + return Optional.of(status); + } + } + return Optional.empty(); + } + + /** The status named {@code value}, or empty for a status this build does not know. */ + public static Optional<MetadataChangeStatus> fromValue(String value) { + for (MetadataChangeStatus status : values()) { + if (status.value.equals(value)) { + return Optional.of(status); + } + } + return Optional.empty(); + } + + public int asCode() { + return code; + } + + public String value() { + return value; + } +}
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChanges.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChanges.java new file mode 100644 index 0000000..73debff --- /dev/null +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChanges.java
@@ -0,0 +1,60 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.system; + +import java.math.BigInteger; +import java.util.List; +import java.util.Optional; + +/** + * One page of the metadata change log, or of the current state. + * + * @param status outcome of the query, only {@code OK} carries records or items + * @param phase whether the page holds change records or current state items + * @param logId the answering node's log id, empty before it applied op 1 + * @param oldestOp lowest whole operation from which the requested groups are complete + * @param retainedSince the requested groups are complete for every timestamp above this value, 0 when nothing was + * evicted + * @param headOp highest metadata operation the answering node applied + * @param next the cursor to send back: present on ok, echoed on behind, empty on expired and reset, and the anchor + * to follow from on the last page of a current state scan + * @param hasMore more entries remain after {@code next}, or another state page exists + * @param records change records in {@code (op, ordinal)} order, empty on a current state page + * @param items current state items, empty on a change page + * @param currentPageToken continuation of a current state scan, empty once it is complete + */ +public record MetadataChanges( + MetadataChangeStatus status, + MetadataChangePhase phase, + Optional<BigInteger> logId, + BigInteger oldestOp, + BigInteger retainedSince, + BigInteger headOp, + Optional<MetadataChangeCursor> next, + boolean hasMore, + List<MetadataChangeRecord> records, + List<MetadataCurrentItem> items, + Optional<MetadataCurrentPageToken> currentPageToken) { + + public MetadataChanges { + records = List.copyOf(records); + items = List.copyOf(items); + } +}
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangesQuery.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangesQuery.java new file mode 100644 index 0000000..19cf1b5 --- /dev/null +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataChangesQuery.java
@@ -0,0 +1,194 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.system; + +import java.math.BigInteger; +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; +import java.util.Optional; +import java.util.stream.Collectors; + +/** + * A metadata changes query. {@code groups} empty means every group. {@code since} and {@code until} keep records + * whose prepare timestamp, in microseconds since the Unix epoch, is in the inclusive range, and apply only to + * {@link MetadataChangeStart#earliest()} and {@link MetadataChangeStart#after(MetadataChangeCursor)}. + * + * @param start where the query starts + * @param groups the groups to read, duplicates ignored, empty for every group + * @param since lowest prepare timestamp to keep + * @param until highest prepare timestamp to keep + * @param limit records or items per page, 1 to 1000 + */ +public record MetadataChangesQuery( + MetadataChangeStart start, + List<MetadataChangeGroup> groups, + Optional<BigInteger> since, + Optional<BigInteger> until, + long limit) { + + /** Records or items per page when the caller does not say. */ + public static final long DEFAULT_LIMIT = 100; + + /** Most records or items one page returns. */ + public static final long MAX_LIMIT = 1000; + + public MetadataChangesQuery { + Objects.requireNonNull(start, "start"); + groups = List.copyOf(groups); + Objects.requireNonNull(since, "since"); + Objects.requireNonNull(until, "until"); + } + + public static Builder builder() { + return new Builder(); + } + + /** The same query starting elsewhere, how a follow loop moves to the cursor an answer returned. */ + public MetadataChangesQuery withStart(MetadataChangeStart newStart) { + return new MetadataChangesQuery(newStart, groups, since, until, limit); + } + + /** + * Applies the rules every transport and the server share, so a refused query is never sent. + * + * @throws org.apache.iggy.exception.IggyValidationException invalid command naming the broken rule + */ + public void validate() { + if (limit < 1 || limit > MAX_LIMIT) { + throw MetadataChangeEncoding.refused("The limit must be 1 to " + MAX_LIMIT + ", got " + limit); + } + validateBounds(); + validateStart(); + } + + /** The query string of {@code GET /metadata/changes}, in the order the server documents. */ + public String httpQuery() { + List<String> pairs = new ArrayList<>(6); + if (start instanceof MetadataChangeStart.After after) { + pairs.add("after=" + after.cursor()); + } else if (start instanceof MetadataChangeStart.Current current) { + pairs.add("from=current"); + current.pageToken().ifPresent(token -> pairs.add("current_after=" + token)); + } else { + pairs.add(start instanceof MetadataChangeStart.Latest ? "from=latest" : "from=earliest"); + } + if (!groups.isEmpty()) { + pairs.add("groups=" + + MetadataChangeGroup.fromMask(MetadataChangeGroup.mask(groups)).stream() + .map(MetadataChangeGroup::value) + .collect(Collectors.joining(","))); + } + since.ifPresent(value -> pairs.add("since=" + value)); + until.ifPresent(value -> pairs.add("until=" + value)); + pairs.add("limit=" + limit); + return String.join("&", pairs); + } + + private void validateBounds() { + if (since.filter(MetadataChangesQuery::outsideU64).isPresent() + || until.filter(MetadataChangesQuery::outsideU64).isPresent()) { + throw MetadataChangeEncoding.refused("Time bounds are unsigned 64-bit microseconds"); + } + if (since.isPresent() && until.isPresent() && since.get().compareTo(until.get()) > 0) { + throw MetadataChangeEncoding.refused("Since must not be above until"); + } + } + + private void validateStart() { + boolean hasTimeBound = since.isPresent() || until.isPresent(); + if (start instanceof MetadataChangeStart.After after && !after.cursor().isCanonical()) { + throw MetadataChangeEncoding.refused("A cursor at op 0 carries no checksum and no ordinal"); + } + boolean timeBoundAllowed = + start instanceof MetadataChangeStart.Earliest || start instanceof MetadataChangeStart.After; + if (hasTimeBound && !timeBoundAllowed) { + throw MetadataChangeEncoding.refused( + "Time bounds apply only when starting at the earliest record or after a cursor"); + } + if (start instanceof MetadataChangeStart.Current current) { + validateCurrent(current); + } + } + + private static boolean outsideU64(BigInteger value) { + return value.signum() < 0 || value.bitLength() > Long.SIZE; + } + + private void validateCurrent(MetadataChangeStart.Current current) { + int mask = MetadataChangeGroup.mask(groups); + for (MetadataChangeGroup group : MetadataChangeGroup.fromMask(mask)) { + for (MetadataChangeGroup parent : group.parents()) { + if ((mask & parent.bit()) == 0) { + throw MetadataChangeEncoding.refused("A current state scan of " + group.value() + + " must include its parent group " + parent.value()); + } + } + } + if (current.pageToken().filter(token -> token.groupsMask() != mask).isPresent()) { + throw MetadataChangeEncoding.refused("The page token was issued for other groups"); + } + } + + /** Builds a query from the defaults: earliest, every group, no time bounds and a limit of 100. */ + public static final class Builder { + private MetadataChangeStart start = MetadataChangeStart.earliest(); + private List<MetadataChangeGroup> groups = List.of(); + private Optional<BigInteger> since = Optional.empty(); + private Optional<BigInteger> until = Optional.empty(); + private long limit = DEFAULT_LIMIT; + + private Builder() {} + + public Builder start(MetadataChangeStart value) { + start = value; + return this; + } + + public Builder groups(List<MetadataChangeGroup> value) { + groups = value; + return this; + } + + public Builder groups(MetadataChangeGroup... value) { + groups = List.of(value); + return this; + } + + public Builder since(BigInteger value) { + since = Optional.of(value); + return this; + } + + public Builder until(BigInteger value) { + until = Optional.of(value); + return this; + } + + public Builder limit(long value) { + limit = value; + return this; + } + + public MetadataChangesQuery build() { + return new MetadataChangesQuery(start, groups, since, until, limit); + } + } +}
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataCurrentItem.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataCurrentItem.java new file mode 100644 index 0000000..b3fa4f8 --- /dev/null +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataCurrentItem.java
@@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.system; + +import java.util.Optional; + +/** + * One current state item. + * + * @param group the group, empty for an item this build does not know + * @param body the typed body, or {@link MetadataChangeBody.Unknown} and {@link MetadataChangeBody.UnknownJson} for + * an item this build does not know + */ +public record MetadataCurrentItem(Optional<MetadataChangeGroup> group, MetadataChangeBody body) {}
diff --git a/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataCurrentPageToken.java b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataCurrentPageToken.java new file mode 100644 index 0000000..272c3b0 --- /dev/null +++ b/foreign/java/java-sdk/src/main/java/org/apache/iggy/system/MetadataCurrentPageToken.java
@@ -0,0 +1,169 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.system; + +import java.math.BigInteger; +import java.nio.ByteBuffer; +import java.nio.charset.CharacterCodingException; +import java.nio.charset.CodingErrorAction; +import java.nio.charset.StandardCharsets; +import java.util.Arrays; +import java.util.Optional; +import java.util.Set; + +/** + * Opaque continuation of a current state scan. Send it back unchanged with the same groups through + * {@link MetadataChangeStart#current(MetadataCurrentPageToken)}. + * + * <p>The text form is {@code cs1_} followed by unpadded Base64url of + * {@code [anchor:44][required_head:u64][groups_mask:u8][group:u8][key_len:u16][last_key]}, little-endian. + */ +public final class MetadataCurrentPageToken { + + /** Text prefix of a current state page token. */ + public static final String PREFIX = "cs1_"; + + /** Longest key: a user id and a personal access token name. */ + public static final int MAX_KEY_SIZE = 4 + 1 + 255; + + /** Longest decoded token payload. */ + public static final int MAX_SIZE = MetadataChangeCursor.SIZE + 8 + 1 + 1 + 2 + MAX_KEY_SIZE; + + private static final int FIXED_SIZE = MetadataChangeCursor.SIZE + 8 + 1 + 1 + 2; + private static final int MAX_TEXT_LENGTH = PREFIX.length() + (MAX_SIZE + 2) / 3 * 4; + private static final int MASK_OFFSET = MetadataChangeCursor.SIZE + 8; + + private final byte[] payload; + + private MetadataCurrentPageToken(byte[] payload) { + this.payload = payload; + } + + /** + * Parses the {@code cs1_} text form, refusing a wrong prefix, padding, non-canonical encodings and a payload + * that breaks the token's own rules. + * + * @throws org.apache.iggy.exception.IggyValidationException invalid command for any refused text + */ + public static MetadataCurrentPageToken parse(String text) { + if (text == null || text.length() > MAX_TEXT_LENGTH || !text.startsWith(PREFIX)) { + throw MetadataChangeEncoding.refused("A page token is cs1_ followed by Base64url"); + } + return MetadataChangeEncoding.decodeCanonical(text.substring(PREFIX.length())) + .flatMap(MetadataCurrentPageToken::fromPayload) + .orElseThrow(() -> MetadataChangeEncoding.refused("A page token is not a canonical page token")); + } + + /** + * Decodes a token payload, requiring it to be exactly one token: a canonical anchor, a known group selected + * by the mask, a head at or above the anchor and a key of the exact width that group uses. + */ + public static Optional<MetadataCurrentPageToken> fromPayload(byte[] payload) { + if (payload.length < FIXED_SIZE) { + return Optional.empty(); + } + int keyLength = (payload[FIXED_SIZE - 2] & 0xFF) | ((payload[FIXED_SIZE - 1] & 0xFF) << Byte.SIZE); + if (keyLength > MAX_KEY_SIZE || payload.length != FIXED_SIZE + keyLength) { + return Optional.empty(); + } + var token = new MetadataCurrentPageToken(MetadataChangeEncoding.copy(payload)); + Optional<MetadataChangeGroup> group = MetadataChangeGroup.fromCode(payload[MASK_OFFSET + 1] & 0xFF); + boolean valid = token.anchor().isCanonical() + && token.requiredHead().compareTo(token.anchor().op()) >= 0 + && group.filter(known -> (token.groupsMask() & known.bit()) != 0) + .filter(known -> keyFitsGroup(known, token.lastKey())) + .isPresent(); + return valid ? Optional.of(token) : Optional.empty(); + } + + /** The anchor captured before the scan's first page. */ + public MetadataChangeCursor anchor() { + return MetadataChangeCursor.fromBytes(payload, 0); + } + + /** The highest head any page of the scan reported so far. */ + public BigInteger requiredHead() { + return MetadataChangeEncoding.readUnsigned(payload, MetadataChangeCursor.SIZE, 8); + } + + /** The groups the scan was started with, as their mask. */ + public int groupsMask() { + return payload[MASK_OFFSET] & 0xFF; + } + + /** The groups the scan was started with. */ + public Set<MetadataChangeGroup> groups() { + return MetadataChangeGroup.fromMask(groupsMask()); + } + + /** The group of the last item returned. */ + public MetadataChangeGroup lastGroup() { + return MetadataChangeGroup.fromCode(payload[MASK_OFFSET + 1] & 0xFF).orElseThrow(); + } + + /** The key of the last item returned, in the page token key layout. */ + public byte[] lastKey() { + return Arrays.copyOfRange(payload, FIXED_SIZE, payload.length); + } + + /** The decoded payload binary requests and responses carry. */ + public byte[] payload() { + return MetadataChangeEncoding.copy(payload); + } + + /** The {@code cs1_} text form. */ + @Override + public String toString() { + return PREFIX + MetadataChangeEncoding.encode(payload); + } + + @Override + public boolean equals(Object other) { + return other instanceof MetadataCurrentPageToken token && Arrays.equals(payload, token.payload); + } + + @Override + public int hashCode() { + return Arrays.hashCode(payload); + } + + private static boolean keyFitsGroup(MetadataChangeGroup group, byte[] key) { + return switch (group) { + case STREAMS, USERS -> key.length == 4; + case TOPICS, PARTITIONS, SESSIONS -> key.length == 8; + case CONSUMER_GROUPS -> key.length == 12; + case CONSUMER_GROUP_MEMBERS -> key.length == 16; + default -> key.length > 5 && key.length == 5 + (key[4] & 0xFF) && isUtf8(key, 5); + }; + } + + private static boolean isUtf8(byte[] bytes, int offset) { + try { + StandardCharsets.UTF_8 + .newDecoder() + .onMalformedInput(CodingErrorAction.REPORT) + .onUnmappableCharacter(CodingErrorAction.REPORT) + .decode(ByteBuffer.wrap(bytes, offset, bytes.length - offset)); + return true; + } catch (CharacterCodingException malformed) { + return false; + } + } +}
diff --git a/foreign/java/java-sdk/src/test/java/org/apache/iggy/MetadataChangesFixture.java b/foreign/java/java-sdk/src/test/java/org/apache/iggy/MetadataChangesFixture.java new file mode 100644 index 0000000..8aabc8d --- /dev/null +++ b/foreign/java/java-sdk/src/test/java/org/apache/iggy/MetadataChangesFixture.java
@@ -0,0 +1,322 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy; + +import com.fasterxml.jackson.annotation.JsonSetter; +import com.fasterxml.jackson.annotation.Nulls; +import io.netty.buffer.ByteBuf; +import io.netty.buffer.Unpooled; +import org.apache.iggy.system.MetadataChangeBody; +import org.apache.iggy.system.MetadataChangeCursor; +import org.apache.iggy.system.MetadataChangeGroup; +import org.apache.iggy.system.MetadataChangeRecord; +import org.apache.iggy.system.MetadataChangeStart; +import org.apache.iggy.system.MetadataChanges; +import org.apache.iggy.system.MetadataChangesQuery; +import org.apache.iggy.system.MetadataCurrentItem; +import org.apache.iggy.system.MetadataCurrentPageToken; +import org.apache.iggy.user.Permissions; +import tools.jackson.databind.JsonNode; +import tools.jackson.databind.PropertyNamingStrategies; +import tools.jackson.databind.json.JsonMapper; + +import java.io.IOException; +import java.io.InputStream; +import java.io.UncheckedIOException; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.RecordComponent; +import java.math.BigInteger; +import java.util.ArrayList; +import java.util.Base64; +import java.util.HexFormat; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Optional; +import java.util.Set; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * The shared metadata changes fixture the Rust encoders generate, read from the test classpath, and the checks that + * a decoded answer holds every value of the JSON the server renders for it. + * + * <p>Body fields are matched by name: a JSON property {@code stream_created_op} is the record component + * {@code streamCreatedOp}, so every field of every body kind is compared without a hand-written list per kind. + */ +public final class MetadataChangesFixture { + + private static final JsonNode ROOT = load(); + private static final JsonMapper MAPPER = JsonMapper.builder() + .propertyNamingStrategy(PropertyNamingStrategies.SNAKE_CASE) + .withConfigOverride(Map.class, map -> map.setNullHandling(JsonSetter.Value.forValueNulls(Nulls.AS_EMPTY))) + .build(); + private static final Set<String> RECORD_ENVELOPE = + Set.of("op", "ordinal", "checksum", "timestamp", "actor_user_id", "group", "kind"); + private static final Set<String> ITEM_ENVELOPE = Set.of("group"); + private static final Set<String> OPTION_COMPONENTS = Set.of("options", "derivedOptions", "optionsUnreadable"); + + private MetadataChangesFixture() {} + + public static JsonNode section(String name) { + return ROOT.get(name); + } + + public static JsonNode refused(String name) { + return ROOT.get("refused").get(name); + } + + public static JsonNode find(JsonNode section, String vectorCase) { + for (JsonNode vector : section) { + if (vector.get("case").asString().equals(vectorCase)) { + return vector; + } + } + throw new IllegalArgumentException("no vector " + vectorCase); + } + + public static List<String> cases(JsonNode section) { + List<String> cases = new ArrayList<>(); + section.forEach(vector -> cases.add(vector.get("case").asString())); + return cases; + } + + public static byte[] bytes(String hex) { + return HexFormat.of().parseHex(hex); + } + + public static ByteBuf buffer(String hex) { + return Unpooled.wrappedBuffer(bytes(hex)); + } + + public static JsonNode parse(String json) { + return MAPPER.readTree(json); + } + + /** The query a fixture vector's {@code input} describes. */ + public static MetadataChangesQuery query(JsonNode input) { + var builder = MetadataChangesQuery.builder() + .start(start(input)) + .limit(input.get("limit").asLong()); + List<MetadataChangeGroup> groups = new ArrayList<>(); + input.get("groups") + .forEach(group -> groups.add( + MetadataChangeGroup.fromValue(group.asString()).orElseThrow())); + builder.groups(groups); + if (!input.get("since").isNull()) { + builder.since(input.get("since").bigIntegerValue()); + } + if (!input.get("until").isNull()) { + builder.until(input.get("until").bigIntegerValue()); + } + return builder.build(); + } + + /** Asserts that {@code actual} holds every value of the answer the server renders as {@code expected}. */ + public static void assertAnswer(JsonNode expected, MetadataChanges actual) { + assertThat(actual.status().value()).isEqualTo(expected.get("status").asString()); + assertThat(actual.phase().value()).isEqualTo(expected.get("phase").asString()); + assertValue("log_id", expected.get("log_id"), actual.logId()); + assertValue("oldest_op", expected.get("oldest_op"), actual.oldestOp()); + assertValue("retained_since", expected.get("retained_since"), actual.retainedSince()); + assertValue("head_op", expected.get("head_op"), actual.headOp()); + assertThat(actual.next().map(MetadataChangeCursor::toString)).isEqualTo(text(expected.get("next"))); + assertThat(actual.hasMore()).isEqualTo(expected.get("has_more").asBoolean()); + assertThat(actual.currentPageToken().map(MetadataCurrentPageToken::toString)) + .isEqualTo(text(expected.get("current_page_token"))); + assertThat(actual.records()).hasSize(expected.get("records").size()); + assertThat(actual.items()).hasSize(expected.get("items").size()); + for (int index = 0; index < actual.records().size(); index++) { + assertRecord(expected.get("records").get(index), actual.records().get(index)); + } + for (int index = 0; index < actual.items().size(); index++) { + assertItem(expected.get("items").get(index), actual.items().get(index)); + } + } + + /** Asserts that {@code actual} holds every value of the record the server renders as {@code expected}. */ + public static void assertRecord(JsonNode expected, MetadataChangeRecord actual) { + if (actual.body() instanceof MetadataChangeBody.UnknownJson unknown) { + assertThat(parse(unknown.json())).isEqualTo(expected); + assertThat(actual.kind()).isEmpty(); + return; + } + assertValue("op", expected.get("op"), actual.op()); + assertValue("ordinal", expected.get("ordinal"), actual.ordinal()); + assertValue("checksum", expected.get("checksum"), actual.checksum()); + assertValue("timestamp", expected.get("timestamp"), actual.timestamp()); + assertThat(actual.actorUserId().map(String::valueOf)).isEqualTo(number(expected.get("actor_user_id"))); + if (actual.body() instanceof MetadataChangeBody.Unknown unknown) { + assertThat(String.valueOf(unknown.group())) + .isEqualTo(expected.get("group").asString()); + assertThat(unknown.kind().map(String::valueOf)) + .contains(expected.get("kind").asString()); + assertThat(Base64.getEncoder().encodeToString(unknown.body())) + .isEqualTo(expected.get("body").asString()); + return; + } + assertThat(actual.group().orElseThrow().value()) + .isEqualTo(expected.get("group").asString()); + assertThat(actual.kind().orElseThrow().value()) + .isEqualTo(expected.get("kind").asString()); + assertBody(expected, actual.body(), RECORD_ENVELOPE); + } + + /** Asserts that {@code actual} holds every value of the item the server renders as {@code expected}. */ + public static void assertItem(JsonNode expected, MetadataCurrentItem actual) { + if (actual.body() instanceof MetadataChangeBody.UnknownJson unknown) { + assertThat(parse(unknown.json())).isEqualTo(expected); + return; + } + assertThat(actual.group().orElseThrow().value()) + .isEqualTo(expected.get("group").asString()); + assertBody(expected, actual.body(), ITEM_ENVELOPE); + } + + private static void assertBody(JsonNode expected, MetadataChangeBody body, Set<String> envelope) { + for (Map.Entry<String, JsonNode> property : expected.properties()) { + String name = property.getKey(); + if (envelope.contains(name) || name.startsWith("options")) { + continue; + } + if ("permissions".equals(name)) { + assertThat(component(body, "permissions")).as(name).isEqualTo(permissions(property.getValue())); + } else { + assertValue(name, property.getValue(), component(body, camelCase(name))); + } + } + for (RecordComponent component : body.getClass().getRecordComponents()) { + String name = snakeCase(component.getName()); + if (!expected.has(name) && !OPTION_COMPONENTS.contains(component.getName())) { + assertThat(component(body, component.getName())).as(name).isEqualTo(Optional.empty()); + } + } + if (expected.has("options") || expected.has("options_unreadable")) { + assertOptions(expected, body); + } + } + + private static void assertOptions(JsonNode expected, MetadataChangeBody body) { + boolean unreadable = expected.has("options_unreadable") + && expected.get("options_unreadable").asBoolean(); + assertThat(component(body, "optionsUnreadable")).isEqualTo(unreadable); + JsonNode options = unreadable ? MAPPER.createObjectNode() : expected.get("options"); + assertThat(component(body, "options")).isEqualTo(ResourceOptionsFixture.optionMap(options, true)); + assertThat(component(body, "derivedOptions")).isEqualTo(ResourceOptionsFixture.optionMap(options, false)); + } + + private static void assertValue(String field, JsonNode expected, Object actual) { + Object value = actual instanceof Optional<?> optional ? optional.orElse(null) : actual; + if (expected.isNull()) { + assertThat(value).as(field).isNull(); + } else if (expected.isArray()) { + List<String> elements = new ArrayList<>(); + expected.forEach(element -> elements.add(element.bigIntegerValue().toString())); + assertThat(((List<?>) value).stream().map(String::valueOf).toList()) + .as(field) + .isEqualTo(elements); + } else if (expected.isNumber()) { + assertThat(String.valueOf(value)) + .as(field) + .isEqualTo(expected.bigIntegerValue().toString()); + } else { + assertThat(text(value)) + .as(field) + .isEqualTo(expected.isBoolean() ? expected.asString() : expected.stringValue()); + } + } + + /** A decoded value in the text the JSON renders it as: 128-bit values in hex, enums by lowercase name. */ + private static String text(Object value) { + if (value instanceof BigInteger wide) { + return String.format("%032x", wide); + } + if (value instanceof Enum<?> constant) { + return constant.name().toLowerCase(Locale.ROOT); + } + return String.valueOf(value); + } + + private static Optional<String> text(JsonNode node) { + return node.isNull() ? Optional.empty() : Optional.of(node.asString()); + } + + private static Optional<String> number(JsonNode node) { + return node == null || node.isNull() + ? Optional.empty() + : Optional.of(node.bigIntegerValue().toString()); + } + + private static Optional<Permissions> permissions(JsonNode node) { + return node.isNull() ? Optional.empty() : Optional.of(MAPPER.treeToValue(node, Permissions.class)); + } + + private static Object component(MetadataChangeBody body, String name) { + try { + return body.getClass().getMethod(name).invoke(body); + } catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException missing) { + throw new AssertionError(body.getClass().getSimpleName() + " has no component " + name, missing); + } + } + + private static String camelCase(String snake) { + StringBuilder camel = new StringBuilder(); + boolean upper = false; + for (char character : snake.toCharArray()) { + if (character == '_') { + upper = true; + } else { + camel.append(upper ? Character.toUpperCase(character) : character); + upper = false; + } + } + return camel.toString(); + } + + private static String snakeCase(String camel) { + return camel.replaceAll("([A-Z])", "_$1").toLowerCase(Locale.ROOT); + } + + private static MetadataChangeStart start(JsonNode input) { + return switch (input.get("start").asString()) { + case "earliest" -> MetadataChangeStart.earliest(); + case "latest" -> MetadataChangeStart.latest(); + case "after" -> + MetadataChangeStart.after( + MetadataChangeCursor.parse(input.get("cursor").asString())); + default -> + input.get("page_token").isNull() + ? MetadataChangeStart.current() + : MetadataChangeStart.current(MetadataCurrentPageToken.parse( + input.get("page_token").asString())); + }; + } + + private static JsonNode load() { + try (InputStream fixture = MetadataChangesFixture.class.getResourceAsStream("/metadata_changes.json")) { + if (fixture == null) { + throw new IllegalStateException("metadata_changes.json is not on the test classpath"); + } + return JsonMapper.builder().build().readTree(fixture); + } catch (IOException unreadable) { + throw new UncheckedIOException(unreadable); + } + } +}
diff --git a/foreign/java/java-sdk/src/test/java/org/apache/iggy/ProtocolParityTest.java b/foreign/java/java-sdk/src/test/java/org/apache/iggy/ProtocolParityTest.java index 7ba82ea..094d219 100644 --- a/foreign/java/java-sdk/src/test/java/org/apache/iggy/ProtocolParityTest.java +++ b/foreign/java/java-sdk/src/test/java/org/apache/iggy/ProtocolParityTest.java
@@ -63,6 +63,7 @@ Map.entry(CommandCode.System.DESCRIBE_OPTIONS, "DESCRIBE_OPTIONS_CODE"), Map.entry(CommandCode.System.BIND_SESSION, "BIND_SESSION_CODE"), Map.entry(CommandCode.System.UPDATE_RESOURCE, "UPDATE_RESOURCE_CODE"), + Map.entry(CommandCode.System.GET_METADATA_CHANGES, "GET_METADATA_CHANGES_CODE"), Map.entry(CommandCode.System.GET_ME, "GET_ME_CODE"), Map.entry(CommandCode.System.GET_CLIENT, "GET_CLIENT_CODE"), Map.entry(CommandCode.System.GET_ALL_CLIENTS, "GET_CLIENTS_CODE"),
diff --git a/foreign/java/java-sdk/src/test/java/org/apache/iggy/client/async/MetadataChangesTest.java b/foreign/java/java-sdk/src/test/java/org/apache/iggy/client/async/MetadataChangesTest.java new file mode 100644 index 0000000..2ce6cbe --- /dev/null +++ b/foreign/java/java-sdk/src/test/java/org/apache/iggy/client/async/MetadataChangesTest.java
@@ -0,0 +1,248 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.client.async; + +import org.apache.iggy.client.BaseIntegrationTest; +import org.apache.iggy.client.async.tcp.AsyncIggyTcpClient; +import org.apache.iggy.exception.IggyErrorCode; +import org.apache.iggy.exception.IggyServerException; +import org.apache.iggy.exception.IggyValidationException; +import org.apache.iggy.stream.EncryptionKeySource; +import org.apache.iggy.stream.EncryptionKeys; +import org.apache.iggy.stream.StreamOptions; +import org.apache.iggy.system.MetadataChangeBody; +import org.apache.iggy.system.MetadataChangeCursor; +import org.apache.iggy.system.MetadataChangeGroup; +import org.apache.iggy.system.MetadataChangeKind; +import org.apache.iggy.system.MetadataChangeRecord; +import org.apache.iggy.system.MetadataChangeStart; +import org.apache.iggy.system.MetadataChangeStatus; +import org.apache.iggy.system.MetadataChanges; +import org.apache.iggy.system.MetadataChangesQuery; +import org.apache.iggy.system.MetadataCurrentItem; +import org.apache.iggy.user.GlobalPermissions; +import org.apache.iggy.user.Permissions; +import org.apache.iggy.user.UserStatus; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.security.SecureRandom; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.UUID; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.TimeUnit; +import java.util.function.Predicate; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** Metadata changes through the async TCP client. */ +class MetadataChangesTest extends BaseIntegrationTest { + + private static final int TIMEOUT_SECONDS = 5; + private static final int FOLLOW_ATTEMPTS = 100; + private static final long FOLLOW_INTERVAL_MILLIS = 100; + + private final List<Long> streams = new ArrayList<>(); + private final List<Long> users = new ArrayList<>(); + private AsyncIggyTcpClient client; + + @BeforeEach + void setUp() throws Exception { + client = connect(); + await(client.users().login("iggy", "iggy")); + } + + @AfterEach + void tearDown() throws Exception { + for (Long stream : streams) { + client.streams().deleteStream(stream).exceptionally(ignored -> null).get(TIMEOUT_SECONDS, TimeUnit.SECONDS); + } + for (Long user : users) { + client.users().deleteUser(user).exceptionally(ignored -> null).get(TIMEOUT_SECONDS, TimeUnit.SECONDS); + } + client.close().get(TIMEOUT_SECONDS, TimeUnit.SECONDS); + } + + @Test + void givenKeyedStreamWhenFollowingFromLatestShouldReturnItsRecordWithFingerprintsAndNoKey() throws Exception { + // given + var latest = await(client.system().getMetadataChanges(query(MetadataChangeStart.latest(), 1))); + assertThat(latest.status()).isEqualTo(MetadataChangeStatus.OK); + byte[] key = new byte[EncryptionKeys.KEY_LENGTH]; + new SecureRandom().nextBytes(key); + String name = unique("async-changes"); + + // when + var created = await(client.streams() + .createStream(name, StreamOptions.builder().encryptionKey(key).build())); + streams.add(created.id()); + var records = follow( + latest.next().orElseThrow(), + followed -> followed.stream().anyMatch(record -> isCreationOf(record, created.id(), name))); + + // then + var record = records.stream() + .filter(followed -> isCreationOf(followed, created.id(), name)) + .findFirst() + .orElseThrow(); + assertThat(record.actorUserId()).contains(0L); + assertThat(record.group()).contains(MetadataChangeGroup.STREAMS); + var stream = (MetadataChangeBody.StreamChange) record.body(); + assertThat(stream.streamCreatedOp()).isEqualTo(record.op()); + assertThat(stream.encryptionKeySource()).contains(EncryptionKeySource.STREAM); + assertThat(stream.encryptionKeyFingerprint()).contains(EncryptionKeys.fingerprint(key)); + assertThat(stream.derivedOptions()).doesNotContainKeys("encryption_key", "encryption_key_wrapped"); + assertThat(stream.options()).doesNotContainKeys("encryption_key", "encryption_key_wrapped"); + } + + @Test + void givenCurrentStateScanWhenPagingShouldEndWithAnAnchorToFollowFrom() throws Exception { + // given + streams.add( + await(client.streams().createStream(unique("async-current"))).id()); + streams.add( + await(client.streams().createStream(unique("async-current"))).id()); + + // when + List<MetadataCurrentItem> items = new ArrayList<>(); + var start = MetadataChangeStart.current(); + MetadataChanges page; + do { + page = await(client.system().getMetadataChanges(query(start, 1))); + assertThat(page.status()).isEqualTo(MetadataChangeStatus.OK); + items.addAll(page.items()); + start = page.currentPageToken().map(MetadataChangeStart::current).orElse(start); + } while (page.currentPageToken().isPresent()); + var followed = await(client.system() + .getMetadataChanges(query(MetadataChangeStart.after(page.next().orElseThrow()), 1))); + + // then + assertThat(items.stream() + .map(MetadataCurrentItem::body) + .filter(MetadataChangeBody.StreamChange.class::isInstance) + .map(body -> ((MetadataChangeBody.StreamChange) body).streamId())) + .containsAll(streams); + assertThat(followed.status()).isEqualTo(MetadataChangeStatus.OK); + } + + @Test + void givenQueryTheRulesRefuseWhenReadingChangesShouldFailTheFutureBeforeSending() { + var query = MetadataChangesQuery.builder() + .start(MetadataChangeStart.current()) + .groups(MetadataChangeGroup.TOPICS) + .build(); + + assertThatThrownBy(() -> await(client.system().getMetadataChanges(query))) + .isInstanceOfSatisfying( + IggyValidationException.class, + refusal -> assertThat(refusal.getErrorCode()).isEqualTo(IggyErrorCode.INVALID_COMMAND)); + } + + @Test + void givenUserWithoutManageServersWhenReadingChangesShouldBeRefusedAsUnauthorized() throws Exception { + // given + String username = unique("async-reader"); + var permissions = new Permissions( + new GlobalPermissions(false, true, false, false, false, false, false, false, false, false), Map.of()); + users.add(await(client.users() + .createUser(username, "changes-secret", UserStatus.Active, Optional.of(permissions))) + .id()); + var reader = connect(); + + try { + await(reader.users().login(username, "changes-secret")); + + // then + assertThatThrownBy(() -> await(reader.system() + .getMetadataChanges(MetadataChangesQuery.builder().build()))) + .isInstanceOfSatisfying( + IggyServerException.class, + refusal -> assertThat(refusal.getErrorCode()).isEqualTo(IggyErrorCode.UNAUTHORIZED)); + } finally { + reader.close().get(TIMEOUT_SECONDS, TimeUnit.SECONDS); + } + } + + private List<MetadataChangeRecord> follow(MetadataChangeCursor cursor, Predicate<List<MetadataChangeRecord>> done) + throws Exception { + List<MetadataChangeRecord> records = new ArrayList<>(); + var position = cursor; + for (int attempt = 0; attempt < FOLLOW_ATTEMPTS; attempt++) { + var page = await(client.system() + .getMetadataChanges(query(MetadataChangeStart.after(position), MetadataChangesQuery.MAX_LIMIT))); + assertThat(page.status()).isEqualTo(MetadataChangeStatus.OK); + position = page.next().orElseThrow(); + records.addAll(page.records()); + if (!page.hasMore() && done.test(records)) { + return records; + } + if (!page.hasMore()) { + Thread.sleep(FOLLOW_INTERVAL_MILLIS); + } + } + throw new AssertionError("The followed metadata changes never reached the expected records"); + } + + private static MetadataChangesQuery query(MetadataChangeStart start, long limit) { + return MetadataChangesQuery.builder() + .start(start) + .groups(MetadataChangeGroup.STREAMS) + .limit(limit) + .build(); + } + + private static boolean isCreationOf(MetadataChangeRecord record, long streamId, String name) { + return record.kind().equals(Optional.of(MetadataChangeKind.STREAM_CREATED)) + && record.body() instanceof MetadataChangeBody.StreamChange stream + && stream.streamId() == streamId + && stream.name().equals(name); + } + + private AsyncIggyTcpClient connect() throws Exception { + AsyncIggyTcpClient connected = AsyncIggyTcpClient.builder() + .host(serverHost()) + .port(serverTcpPort()) + .build(); + connected.connect().get(TIMEOUT_SECONDS, TimeUnit.SECONDS); + return connected; + } + + /** Waits for {@code future}, rethrowing an Iggy failure as itself rather than wrapped. */ + private static <T> T await(CompletableFuture<T> future) throws Exception { + try { + return future.get(TIMEOUT_SECONDS, TimeUnit.SECONDS); + } catch (ExecutionException failed) { + if (failed.getCause() instanceof RuntimeException cause) { + throw cause; + } + throw failed; + } + } + + private static String unique(String prefix) { + return prefix + "-" + UUID.randomUUID().toString().substring(0, 8); + } +}
diff --git a/foreign/java/java-sdk/src/test/java/org/apache/iggy/client/blocking/SystemClientBaseTest.java b/foreign/java/java-sdk/src/test/java/org/apache/iggy/client/blocking/SystemClientBaseTest.java index 93ee820..f827133 100644 --- a/foreign/java/java-sdk/src/test/java/org/apache/iggy/client/blocking/SystemClientBaseTest.java +++ b/foreign/java/java-sdk/src/test/java/org/apache/iggy/client/blocking/SystemClientBaseTest.java
@@ -22,25 +22,61 @@ import org.apache.commons.lang3.StringUtils; import org.apache.iggy.cluster.ClusterNodeRole; import org.apache.iggy.cluster.ClusterNodeStatus; +import org.apache.iggy.exception.IggyErrorCode; +import org.apache.iggy.exception.IggyServerException; +import org.apache.iggy.exception.IggyValidationException; import org.apache.iggy.message.HeaderKind; +import org.apache.iggy.stream.EncryptionKeySource; +import org.apache.iggy.stream.EncryptionKeys; +import org.apache.iggy.stream.StreamBase; import org.apache.iggy.stream.StreamOptionKey; +import org.apache.iggy.stream.StreamOptions; +import org.apache.iggy.stream.StreamUpdate; +import org.apache.iggy.system.MetadataChangeBody; +import org.apache.iggy.system.MetadataChangeCursor; +import org.apache.iggy.system.MetadataChangeGroup; +import org.apache.iggy.system.MetadataChangeKind; +import org.apache.iggy.system.MetadataChangePhase; +import org.apache.iggy.system.MetadataChangeRecord; +import org.apache.iggy.system.MetadataChangeStart; +import org.apache.iggy.system.MetadataChangeStatus; +import org.apache.iggy.system.MetadataChanges; +import org.apache.iggy.system.MetadataChangesQuery; +import org.apache.iggy.system.MetadataCurrentItem; import org.apache.iggy.system.OptionSpec; import org.apache.iggy.system.OptionsScope; +import org.apache.iggy.user.GlobalPermissions; +import org.apache.iggy.user.Permissions; import org.apache.iggy.user.UserOptionKey; +import org.apache.iggy.user.UserOptions; +import org.apache.iggy.user.UserStatus; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import tools.jackson.databind.JsonNode; +import java.math.BigInteger; +import java.security.SecureRandom; import java.util.ArrayList; import java.util.List; import java.util.Locale; +import java.util.Map; +import java.util.Optional; +import java.util.UUID; +import java.util.function.Predicate; import java.util.stream.Collectors; import static org.apache.iggy.ResourceOptionsFixture.section; import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; public abstract class SystemClientBaseTest extends IntegrationTest { + private static final long ROOT_USER_ID = 0; + private static final int FOLLOW_ATTEMPTS = 100; + private static final long FOLLOW_INTERVAL_MILLIS = 100; + private static final List<String> HIDDEN_OPTIONS = + List.of("encryption_key", "encryption_key_wrapped", "critical_keys"); + protected SystemClient systemClient; @BeforeEach @@ -151,4 +187,380 @@ // then assertThat(result).isEqualTo("pong"); } + + @Test + void givenStreamLifecycleWhenFollowingFromLatestShouldReturnItsRecords() throws InterruptedException { + // given + var latest = read(MetadataChangeStart.latest(), List.of(MetadataChangeGroup.STREAMS), 1); + assertThat(latest.status()).isEqualTo(MetadataChangeStatus.OK); + assertThat(latest.phase()).isEqualTo(MetadataChangePhase.CHANGES); + assertThat(latest.records()).isEmpty(); + assertThat(latest.logId()).isEqualTo(latest.next().map(MetadataChangeCursor::logId)); + String name = unique("changes"); + String renamed = name + "-renamed"; + + // when + var created = client.streams().createStream(name); + client.streams() + .updateStream(created.id(), StreamUpdate.builder().name(renamed).build()); + client.streams().deleteStream(created.id()); + var records = follow( + latest.next().orElseThrow(), + List.of(MetadataChangeGroup.STREAMS), + followed -> ofStream(followed, created.id(), name).size() >= 3); + + // then + assertThat(records).allSatisfy(record -> assertThat(record.group()).contains(MetadataChangeGroup.STREAMS)); + var mine = ofStream(records, created.id(), name); + assertThat(mine) + .extracting(record -> record.kind().orElseThrow()) + .containsExactly( + MetadataChangeKind.STREAM_CREATED, + MetadataChangeKind.STREAM_UPDATED, + MetadataChangeKind.STREAM_DELETED); + assertThat(mine).allSatisfy(record -> { + assertThat(record.actorUserId()).contains(ROOT_USER_ID); + assertThat(record.checksum()).isPositive(); + }); + var createdBody = (MetadataChangeBody.StreamChange) mine.get(0).body(); + assertThat(createdBody.streamCreatedOp()).isEqualTo(mine.get(0).op()); + assertThat(createdBody.optionsUnreadable()).isFalse(); + assertThat(((MetadataChangeBody.StreamChange) mine.get(1).body()).name()) + .isEqualTo(renamed); + var deleted = (MetadataChangeBody.StreamDeleted) mine.get(2).body(); + assertThat(deleted.name()).isEqualTo(renamed); + assertThat(deleted.createdAt()).isEqualTo(createdBody.createdAt()); + assertThat(deleted.requestedOp()).isEmpty(); + } + + @Test + void givenTopicsGroupWhenFollowingShouldReturnOnlyTopicRecords() throws InterruptedException { + // given + var cursor = latestCursor(List.of(MetadataChangeGroup.TOPICS)); + var stream = client.streams().createStream(unique("changes-filter")); + trackStream(stream.id()); + + // when + client.topics().createTopic(stream.id(), "events", 1L); + var records = follow( + cursor, + List.of(MetadataChangeGroup.TOPICS), + followed -> followed.stream().anyMatch(record -> isTopicOf(record, stream.id()))); + + // then + assertThat(records).allSatisfy(record -> assertThat(record.group()).contains(MetadataChangeGroup.TOPICS)); + var topic = (MetadataChangeBody.TopicChange) records.stream() + .filter(record -> isTopicOf(record, stream.id())) + .findFirst() + .orElseThrow() + .body(); + assertThat(topic.name()).isEqualTo("events"); + assertThat(topic.partitionsCount()).isEqualTo(1L); + assertThat(topic.derivedOptions()).containsKey("segment_size"); + } + + @Test + void givenLimitOfOneWhenFollowingShouldPageOneRecordAtATime() throws InterruptedException { + // given + var cursor = latestCursor(List.of(MetadataChangeGroup.STREAMS)); + List<String> names = List.of(unique("changes-page"), unique("changes-page"), unique("changes-page")); + names.forEach(name -> trackStream(client.streams().createStream(name).id())); + + // when + List<String> seen = new ArrayList<>(); + for (int attempt = 0; attempt < FOLLOW_ATTEMPTS && seen.size() < names.size(); attempt++) { + var from = cursor; + var page = read(MetadataChangeStart.after(from), List.of(MetadataChangeGroup.STREAMS), 1); + assertThat(page.status()).isEqualTo(MetadataChangeStatus.OK); + assertThat(page.records()).hasSizeLessThanOrEqualTo(1); + var next = page.next().orElseThrow(); + page.records().forEach(record -> { + assertThat(record.op()).isGreaterThan(from.op()); + assertThat(next.op()).isEqualTo(record.op()); + assertThat(next.ordinal()).isEqualTo(record.ordinal()); + if (record.body() instanceof MetadataChangeBody.StreamChange stream && names.contains(stream.name())) { + seen.add(stream.name()); + } + }); + if (page.records().isEmpty() && !page.hasMore()) { + Thread.sleep(FOLLOW_INTERVAL_MILLIS); + } + cursor = next; + } + + // then + assertThat(seen).isEqualTo(names); + } + + @Test + void givenCurrentStateScanWhenPagingShouldListTheStreamsTheListReturns() { + // given + var stream = client.streams().createStream(unique("changes-current")); + trackStream(stream.id()); + trackStream(client.streams().createStream(unique("changes-current")).id()); + client.topics().createTopic(stream.id(), "events", 2L); + var groups = List.of(MetadataChangeGroup.STREAMS, MetadataChangeGroup.TOPICS, MetadataChangeGroup.PARTITIONS); + + // when + // A deferred delete of another test can complete at any time, so the scan is compared with a stream list + // that did not change around it. + List<MetadataCurrentItem> items = List.of(); + Map<Long, String> listed = Map.of(); + for (int attempt = 0; attempt < 3; attempt++) { + Map<Long, String> before = listStreams(); + items = scanCurrent(groups); + listed = listStreams(); + if (before.equals(listed)) { + break; + } + } + + // then + Map<Long, String> projected = items.stream() + .map(MetadataCurrentItem::body) + .filter(MetadataChangeBody.StreamChange.class::isInstance) + .map(MetadataChangeBody.StreamChange.class::cast) + .collect(Collectors.toMap( + MetadataChangeBody.StreamChange::streamId, MetadataChangeBody.StreamChange::name)); + assertThat(projected).isEqualTo(listed); + var topic = items.stream() + .map(MetadataCurrentItem::body) + .filter(MetadataChangeBody.TopicItem.class::isInstance) + .map(MetadataChangeBody.TopicItem.class::cast) + .filter(item -> item.streamId() == stream.id()) + .findFirst() + .orElseThrow(); + assertThat(topic.name()).isEqualTo("events"); + assertThat(items.stream() + .map(MetadataCurrentItem::body) + .filter(MetadataChangeBody.PartitionsItem.class::isInstance) + .map(MetadataChangeBody.PartitionsItem.class::cast) + .filter(item -> item.streamId() == stream.id() && item.topicId() == topic.topicId()) + .findFirst() + .orElseThrow() + .partitionsCount()) + .isEqualTo(2L); + } + + @Test + void givenCursorOfAnotherHistoryOrBeyondTheHeadWhenFollowingShouldAnswerResetOrBehind() { + // given + var latest = latestCursor(List.of(MetadataChangeGroup.STREAMS)); + var otherHistory = new MetadataChangeCursor( + latest.logId().add(BigInteger.ONE), latest.op(), latest.checksum(), latest.ordinal()); + var beyondHead = new MetadataChangeCursor( + latest.logId(), BigInteger.ONE.shiftLeft(64).subtract(BigInteger.ONE), latest.checksum(), 0); + + // when + var reset = read(MetadataChangeStart.after(otherHistory), List.of(MetadataChangeGroup.STREAMS), 100); + var behind = read(MetadataChangeStart.after(beyondHead), List.of(MetadataChangeGroup.STREAMS), 100); + + // then + assertThat(reset.status()).isEqualTo(MetadataChangeStatus.RESET); + assertThat(reset.records()).isEmpty(); + assertThat(reset.next()).isEmpty(); + assertThat(behind.status()).isEqualTo(MetadataChangeStatus.BEHIND); + assertThat(behind.next()).contains(beyondHead); + } + + @Test + void givenQueryTheRulesRefuseWhenReadingChangesShouldRefuseItBeforeSending() { + // given + trackStream(client.streams().createStream(unique("changes-refused")).id()); + trackStream(client.streams().createStream(unique("changes-refused")).id()); + var token = read(MetadataChangeStart.current(), List.of(MetadataChangeGroup.STREAMS), 1) + .currentPageToken() + .orElseThrow(); + var notCanonical = new MetadataChangeCursor(BigInteger.ONE, BigInteger.ZERO, BigInteger.TWO, 0); + List<MetadataChangesQuery> refused = List.of( + MetadataChangesQuery.builder().limit(0).build(), + MetadataChangesQuery.builder() + .limit(MetadataChangesQuery.MAX_LIMIT + 1) + .build(), + MetadataChangesQuery.builder() + .since(BigInteger.TWO) + .until(BigInteger.ONE) + .build(), + MetadataChangesQuery.builder() + .start(MetadataChangeStart.latest()) + .since(BigInteger.ONE) + .build(), + MetadataChangesQuery.builder() + .start(MetadataChangeStart.current()) + .groups(MetadataChangeGroup.TOPICS) + .build(), + MetadataChangesQuery.builder() + .start(MetadataChangeStart.current(token)) + .groups(MetadataChangeGroup.STREAMS, MetadataChangeGroup.TOPICS) + .build(), + MetadataChangesQuery.builder() + .start(MetadataChangeStart.after(notCanonical)) + .build()); + + // then + for (MetadataChangesQuery query : refused) { + assertThatThrownBy(() -> systemClient.getMetadataChanges(query)) + .isInstanceOfSatisfying( + IggyValidationException.class, + refusal -> assertThat(refusal.getErrorCode()).isEqualTo(IggyErrorCode.INVALID_COMMAND)); + } + } + + @Test + void givenUserWithoutManageServersWhenReadingChangesShouldBeRefusedAsUnauthorized() { + // given + String username = unique("changes-reader"); + var permissions = new Permissions( + new GlobalPermissions(false, true, false, false, false, false, false, false, false, false), Map.of()); + var user = client.users() + .createUser( + username, "changes-secret", UserStatus.Active, Optional.of(permissions), UserOptions.empty()); + trackUser(user.id()); + var reader = getClient(); + reader.users().login(username, "changes-secret"); + + // then + assertThatThrownBy(() -> reader.system() + .getMetadataChanges(MetadataChangesQuery.builder().build())) + .isInstanceOfSatisfying( + IggyServerException.class, + refusal -> assertThat(refusal.getErrorCode()).isEqualTo(IggyErrorCode.UNAUTHORIZED)); + } + + @Test + void givenKeyedAndGeneratedStreamKeysWhenFollowingShouldReportFingerprintsAndNoKey() throws InterruptedException { + // given + var cursor = latestCursor(List.of(MetadataChangeGroup.STREAMS)); + byte[] key = new byte[EncryptionKeys.KEY_LENGTH]; + new SecureRandom().nextBytes(key); + String keyedName = unique("changes-keyed"); + String generatedName = unique("changes-generated"); + + // when + var keyed = client.streams() + .createStream( + keyedName, StreamOptions.builder().encryptionKey(key).build()); + trackStream(keyed.id()); + var generated = client.streams().createStream(generatedName); + trackStream(generated.id()); + var records = follow( + cursor, + List.of(MetadataChangeGroup.STREAMS), + followed -> ofStream(followed, keyed.id(), keyedName).size() == 1 + && ofStream(followed, generated.id(), generatedName).size() == 1); + + // then + var keyedStream = (MetadataChangeBody.StreamChange) + ofStream(records, keyed.id(), keyedName).get(0).body(); + assertThat(keyedStream.encryptionKeySource()).contains(EncryptionKeySource.STREAM); + assertThat(keyedStream.encryptionKeyFingerprint()).contains(EncryptionKeys.fingerprint(key)); + var generatedStream = (MetadataChangeBody.StreamChange) + ofStream(records, generated.id(), generatedName).get(0).body(); + assertThat(generatedStream.encryptionKeySource()).contains(EncryptionKeySource.SERVER); + assertThat(generatedStream.encryptionKeyFingerprint()).isPresent(); + for (var stream : List.of(keyedStream, generatedStream)) { + for (var options : List.of(stream.options(), stream.derivedOptions())) { + assertThat(options).doesNotContainKeys(HIDDEN_OPTIONS.toArray(String[]::new)); + assertThat(options.values()) + .noneSatisfy(value -> assertThat(value.value()).isEqualTo(key)); + } + } + } + + private MetadataChanges read(MetadataChangeStart start, List<MetadataChangeGroup> groups, long limit) { + return systemClient.getMetadataChanges(MetadataChangesQuery.builder() + .start(start) + .groups(groups) + .limit(limit) + .build()); + } + + private MetadataChangeCursor latestCursor(List<MetadataChangeGroup> groups) { + var page = read(MetadataChangeStart.latest(), groups, 1); + assertThat(page.status()).isEqualTo(MetadataChangeStatus.OK); + return page.next().orElseThrow(); + } + + /** + * Follows the change log from {@code cursor} until {@code done} holds. A two phase delete commits its record + * after the call returns, so an empty tail is polled again. + */ + private List<MetadataChangeRecord> follow( + MetadataChangeCursor cursor, List<MetadataChangeGroup> groups, Predicate<List<MetadataChangeRecord>> done) + throws InterruptedException { + List<MetadataChangeRecord> records = new ArrayList<>(); + var position = cursor; + for (int attempt = 0; attempt < FOLLOW_ATTEMPTS; attempt++) { + var page = read(MetadataChangeStart.after(position), groups, MetadataChangesQuery.MAX_LIMIT); + assertThat(page.status()).isEqualTo(MetadataChangeStatus.OK); + position = page.next().orElseThrow(); + records.addAll(page.records()); + if (!page.hasMore() && done.test(records)) { + return records; + } + if (!page.hasMore()) { + Thread.sleep(FOLLOW_INTERVAL_MILLIS); + } + } + throw new AssertionError("The followed metadata changes never reached the expected records"); + } + + private List<MetadataCurrentItem> scanCurrent(List<MetadataChangeGroup> groups) { + List<MetadataCurrentItem> items = new ArrayList<>(); + var start = MetadataChangeStart.current(); + while (true) { + var page = read(start, groups, 2); + assertThat(page.status()).isEqualTo(MetadataChangeStatus.OK); + assertThat(page.phase()).isEqualTo(MetadataChangePhase.CURRENT); + assertThat(page.records()).isEmpty(); + assertThat(page.hasMore()).isEqualTo(page.currentPageToken().isPresent()); + items.addAll(page.items()); + if (page.currentPageToken().isEmpty()) { + var anchor = page.next().orElseThrow(); + assertThat(read(MetadataChangeStart.after(anchor), groups, 1).status()) + .isEqualTo(MetadataChangeStatus.OK); + return items; + } + assertThat(page.currentPageToken().orElseThrow().groups()).containsExactlyElementsOf(groups); + start = MetadataChangeStart.current(page.currentPageToken().orElseThrow()); + } + } + + private Map<Long, String> listStreams() { + return client.streams().getStreams().stream().collect(Collectors.toMap(StreamBase::id, StreamBase::name)); + } + + /** The records of the stream created under {@code name}, matched by full identity because ids are reused. */ + private static List<MetadataChangeRecord> ofStream(List<MetadataChangeRecord> records, long streamId, String name) { + Optional<BigInteger> createdOp = records.stream() + .filter(record -> record.kind().equals(Optional.of(MetadataChangeKind.STREAM_CREATED))) + .filter(record -> record.body() instanceof MetadataChangeBody.StreamChange stream + && stream.streamId() == streamId + && stream.name().equals(name)) + .map(MetadataChangeRecord::op) + .findFirst(); + return records.stream() + .filter(record -> createdOp.isPresent() && namesStream(record, streamId, createdOp.get())) + .toList(); + } + + private static boolean namesStream(MetadataChangeRecord record, long streamId, BigInteger createdOp) { + if (record.body() instanceof MetadataChangeBody.StreamChange stream) { + return stream.streamId() == streamId && stream.streamCreatedOp().equals(createdOp); + } + return record.body() instanceof MetadataChangeBody.StreamDeleted deleted + && deleted.streamId() == streamId + && deleted.streamCreatedOp().equals(createdOp); + } + + private static boolean isTopicOf(MetadataChangeRecord record, long streamId) { + return record.body() instanceof MetadataChangeBody.TopicChange topic + && record.kind().equals(Optional.of(MetadataChangeKind.TOPIC_CREATED)) + && topic.streamId() == streamId + && topic.topicCreatedOp().equals(record.op()); + } + + private static String unique(String prefix) { + return prefix + "-" + UUID.randomUUID().toString().replace("-", ""); + } }
diff --git a/foreign/java/java-sdk/src/test/java/org/apache/iggy/client/blocking/http/MetadataChangesJsonTest.java b/foreign/java/java-sdk/src/test/java/org/apache/iggy/client/blocking/http/MetadataChangesJsonTest.java new file mode 100644 index 0000000..fa46443 --- /dev/null +++ b/foreign/java/java-sdk/src/test/java/org/apache/iggy/client/blocking/http/MetadataChangesJsonTest.java
@@ -0,0 +1,113 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.client.blocking.http; + +import org.apache.iggy.exception.IggyMalformedResponseException; +import org.apache.iggy.serde.BytesDeserializer; +import org.apache.iggy.system.MetadataChangeBody; +import org.apache.iggy.system.MetadataChangeKind; +import org.junit.jupiter.api.Test; +import tools.jackson.databind.JsonNode; +import tools.jackson.databind.node.ObjectNode; + +import static org.apache.iggy.MetadataChangesFixture.assertAnswer; +import static org.apache.iggy.MetadataChangesFixture.assertItem; +import static org.apache.iggy.MetadataChangesFixture.assertRecord; +import static org.apache.iggy.MetadataChangesFixture.buffer; +import static org.apache.iggy.MetadataChangesFixture.parse; +import static org.apache.iggy.MetadataChangesFixture.section; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** The shared metadata changes vectors through the HTTP JSON reader, which must agree with the binary decoder. */ +class MetadataChangesJsonTest { + + private static final String EMPTY_PAGE = """ + {"status":"ok","phase":"changes","log_id":null,"oldest_op":0,"retained_since":0,"head_op":0, + "next":null,"has_more":false,"records":[],"items":[],"current_page_token":null} + """; + + @Test + void givenSharedRecordsWhenReadingJsonShouldMatchTheBinaryDecoder() { + for (JsonNode vector : section("records")) { + var record = MetadataChangesJson.record(vector.get("expected")); + var binary = BytesDeserializer.readMetadataChangeRecord( + buffer(vector.get("record_hex").asString())); + + assertRecord(vector.get("expected"), record); + assertThat(record).as(vector.get("case").asString()).isEqualTo(binary); + } + } + + @Test + void givenSharedItemsWhenReadingJsonShouldMatchTheBinaryDecoder() { + for (JsonNode vector : section("items")) { + var item = MetadataChangesJson.item(vector.get("expected")); + var binary = BytesDeserializer.readMetadataCurrentItem( + buffer(vector.get("item_hex").asString())); + + assertItem(vector.get("expected"), item); + assertThat(item).as(vector.get("case").asString()).isEqualTo(binary); + } + } + + @Test + void givenSharedResponsesWhenReadingJsonShouldReachTheServerJson() { + for (JsonNode vector : section("responses")) { + assertAnswer(vector.get("expected"), MetadataChangesJson.read(vector.get("expected"))); + } + } + + @Test + void givenRecordOfAKindThisBuildDoesNotKnowWhenReadingJsonShouldKeepItWhole() { + JsonNode unknown = parse(""" + {"op":9,"ordinal":0,"checksum":"0000000000000000000000000000000a","timestamp":1, + "group":"streams","kind":"stream_archived","archive":"cold"} + """); + + var record = MetadataChangesJson.record(unknown); + + assertThat(record.kind()).isEmpty(); + assertThat(record.group()).isEmpty(); + var body = (MetadataChangeBody.UnknownJson) record.body(); + assertThat(body.group()).isEqualTo("streams"); + assertThat(body.kind()).contains("stream_archived"); + assertThat(parse(body.json())).isEqualTo(unknown); + } + + @Test + void givenKnownKindUnderAnotherGroupWhenReadingJsonShouldRefuseIt() { + ObjectNode record = + (ObjectNode) section("records").get(0).get("expected").deepCopy(); + record.put("group", "topics"); + + assertThat(record.get("kind").asString()).isEqualTo(MetadataChangeKind.STREAM_CREATED.value()); + assertThatThrownBy(() -> MetadataChangesJson.record(record)).isInstanceOf(IggyMalformedResponseException.class); + } + + @Test + void givenValueOfTheWrongTypeWhenReadingJsonShouldRefuseTheAnswer() { + ObjectNode answer = (ObjectNode) parse(EMPTY_PAGE); + answer.put("oldest_op", "1"); + + assertThatThrownBy(() -> MetadataChangesJson.read(answer)).isInstanceOf(IggyMalformedResponseException.class); + assertThat(MetadataChangesJson.read(parse(EMPTY_PAGE)).records()).isEmpty(); + } +}
diff --git a/foreign/java/java-sdk/src/test/java/org/apache/iggy/client/blocking/tcp/TlsConnectionTest.java b/foreign/java/java-sdk/src/test/java/org/apache/iggy/client/blocking/tcp/TlsConnectionTest.java index 941e509..1338153 100644 --- a/foreign/java/java-sdk/src/test/java/org/apache/iggy/client/blocking/tcp/TlsConnectionTest.java +++ b/foreign/java/java-sdk/src/test/java/org/apache/iggy/client/blocking/tcp/TlsConnectionTest.java
@@ -27,6 +27,11 @@ import org.apache.iggy.message.PolledMessages; import org.apache.iggy.message.PollingStrategy; import org.apache.iggy.stream.StreamDetails; +import org.apache.iggy.system.MetadataChangeBody; +import org.apache.iggy.system.MetadataChangeGroup; +import org.apache.iggy.system.MetadataChangeStart; +import org.apache.iggy.system.MetadataChangeStatus; +import org.apache.iggy.system.MetadataChangesQuery; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable; import org.slf4j.Logger; @@ -37,6 +42,7 @@ import java.nio.file.Path; import java.util.List; import java.util.Optional; +import java.util.UUID; import java.util.concurrent.ExecutionException; import java.util.concurrent.Executors; import java.util.concurrent.TimeUnit; @@ -110,6 +116,40 @@ } @Test + void readMetadataChangesOverTlsShouldFollowACreatedStream() { + var client = IggyTcpClient.builder() + .host(TLS_HOST) + .port(serverTcpPort()) + .enableTls() + .tlsCertificate(CERTS_DIR.resolve("iggy_ca_cert.pem").toFile()) + .credentials("iggy", "iggy") + .buildAndLogin(); + var query = MetadataChangesQuery.builder() + .start(MetadataChangeStart.latest()) + .groups(MetadataChangeGroup.STREAMS) + .build(); + var latest = client.system().getMetadataChanges(query); + String name = "tls-changes-" + UUID.randomUUID().toString().substring(0, 8); + StreamDetails stream = client.streams().createStream(name); + + try { + var page = client.system() + .getMetadataChanges(query.withStart( + MetadataChangeStart.after(latest.next().orElseThrow()))); + + assertThat(page.status()).isEqualTo(MetadataChangeStatus.OK); + assertThat(page.records()) + .anySatisfy(record -> assertThat(record.body()) + .isInstanceOfSatisfying(MetadataChangeBody.StreamChange.class, created -> { + assertThat(created.streamId()).isEqualTo(stream.id()); + assertThat(created.name()).isEqualTo(name); + })); + } finally { + client.streams().deleteStream(stream.id()); + } + } + + @Test void sendAndReceiveOverTlsShouldWork() { var client = IggyTcpClient.builder() .host(TLS_HOST)
diff --git a/foreign/java/java-sdk/src/test/java/org/apache/iggy/serde/MetadataChangesTest.java b/foreign/java/java-sdk/src/test/java/org/apache/iggy/serde/MetadataChangesTest.java new file mode 100644 index 0000000..b7746ad --- /dev/null +++ b/foreign/java/java-sdk/src/test/java/org/apache/iggy/serde/MetadataChangesTest.java
@@ -0,0 +1,410 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iggy.serde; + +import io.netty.buffer.ByteBuf; +import io.netty.buffer.Unpooled; +import org.apache.iggy.exception.IggyErrorCode; +import org.apache.iggy.exception.IggyMalformedResponseException; +import org.apache.iggy.exception.IggyValidationException; +import org.apache.iggy.system.MetadataChangeBody; +import org.apache.iggy.system.MetadataChangeCursor; +import org.apache.iggy.system.MetadataChangeGroup; +import org.apache.iggy.system.MetadataChangeKind; +import org.apache.iggy.system.MetadataChangePhase; +import org.apache.iggy.system.MetadataChangeStart; +import org.apache.iggy.system.MetadataChangeStatus; +import org.apache.iggy.system.MetadataChangesQuery; +import org.apache.iggy.system.MetadataCurrentPageToken; +import org.assertj.core.api.ThrowableAssert.ThrowingCallable; +import org.junit.jupiter.api.Test; +import tools.jackson.databind.JsonNode; + +import java.math.BigInteger; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.TreeMap; +import java.util.function.Function; +import java.util.function.ToIntFunction; + +import static org.apache.iggy.MetadataChangesFixture.assertAnswer; +import static org.apache.iggy.MetadataChangesFixture.assertItem; +import static org.apache.iggy.MetadataChangesFixture.assertRecord; +import static org.apache.iggy.MetadataChangesFixture.buffer; +import static org.apache.iggy.MetadataChangesFixture.bytes; +import static org.apache.iggy.MetadataChangesFixture.find; +import static org.apache.iggy.MetadataChangesFixture.query; +import static org.apache.iggy.MetadataChangesFixture.refused; +import static org.apache.iggy.MetadataChangesFixture.section; +import static org.apache.iggy.ResourceOptionsFixture.hex; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** + * The shared metadata changes vectors against this SDK's own encoder and decoder. Every request must match the Rust + * encoder byte for byte and every response must decode to the values the server renders as JSON. + */ +class MetadataChangesTest { + + @Test + void givenManyDuplicateGroupsWhenEncodingShouldPreserveEveryRequestedGroup() { + var groups = new ArrayList<MetadataChangeGroup>(); + for (int index = 0; index < 255; index++) { + groups.add(MetadataChangeGroup.STREAMS); + } + groups.add(MetadataChangeGroup.USERS); + var actual = BytesSerializer.toGetMetadataChanges(MetadataChangesQuery.builder() + .groups(groups.toArray(MetadataChangeGroup[]::new)) + .build()); + var expected = BytesSerializer.toGetMetadataChanges(MetadataChangesQuery.builder() + .groups(MetadataChangeGroup.STREAMS, MetadataChangeGroup.USERS) + .build()); + try { + assertThat(actual).isEqualTo(expected); + } finally { + actual.release(); + expected.release(); + } + } + + @Test + void givenSharedConstantsWhenComparingShouldMatchTheServerCatalog() { + JsonNode constants = section("constants"); + assertThat(CommandCode.System.GET_METADATA_CHANGES.getValue()) + .isEqualTo(constants.get("command_code").asInt()); + assertThat(MetadataChangeCursor.PREFIX) + .isEqualTo(constants.get("cursor_prefix").asString()); + assertThat(MetadataCurrentPageToken.PREFIX) + .isEqualTo(constants.get("page_token_prefix").asString()); + assertThat(MetadataChangeCursor.SIZE) + .isEqualTo(constants.get("cursor_size").asInt()); + assertThat(MetadataCurrentPageToken.MAX_SIZE) + .isEqualTo(constants.get("max_page_token_size").asInt()); + assertThat(MetadataCurrentPageToken.MAX_KEY_SIZE) + .isEqualTo(constants.get("max_page_token_key_size").asInt()); + assertThat(MetadataChangesQuery.DEFAULT_LIMIT) + .isEqualTo(constants.get("default_limit").asLong()); + assertThat(MetadataChangesQuery.MAX_LIMIT) + .isEqualTo(constants.get("max_limit").asLong()); + assertThat(MetadataChangesQuery.builder().build().limit()).isEqualTo(MetadataChangesQuery.DEFAULT_LIMIT); + var zero = new MetadataChangeCursor(BigInteger.ONE, BigInteger.ZERO, BigInteger.ZERO, 0); + assertThat(zero.toString()).hasSize(constants.get("cursor_text_length").asInt()); + constants.get("groups").properties().forEach(entry -> { + var group = MetadataChangeGroup.fromValue(entry.getKey()).orElseThrow(); + assertThat(group.asCode()).isEqualTo(entry.getValue().get("code").asInt()); + List<String> parents = new ArrayList<>(); + entry.getValue().get("parents").forEach(parent -> parents.add(parent.asString())); + assertThat(group.parents().stream().map(MetadataChangeGroup::value).toList()) + .isEqualTo(parents); + }); + assertThat(constants.get("groups").size()).isEqualTo(MetadataChangeGroup.values().length); + constants.get("kinds").properties().forEach(entry -> { + var kind = MetadataChangeKind.fromValue(entry.getKey()).orElseThrow(); + assertThat(kind.asCode()).isEqualTo(entry.getValue().get("code").asInt()); + assertThat(kind.group().value()) + .isEqualTo(entry.getValue().get("group").asString()); + }); + assertThat(constants.get("kinds").size()).isEqualTo(MetadataChangeKind.values().length); + assertThat(codes(MetadataChangeStatus.values(), MetadataChangeStatus::value, MetadataChangeStatus::asCode)) + .isEqualTo(codes(constants.get("statuses"))); + assertThat(codes(MetadataChangePhase.values(), MetadataChangePhase::value, MetadataChangePhase::asCode)) + .isEqualTo(codes(constants.get("phases"))); + assertThat(codes( + MetadataChangeBody.MemberLeftReason.values(), + MetadataChangeBody.MemberLeftReason::value, + MetadataChangeBody.MemberLeftReason::asCode)) + .isEqualTo(codes(constants.get("member_left_reasons"))); + assertThat(codes( + MetadataChangeBody.TokenDeletedReason.values(), + MetadataChangeBody.TokenDeletedReason::value, + MetadataChangeBody.TokenDeletedReason::asCode)) + .isEqualTo(codes(constants.get("token_deleted_reasons"))); + assertThat(codes( + MetadataChangeBody.SessionClosedReason.values(), + MetadataChangeBody.SessionClosedReason::value, + MetadataChangeBody.SessionClosedReason::asCode)) + .isEqualTo(codes(constants.get("session_closed_reasons"))); + assertThat(codes( + MetadataChangeBody.PermissionsUpdatedReason.values(), + MetadataChangeBody.PermissionsUpdatedReason::value, + MetadataChangeBody.PermissionsUpdatedReason::asCode)) + .isEqualTo(codes(constants.get("permissions_updated_reasons"))); + Map<String, Integer> startModes = Map.of( + "earliest", MetadataChangeStart.earliest().asCode(), + "latest", MetadataChangeStart.latest().asCode(), + "after", MetadataChangeStart.after(zero).asCode(), + "current", MetadataChangeStart.current().asCode()); + assertThat(new TreeMap<>(startModes)).isEqualTo(codes(constants.get("start_modes"))); + } + + @Test + void givenSharedRecordsWhenDecodingShouldReachTheServerJson() { + for (JsonNode vector : section("records")) { + String name = vector.get("case").asString(); + var record = BytesDeserializer.readMetadataChangeRecord( + buffer(vector.get("record_hex").asString())); + var body = BytesDeserializer.readMetadataChangeBody( + vector.get("group").asInt(), + vector.get("kind").asInt(), + buffer(vector.get("body_hex").asString())); + + assertRecord(vector.get("expected"), record); + assertThat(body).as(name).isEqualTo(record.body()); + assertThat(record.kind().orElseThrow().asCode()) + .as(name) + .isEqualTo(vector.get("kind").asInt()); + } + } + + @Test + void givenSharedItemsWhenDecodingShouldReachTheServerJson() { + for (JsonNode vector : section("items")) { + var item = BytesDeserializer.readMetadataCurrentItem( + buffer(vector.get("item_hex").asString())); + var body = BytesDeserializer.readMetadataItemBody( + vector.get("group").asInt(), buffer(vector.get("body_hex").asString())); + + assertItem(vector.get("expected"), item); + assertThat(body).as(vector.get("case").asString()).isEqualTo(item.body()); + } + } + + @Test + void givenSharedResponsesWhenDecodingShouldReachTheServerJson() { + for (JsonNode vector : section("responses")) { + var answer = BytesDeserializer.readMetadataChanges( + buffer(vector.get("response_hex").asString())); + + assertAnswer(vector.get("expected"), answer); + } + } + + @Test + void givenLargeValuesWhenDecodingShouldKeepEveryUnsignedBitExact() { + JsonNode vector = section("large_values"); + + var record = BytesDeserializer.readMetadataChangeRecord( + buffer(vector.get("record_hex").asString())); + + assertThat(record.op()).isEqualTo(new BigInteger(vector.get("op").asString())); + assertThat(record.ordinal()).isEqualTo(vector.get("ordinal").asLong()); + assertThat(record.timestamp()) + .isEqualTo(new BigInteger(vector.get("timestamp").asString())); + assertThat(record.checksum()) + .isEqualTo(new BigInteger(vector.get("checksum").asString(), 16)); + var session = (MetadataChangeBody.Session) record.body(); + assertThat(session.userCreatedAt()) + .isEqualTo(new BigInteger(vector.get("user_created_at").asString())); + assertThat(session.session()) + .isEqualTo(new BigInteger(vector.get("session").asString())); + assertThat(session.clientId()) + .isEqualTo(new BigInteger(vector.get("client_id").asString(), 16)); + } + + @Test + void givenSharedCursorsWhenParsingShouldRoundTripTheirTextAndBytes() { + for (JsonNode vector : section("cursors")) { + String text = vector.get("text").asString(); + + var cursor = MetadataChangeCursor.parse(text); + + assertThat(cursor.logId()) + .isEqualTo(new BigInteger(vector.get("log_id").asString(), 16)); + assertThat(cursor.op()).isEqualTo(vector.get("op").bigIntegerValue()); + assertThat(cursor.checksum()) + .isEqualTo(new BigInteger(vector.get("checksum").asString(), 16)); + assertThat(cursor.ordinal()).isEqualTo(vector.get("ordinal").asLong()); + assertThat(cursor.toString()).isEqualTo(text); + assertThat(cursor.toBytes()).isEqualTo(bytes(vector.get("hex").asString())); + } + } + + @Test + void givenSharedPageTokensWhenParsingShouldExposeEveryField() { + for (JsonNode vector : section("page_tokens")) { + String text = vector.get("text").asString(); + + var token = MetadataCurrentPageToken.parse(text); + + assertThat(token.anchor().toString()).isEqualTo(vector.get("anchor").asString()); + assertThat(token.requiredHead()) + .isEqualTo(vector.get("required_head").bigIntegerValue()); + assertThat(token.groupsMask()).isEqualTo(vector.get("groups_mask").asInt()); + assertThat(token.groups()).isEqualTo(MetadataChangeGroup.fromMask(token.groupsMask())); + assertThat(token.lastGroup().value()).isEqualTo(vector.get("group").asString()); + assertThat(token.lastKey()) + .isEqualTo(bytes(vector.get("last_key_hex").asString())); + assertThat(token.payload()).isEqualTo(bytes(vector.get("hex").asString())); + assertThat(token.toString()).isEqualTo(text); + assertThat(token).isEqualTo(MetadataCurrentPageToken.parse(text)); + } + } + + @Test + void givenSharedRequestsWhenEncodingShouldMatchTheRustEncoderAndHttpQuery() { + for (JsonNode vector : section("requests")) { + String name = vector.get("case").asString(); + var query = query(vector.get("input")); + + query.validate(); + + assertThat(hex(BytesSerializer.toGetMetadataChanges(query))) + .as(name) + .isEqualTo(vector.get("request_hex").asString()); + assertThat(query.httpQuery()) + .as(name) + .isEqualTo(vector.get("http_query").asString()); + } + } + + @Test + void givenRefusedCursorsAndTokensWhenParsingShouldRefuseThemAsInvalidCommand() { + for (JsonNode vector : refused("cursors")) { + assertRefused(() -> MetadataChangeCursor.parse(vector.get("text").asString())); + } + for (JsonNode vector : refused("page_tokens")) { + assertRefused( + () -> MetadataCurrentPageToken.parse(vector.get("text").asString())); + } + } + + @Test + void givenRefusedBodiesAndResponsesWhenDecodingShouldRefuseThemAsMalformed() { + for (JsonNode vector : refused("record_bodies")) { + assertThatThrownBy(() -> BytesDeserializer.readMetadataChangeBody( + vector.get("group").asInt(), + vector.get("kind").asInt(), + buffer(vector.get("body_hex").asString()))) + .as(vector.get("case").asString()) + .isInstanceOf(IggyMalformedResponseException.class); + } + for (JsonNode vector : refused("responses")) { + assertThatThrownBy(() -> BytesDeserializer.readMetadataChanges( + buffer(vector.get("hex").asString()))) + .as(vector.get("case").asString()) + .isInstanceOf(IggyMalformedResponseException.class); + } + } + + @Test + void givenRefusedQueriesWhenValidatingShouldRefuseThemBeforeSending() { + for (JsonNode vector : refused("queries")) { + var query = query(vector.get("input")); + assertRefused(query::validate); + } + // Of the refused request payloads the SDK can only express a cursor that is not canonical. + var request = bytes( + find(refused("requests"), "non_canonical_cursor").get("hex").asString()); + var cursor = MetadataChangeCursor.fromBytes(Arrays.copyOfRange(request, 1, 45), 0); + assertRefused(() -> MetadataChangesQuery.builder() + .start(MetadataChangeStart.after(cursor)) + .build() + .validate()); + assertRefused(() -> MetadataChangesQuery.builder() + .since(BigInteger.ONE.shiftLeft(64)) + .build() + .validate()); + } + + @Test + void givenEveryRecordBodyWhenTruncatingItShouldRefuseEveryPrefixAndATrailingByte() { + for (JsonNode vector : section("records")) { + byte[] body = bytes(vector.get("body_hex").asString()); + int group = vector.get("group").asInt(); + int kind = vector.get("kind").asInt(); + for (int length = 0; length < body.length; length++) { + ByteBuf prefix = Unpooled.wrappedBuffer(body, 0, length); + assertThatThrownBy(() -> BytesDeserializer.readMetadataChangeBody(group, kind, prefix)) + .as(vector.get("case").asString() + " truncated to " + length) + .isInstanceOf(IggyMalformedResponseException.class); + } + ByteBuf trailing = Unpooled.wrappedBuffer(Arrays.copyOf(body, body.length + 1)); + assertThatThrownBy(() -> BytesDeserializer.readMetadataChangeBody(group, kind, trailing)) + .isInstanceOf(IggyMalformedResponseException.class); + } + } + + @Test + void givenUnknownGroupOrKindWhenDecodingShouldKeepTheCodesAndRawBody() { + byte[] raw = {(byte) 0xde, (byte) 0xad, (byte) 0xbe, (byte) 0xef}; + + var change = BytesDeserializer.readMetadataChangeBody(9, 1, Unpooled.wrappedBuffer(raw)); + var item = BytesDeserializer.readMetadataItemBody(128, Unpooled.wrappedBuffer(raw)); + + var unknownChange = (MetadataChangeBody.Unknown) change; + assertThat(unknownChange.group()).isEqualTo(9); + assertThat(unknownChange.kind()).contains(1); + assertThat(unknownChange.body()).isEqualTo(raw); + var unknownItem = (MetadataChangeBody.Unknown) item; + assertThat(unknownItem.group()).isEqualTo(128); + assertThat(unknownItem.kind()).isEmpty(); + } + + @Test + void givenUnknownBodiesWithEqualBytesWhenComparingShouldBeEqualAndKeepTheirOwnCopy() { + byte[] raw = {1, 2, 3}; + var unknown = new MetadataChangeBody.Unknown(9, Optional.of(1), raw); + var same = new MetadataChangeBody.Unknown(9, Optional.of(1), new byte[] {1, 2, 3}); + + raw[0] = 7; + unknown.body()[1] = 7; + + assertThat(unknown).isEqualTo(same).hasSameHashCodeAs(same); + assertThat(unknown.body()).containsExactly(1, 2, 3); + assertThat(unknown).isNotEqualTo(new MetadataChangeBody.Unknown(9, Optional.empty(), new byte[] {1, 2, 3})); + } + + @Test + void givenGroupsInAnyOrderWhenEncodingShouldKeepBinaryOrderAndSortTheHttpNames() { + var query = MetadataChangesQuery.builder() + .groups(MetadataChangeGroup.USERS, MetadataChangeGroup.STREAMS, MetadataChangeGroup.USERS) + .build(); + + String request = hex(BytesSerializer.toGetMetadataChanges(query)); + + assertThat(request).endsWith("030601060000"); + assertThat(query.httpQuery()).isEqualTo("from=earliest&groups=streams,users&limit=100"); + } + + private static void assertRefused(ThrowingCallable call) { + assertThatThrownBy(call) + .isInstanceOfSatisfying( + IggyValidationException.class, + refusal -> assertThat(refusal.getErrorCode()).isEqualTo(IggyErrorCode.INVALID_COMMAND)); + } + + private static <T> Map<String, Integer> codes(T[] values, Function<T, String> name, ToIntFunction<T> code) { + Map<String, Integer> codes = new TreeMap<>(); + for (T value : values) { + codes.put(name.apply(value), code.applyAsInt(value)); + } + return codes; + } + + private static Map<String, Integer> codes(JsonNode node) { + Map<String, Integer> codes = new TreeMap<>(); + node.properties() + .forEach(entry -> codes.put(entry.getKey(), entry.getValue().asInt())); + return codes; + } +}
diff --git a/foreign/java/java-sdk/src/test/resources/metadata_changes.json b/foreign/java/java-sdk/src/test/resources/metadata_changes.json new file mode 100644 index 0000000..26e7faf --- /dev/null +++ b/foreign/java/java-sdk/src/test/resources/metadata_changes.json
@@ -0,0 +1,1997 @@ +{ + "schema_version": 1, + "generated_by": "core/common/examples/metadata_changes_fixtures.rs", + "conventions": { + "byte_strings": "Every *_hex field is lowercase hex without separators.", + "expected": "expected is the JSON the HTTP route renders. Binary decoders must reach the same values.", + "record_hex": "record_hex is one record envelope as a response carries it. body_hex is the body alone.", + "refused": "Every refused vector must fail to decode or validate. Query and HTTP refusals are invalid_command.", + "request_hex": "request_hex is the command payload without framing. http_query is the query string of GET /metadata/changes.", + "u128_values": "128-bit values (log ids, checksums, client ids) are 32 lowercase hex characters in JSON.", + "u64_values": "Ops and timestamps are JSON numbers below 2^53 in every vector except large_values, which carries them as decimal strings.", + "unknown_kinds": "A record of an unknown group or kind is kept with its codes and raw body, never refused." + }, + "constants": { + "command_code": 17, + "command_name": "metadata.changes", + "cursor_prefix": "mc1_", + "page_token_prefix": "cs1_", + "cursor_size": 44, + "cursor_text_length": 63, + "max_page_token_size": 316, + "max_page_token_key_size": 260, + "default_limit": 100, + "max_limit": 1000, + "groups": { + "consumer_group_members": { + "code": 5, + "class": "activity", + "parents": [ + "consumer_groups", + "partitions", + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32][member_id:u32]" + }, + "consumer_groups": { + "code": 4, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32]" + }, + "partitions": { + "code": 3, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "personal_access_tokens": { + "code": 7, + "class": "resource", + "parents": [ + "users" + ], + "page_token_key": "[user_id:u32][name_len:u8][name]" + }, + "sessions": { + "code": 8, + "class": "activity", + "parents": [ + "users" + ], + "page_token_key": "[epoch:u64]" + }, + "streams": { + "code": 1, + "class": "resource", + "parents": [], + "page_token_key": "[stream_id:u32]" + }, + "topics": { + "code": 2, + "class": "resource", + "parents": [ + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "users": { + "code": 6, + "class": "resource", + "parents": [], + "page_token_key": "[user_id:u32]" + } + }, + "kinds": { + "consumer_group_created": { + "code": 30, + "group": "consumer_groups" + }, + "consumer_group_deleted": { + "code": 31, + "group": "consumer_groups" + }, + "consumer_group_member_joined": { + "code": 40, + "group": "consumer_group_members" + }, + "consumer_group_member_left": { + "code": 41, + "group": "consumer_group_members" + }, + "consumer_group_partition_handed_over": { + "code": 42, + "group": "consumer_group_members" + }, + "partitions_changed": { + "code": 20, + "group": "partitions" + }, + "personal_access_token_created": { + "code": 60, + "group": "personal_access_tokens" + }, + "personal_access_token_deleted": { + "code": 61, + "group": "personal_access_tokens" + }, + "session_closed": { + "code": 71, + "group": "sessions" + }, + "session_registered": { + "code": 70, + "group": "sessions" + }, + "stream_created": { + "code": 1, + "group": "streams" + }, + "stream_deleted": { + "code": 3, + "group": "streams" + }, + "stream_updated": { + "code": 2, + "group": "streams" + }, + "topic_created": { + "code": 10, + "group": "topics" + }, + "topic_deleted": { + "code": 12, + "group": "topics" + }, + "topic_updated": { + "code": 11, + "group": "topics" + }, + "user_created": { + "code": 50, + "group": "users" + }, + "user_deleted": { + "code": 52, + "group": "users" + }, + "user_password_changed": { + "code": 53, + "group": "users" + }, + "user_permissions_updated": { + "code": 54, + "group": "users" + }, + "user_updated": { + "code": 51, + "group": "users" + } + }, + "statuses": { + "behind": 1, + "expired": 2, + "ok": 0, + "reset": 3 + }, + "phases": { + "changes": 0, + "current": 1 + }, + "start_modes": { + "after": 2, + "current": 3, + "earliest": 0, + "latest": 1 + }, + "member_left_reasons": { + "left": 0, + "session_ended": 1 + }, + "token_deleted_reasons": { + "deleted": 0, + "expired": 1 + }, + "session_closed_reasons": { + "expired": 1, + "logout": 0, + "revoked": 2 + }, + "permissions_updated_reasons": { + "resource_deleted": 1, + "updated": 0 + } + }, + "records": [ + { + "case": "stream_created", + "group": 1, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "record_hex": "191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "stream_updated_unreadable_options", + "group": 1, + "kind": 2, + "body_hex": "03000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "record_hex": "1a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "expected": { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + } + }, + { + "case": "stream_deleted_deferred", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "record_hex": "1b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "expected": { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + }, + { + "case": "topic_created", + "group": 2, + "kind": 10, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "record_hex": "1c1000000000000000000000683d9e8b03f7c1a500000000000000001c00be01094106000104000000020a78000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "op": 4124, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d68", + "timestamp": 1760356800004124, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "partitions_count": 8, + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "topic_updated", + "group": 2, + "kind": 11, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "record_hex": "1d1000000000000000000000693d9e8b03f7c1a500000000000000001d00be01094106000104000000020b3d000000030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "expected": { + "op": 4125, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d69", + "timestamp": 1760356800004125, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_updated", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics-v2", + "partitions_count": 8, + "options": {} + } + }, + { + "case": "topic_deleted_immediate", + "group": 2, + "kind": 12, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "record_hex": "1e10000000000000000000006a3d9e8b03f7c1a500000000000000001e00be01094106000104000000020c2e000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "expected": { + "op": 4126, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6a", + "timestamp": 1760356800004126, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics" + } + }, + { + "case": "partitions_changed_deleted_tail", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "record_hex": "1f10000000000000000000006b3d9e8b03f7c1a500000000000000001f00be01094106000104000000031435000000030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "expected": { + "op": 4127, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6b", + "timestamp": 1760356800004127, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 5, + "removed_partition_ids": [ + 5, + 6, + 7 + ], + "requested_op": 4200 + } + }, + { + "case": "partitions_changed_created", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000a0000000000000000", + "record_hex": "2010000000000000000000006c3d9e8b03f7c1a500000000000000002000be01094106000104000000031421000000030000001910000000000000000000001d100000000000000a0000000000000000", + "expected": { + "op": 4128, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6c", + "timestamp": 1760356800004128, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 10, + "removed_partition_ids": [] + } + }, + { + "case": "consumer_group_created", + "group": 4, + "kind": 30, + "body_hex": "030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "record_hex": "2110000000000000000000006d3d9e8b03f7c1a500000000000000002100be01094106000104000000041e28000000030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "expected": { + "op": 4129, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6d", + "timestamp": 1760356800004129, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 1 + } + }, + { + "case": "consumer_group_deleted", + "group": 4, + "kind": 31, + "body_hex": "030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "record_hex": "2210000000000000000000006e3d9e8b03f7c1a500000000000000002200be01094106000104000000041f31000000030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "expected": { + "op": 4130, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6e", + "timestamp": 1760356800004130, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 0, + "requested_op": 4300 + } + }, + { + "case": "consumer_group_member_joined", + "group": 5, + "kind": 40, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "record_hex": "2310000000000000000000006f3d9e8b03f7c1a500000000000000002300be01094106000109000000052838000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "expected": { + "op": 4131, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6f", + "timestamp": 1760356800004131, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_joined", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 2, + "partition_ids": [] + } + }, + { + "case": "consumer_group_member_left_session_ended", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "record_hex": "241000000000000000000000703d9e8b03f7c1a500000000000000002400be01094106000109000000052935000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "expected": { + "op": 4132, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d70", + "timestamp": 1760356800004132, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_left", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 1, + "reason": "session_ended" + } + }, + { + "case": "consumer_group_partition_handed_over", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d1000000000000000000000020000000101000000", + "record_hex": "251000000000000000000000713d9e8b03f7c1a500000000000000002500be010941060000052a25000000030000001910000000000000000000001d1000000000000000000000020000000101000000", + "expected": { + "op": 4133, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d71", + "timestamp": 1760356800004133, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 2, + "member_id": 1 + } + }, + { + "case": "consumer_group_partition_released", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d10000000000000000000000300000000", + "record_hex": "261000000000000000000000723d9e8b03f7c1a500000000000000002600be010941060000052a21000000030000001910000000000000000000001d10000000000000000000000300000000", + "expected": { + "op": 4134, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d72", + "timestamp": 1760356800004134, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 3 + } + }, + { + "case": "user_created_with_permissions", + "group": 6, + "kind": 50, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "record_hex": "271000000000000000000000733d9e8b03f7c1a500000000000000002700be010941060001000000000632480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "op": 4135, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d73", + "timestamp": 1760356800004135, + "actor_user_id": 0, + "group": "users", + "kind": "user_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "user_updated_inactive", + "group": 6, + "kind": 51, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "record_hex": "281000000000000000000000743d9e8b03f7c1a500000000000000002800be010941060001000000000633280000000700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "expected": { + "op": 4136, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d74", + "timestamp": 1760356800004136, + "actor_user_id": 0, + "group": "users", + "kind": "user_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "inactive", + "permissions": null, + "options": {} + } + }, + { + "case": "user_deleted", + "group": 6, + "kind": 52, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "291000000000000000000000753d9e8b03f7c1a500000000000000002900be0109410600010000000006341d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4137, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d75", + "timestamp": 1760356800004137, + "actor_user_id": 0, + "group": "users", + "kind": "user_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_password_changed", + "group": 6, + "kind": 53, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "2a1000000000000000000000763d9e8b03f7c1a500000000000000002a00be0109410600010700000006351d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4138, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d76", + "timestamp": 1760356800004138, + "actor_user_id": 7, + "group": "users", + "kind": "user_password_changed", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_permissions_pruned", + "group": 6, + "kind": 54, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "record_hex": "2b1000000000000000000000773d9e8b03f7c1a500000000000000002b00be0109410600010400000006362a0000000700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "expected": { + "op": 4139, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d77", + "timestamp": 1760356800004139, + "actor_user_id": 4, + "group": "users", + "kind": "user_permissions_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": false, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": null + }, + "reason": "resource_deleted" + } + }, + { + "case": "personal_access_token_created", + "group": 7, + "kind": 60, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "record_hex": "2c1000000000000000000000783d9e8b03f7c1a500000000000000002c00be01094106000107000000073c1d0000000700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "expected": { + "op": 4140, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d78", + "timestamp": 1760356800004140, + "actor_user_id": 7, + "group": "personal_access_tokens", + "kind": "personal_access_token_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "expiry": 1760400000000000 + } + }, + { + "case": "personal_access_token_deleted_expired", + "group": 7, + "kind": 61, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e01", + "record_hex": "2d1000000000000000000000793d9e8b03f7c1a500000000000000002d00be010941060000073d160000000700000000b832c8fb4006000863692d746f6b656e01", + "expected": { + "op": 4141, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d79", + "timestamp": 1760356800004141, + "group": "personal_access_tokens", + "kind": "personal_access_token_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "reason": "expired" + } + }, + { + "case": "session_registered", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "record_hex": "2e10000000000000000000007a3d9e8b03f7c1a500000000000000002e00be010941060001070000000846240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "op": 4142, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7a", + "timestamp": 1760356800004142, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_registered", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + }, + { + "case": "session_closed_expired", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "record_hex": "2f10000000000000000000007b3d9e8b03f7c1a500000000000000002f00be0109410600000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "expected": { + "op": 4143, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7b", + "timestamp": 1760356800004143, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "expired" + } + }, + { + "case": "session_closed_revoked", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "record_hex": "3010000000000000000000007c3d9e8b03f7c1a500000000000000003000be010941060001070000000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "expected": { + "op": 4144, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7c", + "timestamp": 1760356800004144, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "revoked" + } + } + ], + "items": [ + { + "case": "stream", + "group": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "item_hex": "01b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "topic", + "group": 2, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "item_hex": "0274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "partitions", + "group": 3, + "body_hex": "030000001910000000000000000000001d1000000000000008000000", + "item_hex": "031c000000030000001910000000000000000000001d1000000000000008000000", + "expected": { + "group": "partitions", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 8 + } + }, + { + "case": "consumer_group", + "group": 4, + "body_hex": "030000001910000000000000000000001d10000000000000000000000772656164657273", + "item_hex": "0424000000030000001910000000000000000000001d10000000000000000000000772656164657273", + "expected": { + "group": "consumer_groups", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers" + } + }, + { + "case": "consumer_group_member", + "group": 5, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "item_hex": "0540000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "expected": { + "group": "consumer_group_members", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "partition_ids": [ + 0, + 2, + 4 + ] + } + }, + { + "case": "user", + "group": 6, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "item_hex": "06480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "group": "users", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "personal_access_token_without_expiry", + "group": 7, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "item_hex": "071d0000000700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "expected": { + "group": "personal_access_tokens", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token" + } + }, + { + "case": "session", + "group": 8, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "item_hex": "08240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "group": "sessions", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + } + ], + "cursors": [ + { + "case": "zero", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d000000000000000000000000000000000000000000000000000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 0, + "checksum": "00000000000000000000000000000000", + "ordinal": 0 + }, + { + "case": "first_record_of_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAABgQAAAAAAAApcH3A4ueLUwAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000001810000000000000a5c1f7038b9e2d4c000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4120, + "checksum": "00000000000000004c2d9e8b03f7c1a5", + "ordinal": 0 + }, + { + "case": "inside_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f2000000000000000001000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4187, + "checksum": "0000000000000000f2a4c6e8b0d1f359", + "ordinal": 1 + }, + { + "case": "every_field_at_maximum", + "text": "mc1___________________________________________________________8", + "hex": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + "log_id": "ffffffffffffffffffffffffffffffff", + "op": 18446744073709551615, + "checksum": "ffffffffffffffffffffffffffffffff", + "ordinal": 4294967295 + } + ], + "page_tokens": [ + { + "case": "streams", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000000101040003000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 1, + "group": "streams", + "last_key_hex": "03000000" + }, + { + "case": "topics", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 3, + "group": "topics", + "last_key_hex": "0300000000000000" + }, + { + "case": "partitions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8DCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff0308000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "partitions", + "last_key_hex": "0300000000000000" + }, + { + "case": "consumer_groups", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8EDAADAAAAAAAAAAIAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff040c00030000000000000002000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_groups", + "last_key_hex": "030000000000000002000000" + }, + { + "case": "consumer_group_members", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8FEAADAAAAAAAAAAIAAAABAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff05100003000000000000000200000001000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_group_members", + "last_key_hex": "03000000000000000200000001000000" + }, + { + "case": "users", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAACAGBAAHAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000002006040007000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 32, + "group": "users", + "last_key_hex": "07000000" + }, + { + "case": "personal_access_tokens", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAGAHDQAHAAAACGNpLXRva2Vu", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e1000000000000060070d00070000000863692d746f6b656e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 96, + "group": "personal_access_tokens", + "last_key_hex": "070000000863692d746f6b656e" + }, + { + "case": "sessions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAKAICACUEQAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000a00808009411000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 160, + "group": "sessions", + "last_key_hex": "9411000000000000" + }, + { + "case": "longest_token_name", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8HBAH______25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubg", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff070401ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "personal_access_tokens", + "last_key_hex": "ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e" + } + ], + "requests": [ + { + "case": "earliest_every_group", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000", + "http_query": "from=earliest&limit=100" + }, + { + "case": "earliest_many_duplicate_groups", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "users" + ], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000640000000201060000", + "http_query": "from=earliest&groups=streams,users&limit=100" + }, + { + "case": "earliest_since_users_and_tokens", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "users", + "personal_access_tokens" + ], + "since": 1760353200000000, + "until": null, + "limit": 100 + }, + "request_hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001004c2a2b0841060000640000000206070000", + "http_query": "from=earliest&groups=users,personal_access_tokens&since=1760353200000000&limit=100" + }, + { + "case": "latest_streams", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [ + "streams" + ], + "since": null, + "until": null, + "limit": 1 + }, + "request_hex": "01000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000001010000", + "http_query": "from=latest&groups=streams&limit=1" + }, + { + "case": "after_cursor_with_time_range", + "input": { + "start": "after", + "cursor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "page_token": null, + "groups": [ + "topics", + "streams" + ], + "since": 1760356800000000, + "until": 1760360400000000, + "limit": 500 + }, + "request_hex": "02d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000000100f0bd010941060001009451d809410600f40100000202010000", + "http_query": "after=mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA&groups=streams,topics&since=1760356800000000&until=1760360400000000&limit=500" + }, + { + "case": "current_first_page", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201020000", + "http_query": "from=current&groups=streams,topics&limit=1000" + }, + { + "case": "current_continuation", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201024000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "http_query": "from=current¤t_after=cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA&groups=streams,topics&limit=1000" + } + ], + "responses": [ + { + "case": "changes_page", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001b10000000000000673d9e8b03f7c1a50000000000000000000000000103000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465641a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d7632011b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABsQAAAAAAAAZz2eiwP3waUAAAAAAAAAAAAAAAA", + "has_more": true, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + }, + { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "unknown_kind_and_group", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001910000000000000653d9e8b03f7c1a50000000000000000010000000002000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be01094106000104000000010904000000deadbeef191000000000000001000000653d9e8b03f7c1a500000000000000001900be01094106000104000000c80104000000deadbeef000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABkQAAAAAAAAZT2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": false, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "1", + "kind": "9", + "body": "3q2+7w==" + }, + { + "op": 4121, + "ordinal": 1, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "200", + "kind": "1", + "body": "3q2+7w==" + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "current_page", + "response_hex": "000101d7931c6e2a8f4b0d0000000000000000010000000000000000000000000000005e1000000000000001d7931c6e2a8f4b0d00000000000000005b10000000000000a73d9e8b03f7c1a500000000000000000100000001000000000200000001b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465640274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c0800000000000040000000004000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "expected": { + "status": "ok", + "phase": "current", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 1, + "retained_since": 0, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAApz2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": true, + "records": [], + "items": [ + { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + ], + "current_page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA" + } + }, + { + "case": "behind_before_op_1", + "response_hex": "010000000000000000000000000000000000000000000000000000000000000000000000000000", + "expected": { + "status": "behind", + "phase": "changes", + "log_id": null, + "oldest_op": 0, + "retained_since": 0, + "head_op": 0, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "expired", + "response_hex": "020001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "expired", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "reset", + "response_hex": "030001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "reset", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + } + ], + "large_values": { + "record_hex": "0300000000002000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff01ffffffff084624000000ffffffff0100000000002000feffffffffffffffffffffffffffffff0200000000002000", + "op": "9007199254740995", + "ordinal": 4294967295, + "timestamp": "18446744073709551615", + "checksum": "ffffffffffffffffffffffffffffffff", + "user_created_at": "9007199254740993", + "session": "9007199254740994", + "client_id": "fffffffffffffffffffffffffffffffe" + }, + "refused": { + "cursors": [ + { + "case": "wrong_prefix", + "text": "mc2_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "missing_prefix", + "text": "15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "too_long", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAAA" + }, + { + "case": "too_short", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA" + }, + { + "case": "padded", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA=" + }, + { + "case": "standard_alphabet_character", + "text": "mc1_+5McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "op_zero_with_checksum", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAA" + } + ], + "page_tokens": [ + { + "case": "wrong_prefix", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA" + }, + { + "case": "trailing_byte", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAAAA" + }, + { + "case": "wrong_key_width", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBCAADAAAAAAAAAA" + }, + { + "case": "group_outside_mask", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAIBBAADAAAA" + }, + { + "case": "head_below_anchor", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAABAAAAAAAAAAEBBAADAAAA" + }, + { + "case": "empty", + "text": "cs1_" + } + ], + "record_bodies": [ + { + "case": "unknown_member_left_reason", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000002" + }, + { + "case": "presence_byte_two", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279022c10000000000000" + }, + { + "case": "unsorted_removed_partition_ids", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000090000000600000007000000016810000000000000" + }, + { + "case": "known_kind_under_wrong_group", + "group": 2, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564" + }, + { + "case": "trailing_byte", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000000" + }, + { + "case": "truncated", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a09080706050403020194110000000000" + } + ], + "requests": [ + { + "case": "cursor_bytes_without_after", + "hex": "000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "trailing_byte", + "hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006400000000000000" + }, + { + "case": "unknown_start_mode", + "hex": "040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "non_canonical_cursor", + "hex": "02d7931c6e2a8f4b0d000000000000000000000000000000000100000000000000000000000000000000000000000064000000000000" + } + ], + "responses": [ + { + "case": "items_in_changes_phase", + "hex": "000001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + }, + { + "case": "unknown_status", + "hex": "040001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + } + ], + "queries": [ + { + "case": "limit_zero", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 0 + } + }, + { + "case": "limit_above_maximum", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 1001 + } + }, + { + "case": "since_above_until", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 2, + "until": 1, + "limit": 100 + } + }, + { + "case": "time_bound_with_latest", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 1, + "until": null, + "limit": 100 + } + }, + { + "case": "time_bound_with_current", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": 1, + "limit": 100 + } + }, + { + "case": "current_missing_parent_group", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + }, + { + "case": "page_token_for_other_groups", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + } + ], + "http_queries": [ + { + "case": "no_start", + "query": "groups=streams" + }, + { + "case": "from_and_after", + "query": "from=earliest&after=mc1_x" + }, + { + "case": "unknown_from", + "query": "from=newest" + }, + { + "case": "duplicate_key", + "query": "from=earliest&from=latest" + }, + { + "case": "unknown_key", + "query": "from=earliest&verbose=1" + }, + { + "case": "empty_group_element", + "query": "from=earliest&groups=streams,,topics" + }, + { + "case": "trailing_group_comma", + "query": "from=earliest&groups=streams," + }, + { + "case": "unknown_group", + "query": "from=earliest&groups=everything" + }, + { + "case": "negative_since", + "query": "from=earliest&since=-1" + }, + { + "case": "leading_zero_until", + "query": "from=earliest&until=01" + }, + { + "case": "non_numeric_limit", + "query": "from=earliest&limit=ten" + }, + { + "case": "token_without_current", + "query": "from=latest¤t_after=cs1_AAAA" + } + ] + } +}
diff --git a/foreign/node/README.md b/foreign/node/README.md index b6f5b4c..e7aa6d4 100644 --- a/foreign/node/README.md +++ b/foreign/node/README.md
@@ -22,7 +22,7 @@ npm i --save apache-iggy@edge ``` -Use matching SDK and server builds from the coordinated VSR release. Older builds are incompatible. This release requires fresh server storage (`IGGY-RESOURCE-OPTIONS-1`). Existing incompatible data is refused unchanged. For a local server with the example credentials, follow the [example setup](../../examples/node/README.md#running-examples). +Use matching SDK and server builds from the coordinated VSR release. Older builds are incompatible. This release requires fresh server storage (`IGGY-METADATA-CHANGE-LOG-1`). Existing incompatible data is refused unchanged. For a local server with the example credentials, follow the [example setup](../../examples/node/README.md#running-examples). ## basic usage @@ -362,6 +362,69 @@ Reads return every setting in `options` (keys the client set) and `derivedOptions` (keys the server filled). `u64` values stay `bigint`, so values above 2^53 keep their precision. `system.describeOptions({ scope: OptionsScope.Stream })` lists the keys a server accepts per scope, with `updatable`, `resettable`, `secret`, `readOnly` and `critical`. +### metadata changes + +`system.getMetadataChanges` reads the node's change log: one record per public result of a committed metadata operation, such as a stream created, a topic renamed, a user's permissions changed or a session closed. A mirror keeps a cursor, asks for the records after it, and stores the returned `next` cursor. The call needs root or the global `manage_servers` permission. + +```ts +import { setTimeout as sleep } from "node:timers/promises"; +import { + Client, MetadataChangeStart, MetadataChangeStatus, + type MetadataChangeGroup, type MetadataCurrentPageToken, +} from "apache-iggy"; + +const follow = async (client: Client, groups: MetadataChangeGroup[]) => { + let start: MetadataChangeStart = MetadataChangeStart.Earliest; + for (;;) { + const page = await client.system.getMetadataChanges({ start, groups }); + switch (page.status) { + case MetadataChangeStatus.Ok: + for (const record of page.records) + console.log(record.op, record.kind, record.actorUserId, record.body); + if (page.next) start = MetadataChangeStart.After(page.next); + if (!page.hasMore) await sleep(1000); + break; + case MetadataChangeStatus.Behind: + await sleep(100); + break; + default: + start = await rebuild(client, groups); + } + } +}; + +/** Pages the current state, then returns the position to follow from. */ +const rebuild = async (client: Client, groups: MetadataChangeGroup[]) => { + let pageToken: MetadataCurrentPageToken | undefined; + for (;;) { + const page = await client.system.getMetadataChanges({ + start: MetadataChangeStart.Current(pageToken), groups, + }); + if (page.status === MetadataChangeStatus.Behind) { + await sleep(100); + continue; + } + if (page.status !== MetadataChangeStatus.Ok) { + pageToken = undefined; + continue; + } + for (const item of page.items) + console.log(item.group, item.body); + if (page.currentPageToken) pageToken = page.currentPageToken; + else if (page.next) return MetadataChangeStart.After(page.next); + else pageToken = undefined; + } +}; +``` + +`expired` means the log no longer holds the history after the cursor, and `reset` means the cursor belongs to another history, for example after the data directory was rebuilt. Either way the mirror rebuilds from current state pages and then follows the changes after the anchor they return. + +Current pages need the parent groups of every selected group, for example `topics` with `streams`. Keep a rebuilt state private until a change page answers `ok` with `hasMore` false and a `headOp` at or above the last current page's. + +Records narrow by `kind` and items by `group`, so `record.kind === "stream_created"` types `record.body` as a stream. A group or kind newer than this build arrives as `"unknown"` with the raw body and the wire codes in `groupCode` and `kindCode`, while a record of a newer kind keeps its known `group`, so a mirror that cannot apply it can stop without advancing its cursor. Ops, timestamps and 128-bit ids are `bigint`. + +A query that breaks the request rules, such as a limit outside 1 to 1000 or a time bound with a latest or current start, is refused with error 3 before anything is sent. `MetadataChangeCursor` and `MetadataCurrentPageToken` print and parse as the `mc1_` and `cs1_` text forms the HTTP API and the CLI use, and `JSON.stringify` writes them as that text. + ## use sources Cluster polling uses command 15 (`BindSession`), route discovery 105 and poll @@ -398,7 +461,7 @@ e2e test expect an iggy-server at tcp://127.0.0.1:8090 -The resource options tests also expect the server to run with encryption enabled and a config key (`IGGY_ENCRYPTION_ENABLED=true` and `IGGY_ENCRYPTION_KEY` set to the base64 of 32 bytes), so a stream created without a key reports a server-generated one. +The resource options tests also expect the server to run with encryption enabled and a config key (`IGGY_ENCRYPTION_ENABLED=true` and `IGGY_ENCRYPTION_KEY` set to the base64 of 32 bytes), so a stream created without a key reports a server-generated one. The keyed metadata changes case registered from the same file needs the config key too. ```bash npm run test:e2e
diff --git a/foreign/node/package-lock.json b/foreign/node/package-lock.json index f48f3c7..032e38c 100644 --- a/foreign/node/package-lock.json +++ b/foreign/node/package-lock.json
@@ -1,12 +1,12 @@ { "name": "apache-iggy", - "version": "0.10.1-edge.2", + "version": "0.10.1-edge.3", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "apache-iggy", - "version": "0.10.1-edge.2", + "version": "0.10.1-edge.3", "license": "Apache-2.0", "dependencies": { "@node-rs/xxhash": "1.7.8",
diff --git a/foreign/node/package.json b/foreign/node/package.json index ba9e2af..97de345 100644 --- a/foreign/node/package.json +++ b/foreign/node/package.json
@@ -1,7 +1,7 @@ { "name": "apache-iggy", "type": "module", - "version": "0.10.1-edge.2", + "version": "0.10.1-edge.3", "description": "Official Apache Iggy NodeJS SDK", "keywords": [ "iggy",
diff --git a/foreign/node/src/bdd/README.md b/foreign/node/src/bdd/README.md index 7f723ec..1e1e24e 100644 --- a/foreign/node/src/bdd/README.md +++ b/foreign/node/src/bdd/README.md
@@ -31,6 +31,8 @@ The stream options feature also needs the server to run with encryption enabled and a config key (`IGGY_ENCRYPTION_ENABLED=true`, `IGGY_ENCRYPTION_KEY`), as `scripts/run-bdd-tests.sh` starts it. +The metadata changes feature needs the same key and `IGGY_METADATA_CHANGE_LOG_RESOURCE_CAPACITY=64`, so its expiry scenario can push a held cursor out of the change log. + from [/foreign/node](../../) run ```bash
diff --git a/foreign/node/src/bdd/metadata-changes.ts b/foreign/node/src/bdd/metadata-changes.ts new file mode 100644 index 0000000..5a0b6b3 --- /dev/null +++ b/foreign/node/src/bdd/metadata-changes.ts
@@ -0,0 +1,252 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import assert from 'node:assert/strict'; +import { setTimeout as sleep } from 'node:timers/promises'; +import { Given, When, Then } from '@cucumber/cucumber'; +import { ResponseError } from '../wire/error.utils.js'; +import { + MAX_METADATA_CHANGES_LIMIT, MetadataChangeStart, MetadataChangeStatus, UserStatus, + encryptionKeyFingerprint, + type MetadataChangeGroup, type MetadataChangeRecord, type MetadataCurrentPageToken, + type MetadataStreamChange, type ParsedOptions +} from '../wire/index.js'; +import type { TestWorld } from './world.js'; + +const ROOT_USER_ID = 0; +const UNAUTHORIZED = 41; +// A two phase delete commits its record after the call returns. +const FOLLOW_ATTEMPTS = 100; +const FOLLOW_INTERVAL_MS = 100; +const HIDDEN_OPTIONS = ['encryption_key', 'encryption_key_wrapped', 'critical_keys']; + +const readChanges = (world: TestWorld, start: MetadataChangeStart) => + world.client.system.getMetadataChanges({ + start, groups: world.changeGroups ?? [], limit: MAX_METADATA_CHANGES_LIMIT + }); + +const groupList = (groups: string) => + groups.split(',').map((group) => group.trim() as MetadataChangeGroup); + +/** The stream a stream, topic or partitions record names, null for any other record. */ +const changeStreamId = (record: MetadataChangeRecord): number | null => { + switch (record.kind) { + case 'stream_created': + case 'stream_updated': + case 'stream_deleted': + case 'topic_created': + case 'topic_updated': + case 'topic_deleted': + case 'partitions_changed': + return record.body.streamId; + default: + return null; + } +}; + +const followed = (world: TestWorld, kind: string): MetadataChangeRecord => { + const record = world.followedChanges?.find((candidate) => candidate.kind === kind); + assert.ok(record, `a followed ${kind} change`); + return record; +}; + +const followedStreamOptions = (world: TestWorld, kind: string) => { + const record = followed(world, kind); + assert.ok(record.kind === 'stream_created' || record.kind === 'stream_updated', `${kind} is a stream change`); + const { options } = record.body as MetadataStreamChange; + assert.ok(!options.unreadable, 'stream options stay readable'); + return options; +}; + +Given( + 'I hold the latest metadata change cursor for the groups {string}', + async function (this: TestWorld, groups: string) { + this.changeGroups = groupList(groups); + const changes = await readChanges(this, MetadataChangeStart.Latest); + assert.equal(changes.status, MetadataChangeStatus.Ok); + assert.ok(changes.next, 'latest returns a cursor'); + this.changeCursor = changes.next; + } +); + +When('I delete the stream', async function (this: TestWorld) { + assert.ok(await this.client.stream.delete({ streamId: this.stream.id })); +}); + +When( + 'I create and delete {int} streams named {string}', + async function (this: TestWorld, count: number, prefix: string) { + for (let index = 0; index < count; index++) { + const stream = await this.client.stream.create({ name: `${prefix}-${index}` }); + assert.ok(await this.client.stream.delete({ streamId: stream.id })); + } + } +); + +Then( + 'following the metadata changes should return the kinds {string} for the stream', + async function (this: TestWorld, kinds: string) { + const expected = kinds.split(',').map((kind) => kind.trim()); + const records: MetadataChangeRecord[] = []; + let cursor = this.changeCursor; + assert.ok(cursor, 'a held cursor'); + let mine: MetadataChangeRecord[] = []; + for (let attempt = 0; attempt < FOLLOW_ATTEMPTS; attempt++) { + const changes = await readChanges(this, MetadataChangeStart.After(cursor)); + assert.equal(changes.status, MetadataChangeStatus.Ok); + assert.ok(changes.next, 'a continuation'); + cursor = changes.next; + records.push(...changes.records); + if (changes.hasMore) + continue; + mine = records.filter((record) => changeStreamId(record) === this.stream.id); + if (mine.length >= expected.length) + break; + await sleep(FOLLOW_INTERVAL_MS); + } + for (const record of records) + assert.ok(this.changeGroups?.includes(record.group as MetadataChangeGroup), `${record.group} was requested`); + assert.deepEqual(mine.map((record) => record.kind), expected); + this.followedChanges = mine; + } +); + +Then('each followed change should name the root user as its actor', function (this: TestWorld) { + for (const record of this.followedChanges ?? []) + assert.equal(record.actorUserId, ROOT_USER_ID, `op ${record.op}`); +}); + +Then( + 'the followed {string} change should carry a requested op before its own op', + function (this: TestWorld, kind: string) { + const record = followed(this, kind); + assert.ok(record.kind === 'stream_deleted', `${kind} is a stream deletion`); + assert.ok(record.body.requestedOp !== null, 'a requested op'); + assert.ok(record.body.requestedOp < record.op, `requested op ${record.body.requestedOp} before ${record.op}`); + } +); + +Then( + 'the followed {string} change should report encryption key source {string}', + function (this: TestWorld, kind: string, source: string) { + const options = followedStreamOptions(this, kind); + assert.equal(options.derivedOptions.encryption_key_source ?? options.options.encryption_key_source, source); + } +); + +Then( + 'the followed {string} change should carry the stream key fingerprint', + function (this: TestWorld, kind: string) { + const options = followedStreamOptions(this, kind); + assert.ok(this.streamKey, 'the stream was created with a key'); + assert.equal( + options.derivedOptions.encryption_key_fingerprint ?? options.options.encryption_key_fingerprint, + encryptionKeyFingerprint(this.streamKey) + ); + } +); + +Then('the followed changes should expose no encryption key material', function (this: TestWorld) { + for (const record of this.followedChanges ?? []) { + if (record.kind !== 'stream_created' && record.kind !== 'stream_updated') + continue; + const { options } = record.body; + assert.ok(!options.unreadable, 'stream options stay readable'); + for (const block of [options.options, options.derivedOptions] as ParsedOptions[]) { + for (const hidden of HIDDEN_OPTIONS) + assert.equal(hidden in block, false, `${hidden} must never be returned`); + for (const value of Object.values(block)) + assert.ok(!(Buffer.isBuffer(value) && this.streamKey?.equals(value)), 'a value holds the key'); + } + } +}); + +Then( + 'following the metadata changes should answer {string}', + async function (this: TestWorld, status: string) { + assert.ok(this.changeCursor, 'a held cursor'); + const changes = await readChanges(this, MetadataChangeStart.After(this.changeCursor)); + assert.equal(changes.status, status); + if (changes.status !== MetadataChangeStatus.Ok) { + assert.deepEqual(changes.records, []); + assert.equal(changes.next, null); + } + } +); + +When( + 'I read the current state for the groups {string}', + async function (this: TestWorld, groups: string) { + this.changeGroups = groupList(groups); + this.currentItems = []; + let pageToken: MetadataCurrentPageToken | undefined; + for (;;) { + const changes = await readChanges(this, MetadataChangeStart.Current(pageToken)); + assert.equal(changes.status, MetadataChangeStatus.Ok); + this.currentItems.push(...changes.items); + if (changes.currentPageToken === null) { + assert.ok(changes.next, 'the last page returns the anchor'); + this.changeCursor = changes.next; + return; + } + pageToken = changes.currentPageToken; + } + } +); + +Then('the current state should list the same streams as the stream list', async function (this: TestWorld) { + const listed = new Map((await this.client.stream.list()).map((stream) => [stream.id, stream.name])); + const projected = new Map<number, string>(); + for (const item of this.currentItems ?? []) + if (item.group === 'streams') + projected.set(item.body.streamId, item.body.name); + assert.deepEqual(projected, listed); +}); + +Given( + 'a user named {string} with password {string} and no server permissions exists', + async function (this: TestWorld, username: string, password: string) { + const user = await this.client.user.create({ + username, password, status: UserStatus.Active, + permissions: { + global: { + ManageServers: false, ReadServers: true, ManageUsers: false, ReadUsers: false, + ManageStreams: false, ReadStreams: false, ManageTopics: false, ReadTopics: false, + PollMessages: false, SendMessages: false + }, + streams: [] + } + }); + this.userId = user.id; + } +); + +When( + 'I authenticate as {string} with password {string}', + async function (this: TestWorld, username: string, password: string) { + assert.ok(await this.client.session.logout()); + await this.client.session.login({ username, password }); + } +); + +Then('reading the metadata changes should be refused as unauthorized', async function (this: TestWorld) { + this.changeGroups = []; + await assert.rejects( + readChanges(this, MetadataChangeStart.Earliest), + (error: unknown) => error instanceof ResponseError && error.errorCode === UNAUTHORIZED && error.fromServer + ); +});
diff --git a/foreign/node/src/bdd/world.ts b/foreign/node/src/bdd/world.ts index 4d26e7d..54c4b61 100644 --- a/foreign/node/src/bdd/world.ts +++ b/foreign/node/src/bdd/world.ts
@@ -17,7 +17,10 @@ import type { Stream } from '../wire/stream/stream.utils.js'; import type { Topic } from '../wire/topic/topic.utils.js'; -import type { CreateMessage, Message, OptionSpec } from 'src/wire/index.js'; +import type { + CreateMessage, Message, MetadataChangeCursor, MetadataChangeGroup, MetadataChangeRecord, + MetadataCurrentItem, OptionSpec +} from 'src/wire/index.js'; import type { Client } from '../client/index.js'; export interface TestWorld { @@ -33,5 +36,13 @@ updateError?: unknown, optionSpecs?: OptionSpec[], userId?: number, - callError?: Error + callError?: Error, + /** Groups the metadata change steps read */ + changeGroups?: MetadataChangeGroup[], + /** Cursor the metadata change steps follow from */ + changeCursor?: MetadataChangeCursor, + /** Records of the last stream that the follow step returned */ + followedChanges?: MetadataChangeRecord[], + /** Items of the last current state scan */ + currentItems?: MetadataCurrentItem[] };
diff --git a/foreign/node/src/client/client.socket.test.ts b/foreign/node/src/client/client.socket.test.ts index c4f16c6..37049a0 100644 --- a/foreign/node/src/client/client.socket.test.ts +++ b/foreign/node/src/client/client.socket.test.ts
@@ -32,6 +32,7 @@ import { UPDATE_TOPIC } from '../wire/topic/update-topic.command.js'; import { UPDATE_STREAM } from '../wire/stream/update-stream.command.js'; import { OptionsScope } from '../wire/system/describe-options.command.js'; +import { MetadataChangeStart, getMetadataChanges } from '../wire/system/get-metadata-changes.command.js'; import { CHANGE_PASSWORD } from '../wire/user/change-password.command.js'; import { POLL_MESSAGES, pollMessages } from '../wire/message/poll-messages.command.js'; import { SEND_MESSAGES, sendMessages } from '../wire/message/send-messages.command.js'; @@ -1409,6 +1410,39 @@ } }); + it('resends a metadata changes read after an uncommitted response', async () => { + // A behind answer before op 1: status 1, phase 0, no log id, zero ops, no cursor. + const behind = Buffer.concat([Buffer.from([1, 0, 0]), Buffer.alloc(24), Buffer.alloc(2 + 4 + 4 + 2)]); + let attempts = 0; + const server = await startVsrServer((frame, socket) => { + const operation = frame.readUInt8(REQUEST_OFFSET.operation); + if (operation !== Operation.NonReplicated || + frame.readUInt32LE(REQUEST_OFFSET.reserved) !== COMMAND_CODE.GetMetadataChanges) + return singleNodeHandler(server.port)(frame, socket); + attempts += 1; + socket.write( + attempts === 1 + ? replyFrame(frame, operation, Buffer.alloc(0), 57) + : replyFrame(frame, operation, behind) + ); + }); + const client = new CommandResponseStream(vsrConfig(server.port)); + try { + const changes = await getMetadataChanges(async () => client)({ start: MetadataChangeStart.Latest }); + assert.equal(attempts, 2); + assert.equal(changes.status, 'behind'); + assert.equal(changes.next, null); + const sent = server.frames.filter( + (frame) => frame.readUInt32LE(REQUEST_OFFSET.reserved) === COMMAND_CODE.GetMetadataChanges + ); + assert.equal(sent.length, 2); + assert.deepEqual(sent[0], sent[1]); + } finally { + client.destroy(); + await server.close(); + } + }); + it('routes eviction out of band without desynchronizing replies', async () => { const server = await startVsrServer((frame, socket) => { if (frame.readUInt32LE(REQUEST_OFFSET.reserved) === 60_003) {
diff --git a/foreign/node/src/client/client.socket.ts b/foreign/node/src/client/client.socket.ts index 6f338d8..7dbf8b1 100644 --- a/foreign/node/src/client/client.socket.ts +++ b/foreign/node/src/client/client.socket.ts
@@ -88,8 +88,8 @@ ]); const READ_COMMANDS = new Set<number>([ COMMAND_CODE.Ping, COMMAND_CODE.GetStats, COMMAND_CODE.GetClusterMetadata, - COMMAND_CODE.DescribeOptions, COMMAND_CODE.GetMe, COMMAND_CODE.GetClient, - COMMAND_CODE.GetClients, COMMAND_CODE.GetUser, COMMAND_CODE.GetUsers, + COMMAND_CODE.DescribeOptions, COMMAND_CODE.GetMetadataChanges, COMMAND_CODE.GetMe, + COMMAND_CODE.GetClient, COMMAND_CODE.GetClients, COMMAND_CODE.GetUser, COMMAND_CODE.GetUsers, COMMAND_CODE.GetAccessTokens, COMMAND_CODE.GetPartitionRoute, COMMAND_CODE.GetOffset, COMMAND_CODE.GetStream, COMMAND_CODE.GetStreams, COMMAND_CODE.GetTopic, COMMAND_CODE.GetTopics, COMMAND_CODE.GetGroup, COMMAND_CODE.GetGroups, COMMAND_CODE.SyncGroup
diff --git a/foreign/node/src/e2e/metadata-changes.e2e.ts b/foreign/node/src/e2e/metadata-changes.e2e.ts new file mode 100644 index 0000000..cb3b9ef --- /dev/null +++ b/foreign/node/src/e2e/metadata-changes.e2e.ts
@@ -0,0 +1,29 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +// The keyed stream case needs a server key, so resource-options.e2e.ts and +// tls.system.e2e.ts register it, and the shipped-default lane skips both. + +import { Client } from '../client/client.js'; +import { getIggyAddress } from '../tcp.sm.utils.js'; +import { describeMetadataChanges } from './metadata-changes.suite.js'; + +const [host, port] = getIggyAddress(); + +describeMetadataChanges('e2e -> metadata changes', 'e2e-mc', (credentials) => new Client({ + transport: 'TCP', options: { host, port }, credentials +}));
diff --git a/foreign/node/src/e2e/metadata-changes.suite.ts b/foreign/node/src/e2e/metadata-changes.suite.ts new file mode 100644 index 0000000..6685b44 --- /dev/null +++ b/foreign/node/src/e2e/metadata-changes.suite.ts
@@ -0,0 +1,369 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import { randomBytes } from 'node:crypto'; +import { setTimeout as sleep } from 'node:timers/promises'; +import { after, describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import type { Client } from '../client/client.js'; +import { ResponseError } from '../wire/error.utils.js'; +import { + EncryptionKeySource, MAX_METADATA_CHANGES_LIMIT, MetadataChangeCursor, MetadataChangeKind, + MetadataChangeStart, MetadataChangeStatus, StreamEncryptionKey, UserStatus, + encryptionKeyFingerprint, + type GetMetadataChanges, type MetadataChangeGroup, type MetadataChangeRecord, + type MetadataCurrentPageToken, type ParsedOptions +} from '../wire/index.js'; + +const NEVER_RETURNED = ['encryption_key', 'encryption_key_wrapped', 'critical_keys']; +const FINGERPRINT = /^[0-9a-f]{16}$/; +const INVALID_COMMAND = 3; +const UNAUTHORIZED = 41; +// A two phase delete commits its record after the call returns. +const FOLLOW_BUDGET_MS = 15_000; +const FOLLOW_INTERVAL_MS = 100; + +type Credentials = { username: string, password: string }; +type StreamIdentity = { streamId: number, streamCreatedOp: bigint }; + +const latestCursor = async (c: Client, groups: MetadataChangeGroup[]) => { + const changes = await c.system.getMetadataChanges({ start: MetadataChangeStart.Latest, groups }); + assert.equal(changes.status, MetadataChangeStatus.Ok); + assert.deepEqual(changes.records, []); + assert.ok(changes.next); + assert.equal(changes.next.logId, changes.logId); + return changes.next; +}; + +/** The stream a stream, topic or partitions record names, null for any other record. */ +const recordStream = (record: MetadataChangeRecord): StreamIdentity | null => { + switch (record.kind) { + case 'stream_created': + case 'stream_updated': + case 'stream_deleted': + case 'topic_created': + case 'topic_updated': + case 'topic_deleted': + case 'partitions_changed': + return { streamId: record.body.streamId, streamCreatedOp: record.body.streamCreatedOp }; + default: + return null; + } +}; + +/** + * Follows the change log after `cursor` until `want` records name the stream + * the first record `introduces` accepts. Other e2e files run concurrently and + * stream ids are reused, so the stream key identifies the stream, not its id. + */ +const followStream = async ( + c: Client, cursor: MetadataChangeCursor, groups: MetadataChangeGroup[], + introduces: (record: MetadataChangeRecord) => boolean, want: number +) => { + const records: MetadataChangeRecord[] = []; + let identity: StreamIdentity | undefined; + const deadline = Date.now() + FOLLOW_BUDGET_MS; + for (;;) { + const changes = await c.system.getMetadataChanges({ + start: MetadataChangeStart.After(cursor), groups, limit: MAX_METADATA_CHANGES_LIMIT + }); + assert.equal(changes.status, MetadataChangeStatus.Ok); + assert.equal(changes.phase, 'changes'); + assert.deepEqual(changes.items, []); + assert.equal(changes.currentPageToken, null); + assert.ok(changes.next); + cursor = changes.next; + for (const record of changes.records) { + assert.ok(groups.includes(record.group as MetadataChangeGroup), `${record.group} was not requested`); + const stream = recordStream(record); + if (stream === null) + continue; + if (identity === undefined && introduces(record)) + identity = stream; + if (identity !== undefined && stream.streamId === identity.streamId && + stream.streamCreatedOp === identity.streamCreatedOp) + records.push(record); + } + if (changes.hasMore) + continue; + if (records.length >= want || Date.now() > deadline) + break; + await sleep(FOLLOW_INTERVAL_MS); + } + assert.equal(records.length, want); + return records; +}; + +/** Accepts the stream_created record of a stream by its id and name. */ +const createdStream = (streamId: number, name: string) => (record: MetadataChangeRecord) => + record.kind === 'stream_created' && record.body.streamId === streamId && record.body.name === name; + +/** Pages the streams state one item at a time, returning the names by id. */ +const currentStreams = async (c: Client) => { + const streams = new Map<number, string>(); + let pageToken: MetadataCurrentPageToken | undefined; + for (;;) { + const changes = await c.system.getMetadataChanges({ + start: MetadataChangeStart.Current(pageToken), groups: ['streams'], limit: 1 + }); + assert.equal(changes.status, MetadataChangeStatus.Ok); + assert.equal(changes.phase, 'current'); + assert.deepEqual(changes.records, []); + assert.ok(changes.items.length <= 1); + for (const item of changes.items) { + assert.equal(item.group, 'streams'); + if (item.group === 'streams') + streams.set(item.body.streamId, item.body.name); + } + if (changes.currentPageToken === null) { + assert.ok(changes.next, 'the last page returns the anchor'); + return streams; + } + assert.equal(changes.hasMore, true); + pageToken = changes.currentPageToken; + } +}; + +const isRefusedLocally = (code: number) => (error: unknown) => + error instanceof ResponseError && error.errorCode === code && !error.fromServer; + +/** + * Registers the metadata changes cases for one transport. `connect` builds a + * client of that transport signed in with the given credentials. + */ +export const describeMetadataChanges = ( + label: string, prefix: string, connect: (credentials: Credentials) => Client, + skip: string | false = false +) => describe(label, { skip }, async () => { + + const c = connect({ username: 'iggy', password: 'iggy' }); + + it('given latest cursor when creating renaming and deleting a stream should follow its records', async () => { + const cursor = await latestCursor(c, ['streams']); + const created = await c.stream.create({ name: `${prefix}-follow` }); + assert.ok(await c.stream.update({ streamId: created.id, name: `${prefix}-follow-renamed` })); + assert.ok(await c.stream.delete({ streamId: created.id })); + + const records = await followStream(c, cursor, ['streams'], createdStream(created.id, created.name), 3); + assert.deepEqual(records.map((record) => record.kind), [ + MetadataChangeKind.StreamCreated, MetadataChangeKind.StreamUpdated, MetadataChangeKind.StreamDeleted + ]); + for (const record of records) { + assert.equal(record.actorUserId, 0); + assert.equal(record.group, 'streams'); + assert.ok(record.checksum > 0n); + assert.ok(record.timestamp > 0n); + } + const [createdRecord, updatedRecord, deletedRecord] = records; + assert.ok(createdRecord.kind === 'stream_created' && updatedRecord.kind === 'stream_updated' && + deletedRecord.kind === 'stream_deleted'); + assert.equal(createdRecord.body.name, `${prefix}-follow`); + assert.equal(updatedRecord.body.name, `${prefix}-follow-renamed`); + assert.equal(deletedRecord.body.name, `${prefix}-follow-renamed`); + assert.equal(createdRecord.body.streamCreatedOp, createdRecord.op); + assert.equal(createdRecord.body.options.unreadable, false); + assert.ok(createdRecord.op < updatedRecord.op && updatedRecord.op < deletedRecord.op); + }); + + it('given topics group when creating a stream and topic should return only the topic record', async () => { + const cursor = await latestCursor(c, ['topics']); + const stream = await c.stream.create({ name: `${prefix}-filter` }); + try { + const topic = await c.topic.create({ streamId: stream.id, name: 'events', partitionCount: 2 }); + const introduces = (record: MetadataChangeRecord) => + record.kind === 'topic_created' && record.body.streamId === stream.id && record.body.topicId === topic.id; + const records = await followStream(c, cursor, ['topics'], introduces, 1); + const [record] = records; + assert.equal(record.kind, MetadataChangeKind.TopicCreated); + assert.ok(record.kind === 'topic_created'); + assert.equal(record.body.topicId, topic.id); + assert.equal(record.body.name, 'events'); + assert.equal(record.body.partitionsCount, 2); + assert.equal(record.body.topicCreatedOp, record.op); + } finally { + await c.stream.delete({ streamId: stream.id }); + } + }); + + it('given limit of one when following three streams should page one record at a time', async () => { + let cursor = await latestCursor(c, ['streams']); + const created = new Map<number, string>(); + try { + for (const index of [1, 2, 3]) { + const stream = await c.stream.create({ name: `${prefix}-page-${index}` }); + created.set(stream.id, stream.name); + } + const seen = new Map<number, string>(); + const deadline = Date.now() + FOLLOW_BUDGET_MS; + while (seen.size < created.size) { + assert.ok(Date.now() < deadline, `paged ${[...seen.values()]} of ${[...created.values()]}`); + const changes = await c.system.getMetadataChanges({ + start: MetadataChangeStart.After(cursor), groups: ['streams'], limit: 1 + }); + assert.equal(changes.status, MetadataChangeStatus.Ok); + assert.ok(changes.records.length <= 1); + assert.ok(changes.next); + for (const record of changes.records) { + if (record.kind === 'stream_created' && created.get(record.body.streamId) === record.body.name) + seen.set(record.body.streamId, record.body.name); + assert.equal(record.op, changes.next.op); + assert.equal(record.ordinal, changes.next.ordinal); + } + assert.ok(changes.next.op >= cursor.op); + cursor = changes.next; + if (!changes.hasMore && seen.size < created.size) + await sleep(FOLLOW_INTERVAL_MS); + } + assert.deepEqual(seen, created); + } finally { + for (const streamId of created.keys()) + await c.stream.delete({ streamId }); + } + }); + + it('given streams when paging current state one item at a time should list the streams the list returns', async () => { + const created = [ + await c.stream.create({ name: `${prefix}-current-1` }), + await c.stream.create({ name: `${prefix}-current-2` }) + ]; + try { + const before = await c.stream.list(); + const projected = await currentStreams(c); + const afterwards = await c.stream.list(); + for (const stream of created) + assert.equal(projected.get(stream.id), stream.name); + // Other e2e files change streams meanwhile: a stream listed unchanged + // before and after the scan must be in it. + for (const stream of before) + if (afterwards.some((later) => later.id === stream.id && later.name === stream.name)) + assert.equal(projected.get(stream.id), stream.name, `stream ${stream.id}`); + } finally { + for (const stream of created) + await c.stream.delete({ streamId: stream.id }); + } + }); + + it('given queries breaking the request rules when reading should refuse before sending', async () => { + const created = [ + await c.stream.create({ name: `${prefix}-token-1` }), + await c.stream.create({ name: `${prefix}-token-2` }) + ]; + try { + const first = await c.system.getMetadataChanges({ + start: MetadataChangeStart.Current(), groups: ['streams'], limit: 1 + }); + assert.ok(first.currentPageToken); + const refused: GetMetadataChanges[] = [ + { limit: 0 }, + { limit: MAX_METADATA_CHANGES_LIMIT + 1 }, + { since: 2n, until: 1n }, + { start: MetadataChangeStart.Latest, since: 1n }, + { start: MetadataChangeStart.Current(), until: 1n }, + { start: MetadataChangeStart.Current(), groups: ['topics'] }, + { start: MetadataChangeStart.Current(first.currentPageToken), groups: ['streams', 'users'] } + ]; + for (const query of refused) + await assert.rejects(c.system.getMetadataChanges(query), isRefusedLocally(INVALID_COMMAND)); + assert.throws(() => MetadataChangeCursor.parse('mc1_x'), isRefusedLocally(INVALID_COMMAND)); + + const latest = await c.system.getMetadataChanges({ start: MetadataChangeStart.Latest }); + assert.equal(latest.status, MetadataChangeStatus.Ok, 'the connection stays usable'); + } finally { + for (const stream of created) + await c.stream.delete({ streamId: stream.id }); + } + }); + + it('given user without manage servers when reading changes should be refused as unauthorized', async () => { + const password = 'changes-secret'; + const user = await c.user.create({ + username: `${prefix}-reader`, password, status: UserStatus.Active, + permissions: { + global: { + ManageServers: false, ReadServers: true, ManageUsers: false, ReadUsers: true, + ManageStreams: false, ReadStreams: true, ManageTopics: false, ReadTopics: true, + PollMessages: false, SendMessages: false + }, + streams: [] + } + }); + const reader = connect({ username: `${prefix}-reader`, password }); + try { + await assert.rejects( + reader.system.getMetadataChanges({}), + (error: unknown) => error instanceof ResponseError && error.errorCode === UNAUTHORIZED && error.fromServer + ); + } finally { + await reader.destroy(); + await c.user.delete({ userId: user.id }); + } + }); + + after(() => { + c.destroy(); + }); +}); + +/** + * Registers the keyed stream case, which needs a server with a config key: + * a client key is refused without one. + */ +export const describeKeyedMetadataChanges = ( + label: string, prefix: string, connect: (credentials: Credentials) => Client, + skip: string | false = false +) => describe(label, { skip }, async () => { + + const c = connect({ username: 'iggy', password: 'iggy' }); + + it('given keyed stream when following its creation should expose fingerprints and no key', async () => { + const cursor = await latestCursor(c, ['streams']); + const key = randomBytes(32); + const created = await c.stream.create({ + name: `${prefix}-keyed`, encryptionKey: StreamEncryptionKey.from(key) + }); + try { + const [record] = await followStream(c, cursor, ['streams'], createdStream(created.id, created.name), 1); + assert.equal(record.kind, MetadataChangeKind.StreamCreated); + assert.ok(record.kind === 'stream_created'); + const { options } = record.body; + assert.equal(options.unreadable, false); + assert.ok(!options.unreadable); + const text = (name: string) => options.derivedOptions[name] ?? options.options[name]; + assert.equal(text('encryption_key_source'), EncryptionKeySource.Stream); + assert.equal(text('encryption_key_fingerprint'), encryptionKeyFingerprint(key)); + const keks = String(text('encryption_kek_fingerprint')).split(','); + assert.deepEqual(keks, created.encryptionKekFingerprints); + for (const kek of keks) + assert.match(kek, FINGERPRINT); + for (const block of [options.options, options.derivedOptions] as ParsedOptions[]) { + for (const name of NEVER_RETURNED) + assert.equal(name in block, false, `${name} was returned`); + for (const value of Object.values(block)) { + assert.ok(!(Buffer.isBuffer(value) && value.includes(key)), 'an option holds the key'); + for (const encoded of [key.toString('base64'), key.toString('base64url'), key.toString('hex')]) + assert.ok(!String(value).includes(encoded), 'an option holds the encoded key'); + } + } + } finally { + await c.stream.delete({ streamId: created.id }); + } + }); + + after(() => { + c.destroy(); + }); +});
diff --git a/foreign/node/src/e2e/resource-options.e2e.ts b/foreign/node/src/e2e/resource-options.e2e.ts index 4be2df2..2eb867a 100644 --- a/foreign/node/src/e2e/resource-options.e2e.ts +++ b/foreign/node/src/e2e/resource-options.e2e.ts
@@ -22,6 +22,7 @@ import { Client } from '../client/client.js'; import { getIggyAddress } from '../tcp.sm.utils.js'; import { EncryptionKeySource } from '../wire/index.js'; +import { describeKeyedMetadataChanges } from './metadata-changes.suite.js'; import { describeResourceOptions } from './resource-options.suite.js'; const [host, port] = getIggyAddress(); @@ -29,3 +30,7 @@ describeResourceOptions('e2e -> resource options', 'e2e-ro', (credentials) => new Client({ transport: 'TCP', options: { host, port }, credentials }), EncryptionKeySource.Server); + +describeKeyedMetadataChanges('e2e -> keyed metadata changes', 'e2e-mc', (credentials) => new Client({ + transport: 'TCP', options: { host, port }, credentials +}));
diff --git a/foreign/node/src/e2e/tls.system.e2e.ts b/foreign/node/src/e2e/tls.system.e2e.ts index 71b21b2..9c5c11c 100644 --- a/foreign/node/src/e2e/tls.system.e2e.ts +++ b/foreign/node/src/e2e/tls.system.e2e.ts
@@ -42,6 +42,7 @@ import type { TlsOption } from '../client/client.type.js'; import { Partitioning, Consumer, EncryptionKeySource, PollingStrategy } from '../wire/index.js'; import { getIggyAddress } from '../tcp.sm.utils.js'; +import { describeKeyedMetadataChanges, describeMetadataChanges } from './metadata-changes.suite.js'; import { describeResourceOptions } from './resource-options.suite.js'; const tlsEnabled = process.env.IGGY_TCP_TLS_ENABLED === 'true'; @@ -75,6 +76,14 @@ !tlsEnabled && 'IGGY_TCP_TLS_ENABLED is not set' ); +describeMetadataChanges( + 'e2e -> tls metadata changes', 'tls-e2e-mc', getTlsClient, !tlsEnabled && 'IGGY_TCP_TLS_ENABLED is not set' +); + +describeKeyedMetadataChanges( + 'e2e -> tls keyed metadata changes', 'tls-e2e-mc', getTlsClient, !tlsEnabled && 'IGGY_TCP_TLS_ENABLED is not set' +); + describe('e2e -> tls', { skip: !tlsEnabled && 'IGGY_TCP_TLS_ENABLED is not set' }, async () => { const c = getTlsClient();
diff --git a/foreign/node/src/index.ts b/foreign/node/src/index.ts index 200972f..456e6c1 100644 --- a/foreign/node/src/index.ts +++ b/foreign/node/src/index.ts
@@ -41,6 +41,54 @@ type StreamOptions, type TopicOptions, type UserOptions, + MetadataChangeGroup, + MetadataChangeKind, + MetadataChangeStatus, + MetadataChangePhase, + MetadataMemberLeftReason, + MetadataTokenDeletedReason, + MetadataSessionClosedReason, + MetadataPermissionsUpdatedReason, + MetadataChangeStart, + MetadataChangeCursor, + MetadataCurrentPageToken, + DEFAULT_METADATA_CHANGES_LIMIT, + MAX_METADATA_CHANGES_LIMIT, + METADATA_CHANGE_CURSOR_PREFIX, + METADATA_CURRENT_PAGE_TOKEN_PREFIX, + metadataChangeGroupName, + metadataChangeKindName, + metadataChangeKindGroup, + type GetMetadataChanges, + type MetadataChanges, + type MetadataChange, + type MetadataChangeRecord, + type MetadataCurrentItem, + type MetadataOptions, + type MetadataStreamKey, + type MetadataTopicKey, + type MetadataUserKey, + type MetadataStreamChange, + type MetadataStreamDeleted, + type MetadataTopicChange, + type MetadataTopicItem, + type MetadataTopicDeleted, + type MetadataPartitionsChanged, + type MetadataPartitionsItem, + type MetadataConsumerGroupCreated, + type MetadataConsumerGroupDeleted, + type MetadataConsumerGroupItem, + type MetadataMemberJoined, + type MetadataMemberLeft, + type MetadataPartitionHandedOver, + type MetadataMemberItem, + type MetadataUserChange, + type MetadataUserIdentity, + type MetadataUserPermissionsUpdated, + type MetadataTokenChange, + type MetadataTokenDeleted, + type MetadataSession, + type MetadataSessionClosed, } from "./wire/index.js"; export * from "./client/index.js";
diff --git a/foreign/node/src/wire/command-set.ts b/foreign/node/src/wire/command-set.ts index 7ccf75a..ca6ddad 100644 --- a/foreign/node/src/wire/command-set.ts +++ b/foreign/node/src/wire/command-set.ts
@@ -65,6 +65,7 @@ import { deleteSegments } from './segment/delete-segments.command.js'; import { describeOptions } from './system/describe-options.command.js'; +import { getMetadataChanges } from './system/get-metadata-changes.command.js'; import { getStats } from './system/get-stats.command.js'; import { snapshot } from './system/snapshot.command.js'; import { ping } from './system/ping.command.js'; @@ -187,7 +188,8 @@ ping: ping(c), getStats: getStats(c), snapshot: snapshot(c), - describeOptions: describeOptions(c) + describeOptions: describeOptions(c), + getMetadataChanges: getMetadataChanges(c) }); type SystemAPI = ReturnType<typeof systemAPI>;
diff --git a/foreign/node/src/wire/command.code.ts b/foreign/node/src/wire/command.code.ts index 86924c7..0df76a6 100644 --- a/foreign/node/src/wire/command.code.ts +++ b/foreign/node/src/wire/command.code.ts
@@ -25,6 +25,7 @@ DescribeOptions: 13, BindSession: 15, UpdateResource: 16, + GetMetadataChanges: 17, GetMe: 20, GetClient: 21, GetClients: 22,
diff --git a/foreign/node/src/wire/fixtures/metadata_changes.json b/foreign/node/src/wire/fixtures/metadata_changes.json new file mode 100644 index 0000000..26e7faf --- /dev/null +++ b/foreign/node/src/wire/fixtures/metadata_changes.json
@@ -0,0 +1,1997 @@ +{ + "schema_version": 1, + "generated_by": "core/common/examples/metadata_changes_fixtures.rs", + "conventions": { + "byte_strings": "Every *_hex field is lowercase hex without separators.", + "expected": "expected is the JSON the HTTP route renders. Binary decoders must reach the same values.", + "record_hex": "record_hex is one record envelope as a response carries it. body_hex is the body alone.", + "refused": "Every refused vector must fail to decode or validate. Query and HTTP refusals are invalid_command.", + "request_hex": "request_hex is the command payload without framing. http_query is the query string of GET /metadata/changes.", + "u128_values": "128-bit values (log ids, checksums, client ids) are 32 lowercase hex characters in JSON.", + "u64_values": "Ops and timestamps are JSON numbers below 2^53 in every vector except large_values, which carries them as decimal strings.", + "unknown_kinds": "A record of an unknown group or kind is kept with its codes and raw body, never refused." + }, + "constants": { + "command_code": 17, + "command_name": "metadata.changes", + "cursor_prefix": "mc1_", + "page_token_prefix": "cs1_", + "cursor_size": 44, + "cursor_text_length": 63, + "max_page_token_size": 316, + "max_page_token_key_size": 260, + "default_limit": 100, + "max_limit": 1000, + "groups": { + "consumer_group_members": { + "code": 5, + "class": "activity", + "parents": [ + "consumer_groups", + "partitions", + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32][member_id:u32]" + }, + "consumer_groups": { + "code": 4, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32]" + }, + "partitions": { + "code": 3, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "personal_access_tokens": { + "code": 7, + "class": "resource", + "parents": [ + "users" + ], + "page_token_key": "[user_id:u32][name_len:u8][name]" + }, + "sessions": { + "code": 8, + "class": "activity", + "parents": [ + "users" + ], + "page_token_key": "[epoch:u64]" + }, + "streams": { + "code": 1, + "class": "resource", + "parents": [], + "page_token_key": "[stream_id:u32]" + }, + "topics": { + "code": 2, + "class": "resource", + "parents": [ + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "users": { + "code": 6, + "class": "resource", + "parents": [], + "page_token_key": "[user_id:u32]" + } + }, + "kinds": { + "consumer_group_created": { + "code": 30, + "group": "consumer_groups" + }, + "consumer_group_deleted": { + "code": 31, + "group": "consumer_groups" + }, + "consumer_group_member_joined": { + "code": 40, + "group": "consumer_group_members" + }, + "consumer_group_member_left": { + "code": 41, + "group": "consumer_group_members" + }, + "consumer_group_partition_handed_over": { + "code": 42, + "group": "consumer_group_members" + }, + "partitions_changed": { + "code": 20, + "group": "partitions" + }, + "personal_access_token_created": { + "code": 60, + "group": "personal_access_tokens" + }, + "personal_access_token_deleted": { + "code": 61, + "group": "personal_access_tokens" + }, + "session_closed": { + "code": 71, + "group": "sessions" + }, + "session_registered": { + "code": 70, + "group": "sessions" + }, + "stream_created": { + "code": 1, + "group": "streams" + }, + "stream_deleted": { + "code": 3, + "group": "streams" + }, + "stream_updated": { + "code": 2, + "group": "streams" + }, + "topic_created": { + "code": 10, + "group": "topics" + }, + "topic_deleted": { + "code": 12, + "group": "topics" + }, + "topic_updated": { + "code": 11, + "group": "topics" + }, + "user_created": { + "code": 50, + "group": "users" + }, + "user_deleted": { + "code": 52, + "group": "users" + }, + "user_password_changed": { + "code": 53, + "group": "users" + }, + "user_permissions_updated": { + "code": 54, + "group": "users" + }, + "user_updated": { + "code": 51, + "group": "users" + } + }, + "statuses": { + "behind": 1, + "expired": 2, + "ok": 0, + "reset": 3 + }, + "phases": { + "changes": 0, + "current": 1 + }, + "start_modes": { + "after": 2, + "current": 3, + "earliest": 0, + "latest": 1 + }, + "member_left_reasons": { + "left": 0, + "session_ended": 1 + }, + "token_deleted_reasons": { + "deleted": 0, + "expired": 1 + }, + "session_closed_reasons": { + "expired": 1, + "logout": 0, + "revoked": 2 + }, + "permissions_updated_reasons": { + "resource_deleted": 1, + "updated": 0 + } + }, + "records": [ + { + "case": "stream_created", + "group": 1, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "record_hex": "191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "stream_updated_unreadable_options", + "group": 1, + "kind": 2, + "body_hex": "03000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "record_hex": "1a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "expected": { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + } + }, + { + "case": "stream_deleted_deferred", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "record_hex": "1b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "expected": { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + }, + { + "case": "topic_created", + "group": 2, + "kind": 10, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "record_hex": "1c1000000000000000000000683d9e8b03f7c1a500000000000000001c00be01094106000104000000020a78000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "op": 4124, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d68", + "timestamp": 1760356800004124, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "partitions_count": 8, + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "topic_updated", + "group": 2, + "kind": 11, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "record_hex": "1d1000000000000000000000693d9e8b03f7c1a500000000000000001d00be01094106000104000000020b3d000000030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "expected": { + "op": 4125, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d69", + "timestamp": 1760356800004125, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_updated", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics-v2", + "partitions_count": 8, + "options": {} + } + }, + { + "case": "topic_deleted_immediate", + "group": 2, + "kind": 12, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "record_hex": "1e10000000000000000000006a3d9e8b03f7c1a500000000000000001e00be01094106000104000000020c2e000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "expected": { + "op": 4126, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6a", + "timestamp": 1760356800004126, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics" + } + }, + { + "case": "partitions_changed_deleted_tail", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "record_hex": "1f10000000000000000000006b3d9e8b03f7c1a500000000000000001f00be01094106000104000000031435000000030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "expected": { + "op": 4127, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6b", + "timestamp": 1760356800004127, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 5, + "removed_partition_ids": [ + 5, + 6, + 7 + ], + "requested_op": 4200 + } + }, + { + "case": "partitions_changed_created", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000a0000000000000000", + "record_hex": "2010000000000000000000006c3d9e8b03f7c1a500000000000000002000be01094106000104000000031421000000030000001910000000000000000000001d100000000000000a0000000000000000", + "expected": { + "op": 4128, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6c", + "timestamp": 1760356800004128, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 10, + "removed_partition_ids": [] + } + }, + { + "case": "consumer_group_created", + "group": 4, + "kind": 30, + "body_hex": "030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "record_hex": "2110000000000000000000006d3d9e8b03f7c1a500000000000000002100be01094106000104000000041e28000000030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "expected": { + "op": 4129, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6d", + "timestamp": 1760356800004129, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 1 + } + }, + { + "case": "consumer_group_deleted", + "group": 4, + "kind": 31, + "body_hex": "030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "record_hex": "2210000000000000000000006e3d9e8b03f7c1a500000000000000002200be01094106000104000000041f31000000030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "expected": { + "op": 4130, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6e", + "timestamp": 1760356800004130, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 0, + "requested_op": 4300 + } + }, + { + "case": "consumer_group_member_joined", + "group": 5, + "kind": 40, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "record_hex": "2310000000000000000000006f3d9e8b03f7c1a500000000000000002300be01094106000109000000052838000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "expected": { + "op": 4131, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6f", + "timestamp": 1760356800004131, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_joined", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 2, + "partition_ids": [] + } + }, + { + "case": "consumer_group_member_left_session_ended", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "record_hex": "241000000000000000000000703d9e8b03f7c1a500000000000000002400be01094106000109000000052935000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "expected": { + "op": 4132, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d70", + "timestamp": 1760356800004132, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_left", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 1, + "reason": "session_ended" + } + }, + { + "case": "consumer_group_partition_handed_over", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d1000000000000000000000020000000101000000", + "record_hex": "251000000000000000000000713d9e8b03f7c1a500000000000000002500be010941060000052a25000000030000001910000000000000000000001d1000000000000000000000020000000101000000", + "expected": { + "op": 4133, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d71", + "timestamp": 1760356800004133, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 2, + "member_id": 1 + } + }, + { + "case": "consumer_group_partition_released", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d10000000000000000000000300000000", + "record_hex": "261000000000000000000000723d9e8b03f7c1a500000000000000002600be010941060000052a21000000030000001910000000000000000000001d10000000000000000000000300000000", + "expected": { + "op": 4134, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d72", + "timestamp": 1760356800004134, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 3 + } + }, + { + "case": "user_created_with_permissions", + "group": 6, + "kind": 50, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "record_hex": "271000000000000000000000733d9e8b03f7c1a500000000000000002700be010941060001000000000632480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "op": 4135, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d73", + "timestamp": 1760356800004135, + "actor_user_id": 0, + "group": "users", + "kind": "user_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "user_updated_inactive", + "group": 6, + "kind": 51, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "record_hex": "281000000000000000000000743d9e8b03f7c1a500000000000000002800be010941060001000000000633280000000700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "expected": { + "op": 4136, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d74", + "timestamp": 1760356800004136, + "actor_user_id": 0, + "group": "users", + "kind": "user_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "inactive", + "permissions": null, + "options": {} + } + }, + { + "case": "user_deleted", + "group": 6, + "kind": 52, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "291000000000000000000000753d9e8b03f7c1a500000000000000002900be0109410600010000000006341d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4137, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d75", + "timestamp": 1760356800004137, + "actor_user_id": 0, + "group": "users", + "kind": "user_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_password_changed", + "group": 6, + "kind": 53, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "2a1000000000000000000000763d9e8b03f7c1a500000000000000002a00be0109410600010700000006351d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4138, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d76", + "timestamp": 1760356800004138, + "actor_user_id": 7, + "group": "users", + "kind": "user_password_changed", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_permissions_pruned", + "group": 6, + "kind": 54, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "record_hex": "2b1000000000000000000000773d9e8b03f7c1a500000000000000002b00be0109410600010400000006362a0000000700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "expected": { + "op": 4139, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d77", + "timestamp": 1760356800004139, + "actor_user_id": 4, + "group": "users", + "kind": "user_permissions_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": false, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": null + }, + "reason": "resource_deleted" + } + }, + { + "case": "personal_access_token_created", + "group": 7, + "kind": 60, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "record_hex": "2c1000000000000000000000783d9e8b03f7c1a500000000000000002c00be01094106000107000000073c1d0000000700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "expected": { + "op": 4140, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d78", + "timestamp": 1760356800004140, + "actor_user_id": 7, + "group": "personal_access_tokens", + "kind": "personal_access_token_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "expiry": 1760400000000000 + } + }, + { + "case": "personal_access_token_deleted_expired", + "group": 7, + "kind": 61, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e01", + "record_hex": "2d1000000000000000000000793d9e8b03f7c1a500000000000000002d00be010941060000073d160000000700000000b832c8fb4006000863692d746f6b656e01", + "expected": { + "op": 4141, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d79", + "timestamp": 1760356800004141, + "group": "personal_access_tokens", + "kind": "personal_access_token_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "reason": "expired" + } + }, + { + "case": "session_registered", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "record_hex": "2e10000000000000000000007a3d9e8b03f7c1a500000000000000002e00be010941060001070000000846240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "op": 4142, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7a", + "timestamp": 1760356800004142, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_registered", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + }, + { + "case": "session_closed_expired", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "record_hex": "2f10000000000000000000007b3d9e8b03f7c1a500000000000000002f00be0109410600000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "expected": { + "op": 4143, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7b", + "timestamp": 1760356800004143, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "expired" + } + }, + { + "case": "session_closed_revoked", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "record_hex": "3010000000000000000000007c3d9e8b03f7c1a500000000000000003000be010941060001070000000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "expected": { + "op": 4144, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7c", + "timestamp": 1760356800004144, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "revoked" + } + } + ], + "items": [ + { + "case": "stream", + "group": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "item_hex": "01b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "topic", + "group": 2, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "item_hex": "0274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "partitions", + "group": 3, + "body_hex": "030000001910000000000000000000001d1000000000000008000000", + "item_hex": "031c000000030000001910000000000000000000001d1000000000000008000000", + "expected": { + "group": "partitions", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 8 + } + }, + { + "case": "consumer_group", + "group": 4, + "body_hex": "030000001910000000000000000000001d10000000000000000000000772656164657273", + "item_hex": "0424000000030000001910000000000000000000001d10000000000000000000000772656164657273", + "expected": { + "group": "consumer_groups", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers" + } + }, + { + "case": "consumer_group_member", + "group": 5, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "item_hex": "0540000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "expected": { + "group": "consumer_group_members", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "partition_ids": [ + 0, + 2, + 4 + ] + } + }, + { + "case": "user", + "group": 6, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "item_hex": "06480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "group": "users", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "personal_access_token_without_expiry", + "group": 7, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "item_hex": "071d0000000700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "expected": { + "group": "personal_access_tokens", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token" + } + }, + { + "case": "session", + "group": 8, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "item_hex": "08240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "group": "sessions", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + } + ], + "cursors": [ + { + "case": "zero", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d000000000000000000000000000000000000000000000000000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 0, + "checksum": "00000000000000000000000000000000", + "ordinal": 0 + }, + { + "case": "first_record_of_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAABgQAAAAAAAApcH3A4ueLUwAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000001810000000000000a5c1f7038b9e2d4c000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4120, + "checksum": "00000000000000004c2d9e8b03f7c1a5", + "ordinal": 0 + }, + { + "case": "inside_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f2000000000000000001000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4187, + "checksum": "0000000000000000f2a4c6e8b0d1f359", + "ordinal": 1 + }, + { + "case": "every_field_at_maximum", + "text": "mc1___________________________________________________________8", + "hex": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + "log_id": "ffffffffffffffffffffffffffffffff", + "op": 18446744073709551615, + "checksum": "ffffffffffffffffffffffffffffffff", + "ordinal": 4294967295 + } + ], + "page_tokens": [ + { + "case": "streams", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000000101040003000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 1, + "group": "streams", + "last_key_hex": "03000000" + }, + { + "case": "topics", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 3, + "group": "topics", + "last_key_hex": "0300000000000000" + }, + { + "case": "partitions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8DCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff0308000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "partitions", + "last_key_hex": "0300000000000000" + }, + { + "case": "consumer_groups", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8EDAADAAAAAAAAAAIAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff040c00030000000000000002000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_groups", + "last_key_hex": "030000000000000002000000" + }, + { + "case": "consumer_group_members", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8FEAADAAAAAAAAAAIAAAABAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff05100003000000000000000200000001000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_group_members", + "last_key_hex": "03000000000000000200000001000000" + }, + { + "case": "users", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAACAGBAAHAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000002006040007000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 32, + "group": "users", + "last_key_hex": "07000000" + }, + { + "case": "personal_access_tokens", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAGAHDQAHAAAACGNpLXRva2Vu", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e1000000000000060070d00070000000863692d746f6b656e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 96, + "group": "personal_access_tokens", + "last_key_hex": "070000000863692d746f6b656e" + }, + { + "case": "sessions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAKAICACUEQAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000a00808009411000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 160, + "group": "sessions", + "last_key_hex": "9411000000000000" + }, + { + "case": "longest_token_name", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8HBAH______25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubg", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff070401ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "personal_access_tokens", + "last_key_hex": "ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e" + } + ], + "requests": [ + { + "case": "earliest_every_group", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000", + "http_query": "from=earliest&limit=100" + }, + { + "case": "earliest_many_duplicate_groups", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "users" + ], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000640000000201060000", + "http_query": "from=earliest&groups=streams,users&limit=100" + }, + { + "case": "earliest_since_users_and_tokens", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "users", + "personal_access_tokens" + ], + "since": 1760353200000000, + "until": null, + "limit": 100 + }, + "request_hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001004c2a2b0841060000640000000206070000", + "http_query": "from=earliest&groups=users,personal_access_tokens&since=1760353200000000&limit=100" + }, + { + "case": "latest_streams", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [ + "streams" + ], + "since": null, + "until": null, + "limit": 1 + }, + "request_hex": "01000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000001010000", + "http_query": "from=latest&groups=streams&limit=1" + }, + { + "case": "after_cursor_with_time_range", + "input": { + "start": "after", + "cursor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "page_token": null, + "groups": [ + "topics", + "streams" + ], + "since": 1760356800000000, + "until": 1760360400000000, + "limit": 500 + }, + "request_hex": "02d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000000100f0bd010941060001009451d809410600f40100000202010000", + "http_query": "after=mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA&groups=streams,topics&since=1760356800000000&until=1760360400000000&limit=500" + }, + { + "case": "current_first_page", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201020000", + "http_query": "from=current&groups=streams,topics&limit=1000" + }, + { + "case": "current_continuation", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201024000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "http_query": "from=current¤t_after=cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA&groups=streams,topics&limit=1000" + } + ], + "responses": [ + { + "case": "changes_page", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001b10000000000000673d9e8b03f7c1a50000000000000000000000000103000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465641a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d7632011b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABsQAAAAAAAAZz2eiwP3waUAAAAAAAAAAAAAAAA", + "has_more": true, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + }, + { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "unknown_kind_and_group", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001910000000000000653d9e8b03f7c1a50000000000000000010000000002000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be01094106000104000000010904000000deadbeef191000000000000001000000653d9e8b03f7c1a500000000000000001900be01094106000104000000c80104000000deadbeef000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABkQAAAAAAAAZT2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": false, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "1", + "kind": "9", + "body": "3q2+7w==" + }, + { + "op": 4121, + "ordinal": 1, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "200", + "kind": "1", + "body": "3q2+7w==" + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "current_page", + "response_hex": "000101d7931c6e2a8f4b0d0000000000000000010000000000000000000000000000005e1000000000000001d7931c6e2a8f4b0d00000000000000005b10000000000000a73d9e8b03f7c1a500000000000000000100000001000000000200000001b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465640274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c0800000000000040000000004000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "expected": { + "status": "ok", + "phase": "current", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 1, + "retained_since": 0, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAApz2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": true, + "records": [], + "items": [ + { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + ], + "current_page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA" + } + }, + { + "case": "behind_before_op_1", + "response_hex": "010000000000000000000000000000000000000000000000000000000000000000000000000000", + "expected": { + "status": "behind", + "phase": "changes", + "log_id": null, + "oldest_op": 0, + "retained_since": 0, + "head_op": 0, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "expired", + "response_hex": "020001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "expired", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "reset", + "response_hex": "030001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "reset", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + } + ], + "large_values": { + "record_hex": "0300000000002000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff01ffffffff084624000000ffffffff0100000000002000feffffffffffffffffffffffffffffff0200000000002000", + "op": "9007199254740995", + "ordinal": 4294967295, + "timestamp": "18446744073709551615", + "checksum": "ffffffffffffffffffffffffffffffff", + "user_created_at": "9007199254740993", + "session": "9007199254740994", + "client_id": "fffffffffffffffffffffffffffffffe" + }, + "refused": { + "cursors": [ + { + "case": "wrong_prefix", + "text": "mc2_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "missing_prefix", + "text": "15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "too_long", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAAA" + }, + { + "case": "too_short", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA" + }, + { + "case": "padded", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA=" + }, + { + "case": "standard_alphabet_character", + "text": "mc1_+5McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "op_zero_with_checksum", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAA" + } + ], + "page_tokens": [ + { + "case": "wrong_prefix", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA" + }, + { + "case": "trailing_byte", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAAAA" + }, + { + "case": "wrong_key_width", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBCAADAAAAAAAAAA" + }, + { + "case": "group_outside_mask", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAIBBAADAAAA" + }, + { + "case": "head_below_anchor", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAABAAAAAAAAAAEBBAADAAAA" + }, + { + "case": "empty", + "text": "cs1_" + } + ], + "record_bodies": [ + { + "case": "unknown_member_left_reason", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000002" + }, + { + "case": "presence_byte_two", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279022c10000000000000" + }, + { + "case": "unsorted_removed_partition_ids", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000090000000600000007000000016810000000000000" + }, + { + "case": "known_kind_under_wrong_group", + "group": 2, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564" + }, + { + "case": "trailing_byte", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000000" + }, + { + "case": "truncated", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a09080706050403020194110000000000" + } + ], + "requests": [ + { + "case": "cursor_bytes_without_after", + "hex": "000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "trailing_byte", + "hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006400000000000000" + }, + { + "case": "unknown_start_mode", + "hex": "040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "non_canonical_cursor", + "hex": "02d7931c6e2a8f4b0d000000000000000000000000000000000100000000000000000000000000000000000000000064000000000000" + } + ], + "responses": [ + { + "case": "items_in_changes_phase", + "hex": "000001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + }, + { + "case": "unknown_status", + "hex": "040001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + } + ], + "queries": [ + { + "case": "limit_zero", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 0 + } + }, + { + "case": "limit_above_maximum", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 1001 + } + }, + { + "case": "since_above_until", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 2, + "until": 1, + "limit": 100 + } + }, + { + "case": "time_bound_with_latest", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 1, + "until": null, + "limit": 100 + } + }, + { + "case": "time_bound_with_current", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": 1, + "limit": 100 + } + }, + { + "case": "current_missing_parent_group", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + }, + { + "case": "page_token_for_other_groups", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + } + ], + "http_queries": [ + { + "case": "no_start", + "query": "groups=streams" + }, + { + "case": "from_and_after", + "query": "from=earliest&after=mc1_x" + }, + { + "case": "unknown_from", + "query": "from=newest" + }, + { + "case": "duplicate_key", + "query": "from=earliest&from=latest" + }, + { + "case": "unknown_key", + "query": "from=earliest&verbose=1" + }, + { + "case": "empty_group_element", + "query": "from=earliest&groups=streams,,topics" + }, + { + "case": "trailing_group_comma", + "query": "from=earliest&groups=streams," + }, + { + "case": "unknown_group", + "query": "from=earliest&groups=everything" + }, + { + "case": "negative_since", + "query": "from=earliest&since=-1" + }, + { + "case": "leading_zero_until", + "query": "from=earliest&until=01" + }, + { + "case": "non_numeric_limit", + "query": "from=earliest&limit=ten" + }, + { + "case": "token_without_current", + "query": "from=latest¤t_after=cs1_AAAA" + } + ] + } +}
diff --git a/foreign/node/src/wire/system/get-metadata-changes.command.test.ts b/foreign/node/src/wire/system/get-metadata-changes.command.test.ts new file mode 100644 index 0000000..97efebd --- /dev/null +++ b/foreign/node/src/wire/system/get-metadata-changes.command.test.ts
@@ -0,0 +1,623 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +import type { ClientProvider } from '../../client/client.type.js'; +import { COMMAND_CODE } from '../command.code.js'; +import { DeserializeError, ResponseError } from '../error.utils.js'; +import type { ParsedOptions } from '../options.utils.js'; +import type { UserPermissions } from '../user/permissions.utils.js'; +import { UserStatus } from '../user/user.utils.js'; +import { Operation, operationForCode } from '../vsr/operation.js'; +import { + DEFAULT_METADATA_CHANGES_LIMIT, GET_METADATA_CHANGES, MAX_METADATA_CHANGES_LIMIT, + MetadataChangeStart, deserializeMetadataChanges, getMetadataChanges, + type GetMetadataChanges, type MetadataChanges +} from './get-metadata-changes.command.js'; +import { + METADATA_CHANGE_CURSOR_PREFIX, METADATA_CHANGE_CURSOR_SIZE, METADATA_CHANGE_GROUP_CODE, + METADATA_CHANGE_KIND_CODE, METADATA_CURRENT_PAGE_TOKEN_PREFIX, MetadataChangeCursor, + MetadataChangeGroup, MetadataChangeKind, MetadataChangePhase, MetadataChangeStatus, + MetadataCurrentPageToken, MetadataMemberLeftReason, MetadataPermissionsUpdatedReason, + MetadataSessionClosedReason, MetadataTokenDeletedReason, metadataChangeGroupName, + metadataChangeKindGroup, metadataChangeKindName, + type MetadataChangeRecord, type MetadataCurrentItem, type MetadataOptions +} from './metadata-changes.utils.js'; + +type QueryInput = { + start: 'earliest' | 'latest' | 'after' | 'current', + cursor: string | null, + page_token: string | null, + groups: MetadataChangeGroup[], + since: number | null, + until: number | null, + limit: number +}; + +type Fixture = { + schema_version: number, + constants: { + command_code: number, + cursor_prefix: string, + page_token_prefix: string, + cursor_size: number, + cursor_text_length: number, + max_page_token_size: number, + default_limit: number, + max_limit: number, + groups: Record<string, { code: number, parents: string[] }>, + kinds: Record<string, { code: number, group: string }>, + statuses: Record<string, number>, + phases: Record<string, number>, + start_modes: Record<string, number>, + member_left_reasons: Record<string, number>, + token_deleted_reasons: Record<string, number>, + session_closed_reasons: Record<string, number>, + permissions_updated_reasons: Record<string, number> + }, + records: { + case: string, group: number, kind: number, body_hex: string, record_hex: string, + expected: Record<string, unknown> + }[], + items: { + case: string, group: number, body_hex: string, item_hex: string, expected: Record<string, unknown> + }[], + cursors: { + case: string, text: string, hex: string, log_id: string, op: number, checksum: string, + ordinal: number + }[], + page_tokens: { + case: string, text: string, hex: string, anchor: string, required_head: number, + groups_mask: number, group: string, last_key_hex: string + }[], + requests: { case: string, input: QueryInput, request_hex: string }[], + responses: { case: string, response_hex: string, expected: Record<string, unknown> }[], + large_values: { + record_hex: string, op: string, ordinal: number, timestamp: string, checksum: string, + user_created_at: string, session: string, client_id: string + }, + refused: { + cursors: { case: string, text: string }[], + page_tokens: { case: string, text: string }[], + record_bodies: { case: string, group: number, kind: number, body_hex: string }[], + requests: { case: string, hex: string }[], + responses: { case: string, hex: string }[], + queries: { case: string, input: QueryInput }[] + } +}; + +const fixtureText = readFileSync(new URL('../fixtures/metadata_changes.json', import.meta.url), 'utf8'); +const fixture: Fixture = JSON.parse(fixtureText); +// The cursor vector at u64::MAX does not survive a plain JSON number. +const cursorOps = [...fixtureText.matchAll(/"op": ([0-9]+),\s*"checksum"/g)].map((match) => BigInt(match[1])); +const { constants } = fixture; + +const fromHex = (hex: string) => Buffer.from(hex, 'hex'); +const hex128 = (value: bigint) => value.toString(16).padStart(32, '0'); + +const isInvalidCommand = (error: unknown) => + error instanceof ResponseError && error.errorCode === 3 && + error.commandCode === COMMAND_CODE.GetMetadataChanges && !error.fromServer; + +const isMalformed = (error: unknown) => error instanceof DeserializeError; + +/** An ok changes page without a log id or cursor around the given records. */ +const changesPage = (...records: Buffer[]) => Buffer.concat([ + Buffer.from([0, 0, 0]), Buffer.alloc(24), Buffer.from([0, 0]), + u32(records.length), ...records, u32(0), Buffer.alloc(2) +]); + +/** An ok current page without a log id or cursor around the given items. */ +const currentPage = (...items: Buffer[]) => Buffer.concat([ + Buffer.from([0, 1, 0]), Buffer.alloc(24), Buffer.from([0, 0]), + u32(0), u32(items.length), ...items, Buffer.alloc(2) +]); + +/** A record of op 1 by root around a body. */ +const recordEnvelope = (group: number, kind: number, body: Buffer) => Buffer.concat([ + u64(1n), u32(0), Buffer.alloc(16), u64(7n), Buffer.from([1, 0, 0, 0, 0, group, kind]), + u32(body.length), body +]); + +const u32 = (value: number) => { + const bytes = Buffer.alloc(4); + bytes.writeUInt32LE(value); + return bytes; +}; + +const u64 = (value: bigint) => { + const bytes = Buffer.alloc(8); + bytes.writeBigUInt64LE(value); + return bytes; +}; + +const fixtureQuery = (input: QueryInput): GetMetadataChanges => { + const starts = { + earliest: () => MetadataChangeStart.Earliest, + latest: () => MetadataChangeStart.Latest, + after: () => MetadataChangeStart.After(MetadataChangeCursor.parse(input.cursor!)), + current: () => MetadataChangeStart.Current( + input.page_token === null ? undefined : MetadataCurrentPageToken.parse(input.page_token) + ) + }; + return { + start: starts[input.start](), + groups: input.groups, + since: input.since === null ? undefined : BigInt(input.since), + until: input.until === null ? undefined : BigInt(input.until), + limit: input.limit + }; +}; + +const optionText = (value: unknown): string => + Buffer.isBuffer(value) ? value.toString('base64') : String(value); + +const kindName = (value: unknown): string => { + if (typeof value === 'bigint') return 'uint64'; + if (typeof value === 'number') return 'uint32'; + if (typeof value === 'boolean') return 'bool'; + if (Buffer.isBuffer(value)) return 'raw'; + return 'string'; +}; + +/** Options in the HTTP JSON form, both blocks merged with their provenance. */ +const optionsView = (options: MetadataOptions) => { + if (options.unreadable) + return { options: null, options_unreadable: true }; + const merged: Record<string, unknown> = {}; + for (const [block, explicit] of [[options.derivedOptions, false], [options.options, true]] as [ParsedOptions, boolean][]) + for (const [key, value] of Object.entries(block)) + merged[key] = { value: optionText(value), kind: kindName(value), explicit }; + return { options: merged }; +}; + +/** Permissions in the get_user JSON form. */ +const permissionsView = (permissions: UserPermissions | null) => { + if (permissions === null) + return null; + const { global } = permissions; + return { + global: { + manage_servers: global.ManageServers, read_servers: global.ReadServers, + manage_users: global.ManageUsers, read_users: global.ReadUsers, + manage_streams: global.ManageStreams, read_streams: global.ReadStreams, + manage_topics: global.ManageTopics, read_topics: global.ReadTopics, + poll_messages: global.PollMessages, send_messages: global.SendMessages + }, + streams: permissions.streams.length === 0 ? null : Object.fromEntries(permissions.streams.map((stream) => [ + String(stream.streamId), + { + manage_stream: stream.permissions.manageStream, read_stream: stream.permissions.readStream, + manage_topics: stream.permissions.manageTopics, read_topics: stream.permissions.readTopics, + poll_messages: stream.permissions.pollMessages, send_messages: stream.permissions.sendMessages, + topics: stream.topics.length === 0 ? null : Object.fromEntries(stream.topics.map((topic) => [ + String(topic.topicId), + { + manage_topic: topic.permissions.manage, read_topic: topic.permissions.read, + poll_messages: topic.permissions.pollMessages, send_messages: topic.permissions.sendMessages + } + ])) + } + ])) + }; +}; + +/** + * A body in the HTTP JSON form: snake_case keys, 64-bit values as numbers + * (every fixture vector but large_values stays below 2^53), 128-bit values as + * hex, absent optionals left out. + */ +const bodyView = (body: Record<string, unknown>): Record<string, unknown> => { + const view: Record<string, unknown> = {}; + for (const [key, value] of Object.entries(body)) { + const name = key.replace(/[A-Z]/g, (letter) => `_${letter.toLowerCase()}`); + if (key === 'options') + Object.assign(view, optionsView(value as MetadataOptions)); + else if (key === 'permissions') + view.permissions = permissionsView(value as UserPermissions | null); + else if (key === 'status') + view.status = value === UserStatus.Active ? 'active' : 'inactive'; + else if (key === 'clientId') + view.client_id = hex128(value as bigint); + else if (value === null && ['requestedOp', 'memberId', 'expiry'].includes(key)) + continue; + else + view[name] = typeof value === 'bigint' ? Number(value) : value; + } + return view; +}; + +const recordView = (record: MetadataChangeRecord) => { + const envelope: Record<string, unknown> = { + op: Number(record.op), ordinal: record.ordinal, checksum: hex128(record.checksum), + timestamp: Number(record.timestamp) + }; + if (record.actorUserId !== null) + envelope.actor_user_id = record.actorUserId; + if (record.kind === 'unknown') + return { + ...envelope, group: String(record.groupCode), kind: String(record.kindCode), + body: record.body.toString('base64') + }; + return { ...envelope, group: record.group, kind: record.kind, ...bodyView(record.body) }; +}; + +const itemView = (item: MetadataCurrentItem) => item.group === 'unknown' + ? { group: String(item.groupCode), body: item.body.toString('base64') } + : { group: item.group, ...bodyView(item.body) }; + +const changesView = (changes: MetadataChanges) => ({ + status: changes.status, + phase: changes.phase, + log_id: changes.logId === null ? null : hex128(changes.logId), + oldest_op: Number(changes.oldestOp), + retained_since: Number(changes.retainedSince), + head_op: Number(changes.headOp), + next: changes.next?.toString() ?? null, + has_more: changes.hasMore, + records: changes.records.map(recordView), + items: changes.items.map(itemView), + current_page_token: changes.currentPageToken?.toString() ?? null +}); + +describe('given shared vectors when decoding should match server contract', () => { + it('mirrors the command code, prefixes, sizes, limits and every name', () => { + assert.equal(fixture.schema_version, 1); + assert.equal(COMMAND_CODE.GetMetadataChanges, constants.command_code); + assert.equal(operationForCode(constants.command_code), Operation.NonReplicated); + assert.equal(METADATA_CHANGE_CURSOR_PREFIX, constants.cursor_prefix); + assert.equal(METADATA_CURRENT_PAGE_TOKEN_PREFIX, constants.page_token_prefix); + assert.equal(METADATA_CHANGE_CURSOR_SIZE, constants.cursor_size); + assert.equal(new MetadataChangeCursor(0n, 0n, 0n, 0).toString().length, constants.cursor_text_length); + assert.equal(DEFAULT_METADATA_CHANGES_LIMIT, constants.default_limit); + assert.equal(MAX_METADATA_CHANGES_LIMIT, constants.max_limit); + + assert.deepEqual( + METADATA_CHANGE_GROUP_CODE, + Object.fromEntries(Object.entries(constants.groups).map(([name, { code }]) => [name, code])) + ); + assert.deepEqual(Object.values(MetadataChangeGroup).sort(), Object.keys(constants.groups).sort()); + for (const [name, { code, group }] of Object.entries(constants.kinds)) { + assert.equal(METADATA_CHANGE_KIND_CODE[name as MetadataChangeKind], code, name); + assert.equal(metadataChangeKindName(code), name); + assert.equal(metadataChangeKindGroup(name as MetadataChangeKind), group, name); + } + assert.deepEqual(Object.values(MetadataChangeKind).sort(), Object.keys(constants.kinds).sort()); + for (let code = 0; code < 256; code++) { + assert.equal(metadataChangeGroupName(code) !== 'unknown', code >= 1 && code <= 8, `group ${code}`); + assert.equal( + metadataChangeKindName(code) !== 'unknown', + Object.values(constants.kinds).some((kind) => kind.code === code), + `kind ${code}` + ); + } + + const names = (values: Record<string, string>) => Object.values(values).sort(); + assert.deepEqual(names(MetadataChangeStatus), Object.keys(constants.statuses).sort()); + assert.deepEqual(names(MetadataChangePhase), Object.keys(constants.phases).sort()); + assert.deepEqual(names(MetadataMemberLeftReason), Object.keys(constants.member_left_reasons).sort()); + assert.deepEqual(names(MetadataTokenDeletedReason), Object.keys(constants.token_deleted_reasons).sort()); + assert.deepEqual(names(MetadataSessionClosedReason), Object.keys(constants.session_closed_reasons).sort()); + assert.deepEqual( + names(MetadataPermissionsUpdatedReason), Object.keys(constants.permissions_updated_reasons).sort() + ); + for (const [mode, code] of Object.entries(constants.start_modes)) { + const query: GetMetadataChanges = mode === 'after' + ? { start: MetadataChangeStart.After(new MetadataChangeCursor(0n, 0n, 0n, 0)) } + : { start: { mode } as MetadataChangeStart }; + assert.equal(GET_METADATA_CHANGES.serialize(query)[0], code, mode); + } + }); + + it('decodes every record vector to the fields the HTTP route renders', () => { + for (const vector of fixture.records) { + assert.ok(vector.record_hex.endsWith(vector.body_hex), vector.case); + const [record] = deserializeMetadataChanges(changesPage(fromHex(vector.record_hex))).records; + assert.equal(record.groupCode, vector.group, vector.case); + assert.equal(record.kindCode, vector.kind, vector.case); + assert.deepEqual(recordView(record), vector.expected, vector.case); + } + }); + + it('decodes every item vector to the fields the HTTP route renders', () => { + for (const vector of fixture.items) { + assert.ok(vector.item_hex.endsWith(vector.body_hex), vector.case); + const [item] = deserializeMetadataChanges(currentPage(fromHex(vector.item_hex))).items; + assert.equal(item.groupCode, vector.group, vector.case); + assert.deepEqual(itemView(item), vector.expected, vector.case); + } + }); + + it('parses and prints every cursor vector', () => { + fixture.cursors.forEach((vector, index) => { + const cursor = MetadataChangeCursor.parse(vector.text); + assert.equal(hex128(cursor.logId), vector.log_id, vector.case); + assert.equal(cursor.op, cursorOps[index], vector.case); + assert.equal(hex128(cursor.checksum), vector.checksum, vector.case); + assert.equal(cursor.ordinal, vector.ordinal, vector.case); + assert.equal(cursor.toString(), vector.text, vector.case); + assert.equal(JSON.stringify({ cursor }), JSON.stringify({ cursor: vector.text }), vector.case); + assert.equal(cursor.toBytes().toString('hex'), vector.hex, vector.case); + }); + assert.equal(cursorOps.length, fixture.cursors.length); + }); + + it('parses and prints every page token vector', () => { + for (const vector of fixture.page_tokens) { + const token = MetadataCurrentPageToken.parse(vector.text); + assert.equal(token.anchor.toString(), vector.anchor, vector.case); + assert.equal(token.requiredHead, BigInt(vector.required_head), vector.case); + const mask = token.groups.reduce((bits, group) => bits | (1 << (METADATA_CHANGE_GROUP_CODE[group] - 1)), 0); + assert.equal(mask, vector.groups_mask, vector.case); + assert.equal(token.lastGroup, vector.group, vector.case); + assert.equal(token.lastKey.toString('hex'), vector.last_key_hex, vector.case); + assert.equal(token.toString(), vector.text, vector.case); + assert.equal(token.toBytes().toString('hex'), vector.hex, vector.case); + assert.equal(MetadataCurrentPageToken.fromBytes(fromHex(vector.hex)).toString(), vector.text, vector.case); + } + const longest = fixture.page_tokens.find((vector) => vector.case === 'longest_token_name'); + assert.equal(longest?.hex.length, constants.max_page_token_size * 2); + }); + + // The Node SDK has no HTTP transport, so http_query and the refused HTTP + // queries have no encoder to check. + it('encodes every request vector byte for byte', () => { + for (const vector of fixture.requests) + assert.equal( + GET_METADATA_CHANGES.serialize(fixtureQuery(vector.input)).toString('hex'), vector.request_hex, vector.case + ); + }); + + it('decodes every response vector to the answer the HTTP route renders', () => { + for (const vector of fixture.responses) + assert.deepEqual(changesView(deserializeMetadataChanges(fromHex(vector.response_hex))), vector.expected, vector.case); + }); + + it('keeps 64-bit and 128-bit values above 2^53 exact', () => { + const large = fixture.large_values; + const [record] = deserializeMetadataChanges(changesPage(fromHex(large.record_hex))).records; + assert.equal(record.kind, MetadataChangeKind.SessionRegistered); + if (record.kind !== 'session_registered') + return; + assert.equal(record.op, BigInt(large.op)); + assert.equal(record.ordinal, large.ordinal); + assert.equal(record.timestamp, BigInt(large.timestamp)); + assert.equal(hex128(record.checksum), large.checksum); + assert.equal(record.body.userCreatedAt, BigInt(large.user_created_at)); + assert.equal(record.body.session, BigInt(large.session)); + assert.equal(hex128(record.body.clientId), large.client_id); + }); +}); + +describe('given refused vectors when decoding or validating should refuse', () => { + it('refuses every malformed cursor text', () => { + for (const vector of fixture.refused.cursors) + assert.throws(() => MetadataChangeCursor.parse(vector.text), isInvalidCommand, vector.case); + }); + + it('refuses every malformed page token text', () => { + for (const vector of fixture.refused.page_tokens) + assert.throws(() => MetadataCurrentPageToken.parse(vector.text), isInvalidCommand, vector.case); + }); + + it('refuses every malformed record body', () => { + for (const vector of fixture.refused.record_bodies) + assert.throws( + () => deserializeMetadataChanges(changesPage(recordEnvelope(vector.group, vector.kind, fromHex(vector.body_hex)))), + isMalformed, vector.case + ); + }); + + it('cannot produce any request the server refuses', () => { + const earliest = (): Buffer => GET_METADATA_CHANGES.serialize({}); + for (const vector of fixture.refused.requests) { + const request = fromHex(vector.hex); + const cursor = request.subarray(1, 1 + METADATA_CHANGE_CURSOR_SIZE); + switch (vector.case) { + case 'cursor_bytes_without_after': + assert.ok(cursor.some((byte) => byte !== 0)); + assert.ok(earliest().subarray(1, 1 + METADATA_CHANGE_CURSOR_SIZE).every((byte) => byte === 0)); + break; + case 'trailing_byte': + assert.deepEqual(earliest(), request.subarray(0, request.length - 1)); + break; + case 'unknown_start_mode': + assert.throws( + () => GET_METADATA_CHANGES.serialize({ start: { mode: `mode-${request[0]}` } as unknown as MetadataChangeStart }), + isInvalidCommand + ); + break; + case 'non_canonical_cursor': + assert.throws( + () => new MetadataChangeCursor( + cursor.readBigUInt64LE(0) | (cursor.readBigUInt64LE(8) << 64n), cursor.readBigUInt64LE(16), + cursor.readBigUInt64LE(24) | (cursor.readBigUInt64LE(32) << 64n), cursor.readUInt32LE(40) + ), + isInvalidCommand + ); + break; + default: + assert.fail(`a refused request vector this test does not cover: ${vector.case}`); + } + } + }); + + it('refuses every malformed response', () => { + for (const vector of fixture.refused.responses) + assert.throws(() => deserializeMetadataChanges(fromHex(vector.hex)), isMalformed, vector.case); + }); + + it('refuses every invalid query before network io', async () => { + let sent = 0; + const provider: ClientProvider = async () => { + sent += 1; + throw new Error('must not connect'); + }; + for (const vector of fixture.refused.queries) + await assert.rejects(getMetadataChanges(provider)(fixtureQuery(vector.input)), isInvalidCommand, vector.case); + assert.equal(sent, 0); + }); +}); + +describe('given metadata change queries when encoding should apply the request rules', () => { + const streamsToken = MetadataCurrentPageToken.parse(fixture.page_tokens[0].text); + + it('accepts the bounds of every rule', () => { + const one = 1n; + const accepted: GetMetadataChanges[] = [ + { limit: 1 }, + { limit: MAX_METADATA_CHANGES_LIMIT }, + { start: MetadataChangeStart.After(new MetadataChangeCursor(1n, 0n, 0n, 0)), since: one, until: one }, + { start: MetadataChangeStart.Latest, groups: ['users', 'users'] }, + { start: MetadataChangeStart.Current(streamsToken), groups: ['streams'] }, + { start: MetadataChangeStart.Current() }, + { + start: MetadataChangeStart.Current(), + groups: ['consumer_group_members', 'consumer_groups', 'partitions', 'topics', 'streams'] + } + ]; + for (const query of accepted) + assert.ok(GET_METADATA_CHANGES.serialize(query).length > 0); + }); + + it('refuses what the server refuses', () => { + const refused: GetMetadataChanges[] = [ + { limit: 0 }, + { limit: 1.5 }, + { groups: ['everything' as MetadataChangeGroup] }, + { since: -1n }, + { until: 1n << 64n }, + { start: MetadataChangeStart.Current(), groups: ['sessions'] }, + { start: MetadataChangeStart.Current(streamsToken) }, + { start: MetadataChangeStart.Current(null as unknown as MetadataCurrentPageToken) }, + { start: MetadataChangeStart.Current(fixture.page_tokens[0].text as unknown as MetadataCurrentPageToken) } + ]; + for (const query of refused) + assert.throws(() => GET_METADATA_CHANGES.serialize(query), isInvalidCommand, JSON.stringify(query, (_, value) => + typeof value === 'bigint' ? value.toString() : value)); + }); + + it('requires every parent group of a current scan', () => { + for (const [name, { parents }] of Object.entries(constants.groups)) { + const alone = { start: MetadataChangeStart.Current(), groups: [name as MetadataChangeGroup] }; + if (parents.length === 0) { + assert.ok(GET_METADATA_CHANGES.serialize(alone), name); + continue; + } + assert.throws(() => GET_METADATA_CHANGES.serialize(alone), isInvalidCommand, name); + assert.ok(GET_METADATA_CHANGES.serialize({ + ...alone, groups: [name as MetadataChangeGroup, ...parents as MetadataChangeGroup[]] + }), name); + } + }); + + it('keeps every requested group when duplicates exceed the count byte', () => { + const groups: MetadataChangeGroup[] = ['topics', 'streams', 'topics', ...Array(300).fill('users')]; + const payload = GET_METADATA_CHANGES.serialize({ groups }); + const countAt = 1 + METADATA_CHANGE_CURSOR_SIZE + 1 + 1 + 4; + assert.equal(payload[countAt], 3); + assert.deepEqual([...payload.subarray(countAt + 1, countAt + 4)], [2, 1, 6]); + assert.equal(payload.length, countAt + 1 + 3 + 2); + }); + + it('leaves the cursor out of a latest start and the token out of a non current start', () => { + const latest = GET_METADATA_CHANGES.serialize({ start: MetadataChangeStart.Latest }); + const earliest = GET_METADATA_CHANGES.serialize({}); + assert.equal(latest[0], 1); + assert.deepEqual(latest.subarray(1), earliest.subarray(1)); + }); + + it('refuses canonical text forms only', () => { + const tokenText = fixture.page_tokens[0].text; + for (const text of [ + tokenText.slice(0, 20) + '\n' + tokenText.slice(20), + `${tokenText}=`, + tokenText.replace('cs1_', 'CS1_') + ]) + assert.throws(() => MetadataCurrentPageToken.parse(text), isInvalidCommand, JSON.stringify(text)); + const cursorText = fixture.cursors[0].text; + assert.throws(() => MetadataChangeCursor.parse(`${cursorText.slice(0, -1)}B`), isInvalidCommand); + assert.throws(() => new MetadataChangeCursor(-1n, 1n, 0n, 0), isInvalidCommand); + assert.throws(() => new MetadataChangeCursor(0n, 1n, 0n, 2 ** 32), isInvalidCommand); + }); + + it('copies the token key out', () => { + const token = MetadataCurrentPageToken.parse(fixture.page_tokens[0].text); + token.lastKey.fill(0xff); + token.toBytes().fill(0xff); + assert.equal(token.toString(), fixture.page_tokens[0].text); + }); +}); + +describe('given malformed responses when decoding should refuse every one', () => { + it('refuses every truncated response vector', () => { + for (const vector of fixture.responses) { + const response = fromHex(vector.response_hex); + for (let length = 0; length < response.length; length++) + assert.throws(() => deserializeMetadataChanges(response.subarray(0, length)), isMalformed, + `${vector.case} truncated to ${length} bytes`); + } + }); + + it('refuses every truncated or extended record body', () => { + for (const vector of fixture.records) { + const body = fromHex(vector.body_hex); + for (let length = 0; length < body.length; length++) + assert.throws( + () => deserializeMetadataChanges(changesPage(recordEnvelope(vector.group, vector.kind, body.subarray(0, length)))), + isMalformed, `${vector.case} truncated to ${length} bytes` + ); + assert.throws( + () => deserializeMetadataChanges(changesPage(recordEnvelope(vector.group, vector.kind, Buffer.concat([body, Buffer.from([0])])))), + isMalformed, `${vector.case} with a trailing byte` + ); + } + }); + + it('refuses a huge record count without allocating it', () => { + const payload = changesPage(); + payload.writeUInt32LE(0xffff_ffff, payload.length - 2 - 4 - 4); + assert.throws(() => deserializeMetadataChanges(payload), isMalformed); + }); + + it('keeps an unknown kind of a known group with its raw body', () => { + const [record] = deserializeMetadataChanges(changesPage(recordEnvelope(1, 9, Buffer.from([0xde, 0xad])))).records; + assert.equal(record.kind, 'unknown'); + assert.equal(record.group, 'streams'); + assert.equal(record.groupCode, 1); + assert.equal(record.kindCode, 9); + assert.deepEqual(record.body, Buffer.from([0xde, 0xad])); + }); + + it('keeps an unknown item group with its raw body', () => { + const [item] = deserializeMetadataChanges(currentPage(Buffer.concat([Buffer.from([200]), u32(3), Buffer.from([1, 2, 3])]))).items; + assert.equal(item.group, 'unknown'); + assert.equal(item.groupCode, 200); + assert.deepEqual(item.body, Buffer.from([1, 2, 3])); + }); +}); + +it('given_many_duplicate_groups_when_encoding_should_preserve_every_requested_group', () => { + const groups: MetadataChangeGroup[] = Array(255).fill('streams'); + groups.push('users'); + const payload = GET_METADATA_CHANGES.serialize({ groups }); + const groupCountAt = 1 + METADATA_CHANGE_CURSOR_SIZE + 2 + 4; + const encoded = payload.subarray(groupCountAt + 1, groupCountAt + 1 + payload[groupCountAt]); + assert.deepEqual(new Set(encoded), new Set([1, 6])); +});
diff --git a/foreign/node/src/wire/system/get-metadata-changes.command.ts b/foreign/node/src/wire/system/get-metadata-changes.command.ts new file mode 100644 index 0000000..fb3bc77 --- /dev/null +++ b/foreign/node/src/wire/system/get-metadata-changes.command.ts
@@ -0,0 +1,259 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import type { CommandResponse } from '../../client/index.js'; +import { COMMAND_CODE } from '../command.code.js'; +import { wrapCommand } from '../command.utils.js'; +import { DeserializeError, responseError } from '../error.utils.js'; +import { + ChangeReader, METADATA_CHANGE_CURSOR_SIZE, METADATA_CHANGE_GROUP_CODE, MetadataChangeCursor, + MetadataCurrentPageToken, decodeMetadataChange, decodeMetadataCurrentItem, + metadataChangeGroupsMask, metadataChangePhase, metadataChangeStatus, missesParentGroup, + readMetadataChangeCursor, + type MetadataChangeGroup, type MetadataChangePhase, type MetadataChangeRecord, + type MetadataChangeStatus, type MetadataCurrentItem +} from './metadata-changes.utils.js'; + +/** Records or items per page when the query does not say. */ +export const DEFAULT_METADATA_CHANGES_LIMIT = 100; + +/** Most records or items one page returns. */ +export const MAX_METADATA_CHANGES_LIMIT = 1000; + +const INVALID_COMMAND = 3; +const U64_MAX = 0xffff_ffff_ffff_ffffn; +/** The group count travels in one byte. */ +const MAX_GROUPS = 255; +const START_MODE_CODE = { earliest: 0, latest: 1, after: 2, current: 3 } as const; + +/** Where a change log query starts. */ +export type MetadataChangeStart = + /** The oldest operation the requested groups still hold completely */ + | { mode: 'earliest' } + /** No records, only a cursor at the newest retained record */ + | { mode: 'latest' } + /** The records after a cursor */ + | { mode: 'after', cursor: MetadataChangeCursor } + /** Current state pages: no token starts a scan, the previous page's token continues it */ + | { mode: 'current', pageToken?: MetadataCurrentPageToken }; + +/** Builders of every start. */ +export const MetadataChangeStart = { + Earliest: { mode: 'earliest' } as MetadataChangeStart, + Latest: { mode: 'latest' } as MetadataChangeStart, + After: (cursor: MetadataChangeCursor): MetadataChangeStart => ({ mode: 'after', cursor }), + Current: (pageToken?: MetadataCurrentPageToken): MetadataChangeStart => ({ mode: 'current', pageToken }) +}; + +/** + * One request for a page of the change log or of the current state. + * + * Refused with `ResponseError` code 3 (invalid command) before anything is + * sent: a limit outside 1 to 1000, `since` above `until`, a time bound with + * the latest or current start, an unknown group, a current scan that misses + * a parent group, and a page token that is not a `MetadataCurrentPageToken` + * or was issued for other groups. + */ +export type GetMetadataChanges = { + /** Defaults to the earliest start */ + start?: MetadataChangeStart, + /** Record families to return, every group when absent or empty */ + groups?: MetadataChangeGroup[], + /** Keeps records stamped at or after it, microseconds since the Unix epoch */ + since?: bigint, + /** Keeps records stamped at or before it, microseconds since the Unix epoch */ + until?: bigint, + /** Records or items per page, 100 when absent */ + limit?: number +}; + +/** One page of the change log, or of the current state. */ +export type MetadataChanges = { + status: MetadataChangeStatus, + phase: MetadataChangePhase, + /** The node's log id, null before it applied op 1 */ + logId: bigint | null, + /** Lowest whole operation from which the requested groups are complete */ + oldestOp: bigint, + /** The requested groups are complete above this timestamp, 0 when nothing was evicted */ + retainedSince: bigint, + /** Highest metadata operation the node applied */ + headOp: bigint, + /** The cursor to send back: present on ok, echoed on behind, null on expired and reset */ + next: MetadataChangeCursor | null, + /** More records remain after `next`, or another state page exists */ + hasMore: boolean, + records: MetadataChangeRecord[], + items: MetadataCurrentItem[], + /** Continuation of a current state scan, null once it is complete */ + currentPageToken: MetadataCurrentPageToken | null +}; + +/** + * Decodes a `GetMetadataChanges` response. Bodies are decoded strictly: + * unknown reason codes, presence bytes other than 0 and 1, a known kind under + * another group and trailing body bytes are refused, while a group or kind + * this build does not know keeps its raw body. + * + * @throws DeserializeError for a malformed response + */ +export const deserializeMetadataChanges = (payload: Buffer): MetadataChanges => { + const reader = new ChangeReader(payload); + const status = metadataChangeStatus(reader); + const phase = metadataChangePhase(reader); + const logId = reader.flag() ? reader.u128() : null; + const oldestOp = reader.u64(); + const retainedSince = reader.u64(); + const headOp = reader.u64(); + const next = reader.flag() ? readMetadataChangeCursor(reader) : null; + const hasMore = reader.flag(); + + const records: MetadataChangeRecord[] = []; + for (let remaining = reader.u32(); remaining > 0; remaining--) { + const op = reader.u64(); + const ordinal = reader.u32(); + const checksum = reader.u128(); + const timestamp = reader.u64(); + const actorUserId = reader.optionalU32(); + const groupCode = reader.u8(); + const kindCode = reader.u8(); + const body = reader.take(reader.u32()); + records.push({ + op, ordinal, checksum, timestamp, actorUserId, groupCode, kindCode, + ...decodeMetadataChange(groupCode, kindCode, body) + }); + } + + const items: MetadataCurrentItem[] = []; + for (let remaining = reader.u32(); remaining > 0; remaining--) { + const groupCode = reader.u8(); + items.push(decodeMetadataCurrentItem(groupCode, reader.take(reader.u32()))); + } + + const tokenLength = reader.u16(); + const currentPageToken = tokenLength === 0 + ? null + : MetadataCurrentPageToken.fromBytes(reader.take(tokenLength)); + + const holdsOtherContent = phase === 'changes' + ? items.length > 0 || currentPageToken !== null + : records.length > 0; + if (holdsOtherContent) + throw new DeserializeError( + 'cannot deserialize metadata changes: a changes page carries only records and a current page only items' + ); + return { + status, phase, logId, oldestOp, retainedSince, headOp, next, hasMore, + records, items, currentPageToken + }; +}; + +export const GET_METADATA_CHANGES = { + code: COMMAND_CODE.GetMetadataChanges, + + /** + * `[start:u8][cursor:44][has_since:u8][since:u64][has_until:u8][until:u64] + * [limit:u32][groups_count:u8][group:u8]*[token_len:u16][token]`. The cursor + * is all zero unless the query starts after one, the token is empty unless + * it continues a current scan, and groups keep the caller's order. + */ + serialize: (query: GetMetadataChanges): Buffer => { + const { start = MetadataChangeStart.Earliest, since, until } = query; + const limit = query.limit ?? DEFAULT_METADATA_CHANGES_LIMIT; + const groups = query.groups ?? []; + checkQuery(start, groups, since, until, limit); + + const encodedGroups = groups.length > MAX_GROUPS ? [...new Set(groups)] : groups; + const token = start.mode === 'current' && start.pageToken ? start.pageToken.toBytes() : null; + const payload = Buffer.alloc( + 1 + METADATA_CHANGE_CURSOR_SIZE + 9 + 9 + 4 + 1 + encodedGroups.length + 2 + (token?.length ?? 0) + ); + let position = payload.writeUInt8(START_MODE_CODE[start.mode], 0); + if (start.mode === 'after') + start.cursor.toBytes().copy(payload, position); + position += METADATA_CHANGE_CURSOR_SIZE; + position = writeOptionalU64(payload, position, since); + position = writeOptionalU64(payload, position, until); + position = payload.writeUInt32LE(limit, position); + position = payload.writeUInt8(encodedGroups.length, position); + for (const group of encodedGroups) + position = payload.writeUInt8(METADATA_CHANGE_GROUP_CODE[group], position); + position = payload.writeUInt16LE(token?.length ?? 0, position); + if (token) + position += token.copy(payload, position); + return payload.subarray(0, position); + }, + + deserialize: (r: CommandResponse): MetadataChanges => deserializeMetadataChanges(r.data) +}; + +export const getMetadataChanges = wrapCommand<GetMetadataChanges, MetadataChanges>( + GET_METADATA_CHANGES +); + +/** Applies the request rules the server enforces. */ +const checkQuery = ( + start: MetadataChangeStart, groups: readonly string[], + since: bigint | undefined, until: bigint | undefined, limit: number +): void => { + if (!Number.isInteger(limit) || limit < 1 || limit > MAX_METADATA_CHANGES_LIMIT) + refuse(`limit ${limit} is outside 1 to ${MAX_METADATA_CHANGES_LIMIT}`); + for (const bound of [since, until]) + if (bound !== undefined && (typeof bound !== 'bigint' || bound < 0n || bound > U64_MAX)) + refuse('time bounds are unsigned 64-bit microseconds'); + if (since !== undefined && until !== undefined && since > until) + refuse('since is above until'); + const mask = metadataChangeGroupsMask(groups); + if (mask === undefined) + return refuse(`unknown group in ${groups.join(',')}`); + const hasTimeBound = since !== undefined || until !== undefined; + switch (start.mode) { + case 'earliest': + break; + case 'after': + if (!(start.cursor instanceof MetadataChangeCursor)) + refuse('an after start needs a MetadataChangeCursor'); + break; + case 'latest': + if (hasTimeBound) + refuse('a latest start takes no time bound'); + break; + case 'current': + if (hasTimeBound) + refuse('a current start takes no time bound'); + if (missesParentGroup(mask)) + refuse('a current scan needs the parent groups of every selected group'); + if (start.pageToken !== undefined && !(start.pageToken instanceof MetadataCurrentPageToken)) + refuse('a current start continues with a MetadataCurrentPageToken'); + if (start.pageToken !== undefined && metadataChangeGroupsMask(start.pageToken.groups) !== mask) + refuse('the page token was issued for other groups'); + break; + default: + refuse(`unknown start ${(start as { mode: unknown }).mode}`); + } +}; + +const writeOptionalU64 = (payload: Buffer, position: number, value: bigint | undefined): number => { + if (value === undefined) + return payload.writeUInt8(0, position); + position = payload.writeUInt8(1, position); + return payload.writeBigUInt64LE(value, position); +}; + +const refuse = (detail: string): never => { + throw responseError(COMMAND_CODE.GetMetadataChanges, INVALID_COMMAND, detail); +};
diff --git a/foreign/node/src/wire/system/index.ts b/foreign/node/src/wire/system/index.ts index 949aa97..c486e0c 100644 --- a/foreign/node/src/wire/system/index.ts +++ b/foreign/node/src/wire/system/index.ts
@@ -16,7 +16,24 @@ // under the License. export * from './describe-options.command.js'; +export * from './get-metadata-changes.command.js'; export * from './get-stats.command.js'; export * from './ping.command.js'; +export { + METADATA_CHANGE_CURSOR_PREFIX, METADATA_CURRENT_PAGE_TOKEN_PREFIX, METADATA_CHANGE_CURSOR_SIZE, + MetadataChangeGroup, MetadataChangeKind, MetadataChangeStatus, MetadataChangePhase, + MetadataMemberLeftReason, MetadataTokenDeletedReason, MetadataSessionClosedReason, + MetadataPermissionsUpdatedReason, MetadataChangeCursor, MetadataCurrentPageToken, + metadataChangeGroupName, metadataChangeKindName, metadataChangeKindGroup, + type MetadataChange, type MetadataChangeRecord, type MetadataCurrentItem, type MetadataOptions, + type MetadataStreamKey, type MetadataTopicKey, type MetadataUserKey, + type MetadataStreamChange, type MetadataStreamDeleted, type MetadataTopicChange, + type MetadataTopicItem, type MetadataTopicDeleted, type MetadataPartitionsChanged, + type MetadataPartitionsItem, type MetadataConsumerGroupCreated, type MetadataConsumerGroupDeleted, + type MetadataConsumerGroupItem, type MetadataMemberJoined, type MetadataMemberLeft, + type MetadataPartitionHandedOver, type MetadataMemberItem, type MetadataUserChange, + type MetadataUserIdentity, type MetadataUserPermissionsUpdated, type MetadataTokenChange, + type MetadataTokenDeleted, type MetadataSession, type MetadataSessionClosed +} from './metadata-changes.utils.js'; export * from './snapshot.command.js'; export * from './update-resource.command.js';
diff --git a/foreign/node/src/wire/system/metadata-changes.utils.ts b/foreign/node/src/wire/system/metadata-changes.utils.ts new file mode 100644 index 0000000..3533197 --- /dev/null +++ b/foreign/node/src/wire/system/metadata-changes.utils.ts
@@ -0,0 +1,1164 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import { isUtf8 } from 'node:buffer'; +import { COMMAND_CODE } from '../command.code.js'; +import { DeserializeError, responseError } from '../error.utils.js'; +import { deserializeOptions, type ParsedOptions } from '../options.utils.js'; +import { UserStatus } from '../user/user.utils.js'; +import type { StreamPerms, TopicPerms, UserPermissions } from '../user/permissions.utils.js'; + +/** Text prefix of a change cursor. */ +export const METADATA_CHANGE_CURSOR_PREFIX = 'mc1_'; + +/** Text prefix of a current state page token. */ +export const METADATA_CURRENT_PAGE_TOKEN_PREFIX = 'cs1_'; + +/** Encoded size of a change cursor. */ +export const METADATA_CHANGE_CURSOR_SIZE = 44; + +const CURSOR_TEXT_LENGTH = METADATA_CHANGE_CURSOR_PREFIX.length + 59; +/** A user id and a personal access token name. */ +const MAX_PAGE_TOKEN_KEY_SIZE = 4 + 1 + 255; +const PAGE_TOKEN_FIXED_SIZE = METADATA_CHANGE_CURSOR_SIZE + 8 + 1 + 1 + 2; +const MAX_PAGE_TOKEN_TEXT_LENGTH = METADATA_CURRENT_PAGE_TOKEN_PREFIX.length + + Math.ceil((PAGE_TOKEN_FIXED_SIZE + MAX_PAGE_TOKEN_KEY_SIZE) / 3) * 4; +const INVALID_COMMAND = 3; +const NON_CANONICAL_CURSOR = 'a cursor at op 0 carries no checksum and no ordinal'; +const U32_MAX = 0xffff_ffff; +const U64_MASK = 0xffff_ffff_ffff_ffffn; +const U128_MAX = (1n << 128n) - 1n; + +/** Family of change records and the unit of filtering. */ +export const MetadataChangeGroup = { + Streams: 'streams', + Topics: 'topics', + Partitions: 'partitions', + ConsumerGroups: 'consumer_groups', + ConsumerGroupMembers: 'consumer_group_members', + Users: 'users', + PersonalAccessTokens: 'personal_access_tokens', + Sessions: 'sessions' +} as const; + +export type MetadataChangeGroup = typeof MetadataChangeGroup[keyof typeof MetadataChangeGroup]; + +/** Kind of a change record. A new body layout is a new kind. */ +export const MetadataChangeKind = { + StreamCreated: 'stream_created', + StreamUpdated: 'stream_updated', + StreamDeleted: 'stream_deleted', + TopicCreated: 'topic_created', + TopicUpdated: 'topic_updated', + TopicDeleted: 'topic_deleted', + PartitionsChanged: 'partitions_changed', + ConsumerGroupCreated: 'consumer_group_created', + ConsumerGroupDeleted: 'consumer_group_deleted', + ConsumerGroupMemberJoined: 'consumer_group_member_joined', + ConsumerGroupMemberLeft: 'consumer_group_member_left', + ConsumerGroupPartitionHandedOver: 'consumer_group_partition_handed_over', + UserCreated: 'user_created', + UserUpdated: 'user_updated', + UserDeleted: 'user_deleted', + UserPasswordChanged: 'user_password_changed', + UserPermissionsUpdated: 'user_permissions_updated', + PersonalAccessTokenCreated: 'personal_access_token_created', + PersonalAccessTokenDeleted: 'personal_access_token_deleted', + SessionRegistered: 'session_registered', + SessionClosed: 'session_closed' +} as const; + +export type MetadataChangeKind = typeof MetadataChangeKind[keyof typeof MetadataChangeKind]; + +/** Outcome of a change log query. */ +export const MetadataChangeStatus = { + /** Records or items after the requested position */ + Ok: 'ok', + /** The node has not applied the requested position yet: ask again later or ask another node */ + Behind: 'behind', + /** The node no longer holds every record after the position: rebuild from current state */ + Expired: 'expired', + /** The position belongs to another history: rebuild from current state */ + Reset: 'reset' +} as const; + +export type MetadataChangeStatus = typeof MetadataChangeStatus[keyof typeof MetadataChangeStatus]; + +/** Whether a page holds change records or current state items. */ +export const MetadataChangePhase = { + Changes: 'changes', + Current: 'current' +} as const; + +export type MetadataChangePhase = typeof MetadataChangePhase[keyof typeof MetadataChangePhase]; + +/** Why a consumer group member was removed. */ +export const MetadataMemberLeftReason = { + Left: 'left', + SessionEnded: 'session_ended' +} as const; + +export type MetadataMemberLeftReason = + typeof MetadataMemberLeftReason[keyof typeof MetadataMemberLeftReason]; + +/** Why a personal access token was removed. */ +export const MetadataTokenDeletedReason = { + Deleted: 'deleted', + Expired: 'expired' +} as const; + +export type MetadataTokenDeletedReason = + typeof MetadataTokenDeletedReason[keyof typeof MetadataTokenDeletedReason]; + +/** Why a session was closed. */ +export const MetadataSessionClosedReason = { + Logout: 'logout', + Expired: 'expired', + /** The user was deactivated or deleted */ + Revoked: 'revoked' +} as const; + +export type MetadataSessionClosedReason = + typeof MetadataSessionClosedReason[keyof typeof MetadataSessionClosedReason]; + +/** Why a user's public permission set changed. */ +export const MetadataPermissionsUpdatedReason = { + /** An `updatePermissions` replaced the set */ + Updated: 'updated', + /** A stream or topic delete pruned the user's grants on it */ + ResourceDeleted: 'resource_deleted' +} as const; + +export type MetadataPermissionsUpdatedReason = + typeof MetadataPermissionsUpdatedReason[keyof typeof MetadataPermissionsUpdatedReason]; + +/** Wire code of every group, in code order. */ +export const METADATA_CHANGE_GROUP_CODE: Readonly<Record<MetadataChangeGroup, number>> = { + streams: 1, + topics: 2, + partitions: 3, + consumer_groups: 4, + consumer_group_members: 5, + users: 6, + personal_access_tokens: 7, + sessions: 8 +}; + +/** Wire code and group of every kind. */ +export const METADATA_CHANGE_KIND_CODE: Readonly<Record<MetadataChangeKind, number>> = { + stream_created: 1, + stream_updated: 2, + stream_deleted: 3, + topic_created: 10, + topic_updated: 11, + topic_deleted: 12, + partitions_changed: 20, + consumer_group_created: 30, + consumer_group_deleted: 31, + consumer_group_member_joined: 40, + consumer_group_member_left: 41, + consumer_group_partition_handed_over: 42, + user_created: 50, + user_updated: 51, + user_deleted: 52, + user_password_changed: 53, + user_permissions_updated: 54, + personal_access_token_created: 60, + personal_access_token_deleted: 61, + session_registered: 70, + session_closed: 71 +}; + +const KIND_GROUP: Readonly<Record<MetadataChangeKind, MetadataChangeGroup>> = { + stream_created: 'streams', + stream_updated: 'streams', + stream_deleted: 'streams', + topic_created: 'topics', + topic_updated: 'topics', + topic_deleted: 'topics', + partitions_changed: 'partitions', + consumer_group_created: 'consumer_groups', + consumer_group_deleted: 'consumer_groups', + consumer_group_member_joined: 'consumer_group_members', + consumer_group_member_left: 'consumer_group_members', + consumer_group_partition_handed_over: 'consumer_group_members', + user_created: 'users', + user_updated: 'users', + user_deleted: 'users', + user_password_changed: 'users', + user_permissions_updated: 'users', + personal_access_token_created: 'personal_access_tokens', + personal_access_token_deleted: 'personal_access_tokens', + session_registered: 'sessions', + session_closed: 'sessions' +}; + +/** Stream identity: ids are reused after a delete, the creating op is not. */ +export type MetadataStreamKey = { + streamId: number, + streamCreatedOp: bigint +}; + +/** Topic identity: the stream key plus the topic id and creating op. */ +export type MetadataTopicKey = MetadataStreamKey & { + topicId: number, + topicCreatedOp: bigint +}; + +/** User identity, the one sessions and tokens bind to. */ +export type MetadataUserKey = { + userId: number, + userCreatedAt: bigint +}; + +/** + * Public options of a resource after a change, as the read commands return + * them, or a marker for stored critical state this build cannot read. A + * mirror keeps its previous values for an unreadable resource. + */ +export type MetadataOptions = + | { unreadable: false, options: ParsedOptions, derivedOptions: ParsedOptions } + | { unreadable: true }; + +/** A stream after a create or update, and a `streams` item. */ +export type MetadataStreamChange = MetadataStreamKey & { + createdAt: bigint, + name: string, + options: MetadataOptions +}; + +/** A removed stream. `requestedOp` is the op of a deferred delete's request. */ +export type MetadataStreamDeleted = MetadataStreamKey & { + createdAt: bigint, + name: string, + requestedOp: bigint | null +}; + +/** A topic after a create or update. */ +export type MetadataTopicChange = MetadataTopicKey & { + createdAt: bigint, + name: string, + partitionsCount: number, + options: MetadataOptions +}; + +/** A `topics` item. Partition counts travel as `partitions` items. */ +export type MetadataTopicItem = MetadataTopicKey & { + createdAt: bigint, + name: string, + options: MetadataOptions +}; + +/** A removed topic. */ +export type MetadataTopicDeleted = MetadataTopicKey & { + createdAt: bigint, + name: string, + requestedOp: bigint | null +}; + +/** A partition count change. Removed ids are ascending, empty on create. */ +export type MetadataPartitionsChanged = MetadataTopicKey & { + partitionsCount: number, + removedPartitionIds: number[], + requestedOp: bigint | null +}; + +/** A `partitions` item. */ +export type MetadataPartitionsItem = MetadataTopicKey & { + partitionsCount: number +}; + +/** A created consumer group with the topic's resulting group count. */ +export type MetadataConsumerGroupCreated = MetadataTopicKey & { + groupId: number, + name: string, + consumerGroupsCount: number +}; + +/** A removed consumer group with the topic's resulting group count. */ +export type MetadataConsumerGroupDeleted = MetadataConsumerGroupCreated & { + requestedOp: bigint | null +}; + +/** A `consumer_groups` item. */ +export type MetadataConsumerGroupItem = MetadataTopicKey & { + groupId: number, + name: string +}; + +/** A member that joined, with its installed partitions. */ +export type MetadataMemberJoined = MetadataTopicKey & { + groupId: number, + memberId: number, + clientId: bigint, + membersCount: number, + partitionIds: number[] +}; + +/** A member that left the group. */ +export type MetadataMemberLeft = MetadataTopicKey & { + groupId: number, + memberId: number, + clientId: bigint, + membersCount: number, + reason: MetadataMemberLeftReason +}; + +/** A completed partition handover. `memberId` is null when the last member left. */ +export type MetadataPartitionHandedOver = MetadataTopicKey & { + groupId: number, + partitionId: number, + memberId: number | null +}; + +/** A `consumer_group_members` item. */ +export type MetadataMemberItem = MetadataTopicKey & { + groupId: number, + memberId: number, + clientId: bigint, + partitionIds: number[] +}; + +/** A user after a create or update, and a `users` item. */ +export type MetadataUserChange = MetadataUserKey & { + username: string, + status: UserStatus, + permissions: UserPermissions | null, + options: MetadataOptions +}; + +/** A deleted user, or one whose password changed. */ +export type MetadataUserIdentity = MetadataUserKey & { + username: string +}; + +/** A user's public permission set after a change. */ +export type MetadataUserPermissionsUpdated = MetadataUserKey & { + username: string, + permissions: UserPermissions | null, + reason: MetadataPermissionsUpdatedReason +}; + +/** A created personal access token, and an item. `expiry` is null for a token that never expires. */ +export type MetadataTokenChange = MetadataUserKey & { + name: string, + expiry: bigint | null +}; + +/** A removed personal access token. */ +export type MetadataTokenDeleted = MetadataUserKey & { + name: string, + reason: MetadataTokenDeletedReason +}; + +/** A registered session, and a `sessions` item. */ +export type MetadataSession = MetadataUserKey & { + clientId: bigint, + session: bigint +}; + +/** A closed session. */ +export type MetadataSessionClosed = MetadataSession & { + reason: MetadataSessionClosedReason +}; + +/** The typed body of a change record, narrowed by `kind`. */ +export type MetadataChange = + | { kind: 'stream_created' | 'stream_updated', body: MetadataStreamChange } + | { kind: 'stream_deleted', body: MetadataStreamDeleted } + | { kind: 'topic_created' | 'topic_updated', body: MetadataTopicChange } + | { kind: 'topic_deleted', body: MetadataTopicDeleted } + | { kind: 'partitions_changed', body: MetadataPartitionsChanged } + | { kind: 'consumer_group_created', body: MetadataConsumerGroupCreated } + | { kind: 'consumer_group_deleted', body: MetadataConsumerGroupDeleted } + | { kind: 'consumer_group_member_joined', body: MetadataMemberJoined } + | { kind: 'consumer_group_member_left', body: MetadataMemberLeft } + | { kind: 'consumer_group_partition_handed_over', body: MetadataPartitionHandedOver } + | { kind: 'user_created' | 'user_updated', body: MetadataUserChange } + | { kind: 'user_deleted' | 'user_password_changed', body: MetadataUserIdentity } + | { kind: 'user_permissions_updated', body: MetadataUserPermissionsUpdated } + | { kind: 'personal_access_token_created', body: MetadataTokenChange } + | { kind: 'personal_access_token_deleted', body: MetadataTokenDeleted } + | { kind: 'session_registered', body: MetadataSession } + | { kind: 'session_closed', body: MetadataSessionClosed } + /** A group or kind this build does not know, with its raw body */ + | { kind: 'unknown', body: Buffer }; + +/** + * One public result of a committed metadata operation. `groupCode` and + * `kindCode` are the wire codes, kept for a record this build does not know. + */ +export type MetadataChangeRecord = MetadataChange & { + op: bigint, + ordinal: number, + /** Prepare checksum of `op` */ + checksum: bigint, + /** Prepare timestamp of `op` in microseconds */ + timestamp: bigint, + /** The acting user, null for a server action */ + actorUserId: number | null, + group: MetadataChangeGroup | 'unknown', + groupCode: number, + kindCode: number +}; + +/** One current state item, narrowed by `group`. `groupCode` is the wire code. */ +export type MetadataCurrentItem = { groupCode: number } & ( + | { group: 'streams', body: MetadataStreamChange } + | { group: 'topics', body: MetadataTopicItem } + | { group: 'partitions', body: MetadataPartitionsItem } + | { group: 'consumer_groups', body: MetadataConsumerGroupItem } + | { group: 'consumer_group_members', body: MetadataMemberItem } + | { group: 'users', body: MetadataUserChange } + | { group: 'personal_access_tokens', body: MetadataTokenChange } + | { group: 'sessions', body: MetadataSession } + /** A group this build does not know, with its raw body */ + | { group: 'unknown', body: Buffer } +); + +const GROUP_BY_CODE = new Map<number, MetadataChangeGroup>( + Object.entries(METADATA_CHANGE_GROUP_CODE).map(([name, code]) => [code, name as MetadataChangeGroup]) +); + +const KIND_BY_CODE = new Map<number, MetadataChangeKind>( + Object.entries(METADATA_CHANGE_KIND_CODE).map(([name, code]) => [code, name as MetadataChangeKind]) +); + +/** Groups a current state scan of a group must include, so cascades of their deletes apply. */ +const GROUP_PARENTS: Readonly<Record<MetadataChangeGroup, readonly MetadataChangeGroup[]>> = { + streams: [], + topics: ['streams'], + partitions: ['topics', 'streams'], + consumer_groups: ['topics', 'streams'], + consumer_group_members: ['consumer_groups', 'partitions', 'topics', 'streams'], + users: [], + personal_access_tokens: ['users'], + sessions: ['users'] +}; + +const STATUS_NAMES: readonly MetadataChangeStatus[] = ['ok', 'behind', 'expired', 'reset']; +const PHASE_NAMES: readonly MetadataChangePhase[] = ['changes', 'current']; +const MEMBER_LEFT_REASONS: readonly MetadataMemberLeftReason[] = ['left', 'session_ended']; +const TOKEN_DELETED_REASONS: readonly MetadataTokenDeletedReason[] = ['deleted', 'expired']; +const SESSION_CLOSED_REASONS: readonly MetadataSessionClosedReason[] = ['logout', 'expired', 'revoked']; +const PERMISSIONS_UPDATED_REASONS: readonly MetadataPermissionsUpdatedReason[] = + ['updated', 'resource_deleted']; + +/** + * Position of the last change record a caller consumed or scanned. Op 0 with + * a zero checksum and ordinal names the beginning of the history `logId` + * identifies. + * + * Its text form, `toString()` and `toJSON()`, is `mc1_` followed by unpadded + * Base64url of the 44 encoded bytes. + */ +export class MetadataChangeCursor { + /** Prepare checksum of op 1, the root of this history */ + readonly logId: bigint; + readonly op: bigint; + /** Prepare checksum of `op` */ + readonly checksum: bigint; + /** Position of the record within `op` */ + readonly ordinal: number; + + /** + * @throws ResponseError with code 3 (invalid command) for a value outside its + * unsigned width or an op 0 that carries a checksum or an ordinal + */ + constructor(logId: bigint, op: bigint, checksum: bigint, ordinal: number) { + const inRange = isUnsigned(logId, U128_MAX) && isUnsigned(op, U64_MASK) && + isUnsigned(checksum, U128_MAX) && Number.isInteger(ordinal) && ordinal >= 0 && ordinal <= U32_MAX; + if (!inRange || (op === 0n && (checksum !== 0n || ordinal !== 0))) + throw invalidCommand('a change cursor holds unsigned values, and op 0 carries no checksum or ordinal'); + this.logId = logId; + this.op = op; + this.checksum = checksum; + this.ordinal = ordinal; + } + + /** + * Reads the text form. Anything but the prefix, the exact length and + * canonical unpadded Base64url is refused. + * + * @throws ResponseError with code 3 (invalid command) + */ + static parse(text: string): MetadataChangeCursor { + if (text.length !== CURSOR_TEXT_LENGTH) + throw invalidCommand('a change cursor is 63 characters'); + const cursor = decodeCursor(decodeCanonicalBase64(text, METADATA_CHANGE_CURSOR_PREFIX)); + if (cursor === undefined) + throw invalidCommand(NON_CANONICAL_CURSOR); + return cursor; + } + + /** The 44 wire bytes: `[log_id:u128][op:u64][checksum:u128][ordinal:u32]`, little-endian. */ + toBytes(): Buffer { + const bytes = Buffer.alloc(METADATA_CHANGE_CURSOR_SIZE); + writeUnsigned128(bytes, 0, this.logId); + bytes.writeBigUInt64LE(this.op, 16); + writeUnsigned128(bytes, 24, this.checksum); + bytes.writeUInt32LE(this.ordinal, 40); + return bytes; + } + + toString(): string { + return METADATA_CHANGE_CURSOR_PREFIX + this.toBytes().toString('base64url'); + } + + toJSON(): string { + return this.toString(); + } +} + +/** + * Continuation of a current state scan. Send it back unchanged with the same + * groups. Its text form is `cs1_` followed by unpadded Base64url of the + * token payload. + */ +export class MetadataCurrentPageToken { + readonly #payload: Buffer; + + private constructor(payload: Buffer) { + this.#payload = payload; + } + + /** + * Reads the text form. Anything but the prefix, canonical unpadded Base64url + * and a token that keeps its own invariants is refused. + * + * @throws ResponseError with code 3 (invalid command) + */ + static parse(text: string): MetadataCurrentPageToken { + if (text.length > MAX_PAGE_TOKEN_TEXT_LENGTH) + throw invalidCommand('the page token is too long'); + const payload = decodeCanonicalBase64(text, METADATA_CURRENT_PAGE_TOKEN_PREFIX); + const refusal = pageTokenRefusal(payload); + if (refusal !== undefined) + throw invalidCommand(refusal); + return new MetadataCurrentPageToken(payload); + } + + /** + * Reads the token payload a response or request carries, refusing one that + * breaks its own invariants. + * + * @throws DeserializeError + */ + static fromBytes(payload: Buffer): MetadataCurrentPageToken { + const refusal = pageTokenRefusal(payload); + if (refusal !== undefined) + throw new DeserializeError(`cannot deserialize metadata changes: ${refusal}`); + return new MetadataCurrentPageToken(Buffer.from(payload)); + } + + /** The cursor captured before the scan's first page */ + get anchor(): MetadataChangeCursor { + return decodeCursor(this.#payload)!; + } + + /** The highest head any page of the scan reported so far */ + get requiredHead(): bigint { + return this.#payload.readBigUInt64LE(METADATA_CHANGE_CURSOR_SIZE); + } + + /** The groups the scan was started with, in code order */ + get groups(): MetadataChangeGroup[] { + return groupsOfMask(this.#payload[METADATA_CHANGE_CURSOR_SIZE + 8]); + } + + /** The group of the last item returned */ + get lastGroup(): MetadataChangeGroup { + return GROUP_BY_CODE.get(this.#payload[METADATA_CHANGE_CURSOR_SIZE + 9])!; + } + + /** A copy of the last item's key, in the page token key layout of its group */ + get lastKey(): Buffer { + return Buffer.from(this.#payload.subarray(PAGE_TOKEN_FIXED_SIZE)); + } + + /** A copy of the token payload */ + toBytes(): Buffer { + return Buffer.from(this.#payload); + } + + toString(): string { + return METADATA_CURRENT_PAGE_TOKEN_PREFIX + this.#payload.toString('base64url'); + } + + toJSON(): string { + return this.toString(); + } +} + +/** + * Sequential bounds-checked reader over a response or one body. Every + * refusal is a `DeserializeError`. + */ +export class ChangeReader { + readonly #data: Buffer; + #position: number; + + constructor(data: Buffer, position = 0) { + this.#data = data; + this.#position = position; + } + + get position(): number { + return this.#position; + } + + get remaining(): number { + return this.#data.length - this.#position; + } + + take(count: number): Buffer { + if (count > this.remaining) + this.fail(`need ${count} bytes at offset ${this.#position}, have ${this.remaining}`); + const bytes = this.#data.subarray(this.#position, this.#position + count); + this.#position += count; + return bytes; + } + + u8(): number { + return this.take(1)[0]; + } + + u16(): number { + return this.take(2).readUInt16LE(0); + } + + u32(): number { + return this.take(4).readUInt32LE(0); + } + + u64(): bigint { + return this.take(8).readBigUInt64LE(0); + } + + u128(): bigint { + return readUnsigned128(this.take(16), 0); + } + + /** A presence or boolean byte, refusing anything but 0 and 1. */ + flag(): boolean { + const offset = this.#position; + const value = this.u8(); + if (value > 1) + this.fail(`flag byte ${value} at offset ${offset}`); + return value === 1; + } + + /** A grant byte, read the way the server's permission decoder reads it. */ + grant(): boolean { + return this.u8() !== 0; + } + + /** A code at or below `names.length - 1`, mapped to its name. */ + code<T>(names: readonly T[], field: string): T { + const offset = this.#position; + const code = this.u8(); + if (code >= names.length) + this.fail(`unknown ${field} ${code} at offset ${offset}`); + return names[code]; + } + + optionalU64(): bigint | null { + return this.flag() ? this.u64() : null; + } + + optionalU32(): number | null { + return this.flag() ? this.u32() : null; + } + + /** A 1 to 255 byte UTF-8 name behind its u8 length. */ + name(): string { + const offset = this.#position; + const length = this.u8(); + if (length === 0) + this.fail(`empty name at offset ${offset}`); + const bytes = this.take(length); + if (!isUtf8(bytes)) + this.fail(`name at offset ${offset} is not UTF-8`); + return bytes.toString('utf8'); + } + + /** `[count:u32][id:u32]*`, refusing ids that are not strictly ascending. */ + partitionIds(): number[] { + const count = this.u32(); + const ids: number[] = []; + for (let index = 0; index < count; index++) { + const id = this.u32(); + if (ids.length > 0 && ids[ids.length - 1] >= id) + this.fail(`partition ids must be strictly ascending at offset ${this.#position - 4}`); + ids.push(id); + } + return ids; + } + + /** Refuses bytes left after a body. */ + finish(): void { + if (this.remaining !== 0) + this.fail(`${this.remaining} trailing bytes after a change body`); + } + + fail(detail: string): never { + throw new DeserializeError(`cannot deserialize metadata changes: ${detail}`); + } +} + +/** The snake_case name of a group code, `unknown` for a code this build does not know. */ +export const metadataChangeGroupName = (code: number): MetadataChangeGroup | 'unknown' => + GROUP_BY_CODE.get(code) ?? 'unknown'; + +/** The snake_case name of a kind code, `unknown` for a code this build does not know. */ +export const metadataChangeKindName = (code: number): MetadataChangeKind | 'unknown' => + KIND_BY_CODE.get(code) ?? 'unknown'; + +/** The group every record of a known kind belongs to. */ +export const metadataChangeKindGroup = (kind: MetadataChangeKind): MetadataChangeGroup => + KIND_GROUP[kind]; + +/** + * The group mask a selection names: one bit per group code, an empty + * selection meaning every group, `undefined` for a name this build does not + * know. + */ +export const metadataChangeGroupsMask = (groups: readonly string[]): number | undefined => { + if (groups.length === 0) + return 0xff; + let mask = 0; + for (const group of groups) { + const code = METADATA_CHANGE_GROUP_CODE[group as MetadataChangeGroup]; + if (code === undefined) + return undefined; + mask |= 1 << (code - 1); + } + return mask; +}; + +/** Whether a current state scan of the mask misses a parent group. */ +export const missesParentGroup = (mask: number): boolean => + groupsOfMask(mask).some((group) => + GROUP_PARENTS[group].some((parent) => (mask & (1 << (METADATA_CHANGE_GROUP_CODE[parent] - 1))) === 0)); + +/** + * Decodes a record body by its codes. A group or kind this build does not + * know keeps the raw body and a known group name, a known kind under another + * group is refused. + */ +export const decodeMetadataChange = ( + groupCode: number, kindCode: number, body: Buffer +): MetadataChange & { group: MetadataChangeGroup | 'unknown' } => { + const group = GROUP_BY_CODE.get(groupCode); + const kind = KIND_BY_CODE.get(kindCode); + if (group === undefined || kind === undefined) + return { kind: 'unknown', group: group ?? 'unknown', body: Buffer.from(body) }; + const reader = new ChangeReader(body); + if (metadataChangeKindGroup(kind) !== group) + reader.fail(`change kind ${kindCode} does not belong to group ${groupCode}`); + const change = readChange(reader, kind); + reader.finish(); + return { ...change, group }; +}; + +/** Decodes an item body by its group code. A group this build does not know keeps the raw body. */ +export const decodeMetadataCurrentItem = (groupCode: number, body: Buffer): MetadataCurrentItem => { + const reader = new ChangeReader(body); + let item: MetadataCurrentItem; + switch (GROUP_BY_CODE.get(groupCode)) { + case 'streams': + item = { groupCode, group: 'streams', body: readStreamChange(reader) }; + break; + case 'topics': + item = { + groupCode, group: 'topics', + body: { + ...readTopicKey(reader), createdAt: reader.u64(), name: reader.name(), + options: readOptions(reader) + } + }; + break; + case 'partitions': + item = { + groupCode, group: 'partitions', + body: { ...readTopicKey(reader), partitionsCount: reader.u32() } + }; + break; + case 'consumer_groups': + item = { + groupCode, group: 'consumer_groups', + body: { ...readTopicKey(reader), groupId: reader.u32(), name: reader.name() } + }; + break; + case 'consumer_group_members': + item = { + groupCode, group: 'consumer_group_members', + body: { + ...readTopicKey(reader), groupId: reader.u32(), memberId: reader.u32(), + clientId: reader.u128(), partitionIds: reader.partitionIds() + } + }; + break; + case 'users': + item = { groupCode, group: 'users', body: readUserChange(reader) }; + break; + case 'personal_access_tokens': + item = { groupCode, group: 'personal_access_tokens', body: readTokenChange(reader) }; + break; + case 'sessions': + item = { groupCode, group: 'sessions', body: readSession(reader) }; + break; + default: + return { groupCode, group: 'unknown', body: Buffer.from(body) }; + } + reader.finish(); + return item; +}; + +/** Reads a 44-byte cursor, refusing an op 0 that carries a checksum or an ordinal. */ +export const readMetadataChangeCursor = (reader: ChangeReader): MetadataChangeCursor => + decodeCursor(reader.take(METADATA_CHANGE_CURSOR_SIZE)) ?? reader.fail(NON_CANONICAL_CURSOR); + +export const metadataChangeStatus = (reader: ChangeReader): MetadataChangeStatus => + reader.code(STATUS_NAMES, 'status'); + +export const metadataChangePhase = (reader: ChangeReader): MetadataChangePhase => + reader.code(PHASE_NAMES, 'phase'); + +const readChange = (reader: ChangeReader, kind: MetadataChangeKind): MetadataChange => { + switch (kind) { + case 'stream_created': + case 'stream_updated': + return { kind, body: readStreamChange(reader) }; + case 'stream_deleted': + return { + kind, + body: { + ...readStreamKey(reader), createdAt: reader.u64(), name: reader.name(), + requestedOp: reader.optionalU64() + } + }; + case 'topic_created': + case 'topic_updated': + return { + kind, + body: { + ...readTopicKey(reader), createdAt: reader.u64(), name: reader.name(), + partitionsCount: reader.u32(), options: readOptions(reader) + } + }; + case 'topic_deleted': + return { + kind, + body: { + ...readTopicKey(reader), createdAt: reader.u64(), name: reader.name(), + requestedOp: reader.optionalU64() + } + }; + case 'partitions_changed': + return { + kind, + body: { + ...readTopicKey(reader), partitionsCount: reader.u32(), + removedPartitionIds: reader.partitionIds(), requestedOp: reader.optionalU64() + } + }; + case 'consumer_group_created': + return { + kind, + body: { + ...readTopicKey(reader), groupId: reader.u32(), name: reader.name(), + consumerGroupsCount: reader.u32() + } + }; + case 'consumer_group_deleted': + return { + kind, + body: { + ...readTopicKey(reader), groupId: reader.u32(), name: reader.name(), + consumerGroupsCount: reader.u32(), requestedOp: reader.optionalU64() + } + }; + case 'consumer_group_member_joined': + return { + kind, + body: { + ...readTopicKey(reader), groupId: reader.u32(), memberId: reader.u32(), + clientId: reader.u128(), membersCount: reader.u32(), partitionIds: reader.partitionIds() + } + }; + case 'consumer_group_member_left': + return { + kind, + body: { + ...readTopicKey(reader), groupId: reader.u32(), memberId: reader.u32(), + clientId: reader.u128(), membersCount: reader.u32(), + reason: reader.code(MEMBER_LEFT_REASONS, 'member left reason') + } + }; + case 'consumer_group_partition_handed_over': + return { + kind, + body: { + ...readTopicKey(reader), groupId: reader.u32(), partitionId: reader.u32(), + memberId: reader.optionalU32() + } + }; + case 'user_created': + case 'user_updated': + return { kind, body: readUserChange(reader) }; + case 'user_deleted': + case 'user_password_changed': + return { kind, body: { ...readUserKey(reader), username: reader.name() } }; + case 'user_permissions_updated': + return { + kind, + body: { + ...readUserKey(reader), username: reader.name(), permissions: readPermissions(reader), + reason: reader.code(PERMISSIONS_UPDATED_REASONS, 'permissions updated reason') + } + }; + case 'personal_access_token_created': + return { kind, body: readTokenChange(reader) }; + case 'personal_access_token_deleted': + return { + kind, + body: { + ...readUserKey(reader), name: reader.name(), + reason: reader.code(TOKEN_DELETED_REASONS, 'token deleted reason') + } + }; + case 'session_registered': + return { kind, body: readSession(reader) }; + case 'session_closed': + return { + kind, + body: { + ...readSession(reader), + reason: reader.code(SESSION_CLOSED_REASONS, 'session closed reason') + } + }; + } +}; + +const readStreamKey = (reader: ChangeReader): MetadataStreamKey => ({ + streamId: reader.u32(), + streamCreatedOp: reader.u64() +}); + +const readTopicKey = (reader: ChangeReader): MetadataTopicKey => ({ + ...readStreamKey(reader), + topicId: reader.u32(), + topicCreatedOp: reader.u64() +}); + +const readUserKey = (reader: ChangeReader): MetadataUserKey => ({ + userId: reader.u32(), + userCreatedAt: reader.u64() +}); + +const readStreamChange = (reader: ChangeReader): MetadataStreamChange => ({ + ...readStreamKey(reader), + createdAt: reader.u64(), + name: reader.name(), + options: readOptions(reader) +}); + +const readUserChange = (reader: ChangeReader): MetadataUserChange => ({ + ...readUserKey(reader), + username: reader.name(), + status: readUserStatus(reader), + permissions: readPermissions(reader), + options: readOptions(reader) +}); + +/** A token body, where expiry 0 means the token never expires. */ +const readTokenChange = (reader: ChangeReader): MetadataTokenChange => { + const key = readUserKey(reader); + const name = reader.name(); + const expiry = reader.u64(); + return { ...key, name, expiry: expiry === 0n ? null : expiry }; +}; + +const readSession = (reader: ChangeReader): MetadataSession => ({ + ...readUserKey(reader), + clientId: reader.u128(), + session: reader.u64() +}); + +const readUserStatus = (reader: ChangeReader): UserStatus => { + const offset = reader.position; + const status = reader.u8(); + if (status !== UserStatus.Active && status !== UserStatus.Inactive) + reader.fail(`user status ${status} at offset ${offset}`); + return status; +}; + +/** + * `[state:u8]`, then for state 0 the explicit and derived blocks exactly as + * the stream, topic and user reads carry them. + */ +const readOptions = (reader: ChangeReader): MetadataOptions => { + const offset = reader.position; + const state = reader.u8(); + if (state === 1) + return { unreadable: true }; + if (state !== 0) + reader.fail(`options state ${state} at offset ${offset}`); + const block = () => { + const blockOffset = reader.position; + const bytes = reader.take(reader.u32()); + try { + return deserializeOptions(bytes); + } catch (error) { + return reader.fail(`options at offset ${blockOffset}: ${(error as Error).message}`); + } + }; + const options = block(); + return { unreadable: false, options, derivedOptions: block() }; +}; + +/** `[has:u8]` and the permission set without a length prefix. */ +const readPermissions = (reader: ChangeReader): UserPermissions | null => { + if (!reader.flag()) + return null; + const global = { + ManageServers: reader.grant(), + ReadServers: reader.grant(), + ManageUsers: reader.grant(), + ReadUsers: reader.grant(), + ManageStreams: reader.grant(), + ReadStreams: reader.grant(), + ManageTopics: reader.grant(), + ReadTopics: reader.grant(), + PollMessages: reader.grant(), + SendMessages: reader.grant() + }; + const streams: StreamPerms[] = []; + for (let hasStream = reader.grant(); hasStream; hasStream = reader.grant()) { + const streamId = reader.u32(); + const permissions = { + manageStream: reader.grant(), + readStream: reader.grant(), + manageTopics: reader.grant(), + readTopics: reader.grant(), + pollMessages: reader.grant(), + sendMessages: reader.grant() + }; + const topics: TopicPerms[] = []; + for (let hasTopic = reader.grant(); hasTopic; hasTopic = reader.grant()) { + const topicId = reader.u32(); + topics.push({ + topicId, + permissions: { + manage: reader.grant(), + read: reader.grant(), + pollMessages: reader.grant(), + sendMessages: reader.grant() + } + }); + } + streams.push({ streamId, permissions, topics }); + } + return { global, streams }; +}; + +const groupsOfMask = (mask: number): MetadataChangeGroup[] => + [...GROUP_BY_CODE].filter(([code]) => (mask & (1 << (code - 1))) !== 0).map(([, group]) => group); + +/** Why a page token breaks its own invariants, `undefined` for a valid one. */ +const pageTokenRefusal = (payload: Buffer): string | undefined => { + if (payload.length < PAGE_TOKEN_FIXED_SIZE) + return `a page token is at least ${PAGE_TOKEN_FIXED_SIZE} bytes`; + const anchor = decodeCursor(payload); + if (anchor === undefined) + return 'page token anchor is not canonical'; + const groupsMask = payload[METADATA_CHANGE_CURSOR_SIZE + 8]; + const groupCode = payload[METADATA_CHANGE_CURSOR_SIZE + 9]; + const group = GROUP_BY_CODE.get(groupCode); + if (group === undefined) + return `page token names unknown group ${groupCode}`; + const keyLength = payload.readUInt16LE(METADATA_CHANGE_CURSOR_SIZE + 10); + if (keyLength > MAX_PAGE_TOKEN_KEY_SIZE || payload.length !== PAGE_TOKEN_FIXED_SIZE + keyLength) + return `page token key of ${keyLength} bytes does not fill a ${payload.length} byte token`; + if ((groupsMask & (1 << (groupCode - 1))) === 0) + return 'page token group is not in its group mask'; + if (payload.readBigUInt64LE(METADATA_CHANGE_CURSOR_SIZE) < anchor.op) + return 'page token head is below its anchor'; + if (!keyFitsGroup(group, payload.subarray(PAGE_TOKEN_FIXED_SIZE))) + return `page token key of ${keyLength} bytes does not fit group ${group}`; + return undefined; +}; + +const keyFitsGroup = (group: MetadataChangeGroup, key: Buffer): boolean => { + switch (group) { + case 'streams': + case 'users': + return key.length === 4; + case 'topics': + case 'partitions': + case 'sessions': + return key.length === 8; + case 'consumer_groups': + return key.length === 12; + case 'consumer_group_members': + return key.length === 16; + case 'personal_access_tokens': + return key.length > 5 && key[4] > 0 && key.length === 5 + key[4] && isUtf8(key.subarray(5)); + } +}; + +/** + * Strips `prefix` and decodes unpadded Base64url that encodes back to the + * same text, so one value has exactly one text form. + */ +const decodeCanonicalBase64 = (text: string, prefix: string): Buffer => { + if (!text.startsWith(prefix)) + throw invalidCommand(`expected the ${prefix} prefix`); + const encoded = text.slice(prefix.length); + const bytes = Buffer.from(encoded, 'base64url'); + if (bytes.toString('base64url') !== encoded) + throw invalidCommand('expected canonical unpadded Base64url'); + return bytes; +}; + +/** + * The cursor in the first 44 bytes, `undefined` for an op 0 that carries a + * checksum or an ordinal, so each caller raises its own error. + */ +const decodeCursor = (bytes: Buffer): MetadataChangeCursor | undefined => { + const op = bytes.readBigUInt64LE(16); + const checksum = readUnsigned128(bytes, 24); + const ordinal = bytes.readUInt32LE(40); + if (op === 0n && (checksum !== 0n || ordinal !== 0)) + return undefined; + return new MetadataChangeCursor(readUnsigned128(bytes, 0), op, checksum, ordinal); +}; + +const isUnsigned = (value: bigint, maximum: bigint): boolean => + typeof value === 'bigint' && value >= 0n && value <= maximum; + +const readUnsigned128 = (bytes: Buffer, offset: number): bigint => + bytes.readBigUInt64LE(offset) | (bytes.readBigUInt64LE(offset + 8) << 64n); + +const writeUnsigned128 = (bytes: Buffer, offset: number, value: bigint): void => { + bytes.writeBigUInt64LE(value & U64_MASK, offset); + bytes.writeBigUInt64LE(value >> 64n, offset + 8); +}; + +const invalidCommand = (detail: string) => + responseError(COMMAND_CODE.GetMetadataChanges, INVALID_COMMAND, detail);
diff --git a/foreign/php/Cargo.toml b/foreign/php/Cargo.toml index e0d0d5a..9c7933c 100644 --- a/foreign/php/Cargo.toml +++ b/foreign/php/Cargo.toml
@@ -17,7 +17,7 @@ [package] name = "iggy-php" -version = "0.1.4" +version = "0.1.5" edition = "2024" authors = ["Iggy Committers <dev@iggy.apache.org>"] license = "Apache-2.0" @@ -33,7 +33,9 @@ futures = "0.3.34" ext-php-rs = "=0.16.3" iggy = { path = "../../core/sdk" } +iggy_binary_protocol = { path = "../../core/binary_protocol" } libc = "0.2.191" +serde_json = "1.0.154" tokio = "1.53.1" [profile.release]
diff --git a/foreign/php/README.md b/foreign/php/README.md index 74c8910..51cc55b 100644 --- a/foreign/php/README.md +++ b/foreign/php/README.md
@@ -238,6 +238,79 @@ - `describeOptions(Iggy\OptionsScope)` returns the server catalog as `Iggy\OptionSpec` objects with `key`, `kind`, `updatable`, `resettable`, `secret`, `read_only`, `critical`, `default_value` and `description`. - SDK exceptions carry the server error code in `getCode()`, for example 4103 (option cannot be updated), 4104 (option cannot be reset), 4105 (read-only option), 4108 (root user cannot be changed by this caller) and 4041 (unsupported option key). +## Metadata Changes + +`getMetadataChanges()` reads the node's change log: one record per public result of a committed metadata operation, such as a stream created, a topic renamed, a user's permissions changed or a session closed. A mirror keeps a cursor, asks for the records after it, and stores the returned `next` cursor. The call needs root or the global `manage_servers` permission, and a query the server would refuse throws `Invalid command` (code 3) before anything is sent. + +```php +use Iggy\MetadataChangeGroup; +use Iggy\MetadataChangeStart; +use Iggy\MetadataChangeStatus; + +function follow(Iggy\Client $client, array $groups): void +{ + $start = MetadataChangeStart::earliest(); + while (true) { + $page = $client->getMetadataChanges($start, $groups); + switch ($page->status) { + case MetadataChangeStatus::Ok: + foreach ($page->records as $record) { + echo $record->op, ' ', $record->kind?->value ?? 'unknown', PHP_EOL; + } + if ($page->next !== null) { + $start = MetadataChangeStart::after($page->next); + } + if (!$page->has_more) { + sleep(1); + } + break; + case MetadataChangeStatus::Behind: + usleep(100_000); + break; + default: + $start = rebuild($client, $groups); + } + } +} + +// Pages the current state, then returns the position to follow from. +function rebuild(Iggy\Client $client, array $groups): MetadataChangeStart +{ + $start = MetadataChangeStart::current(); + while (true) { + $page = $client->getMetadataChanges($start, $groups); + if ($page->status === MetadataChangeStatus::Behind) { + usleep(100_000); + continue; + } + if ($page->status !== MetadataChangeStatus::Ok) { + $start = MetadataChangeStart::current(); + continue; + } + foreach ($page->items as $item) { + echo $item->group?->value ?? 'unknown', PHP_EOL; + } + if ($page->current_page_token !== null) { + $start = MetadataChangeStart::current($page->current_page_token); + } elseif ($page->next !== null) { + return MetadataChangeStart::after($page->next); + } else { + $start = MetadataChangeStart::current(); + } + } +} + +follow($client, [MetadataChangeGroup::Streams, MetadataChangeGroup::Topics]); +``` + +- `Expired` means the log no longer holds the history after the cursor, and `Reset` means the cursor belongs to another history, for example after the data directory was rebuilt. Either way the mirror rebuilds from current state pages and then follows the changes after the anchor they return. +- Current pages need the parent groups of every selected group, for example `Topics` with `Streams`. Keep a rebuilt state private until a change page answers `Ok` with `has_more` false and a `head_op` at or above the last current page's. +- Pages, cursors, page tokens, records, items and bodies expose their values as readonly snake_case properties, like `StreamDetails`, for example `$record->change->stream_created_op`, so `==` compares their contents. `$record->kind` names the body class in `$record->change`, for example `Iggy\MetadataStreamChange` for `StreamCreated` and `StreamUpdated`. +- A record or item this build does not know has a null `kind` or `group` and a `MetadataUnknownChange` or `MetadataUnknownItem` body with its codes and raw body, so a mirror can stop there rather than skip a change. +- Stream, topic and user bodies return `options` and `derived_options` arrays like `StreamDetails`: a stream's key source and fingerprints, never its key. +- Unsigned 64-bit values such as ops and timestamps are ints, or decimal strings above `PHP_INT_MAX`. Log ids, checksums and client ids are 128-bit decimal strings. A cursor cast to string gives its `mc1_` text and a page token its `cs1_` text, and `MetadataChangeCursor::parse()` and `MetadataCurrentPageToken::parse()` read them back, so a mirror can store its position as a string. +- `MetadataChanges::decode()` reads the payload `sendBinaryRequest(17, ...)` returns, and `MetadataChanges::fromJson()` the body of `GET /metadata/changes`. + ## API Notes - `new Iggy\Client($address, new Iggy\PendingOperationsConfig(256, 64 * 1024 * 1024))`
diff --git a/foreign/php/iggy-php.stubs.php b/foreign/php/iggy-php.stubs.php index b00d5eb..b7454f8 100644 --- a/foreign/php/iggy-php.stubs.php +++ b/foreign/php/iggy-php.stubs.php
@@ -218,6 +218,25 @@ public static function fromConnectionString(string $connection_string): \Iggy\Client {} /** + * Reads one page of the metadata change log, or of the current state. + * + * Needs root or the global `manage_servers` permission. `groups` takes + * `Iggy\MetadataChangeGroup` cases, empty for every group, and a current state scan + * needs the parent groups of each group. `since` and `until` are microseconds as an int + * or a decimal string, and only apply to `earliest` and `after`. `limit` is 1 to 1000, + * 100 when null. A query the server would refuse throws `Invalid command` (code 3) + * before anything is sent. + * + * @param \Iggy\MetadataChangeStart $start + * @param array $groups + * @param mixed $since + * @param mixed $until + * @param mixed $limit + * @return \Iggy\MetadataChanges + */ + public function getMetadataChanges(\Iggy\MetadataChangeStart $start, array $groups = [], mixed $since = null, mixed $until = null, mixed $limit = null): \Iggy\MetadataChanges {} + + /** * Gets a stream by id or name. * * @param mixed $stream_id @@ -546,6 +565,1126 @@ } /** + * Position of the last change record a reader consumed or scanned. Send it back in + * `MetadataChangeStart::after()` to read the records after it. + * + * Casting it to string gives its `mc1_` text, which `parse()` reads back. + */ + class MetadataChangeCursor { + /** + * Prepare checksum of `op` as a decimal string. + * + * @var string + */ + public readonly string $checksum; + + /** + * Prepare checksum of op 1 as a decimal string, which names the history the cursor + * belongs to. + * + * @var string + */ + public readonly string $log_id; + + /** + * The operation of the last record, 0 before the first record. + * + * @var mixed + */ + public readonly mixed $op; + + /** + * Position of the record within `op`. + * + * @var int + */ + public readonly int $ordinal; + + /** + * Builds a cursor from its fields: 128-bit values as decimal strings, `op` as an int or + * a decimal string. + * + * @param string $log_id + * @param mixed $op + * @param string $checksum + * @param int $ordinal + */ + public function __construct(string $log_id, mixed $op, string $checksum, int $ordinal) {} + + /** + * @return string + */ + public function __toString(): string {} + + /** + * Reads a cursor from its `mc1_` text. Anything else throws `Invalid command` (code 3). + * + * @param string $text + * @return \Iggy\MetadataChangeCursor + */ + public static function parse(string $text): \Iggy\MetadataChangeCursor {} + } + + /** + * Family of change records and the unit of filtering. Values are the HTTP API names. + */ + enum MetadataChangeGroup: string { + case Streams = 'streams'; + case Topics = 'topics'; + case Partitions = 'partitions'; + case ConsumerGroups = 'consumer_groups'; + case ConsumerGroupMembers = 'consumer_group_members'; + case Users = 'users'; + case PersonalAccessTokens = 'personal_access_tokens'; + case Sessions = 'sessions'; + } + + /** + * Kind of a change record. Values are the HTTP API names. + */ + enum MetadataChangeKind: string { + case StreamCreated = 'stream_created'; + case StreamUpdated = 'stream_updated'; + case StreamDeleted = 'stream_deleted'; + case TopicCreated = 'topic_created'; + case TopicUpdated = 'topic_updated'; + case TopicDeleted = 'topic_deleted'; + case PartitionsChanged = 'partitions_changed'; + case ConsumerGroupCreated = 'consumer_group_created'; + case ConsumerGroupDeleted = 'consumer_group_deleted'; + case ConsumerGroupMemberJoined = 'consumer_group_member_joined'; + case ConsumerGroupMemberLeft = 'consumer_group_member_left'; + case ConsumerGroupPartitionHandedOver = 'consumer_group_partition_handed_over'; + case UserCreated = 'user_created'; + case UserUpdated = 'user_updated'; + case UserDeleted = 'user_deleted'; + case UserPasswordChanged = 'user_password_changed'; + case UserPermissionsUpdated = 'user_permissions_updated'; + case PersonalAccessTokenCreated = 'personal_access_token_created'; + case PersonalAccessTokenDeleted = 'personal_access_token_deleted'; + case SessionRegistered = 'session_registered'; + case SessionClosed = 'session_closed'; + } + + /** + * Whether a page carries change records or current state items. + */ + enum MetadataChangePhase: string { + case Changes = 'changes'; + case Current = 'current'; + } + + /** + * One change record: the committed operation that produced it, its position within that + * operation and the typed body of what is true after it. + */ + class MetadataChangeRecord { + /** + * The acting user, null for server actions. + * + * @var int|null + */ + public readonly ?int $actor_user_id; + + /** + * The typed body `kind` names, for example a `MetadataStreamChange` for + * `StreamCreated` and `StreamUpdated`, or a `MetadataUnknownChange`. + * + * @var mixed + */ + public readonly mixed $change; + + /** + * Prepare checksum of `op` as a decimal string. + * + * @var string + */ + public readonly string $checksum; + + /** + * The group, null for a record this build does not know. + * + * @var \Iggy\MetadataChangeGroup|null + */ + public readonly ?\Iggy\MetadataChangeGroup $group; + + /** + * The kind, null for a record this build does not know. + * + * @var \Iggy\MetadataChangeKind|null + */ + public readonly ?\Iggy\MetadataChangeKind $kind; + + /** + * The metadata operation that produced the record. + * + * @var mixed + */ + public readonly mixed $op; + + /** + * Position of the record within `op`. + * + * @var int + */ + public readonly int $ordinal; + + /** + * Prepare timestamp of `op` in microseconds. + * + * @var mixed + */ + public readonly mixed $timestamp; + + public function __construct() {} + } + + /** + * Where `Client::getMetadataChanges()` starts reading. + */ + class MetadataChangeStart { + public function __construct() {} + + /** + * The records after `cursor`. + * + * @param \Iggy\MetadataChangeCursor $cursor + * @return \Iggy\MetadataChangeStart + */ + public static function after(\Iggy\MetadataChangeCursor $cursor): \Iggy\MetadataChangeStart {} + + /** + * Current state pages: null starts a scan, a token continues it. + * + * @param \Iggy\MetadataCurrentPageToken|null $page_token + * @return \Iggy\MetadataChangeStart + */ + public static function current(?\Iggy\MetadataCurrentPageToken $page_token = null): \Iggy\MetadataChangeStart {} + + /** + * The oldest operation the requested groups still hold completely. + * + * @return \Iggy\MetadataChangeStart + */ + public static function earliest(): \Iggy\MetadataChangeStart {} + + /** + * No records, only a cursor at the newest retained record. + * + * @return \Iggy\MetadataChangeStart + */ + public static function latest(): \Iggy\MetadataChangeStart {} + } + + /** + * Outcome of a change log query. + * + * `Ok`: apply the records and keep `next`. `Behind`: the node has not applied the position + * yet, ask again later or ask another node. `Expired` and `Reset`: rebuild from current state. + */ + enum MetadataChangeStatus: string { + case Ok = 'ok'; + case Behind = 'behind'; + case Expired = 'expired'; + case Reset = 'reset'; + } + + /** + * One page of the metadata change log, or of the current state, as + * `Client::getMetadataChanges()` returns it. + * + * Unsigned 64-bit values are ints, or decimal strings above `PHP_INT_MAX`. 128-bit values + * (log ids, checksums, client ids) are decimal strings. + */ + class MetadataChanges { + /** + * Continuation of a current state scan, null once it is complete. + * + * @var \Iggy\MetadataCurrentPageToken|null + */ + public readonly ?\Iggy\MetadataCurrentPageToken $current_page_token; + + /** + * More entries remain after `next`, or another state page exists. + * + * @var bool + */ + public readonly bool $has_more; + + /** + * Highest metadata operation the answering node applied. + * + * @var mixed + */ + public readonly mixed $head_op; + + /** + * Current state items, empty on change pages. + * + * @var array + */ + public readonly array $items; + + /** + * This node's log id as a decimal string, null before it applied op 1. + * + * @var string|null + */ + public readonly ?string $log_id; + + /** + * The cursor to send back. Present on `Ok`, echoed on `Behind`, null on `Expired` and + * `Reset`. The last state page returns the anchor to follow from. + * + * @var \Iggy\MetadataChangeCursor|null + */ + public readonly ?\Iggy\MetadataChangeCursor $next; + + /** + * Lowest whole operation from which the requested groups are complete. + * + * @var mixed + */ + public readonly mixed $oldest_op; + + /** + * `Changes` for `earliest`, `latest` and `after`, `Current` for state pages. + * + * @var \Iggy\MetadataChangePhase + */ + public readonly \Iggy\MetadataChangePhase $phase; + + /** + * Change records in `(op, ordinal)` order, empty on state pages. + * + * @var array + */ + public readonly array $records; + + /** + * The requested groups are complete for every timestamp above this value in + * microseconds, 0 when nothing was evicted. + * + * @var mixed + */ + public readonly mixed $retained_since; + + public readonly \Iggy\MetadataChangeStatus $status; + + public function __construct() {} + + /** + * Decodes the response payload of the raw `metadata.changes` command (code 17), as + * `Client::sendBinaryRequest()` returns it. + * + * @param string $payload + * @return \Iggy\MetadataChanges + */ + public static function decode(string $payload): \Iggy\MetadataChanges {} + + /** + * Reads the JSON body `GET /metadata/changes` returns. + * + * @param string $json + * @return \Iggy\MetadataChanges + */ + public static function fromJson(string $json): \Iggy\MetadataChanges {} + } + + /** + * A created consumer group with the topic's resulting group count. + */ + class MetadataConsumerGroupCreated { + /** + * The topic's consumer group count after the change. + * + * @var int + */ + public readonly int $consumer_groups_count; + + public readonly int $group_id; + + public readonly string $name; + + public readonly mixed $stream_created_op; + + public readonly int $stream_id; + + public readonly mixed $topic_created_op; + + public readonly int $topic_id; + + public function __construct() {} + } + + /** + * A removed consumer group with the topic's resulting group count. + */ + class MetadataConsumerGroupDeleted { + /** + * The topic's consumer group count after the change. + * + * @var int + */ + public readonly int $consumer_groups_count; + + public readonly int $group_id; + + public readonly string $name; + + /** + * The operation of the delete request when the delete completed later, else null. + * + * @var mixed + */ + public readonly mixed $requested_op; + + public readonly mixed $stream_created_op; + + public readonly int $stream_id; + + public readonly mixed $topic_created_op; + + public readonly int $topic_id; + + public function __construct() {} + } + + /** + * A `consumer_groups` current state item. + */ + class MetadataConsumerGroupItem { + public readonly int $group_id; + + public readonly string $name; + + public readonly mixed $stream_created_op; + + public readonly int $stream_id; + + public readonly mixed $topic_created_op; + + public readonly int $topic_id; + + public function __construct() {} + } + + /** + * One current state item of a `MetadataChangeStart::current()` page. + */ + class MetadataCurrentItem { + /** + * The typed body `group` names, or a `MetadataUnknownItem`. + * + * @var mixed + */ + public readonly mixed $body; + + /** + * The group, null for an item this build does not know. + * + * @var \Iggy\MetadataChangeGroup|null + */ + public readonly ?\Iggy\MetadataChangeGroup $group; + + public function __construct() {} + } + + /** + * Opaque continuation of a current state scan. Send it back unchanged in + * `MetadataChangeStart::current()` with the same groups. + * + * Casting it to string gives its `cs1_` text, which `parse()` reads back. + */ + class MetadataCurrentPageToken { + /** + * The cursor captured before the scan's first page. + * + * @var \Iggy\MetadataChangeCursor + */ + public readonly \Iggy\MetadataChangeCursor $anchor; + + /** + * The groups the scan was started with, in code order. + * + * @var array + */ + public readonly array $groups; + + /** + * The group of the last item returned. + * + * @var \Iggy\MetadataChangeGroup + */ + public readonly \Iggy\MetadataChangeGroup $last_group; + + /** + * The key of the last item returned, in the page token key layout. + * + * @var string + */ + public readonly string $last_key; + + /** + * The highest head any page of the scan reported so far. + * + * @var mixed + */ + public readonly mixed $required_head; + + public function __construct() {} + + /** + * @return string + */ + public function __toString(): string {} + + /** + * Reads a page token from its `cs1_` text. Anything else throws `Invalid command` + * (code 3). + * + * @param string $text + * @return \Iggy\MetadataCurrentPageToken + */ + public static function parse(string $text): \Iggy\MetadataCurrentPageToken {} + } + + /** + * A `consumer_group_members` current state item. + */ + class MetadataMemberItem { + /** + * The member's client id as a decimal string. + * + * @var string + */ + public readonly string $client_id; + + public readonly int $group_id; + + public readonly int $member_id; + + /** + * Installed partitions, ascending. + * + * @var array + */ + public readonly array $partition_ids; + + public readonly mixed $stream_created_op; + + public readonly int $stream_id; + + public readonly mixed $topic_created_op; + + public readonly int $topic_id; + + public function __construct() {} + } + + /** + * A member that joined a consumer group. + */ + class MetadataMemberJoined { + /** + * The member's client id as a decimal string. + * + * @var string + */ + public readonly string $client_id; + + public readonly int $group_id; + + public readonly int $member_id; + + public readonly int $members_count; + + /** + * Installed partitions, normally none because a handover installs them. + * + * @var array + */ + public readonly array $partition_ids; + + public readonly mixed $stream_created_op; + + public readonly int $stream_id; + + public readonly mixed $topic_created_op; + + public readonly int $topic_id; + + public function __construct() {} + } + + /** + * A member that left a consumer group. + */ + class MetadataMemberLeft { + /** + * The member's client id as a decimal string. + * + * @var string + */ + public readonly string $client_id; + + public readonly int $group_id; + + public readonly int $member_id; + + public readonly int $members_count; + + public readonly \Iggy\MetadataMemberLeftReason $reason; + + public readonly mixed $stream_created_op; + + public readonly int $stream_id; + + public readonly mixed $topic_created_op; + + public readonly int $topic_id; + + public function __construct() {} + } + + /** + * Why a consumer group member was removed. + */ + enum MetadataMemberLeftReason: string { + case Left = 'left'; + case SessionEnded = 'session_ended'; + } + + /** + * A completed partition handover. + */ + class MetadataPartitionHandedOver { + public readonly int $group_id; + + /** + * The new owner, null when the last member left. + * + * @var int|null + */ + public readonly ?int $member_id; + + public readonly int $partition_id; + + public readonly mixed $stream_created_op; + + public readonly int $stream_id; + + public readonly mixed $topic_created_op; + + public readonly int $topic_id; + + public function __construct() {} + } + + /** + * A partition count change. + */ + class MetadataPartitionsChanged { + /** + * The partition count after the change. + * + * @var int + */ + public readonly int $partitions_count; + + /** + * The deleted partition ids, ascending, empty on create. + * + * @var array + */ + public readonly array $removed_partition_ids; + + /** + * The operation of the delete request when the delete completed later, else null. + * + * @var mixed + */ + public readonly mixed $requested_op; + + public readonly mixed $stream_created_op; + + public readonly int $stream_id; + + public readonly mixed $topic_created_op; + + public readonly int $topic_id; + + public function __construct() {} + } + + /** + * A `partitions` current state item. + */ + class MetadataPartitionsItem { + public readonly int $partitions_count; + + public readonly mixed $stream_created_op; + + public readonly int $stream_id; + + public readonly mixed $topic_created_op; + + public readonly int $topic_id; + + public function __construct() {} + } + + /** + * Why a user's public permission set changed. + */ + enum MetadataPermissionsUpdatedReason: string { + case Updated = 'updated'; + case ResourceDeleted = 'resource_deleted'; + } + + /** + * A registered session, and a `sessions` current state item. + */ + class MetadataSession { + /** + * The session's client id as a decimal string. + * + * @var string + */ + public readonly string $client_id; + + /** + * The session epoch. + * + * @var mixed + */ + public readonly mixed $session; + + public readonly mixed $user_created_at; + + public readonly int $user_id; + + public function __construct() {} + } + + /** + * A closed session. + */ + class MetadataSessionClosed { + /** + * The session's client id as a decimal string. + * + * @var string + */ + public readonly string $client_id; + + public readonly \Iggy\MetadataSessionClosedReason $reason; + + /** + * The session epoch. + * + * @var mixed + */ + public readonly mixed $session; + + public readonly mixed $user_created_at; + + public readonly int $user_id; + + public function __construct() {} + } + + /** + * Why a session was closed. + */ + enum MetadataSessionClosedReason: string { + case Logout = 'logout'; + case Expired = 'expired'; + /** + * The user was deactivated or deleted. + */ + case Revoked = 'revoked'; + } + + /** + * A stream after a create or update, and a `streams` current state item. + */ + class MetadataStreamChange { + /** + * Stream creation time in microseconds. + * + * @var mixed + */ + public readonly mixed $created_at; + + /** + * Options the server filled in, including the key source and fingerprints. + * + * @var array|null + */ + public readonly ?array $derived_options; + + public readonly string $name; + + /** + * Options the creating client set, as `StreamDetails::$options` returns them, or null + * when `options_unreadable`. Key material is never returned. + * + * @var array|null + */ + public readonly ?array $options; + + /** + * The server could not read the stream's stored options. Keep the previous values. + * + * @var bool + */ + public readonly bool $options_unreadable; + + /** + * The operation that created the stream, which tells it from an earlier stream with + * the same id. + * + * @var mixed + */ + public readonly mixed $stream_created_op; + + /** + * Stream id. Ids are reused after a delete. + * + * @var int + */ + public readonly int $stream_id; + + public function __construct() {} + } + + /** + * A removed stream. + */ + class MetadataStreamDeleted { + public readonly mixed $created_at; + + public readonly string $name; + + /** + * The operation of the delete request when the delete completed later, else null. + * + * @var mixed + */ + public readonly mixed $requested_op; + + public readonly mixed $stream_created_op; + + public readonly int $stream_id; + + public function __construct() {} + } + + /** + * A created personal access token, and a `personal_access_tokens` current state item. + * Never the token itself. + */ + class MetadataTokenChange { + /** + * Expiry in microseconds, null for a token that never expires. + * + * @var mixed + */ + public readonly mixed $expiry; + + public readonly string $name; + + public readonly mixed $user_created_at; + + public readonly int $user_id; + + public function __construct() {} + } + + /** + * A removed personal access token. + */ + class MetadataTokenDeleted { + public readonly string $name; + + public readonly \Iggy\MetadataTokenDeletedReason $reason; + + public readonly mixed $user_created_at; + + public readonly int $user_id; + + public function __construct() {} + } + + /** + * Why a personal access token was removed. + */ + enum MetadataTokenDeletedReason: string { + case Deleted = 'deleted'; + case Expired = 'expired'; + } + + /** + * A topic after a create or update. + */ + class MetadataTopicChange { + public readonly mixed $created_at; + + /** + * Options the server filled in from its catalog defaults. + * + * @var array|null + */ + public readonly ?array $derived_options; + + public readonly string $name; + + /** + * Options the creating or updating client set, or null when `options_unreadable`. + * + * @var array|null + */ + public readonly ?array $options; + + /** + * The server could not read the topic's stored options. Keep the previous values. + * + * @var bool + */ + public readonly bool $options_unreadable; + + public readonly int $partitions_count; + + public readonly mixed $stream_created_op; + + public readonly int $stream_id; + + /** + * The operation that created the topic, which tells it from an earlier topic with the + * same id. + * + * @var mixed + */ + public readonly mixed $topic_created_op; + + public readonly int $topic_id; + + public function __construct() {} + } + + /** + * A removed topic. + */ + class MetadataTopicDeleted { + public readonly mixed $created_at; + + public readonly string $name; + + /** + * The operation of the delete request when the delete completed later, else null. + * + * @var mixed + */ + public readonly mixed $requested_op; + + public readonly mixed $stream_created_op; + + public readonly int $stream_id; + + public readonly mixed $topic_created_op; + + public readonly int $topic_id; + + public function __construct() {} + } + + /** + * A `topics` current state item. Partition counts come as `partitions` items. + */ + class MetadataTopicItem { + public readonly mixed $created_at; + + /** + * Options the server filled in from its catalog defaults. + * + * @var array|null + */ + public readonly ?array $derived_options; + + public readonly string $name; + + /** + * Options the creating or updating client set, or null when `options_unreadable`. + * + * @var array|null + */ + public readonly ?array $options; + + /** + * The server could not read the topic's stored options. + * + * @var bool + */ + public readonly bool $options_unreadable; + + public readonly mixed $stream_created_op; + + public readonly int $stream_id; + + public readonly mixed $topic_created_op; + + public readonly int $topic_id; + + public function __construct() {} + } + + /** + * A record of a group or kind this build does not know. A mirror stops on it rather than + * skip a possible state change. + */ + class MetadataUnknownChange { + /** + * The raw body on binary transports, or the record's whole JSON object over HTTP. + * + * @var string + */ + public readonly string $body; + + /** + * The group code on binary transports, or the group name over HTTP. + * + * @var mixed + */ + public readonly mixed $group; + + /** + * The kind code on binary transports, or the kind name over HTTP. + * + * @var mixed + */ + public readonly mixed $kind; + + public function __construct() {} + } + + /** + * A current state item of a group this build does not know. + */ + class MetadataUnknownItem { + /** + * The raw body on binary transports, or the item's whole JSON object over HTTP. + * + * @var string + */ + public readonly string $body; + + /** + * The group code on binary transports, or the group name over HTTP. + * + * @var mixed + */ + public readonly mixed $group; + + public function __construct() {} + } + + /** + * A user after a create or update, and a `users` current state item. + */ + class MetadataUserChange { + /** + * Options the server filled in. + * + * @var array|null + */ + public readonly ?array $derived_options; + + /** + * Options the creating client set, or null when `options_unreadable`. + * + * @var array|null + */ + public readonly ?array $options; + + /** + * The server could not read the user's stored options. + * + * @var bool + */ + public readonly bool $options_unreadable; + + /** + * The public permission set in the shape `GET /users/{id}` returns, null when the + * user has none. + * + * @var array|null + */ + public readonly ?array $permissions; + + /** + * `active` or `inactive`. + * + * @var string + */ + public readonly string $status; + + /** + * User creation time in microseconds, which tells the user from an earlier user with + * the same id. + * + * @var mixed + */ + public readonly mixed $user_created_at; + + /** + * User id. Ids are reused after a delete. + * + * @var int + */ + public readonly int $user_id; + + public readonly string $username; + + public function __construct() {} + } + + /** + * A deleted user, or one whose password changed. + */ + class MetadataUserIdentity { + public readonly mixed $user_created_at; + + public readonly int $user_id; + + public readonly string $username; + + public function __construct() {} + } + + /** + * A user's public permission set after a change. + */ + class MetadataUserPermissionsUpdated { + /** + * The public permission set after the change, null when the user has none. + * + * @var array|null + */ + public readonly ?array $permissions; + + public readonly \Iggy\MetadataPermissionsUpdatedReason $reason; + + public readonly mixed $user_created_at; + + public readonly int $user_id; + + public readonly string $username; + + public function __construct() {} + } + + /** * One option key as the server's catalog describes it. */ class OptionSpec {
diff --git a/foreign/php/src/client.rs b/foreign/php/src/client.rs index 8745fdf..b5b70b1 100644 --- a/foreign/php/src/client.rs +++ b/foreign/php/src/client.rs
@@ -36,6 +36,7 @@ use crate::durability::Durability as PhpDurability; use crate::error::to_php_exception; use crate::identifier::PhpIdentifier; +use crate::metadata_changes::{MetadataChangeGroup, MetadataChangeStart, MetadataChanges}; use crate::options::{OptionSpec, OptionsScope, StreamOptionKey, TopicOptionKey, reset_names}; use crate::receive_message::{PollingStrategy, ReceiveMessage}; use crate::runtime::runtime; @@ -421,6 +422,47 @@ }) } + /// Reads one page of the metadata change log, or of the current state. + /// + /// Needs root or the global `manage_servers` permission. `groups` takes + /// `Iggy\MetadataChangeGroup` cases, empty for every group, and a current state scan + /// needs the parent groups of each group. `since` and `until` are microseconds as an int + /// or a decimal string, and only apply to `earliest` and `after`. `limit` is 1 to 1000, + /// 100 when null. A query the server would refuse throws `Invalid command` (code 3) + /// before anything is sent. + #[php(defaults(groups = Vec::<MetadataChangeGroup>::new()))] + pub fn get_metadata_changes( + &self, + start: &MetadataChangeStart, + groups: Vec<MetadataChangeGroup>, + since: Option<&Zval>, + until: Option<&Zval>, + limit: Option<&Zval>, + ) -> PhpResult<MetadataChanges> { + let timestamp = |micros: u64| { + IggyTimestamp::checked_from_micros(micros) + .ok_or_else(|| to_php_exception(IggyError::InvalidCommand)) + }; + let query = MetadataChangesQuery { + start: start.inner.clone(), + groups: groups.into_iter().map(Into::into).collect(), + since: optional_u64(since, "since")?.map(timestamp).transpose()?, + until: optional_u64(until, "until")?.map(timestamp).transpose()?, + limit: optional_unsigned(limit, "limit")? + .unwrap_or(MetadataChangesQuery::default().limit), + }; + query.validate().map_err(to_php_exception)?; + let inner = self.inner.clone(); + + runtime().block_on(async move { + inner + .get_metadata_changes(&query) + .await + .map(MetadataChanges::from) + .map_err(to_php_exception) + }) + } + /// Gets a topic by stream and topic id/name. pub fn get_topic( &self, @@ -697,9 +739,13 @@ // u64 values above PHP_INT_MAX, such as the never-expire sentinel, only fit a decimal string. fn optional_u64(value: Option<&Zval>, field: &str) -> PhpResult<Option<u64>> { - let Some(value) = value.filter(|value| !value.is_null()) else { - return Ok(None); - }; + value + .filter(|value| !value.is_null()) + .map(|value| unsigned_argument(value, field)) + .transpose() +} + +pub(crate) fn unsigned_argument(value: &Zval, field: &str) -> PhpResult<u64> { let number = match (value.long(), value.str()) { (Some(number), _) => u64::try_from(number).ok(), (None, Some(text)) @@ -709,9 +755,9 @@ } _ => None, }; - number.map(Some).ok_or_else(|| { + number.ok_or_else(|| { to_php_exception(format!( - "'{field}' must be a non-negative integer, a decimal string or null" + "'{field}' must be a non-negative integer or a decimal string" )) }) }
diff --git a/foreign/php/src/lib.rs b/foreign/php/src/lib.rs index 9cc6c5f..b9f860f 100644 --- a/foreign/php/src/lib.rs +++ b/foreign/php/src/lib.rs
@@ -21,6 +21,7 @@ pub mod error; pub mod identifier; pub mod message_iterator; +pub mod metadata_changes; pub mod options; pub mod receive_message; pub mod runtime; @@ -38,6 +39,18 @@ PendingOperationException, TransientException, }; use crate::message_iterator::MessageIterator; +use crate::metadata_changes::{ + MetadataChangeCursor, MetadataChangeGroup, MetadataChangeKind, MetadataChangePhase, + MetadataChangeRecord, MetadataChangeStart, MetadataChangeStatus, MetadataChanges, + MetadataConsumerGroupCreated, MetadataConsumerGroupDeleted, MetadataConsumerGroupItem, + MetadataCurrentItem, MetadataCurrentPageToken, MetadataMemberItem, MetadataMemberJoined, + MetadataMemberLeft, MetadataMemberLeftReason, MetadataPartitionHandedOver, + MetadataPartitionsChanged, MetadataPartitionsItem, MetadataPermissionsUpdatedReason, + MetadataSession, MetadataSessionClosed, MetadataSessionClosedReason, MetadataStreamChange, + MetadataStreamDeleted, MetadataTokenChange, MetadataTokenDeleted, MetadataTokenDeletedReason, + MetadataTopicChange, MetadataTopicDeleted, MetadataTopicItem, MetadataUnknownChange, + MetadataUnknownItem, MetadataUserChange, MetadataUserIdentity, MetadataUserPermissionsUpdated, +}; use crate::options::{OptionSpec, OptionsScope, StreamOptionKey, TopicOptionKey}; use crate::receive_message::{ConsumerPosition, PartitionContext, PollingStrategy, ReceiveMessage}; use crate::send_message::{SendMessage, SendMessagesConfirmation, SendMessagesResponse}; @@ -52,6 +65,14 @@ .enumeration::<StreamOptionKey>() .enumeration::<TopicOptionKey>() .enumeration::<OptionsScope>() + .enumeration::<MetadataChangeGroup>() + .enumeration::<MetadataChangeKind>() + .enumeration::<MetadataChangeStatus>() + .enumeration::<MetadataChangePhase>() + .enumeration::<MetadataMemberLeftReason>() + .enumeration::<MetadataTokenDeletedReason>() + .enumeration::<MetadataSessionClosedReason>() + .enumeration::<MetadataPermissionsUpdatedReason>() // Parent classes must be registered before subclasses because ext-php-rs resolves // the parent ClassEntry during child registration. .class::<IggyException>() @@ -76,4 +97,33 @@ .class::<StreamDetails>() .class::<TopicDetails>() .class::<OptionSpec>() + .class::<MetadataChanges>() + .class::<MetadataChangeRecord>() + .class::<MetadataCurrentItem>() + .class::<MetadataChangeCursor>() + .class::<MetadataCurrentPageToken>() + .class::<MetadataChangeStart>() + .class::<MetadataStreamChange>() + .class::<MetadataStreamDeleted>() + .class::<MetadataTopicChange>() + .class::<MetadataTopicItem>() + .class::<MetadataTopicDeleted>() + .class::<MetadataPartitionsChanged>() + .class::<MetadataPartitionsItem>() + .class::<MetadataConsumerGroupCreated>() + .class::<MetadataConsumerGroupDeleted>() + .class::<MetadataConsumerGroupItem>() + .class::<MetadataMemberJoined>() + .class::<MetadataMemberLeft>() + .class::<MetadataPartitionHandedOver>() + .class::<MetadataMemberItem>() + .class::<MetadataUserChange>() + .class::<MetadataUserIdentity>() + .class::<MetadataUserPermissionsUpdated>() + .class::<MetadataTokenChange>() + .class::<MetadataTokenDeleted>() + .class::<MetadataSession>() + .class::<MetadataSessionClosed>() + .class::<MetadataUnknownChange>() + .class::<MetadataUnknownItem>() }
diff --git a/foreign/php/src/metadata_changes.rs b/foreign/php/src/metadata_changes.rs new file mode 100644 index 0000000..a418dd4 --- /dev/null +++ b/foreign/php/src/metadata_changes.rs
@@ -0,0 +1,2005 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use std::str::FromStr; + +use ext_php_rs::{ + binary::Binary, + boxed::ZBox, + convert::{IntoZval, StubLiteral}, + enum_::RegisteredEnum, + error::Result as ConversionResult, + exception::PhpResult, + flags::DataType, + php_class, php_enum, php_impl, + types::{ZendHashTable, Zval}, +}; +use iggy::prelude::{ + IggyError, MetadataChange as RustMetadataChange, + MetadataChangeCursor as RustMetadataChangeCursor, + MetadataChangeGroup as RustMetadataChangeGroup, MetadataChangeKind as RustMetadataChangeKind, + MetadataChangePhase as RustMetadataChangePhase, + MetadataChangeRecord as RustMetadataChangeRecord, + MetadataChangeStart as RustMetadataChangeStart, + MetadataChangeStatus as RustMetadataChangeStatus, MetadataChanges as RustMetadataChanges, + MetadataConsumerGroupCreated as RustMetadataConsumerGroupCreated, + MetadataConsumerGroupDeleted as RustMetadataConsumerGroupDeleted, + MetadataConsumerGroupItem as RustMetadataConsumerGroupItem, + MetadataCurrentItem as RustMetadataCurrentItem, + MetadataCurrentPageToken as RustMetadataCurrentPageToken, + MetadataMemberItem as RustMetadataMemberItem, MetadataMemberJoined as RustMetadataMemberJoined, + MetadataMemberLeft as RustMetadataMemberLeft, + MetadataMemberLeftReason as RustMetadataMemberLeftReason, MetadataOptions, + MetadataPartitionHandedOver as RustMetadataPartitionHandedOver, + MetadataPartitionsChanged as RustMetadataPartitionsChanged, + MetadataPartitionsItem as RustMetadataPartitionsItem, + MetadataPermissionsUpdatedReason as RustMetadataPermissionsUpdatedReason, + MetadataSession as RustMetadataSession, MetadataSessionClosed as RustMetadataSessionClosed, + MetadataSessionClosedReason as RustMetadataSessionClosedReason, + MetadataStreamChange as RustMetadataStreamChange, + MetadataStreamDeleted as RustMetadataStreamDeleted, + MetadataTokenChange as RustMetadataTokenChange, + MetadataTokenDeleted as RustMetadataTokenDeleted, + MetadataTokenDeletedReason as RustMetadataTokenDeletedReason, + MetadataTopicChange as RustMetadataTopicChange, + MetadataTopicDeleted as RustMetadataTopicDeleted, MetadataTopicItem as RustMetadataTopicItem, + MetadataUserChange as RustMetadataUserChange, MetadataUserIdentity as RustMetadataUserIdentity, + MetadataUserPermissionsUpdated as RustMetadataUserPermissionsUpdated, Permissions, + StreamPermissions, +}; +use iggy_binary_protocol::WireDecode; +use iggy_binary_protocol::responses::system::GetMetadataChangesResponse; + +use crate::client::unsigned_argument; +use crate::error::to_php_exception; +use crate::options::options_array; + +/// One page of the metadata change log, or of the current state, as +/// `Client::getMetadataChanges()` returns it. +/// +/// Unsigned 64-bit values are ints, or decimal strings above `PHP_INT_MAX`. 128-bit values +/// (log ids, checksums, client ids) are decimal strings. +#[php_class] +#[php(name = "Iggy\\MetadataChanges")] +pub struct MetadataChanges { + inner: RustMetadataChanges, +} + +/// One change record: the committed operation that produced it, its position within that +/// operation and the typed body of what is true after it. +#[php_class] +#[php(name = "Iggy\\MetadataChangeRecord")] +pub struct MetadataChangeRecord { + inner: RustMetadataChangeRecord, +} + +/// One current state item of a `MetadataChangeStart::current()` page. +#[php_class] +#[php(name = "Iggy\\MetadataCurrentItem")] +pub struct MetadataCurrentItem { + inner: RustMetadataCurrentItem, +} + +/// Position of the last change record a reader consumed or scanned. Send it back in +/// `MetadataChangeStart::after()` to read the records after it. +/// +/// Casting it to string gives its `mc1_` text, which `parse()` reads back. +#[php_class] +#[php(name = "Iggy\\MetadataChangeCursor")] +#[derive(Clone, Copy)] +pub struct MetadataChangeCursor { + inner: RustMetadataChangeCursor, +} + +/// Opaque continuation of a current state scan. Send it back unchanged in +/// `MetadataChangeStart::current()` with the same groups. +/// +/// Casting it to string gives its `cs1_` text, which `parse()` reads back. +#[php_class] +#[php(name = "Iggy\\MetadataCurrentPageToken")] +#[derive(Clone)] +pub struct MetadataCurrentPageToken { + inner: RustMetadataCurrentPageToken, +} + +/// Where `Client::getMetadataChanges()` starts reading. +#[php_class] +#[php(name = "Iggy\\MetadataChangeStart")] +#[derive(Clone)] +pub struct MetadataChangeStart { + pub(crate) inner: RustMetadataChangeStart, +} + +/// Family of change records and the unit of filtering. Values are the HTTP API names. +#[php_enum] +#[php(name = "Iggy\\MetadataChangeGroup")] +#[derive(Clone, Copy)] +pub enum MetadataChangeGroup { + #[php(value = "streams")] + Streams, + #[php(value = "topics")] + Topics, + #[php(value = "partitions")] + Partitions, + #[php(value = "consumer_groups")] + ConsumerGroups, + #[php(value = "consumer_group_members")] + ConsumerGroupMembers, + #[php(value = "users")] + Users, + #[php(value = "personal_access_tokens")] + PersonalAccessTokens, + #[php(value = "sessions")] + Sessions, +} + +/// Kind of a change record. Values are the HTTP API names. +#[php_enum] +#[php(name = "Iggy\\MetadataChangeKind")] +#[derive(Clone, Copy)] +pub enum MetadataChangeKind { + #[php(value = "stream_created")] + StreamCreated, + #[php(value = "stream_updated")] + StreamUpdated, + #[php(value = "stream_deleted")] + StreamDeleted, + #[php(value = "topic_created")] + TopicCreated, + #[php(value = "topic_updated")] + TopicUpdated, + #[php(value = "topic_deleted")] + TopicDeleted, + #[php(value = "partitions_changed")] + PartitionsChanged, + #[php(value = "consumer_group_created")] + ConsumerGroupCreated, + #[php(value = "consumer_group_deleted")] + ConsumerGroupDeleted, + #[php(value = "consumer_group_member_joined")] + ConsumerGroupMemberJoined, + #[php(value = "consumer_group_member_left")] + ConsumerGroupMemberLeft, + #[php(value = "consumer_group_partition_handed_over")] + ConsumerGroupPartitionHandedOver, + #[php(value = "user_created")] + UserCreated, + #[php(value = "user_updated")] + UserUpdated, + #[php(value = "user_deleted")] + UserDeleted, + #[php(value = "user_password_changed")] + UserPasswordChanged, + #[php(value = "user_permissions_updated")] + UserPermissionsUpdated, + #[php(value = "personal_access_token_created")] + PersonalAccessTokenCreated, + #[php(value = "personal_access_token_deleted")] + PersonalAccessTokenDeleted, + #[php(value = "session_registered")] + SessionRegistered, + #[php(value = "session_closed")] + SessionClosed, +} + +/// Outcome of a change log query. +/// +/// `Ok`: apply the records and keep `next`. `Behind`: the node has not applied the position +/// yet, ask again later or ask another node. `Expired` and `Reset`: rebuild from current state. +#[php_enum] +#[php(name = "Iggy\\MetadataChangeStatus")] +#[derive(Clone, Copy)] +pub enum MetadataChangeStatus { + #[php(value = "ok")] + Ok, + #[php(value = "behind")] + Behind, + #[php(value = "expired")] + Expired, + #[php(value = "reset")] + Reset, +} + +/// Whether a page carries change records or current state items. +#[php_enum] +#[php(name = "Iggy\\MetadataChangePhase")] +#[derive(Clone, Copy)] +pub enum MetadataChangePhase { + #[php(value = "changes")] + Changes, + #[php(value = "current")] + Current, +} + +/// Why a consumer group member was removed. +#[php_enum] +#[php(name = "Iggy\\MetadataMemberLeftReason")] +#[derive(Clone, Copy)] +pub enum MetadataMemberLeftReason { + #[php(value = "left")] + Left, + #[php(value = "session_ended")] + SessionEnded, +} + +/// Why a personal access token was removed. +#[php_enum] +#[php(name = "Iggy\\MetadataTokenDeletedReason")] +#[derive(Clone, Copy)] +pub enum MetadataTokenDeletedReason { + #[php(value = "deleted")] + Deleted, + #[php(value = "expired")] + Expired, +} + +/// Why a session was closed. +#[php_enum] +#[php(name = "Iggy\\MetadataSessionClosedReason")] +#[derive(Clone, Copy)] +pub enum MetadataSessionClosedReason { + #[php(value = "logout")] + Logout, + #[php(value = "expired")] + Expired, + /// The user was deactivated or deleted. + #[php(value = "revoked")] + Revoked, +} + +/// Why a user's public permission set changed. +#[php_enum] +#[php(name = "Iggy\\MetadataPermissionsUpdatedReason")] +#[derive(Clone, Copy)] +pub enum MetadataPermissionsUpdatedReason { + #[php(value = "updated")] + Updated, + #[php(value = "resource_deleted")] + ResourceDeleted, +} + +/// A stream after a create or update, and a `streams` current state item. +#[php_class] +#[php(name = "Iggy\\MetadataStreamChange")] +pub struct MetadataStreamChange { + inner: RustMetadataStreamChange, +} + +/// A removed stream. +#[php_class] +#[php(name = "Iggy\\MetadataStreamDeleted")] +pub struct MetadataStreamDeleted { + inner: RustMetadataStreamDeleted, +} + +/// A topic after a create or update. +#[php_class] +#[php(name = "Iggy\\MetadataTopicChange")] +pub struct MetadataTopicChange { + inner: RustMetadataTopicChange, +} + +/// A `topics` current state item. Partition counts come as `partitions` items. +#[php_class] +#[php(name = "Iggy\\MetadataTopicItem")] +pub struct MetadataTopicItem { + inner: RustMetadataTopicItem, +} + +/// A removed topic. +#[php_class] +#[php(name = "Iggy\\MetadataTopicDeleted")] +pub struct MetadataTopicDeleted { + inner: RustMetadataTopicDeleted, +} + +/// A partition count change. +#[php_class] +#[php(name = "Iggy\\MetadataPartitionsChanged")] +pub struct MetadataPartitionsChanged { + inner: RustMetadataPartitionsChanged, +} + +/// A `partitions` current state item. +#[php_class] +#[php(name = "Iggy\\MetadataPartitionsItem")] +pub struct MetadataPartitionsItem { + inner: RustMetadataPartitionsItem, +} + +/// A created consumer group with the topic's resulting group count. +#[php_class] +#[php(name = "Iggy\\MetadataConsumerGroupCreated")] +pub struct MetadataConsumerGroupCreated { + inner: RustMetadataConsumerGroupCreated, +} + +/// A removed consumer group with the topic's resulting group count. +#[php_class] +#[php(name = "Iggy\\MetadataConsumerGroupDeleted")] +pub struct MetadataConsumerGroupDeleted { + inner: RustMetadataConsumerGroupDeleted, +} + +/// A `consumer_groups` current state item. +#[php_class] +#[php(name = "Iggy\\MetadataConsumerGroupItem")] +pub struct MetadataConsumerGroupItem { + inner: RustMetadataConsumerGroupItem, +} + +/// A member that joined a consumer group. +#[php_class] +#[php(name = "Iggy\\MetadataMemberJoined")] +pub struct MetadataMemberJoined { + inner: RustMetadataMemberJoined, +} + +/// A member that left a consumer group. +#[php_class] +#[php(name = "Iggy\\MetadataMemberLeft")] +pub struct MetadataMemberLeft { + inner: RustMetadataMemberLeft, +} + +/// A completed partition handover. +#[php_class] +#[php(name = "Iggy\\MetadataPartitionHandedOver")] +pub struct MetadataPartitionHandedOver { + inner: RustMetadataPartitionHandedOver, +} + +/// A `consumer_group_members` current state item. +#[php_class] +#[php(name = "Iggy\\MetadataMemberItem")] +pub struct MetadataMemberItem { + inner: RustMetadataMemberItem, +} + +/// A user after a create or update, and a `users` current state item. +#[php_class] +#[php(name = "Iggy\\MetadataUserChange")] +pub struct MetadataUserChange { + inner: RustMetadataUserChange, +} + +/// A deleted user, or one whose password changed. +#[php_class] +#[php(name = "Iggy\\MetadataUserIdentity")] +pub struct MetadataUserIdentity { + inner: RustMetadataUserIdentity, +} + +/// A user's public permission set after a change. +#[php_class] +#[php(name = "Iggy\\MetadataUserPermissionsUpdated")] +pub struct MetadataUserPermissionsUpdated { + inner: RustMetadataUserPermissionsUpdated, +} + +/// A created personal access token, and a `personal_access_tokens` current state item. +/// Never the token itself. +#[php_class] +#[php(name = "Iggy\\MetadataTokenChange")] +pub struct MetadataTokenChange { + inner: RustMetadataTokenChange, +} + +/// A removed personal access token. +#[php_class] +#[php(name = "Iggy\\MetadataTokenDeleted")] +pub struct MetadataTokenDeleted { + inner: RustMetadataTokenDeleted, +} + +/// A registered session, and a `sessions` current state item. +#[php_class] +#[php(name = "Iggy\\MetadataSession")] +pub struct MetadataSession { + inner: RustMetadataSession, +} + +/// A closed session. +#[php_class] +#[php(name = "Iggy\\MetadataSessionClosed")] +pub struct MetadataSessionClosed { + inner: RustMetadataSessionClosed, +} + +/// A record of a group or kind this build does not know. A mirror stops on it rather than +/// skip a possible state change. +#[php_class] +#[php(name = "Iggy\\MetadataUnknownChange")] +pub struct MetadataUnknownChange { + group: UnknownCode, + kind: UnknownCode, + body: Vec<u8>, +} + +/// A current state item of a group this build does not know. +#[php_class] +#[php(name = "Iggy\\MetadataUnknownItem")] +pub struct MetadataUnknownItem { + group: UnknownCode, + body: Vec<u8>, +} + +#[php_impl] +impl MetadataChanges { + /// Decodes the response payload of the raw `metadata.changes` command (code 17), as + /// `Client::sendBinaryRequest()` returns it. + pub fn decode(payload: Binary<u8>) -> PhpResult<Self> { + let payload: Vec<u8> = payload.into(); + let (response, consumed) = GetMetadataChangesResponse::decode(&payload) + .map_err(|_| to_php_exception(IggyError::InvalidFormat))?; + if consumed != payload.len() { + return Err(to_php_exception(IggyError::InvalidFormat)); + } + RustMetadataChanges::try_from(response) + .map(|inner| Self { inner }) + .map_err(to_php_exception) + } + + /// Reads the JSON body `GET /metadata/changes` returns. + pub fn from_json(json: String) -> PhpResult<Self> { + serde_json::from_str(&json) + .map(|inner| Self { inner }) + .map_err(|_| to_php_exception(IggyError::InvalidJsonResponse)) + } + + #[php(getter)] + pub fn status(&self) -> MetadataChangeStatus { + self.inner.status.into() + } + + /// `Changes` for `earliest`, `latest` and `after`, `Current` for state pages. + #[php(getter)] + pub fn phase(&self) -> MetadataChangePhase { + self.inner.phase.into() + } + + /// This node's log id as a decimal string, null before it applied op 1. + #[php(getter, change_case = "snake_case")] + pub fn log_id(&self) -> Option<String> { + self.inner.log_id.map(|log_id| log_id.to_string()) + } + + /// Lowest whole operation from which the requested groups are complete. + #[php(getter, change_case = "snake_case")] + pub fn oldest_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.oldest_op) + } + + /// The requested groups are complete for every timestamp above this value in + /// microseconds, 0 when nothing was evicted. + #[php(getter, change_case = "snake_case")] + pub fn retained_since(&self) -> PhpResult<Zval> { + unsigned(self.inner.retained_since) + } + + /// Highest metadata operation the answering node applied. + #[php(getter, change_case = "snake_case")] + pub fn head_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.head_op) + } + + /// The cursor to send back. Present on `Ok`, echoed on `Behind`, null on `Expired` and + /// `Reset`. The last state page returns the anchor to follow from. + #[php(getter)] + pub fn next(&self) -> Option<MetadataChangeCursor> { + self.inner.next.map(|inner| MetadataChangeCursor { inner }) + } + + /// More entries remain after `next`, or another state page exists. + #[php(getter, change_case = "snake_case")] + pub fn has_more(&self) -> bool { + self.inner.has_more + } + + /// Change records in `(op, ordinal)` order, empty on state pages. + #[php(getter)] + pub fn records(&self) -> Vec<MetadataChangeRecord> { + self.inner + .records + .iter() + .map(|record| MetadataChangeRecord { + inner: record.clone(), + }) + .collect() + } + + /// Current state items, empty on change pages. + #[php(getter)] + pub fn items(&self) -> Vec<MetadataCurrentItem> { + self.inner + .items + .iter() + .map(|item| MetadataCurrentItem { + inner: item.clone(), + }) + .collect() + } + + /// Continuation of a current state scan, null once it is complete. + #[php(getter, change_case = "snake_case")] + pub fn current_page_token(&self) -> Option<MetadataCurrentPageToken> { + self.inner + .current_page_token + .clone() + .map(|inner| MetadataCurrentPageToken { inner }) + } +} + +#[php_impl] +impl MetadataChangeRecord { + /// The metadata operation that produced the record. + #[php(getter)] + pub fn op(&self) -> PhpResult<Zval> { + unsigned(self.inner.op) + } + + /// Position of the record within `op`. + #[php(getter)] + pub fn ordinal(&self) -> u32 { + self.inner.ordinal + } + + /// Prepare checksum of `op` as a decimal string. + #[php(getter)] + pub fn checksum(&self) -> String { + self.inner.checksum.to_string() + } + + /// Prepare timestamp of `op` in microseconds. + #[php(getter)] + pub fn timestamp(&self) -> PhpResult<Zval> { + unsigned(self.inner.timestamp) + } + + /// The acting user, null for server actions. + #[php(getter, change_case = "snake_case")] + pub fn actor_user_id(&self) -> Option<u32> { + self.inner.actor_user_id + } + + /// The group, null for a record this build does not know. + #[php(getter)] + pub fn group(&self) -> Option<MetadataChangeGroup> { + self.inner.change.group().map(Into::into) + } + + /// The kind, null for a record this build does not know. + #[php(getter)] + pub fn kind(&self) -> Option<MetadataChangeKind> { + self.inner.change.kind().map(Into::into) + } + + /// The typed body `kind` names, for example a `MetadataStreamChange` for + /// `StreamCreated` and `StreamUpdated`, or a `MetadataUnknownChange`. + #[php(getter)] + pub fn change(&self) -> PhpResult<Zval> { + change_body(&self.inner.change) + } +} + +#[php_impl] +impl MetadataCurrentItem { + /// The group, null for an item this build does not know. + #[php(getter)] + pub fn group(&self) -> Option<MetadataChangeGroup> { + self.inner.group().map(Into::into) + } + + /// The typed body `group` names, or a `MetadataUnknownItem`. + #[php(getter)] + pub fn body(&self) -> PhpResult<Zval> { + item_body(&self.inner) + } +} + +#[php_impl] +impl MetadataChangeCursor { + /// Builds a cursor from its fields: 128-bit values as decimal strings, `op` as an int or + /// a decimal string. + #[php(constructor)] + pub fn __construct( + log_id: String, + op: &Zval, + checksum: String, + ordinal: u32, + ) -> PhpResult<Self> { + Ok(Self { + inner: RustMetadataChangeCursor { + log_id: decimal_u128(&log_id, "log_id")?, + op: unsigned_argument(op, "op")?, + checksum: decimal_u128(&checksum, "checksum")?, + ordinal, + }, + }) + } + + /// Reads a cursor from its `mc1_` text. Anything else throws `Invalid command` (code 3). + pub fn parse(text: String) -> PhpResult<Self> { + RustMetadataChangeCursor::from_str(&text) + .map(|inner| Self { inner }) + .map_err(to_php_exception) + } + + /// Prepare checksum of op 1 as a decimal string, which names the history the cursor + /// belongs to. + #[php(getter, change_case = "snake_case")] + pub fn log_id(&self) -> String { + self.inner.log_id.to_string() + } + + /// The operation of the last record, 0 before the first record. + #[php(getter)] + pub fn op(&self) -> PhpResult<Zval> { + unsigned(self.inner.op) + } + + /// Prepare checksum of `op` as a decimal string. + #[php(getter)] + pub fn checksum(&self) -> String { + self.inner.checksum.to_string() + } + + /// Position of the record within `op`. + #[php(getter)] + pub fn ordinal(&self) -> u32 { + self.inner.ordinal + } + + pub fn __to_string(&self) -> String { + self.inner.to_string() + } +} + +#[php_impl] +impl MetadataCurrentPageToken { + /// Reads a page token from its `cs1_` text. Anything else throws `Invalid command` + /// (code 3). + pub fn parse(text: String) -> PhpResult<Self> { + RustMetadataCurrentPageToken::from_str(&text) + .map(|inner| Self { inner }) + .map_err(to_php_exception) + } + + /// The cursor captured before the scan's first page. + #[php(getter)] + pub fn anchor(&self) -> MetadataChangeCursor { + MetadataChangeCursor { + inner: self.inner.anchor(), + } + } + + /// The highest head any page of the scan reported so far. + #[php(getter, change_case = "snake_case")] + pub fn required_head(&self) -> PhpResult<Zval> { + unsigned(self.inner.required_head()) + } + + /// The groups the scan was started with, in code order. + #[php(getter)] + pub fn groups(&self) -> Vec<MetadataChangeGroup> { + self.inner.groups().iter().map(Into::into).collect() + } + + /// The group of the last item returned. + #[php(getter, change_case = "snake_case")] + pub fn last_group(&self) -> MetadataChangeGroup { + self.inner.last_group().into() + } + + /// The key of the last item returned, in the page token key layout. + #[php(getter, change_case = "snake_case")] + pub fn last_key(&self) -> Binary<u8> { + Binary::new(self.inner.last_key().to_vec()) + } + + pub fn __to_string(&self) -> String { + self.inner.to_string() + } +} + +#[php_impl] +impl MetadataChangeStart { + /// The oldest operation the requested groups still hold completely. + pub fn earliest() -> Self { + Self { + inner: RustMetadataChangeStart::Earliest, + } + } + + /// No records, only a cursor at the newest retained record. + pub fn latest() -> Self { + Self { + inner: RustMetadataChangeStart::Latest, + } + } + + /// The records after `cursor`. + pub fn after(cursor: &MetadataChangeCursor) -> Self { + Self { + inner: RustMetadataChangeStart::After(cursor.inner), + } + } + + /// Current state pages: null starts a scan, a token continues it. + pub fn current(page_token: Option<&MetadataCurrentPageToken>) -> Self { + Self { + inner: RustMetadataChangeStart::Current(page_token.map(|token| token.inner.clone())), + } + } +} + +#[php_impl] +impl MetadataStreamChange { + /// Stream id. Ids are reused after a delete. + #[php(getter, change_case = "snake_case")] + pub fn stream_id(&self) -> u32 { + self.inner.stream_id + } + + /// The operation that created the stream, which tells it from an earlier stream with + /// the same id. + #[php(getter, change_case = "snake_case")] + pub fn stream_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.stream_created_op) + } + + /// Stream creation time in microseconds. + #[php(getter, change_case = "snake_case")] + pub fn created_at(&self) -> PhpResult<Zval> { + unsigned(self.inner.created_at) + } + + #[php(getter)] + pub fn name(&self) -> String { + self.inner.name.clone() + } + + /// Options the creating client set, as `StreamDetails::$options` returns them, or null + /// when `options_unreadable`. Key material is never returned. + #[php(getter)] + pub fn options(&self) -> PhpResult<Option<ZBox<ZendHashTable>>> { + options_view(&self.inner.options, true) + } + + /// Options the server filled in, including the key source and fingerprints. + #[php(getter, change_case = "snake_case")] + pub fn derived_options(&self) -> PhpResult<Option<ZBox<ZendHashTable>>> { + options_view(&self.inner.options, false) + } + + /// The server could not read the stream's stored options. Keep the previous values. + #[php(getter, change_case = "snake_case")] + pub fn options_unreadable(&self) -> bool { + matches!(self.inner.options, MetadataOptions::Unreadable) + } +} + +#[php_impl] +impl MetadataStreamDeleted { + #[php(getter, change_case = "snake_case")] + pub fn stream_id(&self) -> u32 { + self.inner.stream_id + } + + #[php(getter, change_case = "snake_case")] + pub fn stream_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.stream_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn created_at(&self) -> PhpResult<Zval> { + unsigned(self.inner.created_at) + } + + #[php(getter)] + pub fn name(&self) -> String { + self.inner.name.clone() + } + + /// The operation of the delete request when the delete completed later, else null. + #[php(getter, change_case = "snake_case")] + pub fn requested_op(&self) -> PhpResult<Zval> { + optional_unsigned(self.inner.requested_op) + } +} + +#[php_impl] +impl MetadataTopicChange { + #[php(getter, change_case = "snake_case")] + pub fn stream_id(&self) -> u32 { + self.inner.stream_id + } + + #[php(getter, change_case = "snake_case")] + pub fn stream_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.stream_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_id(&self) -> u32 { + self.inner.topic_id + } + + /// The operation that created the topic, which tells it from an earlier topic with the + /// same id. + #[php(getter, change_case = "snake_case")] + pub fn topic_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.topic_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn created_at(&self) -> PhpResult<Zval> { + unsigned(self.inner.created_at) + } + + #[php(getter)] + pub fn name(&self) -> String { + self.inner.name.clone() + } + + #[php(getter, change_case = "snake_case")] + pub fn partitions_count(&self) -> u32 { + self.inner.partitions_count + } + + /// Options the creating or updating client set, or null when `options_unreadable`. + #[php(getter)] + pub fn options(&self) -> PhpResult<Option<ZBox<ZendHashTable>>> { + options_view(&self.inner.options, true) + } + + /// Options the server filled in from its catalog defaults. + #[php(getter, change_case = "snake_case")] + pub fn derived_options(&self) -> PhpResult<Option<ZBox<ZendHashTable>>> { + options_view(&self.inner.options, false) + } + + /// The server could not read the topic's stored options. Keep the previous values. + #[php(getter, change_case = "snake_case")] + pub fn options_unreadable(&self) -> bool { + matches!(self.inner.options, MetadataOptions::Unreadable) + } +} + +#[php_impl] +impl MetadataTopicItem { + #[php(getter, change_case = "snake_case")] + pub fn stream_id(&self) -> u32 { + self.inner.stream_id + } + + #[php(getter, change_case = "snake_case")] + pub fn stream_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.stream_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_id(&self) -> u32 { + self.inner.topic_id + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.topic_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn created_at(&self) -> PhpResult<Zval> { + unsigned(self.inner.created_at) + } + + #[php(getter)] + pub fn name(&self) -> String { + self.inner.name.clone() + } + + /// Options the creating or updating client set, or null when `options_unreadable`. + #[php(getter)] + pub fn options(&self) -> PhpResult<Option<ZBox<ZendHashTable>>> { + options_view(&self.inner.options, true) + } + + /// Options the server filled in from its catalog defaults. + #[php(getter, change_case = "snake_case")] + pub fn derived_options(&self) -> PhpResult<Option<ZBox<ZendHashTable>>> { + options_view(&self.inner.options, false) + } + + /// The server could not read the topic's stored options. + #[php(getter, change_case = "snake_case")] + pub fn options_unreadable(&self) -> bool { + matches!(self.inner.options, MetadataOptions::Unreadable) + } +} + +#[php_impl] +impl MetadataTopicDeleted { + #[php(getter, change_case = "snake_case")] + pub fn stream_id(&self) -> u32 { + self.inner.stream_id + } + + #[php(getter, change_case = "snake_case")] + pub fn stream_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.stream_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_id(&self) -> u32 { + self.inner.topic_id + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.topic_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn created_at(&self) -> PhpResult<Zval> { + unsigned(self.inner.created_at) + } + + #[php(getter)] + pub fn name(&self) -> String { + self.inner.name.clone() + } + + /// The operation of the delete request when the delete completed later, else null. + #[php(getter, change_case = "snake_case")] + pub fn requested_op(&self) -> PhpResult<Zval> { + optional_unsigned(self.inner.requested_op) + } +} + +#[php_impl] +impl MetadataPartitionsChanged { + #[php(getter, change_case = "snake_case")] + pub fn stream_id(&self) -> u32 { + self.inner.stream_id + } + + #[php(getter, change_case = "snake_case")] + pub fn stream_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.stream_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_id(&self) -> u32 { + self.inner.topic_id + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.topic_created_op) + } + + /// The partition count after the change. + #[php(getter, change_case = "snake_case")] + pub fn partitions_count(&self) -> u32 { + self.inner.partitions_count + } + + /// The deleted partition ids, ascending, empty on create. + #[php(getter, change_case = "snake_case")] + pub fn removed_partition_ids(&self) -> Vec<u32> { + self.inner.removed_partition_ids.clone() + } + + /// The operation of the delete request when the delete completed later, else null. + #[php(getter, change_case = "snake_case")] + pub fn requested_op(&self) -> PhpResult<Zval> { + optional_unsigned(self.inner.requested_op) + } +} + +#[php_impl] +impl MetadataPartitionsItem { + #[php(getter, change_case = "snake_case")] + pub fn stream_id(&self) -> u32 { + self.inner.stream_id + } + + #[php(getter, change_case = "snake_case")] + pub fn stream_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.stream_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_id(&self) -> u32 { + self.inner.topic_id + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.topic_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn partitions_count(&self) -> u32 { + self.inner.partitions_count + } +} + +#[php_impl] +impl MetadataConsumerGroupCreated { + #[php(getter, change_case = "snake_case")] + pub fn stream_id(&self) -> u32 { + self.inner.stream_id + } + + #[php(getter, change_case = "snake_case")] + pub fn stream_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.stream_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_id(&self) -> u32 { + self.inner.topic_id + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.topic_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn group_id(&self) -> u32 { + self.inner.group_id + } + + #[php(getter)] + pub fn name(&self) -> String { + self.inner.name.clone() + } + + /// The topic's consumer group count after the change. + #[php(getter, change_case = "snake_case")] + pub fn consumer_groups_count(&self) -> u32 { + self.inner.consumer_groups_count + } +} + +#[php_impl] +impl MetadataConsumerGroupDeleted { + #[php(getter, change_case = "snake_case")] + pub fn stream_id(&self) -> u32 { + self.inner.stream_id + } + + #[php(getter, change_case = "snake_case")] + pub fn stream_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.stream_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_id(&self) -> u32 { + self.inner.topic_id + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.topic_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn group_id(&self) -> u32 { + self.inner.group_id + } + + #[php(getter)] + pub fn name(&self) -> String { + self.inner.name.clone() + } + + /// The topic's consumer group count after the change. + #[php(getter, change_case = "snake_case")] + pub fn consumer_groups_count(&self) -> u32 { + self.inner.consumer_groups_count + } + + /// The operation of the delete request when the delete completed later, else null. + #[php(getter, change_case = "snake_case")] + pub fn requested_op(&self) -> PhpResult<Zval> { + optional_unsigned(self.inner.requested_op) + } +} + +#[php_impl] +impl MetadataConsumerGroupItem { + #[php(getter, change_case = "snake_case")] + pub fn stream_id(&self) -> u32 { + self.inner.stream_id + } + + #[php(getter, change_case = "snake_case")] + pub fn stream_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.stream_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_id(&self) -> u32 { + self.inner.topic_id + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.topic_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn group_id(&self) -> u32 { + self.inner.group_id + } + + #[php(getter)] + pub fn name(&self) -> String { + self.inner.name.clone() + } +} + +#[php_impl] +impl MetadataMemberJoined { + #[php(getter, change_case = "snake_case")] + pub fn stream_id(&self) -> u32 { + self.inner.stream_id + } + + #[php(getter, change_case = "snake_case")] + pub fn stream_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.stream_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_id(&self) -> u32 { + self.inner.topic_id + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.topic_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn group_id(&self) -> u32 { + self.inner.group_id + } + + #[php(getter, change_case = "snake_case")] + pub fn member_id(&self) -> u32 { + self.inner.member_id + } + + /// The member's client id as a decimal string. + #[php(getter, change_case = "snake_case")] + pub fn client_id(&self) -> String { + self.inner.client_id.to_string() + } + + #[php(getter, change_case = "snake_case")] + pub fn members_count(&self) -> u32 { + self.inner.members_count + } + + /// Installed partitions, normally none because a handover installs them. + #[php(getter, change_case = "snake_case")] + pub fn partition_ids(&self) -> Vec<u32> { + self.inner.partition_ids.clone() + } +} + +#[php_impl] +impl MetadataMemberLeft { + #[php(getter, change_case = "snake_case")] + pub fn stream_id(&self) -> u32 { + self.inner.stream_id + } + + #[php(getter, change_case = "snake_case")] + pub fn stream_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.stream_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_id(&self) -> u32 { + self.inner.topic_id + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.topic_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn group_id(&self) -> u32 { + self.inner.group_id + } + + #[php(getter, change_case = "snake_case")] + pub fn member_id(&self) -> u32 { + self.inner.member_id + } + + /// The member's client id as a decimal string. + #[php(getter, change_case = "snake_case")] + pub fn client_id(&self) -> String { + self.inner.client_id.to_string() + } + + #[php(getter, change_case = "snake_case")] + pub fn members_count(&self) -> u32 { + self.inner.members_count + } + + #[php(getter)] + pub fn reason(&self) -> MetadataMemberLeftReason { + self.inner.reason.into() + } +} + +#[php_impl] +impl MetadataPartitionHandedOver { + #[php(getter, change_case = "snake_case")] + pub fn stream_id(&self) -> u32 { + self.inner.stream_id + } + + #[php(getter, change_case = "snake_case")] + pub fn stream_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.stream_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_id(&self) -> u32 { + self.inner.topic_id + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.topic_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn group_id(&self) -> u32 { + self.inner.group_id + } + + #[php(getter, change_case = "snake_case")] + pub fn partition_id(&self) -> u32 { + self.inner.partition_id + } + + /// The new owner, null when the last member left. + #[php(getter, change_case = "snake_case")] + pub fn member_id(&self) -> Option<u32> { + self.inner.member_id + } +} + +#[php_impl] +impl MetadataMemberItem { + #[php(getter, change_case = "snake_case")] + pub fn stream_id(&self) -> u32 { + self.inner.stream_id + } + + #[php(getter, change_case = "snake_case")] + pub fn stream_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.stream_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_id(&self) -> u32 { + self.inner.topic_id + } + + #[php(getter, change_case = "snake_case")] + pub fn topic_created_op(&self) -> PhpResult<Zval> { + unsigned(self.inner.topic_created_op) + } + + #[php(getter, change_case = "snake_case")] + pub fn group_id(&self) -> u32 { + self.inner.group_id + } + + #[php(getter, change_case = "snake_case")] + pub fn member_id(&self) -> u32 { + self.inner.member_id + } + + /// The member's client id as a decimal string. + #[php(getter, change_case = "snake_case")] + pub fn client_id(&self) -> String { + self.inner.client_id.to_string() + } + + /// Installed partitions, ascending. + #[php(getter, change_case = "snake_case")] + pub fn partition_ids(&self) -> Vec<u32> { + self.inner.partition_ids.clone() + } +} + +#[php_impl] +impl MetadataUserChange { + /// User id. Ids are reused after a delete. + #[php(getter, change_case = "snake_case")] + pub fn user_id(&self) -> u32 { + self.inner.user_id + } + + /// User creation time in microseconds, which tells the user from an earlier user with + /// the same id. + #[php(getter, change_case = "snake_case")] + pub fn user_created_at(&self) -> PhpResult<Zval> { + unsigned(self.inner.user_created_at) + } + + #[php(getter)] + pub fn username(&self) -> String { + self.inner.username.clone() + } + + /// `active` or `inactive`. + #[php(getter)] + pub fn status(&self) -> String { + self.inner.status.to_string() + } + + /// The public permission set in the shape `GET /users/{id}` returns, null when the + /// user has none. + #[php(getter)] + pub fn permissions(&self) -> PhpResult<Option<ZBox<ZendHashTable>>> { + self.inner + .permissions + .as_ref() + .map(permissions_array) + .transpose() + } + + /// Options the creating client set, or null when `options_unreadable`. + #[php(getter)] + pub fn options(&self) -> PhpResult<Option<ZBox<ZendHashTable>>> { + options_view(&self.inner.options, true) + } + + /// Options the server filled in. + #[php(getter, change_case = "snake_case")] + pub fn derived_options(&self) -> PhpResult<Option<ZBox<ZendHashTable>>> { + options_view(&self.inner.options, false) + } + + /// The server could not read the user's stored options. + #[php(getter, change_case = "snake_case")] + pub fn options_unreadable(&self) -> bool { + matches!(self.inner.options, MetadataOptions::Unreadable) + } +} + +#[php_impl] +impl MetadataUserIdentity { + #[php(getter, change_case = "snake_case")] + pub fn user_id(&self) -> u32 { + self.inner.user_id + } + + #[php(getter, change_case = "snake_case")] + pub fn user_created_at(&self) -> PhpResult<Zval> { + unsigned(self.inner.user_created_at) + } + + #[php(getter)] + pub fn username(&self) -> String { + self.inner.username.clone() + } +} + +#[php_impl] +impl MetadataUserPermissionsUpdated { + #[php(getter, change_case = "snake_case")] + pub fn user_id(&self) -> u32 { + self.inner.user_id + } + + #[php(getter, change_case = "snake_case")] + pub fn user_created_at(&self) -> PhpResult<Zval> { + unsigned(self.inner.user_created_at) + } + + #[php(getter)] + pub fn username(&self) -> String { + self.inner.username.clone() + } + + /// The public permission set after the change, null when the user has none. + #[php(getter)] + pub fn permissions(&self) -> PhpResult<Option<ZBox<ZendHashTable>>> { + self.inner + .permissions + .as_ref() + .map(permissions_array) + .transpose() + } + + #[php(getter)] + pub fn reason(&self) -> MetadataPermissionsUpdatedReason { + self.inner.reason.into() + } +} + +#[php_impl] +impl MetadataTokenChange { + #[php(getter, change_case = "snake_case")] + pub fn user_id(&self) -> u32 { + self.inner.user_id + } + + #[php(getter, change_case = "snake_case")] + pub fn user_created_at(&self) -> PhpResult<Zval> { + unsigned(self.inner.user_created_at) + } + + #[php(getter)] + pub fn name(&self) -> String { + self.inner.name.clone() + } + + /// Expiry in microseconds, null for a token that never expires. + #[php(getter)] + pub fn expiry(&self) -> PhpResult<Zval> { + optional_unsigned(self.inner.expiry) + } +} + +#[php_impl] +impl MetadataTokenDeleted { + #[php(getter, change_case = "snake_case")] + pub fn user_id(&self) -> u32 { + self.inner.user_id + } + + #[php(getter, change_case = "snake_case")] + pub fn user_created_at(&self) -> PhpResult<Zval> { + unsigned(self.inner.user_created_at) + } + + #[php(getter)] + pub fn name(&self) -> String { + self.inner.name.clone() + } + + #[php(getter)] + pub fn reason(&self) -> MetadataTokenDeletedReason { + self.inner.reason.into() + } +} + +#[php_impl] +impl MetadataSession { + #[php(getter, change_case = "snake_case")] + pub fn user_id(&self) -> u32 { + self.inner.user_id + } + + #[php(getter, change_case = "snake_case")] + pub fn user_created_at(&self) -> PhpResult<Zval> { + unsigned(self.inner.user_created_at) + } + + /// The session's client id as a decimal string. + #[php(getter, change_case = "snake_case")] + pub fn client_id(&self) -> String { + self.inner.client_id.to_string() + } + + /// The session epoch. + #[php(getter)] + pub fn session(&self) -> PhpResult<Zval> { + unsigned(self.inner.session) + } +} + +#[php_impl] +impl MetadataSessionClosed { + #[php(getter, change_case = "snake_case")] + pub fn user_id(&self) -> u32 { + self.inner.user_id + } + + #[php(getter, change_case = "snake_case")] + pub fn user_created_at(&self) -> PhpResult<Zval> { + unsigned(self.inner.user_created_at) + } + + /// The session's client id as a decimal string. + #[php(getter, change_case = "snake_case")] + pub fn client_id(&self) -> String { + self.inner.client_id.to_string() + } + + /// The session epoch. + #[php(getter)] + pub fn session(&self) -> PhpResult<Zval> { + unsigned(self.inner.session) + } + + #[php(getter)] + pub fn reason(&self) -> MetadataSessionClosedReason { + self.inner.reason.into() + } +} + +#[php_impl] +impl MetadataUnknownChange { + /// The group code on binary transports, or the group name over HTTP. + #[php(getter)] + pub fn group(&self) -> PhpResult<Zval> { + self.group + .clone() + .into_zval(false) + .map_err(to_php_exception) + } + + /// The kind code on binary transports, or the kind name over HTTP. + #[php(getter)] + pub fn kind(&self) -> PhpResult<Zval> { + self.kind.clone().into_zval(false).map_err(to_php_exception) + } + + /// The raw body on binary transports, or the record's whole JSON object over HTTP. + #[php(getter)] + pub fn body(&self) -> Binary<u8> { + Binary::new(self.body.clone()) + } +} + +#[php_impl] +impl MetadataUnknownItem { + /// The group code on binary transports, or the group name over HTTP. + #[php(getter)] + pub fn group(&self) -> PhpResult<Zval> { + self.group + .clone() + .into_zval(false) + .map_err(to_php_exception) + } + + /// The raw body on binary transports, or the item's whole JSON object over HTTP. + #[php(getter)] + pub fn body(&self) -> Binary<u8> { + Binary::new(self.body.clone()) + } +} + +impl From<RustMetadataChanges> for MetadataChanges { + fn from(inner: RustMetadataChanges) -> Self { + Self { inner } + } +} + +// Lets a group list default to `[]` in the generated stub. +impl StubLiteral for MetadataChangeGroup { + fn stub_literal(&self) -> String { + format!("\\Iggy\\MetadataChangeGroup::{}", self.to_name()) + } +} + +impl From<MetadataChangeGroup> for RustMetadataChangeGroup { + fn from(group: MetadataChangeGroup) -> Self { + match group { + MetadataChangeGroup::Streams => Self::Streams, + MetadataChangeGroup::Topics => Self::Topics, + MetadataChangeGroup::Partitions => Self::Partitions, + MetadataChangeGroup::ConsumerGroups => Self::ConsumerGroups, + MetadataChangeGroup::ConsumerGroupMembers => Self::ConsumerGroupMembers, + MetadataChangeGroup::Users => Self::Users, + MetadataChangeGroup::PersonalAccessTokens => Self::PersonalAccessTokens, + MetadataChangeGroup::Sessions => Self::Sessions, + } + } +} + +impl From<RustMetadataChangeGroup> for MetadataChangeGroup { + fn from(group: RustMetadataChangeGroup) -> Self { + match group { + RustMetadataChangeGroup::Streams => Self::Streams, + RustMetadataChangeGroup::Topics => Self::Topics, + RustMetadataChangeGroup::Partitions => Self::Partitions, + RustMetadataChangeGroup::ConsumerGroups => Self::ConsumerGroups, + RustMetadataChangeGroup::ConsumerGroupMembers => Self::ConsumerGroupMembers, + RustMetadataChangeGroup::Users => Self::Users, + RustMetadataChangeGroup::PersonalAccessTokens => Self::PersonalAccessTokens, + RustMetadataChangeGroup::Sessions => Self::Sessions, + } + } +} + +impl From<RustMetadataChangeKind> for MetadataChangeKind { + fn from(kind: RustMetadataChangeKind) -> Self { + match kind { + RustMetadataChangeKind::StreamCreated => Self::StreamCreated, + RustMetadataChangeKind::StreamUpdated => Self::StreamUpdated, + RustMetadataChangeKind::StreamDeleted => Self::StreamDeleted, + RustMetadataChangeKind::TopicCreated => Self::TopicCreated, + RustMetadataChangeKind::TopicUpdated => Self::TopicUpdated, + RustMetadataChangeKind::TopicDeleted => Self::TopicDeleted, + RustMetadataChangeKind::PartitionsChanged => Self::PartitionsChanged, + RustMetadataChangeKind::ConsumerGroupCreated => Self::ConsumerGroupCreated, + RustMetadataChangeKind::ConsumerGroupDeleted => Self::ConsumerGroupDeleted, + RustMetadataChangeKind::ConsumerGroupMemberJoined => Self::ConsumerGroupMemberJoined, + RustMetadataChangeKind::ConsumerGroupMemberLeft => Self::ConsumerGroupMemberLeft, + RustMetadataChangeKind::ConsumerGroupPartitionHandedOver => { + Self::ConsumerGroupPartitionHandedOver + } + RustMetadataChangeKind::UserCreated => Self::UserCreated, + RustMetadataChangeKind::UserUpdated => Self::UserUpdated, + RustMetadataChangeKind::UserDeleted => Self::UserDeleted, + RustMetadataChangeKind::UserPasswordChanged => Self::UserPasswordChanged, + RustMetadataChangeKind::UserPermissionsUpdated => Self::UserPermissionsUpdated, + RustMetadataChangeKind::PersonalAccessTokenCreated => Self::PersonalAccessTokenCreated, + RustMetadataChangeKind::PersonalAccessTokenDeleted => Self::PersonalAccessTokenDeleted, + RustMetadataChangeKind::SessionRegistered => Self::SessionRegistered, + RustMetadataChangeKind::SessionClosed => Self::SessionClosed, + } + } +} + +impl From<RustMetadataChangeStatus> for MetadataChangeStatus { + fn from(status: RustMetadataChangeStatus) -> Self { + match status { + RustMetadataChangeStatus::Ok => Self::Ok, + RustMetadataChangeStatus::Behind => Self::Behind, + RustMetadataChangeStatus::Expired => Self::Expired, + RustMetadataChangeStatus::Reset => Self::Reset, + } + } +} + +impl From<RustMetadataChangePhase> for MetadataChangePhase { + fn from(phase: RustMetadataChangePhase) -> Self { + match phase { + RustMetadataChangePhase::Changes => Self::Changes, + RustMetadataChangePhase::Current => Self::Current, + } + } +} + +impl From<RustMetadataMemberLeftReason> for MetadataMemberLeftReason { + fn from(reason: RustMetadataMemberLeftReason) -> Self { + match reason { + RustMetadataMemberLeftReason::Left => Self::Left, + RustMetadataMemberLeftReason::SessionEnded => Self::SessionEnded, + } + } +} + +impl From<RustMetadataTokenDeletedReason> for MetadataTokenDeletedReason { + fn from(reason: RustMetadataTokenDeletedReason) -> Self { + match reason { + RustMetadataTokenDeletedReason::Deleted => Self::Deleted, + RustMetadataTokenDeletedReason::Expired => Self::Expired, + } + } +} + +impl From<RustMetadataSessionClosedReason> for MetadataSessionClosedReason { + fn from(reason: RustMetadataSessionClosedReason) -> Self { + match reason { + RustMetadataSessionClosedReason::Logout => Self::Logout, + RustMetadataSessionClosedReason::Expired => Self::Expired, + RustMetadataSessionClosedReason::Revoked => Self::Revoked, + } + } +} + +impl From<RustMetadataPermissionsUpdatedReason> for MetadataPermissionsUpdatedReason { + fn from(reason: RustMetadataPermissionsUpdatedReason) -> Self { + match reason { + RustMetadataPermissionsUpdatedReason::Updated => Self::Updated, + RustMetadataPermissionsUpdatedReason::ResourceDeleted => Self::ResourceDeleted, + } + } +} + +/// A group or kind as the transport carried it: a code on binary transports, a name over +/// HTTP. +#[derive(Clone)] +enum UnknownCode { + Binary(u8), + Json(String), +} + +impl IntoZval for UnknownCode { + const TYPE: DataType = DataType::Mixed; + const NULLABLE: bool = false; + + fn set_zval(self, zval: &mut Zval, persistent: bool) -> ConversionResult<()> { + match self { + Self::Binary(code) => i64::from(code).set_zval(zval, persistent), + Self::Json(name) => name.set_zval(zval, persistent), + } + } +} + +fn change_body(change: &RustMetadataChange) -> PhpResult<Zval> { + let body = match change { + RustMetadataChange::StreamCreated(body) | RustMetadataChange::StreamUpdated(body) => { + MetadataStreamChange { + inner: body.clone(), + } + .into_zval(false) + } + RustMetadataChange::StreamDeleted(body) => MetadataStreamDeleted { + inner: body.clone(), + } + .into_zval(false), + RustMetadataChange::TopicCreated(body) | RustMetadataChange::TopicUpdated(body) => { + MetadataTopicChange { + inner: body.clone(), + } + .into_zval(false) + } + RustMetadataChange::TopicDeleted(body) => MetadataTopicDeleted { + inner: body.clone(), + } + .into_zval(false), + RustMetadataChange::PartitionsChanged(body) => MetadataPartitionsChanged { + inner: body.clone(), + } + .into_zval(false), + RustMetadataChange::ConsumerGroupCreated(body) => MetadataConsumerGroupCreated { + inner: body.clone(), + } + .into_zval(false), + RustMetadataChange::ConsumerGroupDeleted(body) => MetadataConsumerGroupDeleted { + inner: body.clone(), + } + .into_zval(false), + RustMetadataChange::ConsumerGroupMemberJoined(body) => MetadataMemberJoined { + inner: body.clone(), + } + .into_zval(false), + RustMetadataChange::ConsumerGroupMemberLeft(body) => MetadataMemberLeft { + inner: body.clone(), + } + .into_zval(false), + RustMetadataChange::ConsumerGroupPartitionHandedOver(body) => MetadataPartitionHandedOver { + inner: body.clone(), + } + .into_zval(false), + RustMetadataChange::UserCreated(body) | RustMetadataChange::UserUpdated(body) => { + MetadataUserChange { + inner: body.clone(), + } + .into_zval(false) + } + RustMetadataChange::UserDeleted(body) | RustMetadataChange::UserPasswordChanged(body) => { + MetadataUserIdentity { + inner: body.clone(), + } + .into_zval(false) + } + RustMetadataChange::UserPermissionsUpdated(body) => MetadataUserPermissionsUpdated { + inner: body.clone(), + } + .into_zval(false), + RustMetadataChange::PersonalAccessTokenCreated(body) => MetadataTokenChange { + inner: body.clone(), + } + .into_zval(false), + RustMetadataChange::PersonalAccessTokenDeleted(body) => MetadataTokenDeleted { + inner: body.clone(), + } + .into_zval(false), + RustMetadataChange::SessionRegistered(body) => MetadataSession { + inner: body.clone(), + } + .into_zval(false), + RustMetadataChange::SessionClosed(body) => MetadataSessionClosed { + inner: body.clone(), + } + .into_zval(false), + RustMetadataChange::Unknown { group, kind, body } => MetadataUnknownChange { + group: UnknownCode::Binary(*group), + kind: UnknownCode::Binary(*kind), + body: body.to_vec(), + } + .into_zval(false), + RustMetadataChange::UnknownJson { group, kind, body } => MetadataUnknownChange { + group: UnknownCode::Json(group.clone()), + kind: UnknownCode::Json(kind.clone()), + body: body.get().as_bytes().to_vec(), + } + .into_zval(false), + }; + body.map_err(to_php_exception) +} + +fn item_body(item: &RustMetadataCurrentItem) -> PhpResult<Zval> { + let body = match item { + RustMetadataCurrentItem::Stream(body) => MetadataStreamChange { + inner: body.clone(), + } + .into_zval(false), + RustMetadataCurrentItem::Topic(body) => MetadataTopicItem { + inner: body.clone(), + } + .into_zval(false), + RustMetadataCurrentItem::Partitions(body) => MetadataPartitionsItem { + inner: body.clone(), + } + .into_zval(false), + RustMetadataCurrentItem::ConsumerGroup(body) => MetadataConsumerGroupItem { + inner: body.clone(), + } + .into_zval(false), + RustMetadataCurrentItem::ConsumerGroupMember(body) => MetadataMemberItem { + inner: body.clone(), + } + .into_zval(false), + RustMetadataCurrentItem::User(body) => MetadataUserChange { + inner: body.clone(), + } + .into_zval(false), + RustMetadataCurrentItem::PersonalAccessToken(body) => MetadataTokenChange { + inner: body.clone(), + } + .into_zval(false), + RustMetadataCurrentItem::Session(body) => MetadataSession { + inner: body.clone(), + } + .into_zval(false), + RustMetadataCurrentItem::Unknown { group, body } => MetadataUnknownItem { + group: UnknownCode::Binary(*group), + body: body.to_vec(), + } + .into_zval(false), + RustMetadataCurrentItem::UnknownJson { group, body } => MetadataUnknownItem { + group: UnknownCode::Json(group.clone()), + body: body.get().as_bytes().to_vec(), + } + .into_zval(false), + }; + body.map_err(to_php_exception) +} + +fn options_view( + options: &MetadataOptions, + explicit: bool, +) -> PhpResult<Option<ZBox<ZendHashTable>>> { + match options { + MetadataOptions::Readable(options) => options_array(options, explicit).map(Some), + MetadataOptions::Unreadable => Ok(None), + } +} + +/// Public permissions as nested arrays keyed like the `GET /users/{id}` JSON, with stream +/// and topic ids as int keys. +fn permissions_array(permissions: &Permissions) -> PhpResult<ZBox<ZendHashTable>> { + let global = &permissions.global; + let mut global_array = ZendHashTable::new(); + for (name, allowed) in [ + ("manage_servers", global.manage_servers), + ("read_servers", global.read_servers), + ("manage_users", global.manage_users), + ("read_users", global.read_users), + ("manage_streams", global.manage_streams), + ("read_streams", global.read_streams), + ("manage_topics", global.manage_topics), + ("read_topics", global.read_topics), + ("poll_messages", global.poll_messages), + ("send_messages", global.send_messages), + ] { + global_array + .insert(name, allowed) + .map_err(to_php_exception)?; + } + let mut array = ZendHashTable::new(); + array + .insert("global", global_array) + .map_err(to_php_exception)?; + let streams = match &permissions.streams { + Some(streams) => { + let mut streams_array = ZendHashTable::new(); + for (stream_id, stream) in streams { + streams_array + .insert_at_index(id_key(*stream_id)?, stream_permissions_array(stream)?) + .map_err(to_php_exception)?; + } + Some(streams_array) + } + None => None, + }; + array.insert("streams", streams).map_err(to_php_exception)?; + Ok(array) +} + +fn stream_permissions_array(stream: &StreamPermissions) -> PhpResult<ZBox<ZendHashTable>> { + let mut array = ZendHashTable::new(); + for (name, allowed) in [ + ("manage_stream", stream.manage_stream), + ("read_stream", stream.read_stream), + ("manage_topics", stream.manage_topics), + ("read_topics", stream.read_topics), + ("poll_messages", stream.poll_messages), + ("send_messages", stream.send_messages), + ] { + array.insert(name, allowed).map_err(to_php_exception)?; + } + let topics = match &stream.topics { + Some(topics) => { + let mut topics_array = ZendHashTable::new(); + for (topic_id, topic) in topics { + let mut topic_array = ZendHashTable::new(); + for (name, allowed) in [ + ("manage_topic", topic.manage_topic), + ("read_topic", topic.read_topic), + ("poll_messages", topic.poll_messages), + ("send_messages", topic.send_messages), + ] { + topic_array + .insert(name, allowed) + .map_err(to_php_exception)?; + } + topics_array + .insert_at_index(id_key(*topic_id)?, topic_array) + .map_err(to_php_exception)?; + } + Some(topics_array) + } + None => None, + }; + array.insert("topics", topics).map_err(to_php_exception)?; + Ok(array) +} + +fn id_key(id: usize) -> PhpResult<i64> { + i64::try_from(id).map_err(|_| to_php_exception(IggyError::InvalidFormat)) +} + +/// An unsigned 64-bit value as an int, or as a decimal string above `PHP_INT_MAX`. +fn unsigned(value: u64) -> PhpResult<Zval> { + match i64::try_from(value) { + Ok(number) => number.into_zval(false), + Err(_) => value.to_string().into_zval(false), + } + .map_err(to_php_exception) +} + +fn optional_unsigned(value: Option<u64>) -> PhpResult<Zval> { + match value { + Some(value) => unsigned(value), + None => Ok(Zval::new()), + } +} + +fn decimal_u128(text: &str, field: &str) -> PhpResult<u128> { + let valid = !text.is_empty() && text.bytes().all(|byte| byte.is_ascii_digit()); + valid + .then(|| text.parse::<u128>().ok()) + .flatten() + .ok_or_else(|| { + to_php_exception(format!( + "'{field}' must be an unsigned 128-bit decimal string" + )) + }) +}
diff --git a/foreign/php/tests/MetadataChangesTest.php b/foreign/php/tests/MetadataChangesTest.php new file mode 100644 index 0000000..ccf6a08 --- /dev/null +++ b/foreign/php/tests/MetadataChangesTest.php
@@ -0,0 +1,840 @@ +<?php +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +declare(strict_types=1); + +use Iggy\Client as IggyClient; +use Iggy\Exception\AuthenticationException; +use Iggy\Exception\IggyException; +use Iggy\MetadataChangeCursor; +use Iggy\MetadataChangeGroup; +use Iggy\MetadataChangeKind; +use Iggy\MetadataChangePhase; +use Iggy\MetadataChangeRecord; +use Iggy\MetadataChanges; +use Iggy\MetadataChangeStart; +use Iggy\MetadataChangeStatus; +use Iggy\MetadataCurrentItem; +use Iggy\MetadataCurrentPageToken; +use Iggy\MetadataMemberLeftReason; +use Iggy\MetadataPermissionsUpdatedReason; +use Iggy\MetadataSessionClosedReason; +use Iggy\MetadataStreamChange; +use Iggy\MetadataStreamDeleted; +use Iggy\MetadataTokenDeletedReason; +use Iggy\MetadataTopicChange; +use Iggy\MetadataUnknownChange; +use Iggy\MetadataUnknownItem; +use PHPUnit\Framework\Attributes\DataProvider; +use PHPUnit\Framework\Attributes\TestDox; +use PHPUnit\Framework\TestCase; + +final class MetadataChangesTest extends TestCase +{ + private const INVALID_COMMAND = 3; + private const UNAUTHORIZED = 41; + private const ROOT_USER_ID = 0; + private const PAGE_LIMIT = 1000; + // A two phase delete commits its record after the call returns. + private const FOLLOW_ATTEMPTS = 100; + private const FOLLOW_INTERVAL_MICROSECONDS = 100_000; + private const ENVELOPE_KEYS = ['op', 'ordinal', 'checksum', 'timestamp', 'actor_user_id', 'group', 'kind']; + private const OPTIONAL_BODY_KEYS = ['requested_op', 'member_id', 'expiry']; + private const NEVER_RETURNED_KEYS = ['encryption_key', 'encryption_key_wrapped', 'critical_keys']; + + private array $createdStreams = []; + + public static function transports(): array + { + return [ + 'tcp' => ['tcp'], + 'quic' => ['quic'], + 'http' => ['http'], + 'websocket' => ['websocket'], + ]; + } + + protected function tearDown(): void + { + if ($this->createdStreams !== []) { + $client = new_client(); + foreach ($this->createdStreams as $stream) { + cleanup_stream($client, $stream); + } + } + + parent::tearDown(); + } + + #[DataProvider('transports')] + #[TestDox('Creating, renaming and deleting a stream comes back as typed changes with the root actor on $transport')] + public function testGivenLatestCursorWhenCreatingRenamingAndDeletingStreamShouldFollowEachChange(string $transport): void + { + $client = new_transport_client($transport); + $name = unique_name('changes-follow'); + $renamed = unique_name('changes-renamed'); + $cursor = self::latestCursor($client, [MetadataChangeGroup::Streams]); + + $streamId = $this->createStream($client, $name); + $client->updateStream($streamId, name: $renamed); + $client->deleteStream($streamId); + + $followed = self::follow($client, $cursor, [MetadataChangeGroup::Streams], $streamId, 3); + foreach ($followed as $record) { + assert_same(MetadataChangeGroup::Streams, $record->group); + } + $records = self::streamRecords($followed, $streamId); + assert_same( + [MetadataChangeKind::StreamCreated, MetadataChangeKind::StreamUpdated, MetadataChangeKind::StreamDeleted], + array_map(static fn (MetadataChangeRecord $record): ?MetadataChangeKind => $record->kind, $records), + ); + [$created, $updated, $deleted] = array_map(static fn (MetadataChangeRecord $record): object => $record->change, $records); + assert_instance_of(MetadataStreamChange::class, $created); + assert_instance_of(MetadataStreamChange::class, $updated); + assert_instance_of(MetadataStreamDeleted::class, $deleted); + assert_same([$name, $renamed, $renamed], [$created->name, $updated->name, $deleted->name]); + assert_same($records[0]->op, $created->stream_created_op); + assert_same($created->stream_created_op, $updated->stream_created_op); + assert_same($created->stream_created_op, $deleted->stream_created_op); + assert_true(!$created->options_unreadable, 'a new stream has readable options'); + $positions = []; + foreach ($records as $record) { + assert_same(self::ROOT_USER_ID, $record->actor_user_id); + self::assertMatchesRegularExpression('/^[1-9][0-9]*$/', $record->checksum); + $positions[] = [$record->op, $record->ordinal]; + } + $sorted = $positions; + sort($sorted); + assert_same($sorted, $positions); + } + + #[DataProvider('transports')] + #[TestDox('A topics group filter leaves out the stream created on the way on $transport')] + public function testGivenTopicsGroupWhenFollowingShouldReturnOnlyTopicRecords(string $transport): void + { + $client = new_transport_client($transport); + $topicName = unique_name('changes-topic'); + $cursor = self::latestCursor($client, [MetadataChangeGroup::Topics]); + + $streamId = $this->createStream($client, unique_name('changes-filter')); + $client->createTopic($streamId, $topicName, 2, null, null, null, null); + + $followed = self::follow($client, $cursor, [MetadataChangeGroup::Topics], $streamId, 1); + foreach ($followed as $record) { + assert_same(MetadataChangeGroup::Topics, $record->group); + } + $records = self::streamRecords($followed, $streamId); + assert_count(1, $records); + assert_same(MetadataChangeKind::TopicCreated, $records[0]->kind); + $topic = $records[0]->change; + assert_instance_of(MetadataTopicChange::class, $topic); + assert_same([$topicName, 2], [$topic->name, $topic->partitions_count]); + assert_same($records[0]->op, $topic->topic_created_op); + } + + #[DataProvider('transports')] + #[TestDox('Pages of one record continue through next until has more drops on $transport')] + public function testGivenLimitOneWhenFollowingShouldPageEveryRecordInOrder(string $transport): void + { + $client = new_transport_client($transport); + $cursor = self::latestCursor($client, [MetadataChangeGroup::Streams]); + $created = []; + foreach (range(1, 3) as $index) { + $created[] = $this->createStream($client, unique_name("changes-page-{$index}")); + } + + $records = []; + do { + $page = $client->getMetadataChanges(MetadataChangeStart::after($cursor), [MetadataChangeGroup::Streams], limit: 1); + assert_same(MetadataChangeStatus::Ok, $page->status); + assert_same(MetadataChangePhase::Changes, $page->phase); + self::assertLessThanOrEqual(1, count($page->records)); + $next = $page->next; + assert_not_null($next); + assert_true( + [$next->op, $next->ordinal] > [$cursor->op, $cursor->ordinal], + 'every page advances the cursor', + ); + $cursor = $next; + array_push($records, ...$page->records); + } while ($page->has_more); + + $streamIds = []; + foreach ($records as $record) { + $change = $record->change; + if ($record->kind === MetadataChangeKind::StreamCreated && $change instanceof MetadataStreamChange) { + $streamIds[] = $change->stream_id; + } + } + assert_same($created, $streamIds); + $last = $records[array_key_last($records)]; + assert_same([$last->op, $last->ordinal], [$cursor->op, $cursor->ordinal]); + } + + #[DataProvider('transports')] + #[TestDox('Since and until keep the records prepared inside the range on $transport')] + public function testGivenTimeRangeWhenReadingEarliestShouldReturnOnlyRecordsInRange(string $transport): void + { + $client = new_transport_client($transport); + $cursor = self::latestCursor($client, [MetadataChangeGroup::Streams]); + $streamId = $this->createStream($client, unique_name('changes-range')); + $timestamp = self::streamRecords( + self::follow($client, $cursor, [MetadataChangeGroup::Streams], $streamId, 1), + $streamId, + )[0]->timestamp; + + $page = $client->getMetadataChanges( + MetadataChangeStart::earliest(), + [MetadataChangeGroup::Streams], + since: $timestamp, + until: (string) $timestamp, + limit: self::PAGE_LIMIT, + ); + + assert_same(MetadataChangeStatus::Ok, $page->status); + assert_true($page->records !== [], 'the range holds the stream record'); + foreach ($page->records as $record) { + assert_same($timestamp, $record->timestamp); + } + assert_count(1, self::streamRecords($page->records, $streamId)); + } + + #[DataProvider('transports')] + #[TestDox('A cursor of another history resets and one past the head is behind on $transport')] + public function testGivenForeignAndFutureCursorsWhenFollowingShouldAnswerResetAndBehind(string $transport): void + { + $client = new_transport_client($transport); + $latest = $client->getMetadataChanges(MetadataChangeStart::latest()); + $logId = $latest->log_id; + assert_not_null($logId); + $headOp = $latest->head_op; + assert_true(is_int($headOp), 'a live head fits an int'); + $foreign = new MetadataChangeCursor($logId === '1' ? '2' : '1', 1, '1', 0); + $ahead = new MetadataChangeCursor($logId, $headOp + 1_000_000, '1', 0); + + $reset = $client->getMetadataChanges(MetadataChangeStart::after($foreign)); + $behind = $client->getMetadataChanges(MetadataChangeStart::after($ahead)); + + assert_same(MetadataChangeStatus::Reset, $reset->status); + assert_null($reset->next); + assert_same([], $reset->records); + assert_same(MetadataChangeStatus::Behind, $behind->status); + assert_same((string) $ahead, (string) $behind->next); + assert_same([], $behind->records); + } + + #[DataProvider('transports')] + #[TestDox('Current state pages of two streams add up to the stream list, then hand an anchor on $transport')] + public function testGivenCurrentScanWhenPagingShouldMatchStreamList(string $transport): void + { + $client = new_transport_client($transport); + foreach (range(1, 3) as $index) { + $this->createStream($client, unique_name("changes-current-{$index}")); + } + + $items = []; + $token = null; + $pages = 0; + while (true) { + $page = $client->getMetadataChanges(MetadataChangeStart::current($token), [MetadataChangeGroup::Streams], limit: 2); + $pages++; + assert_same(MetadataChangeStatus::Ok, $page->status); + assert_same(MetadataChangePhase::Current, $page->phase); + assert_same([], $page->records); + self::assertLessThanOrEqual(2, count($page->items)); + array_push($items, ...$page->items); + $token = $page->current_page_token; + if ($token === null) { + $anchor = $page->next; + break; + } + assert_true($page->has_more, 'a page with a token has more'); + assert_same([MetadataChangeGroup::Streams], $token->groups); + assert_same((string) $token, (string) MetadataCurrentPageToken::parse((string) $token)); + } + + $projected = []; + foreach ($items as $item) { + assert_same(MetadataChangeGroup::Streams, $item->group); + $body = $item->body; + assert_instance_of(MetadataStreamChange::class, $body); + $projected[$body->stream_id] = $body->name; + } + ksort($projected); + assert_same(stream_list(new_client()), $projected); + self::assertGreaterThanOrEqual(2, $pages); + assert_not_null($anchor); + $following = $client->getMetadataChanges(MetadataChangeStart::after($anchor), [MetadataChangeGroup::Streams]); + assert_same(MetadataChangeStatus::Ok, $following->status); + } + + #[DataProvider('transports')] + #[TestDox('Refused query vectors throw invalid command before anything is sent on $transport')] + public function testGivenRefusedQueryVectorsWhenReadingShouldThrowBeforeSending(string $transport): void + { + $client = new_transport_client($transport); + foreach (self::fixture()['refused']['queries'] as $vector) { + $error = assert_throws(static fn () => self::readVector($client, $vector['input']), 'Invalid command'); + assert_same(self::INVALID_COMMAND, $error->getCode(), $vector['case']); + } + } + + #[DataProvider('transports')] + #[TestDox('A user without manage servers is refused as unauthorized on $transport')] + public function testGivenUserWithoutManageServersWhenReadingShouldThrowUnauthorized(string $transport): void + { + $admin = new_client(); + $username = unique_name('changes-reader'); + $password = 'changes-secret'; + create_user_without_permissions($admin, $username, $password); + + try { + $client = new_transport_client($transport, $username, $password); + $error = assert_throws(static fn () => $client->getMetadataChanges(MetadataChangeStart::earliest())); + assert_instance_of(AuthenticationException::class, $error); + assert_same(self::UNAUTHORIZED, $error->getCode()); + } finally { + delete_user_by_name($admin, $username); + } + } + + #[DataProvider('transports')] + #[TestDox('A keyed stream change carries the key source and fingerprint and no key material on $transport')] + public function testGivenStreamKeyWhenFollowingChangesShouldExposeFingerprintsWithoutKey(string $transport): void + { + $key = base64_encode(random_bytes(32)); + $client = new_transport_client($transport); + $cursor = self::latestCursor($client, [MetadataChangeGroup::Streams]); + + $streamId = $this->createStream($client, unique_name('changes-keyed'), $key); + + $records = self::streamRecords(self::follow($client, $cursor, [MetadataChangeGroup::Streams], $streamId, 1), $streamId); + assert_count(1, $records); + assert_same(MetadataChangeKind::StreamCreated, $records[0]->kind); + $stream = $records[0]->change; + assert_instance_of(MetadataStreamChange::class, $stream); + $explicit = $stream->options; + $derived = $stream->derived_options; + assert_not_null($explicit); + assert_not_null($derived); + assert_same('stream', $derived['encryption_key_source']); + assert_same(IggyClient::encryptionKeyFingerprint($key), $derived['encryption_key_fingerprint']); + assert_same($client->getStream($streamId)?->encryption_kek_fingerprint, $derived['encryption_kek_fingerprint']); + foreach (self::NEVER_RETURNED_KEYS as $secret) { + self::assertArrayNotHasKey($secret, $explicit); + self::assertArrayNotHasKey($secret, $derived); + } + $raw = base64_decode($key, true); + foreach ([...array_values($explicit), ...array_values($derived)] as $value) { + self::assertNotContains($value, [$key, $raw, bin2hex((string) $raw)]); + } + } + + #[TestDox('Response vectors decode from binary and JSON to the expected page')] + public function testGivenResponseVectorsWhenDecodingBinaryAndJsonShouldMatchExpectedJson(): void + { + foreach (self::fixture()['responses'] as $index => $vector) { + $binary = MetadataChanges::decode(hex2bin($vector['response_hex'])); + $json = MetadataChanges::fromJson( + json_encode(self::objectFixture()->responses[$index]->expected, JSON_THROW_ON_ERROR), + ); + foreach ([$binary, $json] as $page) { + self::assertPage($page, $vector['expected'], $vector['case']); + } + } + } + + #[TestDox('Record and item vectors decode from binary and JSON to the expected bodies')] + public function testGivenRecordAndItemVectorsWhenDecodingShouldMatchExpectedJson(): void + { + $fixture = self::fixture(); + $objects = self::objectFixture(); + foreach ($fixture['records'] as $index => $vector) { + $pages = [ + MetadataChanges::decode(self::pagePayload(hex2bin($vector['record_hex']), '')), + MetadataChanges::fromJson(self::pageJson([$objects->records[$index]->expected], [])), + ]; + foreach ($pages as $page) { + assert_count(1, $page->records, $vector['case']); + self::assertRecord($page->records[0], $vector['expected'], $vector['case']); + } + } + foreach ($fixture['items'] as $index => $vector) { + $pages = [ + MetadataChanges::decode(self::pagePayload('', hex2bin($vector['item_hex']))), + MetadataChanges::fromJson(self::pageJson([], [$objects->items[$index]->expected])), + ]; + foreach ($pages as $page) { + assert_count(1, $page->items, $vector['case']); + self::assertItem($page->items[0], $vector['expected'], $vector['case']); + } + } + } + + #[TestDox('Values above PHP_INT_MAX and 128-bit values keep every bit as decimal strings')] + public function testGivenLargeValuesWhenDecodingShouldKeepEveryBit(): void + { + $vector = self::fixture()['large_values']; + $record = MetadataChanges::decode(self::pagePayload(hex2bin($vector['record_hex']), ''))->records[0]; + $change = $record->change; + + assert_same((int) $vector['op'], $record->op); + assert_same($vector['timestamp'], $record->timestamp); + assert_same($vector['ordinal'], $record->ordinal); + assert_same(self::hexToDecimal($vector['checksum']), $record->checksum); + assert_same((int) $vector['user_created_at'], $change->user_created_at); + assert_same((int) $vector['session'], $change->session); + assert_same(self::hexToDecimal($vector['client_id']), $change->client_id); + } + + #[TestDox('Cursor vectors round trip their text and every field')] + public function testGivenCursorVectorsWhenParsingShouldRoundTripEveryField(): void + { + $constants = self::fixture()['constants']; + foreach (self::fixture()['cursors'] as $vector) { + $cursor = MetadataChangeCursor::parse($vector['text']); + $fields = [ + self::hexToDecimal($vector['log_id']), + self::unsigned($vector['op']), + self::hexToDecimal($vector['checksum']), + $vector['ordinal'], + ]; + assert_same($fields, [$cursor->log_id, $cursor->op, $cursor->checksum, $cursor->ordinal], $vector['case']); + assert_same($vector['text'], (string) $cursor); + assert_same($constants['cursor_text_length'], strlen((string) $cursor)); + assert_same($vector['text'], (string) new MetadataChangeCursor(...$fields)); + } + } + + #[TestDox('Page token vectors expose their anchor, head, groups and last key')] + public function testGivenPageTokenVectorsWhenParsingShouldExposeEveryField(): void + { + $codes = array_map(static fn (array $group): int => $group['code'], self::fixture()['constants']['groups']); + asort($codes); + foreach (self::fixture()['page_tokens'] as $vector) { + $token = MetadataCurrentPageToken::parse($vector['text']); + assert_same($vector['text'], (string) $token); + assert_same($vector['anchor'], (string) $token->anchor); + assert_same($vector['required_head'], $token->required_head); + assert_same($vector['group'], $token->last_group->value); + assert_same($vector['last_key_hex'], bin2hex($token->last_key)); + $masked = array_keys(array_filter( + $codes, + static fn (int $code): bool => ($vector['groups_mask'] & (1 << ($code - 1))) !== 0, + )); + assert_same( + $masked, + array_map(static fn (MetadataChangeGroup $group): string => $group->value, $token->groups), + $vector['case'], + ); + } + } + + #[TestDox('Equal values compare equal with == and different cursors, tokens, pages, records, items and bodies do not')] + public function testGivenDecodedValuesWhenComparingWithEqualsShouldCompareContents(): void + { + $fixture = self::fixture(); + $unknown = hex2bin($fixture['responses'][1]['response_hex']); + $builders = [ + 'cursor' => static fn (int $index): object => MetadataChangeCursor::parse($fixture['cursors'][$index]['text']), + 'page token' => static fn (int $index): object => MetadataCurrentPageToken::parse($fixture['page_tokens'][$index]['text']), + 'page' => static fn (int $index): object => MetadataChanges::decode(hex2bin($fixture['responses'][$index]['response_hex'])), + 'record' => static fn (int $index): object => MetadataChanges::decode( + self::pagePayload(hex2bin($fixture['records'][$index]['record_hex']), ''), + )->records[0], + 'stream body' => static fn (int $index): object => MetadataChanges::decode( + self::pagePayload(hex2bin($fixture['records'][$index]['record_hex']), ''), + )->records[0]->change, + 'item' => static fn (int $index): object => MetadataChanges::decode( + self::pagePayload('', hex2bin($fixture['items'][$index]['item_hex'])), + )->items[0], + 'unknown body' => static fn (int $index): object => MetadataChanges::decode($unknown)->records[$index]->change, + ]; + $equalButDifferent = []; + foreach ($builders as $label => $build) { + assert_true($build(0) == $build(0), "{$label} equals its copy"); + if ($build(0) == $build(1)) { + $equalButDifferent[] = $label; + } + } + assert_same([], $equalButDifferent, 'different values compare equal'); + } + + #[TestDox('Refused cursor and page token texts throw invalid command')] + public function testGivenRefusedCursorAndTokenTextsWhenParsingShouldThrowInvalidCommand(): void + { + $refused = self::fixture()['refused']; + foreach ($refused['cursors'] as $vector) { + $error = assert_throws(static fn () => MetadataChangeCursor::parse($vector['text']), 'Invalid command'); + assert_same(self::INVALID_COMMAND, $error->getCode(), $vector['case']); + } + foreach ($refused['page_tokens'] as $vector) { + $error = assert_throws(static fn () => MetadataCurrentPageToken::parse($vector['text']), 'Invalid command'); + assert_same(self::INVALID_COMMAND, $error->getCode(), $vector['case']); + } + } + + #[TestDox('Refused query vectors throw before any network I/O')] + public function testGivenRefusedQueryVectorsWhenReadingShouldThrowBeforeNetworkIo(): void + { + $client = new IggyClient('127.0.0.1:1'); + foreach (self::fixture()['refused']['queries'] as $vector) { + $error = assert_throws(static fn () => self::readVector($client, $vector['input']), 'Invalid command'); + assert_instance_of(IggyException::class, $error); + assert_same(self::INVALID_COMMAND, $error->getCode(), $vector['case']); + } + } + + #[TestDox('Request vectors pass validation and fail only on the transport')] + public function testGivenRequestVectorsWhenReadingShouldPassValidation(): void + { + $client = new IggyClient('127.0.0.1:1'); + foreach (self::fixture()['requests'] as $vector) { + $error = assert_throws(static fn () => self::readVector($client, $vector['input'])); + assert_true($error->getCode() !== self::INVALID_COMMAND, $vector['case']); + } + } + + #[TestDox('Refused record bodies and responses fail to decode')] + public function testGivenRefusedBodiesAndResponsesWhenDecodingShouldThrow(): void + { + $refused = self::fixture()['refused']; + foreach ($refused['record_bodies'] as $vector) { + $body = hex2bin($vector['body_hex']); + $record = pack('PVx16P', 1, 0, 0) . chr(0) . chr($vector['group']) . chr($vector['kind']) + . pack('V', strlen($body)) . $body; + assert_throws(static fn () => MetadataChanges::decode(self::pagePayload($record, ''))); + } + foreach ($refused['responses'] as $vector) { + assert_throws(static fn () => MetadataChanges::decode(hex2bin($vector['hex']))); + } + assert_throws(static fn () => MetadataChanges::fromJson('{"status":"ok"}')); + } + + #[TestDox('Every group, kind, status, phase and reason of the fixture has its enum case')] + public function testGivenFixtureConstantsWhenReadingEnumsShouldCoverEveryName(): void + { + $constants = self::fixture()['constants']; + foreach ([ + MetadataChangeGroup::class => 'groups', + MetadataChangeKind::class => 'kinds', + MetadataChangeStatus::class => 'statuses', + MetadataChangePhase::class => 'phases', + MetadataMemberLeftReason::class => 'member_left_reasons', + MetadataTokenDeletedReason::class => 'token_deleted_reasons', + MetadataSessionClosedReason::class => 'session_closed_reasons', + MetadataPermissionsUpdatedReason::class => 'permissions_updated_reasons', + ] as $enum => $section) { + $values = array_map(static fn (BackedEnum $case): string => $case->value, $enum::cases()); + sort($values); + $expected = array_keys($constants[$section]); + sort($expected); + assert_same($expected, $values, $section); + } + } + + private function createStream(IggyClient $client, string $name, ?string $key = null): int + { + $client->createStream($name, $key); + $stream = $client->getStream($name); + assert_not_null($stream); + $this->createdStreams[] = $stream->id; + + return $stream->id; + } + + private static function latestCursor(IggyClient $client, array $groups): MetadataChangeCursor + { + $page = $client->getMetadataChanges(MetadataChangeStart::latest(), $groups); + assert_same(MetadataChangeStatus::Ok, $page->status); + $cursor = $page->next; + assert_not_null($cursor); + + return $cursor; + } + + /** @return list<MetadataChangeRecord> */ + private static function follow( + IggyClient $client, + MetadataChangeCursor $cursor, + array $groups, + int $streamId, + int $count, + ): array { + $records = []; + for ($attempt = 0; $attempt < self::FOLLOW_ATTEMPTS; $attempt++) { + $page = $client->getMetadataChanges(MetadataChangeStart::after($cursor), $groups, limit: self::PAGE_LIMIT); + assert_same(MetadataChangeStatus::Ok, $page->status); + $cursor = $page->next; + assert_not_null($cursor); + array_push($records, ...$page->records); + if ($page->has_more) { + continue; + } + if (count(self::streamRecords($records, $streamId)) >= $count) { + break; + } + usleep(self::FOLLOW_INTERVAL_MICROSECONDS); + } + + return $records; + } + + /** @return list<MetadataChangeRecord> */ + private static function streamRecords(array $records, int $streamId): array + { + return array_values(array_filter( + $records, + static function (MetadataChangeRecord $record) use ($streamId): bool { + $change = $record->change; + + return property_exists($change, 'stream_id') && $change->stream_id === $streamId; + }, + )); + } + + private static function readVector(IggyClient $client, array $query): MetadataChanges + { + $start = match ($query['start']) { + 'earliest' => MetadataChangeStart::earliest(), + 'latest' => MetadataChangeStart::latest(), + 'after' => MetadataChangeStart::after(MetadataChangeCursor::parse($query['cursor'])), + 'current' => MetadataChangeStart::current( + $query['page_token'] === null ? null : MetadataCurrentPageToken::parse($query['page_token']), + ), + }; + + return $client->getMetadataChanges( + $start, + array_map(static fn (string $name): MetadataChangeGroup => MetadataChangeGroup::from($name), $query['groups']), + since: $query['since'], + until: $query['until'], + limit: $query['limit'], + ); + } + + /** A changes page payload around one encoded record, or a current page around one encoded item. */ + private static function pagePayload(string $record, string $item): string + { + return chr(0) . chr($item === '' ? 0 : 1) . chr(0) . pack('PPP', 0, 0, 0) . chr(0) . chr(0) + . pack('V', $record === '' ? 0 : 1) . $record + . pack('V', $item === '' ? 0 : 1) . $item + . pack('v', 0); + } + + private static function pageJson(array $records, array $items): string + { + return json_encode([ + 'status' => 'ok', + 'phase' => $items === [] ? 'changes' : 'current', + 'log_id' => null, + 'oldest_op' => 0, + 'retained_since' => 0, + 'head_op' => 0, + 'next' => null, + 'has_more' => false, + 'records' => $records, + 'items' => $items, + 'current_page_token' => null, + ], JSON_THROW_ON_ERROR); + } + + private static function assertPage(MetadataChanges $page, array $expected, string $case): void + { + assert_same($expected['status'], $page->status->value, $case); + assert_same($expected['phase'], $page->phase->value, $case); + assert_same( + $expected['log_id'] === null ? null : self::hexToDecimal($expected['log_id']), + $page->log_id, + $case, + ); + assert_same( + [$expected['oldest_op'], $expected['retained_since'], $expected['head_op']], + [$page->oldest_op, $page->retained_since, $page->head_op], + $case, + ); + assert_same($expected['next'], $page->next === null ? null : (string) $page->next, $case); + assert_same($expected['has_more'], $page->has_more, $case); + assert_count(count($expected['records']), $page->records, $case); + foreach ($page->records as $index => $record) { + self::assertRecord($record, $expected['records'][$index], $case); + } + assert_count(count($expected['items']), $page->items, $case); + foreach ($page->items as $index => $item) { + self::assertItem($item, $expected['items'][$index], $case); + } + $token = $page->current_page_token; + assert_same($expected['current_page_token'], $token === null ? null : (string) $token, $case); + } + + private static function assertRecord(MetadataChangeRecord $record, array $expected, string $case): void + { + assert_same(self::unsigned($expected['op']), $record->op, $case); + assert_same($expected['ordinal'], $record->ordinal, $case); + assert_same(self::hexToDecimal($expected['checksum']), $record->checksum, $case); + assert_same(self::unsigned($expected['timestamp']), $record->timestamp, $case); + assert_same($expected['actor_user_id'] ?? null, $record->actor_user_id, $case); + $change = $record->change; + if ($record->kind === null) { + assert_null($record->group, $case); + self::assertUnknown($change, $expected, $case); + + return; + } + assert_same($expected['group'], $record->group?->value, $case); + assert_same($expected['kind'], $record->kind->value, $case); + self::assertBody($change, $expected, self::ENVELOPE_KEYS, $case); + } + + private static function assertItem(MetadataCurrentItem $item, array $expected, string $case): void + { + $body = $item->body; + if ($item->group === null) { + self::assertUnknown($body, $expected, $case); + + return; + } + assert_same($expected['group'], $item->group->value, $case); + self::assertBody($body, $expected, ['group'], $case); + } + + /** An unknown record or item keeps its codes and raw body: base64 in the binary rendering, the whole object over JSON. */ + private static function assertUnknown(object $unknown, array $expected, string $case): void + { + self::assertTrue( + $unknown instanceof MetadataUnknownChange || $unknown instanceof MetadataUnknownItem, + $case, + ); + assert_same($expected['group'], (string) $unknown->group, $case); + if ($unknown instanceof MetadataUnknownChange) { + assert_same($expected['kind'], (string) $unknown->kind, $case); + } + if (is_string($unknown->group)) { + assert_same($expected, json_decode($unknown->body, true, 512, JSON_THROW_ON_ERROR), $case); + } else { + assert_same($expected['body'], base64_encode($unknown->body), $case); + } + } + + private static function assertBody(object $body, array $fields, array $skipped, string $case): void + { + foreach ($fields as $key => $value) { + if (in_array($key, $skipped, true)) { + continue; + } + $label = "{$case} {$key}"; + match ($key) { + 'options' => self::assertOptions($body, $value, $label), + 'client_id' => assert_same(self::hexToDecimal($value), $body->client_id, $label), + 'reason' => assert_same($value, $body->reason->value, $label), + default => assert_same(is_int($value) ? self::unsigned($value) : $value, $body->$key, $label), + }; + } + foreach (self::OPTIONAL_BODY_KEYS as $key) { + if (!array_key_exists($key, $fields) && property_exists($body, $key)) { + assert_null($body->$key, "{$case} {$key} must be absent"); + } + } + } + + /** Compare both provenance arrays with the fixture's option map, or the unreadable marker when it is null. */ + private static function assertOptions(object $body, ?array $expected, string $label): void + { + if ($expected === null) { + assert_true($body->options_unreadable, $label); + assert_null($body->options, $label); + assert_null($body->derived_options, $label); + + return; + } + assert_true(!$body->options_unreadable, $label); + foreach (['options' => true, 'derived_options' => false] as $property => $explicit) { + $wanted = []; + foreach ($expected as $key => $option) { + if ($option['explicit'] !== $explicit) { + continue; + } + $wanted[$key] = match ($option['kind']) { + 'uint64', 'uint32' => self::unsigned($option['value']), + 'bool' => $option['value'] === 'true', + default => $option['value'], + }; + } + $actual = $body->$property; + ksort($actual); + ksort($wanted); + assert_same($wanted, $actual, "{$label} {$property}"); + } + } + + /** An unsigned 64-bit value as the SDK returns it: an int, or a decimal string above PHP_INT_MAX. */ + private static function unsigned(int|string $value): int|string + { + $text = (string) $value; + $max = (string) PHP_INT_MAX; + + return strlen($text) < strlen($max) || (strlen($text) === strlen($max) && strcmp($text, $max) <= 0) + ? (int) $text + : $text; + } + + /** A hex value of any width as a decimal string, without bcmath or gmp. */ + private static function hexToDecimal(string $hex): string + { + $digits = [0]; + foreach (str_split($hex) as $nibble) { + $carry = hexdec($nibble); + foreach ($digits as $index => $digit) { + $value = $digit * 16 + $carry; + $digits[$index] = $value % 10; + $carry = intdiv($value, 10); + } + while ($carry > 0) { + $digits[] = $carry % 10; + $carry = intdiv($carry, 10); + } + } + + return implode('', array_reverse($digits)); + } + + /** The fixture with JSON objects kept as objects, so re-encoding keeps an empty `{}` an object. */ + private static function objectFixture(): object + { + static $fixture = null; + + return $fixture ??= json_decode( + (string) file_get_contents(__DIR__ . '/fixtures/metadata_changes.json'), + false, + 512, + JSON_THROW_ON_ERROR | JSON_BIGINT_AS_STRING, + ); + } + + private static function fixture(): array + { + static $fixture = null; + + return $fixture ??= json_decode( + (string) file_get_contents(__DIR__ . '/fixtures/metadata_changes.json'), + true, + 512, + JSON_THROW_ON_ERROR | JSON_BIGINT_AS_STRING, + ); + } +}
diff --git a/foreign/php/tests/TlsTest.php b/foreign/php/tests/TlsTest.php index 371b1ea..8ce63b0 100644 --- a/foreign/php/tests/TlsTest.php +++ b/foreign/php/tests/TlsTest.php
@@ -19,6 +19,11 @@ declare(strict_types=1); use Iggy\Client as IggyClient; +use Iggy\MetadataChangeGroup; +use Iggy\MetadataChangeKind; +use Iggy\MetadataChangeStart; +use Iggy\MetadataChangeStatus; +use Iggy\MetadataStreamChange; use Iggy\PollingStrategy; use Iggy\SendMessage; use PHPUnit\Framework\TestCase; @@ -72,6 +77,36 @@ } } + public function testMetadataChangesOverTls(): void + { + $connectionString = getenv('IGGY_TLS_CONNECTION_STRING'); + if ($connectionString === false || $connectionString === '') { + $this->markTestSkipped('set IGGY_TLS_CONNECTION_STRING to run TLS tests'); + } + + $client = IggyClient::fromConnectionString($connectionString); + $client->connect(); + $groups = [MetadataChangeGroup::Streams]; + $latest = $client->getMetadataChanges(MetadataChangeStart::latest(), $groups); + assert_same(MetadataChangeStatus::Ok, $latest->status); + $streamName = unique_name('tls-changes-stream'); + + try { + $client->createStream($streamName); + $page = $client->getMetadataChanges(MetadataChangeStart::after($latest->next), $groups); + assert_same(MetadataChangeStatus::Ok, $page->status); + $created = array_values(array_filter( + $page->records, + static fn ($record): bool => $record->change instanceof MetadataStreamChange + && $record->change->name === $streamName, + )); + assert_count(1, $created); + assert_same(MetadataChangeKind::StreamCreated, $created[0]->kind); + } finally { + cleanup_stream($client, $streamName); + } + } + public function testConnectWithoutTlsShouldFail(): void { $address = getenv('IGGY_TLS_PLAINTEXT_ADDRESS');
diff --git a/foreign/php/tests/bootstrap.php b/foreign/php/tests/bootstrap.php index 20ce827..5cde652 100644 --- a/foreign/php/tests/bootstrap.php +++ b/foreign/php/tests/bootstrap.php
@@ -142,13 +142,19 @@ } /** - * Logged-in client for one transport. QUIC, HTTP and WebSocket addresses default to the - * server's default ports on the TCP host. + * Logged-in client for one transport, as root unless credentials are given. QUIC, HTTP and + * WebSocket addresses default to the server's default ports on the TCP host. */ -function new_transport_client(string $transport): IggyClient +function new_transport_client(string $transport, ?string $username = null, ?string $password = null): IggyClient { + $username ??= env_or_default('IGGY_ROOT_USERNAME', 'iggy'); + $password ??= env_or_default('IGGY_ROOT_PASSWORD', 'iggy'); if ($transport === 'tcp') { - return new_client(); + $client = new IggyClient(server_address()); + $client->connect(); + $client->loginUser($username, $password); + + return $client; } $schemes = ['quic' => 'iggy+quic', 'http' => 'iggy+http', 'websocket' => 'iggy+ws']; @@ -157,13 +163,15 @@ 'http' => env_or_default('IGGY_HTTP_ADDRESS', server_host() . ':3000'), 'websocket' => env_or_default('IGGY_WEBSOCKET_ADDRESS', server_host() . ':8092'), ]; - $username = rawurlencode(env_or_default('IGGY_ROOT_USERNAME', 'iggy')); - $password = rawurlencode(env_or_default('IGGY_ROOT_PASSWORD', 'iggy')); + $encodedUsername = rawurlencode($username); + $encodedPassword = rawurlencode($password); - $client = IggyClient::fromConnectionString("{$schemes[$transport]}://{$username}:{$password}@{$addresses[$transport]}"); + $client = IggyClient::fromConnectionString( + "{$schemes[$transport]}://{$encodedUsername}:{$encodedPassword}@{$addresses[$transport]}", + ); $client->connect(); if ($transport === 'http') { - $client->loginUser(env_or_default('IGGY_ROOT_USERNAME', 'iggy'), env_or_default('IGGY_ROOT_PASSWORD', 'iggy')); + $client->loginUser($username, $password); } return $client; @@ -200,6 +208,48 @@ cleanup_stream($client, $stream); } +/** + * The SDK has no user management, so tests create a user through the raw CreateUser command + * (33): active and without permissions. + */ +function create_user_without_permissions(IggyClient $client, string $username, string $password): void +{ + $client->sendBinaryRequest( + 33, + chr(strlen($username)) . $username . chr(strlen($password)) . $password . chr(1) . chr(0), + ); +} + +/** Deletes a user by name through the raw DeleteUser command (34). */ +function delete_user_by_name(IggyClient $client, string $username): void +{ + $client->sendBinaryRequest(34, chr(2) . chr(strlen($username)) . $username); +} + +/** + * Stream id to name, from the raw GetStreams answer (201): per stream its id, created_at, + * topics_count, size, messages_count, name and the two length-prefixed option blocks. + */ +function stream_list(IggyClient $client): array +{ + $payload = $client->sendBinaryRequest(201, ''); + $streams = []; + $position = 0; + while ($position < strlen($payload)) { + $id = unpack('V', $payload, $position)[1]; + $position += 4 + 8 + 4 + 8 + 8; + $nameLength = ord($payload[$position]); + $streams[$id] = substr($payload, $position + 1, $nameLength); + $position += 1 + $nameLength; + for ($block = 0; $block < 2; $block++) { + $position += 4 + unpack('V', $payload, $position)[1]; + } + } + ksort($streams); + + return $streams; +} + function collect_payloads(array $messages): array { return array_map(static fn (ReceiveMessage $message): string => $message->payload(), $messages);
diff --git a/foreign/php/tests/fixtures/metadata_changes.json b/foreign/php/tests/fixtures/metadata_changes.json new file mode 100644 index 0000000..26e7faf --- /dev/null +++ b/foreign/php/tests/fixtures/metadata_changes.json
@@ -0,0 +1,1997 @@ +{ + "schema_version": 1, + "generated_by": "core/common/examples/metadata_changes_fixtures.rs", + "conventions": { + "byte_strings": "Every *_hex field is lowercase hex without separators.", + "expected": "expected is the JSON the HTTP route renders. Binary decoders must reach the same values.", + "record_hex": "record_hex is one record envelope as a response carries it. body_hex is the body alone.", + "refused": "Every refused vector must fail to decode or validate. Query and HTTP refusals are invalid_command.", + "request_hex": "request_hex is the command payload without framing. http_query is the query string of GET /metadata/changes.", + "u128_values": "128-bit values (log ids, checksums, client ids) are 32 lowercase hex characters in JSON.", + "u64_values": "Ops and timestamps are JSON numbers below 2^53 in every vector except large_values, which carries them as decimal strings.", + "unknown_kinds": "A record of an unknown group or kind is kept with its codes and raw body, never refused." + }, + "constants": { + "command_code": 17, + "command_name": "metadata.changes", + "cursor_prefix": "mc1_", + "page_token_prefix": "cs1_", + "cursor_size": 44, + "cursor_text_length": 63, + "max_page_token_size": 316, + "max_page_token_key_size": 260, + "default_limit": 100, + "max_limit": 1000, + "groups": { + "consumer_group_members": { + "code": 5, + "class": "activity", + "parents": [ + "consumer_groups", + "partitions", + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32][member_id:u32]" + }, + "consumer_groups": { + "code": 4, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32]" + }, + "partitions": { + "code": 3, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "personal_access_tokens": { + "code": 7, + "class": "resource", + "parents": [ + "users" + ], + "page_token_key": "[user_id:u32][name_len:u8][name]" + }, + "sessions": { + "code": 8, + "class": "activity", + "parents": [ + "users" + ], + "page_token_key": "[epoch:u64]" + }, + "streams": { + "code": 1, + "class": "resource", + "parents": [], + "page_token_key": "[stream_id:u32]" + }, + "topics": { + "code": 2, + "class": "resource", + "parents": [ + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "users": { + "code": 6, + "class": "resource", + "parents": [], + "page_token_key": "[user_id:u32]" + } + }, + "kinds": { + "consumer_group_created": { + "code": 30, + "group": "consumer_groups" + }, + "consumer_group_deleted": { + "code": 31, + "group": "consumer_groups" + }, + "consumer_group_member_joined": { + "code": 40, + "group": "consumer_group_members" + }, + "consumer_group_member_left": { + "code": 41, + "group": "consumer_group_members" + }, + "consumer_group_partition_handed_over": { + "code": 42, + "group": "consumer_group_members" + }, + "partitions_changed": { + "code": 20, + "group": "partitions" + }, + "personal_access_token_created": { + "code": 60, + "group": "personal_access_tokens" + }, + "personal_access_token_deleted": { + "code": 61, + "group": "personal_access_tokens" + }, + "session_closed": { + "code": 71, + "group": "sessions" + }, + "session_registered": { + "code": 70, + "group": "sessions" + }, + "stream_created": { + "code": 1, + "group": "streams" + }, + "stream_deleted": { + "code": 3, + "group": "streams" + }, + "stream_updated": { + "code": 2, + "group": "streams" + }, + "topic_created": { + "code": 10, + "group": "topics" + }, + "topic_deleted": { + "code": 12, + "group": "topics" + }, + "topic_updated": { + "code": 11, + "group": "topics" + }, + "user_created": { + "code": 50, + "group": "users" + }, + "user_deleted": { + "code": 52, + "group": "users" + }, + "user_password_changed": { + "code": 53, + "group": "users" + }, + "user_permissions_updated": { + "code": 54, + "group": "users" + }, + "user_updated": { + "code": 51, + "group": "users" + } + }, + "statuses": { + "behind": 1, + "expired": 2, + "ok": 0, + "reset": 3 + }, + "phases": { + "changes": 0, + "current": 1 + }, + "start_modes": { + "after": 2, + "current": 3, + "earliest": 0, + "latest": 1 + }, + "member_left_reasons": { + "left": 0, + "session_ended": 1 + }, + "token_deleted_reasons": { + "deleted": 0, + "expired": 1 + }, + "session_closed_reasons": { + "expired": 1, + "logout": 0, + "revoked": 2 + }, + "permissions_updated_reasons": { + "resource_deleted": 1, + "updated": 0 + } + }, + "records": [ + { + "case": "stream_created", + "group": 1, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "record_hex": "191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "stream_updated_unreadable_options", + "group": 1, + "kind": 2, + "body_hex": "03000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "record_hex": "1a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "expected": { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + } + }, + { + "case": "stream_deleted_deferred", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "record_hex": "1b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "expected": { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + }, + { + "case": "topic_created", + "group": 2, + "kind": 10, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "record_hex": "1c1000000000000000000000683d9e8b03f7c1a500000000000000001c00be01094106000104000000020a78000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "op": 4124, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d68", + "timestamp": 1760356800004124, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "partitions_count": 8, + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "topic_updated", + "group": 2, + "kind": 11, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "record_hex": "1d1000000000000000000000693d9e8b03f7c1a500000000000000001d00be01094106000104000000020b3d000000030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "expected": { + "op": 4125, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d69", + "timestamp": 1760356800004125, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_updated", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics-v2", + "partitions_count": 8, + "options": {} + } + }, + { + "case": "topic_deleted_immediate", + "group": 2, + "kind": 12, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "record_hex": "1e10000000000000000000006a3d9e8b03f7c1a500000000000000001e00be01094106000104000000020c2e000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "expected": { + "op": 4126, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6a", + "timestamp": 1760356800004126, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics" + } + }, + { + "case": "partitions_changed_deleted_tail", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "record_hex": "1f10000000000000000000006b3d9e8b03f7c1a500000000000000001f00be01094106000104000000031435000000030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "expected": { + "op": 4127, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6b", + "timestamp": 1760356800004127, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 5, + "removed_partition_ids": [ + 5, + 6, + 7 + ], + "requested_op": 4200 + } + }, + { + "case": "partitions_changed_created", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000a0000000000000000", + "record_hex": "2010000000000000000000006c3d9e8b03f7c1a500000000000000002000be01094106000104000000031421000000030000001910000000000000000000001d100000000000000a0000000000000000", + "expected": { + "op": 4128, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6c", + "timestamp": 1760356800004128, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 10, + "removed_partition_ids": [] + } + }, + { + "case": "consumer_group_created", + "group": 4, + "kind": 30, + "body_hex": "030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "record_hex": "2110000000000000000000006d3d9e8b03f7c1a500000000000000002100be01094106000104000000041e28000000030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "expected": { + "op": 4129, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6d", + "timestamp": 1760356800004129, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 1 + } + }, + { + "case": "consumer_group_deleted", + "group": 4, + "kind": 31, + "body_hex": "030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "record_hex": "2210000000000000000000006e3d9e8b03f7c1a500000000000000002200be01094106000104000000041f31000000030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "expected": { + "op": 4130, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6e", + "timestamp": 1760356800004130, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 0, + "requested_op": 4300 + } + }, + { + "case": "consumer_group_member_joined", + "group": 5, + "kind": 40, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "record_hex": "2310000000000000000000006f3d9e8b03f7c1a500000000000000002300be01094106000109000000052838000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "expected": { + "op": 4131, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6f", + "timestamp": 1760356800004131, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_joined", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 2, + "partition_ids": [] + } + }, + { + "case": "consumer_group_member_left_session_ended", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "record_hex": "241000000000000000000000703d9e8b03f7c1a500000000000000002400be01094106000109000000052935000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "expected": { + "op": 4132, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d70", + "timestamp": 1760356800004132, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_left", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 1, + "reason": "session_ended" + } + }, + { + "case": "consumer_group_partition_handed_over", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d1000000000000000000000020000000101000000", + "record_hex": "251000000000000000000000713d9e8b03f7c1a500000000000000002500be010941060000052a25000000030000001910000000000000000000001d1000000000000000000000020000000101000000", + "expected": { + "op": 4133, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d71", + "timestamp": 1760356800004133, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 2, + "member_id": 1 + } + }, + { + "case": "consumer_group_partition_released", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d10000000000000000000000300000000", + "record_hex": "261000000000000000000000723d9e8b03f7c1a500000000000000002600be010941060000052a21000000030000001910000000000000000000001d10000000000000000000000300000000", + "expected": { + "op": 4134, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d72", + "timestamp": 1760356800004134, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 3 + } + }, + { + "case": "user_created_with_permissions", + "group": 6, + "kind": 50, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "record_hex": "271000000000000000000000733d9e8b03f7c1a500000000000000002700be010941060001000000000632480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "op": 4135, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d73", + "timestamp": 1760356800004135, + "actor_user_id": 0, + "group": "users", + "kind": "user_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "user_updated_inactive", + "group": 6, + "kind": 51, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "record_hex": "281000000000000000000000743d9e8b03f7c1a500000000000000002800be010941060001000000000633280000000700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "expected": { + "op": 4136, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d74", + "timestamp": 1760356800004136, + "actor_user_id": 0, + "group": "users", + "kind": "user_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "inactive", + "permissions": null, + "options": {} + } + }, + { + "case": "user_deleted", + "group": 6, + "kind": 52, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "291000000000000000000000753d9e8b03f7c1a500000000000000002900be0109410600010000000006341d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4137, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d75", + "timestamp": 1760356800004137, + "actor_user_id": 0, + "group": "users", + "kind": "user_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_password_changed", + "group": 6, + "kind": 53, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "2a1000000000000000000000763d9e8b03f7c1a500000000000000002a00be0109410600010700000006351d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4138, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d76", + "timestamp": 1760356800004138, + "actor_user_id": 7, + "group": "users", + "kind": "user_password_changed", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_permissions_pruned", + "group": 6, + "kind": 54, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "record_hex": "2b1000000000000000000000773d9e8b03f7c1a500000000000000002b00be0109410600010400000006362a0000000700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "expected": { + "op": 4139, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d77", + "timestamp": 1760356800004139, + "actor_user_id": 4, + "group": "users", + "kind": "user_permissions_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": false, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": null + }, + "reason": "resource_deleted" + } + }, + { + "case": "personal_access_token_created", + "group": 7, + "kind": 60, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "record_hex": "2c1000000000000000000000783d9e8b03f7c1a500000000000000002c00be01094106000107000000073c1d0000000700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "expected": { + "op": 4140, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d78", + "timestamp": 1760356800004140, + "actor_user_id": 7, + "group": "personal_access_tokens", + "kind": "personal_access_token_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "expiry": 1760400000000000 + } + }, + { + "case": "personal_access_token_deleted_expired", + "group": 7, + "kind": 61, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e01", + "record_hex": "2d1000000000000000000000793d9e8b03f7c1a500000000000000002d00be010941060000073d160000000700000000b832c8fb4006000863692d746f6b656e01", + "expected": { + "op": 4141, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d79", + "timestamp": 1760356800004141, + "group": "personal_access_tokens", + "kind": "personal_access_token_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "reason": "expired" + } + }, + { + "case": "session_registered", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "record_hex": "2e10000000000000000000007a3d9e8b03f7c1a500000000000000002e00be010941060001070000000846240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "op": 4142, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7a", + "timestamp": 1760356800004142, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_registered", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + }, + { + "case": "session_closed_expired", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "record_hex": "2f10000000000000000000007b3d9e8b03f7c1a500000000000000002f00be0109410600000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "expected": { + "op": 4143, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7b", + "timestamp": 1760356800004143, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "expired" + } + }, + { + "case": "session_closed_revoked", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "record_hex": "3010000000000000000000007c3d9e8b03f7c1a500000000000000003000be010941060001070000000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "expected": { + "op": 4144, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7c", + "timestamp": 1760356800004144, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "revoked" + } + } + ], + "items": [ + { + "case": "stream", + "group": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "item_hex": "01b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "topic", + "group": 2, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "item_hex": "0274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "partitions", + "group": 3, + "body_hex": "030000001910000000000000000000001d1000000000000008000000", + "item_hex": "031c000000030000001910000000000000000000001d1000000000000008000000", + "expected": { + "group": "partitions", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 8 + } + }, + { + "case": "consumer_group", + "group": 4, + "body_hex": "030000001910000000000000000000001d10000000000000000000000772656164657273", + "item_hex": "0424000000030000001910000000000000000000001d10000000000000000000000772656164657273", + "expected": { + "group": "consumer_groups", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers" + } + }, + { + "case": "consumer_group_member", + "group": 5, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "item_hex": "0540000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "expected": { + "group": "consumer_group_members", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "partition_ids": [ + 0, + 2, + 4 + ] + } + }, + { + "case": "user", + "group": 6, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "item_hex": "06480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "group": "users", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "personal_access_token_without_expiry", + "group": 7, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "item_hex": "071d0000000700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "expected": { + "group": "personal_access_tokens", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token" + } + }, + { + "case": "session", + "group": 8, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "item_hex": "08240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "group": "sessions", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + } + ], + "cursors": [ + { + "case": "zero", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d000000000000000000000000000000000000000000000000000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 0, + "checksum": "00000000000000000000000000000000", + "ordinal": 0 + }, + { + "case": "first_record_of_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAABgQAAAAAAAApcH3A4ueLUwAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000001810000000000000a5c1f7038b9e2d4c000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4120, + "checksum": "00000000000000004c2d9e8b03f7c1a5", + "ordinal": 0 + }, + { + "case": "inside_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f2000000000000000001000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4187, + "checksum": "0000000000000000f2a4c6e8b0d1f359", + "ordinal": 1 + }, + { + "case": "every_field_at_maximum", + "text": "mc1___________________________________________________________8", + "hex": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + "log_id": "ffffffffffffffffffffffffffffffff", + "op": 18446744073709551615, + "checksum": "ffffffffffffffffffffffffffffffff", + "ordinal": 4294967295 + } + ], + "page_tokens": [ + { + "case": "streams", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000000101040003000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 1, + "group": "streams", + "last_key_hex": "03000000" + }, + { + "case": "topics", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 3, + "group": "topics", + "last_key_hex": "0300000000000000" + }, + { + "case": "partitions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8DCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff0308000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "partitions", + "last_key_hex": "0300000000000000" + }, + { + "case": "consumer_groups", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8EDAADAAAAAAAAAAIAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff040c00030000000000000002000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_groups", + "last_key_hex": "030000000000000002000000" + }, + { + "case": "consumer_group_members", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8FEAADAAAAAAAAAAIAAAABAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff05100003000000000000000200000001000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_group_members", + "last_key_hex": "03000000000000000200000001000000" + }, + { + "case": "users", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAACAGBAAHAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000002006040007000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 32, + "group": "users", + "last_key_hex": "07000000" + }, + { + "case": "personal_access_tokens", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAGAHDQAHAAAACGNpLXRva2Vu", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e1000000000000060070d00070000000863692d746f6b656e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 96, + "group": "personal_access_tokens", + "last_key_hex": "070000000863692d746f6b656e" + }, + { + "case": "sessions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAKAICACUEQAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000a00808009411000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 160, + "group": "sessions", + "last_key_hex": "9411000000000000" + }, + { + "case": "longest_token_name", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8HBAH______25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubg", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff070401ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "personal_access_tokens", + "last_key_hex": "ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e" + } + ], + "requests": [ + { + "case": "earliest_every_group", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000", + "http_query": "from=earliest&limit=100" + }, + { + "case": "earliest_many_duplicate_groups", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "users" + ], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000640000000201060000", + "http_query": "from=earliest&groups=streams,users&limit=100" + }, + { + "case": "earliest_since_users_and_tokens", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "users", + "personal_access_tokens" + ], + "since": 1760353200000000, + "until": null, + "limit": 100 + }, + "request_hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001004c2a2b0841060000640000000206070000", + "http_query": "from=earliest&groups=users,personal_access_tokens&since=1760353200000000&limit=100" + }, + { + "case": "latest_streams", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [ + "streams" + ], + "since": null, + "until": null, + "limit": 1 + }, + "request_hex": "01000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000001010000", + "http_query": "from=latest&groups=streams&limit=1" + }, + { + "case": "after_cursor_with_time_range", + "input": { + "start": "after", + "cursor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "page_token": null, + "groups": [ + "topics", + "streams" + ], + "since": 1760356800000000, + "until": 1760360400000000, + "limit": 500 + }, + "request_hex": "02d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000000100f0bd010941060001009451d809410600f40100000202010000", + "http_query": "after=mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA&groups=streams,topics&since=1760356800000000&until=1760360400000000&limit=500" + }, + { + "case": "current_first_page", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201020000", + "http_query": "from=current&groups=streams,topics&limit=1000" + }, + { + "case": "current_continuation", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201024000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "http_query": "from=current¤t_after=cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA&groups=streams,topics&limit=1000" + } + ], + "responses": [ + { + "case": "changes_page", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001b10000000000000673d9e8b03f7c1a50000000000000000000000000103000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465641a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d7632011b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABsQAAAAAAAAZz2eiwP3waUAAAAAAAAAAAAAAAA", + "has_more": true, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + }, + { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "unknown_kind_and_group", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001910000000000000653d9e8b03f7c1a50000000000000000010000000002000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be01094106000104000000010904000000deadbeef191000000000000001000000653d9e8b03f7c1a500000000000000001900be01094106000104000000c80104000000deadbeef000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABkQAAAAAAAAZT2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": false, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "1", + "kind": "9", + "body": "3q2+7w==" + }, + { + "op": 4121, + "ordinal": 1, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "200", + "kind": "1", + "body": "3q2+7w==" + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "current_page", + "response_hex": "000101d7931c6e2a8f4b0d0000000000000000010000000000000000000000000000005e1000000000000001d7931c6e2a8f4b0d00000000000000005b10000000000000a73d9e8b03f7c1a500000000000000000100000001000000000200000001b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465640274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c0800000000000040000000004000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "expected": { + "status": "ok", + "phase": "current", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 1, + "retained_since": 0, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAApz2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": true, + "records": [], + "items": [ + { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + ], + "current_page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA" + } + }, + { + "case": "behind_before_op_1", + "response_hex": "010000000000000000000000000000000000000000000000000000000000000000000000000000", + "expected": { + "status": "behind", + "phase": "changes", + "log_id": null, + "oldest_op": 0, + "retained_since": 0, + "head_op": 0, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "expired", + "response_hex": "020001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "expired", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "reset", + "response_hex": "030001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "reset", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + } + ], + "large_values": { + "record_hex": "0300000000002000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff01ffffffff084624000000ffffffff0100000000002000feffffffffffffffffffffffffffffff0200000000002000", + "op": "9007199254740995", + "ordinal": 4294967295, + "timestamp": "18446744073709551615", + "checksum": "ffffffffffffffffffffffffffffffff", + "user_created_at": "9007199254740993", + "session": "9007199254740994", + "client_id": "fffffffffffffffffffffffffffffffe" + }, + "refused": { + "cursors": [ + { + "case": "wrong_prefix", + "text": "mc2_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "missing_prefix", + "text": "15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "too_long", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAAA" + }, + { + "case": "too_short", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA" + }, + { + "case": "padded", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA=" + }, + { + "case": "standard_alphabet_character", + "text": "mc1_+5McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "op_zero_with_checksum", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAA" + } + ], + "page_tokens": [ + { + "case": "wrong_prefix", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA" + }, + { + "case": "trailing_byte", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAAAA" + }, + { + "case": "wrong_key_width", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBCAADAAAAAAAAAA" + }, + { + "case": "group_outside_mask", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAIBBAADAAAA" + }, + { + "case": "head_below_anchor", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAABAAAAAAAAAAEBBAADAAAA" + }, + { + "case": "empty", + "text": "cs1_" + } + ], + "record_bodies": [ + { + "case": "unknown_member_left_reason", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000002" + }, + { + "case": "presence_byte_two", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279022c10000000000000" + }, + { + "case": "unsorted_removed_partition_ids", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000090000000600000007000000016810000000000000" + }, + { + "case": "known_kind_under_wrong_group", + "group": 2, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564" + }, + { + "case": "trailing_byte", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000000" + }, + { + "case": "truncated", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a09080706050403020194110000000000" + } + ], + "requests": [ + { + "case": "cursor_bytes_without_after", + "hex": "000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "trailing_byte", + "hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006400000000000000" + }, + { + "case": "unknown_start_mode", + "hex": "040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "non_canonical_cursor", + "hex": "02d7931c6e2a8f4b0d000000000000000000000000000000000100000000000000000000000000000000000000000064000000000000" + } + ], + "responses": [ + { + "case": "items_in_changes_phase", + "hex": "000001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + }, + { + "case": "unknown_status", + "hex": "040001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + } + ], + "queries": [ + { + "case": "limit_zero", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 0 + } + }, + { + "case": "limit_above_maximum", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 1001 + } + }, + { + "case": "since_above_until", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 2, + "until": 1, + "limit": 100 + } + }, + { + "case": "time_bound_with_latest", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 1, + "until": null, + "limit": 100 + } + }, + { + "case": "time_bound_with_current", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": 1, + "limit": 100 + } + }, + { + "case": "current_missing_parent_group", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + }, + { + "case": "page_token_for_other_groups", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + } + ], + "http_queries": [ + { + "case": "no_start", + "query": "groups=streams" + }, + { + "case": "from_and_after", + "query": "from=earliest&after=mc1_x" + }, + { + "case": "unknown_from", + "query": "from=newest" + }, + { + "case": "duplicate_key", + "query": "from=earliest&from=latest" + }, + { + "case": "unknown_key", + "query": "from=earliest&verbose=1" + }, + { + "case": "empty_group_element", + "query": "from=earliest&groups=streams,,topics" + }, + { + "case": "trailing_group_comma", + "query": "from=earliest&groups=streams," + }, + { + "case": "unknown_group", + "query": "from=earliest&groups=everything" + }, + { + "case": "negative_since", + "query": "from=earliest&since=-1" + }, + { + "case": "leading_zero_until", + "query": "from=earliest&until=01" + }, + { + "case": "non_numeric_limit", + "query": "from=earliest&limit=ten" + }, + { + "case": "token_without_current", + "query": "from=latest¤t_after=cs1_AAAA" + } + ] + } +}
diff --git a/foreign/python/Cargo.toml b/foreign/python/Cargo.toml index 390e87b..93a3cd1 100644 --- a/foreign/python/Cargo.toml +++ b/foreign/python/Cargo.toml
@@ -17,7 +17,7 @@ [package] name = "apache-iggy" -version = "0.9.1-dev2" +version = "0.9.1-dev3" edition = "2024" authors = ["Iggy Committers <dev@iggy.apache.org>"] license = "Apache-2.0" @@ -37,7 +37,7 @@ [dependencies] bytes = "1.12.1" futures = "0.3.34" -iggy = { path = "../../core/sdk", version = "0.11.1-edge.2" } +iggy = { path = "../../core/sdk", version = "0.11.1-edge.3" } paste = "1" pyo3 = "0.29.3" pyo3-async-runtimes = { version = "0.29.0", features = [
diff --git a/foreign/python/README.md b/foreign/python/README.md index c51d2d4..abeb9a4 100644 --- a/foreign/python/README.md +++ b/foreign/python/README.md
@@ -273,6 +273,76 @@ The server never returns the key, only `encryption_key_source`, `encryption_key_fingerprint` and `encryption_kek_fingerprints`. Without a key, a server with encryption enabled generates one for the stream and reports the source `"server"`. An invalid key raises `ValueError` before anything is sent, and no repr or error message contains it. +## Metadata Changes + +`get_metadata_changes` reads the node's change log: one record per public result of a committed metadata operation, such as a stream created, a topic renamed, a user's permissions changed or a session closed. A mirror keeps a cursor, asks for the records after it, and stores the returned `next` cursor. The call needs root or the global `manage_servers` permission. A query the server would refuse raises `IggyError` with the invalid command code before anything is sent. + +```python +import asyncio + +from apache_iggy import ( + IggyClient, + MetadataChangeGroup, + MetadataChangeStart, + MetadataChangeStatus, +) + + +async def follow(client, groups): + start = MetadataChangeStart.Earliest() + while True: + page = await client.get_metadata_changes(start, groups=groups) + if page.status == MetadataChangeStatus.Ok: + for record in page.records: + print(record.op, record.actor_user_id, record.kind, record.change) + if page.next is not None: + start = MetadataChangeStart.After(page.next) + if not page.has_more: + await asyncio.sleep(1) + elif page.status == MetadataChangeStatus.Behind: + await asyncio.sleep(0.1) + else: + start = await rebuild(client, groups) + + +async def rebuild(client, groups): + """Pages the current state, then returns the position to follow from.""" + start = MetadataChangeStart.Current() + while True: + page = await client.get_metadata_changes(start, groups=groups) + if page.status == MetadataChangeStatus.Behind: + await asyncio.sleep(0.1) + continue + if page.status != MetadataChangeStatus.Ok: + start = MetadataChangeStart.Current() + continue + for item in page.items: + print(item.group, item.body) + if page.current_page_token is not None: + start = MetadataChangeStart.Current(page.current_page_token) + elif page.next is not None: + return MetadataChangeStart.After(page.next) + else: + start = MetadataChangeStart.Current() + + +async def main(): + client = IggyClient.from_connection_string("iggy+tcp://iggy:iggy@127.0.0.1:8090") + await client.connect() + await follow(client, [MetadataChangeGroup.Streams, MetadataChangeGroup.Topics]) + + +asyncio.run(main()) +``` + +`Expired` means the log no longer holds the history after the cursor, and `Reset` means the cursor belongs to another history, for example after the data directory was rebuilt. Either way the mirror rebuilds from current state pages and then follows the changes after the anchor they return. + +Current pages need the parent groups of every selected group, for example `Topics` with `Streams`. Keep a rebuilt state private until a change page answers `Ok` with `has_more` false and a `head_op` at or above the last current page's. + +`record.kind` names the body in `record.change`, for example a `MetadataStreamChange` for `StreamCreated` and `StreamUpdated`. A record or item this build does not know has `kind` or `group` set to `None` and a `MetadataUnknownChange` or `MetadataUnknownItem` body with its codes and raw body, so a mirror can stop there rather than skip a change. Stream, topic and user bodies expose `options` and `derived_options` as `StreamDetails` does: a stream's key source and fingerprints, never its key. + +`str()` of a cursor gives its `mc1_` text and of a page token its `cs1_` text, and `MetadataChangeCursor.parse` and `MetadataCurrentPageToken.parse` read them back, so a mirror can store its position as a string. Log ids, checksums and client ids are Python ints. + ## High-Level Producer The Python high-level producer API is a port of the Rust high-level producer
diff --git a/foreign/python/apache_iggy.pyi b/foreign/python/apache_iggy.pyi index f807d68..ef519c9 100644 --- a/foreign/python/apache_iggy.pyi +++ b/foreign/python/apache_iggy.pyi
@@ -52,6 +52,43 @@ "IggyExpiry", "IggyProducer", "MaxTopicSize", + "MetadataChangeCursor", + "MetadataChangeGroup", + "MetadataChangeKind", + "MetadataChangePhase", + "MetadataChangeRecord", + "MetadataChangeStart", + "MetadataChangeStatus", + "MetadataChanges", + "MetadataConsumerGroupCreated", + "MetadataConsumerGroupDeleted", + "MetadataConsumerGroupItem", + "MetadataCurrentItem", + "MetadataCurrentPageToken", + "MetadataMemberItem", + "MetadataMemberJoined", + "MetadataMemberLeft", + "MetadataMemberLeftReason", + "MetadataPartitionHandedOver", + "MetadataPartitionsChanged", + "MetadataPartitionsItem", + "MetadataPermissionsUpdatedReason", + "MetadataSession", + "MetadataSessionClosed", + "MetadataSessionClosedReason", + "MetadataStreamChange", + "MetadataStreamDeleted", + "MetadataTokenChange", + "MetadataTokenDeleted", + "MetadataTokenDeletedReason", + "MetadataTopicChange", + "MetadataTopicDeleted", + "MetadataTopicItem", + "MetadataUnknownChange", + "MetadataUnknownItem", + "MetadataUserChange", + "MetadataUserIdentity", + "MetadataUserPermissionsUpdated", "OptionError", "OptionKey", "OptionSpec", @@ -1244,6 +1281,41 @@ ValueError: If the scope name is not one of the three above. RuntimeError: If the request fails. """ + def get_metadata_changes( + self, + start: MetadataChangeStart, + *, + groups: builtins.list[MetadataChangeGroup] | None = None, + since: builtins.int | None = None, + until: builtins.int | None = None, + limit: builtins.int | None = None, + ) -> collections.abc.Awaitable[MetadataChanges]: + r""" + Reads one page of the metadata change log, or of the current state. + + Requires root or the global `manage_servers` permission. Keep the + page's `next` cursor and send it back in `MetadataChangeStart.After`. + + Args: + start: Where to start, as `MetadataChangeStart`. + groups: The groups to read as `list[MetadataChangeGroup] | None`. + `None` or an empty list reads every group. A current state scan + needs the parent groups of each group, for example `Topics` + with `Streams`. + since: Keep records prepared at or after this Unix time in + microseconds. Only with `Earliest` and `After`. + until: Keep records prepared at or before this Unix time in + microseconds. Only with `Earliest` and `After`. + limit: Records or items per page, 1 to 1000. Defaults to 100. + + Returns: + An awaitable that resolves to `MetadataChanges`. + + Raises: + IggyError: With the invalid command code before anything is sent + when the query is refused, or when the request fails. A user + without the permission gets the unauthorized code. + """ def login_user( self, username: builtins.str, password: builtins.str ) -> collections.abc.Awaitable[IdentityInfo]: @@ -2340,6 +2412,1018 @@ ... @typing.final +class MetadataChangeCursor: + r""" + Position of the last change record a reader consumed or scanned. Send it + back in `MetadataChangeStart.After` to read the records after it. + + `str()` gives its `mc1_` text, which `parse` reads back. The 128-bit + values are Python ints. + """ + @property + def log_id(self) -> builtins.int: + r""" + Prepare checksum of op 1, which names the history the cursor belongs to. + """ + @property + def op(self) -> builtins.int: + r""" + The operation of the last record, 0 before the first record. + """ + @property + def checksum(self) -> builtins.int: + r""" + Prepare checksum of `op`. + """ + @property + def ordinal(self) -> builtins.int: + r""" + Position of the record within `op`. + """ + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + def __hash__(self) -> builtins.int: ... + def __new__( + cls, + log_id: builtins.int, + op: builtins.int, + checksum: builtins.int, + ordinal: builtins.int, + ) -> MetadataChangeCursor: ... + @staticmethod + def parse(text: builtins.str) -> MetadataChangeCursor: + r""" + Reads a cursor from its `mc1_` text. + + Raises: + ValueError: If the text is not a canonical cursor. + """ + def __str__(self) -> builtins.str: ... + def __repr__(self) -> builtins.str: ... + +@typing.final +class MetadataChangeGroup: + r""" + Family of change records and the unit of filtering. `str()` gives the + name the HTTP API uses. + """ + + Streams: typing.ClassVar[MetadataChangeGroup] + Topics: typing.ClassVar[MetadataChangeGroup] + Partitions: typing.ClassVar[MetadataChangeGroup] + ConsumerGroups: typing.ClassVar[MetadataChangeGroup] + ConsumerGroupMembers: typing.ClassVar[MetadataChangeGroup] + Users: typing.ClassVar[MetadataChangeGroup] + PersonalAccessTokens: typing.ClassVar[MetadataChangeGroup] + Sessions: typing.ClassVar[MetadataChangeGroup] + def __str__(self) -> builtins.str: ... + +@typing.final +class MetadataChangeKind: + r""" + Kind of a change record. `str()` gives the name the HTTP API uses. + """ + + StreamCreated: typing.ClassVar[MetadataChangeKind] + StreamUpdated: typing.ClassVar[MetadataChangeKind] + StreamDeleted: typing.ClassVar[MetadataChangeKind] + TopicCreated: typing.ClassVar[MetadataChangeKind] + TopicUpdated: typing.ClassVar[MetadataChangeKind] + TopicDeleted: typing.ClassVar[MetadataChangeKind] + PartitionsChanged: typing.ClassVar[MetadataChangeKind] + ConsumerGroupCreated: typing.ClassVar[MetadataChangeKind] + ConsumerGroupDeleted: typing.ClassVar[MetadataChangeKind] + ConsumerGroupMemberJoined: typing.ClassVar[MetadataChangeKind] + ConsumerGroupMemberLeft: typing.ClassVar[MetadataChangeKind] + ConsumerGroupPartitionHandedOver: typing.ClassVar[MetadataChangeKind] + UserCreated: typing.ClassVar[MetadataChangeKind] + UserUpdated: typing.ClassVar[MetadataChangeKind] + UserDeleted: typing.ClassVar[MetadataChangeKind] + UserPasswordChanged: typing.ClassVar[MetadataChangeKind] + UserPermissionsUpdated: typing.ClassVar[MetadataChangeKind] + PersonalAccessTokenCreated: typing.ClassVar[MetadataChangeKind] + PersonalAccessTokenDeleted: typing.ClassVar[MetadataChangeKind] + SessionRegistered: typing.ClassVar[MetadataChangeKind] + SessionClosed: typing.ClassVar[MetadataChangeKind] + def __str__(self) -> builtins.str: ... + +@typing.final +class MetadataChangePhase: + r""" + Whether a page carries change records or current state items. + """ + + Changes: typing.ClassVar[MetadataChangePhase] + Current: typing.ClassVar[MetadataChangePhase] + def __str__(self) -> builtins.str: ... + +@typing.final +class MetadataChangeRecord: + r""" + One change record: the committed operation that produced it, its position + within that operation and the typed body of what is true after it. + """ + @property + def op(self) -> builtins.int: + r""" + The metadata operation that produced the record. + """ + @property + def ordinal(self) -> builtins.int: + r""" + Position of the record within `op`. + """ + @property + def checksum(self) -> builtins.int: + r""" + Prepare checksum of `op`. + """ + @property + def timestamp(self) -> builtins.int: + r""" + Prepare timestamp of `op` in microseconds. + """ + @property + def actor_user_id(self) -> builtins.int | None: + r""" + The acting user, `None` for server actions. + """ + @property + def group(self) -> MetadataChangeGroup | None: + r""" + The group, `None` for a record this build does not know. + """ + @property + def kind(self) -> MetadataChangeKind | None: + r""" + The kind, `None` for a record this build does not know. + """ + @property + def change( + self, + ) -> ( + MetadataStreamChange + | MetadataStreamDeleted + | MetadataTopicChange + | MetadataTopicDeleted + | MetadataPartitionsChanged + | MetadataConsumerGroupCreated + | MetadataConsumerGroupDeleted + | MetadataMemberJoined + | MetadataMemberLeft + | MetadataPartitionHandedOver + | MetadataUserChange + | MetadataUserIdentity + | MetadataUserPermissionsUpdated + | MetadataTokenChange + | MetadataTokenDeleted + | MetadataSession + | MetadataSessionClosed + | MetadataUnknownChange + ): + r""" + The typed body `kind` names, or `MetadataUnknownChange`. + """ + def __repr__(self) -> builtins.str: ... + +class MetadataChangeStart: + r""" + Where `get_metadata_changes` starts reading. + """ + @typing.final + class Earliest(MetadataChangeStart): + r""" + The oldest operation the requested groups still hold completely. + """ + + __match_args__ = () + def __new__(cls) -> MetadataChangeStart.Earliest: ... + def __len__(self) -> builtins.int: ... + def __getitem__(self, key: builtins.int, /) -> typing.Any: ... + + @typing.final + class Latest(MetadataChangeStart): + r""" + No records, only a cursor at the newest retained record. + """ + + __match_args__ = () + def __new__(cls) -> MetadataChangeStart.Latest: ... + def __len__(self) -> builtins.int: ... + def __getitem__(self, key: builtins.int, /) -> typing.Any: ... + + @typing.final + class After(MetadataChangeStart): + r""" + The records after `cursor`. + """ + + __match_args__ = ("cursor",) + @property + def cursor(self) -> MetadataChangeCursor: ... + def __new__(cls, cursor: MetadataChangeCursor) -> MetadataChangeStart.After: ... + + @typing.final + class Current(MetadataChangeStart): + r""" + Current state pages: `None` starts a scan, a token continues it. + """ + + __match_args__ = ("page_token",) + @property + def page_token(self) -> MetadataCurrentPageToken | None: ... + def __new__( + cls, page_token: MetadataCurrentPageToken | None = None + ) -> MetadataChangeStart.Current: ... + + ... + +@typing.final +class MetadataChangeStatus: + r""" + Outcome of a change log query. + """ + + Ok: typing.ClassVar[MetadataChangeStatus] + r""" + Records or items after the requested position. Apply them and keep + `next`. + """ + Behind: typing.ClassVar[MetadataChangeStatus] + r""" + The answering node has not applied the requested position yet. Ask + again later or ask another node. + """ + Expired: typing.ClassVar[MetadataChangeStatus] + r""" + The node no longer holds every record after the position. Rebuild + from current state. + """ + Reset: typing.ClassVar[MetadataChangeStatus] + r""" + The position belongs to another history. Rebuild from current state. + """ + def __str__(self) -> builtins.str: ... + +@typing.final +class MetadataChanges: + r""" + One page of the metadata change log, or of the current state, as + `IggyClient.get_metadata_changes` returns it. + + The page is converted once: `records` and `items` hand back the same + objects on every access. + """ + @property + def log_id(self) -> builtins.int | None: + r""" + This node's log id, or `None` before it applied op 1. + """ + @property + def oldest_op(self) -> builtins.int: + r""" + Lowest whole operation from which the requested groups are complete. + """ + @property + def retained_since(self) -> builtins.int: + r""" + The requested groups are complete for every timestamp above this + value in microseconds, 0 when nothing was evicted. + """ + @property + def head_op(self) -> builtins.int: + r""" + Highest metadata operation the answering node applied. + """ + @property + def has_more(self) -> builtins.bool: + r""" + More entries remain after `next`, or another state page exists. + """ + @property + def status(self) -> MetadataChangeStatus: ... + @property + def phase(self) -> MetadataChangePhase: + r""" + `Changes` for `Earliest`, `Latest` and `After`, `Current` for state + pages. + """ + @property + def next(self) -> MetadataChangeCursor | None: + r""" + The cursor to send back. Present on `Ok`, echoed on `Behind`, `None` + on `Expired` and `Reset`. The last state page returns the anchor to + follow from. + """ + @property + def records(self) -> builtins.list[MetadataChangeRecord]: + r""" + Change records in `(op, ordinal)` order, empty on state pages. + """ + @property + def items(self) -> builtins.list[MetadataCurrentItem]: + r""" + Current state items, empty on change pages. + """ + @property + def current_page_token(self) -> MetadataCurrentPageToken | None: + r""" + Continuation of a current state scan, `None` once it is complete. + """ + def __repr__(self) -> builtins.str: ... + +@typing.final +class MetadataConsumerGroupCreated: + r""" + A created consumer group with the topic's resulting group count. + """ + @property + def stream_id(self) -> builtins.int: ... + @property + def stream_created_op(self) -> builtins.int: ... + @property + def topic_id(self) -> builtins.int: ... + @property + def topic_created_op(self) -> builtins.int: ... + @property + def group_id(self) -> builtins.int: ... + @property + def name(self) -> builtins.str: ... + @property + def consumer_groups_count(self) -> builtins.int: ... + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataConsumerGroupDeleted: + r""" + A removed consumer group with the topic's resulting group count. + """ + @property + def stream_id(self) -> builtins.int: ... + @property + def stream_created_op(self) -> builtins.int: ... + @property + def topic_id(self) -> builtins.int: ... + @property + def topic_created_op(self) -> builtins.int: ... + @property + def group_id(self) -> builtins.int: ... + @property + def name(self) -> builtins.str: ... + @property + def consumer_groups_count(self) -> builtins.int: ... + @property + def requested_op(self) -> builtins.int | None: + r""" + The operation of the delete request when the delete completed later. + """ + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataConsumerGroupItem: + r""" + A `consumer_groups` current state item. + """ + @property + def stream_id(self) -> builtins.int: ... + @property + def stream_created_op(self) -> builtins.int: ... + @property + def topic_id(self) -> builtins.int: ... + @property + def topic_created_op(self) -> builtins.int: ... + @property + def group_id(self) -> builtins.int: ... + @property + def name(self) -> builtins.str: ... + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataCurrentItem: + r""" + One current state item of a `MetadataChangeStart.Current` page. + """ + @property + def group(self) -> MetadataChangeGroup | None: + r""" + The group, `None` for an item this build does not know. + """ + @property + def body( + self, + ) -> ( + MetadataStreamChange + | MetadataTopicItem + | MetadataPartitionsItem + | MetadataConsumerGroupItem + | MetadataMemberItem + | MetadataUserChange + | MetadataTokenChange + | MetadataSession + | MetadataUnknownItem + ): + r""" + The typed body `group` names, or `MetadataUnknownItem`. + """ + def __repr__(self) -> builtins.str: ... + +@typing.final +class MetadataCurrentPageToken: + r""" + Opaque continuation of a current state scan. Send it back unchanged in + `MetadataChangeStart.Current` with the same groups. + + `str()` gives its `cs1_` text, which `parse` reads back. + """ + @property + def anchor(self) -> MetadataChangeCursor: + r""" + The cursor captured before the scan's first page. + """ + @property + def required_head(self) -> builtins.int: + r""" + The highest head any page of the scan reported so far. + """ + @property + def groups(self) -> builtins.list[MetadataChangeGroup]: + r""" + The groups the scan was started with, in code order. + """ + @property + def last_group(self) -> MetadataChangeGroup: + r""" + The group of the last item returned. + """ + @property + def last_key(self) -> bytes: + r""" + The key of the last item returned, in the page token key layout. + """ + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + @staticmethod + def parse(text: builtins.str) -> MetadataCurrentPageToken: + r""" + Reads a page token from its `cs1_` text. + + Raises: + ValueError: If the text is not a canonical page token. + """ + def __str__(self) -> builtins.str: ... + def __repr__(self) -> builtins.str: ... + +@typing.final +class MetadataMemberItem: + r""" + A `consumer_group_members` current state item. + """ + @property + def stream_id(self) -> builtins.int: ... + @property + def stream_created_op(self) -> builtins.int: ... + @property + def topic_id(self) -> builtins.int: ... + @property + def topic_created_op(self) -> builtins.int: ... + @property + def group_id(self) -> builtins.int: ... + @property + def member_id(self) -> builtins.int: ... + @property + def client_id(self) -> builtins.int: ... + @property + def partition_ids(self) -> builtins.list[builtins.int]: + r""" + Installed partitions, ascending. + """ + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataMemberJoined: + r""" + A member that joined a consumer group. + """ + @property + def stream_id(self) -> builtins.int: ... + @property + def stream_created_op(self) -> builtins.int: ... + @property + def topic_id(self) -> builtins.int: ... + @property + def topic_created_op(self) -> builtins.int: ... + @property + def group_id(self) -> builtins.int: ... + @property + def member_id(self) -> builtins.int: ... + @property + def client_id(self) -> builtins.int: ... + @property + def members_count(self) -> builtins.int: ... + @property + def partition_ids(self) -> builtins.list[builtins.int]: + r""" + Installed partitions, normally none because a handover installs them. + """ + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataMemberLeft: + r""" + A member that left a consumer group. + """ + @property + def stream_id(self) -> builtins.int: ... + @property + def stream_created_op(self) -> builtins.int: ... + @property + def topic_id(self) -> builtins.int: ... + @property + def topic_created_op(self) -> builtins.int: ... + @property + def group_id(self) -> builtins.int: ... + @property + def member_id(self) -> builtins.int: ... + @property + def client_id(self) -> builtins.int: ... + @property + def members_count(self) -> builtins.int: ... + @property + def reason(self) -> MetadataMemberLeftReason: ... + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataMemberLeftReason: + r""" + Why a consumer group member was removed. + """ + + Left: typing.ClassVar[MetadataMemberLeftReason] + SessionEnded: typing.ClassVar[MetadataMemberLeftReason] + def __str__(self) -> builtins.str: ... + +@typing.final +class MetadataPartitionHandedOver: + r""" + A completed partition handover. + """ + @property + def stream_id(self) -> builtins.int: ... + @property + def stream_created_op(self) -> builtins.int: ... + @property + def topic_id(self) -> builtins.int: ... + @property + def topic_created_op(self) -> builtins.int: ... + @property + def group_id(self) -> builtins.int: ... + @property + def partition_id(self) -> builtins.int: ... + @property + def member_id(self) -> builtins.int | None: + r""" + The new owner, `None` when the last member left. + """ + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataPartitionsChanged: + r""" + A partition count change. + """ + @property + def stream_id(self) -> builtins.int: ... + @property + def stream_created_op(self) -> builtins.int: ... + @property + def topic_id(self) -> builtins.int: ... + @property + def topic_created_op(self) -> builtins.int: ... + @property + def partitions_count(self) -> builtins.int: + r""" + The partition count after the change. + """ + @property + def removed_partition_ids(self) -> builtins.list[builtins.int]: + r""" + The deleted partition ids, ascending, empty on create. + """ + @property + def requested_op(self) -> builtins.int | None: + r""" + The operation of the delete request when the delete completed later. + """ + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataPartitionsItem: + r""" + A `partitions` current state item. + """ + @property + def stream_id(self) -> builtins.int: ... + @property + def stream_created_op(self) -> builtins.int: ... + @property + def topic_id(self) -> builtins.int: ... + @property + def topic_created_op(self) -> builtins.int: ... + @property + def partitions_count(self) -> builtins.int: ... + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataPermissionsUpdatedReason: + r""" + Why a user's public permission set changed. + """ + + Updated: typing.ClassVar[MetadataPermissionsUpdatedReason] + r""" + An `update_permissions` call replaced the set. + """ + ResourceDeleted: typing.ClassVar[MetadataPermissionsUpdatedReason] + r""" + A stream or topic delete pruned the user's grants on it. + """ + def __str__(self) -> builtins.str: ... + +@typing.final +class MetadataSession: + r""" + A registered session, and a `sessions` current state item. + """ + @property + def user_id(self) -> builtins.int: ... + @property + def user_created_at(self) -> builtins.int: ... + @property + def client_id(self) -> builtins.int: ... + @property + def session(self) -> builtins.int: + r""" + The session epoch. + """ + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataSessionClosed: + r""" + A closed session. + """ + @property + def user_id(self) -> builtins.int: ... + @property + def user_created_at(self) -> builtins.int: ... + @property + def client_id(self) -> builtins.int: ... + @property + def session(self) -> builtins.int: ... + @property + def reason(self) -> MetadataSessionClosedReason: ... + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataSessionClosedReason: + r""" + Why a session was closed. + """ + + Logout: typing.ClassVar[MetadataSessionClosedReason] + Expired: typing.ClassVar[MetadataSessionClosedReason] + Revoked: typing.ClassVar[MetadataSessionClosedReason] + r""" + The user was deactivated or deleted. + """ + def __str__(self) -> builtins.str: ... + +@typing.final +class MetadataStreamChange: + r""" + A stream after a create or update, and a `streams` current state item. + """ + @property + def stream_id(self) -> builtins.int: + r""" + Stream id. Ids are reused after a delete. + """ + @property + def stream_created_op(self) -> builtins.int: + r""" + The operation that created the stream, which tells it from an earlier + stream with the same id. + """ + @property + def created_at(self) -> builtins.int: + r""" + Stream creation time as Unix time in microseconds. + """ + @property + def name(self) -> builtins.str: ... + @property + def options(self) -> UserHeaders | None: + r""" + Options the creating client set explicitly, the same mapping + `StreamDetails.options` returns, or `None` when `options_unreadable`. + The stream's encryption key never appears here. + """ + @property + def derived_options(self) -> UserHeaders | None: + r""" + Options the server resolved or computed, including the key source and + fingerprints. Same shape as `options`. + """ + @property + def options_unreadable(self) -> builtins.bool: + r""" + The server could not read the stream's stored options. Keep the + previous values. + """ + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataStreamDeleted: + r""" + A removed stream. + """ + @property + def stream_id(self) -> builtins.int: ... + @property + def stream_created_op(self) -> builtins.int: ... + @property + def created_at(self) -> builtins.int: ... + @property + def name(self) -> builtins.str: ... + @property + def requested_op(self) -> builtins.int | None: + r""" + The operation of the delete request when the delete completed later. + """ + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataTokenChange: + r""" + A created personal access token, and a `personal_access_tokens` current + state item. Never the token itself. + """ + @property + def user_id(self) -> builtins.int: ... + @property + def user_created_at(self) -> builtins.int: ... + @property + def name(self) -> builtins.str: ... + @property + def expiry(self) -> builtins.int | None: + r""" + Expiry in microseconds, `None` for a token that never expires. + """ + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataTokenDeleted: + r""" + A removed personal access token. + """ + @property + def user_id(self) -> builtins.int: ... + @property + def user_created_at(self) -> builtins.int: ... + @property + def name(self) -> builtins.str: ... + @property + def reason(self) -> MetadataTokenDeletedReason: ... + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataTokenDeletedReason: + r""" + Why a personal access token was removed. + """ + + Deleted: typing.ClassVar[MetadataTokenDeletedReason] + Expired: typing.ClassVar[MetadataTokenDeletedReason] + def __str__(self) -> builtins.str: ... + +@typing.final +class MetadataTopicChange: + r""" + A topic after a create or update. + """ + @property + def stream_id(self) -> builtins.int: ... + @property + def stream_created_op(self) -> builtins.int: ... + @property + def topic_id(self) -> builtins.int: ... + @property + def topic_created_op(self) -> builtins.int: + r""" + The operation that created the topic, which tells it from an earlier + topic with the same id. + """ + @property + def created_at(self) -> builtins.int: ... + @property + def name(self) -> builtins.str: ... + @property + def partitions_count(self) -> builtins.int: ... + @property + def options(self) -> UserHeaders | None: + r""" + Options the creating client set explicitly, the same mapping + `TopicDetails.options` returns, or `None` when `options_unreadable`. + """ + @property + def derived_options(self) -> UserHeaders | None: + r""" + Options the server resolved for the keys the client did not send. + """ + @property + def options_unreadable(self) -> builtins.bool: + r""" + The server could not read the topic's stored options. Keep the + previous values. + """ + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataTopicDeleted: + r""" + A removed topic. + """ + @property + def stream_id(self) -> builtins.int: ... + @property + def stream_created_op(self) -> builtins.int: ... + @property + def topic_id(self) -> builtins.int: ... + @property + def topic_created_op(self) -> builtins.int: ... + @property + def created_at(self) -> builtins.int: ... + @property + def name(self) -> builtins.str: ... + @property + def requested_op(self) -> builtins.int | None: + r""" + The operation of the delete request when the delete completed later. + """ + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataTopicItem: + r""" + A `topics` current state item. Partition counts come as `partitions` + items. + """ + @property + def stream_id(self) -> builtins.int: ... + @property + def stream_created_op(self) -> builtins.int: ... + @property + def topic_id(self) -> builtins.int: ... + @property + def topic_created_op(self) -> builtins.int: ... + @property + def created_at(self) -> builtins.int: ... + @property + def name(self) -> builtins.str: ... + @property + def options(self) -> UserHeaders | None: + r""" + Options the creating client set explicitly, or `None` when + `options_unreadable`. + """ + @property + def derived_options(self) -> UserHeaders | None: + r""" + Options the server resolved for the keys the client did not send. + """ + @property + def options_unreadable(self) -> builtins.bool: + r""" + The server could not read the topic's stored options. + """ + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataUnknownChange: + r""" + A record of a group or kind this build does not know. A mirror stops on + it rather than skip a possible state change. + """ + @property + def group(self) -> builtins.int | builtins.str: + r""" + The group code on binary transports, or the group name over HTTP. + """ + @property + def kind(self) -> builtins.int | builtins.str: + r""" + The kind code on binary transports, or the kind name over HTTP. + """ + @property + def body(self) -> builtins.bytes | builtins.str: + r""" + The raw body on binary transports, or the record's whole JSON object + over HTTP. + """ + +@typing.final +class MetadataUnknownItem: + r""" + A current state item of a group this build does not know. + """ + @property + def group(self) -> builtins.int | builtins.str: + r""" + The group code on binary transports, or the group name over HTTP. + """ + @property + def body(self) -> builtins.bytes | builtins.str: + r""" + The raw body on binary transports, or the item's whole JSON object + over HTTP. + """ + +@typing.final +class MetadataUserChange: + r""" + A user after a create or update, and a `users` current state item. + """ + @property + def user_id(self) -> builtins.int: + r""" + User id. Ids are reused after a delete. + """ + @property + def user_created_at(self) -> builtins.int: + r""" + User creation time in microseconds, which tells the user from an + earlier user with the same id. + """ + @property + def username(self) -> builtins.str: ... + @property + def status(self) -> UserStatus: ... + @property + def permissions(self) -> Permissions | None: + r""" + The public permission set, or `None` when the user has none. + """ + @property + def options(self) -> UserHeaders | None: + r""" + Options the creating client set explicitly, or `None` when + `options_unreadable`. + """ + @property + def derived_options(self) -> UserHeaders | None: + r""" + Options the server resolved for the keys the client did not send. + """ + @property + def options_unreadable(self) -> builtins.bool: + r""" + The server could not read the user's stored options. + """ + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataUserIdentity: + r""" + A deleted user, or one whose password changed. + """ + @property + def user_id(self) -> builtins.int: ... + @property + def user_created_at(self) -> builtins.int: ... + @property + def username(self) -> builtins.str: ... + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final +class MetadataUserPermissionsUpdated: + r""" + A user's public permission set after a change. + """ + @property + def user_id(self) -> builtins.int: ... + @property + def user_created_at(self) -> builtins.int: ... + @property + def username(self) -> builtins.str: ... + @property + def permissions(self) -> Permissions | None: + r""" + The public permission set after the change, or `None` when the user + has none. + """ + @property + def reason(self) -> MetadataPermissionsUpdatedReason: ... + def __eq__(self, other: builtins.object, /) -> builtins.bool: ... + +@typing.final class OptionError(IggyError): r""" An option refusal, with the server error code and its snake_case name as `kind`. @@ -2348,6 +3432,84 @@ def kind(self) -> builtins.str: ... @typing.final +class OptionKey: + r""" + A catalog option key, as `update_stream`, `update_topic` and `update_user` + take it in `reset`. `str()` gives the key as it travels on the wire. + """ + + Name: typing.ClassVar[OptionKey] + r""" + Stream or topic name. + """ + Username: typing.ClassVar[OptionKey] + r""" + User name. + """ + Status: typing.ClassVar[OptionKey] + r""" + User status. + """ + CompressionAlgorithm: typing.ClassVar[OptionKey] + r""" + Topic compression algorithm, fixed at creation. + """ + MessageExpiry: typing.ClassVar[OptionKey] + r""" + Topic message expiry, resettable to never expire. + """ + MaxTopicSize: typing.ClassVar[OptionKey] + r""" + Topic size cap, resettable to unlimited. + """ + SegmentSize: typing.ClassVar[OptionKey] + r""" + Topic segment size, fixed at creation. + """ + Durability: typing.ClassVar[OptionKey] + r""" + Topic message durability, fixed at creation. + """ + ConsumerOffsetDurability: typing.ClassVar[OptionKey] + r""" + Topic consumer offset durability, fixed at creation. + """ + PartitionResizePolicy: typing.ClassVar[OptionKey] + r""" + Topic partition resize policy, fixed at creation. + """ + MessagesRequiredToSave: typing.ClassVar[OptionKey] + r""" + Topic message-count flush threshold, fixed at creation. + """ + SizeOfMessagesRequiredToSave: typing.ClassVar[OptionKey] + r""" + Topic byte flush threshold, fixed at creation. + """ + PreallocateSegments: typing.ClassVar[OptionKey] + r""" + Topic segment preallocation, fixed at creation. + """ + EncryptionKey: typing.ClassVar[OptionKey] + r""" + Stream encryption key, write-only and fixed at creation. + """ + EncryptionKeySource: typing.ClassVar[OptionKey] + r""" + Who chose the stream's key, computed by the server. + """ + EncryptionKeyFingerprint: typing.ClassVar[OptionKey] + r""" + Fingerprint of the stream's key, computed by the server. + """ + EncryptionKekFingerprint: typing.ClassVar[OptionKey] + r""" + Fingerprints of the server keys wrapping the stream's key, computed by + the server. + """ + def __str__(self) -> builtins.str: ... + +@typing.final class OptionSpec: r""" One entry of a resource's option catalog, as served by `describe_options`. @@ -2484,6 +3646,21 @@ def __repr__(self) -> builtins.str: ... @typing.final +class PartitionResizePolicy: + r""" + Whether partitions can be added to or deleted from a topic after creation. + """ + + Mutable: typing.ClassVar[PartitionResizePolicy] + r""" + Partitions can be added and deleted. + """ + Fixed: typing.ClassVar[PartitionResizePolicy] + r""" + The partition set is fixed at topic creation. + """ + +@typing.final class Partitioning: r""" Defines how a batch of messages is assigned to a topic partition. @@ -2649,6 +3826,15 @@ """ @typing.final +class ProducerSharding: + r""" + How a background producer distributes sends among its workers. + """ + + ORDERED: typing.ClassVar[ProducerSharding] + BALANCED: typing.ClassVar[ProducerSharding] + +@typing.final class QuicConfig: r""" Configuration for the QUIC transport, accepted by `IggyClient(...)`. @@ -3912,6 +5098,21 @@ """ @typing.final +class UserStatus: + r""" + The status of a user account. + """ + + Active: typing.ClassVar[UserStatus] + r""" + The user account is active and can be used. + """ + Inactive: typing.ClassVar[UserStatus] + r""" + The user account is inactive and cannot be used. + """ + +@typing.final class WebSocketConfig: r""" Configuration for the WebSocket transport, accepted by `IggyClient(...)`. @@ -4093,124 +5294,6 @@ """ def __repr__(self) -> builtins.str: ... -@typing.final -class OptionKey(enum.Enum): - r""" - A catalog option key, as `update_stream`, `update_topic` and `update_user` - take it in `reset`. `str()` gives the key as it travels on the wire. - """ - - Name = ... - r""" - Stream or topic name. - """ - Username = ... - r""" - User name. - """ - Status = ... - r""" - User status. - """ - CompressionAlgorithm = ... - r""" - Topic compression algorithm, fixed at creation. - """ - MessageExpiry = ... - r""" - Topic message expiry, resettable to never expire. - """ - MaxTopicSize = ... - r""" - Topic size cap, resettable to unlimited. - """ - SegmentSize = ... - r""" - Topic segment size, fixed at creation. - """ - Durability = ... - r""" - Topic message durability, fixed at creation. - """ - ConsumerOffsetDurability = ... - r""" - Topic consumer offset durability, fixed at creation. - """ - PartitionResizePolicy = ... - r""" - Topic partition resize policy, fixed at creation. - """ - MessagesRequiredToSave = ... - r""" - Topic message-count flush threshold, fixed at creation. - """ - SizeOfMessagesRequiredToSave = ... - r""" - Topic byte flush threshold, fixed at creation. - """ - PreallocateSegments = ... - r""" - Topic segment preallocation, fixed at creation. - """ - EncryptionKey = ... - r""" - Stream encryption key, write-only and fixed at creation. - """ - EncryptionKeySource = ... - r""" - Who chose the stream's key, computed by the server. - """ - EncryptionKeyFingerprint = ... - r""" - Fingerprint of the stream's key, computed by the server. - """ - EncryptionKekFingerprint = ... - r""" - Fingerprints of the server keys wrapping the stream's key, computed by - the server. - """ - - def __str__(self) -> builtins.str: ... - -@typing.final -class PartitionResizePolicy(enum.Enum): - r""" - Whether partitions can be added to or deleted from a topic after creation. - """ - - Mutable = ... - r""" - Partitions can be added and deleted. - """ - Fixed = ... - r""" - The partition set is fixed at topic creation. - """ - -@typing.final -class ProducerSharding(enum.Enum): - r""" - How a background producer distributes sends among its workers. - """ - - ORDERED = ... - BALANCED = ... - -@typing.final -class UserStatus(enum.Enum): - r""" - The status of a user account. - """ - - Active = ... - r""" - The user account is active and can be used. - """ - Inactive = ... - r""" - The user account is inactive and cannot be used. - """ - def encryption_key_fingerprint(key: builtins.bytes | builtins.str) -> builtins.str: r""" Fingerprint of a stream encryption key: the first 8 bytes of SHA-256 over
diff --git a/foreign/python/pyproject.toml b/foreign/python/pyproject.toml index 3e41184..5914cdc 100644 --- a/foreign/python/pyproject.toml +++ b/foreign/python/pyproject.toml
@@ -22,7 +22,7 @@ [project] name = "apache-iggy" requires-python = ">=3.11" -version = "0.9.1.dev2" +version = "0.9.1.dev3" description = "Apache Iggy is the persistent message streaming platform written in Rust, supporting QUIC, TCP and HTTP transport protocols, capable of processing millions of messages per second." readme = "README.md" license = { file = "LICENSE" }
diff --git a/foreign/python/src/bin/stub_gen.rs b/foreign/python/src/bin/stub_gen.rs index f753191..9f295e2 100644 --- a/foreign/python/src/bin/stub_gen.rs +++ b/foreign/python/src/bin/stub_gen.rs
@@ -17,12 +17,17 @@ */ use std::{ + collections::{HashMap, HashSet}, fs::File, io::{Read, Write}, + mem, path::Path, }; -use pyo3_stub_gen::Result; +use pyo3_stub_gen::{ + Result, TypeInfo, + generate::{ClassDef, EnumDef, MemberDef}, +}; const LICENSE: &str = "# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -45,7 +50,15 @@ fn main() -> Result<()> { // `stub_info` is a function defined by `define_stub_info_gatherer!` macro. - let stub = apache_iggy::client::stub_info()?; + let mut stub = apache_iggy::client::stub_info()?; + for module in stub.modules.values_mut() { + let enums = mem::take(&mut module.enum_); + module.class.extend( + enums + .into_iter() + .map(|(type_id, enum_def)| (type_id, native_class(enum_def))), + ); + } stub.generate()?; // Anchor on the manifest dir: `stub.generate()` writes to the crate root, so // a cwd-relative path leaves the tracked stub without its license header. @@ -54,8 +67,12 @@ let mut content = LICENSE.to_owned(); f.read_to_string(&mut content)?; content = content.replacen("__all__ = [", "__all__ = [\n \"Durability\",", 1); + // Durability is the only `enum.Enum` left once the pyo3 enums are plain classes. + content = content.replacen("import typing\n", "import enum\nimport typing\n", 1); // pyo3-stub-gen splits the hyphenated project name around a base class from this module. content = content.replace("(apacheIggyError)", "(IggyError)"); + // The same split hits a pyclass-typed field of a complex enum variant. + content = content.replace("apacheMetadataChangeCursor", "MetadataChangeCursor"); // The runtime creates a str-backed Enum, whose str() differs from StrEnum. content.push_str("\nclass Durability(str, enum.Enum): # noqa: UP042\n REPLICATED = 'replicated'\n PERSISTED = 'persisted'\n"); @@ -63,3 +80,49 @@ f.write_all(content.as_bytes())?; Ok(()) } + +/// A pyo3 enum is a plain class whose variants are class attributes, not an `enum.Enum`. +fn native_class(enum_def: EnumDef) -> ClassDef { + let variant_type = TypeInfo { + name: format!("typing.ClassVar[{}]", enum_def.name), + source_module: None, + import: HashSet::from(["typing".into()]), + type_refs: HashMap::new(), + }; + let variants = enum_def.variants.iter().map(|&(name, doc)| MemberDef { + name, + r#type: variant_type.clone(), + doc, + default: None, + deprecated: None, + }); + let mut class = ClassDef { + name: enum_def.name, + module: enum_def.module, + doc: enum_def.doc, + attrs: variants.chain(enum_def.attrs).collect(), + getter_setters: Default::default(), + methods: Default::default(), + bases: Vec::new(), + classes: Vec::new(), + match_args: None, + subclass: false, + }; + for getter in enum_def.getters { + class + .getter_setters + .insert(getter.name.to_string(), (Some(getter), None)); + } + for setter in enum_def.setters { + let name = setter.name.to_string(); + class.getter_setters.entry(name).or_default().1 = Some(setter); + } + for method in enum_def.methods { + class + .methods + .entry(method.name.to_string()) + .or_default() + .push(method); + } + class +}
diff --git a/foreign/python/src/client.rs b/foreign/python/src/client.rs index 04b0aef..8faa778 100644 --- a/foreign/python/src/client.rs +++ b/foreign/python/src/client.rs
@@ -42,6 +42,9 @@ }; use crate::duration::{py_delta_to_iggy_duration, reject_zero}; use crate::identifier::PyIdentifier; +use crate::metadata_changes::{ + MetadataChangeGroup, MetadataChangeStart, MetadataChanges, validated_query, +}; use crate::options::OptionSpec as PyOptionSpec; use crate::partitioning::PyPartitioning; use crate::permissions::Permissions as PyPermissions; @@ -345,6 +348,54 @@ }) } + /// Reads one page of the metadata change log, or of the current state. + /// + /// Requires root or the global `manage_servers` permission. Keep the + /// page's `next` cursor and send it back in `MetadataChangeStart.After`. + /// + /// Args: + /// start: Where to start, as `MetadataChangeStart`. + /// groups: The groups to read as `list[MetadataChangeGroup] | None`. + /// `None` or an empty list reads every group. A current state scan + /// needs the parent groups of each group, for example `Topics` + /// with `Streams`. + /// since: Keep records prepared at or after this Unix time in + /// microseconds. Only with `Earliest` and `After`. + /// until: Keep records prepared at or before this Unix time in + /// microseconds. Only with `Earliest` and `After`. + /// limit: Records or items per page, 1 to 1000. Defaults to 100. + /// + /// Returns: + /// An awaitable that resolves to `MetadataChanges`. + /// + /// Raises: + /// IggyError: With the invalid command code before anything is sent + /// when the query is refused, or when the request fails. A user + /// without the permission gets the unauthorized code. + #[pyo3(signature = (start, *, groups=None, since=None, until=None, limit=None))] + #[gen_stub(override_return_type(type_repr="collections.abc.Awaitable[MetadataChanges]", imports=("collections.abc")))] + fn get_metadata_changes<'a>( + &self, + py: Python<'a>, + start: &MetadataChangeStart, + #[gen_stub(override_type(type_repr = "builtins.list[MetadataChangeGroup] | None"))] + groups: Option<Vec<MetadataChangeGroup>>, + #[gen_stub(override_type(type_repr = "builtins.int | None"))] since: Option<u64>, + #[gen_stub(override_type(type_repr = "builtins.int | None"))] until: Option<u64>, + #[gen_stub(override_type(type_repr = "builtins.int | None"))] limit: Option<u32>, + ) -> PyResult<Bound<'a, PyAny>> { + let query = validated_query(start, groups, since, until, limit)?; + let inner = self.inner.clone(); + + future_into_py(py, async move { + let changes = inner + .get_metadata_changes(&query) + .await + .map_err(to_runtime_error)?; + MetadataChanges::try_from(changes) + }) + } + /// Logs in the user with the given credentials. /// /// Returns:
diff --git a/foreign/python/src/lib.rs b/foreign/python/src/lib.rs index b5d25ab..9ed4a4f 100644 --- a/foreign/python/src/lib.rs +++ b/foreign/python/src/lib.rs
@@ -21,6 +21,7 @@ mod durability; mod duration; mod identifier; +mod metadata_changes; mod options; mod partitioning; mod permissions; @@ -43,6 +44,18 @@ AutoCommit, AutoCommitAfter, AutoCommitWhen, Consumer, ConsumerGroup, ConsumerGroupDetails, ConsumerGroupMember, IggyConsumer, ReceiveMessageIterator, }; +use metadata_changes::{ + MetadataChangeCursor, MetadataChangeGroup, MetadataChangeKind, MetadataChangePhase, + MetadataChangeRecord, MetadataChangeStart, MetadataChangeStatus, MetadataChanges, + MetadataConsumerGroupCreated, MetadataConsumerGroupDeleted, MetadataConsumerGroupItem, + MetadataCurrentItem, MetadataCurrentPageToken, MetadataMemberItem, MetadataMemberJoined, + MetadataMemberLeft, MetadataMemberLeftReason, MetadataPartitionHandedOver, + MetadataPartitionsChanged, MetadataPartitionsItem, MetadataPermissionsUpdatedReason, + MetadataSession, MetadataSessionClosed, MetadataSessionClosedReason, MetadataStreamChange, + MetadataStreamDeleted, MetadataTokenChange, MetadataTokenDeleted, MetadataTokenDeletedReason, + MetadataTopicChange, MetadataTopicDeleted, MetadataTopicItem, MetadataUnknownChange, + MetadataUnknownItem, MetadataUserChange, MetadataUserIdentity, MetadataUserPermissionsUpdated, +}; use options::OptionSpec; use partitioning::Partitioning; use permissions::{GlobalPermissions, Permissions, StreamPermissions, TopicPermissions}; @@ -133,5 +146,42 @@ m.add_class::<GlobalPermissions>()?; m.add_class::<StreamPermissions>()?; m.add_class::<TopicPermissions>()?; + m.add_class::<MetadataChanges>()?; + m.add_class::<MetadataChangeRecord>()?; + m.add_class::<MetadataCurrentItem>()?; + m.add_class::<MetadataChangeCursor>()?; + m.add_class::<MetadataCurrentPageToken>()?; + m.add_class::<MetadataChangeStart>()?; + m.add_class::<MetadataChangeGroup>()?; + m.add_class::<MetadataChangeKind>()?; + m.add_class::<MetadataChangeStatus>()?; + m.add_class::<MetadataChangePhase>()?; + m.add_class::<MetadataMemberLeftReason>()?; + m.add_class::<MetadataTokenDeletedReason>()?; + m.add_class::<MetadataSessionClosedReason>()?; + m.add_class::<MetadataPermissionsUpdatedReason>()?; + m.add_class::<MetadataStreamChange>()?; + m.add_class::<MetadataStreamDeleted>()?; + m.add_class::<MetadataTopicChange>()?; + m.add_class::<MetadataTopicItem>()?; + m.add_class::<MetadataTopicDeleted>()?; + m.add_class::<MetadataPartitionsChanged>()?; + m.add_class::<MetadataPartitionsItem>()?; + m.add_class::<MetadataConsumerGroupCreated>()?; + m.add_class::<MetadataConsumerGroupDeleted>()?; + m.add_class::<MetadataConsumerGroupItem>()?; + m.add_class::<MetadataMemberJoined>()?; + m.add_class::<MetadataMemberLeft>()?; + m.add_class::<MetadataPartitionHandedOver>()?; + m.add_class::<MetadataMemberItem>()?; + m.add_class::<MetadataUserChange>()?; + m.add_class::<MetadataUserIdentity>()?; + m.add_class::<MetadataUserPermissionsUpdated>()?; + m.add_class::<MetadataTokenChange>()?; + m.add_class::<MetadataTokenDeleted>()?; + m.add_class::<MetadataSession>()?; + m.add_class::<MetadataSessionClosed>()?; + m.add_class::<MetadataUnknownChange>()?; + m.add_class::<MetadataUnknownItem>()?; Ok(()) }
diff --git a/foreign/python/src/metadata_changes.rs b/foreign/python/src/metadata_changes.rs new file mode 100644 index 0000000..a098687 --- /dev/null +++ b/foreign/python/src/metadata_changes.rs
@@ -0,0 +1,2257 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use iggy::prelude::{ + IggyError, IggyTimestamp, MetadataChange as RustMetadataChange, + MetadataChangeCursor as RustMetadataChangeCursor, + MetadataChangeGroup as RustMetadataChangeGroup, MetadataChangeKind as RustMetadataChangeKind, + MetadataChangePhase as RustMetadataChangePhase, + MetadataChangeRecord as RustMetadataChangeRecord, + MetadataChangeStart as RustMetadataChangeStart, + MetadataChangeStatus as RustMetadataChangeStatus, MetadataChanges as RustMetadataChanges, + MetadataChangesQuery, MetadataCurrentItem as RustMetadataCurrentItem, + MetadataCurrentPageToken as RustMetadataCurrentPageToken, + MetadataMemberLeftReason as RustMetadataMemberLeftReason, MetadataOptions, + MetadataPermissionsUpdatedReason as RustMetadataPermissionsUpdatedReason, + MetadataSessionClosedReason as RustMetadataSessionClosedReason, + MetadataTokenDeletedReason as RustMetadataTokenDeletedReason, Permissions as RustPermissions, + ResourceOptions, +}; +use pyo3::IntoPyObjectExt; +use pyo3::exceptions::PyValueError; +use pyo3::prelude::*; +use pyo3::types::PyBytes; +use pyo3_stub_gen::derive::{ + gen_stub_pyclass, gen_stub_pyclass_complex_enum, gen_stub_pyclass_enum, gen_stub_pymethods, +}; +use std::str::FromStr; + +use crate::client::to_runtime_error; +use crate::options::options_by_provenance; +use crate::permissions::Permissions; +use crate::user::UserStatus; +use crate::user_headers::UserHeaders; + +/// One page of the metadata change log, or of the current state, as +/// `IggyClient.get_metadata_changes` returns it. +/// +/// The page is converted once: `records` and `items` hand back the same +/// objects on every access. +#[gen_stub_pyclass] +#[pyclass(frozen)] +pub struct MetadataChanges { + status: MetadataChangeStatus, + phase: MetadataChangePhase, + /// This node's log id, or `None` before it applied op 1. + #[pyo3(get)] + log_id: Option<u128>, + /// Lowest whole operation from which the requested groups are complete. + #[pyo3(get)] + oldest_op: u64, + /// The requested groups are complete for every timestamp above this + /// value in microseconds, 0 when nothing was evicted. + #[pyo3(get)] + retained_since: u64, + /// Highest metadata operation the answering node applied. + #[pyo3(get)] + head_op: u64, + next: Option<MetadataChangeCursor>, + /// More entries remain after `next`, or another state page exists. + #[pyo3(get)] + has_more: bool, + records: Vec<Py<MetadataChangeRecord>>, + items: Vec<Py<MetadataCurrentItem>>, + current_page_token: Option<MetadataCurrentPageToken>, +} + +/// One change record: the committed operation that produced it, its position +/// within that operation and the typed body of what is true after it. +#[gen_stub_pyclass] +#[pyclass(frozen)] +pub struct MetadataChangeRecord { + /// The metadata operation that produced the record. + #[pyo3(get)] + op: u64, + /// Position of the record within `op`. + #[pyo3(get)] + ordinal: u32, + /// Prepare checksum of `op`. + #[pyo3(get)] + checksum: u128, + /// Prepare timestamp of `op` in microseconds. + #[pyo3(get)] + timestamp: u64, + /// The acting user, `None` for server actions. + #[pyo3(get)] + actor_user_id: Option<u32>, + group: Option<MetadataChangeGroup>, + kind: Option<MetadataChangeKind>, + change: Py<PyAny>, +} + +/// One current state item of a `MetadataChangeStart.Current` page. +#[gen_stub_pyclass] +#[pyclass(frozen)] +pub struct MetadataCurrentItem { + group: Option<MetadataChangeGroup>, + body: Py<PyAny>, +} + +/// Position of the last change record a reader consumed or scanned. Send it +/// back in `MetadataChangeStart.After` to read the records after it. +/// +/// `str()` gives its `mc1_` text, which `parse` reads back. The 128-bit +/// values are Python ints. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen, hash, from_py_object)] +pub struct MetadataChangeCursor { + /// Prepare checksum of op 1, which names the history the cursor belongs to. + #[pyo3(get)] + log_id: u128, + /// The operation of the last record, 0 before the first record. + #[pyo3(get)] + op: u64, + /// Prepare checksum of `op`. + #[pyo3(get)] + checksum: u128, + /// Position of the record within `op`. + #[pyo3(get)] + ordinal: u32, +} + +/// Opaque continuation of a current state scan. Send it back unchanged in +/// `MetadataChangeStart.Current` with the same groups. +/// +/// `str()` gives its `cs1_` text, which `parse` reads back. +#[derive(Clone, PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen, from_py_object)] +pub struct MetadataCurrentPageToken { + inner: RustMetadataCurrentPageToken, +} + +/// Where `get_metadata_changes` starts reading. +#[derive(Clone)] +#[gen_stub_pyclass_complex_enum] +#[pyclass(from_py_object)] +pub enum MetadataChangeStart { + /// The oldest operation the requested groups still hold completely. + Earliest(), + /// No records, only a cursor at the newest retained record. + Latest(), + /// The records after `cursor`. + After { cursor: MetadataChangeCursor }, + /// Current state pages: `None` starts a scan, a token continues it. + #[pyo3(constructor = (page_token = None))] + Current { + page_token: Option<MetadataCurrentPageToken>, + }, +} + +/// Family of change records and the unit of filtering. `str()` gives the +/// name the HTTP API uses. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[gen_stub_pyclass_enum] +#[pyclass(eq, frozen, hash, from_py_object)] +pub enum MetadataChangeGroup { + Streams, + Topics, + Partitions, + ConsumerGroups, + ConsumerGroupMembers, + Users, + PersonalAccessTokens, + Sessions, +} + +/// Kind of a change record. `str()` gives the name the HTTP API uses. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[gen_stub_pyclass_enum] +#[pyclass(eq, frozen, hash, from_py_object)] +pub enum MetadataChangeKind { + StreamCreated, + StreamUpdated, + StreamDeleted, + TopicCreated, + TopicUpdated, + TopicDeleted, + PartitionsChanged, + ConsumerGroupCreated, + ConsumerGroupDeleted, + ConsumerGroupMemberJoined, + ConsumerGroupMemberLeft, + ConsumerGroupPartitionHandedOver, + UserCreated, + UserUpdated, + UserDeleted, + UserPasswordChanged, + UserPermissionsUpdated, + PersonalAccessTokenCreated, + PersonalAccessTokenDeleted, + SessionRegistered, + SessionClosed, +} + +/// Outcome of a change log query. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[gen_stub_pyclass_enum] +#[pyclass(eq, frozen, hash, from_py_object)] +pub enum MetadataChangeStatus { + /// Records or items after the requested position. Apply them and keep + /// `next`. + Ok, + /// The answering node has not applied the requested position yet. Ask + /// again later or ask another node. + Behind, + /// The node no longer holds every record after the position. Rebuild + /// from current state. + Expired, + /// The position belongs to another history. Rebuild from current state. + Reset, +} + +/// Whether a page carries change records or current state items. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[gen_stub_pyclass_enum] +#[pyclass(eq, frozen, hash, from_py_object)] +pub enum MetadataChangePhase { + Changes, + Current, +} + +/// Why a consumer group member was removed. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[gen_stub_pyclass_enum] +#[pyclass(eq, frozen, hash, from_py_object)] +pub enum MetadataMemberLeftReason { + Left, + SessionEnded, +} + +/// Why a personal access token was removed. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[gen_stub_pyclass_enum] +#[pyclass(eq, frozen, hash, from_py_object)] +pub enum MetadataTokenDeletedReason { + Deleted, + Expired, +} + +/// Why a session was closed. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[gen_stub_pyclass_enum] +#[pyclass(eq, frozen, hash, from_py_object)] +pub enum MetadataSessionClosedReason { + Logout, + Expired, + /// The user was deactivated or deleted. + Revoked, +} + +/// Why a user's public permission set changed. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[gen_stub_pyclass_enum] +#[pyclass(eq, frozen, hash, from_py_object)] +pub enum MetadataPermissionsUpdatedReason { + /// An `update_permissions` call replaced the set. + Updated, + /// A stream or topic delete pruned the user's grants on it. + ResourceDeleted, +} + +/// A stream after a create or update, and a `streams` current state item. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataStreamChange { + /// Stream id. Ids are reused after a delete. + #[pyo3(get)] + stream_id: u32, + /// The operation that created the stream, which tells it from an earlier + /// stream with the same id. + #[pyo3(get)] + stream_created_op: u64, + /// Stream creation time as Unix time in microseconds. + #[pyo3(get)] + created_at: u64, + #[pyo3(get)] + name: String, + options: Option<ResourceOptions>, +} + +/// A removed stream. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataStreamDeleted { + #[pyo3(get)] + stream_id: u32, + #[pyo3(get)] + stream_created_op: u64, + #[pyo3(get)] + created_at: u64, + #[pyo3(get)] + name: String, + /// The operation of the delete request when the delete completed later. + #[pyo3(get)] + requested_op: Option<u64>, +} + +/// A topic after a create or update. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataTopicChange { + #[pyo3(get)] + stream_id: u32, + #[pyo3(get)] + stream_created_op: u64, + #[pyo3(get)] + topic_id: u32, + /// The operation that created the topic, which tells it from an earlier + /// topic with the same id. + #[pyo3(get)] + topic_created_op: u64, + #[pyo3(get)] + created_at: u64, + #[pyo3(get)] + name: String, + #[pyo3(get)] + partitions_count: u32, + options: Option<ResourceOptions>, +} + +/// A `topics` current state item. Partition counts come as `partitions` +/// items. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataTopicItem { + #[pyo3(get)] + stream_id: u32, + #[pyo3(get)] + stream_created_op: u64, + #[pyo3(get)] + topic_id: u32, + #[pyo3(get)] + topic_created_op: u64, + #[pyo3(get)] + created_at: u64, + #[pyo3(get)] + name: String, + options: Option<ResourceOptions>, +} + +/// A removed topic. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataTopicDeleted { + #[pyo3(get)] + stream_id: u32, + #[pyo3(get)] + stream_created_op: u64, + #[pyo3(get)] + topic_id: u32, + #[pyo3(get)] + topic_created_op: u64, + #[pyo3(get)] + created_at: u64, + #[pyo3(get)] + name: String, + /// The operation of the delete request when the delete completed later. + #[pyo3(get)] + requested_op: Option<u64>, +} + +/// A partition count change. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataPartitionsChanged { + #[pyo3(get)] + stream_id: u32, + #[pyo3(get)] + stream_created_op: u64, + #[pyo3(get)] + topic_id: u32, + #[pyo3(get)] + topic_created_op: u64, + /// The partition count after the change. + #[pyo3(get)] + partitions_count: u32, + /// The deleted partition ids, ascending, empty on create. + #[pyo3(get)] + removed_partition_ids: Vec<u32>, + /// The operation of the delete request when the delete completed later. + #[pyo3(get)] + requested_op: Option<u64>, +} + +/// A `partitions` current state item. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataPartitionsItem { + #[pyo3(get)] + stream_id: u32, + #[pyo3(get)] + stream_created_op: u64, + #[pyo3(get)] + topic_id: u32, + #[pyo3(get)] + topic_created_op: u64, + #[pyo3(get)] + partitions_count: u32, +} + +/// A created consumer group with the topic's resulting group count. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataConsumerGroupCreated { + #[pyo3(get)] + stream_id: u32, + #[pyo3(get)] + stream_created_op: u64, + #[pyo3(get)] + topic_id: u32, + #[pyo3(get)] + topic_created_op: u64, + #[pyo3(get)] + group_id: u32, + #[pyo3(get)] + name: String, + #[pyo3(get)] + consumer_groups_count: u32, +} + +/// A removed consumer group with the topic's resulting group count. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataConsumerGroupDeleted { + #[pyo3(get)] + stream_id: u32, + #[pyo3(get)] + stream_created_op: u64, + #[pyo3(get)] + topic_id: u32, + #[pyo3(get)] + topic_created_op: u64, + #[pyo3(get)] + group_id: u32, + #[pyo3(get)] + name: String, + #[pyo3(get)] + consumer_groups_count: u32, + /// The operation of the delete request when the delete completed later. + #[pyo3(get)] + requested_op: Option<u64>, +} + +/// A `consumer_groups` current state item. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataConsumerGroupItem { + #[pyo3(get)] + stream_id: u32, + #[pyo3(get)] + stream_created_op: u64, + #[pyo3(get)] + topic_id: u32, + #[pyo3(get)] + topic_created_op: u64, + #[pyo3(get)] + group_id: u32, + #[pyo3(get)] + name: String, +} + +/// A member that joined a consumer group. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataMemberJoined { + #[pyo3(get)] + stream_id: u32, + #[pyo3(get)] + stream_created_op: u64, + #[pyo3(get)] + topic_id: u32, + #[pyo3(get)] + topic_created_op: u64, + #[pyo3(get)] + group_id: u32, + #[pyo3(get)] + member_id: u32, + #[pyo3(get)] + client_id: u128, + #[pyo3(get)] + members_count: u32, + /// Installed partitions, normally none because a handover installs them. + #[pyo3(get)] + partition_ids: Vec<u32>, +} + +/// A member that left a consumer group. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataMemberLeft { + #[pyo3(get)] + stream_id: u32, + #[pyo3(get)] + stream_created_op: u64, + #[pyo3(get)] + topic_id: u32, + #[pyo3(get)] + topic_created_op: u64, + #[pyo3(get)] + group_id: u32, + #[pyo3(get)] + member_id: u32, + #[pyo3(get)] + client_id: u128, + #[pyo3(get)] + members_count: u32, + reason: MetadataMemberLeftReason, +} + +/// A completed partition handover. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataPartitionHandedOver { + #[pyo3(get)] + stream_id: u32, + #[pyo3(get)] + stream_created_op: u64, + #[pyo3(get)] + topic_id: u32, + #[pyo3(get)] + topic_created_op: u64, + #[pyo3(get)] + group_id: u32, + #[pyo3(get)] + partition_id: u32, + /// The new owner, `None` when the last member left. + #[pyo3(get)] + member_id: Option<u32>, +} + +/// A `consumer_group_members` current state item. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataMemberItem { + #[pyo3(get)] + stream_id: u32, + #[pyo3(get)] + stream_created_op: u64, + #[pyo3(get)] + topic_id: u32, + #[pyo3(get)] + topic_created_op: u64, + #[pyo3(get)] + group_id: u32, + #[pyo3(get)] + member_id: u32, + #[pyo3(get)] + client_id: u128, + /// Installed partitions, ascending. + #[pyo3(get)] + partition_ids: Vec<u32>, +} + +/// A user after a create or update, and a `users` current state item. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataUserChange { + /// User id. Ids are reused after a delete. + #[pyo3(get)] + user_id: u32, + /// User creation time in microseconds, which tells the user from an + /// earlier user with the same id. + #[pyo3(get)] + user_created_at: u64, + #[pyo3(get)] + username: String, + status: UserStatus, + permissions: Option<RustPermissions>, + options: Option<ResourceOptions>, +} + +/// A deleted user, or one whose password changed. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataUserIdentity { + #[pyo3(get)] + user_id: u32, + #[pyo3(get)] + user_created_at: u64, + #[pyo3(get)] + username: String, +} + +/// A user's public permission set after a change. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataUserPermissionsUpdated { + #[pyo3(get)] + user_id: u32, + #[pyo3(get)] + user_created_at: u64, + #[pyo3(get)] + username: String, + permissions: Option<RustPermissions>, + reason: MetadataPermissionsUpdatedReason, +} + +/// A created personal access token, and a `personal_access_tokens` current +/// state item. Never the token itself. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataTokenChange { + #[pyo3(get)] + user_id: u32, + #[pyo3(get)] + user_created_at: u64, + #[pyo3(get)] + name: String, + /// Expiry in microseconds, `None` for a token that never expires. + #[pyo3(get)] + expiry: Option<u64>, +} + +/// A removed personal access token. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataTokenDeleted { + #[pyo3(get)] + user_id: u32, + #[pyo3(get)] + user_created_at: u64, + #[pyo3(get)] + name: String, + reason: MetadataTokenDeletedReason, +} + +/// A registered session, and a `sessions` current state item. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataSession { + #[pyo3(get)] + user_id: u32, + #[pyo3(get)] + user_created_at: u64, + #[pyo3(get)] + client_id: u128, + /// The session epoch. + #[pyo3(get)] + session: u64, +} + +/// A closed session. +#[derive(PartialEq)] +#[gen_stub_pyclass] +#[pyclass(eq, frozen)] +pub struct MetadataSessionClosed { + #[pyo3(get)] + user_id: u32, + #[pyo3(get)] + user_created_at: u64, + #[pyo3(get)] + client_id: u128, + #[pyo3(get)] + session: u64, + reason: MetadataSessionClosedReason, +} + +/// A record of a group or kind this build does not know. A mirror stops on +/// it rather than skip a possible state change. +#[gen_stub_pyclass] +#[pyclass(frozen)] +pub struct MetadataUnknownChange { + group: Py<PyAny>, + kind: Py<PyAny>, + body: Py<PyAny>, +} + +/// A current state item of a group this build does not know. +#[gen_stub_pyclass] +#[pyclass(frozen)] +pub struct MetadataUnknownItem { + group: Py<PyAny>, + body: Py<PyAny>, +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataChanges { + #[getter] + fn status(&self) -> MetadataChangeStatus { + self.status + } + + /// `Changes` for `Earliest`, `Latest` and `After`, `Current` for state + /// pages. + #[getter] + fn phase(&self) -> MetadataChangePhase { + self.phase + } + + /// The cursor to send back. Present on `Ok`, echoed on `Behind`, `None` + /// on `Expired` and `Reset`. The last state page returns the anchor to + /// follow from. + #[getter] + #[gen_stub(override_return_type(type_repr = "MetadataChangeCursor | None"))] + fn next(&self) -> Option<MetadataChangeCursor> { + self.next + } + + /// Change records in `(op, ordinal)` order, empty on state pages. + #[getter] + #[gen_stub(override_return_type(type_repr = "builtins.list[MetadataChangeRecord]"))] + fn records(&self, py: Python<'_>) -> Vec<Py<MetadataChangeRecord>> { + self.records + .iter() + .map(|record| record.clone_ref(py)) + .collect() + } + + /// Current state items, empty on change pages. + #[getter] + #[gen_stub(override_return_type(type_repr = "builtins.list[MetadataCurrentItem]"))] + fn items(&self, py: Python<'_>) -> Vec<Py<MetadataCurrentItem>> { + self.items.iter().map(|item| item.clone_ref(py)).collect() + } + + /// Continuation of a current state scan, `None` once it is complete. + #[getter] + #[gen_stub(override_return_type(type_repr = "MetadataCurrentPageToken | None"))] + fn current_page_token(&self) -> Option<MetadataCurrentPageToken> { + self.current_page_token.clone() + } + + fn __repr__(&self) -> String { + format!( + "MetadataChanges(status={}, phase={}, head_op={}, has_more={}, records={}, items={})", + self.status.__str__(), + self.phase.__str__(), + self.head_op, + self.has_more, + self.records.len(), + self.items.len() + ) + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataChangeRecord { + /// The group, `None` for a record this build does not know. + #[getter] + #[gen_stub(override_return_type(type_repr = "MetadataChangeGroup | None"))] + fn group(&self) -> Option<MetadataChangeGroup> { + self.group + } + + /// The kind, `None` for a record this build does not know. + #[getter] + #[gen_stub(override_return_type(type_repr = "MetadataChangeKind | None"))] + fn kind(&self) -> Option<MetadataChangeKind> { + self.kind + } + + /// The typed body `kind` names, or `MetadataUnknownChange`. + #[getter] + #[gen_stub(override_return_type( + type_repr = "MetadataStreamChange | MetadataStreamDeleted | MetadataTopicChange | MetadataTopicDeleted | MetadataPartitionsChanged | MetadataConsumerGroupCreated | MetadataConsumerGroupDeleted | MetadataMemberJoined | MetadataMemberLeft | MetadataPartitionHandedOver | MetadataUserChange | MetadataUserIdentity | MetadataUserPermissionsUpdated | MetadataTokenChange | MetadataTokenDeleted | MetadataSession | MetadataSessionClosed | MetadataUnknownChange" + ))] + fn change(&self, py: Python<'_>) -> Py<PyAny> { + self.change.clone_ref(py) + } + + fn __repr__(&self) -> String { + let kind = self + .kind + .map_or("unknown", |kind| RustMetadataChangeKind::from(kind).into()); + format!( + "MetadataChangeRecord(op={}, ordinal={}, kind={kind})", + self.op, self.ordinal + ) + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataCurrentItem { + /// The group, `None` for an item this build does not know. + #[getter] + #[gen_stub(override_return_type(type_repr = "MetadataChangeGroup | None"))] + fn group(&self) -> Option<MetadataChangeGroup> { + self.group + } + + /// The typed body `group` names, or `MetadataUnknownItem`. + #[getter] + #[gen_stub(override_return_type( + type_repr = "MetadataStreamChange | MetadataTopicItem | MetadataPartitionsItem | MetadataConsumerGroupItem | MetadataMemberItem | MetadataUserChange | MetadataTokenChange | MetadataSession | MetadataUnknownItem" + ))] + fn body(&self, py: Python<'_>) -> Py<PyAny> { + self.body.clone_ref(py) + } + + fn __repr__(&self) -> String { + let group = self.group.map_or("unknown", |group| { + RustMetadataChangeGroup::from(group).into() + }); + format!("MetadataCurrentItem(group={group})") + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataChangeCursor { + #[new] + fn new(log_id: u128, op: u64, checksum: u128, ordinal: u32) -> Self { + Self { + log_id, + op, + checksum, + ordinal, + } + } + + /// Reads a cursor from its `mc1_` text. + /// + /// Raises: + /// ValueError: If the text is not a canonical cursor. + #[staticmethod] + fn parse(text: &str) -> PyResult<Self> { + RustMetadataChangeCursor::from_str(text) + .map(Self::from) + .map_err(|_| PyValueError::new_err("invalid metadata change cursor")) + } + + fn __str__(&self) -> String { + RustMetadataChangeCursor::from(*self).to_string() + } + + fn __repr__(&self) -> String { + format!("MetadataChangeCursor.parse('{}')", self.__str__()) + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataCurrentPageToken { + /// Reads a page token from its `cs1_` text. + /// + /// Raises: + /// ValueError: If the text is not a canonical page token. + #[staticmethod] + fn parse(text: &str) -> PyResult<Self> { + RustMetadataCurrentPageToken::from_str(text) + .map(|inner| Self { inner }) + .map_err(|_| PyValueError::new_err("invalid metadata current page token")) + } + + /// The cursor captured before the scan's first page. + #[getter] + fn anchor(&self) -> MetadataChangeCursor { + self.inner.anchor().into() + } + + /// The highest head any page of the scan reported so far. + #[getter] + fn required_head(&self) -> u64 { + self.inner.required_head() + } + + /// The groups the scan was started with, in code order. + #[getter] + fn groups(&self) -> Vec<MetadataChangeGroup> { + self.inner.groups().iter().map(Into::into).collect() + } + + /// The group of the last item returned. + #[getter] + fn last_group(&self) -> MetadataChangeGroup { + self.inner.last_group().into() + } + + /// The key of the last item returned, in the page token key layout. + #[getter] + fn last_key<'a>(&self, py: Python<'a>) -> Bound<'a, PyBytes> { + PyBytes::new(py, self.inner.last_key()) + } + + fn __str__(&self) -> String { + self.inner.to_string() + } + + fn __repr__(&self) -> String { + format!("MetadataCurrentPageToken.parse('{}')", self.inner) + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataChangeGroup { + fn __str__(&self) -> &'static str { + RustMetadataChangeGroup::from(*self).into() + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataChangeKind { + fn __str__(&self) -> &'static str { + RustMetadataChangeKind::from(*self).into() + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataChangeStatus { + fn __str__(&self) -> &'static str { + RustMetadataChangeStatus::from(*self).into() + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataChangePhase { + fn __str__(&self) -> &'static str { + RustMetadataChangePhase::from(*self).into() + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataMemberLeftReason { + fn __str__(&self) -> &'static str { + RustMetadataMemberLeftReason::from(*self).into() + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataTokenDeletedReason { + fn __str__(&self) -> &'static str { + RustMetadataTokenDeletedReason::from(*self).into() + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataSessionClosedReason { + fn __str__(&self) -> &'static str { + RustMetadataSessionClosedReason::from(*self).into() + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataPermissionsUpdatedReason { + fn __str__(&self) -> &'static str { + RustMetadataPermissionsUpdatedReason::from(*self).into() + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataStreamChange { + /// Options the creating client set explicitly, the same mapping + /// `StreamDetails.options` returns, or `None` when `options_unreadable`. + /// The stream's encryption key never appears here. + #[getter] + #[gen_stub(override_return_type(type_repr = "UserHeaders | None"))] + fn options<'a>(&self, py: Python<'a>) -> PyResult<Option<Bound<'a, UserHeaders>>> { + options_view(py, self.options.as_ref(), true) + } + + /// Options the server resolved or computed, including the key source and + /// fingerprints. Same shape as `options`. + #[getter] + #[gen_stub(override_return_type(type_repr = "UserHeaders | None"))] + fn derived_options<'a>(&self, py: Python<'a>) -> PyResult<Option<Bound<'a, UserHeaders>>> { + options_view(py, self.options.as_ref(), false) + } + + /// The server could not read the stream's stored options. Keep the + /// previous values. + #[getter] + fn options_unreadable(&self) -> bool { + self.options.is_none() + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataTopicChange { + /// Options the creating client set explicitly, the same mapping + /// `TopicDetails.options` returns, or `None` when `options_unreadable`. + #[getter] + #[gen_stub(override_return_type(type_repr = "UserHeaders | None"))] + fn options<'a>(&self, py: Python<'a>) -> PyResult<Option<Bound<'a, UserHeaders>>> { + options_view(py, self.options.as_ref(), true) + } + + /// Options the server resolved for the keys the client did not send. + #[getter] + #[gen_stub(override_return_type(type_repr = "UserHeaders | None"))] + fn derived_options<'a>(&self, py: Python<'a>) -> PyResult<Option<Bound<'a, UserHeaders>>> { + options_view(py, self.options.as_ref(), false) + } + + /// The server could not read the topic's stored options. Keep the + /// previous values. + #[getter] + fn options_unreadable(&self) -> bool { + self.options.is_none() + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataTopicItem { + /// Options the creating client set explicitly, or `None` when + /// `options_unreadable`. + #[getter] + #[gen_stub(override_return_type(type_repr = "UserHeaders | None"))] + fn options<'a>(&self, py: Python<'a>) -> PyResult<Option<Bound<'a, UserHeaders>>> { + options_view(py, self.options.as_ref(), true) + } + + /// Options the server resolved for the keys the client did not send. + #[getter] + #[gen_stub(override_return_type(type_repr = "UserHeaders | None"))] + fn derived_options<'a>(&self, py: Python<'a>) -> PyResult<Option<Bound<'a, UserHeaders>>> { + options_view(py, self.options.as_ref(), false) + } + + /// The server could not read the topic's stored options. + #[getter] + fn options_unreadable(&self) -> bool { + self.options.is_none() + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataMemberLeft { + #[getter] + fn reason(&self) -> MetadataMemberLeftReason { + self.reason + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataUserChange { + #[getter] + fn status(&self) -> UserStatus { + self.status + } + + /// The public permission set, or `None` when the user has none. + #[getter] + #[gen_stub(override_return_type(type_repr = "Permissions | None"))] + fn permissions(&self) -> Option<Permissions> { + self.permissions.clone().map(Permissions::from) + } + + /// Options the creating client set explicitly, or `None` when + /// `options_unreadable`. + #[getter] + #[gen_stub(override_return_type(type_repr = "UserHeaders | None"))] + fn options<'a>(&self, py: Python<'a>) -> PyResult<Option<Bound<'a, UserHeaders>>> { + options_view(py, self.options.as_ref(), true) + } + + /// Options the server resolved for the keys the client did not send. + #[getter] + #[gen_stub(override_return_type(type_repr = "UserHeaders | None"))] + fn derived_options<'a>(&self, py: Python<'a>) -> PyResult<Option<Bound<'a, UserHeaders>>> { + options_view(py, self.options.as_ref(), false) + } + + /// The server could not read the user's stored options. + #[getter] + fn options_unreadable(&self) -> bool { + self.options.is_none() + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataUserPermissionsUpdated { + /// The public permission set after the change, or `None` when the user + /// has none. + #[getter] + #[gen_stub(override_return_type(type_repr = "Permissions | None"))] + fn permissions(&self) -> Option<Permissions> { + self.permissions.clone().map(Permissions::from) + } + + #[getter] + fn reason(&self) -> MetadataPermissionsUpdatedReason { + self.reason + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataTokenDeleted { + #[getter] + fn reason(&self) -> MetadataTokenDeletedReason { + self.reason + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataSessionClosed { + #[getter] + fn reason(&self) -> MetadataSessionClosedReason { + self.reason + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataUnknownChange { + /// The group code on binary transports, or the group name over HTTP. + #[getter] + #[gen_stub(override_return_type(type_repr = "builtins.int | builtins.str"))] + fn group(&self, py: Python<'_>) -> Py<PyAny> { + self.group.clone_ref(py) + } + + /// The kind code on binary transports, or the kind name over HTTP. + #[getter] + #[gen_stub(override_return_type(type_repr = "builtins.int | builtins.str"))] + fn kind(&self, py: Python<'_>) -> Py<PyAny> { + self.kind.clone_ref(py) + } + + /// The raw body on binary transports, or the record's whole JSON object + /// over HTTP. + #[getter] + #[gen_stub(override_return_type(type_repr = "builtins.bytes | builtins.str"))] + fn body(&self, py: Python<'_>) -> Py<PyAny> { + self.body.clone_ref(py) + } +} + +#[gen_stub_pymethods] +#[pymethods] +impl MetadataUnknownItem { + /// The group code on binary transports, or the group name over HTTP. + #[getter] + #[gen_stub(override_return_type(type_repr = "builtins.int | builtins.str"))] + fn group(&self, py: Python<'_>) -> Py<PyAny> { + self.group.clone_ref(py) + } + + /// The raw body on binary transports, or the item's whole JSON object + /// over HTTP. + #[getter] + #[gen_stub(override_return_type(type_repr = "builtins.bytes | builtins.str"))] + fn body(&self, py: Python<'_>) -> Py<PyAny> { + self.body.clone_ref(py) + } +} + +impl TryFrom<RustMetadataChanges> for MetadataChanges { + type Error = PyErr; + + fn try_from(changes: RustMetadataChanges) -> PyResult<Self> { + Python::attach(|py| { + let mut records = Vec::with_capacity(changes.records.len()); + for record in changes.records { + records.push(change_record(py, record)?); + } + let mut items = Vec::with_capacity(changes.items.len()); + for item in changes.items { + items.push(current_item(py, item)?); + } + Ok(Self { + status: changes.status.into(), + phase: changes.phase.into(), + log_id: changes.log_id, + oldest_op: changes.oldest_op, + retained_since: changes.retained_since, + head_op: changes.head_op, + next: changes.next.map(Into::into), + has_more: changes.has_more, + records, + items, + current_page_token: changes + .current_page_token + .map(|inner| MetadataCurrentPageToken { inner }), + }) + }) + } +} + +impl From<RustMetadataChangeCursor> for MetadataChangeCursor { + fn from(cursor: RustMetadataChangeCursor) -> Self { + Self { + log_id: cursor.log_id, + op: cursor.op, + checksum: cursor.checksum, + ordinal: cursor.ordinal, + } + } +} + +impl From<MetadataChangeCursor> for RustMetadataChangeCursor { + fn from(cursor: MetadataChangeCursor) -> Self { + Self { + log_id: cursor.log_id, + op: cursor.op, + checksum: cursor.checksum, + ordinal: cursor.ordinal, + } + } +} + +impl From<&MetadataChangeStart> for RustMetadataChangeStart { + fn from(start: &MetadataChangeStart) -> Self { + match start { + MetadataChangeStart::Earliest() => Self::Earliest, + MetadataChangeStart::Latest() => Self::Latest, + MetadataChangeStart::After { cursor } => Self::After((*cursor).into()), + MetadataChangeStart::Current { page_token } => { + Self::Current(page_token.as_ref().map(|token| token.inner.clone())) + } + } + } +} + +macro_rules! mirror_enum { + ($python:ident, $rust:ident, $($variant:ident),+) => { + impl From<$rust> for $python { + fn from(value: $rust) -> Self { + match value { + $($rust::$variant => Self::$variant,)+ + } + } + } + + impl From<$python> for $rust { + fn from(value: $python) -> Self { + match value { + $($python::$variant => Self::$variant,)+ + } + } + } + }; +} + +mirror_enum!( + MetadataChangeGroup, + RustMetadataChangeGroup, + Streams, + Topics, + Partitions, + ConsumerGroups, + ConsumerGroupMembers, + Users, + PersonalAccessTokens, + Sessions +); +mirror_enum!( + MetadataChangeKind, + RustMetadataChangeKind, + StreamCreated, + StreamUpdated, + StreamDeleted, + TopicCreated, + TopicUpdated, + TopicDeleted, + PartitionsChanged, + ConsumerGroupCreated, + ConsumerGroupDeleted, + ConsumerGroupMemberJoined, + ConsumerGroupMemberLeft, + ConsumerGroupPartitionHandedOver, + UserCreated, + UserUpdated, + UserDeleted, + UserPasswordChanged, + UserPermissionsUpdated, + PersonalAccessTokenCreated, + PersonalAccessTokenDeleted, + SessionRegistered, + SessionClosed +); +mirror_enum!( + MetadataChangeStatus, + RustMetadataChangeStatus, + Ok, + Behind, + Expired, + Reset +); +mirror_enum!( + MetadataChangePhase, + RustMetadataChangePhase, + Changes, + Current +); +mirror_enum!( + MetadataMemberLeftReason, + RustMetadataMemberLeftReason, + Left, + SessionEnded +); +mirror_enum!( + MetadataTokenDeletedReason, + RustMetadataTokenDeletedReason, + Deleted, + Expired +); +mirror_enum!( + MetadataSessionClosedReason, + RustMetadataSessionClosedReason, + Logout, + Expired, + Revoked +); +mirror_enum!( + MetadataPermissionsUpdatedReason, + RustMetadataPermissionsUpdatedReason, + Updated, + ResourceDeleted +); + +/// Build the query `get_metadata_changes` sends and refuse it before any +/// I/O when the Rust SDK would. +/// +/// # Errors +/// +/// Raises `IggyError` with the `InvalidCommand` code for a query +/// `MetadataChangesQuery::validate` refuses, or for a time bound the +/// platform clock cannot hold. +pub(crate) fn validated_query( + start: &MetadataChangeStart, + groups: Option<Vec<MetadataChangeGroup>>, + since: Option<u64>, + until: Option<u64>, + limit: Option<u32>, +) -> PyResult<MetadataChangesQuery> { + let timestamp = |micros: u64| { + IggyTimestamp::checked_from_micros(micros) + .ok_or_else(|| to_runtime_error(IggyError::InvalidCommand)) + }; + let defaults = MetadataChangesQuery::default(); + let query = MetadataChangesQuery { + start: start.into(), + groups: groups.into_iter().flatten().map(Into::into).collect(), + since: since.map(timestamp).transpose()?, + until: until.map(timestamp).transpose()?, + limit: limit.unwrap_or(defaults.limit), + }; + query.validate().map_err(to_runtime_error)?; + Ok(query) +} + +fn change_record( + py: Python<'_>, + record: RustMetadataChangeRecord, +) -> PyResult<Py<MetadataChangeRecord>> { + let kind = record.change.kind(); + Py::new( + py, + MetadataChangeRecord { + op: record.op, + ordinal: record.ordinal, + checksum: record.checksum, + timestamp: record.timestamp, + actor_user_id: record.actor_user_id, + group: kind.map(|kind| kind.group().into()), + kind: kind.map(Into::into), + change: change_body(py, record.change)?, + }, + ) +} + +fn current_item( + py: Python<'_>, + item: RustMetadataCurrentItem, +) -> PyResult<Py<MetadataCurrentItem>> { + Py::new( + py, + MetadataCurrentItem { + group: item.group().map(Into::into), + body: item_body(py, item)?, + }, + ) +} + +fn change_body(py: Python<'_>, change: RustMetadataChange) -> PyResult<Py<PyAny>> { + match change { + RustMetadataChange::StreamCreated(body) | RustMetadataChange::StreamUpdated(body) => { + MetadataStreamChange { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + created_at: body.created_at, + name: body.name, + options: readable_options(body.options), + } + .into_py_any(py) + } + RustMetadataChange::StreamDeleted(body) => MetadataStreamDeleted { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + created_at: body.created_at, + name: body.name, + requested_op: body.requested_op, + } + .into_py_any(py), + RustMetadataChange::TopicCreated(body) | RustMetadataChange::TopicUpdated(body) => { + MetadataTopicChange { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + created_at: body.created_at, + name: body.name, + partitions_count: body.partitions_count, + options: readable_options(body.options), + } + .into_py_any(py) + } + RustMetadataChange::TopicDeleted(body) => MetadataTopicDeleted { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + created_at: body.created_at, + name: body.name, + requested_op: body.requested_op, + } + .into_py_any(py), + RustMetadataChange::PartitionsChanged(body) => MetadataPartitionsChanged { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + partitions_count: body.partitions_count, + removed_partition_ids: body.removed_partition_ids, + requested_op: body.requested_op, + } + .into_py_any(py), + RustMetadataChange::ConsumerGroupCreated(body) => MetadataConsumerGroupCreated { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + group_id: body.group_id, + name: body.name, + consumer_groups_count: body.consumer_groups_count, + } + .into_py_any(py), + RustMetadataChange::ConsumerGroupDeleted(body) => MetadataConsumerGroupDeleted { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + group_id: body.group_id, + name: body.name, + consumer_groups_count: body.consumer_groups_count, + requested_op: body.requested_op, + } + .into_py_any(py), + RustMetadataChange::ConsumerGroupMemberJoined(body) => MetadataMemberJoined { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + group_id: body.group_id, + member_id: body.member_id, + client_id: body.client_id, + members_count: body.members_count, + partition_ids: body.partition_ids, + } + .into_py_any(py), + RustMetadataChange::ConsumerGroupMemberLeft(body) => MetadataMemberLeft { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + group_id: body.group_id, + member_id: body.member_id, + client_id: body.client_id, + members_count: body.members_count, + reason: body.reason.into(), + } + .into_py_any(py), + RustMetadataChange::ConsumerGroupPartitionHandedOver(body) => MetadataPartitionHandedOver { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + group_id: body.group_id, + partition_id: body.partition_id, + member_id: body.member_id, + } + .into_py_any(py), + RustMetadataChange::UserCreated(body) | RustMetadataChange::UserUpdated(body) => { + MetadataUserChange { + user_id: body.user_id, + user_created_at: body.user_created_at, + username: body.username, + status: body.status.into(), + permissions: body.permissions, + options: readable_options(body.options), + } + .into_py_any(py) + } + RustMetadataChange::UserDeleted(body) | RustMetadataChange::UserPasswordChanged(body) => { + MetadataUserIdentity { + user_id: body.user_id, + user_created_at: body.user_created_at, + username: body.username, + } + .into_py_any(py) + } + RustMetadataChange::UserPermissionsUpdated(body) => MetadataUserPermissionsUpdated { + user_id: body.user_id, + user_created_at: body.user_created_at, + username: body.username, + permissions: body.permissions, + reason: body.reason.into(), + } + .into_py_any(py), + RustMetadataChange::PersonalAccessTokenCreated(body) => MetadataTokenChange { + user_id: body.user_id, + user_created_at: body.user_created_at, + name: body.name, + expiry: body.expiry, + } + .into_py_any(py), + RustMetadataChange::PersonalAccessTokenDeleted(body) => MetadataTokenDeleted { + user_id: body.user_id, + user_created_at: body.user_created_at, + name: body.name, + reason: body.reason.into(), + } + .into_py_any(py), + RustMetadataChange::SessionRegistered(body) => MetadataSession { + user_id: body.user_id, + user_created_at: body.user_created_at, + client_id: body.client_id, + session: body.session, + } + .into_py_any(py), + RustMetadataChange::SessionClosed(body) => MetadataSessionClosed { + user_id: body.user_id, + user_created_at: body.user_created_at, + client_id: body.client_id, + session: body.session, + reason: body.reason.into(), + } + .into_py_any(py), + RustMetadataChange::Unknown { group, kind, body } => MetadataUnknownChange { + group: group.into_py_any(py)?, + kind: kind.into_py_any(py)?, + body: PyBytes::new(py, &body).into_any().unbind(), + } + .into_py_any(py), + RustMetadataChange::UnknownJson { group, kind, body } => MetadataUnknownChange { + group: group.into_py_any(py)?, + kind: kind.into_py_any(py)?, + body: body.get().into_py_any(py)?, + } + .into_py_any(py), + } +} + +fn item_body(py: Python<'_>, item: RustMetadataCurrentItem) -> PyResult<Py<PyAny>> { + match item { + RustMetadataCurrentItem::Stream(body) => MetadataStreamChange { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + created_at: body.created_at, + name: body.name, + options: readable_options(body.options), + } + .into_py_any(py), + RustMetadataCurrentItem::Topic(body) => MetadataTopicItem { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + created_at: body.created_at, + name: body.name, + options: readable_options(body.options), + } + .into_py_any(py), + RustMetadataCurrentItem::Partitions(body) => MetadataPartitionsItem { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + partitions_count: body.partitions_count, + } + .into_py_any(py), + RustMetadataCurrentItem::ConsumerGroup(body) => MetadataConsumerGroupItem { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + group_id: body.group_id, + name: body.name, + } + .into_py_any(py), + RustMetadataCurrentItem::ConsumerGroupMember(body) => MetadataMemberItem { + stream_id: body.stream_id, + stream_created_op: body.stream_created_op, + topic_id: body.topic_id, + topic_created_op: body.topic_created_op, + group_id: body.group_id, + member_id: body.member_id, + client_id: body.client_id, + partition_ids: body.partition_ids, + } + .into_py_any(py), + RustMetadataCurrentItem::User(body) => MetadataUserChange { + user_id: body.user_id, + user_created_at: body.user_created_at, + username: body.username, + status: body.status.into(), + permissions: body.permissions, + options: readable_options(body.options), + } + .into_py_any(py), + RustMetadataCurrentItem::PersonalAccessToken(body) => MetadataTokenChange { + user_id: body.user_id, + user_created_at: body.user_created_at, + name: body.name, + expiry: body.expiry, + } + .into_py_any(py), + RustMetadataCurrentItem::Session(body) => MetadataSession { + user_id: body.user_id, + user_created_at: body.user_created_at, + client_id: body.client_id, + session: body.session, + } + .into_py_any(py), + RustMetadataCurrentItem::Unknown { group, body } => MetadataUnknownItem { + group: group.into_py_any(py)?, + body: PyBytes::new(py, &body).into_any().unbind(), + } + .into_py_any(py), + RustMetadataCurrentItem::UnknownJson { group, body } => MetadataUnknownItem { + group: group.into_py_any(py)?, + body: body.get().into_py_any(py)?, + } + .into_py_any(py), + } +} + +fn readable_options(options: MetadataOptions) -> Option<ResourceOptions> { + match options { + MetadataOptions::Readable(options) => Some(options), + MetadataOptions::Unreadable => None, + } +} + +fn options_view<'a>( + py: Python<'a>, + options: Option<&ResourceOptions>, + explicit: bool, +) -> PyResult<Option<Bound<'a, UserHeaders>>> { + options + .map(|options| options_by_provenance(py, options, explicit)) + .transpose() +} + +#[cfg(test)] +mod tests { + use super::*; + use bytes::Bytes; + use iggy::prelude::UserStatus as RustUserStatus; + use iggy_binary_protocol::requests::system::GetMetadataChangesRequest; + use iggy_binary_protocol::responses::system::{ + GetMetadataChangesResponse, WireMetadataChangeRecord, WireMetadataItem, + }; + use iggy_binary_protocol::{WireDecode, WireEncode}; + use pyo3::types::{PyDict, PyString}; + use serde_json::{Map, Value}; + + const FIXTURE: &str = include_str!("../tests/fixtures/metadata_changes.json"); + const ENVELOPE_KEYS: [&str; 7] = [ + "op", + "ordinal", + "checksum", + "timestamp", + "actor_user_id", + "group", + "kind", + ]; + const OPTIONAL_BODY_KEYS: [&str; 3] = ["requested_op", "member_id", "expiry"]; + + #[test] + fn given_record_vectors_when_converting_binary_and_json_should_match_expected_json() { + Python::initialize(); + Python::attach(|py| { + for vector in vectors("records") { + let bytes = decode_hex(&vector["record_hex"]); + let (wire, consumed) = + WireMetadataChangeRecord::decode(&bytes).expect("record decodes"); + assert_eq!(consumed, bytes.len()); + let binary = RustMetadataChangeRecord::try_from(wire).expect("record converts"); + let json: RustMetadataChangeRecord = + serde_json::from_value(vector["expected"].clone()).expect("record parses"); + + for record in [binary, json] { + let page = page_of(py, vec![record], vec![]); + let records = page.getattr("records").expect("records"); + assert_record(&records.get_item(0).expect("record"), &vector["expected"]); + } + } + }); + } + + #[test] + fn given_item_vectors_when_converting_binary_and_json_should_match_expected_json() { + Python::initialize(); + Python::attach(|py| { + for vector in vectors("items") { + let bytes = decode_hex(&vector["item_hex"]); + let (wire, consumed) = WireMetadataItem::decode(&bytes).expect("item decodes"); + assert_eq!(consumed, bytes.len()); + let binary = RustMetadataCurrentItem::try_from(wire).expect("item converts"); + let json: RustMetadataCurrentItem = + serde_json::from_value(vector["expected"].clone()).expect("item parses"); + + for item in [binary, json] { + let page = page_of(py, vec![], vec![item]); + let items = page.getattr("items").expect("items"); + assert_item(&items.get_item(0).expect("item"), &vector["expected"]); + } + } + }); + } + + #[test] + fn given_response_vectors_when_converting_binary_and_json_should_match_expected_json() { + Python::initialize(); + Python::attach(|py| { + for vector in vectors("responses") { + let bytes = decode_hex(&vector["response_hex"]); + let (wire, consumed) = + GetMetadataChangesResponse::decode(&bytes).expect("response decodes"); + assert_eq!(consumed, bytes.len()); + let binary = RustMetadataChanges::try_from(wire).expect("response converts"); + let json: RustMetadataChanges = + serde_json::from_value(vector["expected"].clone()).expect("response parses"); + + for changes in [binary, json] { + let page = Bound::new(py, MetadataChanges::try_from(changes).expect("page")) + .expect("page wraps"); + assert_page(page.as_any(), &vector["expected"]); + } + } + }); + } + + #[test] + fn given_large_values_when_converting_should_keep_every_bit() { + Python::initialize(); + let fixture: Value = serde_json::from_str(FIXTURE).expect("fixture parses"); + let vector = &fixture["large_values"]; + Python::attach(|py| { + let bytes = decode_hex(&vector["record_hex"]); + let (wire, _) = WireMetadataChangeRecord::decode(&bytes).expect("record decodes"); + let record = RustMetadataChangeRecord::try_from(wire).expect("record converts"); + let page = page_of(py, vec![record], vec![]); + let record = page + .getattr("records") + .and_then(|records| records.get_item(0)) + .expect("record"); + let change = record.getattr("change").expect("change"); + + assert_eq!(attribute::<u64>(&record, "op"), decimal(&vector["op"])); + assert_eq!( + attribute::<u64>(&record, "timestamp"), + decimal(&vector["timestamp"]) + ); + assert_eq!( + attribute::<u32>(&record, "ordinal"), + vector["ordinal"] + .as_u64() + .map(|value| value as u32) + .expect("ordinal") + ); + assert_eq!( + attribute::<u128>(&record, "checksum"), + hex(&vector["checksum"]) + ); + assert_eq!( + attribute::<u64>(&change, "user_created_at"), + decimal(&vector["user_created_at"]) + ); + assert_eq!( + attribute::<u64>(&change, "session"), + decimal(&vector["session"]) + ); + assert_eq!( + attribute::<u128>(&change, "client_id"), + hex(&vector["client_id"]) + ); + }); + } + + #[test] + fn given_request_vectors_when_building_query_should_encode_fixture_bytes_and_http_query() { + Python::initialize(); + for vector in vectors("requests") { + let input = &vector["input"]; + let start = match input["start"].as_str().expect("start") { + "earliest" => MetadataChangeStart::Earliest(), + "latest" => MetadataChangeStart::Latest(), + "after" => MetadataChangeStart::After { + cursor: RustMetadataChangeCursor::from_str( + input["cursor"].as_str().expect("cursor"), + ) + .expect("cursor parses") + .into(), + }, + "current" => MetadataChangeStart::Current { + page_token: input["page_token"] + .as_str() + .map(|text| MetadataCurrentPageToken { + inner: RustMetadataCurrentPageToken::from_str(text) + .expect("token parses"), + }), + }, + other => panic!("unknown start {other}"), + }; + let groups = input["groups"] + .as_array() + .expect("groups") + .iter() + .map(|name| { + RustMetadataChangeGroup::from_str(name.as_str().expect("group name")) + .expect("known group") + .into() + }) + .collect(); + + let query = validated_query( + &start, + Some(groups), + input["since"].as_u64(), + input["until"].as_u64(), + input["limit"].as_u64().map(|limit| limit as u32), + ) + .expect("query is accepted"); + + assert_eq!( + GetMetadataChangesRequest::from(&query).to_bytes().to_vec(), + decode_hex(&vector["request_hex"]), + "{}", + vector["case"] + ); + let http_query = query + .http_query_pairs() + .into_iter() + .map(|(key, value)| format!("{key}={value}")) + .collect::<Vec<_>>() + .join("&"); + assert_eq!(http_query, vector["http_query"].as_str().expect("query")); + } + } + + #[test] + fn given_refused_bodies_and_responses_when_decoding_should_fail() { + let fixture: Value = serde_json::from_str(FIXTURE).expect("fixture parses"); + let refused = &fixture["refused"]; + for vector in refused["record_bodies"].as_array().expect("bodies") { + let wire = WireMetadataChangeRecord { + op: 1, + ordinal: 0, + checksum: 0, + timestamp: 0, + actor_user_id: None, + group: vector["group"].as_u64().expect("group") as u8, + kind: vector["kind"].as_u64().expect("kind") as u8, + body: Bytes::from(decode_hex(&vector["body_hex"])), + }; + assert!( + RustMetadataChangeRecord::try_from(wire).is_err(), + "{}", + vector["case"] + ); + } + for vector in refused["responses"].as_array().expect("responses") { + let bytes = decode_hex(&vector["hex"]); + let decoded = GetMetadataChangesResponse::decode(&bytes) + .map_err(|_| ()) + .and_then(|(response, _)| RustMetadataChanges::try_from(response).map_err(|_| ())); + assert!(decoded.is_err(), "{}", vector["case"]); + } + } + + #[test] + fn given_unknown_records_and_items_when_reading_twice_should_return_the_same_objects() { + Python::initialize(); + let vector = vectors("responses") + .into_iter() + .find(|vector| vector["case"] == "unknown_kind_and_group") + .expect("unknown vector"); + let bytes = decode_hex(&vector["response_hex"]); + let (wire, _) = GetMetadataChangesResponse::decode(&bytes).expect("response decodes"); + let binary = RustMetadataChanges::try_from(wire).expect("response converts"); + let json: RustMetadataChanges = + serde_json::from_value(vector["expected"].clone()).expect("response parses"); + let items = vec![ + RustMetadataCurrentItem::Unknown { + group: 200, + body: Bytes::from_static(&[0xde, 0xad]), + }, + serde_json::from_str(r#"{"group":"future","value":1}"#).expect("item parses"), + ]; + Python::attach(|py| { + let records = binary.records.into_iter().chain(json.records).collect(); + let page = page_of(py, records, items); + let records = page.getattr("records").expect("records"); + assert_eq!(records.len().expect("length"), 4); + for record in records.try_iter().expect("records iterate") { + let change = record.and_then(|record| record.getattr("change")); + assert_same_on_every_read(&change.expect("change"), &["group", "kind", "body"]); + } + let items = page.getattr("items").expect("items"); + assert_eq!(items.len().expect("length"), 2); + for item in items.try_iter().expect("items iterate") { + let body = item.and_then(|item| item.getattr("body")); + assert_same_on_every_read(&body.expect("body"), &["group", "body"]); + } + }); + } + + fn page_of<'py>( + py: Python<'py>, + records: Vec<RustMetadataChangeRecord>, + items: Vec<RustMetadataCurrentItem>, + ) -> Bound<'py, PyAny> { + let changes = RustMetadataChanges { + status: RustMetadataChangeStatus::Ok, + phase: RustMetadataChangePhase::Changes, + log_id: None, + oldest_op: 0, + retained_since: 0, + head_op: 0, + next: None, + has_more: false, + records, + items, + current_page_token: None, + }; + Bound::new( + py, + MetadataChanges::try_from(changes).expect("page converts"), + ) + .expect("page wraps") + .into_any() + } + + fn assert_page(page: &Bound<'_, PyAny>, expected: &Value) { + assert_eq!(text(page, "status"), expected["status"]); + assert_eq!(text(page, "phase"), expected["phase"]); + assert_eq!( + attribute::<Option<u128>>(page, "log_id"), + expected["log_id"] + .as_str() + .map(|_| hex(&expected["log_id"])) + ); + for key in ["oldest_op", "retained_since", "head_op"] { + assert_eq!( + attribute::<u64>(page, key), + decimal(&expected[key]), + "{key}" + ); + } + assert_eq!( + optional_text(page, "next"), + expected["next"].as_str().map(str::to_owned) + ); + assert_eq!( + attribute::<bool>(page, "has_more"), + expected["has_more"].as_bool().expect("has_more") + ); + let records = page.getattr("records").expect("records"); + let expected_records = expected["records"].as_array().expect("records"); + assert_eq!(records.len().expect("length"), expected_records.len()); + for (index, expected) in expected_records.iter().enumerate() { + assert_record(&records.get_item(index).expect("record"), expected); + } + let items = page.getattr("items").expect("items"); + let expected_items = expected["items"].as_array().expect("items"); + assert_eq!(items.len().expect("length"), expected_items.len()); + for (index, expected) in expected_items.iter().enumerate() { + assert_item(&items.get_item(index).expect("item"), expected); + } + assert_eq!( + optional_text(page, "current_page_token"), + expected["current_page_token"].as_str().map(str::to_owned) + ); + } + + fn assert_record(record: &Bound<'_, PyAny>, expected: &Value) { + let fields = expected.as_object().expect("record object"); + assert_eq!(attribute::<u64>(record, "op"), decimal(&fields["op"])); + assert_eq!( + attribute::<u32>(record, "ordinal"), + fields["ordinal"] + .as_u64() + .map(|value| value as u32) + .expect("ordinal") + ); + assert_eq!( + attribute::<u128>(record, "checksum"), + hex(&fields["checksum"]) + ); + assert_eq!( + attribute::<u64>(record, "timestamp"), + decimal(&fields["timestamp"]) + ); + assert_eq!( + attribute::<Option<u32>>(record, "actor_user_id"), + fields + .get("actor_user_id") + .and_then(Value::as_u64) + .map(|value| value as u32) + ); + let change = record.getattr("change").expect("change"); + if record.getattr("kind").expect("kind").is_none() { + assert!(record.getattr("group").expect("group").is_none()); + assert_unknown(&change, fields); + return; + } + assert_eq!(text(record, "group"), fields["group"]); + assert_eq!(text(record, "kind"), fields["kind"]); + assert_body(&change, fields, &ENVELOPE_KEYS); + } + + fn assert_item(item: &Bound<'_, PyAny>, expected: &Value) { + let fields = expected.as_object().expect("item object"); + let body = item.getattr("body").expect("body"); + if item.getattr("group").expect("group").is_none() { + assert_unknown(&body, fields); + return; + } + assert_eq!(text(item, "group"), fields["group"]); + assert_body(&body, fields, &["group"]); + } + + /// An unknown record or item keeps its codes and its raw body: base64 in + /// the binary rendering, the whole object over JSON. + fn assert_unknown(unknown: &Bound<'_, PyAny>, fields: &Map<String, Value>) { + let py = unknown.py(); + assert_eq!(text(unknown, "group"), fields["group"]); + if fields.contains_key("kind") { + assert_eq!(text(unknown, "kind"), fields["kind"]); + } + let body = unknown.getattr("body").expect("body"); + if let Ok(json) = body.extract::<String>() { + let parsed: Value = serde_json::from_str(&json).expect("raw json"); + assert_eq!(parsed, Value::Object(fields.clone())); + } else { + let encoded = py + .import("base64") + .and_then(|base64| base64.call_method1("b64encode", (body,))) + .and_then(|encoded| encoded.call_method0("decode")) + .and_then(|encoded| encoded.extract::<String>()) + .expect("base64 body"); + assert_eq!(encoded, fields["body"].as_str().expect("body text")); + } + } + + fn assert_body(body: &Bound<'_, PyAny>, fields: &Map<String, Value>, skipped: &[&str]) { + for (key, value) in fields + .iter() + .filter(|(key, _)| !skipped.contains(&key.as_str())) + { + match key.as_str() { + "options" => assert_options(body, value), + "permissions" => { + let permissions = body.getattr("permissions").expect("permissions"); + let expected: Option<RustPermissions> = + serde_json::from_value(value.clone()).expect("permissions parse"); + let actual = (!permissions.is_none()).then(|| { + permissions + .extract::<Permissions>() + .map_err(Into::<PyErr>::into) + .expect("permissions extract") + .inner + }); + assert_eq!(actual, expected); + } + "status" => { + let expected = RustUserStatus::from_str(value.as_str().expect("status")) + .expect("known status"); + assert_eq!(attribute::<UserStatus>(body, key), expected.into()); + } + "client_id" => assert_eq!(attribute::<u128>(body, key), hex(value)), + "reason" => assert_eq!(text(body, key), *value), + _ => match value { + Value::Number(_) => assert_eq!(attribute::<u64>(body, key), decimal(value)), + Value::String(text) => assert_eq!(&attribute::<String>(body, key), text), + Value::Bool(flag) => assert_eq!(attribute::<bool>(body, key), *flag), + Value::Array(_) => assert_eq!( + attribute::<Vec<u32>>(body, key), + serde_json::from_value::<Vec<u32>>(value.clone()).expect("ids") + ), + Value::Null => assert!(body.getattr(key.as_str()).expect(key).is_none()), + Value::Object(_) => panic!("unexpected object field {key}"), + }, + } + } + for key in OPTIONAL_BODY_KEYS { + if !fields.contains_key(key) + && let Ok(value) = body.getattr(key) + { + assert!(value.is_none(), "{key} must be absent"); + } + } + } + + /// Compare both provenance getters with the fixture's option map, or + /// the unreadable marker when the map is null. + fn assert_options(body: &Bound<'_, PyAny>, expected: &Value) { + let py = body.py(); + if expected.is_null() { + assert!(attribute::<bool>(body, "options_unreadable")); + assert!(body.getattr("options").expect("options").is_none()); + assert!(body.getattr("derived_options").expect("derived").is_none()); + return; + } + assert!(!attribute::<bool>(body, "options_unreadable")); + let expected = expected.as_object().expect("option map"); + for (getter, explicit) in [("options", true), ("derived_options", false)] { + let actual = body + .getattr(getter) + .and_then(|headers| headers.call_method0("to_scalar_dict")) + .expect("scalar options"); + let actual = actual.cast::<PyDict>().expect("dict"); + let wanted = PyDict::new(py); + for (key, option) in expected + .iter() + .filter(|(_, option)| option["explicit"].as_bool() == Some(explicit)) + { + let text = option["value"].as_str().expect("value text"); + let value: Bound<'_, PyAny> = match option["kind"].as_str() { + Some("uint64") => text + .parse::<u64>() + .expect("u64") + .into_bound_py_any(py) + .expect("int"), + Some("uint32") => text + .parse::<u32>() + .expect("u32") + .into_bound_py_any(py) + .expect("int"), + Some("bool") => text + .parse::<bool>() + .expect("bool") + .into_bound_py_any(py) + .expect("bool"), + _ => PyString::new(py, text).into_any(), + }; + wanted.set_item(key, value).expect("set item"); + } + assert!( + actual.eq(&wanted).expect("comparable"), + "{getter}: {actual} != {wanted}" + ); + } + } + + fn assert_same_on_every_read(object: &Bound<'_, PyAny>, names: &[&str]) { + for name in names { + let first = object.getattr(*name).expect("first read"); + let second = object.getattr(*name).expect("second read"); + assert!(first.is(&second), "{name} is rebuilt on every read"); + } + } + + fn vectors(section: &str) -> Vec<Value> { + let fixture: Value = serde_json::from_str(FIXTURE).expect("fixture parses"); + fixture[section].as_array().expect("vector list").clone() + } + + fn attribute<'py, T: FromPyObjectOwned<'py>>(object: &Bound<'py, PyAny>, name: &str) -> T { + object + .getattr(name) + .expect("attribute exists") + .extract::<T>() + .map_err(Into::<PyErr>::into) + .expect("attribute extracts") + } + + fn text(object: &Bound<'_, PyAny>, name: &str) -> String { + object + .getattr(name) + .and_then(|value| value.str()) + .map(|value| value.to_string()) + .expect("attribute text") + } + + fn optional_text(object: &Bound<'_, PyAny>, name: &str) -> Option<String> { + let value = object.getattr(name).expect("attribute exists"); + (!value.is_none()).then(|| value.str().expect("text").to_string()) + } + + /// A u64 the fixture writes as a JSON number, or as decimal text above + /// 2^53. + fn decimal(value: &Value) -> u64 { + value + .as_u64() + .or_else(|| value.as_str().and_then(|text| text.parse().ok())) + .expect("u64 value") + } + + fn hex(value: &Value) -> u128 { + u128::from_str_radix(value.as_str().expect("hex text"), 16).expect("u128 hex") + } + + fn decode_hex(text: &Value) -> Vec<u8> { + let text = text.as_str().expect("hex text"); + (0..text.len()) + .step_by(2) + .map(|index| u8::from_str_radix(&text[index..index + 2], 16).expect("hex byte")) + .collect() + } +}
diff --git a/foreign/python/tests/fixtures/metadata_changes.json b/foreign/python/tests/fixtures/metadata_changes.json new file mode 100644 index 0000000..26e7faf --- /dev/null +++ b/foreign/python/tests/fixtures/metadata_changes.json
@@ -0,0 +1,1997 @@ +{ + "schema_version": 1, + "generated_by": "core/common/examples/metadata_changes_fixtures.rs", + "conventions": { + "byte_strings": "Every *_hex field is lowercase hex without separators.", + "expected": "expected is the JSON the HTTP route renders. Binary decoders must reach the same values.", + "record_hex": "record_hex is one record envelope as a response carries it. body_hex is the body alone.", + "refused": "Every refused vector must fail to decode or validate. Query and HTTP refusals are invalid_command.", + "request_hex": "request_hex is the command payload without framing. http_query is the query string of GET /metadata/changes.", + "u128_values": "128-bit values (log ids, checksums, client ids) are 32 lowercase hex characters in JSON.", + "u64_values": "Ops and timestamps are JSON numbers below 2^53 in every vector except large_values, which carries them as decimal strings.", + "unknown_kinds": "A record of an unknown group or kind is kept with its codes and raw body, never refused." + }, + "constants": { + "command_code": 17, + "command_name": "metadata.changes", + "cursor_prefix": "mc1_", + "page_token_prefix": "cs1_", + "cursor_size": 44, + "cursor_text_length": 63, + "max_page_token_size": 316, + "max_page_token_key_size": 260, + "default_limit": 100, + "max_limit": 1000, + "groups": { + "consumer_group_members": { + "code": 5, + "class": "activity", + "parents": [ + "consumer_groups", + "partitions", + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32][member_id:u32]" + }, + "consumer_groups": { + "code": 4, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32]" + }, + "partitions": { + "code": 3, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "personal_access_tokens": { + "code": 7, + "class": "resource", + "parents": [ + "users" + ], + "page_token_key": "[user_id:u32][name_len:u8][name]" + }, + "sessions": { + "code": 8, + "class": "activity", + "parents": [ + "users" + ], + "page_token_key": "[epoch:u64]" + }, + "streams": { + "code": 1, + "class": "resource", + "parents": [], + "page_token_key": "[stream_id:u32]" + }, + "topics": { + "code": 2, + "class": "resource", + "parents": [ + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "users": { + "code": 6, + "class": "resource", + "parents": [], + "page_token_key": "[user_id:u32]" + } + }, + "kinds": { + "consumer_group_created": { + "code": 30, + "group": "consumer_groups" + }, + "consumer_group_deleted": { + "code": 31, + "group": "consumer_groups" + }, + "consumer_group_member_joined": { + "code": 40, + "group": "consumer_group_members" + }, + "consumer_group_member_left": { + "code": 41, + "group": "consumer_group_members" + }, + "consumer_group_partition_handed_over": { + "code": 42, + "group": "consumer_group_members" + }, + "partitions_changed": { + "code": 20, + "group": "partitions" + }, + "personal_access_token_created": { + "code": 60, + "group": "personal_access_tokens" + }, + "personal_access_token_deleted": { + "code": 61, + "group": "personal_access_tokens" + }, + "session_closed": { + "code": 71, + "group": "sessions" + }, + "session_registered": { + "code": 70, + "group": "sessions" + }, + "stream_created": { + "code": 1, + "group": "streams" + }, + "stream_deleted": { + "code": 3, + "group": "streams" + }, + "stream_updated": { + "code": 2, + "group": "streams" + }, + "topic_created": { + "code": 10, + "group": "topics" + }, + "topic_deleted": { + "code": 12, + "group": "topics" + }, + "topic_updated": { + "code": 11, + "group": "topics" + }, + "user_created": { + "code": 50, + "group": "users" + }, + "user_deleted": { + "code": 52, + "group": "users" + }, + "user_password_changed": { + "code": 53, + "group": "users" + }, + "user_permissions_updated": { + "code": 54, + "group": "users" + }, + "user_updated": { + "code": 51, + "group": "users" + } + }, + "statuses": { + "behind": 1, + "expired": 2, + "ok": 0, + "reset": 3 + }, + "phases": { + "changes": 0, + "current": 1 + }, + "start_modes": { + "after": 2, + "current": 3, + "earliest": 0, + "latest": 1 + }, + "member_left_reasons": { + "left": 0, + "session_ended": 1 + }, + "token_deleted_reasons": { + "deleted": 0, + "expired": 1 + }, + "session_closed_reasons": { + "expired": 1, + "logout": 0, + "revoked": 2 + }, + "permissions_updated_reasons": { + "resource_deleted": 1, + "updated": 0 + } + }, + "records": [ + { + "case": "stream_created", + "group": 1, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "record_hex": "191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "stream_updated_unreadable_options", + "group": 1, + "kind": 2, + "body_hex": "03000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "record_hex": "1a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "expected": { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + } + }, + { + "case": "stream_deleted_deferred", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "record_hex": "1b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "expected": { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + }, + { + "case": "topic_created", + "group": 2, + "kind": 10, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "record_hex": "1c1000000000000000000000683d9e8b03f7c1a500000000000000001c00be01094106000104000000020a78000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "op": 4124, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d68", + "timestamp": 1760356800004124, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "partitions_count": 8, + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "topic_updated", + "group": 2, + "kind": 11, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "record_hex": "1d1000000000000000000000693d9e8b03f7c1a500000000000000001d00be01094106000104000000020b3d000000030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "expected": { + "op": 4125, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d69", + "timestamp": 1760356800004125, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_updated", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics-v2", + "partitions_count": 8, + "options": {} + } + }, + { + "case": "topic_deleted_immediate", + "group": 2, + "kind": 12, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "record_hex": "1e10000000000000000000006a3d9e8b03f7c1a500000000000000001e00be01094106000104000000020c2e000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "expected": { + "op": 4126, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6a", + "timestamp": 1760356800004126, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics" + } + }, + { + "case": "partitions_changed_deleted_tail", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "record_hex": "1f10000000000000000000006b3d9e8b03f7c1a500000000000000001f00be01094106000104000000031435000000030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "expected": { + "op": 4127, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6b", + "timestamp": 1760356800004127, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 5, + "removed_partition_ids": [ + 5, + 6, + 7 + ], + "requested_op": 4200 + } + }, + { + "case": "partitions_changed_created", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000a0000000000000000", + "record_hex": "2010000000000000000000006c3d9e8b03f7c1a500000000000000002000be01094106000104000000031421000000030000001910000000000000000000001d100000000000000a0000000000000000", + "expected": { + "op": 4128, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6c", + "timestamp": 1760356800004128, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 10, + "removed_partition_ids": [] + } + }, + { + "case": "consumer_group_created", + "group": 4, + "kind": 30, + "body_hex": "030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "record_hex": "2110000000000000000000006d3d9e8b03f7c1a500000000000000002100be01094106000104000000041e28000000030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "expected": { + "op": 4129, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6d", + "timestamp": 1760356800004129, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 1 + } + }, + { + "case": "consumer_group_deleted", + "group": 4, + "kind": 31, + "body_hex": "030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "record_hex": "2210000000000000000000006e3d9e8b03f7c1a500000000000000002200be01094106000104000000041f31000000030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "expected": { + "op": 4130, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6e", + "timestamp": 1760356800004130, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 0, + "requested_op": 4300 + } + }, + { + "case": "consumer_group_member_joined", + "group": 5, + "kind": 40, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "record_hex": "2310000000000000000000006f3d9e8b03f7c1a500000000000000002300be01094106000109000000052838000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "expected": { + "op": 4131, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6f", + "timestamp": 1760356800004131, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_joined", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 2, + "partition_ids": [] + } + }, + { + "case": "consumer_group_member_left_session_ended", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "record_hex": "241000000000000000000000703d9e8b03f7c1a500000000000000002400be01094106000109000000052935000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "expected": { + "op": 4132, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d70", + "timestamp": 1760356800004132, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_left", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 1, + "reason": "session_ended" + } + }, + { + "case": "consumer_group_partition_handed_over", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d1000000000000000000000020000000101000000", + "record_hex": "251000000000000000000000713d9e8b03f7c1a500000000000000002500be010941060000052a25000000030000001910000000000000000000001d1000000000000000000000020000000101000000", + "expected": { + "op": 4133, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d71", + "timestamp": 1760356800004133, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 2, + "member_id": 1 + } + }, + { + "case": "consumer_group_partition_released", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d10000000000000000000000300000000", + "record_hex": "261000000000000000000000723d9e8b03f7c1a500000000000000002600be010941060000052a21000000030000001910000000000000000000001d10000000000000000000000300000000", + "expected": { + "op": 4134, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d72", + "timestamp": 1760356800004134, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 3 + } + }, + { + "case": "user_created_with_permissions", + "group": 6, + "kind": 50, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "record_hex": "271000000000000000000000733d9e8b03f7c1a500000000000000002700be010941060001000000000632480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "op": 4135, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d73", + "timestamp": 1760356800004135, + "actor_user_id": 0, + "group": "users", + "kind": "user_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "user_updated_inactive", + "group": 6, + "kind": 51, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "record_hex": "281000000000000000000000743d9e8b03f7c1a500000000000000002800be010941060001000000000633280000000700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "expected": { + "op": 4136, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d74", + "timestamp": 1760356800004136, + "actor_user_id": 0, + "group": "users", + "kind": "user_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "inactive", + "permissions": null, + "options": {} + } + }, + { + "case": "user_deleted", + "group": 6, + "kind": 52, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "291000000000000000000000753d9e8b03f7c1a500000000000000002900be0109410600010000000006341d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4137, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d75", + "timestamp": 1760356800004137, + "actor_user_id": 0, + "group": "users", + "kind": "user_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_password_changed", + "group": 6, + "kind": 53, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "2a1000000000000000000000763d9e8b03f7c1a500000000000000002a00be0109410600010700000006351d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4138, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d76", + "timestamp": 1760356800004138, + "actor_user_id": 7, + "group": "users", + "kind": "user_password_changed", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_permissions_pruned", + "group": 6, + "kind": 54, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "record_hex": "2b1000000000000000000000773d9e8b03f7c1a500000000000000002b00be0109410600010400000006362a0000000700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "expected": { + "op": 4139, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d77", + "timestamp": 1760356800004139, + "actor_user_id": 4, + "group": "users", + "kind": "user_permissions_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": false, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": null + }, + "reason": "resource_deleted" + } + }, + { + "case": "personal_access_token_created", + "group": 7, + "kind": 60, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "record_hex": "2c1000000000000000000000783d9e8b03f7c1a500000000000000002c00be01094106000107000000073c1d0000000700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "expected": { + "op": 4140, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d78", + "timestamp": 1760356800004140, + "actor_user_id": 7, + "group": "personal_access_tokens", + "kind": "personal_access_token_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "expiry": 1760400000000000 + } + }, + { + "case": "personal_access_token_deleted_expired", + "group": 7, + "kind": 61, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e01", + "record_hex": "2d1000000000000000000000793d9e8b03f7c1a500000000000000002d00be010941060000073d160000000700000000b832c8fb4006000863692d746f6b656e01", + "expected": { + "op": 4141, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d79", + "timestamp": 1760356800004141, + "group": "personal_access_tokens", + "kind": "personal_access_token_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "reason": "expired" + } + }, + { + "case": "session_registered", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "record_hex": "2e10000000000000000000007a3d9e8b03f7c1a500000000000000002e00be010941060001070000000846240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "op": 4142, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7a", + "timestamp": 1760356800004142, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_registered", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + }, + { + "case": "session_closed_expired", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "record_hex": "2f10000000000000000000007b3d9e8b03f7c1a500000000000000002f00be0109410600000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "expected": { + "op": 4143, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7b", + "timestamp": 1760356800004143, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "expired" + } + }, + { + "case": "session_closed_revoked", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "record_hex": "3010000000000000000000007c3d9e8b03f7c1a500000000000000003000be010941060001070000000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "expected": { + "op": 4144, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7c", + "timestamp": 1760356800004144, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "revoked" + } + } + ], + "items": [ + { + "case": "stream", + "group": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "item_hex": "01b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "topic", + "group": 2, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "item_hex": "0274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "partitions", + "group": 3, + "body_hex": "030000001910000000000000000000001d1000000000000008000000", + "item_hex": "031c000000030000001910000000000000000000001d1000000000000008000000", + "expected": { + "group": "partitions", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 8 + } + }, + { + "case": "consumer_group", + "group": 4, + "body_hex": "030000001910000000000000000000001d10000000000000000000000772656164657273", + "item_hex": "0424000000030000001910000000000000000000001d10000000000000000000000772656164657273", + "expected": { + "group": "consumer_groups", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers" + } + }, + { + "case": "consumer_group_member", + "group": 5, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "item_hex": "0540000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "expected": { + "group": "consumer_group_members", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "partition_ids": [ + 0, + 2, + 4 + ] + } + }, + { + "case": "user", + "group": 6, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "item_hex": "06480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "group": "users", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "personal_access_token_without_expiry", + "group": 7, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "item_hex": "071d0000000700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "expected": { + "group": "personal_access_tokens", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token" + } + }, + { + "case": "session", + "group": 8, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "item_hex": "08240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "group": "sessions", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + } + ], + "cursors": [ + { + "case": "zero", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d000000000000000000000000000000000000000000000000000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 0, + "checksum": "00000000000000000000000000000000", + "ordinal": 0 + }, + { + "case": "first_record_of_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAABgQAAAAAAAApcH3A4ueLUwAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000001810000000000000a5c1f7038b9e2d4c000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4120, + "checksum": "00000000000000004c2d9e8b03f7c1a5", + "ordinal": 0 + }, + { + "case": "inside_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f2000000000000000001000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4187, + "checksum": "0000000000000000f2a4c6e8b0d1f359", + "ordinal": 1 + }, + { + "case": "every_field_at_maximum", + "text": "mc1___________________________________________________________8", + "hex": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + "log_id": "ffffffffffffffffffffffffffffffff", + "op": 18446744073709551615, + "checksum": "ffffffffffffffffffffffffffffffff", + "ordinal": 4294967295 + } + ], + "page_tokens": [ + { + "case": "streams", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000000101040003000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 1, + "group": "streams", + "last_key_hex": "03000000" + }, + { + "case": "topics", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 3, + "group": "topics", + "last_key_hex": "0300000000000000" + }, + { + "case": "partitions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8DCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff0308000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "partitions", + "last_key_hex": "0300000000000000" + }, + { + "case": "consumer_groups", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8EDAADAAAAAAAAAAIAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff040c00030000000000000002000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_groups", + "last_key_hex": "030000000000000002000000" + }, + { + "case": "consumer_group_members", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8FEAADAAAAAAAAAAIAAAABAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff05100003000000000000000200000001000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_group_members", + "last_key_hex": "03000000000000000200000001000000" + }, + { + "case": "users", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAACAGBAAHAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000002006040007000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 32, + "group": "users", + "last_key_hex": "07000000" + }, + { + "case": "personal_access_tokens", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAGAHDQAHAAAACGNpLXRva2Vu", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e1000000000000060070d00070000000863692d746f6b656e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 96, + "group": "personal_access_tokens", + "last_key_hex": "070000000863692d746f6b656e" + }, + { + "case": "sessions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAKAICACUEQAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000a00808009411000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 160, + "group": "sessions", + "last_key_hex": "9411000000000000" + }, + { + "case": "longest_token_name", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8HBAH______25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubg", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff070401ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "personal_access_tokens", + "last_key_hex": "ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e" + } + ], + "requests": [ + { + "case": "earliest_every_group", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000", + "http_query": "from=earliest&limit=100" + }, + { + "case": "earliest_many_duplicate_groups", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "users" + ], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000640000000201060000", + "http_query": "from=earliest&groups=streams,users&limit=100" + }, + { + "case": "earliest_since_users_and_tokens", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "users", + "personal_access_tokens" + ], + "since": 1760353200000000, + "until": null, + "limit": 100 + }, + "request_hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001004c2a2b0841060000640000000206070000", + "http_query": "from=earliest&groups=users,personal_access_tokens&since=1760353200000000&limit=100" + }, + { + "case": "latest_streams", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [ + "streams" + ], + "since": null, + "until": null, + "limit": 1 + }, + "request_hex": "01000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000001010000", + "http_query": "from=latest&groups=streams&limit=1" + }, + { + "case": "after_cursor_with_time_range", + "input": { + "start": "after", + "cursor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "page_token": null, + "groups": [ + "topics", + "streams" + ], + "since": 1760356800000000, + "until": 1760360400000000, + "limit": 500 + }, + "request_hex": "02d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000000100f0bd010941060001009451d809410600f40100000202010000", + "http_query": "after=mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA&groups=streams,topics&since=1760356800000000&until=1760360400000000&limit=500" + }, + { + "case": "current_first_page", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201020000", + "http_query": "from=current&groups=streams,topics&limit=1000" + }, + { + "case": "current_continuation", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201024000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "http_query": "from=current¤t_after=cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA&groups=streams,topics&limit=1000" + } + ], + "responses": [ + { + "case": "changes_page", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001b10000000000000673d9e8b03f7c1a50000000000000000000000000103000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465641a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d7632011b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABsQAAAAAAAAZz2eiwP3waUAAAAAAAAAAAAAAAA", + "has_more": true, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + }, + { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "unknown_kind_and_group", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001910000000000000653d9e8b03f7c1a50000000000000000010000000002000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be01094106000104000000010904000000deadbeef191000000000000001000000653d9e8b03f7c1a500000000000000001900be01094106000104000000c80104000000deadbeef000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABkQAAAAAAAAZT2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": false, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "1", + "kind": "9", + "body": "3q2+7w==" + }, + { + "op": 4121, + "ordinal": 1, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "200", + "kind": "1", + "body": "3q2+7w==" + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "current_page", + "response_hex": "000101d7931c6e2a8f4b0d0000000000000000010000000000000000000000000000005e1000000000000001d7931c6e2a8f4b0d00000000000000005b10000000000000a73d9e8b03f7c1a500000000000000000100000001000000000200000001b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465640274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c0800000000000040000000004000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "expected": { + "status": "ok", + "phase": "current", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 1, + "retained_since": 0, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAApz2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": true, + "records": [], + "items": [ + { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + ], + "current_page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA" + } + }, + { + "case": "behind_before_op_1", + "response_hex": "010000000000000000000000000000000000000000000000000000000000000000000000000000", + "expected": { + "status": "behind", + "phase": "changes", + "log_id": null, + "oldest_op": 0, + "retained_since": 0, + "head_op": 0, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "expired", + "response_hex": "020001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "expired", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "reset", + "response_hex": "030001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "reset", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + } + ], + "large_values": { + "record_hex": "0300000000002000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff01ffffffff084624000000ffffffff0100000000002000feffffffffffffffffffffffffffffff0200000000002000", + "op": "9007199254740995", + "ordinal": 4294967295, + "timestamp": "18446744073709551615", + "checksum": "ffffffffffffffffffffffffffffffff", + "user_created_at": "9007199254740993", + "session": "9007199254740994", + "client_id": "fffffffffffffffffffffffffffffffe" + }, + "refused": { + "cursors": [ + { + "case": "wrong_prefix", + "text": "mc2_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "missing_prefix", + "text": "15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "too_long", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAAA" + }, + { + "case": "too_short", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA" + }, + { + "case": "padded", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA=" + }, + { + "case": "standard_alphabet_character", + "text": "mc1_+5McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "op_zero_with_checksum", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAA" + } + ], + "page_tokens": [ + { + "case": "wrong_prefix", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA" + }, + { + "case": "trailing_byte", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAAAA" + }, + { + "case": "wrong_key_width", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBCAADAAAAAAAAAA" + }, + { + "case": "group_outside_mask", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAIBBAADAAAA" + }, + { + "case": "head_below_anchor", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAABAAAAAAAAAAEBBAADAAAA" + }, + { + "case": "empty", + "text": "cs1_" + } + ], + "record_bodies": [ + { + "case": "unknown_member_left_reason", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000002" + }, + { + "case": "presence_byte_two", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279022c10000000000000" + }, + { + "case": "unsorted_removed_partition_ids", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000090000000600000007000000016810000000000000" + }, + { + "case": "known_kind_under_wrong_group", + "group": 2, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564" + }, + { + "case": "trailing_byte", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000000" + }, + { + "case": "truncated", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a09080706050403020194110000000000" + } + ], + "requests": [ + { + "case": "cursor_bytes_without_after", + "hex": "000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "trailing_byte", + "hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006400000000000000" + }, + { + "case": "unknown_start_mode", + "hex": "040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "non_canonical_cursor", + "hex": "02d7931c6e2a8f4b0d000000000000000000000000000000000100000000000000000000000000000000000000000064000000000000" + } + ], + "responses": [ + { + "case": "items_in_changes_phase", + "hex": "000001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + }, + { + "case": "unknown_status", + "hex": "040001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + } + ], + "queries": [ + { + "case": "limit_zero", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 0 + } + }, + { + "case": "limit_above_maximum", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 1001 + } + }, + { + "case": "since_above_until", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 2, + "until": 1, + "limit": 100 + } + }, + { + "case": "time_bound_with_latest", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 1, + "until": null, + "limit": 100 + } + }, + { + "case": "time_bound_with_current", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": 1, + "limit": 100 + } + }, + { + "case": "current_missing_parent_group", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + }, + { + "case": "page_token_for_other_groups", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + } + ], + "http_queries": [ + { + "case": "no_start", + "query": "groups=streams" + }, + { + "case": "from_and_after", + "query": "from=earliest&after=mc1_x" + }, + { + "case": "unknown_from", + "query": "from=newest" + }, + { + "case": "duplicate_key", + "query": "from=earliest&from=latest" + }, + { + "case": "unknown_key", + "query": "from=earliest&verbose=1" + }, + { + "case": "empty_group_element", + "query": "from=earliest&groups=streams,,topics" + }, + { + "case": "trailing_group_comma", + "query": "from=earliest&groups=streams," + }, + { + "case": "unknown_group", + "query": "from=earliest&groups=everything" + }, + { + "case": "negative_since", + "query": "from=earliest&since=-1" + }, + { + "case": "leading_zero_until", + "query": "from=earliest&until=01" + }, + { + "case": "non_numeric_limit", + "query": "from=earliest&limit=ten" + }, + { + "case": "token_without_current", + "query": "from=latest¤t_after=cs1_AAAA" + } + ] + } +}
diff --git a/foreign/python/tests/test_metadata_changes.py b/foreign/python/tests/test_metadata_changes.py new file mode 100644 index 0000000..3c2394a --- /dev/null +++ b/foreign/python/tests/test_metadata_changes.py
@@ -0,0 +1,455 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Tests for reading the metadata change log. + +The server cases run once per transport through `transport_client`. The vector +cases read the fixture the Rust encoders generated and need no server. The +keyed stream case needs a server key and lives in `test_resource_options.py`. +""" + +import json +from pathlib import Path + +import pytest + +from apache_iggy import ( + GlobalPermissions, + IggyClient, + IggyError, + MetadataChangeCursor, + MetadataChangeGroup, + MetadataChangeKind, + MetadataChangePhase, + MetadataChangeStart, + MetadataChangeStatus, + MetadataCurrentPageToken, + MetadataMemberLeftReason, + MetadataPermissionsUpdatedReason, + MetadataSessionClosedReason, + MetadataStreamChange, + MetadataStreamDeleted, + MetadataTokenDeletedReason, + MetadataTopicChange, + Permissions, + TopicOptions, +) + +from .utils import ( + METADATA_CHANGES_PAGE_LIMIT, + follow_stream_changes, + latest_metadata_cursor, + stream_records, + unique_credentials, +) + +FIXTURE = json.loads( + (Path(__file__).parent / "fixtures" / "metadata_changes.json").read_text() +) +CONSTANTS = FIXTURE["constants"] +INVALID_COMMAND = 3 +UNAUTHORIZED = 41 +ROOT_USER_ID = 0 +STREAMS = MetadataChangeGroup.Streams +TOPICS = MetadataChangeGroup.Topics +ENUM_SECTIONS = ( + (MetadataChangeGroup, "groups"), + (MetadataChangeKind, "kinds"), + (MetadataChangeStatus, "statuses"), + (MetadataChangePhase, "phases"), + (MetadataMemberLeftReason, "member_left_reasons"), + (MetadataTokenDeletedReason, "token_deleted_reasons"), + (MetadataSessionClosedReason, "session_closed_reasons"), + (MetadataPermissionsUpdatedReason, "permissions_updated_reasons"), +) + + +@pytest.mark.usefixtures("iggy_client") +class TestMetadataChangesFollow: + """Following the change log on every transport.""" + + @pytest.mark.asyncio + async def test_given_latest_cursor_when_creating_renaming_and_deleting_stream_should_follow_each_change( # noqa: E501 + self, transport_client: IggyClient, unique_name + ): + """Each change comes back typed, in order and with the root actor.""" + name = unique_name() + renamed = unique_name() + cursor = await latest_metadata_cursor(transport_client, [STREAMS]) + + stream = await transport_client.create_stream(name) + await transport_client.update_stream(stream.id, name=renamed) + await transport_client.delete_stream(stream.id) + + followed = await follow_stream_changes( + transport_client, cursor, [STREAMS], stream.id, 3 + ) + assert all(record.group == STREAMS for record in followed) + records = stream_records(followed, stream.id) + assert [record.kind for record in records] == [ + MetadataChangeKind.StreamCreated, + MetadataChangeKind.StreamUpdated, + MetadataChangeKind.StreamDeleted, + ] + assert all(record.actor_user_id == ROOT_USER_ID for record in records) + positions = [(record.op, record.ordinal) for record in records] + assert positions == sorted(set(positions)) + assert all(record.timestamp > 0 and record.checksum > 0 for record in records) + + created, updated, deleted = (record.change for record in records) + assert isinstance(created, MetadataStreamChange) + assert isinstance(updated, MetadataStreamChange) + assert isinstance(deleted, MetadataStreamDeleted) + assert (created.name, updated.name, deleted.name) == (name, renamed, renamed) + assert created.stream_created_op == records[0].op + assert ( + updated.stream_created_op + == deleted.stream_created_op + == created.stream_created_op + ) + assert created.created_at == stream.created_at + assert not created.options_unreadable + assert created.options is not None + + @pytest.mark.asyncio + async def test_given_topics_group_when_following_should_return_only_topic_records( # noqa: E501 + self, transport_client: IggyClient, unique_name + ): + """A group filter leaves out the stream created on the way.""" + topic_name = unique_name() + cursor = await latest_metadata_cursor(transport_client, [TOPICS]) + + stream = await transport_client.create_stream(unique_name()) + await transport_client.create_topic( + stream.id, topic_name, TopicOptions(partitions_count=2) + ) + + followed = await follow_stream_changes( + transport_client, cursor, [TOPICS], stream.id, 1 + ) + assert all(record.group == TOPICS for record in followed) + records = stream_records(followed, stream.id) + assert [record.kind for record in records] == [MetadataChangeKind.TopicCreated] + topic = records[0].change + assert isinstance(topic, MetadataTopicChange) + assert (topic.name, topic.partitions_count) == (topic_name, 2) + assert topic.topic_created_op == records[0].op + + @pytest.mark.asyncio + async def test_given_limit_one_when_following_should_page_every_record_in_order( + self, transport_client: IggyClient, unique_name + ): + """Pages of one record continue through `next` until `has_more` drops.""" + cursor = await latest_metadata_cursor(transport_client, [STREAMS]) + created = [ + (await transport_client.create_stream(unique_name())).id for _ in range(3) + ] + + records = [] + while True: + page = await transport_client.get_metadata_changes( + MetadataChangeStart.After(cursor), groups=[STREAMS], limit=1 + ) + assert page.status == MetadataChangeStatus.Ok + assert page.phase == MetadataChangePhase.Changes + assert len(page.records) <= 1 + assert page.next is not None + assert (page.next.op, page.next.ordinal) > (cursor.op, cursor.ordinal) + cursor = page.next + records.extend(page.records) + if not page.has_more: + break + + assert [ + record.change.stream_id + for record in records + if record.kind == MetadataChangeKind.StreamCreated + and isinstance(record.change, MetadataStreamChange) + ] == created + assert (cursor.op, cursor.ordinal) == (records[-1].op, records[-1].ordinal) + + @pytest.mark.asyncio + async def test_given_time_range_when_reading_earliest_should_return_only_records_in_range( # noqa: E501 + self, transport_client: IggyClient, unique_name + ): + """`since` and `until` keep the records prepared inside the range.""" + cursor = await latest_metadata_cursor(transport_client, [STREAMS]) + stream = await transport_client.create_stream(unique_name()) + followed = await follow_stream_changes( + transport_client, cursor, [STREAMS], stream.id, 1 + ) + timestamp = stream_records(followed, stream.id)[0].timestamp + + page = await transport_client.get_metadata_changes( + MetadataChangeStart.Earliest(), + groups=[STREAMS], + since=timestamp, + until=timestamp, + limit=METADATA_CHANGES_PAGE_LIMIT, + ) + + assert page.status == MetadataChangeStatus.Ok + assert page.records + assert all(record.timestamp == timestamp for record in page.records) + assert stream_records(page.records, stream.id) + + @pytest.mark.asyncio + async def test_given_foreign_and_future_cursors_when_following_should_answer_reset_and_behind( # noqa: E501 + self, transport_client: IggyClient + ): + """A cursor of another history resets, one past the head is behind.""" + latest = await transport_client.get_metadata_changes( + MetadataChangeStart.Latest() + ) + assert latest.log_id is not None + foreign = MetadataChangeCursor(latest.log_id ^ 1, 1, 1, 0) + ahead = MetadataChangeCursor(latest.log_id, latest.head_op + 1_000_000, 1, 0) + + reset = await transport_client.get_metadata_changes( + MetadataChangeStart.After(foreign) + ) + behind = await transport_client.get_metadata_changes( + MetadataChangeStart.After(ahead) + ) + + assert reset.status == MetadataChangeStatus.Reset + assert (reset.next, reset.records) == (None, []) + assert behind.status == MetadataChangeStatus.Behind + assert (behind.next, behind.records) == (ahead, []) + + +@pytest.mark.usefixtures("iggy_client") +class TestMetadataChangesCurrentState: + """Current state pages on every transport.""" + + @pytest.mark.asyncio + async def test_given_current_scan_when_paging_should_match_stream_list( + self, transport_client: IggyClient, unique_name + ): + """Pages of two streams add up to the stream list, then hand an anchor.""" + for _ in range(3): + await transport_client.create_stream(unique_name()) + + items = [] + page_token = None + anchor = None + pages = 0 + while True: + page = await transport_client.get_metadata_changes( + MetadataChangeStart.Current(page_token), groups=[STREAMS], limit=2 + ) + pages += 1 + assert page.status == MetadataChangeStatus.Ok + assert page.phase == MetadataChangePhase.Current + assert page.records == [] + assert len(page.items) <= 2 + items.extend(page.items) + page_token = page.current_page_token + if page_token is None: + anchor = page.next + break + assert page.has_more + assert page_token.groups == [STREAMS] + assert MetadataCurrentPageToken.parse(str(page_token)) == page_token + + listed = { + (stream.id, stream.name) for stream in await transport_client.get_streams() + } + projected = set() + for item in items: + assert item.group == STREAMS + body = item.body + assert isinstance(body, MetadataStreamChange) + projected.add((body.stream_id, body.name)) + assert projected == listed + assert pages >= 2 + assert anchor is not None + following = await transport_client.get_metadata_changes( + MetadataChangeStart.After(anchor), groups=[STREAMS] + ) + assert following.status == MetadataChangeStatus.Ok + + +@pytest.mark.usefixtures("iggy_client") +class TestMetadataChangesRefusals: + """Refused queries and callers on every transport.""" + + @pytest.mark.asyncio + async def test_given_refused_query_vectors_when_reading_should_raise_before_sending( # noqa: E501 + self, transport_client: IggyClient + ): + """The client refuses what the server would, before any I/O.""" + for vector in FIXTURE["refused"]["queries"]: + with pytest.raises(IggyError) as refusal: + read_vector(transport_client, vector["input"]) + assert refusal.value.code == INVALID_COMMAND, vector["case"] + + @pytest.mark.asyncio + async def test_given_user_without_manage_servers_when_reading_should_raise_unauthorized( # noqa: E501 + self, transport_client: IggyClient, iggy_client: IggyClient, unique_name + ): + """Read permissions do not authorize the change log.""" + username, password = unique_credentials(unique_name) + user = await iggy_client.create_user( + username, + password, + permissions=Permissions( + global_permissions=GlobalPermissions(read_servers=True) + ), + ) + try: + await transport_client.logout_user() + await transport_client.login_user(username, password) + with pytest.raises(IggyError) as refusal: + await transport_client.get_metadata_changes( + MetadataChangeStart.Earliest() + ) + assert refusal.value.code == UNAUTHORIZED + finally: + await iggy_client.delete_user(user.id) + + +class TestMetadataChangesVectors: + """The binding against the vectors the Rust encoders generated.""" + + @pytest.mark.unit + def test_given_cursor_vectors_when_parsing_should_round_trip_every_field(self): + """Cursor text and fields convert both ways without losing a bit.""" + for vector in FIXTURE["cursors"]: + cursor = MetadataChangeCursor.parse(vector["text"]) + fields = ( + int(vector["log_id"], 16), + int(vector["op"]), + int(vector["checksum"], 16), + vector["ordinal"], + ) + assert (cursor.log_id, cursor.op, cursor.checksum, cursor.ordinal) == fields + assert str(cursor) == vector["text"] + assert len(vector["text"]) == CONSTANTS["cursor_text_length"] + assert MetadataChangeCursor(*fields) == cursor + assert hash(MetadataChangeCursor(*fields)) == hash(cursor) + + @pytest.mark.unit + def test_given_page_token_vectors_when_parsing_should_expose_every_field(self): + """A token names its anchor, head, groups and last key.""" + codes = {name: group["code"] for name, group in CONSTANTS["groups"].items()} + for vector in FIXTURE["page_tokens"]: + token = MetadataCurrentPageToken.parse(vector["text"]) + assert str(token) == vector["text"] + assert str(token.anchor) == vector["anchor"] + assert token.required_head == vector["required_head"] + assert str(token.last_group) == vector["group"] + assert token.last_key.hex() == vector["last_key_hex"] + masked = sorted( + (code, name) + for name, code in codes.items() + if vector["groups_mask"] & (1 << (code - 1)) + ) + assert [str(group) for group in token.groups] == [ + name for _, name in masked + ] + + @pytest.mark.unit + def test_given_refused_cursor_and_token_texts_when_parsing_should_raise_value_error( # noqa: E501 + self, + ): + """Prefix, length, padding, alphabet and canonical form are strict.""" + for vector in FIXTURE["refused"]["cursors"]: + with pytest.raises(ValueError, match="invalid metadata change cursor"): + MetadataChangeCursor.parse(vector["text"]) + for vector in FIXTURE["refused"]["page_tokens"]: + with pytest.raises(ValueError, match="invalid metadata current page token"): + MetadataCurrentPageToken.parse(vector["text"]) + + @pytest.mark.unit + def test_given_refused_query_vectors_when_reading_should_raise_before_network_io( # noqa: E501 + self, + ): + """A client that never connects proves the refusal happens before I/O.""" + client = IggyClient("127.0.0.1:1") + for vector in FIXTURE["refused"]["queries"]: + with pytest.raises(IggyError, match="Invalid command") as refusal: + read_vector(client, vector["input"]) + assert refusal.value.code == INVALID_COMMAND, vector["case"] + + @pytest.mark.unit + @pytest.mark.asyncio + async def test_given_request_vectors_when_reading_should_pass_validation(self): + """Accepted queries get past validation and fail only on the transport.""" + client = IggyClient("127.0.0.1:1") + for vector in FIXTURE["requests"]: + with pytest.raises(IggyError) as failure: + await read_vector(client, vector["input"]) + assert failure.value.code != INVALID_COMMAND, vector["case"] + + @pytest.mark.unit + def test_given_fixture_constants_when_reading_enums_should_cover_every_name(self): + """Every group, kind, status, phase and reason has its member.""" + for enum_type, section in ENUM_SECTIONS: + members = { + str(value) + for value in vars(enum_type).values() + if isinstance(value, enum_type) + } + assert members == set(CONSTANTS[section]), section + + @pytest.mark.unit + def test_given_enum_members_when_used_as_keys_should_hash_by_value(self): + """Members key dicts and sets, and equal members hash alike.""" + for enum_type, section in ENUM_SECTIONS: + members = [ + value + for value in vars(enum_type).values() + if isinstance(value, enum_type) + ] + names = {member: str(member) for member in members} + assert len(set(members)) == len(names) == len(members), section + assert all(names[member] == str(member) for member in members), section + for vector in FIXTURE["page_tokens"]: + last_group = MetadataCurrentPageToken.parse(vector["text"]).last_group + assert last_group in {group(vector["group"])} + assert hash(last_group) == hash(group(vector["group"])) + + +def read_vector(client: IggyClient, query: dict): + """Start `get_metadata_changes` with a fixture query input.""" + start = { + "earliest": lambda: MetadataChangeStart.Earliest(), + "latest": lambda: MetadataChangeStart.Latest(), + "after": lambda: MetadataChangeStart.After( + MetadataChangeCursor.parse(query["cursor"]) + ), + "current": lambda: MetadataChangeStart.Current( + MetadataCurrentPageToken.parse(query["page_token"]) + if query["page_token"] + else None + ), + }[query["start"]]() + return client.get_metadata_changes( + start, + groups=[group(name) for name in query["groups"]], + since=query["since"], + until=query["until"], + limit=query["limit"], + ) + + +def group(name: str) -> MetadataChangeGroup: + return getattr( + MetadataChangeGroup, "".join(part.title() for part in name.split("_")) + )
diff --git a/foreign/python/tests/test_resource_options.py b/foreign/python/tests/test_resource_options.py index 8e56337..f670b2a 100644 --- a/foreign/python/tests/test_resource_options.py +++ b/foreign/python/tests/test_resource_options.py
@@ -41,6 +41,9 @@ IggyClient, IggyExpiry, MaxTopicSize, + MetadataChangeGroup, + MetadataChangeKind, + MetadataStreamChange, OptionError, OptionKey, PartitionResizePolicy, @@ -66,9 +69,12 @@ from apache_iggy import SendMessage as Message from .utils import ( + follow_stream_changes, get_quic_server_config, get_server_config, get_websocket_server_config, + latest_metadata_cursor, + stream_records, unique_credentials, wait_for_ping, ) @@ -212,6 +218,40 @@ assert stream.name == created.name assert_encryption_projection(stream, "stream", encryption_key_fingerprint(key)) + @pytest.mark.asyncio + async def test_given_stream_key_when_following_metadata_changes_should_expose_fingerprints_without_key( # noqa: E501 + self, transport_client: IggyClient, unique_name + ): + """A change record carries the key projections a stream read does.""" + key = secrets.token_bytes(STREAM_KEY_LENGTH) + streams = [MetadataChangeGroup.Streams] + cursor = await latest_metadata_cursor(transport_client, streams) + + created = await transport_client.create_stream( + unique_name(), StreamOptions(encryption_key=key) + ) + + followed = await follow_stream_changes( + transport_client, cursor, streams, created.id, 1 + ) + records = stream_records(followed, created.id) + assert [record.kind for record in records] == [MetadataChangeKind.StreamCreated] + change = records[0].change + assert isinstance(change, MetadataStreamChange) + assert change.options is not None + assert change.derived_options is not None + explicit = change.options.to_scalar_dict() + derived = change.derived_options.to_scalar_dict() + assert derived["encryption_key_source"] == "stream" + assert derived["encryption_key_fingerprint"] == encryption_key_fingerprint(key) + assert derived["encryption_kek_fingerprint"] == ",".join( + created.encryption_kek_fingerprints + ) + assert not NEVER_RETURNED_KEYS & (set(explicit) | set(derived)) + for value in (*explicit.values(), *derived.values()): + assert_secret_absent(str(value), key) + assert_secret_absent(repr(records[0]), key) + @pytest.mark.usefixtures("iggy_client") class TestTopicOptions:
diff --git a/foreign/python/tests/test_stub.py b/foreign/python/tests/test_stub.py new file mode 100644 index 0000000..f37a044 --- /dev/null +++ b/foreign/python/tests/test_stub.py
@@ -0,0 +1,65 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Tests that the tracked type stub describes the native module. +""" + +import ast +import enum +from pathlib import Path + +import pytest + +import apache_iggy + +STUB = ast.parse((Path(__file__).parent.parent / "apache_iggy.pyi").read_text()) +STUB_CLASSES = [node for node in STUB.body if isinstance(node, ast.ClassDef)] + + +class TestStub: + """The stub against the classes the extension module exports.""" + + @pytest.mark.unit + def test_given_stub_enum_bases_when_importing_should_match_native_enums(self): + """A class the stub declares as `enum.Enum` is one at runtime.""" + declared = { + node.name + for node in STUB_CLASSES + if "enum.Enum" in map(ast.unparse, node.bases) + } + native = { + node.name + for node in STUB_CLASSES + if issubclass(getattr(apache_iggy, node.name), enum.Enum) + } + assert declared == native + + @pytest.mark.unit + def test_given_stub_class_constants_when_importing_should_be_native_members(self): + """A `typing.ClassVar` of its own class is a member of that class.""" + for node in STUB_CLASSES: + native_class = getattr(apache_iggy, node.name) + for statement in node.body: + if ( + isinstance(statement, ast.AnnAssign) + and isinstance(statement.target, ast.Name) + and ast.unparse(statement.annotation) + == f"typing.ClassVar[{node.name}]" + ): + member = getattr(native_class, statement.target.id) + assert isinstance(member, native_class), statement.target.id
diff --git a/foreign/python/tests/test_tls.py b/foreign/python/tests/test_tls.py index acbdb19..eadb2f5 100644 --- a/foreign/python/tests/test_tls.py +++ b/foreign/python/tests/test_tls.py
@@ -40,10 +40,22 @@ from testcontainers.core.container import DockerContainer from testcontainers.core.waiting_utils import wait_for_logs -from apache_iggy import Consumer, IggyClient, PollingStrategy +from apache_iggy import ( + Consumer, + IggyClient, + MetadataChangeGroup, + MetadataChangeKind, + PollingStrategy, +) from apache_iggy import SendMessage as Message -from .utils import wait_for_ping, wait_for_server +from .utils import ( + follow_stream_changes, + latest_metadata_cursor, + stream_records, + wait_for_ping, + wait_for_server, +) # Paths resolved relative to this file → repo root REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "..")) @@ -157,6 +169,22 @@ if i < len(polled): assert polled[i].payload().decode("utf-8") == expected_msg + @pytest.mark.asyncio + async def test_follow_metadata_changes_over_tls( + self, tls_client: IggyClient, unique_name + ): + """Test following the metadata change log over TLS.""" + streams = [MetadataChangeGroup.Streams] + cursor = await latest_metadata_cursor(tls_client, streams) + + stream = await tls_client.create_stream(unique_name()) + + followed = await follow_stream_changes( + tls_client, cursor, streams, stream.id, 1 + ) + records = stream_records(followed, stream.id) + assert [record.kind for record in records] == [MetadataChangeKind.StreamCreated] + @pytest.mark.integration class TestTlsConnectionString:
diff --git a/foreign/python/tests/utils.py b/foreign/python/tests/utils.py index fc689f9..9c0a56e 100644 --- a/foreign/python/tests/utils.py +++ b/foreign/python/tests/utils.py
@@ -24,7 +24,14 @@ import socket import time -from apache_iggy import IggyClient +from apache_iggy import ( + IggyClient, + MetadataChangeCursor, + MetadataChangeGroup, + MetadataChangeRecord, + MetadataChangeStart, + MetadataChangeStatus, +) # Server-side limits: usernames are 3-50 bytes, passwords 3-100 bytes. MIN_USERNAME_BYTES = 3 @@ -37,6 +44,11 @@ DEFAULT_HTTP_PORT = 3000 DEFAULT_WEBSOCKET_PORT = 8092 +METADATA_CHANGES_PAGE_LIMIT = 1000 +# A two phase delete commits its record after the call returns. +METADATA_FOLLOW_ATTEMPTS = 100 +METADATA_FOLLOW_INTERVAL_SECONDS = 0.1 + def get_transport_config(port_env_var: str, default_port: int) -> tuple[str, int]: """ @@ -234,3 +246,58 @@ await wait_for_ping(client) await client.login_user(username, password) return client + + +async def latest_metadata_cursor( + client: IggyClient, groups: list[MetadataChangeGroup] +) -> MetadataChangeCursor: + """Return the cursor at the newest retained change of the groups.""" + page = await client.get_metadata_changes( + MetadataChangeStart.Latest(), groups=groups + ) + assert page.status == MetadataChangeStatus.Ok, page + assert page.next is not None + return page.next + + +async def follow_stream_changes( + client: IggyClient, + cursor: MetadataChangeCursor, + groups: list[MetadataChangeGroup], + stream_id: int, + count: int, +) -> list[MetadataChangeRecord]: + """ + Follow the changes after `cursor` until `count` of them name the stream. + + Returns every followed record, so callers can check the ones of other + streams too. + """ + records: list[MetadataChangeRecord] = [] + for _ in range(METADATA_FOLLOW_ATTEMPTS): + page = await client.get_metadata_changes( + MetadataChangeStart.After(cursor), + groups=groups, + limit=METADATA_CHANGES_PAGE_LIMIT, + ) + assert page.status == MetadataChangeStatus.Ok, page + assert page.next is not None + cursor = page.next + records.extend(page.records) + if page.has_more: + continue + if len(stream_records(records, stream_id)) >= count: + break + await asyncio.sleep(METADATA_FOLLOW_INTERVAL_SECONDS) + return records + + +def stream_records( + records: list[MetadataChangeRecord], stream_id: int +) -> list[MetadataChangeRecord]: + """The records whose body names the stream.""" + return [ + record + for record in records + if getattr(record.change, "stream_id", None) == stream_id + ]
diff --git a/foreign/python/uv.lock b/foreign/python/uv.lock index 09e8e6a..cd4c439 100644 --- a/foreign/python/uv.lock +++ b/foreign/python/uv.lock
@@ -11,7 +11,7 @@ [[package]] name = "apache-iggy" -version = "0.9.1.dev2" +version = "0.9.1.dev3" source = { editable = "." } [package.optional-dependencies]
diff --git a/foreign/swift/README.md b/foreign/swift/README.md index 8d4cf79..c8717df 100644 --- a/foreign/swift/README.md +++ b/foreign/swift/README.md
@@ -125,6 +125,67 @@ The server stores it wrapped and never returns it: reads report `encryptionKeySource`, `encryptionKeyFingerprint`, and `encryptionKekFingerprints`, and `EncryptionKey.fingerprint(_:)` computes the fingerprint of a key to compare. Without a key, a server with encryption enabled generates one for the stream. +### Metadata changes + +`getMetadataChanges(_:)` reads the node's change log: one record per public result of a committed metadata operation, such as a stream created, a topic renamed, a user's permissions changed, or a session closed. A mirror keeps a cursor, asks for the records after it, and stores the returned `next` cursor. The call needs root or the global `manageServers` permission: + +```swift +func follow(_ client: IggyClient, groups: [MetadataChangeGroup]) async throws { + var query = MetadataChangesQuery(start: .latest, groups: groups) + while true { + let page = try await client.getMetadataChanges(query) + switch page.status { + case .ok: + for record in page.records { + print("op \(record.op) by \(String(describing: record.actorUserID)): \(record.change)") + } + if let next = page.next { + query.start = .after(next) + } + if !page.hasMore { + try await Task.sleep(for: .seconds(1)) + } + case .behind: + try await Task.sleep(for: .milliseconds(100)) + case .expired, .reset: + query.start = try await rebuild(client, groups: groups) + } + } +} + +/// Pages the current state, then returns the position to follow from. +func rebuild(_ client: IggyClient, groups: [MetadataChangeGroup]) async throws -> MetadataChangeStart { + var query = MetadataChangesQuery(start: .current(nil), groups: groups) + while true { + let page = try await client.getMetadataChanges(query) + switch page.status { + case .ok: + break + case .behind: + try await Task.sleep(for: .milliseconds(100)) + continue + case .expired, .reset: + query.start = .current(nil) + continue + } + for item in page.items { + print(item) + } + if let token = page.currentPageToken { + query.start = .current(token) + } else if let anchor = page.next { + return .after(anchor) + } else { + query.start = .current(nil) + } + } +} +``` + +`expired` means the log no longer holds the history after the cursor, and `reset` means the cursor belongs to another history, for example after the data directory was rebuilt. Either way the mirror rebuilds from current state pages and then follows the changes after the anchor they return. A cursor persists as its `description`, `mc1_` and 59 characters of unpadded Base64url, and `MetadataChangeCursor(text:)` reads it back. + +Current pages need the parent groups of every selected group, for example `topics` with `streams`. Keep a rebuilt state private until a change page answers `ok` with `hasMore` false and a `headOp` at or above the last current page's. An invalid query fails with `invalidCommand` before anything is sent. A record of a kind this build does not know arrives as `.unknown(group:kind:body:)` with its codes and raw body, and records carry option values as the read routes return them, never key material. + ### Reconnection If a request hits a dropped connection, the client reconnects and signs in again within @@ -344,7 +405,9 @@ cargo run --manifest-path Tools/golden-vectors/Cargo.toml -- Tests/IggyTests/Fixtures/golden.json ``` -The resource options tests compare against `Tests/IggyTests/Fixtures/resource_options.json`, the fixture every SDK shares, generated by `core/common/examples/resource_options_fixtures.rs`. Never edit the copy. +The resource options tests compare against `Tests/IggyTests/Fixtures/resource_options.json`, the fixture every SDK shares, generated by `core/common/examples/resource_options_fixtures.rs`. Never edit the copy. The metadata changes tests do the same with `Tests/IggyTests/Fixtures/metadata_changes.json`, generated by `core/common/examples/metadata_changes_fixtures.rs`. + +The metadata changes scenario that churns the change log past its capacity runs only when `IGGY_METADATA_CHANGE_LOG_RESOURCE_CAPACITY` is set for both the server and the test run, for example to 64. ## Contributing
diff --git a/foreign/swift/Sources/Iggy/Client/IggyClient.swift b/foreign/swift/Sources/Iggy/Client/IggyClient.swift index 7bca944..357102a 100644 --- a/foreign/swift/Sources/Iggy/Client/IggyClient.swift +++ b/foreign/swift/Sources/Iggy/Client/IggyClient.swift
@@ -290,6 +290,23 @@ return try Responses.optionSpecs(try await send(.describeOptions, Requests.describeOptions(scope: scope), lineage: lineage)[...]) } + /// Reads one page of the metadata change log or of the current state. + /// + /// The node records one change per public result of a committed metadata + /// operation, such as a stream created, a topic renamed, a user's + /// permissions changed, or a session closed. A mirror keeps a cursor, + /// reads the records after it, and stores the returned + /// ``MetadataChanges/next``. Records carry options as the read routes + /// return them and never key material. Needs root or the global + /// `manageServers` permission, and fails with + /// ``IggyErrorCode/unauthorized`` otherwise. An invalid query fails with + /// ``IggyErrorCode/invalidCommand`` before anything is sent. + public func getMetadataChanges(_ query: MetadataChangesQuery = MetadataChangesQuery()) async throws -> MetadataChanges { + let lineage = try await requireAuthenticated() + try query.validate() + return try Responses.metadataChanges(try await send(.getMetadataChanges, Requests.getMetadataChanges(query), lineage: lineage)[...]) + } + /// Captures a snapshot archive of the server state. public func snapshot(compression: SnapshotCompression = .deflated, types: [SystemSnapshotType] = [.all]) async throws -> Snapshot { let lineage = try await requireAuthenticated()
diff --git a/foreign/swift/Sources/Iggy/Iggy.swift b/foreign/swift/Sources/Iggy/Iggy.swift index 9321385..efd248a 100644 --- a/foreign/swift/Sources/Iggy/Iggy.swift +++ b/foreign/swift/Sources/Iggy/Iggy.swift
@@ -21,5 +21,5 @@ public static let sdkName = "swift-sdk" /// Version of this SDK as reported to the server. Provisional until the /// release wiring lands; the release tooling reads this constant. - public static let sdkVersion = "0.1.1" + public static let sdkVersion = "0.1.2" }
diff --git a/foreign/swift/Sources/Iggy/Models/MetadataChanges.swift b/foreign/swift/Sources/Iggy/Models/MetadataChanges.swift new file mode 100644 index 0000000..ecd0280 --- /dev/null +++ b/foreign/swift/Sources/Iggy/Models/MetadataChanges.swift
@@ -0,0 +1,1510 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import Foundation + +/// Family of metadata change records and the unit of filtering. Raw values +/// are the wire codes. A query without groups selects every group. +public enum MetadataChangeGroup: UInt8, Sendable, Hashable, CaseIterable, CustomStringConvertible { + case streams = 1 + case topics = 2 + case partitions = 3 + case consumerGroups = 4 + case consumerGroupMembers = 5 + case users = 6 + case personalAccessTokens = 7 + case sessions = 8 + + /// The snake_case name the other SDKs and the HTTP API use. + public var description: String { + switch self { + case .streams: "streams" + case .topics: "topics" + case .partitions: "partitions" + case .consumerGroups: "consumer_groups" + case .consumerGroupMembers: "consumer_group_members" + case .users: "users" + case .personalAccessTokens: "personal_access_tokens" + case .sessions: "sessions" + } + } + + public init?(name: String) { + guard let group = Self.allCases.first(where: { $0.description == name }) else { + return nil + } + self = group + } + + /// Groups a current state scan of this group must include, so a mirror + /// can apply the cascades of their deletes. + public var parents: [MetadataChangeGroup] { + switch self { + case .streams, .users: [] + case .topics: [.streams] + case .partitions, .consumerGroups: [.topics, .streams] + case .consumerGroupMembers: [.consumerGroups, .partitions, .topics, .streams] + case .personalAccessTokens, .sessions: [.users] + } + } + + /// Bit of the group in a page token's group mask. + var maskBit: UInt8 { 1 << (rawValue - 1) } +} + +/// Kind of a metadata change record. Raw values are the wire codes. +public enum MetadataChangeKind: UInt8, Sendable, Hashable, CaseIterable, CustomStringConvertible { + case streamCreated = 1 + case streamUpdated = 2 + case streamDeleted = 3 + case topicCreated = 10 + case topicUpdated = 11 + case topicDeleted = 12 + case partitionsChanged = 20 + case consumerGroupCreated = 30 + case consumerGroupDeleted = 31 + case consumerGroupMemberJoined = 40 + case consumerGroupMemberLeft = 41 + case consumerGroupPartitionHandedOver = 42 + case userCreated = 50 + case userUpdated = 51 + case userDeleted = 52 + case userPasswordChanged = 53 + case userPermissionsUpdated = 54 + case personalAccessTokenCreated = 60 + case personalAccessTokenDeleted = 61 + case sessionRegistered = 70 + case sessionClosed = 71 + + /// The snake_case name the other SDKs and the HTTP API use. + public var description: String { + switch self { + case .streamCreated: "stream_created" + case .streamUpdated: "stream_updated" + case .streamDeleted: "stream_deleted" + case .topicCreated: "topic_created" + case .topicUpdated: "topic_updated" + case .topicDeleted: "topic_deleted" + case .partitionsChanged: "partitions_changed" + case .consumerGroupCreated: "consumer_group_created" + case .consumerGroupDeleted: "consumer_group_deleted" + case .consumerGroupMemberJoined: "consumer_group_member_joined" + case .consumerGroupMemberLeft: "consumer_group_member_left" + case .consumerGroupPartitionHandedOver: "consumer_group_partition_handed_over" + case .userCreated: "user_created" + case .userUpdated: "user_updated" + case .userDeleted: "user_deleted" + case .userPasswordChanged: "user_password_changed" + case .userPermissionsUpdated: "user_permissions_updated" + case .personalAccessTokenCreated: "personal_access_token_created" + case .personalAccessTokenDeleted: "personal_access_token_deleted" + case .sessionRegistered: "session_registered" + case .sessionClosed: "session_closed" + } + } + + public init?(name: String) { + guard let kind = Self.allCases.first(where: { $0.description == name }) else { + return nil + } + self = kind + } + + /// The group every record of this kind belongs to. + public var group: MetadataChangeGroup { + switch self { + case .streamCreated, .streamUpdated, .streamDeleted: .streams + case .topicCreated, .topicUpdated, .topicDeleted: .topics + case .partitionsChanged: .partitions + case .consumerGroupCreated, .consumerGroupDeleted: .consumerGroups + case .consumerGroupMemberJoined, .consumerGroupMemberLeft, .consumerGroupPartitionHandedOver: .consumerGroupMembers + case .userCreated, .userUpdated, .userDeleted, .userPasswordChanged, .userPermissionsUpdated: .users + case .personalAccessTokenCreated, .personalAccessTokenDeleted: .personalAccessTokens + case .sessionRegistered, .sessionClosed: .sessions + } + } +} + +/// Outcome of a metadata change query. +public enum MetadataChangeStatus: UInt8, Sendable, Hashable, CaseIterable, CustomStringConvertible { + /// Records or items after the requested position. + case ok = 0 + /// The answering node has not applied the requested position yet. Retry later. + case behind = 1 + /// The node no longer holds every record after the position. Rebuild from current state. + case expired = 2 + /// The position belongs to another history. Rebuild from current state. + case reset = 3 + + public var description: String { + switch self { + case .ok: "ok" + case .behind: "behind" + case .expired: "expired" + case .reset: "reset" + } + } +} + +/// Whether an answer pages change records or current state items. +public enum MetadataChangePhase: UInt8, Sendable, Hashable, CaseIterable, CustomStringConvertible { + case changes = 0 + case current = 1 + + public var description: String { + switch self { + case .changes: "changes" + case .current: "current" + } + } +} + +/// Why a consumer group member was removed. +public enum MetadataMemberLeftReason: UInt8, Sendable, Hashable, CaseIterable, CustomStringConvertible { + case left = 0 + case sessionEnded = 1 + + public var description: String { + switch self { + case .left: "left" + case .sessionEnded: "session_ended" + } + } +} + +/// Why a personal access token was removed. +public enum MetadataTokenDeletedReason: UInt8, Sendable, Hashable, CaseIterable, CustomStringConvertible { + case deleted = 0 + case expired = 1 + + public var description: String { + switch self { + case .deleted: "deleted" + case .expired: "expired" + } + } +} + +/// Why a session was closed. +public enum MetadataSessionClosedReason: UInt8, Sendable, Hashable, CaseIterable, CustomStringConvertible { + case logout = 0 + case expired = 1 + /// The user was deactivated or deleted. + case revoked = 2 + + public var description: String { + switch self { + case .logout: "logout" + case .expired: "expired" + case .revoked: "revoked" + } + } +} + +/// Why a user's public permission set changed. +public enum MetadataPermissionsUpdatedReason: UInt8, Sendable, Hashable, CaseIterable, CustomStringConvertible { + /// An update of the user's permissions replaced the set. + case updated = 0 + /// A stream or topic delete pruned the user's grants on it. + case resourceDeleted = 1 + + public var description: String { + switch self { + case .updated: "updated" + case .resourceDeleted: "resource_deleted" + } + } +} + +/// Position of the last metadata change record a caller consumed. +/// +/// Persist ``description``, `mc1_` followed by 59 characters of unpadded +/// Base64url, and read it back with ``init(text:)``. ``op`` 0 names the +/// beginning of the history ``logID`` identifies. +public struct MetadataChangeCursor: Sendable, Hashable, CustomStringConvertible { + public static let textPrefix = "mc1_" + static let encodedSize = 44 + static let textLength = textPrefix.utf8.count + 59 + + /// Prepare checksum of op 1, the root bootstrap of the history. + public var logID: UInt128Value + public var op: UInt64 + /// Prepare checksum of ``op``. + public var checksum: UInt128Value + /// Position of the record within ``op``. + public var ordinal: UInt32 + + public init(logID: UInt128Value, op: UInt64, checksum: UInt128Value, ordinal: UInt32) { + self.logID = logID + self.op = op + self.checksum = checksum + self.ordinal = ordinal + } + + /// The cursor before the first record of `logID`. + public static func zero(logID: UInt128Value) -> MetadataChangeCursor { + MetadataChangeCursor(logID: logID, op: 0, checksum: .zero, ordinal: 0) + } + + public var isZero: Bool { op == 0 } + + /// Op 0 carries no checksum and no ordinal. + public var isCanonical: Bool { op != 0 || (checksum.isZero && ordinal == 0) } + + /// Parses the `mc1_` text form. Fails with + /// ``IggyErrorCode/invalidCommand`` for any other text, whose error never + /// repeats the text. + public init(text: String) throws { + guard text.utf8.count == Self.textLength, let bytes = Base64URL.decode(text, prefix: Self.textPrefix), bytes.count == Self.encodedSize else { + throw IggyError(.invalidCommand, context: "not a metadata change cursor") + } + var reader = ByteReader(bytes) + guard let cursor = try? Self.decode(from: &reader) else { + throw IggyError(.invalidCommand, context: "not a canonical metadata change cursor") + } + self = cursor + } + + /// The `mc1_` text form. + public var description: String { + var writer = ByteWriter(capacity: Self.encodedSize) + encode(into: &writer) + return Self.textPrefix + Base64URL.encode(writer.bytes) + } + + func encode(into writer: inout ByteWriter) { + writer.write(logID) + writer.write(op) + writer.write(checksum) + writer.write(ordinal) + } + + static func decode(from reader: inout ByteReader) throws -> MetadataChangeCursor { + let cursor = MetadataChangeCursor( + logID: try reader.readUInt128(), op: try reader.readUInt64(), checksum: try reader.readUInt128(), ordinal: try reader.readUInt32()) + guard cursor.isCanonical else { + throw WireError.validation("a cursor at op 0 carries no checksum and no ordinal") + } + return cursor + } +} + +/// Continuation of a current state scan. +/// +/// Pass it back unchanged in ``MetadataChangeStart/current(_:)`` with the +/// same groups to read the next page. Its text form starts with `cs1_`. +public struct MetadataCurrentPageToken: Sendable, Hashable, CustomStringConvertible { + public static let textPrefix = "cs1_" + /// Longest key: a user id and a personal access token name. + static let maxKeySize = 4 + 1 + 255 + static let maxEncodedSize = MetadataChangeCursor.encodedSize + 8 + 1 + 1 + 2 + maxKeySize + + /// The position captured before the scan's first page. + public let anchor: MetadataChangeCursor + /// The highest head any page of the scan reported so far. + public let requiredHead: UInt64 + let groupsMask: UInt8 + /// The group of the last item returned. + public let lastGroup: MetadataChangeGroup + /// The key of the last item returned, in the page token key layout. + public let lastKey: [UInt8] + + /// The groups the scan was started with. + public var groups: Set<MetadataChangeGroup> { + Set(MetadataChangeGroup.allCases.filter { groupsMask & $0.maskBit != 0 }) + } + + /// Parses the `cs1_` text form. Fails with + /// ``IggyErrorCode/invalidCommand`` for any other text, whose error never + /// repeats the text. + public init(text: String) throws { + let longest = Self.textPrefix.utf8.count + (Self.maxEncodedSize + 2) / 3 * 4 + guard text.utf8.count <= longest, let bytes = Base64URL.decode(text, prefix: Self.textPrefix) else { + throw IggyError(.invalidCommand, context: "not a metadata current page token") + } + var reader = ByteReader(bytes) + guard let token = try? Self.decode(from: &reader), reader.isAtEnd else { + throw IggyError(.invalidCommand, context: "not a valid metadata current page token") + } + self = token + } + + init(anchor: MetadataChangeCursor, requiredHead: UInt64, groupsMask: UInt8, lastGroup: MetadataChangeGroup, lastKey: [UInt8]) { + self.anchor = anchor + self.requiredHead = requiredHead + self.groupsMask = groupsMask + self.lastGroup = lastGroup + self.lastKey = lastKey + } + + /// The `cs1_` text form. + public var description: String { + Self.textPrefix + Base64URL.encode(encoded()) + } + + func encoded() -> [UInt8] { + var writer = ByteWriter(capacity: MetadataChangeCursor.encodedSize + 12 + lastKey.count) + anchor.encode(into: &writer) + writer.write(requiredHead) + writer.write(groupsMask) + writer.write(lastGroup.rawValue) + writer.write(UInt16(lastKey.count)) + writer.write(lastKey) + return writer.bytes + } + + /// Decodes `[anchor:44][required_head:u64][groups_mask:u8][group:u8][key_len:u16][last_key]` + /// and checks that the group is selected, the key has that group's width + /// and the head covers the anchor. + static func decode(from reader: inout ByteReader) throws -> MetadataCurrentPageToken { + let anchor = try MetadataChangeCursor.decode(from: &reader) + let requiredHead = try reader.readUInt64() + let groupsMask = try reader.readUInt8() + let groupCode = try reader.readUInt8() + guard let lastGroup = MetadataChangeGroup(rawValue: groupCode) else { + throw WireError.validation("page token names an unknown group") + } + let keyLength = Int(try reader.readUInt16()) + guard keyLength <= maxKeySize else { + throw WireError.validation("page token key of \(keyLength) bytes is too long") + } + let token = MetadataCurrentPageToken( + anchor: anchor, requiredHead: requiredHead, groupsMask: groupsMask, lastGroup: lastGroup, lastKey: Array(try reader.readBytes(keyLength))) + guard groupsMask & lastGroup.maskBit != 0 else { + throw WireError.validation("page token group is not in its group mask") + } + guard requiredHead >= anchor.op else { + throw WireError.validation("page token head is below its anchor") + } + guard token.keyFitsGroup else { + throw WireError.validation("page token key of \(keyLength) bytes does not fit group \(groupCode)") + } + return token + } + + private var keyFitsGroup: Bool { + switch lastGroup { + case .streams, .users: + return lastKey.count == 4 + case .topics, .partitions, .sessions: + return lastKey.count == 8 + case .consumerGroups: + return lastKey.count == 12 + case .consumerGroupMembers: + return lastKey.count == 16 + case .personalAccessTokens: + guard lastKey.count > 4 else { + return false + } + var reader = ByteReader(lastKey[4...]) + return (try? reader.readName()) != nil && reader.isAtEnd + } + } +} + +/// Where a metadata change query starts. The wire code of each case is its +/// position: earliest 0, latest 1, after 2, current 3. +public enum MetadataChangeStart: Sendable, Hashable { + /// The oldest operation the requested groups still hold completely. + case earliest + /// No records, only a cursor at the newest retained record to follow from. + case latest + /// The records after a cursor. + case after(MetadataChangeCursor) + /// Current state pages: nil starts a scan, a token continues it. + case current(MetadataCurrentPageToken?) + + var code: UInt8 { + switch self { + case .earliest: 0 + case .latest: 1 + case .after: 2 + case .current: 3 + } + } +} + +/// One read of the metadata change log or of the current state. +/// +/// Empty ``groups`` selects every group. ``since`` and ``until`` keep records +/// whose prepare timestamp is in the inclusive range and apply to +/// ``MetadataChangeStart/earliest`` and ``MetadataChangeStart/after(_:)`` only. +public struct MetadataChangesQuery: Sendable, Hashable { + /// Records or items per page when the caller does not say. + public static let defaultLimit: UInt32 = 100 + /// Most records or items one page returns. + public static let maxLimit: UInt32 = 1000 + + public var start: MetadataChangeStart + public var groups: [MetadataChangeGroup] + public var since: IggyTimestamp? + public var until: IggyTimestamp? + public var limit: UInt32 + + public init( + start: MetadataChangeStart = .earliest, groups: [MetadataChangeGroup] = [], since: IggyTimestamp? = nil, until: IggyTimestamp? = nil, + limit: UInt32 = defaultLimit + ) { + self.start = start + self.groups = groups + self.since = since + self.until = until + self.limit = limit + } + + /// The checks the server applies, run before anything is sent. Fails with + /// ``IggyErrorCode/invalidCommand`` when the limit is outside 1 to + /// ``maxLimit``, ``since`` is above ``until``, a time bound accompanies + /// latest or current, a cursor is not canonical, a current scan misses a + /// parent group, or a page token was issued for other groups. + public func validate() throws { + guard (1...Self.maxLimit).contains(limit) else { + throw IggyError(.invalidCommand, context: "limit must be 1 to \(Self.maxLimit), got \(limit)") + } + if let since, let until, since > until { + throw IggyError(.invalidCommand, context: "since is above until") + } + let selected = groups.isEmpty ? Set(MetadataChangeGroup.allCases) : Set(groups) + let hasTimeBound = since != nil || until != nil + switch start { + case .earliest: + break + case .after(let cursor): + guard cursor.isCanonical else { + throw IggyError(.invalidCommand, context: "a cursor at op 0 carries no checksum and no ordinal") + } + case .latest: + guard !hasTimeBound else { + throw IggyError(.invalidCommand, context: "time bounds apply to earliest and after only") + } + case .current(let token): + guard !hasTimeBound else { + throw IggyError(.invalidCommand, context: "time bounds apply to earliest and after only") + } + if let group = MetadataChangeGroup.allCases.first(where: { selected.contains($0) && !$0.parents.allSatisfy(selected.contains) }) { + throw IggyError(.invalidCommand, context: "a current scan of \(group) needs its parent groups") + } + if let token, token.groups != selected { + throw IggyError(.invalidCommand, context: "the page token was issued for other groups") + } + } + } +} + +/// Public options of a resource after a change, as the read routes return +/// them. ``unreadable`` marks stored critical state this build cannot read: +/// a mirror keeps its previous values then. +public enum MetadataOptions: Sendable, Hashable { + case readable(ResourceOptions) + case unreadable +} + +/// A stream after a create or update, and a current state streams item. +public struct MetadataStreamChange: Sendable, Hashable { + public var streamID: UInt32 + /// The op that created the stream. A recreated stream with a reused id has another. + public var streamCreatedOp: UInt64 + public var createdAt: IggyTimestamp + public var name: String + public var options: MetadataOptions + + public init(streamID: UInt32, streamCreatedOp: UInt64, createdAt: IggyTimestamp, name: String, options: MetadataOptions) { + self.streamID = streamID + self.streamCreatedOp = streamCreatedOp + self.createdAt = createdAt + self.name = name + self.options = options + } +} + +/// A removed stream. ``requestedOp`` is the op of a deferred delete's request. +public struct MetadataStreamDeleted: Sendable, Hashable { + public var streamID: UInt32 + public var streamCreatedOp: UInt64 + public var createdAt: IggyTimestamp + public var name: String + public var requestedOp: UInt64? + + public init(streamID: UInt32, streamCreatedOp: UInt64, createdAt: IggyTimestamp, name: String, requestedOp: UInt64?) { + self.streamID = streamID + self.streamCreatedOp = streamCreatedOp + self.createdAt = createdAt + self.name = name + self.requestedOp = requestedOp + } +} + +/// A topic after a create or update. +public struct MetadataTopicChange: Sendable, Hashable { + public var streamID: UInt32 + public var streamCreatedOp: UInt64 + public var topicID: UInt32 + public var topicCreatedOp: UInt64 + public var createdAt: IggyTimestamp + public var name: String + public var partitionsCount: UInt32 + public var options: MetadataOptions + + public init( + streamID: UInt32, streamCreatedOp: UInt64, topicID: UInt32, topicCreatedOp: UInt64, createdAt: IggyTimestamp, name: String, + partitionsCount: UInt32, options: MetadataOptions + ) { + self.streamID = streamID + self.streamCreatedOp = streamCreatedOp + self.topicID = topicID + self.topicCreatedOp = topicCreatedOp + self.createdAt = createdAt + self.name = name + self.partitionsCount = partitionsCount + self.options = options + } +} + +/// A current state topics item. Partition counts travel as partitions items. +public struct MetadataTopicItem: Sendable, Hashable { + public var streamID: UInt32 + public var streamCreatedOp: UInt64 + public var topicID: UInt32 + public var topicCreatedOp: UInt64 + public var createdAt: IggyTimestamp + public var name: String + public var options: MetadataOptions + + public init( + streamID: UInt32, streamCreatedOp: UInt64, topicID: UInt32, topicCreatedOp: UInt64, createdAt: IggyTimestamp, name: String, + options: MetadataOptions + ) { + self.streamID = streamID + self.streamCreatedOp = streamCreatedOp + self.topicID = topicID + self.topicCreatedOp = topicCreatedOp + self.createdAt = createdAt + self.name = name + self.options = options + } +} + +/// A removed topic. ``requestedOp`` is the op of a deferred delete's request. +public struct MetadataTopicDeleted: Sendable, Hashable { + public var streamID: UInt32 + public var streamCreatedOp: UInt64 + public var topicID: UInt32 + public var topicCreatedOp: UInt64 + public var createdAt: IggyTimestamp + public var name: String + public var requestedOp: UInt64? + + public init( + streamID: UInt32, streamCreatedOp: UInt64, topicID: UInt32, topicCreatedOp: UInt64, createdAt: IggyTimestamp, name: String, + requestedOp: UInt64? + ) { + self.streamID = streamID + self.streamCreatedOp = streamCreatedOp + self.topicID = topicID + self.topicCreatedOp = topicCreatedOp + self.createdAt = createdAt + self.name = name + self.requestedOp = requestedOp + } +} + +/// A partition count change. ``removedPartitionIDs`` lists the deleted ids in +/// ascending order and is empty on create. +public struct MetadataPartitionsChanged: Sendable, Hashable { + public var streamID: UInt32 + public var streamCreatedOp: UInt64 + public var topicID: UInt32 + public var topicCreatedOp: UInt64 + public var partitionsCount: UInt32 + public var removedPartitionIDs: [UInt32] + public var requestedOp: UInt64? + + public init( + streamID: UInt32, streamCreatedOp: UInt64, topicID: UInt32, topicCreatedOp: UInt64, partitionsCount: UInt32, removedPartitionIDs: [UInt32], + requestedOp: UInt64? + ) { + self.streamID = streamID + self.streamCreatedOp = streamCreatedOp + self.topicID = topicID + self.topicCreatedOp = topicCreatedOp + self.partitionsCount = partitionsCount + self.removedPartitionIDs = removedPartitionIDs + self.requestedOp = requestedOp + } +} + +/// A current state partitions item. +public struct MetadataPartitionsItem: Sendable, Hashable { + public var streamID: UInt32 + public var streamCreatedOp: UInt64 + public var topicID: UInt32 + public var topicCreatedOp: UInt64 + public var partitionsCount: UInt32 + + public init(streamID: UInt32, streamCreatedOp: UInt64, topicID: UInt32, topicCreatedOp: UInt64, partitionsCount: UInt32) { + self.streamID = streamID + self.streamCreatedOp = streamCreatedOp + self.topicID = topicID + self.topicCreatedOp = topicCreatedOp + self.partitionsCount = partitionsCount + } +} + +/// A created consumer group with the topic's resulting group count. +public struct MetadataConsumerGroupCreated: Sendable, Hashable { + public var streamID: UInt32 + public var streamCreatedOp: UInt64 + public var topicID: UInt32 + public var topicCreatedOp: UInt64 + public var groupID: UInt32 + public var name: String + public var consumerGroupsCount: UInt32 + + public init( + streamID: UInt32, streamCreatedOp: UInt64, topicID: UInt32, topicCreatedOp: UInt64, groupID: UInt32, name: String, consumerGroupsCount: UInt32 + ) { + self.streamID = streamID + self.streamCreatedOp = streamCreatedOp + self.topicID = topicID + self.topicCreatedOp = topicCreatedOp + self.groupID = groupID + self.name = name + self.consumerGroupsCount = consumerGroupsCount + } +} + +/// A removed consumer group with the topic's resulting group count. +public struct MetadataConsumerGroupDeleted: Sendable, Hashable { + public var streamID: UInt32 + public var streamCreatedOp: UInt64 + public var topicID: UInt32 + public var topicCreatedOp: UInt64 + public var groupID: UInt32 + public var name: String + public var consumerGroupsCount: UInt32 + public var requestedOp: UInt64? + + public init( + streamID: UInt32, streamCreatedOp: UInt64, topicID: UInt32, topicCreatedOp: UInt64, groupID: UInt32, name: String, consumerGroupsCount: UInt32, + requestedOp: UInt64? + ) { + self.streamID = streamID + self.streamCreatedOp = streamCreatedOp + self.topicID = topicID + self.topicCreatedOp = topicCreatedOp + self.groupID = groupID + self.name = name + self.consumerGroupsCount = consumerGroupsCount + self.requestedOp = requestedOp + } +} + +/// A current state consumer groups item. +public struct MetadataConsumerGroupItem: Sendable, Hashable { + public var streamID: UInt32 + public var streamCreatedOp: UInt64 + public var topicID: UInt32 + public var topicCreatedOp: UInt64 + public var groupID: UInt32 + public var name: String + + public init(streamID: UInt32, streamCreatedOp: UInt64, topicID: UInt32, topicCreatedOp: UInt64, groupID: UInt32, name: String) { + self.streamID = streamID + self.streamCreatedOp = streamCreatedOp + self.topicID = topicID + self.topicCreatedOp = topicCreatedOp + self.groupID = groupID + self.name = name + } +} + +/// A member that joined, with its installed partitions, normally none since +/// a handover installs them. +public struct MetadataMemberJoined: Sendable, Hashable { + public var streamID: UInt32 + public var streamCreatedOp: UInt64 + public var topicID: UInt32 + public var topicCreatedOp: UInt64 + public var groupID: UInt32 + public var memberID: UInt32 + public var clientID: UInt128Value + public var membersCount: UInt32 + public var partitionIDs: [UInt32] + + public init( + streamID: UInt32, streamCreatedOp: UInt64, topicID: UInt32, topicCreatedOp: UInt64, groupID: UInt32, memberID: UInt32, clientID: UInt128Value, + membersCount: UInt32, partitionIDs: [UInt32] + ) { + self.streamID = streamID + self.streamCreatedOp = streamCreatedOp + self.topicID = topicID + self.topicCreatedOp = topicCreatedOp + self.groupID = groupID + self.memberID = memberID + self.clientID = clientID + self.membersCount = membersCount + self.partitionIDs = partitionIDs + } +} + +/// A member that left a consumer group. +public struct MetadataMemberLeft: Sendable, Hashable { + public var streamID: UInt32 + public var streamCreatedOp: UInt64 + public var topicID: UInt32 + public var topicCreatedOp: UInt64 + public var groupID: UInt32 + public var memberID: UInt32 + public var clientID: UInt128Value + public var membersCount: UInt32 + public var reason: MetadataMemberLeftReason + + public init( + streamID: UInt32, streamCreatedOp: UInt64, topicID: UInt32, topicCreatedOp: UInt64, groupID: UInt32, memberID: UInt32, clientID: UInt128Value, + membersCount: UInt32, reason: MetadataMemberLeftReason + ) { + self.streamID = streamID + self.streamCreatedOp = streamCreatedOp + self.topicID = topicID + self.topicCreatedOp = topicCreatedOp + self.groupID = groupID + self.memberID = memberID + self.clientID = clientID + self.membersCount = membersCount + self.reason = reason + } +} + +/// A completed partition handover. ``memberID`` is nil when the last member left. +public struct MetadataPartitionHandedOver: Sendable, Hashable { + public var streamID: UInt32 + public var streamCreatedOp: UInt64 + public var topicID: UInt32 + public var topicCreatedOp: UInt64 + public var groupID: UInt32 + public var partitionID: UInt32 + public var memberID: UInt32? + + public init( + streamID: UInt32, streamCreatedOp: UInt64, topicID: UInt32, topicCreatedOp: UInt64, groupID: UInt32, partitionID: UInt32, memberID: UInt32? + ) { + self.streamID = streamID + self.streamCreatedOp = streamCreatedOp + self.topicID = topicID + self.topicCreatedOp = topicCreatedOp + self.groupID = groupID + self.partitionID = partitionID + self.memberID = memberID + } +} + +/// A current state consumer group members item. +public struct MetadataMemberItem: Sendable, Hashable { + public var streamID: UInt32 + public var streamCreatedOp: UInt64 + public var topicID: UInt32 + public var topicCreatedOp: UInt64 + public var groupID: UInt32 + public var memberID: UInt32 + public var clientID: UInt128Value + public var partitionIDs: [UInt32] + + public init( + streamID: UInt32, streamCreatedOp: UInt64, topicID: UInt32, topicCreatedOp: UInt64, groupID: UInt32, memberID: UInt32, clientID: UInt128Value, + partitionIDs: [UInt32] + ) { + self.streamID = streamID + self.streamCreatedOp = streamCreatedOp + self.topicID = topicID + self.topicCreatedOp = topicCreatedOp + self.groupID = groupID + self.memberID = memberID + self.clientID = clientID + self.partitionIDs = partitionIDs + } +} + +/// A user after a create or update, and a current state users item. +public struct MetadataUserChange: Sendable, Hashable { + public var userID: UInt32 + /// The user's creation time. A recreated user with a reused id has another. + public var userCreatedAt: IggyTimestamp + public var username: String + public var status: UserStatus + /// The complete public permission set, nil when the user has none. + public var permissions: Permissions? + public var options: MetadataOptions + + public init(userID: UInt32, userCreatedAt: IggyTimestamp, username: String, status: UserStatus, permissions: Permissions?, options: MetadataOptions) { + self.userID = userID + self.userCreatedAt = userCreatedAt + self.username = username + self.status = status + self.permissions = permissions + self.options = options + } +} + +/// A deleted user, or one whose password changed. +public struct MetadataUserIdentity: Sendable, Hashable { + public var userID: UInt32 + public var userCreatedAt: IggyTimestamp + public var username: String + + public init(userID: UInt32, userCreatedAt: IggyTimestamp, username: String) { + self.userID = userID + self.userCreatedAt = userCreatedAt + self.username = username + } +} + +/// A user's public permission set after a change. +public struct MetadataUserPermissionsUpdated: Sendable, Hashable { + public var userID: UInt32 + public var userCreatedAt: IggyTimestamp + public var username: String + public var permissions: Permissions? + public var reason: MetadataPermissionsUpdatedReason + + public init( + userID: UInt32, userCreatedAt: IggyTimestamp, username: String, permissions: Permissions?, reason: MetadataPermissionsUpdatedReason + ) { + self.userID = userID + self.userCreatedAt = userCreatedAt + self.username = username + self.permissions = permissions + self.reason = reason + } +} + +/// A created personal access token, and a current state personal access +/// tokens item. ``expiry`` is nil for a token that never expires. +public struct MetadataTokenChange: Sendable, Hashable { + public var userID: UInt32 + public var userCreatedAt: IggyTimestamp + public var name: String + public var expiry: IggyTimestamp? + + public init(userID: UInt32, userCreatedAt: IggyTimestamp, name: String, expiry: IggyTimestamp?) { + self.userID = userID + self.userCreatedAt = userCreatedAt + self.name = name + self.expiry = expiry + } +} + +/// A removed personal access token. +public struct MetadataTokenDeleted: Sendable, Hashable { + public var userID: UInt32 + public var userCreatedAt: IggyTimestamp + public var name: String + public var reason: MetadataTokenDeletedReason + + public init(userID: UInt32, userCreatedAt: IggyTimestamp, name: String, reason: MetadataTokenDeletedReason) { + self.userID = userID + self.userCreatedAt = userCreatedAt + self.name = name + self.reason = reason + } +} + +/// A registered session, and a current state sessions item. +public struct MetadataSession: Sendable, Hashable { + public var userID: UInt32 + public var userCreatedAt: IggyTimestamp + public var clientID: UInt128Value + public var session: UInt64 + + public init(userID: UInt32, userCreatedAt: IggyTimestamp, clientID: UInt128Value, session: UInt64) { + self.userID = userID + self.userCreatedAt = userCreatedAt + self.clientID = clientID + self.session = session + } +} + +/// A closed session. +public struct MetadataSessionClosed: Sendable, Hashable { + public var userID: UInt32 + public var userCreatedAt: IggyTimestamp + public var clientID: UInt128Value + public var session: UInt64 + public var reason: MetadataSessionClosedReason + + public init(userID: UInt32, userCreatedAt: IggyTimestamp, clientID: UInt128Value, session: UInt64, reason: MetadataSessionClosedReason) { + self.userID = userID + self.userCreatedAt = userCreatedAt + self.clientID = clientID + self.session = session + self.reason = reason + } +} + +/// The typed body of a change record. +public enum MetadataChange: Sendable, Hashable { + case streamCreated(MetadataStreamChange) + case streamUpdated(MetadataStreamChange) + case streamDeleted(MetadataStreamDeleted) + case topicCreated(MetadataTopicChange) + case topicUpdated(MetadataTopicChange) + case topicDeleted(MetadataTopicDeleted) + case partitionsChanged(MetadataPartitionsChanged) + case consumerGroupCreated(MetadataConsumerGroupCreated) + case consumerGroupDeleted(MetadataConsumerGroupDeleted) + case consumerGroupMemberJoined(MetadataMemberJoined) + case consumerGroupMemberLeft(MetadataMemberLeft) + case consumerGroupPartitionHandedOver(MetadataPartitionHandedOver) + case userCreated(MetadataUserChange) + case userUpdated(MetadataUserChange) + case userDeleted(MetadataUserIdentity) + case userPasswordChanged(MetadataUserIdentity) + case userPermissionsUpdated(MetadataUserPermissionsUpdated) + case personalAccessTokenCreated(MetadataTokenChange) + case personalAccessTokenDeleted(MetadataTokenDeleted) + case sessionRegistered(MetadataSession) + case sessionClosed(MetadataSessionClosed) + /// A record of a group or kind this build does not know, kept with its + /// wire codes and raw body. A mirror stops before advancing its durable + /// cursor past it. + case unknown(group: UInt8, kind: UInt8, body: [UInt8]) + + /// The kind, nil for a record this build does not know. + public var kind: MetadataChangeKind? { + switch self { + case .streamCreated: .streamCreated + case .streamUpdated: .streamUpdated + case .streamDeleted: .streamDeleted + case .topicCreated: .topicCreated + case .topicUpdated: .topicUpdated + case .topicDeleted: .topicDeleted + case .partitionsChanged: .partitionsChanged + case .consumerGroupCreated: .consumerGroupCreated + case .consumerGroupDeleted: .consumerGroupDeleted + case .consumerGroupMemberJoined: .consumerGroupMemberJoined + case .consumerGroupMemberLeft: .consumerGroupMemberLeft + case .consumerGroupPartitionHandedOver: .consumerGroupPartitionHandedOver + case .userCreated: .userCreated + case .userUpdated: .userUpdated + case .userDeleted: .userDeleted + case .userPasswordChanged: .userPasswordChanged + case .userPermissionsUpdated: .userPermissionsUpdated + case .personalAccessTokenCreated: .personalAccessTokenCreated + case .personalAccessTokenDeleted: .personalAccessTokenDeleted + case .sessionRegistered: .sessionRegistered + case .sessionClosed: .sessionClosed + case .unknown: nil + } + } + + /// The group, nil for a record this build does not know. + public var group: MetadataChangeGroup? { kind?.group } +} + +/// One current state item: a resource or session as it is now. +public enum MetadataCurrentItem: Sendable, Hashable { + case stream(MetadataStreamChange) + case topic(MetadataTopicItem) + case partitions(MetadataPartitionsItem) + case consumerGroup(MetadataConsumerGroupItem) + case consumerGroupMember(MetadataMemberItem) + case user(MetadataUserChange) + case personalAccessToken(MetadataTokenChange) + case session(MetadataSession) + /// An item of a group this build does not know, kept with its wire code + /// and raw body. + case unknown(group: UInt8, body: [UInt8]) + + /// The group, nil for an item this build does not know. + public var group: MetadataChangeGroup? { + switch self { + case .stream: .streams + case .topic: .topics + case .partitions: .partitions + case .consumerGroup: .consumerGroups + case .consumerGroupMember: .consumerGroupMembers + case .user: .users + case .personalAccessToken: .personalAccessTokens + case .session: .sessions + case .unknown: nil + } + } +} + +/// One change record: the committed operation that produced it, its position +/// within that operation, and what is true after it. +public struct MetadataChangeRecord: Sendable, Hashable { + public var op: UInt64 + public var ordinal: UInt32 + /// Prepare checksum of ``op``. + public var checksum: UInt128Value + /// Prepare timestamp of ``op``. + public var timestamp: IggyTimestamp + /// The acting user, nil for server actions. + public var actorUserID: UInt32? + public var change: MetadataChange + + public init(op: UInt64, ordinal: UInt32, checksum: UInt128Value, timestamp: IggyTimestamp, actorUserID: UInt32?, change: MetadataChange) { + self.op = op + self.ordinal = ordinal + self.checksum = checksum + self.timestamp = timestamp + self.actorUserID = actorUserID + self.change = change + } +} + +/// One page of the metadata change log, or of the current state. +/// +/// On ``MetadataChangeStatus/ok`` keep ``next`` and send it back. On +/// ``MetadataChangeStatus/behind`` retry later, on +/// ``MetadataChangeStatus/expired`` or ``MetadataChangeStatus/reset`` rebuild +/// from the current state. +public struct MetadataChanges: Sendable, Hashable { + public var status: MetadataChangeStatus + public var phase: MetadataChangePhase + /// The node's log id, nil before it applied op 1. + public var logID: UInt128Value? + /// Lowest whole operation from which the requested groups are complete. + public var oldestOp: UInt64 + /// The requested groups are complete for every timestamp above this one, + /// zero when nothing was evicted. + public var retainedSince: IggyTimestamp + /// Highest metadata operation the node applied. + public var headOp: UInt64 + /// The cursor to send back: present on ok, echoed on behind, nil on + /// expired and reset. The last current state page carries the anchor to + /// follow the change log from. + public var next: MetadataChangeCursor? + /// More records remain after ``next``, or another state page exists. + public var hasMore: Bool + public var records: [MetadataChangeRecord] + public var items: [MetadataCurrentItem] + /// Continuation of a current state scan, nil once it is complete. + public var currentPageToken: MetadataCurrentPageToken? + + public init( + status: MetadataChangeStatus, phase: MetadataChangePhase, logID: UInt128Value?, oldestOp: UInt64, retainedSince: IggyTimestamp, headOp: UInt64, + next: MetadataChangeCursor?, hasMore: Bool, records: [MetadataChangeRecord], items: [MetadataCurrentItem], + currentPageToken: MetadataCurrentPageToken? + ) { + self.status = status + self.phase = phase + self.logID = logID + self.oldestOp = oldestOp + self.retainedSince = retainedSince + self.headOp = headOp + self.next = next + self.hasMore = hasMore + self.records = records + self.items = items + self.currentPageToken = currentPageToken + } +} + +// MARK: - Wire codecs + +extension MetadataChange { + /// Decodes a record body by its codes, mirroring `WireMetadataChange::decode`. + /// A group or kind this build does not know is kept raw. A known kind + /// under another group, a malformed body, an unknown reason, and trailing + /// bytes are refused. + static func decode(group: UInt8, kind: UInt8, body: ArraySlice<UInt8>) throws -> MetadataChange { + guard let knownGroup = MetadataChangeGroup(rawValue: group), let knownKind = MetadataChangeKind(rawValue: kind) else { + return .unknown(group: group, kind: kind, body: Array(body)) + } + guard knownKind.group == knownGroup else { + throw WireError.validation("change kind \(kind) does not belong to group \(group)") + } + var reader = ByteReader(body) + let change: MetadataChange + switch knownKind { + case .streamCreated: change = .streamCreated(try .decode(from: &reader)) + case .streamUpdated: change = .streamUpdated(try .decode(from: &reader)) + case .streamDeleted: change = .streamDeleted(try .decode(from: &reader)) + case .topicCreated: change = .topicCreated(try .decode(from: &reader)) + case .topicUpdated: change = .topicUpdated(try .decode(from: &reader)) + case .topicDeleted: change = .topicDeleted(try .decode(from: &reader)) + case .partitionsChanged: change = .partitionsChanged(try .decode(from: &reader)) + case .consumerGroupCreated: change = .consumerGroupCreated(try .decode(from: &reader)) + case .consumerGroupDeleted: change = .consumerGroupDeleted(try .decode(from: &reader)) + case .consumerGroupMemberJoined: change = .consumerGroupMemberJoined(try .decode(from: &reader)) + case .consumerGroupMemberLeft: change = .consumerGroupMemberLeft(try .decode(from: &reader)) + case .consumerGroupPartitionHandedOver: change = .consumerGroupPartitionHandedOver(try .decode(from: &reader)) + case .userCreated: change = .userCreated(try .decode(from: &reader)) + case .userUpdated: change = .userUpdated(try .decode(from: &reader)) + case .userDeleted: change = .userDeleted(try .decode(from: &reader)) + case .userPasswordChanged: change = .userPasswordChanged(try .decode(from: &reader)) + case .userPermissionsUpdated: change = .userPermissionsUpdated(try .decode(from: &reader)) + case .personalAccessTokenCreated: change = .personalAccessTokenCreated(try .decode(from: &reader)) + case .personalAccessTokenDeleted: change = .personalAccessTokenDeleted(try .decode(from: &reader)) + case .sessionRegistered: change = .sessionRegistered(try .decode(from: &reader)) + case .sessionClosed: change = .sessionClosed(try .decode(from: &reader)) + } + try reader.requireBodyEnd() + return change + } +} + +extension MetadataCurrentItem { + /// Decodes an item body by its group code. A group this build does not + /// know is kept raw. + static func decode(group: UInt8, body: ArraySlice<UInt8>) throws -> MetadataCurrentItem { + guard let knownGroup = MetadataChangeGroup(rawValue: group) else { + return .unknown(group: group, body: Array(body)) + } + var reader = ByteReader(body) + let item: MetadataCurrentItem + switch knownGroup { + case .streams: item = .stream(try .decode(from: &reader)) + case .topics: item = .topic(try .decode(from: &reader)) + case .partitions: item = .partitions(try .decode(from: &reader)) + case .consumerGroups: item = .consumerGroup(try .decode(from: &reader)) + case .consumerGroupMembers: item = .consumerGroupMember(try .decode(from: &reader)) + case .users: item = .user(try .decode(from: &reader)) + case .personalAccessTokens: item = .personalAccessToken(try .decode(from: &reader)) + case .sessions: item = .session(try .decode(from: &reader)) + } + try reader.requireBodyEnd() + return item + } +} + +extension MetadataStreamChange { + /// `[stream_id:u32][stream_created_op:u64][created_at:u64][name][options]` + static func decode(from reader: inout ByteReader) throws -> MetadataStreamChange { + MetadataStreamChange( + streamID: try reader.readUInt32(), streamCreatedOp: try reader.readUInt64(), createdAt: try reader.readTimestamp(), name: try reader.readName(), + options: try reader.readChangeOptions()) + } +} + +extension MetadataStreamDeleted { + /// `[stream_id:u32][stream_created_op:u64][created_at:u64][name][has:u8][requested_op:u64]` + static func decode(from reader: inout ByteReader) throws -> MetadataStreamDeleted { + MetadataStreamDeleted( + streamID: try reader.readUInt32(), streamCreatedOp: try reader.readUInt64(), createdAt: try reader.readTimestamp(), name: try reader.readName(), + requestedOp: try reader.readOptionalUInt64()) + } +} + +extension MetadataTopicChange { + /// `[topic key][created_at:u64][name][partitions_count:u32][options]` + static func decode(from reader: inout ByteReader) throws -> MetadataTopicChange { + MetadataTopicChange( + streamID: try reader.readUInt32(), streamCreatedOp: try reader.readUInt64(), topicID: try reader.readUInt32(), + topicCreatedOp: try reader.readUInt64(), createdAt: try reader.readTimestamp(), name: try reader.readName(), + partitionsCount: try reader.readUInt32(), options: try reader.readChangeOptions()) + } +} + +extension MetadataTopicItem { + /// `[topic key][created_at:u64][name][options]` + static func decode(from reader: inout ByteReader) throws -> MetadataTopicItem { + MetadataTopicItem( + streamID: try reader.readUInt32(), streamCreatedOp: try reader.readUInt64(), topicID: try reader.readUInt32(), + topicCreatedOp: try reader.readUInt64(), createdAt: try reader.readTimestamp(), name: try reader.readName(), options: try reader.readChangeOptions() + ) + } +} + +extension MetadataTopicDeleted { + /// `[topic key][created_at:u64][name][has:u8][requested_op:u64]` + static func decode(from reader: inout ByteReader) throws -> MetadataTopicDeleted { + MetadataTopicDeleted( + streamID: try reader.readUInt32(), streamCreatedOp: try reader.readUInt64(), topicID: try reader.readUInt32(), + topicCreatedOp: try reader.readUInt64(), createdAt: try reader.readTimestamp(), name: try reader.readName(), + requestedOp: try reader.readOptionalUInt64()) + } +} + +extension MetadataPartitionsChanged { + /// `[topic key][partitions_count:u32][count:u32][id:u32]*[has:u8][requested_op:u64]` + static func decode(from reader: inout ByteReader) throws -> MetadataPartitionsChanged { + MetadataPartitionsChanged( + streamID: try reader.readUInt32(), streamCreatedOp: try reader.readUInt64(), topicID: try reader.readUInt32(), + topicCreatedOp: try reader.readUInt64(), partitionsCount: try reader.readUInt32(), removedPartitionIDs: try reader.readPartitionIDs(), + requestedOp: try reader.readOptionalUInt64()) + } +} + +extension MetadataPartitionsItem { + /// `[topic key][partitions_count:u32]` + static func decode(from reader: inout ByteReader) throws -> MetadataPartitionsItem { + MetadataPartitionsItem( + streamID: try reader.readUInt32(), streamCreatedOp: try reader.readUInt64(), topicID: try reader.readUInt32(), + topicCreatedOp: try reader.readUInt64(), partitionsCount: try reader.readUInt32()) + } +} + +extension MetadataConsumerGroupCreated { + /// `[topic key][group_id:u32][name][consumer_groups_count:u32]` + static func decode(from reader: inout ByteReader) throws -> MetadataConsumerGroupCreated { + MetadataConsumerGroupCreated( + streamID: try reader.readUInt32(), streamCreatedOp: try reader.readUInt64(), topicID: try reader.readUInt32(), + topicCreatedOp: try reader.readUInt64(), groupID: try reader.readUInt32(), name: try reader.readName(), consumerGroupsCount: try reader.readUInt32() + ) + } +} + +extension MetadataConsumerGroupDeleted { + /// `[topic key][group_id:u32][name][consumer_groups_count:u32][has:u8][requested_op:u64]` + static func decode(from reader: inout ByteReader) throws -> MetadataConsumerGroupDeleted { + MetadataConsumerGroupDeleted( + streamID: try reader.readUInt32(), streamCreatedOp: try reader.readUInt64(), topicID: try reader.readUInt32(), + topicCreatedOp: try reader.readUInt64(), groupID: try reader.readUInt32(), name: try reader.readName(), + consumerGroupsCount: try reader.readUInt32(), + requestedOp: try reader.readOptionalUInt64()) + } +} + +extension MetadataConsumerGroupItem { + /// `[topic key][group_id:u32][name]` + static func decode(from reader: inout ByteReader) throws -> MetadataConsumerGroupItem { + MetadataConsumerGroupItem( + streamID: try reader.readUInt32(), streamCreatedOp: try reader.readUInt64(), topicID: try reader.readUInt32(), + topicCreatedOp: try reader.readUInt64(), groupID: try reader.readUInt32(), name: try reader.readName()) + } +} + +extension MetadataMemberJoined { + /// `[topic key][group_id:u32][member_id:u32][client_id:u128][members_count:u32][ids]` + static func decode(from reader: inout ByteReader) throws -> MetadataMemberJoined { + MetadataMemberJoined( + streamID: try reader.readUInt32(), streamCreatedOp: try reader.readUInt64(), topicID: try reader.readUInt32(), + topicCreatedOp: try reader.readUInt64(), groupID: try reader.readUInt32(), memberID: try reader.readUInt32(), clientID: try reader.readUInt128(), + membersCount: try reader.readUInt32(), partitionIDs: try reader.readPartitionIDs()) + } +} + +extension MetadataMemberLeft { + /// `[topic key][group_id:u32][member_id:u32][client_id:u128][members_count:u32][reason:u8]` + static func decode(from reader: inout ByteReader) throws -> MetadataMemberLeft { + MetadataMemberLeft( + streamID: try reader.readUInt32(), streamCreatedOp: try reader.readUInt64(), topicID: try reader.readUInt32(), + topicCreatedOp: try reader.readUInt64(), groupID: try reader.readUInt32(), memberID: try reader.readUInt32(), clientID: try reader.readUInt128(), + membersCount: try reader.readUInt32(), reason: try reader.readReason(MetadataMemberLeftReason.self)) + } +} + +extension MetadataPartitionHandedOver { + /// `[topic key][group_id:u32][partition_id:u32][has:u8][member_id:u32]` + static func decode(from reader: inout ByteReader) throws -> MetadataPartitionHandedOver { + MetadataPartitionHandedOver( + streamID: try reader.readUInt32(), streamCreatedOp: try reader.readUInt64(), topicID: try reader.readUInt32(), + topicCreatedOp: try reader.readUInt64(), groupID: try reader.readUInt32(), partitionID: try reader.readUInt32(), + memberID: try reader.readOptionalUInt32()) + } +} + +extension MetadataMemberItem { + /// `[topic key][group_id:u32][member_id:u32][client_id:u128][ids]` + static func decode(from reader: inout ByteReader) throws -> MetadataMemberItem { + MetadataMemberItem( + streamID: try reader.readUInt32(), streamCreatedOp: try reader.readUInt64(), topicID: try reader.readUInt32(), + topicCreatedOp: try reader.readUInt64(), groupID: try reader.readUInt32(), memberID: try reader.readUInt32(), clientID: try reader.readUInt128(), + partitionIDs: try reader.readPartitionIDs()) + } +} + +extension MetadataUserChange { + /// `[user_id:u32][user_created_at:u64][username][status:u8][has:u8][permissions][options]` + static func decode(from reader: inout ByteReader) throws -> MetadataUserChange { + let userID = try reader.readUInt32() + let userCreatedAt = try reader.readTimestamp() + let username = try reader.readName() + let statusCode = try reader.readUInt8() + guard let status = UserStatus(rawValue: statusCode) else { + throw IggyError(.invalidUserStatus, context: "unknown user status \(statusCode)") + } + return MetadataUserChange( + userID: userID, userCreatedAt: userCreatedAt, username: username, status: status, permissions: try reader.readOptionalPermissions(), + options: try reader.readChangeOptions()) + } +} + +extension MetadataUserIdentity { + /// `[user_id:u32][user_created_at:u64][username]` + static func decode(from reader: inout ByteReader) throws -> MetadataUserIdentity { + MetadataUserIdentity(userID: try reader.readUInt32(), userCreatedAt: try reader.readTimestamp(), username: try reader.readName()) + } +} + +extension MetadataUserPermissionsUpdated { + /// `[user_id:u32][user_created_at:u64][username][has:u8][permissions][reason:u8]` + static func decode(from reader: inout ByteReader) throws -> MetadataUserPermissionsUpdated { + MetadataUserPermissionsUpdated( + userID: try reader.readUInt32(), userCreatedAt: try reader.readTimestamp(), username: try reader.readName(), + permissions: try reader.readOptionalPermissions(), reason: try reader.readReason(MetadataPermissionsUpdatedReason.self)) + } +} + +extension MetadataTokenChange { + /// `[user_id:u32][user_created_at:u64][name][expiry:u64]`, expiry 0 for a token that never expires. + static func decode(from reader: inout ByteReader) throws -> MetadataTokenChange { + let userID = try reader.readUInt32() + let userCreatedAt = try reader.readTimestamp() + let name = try reader.readName() + let expiry = try reader.readUInt64() + return MetadataTokenChange(userID: userID, userCreatedAt: userCreatedAt, name: name, expiry: expiry == 0 ? nil : IggyTimestamp(microseconds: expiry)) + } +} + +extension MetadataTokenDeleted { + /// `[user_id:u32][user_created_at:u64][name][reason:u8]` + static func decode(from reader: inout ByteReader) throws -> MetadataTokenDeleted { + MetadataTokenDeleted( + userID: try reader.readUInt32(), userCreatedAt: try reader.readTimestamp(), name: try reader.readName(), + reason: try reader.readReason(MetadataTokenDeletedReason.self)) + } +} + +extension MetadataSession { + /// `[user_id:u32][user_created_at:u64][client_id:u128][session:u64]` + static func decode(from reader: inout ByteReader) throws -> MetadataSession { + MetadataSession( + userID: try reader.readUInt32(), userCreatedAt: try reader.readTimestamp(), clientID: try reader.readUInt128(), session: try reader.readUInt64()) + } +} + +extension MetadataSessionClosed { + /// `[user_id:u32][user_created_at:u64][client_id:u128][session:u64][reason:u8]` + static func decode(from reader: inout ByteReader) throws -> MetadataSessionClosed { + MetadataSessionClosed( + userID: try reader.readUInt32(), userCreatedAt: try reader.readTimestamp(), clientID: try reader.readUInt128(), session: try reader.readUInt64(), + reason: try reader.readReason(MetadataSessionClosedReason.self)) + } +} + +extension ByteReader { + /// A presence byte, refusing anything but 0 and 1 as the Rust codec does. + mutating func readFlag() throws -> Bool { + let offset = position + switch try readUInt8() { + case 0: return false + case 1: return true + case let value: throw WireError.validation("presence byte \(value) at offset \(offset) is not 0 or 1") + } + } + + mutating func readOptionalUInt64() throws -> UInt64? { + guard try readFlag() else { + return nil + } + return try readUInt64() + } + + mutating func readOptionalUInt32() throws -> UInt32? { + guard try readFlag() else { + return nil + } + return try readUInt32() + } + + mutating func readTimestamp() throws -> IggyTimestamp { + IggyTimestamp(microseconds: try readUInt64()) + } + + /// `[count:u32][id:u32]*`, refusing ids that are not strictly ascending. + mutating func readPartitionIDs() throws -> [UInt32] { + let count = try readCount() + var ids: [UInt32] = [] + ids.reserveCapacity(min(count, remaining / 4)) + for _ in 0..<count { + let id = try readUInt32() + if let previous = ids.last, previous >= id { + throw WireError.validation("partition ids must be strictly ascending") + } + ids.append(id) + } + return ids + } + + /// `[count:u32]`, refusing a count a 32-bit `Int` cannot hold instead of trapping. + mutating func readCount() throws -> Int { + let offset = position + let declared = try readUInt32() + guard let count = Int(exactly: declared) else { + throw WireError.validation("count \(declared) at offset \(offset) does not fit the platform Int") + } + return count + } + + mutating func readReason<Reason: RawRepresentable>(_: Reason.Type) throws -> Reason where Reason.RawValue == UInt8 { + let offset = position + let code = try readUInt8() + guard let reason = Reason(rawValue: code) else { + throw WireError.validation("unknown \(Reason.self) code \(code) at offset \(offset)") + } + return reason + } + + mutating func readOptionalPermissions() throws -> Permissions? { + guard try readFlag() else { + return nil + } + return try Permissions.decode(from: &self) + } + + /// `[state:u8]`, then for state 0 the explicit and derived blocks as the + /// read responses carry them, nothing for state 1. + mutating func readChangeOptions() throws -> MetadataOptions { + let offset = position + switch try readUInt8() { + case 0: + let explicit = try readPrefixedBytes() + let derived = try readPrefixedBytes() + return .readable(try OptionsBlock.decodeSplit(explicit: explicit, derived: derived)) + case 1: + return .unreadable + case let state: + throw WireError.validation("options state \(state) at offset \(offset) is not 0 or 1") + } + } + + /// Refuses bytes left after a change body. + func requireBodyEnd() throws { + guard isAtEnd else { + throw WireError.validation("\(remaining) trailing bytes after a change body") + } + } +} + +/// Unpadded Base64url on top of Foundation. Decoding accepts only the text +/// ``encode(_:)`` writes back, so padding, other alphabets and nonzero +/// trailing bits are refused. +enum Base64URL { + static func encode(_ bytes: [UInt8]) -> String { + Data(bytes).base64EncodedString() + .replacingOccurrences(of: "+", with: "-") + .replacingOccurrences(of: "/", with: "_") + .replacingOccurrences(of: "=", with: "") + } + + /// The bytes after `prefix`, or nil for text that is not canonical. + static func decode(_ text: String, prefix: String) -> [UInt8]? { + guard text.utf8.starts(with: prefix.utf8) else { + return nil + } + let encoded = String(decoding: text.utf8.dropFirst(prefix.utf8.count), as: UTF8.self) + let standard = encoded.replacingOccurrences(of: "-", with: "+").replacingOccurrences(of: "_", with: "/") + let padding = String(repeating: "=", count: (4 - standard.utf8.count % 4) % 4) + guard let data = Data(base64Encoded: standard + padding), encode([UInt8](data)) == encoded else { + return nil + } + return [UInt8](data) + } +}
diff --git a/foreign/swift/Sources/Iggy/Protocol/CommandCode.swift b/foreign/swift/Sources/Iggy/Protocol/CommandCode.swift index b292867..ba8e172 100644 --- a/foreign/swift/Sources/Iggy/Protocol/CommandCode.swift +++ b/foreign/swift/Sources/Iggy/Protocol/CommandCode.swift
@@ -26,6 +26,7 @@ // 14 was attach_consumer_session; the server keeps the code retired. case bindSession = 15 case updateResource = 16 + case getMetadataChanges = 17 case getMe = 20 case getClient = 21 case getClients = 22 @@ -84,6 +85,7 @@ case .describeOptions: "options.describe" case .bindSession: "session.bind" case .updateResource: "resource.update" + case .getMetadataChanges: "metadata.changes" case .getMe: "me" case .getClient: "client.get" case .getClients: "client.list"
diff --git a/foreign/swift/Sources/Iggy/Protocol/Requests.swift b/foreign/swift/Sources/Iggy/Protocol/Requests.swift index d734be9..be2494f 100644 --- a/foreign/swift/Sources/Iggy/Protocol/Requests.swift +++ b/foreign/swift/Sources/Iggy/Protocol/Requests.swift
@@ -369,6 +369,42 @@ static func describeOptions(scope: OptionsScope) -> [UInt8] { [scope.rawValue] } + + /// `[start:u8][cursor:44][has_since:u8][since:u64][has_until:u8][until:u64][limit:u32][groups_count:u8][group:u8]*[token_len:u16][token]`. + /// The cursor is all zero unless the read starts after it, and groups + /// travel in the caller's order, at most 255 of them as in the Rust encoder. + static func getMetadataChanges(_ query: MetadataChangesQuery) -> [UInt8] { + var groups = query.groups + if groups.count > Int(UInt8.max) { + var seen = Set<MetadataChangeGroup>() + groups = groups.filter { seen.insert($0).inserted } + } + var token: [UInt8] = [] + if case .current(let pageToken?) = query.start { + token = pageToken.encoded() + } + var writer = ByteWriter(capacity: 1 + MetadataChangeCursor.encodedSize + 9 + 9 + 4 + 1 + groups.count + 2 + token.count) + writer.write(query.start.code) + if case .after(let cursor) = query.start { + cursor.encode(into: &writer) + } else { + writer.writeZeros(MetadataChangeCursor.encodedSize) + } + for bound in [query.since, query.until] { + writer.write(bound != nil) + if let bound { + writer.write(bound.microseconds) + } + } + writer.write(query.limit) + writer.write(UInt8(groups.count)) + for group in groups { + writer.write(group.rawValue) + } + writer.write(UInt16(token.count)) + writer.write(token) + return writer.bytes + } } /// Validates a wire name and returns it, so the length prefix can never
diff --git a/foreign/swift/Sources/Iggy/Protocol/Responses.swift b/foreign/swift/Sources/Iggy/Protocol/Responses.swift index ba710b7..da710c1 100644 --- a/foreign/swift/Sources/Iggy/Protocol/Responses.swift +++ b/foreign/swift/Sources/Iggy/Protocol/Responses.swift
@@ -420,6 +420,89 @@ } } + // MARK: Metadata changes + + /// Decodes a `GetMetadataChanges` reply: + /// `[status:u8][phase:u8][has_log_id:u8][log_id:u128][oldest_op:u64][retained_since:u64][head_op:u64]` + /// `[has_next:u8][cursor:44][has_more:u8][records_count:u32][record]*[items_count:u32][item]*[token_len:u16][token]`. + /// A changes page carries only records, a current page only items and a token. + static func metadataChanges(_ body: ArraySlice<UInt8>) throws -> MetadataChanges { + try decode(body) { reader in + let statusCode = try reader.readUInt8() + guard let status = MetadataChangeStatus(rawValue: statusCode) else { + throw WireError.validation("unknown metadata change status \(statusCode)") + } + let phaseCode = try reader.readUInt8() + guard let phase = MetadataChangePhase(rawValue: phaseCode) else { + throw WireError.validation("unknown metadata change phase \(phaseCode)") + } + var logID: UInt128Value? + if try reader.readFlag() { + logID = try reader.readUInt128() + } + let oldestOp = try reader.readUInt64() + let retainedSince = try reader.readTimestamp() + let headOp = try reader.readUInt64() + var next: MetadataChangeCursor? + if try reader.readFlag() { + next = try MetadataChangeCursor.decode(from: &reader) + } + let hasMore = try reader.readFlag() + let recordsCount = try reader.readCount() + var records: [MetadataChangeRecord] = [] + records.reserveCapacity(min(recordsCount, reader.remaining / metadataChangeRecordMinimumSize)) + for _ in 0..<recordsCount { + records.append(try metadataChangeRecord(from: &reader)) + } + let itemsCount = try reader.readCount() + var items: [MetadataCurrentItem] = [] + items.reserveCapacity(min(itemsCount, reader.remaining / metadataItemMinimumSize)) + for _ in 0..<itemsCount { + items.append(try metadataItem(from: &reader)) + } + let tokenLength = Int(try reader.readUInt16()) + var currentPageToken: MetadataCurrentPageToken? + if tokenLength != 0 { + var tokenReader = ByteReader(try reader.readBytes(tokenLength)) + currentPageToken = try MetadataCurrentPageToken.decode(from: &tokenReader) + guard tokenReader.isAtEnd else { + throw WireError.validation("trailing bytes after a page token") + } + } + let holdsOtherContent = phase == .changes ? !items.isEmpty || currentPageToken != nil : !records.isEmpty + guard !holdsOtherContent else { + throw WireError.validation("a changes page carries only records and a current page only items") + } + return MetadataChanges( + status: status, phase: phase, logID: logID, oldestOp: oldestOp, retainedSince: retainedSince, headOp: headOp, next: next, hasMore: hasMore, + records: records, items: items, currentPageToken: currentPageToken) + } + } + + /// Smallest record envelope: no actor and an empty body. + private static let metadataChangeRecordMinimumSize = 8 + 4 + 16 + 8 + 1 + 1 + 1 + 4 + /// Smallest item envelope: an empty body. + private static let metadataItemMinimumSize = 1 + 4 + + /// `[op:u64][ordinal:u32][checksum:u128][timestamp:u64][has_actor:u8][actor_user_id:u32][group:u8][kind:u8][body_len:u32][body]` + static func metadataChangeRecord(from reader: inout ByteReader) throws -> MetadataChangeRecord { + let op = try reader.readUInt64() + let ordinal = try reader.readUInt32() + let checksum = try reader.readUInt128() + let timestamp = try reader.readTimestamp() + let actorUserID = try reader.readOptionalUInt32() + let group = try reader.readUInt8() + let kind = try reader.readUInt8() + let change = try MetadataChange.decode(group: group, kind: kind, body: try reader.readPrefixedBytes()) + return MetadataChangeRecord(op: op, ordinal: ordinal, checksum: checksum, timestamp: timestamp, actorUserID: actorUserID, change: change) + } + + /// `[group:u8][body_len:u32][body]` + static func metadataItem(from reader: inout ByteReader) throws -> MetadataCurrentItem { + let group = try reader.readUInt8() + return try MetadataCurrentItem.decode(group: group, body: try reader.readPrefixedBytes()) + } + // MARK: Routing static let partitionRouteSize = 25
diff --git a/foreign/swift/Tests/IggyE2ETests/MetadataChangesE2ETests.swift b/foreign/swift/Tests/IggyE2ETests/MetadataChangesE2ETests.swift new file mode 100644 index 0000000..a606a75 --- /dev/null +++ b/foreign/swift/Tests/IggyE2ETests/MetadataChangesE2ETests.swift
@@ -0,0 +1,375 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import Foundation +import Iggy +import Testing + +/// The shared metadata changes scenarios of `bdd/scenarios/metadata_changes.feature`. +/// Other suites run alongside, so every check names the streams this test +/// created rather than the whole log. The keyed scenario needs a server +/// encryption key and lives with the resource options suite below. +@Suite("Metadata changes", requiresServer) +struct MetadataChangesE2ETests { + static let rootUserID: UInt32 = 0 + static let streamGroups: [MetadataChangeGroup] = [.streams] + + @Test func givenLatestCursorWhenCreatingRenamingAndDeletingStreamShouldFollowEachChangeWithRootActor() async throws { + try await E2EEnvironment.withSession { session in + let client = session.client + let cursor = try await Self.latestCursor(client, groups: Self.streamGroups) + let name = E2EEnvironment.uniqueName("swift-changes-follow") + let renamed = name + "-renamed" + await session.track(stream: try Identifier(named: renamed)) + let stream = try await client.createStream(name: name) + try await client.updateStream(Identifier(numeric: stream.id), options: StreamUpdateOptions(name: renamed)) + try await client.deleteStream(Identifier(numeric: stream.id)) + + let records = try await Self.follow(client, after: cursor, groups: Self.streamGroups, count: 3, first: Self.creates(stream: name)) + #expect(records.map(\.change.kind) == [.streamCreated, .streamUpdated, .streamDeleted]) + #expect(records.allSatisfy { $0.actorUserID == Self.rootUserID }) + #expect(records.map(\.op) == records.map(\.op).sorted()) + guard records.count == 3, case .streamCreated(let created) = records[0].change, case .streamUpdated(let updated) = records[1].change, + case .streamDeleted(let deleted) = records[2].change + else { + Issue.record("unexpected records \(records)") + return + } + #expect(created.name == name) + #expect(updated.name == renamed) + #expect(deleted.name == renamed) + #expect(deleted.streamCreatedOp == records[0].op) + #expect(deleted.requestedOp == nil) + } + } + + @Test func givenTopicsGroupWhenCreatingStreamAndTopicShouldFollowOnlyTheTopic() async throws { + try await E2EEnvironment.withSession { session in + let client = session.client + let groups: [MetadataChangeGroup] = [.topics] + let cursor = try await Self.latestCursor(client, groups: groups) + let name = E2EEnvironment.uniqueName("swift-changes-filter") + let stream = try await client.createStream(name: name) + await session.track(stream: try Identifier(named: name)) + let topicName = E2EEnvironment.uniqueName("swift-changes-events") + _ = try await client.createTopic(streamID: Identifier(numeric: stream.id), name: topicName, options: TopicCreateOptions(partitionsCount: 1)) + + let records = try await Self.follow(client, after: cursor, groups: groups, count: 1) { record in + if case .topicCreated(let topic) = record.change { topic.name == topicName } else { false } + } + #expect(records.map(\.change.kind) == [.topicCreated]) + guard case .topicCreated(let topic) = records.first?.change else { + Issue.record("expected a topic record, got \(records)") + return + } + #expect(topic.streamID == stream.id) + #expect(topic.partitionsCount == 1) + #expect(topic.topicCreatedOp > topic.streamCreatedOp) + } + } + + @Test func givenStreamWithTopicWhenDeletingShouldRecordTheTwoPhaseDeleteAtItsCompletion() async throws { + try await E2EEnvironment.withSession { session in + let client = session.client + let cursor = try await Self.latestCursor(client, groups: Self.streamGroups) + let name = E2EEnvironment.uniqueName("swift-changes-two-phase") + await session.track(stream: try Identifier(named: name)) + let stream = try await client.createStream(name: name) + _ = try await client.createTopic(streamID: Identifier(numeric: stream.id), name: "events", options: TopicCreateOptions(partitionsCount: 1)) + try await client.deleteStream(Identifier(numeric: stream.id)) + + let records = try await Self.follow(client, after: cursor, groups: Self.streamGroups, count: 2, first: Self.creates(stream: name)) + #expect(records.map(\.change.kind) == [.streamCreated, .streamDeleted]) + guard let last = records.last, case .streamDeleted(let deleted) = last.change else { + Issue.record("expected a stream deletion, got \(records)") + return + } + let requestedOp = try #require(deleted.requestedOp) + #expect(requestedOp < last.op) + } + } + + @Test func givenLimitOfOneWhenFollowingShouldPageOneRecordAtATimeInOrder() async throws { + try await E2EEnvironment.withSession { session in + let client = session.client + let cursor = try await Self.latestCursor(client, groups: Self.streamGroups) + var names = Set<String>() + for index in 0..<3 { + let name = E2EEnvironment.uniqueName("swift-changes-page-\(index)") + _ = try await client.createStream(name: name) + names.insert(name) + await session.track(stream: try Identifier(named: name)) + } + // Other tests create streams under reused ids at the same time, so + // only these names count. + func createdHere(_ record: MetadataChangeRecord) -> Bool { + guard case .streamCreated(let stream) = record.change else { + return false + } + return names.contains(stream.name) + } + + var paged: [UInt64] = [] + var position = cursor + while true { + let page = try await client.getMetadataChanges(MetadataChangesQuery(start: .after(position), groups: Self.streamGroups, limit: 1)) + #expect(page.status == .ok) + #expect(page.records.count <= 1) + paged += page.records.filter(createdHere).map(\.op) + position = try #require(page.next) + if !page.hasMore { + break + } + } + let whole = try await client.getMetadataChanges(MetadataChangesQuery(start: .after(cursor), groups: Self.streamGroups)) + let expected = whole.records.filter(createdHere).map(\.op) + #expect(expected.count == names.count) + #expect(paged == expected) + } + } + + @Test func givenStreamsWhenPagingCurrentStateShouldListThemAndAnchorTheChangeLog() async throws { + try await E2EEnvironment.withSession { session in + let client = session.client + var created: [UInt32: String] = [:] + for index in 0..<2 { + let name = E2EEnvironment.uniqueName("swift-changes-current-\(index)") + created[try await client.createStream(name: name).id] = name + await session.track(stream: try Identifier(named: name)) + } + let (items, anchor) = try await Self.currentState(client, groups: Self.streamGroups, limit: 1) + var projected: [UInt32: String] = [:] + for item in items { + guard case .stream(let stream) = item else { + Issue.record("unexpected item \(item)") + continue + } + projected[stream.streamID] = stream.name + } + for (id, name) in created { + #expect(projected[id] == name) + } + let listed = Set(try await client.getStreams().map(\.id)) + #expect(Set(created.keys).isSubset(of: listed)) + let follow = try await client.getMetadataChanges(MetadataChangesQuery(start: .after(anchor), groups: Self.streamGroups)) + #expect(follow.status == .ok) + } + } + + @Test func givenCursorOfAnotherHistoryWhenFollowingShouldAnswerResetThenRebuildFromCurrentState() async throws { + try await E2EEnvironment.withSession { session in + let client = session.client + let cursor = try await Self.latestCursor(client, groups: Self.streamGroups) + let name = E2EEnvironment.uniqueName("swift-changes-reset") + let stream = try await client.createStream(name: name) + await session.track(stream: try Identifier(named: name)) + + var foreign = MetadataChangeCursor.zero(logID: cursor.logID) + foreign.logID.low &+= 1 + let page = try await client.getMetadataChanges(MetadataChangesQuery(start: .after(foreign), groups: Self.streamGroups)) + #expect(page.status == .reset) + #expect(page.records.isEmpty) + #expect(page.next == nil) + + let (items, anchor) = try await Self.currentState(client, groups: Self.streamGroups, limit: MetadataChangesQuery.maxLimit) + #expect( + items.contains { item in + if case .stream(let projected) = item { projected.streamID == stream.id && projected.name == name } else { false } + }) + let follow = try await client.getMetadataChanges(MetadataChangesQuery(start: .after(anchor), groups: Self.streamGroups)) + #expect(follow.status == .ok) + } + } + + /// Runs where the server's resource ring is small enough to churn past, + /// named by the same variable the server reads. + @Test(.enabled(if: MetadataChangesE2ETests.resourceCapacity != nil, "set IGGY_METADATA_CHANGE_LOG_RESOURCE_CAPACITY to the server's value to run")) + func givenChurnBeyondCapacityWhenFollowingShouldAnswerExpiredThenRebuildFromCurrentState() async throws { + let capacity = try #require(Self.resourceCapacity) + try await E2EEnvironment.withSession { session in + let client = session.client + let cursor = try await Self.latestCursor(client, groups: Self.streamGroups) + let prefix = E2EEnvironment.uniqueName("swift-changes-churn") + for index in 0..<(capacity / 2 + 8) { + let stream = try await client.createStream(name: "\(prefix)-\(index)") + try await client.deleteStream(Identifier(numeric: stream.id)) + } + let expired = try await client.getMetadataChanges(MetadataChangesQuery(start: .after(cursor), groups: Self.streamGroups)) + #expect(expired.status == .expired) + #expect(expired.records.isEmpty) + #expect(expired.next == nil) + + let (_, anchor) = try await Self.currentState(client, groups: Self.streamGroups, limit: MetadataChangesQuery.maxLimit) + let follow = try await client.getMetadataChanges(MetadataChangesQuery(start: .after(anchor), groups: Self.streamGroups)) + #expect(follow.status == .ok) + } + } + + @Test func givenInvalidQueriesWhenReadingShouldRefuseEachWithInvalidCommand() async throws { + try await E2EEnvironment.withSession { session in + let client = session.client + let queries = [ + MetadataChangesQuery(limit: 0), + MetadataChangesQuery(limit: MetadataChangesQuery.maxLimit + 1), + MetadataChangesQuery(since: IggyTimestamp(microseconds: 2), until: IggyTimestamp(microseconds: 1)), + MetadataChangesQuery(start: .latest, since: IggyTimestamp(microseconds: 1)), + MetadataChangesQuery(start: .current(nil), until: IggyTimestamp(microseconds: 1)), + MetadataChangesQuery(start: .current(nil), groups: [.topics]), + ] + for query in queries { + await expectIggyError(.invalidCommand) { _ = try await client.getMetadataChanges(query) } + } + } + } + + @Test func givenUserWithoutManageServersWhenReadingShouldBeRefusedAsUnauthorized() async throws { + try await E2EEnvironment.withSession { session in + let root = session.client + let username = String(E2EEnvironment.uniqueName("swift-changes-reader").prefix(50)) + let user = try await root.createUser( + username: username, password: "changes-secret", permissions: Permissions(global: GlobalPermissions(readServers: true))) + let reader = IggyClient( + configuration: E2EEnvironment.configuration(autoLogin: .usernamePassword(username: username, password: "changes-secret"))) + do { + try await reader.connect() + await expectIggyError(.unauthorized) { _ = try await reader.getMetadataChanges() } + } catch { + await reader.shutdown() + try? await root.deleteUser(Identifier(numeric: user.id)) + throw error + } + await reader.shutdown() + try await root.deleteUser(Identifier(numeric: user.id)) + } + } + + static var resourceCapacity: Int? { + ProcessInfo.processInfo.environment["IGGY_METADATA_CHANGE_LOG_RESOURCE_CAPACITY"].flatMap(Int.init) + } + + static func latestCursor(_ client: IggyClient, groups: [MetadataChangeGroup]) async throws -> MetadataChangeCursor { + let page = try await client.getMetadataChanges(MetadataChangesQuery(start: .latest, groups: groups)) + #expect(page.status == .ok) + #expect(page.records.isEmpty) + #expect(page.logID != nil) + return try #require(page.next) + } + + /// Follows the change log after `cursor` until `count` records name the + /// stream whose first record matches `first`. Other tests create and + /// delete streams under reused ids at the same time, so the stream is told + /// apart by its creating op. A two phase delete commits its record after + /// the call returns, so an incomplete read polls again. + static func follow( + _ client: IggyClient, after cursor: MetadataChangeCursor, groups: [MetadataChangeGroup], count: Int, + first: (MetadataChangeRecord) -> Bool + ) async throws -> [MetadataChangeRecord] { + var position = cursor + var mine: [MetadataChangeRecord] = [] + var incarnation: (id: UInt32, createdOp: UInt64)? + let found = try await E2EEnvironment.eventually { + while true { + let page = try await client.getMetadataChanges( + MetadataChangesQuery(start: .after(position), groups: groups, limit: MetadataChangesQuery.maxLimit)) + #expect(page.status == .ok) + position = try #require(page.next) + for record in page.records { + #expect(record.change.group.map(groups.contains) == true, "record of op \(record.op) is outside the requested groups") + guard let stream = Self.stream(of: record.change) else { + continue + } + if incarnation == nil, first(record) { + incarnation = stream + } + if let incarnation, stream == incarnation { + mine.append(record) + } + } + if !page.hasMore { + return mine.count >= count + } + } + } + #expect(found, "\(mine.count) of \(count) records") + return mine + } + + /// Whether `record` created the stream named `name`. + static func creates(stream name: String) -> (MetadataChangeRecord) -> Bool { + { record in + if case .streamCreated(let stream) = record.change { stream.name == name } else { false } + } + } + + /// Pages the current state of `groups` and returns its items with the + /// anchor to follow the change log from. + static func currentState(_ client: IggyClient, groups: [MetadataChangeGroup], limit: UInt32) async throws -> ([MetadataCurrentItem], MetadataChangeCursor) { + var query = MetadataChangesQuery(start: .current(nil), groups: groups, limit: limit) + var items: [MetadataCurrentItem] = [] + while true { + let page = try await client.getMetadataChanges(query) + #expect(page.status == .ok) + #expect(page.phase == .current) + #expect(page.records.isEmpty) + #expect(page.items.count <= Int(limit)) + items += page.items + guard let token = page.currentPageToken else { + return (items, try #require(page.next)) + } + query.start = .current(token) + } + } + + static func stream(of change: MetadataChange) -> (id: UInt32, createdOp: UInt64)? { + switch change { + case .streamCreated(let body), .streamUpdated(let body): (body.streamID, body.streamCreatedOp) + case .streamDeleted(let body): (body.streamID, body.streamCreatedOp) + case .topicCreated(let body), .topicUpdated(let body): (body.streamID, body.streamCreatedOp) + case .topicDeleted(let body): (body.streamID, body.streamCreatedOp) + case .partitionsChanged(let body): (body.streamID, body.streamCreatedOp) + default: nil + } + } +} + +/// The keyed scenario needs a server encryption key, so it belongs to the +/// resource options suite the shipped default lane skips. +extension ResourceOptionsE2ETests { + @Test func givenKeyedStreamWhenFollowingMetadataChangesShouldCarryFingerprintsAndNoKeyMaterial() async throws { + try await E2EEnvironment.withSession { session in + let client = session.client + let groups = MetadataChangesE2ETests.streamGroups + let cursor = try await MetadataChangesE2ETests.latestCursor(client, groups: groups) + let key = Self.randomKey() + let name = E2EEnvironment.uniqueName("swift-changes-keyed") + _ = try await client.createStream(name: name, options: StreamCreateOptions(encryptionKey: key)) + await session.track(stream: try Identifier(named: name)) + + let records = try await MetadataChangesE2ETests.follow( + client, after: cursor, groups: groups, count: 1, first: MetadataChangesE2ETests.creates(stream: name)) + #expect(records.map(\.change.kind) == [.streamCreated]) + guard case .streamCreated(let created) = records.first?.change, case .readable(let options) = created.options else { + Issue.record("expected a readable stream record, got \(records)") + return + } + #expect(options[StreamOptionKey.encryptionKeySource.rawValue]?.value == "stream") + #expect(options[StreamOptionKey.encryptionKeyFingerprint.rawValue]?.value == (try HeaderValue.string(try EncryptionKey.fingerprint(key)))) + #expect(options[StreamOptionKey.encryptionKekFingerprint.rawValue].map { Self.isFingerprint($0.value.description) } == true) + Self.expectNoKeyMaterial(options, key: key) + } + } +}
diff --git a/foreign/swift/Tests/IggyTests/Fixtures/golden.json b/foreign/swift/Tests/IggyTests/Fixtures/golden.json index 83235eb..5303079 100644 --- a/foreign/swift/Tests/IggyTests/Fixtures/golden.json +++ b/foreign/swift/Tests/IggyTests/Fixtures/golden.json
@@ -927,7 +927,7 @@ "polling.next": "050000000000000000", "polling.offset.100": "016400000000000000", "polling.timestamp.1700000000000": "020068e5cf8b010000", - "request.bind_session": "003000000973776966742d73646b05302e312e3101000000000000000000000000000000640000000000000078000000000000005a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", + "request.bind_session": "003000000973776966742d73646b05302e312e3201000000000000000000000000000000640000000000000078000000000000005a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a", "request.change_password": "010401000000086f6c642d706173730c6e65772d706173732d313233", "request.create_consumer_group": "01040100000001040200000003677270", "request.create_partitions": "01040100000001040200000005000000", @@ -954,6 +954,7 @@ "request.get_consumer_groups": "010401000000010402000000", "request.get_consumer_offset": "01020b6d792d636f6e73756d6572020873747265616d2d310207746f7069632d310100000000", "request.get_me": "", + "request.get_metadata_changes": "02d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000000100f0bd010941060000f40100000202010000", "request.get_partition_route": "01040a00000002066f726465727305000000", "request.get_personal_access_tokens": "", "request.get_snapshot": "02020105", @@ -966,8 +967,8 @@ "request.get_users": "", "request.join_consumer_group": "010401000000010402000000010403000000", "request.leave_consumer_group": "010401000000010402000000020167", - "request.login_register": "003000000973776966742d73646b05302e312e315a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a0469676779046967677900000000", - "request.login_register_with_pat": "003000000973776966742d73646b05302e312e315a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a107061742d61626331323364656634353600000000", + "request.login_register": "003000000973776966742d73646b05302e312e325a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a0469676779046967677900000000", + "request.login_register_with_pat": "003000000973776966742d73646b05302e312e325a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a107061742d61626331323364656634353600000000", "request.logout_user": "", "request.ping": "", "request.poll_messages": "0101040100000001040a00000001041400000001050000000164000000000000003200000001", @@ -993,6 +994,7 @@ "response.describe_options": "040000000c7365676d656e745f73697a650c000800000000000040000000000c0000005365676d656e742073697a650a6475726162696c69747902000a0000007265706c696361746564000000000e6d6573736167655f6578706972790c0308000000ffffffffffffffff000000000e656e6372797074696f6e5f6b657901140000000000000000", "response.get_clients": "010000000a000000010e0000003132372e302e302e313a38303830020000000200000014000000020d00000031302e302e302e313a3630303003000000", "response.get_consumer_groups": "0100000004000000020000000767726f75702d610200000004000000000000000767726f75702d62", + "response.get_metadata_changes": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000000000000000005e1000000000000001d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000000001000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be0109410600010000000001012700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000000000000000000000000", "response.get_personal_access_tokens": "07746f6b656e2d6100e090663c13060007746f6b656e2d620000000000000000", "response.get_stream": "0100000000e090663c130600020000000008000000000000c800000000000000096d792d73747265616d000000009e000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e740221000000333965346630663935633336393363372c64353564343739383239626332393336021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000383434316533383533386362666135370215000000656e6372797074696f6e5f6b65795f736f75726365020600000073747265616d0100000000e090663c130600030000000010000000000000640000000000000007746f7069632d615b000000020a0000006475726162696c6974790209000000706572736973746564020e0000006d6573736167655f6578706972790c0800000000a0e3d08c000000020c0000007365676d656e745f73697a650c080000000000004000000000430000000215000000636f6d7072657373696f6e5f616c676f726974686d02040000006e6f6e65020e0000006d61785f746f7069635f73697a650c08000000ffffffffffffffff0200000000e090663c130600030000000010000000000000640000000000000007746f7069632d625b000000020a0000006475726162696c6974790209000000706572736973746564020e0000006d6573736167655f6578706972790c0800000000a0e3d08c000000020c0000007365676d656e745f73697a650c080000000000004000000000430000000215000000636f6d7072657373696f6e5f616c676f726974686d02040000006e6f6e65020e0000006d61785f746f7069635f73697a650c08000000ffffffffffffffff", "response.get_streams": "0100000000e090663c130600020000000008000000000000c800000000000000096d792d73747265616d000000009e000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e740221000000333965346630663935633336393363372c64353564343739383239626332393336021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000383434316533383533386362666135370215000000656e6372797074696f6e5f6b65795f736f75726365020600000073747265616d0200000000e090663c130600000000000008000000000000c800000000000000096d792d73747265616d000000009e000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e740221000000333965346630663935633336393363372c64353564343739383239626332393336021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000383434316533383533386362666135370215000000656e6372797074696f6e5f6b65795f736f75726365020600000073747265616d", @@ -1017,7 +1019,7 @@ "response.user_details.no_permissions": "0100000000e090663c130600010561646d696e000000000000000000000000", "stream_key.fingerprint": "8441e38538cbfa57", "user_headers.sample": "0207000000617474656d70740b0400000003000000020800000074726163652d69640203000000616263", - "version_info": "003000000973776966742d73646b05302e312e31", + "version_info": "003000000973776966742d73646b05302e312e32", "vsr.eviction.incompatible_protocol": "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000d000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001032547698badcfeefcdab89674523010030000000300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e", "vsr.eviction.stale_client": "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000d000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001032547698badcfeefcdab89674523010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000d", "vsr.reply.committed": "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000c0100000000000000000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001032547698badcfeefcdab896745230100000000000000000c0000000000000000000000000000000700000000000000800000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000",
diff --git a/foreign/swift/Tests/IggyTests/Fixtures/metadata_changes.json b/foreign/swift/Tests/IggyTests/Fixtures/metadata_changes.json new file mode 100644 index 0000000..26e7faf --- /dev/null +++ b/foreign/swift/Tests/IggyTests/Fixtures/metadata_changes.json
@@ -0,0 +1,1997 @@ +{ + "schema_version": 1, + "generated_by": "core/common/examples/metadata_changes_fixtures.rs", + "conventions": { + "byte_strings": "Every *_hex field is lowercase hex without separators.", + "expected": "expected is the JSON the HTTP route renders. Binary decoders must reach the same values.", + "record_hex": "record_hex is one record envelope as a response carries it. body_hex is the body alone.", + "refused": "Every refused vector must fail to decode or validate. Query and HTTP refusals are invalid_command.", + "request_hex": "request_hex is the command payload without framing. http_query is the query string of GET /metadata/changes.", + "u128_values": "128-bit values (log ids, checksums, client ids) are 32 lowercase hex characters in JSON.", + "u64_values": "Ops and timestamps are JSON numbers below 2^53 in every vector except large_values, which carries them as decimal strings.", + "unknown_kinds": "A record of an unknown group or kind is kept with its codes and raw body, never refused." + }, + "constants": { + "command_code": 17, + "command_name": "metadata.changes", + "cursor_prefix": "mc1_", + "page_token_prefix": "cs1_", + "cursor_size": 44, + "cursor_text_length": 63, + "max_page_token_size": 316, + "max_page_token_key_size": 260, + "default_limit": 100, + "max_limit": 1000, + "groups": { + "consumer_group_members": { + "code": 5, + "class": "activity", + "parents": [ + "consumer_groups", + "partitions", + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32][member_id:u32]" + }, + "consumer_groups": { + "code": 4, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32][group_id:u32]" + }, + "partitions": { + "code": 3, + "class": "resource", + "parents": [ + "topics", + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "personal_access_tokens": { + "code": 7, + "class": "resource", + "parents": [ + "users" + ], + "page_token_key": "[user_id:u32][name_len:u8][name]" + }, + "sessions": { + "code": 8, + "class": "activity", + "parents": [ + "users" + ], + "page_token_key": "[epoch:u64]" + }, + "streams": { + "code": 1, + "class": "resource", + "parents": [], + "page_token_key": "[stream_id:u32]" + }, + "topics": { + "code": 2, + "class": "resource", + "parents": [ + "streams" + ], + "page_token_key": "[stream_id:u32][topic_id:u32]" + }, + "users": { + "code": 6, + "class": "resource", + "parents": [], + "page_token_key": "[user_id:u32]" + } + }, + "kinds": { + "consumer_group_created": { + "code": 30, + "group": "consumer_groups" + }, + "consumer_group_deleted": { + "code": 31, + "group": "consumer_groups" + }, + "consumer_group_member_joined": { + "code": 40, + "group": "consumer_group_members" + }, + "consumer_group_member_left": { + "code": 41, + "group": "consumer_group_members" + }, + "consumer_group_partition_handed_over": { + "code": 42, + "group": "consumer_group_members" + }, + "partitions_changed": { + "code": 20, + "group": "partitions" + }, + "personal_access_token_created": { + "code": 60, + "group": "personal_access_tokens" + }, + "personal_access_token_deleted": { + "code": 61, + "group": "personal_access_tokens" + }, + "session_closed": { + "code": 71, + "group": "sessions" + }, + "session_registered": { + "code": 70, + "group": "sessions" + }, + "stream_created": { + "code": 1, + "group": "streams" + }, + "stream_deleted": { + "code": 3, + "group": "streams" + }, + "stream_updated": { + "code": 2, + "group": "streams" + }, + "topic_created": { + "code": 10, + "group": "topics" + }, + "topic_deleted": { + "code": 12, + "group": "topics" + }, + "topic_updated": { + "code": 11, + "group": "topics" + }, + "user_created": { + "code": 50, + "group": "users" + }, + "user_deleted": { + "code": 52, + "group": "users" + }, + "user_password_changed": { + "code": 53, + "group": "users" + }, + "user_permissions_updated": { + "code": 54, + "group": "users" + }, + "user_updated": { + "code": 51, + "group": "users" + } + }, + "statuses": { + "behind": 1, + "expired": 2, + "ok": 0, + "reset": 3 + }, + "phases": { + "changes": 0, + "current": 1 + }, + "start_modes": { + "after": 2, + "current": 3, + "earliest": 0, + "latest": 1 + }, + "member_left_reasons": { + "left": 0, + "session_ended": 1 + }, + "token_deleted_reasons": { + "deleted": 0, + "expired": 1 + }, + "session_closed_reasons": { + "expired": 1, + "logout": 0, + "revoked": 2 + }, + "permissions_updated_reasons": { + "resource_deleted": 1, + "updated": 0 + } + }, + "records": [ + { + "case": "stream_created", + "group": 1, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "record_hex": "191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "stream_updated_unreadable_options", + "group": 1, + "kind": 2, + "body_hex": "03000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "record_hex": "1a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d763201", + "expected": { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + } + }, + { + "case": "stream_deleted_deferred", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "record_hex": "1b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000", + "expected": { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + }, + { + "case": "topic_created", + "group": 2, + "kind": 10, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "record_hex": "1c1000000000000000000000683d9e8b03f7c1a500000000000000001c00be01094106000104000000020a78000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d657472696373080000000020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "op": 4124, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d68", + "timestamp": 1760356800004124, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "partitions_count": 8, + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "topic_updated", + "group": 2, + "kind": 11, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "record_hex": "1d1000000000000000000000693d9e8b03f7c1a500000000000000001d00be01094106000104000000020b3d000000030000001910000000000000000000001d100000000000004032cd01094106000f6e6f64652d6d6574726963732d763208000000000000000000000000", + "expected": { + "op": 4125, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d69", + "timestamp": 1760356800004125, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_updated", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics-v2", + "partitions_count": 8, + "options": {} + } + }, + { + "case": "topic_deleted_immediate", + "group": 2, + "kind": 12, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "record_hex": "1e10000000000000000000006a3d9e8b03f7c1a500000000000000001e00be01094106000104000000020c2e000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d65747269637300", + "expected": { + "op": 4126, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6a", + "timestamp": 1760356800004126, + "actor_user_id": 4, + "group": "topics", + "kind": "topic_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics" + } + }, + { + "case": "partitions_changed_deleted_tail", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "record_hex": "1f10000000000000000000006b3d9e8b03f7c1a500000000000000001f00be01094106000104000000031435000000030000001910000000000000000000001d100000000000000500000003000000050000000600000007000000016810000000000000", + "expected": { + "op": 4127, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6b", + "timestamp": 1760356800004127, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 5, + "removed_partition_ids": [ + 5, + 6, + 7 + ], + "requested_op": 4200 + } + }, + { + "case": "partitions_changed_created", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000a0000000000000000", + "record_hex": "2010000000000000000000006c3d9e8b03f7c1a500000000000000002000be01094106000104000000031421000000030000001910000000000000000000001d100000000000000a0000000000000000", + "expected": { + "op": 4128, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6c", + "timestamp": 1760356800004128, + "actor_user_id": 4, + "group": "partitions", + "kind": "partitions_changed", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 10, + "removed_partition_ids": [] + } + }, + { + "case": "consumer_group_created", + "group": 4, + "kind": 30, + "body_hex": "030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "record_hex": "2110000000000000000000006d3d9e8b03f7c1a500000000000000002100be01094106000104000000041e28000000030000001910000000000000000000001d1000000000000000000000077265616465727301000000", + "expected": { + "op": 4129, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6d", + "timestamp": 1760356800004129, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_created", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 1 + } + }, + { + "case": "consumer_group_deleted", + "group": 4, + "kind": 31, + "body_hex": "030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "record_hex": "2210000000000000000000006e3d9e8b03f7c1a500000000000000002200be01094106000104000000041f31000000030000001910000000000000000000001d100000000000000000000007726561646572730000000001cc10000000000000", + "expected": { + "op": 4130, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6e", + "timestamp": 1760356800004130, + "actor_user_id": 4, + "group": "consumer_groups", + "kind": "consumer_group_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers", + "consumer_groups_count": 0, + "requested_op": 4300 + } + }, + { + "case": "consumer_group_member_joined", + "group": 5, + "kind": 40, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "record_hex": "2310000000000000000000006f3d9e8b03f7c1a500000000000000002300be01094106000109000000052838000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010200000000000000", + "expected": { + "op": 4131, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d6f", + "timestamp": 1760356800004131, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_joined", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 2, + "partition_ids": [] + } + }, + { + "case": "consumer_group_member_left_session_ended", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "record_hex": "241000000000000000000000703d9e8b03f7c1a500000000000000002400be01094106000109000000052935000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000001", + "expected": { + "op": 4132, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d70", + "timestamp": 1760356800004132, + "actor_user_id": 9, + "group": "consumer_group_members", + "kind": "consumer_group_member_left", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "members_count": 1, + "reason": "session_ended" + } + }, + { + "case": "consumer_group_partition_handed_over", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d1000000000000000000000020000000101000000", + "record_hex": "251000000000000000000000713d9e8b03f7c1a500000000000000002500be010941060000052a25000000030000001910000000000000000000001d1000000000000000000000020000000101000000", + "expected": { + "op": 4133, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d71", + "timestamp": 1760356800004133, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 2, + "member_id": 1 + } + }, + { + "case": "consumer_group_partition_released", + "group": 5, + "kind": 42, + "body_hex": "030000001910000000000000000000001d10000000000000000000000300000000", + "record_hex": "261000000000000000000000723d9e8b03f7c1a500000000000000002600be010941060000052a21000000030000001910000000000000000000001d10000000000000000000000300000000", + "expected": { + "op": 4134, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d72", + "timestamp": 1760356800004134, + "group": "consumer_group_members", + "kind": "consumer_group_partition_handed_over", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "partition_id": 3 + } + }, + { + "case": "user_created_with_permissions", + "group": 6, + "kind": 50, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "record_hex": "271000000000000000000000733d9e8b03f7c1a500000000000000002700be010941060001000000000632480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "op": 4135, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d73", + "timestamp": 1760356800004135, + "actor_user_id": 0, + "group": "users", + "kind": "user_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "user_updated_inactive", + "group": 6, + "kind": 51, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "record_hex": "281000000000000000000000743d9e8b03f7c1a500000000000000002800be010941060001000000000633280000000700000000b832c8fb4006001074656c656d657472792d7772697465720200000000000000000000", + "expected": { + "op": 4136, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d74", + "timestamp": 1760356800004136, + "actor_user_id": 0, + "group": "users", + "kind": "user_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "inactive", + "permissions": null, + "options": {} + } + }, + { + "case": "user_deleted", + "group": 6, + "kind": 52, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "291000000000000000000000753d9e8b03f7c1a500000000000000002900be0109410600010000000006341d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4137, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d75", + "timestamp": 1760356800004137, + "actor_user_id": 0, + "group": "users", + "kind": "user_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_password_changed", + "group": 6, + "kind": 53, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d777269746572", + "record_hex": "2a1000000000000000000000763d9e8b03f7c1a500000000000000002a00be0109410600010700000006351d0000000700000000b832c8fb4006001074656c656d657472792d777269746572", + "expected": { + "op": 4138, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d76", + "timestamp": 1760356800004138, + "actor_user_id": 7, + "group": "users", + "kind": "user_password_changed", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer" + } + }, + { + "case": "user_permissions_pruned", + "group": 6, + "kind": 54, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "record_hex": "2b1000000000000000000000773d9e8b03f7c1a500000000000000002b00be0109410600010400000006362a0000000700000000b832c8fb4006001074656c656d657472792d77726974657201000000000000000000000001", + "expected": { + "op": 4139, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d77", + "timestamp": 1760356800004139, + "actor_user_id": 4, + "group": "users", + "kind": "user_permissions_updated", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": false, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": null + }, + "reason": "resource_deleted" + } + }, + { + "case": "personal_access_token_created", + "group": 7, + "kind": 60, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "record_hex": "2c1000000000000000000000783d9e8b03f7c1a500000000000000002c00be01094106000107000000073c1d0000000700000000b832c8fb4006000863692d746f6b656e00a0a91013410600", + "expected": { + "op": 4140, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d78", + "timestamp": 1760356800004140, + "actor_user_id": 7, + "group": "personal_access_tokens", + "kind": "personal_access_token_created", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "expiry": 1760400000000000 + } + }, + { + "case": "personal_access_token_deleted_expired", + "group": 7, + "kind": 61, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e01", + "record_hex": "2d1000000000000000000000793d9e8b03f7c1a500000000000000002d00be010941060000073d160000000700000000b832c8fb4006000863692d746f6b656e01", + "expected": { + "op": 4141, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d79", + "timestamp": 1760356800004141, + "group": "personal_access_tokens", + "kind": "personal_access_token_deleted", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token", + "reason": "expired" + } + }, + { + "case": "session_registered", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "record_hex": "2e10000000000000000000007a3d9e8b03f7c1a500000000000000002e00be010941060001070000000846240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "op": 4142, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7a", + "timestamp": 1760356800004142, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_registered", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + }, + { + "case": "session_closed_expired", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "record_hex": "2f10000000000000000000007b3d9e8b03f7c1a500000000000000002f00be0109410600000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000001", + "expected": { + "op": 4143, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7b", + "timestamp": 1760356800004143, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "expired" + } + }, + { + "case": "session_closed_revoked", + "group": 8, + "kind": 71, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "record_hex": "3010000000000000000000007c3d9e8b03f7c1a500000000000000003000be010941060001070000000847250000000700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000002", + "expected": { + "op": 4144, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d7c", + "timestamp": 1760356800004144, + "actor_user_id": 7, + "group": "sessions", + "kind": "session_closed", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500, + "reason": "revoked" + } + } + ], + "items": [ + { + "case": "stream", + "group": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "item_hex": "01b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564", + "expected": { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + } + }, + { + "case": "topic", + "group": 2, + "body_hex": "030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "item_hex": "0274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c080000000000004000000000", + "expected": { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + }, + { + "case": "partitions", + "group": 3, + "body_hex": "030000001910000000000000000000001d1000000000000008000000", + "item_hex": "031c000000030000001910000000000000000000001d1000000000000008000000", + "expected": { + "group": "partitions", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "partitions_count": 8 + } + }, + { + "case": "consumer_group", + "group": 4, + "body_hex": "030000001910000000000000000000001d10000000000000000000000772656164657273", + "item_hex": "0424000000030000001910000000000000000000001d10000000000000000000000772656164657273", + "expected": { + "group": "consumer_groups", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "name": "readers" + } + }, + { + "case": "consumer_group_member", + "group": 5, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "item_hex": "0540000000030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a09080706050403020103000000000000000200000004000000", + "expected": { + "group": "consumer_group_members", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "group_id": 0, + "member_id": 1, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "partition_ids": [ + 0, + 2, + 4 + ] + } + }, + { + "case": "user", + "group": 6, + "body_hex": "0700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "item_hex": "06480000000700000000b832c8fb4006001074656c656d657472792d77726974657201010000000000010000000001030000000001000000000100000000000001000000000000000000000000", + "expected": { + "group": "users", + "user_id": 7, + "user_created_at": 1760300000000000, + "username": "telemetry-writer", + "status": "active", + "permissions": { + "global": { + "manage_servers": false, + "read_servers": false, + "manage_users": false, + "read_users": false, + "manage_streams": false, + "read_streams": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false + }, + "streams": { + "3": { + "manage_stream": false, + "read_stream": true, + "manage_topics": false, + "read_topics": false, + "poll_messages": false, + "send_messages": false, + "topics": { + "0": { + "manage_topic": false, + "read_topic": false, + "poll_messages": true, + "send_messages": false + } + } + } + } + }, + "options": {} + } + }, + { + "case": "personal_access_token_without_expiry", + "group": 7, + "body_hex": "0700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "item_hex": "071d0000000700000000b832c8fb4006000863692d746f6b656e0000000000000000", + "expected": { + "group": "personal_access_tokens", + "user_id": 7, + "user_created_at": 1760300000000000, + "name": "ci-token" + } + }, + { + "case": "session", + "group": 8, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "item_hex": "08240000000700000000b832c8fb400600100f0e0d0c0b0a0908070605040302019411000000000000", + "expected": { + "group": "sessions", + "user_id": 7, + "user_created_at": 1760300000000000, + "client_id": "0102030405060708090a0b0c0d0e0f10", + "session": 4500 + } + } + ], + "cursors": [ + { + "case": "zero", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d000000000000000000000000000000000000000000000000000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 0, + "checksum": "00000000000000000000000000000000", + "ordinal": 0 + }, + { + "case": "first_record_of_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAABgQAAAAAAAApcH3A4ueLUwAAAAAAAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000001810000000000000a5c1f7038b9e2d4c000000000000000000000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4120, + "checksum": "00000000000000004c2d9e8b03f7c1a5", + "ordinal": 0 + }, + { + "case": "inside_op", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f2000000000000000001000000", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "op": 4187, + "checksum": "0000000000000000f2a4c6e8b0d1f359", + "ordinal": 1 + }, + { + "case": "every_field_at_maximum", + "text": "mc1___________________________________________________________8", + "hex": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + "log_id": "ffffffffffffffffffffffffffffffff", + "op": 18446744073709551615, + "checksum": "ffffffffffffffffffffffffffffffff", + "ordinal": 4294967295 + } + ], + "page_tokens": [ + { + "case": "streams", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000000101040003000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 1, + "group": "streams", + "last_key_hex": "03000000" + }, + { + "case": "topics", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 3, + "group": "topics", + "last_key_hex": "0300000000000000" + }, + { + "case": "partitions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8DCAADAAAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff0308000300000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "partitions", + "last_key_hex": "0300000000000000" + }, + { + "case": "consumer_groups", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8EDAADAAAAAAAAAAIAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff040c00030000000000000002000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_groups", + "last_key_hex": "030000000000000002000000" + }, + { + "case": "consumer_group_members", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8FEAADAAAAAAAAAAIAAAABAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff05100003000000000000000200000001000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "consumer_group_members", + "last_key_hex": "03000000000000000200000001000000" + }, + { + "case": "users", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAACAGBAAHAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e100000000000002006040007000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 32, + "group": "users", + "last_key_hex": "07000000" + }, + { + "case": "personal_access_tokens", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAGAHDQAHAAAACGNpLXRva2Vu", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e1000000000000060070d00070000000863692d746f6b656e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 96, + "group": "personal_access_tokens", + "last_key_hex": "070000000863692d746f6b656e" + }, + { + "case": "sessions", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAKAICACUEQAAAAAAAA", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000a00808009411000000000000", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 160, + "group": "sessions", + "last_key_hex": "9411000000000000" + }, + { + "case": "longest_token_name", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAP8HBAH______25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubg", + "hex": "d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000ff070401ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e", + "anchor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "required_head": 4190, + "groups_mask": 255, + "group": "personal_access_tokens", + "last_key_hex": "ffffffffff6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e6e" + } + ], + "requests": [ + { + "case": "earliest_every_group", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000", + "http_query": "from=earliest&limit=100" + }, + { + "case": "earliest_many_duplicate_groups", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "streams", + "users" + ], + "since": null, + "until": null, + "limit": 100 + }, + "request_hex": "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000640000000201060000", + "http_query": "from=earliest&groups=streams,users&limit=100" + }, + { + "case": "earliest_since_users_and_tokens", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [ + "users", + "personal_access_tokens" + ], + "since": 1760353200000000, + "until": null, + "limit": 100 + }, + "request_hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001004c2a2b0841060000640000000206070000", + "http_query": "from=earliest&groups=users,personal_access_tokens&since=1760353200000000&limit=100" + }, + { + "case": "latest_streams", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [ + "streams" + ], + "since": null, + "until": null, + "limit": 1 + }, + "request_hex": "01000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000001010000", + "http_query": "from=latest&groups=streams&limit=1" + }, + { + "case": "after_cursor_with_time_range", + "input": { + "start": "after", + "cursor": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA", + "page_token": null, + "groups": [ + "topics", + "streams" + ], + "since": 1760356800000000, + "until": 1760360400000000, + "limit": 500 + }, + "request_hex": "02d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000000100f0bd010941060001009451d809410600f40100000202010000", + "http_query": "after=mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA&groups=streams,topics&since=1760356800000000&until=1760360400000000&limit=500" + }, + { + "case": "current_first_page", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201020000", + "http_query": "from=current&groups=streams,topics&limit=1000" + }, + { + "case": "current_continuation", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 1000 + }, + "request_hex": "0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e80300000201024000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "http_query": "from=current¤t_after=cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA&groups=streams,topics&limit=1000" + } + ], + "responses": [ + { + "case": "changes_page", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001b10000000000000673d9e8b03f7c1a50000000000000000000000000103000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be010941060001040000000101b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465641a1000000000000000000000663d9e8b03f7c1a500000000000000001a00be01094106000001022200000003000000191000000000000040d2bf01094106000c74656c656d657472792d7632011b1000000000000000000000673d9e8b03f7c1a500000000000000001b00be0109410600010400000001032700000003000000191000000000000040d2bf01094106000974656c656d65747279012c10000000000000000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABsQAAAAAAAAZz2eiwP3waUAAAAAAAAAAAAAAAA", + "has_more": true, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_created", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "op": 4122, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d66", + "timestamp": 1760356800004122, + "group": "streams", + "kind": "stream_updated", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry-v2", + "options": null, + "options_unreadable": true + }, + { + "op": 4123, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d67", + "timestamp": 1760356800004123, + "actor_user_id": 4, + "group": "streams", + "kind": "stream_deleted", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "requested_op": 4140 + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "unknown_kind_and_group", + "response_hex": "000001d7931c6e2a8f4b0d00000000000000001f0f00000000000000c0a3e3074106005e1000000000000001d7931c6e2a8f4b0d00000000000000001910000000000000653d9e8b03f7c1a50000000000000000010000000002000000191000000000000000000000653d9e8b03f7c1a500000000000000001900be01094106000104000000010904000000deadbeef191000000000000001000000653d9e8b03f7c1a500000000000000001900be01094106000104000000c80104000000deadbeef000000000000", + "expected": { + "status": "ok", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 3871, + "retained_since": 1760352000000000, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAABkQAAAAAAAAZT2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": false, + "records": [ + { + "op": 4121, + "ordinal": 0, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "1", + "kind": "9", + "body": "3q2+7w==" + }, + { + "op": 4121, + "ordinal": 1, + "checksum": "0000000000000000a5c1f7038b9e3d65", + "timestamp": 1760356800004121, + "actor_user_id": 4, + "group": "200", + "kind": "1", + "body": "3q2+7w==" + } + ], + "items": [], + "current_page_token": null + } + }, + { + "case": "current_page", + "response_hex": "000101d7931c6e2a8f4b0d0000000000000000010000000000000000000000000000005e1000000000000001d7931c6e2a8f4b0d00000000000000005b10000000000000a73d9e8b03f7c1a500000000000000000100000001000000000200000001b700000003000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e6572617465640274000000030000001910000000000000000000001d100000000000004032cd01094106000c6e6f64652d6d6574726963730020000000020e0000006d6573736167655f6578706972790c0800000000a0e3d08c0000001e000000020c0000007365676d656e745f73697a650c0800000000000040000000004000d7931c6e2a8f4b0d00000000000000005b1000000000000059f3d1b0e8c6a4f20000000000000000010000005e10000000000000030208000300000000000000", + "expected": { + "status": "ok", + "phase": "current", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 1, + "retained_since": 0, + "head_op": 4190, + "next": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAApz2eiwP3waUAAAAAAAAAAAEAAAA", + "has_more": true, + "records": [], + "items": [ + { + "group": "streams", + "stream_id": 3, + "stream_created_op": 4121, + "created_at": 1760356800123456, + "name": "telemetry", + "options": { + "encryption_kek_fingerprint": { + "value": "51d0e3a9c27b6f48", + "kind": "string", + "explicit": false + }, + "encryption_key_fingerprint": { + "value": "9f2c1a7e44b0d816", + "kind": "string", + "explicit": false + }, + "encryption_key_source": { + "value": "generated", + "kind": "string", + "explicit": false + } + } + }, + { + "group": "topics", + "stream_id": 3, + "stream_created_op": 4121, + "topic_id": 0, + "topic_created_op": 4125, + "created_at": 1760356801000000, + "name": "node-metrics", + "options": { + "message_expiry": { + "value": "604800000000", + "kind": "uint64", + "explicit": true + }, + "segment_size": { + "value": "1073741824", + "kind": "uint64", + "explicit": false + } + } + } + ], + "current_page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAMCCAADAAAAAAAAAA" + } + }, + { + "case": "behind_before_op_1", + "response_hex": "010000000000000000000000000000000000000000000000000000000000000000000000000000", + "expected": { + "status": "behind", + "phase": "changes", + "log_id": null, + "oldest_op": 0, + "retained_since": 0, + "head_op": 0, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "expired", + "response_hex": "020001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "expired", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + }, + { + "case": "reset", + "response_hex": "030001d7931c6e2a8f4b0d0000000000000000881300000000000000107ac0094106002823000000000000000000000000000000000000", + "expected": { + "status": "reset", + "phase": "changes", + "log_id": "00000000000000000d4b8f2a6e1c93d7", + "oldest_op": 5000, + "retained_since": 1760360000000000, + "head_op": 9000, + "next": null, + "has_more": false, + "records": [], + "items": [], + "current_page_token": null + } + } + ], + "large_values": { + "record_hex": "0300000000002000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff01ffffffff084624000000ffffffff0100000000002000feffffffffffffffffffffffffffffff0200000000002000", + "op": "9007199254740995", + "ordinal": 4294967295, + "timestamp": "18446744073709551615", + "checksum": "ffffffffffffffffffffffffffffffff", + "user_created_at": "9007199254740993", + "session": "9007199254740994", + "client_id": "fffffffffffffffffffffffffffffffe" + }, + "refused": { + "cursors": [ + { + "case": "wrong_prefix", + "text": "mc2_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "missing_prefix", + "text": "15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "too_long", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAAA" + }, + { + "case": "too_short", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA" + }, + { + "case": "padded", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAA=" + }, + { + "case": "standard_alphabet_character", + "text": "mc1_+5McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAA" + }, + { + "case": "op_zero_with_checksum", + "text": "mc1_15McbiqPSw0AAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAA" + } + ], + "page_tokens": [ + { + "case": "wrong_prefix", + "text": "mc1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA" + }, + { + "case": "trailing_byte", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAAAA" + }, + { + "case": "wrong_key_width", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBCAADAAAAAAAAAA" + }, + { + "case": "group_outside_mask", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAIBBAADAAAA" + }, + { + "case": "head_below_anchor", + "text": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAAABAAAAAAAAAAEBBAADAAAA" + }, + { + "case": "empty", + "text": "cs1_" + } + ], + "record_bodies": [ + { + "case": "unknown_member_left_reason", + "group": 5, + "kind": 41, + "body_hex": "030000001910000000000000000000001d100000000000000000000001000000100f0e0d0c0b0a0908070605040302010100000002" + }, + { + "case": "presence_byte_two", + "group": 1, + "kind": 3, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279022c10000000000000" + }, + { + "case": "unsorted_removed_partition_ids", + "group": 3, + "kind": 20, + "body_hex": "030000001910000000000000000000001d100000000000000500000003000000090000000600000007000000016810000000000000" + }, + { + "case": "known_kind_under_wrong_group", + "group": 2, + "kind": 1, + "body_hex": "03000000191000000000000040d2bf01094106000974656c656d65747279000000000090000000021a000000656e6372797074696f6e5f6b656b5f66696e6765727072696e74021000000035316430653361396332376236663438021a000000656e6372797074696f6e5f6b65795f66696e6765727072696e740210000000396632633161376534346230643831360215000000656e6372797074696f6e5f6b65795f736f75726365020900000067656e657261746564" + }, + { + "case": "trailing_byte", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a090807060504030201941100000000000000" + }, + { + "case": "truncated", + "group": 8, + "kind": 70, + "body_hex": "0700000000b832c8fb400600100f0e0d0c0b0a09080706050403020194110000000000" + } + ], + "requests": [ + { + "case": "cursor_bytes_without_after", + "hex": "000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "trailing_byte", + "hex": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006400000000000000" + }, + { + "case": "unknown_start_mode", + "hex": "040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000064000000000000" + }, + { + "case": "non_canonical_cursor", + "hex": "02d7931c6e2a8f4b0d000000000000000000000000000000000100000000000000000000000000000000000000000064000000000000" + } + ], + "responses": [ + { + "case": "items_in_changes_phase", + "hex": "000001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + }, + { + "case": "unknown_status", + "hex": "040001d7931c6e2a8f4b0d000000000000000001000000000000000000000000000000020000000000000001d7931c6e2a8f4b0d000000000000000002000000000000004e2d9e8b03f7c1a500000000000000000000000000000000000100000001240000000300000019100000000000000100000000000000066f72646572730000000000000000000000" + } + ], + "queries": [ + { + "case": "limit_zero", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 0 + } + }, + { + "case": "limit_above_maximum", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": null, + "limit": 1001 + } + }, + { + "case": "since_above_until", + "input": { + "start": "earliest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 2, + "until": 1, + "limit": 100 + } + }, + { + "case": "time_bound_with_latest", + "input": { + "start": "latest", + "cursor": null, + "page_token": null, + "groups": [], + "since": 1, + "until": null, + "limit": 100 + } + }, + { + "case": "time_bound_with_current", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [], + "since": null, + "until": 1, + "limit": 100 + } + }, + { + "case": "current_missing_parent_group", + "input": { + "start": "current", + "cursor": null, + "page_token": null, + "groups": [ + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + }, + { + "case": "page_token_for_other_groups", + "input": { + "start": "current", + "cursor": null, + "page_token": "cs1_15McbiqPSw0AAAAAAAAAAFsQAAAAAAAAWfPRsOjGpPIAAAAAAAAAAAEAAABeEAAAAAAAAAEBBAADAAAA", + "groups": [ + "streams", + "topics" + ], + "since": null, + "until": null, + "limit": 100 + } + } + ], + "http_queries": [ + { + "case": "no_start", + "query": "groups=streams" + }, + { + "case": "from_and_after", + "query": "from=earliest&after=mc1_x" + }, + { + "case": "unknown_from", + "query": "from=newest" + }, + { + "case": "duplicate_key", + "query": "from=earliest&from=latest" + }, + { + "case": "unknown_key", + "query": "from=earliest&verbose=1" + }, + { + "case": "empty_group_element", + "query": "from=earliest&groups=streams,,topics" + }, + { + "case": "trailing_group_comma", + "query": "from=earliest&groups=streams," + }, + { + "case": "unknown_group", + "query": "from=earliest&groups=everything" + }, + { + "case": "negative_since", + "query": "from=earliest&since=-1" + }, + { + "case": "leading_zero_until", + "query": "from=earliest&until=01" + }, + { + "case": "non_numeric_limit", + "query": "from=earliest&limit=ten" + }, + { + "case": "token_without_current", + "query": "from=latest¤t_after=cs1_AAAA" + } + ] + } +}
diff --git a/foreign/swift/Tests/IggyTests/MetadataChangesTests.swift b/foreign/swift/Tests/IggyTests/MetadataChangesTests.swift new file mode 100644 index 0000000..b2b6e48 --- /dev/null +++ b/foreign/swift/Tests/IggyTests/MetadataChangesTests.swift
@@ -0,0 +1,678 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import Foundation +import Testing + +@testable import Iggy + +/// The metadata changes contract every SDK shares, checked against +/// `Fixtures/metadata_changes.json` from the Rust codecs. The Swift encoder +/// and decoders are independent, so these bytes are the contract. Refused +/// request bytes and HTTP queries are left out: this SDK never decodes a +/// request and has no HTTP transport. +@Suite("Metadata changes match the shared fixture") +struct MetadataChangesFixtureTests { + let fixture = MetadataChangesFixture.shared + + @Test func givenManyDuplicateGroupsWhenEncodingShouldPreserveEveryRequestedGroup() { + let groups = Array(repeating: MetadataChangeGroup.streams, count: 255) + [.users] + let query = MetadataChangesQuery(groups: groups) + let expected = MetadataChangesQuery(groups: [.streams, .users]) + #expect(Requests.getMetadataChanges(query) == Requests.getMetadataChanges(expected)) + } + + @Test func givenFixtureWhenLoadingShouldHoldEveryVectorSection() { + for section in ["records", "items", "cursors", "page_tokens", "requests", "responses"] { + #expect(!fixture[section].array.isEmpty, "\(section)") + } + for section in ["cursors", "page_tokens", "record_bodies", "responses", "queries"] { + #expect(!fixture["refused"][section].array.isEmpty, "refused \(section)") + } + } + + @Test func givenFixtureConstantsWhenComparingPublicTypesShouldMatchCodesAndNames() throws { + let constants = fixture["constants"] + #expect(fixture["schema_version"].number == 1) + let command = try #require(CommandCode(rawValue: UInt32(constants["command_code"].number))) + #expect(command == .getMetadataChanges) + #expect(command.name == constants["command_name"].string) + #expect(ConsensusOperation.forCode(command.rawValue) == .nonReplicated) + + #expect(Set(constants["groups"].members.keys) == Set(MetadataChangeGroup.allCases.map(\.description))) + for (name, entry) in constants["groups"].members { + let group = try #require(MetadataChangeGroup(name: name), "\(name)") + #expect(UInt64(group.rawValue) == entry["code"].number, "\(name)") + #expect(group.parents.map(\.description) == entry["parents"].array.map(\.string), "\(name)") + } + #expect(Set(constants["kinds"].members.keys) == Set(MetadataChangeKind.allCases.map(\.description))) + for (name, entry) in constants["kinds"].members { + let kind = try #require(MetadataChangeKind(name: name), "\(name)") + #expect(UInt64(kind.rawValue) == entry["code"].number, "\(name)") + #expect(kind.group.description == entry["group"].string, "\(name)") + } + expectCodes(MetadataChangeStatus.self, constants["statuses"]) + expectCodes(MetadataChangePhase.self, constants["phases"]) + expectCodes(MetadataMemberLeftReason.self, constants["member_left_reasons"]) + expectCodes(MetadataTokenDeletedReason.self, constants["token_deleted_reasons"]) + expectCodes(MetadataSessionClosedReason.self, constants["session_closed_reasons"]) + expectCodes(MetadataPermissionsUpdatedReason.self, constants["permissions_updated_reasons"]) + let startModes: [String: MetadataChangeStart] = [ + "earliest": .earliest, "latest": .latest, "after": .after(.zero(logID: 1)), "current": .current(nil), + ] + #expect(Set(constants["start_modes"].members.keys) == Set(startModes.keys)) + for (name, start) in startModes { + #expect(UInt64(start.code) == constants["start_modes"][name].number, "\(name)") + } + + #expect(MetadataChangeCursor.textPrefix == constants["cursor_prefix"].string) + #expect(MetadataCurrentPageToken.textPrefix == constants["page_token_prefix"].string) + #expect(UInt64(MetadataChangeCursor.encodedSize) == constants["cursor_size"].number) + #expect(UInt64(MetadataChangeCursor.textLength) == constants["cursor_text_length"].number) + #expect(UInt64(MetadataCurrentPageToken.maxEncodedSize) == constants["max_page_token_size"].number) + #expect(UInt64(MetadataCurrentPageToken.maxKeySize) == constants["max_page_token_key_size"].number) + #expect(UInt64(MetadataChangesQuery.defaultLimit) == constants["default_limit"].number) + #expect(UInt64(MetadataChangesQuery.maxLimit) == constants["max_limit"].number) + #expect(MetadataChangesQuery().limit == MetadataChangesQuery.defaultLimit) + } + + @Test func givenRecordVectorsWhenDecodingShouldReachTheExpectedJsonForEveryKind() throws { + var kinds = Set<MetadataChangeKind>() + for vector in fixture["records"].array { + let name = vector["case"].string + var reader = ByteReader([UInt8](hex: vector["record_hex"].string)) + let record = try Responses.metadataChangeRecord(from: &reader) + #expect(reader.isAtEnd, "\(name)") + #expect(json(record) == vector["expected"], "\(name)") + let kind = try #require(record.change.kind, "\(name)") + #expect(UInt64(kind.rawValue) == vector["kind"].number, "\(name)") + #expect(UInt64(kind.group.rawValue) == vector["group"].number, "\(name)") + let body = try MetadataChange.decode(group: kind.group.rawValue, kind: kind.rawValue, body: [UInt8](hex: vector["body_hex"].string)[...]) + #expect(body == record.change, "\(name)") + kinds.insert(kind) + } + #expect(kinds == Set(MetadataChangeKind.allCases)) + } + + @Test func givenItemVectorsWhenDecodingShouldReachTheExpectedJsonForEveryGroup() throws { + var groups = Set<MetadataChangeGroup>() + for vector in fixture["items"].array { + let name = vector["case"].string + var reader = ByteReader([UInt8](hex: vector["item_hex"].string)) + let item = try Responses.metadataItem(from: &reader) + #expect(reader.isAtEnd, "\(name)") + #expect(json(item) == vector["expected"], "\(name)") + let group = try #require(item.group, "\(name)") + #expect(UInt64(group.rawValue) == vector["group"].number, "\(name)") + #expect(try MetadataCurrentItem.decode(group: group.rawValue, body: [UInt8](hex: vector["body_hex"].string)[...]) == item, "\(name)") + groups.insert(group) + } + #expect(groups == Set(MetadataChangeGroup.allCases)) + } + + @Test func givenResponseVectorsWhenDecodingShouldReachTheExpectedAnswers() throws { + for vector in fixture["responses"].array { + let answer = try Responses.metadataChanges([UInt8](hex: vector["response_hex"].string)[...]) + #expect(json(answer) == vector["expected"], "\(vector["case"].string)") + } + } + + @Test func givenUnknownKindAndGroupWhenDecodingShouldKeepCodesAndRawBody() throws { + let vector = try #require(fixture["responses"].array.first { $0["case"].string == "unknown_kind_and_group" }) + let answer = try Responses.metadataChanges([UInt8](hex: vector["response_hex"].string)[...]) + #expect( + answer.records.map(\.change) == [ + .unknown(group: 1, kind: 9, body: [0xde, 0xad, 0xbe, 0xef]), .unknown(group: 200, kind: 1, body: [0xde, 0xad, 0xbe, 0xef]), + ]) + #expect(answer.records.allSatisfy { $0.change.kind == nil && $0.change.group == nil }) + #expect(try MetadataCurrentItem.decode(group: 9, body: [1, 2][...]) == .unknown(group: 9, body: [1, 2])) + } + + @Test func givenLargeValuesWhenDecodingShouldKeepEveryBit() throws { + let vector = fixture["large_values"] + var reader = ByteReader([UInt8](hex: vector["record_hex"].string)) + let record = try Responses.metadataChangeRecord(from: &reader) + #expect(String(record.op) == vector["op"].string) + #expect(UInt64(record.ordinal) == vector["ordinal"].number) + #expect(String(record.timestamp.microseconds) == vector["timestamp"].string) + #expect(hex(record.checksum) == vector["checksum"].string) + guard case .sessionRegistered(let session) = record.change else { + Issue.record("expected a session record, got \(record.change)") + return + } + #expect(String(session.userCreatedAt.microseconds) == vector["user_created_at"].string) + #expect(String(session.session) == vector["session"].string) + #expect(hex(session.clientID) == vector["client_id"].string) + } + + @Test func givenCursorVectorsWhenParsingShouldMatchTheirFieldsBytesAndText() throws { + for vector in fixture["cursors"].array { + let name = vector["case"].string + let cursor = try MetadataChangeCursor(text: vector["text"].string) + #expect(hex(cursor.logID) == vector["log_id"].string, "\(name)") + #expect(cursor.op == vector["op"].number, "\(name)") + #expect(hex(cursor.checksum) == vector["checksum"].string, "\(name)") + #expect(UInt64(cursor.ordinal) == vector["ordinal"].number, "\(name)") + #expect(cursor.isZero == (cursor.op == 0), "\(name)") + #expect(cursor.description == vector["text"].string, "\(name)") + var writer = ByteWriter() + cursor.encode(into: &writer) + #expect(writer.bytes == [UInt8](hex: vector["hex"].string), "\(name)") + } + } + + @Test func givenPageTokenVectorsWhenParsingShouldMatchTheirFieldsBytesAndText() throws { + for vector in fixture["page_tokens"].array { + let name = vector["case"].string + let token = try MetadataCurrentPageToken(text: vector["text"].string) + #expect(token.description == vector["text"].string, "\(name)") + #expect(token.encoded() == [UInt8](hex: vector["hex"].string), "\(name)") + #expect(token.anchor.description == vector["anchor"].string, "\(name)") + #expect(token.requiredHead == vector["required_head"].number, "\(name)") + #expect(UInt64(token.groupsMask) == vector["groups_mask"].number, "\(name)") + #expect(token.groups == Set(MetadataChangeGroup.allCases.filter { UInt64($0.maskBit) & vector["groups_mask"].number != 0 }), "\(name)") + #expect(token.lastGroup.description == vector["group"].string, "\(name)") + #expect(token.lastKey == [UInt8](hex: vector["last_key_hex"].string), "\(name)") + } + } + + @Test func givenRequestVectorsWhenEncodingShouldValidateAndMatchTheRequestBytes() throws { + for vector in fixture["requests"].array { + let query = try query(vector["input"]) + try query.validate() + #expect(Requests.getMetadataChanges(query) == [UInt8](hex: vector["request_hex"].string), "\(vector["case"].string)") + } + } + + @Test func givenRefusedCursorAndTokenTextWhenParsingShouldRefuseWithoutTheText() { + for vector in fixture["refused"]["cursors"].array { + expectRefusedText(vector["text"].string, vector["case"].string) { try MetadataChangeCursor(text: $0) } + } + for vector in fixture["refused"]["page_tokens"].array { + expectRefusedText(vector["text"].string, vector["case"].string) { try MetadataCurrentPageToken(text: $0) } + } + } + + @Test func givenRefusedRecordBodiesWhenDecodingShouldRefuseEach() { + for vector in fixture["refused"]["record_bodies"].array { + let body = [UInt8](hex: vector["body_hex"].string) + #expect(throws: (any Error).self, "\(vector["case"].string)") { + try MetadataChange.decode(group: UInt8(vector["group"].number), kind: UInt8(vector["kind"].number), body: body[...]) + } + } + } + + @Test func givenRefusedResponsesWhenDecodingShouldRefuseEach() { + for vector in fixture["refused"]["responses"].array { + #expect(throws: IggyError(.invalidCommand), "\(vector["case"].string)") { + try Responses.metadataChanges([UInt8](hex: vector["hex"].string)[...]) + } + } + } + + @Test func givenRefusedQueriesWhenValidatingShouldRefuseWithInvalidCommand() throws { + for vector in fixture["refused"]["queries"].array { + let query = try query(vector["input"]) + #expect(throws: IggyError(.invalidCommand), "\(vector["case"].string)") { try query.validate() } + } + } + + // MARK: Helpers + + func expectCodes<Code: RawRepresentable & CaseIterable & CustomStringConvertible>( + _: Code.Type, _ table: JSONValue, sourceLocation: SourceLocation = #_sourceLocation + ) where Code.RawValue == UInt8 { + let expected = table.members.mapValues(\.number) + let actual = Dictionary(uniqueKeysWithValues: Code.allCases.map { ($0.description, UInt64($0.rawValue)) }) + #expect(actual == expected, sourceLocation: sourceLocation) + } + + func expectRefusedText(_ text: String, _ name: String, _ parse: (String) throws -> some Any) { + do { + _ = try parse(text) + Issue.record("\(name) parsed") + } catch let error as IggyError { + #expect(error.code == .invalidCommand, "\(name)") + #expect(!"\(error)".contains(text), "\(name)") + #expect(!String(reflecting: error).contains(text), "\(name)") + } catch { + Issue.record("\(name): \(error)") + } + } + + func query(_ input: JSONValue) throws -> MetadataChangesQuery { + let start: MetadataChangeStart + switch input["start"].string { + case "earliest": start = .earliest + case "latest": start = .latest + case "after": start = .after(try MetadataChangeCursor(text: input["cursor"].string)) + default: start = .current(input["page_token"].isNull ? nil : try MetadataCurrentPageToken(text: input["page_token"].string)) + } + return MetadataChangesQuery( + start: start, groups: try input["groups"].array.map { try #require(MetadataChangeGroup(name: $0.string)) }, + since: input["since"].isNull ? nil : IggyTimestamp(microseconds: input["since"].number), + until: input["until"].isNull ? nil : IggyTimestamp(microseconds: input["until"].number), limit: UInt32(input["limit"].number)) + } +} + +/// The Swift codecs and the client call on their own: query rules, the text +/// forms, and a round trip through a scripted server. +@Suite("Metadata changes codecs and client", .serialized) +struct MetadataChangesTests { + let cursor = MetadataChangeCursor(logID: UInt128Value(low: 7, high: 9), op: 4187, checksum: UInt128Value(low: 11, high: 13), ordinal: 1) + + @Test func givenQueryRulesWhenValidatingShouldRefuseOnlyInvalidQueries() throws { + let refused: [MetadataChangesQuery] = [ + MetadataChangesQuery(limit: 0), + MetadataChangesQuery(limit: MetadataChangesQuery.maxLimit + 1), + MetadataChangesQuery(since: IggyTimestamp(microseconds: 5), until: IggyTimestamp(microseconds: 4)), + MetadataChangesQuery(start: .latest, until: IggyTimestamp(microseconds: 1)), + MetadataChangesQuery(start: .current(nil), since: IggyTimestamp(microseconds: 1)), + MetadataChangesQuery(start: .current(nil), groups: [.consumerGroupMembers, .consumerGroups, .topics, .streams]), + MetadataChangesQuery(start: .after(MetadataChangeCursor(logID: 1, op: 0, checksum: 0, ordinal: 3))), + ] + for query in refused { + #expect(throws: IggyError(.invalidCommand), "\(query)") { try query.validate() } + } + let accepted: [MetadataChangesQuery] = [ + MetadataChangesQuery(), + MetadataChangesQuery(limit: 1), + MetadataChangesQuery(limit: MetadataChangesQuery.maxLimit), + MetadataChangesQuery(start: .after(cursor), until: IggyTimestamp(microseconds: 1)), + MetadataChangesQuery(start: .after(.zero(logID: 1))), + MetadataChangesQuery(start: .current(nil), groups: [.sessions, .users]), + MetadataChangesQuery(start: .earliest, groups: [.topics]), + ] + for query in accepted { + try query.validate() + } + } + + @Test func givenPageTokenWhenGroupsDifferShouldRefuseTheContinuation() throws { + let token = MetadataCurrentPageToken(anchor: cursor, requiredHead: 4200, groupsMask: 0b11, lastGroup: .topics, lastKey: [3, 0, 0, 0, 1, 0, 0, 0]) + let parsed = try MetadataCurrentPageToken(text: token.description) + #expect(parsed == token) + #expect(parsed.groups == [.streams, .topics]) + try MetadataChangesQuery(start: .current(parsed), groups: [.topics, .streams]).validate() + #expect(throws: IggyError(.invalidCommand)) { try MetadataChangesQuery(start: .current(parsed)).validate() } + #expect(throws: IggyError(.invalidCommand)) { try MetadataChangesQuery(start: .current(parsed), groups: [.streams]).validate() } + + var request = ByteReader(Requests.getMetadataChanges(MetadataChangesQuery(start: .current(parsed), groups: [.streams, .topics]))) + #expect(try request.readUInt8() == 3) + #expect(try request.readBytes(MetadataChangeCursor.encodedSize).allSatisfy { $0 == 0 }) + try request.skip(1 + 1 + 4) + #expect(Array(try request.readBytes(Int(try request.readUInt8()))) == [1, 2]) + #expect(Array(try request.readBytes(Int(try request.readUInt16()))) == token.encoded()) + #expect(request.isAtEnd) + } + + @Test func givenCursorTextWhenTamperingShouldRefuseEveryNonCanonicalForm() throws { + let text = cursor.description + #expect(text.count == MetadataChangeCursor.textLength) + #expect(try MetadataChangeCursor(text: text) == cursor) + let zero = MetadataChangeCursor.zero(logID: cursor.logID) + #expect(try MetadataChangeCursor(text: zero.description).isZero) + + let alphabet = Array("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_") + let last = try #require(alphabet.firstIndex(of: text.last!)) + let encoded = String(text.dropFirst(4)) + let tampered = [ + "mc2_" + encoded, encoded, text + "A", String(text.dropLast()), "mc1_" + encoded.dropLast() + "=", + text.replacingOccurrences(of: "_", with: "/"), String(text.dropLast()) + String(alphabet[last ^ 1]), + MetadataChangeCursor(logID: 1, op: 0, checksum: 2, ordinal: 0).description, + ] + for candidate in tampered { + #expect(throws: IggyError(.invalidCommand), "\(candidate)") { try MetadataChangeCursor(text: candidate) } + } + } + + @Test func givenRandomBytesWhenEncodingBase64URLShouldMatchTheStandardAlphabetUnpadded() { + var generator = SystemRandomNumberGenerator() + for length in 0..<70 { + let bytes = (0..<length).map { _ in UInt8.random(in: .min ... .max, using: &generator) } + let expected = Data(bytes).base64EncodedString().replacingOccurrences(of: "+", with: "-").replacingOccurrences(of: "/", with: "_") + .replacingOccurrences(of: "=", with: "") + let encoded = Base64URL.encode(bytes) + #expect(encoded == expected, "\(length)") + #expect(Base64URL.decode("cs1_" + encoded, prefix: "cs1_") == bytes, "\(length)") + } + #expect(Base64URL.decode("cs1_A", prefix: "cs1_") == nil) + #expect(Base64URL.decode("cs1_AA==", prefix: "cs1_") == nil) + #expect(Base64URL.decode("cs1_+w", prefix: "cs1_") == nil) + } + + @Test func givenScriptedServerWhenReadingChangesShouldSendCode17AndDecodeTheReply() async throws { + let reply = GoldenFixture.shared.bytes("response.get_metadata_changes") + let server = try await FakeVSRServer.started() + defer { Task { await server.stop() } } + await server.respond { request in + if request.code == CommandCode.getMetadataChanges.rawValue { + return .ok(reply) + } + return FakeVSRServer.defaultReply(request, address: server.address) + } + var configuration = ClientConfiguration(address: server.address, autoLogin: .usernamePassword(username: "iggy", password: "iggy")) + configuration.connectTimeout = .seconds(2) + configuration.heartbeatInterval = .seconds(60) + let client = IggyClient(configuration: configuration, connector: NIOConnector(), logger: testLogger) + try await client.connect() + + let query = MetadataChangesQuery(start: .after(cursor), groups: [.streams], limit: 10) + let page = try await client.getMetadataChanges(query) + #expect(page == (try Responses.metadataChanges(reply[...]))) + let sent = await server.requests.filter { $0.code == CommandCode.getMetadataChanges.rawValue } + #expect(sent.map(\.payload) == [Requests.getMetadataChanges(query)]) + #expect(sent.allSatisfy { $0.operation == .nonReplicated }) + + // A refused query never reaches the connection. + await expectCode(.invalidCommand) { _ = try await client.getMetadataChanges(MetadataChangesQuery(limit: 0)) } + await expectCode(.invalidCommand) { _ = try await client.getMetadataChanges(MetadataChangesQuery(start: .latest, since: .zero)) } + #expect(await server.requests.filter { $0.code == CommandCode.getMetadataChanges.rawValue }.count == 1) + await client.shutdown() + } +} + +/// The 32 lowercase hex characters the fixture renders a 128-bit value as. +func hex(_ value: UInt128Value) -> String { + String(format: "%016llx%016llx", value.high, value.low) +} + +/// The JSON the HTTP route renders for a record, so a decoded record compares +/// whole with the fixture's `expected` object. +func json(_ record: MetadataChangeRecord) -> JSONValue { + var members: [String: JSONValue] = [ + "op": .number(record.op), "ordinal": .number(UInt64(record.ordinal)), "checksum": .string(hex(record.checksum)), + "timestamp": .number(record.timestamp.microseconds), + ] + if let actor = record.actorUserID { + members["actor_user_id"] = .number(UInt64(actor)) + } + if let kind = record.change.kind { + members["group"] = .string(kind.group.description) + members["kind"] = .string(kind.description) + } + members.merge(body(record.change)) { _, body in body } + return .object(members) +} + +func json(_ item: MetadataCurrentItem) -> JSONValue { + var members = body(item) + if let group = item.group { + members["group"] = .string(group.description) + } + return .object(members) +} + +func json(_ answer: MetadataChanges) -> JSONValue { + .object([ + "status": .string(answer.status.description), "phase": .string(answer.phase.description), "log_id": answer.logID.map { .string(hex($0)) } ?? .null, + "oldest_op": .number(answer.oldestOp), "retained_since": .number(answer.retainedSince.microseconds), "head_op": .number(answer.headOp), + "next": answer.next.map { .string($0.description) } ?? .null, "has_more": .bool(answer.hasMore), "records": .array(answer.records.map(json)), + "items": .array(answer.items.map(json)), "current_page_token": answer.currentPageToken.map { .string($0.description) } ?? .null, + ]) +} + +private func body(_ change: MetadataChange) -> [String: JSONValue] { + switch change { + case .streamCreated(let body), .streamUpdated(let body): stream(body) + case .streamDeleted(let body): + optional( + streamKey(body.streamID, body.streamCreatedOp).merging(["created_at": .number(body.createdAt.microseconds), "name": .string(body.name)]) { $1 }, + "requested_op", body.requestedOp) + case .topicCreated(let body), .topicUpdated(let body): + options( + topicKey(body.streamID, body.streamCreatedOp, body.topicID, body.topicCreatedOp).merging([ + "created_at": .number(body.createdAt.microseconds), "name": .string(body.name), "partitions_count": .number(UInt64(body.partitionsCount)), + ]) { $1 }, body.options) + case .topicDeleted(let body): + optional( + topicKey(body.streamID, body.streamCreatedOp, body.topicID, body.topicCreatedOp).merging([ + "created_at": .number(body.createdAt.microseconds), "name": .string(body.name), + ]) { $1 }, "requested_op", body.requestedOp) + case .partitionsChanged(let body): + optional( + topicKey(body.streamID, body.streamCreatedOp, body.topicID, body.topicCreatedOp).merging([ + "partitions_count": .number(UInt64(body.partitionsCount)), "removed_partition_ids": ids(body.removedPartitionIDs), + ]) { $1 }, "requested_op", body.requestedOp) + case .consumerGroupCreated(let body): + topicKey(body.streamID, body.streamCreatedOp, body.topicID, body.topicCreatedOp).merging([ + "group_id": .number(UInt64(body.groupID)), "name": .string(body.name), "consumer_groups_count": .number(UInt64(body.consumerGroupsCount)), + ]) { $1 } + case .consumerGroupDeleted(let body): + optional( + topicKey(body.streamID, body.streamCreatedOp, body.topicID, body.topicCreatedOp).merging([ + "group_id": .number(UInt64(body.groupID)), "name": .string(body.name), "consumer_groups_count": .number(UInt64(body.consumerGroupsCount)), + ]) { $1 }, "requested_op", body.requestedOp) + case .consumerGroupMemberJoined(let body): + topicKey(body.streamID, body.streamCreatedOp, body.topicID, body.topicCreatedOp).merging([ + "group_id": .number(UInt64(body.groupID)), "member_id": .number(UInt64(body.memberID)), "client_id": .string(hex(body.clientID)), + "members_count": .number(UInt64(body.membersCount)), "partition_ids": ids(body.partitionIDs), + ]) { $1 } + case .consumerGroupMemberLeft(let body): + topicKey(body.streamID, body.streamCreatedOp, body.topicID, body.topicCreatedOp).merging([ + "group_id": .number(UInt64(body.groupID)), "member_id": .number(UInt64(body.memberID)), "client_id": .string(hex(body.clientID)), + "members_count": .number(UInt64(body.membersCount)), "reason": .string(body.reason.description), + ]) { $1 } + case .consumerGroupPartitionHandedOver(let body): + optional( + topicKey(body.streamID, body.streamCreatedOp, body.topicID, body.topicCreatedOp).merging([ + "group_id": .number(UInt64(body.groupID)), "partition_id": .number(UInt64(body.partitionID)), + ]) { $1 }, "member_id", body.memberID.map(UInt64.init)) + case .userCreated(let body), .userUpdated(let body): user(body) + case .userDeleted(let body), .userPasswordChanged(let body): + userKey(body.userID, body.userCreatedAt).merging(["username": .string(body.username)]) { $1 } + case .userPermissionsUpdated(let body): + userKey(body.userID, body.userCreatedAt).merging([ + "username": .string(body.username), "permissions": permissions(body.permissions), "reason": .string(body.reason.description), + ]) { $1 } + case .personalAccessTokenCreated(let body): token(body) + case .personalAccessTokenDeleted(let body): + userKey(body.userID, body.userCreatedAt).merging(["name": .string(body.name), "reason": .string(body.reason.description)]) { $1 } + case .sessionRegistered(let body): session(body) + case .sessionClosed(let body): + session(MetadataSession(userID: body.userID, userCreatedAt: body.userCreatedAt, clientID: body.clientID, session: body.session)).merging([ + "reason": .string(body.reason.description) + ]) { $1 } + case .unknown(let group, let kind, let body): + ["group": .string(String(group)), "kind": .string(String(kind)), "body": .string(Data(body).base64EncodedString())] + } +} + +private func body(_ item: MetadataCurrentItem) -> [String: JSONValue] { + switch item { + case .stream(let body): stream(body) + case .topic(let body): + options( + topicKey(body.streamID, body.streamCreatedOp, body.topicID, body.topicCreatedOp).merging([ + "created_at": .number(body.createdAt.microseconds), "name": .string(body.name), + ]) { $1 }, body.options) + case .partitions(let body): + topicKey(body.streamID, body.streamCreatedOp, body.topicID, body.topicCreatedOp).merging(["partitions_count": .number(UInt64(body.partitionsCount))]) { + $1 + } + case .consumerGroup(let body): + topicKey(body.streamID, body.streamCreatedOp, body.topicID, body.topicCreatedOp).merging([ + "group_id": .number(UInt64(body.groupID)), "name": .string(body.name), + ]) { $1 } + case .consumerGroupMember(let body): + topicKey(body.streamID, body.streamCreatedOp, body.topicID, body.topicCreatedOp).merging([ + "group_id": .number(UInt64(body.groupID)), "member_id": .number(UInt64(body.memberID)), "client_id": .string(hex(body.clientID)), + "partition_ids": ids(body.partitionIDs), + ]) { $1 } + case .user(let body): user(body) + case .personalAccessToken(let body): token(body) + case .session(let body): session(body) + case .unknown(let group, let body): ["group": .string(String(group)), "body": .string(Data(body).base64EncodedString())] + } +} + +private func streamKey(_ streamID: UInt32, _ streamCreatedOp: UInt64) -> [String: JSONValue] { + ["stream_id": .number(UInt64(streamID)), "stream_created_op": .number(streamCreatedOp)] +} + +private func topicKey(_ streamID: UInt32, _ streamCreatedOp: UInt64, _ topicID: UInt32, _ topicCreatedOp: UInt64) -> [String: JSONValue] { + streamKey(streamID, streamCreatedOp).merging(["topic_id": .number(UInt64(topicID)), "topic_created_op": .number(topicCreatedOp)]) { $1 } +} + +private func userKey(_ userID: UInt32, _ userCreatedAt: IggyTimestamp) -> [String: JSONValue] { + ["user_id": .number(UInt64(userID)), "user_created_at": .number(userCreatedAt.microseconds)] +} + +private func stream(_ body: MetadataStreamChange) -> [String: JSONValue] { + options( + streamKey(body.streamID, body.streamCreatedOp).merging(["created_at": .number(body.createdAt.microseconds), "name": .string(body.name)]) { $1 }, + body.options) +} + +private func user(_ body: MetadataUserChange) -> [String: JSONValue] { + options( + userKey(body.userID, body.userCreatedAt).merging([ + "username": .string(body.username), "status": .string(body.status.description), "permissions": permissions(body.permissions), + ]) { $1 }, body.options) +} + +private func token(_ body: MetadataTokenChange) -> [String: JSONValue] { + optional(userKey(body.userID, body.userCreatedAt).merging(["name": .string(body.name)]) { $1 }, "expiry", body.expiry?.microseconds) +} + +private func session(_ body: MetadataSession) -> [String: JSONValue] { + userKey(body.userID, body.userCreatedAt).merging(["client_id": .string(hex(body.clientID)), "session": .number(body.session)]) { $1 } +} + +private func optional(_ members: [String: JSONValue], _ key: String, _ value: UInt64?) -> [String: JSONValue] { + var members = members + if let value { + members[key] = .number(value) + } + return members +} + +private func ids(_ ids: [UInt32]) -> JSONValue { + .array(ids.map { .number(UInt64($0)) }) +} + +private func options(_ members: [String: JSONValue], _ options: MetadataOptions) -> [String: JSONValue] { + var members = members + switch options { + case .readable(let options): + members["options"] = .object( + options.mapValues { option in + let text = option.value.kind == .raw ? Data(option.value.bytes).base64EncodedString() : option.value.description + return .object(["value": .string(text), "kind": .string(option.value.kind.description), "explicit": .bool(option.explicit)]) + }) + case .unreadable: + members["options"] = .null + members["options_unreadable"] = .bool(true) + } + return members +} + +private func permissions(_ permissions: Permissions?) -> JSONValue { + guard let permissions else { + return .null + } + let global = permissions.global + var streams: JSONValue = .null + if !permissions.streams.isEmpty { + streams = .object( + Dictionary( + uniqueKeysWithValues: permissions.streams.map { streamID, stream in + let topics: JSONValue = + stream.topics.isEmpty + ? .null + : .object( + Dictionary( + uniqueKeysWithValues: stream.topics.map { topicID, topic in + ( + String(topicID), + JSONValue.object([ + "manage_topic": .bool(topic.manageTopic), "read_topic": .bool(topic.readTopic), + "poll_messages": .bool(topic.pollMessages), "send_messages": .bool(topic.sendMessages), + ]) + ) + })) + return ( + String(streamID), + JSONValue.object([ + "manage_stream": .bool(stream.manageStream), "read_stream": .bool(stream.readStream), "manage_topics": .bool(stream.manageTopics), + "read_topics": .bool(stream.readTopics), "poll_messages": .bool(stream.pollMessages), "send_messages": .bool(stream.sendMessages), + "topics": topics, + ]) + ) + })) + } + return .object([ + "global": .object([ + "manage_servers": .bool(global.manageServers), "read_servers": .bool(global.readServers), "manage_users": .bool(global.manageUsers), + "read_users": .bool(global.readUsers), "manage_streams": .bool(global.manageStreams), "read_streams": .bool(global.readStreams), + "manage_topics": .bool(global.manageTopics), "read_topics": .bool(global.readTopics), "poll_messages": .bool(global.pollMessages), + "send_messages": .bool(global.sendMessages), + ]), + "streams": streams, + ]) +} + +// The README sample, never called: it only has to compile, so it cannot drift from the API. +private func follow(_ client: IggyClient, groups: [MetadataChangeGroup]) async throws { + var query = MetadataChangesQuery(start: .latest, groups: groups) + while true { + let page = try await client.getMetadataChanges(query) + switch page.status { + case .ok: + for record in page.records { + print("op \(record.op) by \(String(describing: record.actorUserID)): \(record.change)") + } + if let next = page.next { + query.start = .after(next) + } + if !page.hasMore { + try await Task.sleep(for: .seconds(1)) + } + case .behind: + try await Task.sleep(for: .milliseconds(100)) + case .expired, .reset: + query.start = try await rebuild(client, groups: groups) + } + } +} + +private func rebuild(_ client: IggyClient, groups: [MetadataChangeGroup]) async throws -> MetadataChangeStart { + var query = MetadataChangesQuery(start: .current(nil), groups: groups) + while true { + let page = try await client.getMetadataChanges(query) + switch page.status { + case .ok: + break + case .behind: + try await Task.sleep(for: .milliseconds(100)) + continue + case .expired, .reset: + query.start = .current(nil) + continue + } + for item in page.items { + print(item) + } + if let token = page.currentPageToken { + query.start = .current(token) + } else if let anchor = page.next { + return .after(anchor) + } else { + query.start = .current(nil) + } + } +}
diff --git a/foreign/swift/Tests/IggyTests/ResponseVectorTests.swift b/foreign/swift/Tests/IggyTests/ResponseVectorTests.swift index 3702c98..8288d99 100644 --- a/foreign/swift/Tests/IggyTests/ResponseVectorTests.swift +++ b/foreign/swift/Tests/IggyTests/ResponseVectorTests.swift
@@ -243,6 +243,32 @@ #expect(specs[1].description.isEmpty) } + @Test func metadataChanges() throws { + let page = try Responses.metadataChanges(body("response.get_metadata_changes")) + let cursor = MetadataChangeCursor( + logID: UInt128Value(0x0d4b_8f2a_6e1c_93d7), op: 4187, checksum: UInt128Value(0xf2a4_c6e8_b0d1_f359), ordinal: 1) + #expect(page.status == .ok) + #expect(page.phase == .changes) + #expect(page.logID == cursor.logID) + #expect(page.oldestOp == 3871) + #expect(page.retainedSince == .zero) + #expect(page.headOp == 4190) + #expect(page.next == cursor) + #expect(page.hasMore == false) + #expect(page.items.isEmpty) + #expect(page.currentPageToken == nil) + #expect( + page.records == [ + MetadataChangeRecord( + op: 4121, ordinal: 0, checksum: UInt128Value(0xa5c1_f703_8b9e_3d65), timestamp: IggyTimestamp(microseconds: 1_760_356_800_004_121), + actorUserID: 0, + change: .streamCreated( + MetadataStreamChange( + streamID: 3, streamCreatedOp: 4121, createdAt: IggyTimestamp(microseconds: 1_760_356_800_123_456), name: "telemetry", + options: .readable([:])))) + ]) + } + @Test func sendConfirmations() throws { let response = try Responses.sendConfirmations(body("response.send_messages")) #expect(
diff --git a/foreign/swift/Tests/IggyTests/Support/MetadataChangesFixture.swift b/foreign/swift/Tests/IggyTests/Support/MetadataChangesFixture.swift new file mode 100644 index 0000000..9018776 --- /dev/null +++ b/foreign/swift/Tests/IggyTests/Support/MetadataChangesFixture.swift
@@ -0,0 +1,107 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import Foundation + +/// The cross-SDK metadata changes fixture, generated by +/// `core/common/examples/metadata_changes_fixtures.rs` from the real Rust +/// codecs and copied byte for byte into every SDK. Never edit the copy. Its +/// `expected` objects are the JSON the HTTP route renders, so they load as +/// ``JSONValue`` trees the tests compare whole. +enum MetadataChangesFixture { + static let shared: JSONValue = { + let url = Bundle.module.url(forResource: "metadata_changes", withExtension: "json", subdirectory: "Fixtures")! + let data = try! Data(contentsOf: url) + return try! JSONDecoder().decode(JSONValue.self, from: data) + }() +} + +/// A JSON document. Every number in the fixture is an unsigned integer. +enum JSONValue: Decodable, Equatable, Sendable, CustomStringConvertible { + case null + case bool(Bool) + case number(UInt64) + case string(String) + case array([JSONValue]) + case object([String: JSONValue]) + + init(from decoder: any Decoder) throws { + let container = try decoder.singleValueContainer() + if container.decodeNil() { + self = .null + } else if let value = try? container.decode(Bool.self) { + self = .bool(value) + } else if let value = try? container.decode(UInt64.self) { + self = .number(value) + } else if let value = try? container.decode(String.self) { + self = .string(value) + } else if let value = try? container.decode([JSONValue].self) { + self = .array(value) + } else { + self = .object(try container.decode([String: JSONValue].self)) + } + } + + /// The member named `key`, or null when this is not an object or has no such member. + subscript(key: String) -> JSONValue { + if case .object(let members) = self { + return members[key] ?? .null + } + return .null + } + + var isNull: Bool { self == .null } + + var string: String { + if case .string(let value) = self { + return value + } + return "" + } + + var number: UInt64 { + if case .number(let value) = self { + return value + } + return 0 + } + + var array: [JSONValue] { + if case .array(let values) = self { + return values + } + return [] + } + + var members: [String: JSONValue] { + if case .object(let members) = self { + return members + } + return [:] + } + + var description: String { + switch self { + case .null: "null" + case .bool(let value): String(value) + case .number(let value): String(value) + case .string(let value): "\"\(value)\"" + case .array(let values): "[\(values.map(\.description).joined(separator: ","))]" + case .object(let members): "{\(members.sorted { $0.key < $1.key }.map { "\"\($0.key)\":\($0.value)" }.joined(separator: ","))}" + } + } +}
diff --git a/foreign/swift/Tests/IggyTests/WireVectorTests.swift b/foreign/swift/Tests/IggyTests/WireVectorTests.swift index 370cf94..d103719 100644 --- a/foreign/swift/Tests/IggyTests/WireVectorTests.swift +++ b/foreign/swift/Tests/IggyTests/WireVectorTests.swift
@@ -182,6 +182,12 @@ #expect(golden.bytes("request.get_cluster_metadata").isEmpty) #expect(try Requests.getSnapshot(compression: .deflated, types: [.filesystemOverview, .serverLogs]) == golden.bytes("request.get_snapshot")) #expect(Requests.describeOptions(scope: .topic) == golden.bytes("request.describe_options")) + let cursor = MetadataChangeCursor( + logID: UInt128Value(0x0d4b_8f2a_6e1c_93d7), op: 4187, checksum: UInt128Value(0xf2a4_c6e8_b0d1_f359), ordinal: 1) + #expect( + Requests.getMetadataChanges( + MetadataChangesQuery(start: .after(cursor), groups: [.topics, .streams], since: IggyTimestamp(microseconds: 1_760_356_800_000_000), limit: 500)) + == golden.bytes("request.get_metadata_changes")) let testBindSecret = Array(repeating: UInt8(0x5a), count: Requests.bindSecretSize) let testClientID = UInt128Value(low: 1, high: 0) #expect(
diff --git a/foreign/swift/Tools/golden-vectors/src/main.rs b/foreign/swift/Tools/golden-vectors/src/main.rs index 7f804a9..d850acb 100644 --- a/foreign/swift/Tools/golden-vectors/src/main.rs +++ b/foreign/swift/Tools/golden-vectors/src/main.rs
@@ -35,6 +35,10 @@ use iggy_binary_protocol::consensus::{ Command, EvictionHeader, EvictionReason, HEADER_SIZE, Operation, ReplyHeader, RequestHeader, }; +use iggy_binary_protocol::primitives::metadata_changes::{ + WireChangeOptions, WireMetadataChange, WireMetadataChangeCursor, WireMetadataChangeGroup, + WireMetadataChangePhase, WireMetadataChangeStatus, WireStreamBody, WireStreamKey, +}; use iggy_binary_protocol::primitives::options::WireOptions; use iggy_binary_protocol::primitives::partition_history::PartitionContext; use iggy_binary_protocol::primitives::permissions::{ @@ -78,7 +82,8 @@ }; use iggy_binary_protocol::responses::system::get_stats::{CacheMetricEntry, StatsResponse}; use iggy_binary_protocol::responses::system::{ - ClusterMetadataResponse, ClusterNodeResponse, DescribeOptionsResponse, OptionDescriptor, + ClusterMetadataResponse, ClusterNodeResponse, DescribeOptionsResponse, + GetMetadataChangesResponse, OptionDescriptor, WireMetadataChangeRecord, }; use iggy_binary_protocol::responses::topics::{ GetTopicResponse, GetTopicsResponse, PartitionResponse, @@ -138,6 +143,15 @@ WireName::new(value).expect("valid name") } +fn metadata_cursor() -> WireMetadataChangeCursor { + WireMetadataChangeCursor { + log_id: 0x0d4b_8f2a_6e1c_93d7, + op: 4187, + checksum: 0xf2a4_c6e8_b0d1_f359, + ordinal: 1, + } +} + fn options(entries: &[(u8, &[u8], u8, &[u8])]) -> WireOptions { let mut buf = BytesMut::new(); encode_user_headers(entries, &mut buf); @@ -194,7 +208,7 @@ ClientVersionInfo { protocol_version: IGGY_PROTOCOL_VERSION, sdk_name: name("swift-sdk"), - sdk_version: name("0.1.1"), + sdk_version: name("0.1.2"), } } @@ -907,6 +921,20 @@ "request.describe_options", &DescribeOptionsRequest { scope: 1 }, ); + add( + &mut vectors, + "request.get_metadata_changes", + &GetMetadataChangesRequest { + start: WireMetadataChangeStart::After(metadata_cursor()), + since: Some(1_760_356_800_000_000), + until: None, + limit: 500, + groups: vec![ + WireMetadataChangeGroup::Topics.as_u8(), + WireMetadataChangeGroup::Streams.as_u8(), + ], + }, + ); // Send messages: the full body a producer writes. { @@ -1469,6 +1497,44 @@ ], }, ); + let stream_created = WireMetadataChange::StreamCreated(WireStreamBody { + key: WireStreamKey { + stream_id: 3, + stream_created_op: 4121, + }, + created_at: 1_760_356_800_123_456, + name: name("telemetry"), + options: WireChangeOptions::Readable { + explicit: WireOptions::empty(), + derived: WireOptions::empty(), + }, + }); + add( + &mut vectors, + "response.get_metadata_changes", + &GetMetadataChangesResponse { + status: WireMetadataChangeStatus::Ok, + phase: WireMetadataChangePhase::Changes, + log_id: Some(metadata_cursor().log_id), + oldest_op: 3871, + retained_since: 0, + head_op: 4190, + next: Some(metadata_cursor()), + has_more: false, + records: vec![WireMetadataChangeRecord { + op: 4121, + ordinal: 0, + checksum: 0xa5c1_f703_8b9e_3d65, + timestamp: 1_760_356_800_004_121, + actor_user_id: Some(0), + group: stream_created.group().as_u8(), + kind: stream_created.kind().as_u8(), + body: stream_created.body_bytes(), + }], + items: Vec::new(), + current_page_token: None, + }, + ); add( &mut vectors, "response.describe_options",
diff --git a/scripts/run-bdd-tests.sh b/scripts/run-bdd-tests.sh index e56f032..7ceb993 100755 --- a/scripts/run-bdd-tests.sh +++ b/scripts/run-bdd-tests.sh
@@ -36,7 +36,7 @@ log "Usage: $0 [--coverage] <sdk> [feature]" log "" log " sdk: rust | python | php | go | go-race | node | csharp | java | cpp | all | clean (default: all)" - log " feature: basic_messaging | leader_redirection | raw_command | session_contract | stream_crud | stream_options | resource_options_patch | all (default: all)" + log " feature: basic_messaging | leader_redirection | raw_command | session_contract | stream_crud | stream_options | resource_options_patch | metadata_changes | all (default: all)" log "" log " Every suite runs against iggy-server, taken from IGGY_SERVER_PATH" log " (default: target/debug/iggy-server) with an iggy CLI at IGGY_CLI_PATH." @@ -49,7 +49,7 @@ } case "$FEATURE" in - basic_messaging|leader_redirection|raw_command|session_contract|stream_crud|stream_options|resource_options_patch|all) ;; + basic_messaging|leader_redirection|raw_command|session_contract|stream_crud|stream_options|resource_options_patch|metadata_changes|all) ;; *) log "Unknown feature: ${FEATURE}" usage @@ -70,7 +70,7 @@ COMPOSE_FILES=(-f docker-compose.yml) case "$FEATURE" in - basic_messaging|leader_redirection|raw_command|session_contract|stream_crud|stream_options|resource_options_patch|all) + basic_messaging|leader_redirection|raw_command|session_contract|stream_crud|stream_options|resource_options_patch|metadata_changes|all) COMPOSE_FILES+=(-f docker-compose.server.yml) ;; esac case "$FEATURE" in
diff --git a/web/docs/server-api.http b/web/docs/server-api.http index 6fe2cf0..ecbf920 100644 --- a/web/docs/server-api.http +++ b/web/docs/server-api.http
@@ -59,6 +59,22 @@ Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJqdGkiOiI5NmJlY2QwNy03ZTYyLTRiYTktYTcyZS02NzgwOThmMjE1MmMiLCJpc3MiOiJpZ2d5LnJzIiwiYXVkIjoiaWdneS5ycyIsInN1YiI6MSwiaWF0IjoxNzAwNTk4NTI5LCJleHAiOjE3MDA2MDIxMjksIm5iZiI6MTcwMDU5ODUyOX0.IYaIwFA1VajunF_P2KLQWOnnEzCYHJT31QuIQa__TS0 +### Metadata changes from the oldest retained operation (root or manage_servers) +GET {{url}}/metadata/changes?from=earliest&limit=100 +Authorization: Bearer {{access_token}} + +### Metadata changes: a cursor at the newest record, no records +GET {{url}}/metadata/changes?from=latest&groups=streams,topics +Authorization: Bearer {{access_token}} + +### Metadata changes after a cursor from an earlier page +GET {{url}}/metadata/changes?after=mc1_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA&limit=100 +Authorization: Bearer {{access_token}} + +### Current state pages, continued with the current_page_token of the previous answer +GET {{url}}/metadata/changes?from=current&groups=streams,topics&limit=100 +Authorization: Bearer {{access_token}} + ### Create stream POST {{url}}/streams Content-Type: application/json
diff --git a/web/package-lock.json b/web/package-lock.json index d39fec3..ca27991 100644 --- a/web/package-lock.json +++ b/web/package-lock.json
@@ -1,12 +1,12 @@ { "name": "iggy-web-ui", - "version": "0.4.1-edge.2", + "version": "0.4.1-edge.3", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "iggy-web-ui", - "version": "0.4.1-edge.2", + "version": "0.4.1-edge.3", "dependencies": { "@floating-ui/dom": "^1.8.0", "@tailwindcss/postcss": "^4.3.3",
diff --git a/web/package.json b/web/package.json index 0e2c4a7..9eb8973 100644 --- a/web/package.json +++ b/web/package.json
@@ -1,6 +1,6 @@ { "name": "iggy-web-ui", - "version": "0.4.1-edge.2", + "version": "0.4.1-edge.3", "private": true, "scripts": { "dev": "vite dev --port 3050",
diff --git a/web/src/lib/api/fetchMetadataChanges.ts b/web/src/lib/api/fetchMetadataChanges.ts new file mode 100644 index 0000000..cf7dea2 --- /dev/null +++ b/web/src/lib/api/fetchMetadataChanges.ts
@@ -0,0 +1,65 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import { PUBLIC_IGGY_API_URL } from '$app/env/public'; +import { authStore } from '#lib/auth/authStore.svelte.js'; +import { getJson } from './getJson'; +import type { MetadataChangesPage } from './metadataChanges'; + +export type MetadataChangesResult = + | { kind: 'page'; page: MetadataChangesPage } + | { kind: 'forbidden' } + | { kind: 'refused'; reason: string } + | { kind: 'failed'; reason: string }; + +/** + * One page of `GET /metadata/changes`. Unlike `clientApi`, a 403 keeps the + * session: it means the account lacks `manage_servers`, not that it expired. + */ +export async function fetchMetadataChanges( + query: Record<string, string>, + signal?: AbortSignal +): Promise<MetadataChangesResult> { + const headers = new Headers(); + const token = authStore.getAccessToken(); + if (token) { + headers.set('Authorization', `Bearer ${token}`); + } + const url = `${PUBLIC_IGGY_API_URL || ''}/metadata/changes?${new URLSearchParams(Object.entries(query))}`; + let response: Response; + let data: { reason?: string } | null; + try { + response = await fetch(url, { headers, signal }); + data = (await getJson(response)) as { reason?: string } | null; + } catch { + return { kind: 'failed', reason: 'The server did not answer' }; + } + if (response.ok) { + return { kind: 'page', page: data as MetadataChangesPage }; + } + switch (response.status) { + case 401: + authStore.logout(); + return { kind: 'failed', reason: 'The session ended' }; + case 403: + return { kind: 'forbidden' }; + case 400: + return { kind: 'refused', reason: data?.reason ?? 'The server refused the filter' }; + default: + return { kind: 'failed', reason: `The server answered ${response.status}` }; + } +}
diff --git a/web/src/lib/api/metadataChanges.test.mjs b/web/src/lib/api/metadataChanges.test.mjs new file mode 100644 index 0000000..e68d227 --- /dev/null +++ b/web/src/lib/api/metadataChanges.test.mjs
@@ -0,0 +1,279 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import assert from 'node:assert/strict'; +import { test } from 'node:test'; +import { + MAX_METADATA_CHANGES_LIMIT, + canReadMetadataChanges, + describeChange, + filterProblem, + metadataChangesQuery, + microsFromDateTimeLocal, + statusMessage, + scanRecentMetadataChanges +} from './metadataChanges.ts'; + +/** @type {import('./metadataChanges.ts').MetadataChangesFilter} */ +const filter = { start: 'earliest', groups: [], limit: 100 }; +const cursor = 'mc1_15McbiqPSw0AAAAAAAAAABsQAAAAAAAAZz2eiwP3waUAAAAAAAAAAAAAAAA'; + +test('given_earliest_filter_when_building_query_should_send_groups_in_code_order_and_bounds', () => { + assert.deepEqual( + metadataChangesQuery({ + ...filter, + groups: /** @type {const} */ (['sessions', 'streams', 'sessions']), + since: 5, + until: 9 + }), + { from: 'earliest', groups: 'streams,sessions', since: '5', until: '9', limit: '100' } + ); +}); + +test('given_cursor_when_building_query_should_continue_after_it_and_keep_bounds', () => { + assert.deepEqual( + metadataChangesQuery({ ...filter, start: /** @type {const} */ ('latest'), since: 5 }, cursor), + { + after: cursor, + since: '5', + limit: '100' + } + ); +}); + +test('given_latest_filter_when_building_query_should_drop_time_bounds_the_server_refuses', () => { + assert.deepEqual( + metadataChangesQuery({ ...filter, start: /** @type {const} */ ('latest'), since: 5, until: 9 }), + { + from: 'latest', + limit: '100' + } + ); +}); + +test('given_out_of_range_limit_or_inverted_range_when_checking_should_name_the_problem', () => { + for (const limit of [0, MAX_METADATA_CHANGES_LIMIT + 1, 1.5]) { + assert.match(filterProblem({ ...filter, limit }) ?? '', /Limit/); + } + assert.match(filterProblem({ ...filter, since: 9, until: 5 }) ?? '', /time range/); + assert.equal(filterProblem({ ...filter, since: 5, until: 5 }), undefined); +}); + +test('given_datetime_local_when_converting_should_return_microseconds_or_nothing', () => { + assert.equal( + microsFromDateTimeLocal('2026-10-23T10:00'), + new Date('2026-10-23T10:00').getTime() * 1000 + ); + assert.equal(microsFromDateTimeLocal(''), undefined); + assert.equal(microsFromDateTimeLocal('not a date'), undefined); +}); + +test('given_user_when_checking_access_should_allow_only_root_and_manage_servers', () => { + assert.equal(canReadMetadataChanges(0, undefined), true); + assert.equal(canReadMetadataChanges(7, true), true); + assert.equal(canReadMetadataChanges(7, false), false); + assert.equal(canReadMetadataChanges(null, undefined), false); +}); + +test('given_non_ok_status_when_describing_should_explain_what_to_do', () => { + assert.equal(statusMessage('ok', false), undefined); + for (const status of /** @type {const} */ (['behind', 'expired', 'reset'])) { + assert.ok((statusMessage(status, false) ?? '').length > 0); + } +}); + +test('given_behind_status_when_describing_should_promise_a_retry_only_while_following', () => { + assert.match(statusMessage('behind', true) ?? '', /Retrying/); + assert.doesNotMatch(statusMessage('behind', false) ?? '', /Retrying/); + assert.match(statusMessage('behind', false) ?? '', /Load again/); +}); + +test('given_records_when_describing_should_name_the_changed_resource', () => { + const envelope = { op: 4121, ordinal: 0, checksum: '00', timestamp: 1 }; + assert.equal( + describeChange({ + ...envelope, + group: 'streams', + kind: 'stream_created', + stream_id: 3, + name: 'telemetry' + }), + 'stream "telemetry" (3)' + ); + assert.equal( + describeChange({ + ...envelope, + group: 'consumer_group_members', + kind: 'consumer_group_partition_handed_over', + stream_id: 3, + topic_id: 1, + group_id: 2, + partition_id: 5 + }), + 'partition 5 of group 2 on topic 3/1 to no member' + ); + assert.equal(describeChange({ ...envelope, group: '200', kind: '7', body: 'AA' }), '200 7'); +}); + +function position(operation, ordinal = 0) { + const bytes = Buffer.alloc(44); + bytes.writeBigUInt64LE(1n, 0); + bytes.writeBigUInt64LE(BigInt(operation), 16); + bytes.writeUInt32LE(ordinal, 40); + return `mc1_${bytes.toString('base64url')}`; +} + +function change(operation, ordinal = 0) { + return { + op: String(operation), + ordinal, + timestamp: 1, + checksum: '00', + group: 'streams', + kind: 'stream_created' + }; +} + +/** @returns {import('./metadataChanges.ts').MetadataChangesPage} */ +function changesPage(records, next, hasMore = false) { + return { + status: 'ok', + phase: 'changes', + log_id: '01', + head_op: '10', + oldest_op: 1, + retained_since: 0, + next, + has_more: hasMore, + records, + items: [], + current_page_token: null + }; +} + +test('given_recent_scan_when_pages_cross_the_captured_head_should_keep_only_the_newest_bounded_records', async () => { + const pages = [ + changesPage([], position(10)), + changesPage([change(1), change(2)], position(2), true), + changesPage([change(3), change(10), change(11)], position(11), true) + ]; + const queries = []; + const result = await scanRecentMetadataChanges( + filter, + async (query) => { + queries.push(query); + return pages.shift() ?? assert.fail('unexpected page request'); + }, + { capacity: 2 } + ); + assert.deepEqual( + result.page.records.map((record) => record.op), + ['3', '10'] + ); + assert.equal(result.page.next, position(10)); + assert.equal(result.page.has_more, false); + assert.equal(result.trimmed, true); + assert.equal(queries[0].from, 'latest'); + assert.equal(queries[1].from, 'earliest'); + assert.equal(queries[2].after, position(2)); +}); + +test('given_empty_filtered_pages_when_scanning_recent_should_stop_at_the_captured_cursor_without_waiting_for_new_writes', async () => { + const large = 9007199254740993n; + const pages = [ + { ...changesPage([], position(large, 2)), head_op: String(large) }, + changesPage([], position(large - 1n), true), + changesPage([], position(large, 3), true) + ]; + const result = await scanRecentMetadataChanges( + filter, + async () => pages.shift() ?? assert.fail('unexpected page request') + ); + assert.equal(pages.length, 0); + assert.deepEqual(result.page.records, []); + assert.equal(result.page.next, position(large, 2)); +}); + +test('given_node_behind_the_captured_head_when_scanning_recent_should_continue_from_its_own_position', async () => { + const pages = [ + changesPage([], position(10)), + { ...changesPage([change(4)], position(4)), head_op: '6' } + ]; + const result = await scanRecentMetadataChanges( + filter, + async () => pages.shift() ?? assert.fail('unexpected page request') + ); + assert.deepEqual( + result.page.records.map((record) => record.op), + ['4'] + ); + assert.equal(result.page.next, position(4)); + assert.equal(result.page.head_op, '6'); + assert.equal(result.page.has_more, false); +}); + +test('given_expiry_or_changed_history_during_recent_scan_should_discard_partial_results', async () => { + for (const status of /** @type {const} */ (['expired', 'reset', 'behind', 'changed_history'])) { + /** @type {import('./metadataChanges.ts').MetadataChangesPage[]} */ + const pages = [ + changesPage([], position(10)), + changesPage([change(1)], position(1), true), + { + ...changesPage([], null), + status: status === 'changed_history' ? 'ok' : status, + log_id: status === 'changed_history' ? '02' : '01' + } + ]; + const result = await scanRecentMetadataChanges( + filter, + async () => pages.shift() ?? assert.fail('unexpected page request') + ); + assert.equal(result.page.status, status === 'changed_history' ? 'reset' : status); + assert.deepEqual(result.page.records, []); + } +}); + +test('given_recent_time_filters_when_capturing_head_should_omit_bounds_only_for_latest', async () => { + const queries = []; + const pages = [changesPage([], position(10)), changesPage([change(4)], position(4))]; + await scanRecentMetadataChanges({ ...filter, since: 2, until: 9 }, async (query) => { + queries.push(query); + return pages.shift() ?? assert.fail('unexpected page request'); + }); + assert.equal(queries[0].since, undefined); + assert.equal(queries[0].until, undefined); + assert.equal(queries[1].since, '2'); + assert.equal(queries[1].until, '9'); +}); + +test('given_canceled_recent_scan_should_stop_before_another_request', async () => { + const controller = new AbortController(); + let calls = 0; + await assert.rejects( + scanRecentMetadataChanges( + filter, + async () => { + calls++; + controller.abort(); + return changesPage([], position(10)); + }, + { signal: controller.signal } + ), + { name: 'AbortError' } + ); + assert.equal(calls, 1); +});
diff --git a/web/src/lib/api/metadataChanges.ts b/web/src/lib/api/metadataChanges.ts new file mode 100644 index 0000000..4bff5ea --- /dev/null +++ b/web/src/lib/api/metadataChanges.ts
@@ -0,0 +1,251 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +export const METADATA_CHANGE_GROUPS = [ + 'streams', + 'topics', + 'partitions', + 'consumer_groups', + 'consumer_group_members', + 'users', + 'personal_access_tokens', + 'sessions' +] as const; +export const DEFAULT_METADATA_CHANGES_LIMIT = 100; +export const MAX_METADATA_CHANGES_LIMIT = 1000; +export const ROOT_USER_ID = 0; +export const MAX_SHOWN_METADATA_CHANGES = 5000; + +export type MetadataChangeGroup = (typeof METADATA_CHANGE_GROUPS)[number]; +export type MetadataChangeStatus = 'ok' | 'behind' | 'expired' | 'reset'; +export type MetadataChangeStart = 'earliest' | 'latest'; +// Values above 2^53 arrive as decimal strings, see `getJson`. +export type WireInteger = number | string; + +export type MetadataChangeRecord = { + op: WireInteger; + ordinal: number; + checksum: string; + timestamp: WireInteger; + actor_user_id?: number; + group: string; + kind: string; + [field: string]: unknown; +}; + +export type MetadataChangesPage = { + status: MetadataChangeStatus; + phase: 'changes' | 'current'; + log_id: string | null; + oldest_op: WireInteger; + retained_since: WireInteger; + head_op: WireInteger; + next: string | null; + has_more: boolean; + records: MetadataChangeRecord[]; + items: unknown[]; + current_page_token: string | null; +}; + +export type MetadataChangesFilter = { + start: MetadataChangeStart; + groups: MetadataChangeGroup[]; + since?: number; + until?: number; + limit: number; +}; + +/** Query parameters of `GET /metadata/changes`, continuing after `cursor` when given. */ +export function metadataChangesQuery( + filter: MetadataChangesFilter, + cursor?: string +): Record<string, string> { + const query: Record<string, string> = cursor ? { after: cursor } : { from: filter.start }; + const groups = METADATA_CHANGE_GROUPS.filter((group) => filter.groups.includes(group)); + if (groups.length > 0) { + query.groups = groups.join(','); + } + // `latest` returns a position only, the server refuses time bounds with it. + if (cursor || filter.start === 'earliest') { + if (filter.since !== undefined) { + query.since = `${filter.since}`; + } + if (filter.until !== undefined) { + query.until = `${filter.until}`; + } + } + query.limit = `${filter.limit}`; + return query; +} + +/** Why a filter cannot be sent, or `undefined` when the server would accept it. */ +export function filterProblem(filter: MetadataChangesFilter): string | undefined { + if ( + !Number.isInteger(filter.limit) || + filter.limit < 1 || + filter.limit > MAX_METADATA_CHANGES_LIMIT + ) { + return `Limit must be a whole number from 1 to ${MAX_METADATA_CHANGES_LIMIT}`; + } + if (filter.since !== undefined && filter.until !== undefined && filter.since > filter.until) { + return 'The start of the time range is after its end'; + } + return undefined; +} + +/** Microseconds since the epoch of a `datetime-local` input value, `undefined` when empty. */ +export function microsFromDateTimeLocal(value: string): number | undefined { + if (value.trim() === '') { + return undefined; + } + const millis = new Date(value).getTime(); + return Number.isNaN(millis) ? undefined : millis * 1000; +} + +/** Root and holders of the global `manage_servers` permission may read the log. */ +export function canReadMetadataChanges( + userId: number | null, + manageServers: boolean | undefined +): boolean { + return userId === ROOT_USER_ID || manageServers === true; +} + +/** What to tell the reader about a page that carries no records to show. */ +export function statusMessage( + status: MetadataChangeStatus, + following: boolean +): string | undefined { + switch (status) { + case 'ok': + return undefined; + case 'behind': + return following + ? 'This node has not applied that position yet. Retrying.' + : 'This node has not applied that position yet. Load again shortly.'; + case 'expired': + return 'The node no longer holds every change after this position. Load again from the earliest retained change.'; + case 'reset': + return 'The position belongs to another history, for example a rebuilt data directory. Load again.'; + } +} + +/** One line naming what a record changed. */ +export function describeChange(record: MetadataChangeRecord): string { + const text = (field: string) => (record[field] === undefined ? '?' : `${record[field]}`); + const topic = `topic ${text('stream_id')}/${text('topic_id')}`; + const group = `group ${text('group_id')} on ${topic}`; + switch (record.group) { + case 'streams': + return `stream "${text('name')}" (${text('stream_id')})`; + case 'topics': + return `${topic} "${text('name')}"`; + case 'partitions': + return `${topic} now has ${text('partitions_count')} partitions`; + case 'consumer_groups': + return `${group} "${text('name')}"`; + case 'consumer_group_members': + if (record.kind === 'consumer_group_partition_handed_over') { + const owner = record.member_id === undefined ? 'no member' : `member ${text('member_id')}`; + return `partition ${text('partition_id')} of ${group} to ${owner}`; + } + return `member ${text('member_id')} of ${group}`; + case 'users': + return `user "${text('username')}" (${text('user_id')})`; + case 'personal_access_tokens': + return `token "${text('name')}" of user ${text('user_id')}`; + case 'sessions': + return `session ${text('session')} of user ${text('user_id')}`; + default: + return `${record.group} ${record.kind}`; + } +} + +export async function scanRecentMetadataChanges( + filter: MetadataChangesFilter, + readPage: (query: Record<string, string>) => Promise<MetadataChangesPage>, + options: { capacity?: number; signal?: AbortSignal; onPage?: (count: number) => void } = {} +): Promise<{ page: MetadataChangesPage; trimmed: boolean }> { + const capacity = options.capacity ?? MAX_SHOWN_METADATA_CHANGES; + options.signal?.throwIfAborted(); + const boundary = await readPage(metadataChangesQuery({ ...filter, start: 'latest' })); + options.signal?.throwIfAborted(); + if (boundary.status !== 'ok' || !boundary.next) return { page: boundary, trimmed: false }; + const end = cursorPosition(boundary.next); + let after: string | undefined; + let records: MetadataChangeRecord[] = []; + let trimmed = false; + let pages = 0; + for (;;) { + options.signal?.throwIfAborted(); + const page = await readPage( + metadataChangesQuery( + { ...filter, start: 'earliest', limit: MAX_METADATA_CHANGES_LIMIT }, + after + ) + ); + options.signal?.throwIfAborted(); + options.onPage?.(++pages); + if (page.status !== 'ok') return { page: { ...page, records: [] }, trimmed: false }; + if (page.log_id !== boundary.log_id) { + return { + page: { ...page, status: 'reset', records: [], next: null, has_more: false }, + trimmed: false + }; + } + const matching = page.records.filter((record) => comparePosition(record, end) <= 0); + trimmed ||= records.length + matching.length > capacity; + records = [...records, ...matching].slice(-capacity); + const passedEnd = page.next !== null && comparePosition(cursorPosition(page.next), end) >= 0; + if (!page.has_more || passedEnd) { + // A node that has not applied the captured head stops short of it, so keep its own position. + const reachedEnd = passedEnd || BigInt(page.head_op) >= BigInt(end.op); + return { + page: reachedEnd + ? { ...page, head_op: boundary.head_op, records, next: boundary.next, has_more: false } + : { ...page, records, has_more: false }, + trimmed + }; + } + if (!page.next || page.next === after) + throw new Error('The server did not advance the change position. Load again.'); + after = page.next; + } +} + +function cursorPosition(cursor: string): { op: string; ordinal: number } { + const bytes = Uint8Array.from( + atob(cursor.slice(4).replaceAll('-', '+').replaceAll('_', '/')), + (char) => char.charCodeAt(0) + ); + if (!cursor.startsWith('mc1_') || bytes.length !== 44) + throw new Error('The server returned an invalid change position.'); + const view = new DataView(bytes.buffer); + return { op: view.getBigUint64(16, true).toString(), ordinal: view.getUint32(40, true) }; +} + +function comparePosition( + left: { op: WireInteger; ordinal: number }, + right: { op: WireInteger; ordinal: number } +): number { + const leftOperation = BigInt(left.op); + const rightOperation = BigInt(right.op); + return leftOperation === rightOperation + ? left.ordinal - right.ordinal + : leftOperation < rightOperation + ? -1 + : 1; +}
diff --git a/web/src/lib/components/Navbar.svelte b/web/src/lib/components/Navbar.svelte index d3e0e3b..32df83d 100644 --- a/web/src/lib/components/Navbar.svelte +++ b/web/src/lib/components/Navbar.svelte
@@ -27,6 +27,12 @@ import { typedRoute } from '#lib/types/appRoutes.js'; import LogoType from '#lib/components/Logo/LogoType.svelte'; import LogoMark from '#lib/components/Logo/LogoMark.svelte'; + import { authStore } from '#lib/auth/authStore.svelte.js'; + import { canReadMetadataChanges } from '#lib/api/metadataChanges.js'; + + let showChanges = $derived( + canReadMetadataChanges(authStore.userId, page.data.user?.permissions?.global?.manageServers) + ); let navItems = $derived([ { name: 'Overview', @@ -40,6 +46,16 @@ path: typedRoute('dashboard/streams'), active: page.url.pathname.includes(typedRoute('dashboard/streams')) }, + ...(showChanges + ? [ + { + name: 'Metadata changes', + icon: 'timeline' as const, + path: typedRoute('dashboard/changes'), + active: page.url.pathname.includes(typedRoute('dashboard/changes')) + } + ] + : []), { name: 'Settings', icon: 'settings',
diff --git a/web/src/lib/types/appRoutes.ts b/web/src/lib/types/appRoutes.ts index 195dd43..b62ff11 100644 --- a/web/src/lib/types/appRoutes.ts +++ b/web/src/lib/types/appRoutes.ts
@@ -24,6 +24,7 @@ | `streams/${number}/topics/${number}` | `streams/${number}/topics/${number}/partitions/${number}/messages` | 'clients' + | 'changes' | 'logs' | SettingsSegment}`;
diff --git a/web/src/routes/dashboard/+layout.ts b/web/src/routes/dashboard/+layout.ts index 6b0da92..1c35525 100644 --- a/web/src/routes/dashboard/+layout.ts +++ b/web/src/routes/dashboard/+layout.ts
@@ -39,7 +39,7 @@ const getDetailedUser = async () => { let userId = authStore.userId; - if (!userId && token) { + if (userId === null && token) { try { const decoded = jwtDecode(token); userId = decoded.sub ? parseInt(decoded.sub, 10) : null; @@ -48,7 +48,7 @@ } } - if (!userId) return null; + if (userId === null) return null; try { const data = await clientApi({ method: 'GET', path: `/users/${userId}` });
diff --git a/web/src/routes/dashboard/changes/+page.svelte b/web/src/routes/dashboard/changes/+page.svelte new file mode 100644 index 0000000..dd5055a --- /dev/null +++ b/web/src/routes/dashboard/changes/+page.svelte
@@ -0,0 +1,564 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +<script lang="ts"> + import { page } from '$app/state'; + import Button from '#lib/components/Button.svelte'; + import Checkbox from '#lib/components/Checkbox.svelte'; + import { authStore } from '#lib/auth/authStore.svelte.js'; + import { fetchMetadataChanges } from '#lib/api/fetchMetadataChanges.js'; + import { + DEFAULT_METADATA_CHANGES_LIMIT, + METADATA_CHANGE_GROUPS, + MAX_SHOWN_METADATA_CHANGES, + scanRecentMetadataChanges, + canReadMetadataChanges, + describeChange, + filterProblem, + metadataChangesQuery, + microsFromDateTimeLocal, + statusMessage, + type MetadataChangeGroup, + type MetadataChangeRecord, + type MetadataChangesFilter, + type MetadataChangesPage + } from '#lib/api/metadataChanges.js'; + import { onDestroy, onMount } from 'svelte'; + import { fly } from 'svelte/transition'; + + const FOLLOW_INTERVAL_MILLIS = 2000; + const inputClass = + 'rounded-md dark:bg-shade-d400 ring-1 ring-gray-300 dark:ring-gray-500 h-[40px] px-3 text-color bg-transparent'; + + let selectedRecord = $state<MetadataChangeRecord | null>(null); + let copyFeedback = $state(''); + let copyFeedbackTimer: ReturnType<typeof setTimeout> | undefined; + let pageContainer: HTMLDivElement; + const selectedOptions = $derived( + selectedRecord?.options && typeof selectedRecord.options === 'object' + ? Object.entries(selectedRecord.options as Record<string, unknown>).map(([key, option]) => { + const field = + option && typeof option === 'object' ? (option as Record<string, unknown>) : {}; + return { + key, + value: 'value' in field ? field.value : option, + kind: field.kind, + explicit: field.explicit + }; + }) + : [] + ); + + function inspect(record: MetadataChangeRecord) { + selectedRecord = record; + copyFeedback = ''; + clearTimeout(copyFeedbackTimer); + } + + function openDetails(dialog: HTMLDialogElement) { + const trigger = document.activeElement; + const overflow = pageContainer.style.overflow; + pageContainer.style.overflow = 'hidden'; + dialog.showModal(); + return { + destroy() { + dialog.close(); + pageContainer.style.overflow = overflow; + if (trigger instanceof HTMLElement) trigger.focus({ preventScroll: true }); + } + }; + } + + function closeOnBackdrop(event: MouseEvent) { + if (event.target !== event.currentTarget) return; + const bounds = (event.currentTarget as HTMLDialogElement).getBoundingClientRect(); + if ( + event.clientX < bounds.left || + event.clientX > bounds.right || + event.clientY < bounds.top || + event.clientY > bounds.bottom + ) { + selectedRecord = null; + } + } + + function displayValue(value: unknown): string { + return typeof value === 'string' ? value : (JSON.stringify(value, null, 2) ?? ''); + } + + async function copy(value: unknown, label: string) { + clearTimeout(copyFeedbackTimer); + try { + await navigator.clipboard.writeText(displayValue(value)); + copyFeedback = `${label} copied`; + } catch { + copyFeedback = 'Copy is unavailable. You can select and copy the text below.'; + } + copyFeedbackTimer = setTimeout(() => { + copyFeedback = ''; + }, 3000); + } + + let start = $state<'earliest' | 'recent'>('earliest'); + const order = $derived(start === 'recent' ? 'newest' : 'oldest'); + let recentLoaded = $state(false); + let visibleCount = $state(DEFAULT_METADATA_CHANGES_LIMIT); + let scannedPages = $state(0); + let recentController = $state<AbortController | undefined>(); + let groups = $state<string[]>([]); + let since = $state(''); + let until = $state(''); + let limit = $state(DEFAULT_METADATA_CHANGES_LIMIT); + let records = $state<MetadataChangeRecord[]>([]); + const shownRecords = $derived( + (order === 'newest' ? records.slice().reverse() : records).slice( + 0, + recentLoaded ? visibleCount : records.length + ) + ); + let cursor = $state<string | null>(null); + let hasMore = $state(false); + let lastPage = $state<MetadataChangesPage | null>(null); + let notice = $state<string | undefined>(undefined); + let problem = $state<string | undefined>(undefined); + let forbidden = $state(false); + let loading = $state(false); + let following = $state(false); + let activeFilter: MetadataChangesFilter | null = null; + let followTimer: ReturnType<typeof setInterval> | undefined; + + let allowed = $derived( + canReadMetadataChanges(authStore.userId, page.data.user?.permissions?.global?.manageServers) + ); + + function currentFilter(): MetadataChangesFilter { + return { + start: start === 'recent' ? 'earliest' : start, + groups: groups as MetadataChangeGroup[], + since: microsFromDateTimeLocal(since), + until: microsFromDateTimeLocal(until), + limit: Number(limit) + }; + } + + async function read(filter: MetadataChangesFilter, after: string | null) { + loading = true; + const result = await fetchMetadataChanges(metadataChangesQuery(filter, after ?? undefined)); + loading = false; + switch (result.kind) { + case 'forbidden': + forbidden = true; + stopFollowing(); + return; + case 'refused': + case 'failed': + problem = result.reason; + stopFollowing(); + return; + case 'page': + break; + } + const answer = result.page; + lastPage = answer; + notice = statusMessage(answer.status, following); + if (answer.status === 'ok') { + records = [...records, ...answer.records].slice(-MAX_SHOWN_METADATA_CHANGES); + cursor = answer.next; + hasMore = answer.has_more; + } else if (answer.status !== 'behind') { + cursor = null; + hasMore = false; + stopFollowing(); + } + } + + async function load() { + const filter = currentFilter(); + problem = filterProblem(filter); + if (problem) { + return; + } + stopFollowing(); + activeFilter = filter; + records = []; + cursor = null; + hasMore = false; + lastPage = null; + notice = undefined; + recentLoaded = start === 'recent'; + visibleCount = filter.limit; + if (!recentLoaded) { + await read(filter, null); + return; + } + loading = true; + scannedPages = 0; + const controller = new AbortController(); + recentController = controller; + try { + const result = await scanRecentMetadataChanges( + filter, + async (query) => { + const response = await fetchMetadataChanges(query, controller.signal); + if (response.kind === 'page') return response.page; + if (response.kind === 'forbidden') { + forbidden = true; + throw new Error('Reading metadata changes is not allowed.'); + } + throw new Error(response.reason); + }, + { + signal: controller.signal, + onPage: (count) => { + scannedPages = count; + } + } + ); + lastPage = result.page; + if (result.page.status === 'ok') { + records = result.page.records; + cursor = result.page.next; + if (result.trimmed) + notice = `Showing the newest ${MAX_SHOWN_METADATA_CHANGES} matching records. Narrow the time range or load from the earliest retained change to browse further back.`; + } else { + notice = + result.page.status === 'behind' + ? 'This node is behind the captured position. Load again shortly.' + : statusMessage(result.page.status, false); + } + } catch (error) { + if (controller.signal.aborted) notice = 'Loading canceled.'; + else problem = error instanceof Error ? error.message : 'Could not load recent changes.'; + } finally { + recentController = undefined; + loading = false; + } + } + + async function loadMore() { + if (recentLoaded) { + visibleCount = Math.min(records.length, visibleCount + (activeFilter?.limit ?? limit)); + } else { + await pollChanges(); + } + } + + async function pollChanges() { + if (activeFilter && cursor && !loading) { + await read(activeFilter, cursor); + } + } + + function startFollowing() { + following = true; + followTimer = setInterval(pollChanges, FOLLOW_INTERVAL_MILLIS); + } + + function stopFollowing() { + following = false; + clearInterval(followTimer); + followTimer = undefined; + } + + onMount(() => { + if (allowed) void load(); + }); + + onDestroy(() => { + stopFollowing(); + recentController?.abort(); + clearTimeout(copyFeedbackTimer); + }); +</script> + +<div bind:this={pageContainer} class="p-5 flex flex-col gap-5 overflow-auto h-full text-color"> + <h1 class="text-2xl font-semibold">Metadata changes</h1> + + {#if !allowed || forbidden} + <p class="text-shade-d200 dark:text-shade-l700"> + Reading metadata changes needs the root user or the global manage servers permission. + </p> + {:else} + <section class="border rounded-md p-5 flex flex-col gap-4"> + <div class="flex flex-wrap gap-5 items-end"> + <label class="flex flex-col gap-2 text-base font-medium"> + Order + <select + value={start} + aria-label="Order" + class="{inputClass} cursor-pointer" + disabled={loading} + onchange={(event) => { + start = event.currentTarget.value as 'earliest' | 'recent'; + void load(); + }} + > + <option value="recent">Newest first</option> + <option value="earliest">Oldest first</option> + </select> + </label> + <label class="flex flex-col gap-2 text-base font-medium"> + Since + <input type="datetime-local" bind:value={since} class={inputClass} /> + </label> + <label class="flex flex-col gap-2 text-base font-medium"> + Until + <input type="datetime-local" bind:value={until} class={inputClass} /> + </label> + <label class="flex flex-col gap-2 text-base font-medium"> + Page size + <input + type="number" + min="1" + max="1000" + bind:value={limit} + class="{inputClass} w-[110px]" + /> + </label> + </div> + <fieldset class="flex flex-wrap gap-4 text-base"> + <legend class="mb-2">Groups (none selected means all)</legend> + {#each METADATA_CHANGE_GROUPS as group (group)} + <label class="flex items-center gap-2 cursor-pointer"> + <Checkbox value={group} bind:bindGroup={groups} /> + {group.replaceAll('_', ' ')} + </label> + {/each} + </fieldset> + <div class="flex gap-3 items-center"> + <Button variant="contained" onclick={load} disabled={loading}>Refresh</Button> + <Button + variant="outlined" + onclick={loadMore} + disabled={loading || + (recentLoaded ? shownRecords.length >= records.length : !cursor || !hasMore)} + > + Show more + </Button> + <Button + variant="outlined" + onclick={() => (following ? stopFollowing() : startFollowing())} + disabled={!cursor && !following} + > + {following ? 'Stop following' : 'Follow'} + </Button> + {#if recentController} + <Button variant="outlined" onclick={() => recentController?.abort()}>Cancel</Button> + <span role="status" class="text-sm" + >Loading recent changes, {scannedPages} pages read</span + > + {/if} + {#if lastPage} + <span class="text-sm"> + Showing {shownRecords.length} of {records.length} loaded changes + </span> + {/if} + </div> + {#if start === 'recent'} + <p class="text-base leading-6 text-gray-700 dark:text-gray-200"> + Showing up to {MAX_SHOWN_METADATA_CHANGES.toLocaleString()} recent changes. Choose Oldest first + to browse from the beginning. + </p> + {/if} + {#if problem} + <p class="text-red-600 text-sm">{problem}</p> + {/if} + {#if notice} + <p class="text-sm">{notice}</p> + {/if} + </section> + + <table class="w-full text-base border-collapse"> + <thead> + <tr class="text-left border-b"> + <th class="py-2 pr-4">Operation</th> + <th class="py-2 pr-4">Time</th> + <th class="py-2 pr-4">Kind</th> + <th class="py-2 pr-4">Actor</th> + <th class="py-2">Change</th> + </tr> + </thead> + <tbody> + {#each shownRecords as record (`${record.op}:${record.ordinal}`)} + <tr + class="border-b align-top transition-colors hover:bg-shade-l300 dark:hover:bg-shade-d600" + > + <td class="py-2 pr-4"> + <span class="block font-mono">{record.op}</span> + <span class="text-sm text-gray-700 dark:text-gray-200">Ordinal {record.ordinal}</span> + </td> + <td class="py-2 pr-4 whitespace-nowrap"> + {new Date(Number(record.timestamp) / 1000).toLocaleString()} + </td> + <td class="py-3 pr-4" + ><span + class="inline-block rounded-full bg-shade-l300 dark:bg-shade-d500 px-2.5 py-1 text-sm whitespace-nowrap" + >{record.kind.replaceAll('_', ' ')}</span + ></td + > + <td class="py-2 pr-4"> + {record.actor_user_id === undefined ? 'server' : `user ${record.actor_user_id}`} + </td> + <td class="py-2"> + <button + class="group cursor-pointer flex w-full items-center justify-between gap-4 rounded-md px-2 py-1 text-left hover:bg-shade-l300 dark:hover:bg-shade-d500 focus-visible:outline-2 focus-visible:outline-offset-2" + onclick={() => inspect(record)} + aria-label={`Inspect ${record.kind.replaceAll('_', ' ')} at operation ${record.op}, ordinal ${record.ordinal}`} + aria-haspopup="dialog" + > + <span class="break-words">{describeChange(record)}</span> + <span + class="shrink-0 text-sm text-gray-700 dark:text-gray-200 group-hover:text-current" + >Details</span + > + </button> + </td> + </tr> + {:else} + <tr> + <td colspan="5" class="py-5 text-center text-gray-700 dark:text-gray-200"> + {loading ? 'Loading changes...' : 'No changes match these filters.'} + </td> + </tr> + {/each} + </tbody> + </table> + {/if} +</div> + +{#if selectedRecord} + <dialog + use:openDetails + transition:fly={{ x: 16, duration: 120 }} + class="details-panel fixed inset-y-0 left-auto right-0 m-0 h-dvh max-h-none w-full max-w-2xl border-0 bg-shade-l100 dark:bg-shade-d700 p-0 text-color shadow-2xl" + aria-labelledby="change-details-title" + oncancel={(event) => { + event.preventDefault(); + selectedRecord = null; + }} + onclick={closeOnBackdrop} + > + <div class="flex h-full flex-col"> + <header class="shrink-0 border-b p-6"> + <div class="mb-4 flex items-center justify-between gap-4"> + <span + class="text-sm font-medium uppercase tracking-wider text-gray-700 dark:text-gray-200" + >Record details</span + > + <button + class="cursor-pointer rounded-lg px-3 py-2 text-sm hover:bg-shade-l300 dark:hover:bg-shade-d500" + onclick={() => { + selectedRecord = null; + }} + aria-label="Close record details">Close ×</button + > + </div> + <h2 id="change-details-title" class="text-xl font-semibold"> + {selectedRecord.kind.replaceAll('_', ' ')} + </h2> + <p class="mt-2 break-words text-sm text-gray-700 dark:text-gray-200"> + {describeChange(selectedRecord)} + </p> + </header> + <div class="min-h-0 flex-1 overflow-auto overscroll-contain p-6"> + <dl + class="mb-6 grid grid-cols-2 gap-4 rounded-xl border bg-white dark:bg-shade-d600 p-4 text-sm" + > + <div> + <dt class="mb-1 text-sm text-gray-700 dark:text-gray-200">Operation</dt> + <dd class="font-mono">{selectedRecord.op}</dd> + </div> + <div> + <dt class="mb-1 text-sm text-gray-700 dark:text-gray-200">Record ordinal</dt> + <dd class="font-mono">{selectedRecord.ordinal}</dd> + </div> + <div> + <dt class="mb-1 text-sm text-gray-700 dark:text-gray-200">Actor</dt> + <dd> + {selectedRecord.actor_user_id === undefined + ? 'Server' + : `User ${selectedRecord.actor_user_id}`} + </dd> + </div> + <div class="col-span-2"> + <dt class="mb-1 text-sm text-gray-700 dark:text-gray-200">Time</dt> + <dd>{new Date(Number(selectedRecord.timestamp) / 1000).toLocaleString()}</dd> + </div> + </dl> + {#if selectedOptions.length} + <section class="mb-6"> + <h3 class="mb-3 text-sm font-semibold">Options</h3> + <div class="overflow-hidden rounded-xl border bg-white dark:bg-shade-d600"> + {#each selectedOptions as option (option.key)} + <div class="border-b p-4 last:border-b-0"> + <div class="mb-2 flex flex-wrap items-center gap-2"> + <span class="font-mono text-sm font-medium">{option.key}</span> + {#if option.kind}<span + class="rounded bg-shade-l300 dark:bg-shade-d500 px-1.5 py-0.5 text-sm text-gray-700 dark:text-gray-200" + >{String(option.kind)}</span + >{/if} + {#if typeof option.explicit === 'boolean'}<span + class="text-sm text-gray-700 dark:text-gray-200" + >{option.explicit ? 'Explicit' : 'Default / derived'}</span + >{/if} + <button + class="cursor-pointer ml-auto rounded px-2 py-1 text-sm hover:bg-shade-l300 dark:hover:bg-shade-d500" + onclick={() => copy(option.value, option.key)} + aria-label={`Copy ${option.key}`}>Copy value</button + > + </div> + <pre + class="whitespace-pre-wrap break-all font-mono text-sm leading-relaxed">{displayValue( + option.value + )}</pre> + </div> + {/each} + </div> + </section> + {/if} + <section> + <h3 class="mb-3 text-sm font-semibold">Full payload</h3> + <pre + class="overflow-x-auto rounded-xl border bg-white dark:bg-shade-d800 p-4 font-mono text-sm leading-6" + aria-label="Full JSON payload">{JSON.stringify(selectedRecord, null, 2)}</pre> + <div class="mt-4 flex items-center gap-3"> + <Button variant="contained" onclick={() => copy(selectedRecord, 'Payload')} + >Copy payload</Button + > + <span role="status" class="text-sm text-gray-700 dark:text-gray-200" + >{copyFeedback}</span + > + </div> + </section> + </div> + </div> + </dialog> +{/if} + +<style> + .details-panel::backdrop { + background: rgb(0 0 0 / 25%); + } + + @media (prefers-reduced-motion: reduce) { + .details-panel { + animation-duration: 0.01ms !important; + } + } +</style>