Banyandb support update tag family (#13879)
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index 322ada7..ef2c809 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -279,7 +279,7 @@
* LAL: support full arithmetic (`+`, `-`, `*`, `/`) on numeric operands and fix the original bug where `(tag("x") as Integer) + (tag("y") as Integer)` was treated as string concatenation — expressions like `input_tokens + output_tokens < 10000` produced the concatenated string `"2589115"` rather than the integer sum `2704`, so token-threshold conditions never triggered `abort {}`. Operand types are now inferred from explicit casts (`as Integer` / `as Long` / `as Float` / `as Double`), typed proto fields, or numeric literal shape (with `L` / `F` / `D` suffix support, e.g. `1000L`). The compiler honours JLS-style binary numeric promotion and emits Java arithmetic in the declared primitive type — `(x as Integer) + (y as Integer)` compiles to `int + int` (not widened to `long`). `+` with any String operand falls back to string concatenation; `-` / `*` / `/` against non-numeric operands produces a compile-time error. The `as Double` and `as Float` casts are accepted in `typeCast` clauses, including in `def` declarations. Numeric comparisons honour declared casts on both sides (no more universal `h.toLong()` wrapper).
* Fix: `avgHistogramPercentile` / `sumHistogramPercentile` meter functions reported the smallest finite bucket boundary (e.g. `10` for OTel `gen_ai_server_request_duration` whose `le` is rewritten from `0.01s` → `10ms`) for every rank when no samples were observed in any bucket. The percentile loop's `count >= roof` check matched on the first sorted bucket because both sides were `0`. `calculate()` now short-circuits to `0` for every rank when the windowed total is `0`.
* Fix: MAL `expPrefix` now applies to every metric source in `exp`, not just the leading one. Previously the prefix was spliced after the first `.`, so secondary metrics inside arguments (e.g. the divisor in `a.sum(['s']).safeDiv(b.sum(['s']))`) silently skipped the prefix — a rule like envoy-ai-gateway's `request_latency_avg` (`sum / count`) would tag-rewrite only the dividend. The injection is now AST-aware: every bare-IDENTIFIER metric source is wrapped, while downsampling-type constants (`SUM`, `AVG`, `LATEST`, `SUM_PER_MIN`, `MAX`, `MIN`) are skipped.
-* Add `@Stream(allowBootReshape = true)` opt-in for additive boot-time reshape of BanyanDB streams / measures. Code-defined stream classes (e.g. `AlarmRecord`) can now annotate their schema as eligible for in-place additive update at OAP boot — a new `@Column` is appended to the live tag-family / fields via `client.update` instead of being silently rejected with `SKIPPED_SHAPE_MISMATCH` (which previously forced operators to drop the measure / stream and lose historical rows). The opt-in is per-stream and gated by an `isPurelyAdditive` shape diff: type changes, drops, kind flips, entity / interval / sharding-key changes, and field re-typing still skip with `SKIPPED_SHAPE_MISMATCH`, so identity-breaking edits remain explicit operator actions. Only the init / standalone OAP performs the reshape; non-init peers continue through the existing poll-and-wait loop so a single node drives DDL. `AlarmRecord` is opted in. Default remains `false` for all other models — boot-time reshape stays off unless the annotation is explicitly set.
+* Add `@Stream(allowBootReshape = true)` opt-in for additive boot-time reshape of BanyanDB streams / measures. Code-defined stream classes (e.g. `AlarmRecord`) can now annotate their schema as eligible for in-place additive update at OAP boot — a new `@Column` is appended to the live tag-family / fields via `client.update` instead of being silently rejected with `SKIPPED_SHAPE_MISMATCH` (which previously forced operators to drop the measure / stream and lose historical rows). Additive includes both new tags / fields **and** relocating an existing tag between families when a `@Column`'s `storageOnly` flag flips (e.g. `id1` moving from `storage-only` → `searchable` when it becomes indexed). The opt-in is per-stream and gated by an `isPurelyAdditive` shape diff: tag type changes, tag drops, kind flips (tag↔field), entity / interval / sharding-key changes, and field re-typing still skip with `SKIPPED_SHAPE_MISMATCH`, so identity-breaking edits remain explicit operator actions. Only the init / standalone OAP performs the reshape; non-init peers continue through the existing poll-and-wait loop so a single node drives DDL. When a `check*` records `SKIPPED_SHAPE_MISMATCH` the dependent `IndexRule` / `IndexRuleBinding` reconciliation is also skipped — preventing the previous gap where the binding silently updated to a tag list that diverged from the live tag-family layout. `AlarmRecord` is opted in. Default remains `false` for all other models — boot-time reshape stays off unless the annotation is explicitly set. **Operator caveat:** BanyanDB does not physically migrate existing rows when a tag's family changes; pre-existing data stays in its original on-disk location while new writes go to the declared family — expect a backfill window for queries that route through new IndexRules on relocated tags.
#### UI
* Add mobile menu icon and i18n labels for the iOS layer.
diff --git a/docs/en/concepts-and-designs/runtime-rule-hot-update.md b/docs/en/concepts-and-designs/runtime-rule-hot-update.md
index d20fc5e..78ee77f 100644
--- a/docs/en/concepts-and-designs/runtime-rule-hot-update.md
+++ b/docs/en/concepts-and-designs/runtime-rule-hot-update.md
@@ -169,18 +169,43 @@
Streams whose schema lives in OAP source code (e.g. `AlarmRecord`) can opt in
to **additive** boot-time reshape via
`@Stream(allowBootReshape = true)`. When the flag is on and the diff is
-purely additive (new tag / new field; no type changes, no drops, no entity /
-interval / sharding-key flips), the installer calls `client.update` at boot
-to append the new column to the live measure / stream; non-additive
-divergences still record `SKIPPED_SHAPE_MISMATCH` and require an operator
-drop+recreate. Only the init / standalone OAP performs the reshape; non-init
-peers continue through the existing poll-and-wait loop so a single node
-drives DDL during a rolling restart.
+purely additive, the installer calls `client.update` at boot to extend the
+live measure / stream; non-additive divergences still record
+`SKIPPED_SHAPE_MISMATCH` and require an operator drop+recreate. Only the
+init / standalone OAP performs the reshape; non-init peers continue through
+the existing poll-and-wait loop so a single node drives DDL during a rolling
+restart.
+
+"Additive" includes two cases:
+
+1. **New tag / new field** — a brand-new `@Column` is appended to the live
+ tag family (or fields list, for measures).
+2. **Tag relocation between families** — a `@Column`'s `storageOnly` flag
+ flips, moving the tag between the `storage-only` and `searchable`
+ families. The tag identity and type are preserved; only its on-disk
+ family location changes.
+
+Drops, tag-type changes, kind flips (tag↔field), and entity / interval /
+sharding-key changes are still rejected with `SKIPPED_SHAPE_MISMATCH`.
+
+When the primary `check*` records `SKIPPED_SHAPE_MISMATCH`, the dependent
+`IndexRule` and `IndexRuleBinding` reconciliation is **also skipped**. This
+preserves coherence between the stream / measure tag layout and the binding
+that points into it — without the gate, the binding would silently update to
+reference the new declared tag list while the live tag families still carry
+the old shape, leaving operators with a binding routing to tags that don't
+exist in the live family layout.
This opt-in is **BanyanDB-only**. JDBC and Elasticsearch are append-only on
the data path and already accept additive column / mapping additions at boot
without operator intervention, so the flag is unread on those backends.
+> **Operator caveat:** BanyanDB does not physically migrate existing rows
+> when a tag's family changes. Pre-existing data for the relocated tag stays
+> in its original on-disk family location; new writes go to the declared
+> family. Queries that route through a new IndexRule on the relocated tag
+> will only see post-reshape rows until historical data ages out via TTL.
+
## On-demand workflow
Triggered by an HTTP call to one of the admin endpoints. A request arriving at any
diff --git a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBIndexInstaller.java b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBIndexInstaller.java
index 0922149..47ceac8 100644
--- a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBIndexInstaller.java
+++ b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBIndexInstaller.java
@@ -160,37 +160,46 @@
return installInfo;
}
if (runShapeChecks) {
- checkTrace(traceModel.getTrace(), c, opt);
- checkIndexRules(model.getName(), traceModel.getIndexRules(), c, opt);
- checkIndexRuleBinding(
- traceModel.getIndexRules(), metadata.getGroup(), metadata.name(),
- BanyandbCommon.Catalog.CATALOG_TRACE, c, opt
- );
+ if (checkTrace(traceModel.getTrace(), c, opt)) {
+ checkIndexRules(model.getName(), traceModel.getIndexRules(), c, opt);
+ checkIndexRuleBinding(
+ traceModel.getIndexRules(), metadata.getGroup(), metadata.name(),
+ BanyandbCommon.Catalog.CATALOG_TRACE, c, opt
+ );
+ } else {
+ skipDependentReconcile(opt, "trace", metadata.name());
+ }
}
} else {
// stream
StreamModel streamModel = MetadataRegistry.INSTANCE.registerStreamModel(
model, config);
if (runShapeChecks) {
- checkStream(model, streamModel.getStream(), c, opt);
- checkIndexRules(model.getName(), streamModel.getIndexRules(), c, opt);
- checkIndexRuleBinding(
- streamModel.getIndexRules(), metadata.getGroup(), metadata.name(),
- BanyandbCommon.Catalog.CATALOG_STREAM, c, opt
- );
- // Stream not support server side TopN pre-aggregation
+ if (checkStream(model, streamModel.getStream(), c, opt)) {
+ checkIndexRules(model.getName(), streamModel.getIndexRules(), c, opt);
+ checkIndexRuleBinding(
+ streamModel.getIndexRules(), metadata.getGroup(), metadata.name(),
+ BanyandbCommon.Catalog.CATALOG_STREAM, c, opt
+ );
+ // Stream not support server side TopN pre-aggregation
+ } else {
+ skipDependentReconcile(opt, "stream", metadata.name());
+ }
}
}
} else { // measure
MeasureModel measureModel = MetadataRegistry.INSTANCE.registerMeasureModel(model, config, downSamplingConfigService);
if (runShapeChecks) {
- checkMeasure(model, measureModel.getMeasure(), c, opt);
- checkIndexRules(model.getName(), measureModel.getIndexRules(), c, opt);
- checkIndexRuleBinding(
- measureModel.getIndexRules(), metadata.getGroup(), metadata.name(),
- BanyandbCommon.Catalog.CATALOG_MEASURE, c, opt
- );
- checkTopNAggregation(model, c, opt);
+ if (checkMeasure(model, measureModel.getMeasure(), c, opt)) {
+ checkIndexRules(model.getName(), measureModel.getIndexRules(), c, opt);
+ checkIndexRuleBinding(
+ measureModel.getIndexRules(), metadata.getGroup(), metadata.name(),
+ BanyandbCommon.Catalog.CATALOG_MEASURE, c, opt
+ );
+ checkTopNAggregation(model, c, opt);
+ } else {
+ skipDependentReconcile(opt, "measure", metadata.name());
+ }
}
}
} else {
@@ -770,13 +779,22 @@
* NOT reshape the backend by default — reshape is an explicit operator action.
*
* <p>Exception: when the model opts in via {@link Model#isAllowBootReshape()} and the diff
- * is purely additive (new tag / new field, no type changes, no drops, identity preserved),
- * the init OAP is allowed to apply the additive update during boot. Non-init OAPs continue
- * through the poll-and-wait loop in
+ * is purely additive (new tag / new field, or tag relocation between families via a
+ * {@code storageOnly} toggle; no type changes, no drops, identity preserved), the init OAP
+ * is allowed to apply the additive update during boot. Non-init OAPs continue through the
+ * poll-and-wait loop in
* {@link org.apache.skywalking.oap.server.core.storage.model.ModelInstaller#whenCreating}
* so only one node races on the DDL.
+ *
+ * @return {@code true} when the live measure is now aligned with the declared shape
+ * (either it already matched, or the installer successfully applied an update);
+ * {@code false} when the shape diverged and the installer recorded
+ * {@link StorageManipulationOpt.Outcome#SKIPPED_SHAPE_MISMATCH}. Callers use the
+ * return value to skip dependent resources (index rules, binding, TopN) so a
+ * non-additive divergence doesn't leave the binding pointing at a stream/measure
+ * that no longer agrees with it.
*/
- private void checkMeasure(Model model, Measure measure, BanyanDBClient client, StorageManipulationOpt opt) throws BanyanDBException {
+ private boolean checkMeasure(Model model, Measure measure, BanyanDBClient client, StorageManipulationOpt opt) throws BanyanDBException {
Measure hisMeasure = client.findMeasure(measure.getMetadata().getGroup(), measure.getMetadata().getName());
if (hisMeasure == null) {
throw new IllegalStateException("Measure: " + measure.getMetadata().getName() + " exist but can't find it from BanyanDB server");
@@ -795,8 +813,8 @@
hisMeasure.getMetadata().getName(), hisMeasure, measure);
opt.recordOutcome("measure", hisMeasure.getMetadata().getName(),
StorageManipulationOpt.Outcome.UPDATED,
- "additive boot reshape: new tag / field added");
- return;
+ "additive boot reshape: new tag / field added or tag relocated between families");
+ return true;
}
log.error("BanyanDB measure {} shape mismatch at boot — backend holds a "
+ "different shape than the declared rule. SKIPPING metric; operator "
@@ -806,21 +824,28 @@
opt.recordOutcome("measure", hisMeasure.getMetadata().getName(),
StorageManipulationOpt.Outcome.SKIPPED_SHAPE_MISMATCH,
"backend shape differs from declared shape; use /runtime/rule/addOrUpdate to reshape");
- return;
+ return false;
}
// banyanDB server can not delete or update Tags.
opt.recordModRevision(client.update(measure));
log.info("update Measure: {} from: {} to: {}", hisMeasure.getMetadata().getName(), hisMeasure, measure);
}
}
+ return true;
}
/**
- * Check if the stream exists and update (or record shape mismatch) per mode.
- * See {@link #checkMeasure} for the create-if-absent vs full-install contract,
- * including the {@link Model#isAllowBootReshape()} additive opt-in.
+ * Check if the stream exists and update (or record shape mismatch) per mode. See
+ * {@link #checkMeasure} for the create-if-absent vs full-install contract, including the
+ * {@link Model#isAllowBootReshape()} additive opt-in.
+ *
+ * @return {@code true} when the live stream is now aligned with the declared shape
+ * (already matched or successfully updated); {@code false} when the installer
+ * recorded {@link StorageManipulationOpt.Outcome#SKIPPED_SHAPE_MISMATCH}. See
+ * {@link #checkMeasure} for why callers must gate dependent index-rule /
+ * binding reconciliation on this signal.
*/
- private void checkStream(Model model, Stream stream, BanyanDBClient client, StorageManipulationOpt opt) throws BanyanDBException {
+ private boolean checkStream(Model model, Stream stream, BanyanDBClient client, StorageManipulationOpt opt) throws BanyanDBException {
Stream hisStream = client.findStream(stream.getMetadata().getGroup(), stream.getMetadata().getName());
if (hisStream == null) {
throw new IllegalStateException("Stream: " + stream.getMetadata().getName() + " exist but can't find it from BanyanDB server");
@@ -839,8 +864,8 @@
hisStream.getMetadata().getName(), hisStream, stream);
opt.recordOutcome("stream", hisStream.getMetadata().getName(),
StorageManipulationOpt.Outcome.UPDATED,
- "additive boot reshape: new tag added");
- return;
+ "additive boot reshape: new tag added or tag relocated between families");
+ return true;
}
log.error("BanyanDB stream {} shape mismatch at boot — backend holds a "
+ "different shape than the declared rule. SKIPPING; operator must "
@@ -849,12 +874,13 @@
opt.recordOutcome("stream", hisStream.getMetadata().getName(),
StorageManipulationOpt.Outcome.SKIPPED_SHAPE_MISMATCH,
"backend shape differs from declared shape; use /runtime/rule/addOrUpdate to reshape");
- return;
+ return false;
}
opt.recordModRevision(client.update(stream));
log.info("update Stream: {} from: {} to: {}", hisStream.getMetadata().getName(), hisStream, stream);
}
}
+ return true;
}
/**
@@ -878,12 +904,48 @@
}
/**
- * Purely-additive diff for a BanyanDB {@link Stream}: declared may add tag families or
- * tags, but every tag-family / tag that already exists on the backend must be present
- * with the same name and {@link BanyandbDatabase.TagType type}, the {@link BanyandbDatabase.Entity entity}
- * column list must match exactly (reshape can't change shard / series-id semantics),
- * and no tag may be dropped. Returns false for any non-additive divergence so the caller
- * falls back to {@link StorageManipulationOpt.Outcome#SKIPPED_SHAPE_MISMATCH}.
+ * Record a parallel {@link StorageManipulationOpt.Outcome#SKIPPED_SHAPE_MISMATCH} for the
+ * dependent {@code indexRule} + {@code indexRuleBinding} resources of a stream / measure
+ * / trace whose primary {@code check*} just skipped. Calling
+ * {@code checkIndexRules} / {@code checkIndexRuleBinding} unconditionally after a primary
+ * skip would silently update the binding to reference the new declared rule list while
+ * the underlying schema still carries the old shape — operators end up with a binding
+ * pointing at tags / fields that don't agree with the live tag family layout (e.g. a tag
+ * was dropped from the declared model but kept on the backend, the binding loses its
+ * reference, and the orphan IndexRule becomes unqueryable).
+ *
+ * <p>Skipping the dependent reconcile keeps live state coherent: either everything
+ * matches the declared shape, or nothing on this resource is touched until the operator
+ * drops + recreates. The resource-type labels (`indexRule`, `indexRuleBinding`) match the
+ * names {@link StorageManipulationOpt.ResourceOutcome} uses elsewhere so operator-facing
+ * outcome filtering stays consistent.
+ *
+ * <p>{@code TopNAggregation} doesn't need a parallel skip — it's only invoked for
+ * measures, only when the primary {@code checkMeasure} returns {@code true}, and its
+ * own gating cascades through the dispatch in {@link #isExists}.
+ */
+ private void skipDependentReconcile(StorageManipulationOpt opt, String resourceType, String resourceName) {
+ log.warn("BanyanDB {} {} shape mismatch — skipping dependent IndexRule / IndexRuleBinding "
+ + "reconciliation to avoid partial reshape (binding would point at the new tag "
+ + "list while the live tag families still carry the old shape).",
+ resourceType, resourceName);
+ opt.recordOutcome("indexRule", resourceName,
+ StorageManipulationOpt.Outcome.SKIPPED_SHAPE_MISMATCH,
+ resourceType + " shape mismatch; index-rule reconcile skipped");
+ opt.recordOutcome("indexRuleBinding", resourceName,
+ StorageManipulationOpt.Outcome.SKIPPED_SHAPE_MISMATCH,
+ resourceType + " shape mismatch; binding reconcile skipped");
+ }
+
+ /**
+ * Purely-additive diff for a BanyanDB {@link Stream}: declared may add tags or relocate
+ * existing tags between families (a {@code storageOnly} toggle on a {@code @Column}
+ * moves a tag between {@code storage-only} and {@code searchable}; the tag identity is
+ * preserved, only its on-disk family location changes). The {@link BanyandbDatabase.Entity entity}
+ * column list must still match exactly (reshape can't change shard / series-id semantics),
+ * existing tag types may not change, and no tag may be dropped. Returns false for any
+ * non-additive divergence so the caller falls back to
+ * {@link StorageManipulationOpt.Outcome#SKIPPED_SHAPE_MISMATCH}.
*/
private boolean isPurelyAdditiveStream(Stream declared, Stream live) {
if (!declared.getEntity().equals(live.getEntity())) {
@@ -928,23 +990,37 @@
return true;
}
+ /**
+ * Tag-family compatibility check used by {@link #isPurelyAdditiveStream} /
+ * {@link #isPurelyAdditiveMeasure}. The check is name+type oriented, not family-position
+ * oriented — a tag may move between families (e.g. a {@code @Column} flips from
+ * {@code storageOnly = true} → {@code false}, which relocates it from the
+ * {@code storage-only} family to {@code searchable}) and is still considered safe to
+ * apply at boot. Drops (tag missing from declared entirely) and type changes still
+ * return false.
+ *
+ * <p><strong>Operator caveat:</strong> BanyanDB does NOT physically migrate existing
+ * rows when a tag's family changes. Pre-existing data for that tag stays in the old
+ * family's on-disk segment; new writes go to the declared family. Queries that route
+ * through a new IndexRule on the relocated tag will only see post-reshape rows.
+ * Operators should expect a backfill window after a storageOnly toggle.
+ */
private boolean isPurelyAdditiveTagFamilies(List<BanyandbDatabase.TagFamilySpec> declared,
List<BanyandbDatabase.TagFamilySpec> live) {
- final Map<String, BanyandbDatabase.TagFamilySpec> declaredByName = declared.stream()
- .collect(Collectors.toMap(BanyandbDatabase.TagFamilySpec::getName, f -> f, (a, b) -> a));
+ // Collapse declared tags across all families: (name -> TagSpec). A tag is allowed
+ // to move between families, so a per-family lookup would falsely reject the move.
+ final Map<String, BanyandbDatabase.TagSpec> declaredTagsByName = declared.stream()
+ .flatMap(f -> f.getTagsList().stream())
+ .collect(Collectors.toMap(BanyandbDatabase.TagSpec::getName, t -> t, (a, b) -> a));
for (BanyandbDatabase.TagFamilySpec liveFamily : live) {
- BanyandbDatabase.TagFamilySpec declaredFamily = declaredByName.get(liveFamily.getName());
- if (declaredFamily == null) {
- return false;
- }
- final Map<String, BanyandbDatabase.TagSpec> declaredTags = declaredFamily.getTagsList().stream()
- .collect(Collectors.toMap(BanyandbDatabase.TagSpec::getName, t -> t, (a, b) -> a));
for (BanyandbDatabase.TagSpec liveTag : liveFamily.getTagsList()) {
- BanyandbDatabase.TagSpec declaredTag = declaredTags.get(liveTag.getName());
+ BanyandbDatabase.TagSpec declaredTag = declaredTagsByName.get(liveTag.getName());
if (declaredTag == null) {
+ // Tag dropped entirely from the declared model — non-additive.
return false;
}
if (declaredTag.getType() != liveTag.getType()) {
+ // Type changed — non-additive, requires drop+recreate.
return false;
}
}
@@ -952,7 +1028,12 @@
return true;
}
- private void checkTrace(Trace trace, BanyanDBClient client, StorageManipulationOpt opt) throws BanyanDBException {
+ /**
+ * @return {@code true} when the live trace is now aligned with the declared shape;
+ * {@code false} on {@link StorageManipulationOpt.Outcome#SKIPPED_SHAPE_MISMATCH}.
+ * See {@link #checkMeasure} for the dependent-resource gating rationale.
+ */
+ private boolean checkTrace(Trace trace, BanyanDBClient client, StorageManipulationOpt opt) throws BanyanDBException {
Trace hisTrace = client.findTrace(trace.getMetadata().getGroup(), trace.getMetadata().getName());
if (hisTrace == null) {
throw new IllegalStateException("Trace: " + trace.getMetadata().getName() + " exist but can't find it from BanyanDB server");
@@ -972,12 +1053,13 @@
opt.recordOutcome("trace", hisTrace.getMetadata().getName(),
StorageManipulationOpt.Outcome.SKIPPED_SHAPE_MISMATCH,
"backend shape differs from declared shape; use /runtime/rule/addOrUpdate to reshape");
- return;
+ return false;
}
opt.recordModRevision(client.update(trace));
log.info("update Trace: {} from: {} to: {}", hisTrace.getMetadata().getName(), hisTrace, trace);
}
}
+ return true;
}
/**
diff --git a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBIT.java b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBIT.java
index 58e19d4..5ddd1d2 100644
--- a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBIT.java
+++ b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBIT.java
@@ -609,6 +609,155 @@
}
/**
+ * Toggling {@code storageOnly} on an existing {@code @Column} moves the tag from
+ * {@code storage-only} → {@code searchable} (or vice versa). Although the live tag
+ * family no longer contains the tag at its old position, the tag identity + type are
+ * preserved, so {@link BanyanDBIndexInstaller#isPurelyAdditiveStream} (via
+ * {@code isPurelyAdditiveTagFamilies}) should accept the relocation when
+ * {@code allowBootReshape = true} and the OAP is in the init / standalone path. The
+ * dependent IndexRule for the now-indexed tag should also be created.
+ */
+ @Test
+ public void testStreamStorageOnlyTogglePathBootReshape() throws Exception {
+ DownSamplingConfigService downSamplingConfigService = new DownSamplingConfigService(Arrays.asList("minute"));
+ ModuleManager moduleManager = mock(ModuleManager.class);
+ ModuleProviderHolder moduleProviderHolder = mock(ModuleProviderHolder.class);
+ ModuleServiceHolder moduleServiceHolder = mock(ModuleServiceHolder.class);
+ when(moduleManager.find(CoreModule.NAME)).thenReturn(moduleProviderHolder);
+ when(moduleProviderHolder.provider()).thenReturn(moduleServiceHolder);
+ when(moduleServiceHolder.getService(DownSamplingConfigService.class)).thenReturn(downSamplingConfigService);
+
+ StorageModels models = new StorageModels();
+ Model baseModel = models.add(TestStreamStorageOnly.class, DefaultScopeDefine.SERVICE,
+ new Storage("relocStream", true, DownSampling.Second),
+ StorageManipulationOpt.withSchemaChange());
+ BanyanDBIndexInstaller installer = new BanyanDBIndexInstaller(client, moduleManager, config);
+ installer.isExists(baseModel, StorageManipulationOpt.withSchemaChange());
+ installer.createTable(baseModel);
+
+ String groupName = MetadataRegistry.convertGroupName(
+ config.getGlobal().getNamespace(), BanyanDB.StreamGroup.RECORDS_LOG.getName());
+ BanyandbDatabase.Stream initial = client.client.findStream(groupName, "relocStream");
+ // payload starts in storage-only family
+ assertTrue(initial.getTagFamiliesList().stream()
+ .filter(f -> "storage-only".equals(f.getName()))
+ .flatMap(f -> f.getTagsList().stream())
+ .anyMatch(t -> "payload".equals(t.getName())),
+ "expected payload tag in storage-only family initially, got " + initial);
+
+ models.remove(TestStreamStorageOnly.class, StorageManipulationOpt.withSchemaChange());
+ Model reshapedModel = models.add(TestStreamStorageOnlyOff.class, DefaultScopeDefine.SERVICE,
+ new Storage("relocStream", true, DownSampling.Second),
+ StorageManipulationOpt.withSchemaChange());
+ assertTrue(reshapedModel.isAllowBootReshape());
+
+ StorageManipulationOpt bootOpt = StorageManipulationOpt.schemaCreateIfAbsent();
+ new BanyanDBIndexInstaller(client, moduleManager, config).isExists(reshapedModel, bootOpt);
+
+ BanyandbDatabase.Stream reshaped = client.client.findStream(groupName, "relocStream");
+ // payload is now in searchable family
+ assertTrue(reshaped.getTagFamiliesList().stream()
+ .filter(f -> "searchable".equals(f.getName()))
+ .flatMap(f -> f.getTagsList().stream())
+ .anyMatch(t -> "payload".equals(t.getName())),
+ "expected payload tag relocated to searchable family after reshape, got " + reshaped);
+ assertFalse(reshaped.getTagFamiliesList().stream()
+ .filter(f -> "storage-only".equals(f.getName()))
+ .flatMap(f -> f.getTagsList().stream())
+ .anyMatch(t -> "payload".equals(t.getName())),
+ "expected payload tag no longer in storage-only family after reshape, got " + reshaped);
+
+ boolean updatedRecorded = bootOpt.getOutcomes().stream()
+ .anyMatch(o -> "stream".equals(o.getResourceType())
+ && "relocStream".equals(o.getResourceName())
+ && o.getStatus() == StorageManipulationOpt.Outcome.UPDATED);
+ assertTrue(updatedRecorded, "expected UPDATED outcome for storageOnly relocation, got " + bootOpt.getOutcomes());
+
+ // Relocation is "additive" → checkStream returns true → dependent index-rule
+ // reconciliation runs. Verify the IndexRule for the newly-indexed `payload` tag was
+ // created and that the IndexRuleBinding now references it. This is the behavior the
+ // dependent-reconcile gate is supposed to permit when the primary shape change is
+ // accepted.
+ BanyandbDatabase.IndexRule payloadIndexRule = client.client.findIndexRule(groupName, "payload");
+ assertNotNull(payloadIndexRule, "expected IndexRule 'payload' to be created after relocation");
+ BanyandbDatabase.IndexRuleBinding binding = client.client.findIndexRuleBinding(groupName, "relocStream");
+ assertNotNull(binding, "expected IndexRuleBinding for relocStream to be present after relocation");
+ assertTrue(binding.getRulesList().contains("payload"),
+ "expected IndexRuleBinding to reference 'payload', got rules=" + binding.getRulesList());
+ }
+
+ /**
+ * Initial state for {@link #testStreamStorageOnlyTogglePathBootReshape}: {@code payload}
+ * declared with {@code storageOnly = true}, so it lands in the {@code storage-only}
+ * tag family.
+ */
+ @Stream(name = "relocStream", scopeId = DefaultScopeDefine.SERVICE,
+ builder = TestStreamStorageOnly.Builder.class, processor = RecordStreamProcessor.class)
+ @BanyanDB.Group(streamGroup = BanyanDB.StreamGroup.RECORDS_LOG)
+ @BanyanDB.TimestampColumn("timestamp")
+ private static class TestStreamStorageOnly extends Record {
+ @Column(name = "service_id")
+ @BanyanDB.SeriesID(index = 0)
+ private String serviceId;
+ @Column(name = "payload", storageOnly = true)
+ private String payload;
+ @Column(name = "timestamp")
+ private long timestamp;
+
+ @Override
+ public StorageID id() {
+ return new StorageID();
+ }
+
+ static class Builder implements StorageBuilder<StorageData> {
+ @Override
+ public StorageData storage2Entity(final Convert2Entity converter) {
+ return null;
+ }
+
+ @Override
+ public void entity2Storage(final StorageData entity, final Convert2Storage converter) {
+ }
+ }
+ }
+
+ /**
+ * Reshape target: same {@code payload} column, but {@code storageOnly} is gone so the
+ * tag relocates to the {@code searchable} family. Opted in via
+ * {@code allowBootReshape = true}.
+ */
+ @Stream(name = "relocStream", scopeId = DefaultScopeDefine.SERVICE,
+ builder = TestStreamStorageOnlyOff.Builder.class, processor = RecordStreamProcessor.class,
+ allowBootReshape = true)
+ @BanyanDB.Group(streamGroup = BanyanDB.StreamGroup.RECORDS_LOG)
+ @BanyanDB.TimestampColumn("timestamp")
+ private static class TestStreamStorageOnlyOff extends Record {
+ @Column(name = "service_id")
+ @BanyanDB.SeriesID(index = 0)
+ private String serviceId;
+ @Column(name = "payload")
+ private String payload;
+ @Column(name = "timestamp")
+ private long timestamp;
+
+ @Override
+ public StorageID id() {
+ return new StorageID();
+ }
+
+ static class Builder implements StorageBuilder<StorageData> {
+ @Override
+ public StorageData storage2Entity(final Convert2Entity converter) {
+ return null;
+ }
+
+ @Override
+ public void entity2Storage(final StorageData entity, final Convert2Storage converter) {
+ }
+ }
+ }
+
+ /**
* Non-additive variant: {@code tag} is now {@code long} (TAG_TYPE_INT) where the live
* stream has it as {@code String} (TAG_TYPE_STRING). Boot must refuse to reshape even
* with {@code allowBootReshape = true}.
diff --git a/test/e2e-v2/cases/alarm/mysql/e2e.yaml b/test/e2e-v2/cases/alarm/mysql/e2e.yaml
index 2dab123..eec8f1e 100644
--- a/test/e2e-v2/cases/alarm/mysql/e2e.yaml
+++ b/test/e2e-v2/cases/alarm/mysql/e2e.yaml
@@ -41,7 +41,7 @@
verify:
retry:
count: 20
- interval: 3s
+ interval: 10s
cases:
- includes:
- ../alarm-cases.yaml
diff --git a/test/e2e-v2/cases/alarm/postgres/e2e.yaml b/test/e2e-v2/cases/alarm/postgres/e2e.yaml
index 2dab123..eec8f1e 100644
--- a/test/e2e-v2/cases/alarm/postgres/e2e.yaml
+++ b/test/e2e-v2/cases/alarm/postgres/e2e.yaml
@@ -41,7 +41,7 @@
verify:
retry:
count: 20
- interval: 3s
+ interval: 10s
cases:
- includes:
- ../alarm-cases.yaml
diff --git a/test/e2e-v2/script/env b/test/e2e-v2/script/env
index debb001..0944fdd 100644
--- a/test/e2e-v2/script/env
+++ b/test/e2e-v2/script/env
@@ -23,7 +23,7 @@
SW_AGENT_CLIENT_JS_TEST_COMMIT=4f1eb1dcdbde3ec4a38534bf01dded4ab5d2f016
SW_KUBERNETES_COMMIT_SHA=da0e267f877b9b8e5f7728ae4ea7dc7723a2a073
SW_ROVER_COMMIT=79292fe07f17f98f486e0c4471213e1961fb2d1d
-SW_BANYANDB_COMMIT=69c8f4d20ebb6532ea4c16a7ed7114dd6ec9770b
+SW_BANYANDB_COMMIT=84b919efca3fee3d51df9e97a734a9f10ae6f1d2
SW_AGENT_PHP_COMMIT=d1114e7be5d89881eec76e5b56e69ff844691e35
SW_PREDICTOR_COMMIT=54a0197654a3781a6f73ce35146c712af297c994
