| ## 9.3.0 |
| |
| #### Project |
| |
| * Bump up the embedded `swctl` version in OAP Docker image. |
| |
| #### OAP Server |
| |
| * Add component ID(133) for impala JDBC Java agent plugin and component ID(134) for impala server. |
| * Use prepareStatement in H2SQLExecutor#getByIDs.(No function change). |
| * Bump up snakeyaml to 1.32 for fixing CVE. |
| * Fix `DurationUtils.convertToTimeBucket` missed verify date format. |
| * Enhance LAL to support converting LogData to DatabaseSlowStatement. |
| * [**Breaking Change**] Change the LAL script format(Add layer property). |
| * Adapt ElasticSearch 8.1+, migrate from removed APIs to recommended APIs. |
| * Support monitoring MySQL slow SQLs. |
| * Support analyzing cache related spans to provide metrics and slow commands for cache services from client side |
| * Optimize virtual database, fix dynamic config watcher NPE when default value is null |
| * Remove physical index existing check and keep template existing check only to avoid meaningless `retry wait` |
| in `no-init` mode. |
| * Make sure instance list ordered in TTL processor to avoid TTL timer never runs. |
| * Support monitoring PostgreSQL slow SQLs. |
| * [**Breaking Change**] Support sharding MySQL database instances and tables |
| by [Shardingsphere-Proxy](https://shardingsphere.apache.org/document/current/en/overview/#shardingsphere-proxy). |
| SQL-Database requires removing tables `log_tag/segment_tag/zipkin_query` before OAP starts, if bump up from previous |
| releases. |
| * Fix meter functions `avgHistogram`, `avgHistogramPercentile`, `avgLabeled`, `sumHistogram` having data conflict when |
| downsampling. |
| * Do sorting `readLabeledMetricsValues` result forcedly in case the storage(database) doesn't return data consistent |
| with the parameter list. |
| * Fix the wrong watch semantics in Kubernetes watchers, which causes heavy traffic to API server in some Kubernetes |
| clusters, |
| we should use `Get State and Start at Most Recent` semantic instead of `Start at Exact` |
| because we don't need the changing history events, |
| see https://kubernetes.io/docs/reference/using-api/api-concepts/#semantics-for-watch. |
| * Unify query services and DAOs codes time range condition to `Duration`. |
| * [**Breaking Change**]: Remove prometheus-fetcher plugin, please use OpenTelemetry to scrape Prometheus metrics and |
| set up SkyWalking OpenTelemetry receiver instead. |
| * BugFix: histogram metrics sent to MAL should be treated as OpenTelemetry style, not Prometheus style: |
| ``` |
| (-infinity, explicit_bounds[i]] for i == 0 |
| (explicit_bounds[i-1], explicit_bounds[i]] for 0 < i < size(explicit_bounds) |
| (explicit_bounds[i-1], +infinity) for i == size(explicit_bounds) |
| ``` |
| * Support Golang runtime metrics analysis. |
| * Add APISIX metrics monitoring |
| * Support skywalking-client-js report empty `service version` and `page path` , set default version as `latest` and |
| default page path as `/`(root). Fix the |
| error `fetching data (/browser_app_page_pv0) : Can't split endpoint id into 2 parts`. |
| * [**Breaking Change**] Limit the max length of trace/log/alarm tag's `key=value`, set the max length of column `tags` |
| in tables`log_tag/segment_tag/alarm_record_tag` and column `query` in `zipkin_query` and column `tag_value` in `tag_autocomplete` to 256. |
| SQL-Database requires altering these columns' length or removing these tables before OAP starts, if bump up from previous releases. |
| * Optimize the creation conditions of profiling task. |
| * Lazy load the Kubernetes metadata and switch from event-driven to polling. |
| Previously we set up watchers to watch the Kubernetes metadata changes, this is perfect when there are deployments changes and |
| SkyWalking can react to the changes in real time. However when the cluster has many events (such as in large cluster |
| or some special Kubernetes engine like OpenShift), the requests sent from SkyWalking becomes unpredictable, i.e. SkyWalking might |
| send massive requests to Kubernetes API server, causing heavy load to the API server. |
| This PR switches from the watcher mechanism to polling mechanism, SkyWalking polls the metadata in a specified interval, |
| so that the requests sent to API server is predictable (~10 requests every `interval`, 3 minutes), and the requests count is constant |
| regardless of the cluster's changes. However with this change SkyWalking can't react to the cluster changes in time, but the delay |
| is acceptable in our case. |
| * Optimize the query time of tasks in ProfileTaskCache. |
| * Fix metrics was put into wrong slot of the window in the alerting kernel. |
| * Support `sumPerMinLabeled` in `MAL`. |
| * Bump up jackson databind, snakeyaml, grpc dependencies. |
| * Support export `Trace` and `Log` through Kafka. |
| * Add new config initialization mechanism of module provider. This is a ModuleManager lib kernel level change. |
| * [**Breaking Change**] Support new records query protocol, rename the column named `service_id` to `entity_id` for support difference entity. |
| Please re-create `top_n_database_statement` index/table. |
| * Remove improper self-obs metrics in JvmMetricsHandler(for Kafka channel). |
| * gRPC stream canceling code is not logged as an error when the client cancels the stream. The client |
| cancels the stream when the pod is terminated. |
| * [**Breaking Change**] Change the way of loading MAL rules(support pattern). |
| * Move k8s relative MAL files into `/otel-rules/k8s`. |
| * [**Breaking Change**] Refactor service mesh protobuf definitions and split TCP-related metrics to individual definition. |
| * Add `TCP{Service,ServiceInstance,ServiceRelation,ServiceInstanceRelation}` sources and split TCP-related entities out from |
| original `Service,ServiceInstance,ServiceRelation,ServiceInstanceRelation`. |
| * [**Breaking Change**] TCP-related source names are changed, fields of TCP-related sources are changed, please refer to the latest `oal/tcp.oal` file. |
| * Do not log error logs when failed to create ElasticSearch index because the index is created already. |
| * Add virtual MQ analysis for native traces. |
| * Support Python runtime metrics analysis. |
| * Support `sampledTrace` in LAL. |
| * Support multiple rules with different names under the same layer of LAL script. |
| * (Optimization) Reduce the buffer size(queue) of MAL(only) metric streams. Set L1 queue size as 1/20, L2 queue size as 1/2. |
| * Support monitoring MySQL/PostgreSQL in the cluster mode. |
| * [**Breaking Change**] Migrate to BanyanDB v0.2.0. |
| * Adopt new OR logical operator for, |
| 1. `MeasureIDs` query |
| 2. `BanyanDBProfileThreadSnapshotQueryDAO` query |
| 3. Multiple `Event` conditions query |
| 4. Metrics query |
| * Simplify Group check and creation |
| * Partially apply `UITemplate` changes |
| * Support `index_only` |
| * Return `CompletableFuture<Void>` directly from BanyanDB client |
| * Optimize data binary parse methods in *LogQueryDAO |
| * Support different indexType |
| * Support configuration for TTL and (block|segment) intervals |
| * Elasticsearch storage: Provide system environment variable(`SW_STORAGE_ES_SPECIFIC_INDEX_SETTINGS`) and support specify the settings `(number_of_shards/number_of_replicas)` for each index individually. |
| * Elasticsearch storage: Support update index settings `(number_of_shards/number_of_replicas)` for the index template after rebooting. |
| * Optimize MQ Topology analysis. Use entry span's peer from the consumer side as source service when no producer instrumentation(no cross-process reference). |
| * Refactor JDBC storage implementations to reuse logics. |
| * Fix `ClassCastException` in `LoggingConfigWatcher`. |
| * Support span attached event concept in Zipkin and SkyWalking trace query. |
| * Support span attached events on Zipkin lens UI. |
| * Force UTF-8 encoding in `JsonLogHandler` of `kafka-fetcher-plugin`. |
| * Fix max length to 512 of entity, instance and endpoint IDs in trace, log, profiling, topN tables(JDBC storages). The value was 200 by default. |
| * Add component IDs(135, 136, 137) for EventMesh server and client-side plugins. |
| * Bump up Kafka client to 2.8.1 to fix CVE-2021-38153. |
| * Remove `lengthEnvVariable` for `Column` as it never works as expected. |
| * Add `LongText` to support longer logs persistent as a text type in ElasticSearch, instead of a keyword, to avoid length limitation. |
| * Fix wrong system variable name `SW_CORE_ENABLE_ENDPOINT_NAME_GROUPING_BY_OPENAPI`. It was **opaenapi**. |
| * Fix not-time-series model blocking OAP boots in no-init mode. |
| * Fix `ShardingTopologyQueryDAO.loadServiceRelationsDetectedAtServerSide` invoke backend miss parameter `serviceIds`. |
| * Changed system variable `SW_SUPERDATASET_STORAGE_DAY_STEP` to `SW_STORAGE_ES_SUPER_DATASET_DAY_STEP` to be consistent with other ES storage related variables. |
| * Fix ESEventQueryDAO missing metric_table boolQuery criteria. |
| * Add default entity name(`_blank`) if absent to avoid NPE in the decoding. This caused `Can't split xxx id into 2 parts`. |
| * Support dynamic config the sampling strategy in network profiling. |
| * Zipkin module support BanyanDB storage. |
| * Zipkin traces query API, sort the result set by start time by default. |
| * Enhance the cache mechanism in the metric persistent process. |
| * This cache only worked when the metric is accessible(readable) from the database. Once the insert execution is delayed |
| due to the scale, the cache loses efficacy. It only works for the last time update per minute, considering our |
| 25s period. |
| * Fix ID conflicts for all JDBC storage implementations. Due to the insert delay, the JDBC storage implementation would |
| still generate another new insert statement. |
| * [**Breaking Change**] Remove `core/default/enableDatabaseSession` config. |
| * [**Breaking Change**] Add `@BanyanDB.TimestampColumn` to identify `which column in Record` is providing the timestamp(milliseconds) for BanyanDB, |
| since BanyanDB stream requires a timestamp in milliseconds. |
| For SQL-Database: add new column `timestamp` for tables `profile_task_log/top_n_database_statement`, |
| requires altering this column or removing these tables before OAP starts, if bump up from previous releases. |
| * Fix Elasticsearch storage: In `No-Sharding Mode`, add specific analyzer to the template before index creation to avoid update index error. |
| * Internal API: remove undocumented ElasticSearch API usage and use documented one. |
| * Fix `BanyanDB.ShardingKey` annotation missed in the generated OAL metrics classes. |
| * Fix Elasticsearch storage: Query `sortMetrics` missing transform real index column name. |
| * Rename `BanyanDB.ShardingKey` to `BanyanDB.SeriesID`. |
| * Self-Observability: Add counters for metrics reading from DB or cached. Dashboard:`Metrics Persistent Cache Count`. |
| * Self-Observability: Fix `GC Time` calculation. |
| * Fix Elasticsearch storage: In `No-Sharding Mode`, column's property `indexOnly` not applied and cannot be updated. |
| * Update the `trace_id` field as storage only(cannot be queried) in `top_n_database_statement`, `top_n_cache_read_command`, `top_n_cache_read_command` index. |
| |
| #### UI |
| |
| * Fix: tab active incorrectly, when click tab space |
| * Add impala icon for impala JDBC Java agent plugin. |
| * (Webapp)Bump up snakeyaml to 1.31 for fixing CVE-2022-25857 |
| * [Breaking Change]: migrate from Spring Web to Armeria, now you should use the environment variable |
| name `SW_OAP_ADDRESS` |
| to change the OAP backend service addresses, like `SW_OAP_ADDRESS=localhost:12800,localhost:12801`, and use |
| environment |
| variable `SW_SERVER_PORT` to change the port. Other Spring-related configurations don't take effect anymore. |
| * Polish the endpoint list graph. |
| * Fix styles for an adaptive height. |
| * Fix setting up a new time range after clicking the refresh button. |
| * Enhance the process topology graph to support dragging nodes. |
| * UI-template: Fix metrics calculation in `general-service/mesh-service/faas-function` top-list dashboard. |
| * Update MySQL dashboard to visualize collected slow SQLs. |
| * Add virtual cache dashboard. |
| * Remove `responseCode` fields of all OAL sources, as well as examples to avoid user's confusion. |
| * Remove All from the endpoints selector. |
| * Enhance menu configurations to make it easier to change. |
| * Update PostgreSQL dashboard to visualize collected slow SQLs. |
| * Add Golang runtime metrics and cpu/memory used rate panels in General-Instance dashboard. |
| * Add gateway apisix menu. |
| * Query logs with the specific service ID. |
| * Bump d3-color from 3.0.1 to 3.1.0. |
| * Add Golang runtime metrics and cpu/memory used rate panels in FaaS-Instance dashboard. |
| * Revert logs on trace widget. |
| * Add a sub-menu for virtual mq. |
| * Add `readRecords` to metric types. |
| * Verify dashboard names for new dashboards. |
| * Associate metrics with the trace widget on dashboards. |
| * Fix configuration panel styles. |
| * Remove a un-use icon. |
| * Support labeled value on the service/instance/endpoint list widgets. |
| * Add menu for virtual MQ. |
| * Set selector props and update configuration panel styles. |
| * Add Python runtime metrics and cpu/memory utilization panels to General-Instance and Fass-Instance dashboards. |
| * Enhance the legend of metrics graph widget with the summary table. |
| * Add apache eventMesh logo file. |
| * Fix conditions for trace profiling. |
| * Fix tag keys list and duration condition. |
| * Fix typo. |
| * Fix condition logic for trace tree data. |
| * Enhance tags component to search tags with the input value. |
| * Fix topology loading style. |
| * Fix update metric processor for the readRecords and remove readSampledRecords from metrics selector. |
| * Add trace association for FAAS dashboards. |
| * Visualize attached events on the trace widget. |
| * Add HTTP/1.x metrics and HTTP req/resp body collecting tabs on the network profiling widget. |
| * Implement creating tasks ui for network profiling widget. |
| * Fix entity types for ProcessRelation. |
| * Add trace association for general service dashboards. |
| |
| #### Documentation |
| |
| * Add `metadata-uid` setup doc about Kubernetes coordinator in the cluster management. |
| * Add a doc for adding menus to booster UI. |
| * Move general good read blogs from `Agent Introduction` to `Academy`. |
| * Add re-post for blog `Scaling with Apache SkyWalking` in the academy list. |
| * Add re-post for blog `Diagnose Service Mesh Network Performance with eBPF` in the academy list. |
| * Add **Security Notice** doc. |
| * Add new docs for `Report Span Attached Events` data collecting protocol. |
| * Add new docs for `Record` query protocol |
| * Update `Server Agents` and `Compatibility` for PHP agent. |
| * Add docs for profiling. |
| * Update the network profiling documentation. |
| |
| All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/149?closed=1) |
