[Build] Remove Scala overall regression test and post-migration cleanup (#4471)

* [Build] Drop Scala build toolchain from Flink and Console modules

Flink modules are fully Java after #4467; remove the common-scala-bridge
dependency, enumeratum/scalatest test deps, and scala-maven-plugin from
Flink parent, sqlclient, udf, and console-service. Bridge module is
retained for Spark until #4453.

* [Test][Common][Flink] Add migration regression tests and small cleanups

Add unit tests for AutoCloseUtils, ClientBeanUtils, SubmitRequest, and
ExitSecurityManager. Remove unused FlinkVersion.flinkLibs Scala bridge,
drop unused LoggerSupport from FlinkClient, and minor SubmitRequest tidy.


* [Common][Flink] Sonar-oriented fixes and expand migration regression tests

Replace printStackTrace and generic RuntimeException wraps with specific
exceptions, null-safe SubmitRequest accessors, FlinkShimsProxy URL helper,
and TrackId validation cleanup. Add ParameterCliTest, TrackIdTest, and
more SqlClient/SubmitRequest coverage.

* [Test] Keep unit-test-only coverage; revert production Sonar edits

Restore production code from the prior commit and retain focused unit
tests for ParameterCli, TrackId, SqlClient, and SubmitRequest. Complex
deploy-mode and cluster scenarios are deferred to E2E.


* [Common][Flink] Re-apply Sonar Java quality fixes with unit test sync

Restore production Sonar improvements (specific exceptions, null-safe
accessors, toUrl helper, lambda cleanup) while keeping unit tests focused
on pure logic; complex deploy scenarios remain deferred to E2E.

* [Flink] Fix shims Sonar issues in code and remove migration Sonar configs

Remove @SuppressWarnings and shims-specific sonar-project/.sonarcloud
properties added during Scala-to-Java migration; address deprecations
and unchecked casts directly in shims source instead.

* [Flink] Remove unused imports after TableContext refactor

Spotless check requires dropping stale Utils imports from TableContext.

* [Flink] Drop invalid @Override on FlinkStreamTableTrait.execute

execute(String) is not part of StreamTableEnvironment in all Flink
versions; keep the method public without @Override for compatibility.

* [Flink] Address Sonar new-code findings in shims-base

Fix unused parameters, replace generic exceptions, reduce cognitive
complexity, return empty collections, simplify regex matching, and
deduplicate TableExt conversion helpers per SonarCloud dev analysis.

* [Flink] Continue Sonar cleanup for shims, client, and SQL splitter

Refactor SqlSplitter with QueryExtractor, fix generic type names,
ParameterCli stdout emission, SubmitRequest exceptions, and remaining
Sonar findings from the dev new-code report.

* [Flink] Hoist shared Table API to traits and slim version shims contexts

Move common TableEnvironment delegates into FlinkTableTrait and
FlinkStreamTableTrait so version-specific TableContext and
StreamTableContext only retain legacy or version-gated APIs.
Refactor FlinkSqlValidator validation flow for lower complexity.

* [Flink] Hoist stream-table bridge APIs and collapse version contexts

Move Schema-based stream-table conversions and version-gated Table API
delegates into FlinkStreamTableTrait and FlinkTableTrait. Reduce
1.15–1.20 context classes to constructors plus version-specific legacy
overrides only where older Flink APIs require them.

* [Flink] Remove unchecked proxy casts and tighten error handling

Use Class.cast in FlinkShimsProxy.getObject and typed FlinkClient
invocations to drop @SuppressWarnings. Fix SqlClient unzip validation,
SubmitRequest exception handling, and FlinkSqlValidator caused-by parsing.

* [Flink] Clean up packer and kubernetes Sonar findings

Replace unchecked casts with Class.cast or typed YAML helpers, use
IllegalStateException for operational failures, and document enum constants
to remove @SuppressWarnings suppressions.

* [Flink] Replace RuntimeException in Yarn client deploy paths

* [Console] Replace Flink RuntimeException usages with typed failures

Use IllegalStateException with causes for Yarn/K8s deploy and watcher
paths, fix FlinkSql rollback error chaining, and remove unchecked JSON
casts in FlinkApplication via TypeReference.

* [Common] Replace RuntimeException with typed failures in shared utilities

Use IllegalStateException/IllegalArgumentException with context across common
util classes, refactor StringCastUtils with Class.cast, and align Console
shared infrastructure (Shiro, build task, Throws) for Sonar compliance.

* [Flink] Fix Sonar duplication and code smell findings for PR #4471

Extract shared Yarn jar upload and K8s Docker build helpers to reduce new-code
duplication below the 3% quality gate, rename pipeline enum constants to
UPPER_CASE, and address SqlSplitter/StringCastUtils Sonar issues.

* [Flink] Extract shared Yarn/K8s build pipeline steps to fix Sonar duplication

Hoist Yarn SQL build steps into BuildPipeline, K8s docker/pod-template steps
into AbstractK8sApplicationBuildPipeline, and AppControl factory to bring
new-code duplication below the 3% quality gate threshold.


* [Console] Extract shared application and build pipeline helpers to fix Sonar duplication

Consolidate duplicated logic between Flink/Spark entities and build pipeline services so PR #4471 can pass the new-code duplication quality gate.


* [Console] Further deduplicate entities, build pipelines, and watchers for Sonar

Move shared entity defaults to ApplicationEntitySupport, extract release outcome
and watcher init helpers to bring PR #4471 new-code duplication below 3%.

* [Common] Fix StringCastUtils generic cast and restore Flink build log level

Accept Class<?> in cast() so common_2.12 compiles with ConfigOption callers, and keep Flink dependency-merge failures logged at error while Spark stays at warn.
128 files changed
tree: 55a3c9ae786f9566da2c5c5d907d2e62b49ae1b8
  1. .github/
  2. .idea/
  3. .mvn/
  4. dist-material/
  5. docker/
  6. streampark-common/
  7. streampark-common-scala-bridge/
  8. streampark-console/
  9. streampark-e2e/
  10. streampark-flink/
  11. streampark-shaded/
  12. streampark-spark/
  13. tools/
  14. .asf.yaml
  15. .dlc.json
  16. .editorconfig
  17. .git-blame-ignore-revs
  18. .gitattributes
  19. .gitignore
  20. .gitmodules
  21. .licenserc.yaml
  22. AGENTS.md
  23. build.sh
  24. LICENSE
  25. mvnw
  26. mvnw.cmd
  27. NOTICE
  28. pom.xml
  29. qodana.yaml
  30. README.md
README.md

Apache StreamPark

StreamPark: a streaming application development framework and cloud-native real-time computing platform.

GitHub stars Latest release X Follow

📊 Overview

StreamPark is an open-source framework for building and managing real-time streaming applications, designed to simplify the end-to-end lifecycle of stream processing. It provides a unified development framework for Apache Flink and Apache Spark, along with a powerful cloud-native platform for application management.

StreamPark enables users to develop, debug, deploy, and operate large-scale streaming applications efficiently and consistently. Originally named StreamX, the project was renamed StreamPark in August 2022 and became an Apache Top-Level Project (TLP) in January 2025.

  • Streaming Application Development Framework
    • Simplifies Flink and Spark streaming development with prebuilt APIs, connectors, and templates.
  • Cloud-Native Real-Time Computing Platform
    • Offers a one-stop real-time computing platform for development, deployment, monitoring, and operations.
  • Unified Batch & Streaming Processing
    • Supports Apache Flink and Apache Spark, enabling both stream processing and batch processing.
  • Multi-Engine & Multi-Version Support
    • Run and manage multi-version Flink/Spark applications.
  • Multi-Environment Compatibility
    • Works on Standalone, YARN (Hadoop 2.x/3.x), and Kubernetes.
  • Rich Ecosystems
    • Compatible with big-data ecosystem tools (e.g., Apache Flink/Spark/Paimon/Doris) and ML/AI ecosystems.
  • Easy to use
    • Single-service deployment; go from zero to running jobs in minutes.

🚀 QuickStart

🐳 Play StreamPark in Docker

  docker run -d -p 10000:10000 apache/streampark:latest

🖥️ Local Quick Installation Experience

  curl -L https://streampark.apache.org/quickstart.sh | sh

https://github.com/user-attachments/assets/dd7d5a89-bc28-4ccc-9ad5-258925fc4f34

🔨 How to Build

 ./build.sh

🗄 how to Development

⬇️ Downloads

Please head to the releases page to download a release of Apache StreamPark.

📚 Docs

Official Documentation

💋 Our users

Various companies and organizations use Apache StreamPark for research, production and commercial products. Are you using this project? Welcome to add your company!

Our users

🤝 Contribution

PRs Welcome

🙋 Submit Pull Request and Issues

You can submit any ideas as pull requests or as issues.

If you're new to posting issues, we ask that you read How To Ask Questions The Smart Way (This guide does not provide actual support services for this project!), How to Report Bugs Effectively prior to posting. Well written bug reports help us help you!

🍻 How to Contribute

We welcome your suggestions, comments (including criticisms), comments and contributions. See How to Contribute and Code Submission Guide

Thank you to all the people who already contributed to Apache StreamPark!

💬 Contact Us

Contact us through the following mailing list.

NameMailing list
dev@streampark.apache.orgSubscribeUnsubscribeArchives

💿 Social Media


📜 License

Licensed under the Apache License, Version 2.0