Apache StreamPark — Agent Instructions

Project conventions, architecture, and coding patterns for the StreamPark codebase.

Architecture

Module Boundaries

StreamPark is a Maven multi-module project with four top-level modules. Each has a clear responsibility boundary.

  • streampark-common (streampark-common/): Shared foundation layer. Contains configuration management (ConfigKeys, ConfigOption), utility classes (Utils, HadoopUtils, JsonUtils, YarnUtils), file system abstraction (FsOperator, HdfsOperator, LfsOperator), and shared enums (FlinkDeployMode, ApplicationType, etc.). Must be engine-agnostic — no Flink or Spark core API dependencies. All other modules depend on this module.

  • streampark-flink (streampark-flink/): Flink development framework and runtime integration. Contains the core development API (FlinkStreaming, FlinkTable, FlinkSQL), version-specific shims layers (streampark-flink-shims_flink-1.xx), job submission clients (streampark-flink-client), Kubernetes integration (streampark-flink-kubernetes), application packer (streampark-flink-packer), and connectors. The shims proxy (FlinkShimsProxy) is the central mechanism for multi-version Flink support.

  • streampark-spark (streampark-spark/): Spark development framework. Optional module, activated via -Pspark Maven profile. Contains SparkStreaming, SparkBatch core traits, Spark SQL client, and connectors. Follows the same lifecycle pattern as Flink modules.

  • streampark-console (streampark-console/): Web management platform. Contains two submodules:

    • streampark-console-service: Spring Boot 2.7 backend, with base/ (infrastructure), core/ (business logic, controllers, services), and system/ (authentication, user/role/team management) packages.
    • streampark-console-webapp: Vue 3 + Vite frontend, with api/ (API layer), views/ (pages), components/ (reusable UI), store/ (Pinia state).

The streampark-common module has the strongest stability guarantees — changes here affect all other modules. streampark-flink/streampark-flink-core public API (the FlinkStreaming, FlinkTable traits) should also be treated as stable — breaking changes require careful migration planning.

High-Sensitivity Areas

  • FlinkShimsProxy: The multi-version classloader isolation mechanism. Uses ChildFirstClassLoader to dynamically load version-specific shims JARs. Cached per Flink version. Changes here affect all Flink jobs across all versions. Never introduce static state that could leak across classloader boundaries.

  • FlinkStreaming / FlinkTable lifecycle: The main -> init -> ready -> handle -> destroy lifecycle is the contract all user applications depend on. Changes to the execution order or initialization behavior can break existing applications in production.

  • ConfigKeys / CommonConfig: Central configuration key definitions. Adding, removing, or renaming keys affects application configuration files, the console UI, and deployment scripts. Key names must remain backward-compatible.

  • FlinkApplicationController / FlinkApplicationManageService / FlinkApplicationActionService: The core application management flow. Operations (start, stop, cancel, deploy) must be idempotent and handle all Flink states correctly. The AppChangeEvent annotation triggers state synchronization.

  • SQL parsing and validation (FlinkSql, FlinkSqlService, SqlConvertUtils): SQL validation must be version-aware (Flink 1.12-1.20 have different SQL syntax). The sql-rev.dict file handles MySQL-to-PostgreSQL dialect conversion.

  • Kubernetes integration (FlinkK8sWatchController): Uses Caffeine caches (TrackIdCache, JobStatusCache, MetricCache) for tracking K8s-deployed Flink jobs. Cache invalidation and TTL must be correct to avoid stale state.

  • Database schema changes: All schema changes must have corresponding upgrade scripts in streampark-console/.../script/upgrade/ for both MySQL and PostgreSQL. The sql-rev.dict file must be updated if new SQL dialect differences are introduced.

  • Authentication & Authorization: ShiroConfig, JWTUtil, ShiroRealm — changes here affect all user access. The @Permission annotation and PermissionAspect enforce team-level resource isolation. Never weaken RBAC checks.

Design Patterns

  • Lifecycle trait pattern: FlinkStreaming, FlinkTable, SparkStreaming, SparkBatch all follow the same trait-based lifecycle: main -> init -> ready -> handle -> start -> destroy. Users override handle() (required) and optionally ready(), config(), destroy(). Never add mandatory lifecycle methods to existing traits.

  • Shims / Proxy pattern: FlinkShimsProxy.proxy(flinkVersion, func) isolates version-specific Flink API calls behind a ChildFirstClassLoader. Each Flink version has its own shims module (streampark-flink-shims_flink-1.xx) with the same interface. New shims methods must be added to all version modules.

  • Service layer separation: Console services are split by responsibility — FlinkApplicationManageService (CRUD), FlinkApplicationActionService (start/stop/cancel), FlinkApplicationInfoService (query/info). Follow this pattern when adding new application operations.

  • Implicit enrichment: Scala implicit conversions are used to extend Flink/Spark APIs (e.g., DataStreamExt adds methods to DataStream). New implicit conversions must be scoped to avoid polluting the global namespace.

  • MyBatis-Plus entity pattern: Entities extend BaseEntity (auto-fill createTime/modifyTime). Mappers extend MyBatis-Plus BaseMapper. Pagination uses MybatisPager + PaginationInterceptor. Follow existing patterns rather than introducing new ORM approaches.

  • Enum-based configuration: Both Java enums (FlinkDeployMode, ApplicationType) and Scala enumeratum enums (ApiType, PlannerType) are used. Prefer Scala enumeratum for new Scala-side enums — it provides better type safety and serialization.

  • REST response pattern: All controller methods return RestResponse.success(data) or RestResponse.fail(...). Never return raw objects. Use @Permission annotation for access control, @AppChangeEvent for state-change auditing.

  • File system abstraction: Use FsOperator (with HdfsOperator / LfsOperator implementations) for file operations. Never use raw java.io.File or Hadoop FileSystem directly in business logic.

Coding Conventions

Java (Backend)

  • Formatting: Eclipse formatter via Spotless (tools/checkstyle/spotless_streampark_formatter.xml). Run ./mvnw spotless:apply before committing.
  • Import order: org.apache.streampark, org.apache.streampark.shaded, org.apache, javax, java, scala, \# (all others).
  • Static checks: Checkstyle (tools/checkstyle/checkstyle.xml) + Spotless. No wildcard imports. No @author tags. No JUnit 4 imports.
  • Lombok: Use @Slf4j, @Data, @Builder where appropriate. Do not use @EqualsAndHashCode on JPA/Hibernate entities.
  • Testing: JUnit 5 (org.junit.jupiter) + AssertJ. Use @Test (not @Test from JUnit 4). Use assertThat(...).isEqualTo(...) style. Test classes should be in the same package as the code under test.
  • Package structure: Controllers in controller/, service interfaces in service/, implementations in service/impl/, entities in entity/, mappers in mapper/, enums in enums/.

Scala (Framework / Common)

  • Formatting: Scalafmt 3.7.5 (tools/checkstyle/.scalafmt.conf). Max column 160. Run ./mvnw spotless:apply to format.
  • Import ordering: org.apache.streampark.* first, then other third-party, then javax.*, java.*, scala.*.
  • Static checks: Scalastyle (tools/checkstyle/scalastyle-config.xml). No wildcard imports. No println statements (use Logger trait).
  • Testing: ScalaTest 3.2.9. Use FlatSpec or FunSuite style consistent with existing tests.
  • Style: Use val over var. Prefer Option over null in public APIs. Use lazy val for expensive initialization. Use pattern matching instead of isInstanceOf/asInstanceOf.

TypeScript / Vue (Frontend)

  • Formatting: ESLint + Prettier. Run pnpm lint:eslint and pnpm lint:prettier.
  • Vue 3 Composition API: Use <script setup lang="ts"> for new components. Use Pinia for state management (not Vuex).
  • API layer: All HTTP calls go through src/api/ modules using defHttp. API URLs are defined as enum constants. Never use axios or fetch directly in components.
  • Component naming: PascalCase for component files and names. Use index.ts barrel exports in component directories.
  • Unused variables: Prefix with _ to suppress ESLint warnings (argsIgnorePattern: '^_').

General

  • Apache License header: Required on all new files. Use the copyright header from tools/checkstyle/copyright.txt. Enforced by Spotless and Apache RAT.
  • No wildcard imports: Prohibited in both Java and Scala (enforced by Spotless).
  • No personal pronouns in comments: Use descriptive, impersonal documentation.
  • Database: Support both MySQL and PostgreSQL. All new SQL must be tested against both. Use sql-rev.dict for dialect differences.
  • Logging: Use Lombok @Slf4j (Java) or Logger trait (Scala). Use log.info/log.error with parameterized messages. Never log credentials or sensitive data.

Commands

  • Full build (backend + frontend, skip tests):

    ./build.sh

    Equivalent to: ./mvnw -Pshaded,webapp,dist -DskipTests clean install

  • Fast build (backend only, skip all checks):

    ./mvnw -Pfast clean install -DskipTests

  • Build with Spark module:

    ./mvnw -Pspark,shaded clean install -DskipTests

  • Build backend only:

    ./mvnw clean install -DskipTests -pl '!streampark-console/streampark-console-webapp'

  • Run single test class (Java):

    ./mvnw test -pl streampark-console/streampark-console-service -Dtest=FlinkApplicationControllerTest

  • Run single test class (Scala):

    ./mvnw test -pl streampark-common -Dtest=org.apache.streampark.common.util.CommandUtilsTest

  • Format code (Java + Scala):

    ./mvnw spotless:apply

  • Check formatting:

    ./mvnw spotless:check

  • Run Checkstyle:

    ./mvnw checkstyle:check

  • Frontend development server:

    cd streampark-console/streampark-console-webapp && pnpm dev

  • Frontend lint:

    cd streampark-console/streampark-console-webapp && pnpm lint:eslint && pnpm lint:prettier

  • Frontend build:

    cd streampark-console/streampark-console-webapp && pnpm build

  • Run Docker Compose (local):

    docker compose -f docker/docker-compose.yaml up -d

PR & Commit Conventions

  • PR title format: [Module] Description (e.g., [Flink] Fix shims classloader isolation, [Console] Add team-level resource filtering, [Common] Support Hadoop 3.x configuration).
  • Module prefixes: [Common], [Flink], [Spark], [Console], [K8s], [Docs], [CI], [Build].
  • One concern per PR: Unrelated whitespace, import, or formatting changes go in separate PRs. Do not mix refactoring with feature work.
  • Commit messages: Describe the what and why, not implementation details. Reference related GitHub issues with #xxx.
  • Apache License header: Required on all new files (enforced by Spotless and Apache RAT). The spotless:check and apache-rat:check goals run in CI.
  • Schema changes: Must include upgrade scripts for both MySQL and PostgreSQL in streampark-console/.../script/upgrade/.
  • New shims methods: When adding a new method to the shims interface, add implementations to all version-specific shims modules (streampark-flink-shims_flink-1.12 through streampark-flink-shims_flink-1.20).

Boundaries

Never Without Explicit Discussion

  • Never modify .asf.yaml, LICENSE, NOTICE, or .gitignore without explicit discussion.
  • Never upgrade Flink, Spark, Scala, Spring Boot, or other major dependency versions without discussion — these changes have broad impact across the entire project.
  • Never remove or rename keys in ConfigKeys or CommonConfig — these are user-facing configuration contracts.
  • Never change the FlinkStreaming / FlinkTable lifecycle method signatures — this breaks all user applications.
  • Never add new required lifecycle methods to the FlinkStreaming / FlinkTable traits.
  • Never commit secrets, credentials, API keys, or cloud-specific tokens.
  • Never introduce a new Flink version shims module without adding the corresponding CI build configuration.
  • Never add a new database migration without providing both MySQL and PostgreSQL upgrade scripts.
  • Never change the SqlConvertUtils dialect conversion logic without testing against both MySQL and PostgreSQL.

Ask First

  • Ask first before adding new third-party dependencies — license compatibility with Apache 2.0 matters.
  • Ask first before promoting package-private classes/methods to public.
  • Ask first before adding new Maven modules or restructuring the module hierarchy.
  • Ask first before introducing new Scala implicits in the common module — they affect all downstream code.
  • Ask first before changing the authentication model (Shiro/JWT/Pac4j configuration).