title: Plugin Discovery and Class Loading

Plugin Discovery and Class Loading

Why This Page Exists

SeaTunnel documentation already explains how to configure connectors and how isolated dependencies are laid out in the binary package. What is still easy to miss is the runtime path between a job config and a loaded plugin instance.

This page explains that path at a system level:

  • how a configured plugin name becomes a factory implementation
  • how plugin metadata is discovered
  • how connector jars and dependency jars are located
  • why class loader isolation matters, especially in Zeta

The Problem SeaTunnel Solves

SeaTunnel needs to support a large plugin ecosystem across sources, sinks, transforms, and formats. At runtime it has to answer a few questions safely:

  • Which plugin implementation matches the user config?
  • Where are that plugin's jars and isolated dependencies?
  • How do we load them without polluting the whole process classpath?
  • How do we expose plugin metadata to CLI, REST API, and UI features?

Without a discovery and class-loading layer, adding connectors would quickly lead to dependency conflicts and fragile startup behavior.

End-to-End Runtime Path

At a high level, a plugin goes through this path:

Job Config
  -> parse plugin name
  -> discover factory
  -> validate options
  -> resolve connector jar and isolated dependencies
  -> create plugin classloader
  -> instantiate source / sink / transform
  -> execute inside engine runtime

Each step matters for a different reason:

  • discovery finds the correct plugin implementation
  • validation ensures required options are present
  • jar resolution determines what code is available
  • classloader isolation reduces dependency conflicts

Discovery Model

Plugin Identity

At runtime SeaTunnel distinguishes plugins by a logical identity, typically combining:

  • engine type
  • plugin type, such as source or sink
  • plugin name

This identity is used in multiple places:

  • config parsing
  • plugin lookup
  • option metadata exposure
  • log messages and diagnostics

Factory Discovery

Most user-visible plugins are created through factory interfaces. In practice, SeaTunnel relies on Java SPI and factory discovery helpers to locate implementations declared under META-INF/services.

Typical examples include:

  • TableSourceFactory
  • TableSinkFactory
  • transform factories
  • catalog and format factories where applicable

This is why connector development docs require factory registration and service metadata.

Related docs:

Jar Resolution and Packaging

SeaTunnel separates the plugin jar from connector-specific third-party dependencies.

Typical runtime layout:

SEATUNNEL_HOME/
  connectors/
    connector-jdbc-<version>.jar
  plugins/
    connector-jdbc/
      dependency-a.jar
      dependency-b.jar

The mapping between plugin name and dependency directory is managed through plugin-mapping.properties.

This gives SeaTunnel two useful properties:

  • the connector implementation jar can be shipped centrally
  • connector-specific dependency jars can stay isolated from each other

Related docs:

Class Loading in Practice

Why Isolation Matters

Different connectors may depend on different versions of the same third-party library. If all jars were loaded into one flat classpath, a single dependency conflict could break unrelated connectors in the same job.

Isolation helps in three places:

  • connector startup
  • task deployment
  • long-running multi-connector jobs

Zeta vs Flink / Spark

The most important practical difference is that SeaTunnel Engine (Zeta) gives stronger connector dependency isolation for job execution. The existing isolated dependency document already calls out that Spark and Flink still have tighter classpath sharing, so mixed versions remain more risky there.

That distinction is important when:

  • adding a connector with a heavy dependency tree
  • troubleshooting ClassNotFoundException or NoSuchMethodError
  • packaging jobs for a distributed cluster

Metadata Discovery and Option Exposure

Plugin discovery is not only used to instantiate runtime classes. It also powers metadata-oriented capabilities, such as exposing connector options to REST API and Web UI clients.

This is why the plugin system needs access to factory-level metadata such as:

  • supported plugin identifier
  • OptionRule
  • required and optional fields
  • grouping and validation semantics

Related docs:

Failure Modes and What They Usually Mean

When plugin loading fails, the symptom often tells you which layer is broken.

Discovery Failures

Typical symptoms:

  • plugin not found
  • connector name recognized in docs but not in runtime

Usually means:

  • missing connector jar
  • missing SPI registration
  • wrong plugin identifier

Validation Failures

Typical symptoms:

  • job submission fails before execution
  • REST or UI metadata exists, but config is rejected

Usually means:

  • required options missing
  • mutually exclusive options violated
  • option names differ from factory definitions

Classpath / ClassLoader Failures

Typical symptoms:

  • ClassNotFoundException
  • NoClassDefFoundError
  • NoSuchMethodError
  • version conflicts only on some engines

Usually means:

  • dependency jar missing from plugin directory
  • connector packaged with incompatible dependency versions
  • cluster nodes do not have consistent plugin layouts

Operator Checklist

When a plugin cannot be loaded correctly, check these items in order:

  1. Is the connector jar present in the binary package?
  2. Is the dependency directory under ${SEATUNNEL_HOME}/plugins/ correct?
  3. Does plugin-mapping.properties map the plugin to the expected directory?
  4. Are all cluster nodes using the same plugin layout?
  5. Is the plugin name in the job config exactly the same as the factory identifier?
  6. Does the failure happen only in Flink or Spark, but not in Zeta?

Developer Checklist

When adding a new plugin, verify these items before debugging engine internals:

  • the factory is registered correctly
  • the factory identifier matches the doc examples
  • OptionRule covers required, optional, and exclusive fields
  • the connector is added to packaging and plugin mapping
  • isolated dependencies are placed under the expected plugin directory

Code References

Useful code entry points:

  • seatunnel-api/src/main/java/org/apache/seatunnel/api/table/factory/FactoryUtil.java
  • seatunnel-engine/seatunnel-engine-common/src/main/java/org/apache/seatunnel/engine/common/utils/FactoryUtil.java
  • seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/classloader/ClassLoaderService.java
  • seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/classloader/DefaultClassLoaderService.java
  • seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/service/OptionRulesService.java

Recommended Reading Path

  1. this page for the runtime map
  2. Connector Isolated Dependency Loading Mechanism
  3. Configuration And Option System
  4. Source Connector Development or Sink Connector Development