layout: section title: “Direct Runner” permalink: /documentation/runners/direct/ section_menu: section-menu/runners.html redirect_from: /learn/runners/direct/

Using the Direct Runner

The Direct Runner executes pipelines on your machine and is designed to validate that pipelines adhere to the Apache Beam model as closely as possible. Instead of focusing on efficient pipeline execution, the Direct Runner performs additional checks to ensure that users do not rely on semantics that are not guaranteed by the model. Some of these checks include:

  • enforcing immutability of elements
  • enforcing encodability of elements
  • elements are processed in an arbitrary order at all points
  • serialization of user functions (DoFn, CombineFn, etc.)

Using the Direct Runner for testing and development helps ensure that pipelines are robust across different Beam runners. In addition, debugging failed runs can be a non-trivial task when a pipeline executes on a remote cluster. Instead, it is often faster and simpler to perform local unit testing on your pipeline code. Unit testing your pipeline locally also allows you to use your preferred local debugging tools.

Here are some resources with information about how to test your pipelines.

Direct Runner prerequisites and setup

Specify your dependency

When using Java, you must specify your dependency on the Direct Runner in your pom.xml.

<dependency>
   <groupId>org.apache.beam</groupId>
   <artifactId>beam-runners-direct-java</artifactId>
   <version>{{ site.release_latest }}</version>
   <scope>runtime</scope>
</dependency>

This section is not applicable to the Beam SDK for Python.

Pipeline options for the Direct Runner

When executing your pipeline from the command-line, set runner to direct or DirectRunner. The default values for the other pipeline options are generally sufficient.

See the reference documentation for the [DirectOptions]({{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/index.html?org/apache/beam/runners/direct/DirectOptions.html) [DirectOptions]({{ site.baseurl }}/documentation/sdks/pydoc/{{ site.release_latest }}/apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.DirectOptions) interface for defaults and additional pipeline configuration options.

Additional information and caveats

Memory considerations

Local execution is limited by the memory available in your local environment. It is highly recommended that you run your pipeline with data sets small enough to fit in local memory. You can create a small in-memory data set using a [Create]({{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/index.html?org/apache/beam/sdk/transforms/Create.html)Create transform, or you can use a [Read]({{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/index.html?org/apache/beam/sdk/io/Read.html)Read transform to work with small local or remote files.

Streaming execution

If your pipeline uses an unbounded data source or sink, you must set the streaming option to true.