--job_endpoint
to be set when using --runner=PortableRunner
(BEAM-9860). Users seeking the old default behavior should set --runner=FlinkRunner
instead.apache_beam.io.gcp.datastore.v1
has been removed as the client it uses is out of date and does not support Python 3 (BEAM-9529). Please migrate your code to use apache_beam.io.gcp.datastore.v1new. See the updated datastore_wordcount for example usage.Python SDK will now use Python 3 type annotations as pipeline type hints. (#10717)
If you suspect that this feature is causing your pipeline to fail, calling apache_beam.typehints.disable_type_annotations()
before pipeline creation will disable is completely, and decorating specific functions (such as process()
) with @apache_beam.typehints.no_annotations
will disable it for that function.
More details will be in Ensuring Python Type Safety and an upcoming blog post.
Java SDK: Introducing the concept of options in Beam Schema’s. These options add extra context to fields and schemas. This replaces the current Beam metadata that is present in a FieldType only, options are available in fields and row schemas. Schema options are fully typed and can contain complex rows. Remark: Schema aware is still experimental. (BEAM-9035)
Java SDK: The protobuf extension is fully schema aware and also includes protobuf option conversion to beam schema options. Remark: Schema aware is still experimental. (BEAM-9044)
Added ability to write to BigQuery via Avro file loads (Python) (BEAM-8841)
By default, file loads will be done using JSON, but it is possible to specify the temp_file_format parameter to perform file exports with AVRO. AVRO-based file loads work by exporting Python types into Avro types, so to switch to Avro-based loads, you will need to change your data types from Json-compatible types (string-type dates and timestamp, long numeric values as strings) into Python native types that are written to Avro (Python's date, datetime types, decimal, etc). For more information see https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro#avro_conversions.
Added integration of Java SDK with Google Cloud AI VideoIntelligence service (BEAM-9147)
--region
option to be set, unless a default value is set in the environment (BEAM-9199). See here for more details.2.23.0
. (BEAM-9704)SpannerConfig.connectToSpanner
has been moved to SpannerAccessor.create
. (BEAM-9310).force_generated_pcollection_output_ids
experiment.