learning/prompts/documentation-lookup-nolinks/35_coders_data_encoding.md

Prompt:
What is a coder in Apache Beam?

Response:
In Apache Beam, a `Coder` is an object responsible for specifying how elements in a given `PCollection` are encoded and decoded into byte strings. The `Coder` objects provide type consistency and safety during distributed processing between workers, ensuring the correct interpretation of data based on its original types.

Every `PCollection` in an Apache Beam pipeline requires a corresponding `Coder`. While the Apache Beam SDKs can automatically infer a `Coder` for a `PCollection` based on its element type or the producing transform, there are cases where explicit specification or custom `Coder` development is necessary. It is important to note that multiple `Coder` objects can exist for a single data type.

The Apache Beam SDKs use various mechanisms to automatically infer the `Coder` for a `PCollection`. Each pipeline object contains a `CoderRegistry` object representing a mapping of language types to the default coder for those types.

In the Apache Beam SDKs for Python and Java, the `Coder` type provides the necessary methods for encoding and decoding data. The SDKs offer various `Coder` subclasses working with standard Python and Java types, available in the `apache_beam.coders` package for Python and the `Coder` package for Java.

By default, the Beam SDKs use the typehints (Python) or the type parameters (Java) from the transform's function object (such as `DoFn`) to infer the `Coder` for elements in a `PCollection`. For example, in Apache Beam Python SDK, a `ParDo` annotated with the typehints `@beam.typehints.with_input_types(int)` and `@beam.typehints.with_output_types(str)` indicates that it accepts `int` inputs and produces `str` outputs. The Python SDK automatically infers the default `Coder` for the output `PCollection`, in this case, `BytesCoder` (based on the default `CoderRegistry`).

You can use the `CoderRegistry` to look up the default coder for a given type or register a new default coder. The following tables show the default mappings of coders to standard types for any pipeline created using the Beam SDK for Java and Python:

| **Java Type** | **Default Coder** |
|---------------|-------------------|
| Double        | DoubleCoder       |
| Instant       | InstantCoder      |
| Integer       | VarIntCoder       |
| Iterable      | IterableCoder     |
| KV            | KvCoder           |
| List          | ListCoder         |
| Map           | MapCoder          |
| Long          | VarLongCoder      |
| String        | StringUtf8Coder   |

| **Python Type** | **Default Coder** |
|-----------------|-------------------|
| int             | VarIntCoder       |
| float           | FloatCoder        |
| str             | BytesCoder        |
| bytes           | StrUtf8Coder      |
| Tuple           | TupleCoder        |