| .. Licensed to the Apache Software Foundation (ASF) under one | |
| .. or more contributor license agreements. See the NOTICE file | |
| .. distributed with this work for additional information | |
| .. regarding copyright ownership. The ASF licenses this file | |
| .. to you under the Apache License, Version 2.0 (the | |
| .. "License"); you may not use this file except in compliance | |
| .. with the License. You may obtain a copy of the License at | |
| .. http://www.apache.org/licenses/LICENSE-2.0 | |
| .. Unless required by applicable law or agreed to in writing, | |
| .. software distributed under the License is distributed on an | |
| .. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | |
| .. KIND, either express or implied. See the License for the | |
| .. specific language governing permissions and limitations | |
| .. under the License. | |
| =================== | |
| Working with Schema | |
| =================== | |
| Arrow automatically infers the most appropriate data type when reading in data | |
| or converting Python objects to Arrow objects. | |
| However, you might want to manually tell Arrow which data types to | |
| use, for example, to ensure interoperability with databases and data warehouse | |
| systems. This chapter includes recipes for dealing with schemas. | |
| .. contents:: | |
| Setting the data type of an Arrow Array | |
| ======================================= | |
| If you have an existing array and want to change its data type, | |
| that can be done through the ``cast`` function: | |
| .. testcode:: | |
| import pyarrow as pa | |
| arr = pa.array([1, 2, 3, 4, 5]) | |
| print(arr.type) | |
| .. testoutput:: | |
| int64 | |
| .. testcode:: | |
| arr = arr.cast(pa.int8()) | |
| print(arr.type) | |
| .. testoutput:: | |
| int8 | |
| You can also create an array of the requested type by providing | |
| the type at array creation | |
| .. testcode:: | |
| import pyarrow as pa | |
| arr = pa.array([1, 2, 3, 4, 5], type=pa.int8()) | |
| print(arr.type) | |
| .. testoutput:: | |
| int8 | |
| Setting the schema of a Table | |
| ============================= | |
| Tables detain multiple columns, each with its own name | |
| and type. The union of types and names is what defines a schema. | |
| A schema in Arrow can be defined using :meth:`pyarrow.schema` | |
| .. testcode:: | |
| import pyarrow as pa | |
| schema = pa.schema([ | |
| ("col1", pa.int8()), | |
| ("col2", pa.string()), | |
| ("col3", pa.float64()) | |
| ]) | |
| The schema can then be provided to a table when created: | |
| .. testcode:: | |
| table = pa.table([ | |
| [1, 2, 3, 4, 5], | |
| ["a", "b", "c", "d", "e"], | |
| [1.0, 2.0, 3.0, 4.0, 5.0] | |
| ], schema=schema) | |
| print(table) | |
| .. testoutput:: | |
| pyarrow.Table | |
| col1: int8 | |
| col2: string | |
| col3: double | |
| ---- | |
| col1: [[1,2,3,4,5]] | |
| col2: [["a","b","c","d","e"]] | |
| col3: [[1,2,3,4,5]] | |
| Like for arrays, it's possible to cast tables to different schemas | |
| as far as they are compatible | |
| .. testcode:: | |
| schema_int32 = pa.schema([ | |
| ("col1", pa.int32()), | |
| ("col2", pa.string()), | |
| ("col3", pa.float64()) | |
| ]) | |
| table = table.cast(schema_int32) | |
| print(table) | |
| .. testoutput:: | |
| pyarrow.Table | |
| col1: int32 | |
| col2: string | |
| col3: double | |
| ---- | |
| col1: [[1,2,3,4,5]] | |
| col2: [["a","b","c","d","e"]] | |
| col3: [[1,2,3,4,5]] | |
| Merging multiple schemas | |
| ======================== | |
| When you have multiple separate groups of data that you want to combine | |
| it might be necessary to unify their schemas to create a superset of them | |
| that applies to all data sources. | |
| .. testcode:: | |
| import pyarrow as pa | |
| first_schema = pa.schema([ | |
| ("country", pa.string()), | |
| ("population", pa.int32()) | |
| ]) | |
| second_schema = pa.schema([ | |
| ("country_code", pa.string()), | |
| ("language", pa.string()) | |
| ]) | |
| :func:`unify_schemas` can be used to combine multiple schemas into | |
| a single one: | |
| .. testcode:: | |
| union_schema = pa.unify_schemas([first_schema, second_schema]) | |
| print(union_schema) | |
| .. testoutput:: | |
| country: string | |
| population: int32 | |
| country_code: string | |
| language: string | |
| If the combined schemas have overlapping columns, they can still be combined | |
| as far as the colliding columns retain the same type (``country_code``): | |
| .. testcode:: | |
| third_schema = pa.schema([ | |
| ("country_code", pa.string()), | |
| ("lat", pa.float32()), | |
| ("long", pa.float32()), | |
| ]) | |
| union_schema = pa.unify_schemas([first_schema, second_schema, third_schema]) | |
| print(union_schema) | |
| .. testoutput:: | |
| country: string | |
| population: int32 | |
| country_code: string | |
| language: string | |
| lat: float | |
| long: float | |
| If a merged field has instead diverging types in the combined schemas | |
| then trying to merge the schemas will fail. For example if ``country_code`` | |
| was a numeric instead of a string we would be unable to unify the schemas | |
| because in ``second_schema`` it was already declared as a ``pa.string()`` | |
| .. testcode:: | |
| third_schema = pa.schema([ | |
| ("country_code", pa.int32()), | |
| ("lat", pa.float32()), | |
| ("long", pa.float32()), | |
| ]) | |
| try: | |
| union_schema = pa.unify_schemas([first_schema, second_schema, third_schema]) | |
| except (pa.ArrowInvalid, pa.ArrowTypeError) as e: | |
| print(e) | |
| .. testoutput:: | |
| Unable to merge: Field country_code has incompatible types: string vs int32 |