| .. Licensed to the Apache Software Foundation (ASF) under one | |
| .. or more contributor license agreements. See the NOTICE file | |
| .. distributed with this work for additional information | |
| .. regarding copyright ownership. The ASF licenses this file | |
| .. to you under the Apache License, Version 2.0 (the | |
| .. "License"); you may not use this file except in compliance | |
| .. with the License. You may obtain a copy of the License at | |
| .. http://www.apache.org/licenses/LICENSE-2.0 | |
| .. Unless required by applicable law or agreed to in writing, | |
| .. software distributed under the License is distributed on an | |
| .. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | |
| .. KIND, either express or implied. See the License for the | |
| .. specific language governing permissions and limitations | |
| .. under the License. | |
| ====================== | |
| Creating Arrow Objects | |
| ====================== | |
| Recipes related to the creation of Arrays, Tables, | |
| Tensors and all other Arrow entities. | |
| .. contents:: | |
| Creating Arrays | |
| =============== | |
| Arrow keeps data in continuous arrays optimised for memory footprint | |
| and SIMD analyses. In Python it's possible to build :class:`pyarrow.Array` | |
| starting from Python ``lists`` (or sequence types in general), | |
| ``numpy`` arrays and ``pandas`` Series. | |
| .. testcode:: | |
| import pyarrow as pa | |
| array = pa.array([1, 2, 3, 4, 5]) | |
| .. testcode:: | |
| print(array) | |
| .. testoutput:: | |
| [ | |
| 1, | |
| 2, | |
| 3, | |
| 4, | |
| 5 | |
| ] | |
| Arrays can also provide a ``mask`` to specify which values should | |
| be considered nulls | |
| .. testcode:: | |
| import numpy as np | |
| array = pa.array([1, 2, 3, 4, 5], | |
| mask=np.array([True, False, True, False, True])) | |
| print(array) | |
| .. testoutput:: | |
| [ | |
| null, | |
| 2, | |
| null, | |
| 4, | |
| null | |
| ] | |
| When building arrays from ``numpy`` or ``pandas``, Arrow will leverage | |
| optimized code paths that rely on the internal in-memory representation | |
| of the data by ``numpy`` and ``pandas`` | |
| .. testcode:: | |
| import numpy as np | |
| import pandas as pd | |
| array_from_numpy = pa.array(np.arange(5)) | |
| array_from_pandas = pa.array(pd.Series([1, 2, 3, 4, 5])) | |
| Creating Tables | |
| =============== | |
| Arrow supports tabular data in :class:`pyarrow.Table`: each column | |
| is represented by a :class:`pyarrow.ChunkedArray` and tables can be created | |
| by pairing multiple arrays with names for their columns | |
| .. testcode:: | |
| import pyarrow as pa | |
| table = pa.table([ | |
| pa.array([1, 2, 3, 4, 5]), | |
| pa.array(["a", "b", "c", "d", "e"]), | |
| pa.array([1.0, 2.0, 3.0, 4.0, 5.0]) | |
| ], names=["col1", "col2", "col3"]) | |
| print(table) | |
| .. testoutput:: | |
| pyarrow.Table | |
| col1: int64 | |
| col2: string | |
| col3: double | |
| ---- | |
| col1: [[1,2,3,4,5]] | |
| col2: [["a","b","c","d","e"]] | |
| col3: [[1,2,3,4,5]] | |
| Create Table from Plain Types | |
| ============================= | |
| Arrow allows fast zero copy creation of arrow arrays | |
| from numpy and pandas arrays and series, but it's also | |
| possible to create Arrow Arrays and Tables from | |
| plain Python structures. | |
| The :func:`pyarrow.table` function allows creation of Tables | |
| from a variety of inputs, including plain python objects | |
| .. testcode:: | |
| import pyarrow as pa | |
| table = pa.table({ | |
| "col1": [1, 2, 3, 4, 5], | |
| "col2": ["a", "b", "c", "d", "e"] | |
| }) | |
| print(table) | |
| .. testoutput:: | |
| pyarrow.Table | |
| col1: int64 | |
| col2: string | |
| ---- | |
| col1: [[1,2,3,4,5]] | |
| col2: [["a","b","c","d","e"]] | |
| .. note:: | |
| All values provided in the dictionary will be passed to | |
| :func:`pyarrow.array` for conversion to Arrow arrays, | |
| and will benefit from zero copy behaviour when possible. | |
| The :meth:`pyarrow.Table.from_pylist` method allows the creation | |
| of Tables from python lists of row dicts. Types are inferred if a | |
| schema is not explicitly passed. | |
| .. testcode:: | |
| import pyarrow as pa | |
| table = pa.Table.from_pylist([ | |
| {"col1": 1, "col2": "a"}, | |
| {"col1": 2, "col2": "b"}, | |
| {"col1": 3, "col2": "c"}, | |
| {"col1": 4, "col2": "d"}, | |
| {"col1": 5, "col2": "e"} | |
| ]) | |
| print(table) | |
| .. testoutput:: | |
| pyarrow.Table | |
| col1: int64 | |
| col2: string | |
| ---- | |
| col1: [[1,2,3,4,5]] | |
| col2: [["a","b","c","d","e"]] | |
| Creating Record Batches | |
| ======================= | |
| Most I/O operations in Arrow happen when shipping batches of data | |
| to their destination. :class:`pyarrow.RecordBatch` is the way | |
| Arrow represents batches of data. A RecordBatch can be seen as a slice | |
| of a table. | |
| .. testcode:: | |
| import pyarrow as pa | |
| batch = pa.RecordBatch.from_arrays([ | |
| pa.array([1, 3, 5, 7, 9]), | |
| pa.array([2, 4, 6, 8, 10]) | |
| ], names=["odd", "even"]) | |
| Multiple batches can be combined into a table using | |
| :meth:`pyarrow.Table.from_batches` | |
| .. testcode:: | |
| second_batch = pa.RecordBatch.from_arrays([ | |
| pa.array([11, 13, 15, 17, 19]), | |
| pa.array([12, 14, 16, 18, 20]) | |
| ], names=["odd", "even"]) | |
| table = pa.Table.from_batches([batch, second_batch]) | |
| .. testcode:: | |
| print(table) | |
| .. testoutput:: | |
| pyarrow.Table | |
| odd: int64 | |
| even: int64 | |
| ---- | |
| odd: [[1,3,5,7,9],[11,13,15,17,19]] | |
| even: [[2,4,6,8,10],[12,14,16,18,20]] | |
| Equally, :class:`pyarrow.Table` can be converted to a list of | |
| :class:`pyarrow.RecordBatch` using the :meth:`pyarrow.Table.to_batches` | |
| method | |
| .. testcode:: | |
| record_batches = table.to_batches(max_chunksize=5) | |
| print(len(record_batches)) | |
| .. testoutput:: | |
| 2 | |
| Store Categorical Data | |
| ====================== | |
| Arrow provides the :class:`pyarrow.DictionaryArray` type | |
| to represent categorical data without the cost of | |
| storing and repeating the categories over and over. This can reduce memory use | |
| when columns might have large values (such as text). | |
| If you have an array containing repeated categorical data, | |
| it is possible to convert it to a :class:`pyarrow.DictionaryArray` | |
| using :meth:`pyarrow.Array.dictionary_encode` | |
| .. testcode:: | |
| arr = pa.array(["red", "green", "blue", "blue", "green", "red"]) | |
| categorical = arr.dictionary_encode() | |
| print(categorical) | |
| .. testoutput:: | |
| ... | |
| -- dictionary: | |
| [ | |
| "red", | |
| "green", | |
| "blue" | |
| ] | |
| -- indices: | |
| [ | |
| 0, | |
| 1, | |
| 2, | |
| 2, | |
| 1, | |
| 0 | |
| ] | |
| If you already know the categories and indices then you can skip the encode | |
| step and directly create the ``DictionaryArray`` using | |
| :meth:`pyarrow.DictionaryArray.from_arrays` | |
| .. testcode:: | |
| categorical = pa.DictionaryArray.from_arrays( | |
| indices=[0, 1, 2, 2, 1, 0], | |
| dictionary=["red", "green", "blue"] | |
| ) | |
| print(categorical) | |
| .. testoutput:: | |
| ... | |
| -- dictionary: | |
| [ | |
| "red", | |
| "green", | |
| "blue" | |
| ] | |
| -- indices: | |
| [ | |
| 0, | |
| 1, | |
| 2, | |
| 2, | |
| 1, | |
| 0 | |
| ] |