py/_sources/create.rst.txt - arrow-cookbook - Git at Google

 .. Licensed to the Apache Software Foundation (ASF) under one
 .. or more contributor license agreements.  See the NOTICE file
 .. distributed with this work for additional information
 .. regarding copyright ownership.  The ASF licenses this file
 .. to you under the Apache License, Version 2.0 (the
 .. "License"); you may not use this file except in compliance
 .. with the License.  You may obtain a copy of the License at

 ..   http://www.apache.org/licenses/LICENSE-2.0

 .. Unless required by applicable law or agreed to in writing,
 .. software distributed under the License is distributed on an
 .. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 .. KIND, either express or implied.  See the License for the
 .. specific language governing permissions and limitations
 .. under the License.

 ======================
 Creating Arrow Objects
 ======================

 Recipes related to the creation of Arrays, Tables,
 Tensors and all other Arrow entities.

 .. contents::

 Creating Arrays
 ===============

 Arrow keeps data in continuous arrays optimised for memory footprint
 and SIMD analyses. In Python it's possible to build :class:`pyarrow.Array`
 starting from Python ``lists`` (or sequence types in general),
 ``numpy`` arrays and ``pandas`` Series.

 .. testcode::

     import pyarrow as pa

     array = pa.array([1, 2, 3, 4, 5])

 .. testcode::

     print(array)

 .. testoutput::

     [
       1,
       2,
       3,
       4,
       5
     ]

 Arrays can also provide a ``mask`` to specify which values should
 be considered nulls

 .. testcode::

     import numpy as np

     array = pa.array([1, 2, 3, 4, 5],
                      mask=np.array([True, False, True, False, True]))

     print(array)

 .. testoutput::

     [
       null,
       2,
       null,
       4,
       null
     ]

 When building arrays from ``numpy`` or ``pandas``, Arrow will leverage
 optimized code paths that rely on the internal in-memory representation
 of the data by ``numpy`` and ``pandas``

 .. testcode::

     import numpy as np
     import pandas as pd

     array_from_numpy = pa.array(np.arange(5))
     array_from_pandas = pa.array(pd.Series([1, 2, 3, 4, 5]))

 Creating Tables
 ===============

 Arrow supports tabular data in :class:`pyarrow.Table`: each column
 is represented by a :class:`pyarrow.ChunkedArray` and tables can be created
 by pairing multiple arrays with names for their columns

 .. testcode::

     import pyarrow as pa

     table = pa.table([
         pa.array([1, 2, 3, 4, 5]),
         pa.array(["a", "b", "c", "d", "e"]),
         pa.array([1.0, 2.0, 3.0, 4.0, 5.0])
     ], names=["col1", "col2", "col3"])

     print(table)

 .. testoutput::

     pyarrow.Table
     col1: int64
     col2: string
     col3: double
     ----
     col1: [[1,2,3,4,5]]
     col2: [["a","b","c","d","e"]]
     col3: [[1,2,3,4,5]]

 Create Table from Plain Types
 =============================

 Arrow allows fast zero copy creation of arrow arrays
 from numpy and pandas arrays and series, but it's also
 possible to create Arrow Arrays and Tables from
 plain Python structures.

 The :func:`pyarrow.table` function allows creation of Tables
 from a variety of inputs, including plain python objects

 .. testcode::

     import pyarrow as pa

     table = pa.table({
         "col1": [1, 2, 3, 4, 5],
         "col2": ["a", "b", "c", "d", "e"]
     })

     print(table)

 .. testoutput::

     pyarrow.Table
     col1: int64
     col2: string
     ----
     col1: [[1,2,3,4,5]]
     col2: [["a","b","c","d","e"]]

 .. note::

     All values provided in the dictionary will be passed to
     :func:`pyarrow.array` for conversion to Arrow arrays,
     and will benefit from zero copy behaviour when possible.

 The :meth:`pyarrow.Table.from_pylist` method allows the creation
 of Tables from python lists of row dicts. Types are inferred if a
 schema is not explicitly passed.

 .. testcode::

     import pyarrow as pa

     table = pa.Table.from_pylist([
         {"col1": 1, "col2": "a"},
         {"col1": 2, "col2": "b"},
         {"col1": 3, "col2": "c"},
         {"col1": 4, "col2": "d"},
         {"col1": 5, "col2": "e"}
     ])

     print(table)

 .. testoutput::

     pyarrow.Table
     col1: int64
     col2: string
     ----
     col1: [[1,2,3,4,5]]
     col2: [["a","b","c","d","e"]]

 Creating Record Batches
 =======================

 Most I/O operations in Arrow happen when shipping batches of data
 to their destination.  :class:`pyarrow.RecordBatch` is the way
 Arrow represents batches of data.  A RecordBatch can be seen as a slice
 of a table.

 .. testcode::

     import pyarrow as pa

     batch = pa.RecordBatch.from_arrays([
         pa.array([1, 3, 5, 7, 9]),
         pa.array([2, 4, 6, 8, 10])
     ], names=["odd", "even"])

 Multiple batches can be combined into a table using
 :meth:`pyarrow.Table.from_batches`

 .. testcode::

     second_batch = pa.RecordBatch.from_arrays([
         pa.array([11, 13, 15, 17, 19]),
         pa.array([12, 14, 16, 18, 20])
     ], names=["odd", "even"])

     table = pa.Table.from_batches([batch, second_batch])

 .. testcode::

     print(table)

 .. testoutput::

     pyarrow.Table
     odd: int64
     even: int64
     ----
     odd: [[1,3,5,7,9],[11,13,15,17,19]]
     even: [[2,4,6,8,10],[12,14,16,18,20]]

 Equally, :class:`pyarrow.Table` can be converted to a list of
 :class:`pyarrow.RecordBatch` using the :meth:`pyarrow.Table.to_batches`
 method

 .. testcode::

     record_batches = table.to_batches(max_chunksize=5)
     print(len(record_batches))

 .. testoutput::

     2

 Store Categorical Data
 ======================

 Arrow provides the :class:`pyarrow.DictionaryArray` type
 to represent categorical data without the cost of
 storing and repeating the categories over and over.  This can reduce memory use
 when columns might have large values (such as text).

 If you have an array containing repeated categorical data,
 it is possible to convert it to a :class:`pyarrow.DictionaryArray`
 using :meth:`pyarrow.Array.dictionary_encode`

 .. testcode::

     arr = pa.array(["red", "green", "blue", "blue", "green", "red"])

     categorical = arr.dictionary_encode()
     print(categorical)

 .. testoutput::

     ...
     -- dictionary:
       [
         "red",
         "green",
         "blue"
       ]
     -- indices:
       [
         0,
         1,
         2,
         2,
         1,
         0
       ]

 If you already know the categories and indices then you can skip the encode
 step and directly create the ``DictionaryArray`` using
 :meth:`pyarrow.DictionaryArray.from_arrays`

 .. testcode::

     categorical = pa.DictionaryArray.from_arrays(
         indices=[0, 1, 2, 2, 1, 0],
         dictionary=["red", "green", "blue"]
     )
     print(categorical)

 .. testoutput::

     ...
     -- dictionary:
       [
         "red",
         "green",
         "blue"
       ]
     -- indices:
       [
         0,
         1,
         2,
         2,
         1,
         0
       ]
	.. Licensed to the Apache Software Foundation (ASF) under one
	.. or more contributor license agreements. See the NOTICE file
	.. distributed with this work for additional information
	.. regarding copyright ownership. The ASF licenses this file
	.. to you under the Apache License, Version 2.0 (the
	.. "License"); you may not use this file except in compliance
	.. with the License. You may obtain a copy of the License at

	.. http://www.apache.org/licenses/LICENSE-2.0

	.. Unless required by applicable law or agreed to in writing,
	.. software distributed under the License is distributed on an
	.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
	.. KIND, either express or implied. See the License for the
	.. specific language governing permissions and limitations
	.. under the License.

	======================
	Creating Arrow Objects
	======================

	Recipes related to the creation of Arrays, Tables,
	Tensors and all other Arrow entities.

	.. contents::

	Creating Arrays
	===============

	Arrow keeps data in continuous arrays optimised for memory footprint
	and SIMD analyses. In Python it's possible to build :class:`pyarrow.Array`
	starting from Python ``lists`` (or sequence types in general),
	``numpy`` arrays and ``pandas`` Series.

	.. testcode::

	import pyarrow as pa

	array = pa.array([1, 2, 3, 4, 5])

	.. testcode::

	print(array)

	.. testoutput::

	[
	1,
	2,
	3,
	4,
	5
	]

	Arrays can also provide a ``mask`` to specify which values should
	be considered nulls

	.. testcode::

	import numpy as np

	array = pa.array([1, 2, 3, 4, 5],
	mask=np.array([True, False, True, False, True]))

	print(array)

	.. testoutput::

	[
	null,
	2,
	null,
	4,
	null
	]

	When building arrays from ``numpy`` or ``pandas``, Arrow will leverage
	optimized code paths that rely on the internal in-memory representation
	of the data by ``numpy`` and ``pandas``

	.. testcode::

	import numpy as np
	import pandas as pd

	array_from_numpy = pa.array(np.arange(5))
	array_from_pandas = pa.array(pd.Series([1, 2, 3, 4, 5]))

	Creating Tables
	===============

	Arrow supports tabular data in :class:`pyarrow.Table`: each column
	is represented by a :class:`pyarrow.ChunkedArray` and tables can be created
	by pairing multiple arrays with names for their columns

	.. testcode::

	import pyarrow as pa

	table = pa.table([
	pa.array([1, 2, 3, 4, 5]),
	pa.array(["a", "b", "c", "d", "e"]),
	pa.array([1.0, 2.0, 3.0, 4.0, 5.0])
	], names=["col1", "col2", "col3"])

	print(table)

	.. testoutput::

	pyarrow.Table
	col1: int64
	col2: string
	col3: double
	----
	col1: [[1,2,3,4,5]]
	col2: [["a","b","c","d","e"]]
	col3: [[1,2,3,4,5]]

	Create Table from Plain Types
	=============================

	Arrow allows fast zero copy creation of arrow arrays
	from numpy and pandas arrays and series, but it's also
	possible to create Arrow Arrays and Tables from
	plain Python structures.

	The :func:`pyarrow.table` function allows creation of Tables
	from a variety of inputs, including plain python objects

	.. testcode::

	import pyarrow as pa

	table = pa.table({
	"col1": [1, 2, 3, 4, 5],
	"col2": ["a", "b", "c", "d", "e"]
	})

	print(table)

	.. testoutput::

	pyarrow.Table
	col1: int64
	col2: string
	----
	col1: [[1,2,3,4,5]]
	col2: [["a","b","c","d","e"]]

	.. note::

	All values provided in the dictionary will be passed to
	:func:`pyarrow.array` for conversion to Arrow arrays,
	and will benefit from zero copy behaviour when possible.

	The :meth:`pyarrow.Table.from_pylist` method allows the creation
	of Tables from python lists of row dicts. Types are inferred if a
	schema is not explicitly passed.

	.. testcode::

	import pyarrow as pa

	table = pa.Table.from_pylist([
	{"col1": 1, "col2": "a"},
	{"col1": 2, "col2": "b"},
	{"col1": 3, "col2": "c"},
	{"col1": 4, "col2": "d"},
	{"col1": 5, "col2": "e"}
	])

	print(table)

	.. testoutput::

	pyarrow.Table
	col1: int64
	col2: string
	----
	col1: [[1,2,3,4,5]]
	col2: [["a","b","c","d","e"]]

	Creating Record Batches
	=======================

	Most I/O operations in Arrow happen when shipping batches of data
	to their destination. :class:`pyarrow.RecordBatch` is the way
	Arrow represents batches of data. A RecordBatch can be seen as a slice
	of a table.

	.. testcode::

	import pyarrow as pa

	batch = pa.RecordBatch.from_arrays([
	pa.array([1, 3, 5, 7, 9]),
	pa.array([2, 4, 6, 8, 10])
	], names=["odd", "even"])

	Multiple batches can be combined into a table using
	:meth:`pyarrow.Table.from_batches`

	.. testcode::

	second_batch = pa.RecordBatch.from_arrays([
	pa.array([11, 13, 15, 17, 19]),
	pa.array([12, 14, 16, 18, 20])
	], names=["odd", "even"])

	table = pa.Table.from_batches([batch, second_batch])

	.. testcode::

	print(table)

	.. testoutput::

	pyarrow.Table
	odd: int64
	even: int64
	----
	odd: [[1,3,5,7,9],[11,13,15,17,19]]
	even: [[2,4,6,8,10],[12,14,16,18,20]]

	Equally, :class:`pyarrow.Table` can be converted to a list of
	:class:`pyarrow.RecordBatch` using the :meth:`pyarrow.Table.to_batches`
	method

	.. testcode::

	record_batches = table.to_batches(max_chunksize=5)
	print(len(record_batches))

	.. testoutput::

	2

	Store Categorical Data
	======================

	Arrow provides the :class:`pyarrow.DictionaryArray` type
	to represent categorical data without the cost of
	storing and repeating the categories over and over. This can reduce memory use
	when columns might have large values (such as text).

	If you have an array containing repeated categorical data,
	it is possible to convert it to a :class:`pyarrow.DictionaryArray`
	using :meth:`pyarrow.Array.dictionary_encode`

	.. testcode::

	arr = pa.array(["red", "green", "blue", "blue", "green", "red"])

	categorical = arr.dictionary_encode()
	print(categorical)

	.. testoutput::

	...
	-- dictionary:
	[
	"red",
	"green",
	"blue"
	]
	-- indices:
	[
	0,
	1,
	2,
	2,
	1,
	0
	]

	If you already know the categories and indices then you can skip the encode
	step and directly create the ``DictionaryArray`` using
	:meth:`pyarrow.DictionaryArray.from_arrays`

	.. testcode::

	categorical = pa.DictionaryArray.from_arrays(
	indices=[0, 1, 2, 2, 1, 0],
	dictionary=["red", "green", "blue"]
	)
	print(categorical)

	.. testoutput::

	...
	-- dictionary:
	[
	"red",
	"green",
	"blue"
	]
	-- indices:
	[
	0,
	1,
	2,
	2,
	1,
	0
	]