.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements.  See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership.  The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License.  You may obtain a copy of the License at

..   http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing,
.. software distributed under the License is distributed on an
.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
.. KIND, either express or implied.  See the License for the
.. specific language governing permissions and limitations
.. under the License.

.. currentmodule:: pyarrow.json
.. _json:

Reading JSON files
==================

Arrow supports reading columnar data from line-delimited JSON files.
In this context, a JSON file consists of multiple JSON objects, one per line,
representing individual data rows.  For example, this file represents
two rows of data with four columns "a", "b", "c", "d":

.. code-block:: json

   {"a": 1, "b": 2.0, "c": "foo", "d": false}
   {"a": 4, "b": -5.5, "c": null, "d": true}

The features currently offered are the following:

* multi-threaded or single-threaded reading
* automatic decompression of input files (based on the filename extension,
  such as ``my_data.json.gz``)
* sophisticated type inference (see below)

.. note::
   Currently only the line-delimited JSON format is supported.


Usage
-----

JSON reading functionality is available through the :mod:`pyarrow.json` module.
In many cases, you will simply call the :func:`read_json` function
with the file path you want to read from:

.. code-block:: python

   >>> from pyarrow import json
   >>> fn = 'my_data.json'  # doctest: +SKIP
   >>> table = json.read_json(fn)  # doctest: +SKIP
   >>> table  # doctest: +SKIP
   pyarrow.Table
   a: int64
   b: double
   c: string
   d: bool
   >>> table.to_pandas()  # doctest: +SKIP
      a    b     c      d
   0  1  2.0   foo  False
   1  4 -5.5  None   True


Automatic Type Inference
------------------------

Arrow :ref:`data types <data.types>` are inferred from the JSON types and
values of each column:

* JSON null values convert to the ``null`` type, but can fall back to any
  other type.
* JSON booleans convert to ``bool_``.
* JSON numbers convert to ``int64``, falling back to ``float64`` if a
  non-integer is encountered.
* JSON strings of the kind "YYYY-MM-DD" and "YYYY-MM-DD hh:mm:ss" convert
  to ``timestamp[s]``, falling back to ``utf8`` if a conversion error occurs.
* JSON arrays convert to a ``list`` type, and inference proceeds recursively
  on the JSON arrays' values.
* Nested JSON objects convert to a ``struct`` type, and inference proceeds
  recursively on the JSON objects' values.

Thus, reading this JSON file:

.. code-block:: json

   {"a": [1, 2], "b": {"c": true, "d": "1991-02-03"}}
   {"a": [3, 4, 5], "b": {"c": false, "d": "2019-04-01"}}

returns the following data:

.. code-block:: python

   >>> table = json.read_json("my_data.json")  # doctest: +SKIP
   >>> table  # doctest: +SKIP
   pyarrow.Table
   a: list<item: int64>
     child 0, item: int64
   b: struct<c: bool, d: timestamp[s]>
     child 0, c: bool
     child 1, d: timestamp[s]
   >>> table.to_pandas()  # doctest: +SKIP
              a                                       b
   0     [1, 2]   {'c': True, 'd': 1991-02-03 00:00:00}
   1  [3, 4, 5]  {'c': False, 'd': 2019-04-01 00:00:00}


Customized parsing
------------------

To alter the default parsing settings in case of reading JSON files with an
unusual structure, you should create a :class:`ParseOptions` instance
and pass it to :func:`read_json`.  For example, you can pass an explicit
:ref:`schema <data.schema>` in order to bypass automatic type inference.

Similarly, you can choose performance settings by passing a
:class:`ReadOptions` instance to :func:`read_json`.


Incremental reading
-------------------

For memory-constrained environments, it is also possible to read a JSON file
one batch at a time, using :func:`open_json`.

In this case, type inference is done on the first block and types are frozen afterwards.
To make sure the right data types are inferred, either set
:attr:`ReadOptions.block_size` to a large enough value, or use
:attr:`ParseOptions.explicit_schema` to set the desired data types explicitly.