docs/source/java/vector.rst

.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements.  See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership.  The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License.  You may obtain a copy of the License at

..   http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing,
.. software distributed under the License is distributed on an
.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
.. KIND, either express or implied.  See the License for the
.. specific language governing permissions and limitations
.. under the License.

===========
ValueVector
===========

:class:`ValueVector` interface (which called Array in C++ implementation and
the :doc:`the specification <../format/Columnar>`) is an abstraction that is used to store a
sequence of values having the same type in an individual column. Internally, those values are
represented by one or several buffers, the number and meaning of which depend on the vector’s data type.

There are concrete subclasses of :class:`ValueVector` for each primitive data type
and nested type described in the specification. There are a few differences in naming
with the type names described in the specification:
Table with non-intuitive names (BigInt = 64 bit integer, etc).

It is important that vector is allocated before attempting to read or write,
:class:`ValueVector` "should" strive to guarantee this order of operation:
create > allocate > mutate > set value count > access > clear (or allocate to start the process over).
We will go through a concrete example to demonstrate each operation in the next section.

Vector Life Cycle
=================

As discussed above, each vector goes through several steps in its life cycle,
and each step is triggered by a vector operation. In particular, we have the following vector operations:

1. **Vector creation**: we create a new vector object by, for example, the vector constructor.
The following code creates a new ``IntVector`` by the constructor:

.. code-block:: Java

    RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
    ...
    IntVector vector = new IntVector("int vector", allocator);

By now, a vector object is created. However, no underlying memory has been allocated, so we need the
following step.

2. **Vector allocation**: in this step, we allocate memory for the vector. For most vectors, we
have two options: 1) if we know the maximum vector capacity, we can specify it by calling the
``allocateNew(int)`` method; 2) otherwise, we should call the ``allocateNew()`` method, and  a default
capacity will be allocated for it. For our running example, we assume that the vector capacity never
exceeds 10:

.. code-block:: Java

    vector.allocateNew(10);

3. **Vector mutation**: now we can populate the vector with values we desire. For all vectors, we can populate
vector values through vector writers (An example will be given in the next section). For primitive types,
we can also mutate the vector by the set methods. There are two classes of set methods: 1) if we can
be sure the vector has enough capacity, we can call the ``set(index, value)`` method. 2) if we are not sure
about the vector capacity, we should call the ``setSafe(index, value)`` method, which will automatically
take care of vector reallocation, if the capacity is not sufficient. For our running example, we know the
vector has enough capacity, so we can call

.. code-block:: Java

    vector.set(/*index*/5, /*value*/25);

4. **Set value count**: for this step, we set the value count of the vector by calling the
``setValueCount(int)`` method:

.. code-block:: Java

    vector.setValueCount(10);

After this step, the vector enters an immutable state. In other words, we should no longer mutate it.
(Unless we reuse the vector by allocating it again. This will be discussed shortly.)

5. **Vector access**: it is time to access vector values. Similarly, we have two options to access values:
1) get methods and 2) vector reader. Vector reader works for all types of vectors, while get methods are
only available for primitive vectors. A concrete example for vector reader will be given in the next section.
Below is an example of vector access by get method:

.. code-block:: Java

    int value = vector.get(5);  // value == 25

6. **Vector clear**: when we are done with the vector, we should clear it to release its memory. This is done by
calling the ``close()`` method:

.. code-block:: Java

    vector.close();

Some points to note about the steps above:

* The steps are not necessarily performed in a linear sequence. Instead, they can be in a loop. For example,
  when a vector enters the access step, we can also go back to the vector mutation step, and then set value
  count, access vector, and so on.

* We should try to make sure the above steps are carried out in order. Otherwise, the vector
  may be in an undefined state, and some unexpected behavior may occur. However, this restriction
  is not strict. That means it is possible that we violates the order above, but still get
  correct results.

* When mutating vector values through set methods, we should prefer ``set(index, value)`` methods to
  ``setSafe(index, value)`` methods whenever possible, to avoid unnecessary performance overhead of handling
  vector capacity.

* All vectors implement the ``AutoCloseable`` interface. So they must be closed explicitly when they are
  no longer used, to avoid resource leak. To make sure of this, it is recommended to place vector related operations
  into a try-with-resources block.

* For fixed width vectors (e.g. IntVector), we can set values at different indices in arbitrary orders.
  For variable width vectors (e.g. VarCharVector), however, we must set values in non-decreasing order of the
  indices. Otherwise, the values after the set position will become invalid. For example, suppose we use the
  following statements to populate a variable width vector:

.. code-block:: Java

    VarCharVector vector = new VarCharVector("vector", allocator);
    vector.allocateNew();
    vector.setSafe(0, "zero");
    vector.setSafe(1, "one");
    ...
    vector.setSafe(9, "nine");

Then we set the value at position 5 again:

.. code-block:: Java

    vector.setSafe(5, "5");

After that, the values at positions 6, 7, 8, and 9 of the vector will become invalid.

Building ValueVector
====================

Note that the current implementation doesn't enforce the rule that Arrow objects are immutable.
:class:`ValueVector` instances could be created directly by using new keyword, there are
set/setSafe APIs and concrete subclasses of FieldWriter for populating values.

For example, the code below shows how to build a :class:`BigIntVector`, in this case, we build a
vector of the range 0 to 7 where the element that should hold the fourth value is nulled

.. code-block:: Java

    try (BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE);
      BigIntVector vector = new BigIntVector("vector", allocator)) {
      vector.allocateNew(8);
      vector.set(0, 1);
      vector.set(1, 2);
      vector.set(2, 3);
      vector.setNull(3);
      vector.set(4, 5);
      vector.set(5, 6);
      vector.set(6, 7);
      vector.set(7, 8);
      vector.setValueCount(8); // this will finalizes the vector by convention.
      ...
    }

The :class:`BigIntVector` holds two ArrowBufs. The first buffer holds the null bitmap, which consists
here of a single byte with the bits 1|1|1|1|0|1|1|1 (the bit is 1 if the value is non-null).
The second buffer contains all the above values. As the fourth entry is null, the value at that position
in the buffer is undefined. Note compared with set API, setSafe API would check value capacity before setting
values and reallocate buffers if necessary.

Here is how to build a vector using writer

.. code-block:: Java

    try (BigIntVector vector = new BigIntVector("vector", allocator);
      BigIntWriter writer = new BigIntWriterImpl(vector)) {
      writer.setPosition(0);
      writer.writeBigInt(1);
      writer.setPosition(1);
      writer.writeBigInt(2);
      writer.setPosition(2);
      writer.writeBigInt(3);
      // writer.setPosition(3) is not called which means the forth value is null.
      writer.setPosition(4);
      writer.writeBigInt(5);
      writer.setPosition(5);
      writer.writeBigInt(6);
      writer.setPosition(6);
      writer.writeBigInt(7);
      writer.setPosition(7);
      writer.writeBigInt(8);
    }

There are get API and concrete subclasses of :class:`FieldReader` for accessing vector values, what needs
to be declared is that writer/reader is not as efficient as direct access

.. code-block:: Java

    // access via get API
    for (int i = 0; i < vector.getValueCount(); i++) {
      if (!vector.isNull(i)) {
        System.out.println(vector.get(i));
      }
    }

    // access via reader
    BigIntReader reader = vector.getReader();
    for (int i = 0; i < vector.getValueCount(); i++) {
      reader.setPosition(i);
      if (reader.isSet()) {
        System.out.println(reader.readLong());
      }
    }

Building ListVector
===================

A :class:`ListVector` is a vector that holds a list of values for each index. Working with one you need to handle the same steps as mentioned above (create > allocate > mutate > set value count > access > clear), but the details of how you accomplish this are slightly different since you need to both create the vector and set the list of values for each index.

For example, the code below shows how to build a :class:`ListVector` of int's using the writer :class:`UnionListWriter`. We build a vector from 0 to 9 and each index contains a list with values [[0, 0, 0, 0, 0], [0, 1, 2, 3, 4], [0, 2, 4, 6, 8], …, [0, 9, 18, 27, 36]]. List values can be added in any order so writing a list such as [3, 1, 2] would be just as valid.

.. code-block:: Java
  
  try (BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE);
    ListVector listVector = ListVector.empty("vector", allocator)) {
    UnionListWriter writer = listVector.getWriter();
    for (int i = 0; i < 10; i++) {
       writer.startList();
       writer.setPosition(i);
       for (int j = 0; j < 5; j++) {
           writer.writeInt(j * i);
       }
       writer.setValueCount(5);
       writer.endList();
    }
    listVector.setValueCount(10);
  }    

:class:`ListVector` values can be accessed either through the get API or through the reader class :class:`UnionListReader`. To read all the values, first enumerate through the indexes, and then enumerate through the inner list values.

.. code-block:: Java

  // access via get API
  for (int i = 0; i < listVector.getValueCount(); i++) {
     if (!listVector.isNull(i)) {
         ArrayList<Integer> elements = (ArrayList<Integer>) listVector.getObject(i);
         for (Integer element : elements) {
             System.out.println(element);
         }
     }
  }

  // access via reader
  UnionListReader reader = listVector.getReader();
  for (int i = 0; i < listVector.getValueCount(); i++) {
     reader.setPosition(i);
     while (reader.next()) {
         IntReader intReader = reader.reader();
         if (intReader.isSet()) {
             System.out.println(intReader.readInteger());
         }
     }
  }

Slicing
=======

Similar with C++ implementation, it is possible to make zero-copy slices of vectors to obtain a vector
referring to some logical sub-sequence of the data through :class:`TransferPair`

.. code-block:: Java

    IntVector vector = new IntVector("intVector", allocator);
    for (int i = 0; i < 10; i++) {
      vector.setSafe(i, i);
    }
    vector.setValueCount(10);

    TransferPair tp = vector.getTransferPair(allocator);
    tp.splitAndTransfer(0, 5);
    IntVector sliced = (IntVector) tp.getTo();
    // In this case, the vector values are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] and the sliceVector values are [0, 1, 2, 3, 4].