bw_processing.io_pyarrow_helpers
================================

.. py:module:: bw_processing.io_pyarrow_helpers

.. autoapi-nested-parse::

   This module contains some helpers to convert `nympy.ndarrays` to/from Apache `Arrow` `Table`.

   We use `pyarrow.Table` objects to save/retrieve data into/from `parquet` format files.
   We use a `metadata` section in the `pyarrow.Table` (and the `parquet` files) to be able
   to recognize what type of data was serialized. Specific and generic codes exist.

   The metadata object is a `dict` object that looks like this:

   `{"object": "vector", "type": "generic"}`. `object` can be `vector` (`ndim == 1`) or `matrix` (`ndim == 2`),
   `type` can be:

   - `indices` (`dtype` is `INDICES_DTYPE`);
   - `distributions` (`dtype` is `UNCERTAINTY_DTYPE`);
   - `generic` (`dtype` is a common type);



Attributes
----------

.. autoapisummary::

   bw_processing.io_pyarrow_helpers.INDICES_SCHEMA
   bw_processing.io_pyarrow_helpers.NBR_UNCERTAINTY_FIELDS
   bw_processing.io_pyarrow_helpers.PA_UNCERTAINTY_FIELDS
   bw_processing.io_pyarrow_helpers.UNCERTAINTY_FIELDS_NAMES
   bw_processing.io_pyarrow_helpers.UNCERTAINTY_SCHEMA


Functions
---------

.. autoapisummary::

   bw_processing.io_pyarrow_helpers.numpy_distributions_vector_to_pyarrow_distributions_vector_table
   bw_processing.io_pyarrow_helpers.numpy_generic_matrix_to_pyarrow_generic_matrix_table
   bw_processing.io_pyarrow_helpers.numpy_generic_vector_to_pyarrow_generic_vector_table
   bw_processing.io_pyarrow_helpers.numpy_indices_vector_to_pyarrow_indices_vector_table
   bw_processing.io_pyarrow_helpers.pyarrow_distributions_vector_table_to_numpy_distributions_vector
   bw_processing.io_pyarrow_helpers.pyarrow_generic_matrix_table_to_numpy_generic_matrix
   bw_processing.io_pyarrow_helpers.pyarrow_generic_vector_table_to_numpy_generic_vector
   bw_processing.io_pyarrow_helpers.pyarrow_indices_vector_table_to_numpy_indices_vector


Module Contents
---------------

.. py:function:: numpy_distributions_vector_to_pyarrow_distributions_vector_table(arr: numpy.ndarray) -> pyarrow.Table

   Convert a specific distributions (numpy) vector to a (arrow) table.

   :param arr: A numpy array that corresponds to a distributions vector, i.e. its dimension is 1 and its
               `dtype` is `UNCERTAINTY_DTYPE`.
   :type arr: np.ndarray

   See:
       `pyarrow_distributions_vector_table_to_numpy_distributions_vector`

   :returns: The corresponding `pyarrow.Table` object.


.. py:function:: numpy_generic_matrix_to_pyarrow_generic_matrix_table(arr: numpy.ndarray) -> pyarrow.Table

   Convert a generic (numpy) matrix to a (arrow) table.

   :param arr: A numpy array that corresponds to a generic matrix, i.e. its dimension is 2.
   :type arr: ndarray

   See:
       `pyarrow_generic_matrix_table_to_numpy_generic_matrix`.

   :returns: The corresponding `pyarrow.Table` object.


.. py:function:: numpy_generic_vector_to_pyarrow_generic_vector_table(arr: numpy.ndarray) -> pyarrow.Table

   Convert a generic (numpy) vector to a (arrow) table.

   :param arr: A numpy array that corresponds to a vector, i.e. its dimension is 1.
   :type arr: ndarray

   See:
       `pyarrow_generic_vector_table_to_numpy_generic_vector`.

   :returns: The corresponding `pyarrow.Table` object.


.. py:function:: numpy_indices_vector_to_pyarrow_indices_vector_table(arr: numpy.ndarray) -> pyarrow.Table

   Convert a specific indices (numpy) vector to a (arrow) table.

   :param arr: A numpy array that corresponds to an indices vector, i.e. its dimension is 1 and its
               `dtype` is `INDICES_DTYPE`.
   :type arr: ndarray

   See:
       `pyarrow_indices_vector_table_to_numpy_indices_vector`.

   :returns: The corresponding `pyarrow.Table` object.


.. py:function:: pyarrow_distributions_vector_table_to_numpy_distributions_vector(table: pyarrow.Table) -> numpy.ndarray

   Convert a specific distributions (arrow) vector table to a (numpy) array.

   :param table: A `pyarrow` table that corresponds to a distributions vector.
   :type table: pa.Table

   See:
       `numpy_distributions_vector_to_pyarrow_distributions_vector_table`.

   :returns: The corresponding `np.ndarray` object.


.. py:function:: pyarrow_generic_matrix_table_to_numpy_generic_matrix(table: pyarrow.Table) -> numpy.ndarray

   Convert a generic (arrow) matrix table to a (numpy) array.

   :param table: A `pyarrow` table that corresponds to a generic matrix.
   :type table: pa.Table

   See:
       `numpy_generic_matrix_to_pyarrow_generic_matrix_table`.

   :returns: The corresponding `np.ndarray` object.


.. py:function:: pyarrow_generic_vector_table_to_numpy_generic_vector(table: pyarrow.Table) -> numpy.ndarray

   Convert a generic (arrow) vector table to a (numpy) array.

   :param table: A `pyarrow` table that corresponds to a vector.
   :type table: pa.Table

   See:
       `numpy_generic_vector_to_pyarrow_generic_vector_table`.

   :returns: The corresponding `np.ndarray` object.


.. py:function:: pyarrow_indices_vector_table_to_numpy_indices_vector(table: pyarrow.Table) -> numpy.ndarray

   Convert a specific indices (arrow) vector table to a (numpy) array.

   :param table: A `pyarrow` table that corresponds to an indices vector.
   :type table: pa.Table

   See:
       `numpy_indices_vector_to_pyarrow_indices_vector_table`.

   :returns: The corresponding `np.ndarray` object.


.. py:data:: INDICES_SCHEMA

.. py:data:: NBR_UNCERTAINTY_FIELDS

.. py:data:: PA_UNCERTAINTY_FIELDS

.. py:data:: UNCERTAINTY_FIELDS_NAMES

.. py:data:: UNCERTAINTY_SCHEMA