bw_processing.datapackage

Classes

Datapackage	Interface for creating, loading, and using numerical datapackages for Brightway.
DatapackageBase	Base class for datapackages. Not for normal use - you should use either Datapackage or FilteredDatapackage.
FilteredDatapackage	A subset of a datapackage. Used in matrix construction or other data manipulation operations.

Functions

create_datapackage(→ Datapackage)	Create a new data package.
load_datapackage(→ Datapackage)	Load an existing datapackage.
simple_graph(→ Datapackage)	Easy creation of simple datapackages with only persistent vectors.

Module Contents

class bw_processing.datapackage.Datapackage

Bases: DatapackageBase

Interface for creating, loading, and using numerical datapackages for Brightway.

Note that there are two entry points to using this class, both separate functions: create_datapackage() and load_datapackage(). Do not create an instance of the class with Datapackage(), unless you like playing with danger :)

Data packages can be stored in memory, in a directory, or in a zip file. When creating data packages for use later, don’t forget to call .finalize_serialization(), or the metadata won’t be written and the data package won’t be usable.

Potential gotchas:

• There is currently no way to modify a zipped data package once it is finalized.

• Resources that are interfaces to external data sources (either in Python or other) can’t be saved, but must be recreated each time a data package is used.

_add_numpy_array_resource(*, array: numpy.ndarray, name: str, matrix: str, kind: str, keep_proxy: bool = False, matrix_serialize_format_type: bw_processing.constants.MatrixSerializeFormat | None = None, meta_object: str | None = None, meta_type: str | None = None, **kwargs) → None

_check_length_consistency() → None

_create(fs: fsspec.AbstractFileSystem | None, name: str | None, id_: str | None, metadata: dict | None, combinatorial: bool = False, sequential: bool = False, seed: int | None = None, sum_intra_duplicates: bool = True, sum_inter_duplicates: bool = False, matrix_serialize_format_type: bw_processing.constants.MatrixSerializeFormat = MatrixSerializeFormat.NUMPY) → None

Start a new data package.

All metadata elements should follow the datapackage specification.

Licenses are specified as a list in metadata. The default license is the Open Data Commons Public Domain Dedication and License v1.0.

_load(fs: fsspec.AbstractFileSystem, mmap_mode: str | None = None, proxy: bool = False) → None

_load_all(mmap_mode: str | None = None, proxy: bool = False) → None

_prepare_modifications() → None

_prepare_name(name: str) → str

add_csv_metadata(*, dataframe: pandas.DataFrame, valid_for: list, name: str = None, **kwargs) → None

Add an iterable metadata object to be stored as a CSV file.

The purpose of storing metadata is to enable data exchange; therefore, this method assumes that data is written to disk.

The normal use case of this method is to link integer indices from either structured or presample arrays to a set of fields that uniquely identifies each object. This allows for matching based on object attributes from computer to computer, where database ids or other computer-generated codes might not be consistent.

Uses pandas to store and load data; therefore, metadata must already be a pandas dataframe.

In contrast with presamples arrays, iterable_data_source cannot be an infinite generator. We need a finite set of data to build a matrix.

In contrast to self.create_structured_array, this always stores the dataframe in self.data; no proxies are used.

Parameters:

• dataframe (*) – Dataframe to be persisted to disk.

• valid_for (*) – List of resource names that this metadata is valid for; must be either structured or presample indices arrays. Each item in valid_for has the form ("resource_name", "rows" or "cols"). resource_name should be either a structured or a presamples indices array.

• name (*) – The name of this resource. Names must be unique in a given data package

• extra (*) – Dict of extra metadata

Returns:

Nothing, but appends objects to self.metadata['resources'] and self.data.

Raises:

• * AssertionError – If inputs are not in correct form

• * AssertionError – If valid_for refers to unavailable resources

add_dynamic_array(*, matrix: str, interface: Any, indices_array: numpy.ndarray, name: str | None = None, flip_array: numpy.ndarray | None = None, keep_proxy: bool = False, matrix_serialize_format_type: bw_processing.constants.MatrixSerializeFormat | None = None, **kwargs) → None

interface must support the presamples API.

add_dynamic_vector(*, matrix: str, interface: Any, indices_array: numpy.ndarray, name: str | None = None, flip_array: numpy.ndarray | None = None, keep_proxy: bool = False, matrix_serialize_format_type: bw_processing.constants.MatrixSerializeFormat | None = None, **kwargs) → None

add_entries(*, matrix: str, entries: list[bw_processing.matrix_entry.MatrixEntry], name: str | None = None) → None

Add matrix data from a list of MatrixEntry objects.

High-level convenience method that does not require working directly with NumPy arrays.

Parameters:

• matrix – Name of the target matrix (e.g. "technosphere").

• entries – List of MatrixEntry instances.

• name – Optional resource group name; auto-generated if omitted.

add_json_metadata(*, data: Any, valid_for: str, name: str = None, **kwargs) → None

Add an iterable metadata object to be stored as a JSON file.

The purpose of storing metadata is to enable data exchange; therefore, this method assumes that data is written to disk.

The normal use case of this method is to provide names and other metadata for parameters whose values are stored as presamples arrays. The length of data should match the number of rows in the corresponding presamples array, and data is just a list of string labels for the parameters. However, this method can also be used to store other metadata, e.g. for external data resources.

In contrast to self.create_structured_array, this always stores the dataframe in self.data; no proxies are used.

Parameters:

• data (*) – Data to be persisted to disk.

• valid_for (*) – Name of structured data or presample array that this metadata is valid for.

• name (*) – The name of this resource. Names must be unique in a given data package

• extra (*) – Dict of extra metadata

Returns:

Nothing, but appends objects to self.metadata['resources'] and self.data.

Raises:

• * AssertionError – If inputs are not in correct form

• * AssertionError – If valid_for refers to unavailable resources

add_persistent_array(*, matrix: str, data_array: numpy.ndarray, indices_array: numpy.ndarray, name: str | None = None, flip_array: numpy.ndarray | None = None, keep_proxy: bool = False, matrix_serialize_format_type: bw_processing.constants.MatrixSerializeFormat | None = None, **kwargs) → None

add_persistent_vector(*, matrix: str, indices_array: numpy.ndarray, name: str | None = None, data_array: numpy.ndarray | None = None, flip_array: numpy.ndarray | None = None, distributions_array: numpy.ndarray | None = None, keep_proxy: bool = False, matrix_serialize_format_type: bw_processing.constants.MatrixSerializeFormat | None = None, **kwargs) → None

add_persistent_vector_from_iterator(*, matrix: str = None, name: str | None = None, dict_iterator: Any = None, nrows: int | None = None, matrix_serialize_format_type: bw_processing.constants.MatrixSerializeFormat | None = None, **kwargs) → None

Create a persistant vector from an iterator. Uses the utility function resolve_dict_iterator.

This is the only array creation method which produces sorted arrays.

finalize_serialization() → None

write_modified()

Write the data in modified files to the filesystem (if allowed).

class bw_processing.datapackage.DatapackageBase

Bases: abc.ABC

Base class for datapackages. Not for normal use - you should use either Datapackage or FilteredDatapackage.

__get_resources() → list

__set_resources(dct: dict) → None

_dehydrate_interfaces() → None

Substitute an interface resource with UndefinedInterface, in preparation for finalizing data on disk.

_get_index(name_or_index: str | int) → int

Get index of a resource by name or index.

Returning the same number is a bit silly, but makes the other code simpler :)

Raises:

• * IndexError – name_or_index was too big

• * ValueError – Name name_or_index not found

• * NonUnique – Name name_or_index not unique in given resources

dehydrated_interfaces() → List[str]

Return a list of the resource groups which have dehydrated interfaces

del_resource(name_or_index: str | int) → None

Remove a resource, and delete its data file, if any.

del_resource_group(name: str) → None

Remove a resource group, and delete its data files, if any.

Use exclude_resource_group if you want to keep the underlying resource in the filesystem.

exclude(filters: Dict[str, str]) → FilteredDatapackage

Filter a datapackage to exclude resources matching a filter.

Usage cases:

Filter out a given resource:

exclude_generic({“matrix’: “some_label”})

Filter out a resource group with a given kind:

exclude_generic({“group’: “some_group”, “kind”: “some_kind”})

filter_by_attribute(key: str, value: Any) → FilteredDatapackage

Create a new FilteredDatapackage which satisfies the filter resource[key] == value.

All included objects are the same as in the original data package, i.e. no copies are made. No checks are made to ensure consistency with modifications to the original datapackage after the creation of this filtered datapackage.

This method was introduced to allow for the efficient construction of matrices; each datapackage can have data for multiple matrices, and we can then create filtered datapackages which exclusively have data for the matrix of interest. As such, they should be considered read-only, though this is not enforced.

get_max_index_value() → int

Get maximum index value (max signed 32 or 64 bit integer) for this datapackage

get_resource(name_or_index: str | int)

Return data and metadata for name_or_index.

Parameters:

name_or_index (*) – Name (str) or index (int) of a resource in the existing metadata.

Raises:

• * IndexError – Integer index out of range of given metadata

• * ValueError – String name not present in metadata

• * NonUnique – String name present in two resource metadata sections

Returns:

(data object, metadata dict)

rehydrate_interface(name_or_index: str | int, resource: Any, initialize_with_config: bool = False) → None

Substitute the undefined interface in this datapackage with the actual interface resource resource. Loading a datapackage with an interface loads an instance of UndefinedInterface, which should be substituted (rehydrated) with an actual interface instance.

If initialize_with_config is true, the resource is initialized (i.e. resource(**config_data)) with the resource data under the key config. If config is missing, a KeyError is raised.

name_or_index should be the data source name. If this value is a string and doesn’t end with .data, .data is automatically added.

_finalized = False

_matrix_serialize_format_type

_modified

property groups: dict

Return a dictionary of {group label: filtered datapackage} in the same order as the group labels are first encountered in the datapackage metadata.

Ignores resources which don’t have group labels.

resources

class bw_processing.datapackage.FilteredDatapackage

Bases: DatapackageBase

A subset of a datapackage. Used in matrix construction or other data manipulation operations.

Should be treated as read-only.

bw_processing.datapackage.create_datapackage(fs: fsspec.AbstractFileSystem | None = None, name: str | None = None, id_: str | None = None, metadata: dict | None = None, combinatorial: bool = False, sequential: bool = False, seed: int | None = None, sum_intra_duplicates: bool = True, sum_inter_duplicates: bool = False, matrix_serialize_format_type: bw_processing.constants.MatrixSerializeFormat = MatrixSerializeFormat.NUMPY) → Datapackage

Create a new data package.

All arguments are optional; if a fsspec filesystem is not provided, an in-memory DictFS will be used.

All metadata elements should follow the datapackage specification.

Licenses are specified as a list in metadata. The default license is the Open Data Commons Public Domain Dedication and License v1.0.

Parameters:

• fs (*) – A Filesystem, optional. A new DictFS is used if not provided.

• name (*) – str, optional. A new uuid is used if not provided.

• id_ (*) – str, optional. A new uuid is used if not provided.

• metadata (*) – dict, optional. Metadata dictionary following datapackage specification; see above.

• combinatorial (*) – bool, default False .: Policy on how to sample columns across multiple data arrays; see readme.

• sequential (*) – bool, default False .: Policy on how to sample columns in data arrays; see readme.

• seed (*) – int, optional. Seed to use in random number generator.

• sum_intra_duplicates (*) – bool, default True. Should duplicate elements in a single data resource be summed together, or should the last value replace previous values.

• sum_inter_duplicates (*) – bool, default False. Should duplicate elements in across data resources be summed together, or should the last value replace previous values. Order of data resources is given by the order they are added to the data package.

• matrix_serialize_format_type (*) – MatrixSerializeFormat, default MatrixSerializeFormat.NUMPY. Matrix serialization format type.

Returns:

A Datapackage instance.

bw_processing.datapackage.load_datapackage(fs_or_obj: DatapackageBase | fsspec.AbstractFileSystem, mmap_mode: str | None = None, proxy: bool = False) → Datapackage

Load an existing datapackage.

Can load proxies to data instead of the data itself, which can be useful when interacting with large arrays or large packages where only a subset of the data will be accessed.

Proxies use something similar to functools.partial to create a callable class instead of returning the raw data (see https://github.com/brightway-lca/bw_processing/issues/9 for why we can’t just use partial). datapackage access methods (i.e. .get_resource) will automatically resolve proxies when needed.

Parameters:

• fs_or_obj (*) – A Filesystem or an instance of DatapackageBase.

• mmap_mode (*) – str, optional. Define memory mapping mode to use when loading Numpy arrays.

• proxy (*) – bool, default False. Load proxies instead of complete Numpy arrays; see above.

Returns:

A Datapackage instance.

bw_processing.datapackage.simple_graph(data: dict, fs: fsspec.AbstractFileSystem | None = None, **metadata) → Datapackage

Easy creation of simple datapackages with only persistent vectors.

Deprecated since version Use: bw_processing.matrix_entry.create_datapackage_from_entries() with bw_processing.matrix_entry.MatrixEntry objects instead.

Parameters:

• data (*) –

  is a dictionary. The data dictionary has the form:

  {
      matrix_name: [
      (row_id, col_id, value, flip)
      ]
  }
  

  Where row_id and col_id are an `int s, value is a float and flip is a bool (False by default).

• fs (*) – is a filesystem.

• metadata (*) – are passed as kwargs to create_datapackage().

Returns:

the datapackage.