MetadataInventory

#include <sbio/core/metadata.hh>

template<typename FTraits, hd_std::size_t MaxGroups = 128, hd_std::size_t MaxFields = 1024>
struct MetadataInventory

Defined in src/lib/sbio/core/metadata.hh:167

The MetadataInventory provides a lookup of a StreamBroker’s fields.

The inventory stores the names, types, and segment identifiers of discovered groups. It additionally associates these group segments to any data fields that they may have. The field lookup and association is data-format-specific and uses the same mechanisms to generalize this as the DataRequest - the DataRequest is in fact used to query the lookup tables.

The inventory has a two-stage life-cycle. During the discovery stage of the StreamBroker’s state machine, it is filled dynamically - i.e., there is an upper-bound on the capacity, but the size grows during discovery. Once the discovery phase completes, the inventory is “baked”, or finalized, which consists of sorting the constructed lookups for fast access.

The inventory maintains epoch counters to help establish if the stored data is changing over time - e.g., if fields associated to a group segment may return variably sized data, then offsets cannot be used directly if stale. These counters can help inform decisions about crossing memory space boundaries (e.g. performing GPU<->CPU copies) and similar activities.

Template Parameters

FTraits The data format to create the inventory for.
MaxGroups The maximum capacity for the number of groups (BrokerGroups).
MaxFields The maximum capacity for the number of fields.

Public Attributes

Return	Name	Description
`hd_std::uint64_t`	`m_revision`
std::vector< `GroupEntry` >	`m_groups`
std::vector< `FieldEntry` >	`m_fields`

m_revision

hd_std::uint64_t m_revision { 0 }

Defined in src/lib/sbio/core/metadata.hh:307

m_groups

std::vector< GroupEntry > m_groups

Type: std::vector< GroupEntry >

Defined in src/lib/sbio/core/metadata.hh:315

m_fields

std::vector< FieldEntry > m_fields

Type: std::vector< FieldEntry >

Defined in src/lib/sbio/core/metadata.hh:316

Public Methods

Return	Name	Description
`SBIO_HD` void	`touch` `inline`
`SBIO_HD` const auto &	`groups` `const` `inline`	Access the registered groups directly.
`SBIO_HD` const auto &	`fields` `const` `inline`	Access the registered fields directly.
`SBIO_HD` hd_std::uint32_t	`register_group` `inline` `requires (sizeof...(GroupKeyStringArgs) == GroupKeys::count)`	Add a new group segment into the inventory.
`SBIO_HD` hd_std::uint32_t	`register_group_alias` `inline` `requires (sizeof...(GroupKeyStringArgs) == GroupKeys::count)`	Add a new group segment into the inventory.
`SBIO_HD` void	`add_field` `inline` `requires (sizeof...(KeyStringArgs) == KeyCount)`	Add a new field into the inventory.
`SBIO_HD` void	`finalize` `inline`	Sort and finalize metadata after the discovery phase is completed.
`SBIO_HD` hd_std::uint64_t	`revision` `const` `inline`	An number for the inventory which changes whenver the inventory does.
`SBIO_HD` const `FieldEntry` *	`lookup` `const` `inline`	Return a descriptor for the requested data.
`SBIO_HD` hd_std::uint32_t	`field_id` `const` `inline`	The identifier of a field entry (position in the inventory.)
`SBIO_HD` hd_std::uint64_t	`get_epoch` `const` `inline`	For the specified group, retrieve the epoch counter for its descriptors.
`SBIO_HD` hd_std::size_t	`num_entries` `const` `inline`	Return the total number of groups inventoried.
`SBIO_HD` bool	`entry_matches` `const` `inline`	Query whether a specific entry index matches a name/access pattern.
`SBIO_HD` auto	`metadata_for` `const` `inline`	Retrieve the group name and segment number for an entry index.

touch

inline

inline SBIO_HD void touch()

Defined in src/lib/sbio/core/metadata.hh:299

groups

const inline

inline SBIO_HD const auto & groups() const

Defined in src/lib/sbio/core/metadata.hh:324

Access the registered groups directly.

Returns

All registered groups.

fields

const inline

inline SBIO_HD const auto & fields() const

Defined in src/lib/sbio/core/metadata.hh:331

Access the registered fields directly.

Returns

All registered fields.

register_group

inline requires (sizeof...(GroupKeyStringArgs) == GroupKeys::count)

template<typename... GroupKeyStringArgs> inline SBIO_HD hd_std::uint32_t register_group(const char * name, const char * type, hd_std::uint32_t segment, DataAccessPtn ptn, GroupKeyStringArgs... grp_meta) requires (sizeof...(GroupKeyStringArgs) == GroupKeys::count)

Defined in src/lib/sbio/core/metadata.hh:350

Add a new group segment into the inventory.

This function only adds the group segment’s entry. It does not associate any fields to that segment. If that is needed, see add_field below as well.

Returns

The entry of the group segment in the inventory. This can be used to further inventory associated fields as needed.

Parameters

Parameter	Type	Description
`name`	`const char *`	The name of the group.
`type`	`const char *`	The group type.
`segment`	`hd_std::uint32_t`	The segment number associated to the entry.
`ptn`	`DataAccessPtn`	The DataAccessPtn for the lookup strategy for the group segment.
`grp_meta`	`GroupKeyStringArgs...`	The values for any additional metadata keys defined by the format.

register_group_alias

inline requires (sizeof...(GroupKeyStringArgs) == GroupKeys::count)

template<typename... GroupKeyStringArgs> inline SBIO_HD hd_std::uint32_t register_group_alias(const char * alias_name, const char * parent_type, hd_std::uint32_t segment, DataAccessPtn ptn, GroupKeyStringArgs... grp_meta) requires (sizeof...(GroupKeyStringArgs) == GroupKeys::count)

Defined in src/lib/sbio/core/metadata.hh:400

Add a new group segment into the inventory.

This function only adds the group segment’s entry. It does not associate any fields to that segment. If that is needed, see add_field below as well.

Returns

The entry of the group segment in the inventory. This can be used to further inventory associated fields as needed.

Parameters

Parameter	Type	Description
`segment`	`hd_std::uint32_t`	The segment number associated to the entry.
`ptn`	`DataAccessPtn`	The DataAccessPtn for the lookup strategy for the group segment.
`grp_meta`	`GroupKeyStringArgs...`	The values for any additional metadata keys defined by the format.

add_field

inline requires (sizeof...(KeyStringArgs) == KeyCount)

template<typename... KeyStringArgs> inline SBIO_HD void add_field(hd_std::uint32_t group_id, hd_std::uint32_t offset, ncarray::DType dtype, hd_std::uint16_t rank, const hd_std::uint32_t * shape, FormatMetadata fmt_meta, KeyStringArgs... key_values) requires (sizeof...(KeyStringArgs) == KeyCount)

Defined in src/lib/sbio/core/metadata.hh:429

Add a new field into the inventory.

Parameters

Parameter	Type	Description
`group_id`	`hd_std::uint32_t`	The group index to which the field belongs.
`offset`	`hd_std::uint32_t`	The byte offset to use to read this field.
`dtype`	`ncarray::DType`	The datatype of the field’s data.
`rank`	`hd_std::uint16_t`	The total number of dimensions (rank) of the field’s data.
`shape`	`const hd_std::uint32_t *`	The full shape of the field’s data.
`fmt_meta`	`FormatMetadata`	Any format-specific metadata required. This may be an empty struct.

finalize

inline

inline SBIO_HD void finalize()

Defined in src/lib/sbio/core/metadata.hh:464

Sort and finalize metadata after the discovery phase is completed.

Containers are sorted after the discovery is complete to facilitate faster traversal and lookup.

revision

const inline

inline SBIO_HD hd_std::uint64_t revision() const

Defined in src/lib/sbio/core/metadata.hh:480

An number for the inventory which changes whenver the inventory does.

Revision numbers are unique among all the inventories on the host. This applies to the host ONLY though. As a result, if an inventory is copied to a device (GPU) it can be verified if the revision matches by checking only the revision number (and address) against the one on the host.

Returns

The revision number for the MetadataInventory.

lookup

const inline

inline SBIO_HD const FieldEntry * lookup(const DataRequest & req) const

Defined in src/lib/sbio/core/metadata.hh:493

Return a descriptor for the requested data.

For a given request, the associated descriptor points to a location in a DataUnit read, and the appropriate size, shape, and type information to interpret the pointed to payload.

Returns

The entry with key (for indices), and data descriptor to find the requested data.

Parameters

Parameter	Type	Description
`req`	const `DataRequest` &	The DataRequest with any format-specific fields included.

field_id

const inline

inline SBIO_HD hd_std::uint32_t field_id(const FieldEntry * entry) const

Defined in src/lib/sbio/core/metadata.hh:535

The identifier of a field entry (position in the inventory.)

The identifier is stable as long as the inventory is not rebuilt. It is used as a key for per-unit cache lookups (PayloadOffsetCache). The lookup result must be validated regardless, however.

Returns

The entry’s identifier.

Parameters

Parameter	Type	Description
`entry`	const `FieldEntry` *	An entry returned by `lookup`

get_epoch

const inline

inline SBIO_HD hd_std::uint64_t get_epoch(hd_std::uint32_t group_id) const

Defined in src/lib/sbio/core/metadata.hh:551

For the specified group, retrieve the epoch counter for its descriptors.

The epoch is incremented when its determined that the descriptor data has changed. If the data format does not change over the course of the read steps, then the value will always be 0 for the epoch. If it does, for instance if the returned payloads may change in size, then the epoch will be incremented accordingly.

Returns

The epoch counter.

Parameters

Parameter	Type	Description
`group_id`	`hd_std::uint32_t`	The index for the group to retrieve the epoch for.

num_entries

const inline

inline SBIO_HD hd_std::size_t num_entries() const

Defined in src/lib/sbio/core/metadata.hh:566

Return the total number of groups inventoried.

Returns

The total number of groups inventoried.

entry_matches

const inline

inline SBIO_HD bool entry_matches(hd_std::size_t entry_no, const char * name_query, DataAccessPtn ptn) const

Defined in src/lib/sbio/core/metadata.hh:576

Query whether a specific entry index matches a name/access pattern.

Returns

Whether the query matches.

Parameters

Parameter	Type	Description
`entry_no`	`hd_std::size_t`	The entry index.
`name_query`	`const char *`	The group name to test for.
`ptn`	`DataAccessPtn`	The access pattern to test for.

metadata_for

const inline

inline SBIO_HD auto metadata_for(hd_std::size_t entry_no) const

Defined in src/lib/sbio/core/metadata.hh:600

Retrieve the group name and segment number for an entry index.

Returns

The group name, segment number pair for the provided index.

Parameters

Parameter	Type	Description
`entry_no`	`hd_std::size_t`	The entry index.

Public Static Attributes

Return	Name	Description
`constexpr hd_std::size_t`	`KeyCount` `static` `constexpr`

KeyCount

static constexpr

constexpr hd_std::size_t KeyCount { RequestSchema::count }

Defined in src/lib/sbio/core/metadata.hh:171

Public Types

Name	Description
`RequestSchema`
`DataRequest`
`DataAccessPtn`
`GroupKeys`
`FormatMetadata`

RequestSchema

using RequestSchema = typename FTraits::RequestSchema

Defined in src/lib/sbio/core/metadata.hh:168

DataRequest

using DataRequest = sbio::DataRequest< RequestSchema >

Type: sbio::DataRequest< RequestSchema >

Defined in src/lib/sbio/core/metadata.hh:169

DataAccessPtn

using DataAccessPtn = typename FTraits::DataAccessPtn

Defined in src/lib/sbio/core/metadata.hh:173

GroupKeys

using GroupKeys = typename FTraits::GroupKeys

Defined in src/lib/sbio/core/metadata.hh:175

FormatMetadata

using FormatMetadata = typename FTraits::FieldMetadata

Defined in src/lib/sbio/core/metadata.hh:180