BrokerGroup

#include <sbio/core/broker_group.hh>

template<IsStreamBroker BrokerType, FormatTraits< typename BrokerType::IOPolicy, typename BrokerType::ExecutionPolicy > FTraits, std::size_t MaxSegments = 128>
class BrokerGroup

Defined in src/lib/sbio/core/broker_group.hh:66

The logical grouping of StreamBrokers which should be queried together.

While the StreamBroker represents the smallest logical unit that makes sense to query data for, the BrokerGroup represents the combination of various StreamBrokers. Frequently, it makes more sense at higher levels of abstraction to consider a group of StreamBrokers together as a single unit. For example, each StreamBroker may be managing only a portion of a larger data construct (a part of an image, e.g.). The BrokerGroup then provides a single interface to query all logically related StreamBrokers simulatneously. Note that a StreamBroker may belong to multiple BrokerGroups.

Template Parameters

BrokerType The type of the underlying StreamBroker in the group.
FTraits The data format being read.
MaxSegments The maximum number of members in the group.

List of all members

Name	Kind	Owner
`BrokerGroup`	`function`	Declared here
`BrokerGroup`	`function`	Declared here
`BrokerGroup`	`function`	Declared here
`group_name`	`function`	Declared here
`group_type`	`function`	Declared here
`num_segments`	`function`	Declared here
`segments`	`function`	Declared here
`segment`	`function`	Declared here
`stream_brokers`	`function`	Declared here
`num_stream_brokers`	`function`	Declared here
`fetch_next_for`	`function`	Declared here
`fetch_steps_for`	`function`	Declared here
`get_data_for`	`function`	Declared here
`get_data_for`	`function`	Declared here
`get_data`	`function`	Declared here
`get_data`	`function`	Declared here
`get_multi_data`	`function`	Declared here
`get_multi_data`	`function`	Declared here
`IOPolicy`	`typedef`	Declared here
`ExecutionPolicy`	`typedef`	Declared here
`DataFormat`	`typedef`	Declared here
`StreamType`	`typedef`	Declared here
`SBStorageType`	`typedef`	Declared here
`EPolicyConfig`	`typedef`	Declared here
`StreamConfig`	`typedef`	Declared here
`DSConfig`	`typedef`	Declared here
`StreamState`	`typedef`	Declared here
`StreamMetadata`	`typedef`	Declared here
`DataAccessPtn`	`typedef`	Declared here
`DataRequest`	`typedef`	Declared here
`StepIdxType`	`typedef`	Declared here
`PtrTableRequirements`	`typedef`	Declared here
`PtrStorageType`	`typedef`	Declared here
`MemTag`	`typedef`	Declared here
`DataSegmentRef`	`typedef`	Declared here
`m_ptrs`	`variable`	Declared here
`m_topology`	`variable`	Declared here
`m_ptr_storage`	`variable`	Declared here

Public Methods

Return	Name	Description
	`BrokerGroup`	Defaulted constructor.
	`BrokerGroup` `inline`
	`BrokerGroup` `inline`
`const char *`	`group_name` `const` `inline`
`const char *`	`group_type` `const` `inline`
`std::size_t`	`num_segments` `const` `inline`
`const DataSegmentRef *`	`segments` `const` `inline`
`const DataSegmentRef &`	`segment` `const` `inline`
`BrokerType **`	`stream_brokers` `inline`
`std::size_t`	`num_stream_brokers` `const` `inline`
`IOStatus`	`fetch_next_for` `const` `inline`
`IOStatus`	`fetch_steps_for` `const` `inline`
`DataResult`	`get_data_for` `const` `inline`
`DataResult`	`get_data_for` `const` `inline`
`ncarray::SOViewFor< MemTag >`	`get_data` `const` `inline` `requires std::invocable<CBType, DataResult>`	Fetch data for a given index AND read into the fetched data to pull specific data out.
`ncarray::SOViewFor< MemTag >`	`get_data` `const` `inline`	Fetch data for a given index AND read into the fetched data to pull specific data out.
`ncarray::SOViewFor< MemTag >`	`get_multi_data` `const` `inline` `requires std::invocable<CBType, DataResult>`
`ncarray::SOViewFor< MemTag >`	`get_multi_data` `const` `inline`

BrokerGroup

BrokerGroup() = default

Defined in src/lib/sbio/core/broker_group.hh:162

Defaulted constructor.

BrokerGroup

inline

inline BrokerGroup(const char * name, const char * type, std::size_t num_segments, DataSegmentRef * segments)

Defined in src/lib/sbio/core/broker_group.hh:164

BrokerGroup

inline

inline BrokerGroup(const GroupTopology< BrokerType, MaxSegments > & topo)

Defined in src/lib/sbio/core/broker_group.hh:241

group_name

const inline

inline const char * group_name() const

Defined in src/lib/sbio/core/broker_group.hh:255

group_type

const inline

inline const char * group_type() const

Defined in src/lib/sbio/core/broker_group.hh:256

num_segments

const inline

inline std::size_t num_segments() const

Defined in src/lib/sbio/core/broker_group.hh:258

segments

const inline

inline const DataSegmentRef * segments() const

Defined in src/lib/sbio/core/broker_group.hh:259

segment

const inline

inline const DataSegmentRef & segment(std::size_t i) const

Defined in src/lib/sbio/core/broker_group.hh:266

stream_brokers

inline

inline BrokerType ** stream_brokers()

Defined in src/lib/sbio/core/broker_group.hh:270

num_stream_brokers

const inline

inline std::size_t num_stream_brokers() const

Defined in src/lib/sbio/core/broker_group.hh:277

fetch_next_for

const inline

inline IOStatus fetch_next_for(StepIdxType & step_idx, std::size_t broker_no) const

Defined in src/lib/sbio/core/broker_group.hh:279

fetch_steps_for

const inline

inline IOStatus fetch_steps_for(std::initializer_list< StepIdxType > steps, std::size_t broker_no) const

Defined in src/lib/sbio/core/broker_group.hh:286

get_data_for

const inline

template<class CBType> inline DataResult get_data_for(DataRequest & req, std::size_t segment_no, CBType && callback, std::size_t batch_idx = 0) const

Defined in src/lib/sbio/core/broker_group.hh:295

get_data_for

const inline

inline DataResult get_data_for(DataRequest & req, std::size_t segment_no, std::size_t batch_idx = 0) const

Defined in src/lib/sbio/core/broker_group.hh:314

get_data

const inline requires std::invocable<CBType, DataResult>

template<class CBType, typename... Args> inline ncarray::SOViewFor< MemTag > get_data(StepIdxType & step_idx, CBType && callback, Args &&... args) const requires std::invocable<CBType, DataResult>

Defined in src/lib/sbio/core/broker_group.hh:349

Fetch data for a given index AND read into the fetched data to pull specific data out.

NOTE: We add the concept restricting the CBType to invocable to avoid overload ambiguities. This is likely not needed, as the partial ordering will treat the Args pack as less-specialized, and prefer this overload. Adding it, though, does not hurt and future-proofs against future overloads.

Returns

array The requested data as an NCArrayView or NCDevArrayView depending on whether MemTag is HostTag or DevTag, respectively.

Parameters

Parameter	Type	Description
`step_idx`	`[StepIdxType](#stepidxtype) &`	The index for the data to fetch.
`callback`	`CBType &&`	Optionally provide a callback to be run on every segment independently before the data is combined.
`args`	`Args &&...`	The arguments for the DataRequest constructor.

get_data

const inline

template<typename... Args> inline ncarray::SOViewFor< MemTag > get_data(StepIdxType & step_idx, Args &&... args) const

Defined in src/lib/sbio/core/broker_group.hh:456

Fetch data for a given index AND read into the fetched data to pull specific data out.

This is an identical get_data call to the other overload, without the application of a lambda callback.

Returns

array The requested data as an NCArrayView or NCDevArrayView depending on whether MemTag is HostTag or DevTag, respectively.

Parameters

Parameter	Type	Description
`step_idx`	`[StepIdxType](#stepidxtype) &`	The index for the data to fetch.
`args`	`Args &&...`	The arguments for the DataRequest constructor.

get_multi_data

const inline requires std::invocable<CBType, DataResult>

template<typename MemTag = ncarray::HostTag, class CBType, typename... Args> inline ncarray::SOViewFor< MemTag > get_multi_data(const std::initializer_list< StepIdxType > & steps, CBType && callback, Args &&... args) const requires std::invocable<CBType, DataResult>

Defined in src/lib/sbio/core/broker_group.hh:542

get_multi_data

const inline

template<typename MemTag = ncarray::HostTag, typename... Args> inline ncarray::SOViewFor< MemTag > get_multi_data(const std::initializer_list< StepIdxType > & steps, Args &&... args) const

Defined in src/lib/sbio/core/broker_group.hh:629

Public Types

Name	Description
`IOPolicy`	The type of the IO strategy being used.
`ExecutionPolicy`	The Execution policy type.
`DataFormat`	The type of data being read.
`StreamType`	The type of Stream: I.e., the IO strategy and data format being read.
`SBStorageType`	The type of the StreamBroker’s Storage.
`EPolicyConfig`	The Execution policy configuration object type.
`StreamConfig`	The individual Stream configuration object type.
`DSConfig`	The DataSource configuration object type.
`StreamState`	The type of state tracking object for the data format’s Stream.
`StreamMetadata`	The type of the general metadata object for the data format’s Stream.
`DataAccessPtn`	The type of the enumerator used to specify access patterns used for the format.
`DataRequest`	The type of a request object used to query for data.
`StepIdxType`	The type used to request a specific step from the Stream.
`PtrTableRequirements`
`PtrStorageType`
`MemTag`	Based on choice of ExecutionPolicy, data will return in host or device buffers.
`DataSegmentRef`

IOPolicy

using IOPolicy = typename BrokerType::IOPolicy

Defined in src/lib/sbio/core/broker_group.hh:71

The type of the IO strategy being used.

ExecutionPolicy

using ExecutionPolicy = typename BrokerType::ExecutionPolicy

Defined in src/lib/sbio/core/broker_group.hh:75

The Execution policy type.

DataFormat

using DataFormat = FTraits

Defined in src/lib/sbio/core/broker_group.hh:79

The type of data being read.

StreamType

using StreamType = Stream< IOPolicy, FTraits >

Defined in src/lib/sbio/core/broker_group.hh:84

The type of Stream: I.e., the IO strategy and data format being read.

SBStorageType

using SBStorageType = Storage< typename FTraits::BrokerBufferRequirements, ExecutionPolicy >

Defined in src/lib/sbio/core/broker_group.hh:89

The type of the StreamBroker’s Storage.

EPolicyConfig

using EPolicyConfig = typename ExecutionPolicy::Config

Defined in src/lib/sbio/core/broker_group.hh:101

The Execution policy configuration object type.

epolicy_config objects configure the global behaviour of the Execution policy being used. The Execution policy must be configured before any Streams are opened as it controls all aspects of IO down to the allocation of Storage.

StreamConfig

using StreamConfig = typename FTraits::StreamParameters

Defined in src/lib/sbio/core/broker_group.hh:108

The individual Stream configuration object type.

stream_config objects are used to set up each individual Stream so that it can connect and read from its individual data.

DSConfig

using DSConfig = typename FTraits::DataSourceParameters

Defined in src/lib/sbio/core/broker_group.hh:115

The DataSource configuration object type.

ds_config objects are used for initial discovery and connection of the full set of Streams.

StreamState

using StreamState = typename FTraits::DiscoveryState

Defined in src/lib/sbio/core/broker_group.hh:120

The type of state tracking object for the data format’s Stream.

StreamMetadata

using StreamMetadata = typename FTraits::MetadataInventory

Defined in src/lib/sbio/core/broker_group.hh:124

The type of the general metadata object for the data format’s Stream.

DataAccessPtn

using DataAccessPtn = typename FTraits::DataAccessPtn

Defined in src/lib/sbio/core/broker_group.hh:129

The type of the enumerator used to specify access patterns used for the format.

DataRequest

using DataRequest = typename FTraits::DataRequest

Defined in src/lib/sbio/core/broker_group.hh:133

The type of a request object used to query for data.

StepIdxType

using StepIdxType = typename FTraits::StepIdxType

Defined in src/lib/sbio/core/broker_group.hh:140

The type used to request a specific step from the Stream.

This type is required and guaranteed to be convertible std::size_t; however, different data format’s may use different underlying types.

PtrTableRequirements

using PtrTableRequirements = TypeList< BufferDescriptor< TableRole, 0, sizeof(void *)> >

Defined in src/lib/sbio/core/broker_group.hh:142

PtrStorageType

using PtrStorageType = Storage< PtrTableRequirements, ExecutionPolicy >

Defined in src/lib/sbio/core/broker_group.hh:144

MemTag

using MemTag = std::conditional_t< ExecutionPolicy::result_memory_space()==MemorySpace::Host, ncarray::HostTag, ncarray::DevTag >

Defined in src/lib/sbio/core/broker_group.hh:153

Based on choice of ExecutionPolicy, data will return in host or device buffers.

As data is returned via ncarray array objects, the MemTag converts a MemorySpace indicator in sbio to the tagging system used for host/device array constructs in ncarray.

DataSegmentRef

using DataSegmentRef = SegmentRef< BrokerType >

Defined in src/lib/sbio/core/broker_group.hh:159

Private Attributes

Return	Name	Description
`const void *`	`m_ptrs`
`GroupTopology< BrokerType, MaxSegments >`	`m_topology`
`PtrStorageType`	`m_ptr_storage`

m_ptrs

const void * m_ptrs

Defined in src/lib/sbio/core/broker_group.hh:711

m_topology

GroupTopology< BrokerType, MaxSegments > m_topology

Defined in src/lib/sbio/core/broker_group.hh:713

m_ptr_storage

PtrStorageType m_ptr_storage

Defined in src/lib/sbio/core/broker_group.hh:715