StreamBroker

#include <sbio/core/broker.hh>

template<IOTraits IO, class EPolicy, FormatTraits< IO, EPolicy > FTraits, class Derived = void>
class StreamBroker

Defined in src/lib/sbio/core/broker.hh:117

A StreamBroker organizes Streams and mediates data requests.

The StreamBroker has a state machine and exposes data-format dependent metadata and data buffers.

Template Parameters

IO The type of the IO strategy being used.
EPolicy The Execution policy to use for reading data.
FTraits The data format to read.
Derived The sub-class type if defined. All functionality can be exposed purely from the FTraits template parameter, so sub-classing is not strictly necessary. I.e., this class can be instantiated directly.

List of all members

Name	Kind	Owner
`StreamBroker`	`function`	Declared here
`StreamBroker`	`function`	Declared here
`configure_broker`	`function`	Declared here
`allocate_storage`	`function`	Declared here
`open_data_stream`	`function`	Declared here
`discover_metadata`	`function`	Declared here
`prepare`	`function`	Declared here
`index_stream`	`function`	Declared here
`fetch_step`	`function`	Declared here
`fetch_steps`	`function`	Declared here
`process`	`function`	Declared here
`run`	`function`	Declared here
`step`	`function`	Declared here
`state`	`function`	Declared here
`stream_state`	`function`	Declared here
`capacity`	`function`	Declared here
`current_buffer`	`function`	Declared here
`get_data_in_buffer`	`function`	Declared here
`metadata`	`function`	Declared here
`metadata`	`function`	Declared here
`config`	`function`	Declared here
`config`	`function`	Declared here
`set_metadata`	`function`	Declared here
`stream`	`function`	Declared here
`stream`	`function`	Declared here
`sync_vars`	`function`	Declared here
`stream_idx`	`function`	Declared here
`StreamCount`	`variable`	Declared here
`m_streams`	`variable`	Declared here
`m_config`	`variable`	Declared here
`m_broker_state`	`variable`	Declared here
`m_stream_state`	`variable`	Declared here
`m_metadata_inv`	`variable`	Declared here
`m_storage`	`variable`	Declared here
`IOPolicy`	`typedef`	Declared here
`ExecutionPolicy`	`typedef`	Declared here
`DataFormat`	`typedef`	Declared here
`StreamType`	`typedef`	Declared here
`SBStorageType`	`typedef`	Declared here
`EPolicyConfig`	`typedef`	Declared here
`StreamConfig`	`typedef`	Declared here
`DSConfig`	`typedef`	Declared here
`StreamState`	`typedef`	Declared here
`StreamMetadata`	`typedef`	Declared here
`DataAccessPtn`	`typedef`	Declared here
`DataRequest`	`typedef`	Declared here
`StepIdxType`	`typedef`	Declared here

Public Methods

Return	Name	Description
	`StreamBroker` `inline`	A default constructor is provided for simplicity.
	`StreamBroker` `inline`	Construct the Broker with a specific, data-format-dependent, set of parameters.
`SBIO_HD void`	`configure_broker` `inline`	If a Broker has been default-constructed, it can be moved through the INIT state via this function.
`SBIO_HD void`	`allocate_storage` `inline`	Perform data format specific allocations in preparation for streaming data.
`SBIO_HDIOStatus`	`open_data_stream` `inline`	Connect and open brokered Streams.
`SBIO_HDIOStatus`	`discover_metadata` `inline`	After Stream connection, perform metadata discovery.
`SBIO_HDIOStatus`	`prepare` `inline`	Run through the initial allocate, connect, discover steps in one.
`SBIO_HDIOStatus`	`index_stream` `inline`	Index the data Stream if appropriate and implemented.
`SBIO_HDIOStatus`	`fetch_step` `inline`	Retrieve data for the specified index using the provided lookup pattern.
`SBIO_HDIOStatus`	`fetch_steps` `inline`
`SBIO_HDIOStatus`	`process` `inline`
`SBIO_HDIOStatus`	`run` `inline`
`SBIO_HDIOStatus`	`step` `inline`
`SBIO_HDBrokerState`	`state` `const` `inline`	Return the current broker state along the state machine.
`SBIO_HDStreamState`	`stream_state` `const` `inline`	Return the underlying StreamState of the brokered stream(s).
`SBIO_HD std::size_t`	`capacity` `const` `inline`	Return the current capacity for data formats that support indexing.
`SBIO_HD FTraits::DataUnit *`	`current_buffer` `inline`	Access the current data after a fetch.
`SBIO_HDDataResult`	`get_data_in_buffer` `inline`	Parse specific data from the buffer after a fetch.
`SBIO_HDStreamMetadata &`	`metadata` `inline`	The compiled metadata from the brokered Stream(s).
`SBIO_HD const StreamMetadata &`	`metadata` `const` `inline`	The compiled metadata from the brokered Stream(s).
`SBIO_HDStreamConfig &`	`config` `inline`	The set of StreamParameters configuration used to instantiate the broker.
`SBIO_HD const StreamConfig &`	`config` `const` `inline`	The set of StreamParameters configuration used to instantiate the broker.
`SBIO_HD void`	`set_metadata` `inline`	Configure the Broker with a set of metadata.
`SBIO_HDStreamType &`	`stream` `inline`
`SBIO_HD const StreamType &`	`stream` `const` `inline`
`SBIO_HD auto`	`sync_vars` `inline`
`SBIO_HD std::uint32_t`	`stream_idx` `const` `inline`

StreamBroker

inline

inline StreamBroker()

Defined in src/lib/sbio/core/broker.hh:198

A default constructor is provided for simplicity.

The Broker will remain in the INIT state until explicitly configured, if using this constructor.

StreamBroker

inline

inline StreamBroker(const StreamConfig & cfg)

Defined in src/lib/sbio/core/broker.hh:207

Construct the Broker with a specific, data-format-dependent, set of parameters.

Parameters

Parameter	Type	Description
`cfg`	`const [StreamConfig](#streamconfig-2) &`	The data-format-dependent configuration parameters.

configure_broker

inline

inline SBIO_HD void configure_broker(const StreamConfig & cfg)

Defined in src/lib/sbio/core/broker.hh:220

If a Broker has been default-constructed, it can be moved through the INIT state via this function.

This corresponds to the INIT stage of the state machine.

Parameters

Parameter	Type	Description
`cfg`	`const [StreamConfig](#streamconfig-2) &`	The data format specif configuration parameters.

allocate_storage

inline

inline SBIO_HD void allocate_storage()

Defined in src/lib/sbio/core/broker.hh:230

Perform data format specific allocations in preparation for streaming data.

This corresponds to the ALLOCATE stage of the state machine.

open_data_stream

inline

inline SBIO_HDIOStatus open_data_stream()

Defined in src/lib/sbio/core/broker.hh:257

Connect and open brokered Streams.

This corresponds to the CONNECT stage of the state machine.

Returns

The IOStatus result from connecting.

discover_metadata

inline

inline SBIO_HDIOStatus discover_metadata()

Defined in src/lib/sbio/core/broker.hh:276

After Stream connection, perform metadata discovery.

This function determines what contents are available from the brokered Streams. This information is then available to the caller (or higher-level abstractions).

This corresponds to the DISCOVERY stage of the state machine.

Returns

The IOStatus result for whether metadata reads were successful.

prepare

inline

inline SBIO_HDIOStatus prepare()

Defined in src/lib/sbio/core/broker.hh:306

Run through the initial allocate, connect, discover steps in one.

This is simply a convenience wrapper instead of calling each of the early state transitions independently. The down-side is the status will be cummulative so a failure may possibly not be immediately clear as originiating from one step or another.

Returns

The IOStatus result from all 3 steps.

index_stream

inline

inline SBIO_HDIOStatus index_stream()

Defined in src/lib/sbio/core/broker.hh:343

Index the data Stream if appropriate and implemented.

Indexing affords the possibility of ordering the data from (a) Stream(s) for indexable lookup. Not all data formats support a notion of indexing.

This corresponds to the INDEXING stage of the state machine.

Returns

An IOStatus for whether indexing was successful.

fetch_step

inline

inline SBIO_HDIOStatus fetch_step(StepIdxType step_idx, const DataAccessPtn ptn)

Defined in src/lib/sbio/core/broker.hh:387

Retrieve data for the specified index using the provided lookup pattern.

The index used for lookup is data format dependent. If the format supports indexing, then it may correspond to a chunk of data other than the most recent. If not, then this will always return the most recent data from the Stream.

Returns

An IOStatus for whether the lookup was succesful.

Parameters

Parameter	Type	Description
`step_idx`	`[StepIdxType](#stepidxtype-4)`	The data format-dependent data chunk index.
`ptn`	`const [DataAccessPtn](#dataaccessptn-5)`	A lookup pattern. Some data formats support access in various ways.

fetch_steps

inline

inline SBIO_HDIOStatus fetch_steps(std::initializer_list< StepIdxType > steps, const DataAccessPtn ptn)

Defined in src/lib/sbio/core/broker.hh:412

process

inline

inline SBIO_HDIOStatus process()

Defined in src/lib/sbio/core/broker.hh:460

run

inline

inline SBIO_HDIOStatus run()

Defined in src/lib/sbio/core/broker.hh:468

step

inline

template<class CBType> inline SBIO_HDIOStatus step(CBType && callback)

Defined in src/lib/sbio/core/broker.hh:478

state

const inline

inline SBIO_HDBrokerState state() const

Defined in src/lib/sbio/core/broker.hh:495

Return the current broker state along the state machine.

Returns

The current broker state along the state machine.

stream_state

const inline

inline SBIO_HDStreamState stream_state() const

Defined in src/lib/sbio/core/broker.hh:510

Return the underlying StreamState of the brokered stream(s).

The StreamState tracks data format-specific information about the streamed data. This may include information such as counters, whether certain transitions have been encountered, or whether the stream has been exhausted/will be soon. Refer to the specific FormatTraits for the format of interest for more information.

In cases where the broker manages multiple Streams, there is still one shared StreamState which encompasses all of them.

Returns

The current Stream(s) StreamState.

capacity

const inline

inline SBIO_HD std::size_t capacity() const

Defined in src/lib/sbio/core/broker.hh:522

Return the current capacity for data formats that support indexing.

When indexing is supported, the capacity indicates how many indices are immediately available to jump to in the stream. Reading beyond the capacity requires another INDEXING transition, or a move to reading in true streaming manner.

Returns

The current capacity: the number of immediately available indices.

current_buffer

inline

inline SBIO_HD FTraits::DataUnit * current_buffer()

Defined in src/lib/sbio/core/broker.hh:544

Access the current data after a fetch.

The action of performing a fetch pulls the data from the stream into available memory managed by the Broker (in conjunction with the Execution policy). The fetch APIs, however, do not return access to this memory directly. Instead, this function should be used to access data after a fetch if the raw buffer is required. Otherwise, more specific APIs can be used to parse specific components from the buffer.

Returns

A pointer to the buffer filled after a fetch from (a) Stream(s).

get_data_in_buffer

inline

inline SBIO_HDDataResult get_data_in_buffer(const DataRequest & req, const DataAccessPtn ptn, std::size_t batch_idx = 0)

Defined in src/lib/sbio/core/broker.hh:567

Parse specific data from the buffer after a fetch.

This function parses, in a data format-dependent manner, a portion of data from a buffer fetched from (a) Stream(s).

Returns

A data-format-dependent result object with the requested data.

Parameters

Parameter	Type	Description
`req`	`const [DataRequest](#datarequest-7) &`	A data-format-dependent struct with a parse request.
`ptn`	`const [DataAccessPtn](#dataaccessptn-5)`	A data-format-dependent access pattern for formats that can be read in different ways.
`batch_idx`	`std::size_t`	If reading by batches, the index for which of the steps in the batch to be resolved.

metadata

inline

inline SBIO_HDStreamMetadata & metadata()

Defined in src/lib/sbio/core/broker.hh:586

The compiled metadata from the brokered Stream(s).

This function should only be used after having passed the DISCOVERY state, or having set the metadata explicitly.

Returns

The compiled metadata from the brokered Stream(s).

metadata

const inline

inline SBIO_HD const StreamMetadata & metadata() const

Defined in src/lib/sbio/core/broker.hh:595

The compiled metadata from the brokered Stream(s).

This function should only be used after having passed the DISCOVERY state, or having set the metadata explicitly.

Returns

The compiled metadata from the brokered Stream(s).

config

inline

inline SBIO_HDStreamConfig & config()

Defined in src/lib/sbio/core/broker.hh:602

The set of StreamParameters configuration used to instantiate the broker.

Returns

The StreamParameters configuration.

config

const inline

inline SBIO_HD const StreamConfig & config() const

Defined in src/lib/sbio/core/broker.hh:608

The set of StreamParameters configuration used to instantiate the broker.

Returns

The StreamParameters configuration.

set_metadata

inline

inline SBIO_HD void set_metadata(StreamMetadata & metadata)

Defined in src/lib/sbio/core/broker.hh:624

Configure the Broker with a set of metadata.

Under some Execution policies, not all Brokers will necessarily move through all states of the state machine. In case the DISCOVERY state is not explicitly moved to, then the metadata for the Broker can be set using this function. For example, multiple Brokers may be reading different parts of the same Stream(s) in parallel.

The metadata is not only used by the caller, but also internally by the Broker, or at least, it may be used. For that reason, this function is made available.

stream

inline

inline SBIO_HDStreamType & stream(std::size_t role_idx)

Defined in src/lib/sbio/core/broker.hh:628

stream

const inline

inline SBIO_HD const StreamType & stream(std::size_t role_idx) const

Defined in src/lib/sbio/core/broker.hh:632

sync_vars

inline

inline SBIO_HD auto sync_vars()

Defined in src/lib/sbio/core/broker.hh:636

stream_idx

const inline

inline SBIO_HD std::uint32_t stream_idx() const

Defined in src/lib/sbio/core/broker.hh:649

Public Static Attributes

Return	Name	Description
`constexpr std::size_t`	`StreamCount` `static` `constexpr`

StreamCount

static constexpr

constexpr std::size_t StreamCount = FTraits::RoleCount

Defined in src/lib/sbio/core/broker.hh:190

Protected Attributes

Return	Name	Description
`StreamType`	`m_streams`
`StreamConfig`	`m_config`
`BrokerState`	`m_broker_state`
`StreamState`	`m_stream_state`
`StreamMetadata`	`m_metadata_inv`
`SBStorageType`	`m_storage`

m_streams

StreamType m_streams

Defined in src/lib/sbio/core/broker.hh:652

m_config

StreamConfig m_config

Defined in src/lib/sbio/core/broker.hh:653

m_broker_state

BrokerState m_broker_state

Defined in src/lib/sbio/core/broker.hh:654

m_stream_state

StreamState m_stream_state

Defined in src/lib/sbio/core/broker.hh:655

m_metadata_inv

StreamMetadata m_metadata_inv

Defined in src/lib/sbio/core/broker.hh:656

m_storage

SBStorageType m_storage

Defined in src/lib/sbio/core/broker.hh:657

Public Types

Name	Description
`IOPolicy`	The type of the IO strategy being used.
`ExecutionPolicy`	The Execution policy type.
`DataFormat`	The type of data being read.
`StreamType`	The type of Stream: I.e., the IO strategy and data format being read.
`SBStorageType`	The type of the StreamBroker’s Storage.
`EPolicyConfig`	The Execution policy configuration object type.
`StreamConfig`	The individual Stream configuration object type.
`DSConfig`	The DataSource configuration object type.
`StreamState`	The type of state tracking object for the data format’s Stream.
`StreamMetadata`	The type of the general metadata object for the data format’s Stream.
`DataAccessPtn`	The type of the enumerator used to specify access patterns used for the format.
`DataRequest`	The type of a request object used to query for data.
`StepIdxType`	The type used to request a specific step from the Stream.

IOPolicy

using IOPolicy = IO

Defined in src/lib/sbio/core/broker.hh:122

The type of the IO strategy being used.

ExecutionPolicy

using ExecutionPolicy = EPolicy

Defined in src/lib/sbio/core/broker.hh:126

The Execution policy type.

DataFormat

using DataFormat = FTraits

Defined in src/lib/sbio/core/broker.hh:130

The type of data being read.

StreamType

using StreamType = Stream< IO, FTraits >

Defined in src/lib/sbio/core/broker.hh:135

The type of Stream: I.e., the IO strategy and data format being read.

SBStorageType

using SBStorageType = Storage< typename FTraits::BrokerBufferRequirements, EPolicy >

Defined in src/lib/sbio/core/broker.hh:140

The type of the StreamBroker’s Storage.

EPolicyConfig

using EPolicyConfig = typename EPolicy::Config

Defined in src/lib/sbio/core/broker.hh:149

The Execution policy configuration object type.

epolicy_config objects configure the global behaviour of the Execution policy being used. The Execution policy must be configured before any Streams are opened as it controls all aspects of IO down to the allocation of Storage.

StreamConfig

using StreamConfig = typename FTraits::StreamParameters

Defined in src/lib/sbio/core/broker.hh:156

The individual Stream configuration object type.

stream_config objects are used to set up each individual Stream so that it can connect and read from its individual data.

DSConfig

using DSConfig = typename FTraits::DataSourceParameters

Defined in src/lib/sbio/core/broker.hh:163

The DataSource configuration object type.

ds_config objects are used for initial discovery and connection of the full set of Streams.

StreamState

using StreamState = typename FTraits::DiscoveryState

Defined in src/lib/sbio/core/broker.hh:168

The type of state tracking object for the data format’s Stream.

StreamMetadata

using StreamMetadata = typename FTraits::MetadataInventory

Defined in src/lib/sbio/core/broker.hh:172

The type of the general metadata object for the data format’s Stream.

DataAccessPtn

using DataAccessPtn = typename FTraits::DataAccessPtn

Defined in src/lib/sbio/core/broker.hh:177

The type of the enumerator used to specify access patterns used for the format.

DataRequest

using DataRequest = typename FTraits::DataRequest

Defined in src/lib/sbio/core/broker.hh:181

The type of a request object used to query for data.

StepIdxType

using StepIdxType = typename FTraits::StepIdxType

Defined in src/lib/sbio/core/broker.hh:188

The type used to request a specific step from the Stream.

This type is required and guaranteed to be convertible std::size_t; however, different data format’s may use different underlying types.