DataSource

#include <sbio/core/datasource.hh>

template<IOTraits IO, class EPolicy, FormatTraits< IO, EPolicy > FTraits, IsStreamBroker BrokerType = StreamBroker<IO, EPolicy, FTraits>, std::size_t MaxDataStreams = 128>
class DataSource

Defined in src/lib/sbio/core/datasource.hh:79

The highest-level abstraction for defining the set of StreamBrokers that will be used.

The DataSource is used to find and collect together the group of all StreamBrokers that will be relevant for the IO of a specific dataset. The DataSource can be used to spawn BrokerGroups to organize the StreamBrokers into logical groupings, and can be used to generate indices to distribute among said groups to order the way in which data is fetched and queried.

Template Parameters

IO The type of the IO strategy being used.
EPolicy The Execution policy to use for reading data.
FTraits The data format to read.
BrokerType The type of the StreamBroker being used. In general, the StreamBroker does not need to be subclassed, and can be instantiated directly. In that case, this template parameter can be left on its default value. Only if using a StreamBroker subclass does it need to be changed.

List of all members

Name	Kind	Owner
`DataSource`	`function`	Declared here
`DataSource`	`function`	Declared here
`configure_execution_policy`	`function`	Declared here
`add_data_stream`	`function`	Declared here
`load_run`	`function`	Declared here
`discover_metadata`	`function`	Declared here
`next`	`function`	Declared here
`get_stream_group`	`function`	Declared here
`num_data_streams`	`function`	Declared here
`data_streams`	`function`	Declared here
`data_streams`	`function`	Declared here
`data_stream`	`function`	Declared here
`data_stream`	`function`	Declared here
`begin`	`function`	Declared here
`end`	`function`	Declared here
`begin`	`function`	Declared here
`end`	`function`	Declared here
`IOPolicy`	`typedef`	Declared here
`ExecutionPolicy`	`typedef`	Declared here
`DataFormat`	`typedef`	Declared here
`StreamType`	`typedef`	Declared here
`SBStorageType`	`typedef`	Declared here
`EPolicyConfig`	`typedef`	Declared here
`StreamConfig`	`typedef`	Declared here
`DSConfig`	`typedef`	Declared here
`StreamState`	`typedef`	Declared here
`StreamMetadata`	`typedef`	Declared here
`DataAccessPtn`	`typedef`	Declared here
`DataRequest`	`typedef`	Declared here
`StepIdxType`	`typedef`	Declared here
`Iterator`	`typedef`	Declared here
`ConstIterator`	`typedef`	Declared here
`m_data_streams`	`variable`	Declared here
`m_num_data_streams`	`variable`	Declared here
`m_steps_capacity`	`variable`	Declared here
`m_epolicy_configured`	`variable`	Declared here

Public Methods

Return	Name	Description
	`DataSource`	Defaulted constructor.
	`DataSource` `inline` `explicit`	Construct a DataSource and initialize the Execution policy.
`void`	`configure_execution_policy` `inline`	Configure the Execution policy.
`SBIO_HD bool`	`add_data_stream` `inline`	Add a new StreamBroker to the set tracked by the DataSource.
`bool`	`load_run` `inline`	load_run: Scans the standard hutch directory and automatically registers all streams for a given experiment and run number.
`SBIO_HDIOStatus`	`discover_metadata` `inline`	After finding StreamBrokers and opening connections, determine available metadata.
`SBIO_HD FTraits::StepIdxType`	`next` `const` `inline`	Request the next index for a step to read data for.
`SBIO_HDBrokerGroup< BrokerType, FTraits, MaxSegments >`	`get_stream_group` `inline`	Construct a BrokerGroup by name from the set of StreamBrokers.
`SBIO_HD std::size_t`	`num_data_streams` `const` `inline`	The number of data Streams in the DataSource’s set.
`SBIO_HD const BrokerType *`	`data_streams` `const` `inline`	A pointer to the first StreamBroker of the set.
`SBIO_HD BrokerType *`	`data_streams` `inline`	A pointer to the first StreamBroker of the set.
`SBIO_HD const BrokerType &`	`data_stream` `const` `inline`	The requested StreamBroker.
`SBIO_HD BrokerType &`	`data_stream` `inline`	The requested StreamBroker.
`Iterator`	`begin` `inline`	Return an iterator at the first step index.
`Iterator`	`end` `inline`	Return an iterator pointing to the ExhasutedSentinel.
`ConstIterator`	`begin` `const` `inline`
`ConstIterator`	`end` `const` `inline`

DataSource

DataSource() = default

Defined in src/lib/sbio/core/datasource.hh:152

Defaulted constructor.

DataSource

inline explicit

inline explicit DataSource(const EPolicyConfig & ecfg)

Defined in src/lib/sbio/core/datasource.hh:159

Construct a DataSource and initialize the Execution policy.

Parameters

Parameter	Type	Description
`ecfg`	`const [EPolicyConfig](#epolicyconfig-1) &`	The configuration for the Execution policy.

configure_execution_policy

inline

inline void configure_execution_policy(const EPolicyConfig & ecfg)

Defined in src/lib/sbio/core/datasource.hh:172

Configure the Execution policy.

The Execution policy must be configured before Streams are opened and read from as the policy controls the allocation of the buffers that will be read into.

Parameters

Parameter	Type	Description
`ecfg`	`const [EPolicyConfig](#epolicyconfig-1) &`	The configuration for the Execution policy.

add_data_stream

inline

inline SBIO_HD bool add_data_stream(const StreamConfig & cfg)

Defined in src/lib/sbio/core/datasource.hh:187

Add a new StreamBroker to the set tracked by the DataSource.

Returns

Whether the incorporation of the StreamBroker, and subsequent configuration was successful.

Parameters

Parameter	Type	Description
`cfg`	`const [StreamConfig](#streamconfig-1) &`	The base configuration for the StreamBroker’s Stream(s). This configuration may be modified slightly for the parameters that are required by the Stream, but do not make sense to be set directly by the user.

load_run

inline

template<typename... Args> inline bool load_run(StreamConfig base_cfg, Args &&... args)

Defined in src/lib/sbio/core/datasource.hh:203

load_run: Scans the standard hutch directory and automatically registers all streams for a given experiment and run number.

discover_metadata

inline

inline SBIO_HDIOStatus discover_metadata()

Defined in src/lib/sbio/core/datasource.hh:218

After finding StreamBrokers and opening connections, determine available metadata.

The metadata discovery stage is crucial to be able to organize StreamBrokers into their logical BrokerGroups. The metadata informs the rest of the infrastructure what kind of data is available in the set of Streams beign read from.

Returns

The IOStatus result from performing metadata discovery.

const inline

inline SBIO_HD FTraits::StepIdxType next() const

Defined in src/lib/sbio/core/datasource.hh:264

Request the next index for a step to read data for.

The step indices are used to request data for a specific step/event. In general, this corresponds to chronological ordering (i.e. a step is mappable to a time point); however, this is by no means a requirement.

Returns

The next step index to fetch and query data for, or the ExhaustedSentinel if no more data is available.

get_stream_group

inline

template<std::size_t MaxSegments = 128> inline SBIO_HDBrokerGroup< BrokerType, FTraits, MaxSegments > get_stream_group(const char * name)

Defined in src/lib/sbio/core/datasource.hh:318

Construct a BrokerGroup by name from the set of StreamBrokers.

Returns

The BrokerGroup of the given name. The success of the group formation can be checked by inspecting the number of segments the returned group has (0 indicating no group could be formed).

num_data_streams

const inline

inline SBIO_HD std::size_t num_data_streams() const

Defined in src/lib/sbio/core/datasource.hh:330

The number of data Streams in the DataSource’s set.

data_streams

const inline

inline SBIO_HD const BrokerType * data_streams() const

Defined in src/lib/sbio/core/datasource.hh:337

A pointer to the first StreamBroker of the set.

Returns

A pointer to the first StreamBroker of the set.

data_streams

inline

inline SBIO_HD BrokerType * data_streams()

Defined in src/lib/sbio/core/datasource.hh:343

A pointer to the first StreamBroker of the set.

Returns

A pointer to the first StreamBroker of the set.

data_stream

const inline

inline SBIO_HD const BrokerType & data_stream(std::size_t i) const

Defined in src/lib/sbio/core/datasource.hh:353

The requested StreamBroker.

Returns

The requested StreamBroker.

Parameters

Parameter	Type	Description
`i`	`std::size_t`	The StreamBroker to return.

data_stream

inline

inline SBIO_HD BrokerType & data_stream(std::size_t i)

Defined in src/lib/sbio/core/datasource.hh:363

The requested StreamBroker.

Returns

The requested StreamBroker.

Parameters

Parameter	Type	Description
`i`	`std::size_t`	The StreamBroker to return.

begin

inline

inline Iterator begin()

Defined in src/lib/sbio/core/datasource.hh:432

Return an iterator at the first step index.

Returns

An iterator at the first step index.

end

inline

inline Iterator end()

Defined in src/lib/sbio/core/datasource.hh:438

Return an iterator pointing to the ExhasutedSentinel.

Returns

An iterator pointing to the ExhaustedSentinel.

begin

const inline

inline ConstIterator begin() const

Defined in src/lib/sbio/core/datasource.hh:440

end

const inline

inline ConstIterator end() const

Defined in src/lib/sbio/core/datasource.hh:441

Public Types

Name	Description
`IOPolicy`	The type of the IO strategy being used.
`ExecutionPolicy`	The Execution policy type.
`DataFormat`	The type of data being read.
`StreamType`	The type of Stream: I.e., the IO strategy and data format being read.
`SBStorageType`	The type of the StreamBroker’s Storage.
`EPolicyConfig`	The Execution policy configuration object type.
`StreamConfig`	The individual Stream configuration object type.
`DSConfig`	The DataSource configuration object type.
`StreamState`	The type of state tracking object for the data format’s Stream.
`StreamMetadata`	The type of the general metadata object for the data format’s Stream.
`DataAccessPtn`	The type of the enumerator used to specify access patterns used for the format.
`DataRequest`	The type of a request object used to query for data.
`StepIdxType`	The type used to request a specific step from the Stream.
`Iterator`
`ConstIterator`

IOPolicy

using IOPolicy = IO

Defined in src/lib/sbio/core/datasource.hh:84

The type of the IO strategy being used.

ExecutionPolicy

using ExecutionPolicy = EPolicy

Defined in src/lib/sbio/core/datasource.hh:88

The Execution policy type.

DataFormat

using DataFormat = FTraits

Defined in src/lib/sbio/core/datasource.hh:92

The type of data being read.

StreamType

using StreamType = Stream< IO, FTraits >

Defined in src/lib/sbio/core/datasource.hh:97

The type of Stream: I.e., the IO strategy and data format being read.

SBStorageType

using SBStorageType = Storage< typename FTraits::BrokerBufferRequirements, EPolicy >

Defined in src/lib/sbio/core/datasource.hh:102

The type of the StreamBroker’s Storage.

EPolicyConfig

using EPolicyConfig = typename EPolicy::Config

Defined in src/lib/sbio/core/datasource.hh:111

The Execution policy configuration object type.

epolicy_config objects configure the global behaviour of the Execution policy being used. The Execution policy must be configured before any Streams are opened as it controls all aspects of IO down to the allocation of Storage.

StreamConfig

using StreamConfig = typename FTraits::StreamParameters

Defined in src/lib/sbio/core/datasource.hh:118

The individual Stream configuration object type.

stream_config objects are used to set up each individual Stream so that it can connect and read from its individual data.

DSConfig

using DSConfig = typename FTraits::DataSourceParameters

Defined in src/lib/sbio/core/datasource.hh:125

The DataSource configuration object type.

ds_config objects are used for initial discovery and connection of the full set of Streams.

StreamState

using StreamState = typename FTraits::DiscoveryState

Defined in src/lib/sbio/core/datasource.hh:130

The type of state tracking object for the data format’s Stream.

StreamMetadata

using StreamMetadata = typename FTraits::MetadataInventory

Defined in src/lib/sbio/core/datasource.hh:134

The type of the general metadata object for the data format’s Stream.

DataAccessPtn

using DataAccessPtn = typename FTraits::DataAccessPtn

Defined in src/lib/sbio/core/datasource.hh:139

The type of the enumerator used to specify access patterns used for the format.

DataRequest

using DataRequest = typename FTraits::DataRequest

Defined in src/lib/sbio/core/datasource.hh:143

The type of a request object used to query for data.

StepIdxType

using StepIdxType = typename FTraits::StepIdxType

Defined in src/lib/sbio/core/datasource.hh:150

The type used to request a specific step from the Stream.

This type is required and guaranteed to be convertible std::size_t; however, different data format’s may use different underlying types.

Iterator

using Iterator = IteratorImpl< DataSource >

Defined in src/lib/sbio/core/datasource.hh:424

ConstIterator

using ConstIterator = IteratorImpl< const DataSource >

Defined in src/lib/sbio/core/datasource.hh:425

Private Attributes

Return	Name	Description
`BrokerType`	`m_data_streams`	The set of StreamBrokers in the DataSource
`std::size_t`	`m_num_data_streams`	The total number of StreamBrokers in the DataSource
`std::size_t`	`m_steps_capacity`	The current steps capacity before reindexing is required.
`bool`	`m_epolicy_configured`	Flag to track whether the Execution policy has been configured.

m_data_streams

BrokerType m_data_streams

Defined in src/lib/sbio/core/datasource.hh:447

The set of StreamBrokers in the DataSource

m_num_data_streams

std::size_t m_num_data_streams { 0 }

Defined in src/lib/sbio/core/datasource.hh:451

The total number of StreamBrokers in the DataSource

m_steps_capacity

std::size_t m_steps_capacity { 0 }

Defined in src/lib/sbio/core/datasource.hh:455

The current steps capacity before reindexing is required.

m_epolicy_configured

bool m_epolicy_configured { false }

Defined in src/lib/sbio/core/datasource.hh:465

Flag to track whether the Execution policy has been configured.

Since there may be global state in the policy, ensure that a configuration gets called. If its explicitly provided, then that gets used. Otherwise, a default configuration will be invoked before any IO. This allows any global state to be reset if iteratively creating multiple DataSource instantiations over time.