DataSource

DataSource

#include <sbio/core/datasource.hh>

template<IOTraits IO, class EPolicy, FormatTraits< IO, EPolicy > FTraits, IsStreamBroker BrokerType = StreamBroker<IO, EPolicy, FTraits>, std::size_t MaxDataStreams = 128>
class DataSource

Defined in src/lib/sbio/core/datasource.hh:90

The highest-level abstraction for defining the set of StreamBrokers that will be used.

The DataSource is used to find and collect together the group of all StreamBrokers that will be relevant for the IO of a specific dataset. The DataSource can be used to spawn BrokerGroups to organize the StreamBrokers into logical groupings, and can be used to generate indices to distribute among said groups to order the way in which data is fetched and queried.

Template Parameters

• IO The type of the IO strategy being used.

• EPolicy The Execution policy to use for reading data.

• FTraits The data format to read.

• BrokerType The type of the StreamBroker being used. In general, the StreamBroker does not need to be subclassed, and can be instantiated directly. In that case, this template parameter can be left on its default value. Only if using a StreamBroker subclass does it need to be changed.

Friends

Name	Description
SourceSet	A SourceSet drives the reindexing of the DataSources it iterates.

---

SourceSet

template<class... Sources> friend class SourceSet

Defined in src/lib/sbio/core/datasource.hh:157

A SourceSet drives the reindexing of the DataSources it iterates.

Reindexing is private - it was easiest to let the SourceSet be a friend.

Public Methods

Return	Name	Description
	DataSource	Defaulted constructor.
	DataSource inline explicit	Construct a DataSource, with the execution policy already configured.
	DataSource inline explicit	Construct a DataSource and initialize the Execution policy.
void	configure_execution_policy inline	Configure the Execution policy.
SBIO_HD bool	add_data_stream inline	Add a new StreamBroker to the set tracked by the DataSource.
bool	load_source inline
bool	load_source_with inline
SBIO_HDIOStatus	discover_metadata inline	After finding StreamBrokers and opening connections, determine available metadata.
SBIO_HD FTraits::StepIdxType	next const inline	Request the next index for a step to read data for.
SBIO_HDStepBatch< typename FTraits::StepIdxType >	next_batch const inline	Request the next contiguous batch of indices for steps to read data for.
SBIO_HDBrokerGroup< BrokerType, FTraits, MaxSegments >	get_stream_group inline	Construct a BrokerGroup by name from the set of StreamBrokers.
SBIO_HD std::size_t	num_data_streams const inline	The number of data Streams in the DataSource’s set.
SBIO_HD const BrokerType *	data_streams const inline	A pointer to the first StreamBroker of the set.
SBIO_HD BrokerType *	data_streams inline	A pointer to the first StreamBroker of the set.
SBIO_HD const BrokerType &	data_stream const inline	The requested StreamBroker.
SBIO_HD BrokerType &	data_stream inline	The requested StreamBroker.
SBIO_HDStepRange	steps inline	Return a range over the steps of this DataSource for the calling unit.
SBIO_HDConstStepRange	steps const inline
SBIO_HDConstStepRange	csteps const inline
SBIO_HDIterator	begin inline	Return an iterator at the first step index.
SBIO_HDConstIterator	begin const inline
SBIO_HDIterator	end inline	Return an iterator pointing to the ExhasutedSentinel.
SBIO_HDConstIterator	end const inline
SBIO_HDConstIterator	cbegin const inline
SBIO_HDConstIterator	cend const inline
SBIO_HDBatchRange	batches inline	Return a range over contiguous batches of steps for the calling unit.
SBIO_HDConstBatchRange	batches const inline
SBIO_HDConstBatchRange	cbatches const inline

---

DataSource

DataSource() = default

Defined in src/lib/sbio/core/datasource.hh:159

Defaulted constructor.

---

DataSource

inline explicit

inline explicit DataSource(ExecutionPolicyConfigured)

Defined in src/lib/sbio/core/datasource.hh:167

Construct a DataSource, with the execution policy already configured.

Note

This constructor is used by the [Context](api-sbio-Context.md#context), which manages various configuration steps itself.

---

DataSource

inline explicit

inline explicit DataSource(const EPolicyConfig & ecfg)

Defined in src/lib/sbio/core/datasource.hh:176

Construct a DataSource and initialize the Execution policy.

Parameters

Parameter	Type	Description
ecfg	const EPolicyConfig &	The configuration for the Execution policy.

---

configure_execution_policy

inline

inline void configure_execution_policy(const EPolicyConfig & ecfg)

Defined in src/lib/sbio/core/datasource.hh:189

Configure the Execution policy.

The Execution policy must be configured before Streams are opened and read from as the policy controls the allocation of the buffers that will be read into.

Parameters

Parameter	Type	Description
ecfg	const EPolicyConfig &	The configuration for the Execution policy.

---

add_data_stream

inline

inline SBIO_HD bool add_data_stream(const GenericStreamConfig< DataFormat > & cfg)

Defined in src/lib/sbio/core/datasource.hh:204

Add a new StreamBroker to the set tracked by the DataSource.

Returns

Whether the incorporation of the StreamBroker, and subsequent configuration was successful.

Parameters

Parameter	Type	Description
cfg	const GenericStreamConfig< DataFormat > &	The base configuration for the StreamBroker’s Stream(s). This configuration may be modified slightly for the parameters that are required by the Stream, but do not make sense to be set directly by the user.

---

load_source

inline

template<typename... Args> inline bool load_source(const StreamConfig & cfg, Args &&... args)

Defined in src/lib/sbio/core/datasource.hh:216

---

load_source_with

inline

template<IsLocator< DataSource > Locator, typename... Args> inline bool load_source_with(const StreamConfig & cfg, Args &&... args)

Defined in src/lib/sbio/core/datasource.hh:225

---

discover_metadata

inline

inline SBIO_HDIOStatus discover_metadata()

Defined in src/lib/sbio/core/datasource.hh:241

After finding StreamBrokers and opening connections, determine available metadata.

The metadata discovery stage is crucial to be able to organize StreamBrokers into their logical BrokerGroups. The metadata informs the rest of the infrastructure what kind of data is available in the set of Streams beign read from.

Returns

The IOStatus result from performing metadata discovery.

---

next

const inline

inline SBIO_HD FTraits::StepIdxType next() const

Defined in src/lib/sbio/core/datasource.hh:287

Request the next index for a step to read data for.

The step indices are used to request data for a specific step/event. In general, this corresponds to chronological ordering (i.e. a step is mappable to a time point); however, this is by no means a requirement.

Returns

The next step index to fetch and query data for, or the ExhaustedSentinel if no more data is available.

---

next_batch

const inline

inline SBIO_HDStepBatch< typename FTraits::StepIdxType > next_batch(std::size_t batch_size = 0) const

Defined in src/lib/sbio/core/datasource.hh:311

Request the next contiguous batch of indices for steps to read data for.

Note

A batch is provided with a maximum size - it may be smaller, as it will not extend beyond the current maximum capactiy.

Note

If batch_size is left to 0, then the default size will be max_batch_size used in the initial configuration of the StreamBrokers.

Note

The batch_size will never be larger than max_batch_size either.

Returns

The next batch of step indices [first, last) to fetch data for. The Exhausted batch is returned as first and last equal to ExhaustedSentinel

Parameters

Parameter	Type	Description
batch_size	std::size_t	The maximum size of the batch of step indices to read.

---

get_stream_group

inline

template<std::size_t MaxSegments = 128> inline SBIO_HDBrokerGroup< BrokerType, FTraits, MaxSegments > get_stream_group(const char * name)

Defined in src/lib/sbio/core/datasource.hh:338

Construct a BrokerGroup by name from the set of StreamBrokers.

Note

The metadata discovery step MUST have been passed through for BrokerGroup construction to work. A group cannot be formed until it is known what data is available from the set of Streams.

Returns

The BrokerGroup of the given name. The success of the group formation can be checked by inspecting the number of segments the returned group has (0 indicating no group could be formed).

---

num_data_streams

const inline

inline SBIO_HD std::size_t num_data_streams() const

Defined in src/lib/sbio/core/datasource.hh:350

The number of data Streams in the DataSource’s set.

---

data_streams

const inline

inline SBIO_HD const BrokerType * data_streams() const

Defined in src/lib/sbio/core/datasource.hh:357

A pointer to the first StreamBroker of the set.

Returns

A pointer to the first StreamBroker of the set.

---

data_streams

inline

inline SBIO_HD BrokerType * data_streams()

Defined in src/lib/sbio/core/datasource.hh:363

A pointer to the first StreamBroker of the set.

Returns

A pointer to the first StreamBroker of the set.

---

data_stream

const inline

inline SBIO_HD const BrokerType & data_stream(std::size_t i) const

Defined in src/lib/sbio/core/datasource.hh:373

The requested StreamBroker.

Note

This function does not bounds check! Be sure to inspect the number of data streams before indexing out of bounds!

Returns

The requested StreamBroker.

Parameters

Parameter	Type	Description
i	std::size_t	The StreamBroker to return.

---

data_stream

inline

inline SBIO_HD BrokerType & data_stream(std::size_t i)

Defined in src/lib/sbio/core/datasource.hh:383

The requested StreamBroker.

Note

This function does not bounds check! Be sure to inspect the number of data streams before indexing out of bounds!

Returns

The requested StreamBroker.

Parameters

Parameter	Type	Description
i	std::size_t	The StreamBroker to return.

---

steps

inline

inline SBIO_HDStepRange steps()

Defined in src/lib/sbio/core/datasource.hh:506

Return a range over the steps of this DataSource for the calling unit.

Returns

A range to iterate over in a range-based for loop.

---

steps

const inline

inline SBIO_HDConstStepRange steps() const

Defined in src/lib/sbio/core/datasource.hh:507

---

csteps

const inline

inline SBIO_HDConstStepRange csteps() const

Defined in src/lib/sbio/core/datasource.hh:508

---

begin

inline

inline SBIO_HDIterator begin()

Defined in src/lib/sbio/core/datasource.hh:520

Return an iterator at the first step index.

Note

This will invoke next to get the first step for this unit

Note

Using the DataSource directly for iteration doesn’t use the EPolicy’s end iteration implementation for early exits. This must be manually managed then, or instead, prefer the range-based loop with [steps()](#steps).

Returns

An iterator at the first step index.

---

begin

const inline

inline SBIO_HDConstIterator begin() const

Defined in src/lib/sbio/core/datasource.hh:521

---

end

inline

inline SBIO_HDIterator end()

Defined in src/lib/sbio/core/datasource.hh:527

Return an iterator pointing to the ExhasutedSentinel.

Returns

An iterator pointing to the ExhaustedSentinel.

---

end

const inline

inline SBIO_HDConstIterator end() const

Defined in src/lib/sbio/core/datasource.hh:528

---

cbegin

const inline

inline SBIO_HDConstIterator cbegin() const

Defined in src/lib/sbio/core/datasource.hh:530

---

cend

const inline

inline SBIO_HDConstIterator cend() const

Defined in src/lib/sbio/core/datasource.hh:531

---

batches

inline

inline SBIO_HDBatchRange batches(std::size_t batch_size = 0)

Defined in src/lib/sbio/core/datasource.hh:644

Return a range over contiguous batches of steps for the calling unit.

Returns

A range to iterate over in a range-based for loop.

Parameters

Parameter	Type	Description
batch_size	std::size_t	The maximum number of steps per batch. The default is the configured max_batch_size.

---

batches

const inline

inline SBIO_HDConstBatchRange batches(std::size_t batch_size = 0) const

Defined in src/lib/sbio/core/datasource.hh:647

---

cbatches

const inline

inline SBIO_HDConstBatchRange cbatches(std::size_t batch_size = 0) const

Defined in src/lib/sbio/core/datasource.hh:650

Public Types

Name	Description
IOPolicy	The type of the IO strategy being used.
ExecutionPolicy	The Execution policy type.
DataFormat	The type of data being read.
StreamType	The type of Stream: I.e., the IO strategy and data format being read.
SBStorageType	The type of the StreamBroker’s Storage.
EPolicyConfig	The Execution policy configuration object type.
StreamConfig	The individual Stream configuration object type.
StreamMetadata	The type of the general metadata object for the data format’s Stream.
DataAccessPtn	The type of the enumerator used to specify access patterns used for the format.
DataRequest	The type of a request object used to query for data.
StepIdxType	The type used to request a specific step from the Stream.
Iterator
ConstIterator
StepRange
ConstStepRange
BatchIterator
ConstBatchIterator
BatchRange
ConstBatchRange

---

IOPolicy

using IOPolicy = IO

Defined in src/lib/sbio/core/datasource.hh:95

The type of the IO strategy being used.

---

ExecutionPolicy

using ExecutionPolicy = EPolicy

Defined in src/lib/sbio/core/datasource.hh:99

The Execution policy type.

---

DataFormat

using DataFormat = FTraits

Defined in src/lib/sbio/core/datasource.hh:103

The type of data being read.

---

StreamType

using StreamType = Stream< IO, FTraits >

Type: Stream< IO, FTraits >

Defined in src/lib/sbio/core/datasource.hh:108

The type of Stream: I.e., the IO strategy and data format being read.

---

SBStorageType

using SBStorageType = Storage< typename FTraits::BrokerBufferRequirements, EPolicy >

Type: Storage< typename FTraits::BrokerBufferRequirements, EPolicy >

Defined in src/lib/sbio/core/datasource.hh:113

The type of the StreamBroker’s Storage.

---

EPolicyConfig

using EPolicyConfig = typename EPolicy::Config

Defined in src/lib/sbio/core/datasource.hh:122

The Execution policy configuration object type.

epolicy_config objects configure the global behaviour of the Execution policy being used. The Execution policy must be configured before any Streams are opened as it controls all aspects of IO down to the allocation of Storage.

---

StreamConfig

using StreamConfig = GenericStreamConfig< FTraits >

Type: GenericStreamConfig< FTraits >

Defined in src/lib/sbio/core/datasource.hh:129

The individual Stream configuration object type.

stream_config objects are used to set up each individual Stream so that it can connect and read from its individual data.

---

StreamMetadata

using StreamMetadata = MetadataInventory< FTraits >

Type: MetadataInventory< FTraits >

Defined in src/lib/sbio/core/datasource.hh:134

The type of the general metadata object for the data format’s Stream.

---

DataAccessPtn

using DataAccessPtn = typename FTraits::DataAccessPtn

Defined in src/lib/sbio/core/datasource.hh:139

The type of the enumerator used to specify access patterns used for the format.

---

DataRequest

using DataRequest = typename FTraits::DataRequest

Defined in src/lib/sbio/core/datasource.hh:143

The type of a request object used to query for data.

---

StepIdxType

using StepIdxType = typename FTraits::StepIdxType

Defined in src/lib/sbio/core/datasource.hh:150

The type used to request a specific step from the Stream.

This type is required and guaranteed to be convertible std::size_t; however, different data format’s may use different underlying types.

---

Iterator

using Iterator = IteratorImpl< DataSource >

Type: IteratorImpl< DataSource >

Defined in src/lib/sbio/core/datasource.hh:444

---

ConstIterator

using ConstIterator = IteratorImpl< const DataSource >

Type: IteratorImpl< const DataSource >

Defined in src/lib/sbio/core/datasource.hh:445

---

StepRange

using StepRange = StepRangeImpl< DataSource >

Type: StepRangeImpl< DataSource >

Defined in src/lib/sbio/core/datasource.hh:498

---

ConstStepRange

using ConstStepRange = StepRangeImpl< const DataSource >

Type: StepRangeImpl< const DataSource >

Defined in src/lib/sbio/core/datasource.hh:499

---

BatchIterator

using BatchIterator = BatchIteratorImpl< DataSource >

Type: BatchIteratorImpl< DataSource >

Defined in src/lib/sbio/core/datasource.hh:578

---

ConstBatchIterator

using ConstBatchIterator = BatchIteratorImpl< const DataSource >

Type: BatchIteratorImpl< const DataSource >

Defined in src/lib/sbio/core/datasource.hh:579

---

BatchRange

using BatchRange = BatchRangeImpl< DataSource >

Type: BatchRangeImpl< DataSource >

Defined in src/lib/sbio/core/datasource.hh:634

---

ConstBatchRange

using ConstBatchRange = BatchRangeImpl< const DataSource >

Type: BatchRangeImpl< const DataSource >

Defined in src/lib/sbio/core/datasource.hh:635

Private Attributes

Return	Name	Description
BrokerType	m_data_streams	The set of StreamBrokers in the DataSource
std::size_t	m_num_data_streams	The total number of StreamBrokers in the DataSource
std::size_t	m_steps_capacity	The current steps capacity before reindexing is required.
EPolicy::IterationState	m_iteration_state	The EPolicy’s state for iterating and distribution steps from this DataSource.
bool	m_epolicy_configured	Flag to track whether the Execution policy has been configured.

---

m_data_streams

BrokerType m_data_streams[MaxDataStreams]

Defined in src/lib/sbio/core/datasource.hh:692

The set of StreamBrokers in the DataSource

---

m_num_data_streams

std::size_t m_num_data_streams { 0 }

Defined in src/lib/sbio/core/datasource.hh:697

The total number of StreamBrokers in the DataSource

---

m_steps_capacity

std::size_t m_steps_capacity { 0 }

Defined in src/lib/sbio/core/datasource.hh:702

The current steps capacity before reindexing is required.

---

m_iteration_state

EPolicy::IterationState m_iteration_state {}

Defined in src/lib/sbio/core/datasource.hh:707

The EPolicy’s state for iterating and distribution steps from this DataSource.

---

m_epolicy_configured

bool m_epolicy_configured { false }

Defined in src/lib/sbio/core/datasource.hh:719

Flag to track whether the Execution policy has been configured.

Since there may be global state in the policy, ensure that a configuration gets called. If its explicitly provided, then that gets used. Otherwise, a default configuration will be invoked before any IO. This allows any global state to be reset if iteratively creating multiple DataSource instantiations over time. This only applies to static state (generally, process and environment). The remaining state is held in the member m_iteration_state.

Private Methods

Return	Name	Description
SBIO_HD bool	reindex_trigger const inline

---

reindex_trigger

const inline

inline SBIO_HD bool reindex_trigger() const

Defined in src/lib/sbio/core/datasource.hh:655