SourceSet

SourceSet

#include <sbio/core/iteration.hh>

template<class... Sources>
class SourceSet

Defined in src/lib/sbio/core/iteration.hh:752

—SourceSet Implementations (Zipping multiple DataSources) —///

A set of DataSources iterated together, in lockstep by step index.

The SourceSet allows grouping multiple DataSources, even those that are viewing different underlying data. Iteration over a SourceSet means iterating over each component DataSource simultaneously - ie.e when a SoruceSet returns a step n, it corresponds to step n for each DataSource. No remapping/alignment occurs.

Note

If using a SourceSet for iteration, the individual DataSources must NOT be iterated independently!

Note

As the DataSources are iterated together without alignment the capacity is set to the minimum capacity of all the component DataSources. If reindexing can occur, though, only the DataSources who reached the capacity are reindexed to avoid overwriting the step offset data.

Note

The SourceSet is restricted to DataSources with the same EPolicy and StepIdxType.

Note

The SourceSet must be constructed from DataSources after the metadata discovery process.

The SourceSet holds its own EPolicy IterationState and therefore behaves like an independent DataSource.

Each parallel executing unit (e.g. a thread or rank, etc.) iterating will setup its own range-based for loop over the zipped set. First create the set, then zip the BrokerGroups of intereset.

auto set = source_set(ds_a, ds_b);
for (auto step : zip(set, bg1, bg2)) {
  auto hdl1 = step.get(bg1);
  auto hdl2 = step.get(bg2);
}

Note

Early exit from a loop invokes the EPolicy end iteration implementation, the same as using [DataSource::next()](api-sbio-DataSource.md#next) (or [steps()](#steps-2), etc.)

Template Parameters

• Sources The DataSource types. They must use the same EPolicy and StepIdxType.

Public Methods

Return	Name	Description
SBIO_HD	SourceSet inline explicit	Construct a set over DataSources which have already discovered their metadata.
	SourceSet	Deleted constructor.
SourceSet &	operator=	Deleted assignment operator.
SBIO_HDStepIdxType	next const inline	Request the next index for a step to read data for, common to all DataSources.
SBIO_HDStepBatch< StepIdxType >	next_batch const inline	Request the next contiguous batch of indices for steps, common to all DataSources.
SBIO_HDStepRange	steps inline	Return a range over the common steps of the set for the calling unit.
SBIO_HDConstStepRange	steps const inline
SBIO_HDConstStepRange	csteps const inline
SBIO_HDBatchRange	batches inline	Return a range over contiguous batches of steps, common to the DataSources.
SBIO_HDConstBatchRange	batches const inline
SBIO_HDConstBatchRange	cbatches const inline

---

SourceSet

inline explicit

inline explicit SBIO_HD SourceSet(Sources &... sources)

Defined in src/lib/sbio/core/iteration.hh:772

Construct a set over DataSources which have already discovered their metadata.

---

SourceSet

SourceSet(const SourceSet &) = delete

Defined in src/lib/sbio/core/iteration.hh:778

Deleted constructor.

---

operator=

SourceSet & operator=(const SourceSet &) = delete

Defined in src/lib/sbio/core/iteration.hh:779

Deleted assignment operator.

---

next

const inline

inline SBIO_HDStepIdxType next() const

Defined in src/lib/sbio/core/iteration.hh:787

Request the next index for a step to read data for, common to all DataSources.

Returns

The next step index to fetch and query data for, or the ExhaustedSentinel if no more data is available.

---

next_batch

const inline

inline SBIO_HDStepBatch< StepIdxType > next_batch(hd_std::size_t batch_size = 0) const

Defined in src/lib/sbio/core/iteration.hh:809

Request the next contiguous batch of indices for steps, common to all DataSources.

Note

A batch is providfed with a maximum size - it may be smaller, as it will not extend beyond the current maximum capacity.

Note

If batch_size is left to 0, then the default size will be the smallest max_batch_size of the component DataSources in the set.

Note

The batch_size will never be larger than max_batch_size either.

Returns

The next back of step indices [first, last) to fetch data for. The Exhausted batch is returned as first and last equal to ExhaustedSentinel

Parameters

Parameter	Type	Description
batch_size	hd_std::size_t	The maximum size of the batch of step indcies to read.

---

steps

inline

inline SBIO_HDStepRange steps()

Defined in src/lib/sbio/core/iteration.hh:946

Return a range over the common steps of the set for the calling unit.

Returns

A range to iterate over in a range-based for loop.

---

steps

const inline

inline SBIO_HDConstStepRange steps() const

Defined in src/lib/sbio/core/iteration.hh:947

---

csteps

const inline

inline SBIO_HDConstStepRange csteps() const

Defined in src/lib/sbio/core/iteration.hh:948

---

batches

inline

inline SBIO_HDBatchRange batches(hd_std::size_t batch_size = 0)

Defined in src/lib/sbio/core/iteration.hh:957

Return a range over contiguous batches of steps, common to the DataSources.

Returns

A range to iterate over in a range-based for loop.

Parameters

Parameter	Type	Description
batch_size	hd_std::size_t	The maximum number of steps per batch. The default is the smallest configured max_batch_size.

---

batches

const inline

inline SBIO_HDConstBatchRange batches(hd_std::size_t batch_size = 0) const

Defined in src/lib/sbio/core/iteration.hh:960

---

cbatches

const inline

inline SBIO_HDConstBatchRange cbatches(hd_std::size_t batch_size = 0) const

Defined in src/lib/sbio/core/iteration.hh:963

Public Types

Name	Description
DataFormat
ExecutionPolicy
StepIdxType
Iterator
ConstIterator
BatchIterator
ConstBatchIterator
StepRange
ConstStepRange
BatchRange
ConstBatchRange

---

DataFormat

using DataFormat = typename First::DataFormat

Defined in src/lib/sbio/core/iteration.hh:758

---

ExecutionPolicy

using ExecutionPolicy = typename First::ExecutionPolicy

Defined in src/lib/sbio/core/iteration.hh:759

---

StepIdxType

using StepIdxType = typename First::StepIdxType

Defined in src/lib/sbio/core/iteration.hh:760

---

Iterator

using Iterator = IteratorImpl< SourceSet, false >

Type: IteratorImpl< SourceSet, false >

Defined in src/lib/sbio/core/iteration.hh:874

---

ConstIterator

using ConstIterator = IteratorImpl< const SourceSet, false >

Type: IteratorImpl< const SourceSet, false >

Defined in src/lib/sbio/core/iteration.hh:875

---

BatchIterator

using BatchIterator = IteratorImpl< SourceSet, true >

Type: IteratorImpl< SourceSet, true >

Defined in src/lib/sbio/core/iteration.hh:877

---

ConstBatchIterator

using ConstBatchIterator = IteratorImpl< const SourceSet, true >

Type: IteratorImpl< const SourceSet, true >

Defined in src/lib/sbio/core/iteration.hh:878

---

StepRange

using StepRange = RangeImpl< SourceSet, false >

Type: RangeImpl< SourceSet, false >

Defined in src/lib/sbio/core/iteration.hh:935

---

ConstStepRange

using ConstStepRange = RangeImpl< const SourceSet, false >

Type: RangeImpl< const SourceSet, false >

Defined in src/lib/sbio/core/iteration.hh:936

---

BatchRange

using BatchRange = RangeImpl< SourceSet, true >

Type: RangeImpl< SourceSet, true >

Defined in src/lib/sbio/core/iteration.hh:938

---

ConstBatchRange

using ConstBatchRange = RangeImpl< const SourceSet, true >

Type: RangeImpl< const SourceSet, true >

Defined in src/lib/sbio/core/iteration.hh:939

Private Attributes

Return	Name	Description
hd_std::tuple< Sources *... >	m_sources
ExecutionPolicy::IterationState	m_iteration_state
StepIdxType	m_capacity

---

m_sources

hd_std::tuple< Sources *... > m_sources

Defined in src/lib/sbio/core/iteration.hh:1019

---

m_iteration_state

ExecutionPolicy::IterationState m_iteration_state {}

Defined in src/lib/sbio/core/iteration.hh:1020

---

m_capacity

StepIdxType m_capacity { 0 }

Type: StepIdxType

Defined in src/lib/sbio/core/iteration.hh:1021

Private Methods

Return	Name	Description
SBIO_HD bool	reindex_trigger const inline	Reindex the DataSources which reached the common capacity.
SBIO_HDStepIdxType	min_capacity const inline
SBIO_HD hd_std::size_t	min_batch_size const inline

---

reindex_trigger

const inline

inline SBIO_HD bool reindex_trigger() const

Defined in src/lib/sbio/core/iteration.hh:973

Reindex the DataSources which reached the common capacity.

Returns

Whether more common steps are available (false once any is exhausted).

---

min_capacity

const inline

inline SBIO_HDStepIdxType min_capacity() const

Defined in src/lib/sbio/core/iteration.hh:989

---

min_batch_size

const inline

inline SBIO_HD hd_std::size_t min_batch_size() const

Defined in src/lib/sbio/core/iteration.hh:1004