MPIDistribution

MPIDistribution

#include <sbio/execution/mpi.hh>

class MPIDistribution

Defined in src/lib/sbio/execution/mpi.hh:67

A basic shared MPI execution policy.

This policy implements some basic functionality for MPI-based workflows. The main features are:

• Allocations for types of the [roles::Index](api-sbio-roles-Index.md#index) are made over shared MPI windows.
• Shared values are accordingly synchronized via the sync group mechanism on pre/post update hooks.
• As IndexRole buffers are shared, redundant INDEXING is avoided by only allowing that Broker state for rank 0.
• Steps are processed in a simple round-robin fashion with a modulo world-size increment for each rank.
• Exhaustion of available indexed capacity automatically retriggers INDEXING if applicable for the Brokers.

Public Static Attributes

Return	Name	Description
constexpr std::size_t	MaxInactiveRanks static constexpr
constexpr std::bitset< static_cast< std::size_t >(ParallelizationMethods::NUM_METHODS) >	ParallelSupport static constexpr

---

MaxInactiveRanks

static constexpr

constexpr std::size_t MaxInactiveRanks { 1024 }

Defined in src/lib/sbio/execution/mpi.hh:69

---

ParallelSupport

static constexpr

constexpr std::bitset< static_cast< std::size_t >(ParallelizationMethods::NUM_METHODS) > ParallelSupport { 0x2 }

Type: constexpr std::bitset< static_cast< std::size_t >(ParallelizationMethods::NUM_METHODS) >

Defined in src/lib/sbio/execution/mpi.hh:98

Public Static Methods

Return	Name	Description
constexpr BufferScope	scope_for static inline constexpr	Map IndexRole and Shareable to MPISharedBuffer and all other sahred within process.
bool	validate_environment_impl static inline	The only requirement is that MPI was not finalized before launch.
void	configure_impl static inline
std::size_t	unit_ordinal static inline	The ordinal (associated number) for the calling parallel unit.
void	init_logger static inline	Setup a spdlog logger.
void	allocate_node_shared static inline	Setup a node-wide shared buffer.
void	pre_update_impl static inline	This policy establishes a shared memory window over index data. The pre_update hook includes a fence on that window so we can synchronize updates to the memory across ranks.
void	post_update_impl static inline	Paired with the pre_update hook, the post_update ensures the synchronization of IndexRole storage is completed.
bool	should_index_impl static inline	Check on if indexing should be done by the Broker.
bool	is_current_rank_inactive static inline
FTraits::StepIdxType	next_impl static inline	Retrieve the next step index to process.
StepBatch< typename FTraits::StepIdxType >	next_batch_impl static inline	Retrieve the next contiguous batch of steps to process.
void	end_iteration_impl static inline	Exit the iteration with state, releasing any held steps.

---

scope_for

static inline constexpr

template<typename Descriptor> constexpr static inline constexpr BufferScope scope_for()

Defined in src/lib/sbio/execution/mpi.hh:75

Map IndexRole and Shareable to MPISharedBuffer and all other sahred within process.

---

validate_environment_impl

static inline

static inline bool validate_environment_impl(const Config & config, const char *& error)

Defined in src/lib/sbio/execution/mpi.hh:103

The only requirement is that MPI was not finalized before launch.

---

configure_impl

static inline

static inline void configure_impl(const Config & config)

Defined in src/lib/sbio/execution/mpi.hh:114

---

unit_ordinal

static inline

static inline std::size_t unit_ordinal()

Defined in src/lib/sbio/execution/mpi.hh:248

The ordinal (associated number) for the calling parallel unit.

Ranks of a node are numbered using the node-rank.

Note

If there is no node-communicator, the world rank is used.

---

init_logger

static inline

static inline void init_logger()

Defined in src/lib/sbio/execution/mpi.hh:314

Setup a spdlog logger.

---

allocate_node_shared

static inline

static inline void allocate_node_shared(MPISharedBuffer & buf, std::size_t sz)

Defined in src/lib/sbio/execution/mpi.hh:335

Setup a node-wide shared buffer.

For the MPI distribution strategy, an MPI Window is used for sharing. Within the window an area is reserved by the distribution manager (a “catalog”) for distributing state among the participant ranks. If a rank is inactive, it instead will have a private buffer in lieu of the window.

Parameters

Parameter	Type	Description
sz	std::size_t	The size of the buffer.

---

pre_update_impl

static inline

template<class Role, class StorageT> static inline void pre_update_impl(StorageT & storage)

Defined in src/lib/sbio/execution/mpi.hh:368

This policy establishes a shared memory window over index data. The pre_update hook includes a fence on that window so we can synchronize updates to the memory across ranks.

Parameters

Parameter	Type	Description
storage	StorageT &	The storage to synchronize.

---

post_update_impl

static inline

template<class Role, class StorageT, class SyncT> static inline void post_update_impl(StorageT & storage, SyncT && sync_vars, IOStatus status)

Defined in src/lib/sbio/execution/mpi.hh:415

Paired with the pre_update hook, the post_update ensures the synchronization of IndexRole storage is completed.

Remaining sync_vars (including index catalog) get shared through the window as well, via a special catalog area. The catalog area is setup by the indexer for the node, and then it publishes a new sequence number. The other ranks can read back after this. (No additional messages are passed, so communication cannot end up causing a stall if a rank leaves early.)

Parameters

Parameter	Type	Description
storage	StorageT &	The storage to synchronize.
sync_vars	SyncT &&	The attributes that require synchronization
status	IOStatus	The IOStatus from the INDEXING state.

---

should_index_impl

static inline

static inline bool should_index_impl()

Defined in src/lib/sbio/execution/mpi.hh:485

Check on if indexing should be done by the Broker.

File indexing is performed by one rank on each individual node (machine/server). The first rank of the node (rank 0, or lowest, of a node-wide communicator) fills the indexing role.

The EPolicy provides a shared-memory buffer for the index storage, so the other ranks can read what the indexer puts into it.

Note

A rank without their own node-index, still then create and own a private storage buffer for the offsets.

Returns

true for the node’s indexing rank (or an inactive rank), else false.

---

is_current_rank_inactive

static inline

static inline bool is_current_rank_inactive()

Defined in src/lib/sbio/execution/mpi.hh:489

---

next_impl

static inline

template<class FTraits, class IndexTrigger> static inline FTraits::StepIdxType next_impl(IterationState & state, typename FTraits::StepIdxType & max_capacity, IndexTrigger && trigger)

Defined in src/lib/sbio/execution/mpi.hh:515

Retrieve the next step index to process.

Steps are processed using a fixed offset of the rank, with increments of the world size. E.g. for a world size of 2:

• Step 1: Rank 0 processes 0, Rank 1 processes 1
• Step 2: Rank 0 processes 2, Rank 1 processes 3 … and so on.

Returns

The next step to process using the fixed offset of the world size.

Parameters

Parameter	Type	Description
max_capacity	typename FTraits::StepIdxType &	The current max capacity (i.e., already indexed steps).
trigger	IndexTrigger &&	A callback to reindex (if appropriate) when capacity is exhausted.

---

next_batch_impl

static inline

template<class FTraits, class IndexTrigger> static inline StepBatch< typename FTraits::StepIdxType > next_batch_impl(IterationState & state, std::size_t batch_size, typename FTraits::StepIdxType & max_capacity, IndexTrigger && trigger)

Defined in src/lib/sbio/execution/mpi.hh:573

Retrieve the next contiguous batch of steps to process.

Batches are distributed per capacity window (the steps added by one reindexing), as a batch cannot span a reindexing. Within a window, rank r receives the blocks of batch_size steps starting at window_start + (round * world_size + r) * batch_size. The last block of a window may be shorter. Once a rank has no block left in the current window it reindexes, and the next window starts at the previous capacity.

Note

Batches and single steps (next_impl) should not be mixed on one iteration state.

Returns

The next batch of steps for this rank.

Parameters

Parameter	Type	Description
batch_size	std::size_t	The maximum number of steps in the batch.
max_capacity	typename FTraits::StepIdxType &	The current max capacity (i.e., already indexed steps).
trigger	IndexTrigger &&	A callback to reindex (if appropriate) when capacity is exhausted.

---

end_iteration_impl

static inline

template<class FTraits, class IndexTrigger> static inline void end_iteration_impl(IterationState & state, typename FTraits::StepIdxType & max_capacity, IndexTrigger && trigger)

Defined in src/lib/sbio/execution/mpi.hh:649

Exit the iteration with state, releasing any held steps.

Public Types

Name	Description
NodeSharedBuffer

---

NodeSharedBuffer

using NodeSharedBuffer = MPISharedBuffer

Type: MPISharedBuffer

Defined in src/lib/sbio/execution/mpi.hh:85

Private Static Attributes

Return	Name	Description
constexpr std::size_t	CatalogBytes static constexpr	Window area for sync_vars.
constexpr std::size_t	CatalogHeaderBytes static constexpr	Sequence number slot.
constexpr int	NodeRelease static constexpr	Done with the window - may be rewritten.
constexpr int	NodeDeparted static constexpr	Left the iteration early.
MPI_Comm	m_world_comm static	Communicator for synchronizing across the whole MPI world.
MPI_Comm	m_active_comm static
int	m_active_rank static
int	m_active_size static
int	m_inactive_ranks static
std::size_t	m_num_inactive_ranks static
int	m_main_rank static
bool	m_main_rank_loops static
MPI_Comm	m_shmem_comm static	Communicator used when generating shareable buffers.
int	m_shmem_rank static	This rank within its node (m_shmem_comm).
int	m_shmem_size static	The number of (active) ranks on this node.
int	m_node_indexer static	The node rank which indexes for the node.
std::vector< int >	m_node_iterates static	For each rank on the node, whether it iterates (calls next).
int	m_rank static	This processes rank in the MPI world.
int	m_size static	The size of the MPI world.
std::shared_ptr< spdlog::logger >	m_logger static

---

CatalogBytes

static constexpr

constexpr std::size_t CatalogBytes { 4096 }

Defined in src/lib/sbio/execution/mpi.hh:688

Window area for sync_vars.

---

CatalogHeaderBytes

static constexpr

constexpr std::size_t CatalogHeaderBytes { 64 }

Defined in src/lib/sbio/execution/mpi.hh:689

Sequence number slot.

---

NodeRelease

static constexpr

constexpr int NodeRelease { 1 }

Defined in src/lib/sbio/execution/mpi.hh:717

Done with the window - may be rewritten.

---

NodeDeparted

static constexpr

constexpr int NodeDeparted { 2 }

Defined in src/lib/sbio/execution/mpi.hh:718

Left the iteration early.

---

m_world_comm

static

MPI_Comm m_world_comm { MPI_COMM_NULL }

Defined in src/lib/sbio/execution/mpi.hh:808

Communicator for synchronizing across the whole MPI world.

---

m_active_comm

static

MPI_Comm m_active_comm { MPI_COMM_NULL }

Defined in src/lib/sbio/execution/mpi.hh:809

---

m_active_rank

static

int m_active_rank { -1 }

Defined in src/lib/sbio/execution/mpi.hh:810

---

m_active_size

static

int m_active_size { -1 }

Defined in src/lib/sbio/execution/mpi.hh:811

---

m_inactive_ranks

static

int m_inactive_ranks[MaxInactiveRanks]

Defined in src/lib/sbio/execution/mpi.hh:812

---

m_num_inactive_ranks

static

std::size_t m_num_inactive_ranks { 0 }

Defined in src/lib/sbio/execution/mpi.hh:813

---

m_main_rank

static

int m_main_rank { 0 }

Defined in src/lib/sbio/execution/mpi.hh:814

---

m_main_rank_loops

static

bool m_main_rank_loops { true }

Defined in src/lib/sbio/execution/mpi.hh:815

---

m_shmem_comm

static

MPI_Comm m_shmem_comm { MPI_COMM_NULL }

Defined in src/lib/sbio/execution/mpi.hh:819

Communicator used when generating shareable buffers.

---

m_shmem_rank

static

int m_shmem_rank { -1 }

Defined in src/lib/sbio/execution/mpi.hh:820

This rank within its node (m_shmem_comm).

---

m_shmem_size

static

int m_shmem_size { 0 }

Defined in src/lib/sbio/execution/mpi.hh:821

The number of (active) ranks on this node.

---

m_node_indexer

static

int m_node_indexer { -1 }

Defined in src/lib/sbio/execution/mpi.hh:822

The node rank which indexes for the node.

---

m_node_iterates

static

std::vector< int > m_node_iterates {}

Defined in src/lib/sbio/execution/mpi.hh:828

For each rank on the node, whether it iterates (calls next).

Note

This indicates whether the rank participates in collective reindexing.

---

m_rank

static

int m_rank { -1 }

Defined in src/lib/sbio/execution/mpi.hh:829

This processes rank in the MPI world.

---

m_size

static

int m_size { -1 }

Defined in src/lib/sbio/execution/mpi.hh:830

The size of the MPI world.

---

m_logger

static

std::shared_ptr< spdlog::logger > m_logger

Defined in src/lib/sbio/execution/mpi.hh:832

Private Static Methods

Return	Name	Description
constexpr std::size_t	catalog_offset static inline constexpr	Offset of the catalog area after a shared buffer of size bytes.
char *	catalog_area static inline	The catalog used by the processing ranks (inset into the rest of the buffer).
bool	rank_iterates static inline	Whether this rank takes steps (calls next) at all.
bool	is_node_indexer static inline
int	node_tag static inline	Message tag of the node messages of an iteration.
void	release_window static inline	Release the current index window of the node before it is rewritten.
bool	others_iterating static inline	Whether other ranks of the node still iterate.
bool	join_iteration static inline	Count this thread as iterating with state (once per iteration).

---

catalog_offset

static inline constexpr

constexpr static inline constexpr std::size_t catalog_offset(std::size_t size)

Defined in src/lib/sbio/execution/mpi.hh:694

Offset of the catalog area after a shared buffer of size bytes.

---

catalog_area

static inline

template<class BufT> static inline char * catalog_area(BufT & buf)

Defined in src/lib/sbio/execution/mpi.hh:702

The catalog used by the processing ranks (inset into the rest of the buffer).

---

rank_iterates

static inline

static inline bool rank_iterates()

Defined in src/lib/sbio/execution/mpi.hh:709

Whether this rank takes steps (calls next) at all.

---

is_node_indexer

static inline

static inline bool is_node_indexer()

Defined in src/lib/sbio/execution/mpi.hh:713

---

node_tag

static inline

static inline int node_tag(const IterationState & state)

Defined in src/lib/sbio/execution/mpi.hh:725

Message tag of the node messages of an iteration.

Note

Iteration states are expected to be created in the same order on each rank.

---

release_window

static inline

static inline void release_window(IterationState & state)

Defined in src/lib/sbio/execution/mpi.hh:736

Release the current index window of the node before it is rewritten.

The other ranks of the node tell the node indexing rank that they are done with the window (or left the iteration). The indexing rank waits for all of them. Called after the rank drained its handed out steps.

---

others_iterating

static inline

static inline bool others_iterating(const IterationState & state)

Defined in src/lib/sbio/execution/mpi.hh:771

Whether other ranks of the node still iterate.

Note

Used by the node’s indexer rank only.

---

join_iteration

static inline

static inline bool join_iteration(IterationState & state)

Defined in src/lib/sbio/execution/mpi.hh:791

Count this thread as iterating with state (once per iteration).

Note

A thread starting after all others of its rank left finds the rank departed.

Returns

false if the rank departed the iteration.