sbio

Classes

Name	Description
`BatchedSequential`	Read a long batch of bytes. This only requires provided an offset and number of bytes.
`BrokerGroup`	The logical grouping of StreamBrokers which should be queried together.
`DataSource`	The highest-level abstraction for defining the set of StreamBrokers that will be used.
`Execution`	The Execution policy is the central point for the management of IO activity.
`IAccessStyle`
`MemPool`
`MPIExecution`	A basic shared MPI execution policy.
`MPIThreadedExecution`	The MPIThreadedExecution policy is designed for a hybrid MPI/threaded approach.
`RandomAccess`
`SerialExecution`	SerialExecution is a simple policy, with single-process single-thread host buffers.
`StorageView`
`Stream`	A light-weight wrapper over an IO engine.
`StreamBroker`	A StreamBroker organizes Streams and mediates data requests.
`ThreadedExecution`	ThreadedExecution manages a single-process/multi-thread IO strategy.
`Transaction`
`UnitSequential`	Read a single unit. This will check a header and payload for you.
`AllocationRequest`
`BaseTraits`	Base class tag for a data-format implementation.
`BufferDescriptor`
`BufferMember`
`CalibrationRole`
`CompositeDataResult`	An organized result returned from a BrokerGroup managing one or more brokers.
`cuFileIO`	A IO protocol using cuFile APIs in conjunction with POSIX descriptors.
`DataResult`	A result returned from a request to a single StreamBroker.
`DataRole`
`DeviceBound`
`DoubleBuffered`
`ExtractDescriptorsList`
`ExtractDescriptorsList< Storage< List, Policy > >`
`FindDescriptor`
`FindDescriptor< Role, Id, TypeList< Head, Tail... > >`
`FindDescriptor< Role, Id, TypeList<> >`
`GetHint`
`GetHint< BufferDescriptor< Role, Id, MinSize, Hint > >`
`GroupRole`
`GroupTopology`	The layout, post any sorting, of all segment references in a BrokerGroup.
`HostBuffer`
`IndexRole`
`IOPolicy`
`is_type_list`
`is_type_list< TypeList< Ts... > >`
`IsDescriptor`
`IsDescriptor< BufferDescriptor< Role, Id, Size, H > >`
`MetadataRole`
`MPISharedBuffer`	A buffer implementation backed by an MPI Shared Window.
`MPIWinAllocator`	A custom Allocator for an RC managed MPI_Win.
`MPIWinDeleter`	A custom Deleter for an RC managed MPI_Win.
`RandomTraits`
`RC`	A simple reference counted container for an object.
`SegmentRef`	Map a StreamBroker to a logical position within a BrokerGroup.
`Shareable`
`Storage`
`Storage< TypeList< Roles... >, Policy >`
`SyncGroup`	The SyncGroup specifies a series of references that may require synchronization. This can be exposed by any data stream broker, allowing an execution policy to to generically use appropriate synchronization mechanisms without apriori knowledge of the number, or type.
`SyncGroup< T, Rest... >`
`SyncGroup<>`
`SyncPOSIXIO`	A very basic IO implementation over standard synchronous POSIX APIs.
`TableRole`
`ThreadLocalBuffer`
`TypeList`
`use_rc_alloc_del_t`	A small struct for tag dispatching a custom-allocator/deleter RC creation function.
`use_rc_alloc_t`	A small struct for tag dispatching a custom-allocator RC creation function.
`use_rc_del_t`	A small struct for tag dispatching a custom-deleter RC creation function.
`XTC1Traits`
`XTC2Traits`

Enumerations

Name	Description
`AccessType`
`BrokerState`	The indicators of the StreamBroker state machine.
`ParallelizationMethods`	The set of parallelization methods that may be supported by Execution policies.
`IOStatus`
`MemorySpace`	The MemorySpace an individual buffer lives in.
`AcquireIntent`	Operating through StorageView’s acquisition model, specify MemorySpace to use.
`StreamPartitioningStrategy`	Indicates the strategy used to partition data across streams.
`StreamSentinels`

AccessType

enum AccessType

Value	Description
`NoOp`
`UnitSequential`
`BatchedSequential`
`RandomAccess`

BrokerState

enum BrokerState

The indicators of the StreamBroker state machine.

As data is read from the Stream(s) the Broker will transition through these various states depending on the exact actions that are being undertaken.

The Execution policy can also move the Broker through states, or make decisions, i.e., be held up, or advanced, based on the state.

Not all data formats require the same things, so the exact actions performed by the Broker in each state will change. E.g., “INDEXING” may not always make sense for all data formats (some may not be indexable).

Value	Description
`INIT`	Initialization of any non-stream setup the Broker must do.
`ALLOCATE`	Allocation of memory required by the data format to read properly.
`CONNECT`	Connection/opening of the Streams managed by the Broker.
`DISCOVERY`	Determines the contents of the Stream.
`READY`	Ready to begin reading/brokering/transmitting streamed data.
`INDEXING`	Indexing the data stream (if meaningful).
`STREAMING`	In the process of fetching/transmitting streamed data.
`ERROR`	An error state has been encountered and should be investigated.
`DONE`	The Stream is exhausted and has been closed.

ParallelizationMethods

enum ParallelizationMethods

The set of parallelization methods that may be supported by Execution policies.

A given policy may support one or more of the following parallelization strategies.

Value	Description
`THREADS`	Support for multi-threaded programs.
`MPI`	Support for MPI programs.
`CUDA`	Support for single GPU-coprocessor with CUDA.
`CUDA_MULTIGPU`	Support for multiple GPUs (per process) with CUDA.
`CUDA_FROMDEV`	Support for being invoked from inside CUDA device code.
`NUM_METHODS`

IOStatus

enum IOStatus

Value	Description
`Success`
`Pending`
`WouldBlock`
`OpenFailed`
`GeneralIOError`
`ZeroBytesRead`
`TruncatedRead`
`FunctionUnavailable`
`NoOffsetInData`
`HeaderReadError`
`PayloadTruncatedError`
`AllRequestedRead`

MemorySpace

enum MemorySpace

The MemorySpace an individual buffer lives in.

Each buffer satisfying ValidBuffer (below) indicates the memory space that it resides in. This allows querying, and making decisions if/when memory may need to be copied or moved into different spaces.

Value	Description
`Host`	Resides in host memory.
`Pinned`
`Device`	Resides in device (GPU) memory.
`Managed`

AcquireIntent

enum AcquireIntent

Operating through StorageView’s acquisition model, specify MemorySpace to use.

A caller can specify a view of a buffer must be returned in the caller’s native memory space (GPU/CPU etc), or alternatively, that it can be left in the buffer’s native space. The latter option may at first seem not so useful (why would a caller request access to a buffer it cannot read?) - however, callers in the sbio infrastructure are often orchestrating other components. They can specify acquisition, initiating a transaction, and proceed to direct how the buffer be filled/read elsewhere, without actually reading it themselves. This allows for major performance improvements where the alternative would require copies back and forth between CPU and GPU memory (as an example).

Value	Description
`CallerMemorySpace`	On acquire, view should be in the caller’s space.
`BufferMemorySpace`	On acquire, view can be left in the buffer’s space.

StreamPartitioningStrategy

enum StreamPartitioningStrategy

Indicates the strategy used to partition data across streams.

When partitioning data from a single “logical unit” across multiple data streams written over time, one can imagine two basic strategies:

At each point in time, the logical unit is sub-divided and a portion is written to each data stream.
At each point in time, the entire logical unit is written to one stream, and then the data is round-robined (or via some other selection mechanism distributed) across the other streams.

SubDivide indicates the format uses strategy 1.

Chronological indicates the format uses strategy 2.

Value	Description
`SubDivide`	A single logical unit is sub-divided into multiple streams.
`Chronological`	A complete logical unit is written to one stream at a time.

StreamSentinels

enum StreamSentinels

Value	Description
`RequestExhausted`	All data units from this read request were read.
`StreamExhausted`	The stream is entirely exhausted (no more data)

Typedefs

Return	Name	Description
`typename ExtractDescriptorsList< T >::type`	`ExtractDescriptorsListT`

ExtractDescriptorsListT

using ExtractDescriptorsListT = typename ExtractDescriptorsList< T >::type

Functions

Return	Name	Description
`bool`	`is_locality_match`
`auto`	`make_sync_group`	A variadic helper function to transform any number of objects into a SyncGroup.
`SBIO_HD void`	`safe_strncpy` `inline`
`str`	`get_pkg_config`
`None`	`sbio_pkg_config`
`str`	`get_include`
`str`	`get_lib_dir`

is_locality_match

template<class MemTag> bool is_locality_match(MemorySpace memory_space)

make_sync_group

template<typename... Args> auto make_sync_group(Args &... args)

A variadic helper function to transform any number of objects into a SyncGroup.

safe_strncpy

inline

inline SBIO_HD void safe_strncpy(char * dst, const char * src, std::size_t max_len)

get_pkg_config

str get_pkg_config()

sbio_pkg_config

None sbio_pkg_config()

get_include

str get_include()

get_lib_dir

str get_lib_dir()

Variables

Return	Name	Description
`constexpr bool`	`is_type_list_v` `static` `constexpr`
`constexpr use_rc_alloc_t`	`use_rc_alloc` `constexpr`	Tag to provide for custom allocator RC creation.
`constexpr use_rc_del_t`	`use_rc_del` `constexpr`	Tag to provide for custom deleter RC creation.
`constexpr use_rc_alloc_del_t`	`use_rc_alloc_del` `constexpr`	Tag to provide for custom allocator and deleter RC creation.
`str`	`lib_dir`
`bool`	`has_driver`
`str`	`stub_path`
	`mode`

is_type_list_v

static constexpr

constexpr bool is_type_list_v = <T>::value

use_rc_alloc

constexpr

constexpr use_rc_alloc_t use_rc_alloc {}

Tag to provide for custom allocator RC creation.

use_rc_del

constexpr

constexpr use_rc_del_t use_rc_del {}

Tag to provide for custom deleter RC creation.

use_rc_alloc_del

constexpr

constexpr use_rc_alloc_del_t use_rc_alloc_del {}

Tag to provide for custom allocator and deleter RC creation.

lib_dir

str lib_dir =  ()

has_driver

bool has_driver =  ctypes.util.find_library("cuda") is not None

stub_path

str stub_path =  os.path.join(
            os.path.dirname(__file__), "core", "stubs", "libcuda.so"
        )

mode

 mode