Building and Building Against sbio

Recommendations for Builds and Installs

Unlike with ncarray, the sbio package is often better built from source. This is because the various components are all optional, and choosing to build and link against a set of components in the environment you are working in is easier than incorporating a pre-built library. E.g., sbio can be built with MPI support (a feature very likely to be used); however, the implementation and version of MPI can vary over a wide range of supported options. If you are interested in sbio you very likely already have a working environment with an MPI installation, in which case building against that may be preferrable. The sbio build is generally quite reasonable in terms of time and computational cost - 5-10 minutes with modest hardware for compilation should generally be sufficient.

Nonetheless, like all XFELPP projects, sbio has releases bundled as wheels available through the XFELPP indices. Currently, only the host builds are released. You can install from the index on the command-line using:

CPU Only

pip install sbio --extra-index-url https://pypi.xfelpp.org/host

The host-builds are currently configured as follows:

Linux

1. glibc 2.28 (With GCC 14.2.1) and/or musl 1.2 (With GCC 14.2.0) for x86 only
2. Built against ncarray == 0.7.6
3. MPI Support with MPICH 5.0.1
4. XTC2 Data Format Support
5. Python Wheels for Python 3.8-3.15t

Windows

1. msvc 19.44.35228 for x86 only
2. Built against ncarray == 0.7.6
3. MPI support with MS-MPI 10.1.3
4. XTC2 Data Format Support
5. Python Wheels for Python 3.8-3.15t

macOS

1. clang 21.0.0 with MACOSX_DEPLOYMENT_TARGET=11.0 for Apple Silicon (AArch64) only
2. Built against ncarray == 0.7.6
3. MPI support with MPICH 5.0.1
4. XTC2 Data Format Support
5. Python Wheels for Python 3.8-3.15t

Releases for the four core library builds (glibc and musl Linux, macOS, Windows) are also uploaded to the GitHub release independently of the Python wheel if that is preferred.

General Information

sbio uses meson as its build system. It will also require a relatively modern compiler for C++23 support. If building with GPU support it then also requires the CUDA toolkit.

In general, the steps for building are:

meson setup $MY_BUILD_DIR # --prefix=$MY_INSTALL_DIR #-Dbuildtype=debug # (or release etc).
meson compile -C $MY_BUILD_DIR
meson install -C $MY_BUILD_DIR
# For tests:
meson test -C $MY_BUILD_DIR

Requirements

In addition to meson the following dependencies are needed:

• meson
• ninja
• meson-python

The remaining dependencies are vendored through the meson subprojects system. ncarray requires Python >= 3.8. The core dependencies are:

• ncarray - the documentation is available here

Additionally, there are further dependencies that are optional given the build options. However, they are likely to be needed to get the most utility from the project:

• MPI - MPICH and OpenMPI can both be used directly from the project and have been tested. Any suitable MPI implementation with a conforming specification and support for OSC/shared memory APIs will likely also work. This would need to be provided by your environment, however.
• CUDA - CUDA Toolkit 12 or 13 would be recommended.

Compiler Matrix

sbio requires at least gcc >= 12 (although gcc >= 13 is recommended, if permitted by CUDA version), clang > 15, or msvc >= 19.30. Other compilers may work but are not tested.

If building with CUDA support then for CUDA Version:

• CUDA < 12.4 you must use gcc > 12 && gcc < 13, clang > 16 && clang < 17, msvc >= 19.30 && msvc < 19.38 (VS 2022 < 17.8)
• CUDA >= 12.4 && CUDA < 12.8 you must use gcc > 12 && gcc < 14, clang > 16 && clang < 19, msvc >= 19.30 && msvc < 19.41 (VS 2022 < 17.11)
• CUDA >= 12.8 you must use gcc > 12 && gcc < 15, clang > 16 && clang < 20, msvc >= 19.30 && msvc < 19.50 (VS 2022 < 18.0)

Buid Options and Setup

There are a number of flags which can be provided when running the meson setup:

• build_core : bool flag for whether to build the C++ libraries.
• sbio_data_formats : string for the core data formats to be built. Currently xtc1, xtc2, or all (for both xtc1 and xtc2) are supported
• build_mpi : string for if and how to build MPI. Passing none will disable MPI support. auto will search for an existing MPI installation, then try MPICH and OpenMPI in that order. mpich or ompi can alternatively be provided to build either of those two options.
• build_python : bool flag for whether to build the Python bindings.
• sbio_cuda_archs : A string of architectures for building CUDA fat binaries, or, alternatively, the string all, in which case architectures 75,80,86,89 and 90 are built (with compute_90 at the end for PTX forward compatibility).
• sbio_as_wheel : bool flag for whether to build as the combined standard wheel.
  • NOTE: If not using this flag, it is expected that the wheel will built as build_core=true build_python=false in one phase, and build_core=false build_python=true in a second.
  • NOTE: It is possible to do two-phase builds with sbio_as_wheel set to true. This can be somewhat more error-prone, but build.sh may have an example for Linux builds.
• build_examples : bool flag for whether to build the example programs. These are small executables and in general are not needed. The tests are always built as part of build_core.

Each of these flags is passed as an option during the meson setup stage using the -D prefix. E.g.

meson setup <builddir> -Dbuild_core=true -Dbuild_python=false -Dbuild_mpi=auto

You can use any build directory of your choosing for <builddir>

Feature Flags and Your Own Projects

Currently, the meson feature flags for incorporating data formats, or MPI support, e.g. are implemented as macro defines internally. This can pose an issue when including headers in your own project. E.g. a build of sbio with MPI support has a SBIO_HAS_MPI macro internally - it may be required to define the macro yourself as well if including headers. This is a clear experience problem and is being worked on for future releases.

Defining the following headers yourself, in the meantime, will avoid problems if you built sbio with these features and are including headers in your own projects:

• SBIO_HAS_XTC1 : For the XTC1 data format
• SBIO_HAS_XTC2 : For the XTC2 data format
• SBIO_HAS_MPI : For MPI support
• SBIO_HAS_CUDA : For CUDA support

Compilation

After running the meson setup command using the choice of options you prefer, compilation can be done using:

meson compile -C <builddir>

The number of parallel jobs can be controlled with -j <jobs> during this step. -l <load> can also be used to try and tweak load. By default as many jobs will be launched as independent processors are available.

In general, it is not necessary to override the default for sbio. The most intensive compilation steps are if building a vendored MPI version; however, it will likely be fine to use the defaults, even in this case.

Building via pip

You can also run pip install . directly after cloning the repo:

git clone [email protected]:XFELPP/sbio
cd sbio
pip install . # --prefix=....

This will be moderately slow compared to parallel builds with meson directly.