A Simple Introduction to std::simd [WIP]

So C++ 26 has now finalized and becomes the latest standard for C++. There are a lot of things to be excited about. Arguably the “big 3” features of this new standard would be reflection, contracts and execution library, which focus on the language expressivity, safety and concurrency. There have been many great articles and videos online already that talks about the “big 3” features. However, I find it rather difficult to find relevant matrial on one of my most anticipated features in the new standard: The Data-Parallel Types (SIMD).

What is SIMD

As a programmer working in the field of high frequency trading. Latency is often one of the most important and impactful metric of the trading system. While the trading servers are pretty beefy with ample number of CPU core. We often tend to avoid multi-threading operations in the critical path to minimize the context switching/resource contentions etc. Instead, we try to squeeze every laste juice in a single CPU core. SIMD is an unavoidable topic in this space. As the name suggests, there are sort of “parallelism” in SIMD. However, the parallelism is limited to the “data” level. It is a little hard to explain the “data” level parallellism, but we can first take a look at the opposite: instruction-level parallelism. As the name suggests. It naturally involves simultaneous executions of instructions, which unavoidably involves multiple physical CPU cores. In SIMD, only a single CPU core is used. So where does the parallelism come from ? Well it comes from the modern CPUs’ natural capabilities of dealing with more “bytes” simultaneously in each cycle.

In a very simple example, you want to add 4 integers. You normally think of 4 CPU cycles: 1: add the first number to original number (0 here) and store it in some register; 2: add the second number to what’s in the register; 3: add the third number to what’s in the register; 4: add the last number to what’s in the register. Now you are end up with 4 cycles. But modern CPUs often comes with “wide registers” They can actually do it this way: 1: load that 4 numebrs all together into one wide register. 2: reduce (sum) the four integers within it. Just like that. 2x reduction in instruction counts. It’s just like magic. But no, it’s just one of the modern CPUs’ hardware features.

The Standardization

As mentioned above. The SIMD functionality is tied to the CPU hardware. So why is C++26 relevant in this context ? Well, it’s because C++26 standardizes the simd and provides a standard way of explicitly leveraging the SIMD capability of CPU. Before C++26. There have been several non-standard way of doing this:

• Rely on the compiler’s auto-vectorization
• Use intrinsics
• Third-party libraries

For option 1: It’s not a guarantee. And it’s non-portable. You need to curate your compiler flags and investigate the disassembly to be sure the intended auto vectorization is actually generated. And to maintain this, you need to validate pretty often after you’ve updated any code in the hot path. When it’s in a big code base. It’s almost non-maintainable.

For option 2: It’s the very basic way of explicitly using intrinsics. On intel x86 platforms, you’ll need #include <immintrin.h>. And you are going to need to invoke all the double-underscore types and single-underscore functions. It really clutters the eyes.

For option 3: It’s degined to address the problems of the first 2 approaches. There are many choices, Highway, Vc, etc, you name it. These thirdparty libraries provide just good enough abstractions to make it easy enough to use simd operations in an elegant enough way. These libraries are good enough more often than not. But as mentioned, they are not standardized. If you happen to disagree with your colleagues on your favourite libraries, there can be frictions caused becuase of this mismatch.

Entering C++26, as simd now already enters the standard. I think it would be a great way that we can start using the SIMD capabilities of our CPUs.

New Headers, Namespace and Types

As the time of writing. It seems like only GCC > 16.1.0 now supports C++26 SIMD. When you compile your program. C++26 needs to be used: -std=c++26.

The new simd facility comes within the <simd> header. You are going to need to include this header if you want to use any functionality of the new SIMD facility of C++26.

All the std::simd facilities are in the std::simd namespace, including the types and functions.

The very basic new type introduced in the new stadard is std::simd::basic_vec with the following signature:

template< class T, class Abi = /*native-abi*/<T> >
class basic_vec;

The T is the so-called “vectorizable type”. There is nothing too mythical about it. It’s specified in a white-list form in the stadard. in a word, it’s one of the followings:

• all integer types
• most of the floating-point types
• std::complex<S>, where S belongs to type 2.

The type Abi is just a tag type. Here are some of the most common ones:

• std::simd::deduce_abi_t<T, N>. Where N maens: How many values of type T you want to squeeze into this simd_vec type. A normal SIMD register has bit width starting from 128 bits and tops at 512 bits, depending on your specific hardware. If you’ve specified a type wider than what your system supports. It leads to a compile-time error. As the full name of this is very verbose, the std::simd::vec<T, N> alias is introduced to make it more concise. As you can tell from it’s concise name. It’s the centric data structure of the whole std::simd library. Kind of like that std::string is just an alias of std::basic_string.

• std::simd::native_abi<T>. This means the ‘native’ simd register of your platform. For example, if your platform has the 256 bit register ready for you as the native, and T is std::int32_t. Then you will have a simd vector with 8 32 bit integers side-by-side for you to use. Note: As you can see in the signature of the type std::simd::basic_vec. This type is the default type of the Abi tag. So you can simply do: std::simd::basic_vec<T> and it will default to a native abi. In older online tutorials, you may often see the type std::simd::native_vec<T>. This has been removed in the finalized C++26 standard.

The companion type is the mask type:

template< class T, class Abi = /*native-abi*/<T> >
class basic_mask;

Almost exactly the same signature. Almost exactly the same meaning. The difference is that the mask types are just colelctions fo boolean flags. Although represented in type of T. The mask types are the second most common type in all of std::simd. It’s like the if-else statements in the SIMD world. You can’t really do much meaningful operations better than the scalar counterpart without the usage of this mask type. Actually, if you are familiar with numpy, I think you will be fairly faimilar with this mask entity. They are about the same thing.

Useful Basic Functions/Methods regarding the vec and mask Types

The Size of a vec/mask

template <typename T, typename Abi>
std::size_t std::simd::basic_vec::size() const;

template <typename T, typename Abi>
std::size_t std::simd::basic_mask::size() const;

These functions return how many elements of the type T there are in a single basic_vec or basic_mask type.

Standard Library Format/Print Support

You can do this:

std::simd::vec<float, 4> my_simd_vec{ 1.0 };
std::println( "my_simd_vec: {}", my_simd_vec ); // will print: my_simd_vec: [1.0, 1.0, 1.0, 1.0]

std::simd::mask<int, 4> my_simd_mask{ true };
std::println( "my_simd_mask: {}", my_simd_mask ); // will print: my_simd_mask: [true, true, true, true]

The Bracket Operator

Just like a normal C++ container. You can use operator[] to retrieve an element in the collection.

std::simd::vec<float, 4> my_simd_vec{1.0};
const auto first_element = my_simd_vec[0];

std::simd::mask<float, 4> my_simd_mask{true};
const auto first_mask = my_simd_mask[0];

Constructor

For std::simd::basic_vec. You have two constructors to choose from: The fill constructor and the generator constructor.

std::simd::vec<int, 4> fill_simd_vec{1.0}; // all 1
std::simd::vec<int, 4> gen_simd_vec{[]( int ind ) { return i + 1; }}; // {1, 2, 3, 4}

In the generator constructor, you pass a callable that takes the index and returns the desired element and your compiler will help you fill your simd vector.

Basic Arithematics

Basic arithematics are supported with broadcasting behaviors:

std::simd::vec<float, 4> first_vec{ 1.0 };
auto second_vec = first_vec + 2.0f; // [ 3.0, 3.0, 3.0, 3.0 ]

std::simd::vec<float, 4> third_vec{ [](int ind){ return ind; } }; // [0.0, 1.0, 2.0, 3.0]
auto forth_vec = first_vec + third_vec; // [ 1.0, 2.0, 3.0, 4.0 ]

Some element-wise operations

Here are some useful element-wise functions that works on std::simd::vec<T>.

// element-wise min/max/minmax
std::simd::vec<int> a ([](int ind){ return 2 - i }); // [ 2, 1, 0, -1 ]
std::simd::vec<int> b ([](int ind){ return i }); // [ 0, 1, 2, 3 ]

const auto min_val = std::simd::min(a, b); // [ 0, 1, 0, -1 ];
const auto max_val = std::simd::max(a, b); // [ 2, 1, 2, 3 ];
const auto [min_val, max_val] = std::simd::minmax(a, b); // [ 0, 1, 0, -1 ], [ 2, 1, 2, 3 ];

How SIMD Usually Works in Real Code

std::simd usually works contiguous memory blocks and it does not changed the common logic of traditional simd libraries about how simd instructions are usually used in a program. It normally works liked this: You orignally have a program that works on a continugous chunk of memory and have a non-simd version of your algorithm. Most of the time it is a loop with either an index or a pointer as the control variable. How simd emerges into the place is that it will (explicitly) try to load not only 1 element, but more than 1 element given its wider bitwidth, and then do whatever simd operations on that small blob. After which the chunk is written back to to memory, or the originally loaded chunk just gooes through some reduction operation and contributes to the update of some state.

It’s nothing much different from the original scale loops version. It’s just using std::simd helps working on more than 1 elements at a step by virtue of the wide registers in your CPU.

But also, In some cases. It’s not always obvious that a scalar algorithm can be directly translated into simd-empowered implementations. Sometimes you need to rethink your algorithm because by using simd explicitly, some operations may suddenly appear less expensive such as a full pass over the whole buffer. And maybe by doing more work, you can actually make it faster because those additional work is actually easier to be parallelized via manual simd code. A prominent example in this direction is the simdjson library. Which is C++’s answer to the problem of performant json serialization.

** Note that std::simd types should be avoided in the public interface in a library design because it can cause abi problems. It’s also almost always simply an implementation-only technique and adds little value to the public interface of a library.

Interfacing with Memory

As mentioned above, std::simd is almost always an implementaion detail and it usually works on a chunk of memory. So the first import question is naturally how it loads from and writes into memory positins.

The Control Flow

Most of the time. Standard if-else should be avoided within the hot path of native std::simd kernel’s hot path. Instread, the native control flow in the simd world is std::simd::mask. As the name suggests. It works like an array of booleans that can be used to do a lot of work that resembles traditional if-else control flow. People familiar with linear algebra libraries should be fairly exposed to these kind of technique such as the mask in Numpy. It’s basically the same idea, but just in the simd world. Which works on a single register worth of data.