SIMD is available in all major CPUs and does N things at a time
What is SIMD: e.g. 16 operations per instruction
What for? More power efficient, amortizes per-instruction cost
Why Highway? Portable and reliable.
Typical example of where to use it: element-wise sum of two arrays.
Some commonly used SIMD methods are auto-vectorization and intrinsics. The former is a compiler technique that attempts to convert normal C++ into SIMD, but this often fails in nontrivial cases. The latter are built-in C++ functions that map closely to SIMD instructions, but are not portable and often verbose or non type-safe.
C++ template library with ‘portable intrinsics’
Highway is a C++ template library which allows for portable, type-safe, and readable SIMD code. It also includes a toolbox of components which are useful for high-performance code: aligned memory allocation, bit sets, dynamic dispatch, prefetching, benchmarking, performance counters, profiling, statistics for benchmarking, and timing.
The contrib/ directory also includes related libraries: a standard library subset, base64 encoding/decoding, bit packing, B-tree, dot products, hashing, an image class, integer division, math functions, matrix-vector products, random generation, sorting, OpenMP-like thread pool, and loop unroller.
Before going into each concept, we here list which ones are coming.
This tutorial is grouped into eight sections. The first two are prerequisites, everything else can be read in any order, depending on what operations your use case requires.
1) The most fundamental SIMD operations are element-wise operations on arrays (map pattern) and reductions (fold pattern), plus how to write loops and handle remainders.
2) To be able to use Highway, we must first understand some incidental complexity required by C++: tag-based dispatch for function overloading, plus namespaces and attributes, and either dynamic or static dispatch.
3) Writing SIMD implies we care about performance, hence we discuss important optimizations such as loop unrolling, restrict pointers, alignment and inlining.
4) More complex SIMD code may involve conditions. When porting if() expressions to SIMD, we use per-element ‘masks’ to skip operations on certain lanes. Masks also have specialized reductions.
5) The fixed length of vectors necessitates some changes when type-casting: we typically only halve or double the size at a time, using two or half an input vector, respectively.
6) SIMD offers special instructions for accelerating bit-shifts, dot products, integer multiplication, and saturating arithmetic. All of these differ from typical C semantics.
7) Some applications require moving data between lanes.
8) We conclude with less-common patterns including Gather/Scatter, Iota, sign manipulation and cryptography.
After working through these sections, you will be well on the way toward SIMD expertise!
Note that the sections reference source files which can be found in hwy/examples.
SIMD usually applies the same operation to all lanes simultaneously.
Apply a pure function (such as addition) to the same lane of two vectors: c[i] = f(a[i], b[i]). Lanes do not interact.
Code: sum_array_simple.cc
Use compile-time-unknown vector length as loop step to support RISC-V/SVE.
One of the key benefits of Highway is its ability to adapt the loop step size to the CPU's available vector width, using portable code. Use N = Lanes(d) as the loop increment. It is always a power of two, and constexpr on older ISAs but not on RISC-V V and Arm SVE.
Code: sum_array_simple.cc
Handle any iteration count with separate remainder implementation.
To prevent static analysis warning, use if+for structure:
size_t i = 0; if (num >= N) { for (; i <= num - N; i += N) .. } Remainder = num - i; if (HWY_UNLIKELY(remainder != 0)) { // branch prediction hint // Same loop body, but use LoadN/StoreN }
Prefer LoadN/StoreN: always safe, fast on modern ISA, slow on x86 AVX2 or earlier - but OK if used after a long loop. Later we will introduce masking (MaskedLoad).
The branch prediction hint may improve code layout by telling the compiler that there usually are no leftovers.
If you know your num >= N, and the operation being performed is idempotent (safe to repeat, e.g. ORing with 0x80), you can also replace the last iteration with a vector load starting at num - N. Note that this potentially overlaps previously processed elements.
Code: sum_array_advanced.cc
Summarize vectors into a single result (fold pattern).
Reduction applies an operation to each lane and updates the result, for example: the sum. Often requires a tree-shaped permute pattern plus the reduction operation (add).
This is usually expensive. Prefer to reduce once after a loop, not inside the loop. SumOfLanes/MaxOfLanes/MinOfLanes broadcast to each element. ReduceSum/ReduceMin/ReduceMax return a single result. GetLane also returns lane 0; this can be useful if for some reason you want both the broadcasting behavior of SumOfLanes, but also a scalar result.
Code: sum_array_advanced.cc.
Return scalar sum of array of float.
This is mainly the map pattern with loop unrolling plus reduction.
Code: sum_array_advanced.cc.
Insert/Extract is slow; you can also Print().
If we want to check the value of a particular lane, GetLane() returns lane 0 and ExtractLane(v, idx) returns the lane with the given index. The latter is fairly slow. If you are going to access multiple elements, it is probably better to Store to a temporary aligned array. But remember that this should not be done in time-critical loops, which would preferably only involve true SIMD operations.
Similarly, you can overwrite a lane using InsertLane, but if doing that for multiple lanes, it may be better to do so on an array, then Load it. However, this is still likely to be slow, because the individual stores may not forward to a larger load, leading to stalls. We recommend this only for debugging or testing.
It is common to want to see the values of a vector for debugging. print-inl.h provides a Print(d, "caption", v) function for this. There are two extra optional arguments specifying the first lane to print, and the maximum number. The latter is limited to 7, hence if the lane type is small and you want to see all of them, you can pass [0, Lanes(d)).
Code: sum_array_simple.cc.
Highway requires your code to reside in a namespace called HWY_NAMESPACE.
To enable dynamic dispatch (section below), your code must reside inside a namespace called HWY_NAMESPACE. Note that this is a macro, but it should just be used as “the namespace your code lives in”, without caring about what the macro evaluates to. Users typically also wrap their code in an additional project-specific namespace, e.g. project, to avoid clashes between their definitions and other projects, but this is optional.
Thus your .cc file will include, after the headers:
namespace project { // optional namespace HWY_NAMESPACE { // … } // namespace HWY_NAMESPACE } // namespace project
Compilers require extra annotations (pragmas) in your source code for functions using SIMD.
clang and GCC used to require compiler flags in order to use SIMD, but this no longer works if we want dynamic dispatch. Instead, compilers have started supporting #pragma. Highway offers two mechanisms for hiding both these requirements from users.
1) You can prepend a HWY_ATTR macro to each function. This is brittle and not recommended, especially when there are many functions.
2) Recommended: insert HWY_BEFORE_NAMESPACE and HWY_AFTER_NAMESPACE before/after the above namespaces. The effect of this is as if HWY_ATTR had been written before any member or free functions residing between them.
(Due to a prior compiler bug, it was necessary to place HWY_BEFORE_NAMESPACE at the root (top-level) namespace scope, hence the name. The abstraction leaks in one way: lambda functions are unfortunately not covered due to compiler limitations; they require a HWY_ATTR annotation before their opening brace.)
For users that know exactly which instructions are available on their CPU (such as personal desktops or supercomputers), Highway allows static dispatch. An #include "hwy/highway.h" is sufficient before your actual SIMD code. You will have an entry point (CallMyFunction), which then uses HWY_STATIC_DISPATCH to call your SIMD functions.
You can tell the compiler which instructions to generate using -m compiler flags such as -march=skx. Highway will use the best available ‘target’ (combination of instruction sets) enabled by the -m flags, which it detects using predefined macros. Some targets require multiple flags. For example, HWY_AVX2 requires both -march=haswell and -maes because not all Haswell CPUs support AES, but Highway guarantees AES is available if AVX2 is. You can use the list_targets.cc program to print which target is supported/enabled.
See sum_array_simple.cc for an example.
Highway strength: choosing at runtime the instruction set to use.
Runtime dispatch, aka. dynamic dispatch, means compiling multiple codepaths, one per instruction set, and choosing the best available one at runtime. This is important when deploying to unknown hardware such as cloud or user devices. Highway supports this without having to write your code multiple times. It builds a table of function pointers to each version of your code, and calls them via HWY_DYNAMIC_DISPATCH. Some boilerplate is required in addition to the static dispatch.
First, we tell foreach_target.h to re-include this file multiple times, near the top of the source file. We define a HWY_TARGET_INCLUDE macro to the name or path to the current file. This is effectively an ‘argument’ passed to and used by foreach_target.h. The source path is relative to the compiler's include path(s).
System headers can reside before this, at the top of the file. Any includes of highway.h, typically via headers we call *-inl.h, must come after foreach_target.h.
// 1. Define HWY_TARGET_INCLUDE pointing to this file #undef HWY_TARGET_INCLUDE #define HWY_TARGET_INCLUDE "path/to/this_file.cc" #include "hwy/foreach_target.h" // 2. Include highway.h AFTER foreach_target.h #include "hwy/highway.h"
Then comes the actual ‘business logic’, the SIMD portions of your code, wrapped in namespaces and attributes per the prior two sections. Let's say it defines a function called MyFunction, referenced below.
Finally, at the end of the source file, we generate the entry point into the SIMD code for use by callers outside this .cc file. Everything that should only be compiled once must reside in the #if HWY_ONCE block. This includes HWY_EXPORT, which generates a table of function pointers, one per SIMD target, and CallMyFunction, which dispatches to the version of MyFunction compiled for the best-available instruction set. This is accomplished via HWY_DYNAMIC_DISPATCH, which expands to code that queries the current CPU and calls the corresponding function pointer in the table. CallMyFunction is typically also declared in a header for calling by other .cc files.
#if HWY_ONCE namespace project { HWY_EXPORT(MyFunction); void CallMyFunction(...) { return HWY_DYNAMIC_DISPATCH(MyFunction)(...); } } #endif
Highway ops require namespace-using, or hn:: prefix in front of the function calls.
Highway ops (functions) reside in namespace hwy::HWY_NAMESPACE. If your project namespace is chosen to be hwy, then you can call Highway ops directly like normal functions.
Otherwise, thanks to C++ lookup rules (Argument Dependent Lookup), functions with arguments residing in a namespace are automatically found. This means functions with a tag argument (see next section) such as Load can also be called directly. On some platforms, this also works for vectors if they are wrapper classes defined in the same namespace. However, on others (SVE and RISC-V) vectors are built-in types. Thus functions such as Add which have only vector arguments would not be found. Users must instead call hwy::HWY_NAMESPACE::Add, which is typically abbreviated with namespace aliases to hn::Add after a namespace hn = hwy::HWY_NAMESPACE;. We recommend this prefix for all ops, even Load, for consistency.
One possible alternative is to add using hwy::HWY_NAMESPACE::Add; directives for every function that will be called, but missing directives are hard to spot and will cause build failures on other platforms, making this a more brittle approach.
Overloaded functions instead of classes.
For technical reasons, the Highway API is based on overloaded functions, which have the same name, but differing argument types. Highway uses an empty ‘tag’ struct to select the correct function.
(Most other libraries are instead built around vector class wrappers, but this does not work for scalable RISC-V V and Arm SVE vectors because compilers do not allow such vectors to be class members.)
Let D denote the ‘tag type’, most commonly using D = hn::ScalableTag<T>;, where T is the desired lane type. Your SIMD code typically starts by declaring an lvalue d of type `D`. Note that hn::Vec<D> is the actual type of a vector. In specific situations, e.g. when using AES (which often uses 128-bit vectors), we instead want using D = hn::FixedTag<uint8_t, 16>;.
Using this d, we can now construct vectors. The function hn::Set(d, val) returns a vector with all lanes set to val. hn::Zero(d) is shorthand for hn::Set(d, T{0}).
Compilers also do not allow arrays of vectors. If you want that, use arrays of the lane type instead. What is allowed: local vector variables, returning vectors, and pointers/references to vectors.
You are now ready to write your first Highway code! We strongly recommend you copy an existing example such as hwy/examples/skeleton*, because the order of these components is important.
The simplest approach is to write SIMD code entirely contained within a single .cc file. This section is only relevant if you want to reuse SIMD code between multiple .cc files.
Due to the multiple-compilation approach (see Dynamic Dispatch), special header include guards are required. Highway toggles the HWY_TARGET_TOGGLE macro between defined and undefined between every compilation. As a result, the include guard will be re-enabled for that target, but only once because the user code also toggles its include guard.
For more information and guidance, see “When to use -inl.h” in impl_details.md. See skeleton-inl.h for an example.
#if defined(HWY_PATH_NAME_INL_H_) == defined(HWY_TARGET_TOGGLE) // Flip the is-defined state of the ‘include guard’ #ifdef HWY_PATH_NAME_INL_H_ #undef HWY_PATH_NAME_INL_H_ #else #define HWY_PATH_NAME_INL_H_ #endif HWY_BEFORE_NAMESPACE(); namespace hwy { namespace HWY_NAMESPACE { // Header contents // NOLINTNEXTLINE(google-readability-namespace-comments) } // namespace HWY_NAMESPACE } // namespace hwy HWY_AFTER_NAMESPACE(); #endif // per-target include guard
Data type used for comparison results, with potentially different representation than vectors.
Highway provides a Mask type for the result of comparison ops such as Gt/Eq. This is because CPUs differ in how they represent these results:
To bridge these differences, Highway provides a rich set of operations operating on masks:
We can convert between vectors and masks using VecFromMask/MaskFromVec, assuming vector is all-1-bits or all-0. MaskFalse(d), SetMask(d, bool): initializes all elements to same bool FirstN(d, num): true for first Lanes(d) elements, false otherwise RebindMask: type-cast e.g. int32 to float (same number of lanes)
Can also convert to/from bit array: LoadMaskBits/StoreMaskBits. For <= 512-bit vectors BitsFromMask is also available.
Code: masks_and_logic.cc.
Per-lane “maybe do something” decisions.
Masks are most typically used to “select/blend” between two vectors, akin to the ternary operator in C++. This is expressed as IfThenElse, or its variants IfThenElseZero/IfThenZeroElse which have an implicit zero operand.
Masks can also be used to skip certain lanes during ops such as MaskedAddOr; this is especially relevant for division.
The convention is that an “Or” suffix means ‘use the first argument if mask is false’, and otherwise, the default value is zero. A common pattern for conditional addition is val = MaskedAddOr(val, condition, val, increment), corresponding to if (condition) val += increment.
Masked comparisons can also be useful for fusing an AND (e.g. with FirstN so that only valid lanes are considered) into the comparison result, for example MaskedEq.
MaskedLoad conditionally loads, or at least zeros out the result. Note that this is unsafe for unaligned vectors that may cross a page boundary into an unmapped page, if HWY_MEM_OPS_MIGHT_FAULT is true. AMD AVX2 CPUs indeed reserve the right to page-fault in this case. We can avoid this problem by ensuring vectors are aligned, or using LoadN instead for the common case where we want to load no more than the given number of elements (e.g. in the last loop iteration).
Code: masks_and_logic.cc.
No per-element branching, can only decide on the entire vector.
Highway provides special Reduction-like operators for masks:
AllTrue, AllFalse: decide if no matching lane found
Can also ‘reduce’ masks: CountTrue, FindFirstTrue/FindLastTrue.
As with reductions, these are slower than elementwise operations and should preferably be executed only rarely, e.g. after a loop.
Code: mandelbrot.cc.
Highway has a mini-language for compile-time type manipulation.
Simple programs use only a single type. However, libraries may accept a tag argument which determines the vector length, and then want to derive from that new tags for different types or vector lengths.
Typically we use RebindToSigned<D> to generate an additional type tag for signed ints alongside e.g. a float tag, and RepartitionToWide<D>/RepartitionToNarrow<D> for wider/narrower types. You can also Rebind<T2, D> or Repartition<T2, D> for any new type T2. These templates change the type of the tag to T2; Rebind keeps the same number of lanes (hence T2 must be the same size as the previous T), and Repartition keeps the overall vector size the same, which means the number of lanes halves when T2 is twice as large.
Half<decltype(d_full)> and Twice<D> respectively halve and double the number of lanes in a vector.
Sometimes we do not even have a tag argument. Tags can be deduced from vector arguments: DFromV<decltype(v)> d; . For symmetry, we also provide VFromD<D> which is equivalent to Vec<D>. You can also obtain the lane type from a tag with TFromD<D>.
Code: see float_distribution.cc.
We can split/combine vector halves, for example when type-casting (see below).
LowerHalf(v) returns a half-vector. UpperHalf(d, v) requires a tag argument (which is optional for LowerHalf). Combine(d, hi, lo) rejoins these halves. ZeroExtendVector(d, lo) is equivalent to Combine(d, Zero(Half<decltype(d)>()), lo)
Code: baker_mix.cc
Half/twice as many lanes for promotion/demotion.
‘Promote’ means a widening cast and ‘demote’ means narrowing.
Most ISAs have a native PromoteTo, which promote the lower half of an input (e.g. int16) into a full output vector (e.g. int32). DemoteTo similarly takes one input vector and return only a half-vector.
Similarly, PromoteMaskTo/DemoteMaskTo are provided for masks.
Code: skeleton.cc.
BitCast treats the underlying representation as a different type.
Sometimes we want to operate on the bit representation of floats, or Gather pairs of smaller types. This requires treating the bits in a vector as if they were a different type. Unlike PromoteTo/DemoteTo/ConvertTo, BitCast does not execute any instruction. It only returns a copy of the same vector bits, but reinterpreted as another type.
There are several forms of bit-shifts.
Historically, CPUs preferred shifts of the form x << 13, where the shift amount is hard-coded into the instruction. This is expressed as ShiftLeft<13>(x). More recently, support was added for x << y, where the shift amount is runtime variable and also per-lane. This is expressed as Shl(x, y). There is also an intermediate form which accepts a runtime variable, applied to all lanes: ShiftLeft(x, scalar).
Highway also supports bit rotation: RotateLeft, Rol, RotateLeftSame. All of the above also have variants for rightward shifts/rotations (e.g. ShiftRight).
Code: float_distribution.cc.
We can avoid slowdowns caused by ISA differences for use cases that tolerate lane reordering.
For a dot product, we only want the sum and are willing to reorder the lanes (even though this can slightly change the results, see Loop unrolling). This helps deal with differences in the ISA. For example, NEON typically processes the upper or lower half of a vector, whereas SVE prefers to process odd or even-indexed lanes.
Demoting two vectors (e.g. int16) into one full output vector (e.g. int8) can be more efficient than DemoteTo, but is not supported for all types. There are two variants: OrderedDemote2To preserves the lane order, whereas ReorderDemote2To is allowed to change order (useful we we know all lanes are equal, or we will later reduce).
Code: float_distribution.cc.
CPUs provide a fused promote/widen/multiply/add operation crucial for ML workloads.
As of 2026, this is the most important operation for ML, and has one of the highest compute densities of any instruction. sum0 = ReorderWidenMulAccumulate(d, a, b, sum0, sum1) multiplies a[i] * b[i] after promoting them to the next wider type, and then adds the result to some lane of (see Reorder pattern) sum0 or sum1. It returns the updated sum0 and updates sum1 in-place. This reordering is fine for dot products. If you want to guarantee a certain order, namely that each wider lane of the result contains the result of multiplying and then pairwise-adding the two narrow lanes that cover the wider lane, then this will be the return value of RearrangeToOddPlusEven(sum0, sum1), which should be called outside the loop. We currently support this operation for signed/unsigned int16 and BF16; FP16 is anticipated to be added. Due to the high practical relevance of int8*int8 + int32, CPUs also support a related operation that multiplies and then adds four adjacent 8-bit lanes. This is SumOfMulQuadAccumulate.
Code: dot_product_mixed_precision.cc
CPUs provide special instructions for integer multiplication.
With integer arithmetic, multiplying 8-bit numbers produces a 16-bit result. How does this co-exist with the fixed size of a vector? One possibility is returning the lower or upper half of the result using Mul and MulHigh, respectively. Another is that MulEven returns a double-width result for every even lane. Similarly, but less likely to be supported natively, MulOdd multiplies every odd lane and places the result into the lane pair whose upper half it is.
Code: mandelbrot.cc
Integer overflow is well-defined in SIMD, but sometimes we want to detect/prevent it.
Addition and subtraction always wrap around for both signed and unsigned types. It can be useful to instead have sums ‘stuck’ to the largest or smallest value - either to check whether overflow happened, as a debug aid, or as part of a calculation. For example, broadcasting the sign bit of a byte can be accomplished via signed min, which replaces positive values with zero, then unsigned saturating addition with itself. If the sign bit was set, then the sum would overflow and is instead replaced with the largest unsigned value, 0xFF; otherwise, it remains zero. Saturating add/subtract is only supported for 8 or 16-bit elements, and is expressed as SaturatedAdd/SaturatedSub.
Code: sum_hex.cc.
Elementwise operations are not always sufficient. Note that the CUDA programming model attempts to pretend that there is only a simple element at a time, but ‘warp shuffles’ were added to allow passing data between adjacent elements. Highway explicitly acknowledges the desirability of such movement, which is often called shuffling or swizzling, and guarantees there are at least 128 bits of elements in a full vector. Note that they tend to be slower (higher latency) than simple logic or integer arithmetic, but they also tend to run on different functional units, hence might be able to run in parallel with other computations.
CPUs efficiently support gather operations within one or two registers.
(This section is the most general case of cross-lane movement; the following sections offer specializations which can be faster if they match the desired pattern.)
All CPUs have good support for looking up one of 16 bytes, within 128-bit blocks. This is expressed as TableLookupBytes: the result for a lane i is table[indices[i]], which requires indices[i] < 16. In other words, each index lane says which input lane to take to produce the resulting lane. You can also request that the output should be zero if the index is >= 128 using TableLookupBytesOr0.
Modern ISAs also support other element types, expressed as the TableLookupLanes op. Except for RISC-V, lookups are usually constant-time, and much faster than Gather/Scatter. TwoTablesLookupLanes concatenates two table registers. When not directly supported by the CPU, Highway emulates these ops via TableLookupBytes. This may require some preprocessing, which is why indices must be obtained from registers or pointers using IndicesFromVec or SetTableIndices, respectively.
It can be complicated to reconcile large tables with unknown vector lengths. It is generally safe to assume 128-bit vectors (except for the deprecated HWY_SCALAR target), which when concatenated yields 256-bit tables. Highway provides a Lookup8 (plus variants for 16/32/64 elements) which can be more efficient (may be able to use a single register) as well as more powerful (may support larger tables) than using TwoTablesLookupLanes directly. The prerequisite is that HWY_MIN_BYTES / sizeof(T) should be at least 4. This is always the case for 32-bit T.
Code: sum_hex.cc.
Partition: special form of Swizzling driven by predicates.
QuickSort/Select involve a partition step that moves to the front of an array all elements that match a predicate. Special Compress instructions compact all lanes whose predicate is true. You can StoreU those to memory and increment by CountTrue(d, mask), or better yet use the fused CompressStore(v, mask, d, ptr) which returns the number of elements written.
Expand is the opposite: depositing consecutive elements at the lanes whose predicate is true. We also provide a fused LoadExpand. This is useful for unpacking data that was compressed by omitting certain elements (e.g. zero-valued or negative etc.).
Code: float_distribution.cc.
For historical reasons, many swizzles split into separate 128-bit blocks.
To avoid overhead on x86, many Highway operations involving cross-lane movement are defined separately for each 128-bit ‘block’. The whole-vector variants may be more convenient, but often slower.
For example, Broadcast<0> copies into all other lanes - but per 128-bit block. With AVX2 and int64, lane 0 is copied to lanes 0..1, but lane 2 is copied to lanes 2..3. This behavior is somewhat surprising, and the quick_reference.md calls it “blockwise”. Conversely, we also provide ops defined across the entire vector, without being split into blocks. We now list similar ops and their whole-vector equivalents:
Broadcast | BroadcastLane TableLookupBytes | TableLookupLanes, TwoTablesLookupLanes InterleaveLower | InterleaveWholeLower InterleaveUpper | InterleaveWholeUpper ShiftLeftBytes/ShiftLeftLanes | SlideUpLanes ShiftRightBytes/ShiftRightLanes | SlideDownLanes
Some blockwise functions do not yet have whole-vector equivalents, including InterleaveEven/InterleaveOdd and CombineShiftRightBytes/CombineShiftRightLanes.
Code: benchmark.cc
Supporting ‘stencils’ (computations involving neighboring elements) via loads or swizzles.
In image processing (filters) or scientific computations (fluid dynamics), computations typically involve neighboring elements. The simplest approach is to LoadU again with an index offset of +1 or -1. This can actually be performant on most CPUs for small stencils (just two neighbors), assuming the array is padded such that it is safe to load an entire vector.
For larger stencils or if there are already many loads, it can be better to synthesize the vectors of left/right neighbors from aligned loads. CombineShiftRightBytes does exactly this, but blockwise, hence it is only practical to use with vectors capped to 128 bits. The more recently added Slide1Up/Slide1Down ops could also be helpful because they operate on the whole register, not blocks, but they insert a zero into the bottom/top lane, respectively, which is likely not what the application requires, and may require extra blending/insertion to fix. SlideUpLanesOr(low, d, v, 1) solves this by inserting a value from another vector low into the lower lane of v.
Code: benchmark.cc, game_of_life.cc see also Tim Mattson's heat.c.
Inlining small functions is extremely important for avoiding stack frame setup overhead.
Compilers decide whether to ‘inline’ functions (insert their code into the caller), or instead generate a call to the subroutine. The latter can lead to more compact code if the subroutine would be called multiple times. However, the compiler might not have accurate knowledge of whether this happens. Hence Highway allows specifying HWY_INLINE or HWY_NOINLINE in front of functions to forcibly enable or disable inlining. We recommend HWY_INLINE for functions only called once, or small functions (a few instructions). Conversely, HWY_NOINLINE should be used for large functions, or for the top-level function in a .cc file. The latter prevents warnings/errors about mismatches between targets during inlining, relating toHWY_ATTR (see Required Attributes).
Manually unrolling loops is a very important optimization.
CPUs can often issue 2, 4, or even 6 vector operations per cycle, but it usually takes 2-4 cycles until the result is complete. If we have a loop updating a single sum = Add(sum, Load(d, p) per iteration, then the next iteration must wait these 2-4 cycles until the result is ready. This underutilizes the CPU. Compilers are usually unable to fix this because adding several sums can change the results due to numerics (floats are not associative, and adding separate sums changes the order of additions). Thus it is up to the programmer to manually introduce independent sum variables, update them in an unrolled loop, and then combine them into a single result after the loop. Moreover, because compilers may disallow arrays of vectors, this really does require individual variables. On the plus side, autocomplete is quite good at updating indices sum0, sum1 etc. after copy-pasting. This optimization is also very worthwhile, with potential for 2-5x speedups. Choose the number of accumulators as the throughput (instructions per cycle) times the latency (number of cycles until a result is ready), typically 4 or 8. We also want to ensure that all these sums, plus any active variables from each unrolling of the loop, fit within the typically 32 registers provided by the architecture.
Code: dot_product_unroll.cc.
Annotate pointers which are the only way to access an object/array.
The presence of pointers in C++ makes it difficult for compilers to reason about object lifetimes and references. For example, if a pointer is used to change an object, any local copies made from that object must be re-loaded. As a result, compilers may pessimistically always reload the object rather than keeping it in a register, leading to larger and slower code. It is good practice to annotate pointers using HWY_RESTRICT whenever possible. This tells the compiler that the pointer is the only way to access the object to which it points. Hence the compiler can keep the object in a register as long as that pointer is not modified. HWY_RESTRICT should not be used when multiple pointers point to the same object. Thus the presence of this modifier is helpful documentation for readers, in addition to its benefits for the compiler.
Vectors should usually be aligned for performance.
Vector-aligned means their address is a multiple of their size. Sizes are powers of two, so aligning to large pow2 is enough. Use aligned_allocator.h for allocating dynamically, and HWY_ALIGN for stack arrays. Load may crash if not aligned, to ensure this is noticed. LoadU does not crash. Other memory-access functions including LoadN/CompressStore do not require alignment.
The main allocation API is auto arr = hwy::AllocateAligned<float>(count); // returns unique_ptr. This aligns to a multiple of 128 bytes, with some effort to avoid cache conflicts caused by pointers always being congruent modulo some power of two (i.e. having common factors of 2048 or 4096, which look the same to parts of the CPU, which must thus assume they conflict).
Separately, even ‘unaligned’ vectors should still be aligned to their element size, otherwise C++ considers this UB. HWY_RCAST_ALIGNED tells the compiler that a byte pointer is actually aligned to the element size, and should be used instead of C++ reinterpret_cast to avoid warnings.
Note that you might not see a speedup from aligning for some code on some CPUs. For example, x86 does a good job of hiding the extra cost in many cases. However, if your code is limited by accesses to the L1 cache, pointers that happen to be unaligned may halve your performance.
Code: benchmark and ctf_aes
For best results, choose planar (Structure of Array) data layouts.
Elementwise operations in SIMD tend to require the same type of data in each element. Hence pixel or point data should have red channel or x coordinate in all elements of one register. Unfortunately, some image formats are defined as red, green, blue channels, and some input data may be arranged as Point2D {x, y}. These are known as Array of Structure. If at all possible, these should be replaced with planar, Structure of Array layouts.
When this is infeasible, Highway's LoadInterleaved2/LoadInterleaved3/LoadInterleaved4 ops can deinterleave (for example) XY, RGB, or RGBA data into 2, 3 or 4 registers, each containing the same type of component. Conversely, StoreInterleaved2/StoreInterleaved3/StoreInterleaved4 interleaves planar vectors and stores to memory in that format. These are likely much faster than Gather/Scatter.
Code: baker_mix.cc and mandelbrot.cc
CPUs can often perform multiple operations per instruction.
For good reason, SIMD instruction sets are more complex than a pure “RISC” (reduced instruction set). Fusing two or even three operations into a single instruction reduces the number of instructions. Unfortunately, CPUs differ in what exactly they can fuse. Highway includes a carefully chosen set of operators which benefit from commonly available fusions without penalizing other processors that lack them. The most important are fused-multiple add/subtract, which are also more accurate in addition to faster. Note that some compilers may automatically contract Mul+Add into MulAdd, but we recommend using the pre-fused form for safety and pattern-matching for readers.
MulAdd(f, m, a) = Add(Mul(f, m), a) MulSub(f, m, s) = Sub(Mul(f, m), s) AbsDiff(a, b) = Abs(Sub(a, b) AndNot(a1, a2) = And(Not(a1), a2) OrAnd(o1, a1, a2) = Or(o1, And(a1, a2) Xor3(x1, x2, x3) = Xor(x1, Xor(x2, x3) Or3(o1, o2, o3) = Or(o1, Or(o2, o3) - Xor3 should be preferred if applicable. XorAndNot(x1, a1, a2) = Xor(x1, AndNot(a1, a2)) AndXor(a, x1, x2) = And(a, Xor(x1, x2)) IsNegative(v) = Lt(v, Zero(DFromV<decltype(v)>())) IsEitherNaN(a, b) = Or(IsNaN(a), IsNaN(b)) CompressStore(v, m, d, p) = StoreU(Compress(v, m), d, p) LoadExpand(m, d, p) = Expand(LoadU(d, p), m) ConcatLowerLower(d, hi, lo) = Combine(d, LowerHalf(hi), LowerHalf(lo)) ConcatUpperUpper(d, hi, lo) = Combine(d, UpperHalf(dh, hi), UpperHalf(dh, lo)) ConcatUpperLower(d, hi, lo) = Combine(d, UpperHalf(dh, hi), LowerHalf(dh, lo)) ConcatLowerUpper(d, hi, lo) = Combine(d, LowerHalf(dh, hi), UpperHalf(dh, lo)) LoadDup128(d, p) = BroadcastBlock(ResizeBitCast(d, LoadU(FixedTag<T, 16/sizeof(T)>(), p)))
Also, all the Masked\* functions are arguably fused, and typically fold an AND into the operation.
Code: dot_product_unroll.cc.
These are less common, but sometimes necessary.
You can also request ‘partial vectors’.
Highway encourages scalable vectors: whenever possible, user code should conform to the number of lanes provided by the CPU. However, there are use cases where we may want to use fewer lanes. One option is to cap the number of lanes to a given upper bound. Highway only guarantees ‘as-if’ behavior: loads/stores do not exceed the given amount, but vectors may still be larger and perform operations on all elements. One common example is when ‘reducing’ (see Reduction) four separate vectors. We can first StoreInterleaved4 to a buffer, then load groups of four values at a time and reduce them e.g. into a global sum. This has the advantage of still partially vectorizing (four lanes at a time) without vector-length-specific handling.
This pattern requires caution because capped only guarantees Lanes(d) does not exceed the cap. However, if you test on a CPU that actually has that many lanes, your code may rely on this and then break when running on CPUs with shorter vectors. It is thus safer to use FixedTag<T, 16/sizeof(T)>. Such 128-bit vectors are guaranteed to be supported on all Highway targets (except the deprecated HWY_SCALAR). One common use case is a prefix sum of 32-bit lanes. This involves a tree of additions which relies on the fact that there are exactly four lanes. Another use case is AES-128, which requires vectors of exactly 16 bytes.
Code: ctf_aes.cc.
Generate ascending integer sequences using Iota.
Sometimes we want to generate consecutive integers 0..N-1, or more generally j..j+N-1. This can be done with Iota(d, j). It also works for floating-point types, for which the integer j is converted to that type.
This is most commonly used as inputs to tests, or when vectorizing a loop which uses/stores the value of the loop counter, for example argmax. Another interesting usage is to turn an index into a ‘one-hot’ representation, where a single predicate is true. For example, Eq(Set(d, val), Iota(d, 0)) is true for the lane whose index is equal to val.
Code: baker_mix.cc
Elementwise loads/stores: slow, but sometimes required for vectorizing a loop.
For SIMD, we strongly prefer when lanes are loaded/stored to consecutive memory addresses. If not, e.g. addresses depend on a different value in each lane, then loading/storing is called a Gather/Scatter. This is much slower than a regular Load/Store and should be avoided whenever possible. However, it can still make sense if the values being produced or consumed require considerable (SIMD) computation before/after the Gather/Scatter. Conversely, it is not worthwhile to use a Gather/Scatter instruction simply to perform the memory loads/stores. CPUs can do about as well just using normal single-element instructions.
Two variants exist: GatherIndex is when the vector represents array indices: base + indices[i]. GatherOffset is for byte offsets.
Code: matrix_transpose_scatter_gather.cc.
Transposing matrices is a common operation which is difficult to make vector-agnostic.
One tempting approach is to use GatherIndex with a vector equal to Mul(Iota(di), Set(di, stride)). This works, but is not especially fast, though likely slightly better than ScatterIndex. For width-4 or 8 transpose, it is usually faster to instead use StoreInterleaved4, which permutes registers and then performs normal stores.
Code: matrix_transpose_scatter_gather.cc.
Highway provides various ops to read/modify floating-point signs.
CopySign and its more efficient variant CopySignToAbs (when the first operand is non-negative) set the sign to that of the second argument. These are useful for implementing math functions.
Neg and Abs (also available for integers) negate and return the absolute value.
Code: float_distribution.cc
AESRound is the fastest way to thoroughly scramble data for hashing/encryption/RNG.
Highway provides ops to accelerate the AES symmetric block cipher, used in Wifi and TLS. AESRound and AESRoundInv each implement one ‘round’ of encryption/decryption. They are also useful for implementing a fast random generator (AesCtrEngine) or long-string hash.
Code: ctf_aes.cc.
Highway also provides ops to exercise limited control over the otherwise transparent cache.
The cache hierarchy is critical for performance, but it is mostly managed transparently. Highway provides a few means of influencing it directly. Stream writes to aligned memory, bypassing the cache. This is sometimes useful if we know the data will be write-only and not accessed again, because it avoids evicting useful data in favor of the write-only data. After all such writes and before loading, the FlushStream op must be called to ensure the writes are visible on the current core (this has nothing to do with cross-core synchronization, and provides no guarantees in that regard).
Prefetch requests that the memory system bring the data into cache(s). The CPU typically already does this automatically, especially when the access pattern is predictable, for example an ascending or descending sequence. However, your program might know before the actual load what the next addresses will be. In particular, unpredictable sequences as seen in sparse data structures might benefit from prefetching already while processing the previous data. However, one vexing question is how far in advance to prefetch, which depends on the CPU. A good answer might be to auto-tune the best distance at runtime (measure what works best).
You can also FlushCacheline, forcing the cache line to be written back to memory. This is rarely done in application software, but useful for systems software such as device drivers.
We advise first making sure that your code is efficient on a single thread/core. As a second step, utilizing other cores is necessary to maximize throughput. Modern server CPUs have a surprising number of cores: 128 for AMD Turin, twice that for Venice (coming in 2026). This leads to two concerns: first, each core's share of the per-socket memory bandwidth is small and shrinking. Second, keeping a coherent view of shared memory entails overhead. Ideally, each core will work on its own, cache-sized data. This solves both problems by entirely minimizing traffic on the narrow links to memory and other chiplets. This ideal is not always feasible in practice, but staying as close as possible to a ‘shared-nothing’ model of independent cores and data is helpful.
How to parallelize? This is mostly orthogonal to the SIMD code, except that the distribution of work should still be a multiple of the vector (and preferably cache line/page size). Although out of scope of this tutorial, we briefly mention some options:
OpenMP introduced a successful approach where you add minimal source code annotations right before your loop. This is very often sufficient, including where reduction is required. Where available, we suggest using OpenMP because it is standardized, simple and well-known. Some environments do not support OpenMP and instead provide a ParallelFor function that loops over work items [0, N). For example, Highway provides a ThreadPool class which is sufficient for many use cases.
Code: stream.cc
Other frameworks for specific patterns or use cases include:
This tutorial focuses on SIMD, not all the topics required for performance. However, there are two interactions between threading and SIMD:
1) The start/end of ranges processed by each thread should be vector-aligned, and better yet cache-aligned to prevent conflicts, unalignment or remainder handling.
2) SIMD benefits from aligned allocations, but these are not always aware of NUMA effects (e.g. multiple CPU sockets). A simple and mostly sufficient workaround is to have each thread allocate the memory it will use, e.g. via hwy::AllocateAligned. Note that allocation can be slow and should preferably be done only once, during initialization.
It can be beneficial to align very hot loops to 32 or 64 bytes, but this is highly CPU-specific. When using clang, you can insert [[clang::code_align(64)]] before the loop to see if it helps.