## Overview Types used here follow `[VET][ISUF]?\d?`. First character: - `V`: some vector type - `E`: some scalar type - `T`: either scalar or vector type Second character, if present: - `F`: float - `S`: signed integer - `U`: unsigned integer (may include `u1`) - `I`: any signedness integer (may include `u1`) Equally-named types should match. A number is added to the type to separate unequal types. `mt{T}` is a mask version of the argument - either `ty_u{T}` or `[vcount{T}]u1` depending on the target. The SIMD operations listed aren't guaranteed to be supported on all targets, nor list all possible argument types. ## General definitons - `has_simd` `0` or `1` depending on whether SIMD types are available. - `arch_defvw` Target vector width for the current target architecture. If `has_simd==0`, should not be used. Otherwise, vectors of all element types (except `u1`) whose width is this must be supported, but other widths (both larger and smaller) may be too. - `reinterpret` is available as `~~` - `exportN{f, 'name1', 'name2', ...}` - export single function under many names - `exportT{'name', fs}` - export list of functions as the specified name - `iota{knum}` - tuple `tup{0,1,...,knum-1}` - `broadcast{knum,v}` (`knum**v`) - tuple with `knum` items, all `v` - `tern{c, t, f}` - ternary - if `c`, return `t`, else `f` - `eachx{F, ...args}` - `each` but broadcasting non-tuples - `undef{T}` - return register of type `T` with undefined value; maps over a tuple of types - `undef{T, n}` - a tuple of `n` different undefined registers - `oneVal{xs}` - assert that `xs` is a tuple of equivalent (as per `==`) items, and return the first - `oneType{xs}` - `oneVal` but over the types of the items ## Loops & branches - `@for` - generate body once, with indices `s,s+1,...,e-2,e-1` - `@forNZ` - `@for`, but assumed to run at least once - `@for_backwards` - `@for`, but indices go backwards `e-1,e-2,...,s+1,s` - `@forUnroll{exp, unr}` - generate a main loop body that takes a tuple of `unr` indices to process. Handle the tail via processing one index at a time; if `exp==1`, that's done via `unr` generated bodies, otherwise it's a loop - `@unroll(n)` - generate body `n` times - `@collect(n)` - generate body `n` times, and collect the results as a tuple - `makeBranch{Ts, G=={...x:Ts}=>{}} == {...x:Ts}=>{}` - make a code segment that takes arguments of types `Ts`, and can be jumped to by invoking the result of the `makeBranch` - `makeOptBranch{enable, Ts, G}` - `makeBranch` but only generated & invokable when `enable==1` ## Scalar operations - `popc{x:EI} : uint` - population count - `popcRand{x:EI} : uint` - population count; under valgrind, return a random value in the set of possible ones if the input is partly undefined - `ctz{x:EI} : uint` - count trailing zeroes; if `x==0`, the result is undefined - `clz{x:EI} : uint` - count leading zeroes; if `x==0`, the result is undefined - `clzc{x:EI} : uint` - `width{EI}-clz{x}`; if `x==0`, the result is undefined - `zlow{n, x}` - zero low `n` bits of `x` - `tail{n, x}` - keep low `n` bits of `x` - `bit{n, x}` - get `n`-th bit of `x` - `lb{n}` - base-2 log of a constant power of two - `base{b, x}` - convert tuple to number in little-endian base `b` - `tree_fold{G, xs}` - fold tuple `xs` with a tree-like pattern with `G` ## Hints - `rare{x:u1} : u1` - Expect that `x` is `0`. Returns `x`. - `likely{x:u1} : u1` - Expect that `x` is `1`. Returns `x`. - `assert{x:u1} : void` - assume `x` is `1`. If `x` is a compile-time constant, fails to compile if it's not `1`. If at runtime it is not `1`, undefined behavior happens (unless debug mode, via either `include './debug'` or CBQN-wide `DEBUG` mode) ## Type operations Of the below functions which take & return vector types of the same width, they can instead take a vector value & reinterpret it to the result type. - `w_n{E, w} == E2` - a scalar type with the quality of `E` and width `w` - `w_d{E} == E2`, `w_h{E} == E2` - double/halve the width of a scalar type - `n_d{V} == V2`, `n_h{V} == V2` - double/halve element count in vector (keeps the same element width) - `el_d{V} == V2`, `el_h{V} == V2` - double/halve element width in vector (keeps the same count) - `el_m{V} == V2`, `el_s{V} == V2` - vector with the same total width, but element width doubled (`el_m` - elements merged) or halved (`el_s` - elements split) - `ty_u{T} == TU` - unsigned integer version of the type - `ty_s{T} == TS` - signed integer version of the type - `ty_f{T} == TF` - float version of the type - `re_el{E, V} == V2` - reinterpret vector to one with the specified element - `eltype{V2}==E and width{V2}==width{V}` - `minvalue{EI}`, `maxvalue{EI}` - smallest & biggest value of the specified type ## Arithmetic Some may also support one scalar argument or arguments with different widths. - `__neg{a}` - `-a` - `__add{a:V, b:V} : V` - `a + b` - `__sub{a:VI, b:VI} : VI` - `a - b` - `__div{a:V, b:V} : V` - `a / b` - `__mul{a:V, b:V} : V` - `a * b` - `mulHi{a:VI, b:VI} : VI` - high half of multiplication - `__eq`, `__ne`, `__ge`, `__gt`, `__le`, `__lt` - comparison; `{a:V, b:V} : mt{V}` - `__not{a:VI} : VI` - `~a` - `__and{a:VI, b:VI} : VI` - `a&b` - `__or{a:VI, b:VI} : VI` - `a|b` - `__xor{a:VI, b:VI} : VI` - `a^b` - `andnot{a:VI, b:VI} : VI` - `a&~b` - `__shl{a:VI, b:TI} : VI` - `a<>b`; arithmetic shift right - `__shr{a:VU, b:TI} : VU` - `a>>b`; logical shift right - `abs{a:V} : V` - absolute value - `absu{a:VI} : ty_u{VI}` - absolute value, reinterpreted as unsigned - `adds{a:VI, b:VI} : VI` - saturating add - `subs{a:VI, b:VI} : VI` - saturating subtract - `rcpE{a:VF} : VF` - estimate of `1/a`; precision depends - `rsqrtE{a:VF} : VF` - estimate of `sqrt{1/a}`; precision depends - `sqrt{a:VT} : VT` - square root - `floor{a:VF} : VF` - floor - `ceil{a:VF} : VF` - ceiling - `max{a:V, b:V} : V` - minimum value; arch-specific behavior on NaNs - `min{a:V, b:V} : V` - maximum value; arch-specific behavior on NaNs - `shl{U, a:V, n}` - shift vector elements within blocks of `U` - `shr{U, a:V, n}` - shift vector elements within blocks of `U` - `unord{a:VF, b:VF} : mt{VI}` - `(a==NaN) | (b==NaN)` - `andAllZero{a:VI, b:VI} : u1` - whether `a&b` is all zeroes ## Structural operations - `broadcast{V, a:eltype{V}} : V` (`V**a`) - broadcast (splat) the scalar to all elements of the result - `make{V, ...vs} : V` or `make{V, tup{...vs}}` - initialize a vector with the specified elements - `iota{V} : V` - `make{V, range{vcount{V}}}` - `extract{a:V, n} : eltype{V}` - extract the n-th element - `insert{a:V, i, b:eltype{V}} : V` - a copy of `a` with the `i`-th element replaced with `b` - `half{a:V, n} : count_half{V}` - extract either the low (`n==0`) or high (`n==1`) half of `a` - `pair{a:V, b:V} : count_dbl{V}` - make a vector from halves - `undefPromote{V2, a:V} : V` - promote `a` to a wider type, leaving the upper part undefined - `vshl{a:V, b:V, n} : V` - `vshl{[0,1,2,3], [4,5,6,7], 1} → [1,2,3,4]` - `sel{VI1, a:V, b:VI2} : V2` - shuffle `a` by indices `b` in `VI1`-long lanes; arch-specific behavior on out-of-bounds values - `zip{a:T,b:T} : tup{T, T}` - `zip{[0,1,2,3], [4,5,6,7]} → tup{[0,4,1,5], [2,6,3,7]}` - `mzip{a:T,b:T} : tup{el_dbl{T}, el_dbl{T}}` - reinterpreted `zip{a, b}` - `zipLo{a:T,b:T} : T` - `tupsel{0, zip{a, b}}` - `zipHi{a:T,b:T} : T` - `tupsel{1, zip{a, b}}` - `mzipLo{a:T,b:T} : T` - `tupsel{0, mzip{a, b}}` - `mzipHi{a:T,b:T} : T` - `tupsel{1, mzip{a, b}}` ## Mask stuff Homogeneous definitions (i.e. ones with `hom` in their name) assume that each element in the mask type has all its bits equal. - `homAll{a:VI} : u1` - whether all elements are set - `homAny{a:VI} : u1` - whether any element is set - `topAny{a:VI} : u1` - whether all elements have their top bit set - `topAll{a:VI} : u1` - whether any element has its top bit set - `homBlend{f:V, t:V, m:mt{V}} : V` - blend by `m`, setting to `f` where `0` and `t` where `1` - `topBlend{f:V, t:V, m:V} : V` - blend by top bit of `m` - `homMask{a:VI} : uint` - integer mask of whether each element is set (assumes each element has all its bits equal) - `topMask{a:VI} : uint` - integer mask of the top bit of each element - `homMaskX{a:VI} : tup{knum, uint}` - integer mask where each element is represented by `knum` bits (possibly more efficient to calculate than `homMask`) - `ctzX{tup{knum, uint}}` - count trailing zeroes from a result of `homMaskX` ## Load/store Alignment requirements are target-specific, but at most one element. For unaligned scalar loads & stores, `loadu` & `storeu` should be used. - `loadu{p:*E} : E` - load scalar from unaligned memory - `storeu{p:*E, a:V} : void` - store scalar to unaligned memory - `load{p:*V} : V` - load full vector - `store{p:*V, a:V} : void` - store full vector - `loadLow{p:*V, w} : V` - load to low `w` bits - `storeLow{p:*V, w, a:V}` - store low `w` bits - `homMaskStore{p:*V, m:mt{V}, a:V}` - conditionally store elements based on mask; won't touch masked-off elements - `topMaskStore{p:*V, m:V, a:V}` - conditionally store elements based on top bit of `m`; won't touch masked-off elements - `homMaskStoreF` - `homMaskStore` but may touch masked-off elements and thus be supported on more types - `topMaskStoreF` - `topMaskStore` but may touch masked-off elements and thus be supported on more types ## Conversion For float conversions, the used rounding mode is unspecified. - `narrow{eltype{V2}, a:V} : V2` - narrow elements of `a` to desired type. For integer-to-integer conversion, completely undefined result if the values don't exactly fit. Provides `vcount{V2} >= vcount{V}`. - `widen{V2, a:V} : V2` - widen elements of `a` to desired type. Discards elements of `a` past `vcount{V2}`. - `cvt{eltype{V2}, a:V} : V2` - convert element type at the same width. Requires `vcount{V}==vcount{V2} and elwidth{V}==elwidth{V2}`. ## x86-specific - `mul32{a:VI, b:VI} : VI` - multiply, reading only low 32 bits - `blend{L, a:V, b:V, m}` - blend `L`-sized blocks via the immediate - `shuf{L, x:T, n} : T` - shuffle by immediate in `L`-sized lanes - `packQ` - pack 128-bit lanes (`packs`/`packus`) for 16-bit & 32-bit elements - `packQQ` - `packQ` but also defined for 64-bit elements, assuming the high halves are zeroes - `packs` - 128-bit `packs`/`packus` - `unpackQ` - unpack in 128-bit lanes (like `mzip`) - `shuf16Hi`, `shuf16Lo` - 16-bit shuffles with immediate - `shufHalves` ## NEON-specific ### folds - `fold_add{x:V} : eltype{V}` - `fold_addw{x:V} : ty_dbl{eltype{V}}` - `fold_max{x:V} : eltype{V}` - `fold_min{x:V} : eltype{V}` - `vfold{F, x:V} : eltype{V}` for F in `min`, `max`, `__add` ### arithmetic - `addp` - add pairwise - `addpw` - add pairwise, widening - `addpwa` - add pairwise, widening; accumulate - `addwLo` - widening add - `subwLo` - widening subtract - `mulwLo`, `mulwHi`, `mulw` - widening multiply - `andnz` - element-wise (a&b)!=0? ~0 : 0 - `bitAll` - are all bits 1s - `bitAny` - is any bit a 1 - `bitBlend` - blend by bits - `clz` - count leading zeroes - `cls` - count leading sign bits - `copyLane` - `mla` - multiply-add - `mls` - multiply-subtract - `ornot` - a|~b - `packLo`, packHi - `popc` - popcount of each byte - `rbit` - reverse bits in bytes - `rev` - reverse elements in lanes - `shlm`, shrm - fun shifts with defaults - `shrn` - narrowing shift right - `trn1`, `trn2` - 2×2 transposes from pairs of elements across the two arguments ## mask.singeli - `maskOf{TU, n}` - get a mask of type `TU` whose first `n` elements are all `1`s, and the remaining are `0`s. `0≤n≤vcount{TU}` - `maskNone` - a mask generator that matches all items - `maskAfter{n}` - a mask generator that patches the first `n` items - `loadBatch{p:*E, n, V}` - load the `n`-th batch of `vcount{V}` elements from `*E`; if `E` isn't `eltype{V}`, the result is sign- or zero-extended. - `storeBatch{p:*E, n, x:V, M}` - store equivalent of the `loadBatch`; if `E` isn't `eltype{V}`, `x` is narrowed via `narrow{}` To test whether a mask object `M` is `maskNone` or `maskAfter`, `M{0}` can be used - `maskNone{0}` is `0`, but `maskAfter{n}{0}` is `1`. ### Loops - `@maskedLoop{bulk}` - loop that generates its body twice, once with a `maskNone` mask, and once with a `maskAfter{n}` one to handle the tail. - `@muLoop{bulk, unr}` - masked & unrolled loop - generates its body three times (or two if `unr==1`) - once for unrolled main loop, once for the unrolling leftover, and once for the masked end. Unrolling is handled by passing a tuple of indices to process as the index variable (the tail generated bodies get a tuple of the one index) - `@muLoop{bulk, unr, fromunr}` - `fromunr` is additionally ran after exiting the unrolled loop (such that you can have separate accumulators for the unrolled bit, and convert them to single-vector acccumulators for the tail). Tuples can be used in the iterated variable list for various things: ``` p:*T - regular pointer tup{VT,p:*T} - loadBatch/storeBatch vector data tup{'b',p:P} - b_getBatch (bits.singeli) tup{'b',VT,p:P} - loadBatchBit (bits.singeli) tup{'g',VT,p:*T} - gives a generator, used as g{} to loadBatch and g{newValue} to storeBatch tup{'g',p:*T} - the above, but without load support 'm' - get the mask object - either maskNone or some maskAfter{n} ``` Stores via those will implicitly do a masked store when required. Loads will load past the end, so `M in 'm'` must be used to mask off elements if they're used for something other than the above stores. For `muLoop`, `M in 'm'` still gives a single generator, but it's only ever `maskAfter{n}` when there's only one index. Example usage for a loop that adds `u16` and bit boolean elements to a `u32` accumulator, early-exiting on overflow: ``` fn acc_u32_u16_bit(r:*u32, x:*u16, bits:*u64, len:u64) : u1 = { # @maskedLoop def bulk = 8 def VT = [8]u32 @maskedLoop{bulk}( r in tup{VT, r}, x in tup{VT, x}, b in tup{'b', VT, bits}, M in 'm' over i to len ) { # x, r, and b now have type VT r1:= r + x - b # subtracting b because it's all 1s for a one if (homAny{M{r1 < r}}) return{0} # detect overflow r = r1 # will be mask-stored if needed } 1 } fn acc_u32_u16_bit(r:*u32, x:*u16, bits:*u64, len:u64) : u1 = { # @muLoop, 2x unrolled def bulk = 8 def VT = [8]u32 @muLoop{bulk,2}( r in tup{'g', VT, r}, x in tup{VT, x}, b in tup{'b', VT, bits}, M in 'm' over i to len ) { def r0 = r{} def r1 = each{-, each{+,r0,x}, b} if (homAny{M{tree_fold{|, each{<,r1,r0}}}}) return{0} r{r1} } 1 } ```