| .. | ||
| c | ||
| src | ||
| README.md | ||
Overview
Types used here follow [VET][ISUF]?\d?.
First character:
V: some vector typeE: some scalar typeT: either scalar or vector type
Second character, if present:
F: floatS: signed integerU: unsigned integer (may includeu1)I: any signedness integer (may includeu1)
Equally-named types should match. A number is added to the type to separate unequal types.
mt{T} is a mask version of the argument - either ty_u{T} or [vcount{T}]u1 depending on the target.
The SIMD operations listed aren't guaranteed to be supported on all targets, nor list all possible argument types.
General definitons
-
has_simd
0or1depending on whether SIMD types are available. -
arch_defvw
Target vector width for the current target architecture.
Ifhas_simd==0, should not be used. Otherwise, vectors of all element types (exceptu1) whose width is this must be supported, but other widths (both larger and smaller) may be too. -
reinterpretis available as~~
export_tab{'name', fs}- export list of functions as the specified nameiota{knum}- tupletup{0,1,...,knum-1}broadcast{knum,v}(knum**v) - tuple withknumitems, allvtern{c, t, f}- ternary - ifc, returnt, elsefeachx{F, ...args}-eachbut broadcasting non-tuplesundef{T}- return register of typeTwith undefined value; maps over a tuple of typesundef{T, n}- a tuple ofndifferent undefined registersone_val{xs}- assert thatxsis a tuple of equivalent (as persame) items, and return the firstone_type{xs}-oneValbut over the types of the itemsall_same{xs}- whether all values ofxsare the same
Loops & branches
@for- generate body once, with indicess,s+1,...,e-2,e-1@for_nz-@for, but assumed to run at least once@for_unroll_block{exp, unr}- generate a main loop body that takes a tuple ofunrindices to process. Handle the tail via processing one index at a time; ifexp==1, that's done viaunrgenerated bodies, otherwise it's a loop@unroll(n)- generate bodyntimes
make_branch{Ts, G=={...x:Ts}=>{}} == {...x:Ts}=>{}- make a code segment that takes arguments of typesTs, and can be jumped to by invoking the result of themake_branchmake_opt_ranch{enable, Ts, G}-make_branchbut only generated & invokable whenenable==1
Scalar operations
popc{x:EI} : uint- population countpopc_rand{x:EI} : uint- population count; under valgrind, return a random value in the set of possible ones if the input is partly undefinedctz{x:EI} : uint- count trailing zeroes; ifx==0, the result is undefinedclz{x:EI} : uint- count leading zeroes; ifx==0, the result is undefinedclzc{x:EI} : uint-width{EI}-clz{x}; ifx==0, the result is undefined
zlow{n, x}- zero lownbits ofxtail{n, x}- keep lownbits ofxbit{n, x}- getn-th bit ofx
lb{n}- base-2 log of a constant power of twobase{b, x}- convert tuple to number in little-endian basebtree_fold{G, xs}- fold tuplexswith a tree-like pattern withG
Hints
rare{x:u1} : u1- Expect thatxis0. Returnsx.likely{x:u1} : u1- Expect thatxis1. Returnsx.assert{x:u1} : void- assumexis1. Ifxis a compile-time constant, fails to compile if it's not1. If at runtime it is not1, undefined behavior happens (unless debug mode, via eitherinclude './debug'or CBQN-wideDEBUGmode)
Type operations
Of the below functions which take & return vector types of the same width, they can instead take a vector value & reinterpret it to the result type.
w_n{E, w} == E2- a scalar type with the quality ofEand widthww_d{E} == E2,w_h{E} == E2- double/halve the width of a scalar typen_d{V} == V2,n_h{V} == V2- double/halve element count in vector (keeps the same element width)el_d{V} == V2,el_h{V} == V2- double/halve element width in vector (keeps the same count)el_m{V} == V2,el_s{V} == V2- vector with the same total width, but element width doubled (el_m- elements merged) or halved (el_s- elements split)
ty_u{T} == TU- unsigned integer version of the typety_s{T} == TS- signed integer version of the typety_f{T} == TF- float version of the typere_el{E, V} == V2- reinterpret vector to one with the specified element -eltype{V2}==E and width{V2}==width{V}
minvalue{EI},maxvalue{EI}- smallest & biggest value of the specified type
Arithmetic
Some may also support one scalar argument or arguments with different widths.
__neg{a}--a__add{a:V, b:V} : V-a + b__sub{a:VI, b:VI} : VI-a - b__div{a:V, b:V} : V-a / b__mul{a:V, b:V} : V-a * bmulh{a:VI, b:VI} : VI- high half of multiplication
__eq,__ne,__ge,__gt,__le,__lt- comparison;{a:V, b:V} : mt{V}
__not{a:VI} : VI-~a__and{a:VI, b:VI} : VI-a&b__or{a:VI, b:VI} : VI-a|b__xor{a:VI, b:VI} : VI-a^bandnot{a:VI, b:VI} : VI-a&~b__shl{a:VI, b:TI} : VI-a<<b; shift left__shr{a:VS, b:TI} : VS-a>>b; arithmetic shift right__shr{a:VU, b:TI} : VU-a>>b; logical shift right
abs{a:V} : V- absolute valueabsu{a:VI} : ty_u{VI}- absolute value, reinterpreted as unsignedadds{a:VI, b:VI} : VI- saturating addsubs{a:VI, b:VI} : VI- saturating subtract
rcpE{a:VF} : VF- estimate of1/a; precision dependsrsqrtE{a:VF} : VF- estimate ofsqrt{1/a}; precision depends
sqrt{a:VT} : VT- square rootfloor{a:VF} : VF- floorceil{a:VF} : VF- ceilingmax{a:V, b:V} : V- minimum value; arch-specific behavior on NaNsmin{a:V, b:V} : V- maximum value; arch-specific behavior on NaNs
shl{U, a:V, n}- shift vector elements within blocks ofUshr{U, a:V, n}- shift vector elements within blocks ofUunord{a:VF, b:VF} : mt{VI}-(a==NaN) | (b==NaN)
and_bit_none{a:VI, b:VI} : u1-~any_bit{a & b}
Structural operations
broadcast{V, a:eltype{V}} : V(V**a) - broadcast (splat) the scalar to all elements of the resultmake{V, ...vs} : Vormake{V, tup{...vs}}- initialize a vector with the specified elementsiota{V} : V-make{V, range{vcount{V}}}extract{a:V, n} : eltype{V}- extract the n-th elementinsert{a:V, i, b:eltype{V}} : V- a copy ofawith thei-th element replaced withbhalf{a:V, n} : count_half{V}- extract either the low (n==0) or high (n==1) half ofapair{a:V, b:V} : count_dbl{V}- make a vector from halvesundef_promote{V2, a:V} : V- promoteato a wider type, leaving the upper part undefinedvshl{a:V, b:V, n} : V-vshl{[0,1,2,3], [4,5,6,7], 1}→[1,2,3,4]sel{VI1, a:V, b:VI2} : V2- shuffleaby indicesbinVI1-long lanes; arch-specific behavior on out-of-bounds values
Zip & unzip
All of these optionally accept a third argument of a constant 0 or 1 indicating which result element to give.
Some of them define variants which operate within 128-bit lanes, via a 128 postfix to the name (e.g. zip128).
zip{a:V, b:V} : tup{V, V}-zip{[0,1,2,3], [4,5,6,7]}→tup{[0,4,1,5], [2,6,3,7]}mzip{a:V, b:V} : tup{el_m{V}, el_m{V}}- reinterpretedzip{a, b}unzip{a:V, b:V} : tup{V, V}-unzip{[0,1,2,3], [4,5,6,7]}→tup{[0,2,4,6], [1,3,5,7]}pack{a:V, b:V} : tup{el_s{V}, el_s{V}}-unzip{el_s{V}~~a, el_s{V}~~b}; i.e. low half of each element element in first result, high in second
Mask stuff
Homogeneous definitions (i.e. ones with hom in their name) assume that each element in the mask type has all its bits equal.
all_hom{a:VI} : u1- whether all elements are setany_hom{a:VI} : u1- whether any element is setany_top{a:VI} : u1- whether all elements have their top bit setall_top{a:VI} : u1- whether any element has its top bit setany_bit{a:VI} : u1- whether any bit in any element is setall_bit{a:VI} : u1- whether all bits in all elements is setblend_hom{f:V, t:V, m:mt{V}} : V- blend bym, setting tofwhere0andtwhere1blend_top{f:V, t:V, m:V} : V- blend by top bit ofmblend_bit{f:V, t:V, m:M} : V- bitwise blendhom_to_int{a:VI} : uint- integer mask of whether each element is set (assumes each element has all its bits equal)hom_to_int{...vs} : uint- merged mask ofeach{hom_to_int,vs}top_to_int{a:VI} : uint- integer mask of the top bit of each elementhom_to_int_ext{a:VI} : tup{knum, uint}- integer mask where each element is represented byknumbits (possibly more efficient to calculate thanhom_to_int)ctz_ext{tup{knum, uint}}- count trailing zeroes from a result ofhom_to_int_ext
Load/store
Alignment requirements are target-specific, but at most one element.
For unaligned scalar loads & stores, loadu & storeu should be used.
loadu{p:*E} : E- load scalar from unaligned memorystoreu{p:*E, a:E} : void- store scalar to unaligned memoryload{[k]E, p:*E} : [k]E- load full vectorstore{p:*E, a:[k]E} : void- store full vectorload{[k]E, p:*E, vl} : [k]E- load firstvlelements of vector (memory of upper ones won't be touched, and their values are unspecified)store{p:*E, a:[k]E, vl} : void- store firstvlelements (upper ones won't be touched)store_masked_hom{p:*E, m:mt{V}, a:V=[_]E}- conditionally store elements based on mask; won't touch masked-off elements;pmay also bep:*Vstore_masked_top{p:*E, m:V, a:V=[_]E}- conditionally store elements based on top bit ofm; won't touch masked-off elements;pmay also bep:*Vstore_blended_hom-store_masked_hombut may touch masked-off elements and thus be supported on more typesstore_blended_top-store_masked_topbut may touch masked-off elements and thus be supported on more types
Conversion
For float conversions, the used rounding mode is unspecified.
narrow{eltype{V2}, a:V} : V2- narrow elements ofato desired type.
For integer-to-integer conversion, completely undefined result if the values don't exactly fit.
Providesvcount{V2} >= vcount{V}.widen{V2, a:V} : V2- widen elements ofato desired type.
Discards elements ofapastvcount{V2}.cvt{eltype{V2}, a:V} : V2- convert element type at the same width.
Requiresvcount{V}==vcount{V2} and elwidth{V}==elwidth{V2}.
x86-specific
mul32{a:VI, b:VI} : VI- multiply, reading only low 32 bitsblend{L, a:V, b:V, m}- blendL-sized blocks via the immediateshuf{L, x:V, n} : V- shuffle by immediate inL-sized lanespacks128- saturating pack 128-bit lanes (packs/packus) for 16-bit & 32-bit elementspackQQ-packs128but also defined for 64-bit elements, assuming the high halves are zeroespacks- 128-bitpacks/packusshuf16Hi,shuf16Lo- 16-bit shuffles with immediateshufHalves
NEON-specific
folds
fold_add{x:V} : eltype{V}fold_addw{x:V} : ty_dbl{eltype{V}}fold_max{x:V} : eltype{V}fold_min{x:V} : eltype{V}vfold{F, x:V} : eltype{V}for F inmin,max,__add
arithmetic
addp- add pairwiseaddpw- add pairwise, wideningaddpwa- add pairwise, widening; accumulateaddw,subw,mulw- widening add/subtract/multiplyandnz- element-wise (a&b)!=0? ~0 : 0
clz- count leading zeroescls- count leading sign bitscopy_lanemla- multiply-addmls- multiply-subtractornot- a|~bpopc- popcount of each byterbit- reverse bits in bytesrev- reverse elements in lanesshlm, shrm - fun shifts with defaultsshrn- narrowing shift righttrn1,trn2- 2×2 transposes from pairs of elements across the two arguments
mask.singeli
mask_of_first{TU, n}- get a mask of typeTUwhose firstnelements are all1s, and the remaining are0s.0≤n≤vcount{TU}mask_none- a mask generator that matches all itemsmask_first{n}- a mask generator that enables the firstnitemsload_widen{p:*E, n, [k]W}- load then-th batch ofkelements fromp; ifWisn'tE, the result is sign- or zero-extended.store_narrow{p:*E, n, x:[k]N, M}- store equivalent of theload_widen; ifEisn'tN,xis narrowed vianarrow{}
To test whether a mask object M is mask_none or mask_first, M{0} can be used - mask_none{0} is 0, but mask_first{n}{0} is 1.
Loops
-
@for_masked{bulk}- loop that generates its body twice, once with amask_nonemask, and once with amask_first{n}one to handle the tail. -
@for_mu{bulk, unr}- masked & unrolled loop - generates its body three times (or two ifunr==1) - once for unrolled main loop, once for the unrolling leftover, and once for the masked end. Unrolling is handled by passing a tuple of indices to process as the index variable (the tail generated bodies get a tuple of the one index) -
@for_mu{bulk, unr, fromunr}-fromunris additionally ran after exiting the unrolled loop (such that you can have separate accumulators for the unrolled bit, and convert them to single-vector acccumulators for the tail).
Tuples can be used in the iterated variable list for various things:
p:*T - regular pointer
tup{VT,p:*T} - load_widen/store_narrow vector data
tup{'b',p:P} - load_bits (bits.singeli)
tup{'b',VT,p:P} - load_expand_bits (bits.singeli)
tup{'g',VT,p:*T} - gives a generator, used as g{} to load_widen and g{newValue} to store_narrow
tup{'g',p:*T} - the above, but without load support
'm' - get the mask object - either mask_none or some mask_first{n}
Stores via those will implicitly do a masked store when required.
Loads will load past the end, so M in 'm' must be used to mask off elements if they're used for something other than the above stores.
For for_mu, M in 'm' still gives a single generator, but it's only ever mask_first{n} when there's only one index.
Example usage for a loop that adds u16 and bit boolean elements to a u32 accumulator, early-exiting on overflow:
fn acc_u32_u16_bit_u1(r:*u32, x:*u16, bits:*u64, len:u64) : u1 = { # @for_masked
def bulk = 8
def VT = [8]u32
@for_masked{bulk}(
r in tup{VT, r},
x in tup{VT, x},
b in tup{'b', VT, bits},
M in 'm'
over i to len
) {
# x, r, and b now have type VT
r1:= r + x - b # subtracting b because it's all 1s for a one
if (any_hom{M{r1 < r}}) return{0} # detect overflow
r = r1 # will be mask-stored if needed
}
1
}
export {'acc_u32_u16_bit_u1', acc_u32_u16_bit_u1}
fn acc_u32_u16_bit_u2(r:*u32, x:*u16, bits:*u64, len:u64) : u1 = { # @for_mu, 2x unrolled
def bulk = 8
def VT = [8]u32
@for_mu{bulk,2}(
r in tup{'g', VT, r},
x in tup{VT, x},
b in tup{'b', VT, bits},
M in 'm'
over i to len
) {
def r0 = r{}
def r1 = each{-, each{+,r0,x}, b}
if (any_hom{M{tree_fold{|, each{<,r1,r0}}}}) return{0}
r{r1}
}
1
}
export {'acc_u32_u16_bit_u2', acc_u32_u16_bit_u2}