| .. | ||
| c | ||
| src | ||
| README.md | ||
Overview
Types used here follow [VET][ISUF]?\d?.
First character:
V: some vector typeE: some scalar typeT: either scalar or vector type
Second character, if present:
F: floatS: signed integerU: unsigned integer (may includeu1)I: any signedness integer (may includeu1)
Equally-named types should match. A number is added to the type to separate unequal types.
mt{T} is a mask version of the argument - either ty_u{T} or [vcount{T}]u1 depending on the target.
The SIMD operations listed aren't guaranteed to be supported on all targets, nor list all possible argument types.
General definitons
-
has_simd
0or1depending on whether SIMD types are available. -
arch_defvw
Target vector width for the current target architecture.
Ifhas_simd==0, should not be used. Otherwise, vectors of all element types (exceptu1) whose width is this must be supported, but other widths (both larger and smaller) may be too. -
reinterpretis available as~~
export_tab{'name', fs}- export list of functions as the specified nameiota{knum}- tupletup{0,1,...,knum-1}broadcast{knum,v}(knum**v) - tuple withknumitems, allvtern{c, t, f}- ternary - ifc, returnt, elsefeachx{F, ...args}-eachbut broadcasting non-tuplesundef{T}- return register of typeTwith undefined value; maps over a tuple of typesundef{T, n}- a tuple ofndifferent undefined registersone_val{xs}- assert thatxsis a tuple of equivalent (as persame) items, and return the firstone_type{xs}-oneValbut over the types of the itemsall_same{xs}- whether all values ofxsare the same
Loops & branches
@for- generate body once, with indicess,s+1,...,e-2,e-1@forNZ-@for, but assumed to run at least once@for_backwards-@for, but indices go backwardse-1,e-2,...,s+1,s@forUnroll{exp, unr}- generate a main loop body that takes a tuple ofunrindices to process. Handle the tail via processing one index at a time; ifexp==1, that's done viaunrgenerated bodies, otherwise it's a loop@unroll(n)- generate bodyntimes@collect(n)- generate bodyntimes, and collect the results as a tuple
makeBranch{Ts, G=={...x:Ts}=>{}} == {...x:Ts}=>{}- make a code segment that takes arguments of typesTs, and can be jumped to by invoking the result of themakeBranchmakeOptBranch{enable, Ts, G}-makeBranchbut only generated & invokable whenenable==1
Scalar operations
popc{x:EI} : uint- population countpopcRand{x:EI} : uint- population count; under valgrind, return a random value in the set of possible ones if the input is partly undefinedctz{x:EI} : uint- count trailing zeroes; ifx==0, the result is undefinedclz{x:EI} : uint- count leading zeroes; ifx==0, the result is undefinedclzc{x:EI} : uint-width{EI}-clz{x}; ifx==0, the result is undefined
zlow{n, x}- zero lownbits ofxtail{n, x}- keep lownbits ofxbit{n, x}- getn-th bit ofx
lb{n}- base-2 log of a constant power of twobase{b, x}- convert tuple to number in little-endian basebtree_fold{G, xs}- fold tuplexswith a tree-like pattern withG
Hints
rare{x:u1} : u1- Expect thatxis0. Returnsx.likely{x:u1} : u1- Expect thatxis1. Returnsx.assert{x:u1} : void- assumexis1. Ifxis a compile-time constant, fails to compile if it's not1. If at runtime it is not1, undefined behavior happens (unless debug mode, via eitherinclude './debug'or CBQN-wideDEBUGmode)
Type operations
Of the below functions which take & return vector types of the same width, they can instead take a vector value & reinterpret it to the result type.
w_n{E, w} == E2- a scalar type with the quality ofEand widthww_d{E} == E2,w_h{E} == E2- double/halve the width of a scalar typen_d{V} == V2,n_h{V} == V2- double/halve element count in vector (keeps the same element width)el_d{V} == V2,el_h{V} == V2- double/halve element width in vector (keeps the same count)el_m{V} == V2,el_s{V} == V2- vector with the same total width, but element width doubled (el_m- elements merged) or halved (el_s- elements split)
ty_u{T} == TU- unsigned integer version of the typety_s{T} == TS- signed integer version of the typety_f{T} == TF- float version of the typere_el{E, V} == V2- reinterpret vector to one with the specified element -eltype{V2}==E and width{V2}==width{V}
minvalue{EI},maxvalue{EI}- smallest & biggest value of the specified type
Arithmetic
Some may also support one scalar argument or arguments with different widths.
__neg{a}--a__add{a:V, b:V} : V-a + b__sub{a:VI, b:VI} : VI-a - b__div{a:V, b:V} : V-a / b__mul{a:V, b:V} : V-a * bmulh{a:VI, b:VI} : VI- high half of multiplication
__eq,__ne,__ge,__gt,__le,__lt- comparison;{a:V, b:V} : mt{V}
__not{a:VI} : VI-~a__and{a:VI, b:VI} : VI-a&b__or{a:VI, b:VI} : VI-a|b__xor{a:VI, b:VI} : VI-a^bandnot{a:VI, b:VI} : VI-a&~b__shl{a:VI, b:TI} : VI-a<<b; shift left__shr{a:VS, b:TI} : VS-a>>b; arithmetic shift right__shr{a:VU, b:TI} : VU-a>>b; logical shift right
abs{a:V} : V- absolute valueabsu{a:VI} : ty_u{VI}- absolute value, reinterpreted as unsignedadds{a:VI, b:VI} : VI- saturating addsubs{a:VI, b:VI} : VI- saturating subtract
rcpE{a:VF} : VF- estimate of1/a; precision dependsrsqrtE{a:VF} : VF- estimate ofsqrt{1/a}; precision depends
sqrt{a:VT} : VT- square rootfloor{a:VF} : VF- floorceil{a:VF} : VF- ceilingmax{a:V, b:V} : V- minimum value; arch-specific behavior on NaNsmin{a:V, b:V} : V- maximum value; arch-specific behavior on NaNs
shl{U, a:V, n}- shift vector elements within blocks ofUshr{U, a:V, n}- shift vector elements within blocks ofUunord{a:VF, b:VF} : mt{VI}-(a==NaN) | (b==NaN)
and_bit_none{a:VI, b:VI} : u1-~any_bit{a & b}
Structural operations
broadcast{V, a:eltype{V}} : V(V**a) - broadcast (splat) the scalar to all elements of the resultmake{V, ...vs} : Vormake{V, tup{...vs}}- initialize a vector with the specified elementsiota{V} : V-make{V, range{vcount{V}}}extract{a:V, n} : eltype{V}- extract the n-th elementinsert{a:V, i, b:eltype{V}} : V- a copy ofawith thei-th element replaced withbhalf{a:V, n} : count_half{V}- extract either the low (n==0) or high (n==1) half ofapair{a:V, b:V} : count_dbl{V}- make a vector from halvesundefPromote{V2, a:V} : V- promoteato a wider type, leaving the upper part undefinedvshl{a:V, b:V, n} : V-vshl{[0,1,2,3], [4,5,6,7], 1}→[1,2,3,4]sel{VI1, a:V, b:VI2} : V2- shuffleaby indicesbinVI1-long lanes; arch-specific behavior on out-of-bounds values
Zip & unzip
All of these optionally accept a third argument of a constant 0 or 1 indicating which result element to give.
Some of them define variants which operate within 128-bit lanes, via a 128 postfix to the name (e.g. zip128).
zip{a:V, b:V} : tup{V, V}-zip{[0,1,2,3], [4,5,6,7]}→tup{[0,4,1,5], [2,6,3,7]}mzip{a:V, b:V} : tup{el_m{V}, el_m{V}}- reinterpretedzip{a, b}unzip{a:V, b:V} : tup{V, V}-unzip{[0,1,2,3], [4,5,6,7]}→tup{[0,2,4,6], [1,3,5,7]}pack{a:V, b:V} : tup{el_s{V}, el_s{V}}-unzip{el_s{V}~~a, el_s{V}~~b}; i.e. low half of each element element in first result, high in second
Mask stuff
Homogeneous definitions (i.e. ones with hom in their name) assume that each element in the mask type has all its bits equal.
all_hom{a:VI} : u1- whether all elements are setany_hom{a:VI} : u1- whether any element is setany_top{a:VI} : u1- whether all elements have their top bit setall_top{a:VI} : u1- whether any element has its top bit setany_bit{a:VI} : u1- whether any bit in any element is setall_bit{a:VI} : u1- whether all bits in all elements is setblend_hom{f:V, t:V, m:mt{V}} : V- blend bym, setting tofwhere0andtwhere1blend_top{f:V, t:V, m:V} : V- blend by top bit ofmblend_bit{f:V, t:V, m:M} : V- bitwise blendhom_to_int{a:VI} : uint- integer mask of whether each element is set (assumes each element has all its bits equal)hom_to_int{...vs} : uint- merged mask ofeach{hom_to_int,vs}top_to_int{a:VI} : uint- integer mask of the top bit of each elementhom_to_int_ext{a:VI} : tup{knum, uint}- integer mask where each element is represented byknumbits (possibly more efficient to calculate thanhom_to_int)ctz_ext{tup{knum, uint}}- count trailing zeroes from a result ofhom_to_int_ext
Load/store
Alignment requirements are target-specific, but at most one element.
For unaligned scalar loads & stores, loadu & storeu should be used.
loadu{p:*E} : E- load scalar from unaligned memorystoreu{p:*E, a:E} : void- store scalar to unaligned memoryload{p:*V} : V- load full vectorstore{p:*V, a:V} : void- store full vectorload_low{p:*V, w} : V- load to lowwbitsstore_low{p:*E, w, a:[n]E}- store lowwbitsstore_masked_hom{p:*V, m:mt{V}, a:V}- conditionally store elements based on mask; won't touch masked-off elementsstore_masked_top{p:*V, m:V, a:V}- conditionally store elements based on top bit ofm; won't touch masked-off elementsstore_blended_hom-store_masked_hombut may touch masked-off elements and thus be supported on more typesstore_blended_top-store_masked_topbut may touch masked-off elements and thus be supported on more types
Conversion
For float conversions, the used rounding mode is unspecified.
narrow{eltype{V2}, a:V} : V2- narrow elements ofato desired type.
For integer-to-integer conversion, completely undefined result if the values don't exactly fit.
Providesvcount{V2} >= vcount{V}.widen{V2, a:V} : V2- widen elements ofato desired type.
Discards elements ofapastvcount{V2}.cvt{eltype{V2}, a:V} : V2- convert element type at the same width.
Requiresvcount{V}==vcount{V2} and elwidth{V}==elwidth{V2}.
x86-specific
mul32{a:VI, b:VI} : VI- multiply, reading only low 32 bitsblend{L, a:V, b:V, m}- blendL-sized blocks via the immediateshuf{L, x:V, n} : V- shuffle by immediate inL-sized lanespackQ- pack 128-bit lanes (packs/packus) for 16-bit & 32-bit elementspackQQ-packQbut also defined for 64-bit elements, assuming the high halves are zeroespacks- 128-bitpacks/packusshuf16Hi,shuf16Lo- 16-bit shuffles with immediateshufHalves
NEON-specific
folds
fold_add{x:V} : eltype{V}fold_addw{x:V} : ty_dbl{eltype{V}}fold_max{x:V} : eltype{V}fold_min{x:V} : eltype{V}vfold{F, x:V} : eltype{V}for F inmin,max,__add
arithmetic
addp- add pairwiseaddpw- add pairwise, wideningaddpwa- add pairwise, widening; accumulateaddwLo- widening addsubwLo- widening subtractmulwLo,mulwHi,mulw- widening multiplyandnz- element-wise (a&b)!=0? ~0 : 0
clz- count leading zeroescls- count leading sign bitscopyLanemla- multiply-addmls- multiply-subtractornot- a|~bpopc- popcount of each byterbit- reverse bits in bytesrev- reverse elements in lanesshlm, shrm - fun shifts with defaultsshrn- narrowing shift righttrn1,trn2- 2×2 transposes from pairs of elements across the two arguments
mask.singeli
maskOf{TU, n}- get a mask of typeTUwhose firstnelements are all1s, and the remaining are0s.0≤n≤vcount{TU}maskNone- a mask generator that matches all itemsmaskAfter{n}- a mask generator that patches the firstnitemsloadBatch{p:*E, n, V}- load then-th batch ofvcount{V}elements from*E; ifEisn'teltype{V}, the result is sign- or zero-extended.storeBatch{p:*E, n, x:V, M}- store equivalent of theloadBatch; ifEisn'teltype{V},xis narrowed vianarrow{}
To test whether a mask object M is maskNone or maskAfter, M{0} can be used - maskNone{0} is 0, but maskAfter{n}{0} is 1.
Loops
-
@maskedLoop{bulk}- loop that generates its body twice, once with amaskNonemask, and once with amaskAfter{n}one to handle the tail. -
@muLoop{bulk, unr}- masked & unrolled loop - generates its body three times (or two ifunr==1) - once for unrolled main loop, once for the unrolling leftover, and once for the masked end. Unrolling is handled by passing a tuple of indices to process as the index variable (the tail generated bodies get a tuple of the one index) -
@muLoop{bulk, unr, fromunr}-fromunris additionally ran after exiting the unrolled loop (such that you can have separate accumulators for the unrolled bit, and convert them to single-vector acccumulators for the tail).
Tuples can be used in the iterated variable list for various things:
p:*T - regular pointer
tup{VT,p:*T} - loadBatch/storeBatch vector data
tup{'b',p:P} - b_getBatch (bits.singeli)
tup{'b',VT,p:P} - loadBatchBit (bits.singeli)
tup{'g',VT,p:*T} - gives a generator, used as g{} to loadBatch and g{newValue} to storeBatch
tup{'g',p:*T} - the above, but without load support
'm' - get the mask object - either maskNone or some maskAfter{n}
Stores via those will implicitly do a masked store when required.
Loads will load past the end, so M in 'm' must be used to mask off elements if they're used for something other than the above stores.
For muLoop, M in 'm' still gives a single generator, but it's only ever maskAfter{n} when there's only one index.
Example usage for a loop that adds u16 and bit boolean elements to a u32 accumulator, early-exiting on overflow:
fn acc_u32_u16_bit(r:*u32, x:*u16, bits:*u64, len:u64) : u1 = { # @maskedLoop
def bulk = 8
def VT = [8]u32
@maskedLoop{bulk}(
r in tup{VT, r},
x in tup{VT, x},
b in tup{'b', VT, bits},
M in 'm'
over i to len
) {
# x, r, and b now have type VT
r1:= r + x - b # subtracting b because it's all 1s for a one
if (any_hom{M{r1 < r}}) return{0} # detect overflow
r = r1 # will be mask-stored if needed
}
1
}
fn acc_u32_u16_bit(r:*u32, x:*u16, bits:*u64, len:u64) : u1 = { # @muLoop, 2x unrolled
def bulk = 8
def VT = [8]u32
@muLoop{bulk,2}(
r in tup{'g', VT, r},
x in tup{VT, x},
b in tup{'b', VT, bits},
M in 'm'
over i to len
) {
def r0 = r{}
def r1 = each{-, each{+,r0,x}, b}
if (any_hom{M{tree_fold{|, each{<,r1,r0}}}}) return{0}
r{r1}
}
1
}