| .. | ||
| c | ||
| src | ||
| README.md | ||
Overview
Types used here follow [VET][ISUF]?\d?.
First character:
V: some vector typeE: some scalar typeT: either scalar or vector type
Second character, if present:
F: floatS: signed integerU: unsigned integer (may includeu1)I: any signedness integer (may includeu1)
Equally-named types should match. A number is added to the type to separate unequal types.
mt{T} is a mask version of the argument - either ty_u{T} or [vcount{T}]u1 depending on the target.
The SIMD operations listed aren't guaranteed to be supported on all targets, nor list all possible argument types.
General definitons
-
has_simd
0or1depending on whether SIMD types are available. -
arch_defvw
Target vector width for the current target architecture.
Ifhas_simd==0, should not be used. Otherwise, vectors of all element types (exceptu1) whose width is this must be supported, but other widths (both larger and smaller) may be too. -
reinterpretis available as~~
exportN{f, 'name1', 'name2', ...}- export single function under many namesexportT{'name', fs}- export list of functions as the specified nameiota{knum}- tupletup{0,1,...,knum-1}broadcast{knum,v}(knum**v) - tuple withknumitems, allvtern{c, t, f}- ternary - ifc, returnt, elsefeachx{F, ...args}-eachbut broadcasting non-tuplesundef{T}- return register of typeTwith undefined value; maps over a tuple of typesundef{T, n}- a tuple ofndifferent undefined registersoneVal{xs}- assert thatxsis a tuple of equivalent (as per==) items, and return the firstoneType{xs}-oneValbut over the types of the items
Loops & branches
@for- generate body once, with indicess,s+1,...,e-2,e-1@forNZ-@for, but assumed to run at least once@for_backwards-@for, but indices go backwardse-1,e-2,...,s+1,s@forUnroll{exp, unr}- generate a main loop body that takes a tuple ofunrindices to process. Handle the tail via processing one index at a time; ifexp==1, that's done viaunrgenerated bodies, otherwise it's a loop@unroll(n)- generate bodyntimes@collect(n)- generate bodyntimes, and collect the results as a tuple
makeBranch{Ts, G=={...x:Ts}=>{}} == {...x:Ts}=>{}- make a code segment that takes arguments of typesTs, and can be jumped to by invoking the result of themakeBranchmakeOptBranch{enable, Ts, G}-makeBranchbut only generated & invokable whenenable==1
Scalar operations
popc{x:EI} : uint- population countpopcRand{x:EI} : uint- population count; under valgrind, return a random value in the set of possible ones if the input is partly undefinedctz{x:EI} : uint- count trailing zeroes; ifx==0, the result is undefinedclz{x:EI} : uint- count leading zeroes; ifx==0, the result is undefinedclzc{x:EI} : uint-width{EI}-clz{x}; ifx==0, the result is undefined
zlow{n, x}- zero lownbits ofxtail{n, x}- keep lownbits ofxbit{n, x}- getn-th bit ofx
lb{n}- base-2 log of a constant power of twobase{b, x}- convert tuple to number in little-endian basebtree_fold{G, xs}- fold tuplexswith a tree-like pattern withG
Hints
rare{x:u1} : u1- Expect thatxis0. Returnsx.likely{x:u1} : u1- Expect thatxis1. Returnsx.assert{x:u1} : void- assumexis1. Ifxis a compile-time constant, fails to compile if it's not1. If at runtime it is not1, undefined behavior happens (unless debug mode, via eitherinclude './debug'or CBQN-wideDEBUGmode)
Type operations
Of the below functions which take & return vector types of the same width, they can instead take a vector value & reinterpret it to the result type.
w_n{E, w} == E2- a scalar type with the quality ofEand widthww_d{E} == E2,w_h{E} == E2- double/halve the width of a scalar typen_d{V} == V2,n_h{V} == V2- double/halve element count in vector (keeps the same element width)el_d{V} == V2,el_h{V} == V2- double/halve element width in vector (keeps the same count)el_m{V} == V2,el_s{V} == V2- vector with the same total width, but element width doubled (el_m- elements merged) or halved (el_s- elements split)
ty_u{T} == TU- unsigned integer version of the typety_s{T} == TS- signed integer version of the typety_f{T} == TF- float version of the typere_el{E, V} == V2- reinterpret vector to one with the specified element -eltype{V2}==E and width{V2}==width{V}
minvalue{EI},maxvalue{EI}- smallest & biggest value of the specified type
Arithmetic
Some may also support one scalar argument or arguments with different widths.
__neg{a}--a__add{a:V, b:V} : V-a + b__sub{a:VI, b:VI} : VI-a - b__div{a:V, b:V} : V-a / b__mul{a:V, b:V} : V-a * bmulh{a:VI, b:VI} : VI- high half of multiplication
__eq,__ne,__ge,__gt,__le,__lt- comparison;{a:V, b:V} : mt{V}
__not{a:VI} : VI-~a__and{a:VI, b:VI} : VI-a&b__or{a:VI, b:VI} : VI-a|b__xor{a:VI, b:VI} : VI-a^bandnot{a:VI, b:VI} : VI-a&~b__shl{a:VI, b:TI} : VI-a<<b; shift left__shr{a:VS, b:TI} : VS-a>>b; arithmetic shift right__shr{a:VU, b:TI} : VU-a>>b; logical shift right
abs{a:V} : V- absolute valueabsu{a:VI} : ty_u{VI}- absolute value, reinterpreted as unsignedadds{a:VI, b:VI} : VI- saturating addsubs{a:VI, b:VI} : VI- saturating subtract
rcpE{a:VF} : VF- estimate of1/a; precision dependsrsqrtE{a:VF} : VF- estimate ofsqrt{1/a}; precision depends
sqrt{a:VT} : VT- square rootfloor{a:VF} : VF- floorceil{a:VF} : VF- ceilingmax{a:V, b:V} : V- minimum value; arch-specific behavior on NaNsmin{a:V, b:V} : V- maximum value; arch-specific behavior on NaNs
shl{U, a:V, n}- shift vector elements within blocks ofUshr{U, a:V, n}- shift vector elements within blocks ofUunord{a:VF, b:VF} : mt{VI}-(a==NaN) | (b==NaN)
andAllZero{a:VI, b:VI} : u1- whethera&bis all zeroes
Structural operations
broadcast{V, a:eltype{V}} : V(V**a) - broadcast (splat) the scalar to all elements of the resultmake{V, ...vs} : Vormake{V, tup{...vs}}- initialize a vector with the specified elementsiota{V} : V-make{V, range{vcount{V}}}extract{a:V, n} : eltype{V}- extract the n-th elementinsert{a:V, i, b:eltype{V}} : V- a copy ofawith thei-th element replaced withbhalf{a:V, n} : count_half{V}- extract either the low (n==0) or high (n==1) half ofapair{a:V, b:V} : count_dbl{V}- make a vector from halvesundefPromote{V2, a:V} : V- promoteato a wider type, leaving the upper part undefinedvshl{a:V, b:V, n} : V-vshl{[0,1,2,3], [4,5,6,7], 1} → [1,2,3,4]sel{VI1, a:V, b:VI2} : V2- shuffleaby indicesbinVI1-long lanes; arch-specific behavior on out-of-bounds values
zip{a:V,b:V} : tup{V, V}-zip{[0,1,2,3], [4,5,6,7]} → tup{[0,4,1,5], [2,6,3,7]}mzip{a:V,b:V} : tup{el_dbl{V}, el_dbl{V}}- reinterpretedzip{a, b}zipLo{a:V,b:V} : V-select{zip{a, b}, 0}zipHi{a:V,b:V} : V-select{zip{a, b}, 1}mzipLo{a:V,b:V} : V-select{mzip{a, b}, 0}mzipHi{a:V,b:V} : V-select{mzip{a, b}, 1}
Mask stuff
Homogeneous definitions (i.e. ones with hom in their name) assume that each element in the mask type has all its bits equal.
homAll{a:VI} : u1- whether all elements are sethomAny{a:VI} : u1- whether any element is settopAny{a:VI} : u1- whether all elements have their top bit settopAll{a:VI} : u1- whether any element has its top bit sethomBlend{f:V, t:V, m:mt{V}} : V- blend bym, setting tofwhere0andtwhere1topBlend{f:V, t:V, m:V} : V- blend by top bit ofmhomMask{a:VI} : uint- integer mask of whether each element is set (assumes each element has all its bits equal)homMask{...vs} : uint- merged mask ofeach{homMask,vs}topMask{a:VI} : uint- integer mask of the top bit of each elementhomMaskX{a:VI} : tup{knum, uint}- integer mask where each element is represented byknumbits (possibly more efficient to calculate thanhomMask)ctzX{tup{knum, uint}}- count trailing zeroes from a result ofhomMaskX
Load/store
Alignment requirements are target-specific, but at most one element.
For unaligned scalar loads & stores, loadu & storeu should be used.
loadu{p:*E} : E- load scalar from unaligned memorystoreu{p:*E, a:E} : void- store scalar to unaligned memoryload{p:*V} : V- load full vectorstore{p:*V, a:V} : void- store full vectorloadLow{p:*V, w} : V- load to lowwbitsstoreLow{p:*E, w, a:[n]E}- store lowwbitshomMaskStore{p:*V, m:mt{V}, a:V}- conditionally store elements based on mask; won't touch masked-off elementstopMaskStore{p:*V, m:V, a:V}- conditionally store elements based on top bit ofm; won't touch masked-off elementshomMaskStoreF-homMaskStorebut may touch masked-off elements and thus be supported on more typestopMaskStoreF-topMaskStorebut may touch masked-off elements and thus be supported on more types
Conversion
For float conversions, the used rounding mode is unspecified.
narrow{eltype{V2}, a:V} : V2- narrow elements ofato desired type.
For integer-to-integer conversion, completely undefined result if the values don't exactly fit.
Providesvcount{V2} >= vcount{V}.widen{V2, a:V} : V2- widen elements ofato desired type.
Discards elements ofapastvcount{V2}.cvt{eltype{V2}, a:V} : V2- convert element type at the same width.
Requiresvcount{V}==vcount{V2} and elwidth{V}==elwidth{V2}.
x86-specific
mul32{a:VI, b:VI} : VI- multiply, reading only low 32 bitsblend{L, a:V, b:V, m}- blendL-sized blocks via the immediateshuf{L, x:V, n} : V- shuffle by immediate inL-sized lanespackQ- pack 128-bit lanes (packs/packus) for 16-bit & 32-bit elementspackQQ-packQbut also defined for 64-bit elements, assuming the high halves are zeroespacks- 128-bitpacks/packusunpackQ- unpack in 128-bit lanes (likemzip)shuf16Hi,shuf16Lo- 16-bit shuffles with immediateshufHalves
NEON-specific
folds
fold_add{x:V} : eltype{V}fold_addw{x:V} : ty_dbl{eltype{V}}fold_max{x:V} : eltype{V}fold_min{x:V} : eltype{V}vfold{F, x:V} : eltype{V}for F inmin,max,__add
arithmetic
addp- add pairwiseaddpw- add pairwise, wideningaddpwa- add pairwise, widening; accumulateaddwLo- widening addsubwLo- widening subtractmulwLo,mulwHi,mulw- widening multiplyandnz- element-wise (a&b)!=0? ~0 : 0
bitAll- are all bits 1sbitAny- is any bit a 1bitBlend- blend by bitsclz- count leading zeroescls- count leading sign bitscopyLanemla- multiply-addmls- multiply-subtractornot- a|~bpackLo, packHipopc- popcount of each byterbit- reverse bits in bytesrev- reverse elements in lanesshlm, shrm - fun shifts with defaultsshrn- narrowing shift righttrn1,trn2- 2×2 transposes from pairs of elements across the two arguments
mask.singeli
maskOf{TU, n}- get a mask of typeTUwhose firstnelements are all1s, and the remaining are0s.0≤n≤vcount{TU}maskNone- a mask generator that matches all itemsmaskAfter{n}- a mask generator that patches the firstnitemsloadBatch{p:*E, n, V}- load then-th batch ofvcount{V}elements from*E; ifEisn'teltype{V}, the result is sign- or zero-extended.storeBatch{p:*E, n, x:V, M}- store equivalent of theloadBatch; ifEisn'teltype{V},xis narrowed vianarrow{}
To test whether a mask object M is maskNone or maskAfter, M{0} can be used - maskNone{0} is 0, but maskAfter{n}{0} is 1.
Loops
-
@maskedLoop{bulk}- loop that generates its body twice, once with amaskNonemask, and once with amaskAfter{n}one to handle the tail. -
@muLoop{bulk, unr}- masked & unrolled loop - generates its body three times (or two ifunr==1) - once for unrolled main loop, once for the unrolling leftover, and once for the masked end. Unrolling is handled by passing a tuple of indices to process as the index variable (the tail generated bodies get a tuple of the one index) -
@muLoop{bulk, unr, fromunr}-fromunris additionally ran after exiting the unrolled loop (such that you can have separate accumulators for the unrolled bit, and convert them to single-vector acccumulators for the tail).
Tuples can be used in the iterated variable list for various things:
p:*T - regular pointer
tup{VT,p:*T} - loadBatch/storeBatch vector data
tup{'b',p:P} - b_getBatch (bits.singeli)
tup{'b',VT,p:P} - loadBatchBit (bits.singeli)
tup{'g',VT,p:*T} - gives a generator, used as g{} to loadBatch and g{newValue} to storeBatch
tup{'g',p:*T} - the above, but without load support
'm' - get the mask object - either maskNone or some maskAfter{n}
Stores via those will implicitly do a masked store when required.
Loads will load past the end, so M in 'm' must be used to mask off elements if they're used for something other than the above stores.
For muLoop, M in 'm' still gives a single generator, but it's only ever maskAfter{n} when there's only one index.
Example usage for a loop that adds u16 and bit boolean elements to a u32 accumulator, early-exiting on overflow:
fn acc_u32_u16_bit(r:*u32, x:*u16, bits:*u64, len:u64) : u1 = { # @maskedLoop
def bulk = 8
def VT = [8]u32
@maskedLoop{bulk}(
r in tup{VT, r},
x in tup{VT, x},
b in tup{'b', VT, bits},
M in 'm'
over i to len
) {
# x, r, and b now have type VT
r1:= r + x - b # subtracting b because it's all 1s for a one
if (homAny{M{r1 < r}}) return{0} # detect overflow
r = r1 # will be mask-stored if needed
}
1
}
fn acc_u32_u16_bit(r:*u32, x:*u16, bits:*u64, len:u64) : u1 = { # @muLoop, 2x unrolled
def bulk = 8
def VT = [8]u32
@muLoop{bulk,2}(
r in tup{'g', VT, r},
x in tup{VT, x},
b in tup{'b', VT, bits},
M in 'm'
over i to len
) {
def r0 = r{}
def r1 = each{-, each{+,r0,x}, b}
if (homAny{M{tree_fold{|, each{<,r1,r0}}}}) return{0}
r{r1}
}
1
}