uCBQN/src/singeli
2023-11-07 14:54:16 -05:00
..
c fix completely wrong dyTable lookup 2023-08-15 14:45:27 +03:00
src 64-bit hashes with float normalization 2023-11-07 14:54:16 -05:00
README.md Singeli renames & docs 2023-07-22 17:37:20 +03:00

Overview

Types used here follow [VET][ISUF]?\d?.
First character:

  • V: some vector type
  • E: some scalar type
  • T: either scalar or vector type

Second character, if present:

  • F: float
  • S: signed integer
  • U: unsigned integer (may include u1)
  • I: any signedness integer (may include u1)

Equally-named types should match. A number is added to the type to separate unequal types.

mt{T} is a mask version of the argument - either ty_u{T} or [vcount{T}]u1 depending on the target.

The SIMD operations listed aren't guaranteed to be supported on all targets, nor list all possible argument types.

General definitons

  • has_simd
    0 or 1 depending on whether SIMD types are available.

  • arch_defvw
    Target vector width for the current target architecture.
    If has_simd==0, should not be used. Otherwise, vectors of all element types (except u1) whose width is this must be supported, but other widths (both larger and smaller) may be too.

  • reinterpret is available as ~~

  • exportN{f, 'name1', 'name2', ...} - export single function under many names
  • exportT{'name', fs} - export list of functions as the specified name
  • iota{knum} - tuple tup{0,1,...,knum-1}
  • broadcast{knum,v} (knum**v) - tuple with knum items, all v
  • tern{c, t, f} - ternary - if c, return t, else f
  • eachx{F, ...args} - each but broadcasting non-tuples
  • undef{T} - return register of type T with undefined value; maps over a tuple of types
  • undef{T, n} - a tuple of n different undefined registers
  • oneVal{xs} - assert that xs is a tuple of equivalent (as per ==) items, and return the first
  • oneType{xs} - oneVal but over the types of the items

Loops & branches

  • @for - generate body once, with indices s,s+1,...,e-2,e-1
  • @forNZ - @for, but assumed to run at least once
  • @for_backwards - @for, but indices go backwards e-1,e-2,...,s+1,s
  • @forUnroll{exp, unr} - generate a main loop body that takes a tuple of unr indices to process. Handle the tail via processing one index at a time; if exp==1, that's done via unr generated bodies, otherwise it's a loop
  • @unroll(n) - generate body n times
  • @collect(n) - generate body n times, and collect the results as a tuple
  • makeBranch{Ts, G=={...x:Ts}=>{}} == {...x:Ts}=>{} - make a code segment that takes arguments of types Ts, and can be jumped to by invoking the result of the makeBranch
  • makeOptBranch{enable, Ts, G} - makeBranch but only generated & invokable when enable==1

Scalar operations

  • popc{x:EI} : uint - population count
  • popcRand{x:EI} : uint - population count; under valgrind, return a random value in the set of possible ones if the input is partly undefined
  • ctz{x:EI} : uint - count trailing zeroes; if x==0, the result is undefined
  • clz{x:EI} : uint - count leading zeroes; if x==0, the result is undefined
  • clzc{x:EI} : uint - width{EI}-clz{x}; if x==0, the result is undefined
  • zlow{n, x} - zero low n bits of x
  • tail{n, x} - keep low n bits of x
  • bit{n, x} - get n-th bit of x
  • lb{n} - base-2 log of a constant power of two
  • base{b, x} - convert tuple to number in little-endian base b
  • tree_fold{G, xs} - fold tuple xs with a tree-like pattern with G

Hints

  • rare{x:u1} : u1 - Expect that x is 0. Returns x.
  • likely{x:u1} : u1 - Expect that x is 1. Returns x.
  • assert{x:u1} : void - assume x is 1. If x is a compile-time constant, fails to compile if it's not 1. If at runtime it is not 1, undefined behavior happens (unless debug mode, via either include './debug' or CBQN-wide DEBUG mode)

Type operations

Of the below functions which take & return vector types of the same width, they can instead take a vector value & reinterpret it to the result type.

  • w_n{E, w} == E2 - a scalar type with the quality of E and width w
  • w_d{E} == E2, w_h{E} == E2 - double/halve the width of a scalar type
  • n_d{V} == V2, n_h{V} == V2 - double/halve element count in vector (keeps the same element width)
  • el_d{V} == V2, el_h{V} == V2 - double/halve element width in vector (keeps the same count)
  • el_m{V} == V2, el_s{V} == V2 - vector with the same total width, but element width doubled (el_m - elements merged) or halved (el_s - elements split)
  • ty_u{T} == TU - unsigned integer version of the type
  • ty_s{T} == TS - signed integer version of the type
  • ty_f{T} == TF - float version of the type
  • re_el{E, V} == V2 - reinterpret vector to one with the specified element - eltype{V2}==E and width{V2}==width{V}
  • minvalue{EI}, maxvalue{EI} - smallest & biggest value of the specified type

Arithmetic

Some may also support one scalar argument or arguments with different widths.

  • __neg{a} - -a
  • __add{a:V, b:V} : V - a + b
  • __sub{a:VI, b:VI} : VI - a - b
  • __div{a:V, b:V} : V - a / b
  • __mul{a:V, b:V} : V - a * b
  • mulHi{a:VI, b:VI} : VI - high half of multiplication
  • __eq, __ne, __ge, __gt, __le, __lt - comparison; {a:V, b:V} : mt{V}
  • __not{a:VI} : VI - ~a
  • __and{a:VI, b:VI} : VI - a&b
  • __or{a:VI, b:VI} : VI - a|b
  • __xor{a:VI, b:VI} : VI - a^b
  • andnot{a:VI, b:VI} : VI - a&~b
  • __shl{a:VI, b:TI} : VI - a<<b; shift left
  • __shr{a:VS, b:TI} : VS - a>>b; arithmetic shift right
  • __shr{a:VU, b:TI} : VU - a>>b; logical shift right
  • abs{a:V} : V - absolute value
  • absu{a:VI} : ty_u{VI} - absolute value, reinterpreted as unsigned
  • adds{a:VI, b:VI} : VI - saturating add
  • subs{a:VI, b:VI} : VI - saturating subtract
  • rcpE{a:VF} : VF - estimate of 1/a; precision depends
  • rsqrtE{a:VF} : VF - estimate of sqrt{1/a}; precision depends
  • sqrt{a:VT} : VT - square root
  • floor{a:VF} : VF - floor
  • ceil{a:VF} : VF - ceiling
  • max{a:V, b:V} : V - minimum value; arch-specific behavior on NaNs
  • min{a:V, b:V} : V - maximum value; arch-specific behavior on NaNs
  • shl{U, a:V, n} - shift vector elements within blocks of U
  • shr{U, a:V, n} - shift vector elements within blocks of U
  • unord{a:VF, b:VF} : mt{VI} - (a==NaN) | (b==NaN)
  • andAllZero{a:VI, b:VI} : u1 - whether a&b is all zeroes

Structural operations

  • broadcast{V, a:eltype{V}} : V (V**a) - broadcast (splat) the scalar to all elements of the result
  • make{V, ...vs} : V or make{V, tup{...vs}} - initialize a vector with the specified elements
  • iota{V} : V - make{V, range{vcount{V}}}
  • extract{a:V, n} : eltype{V} - extract the n-th element
  • insert{a:V, i, b:eltype{V}} : V - a copy of a with the i-th element replaced with b
  • half{a:V, n} : count_half{V} - extract either the low (n==0) or high (n==1) half of a
  • pair{a:V, b:V} : count_dbl{V} - make a vector from halves
  • undefPromote{V2, a:V} : V - promote a to a wider type, leaving the upper part undefined
  • vshl{a:V, b:V, n} : V - vshl{[0,1,2,3], [4,5,6,7], 1} → [1,2,3,4]
  • sel{VI1, a:V, b:VI2} : V2 - shuffle a by indices b in VI1-long lanes; arch-specific behavior on out-of-bounds values
  • zip{a:T,b:T} : tup{T, T} - zip{[0,1,2,3], [4,5,6,7]} → tup{[0,4,1,5], [2,6,3,7]}
  • mzip{a:T,b:T} : tup{el_dbl{T}, el_dbl{T}} - reinterpreted zip{a, b}
  • zipLo{a:T,b:T} : T - tupsel{0, zip{a, b}}
  • zipHi{a:T,b:T} : T - tupsel{1, zip{a, b}}
  • mzipLo{a:T,b:T} : T - tupsel{0, mzip{a, b}}
  • mzipHi{a:T,b:T} : T - tupsel{1, mzip{a, b}}

Mask stuff

Homogeneous definitions (i.e. ones with hom in their name) assume that each element in the mask type has all its bits equal.

  • homAll{a:VI} : u1 - whether all elements are set
  • homAny{a:VI} : u1 - whether any element is set
  • topAny{a:VI} : u1 - whether all elements have their top bit set
  • topAll{a:VI} : u1 - whether any element has its top bit set
  • homBlend{f:V, t:V, m:mt{V}} : V - blend by m, setting to f where 0 and t where 1
  • topBlend{f:V, t:V, m:V} : V - blend by top bit of m
  • homMask{a:VI} : uint - integer mask of whether each element is set (assumes each element has all its bits equal)
  • topMask{a:VI} : uint - integer mask of the top bit of each element
  • homMaskX{a:VI} : tup{knum, uint} - integer mask where each element is represented by knum bits (possibly more efficient to calculate than homMask)
  • ctzX{tup{knum, uint}} - count trailing zeroes from a result of homMaskX

Load/store

Alignment requirements are target-specific, but at most one element.

For unaligned scalar loads & stores, loadu & storeu should be used.

  • loadu{p:*E} : E - load scalar from unaligned memory
  • storeu{p:*E, a:V} : void - store scalar to unaligned memory
  • load{p:*V} : V - load full vector
  • store{p:*V, a:V} : void - store full vector
  • loadLow{p:*V, w} : V - load to low w bits
  • storeLow{p:*V, w, a:V} - store low w bits
  • homMaskStore{p:*V, m:mt{V}, a:V} - conditionally store elements based on mask; won't touch masked-off elements
  • topMaskStore{p:*V, m:V, a:V} - conditionally store elements based on top bit of m; won't touch masked-off elements
  • homMaskStoreF - homMaskStore but may touch masked-off elements and thus be supported on more types
  • topMaskStoreF - topMaskStore but may touch masked-off elements and thus be supported on more types

Conversion

For float conversions, the used rounding mode is unspecified.

  • narrow{eltype{V2}, a:V} : V2 - narrow elements of a to desired type.
    For integer-to-integer conversion, completely undefined result if the values don't exactly fit.
    Provides vcount{V2} >= vcount{V}.
  • widen{V2, a:V} : V2 - widen elements of a to desired type.
    Discards elements of a past vcount{V2}.
  • cvt{eltype{V2}, a:V} : V2 - convert element type at the same width.
    Requires vcount{V}==vcount{V2} and elwidth{V}==elwidth{V2}.

x86-specific

  • mul32{a:VI, b:VI} : VI - multiply, reading only low 32 bits
  • blend{L, a:V, b:V, m} - blend L-sized blocks via the immediate
  • shuf{L, x:T, n} : T - shuffle by immediate in L-sized lanes
  • packQ - pack 128-bit lanes (packs/packus) for 16-bit & 32-bit elements
  • packQQ - packQ but also defined for 64-bit elements, assuming the high halves are zeroes
  • packs - 128-bit packs/packus
  • unpackQ - unpack in 128-bit lanes (like mzip)
  • shuf16Hi, shuf16Lo - 16-bit shuffles with immediate
  • shufHalves

NEON-specific

folds

  • fold_add{x:V} : eltype{V}
  • fold_addw{x:V} : ty_dbl{eltype{V}}
  • fold_max{x:V} : eltype{V}
  • fold_min{x:V} : eltype{V}
  • vfold{F, x:V} : eltype{V} for F in min, max, __add

arithmetic

  • addp - add pairwise
  • addpw - add pairwise, widening
  • addpwa - add pairwise, widening; accumulate
  • addwLo - widening add
  • subwLo - widening subtract
  • mulwLo, mulwHi, mulw - widening multiply
  • andnz - element-wise (a&b)!=0? ~0 : 0
  • bitAll - are all bits 1s
  • bitAny - is any bit a 1
  • bitBlend - blend by bits
  • clz - count leading zeroes
  • cls - count leading sign bits
  • copyLane
  • mla - multiply-add
  • mls - multiply-subtract
  • ornot - a|~b
  • packLo, packHi
  • popc - popcount of each byte
  • rbit - reverse bits in bytes
  • rev - reverse elements in lanes
  • shlm, shrm - fun shifts with defaults
  • shrn - narrowing shift right
  • trn1, trn2 - 2×2 transposes from pairs of elements across the two arguments

mask.singeli

  • maskOf{TU, n} - get a mask of type TU whose first n elements are all 1s, and the remaining are 0s. 0≤n≤vcount{TU}
  • maskNone - a mask generator that matches all items
  • maskAfter{n} - a mask generator that patches the first n items
  • loadBatch{p:*E, n, V} - load the n-th batch of vcount{V} elements from *E; if E isn't eltype{V}, the result is sign- or zero-extended.
  • storeBatch{p:*E, n, x:V, M} - store equivalent of the loadBatch; if E isn't eltype{V}, x is narrowed via narrow{}

To test whether a mask object M is maskNone or maskAfter, M{0} can be used - maskNone{0} is 0, but maskAfter{n}{0} is 1.

Loops

  • @maskedLoop{bulk} - loop that generates its body twice, once with a maskNone mask, and once with a maskAfter{n} one to handle the tail.

  • @muLoop{bulk, unr} - masked & unrolled loop - generates its body three times (or two if unr==1) - once for unrolled main loop, once for the unrolling leftover, and once for the masked end. Unrolling is handled by passing a tuple of indices to process as the index variable (the tail generated bodies get a tuple of the one index)

  • @muLoop{bulk, unr, fromunr} - fromunr is additionally ran after exiting the unrolled loop (such that you can have separate accumulators for the unrolled bit, and convert them to single-vector acccumulators for the tail).

Tuples can be used in the iterated variable list for various things:

p:*T - regular pointer
tup{VT,p:*T} - loadBatch/storeBatch vector data
tup{'b',p:P} - b_getBatch (bits.singeli)
tup{'b',VT,p:P} - loadBatchBit (bits.singeli)
tup{'g',VT,p:*T} - gives a generator, used as g{} to loadBatch and g{newValue} to storeBatch
tup{'g',p:*T} - the above, but without load support
'm' - get the mask object - either maskNone or some maskAfter{n}

Stores via those will implicitly do a masked store when required.

Loads will load past the end, so M in 'm' must be used to mask off elements if they're used for something other than the above stores.

For muLoop, M in 'm' still gives a single generator, but it's only ever maskAfter{n} when there's only one index.

Example usage for a loop that adds u16 and bit boolean elements to a u32 accumulator, early-exiting on overflow:

fn acc_u32_u16_bit(r:*u32, x:*u16, bits:*u64, len:u64) : u1 = { # @maskedLoop
  def bulk = 8
  def VT = [8]u32
  
  @maskedLoop{bulk}(
    r in tup{VT, r},
    x in tup{VT, x},
    b in tup{'b', VT, bits},
    M in 'm'
    over i to len
  ) {
    # x, r, and b now have type VT
    r1:= r + x - b # subtracting b because it's all 1s for a one
    if (homAny{M{r1 < r}}) return{0} # detect overflow
    r = r1 # will be mask-stored if needed
  }
  1
}

fn acc_u32_u16_bit(r:*u32, x:*u16, bits:*u64, len:u64) : u1 = { # @muLoop, 2x unrolled
  def bulk = 8
  def VT = [8]u32
  
  @muLoop{bulk,2}(
    r in tup{'g', VT, r},
    x in tup{VT, x},
    b in tup{'b', VT, bits},
    M in 'm'
    over i to len
  ) {
    def r0 = r{}
    def r1 = each{-, each{+,r0,x}, b}
    if (homAny{M{tree_fold{|, each{<,r1,r0}}}}) return{0}
    r{r1}
  }
  1
}