Singeli renames & docs

This commit is contained in:
dzaima 2023-07-22 14:11:28 +03:00
parent 6e96600878
commit 446b2d09ed
17 changed files with 456 additions and 112 deletions

355
src/singeli/README.md Normal file
View File

@ -0,0 +1,355 @@
## Overview
Types used here follow `[VET][ISUF]?\d?`.
First character:
- `V`: some vector type
- `E`: some scalar type
- `T`: either scalar or vector type
Second character, if present:
- `F`: float
- `S`: signed integer
- `U`: unsigned integer (may include `u1`)
- `I`: any signedness integer (may include `u1`)
Equally-named types should match. A number is added to the type to separate unequal types.
`mt{T}` is a mask version of the argument - either `ty_u{T}` or `[vcount{T}]u1` depending on the target.
The SIMD operations listed aren't guaranteed to be supported on all targets, nor list all possible argument types.
## General definitons
- `has_simd`
`0` or `1` depending on whether SIMD types are available.
- `arch_defvw`
Target vector width for the current target architecture.
If `has_simd==0`, should not be used. Otherwise, vectors of all element types (except `u1`) whose width is this must be supported, but other widths (both larger and smaller) may be too.
- `reinterpret` is available as `~~`
<!-- -->
- `exportN{f, 'name1', 'name2', ...}` - export single function under many names
- `exportT{'name', fs}` - export list of functions as the specified name
- `iota{knum}` - tuple `tup{0,1,...,knum-1}`
- `broadcast{knum,v}` (`knum**v`) - tuple with `knum` items, all `v`
- `tern{c, t, f}` - ternary - if `c`, return `t`, else `f`
- `eachx{F, ...args}` - `each` but broadcasting non-tuples
- `undef{T}` - return register of type `T` with undefined value; maps over a tuple of types
- `undef{T, n}` - a tuple of `n` different undefined registers
- `oneVal{xs}` - assert that `xs` is a tuple of equivalent (as per `==`) items, and return the first
- `oneType{xs}` - `oneVal` but over the types of the items
## Loops & branches
- `@for` - generate body once, with indices `s,s+1,...,e-2,e-1`
- `@forNZ` - `@for`, but assumed to run at least once
- `@for_backwards` - `@for`, but indices go backwards `e-1,e-2,...,s+1,s`
- `@forUnroll{exp, unr}` - generate a main loop body that takes a tuple of `unr` indices to process. Handle the tail via processing one index at a time; if `exp==1`, that's done via `unr` generated bodies, otherwise it's a loop
- `@unroll(n)` - generate body `n` times
- `@collect(n)` - generate body `n` times, and collect the results as a tuple
<!-- -->
- `makeBranch{Ts, G=={...x:Ts}=>{}} == {...x:Ts}=>{}` - make a code segment that takes arguments of types `Ts`, and can be jumped to by invoking the result of the `makeBranch`
- `makeOptBranch{enable, Ts, G}` - `makeBranch` but only generated & invokable when `enable==1`
## Scalar operations
- `popc{x:EI} : uint` - population count
- `popcRand{x:EI} : uint` - population count; under valgrind, return a random value in the set of possible ones if the input is partly undefined
- `ctz{x:EI} : uint` - count trailing zeroes; if `x==0`, the result is undefined
- `clz{x:EI} : uint` - count leading zeroes; if `x==0`, the result is undefined
- `clzc{x:EI} : uint` - `width{EI}-clz{x}`; if `x==0`, the result is undefined
<!-- -->
- `zlow{n, x}` - zero low `n` bits of `x`
- `tail{n, x}` - keep low `n` bits of `x`
- `bit{n, x}` - get `n`-th bit of `x`
<!-- -->
- `lb{n}` - base-2 log of a constant power of two
- `base{b, x}` - convert tuple to number in little-endian base `b`
- `tree_fold{G, xs}` - fold tuple `xs` with a tree-like pattern with `G`
## Hints
- `rare{x:u1} : u1` - Expect that `x` is `0`. Returns `x`.
- `likely{x:u1} : u1` - Expect that `x` is `1`. Returns `x`.
- `assert{x:u1} : void` - assume `x` is `1`. If `x` is a compile-time constant, fails to compile if it's not `1`. If at runtime it is not `1`, undefined behavior happens (unless debug mode, via either `include './debug'` or CBQN-wide `DEBUG` mode)
## Type operations
Of the below functions which take & return vector types of the same width, they can instead take a vector value & reinterpret it to the result type.
- `w_n{E, w} == E2` - a scalar type with the quality of `E` and width `w`
- `w_d{E} == E2`, `w_h{E} == E2` - double/halve the width of a scalar type
- `n_d{V} == V2`, `n_h{V} == V2` - double/halve element count in vector (keeps the same element width)
- `el_d{V} == V2`, `el_h{V} == V2` - double/halve element width in vector (keeps the same count)
- `el_m{V} == V2`, `el_s{V} == V2` - vector with the same total width, but element width doubled (`el_m` - elements merged) or halved (`el_s` - elements split)
<!-- -->
- `ty_u{T} == TU` - unsigned integer version of the type
- `ty_s{T} == TS` - signed integer version of the type
- `ty_f{T} == TF` - float version of the type
- `re_el{E, V} == V2` - reinterpret vector to one with the specified element - `eltype{V2}==E and width{V2}==width{V}`
<!-- -->
- `minvalue{EI}`, `maxvalue{EI}` - smallest & biggest value of the specified type
## Arithmetic
Some may also support one scalar argument or arguments with different widths.
- `__neg{a}` - `-a`
- `__add{a:V, b:V} : V` - `a + b`
- `__sub{a:VI, b:VI} : VI` - `a - b`
- `__div{a:V, b:V} : V` - `a / b`
- `__mul{a:V, b:V} : V` - `a * b`
- `mulHi{a:VI, b:VI} : VI` - high half of multiplication
<!-- -->
- `__eq`, `__ne`, `__ge`, `__gt`, `__le`, `__lt` - comparison; `{a:V, b:V} : mt{V}`
<!-- -->
- `__not{a:VI} : VI` - `~a`
- `__and{a:VI, b:VI} : VI` - `a&b`
- `__or{a:VI, b:VI} : VI` - `a|b`
- `__xor{a:VI, b:VI} : VI` - `a^b`
- `andnot{a:VI, b:VI} : VI` - `a&~b`
- `__shl{a:VI, b:TI} : VI` - `a<<b`; shift left
- `__shr{a:VS, b:TI} : VS` - `a>>b`; arithmetic shift right
- `__shr{a:VU, b:TI} : VU` - `a>>b`; logical shift right
<!-- -->
- `abs{a:V} : V` - absolute value
- `absu{a:VI} : ty_u{VI}` - absolute value, reinterpreted as unsigned
- `adds{a:VI, b:VI} : VI` - saturating add
- `subs{a:VI, b:VI} : VI` - saturating subtract
<!-- -->
- `rcpE{a:VF} : VF` - estimate of `1/a`; precision depends
- `rsqrtE{a:VF} : VF` - estimate of `sqrt{1/a}`; precision depends
<!-- -->
- `sqrt{a:VT} : VT` - square root
- `floor{a:VF} : VF` - floor
- `ceil{a:VF} : VF` - ceiling
- `max{a:V, b:V} : V` - minimum value; arch-specific behavior on NaNs
- `min{a:V, b:V} : V` - maximum value; arch-specific behavior on NaNs
<!-- -->
- `shl{U, a:V, n}` - shift vector elements within blocks of `U`
- `shr{U, a:V, n}` - shift vector elements within blocks of `U`
- `unord{a:VF, b:VF} : mt{VI}` - `(a==NaN) | (b==NaN)`
<!-- -->
- `andAllZero{a:VI, b:VI} : u1` - whether `a&b` is all zeroes
## Structural operations
- `broadcast{V, a:eltype{V}} : V` (`V**a`) - broadcast (splat) the scalar to all elements of the result
- `make{V, ...vs} : V` or `make{V, tup{...vs}}` - initialize a vector with the specified elements
- `iota{V} : V` - `make{V, range{vcount{V}}}`
- `extract{a:V, n} : eltype{V}` - extract the n-th element
- `insert{a:V, i, b:eltype{V}} : V` - a copy of `a` with the `i`-th element replaced with `b`
- `half{a:V, n} : count_half{V}` - extract either the low (`n==0`) or high (`n==1`) half of `a`
- `pair{a:V, b:V} : count_dbl{V}` - make a vector from halves
- `undefPromote{V2, a:V} : V` - promote `a` to a wider type, leaving the upper part undefined
- `vshl{a:V, b:V, n} : V` - `vshl{[0,1,2,3], [4,5,6,7], 1} → [1,2,3,4]`
- `sel{VI1, a:V, b:VI2} : V2` - shuffle `a` by indices `b` in `VI1`-long lanes; arch-specific behavior on out-of-bounds values
<!-- -->
- `zip{a:T,b:T} : tup{T, T}` - `zip{[0,1,2,3], [4,5,6,7]} → tup{[0,4,1,5], [2,6,3,7]}`
- `mzip{a:T,b:T} : tup{el_dbl{T}, el_dbl{T}}` - reinterpreted `zip{a, b}`
- `zipLo{a:T,b:T} : T` - `tupsel{0, zip{a, b}}`
- `zipHi{a:T,b:T} : T` - `tupsel{1, zip{a, b}}`
- `mzipLo{a:T,b:T} : T` - `tupsel{0, mzip{a, b}}`
- `mzipHi{a:T,b:T} : T` - `tupsel{1, mzip{a, b}}`
## Mask stuff
Homogeneous definitions (i.e. ones with `hom` in their name) assume that each element in the mask type has all its bits equal.
- `homAll{a:VI} : u1` - whether all elements are set
- `homAny{a:VI} : u1` - whether any element is set
- `topAny{a:VI} : u1` - whether all elements have their top bit set
- `topAll{a:VI} : u1` - whether any element has its top bit set
- `homBlend{f:V, t:V, m:mt{V}} : V` - blend by `m`, setting to `f` where `0` and `t` where `1`
- `topBlend{f:V, t:V, m:V} : V` - blend by top bit of `m`
- `homMask{a:VI} : uint` - integer mask of whether each element is set (assumes each element has all its bits equal)
- `topMask{a:VI} : uint` - integer mask of the top bit of each element
- `homMaskX{a:VI} : tup{knum, uint}` - integer mask where each element is represented by `knum` bits (possibly more efficient to calculate than `homMask`)
- `ctzX{tup{knum, uint}}` - count trailing zeroes from a result of `homMaskX`
## Load/store
Alignment requirements are target-specific, but at most one element.
For unaligned scalar loads & stores, `loadu` & `storeu` should be used.
- `loadu{p:*E} : E` - load scalar from unaligned memory
- `storeu{p:*E, a:V} : void` - store scalar to unaligned memory
- `load{p:*V} : V` - load full vector
- `store{p:*V, a:V} : void` - store full vector
- `loadLow{p:*V, w} : V` - load to low `w` bits
- `storeLow{p:*V, w, a:V}` - store low `w` bits
- `homMaskStore{p:*V, m:mt{V}, a:V}` - conditionally store elements based on mask; won't touch masked-off elements
- `topMaskStore{p:*V, m:V, a:V}` - conditionally store elements based on top bit of `m`; won't touch masked-off elements
- `homMaskStoreF` - `homMaskStore` but may touch masked-off elements and thus be supported on more types
- `topMaskStoreF` - `topMaskStore` but may touch masked-off elements and thus be supported on more types
<!-- useless x86 defs for vector-width-aligned load/store: loada storea -->
## Conversion
For float conversions, the used rounding mode is unspecified.
- `narrow{eltype{V2}, a:V} : V2` - narrow elements of `a` to desired type.
For integer-to-integer conversion, completely undefined result if the values don't exactly fit.
Provides `vcount{V2} >= vcount{V}`.
- `widen{V2, a:V} : V2` - widen elements of `a` to desired type.
Discards elements of `a` past `vcount{V2}`.
- `cvt{eltype{V2}, a:V} : V2` - convert element type at the same width.
Requires `vcount{V}==vcount{V2} and elwidth{V}==elwidth{V2}`.
<!-- cvt2 -->
## x86-specific
- `mul32{a:VI, b:VI} : VI` - multiply, reading only low 32 bits
- `blend{L, a:V, b:V, m}` - blend `L`-sized blocks via the immediate
- `shuf{L, x:T, n} : T` - shuffle by immediate in `L`-sized lanes
- `packQ` - pack 128-bit lanes (`packs`/`packus`) for 16-bit & 32-bit elements
- `packQQ` - `packQ` but also defined for 64-bit elements, assuming the high halves are zeroes
- `packs` - 128-bit `packs`/`packus`
- `unpackQ` - unpack in 128-bit lanes (like `mzip`)
- `shuf16Hi`, `shuf16Lo` - 16-bit shuffles with immediate
- `shufHalves`
## NEON-specific
### folds
- `fold_add{x:V} : eltype{V}`
- `fold_addw{x:V} : ty_dbl{eltype{V}}`
- `fold_max{x:V} : eltype{V}`
- `fold_min{x:V} : eltype{V}`
- `vfold{F, x:V} : eltype{V}` for F in `min`, `max`, `__add`
### arithmetic
- `addp` - add pairwise
- `addpw` - add pairwise, widening
- `addpwa` - add pairwise, widening; accumulate
- `addwLo` - widening add
- `subwLo` - widening subtract
- `mulwLo`, `mulwHi`, `mulw` - widening multiply
- `andnz` - element-wise (a&b)!=0? ~0 : 0
<!-- -->
- `bitAll` - are all bits 1s
- `bitAny` - is any bit a 1
- `bitBlend` - blend by bits
- `clz` - count leading zeroes
- `cls` - count leading sign bits
- `copyLane`
- `mla` - multiply-add
- `mls` - multiply-subtract
- `ornot` - a|~b
- `packLo`, packHi
- `popc` - popcount of each byte
- `rbit` - reverse bits in bytes
- `rev` - reverse elements in lanes
- `shlm`, shrm - fun shifts with defaults
- `shrn` - narrowing shift right
- `trn1`, `trn2` - 2×2 transposes from pairs of elements across the two arguments
<!--
widenUpper{x:V}, widen{x:V}
narrowPair, narrowUpper
-->
## mask.singeli
- `maskOf{TU, n}` - get a mask of type `TU` whose first `n` elements are all `1`s, and the remaining are `0`s. `0≤n≤vcount{TU}`
- `maskNone` - a mask generator that matches all items
- `maskAfter{n}` - a mask generator that patches the first `n` items
- `loadBatch{p:*E, n, V}` - load the `n`-th batch of `vcount{V}` elements from `*E`; if `E` isn't `eltype{V}`, the result is sign- or zero-extended.
- `storeBatch{p:*E, n, x:V, M}` - store equivalent of the `loadBatch`; if `E` isn't `eltype{V}`, `x` is narrowed via `narrow{}`
To test whether a mask object `M` is `maskNone` or `maskAfter`, `M{0}` can be used - `maskNone{0}` is `0`, but `maskAfter{n}{0}` is `1`.
### Loops
- `@maskedLoop{bulk}` - loop that generates its body twice, once with a `maskNone` mask, and once with a `maskAfter{n}` one to handle the tail.
- `@muLoop{bulk, unr}` - masked & unrolled loop - generates its body three times (or two if `unr==1`) - once for unrolled main loop, once for the unrolling leftover, and once for the masked end.
Unrolling is handled by passing a tuple of indices to process as the index variable (the tail generated bodies get a tuple of the one index)
- `@muLoop{bulk, unr, fromunr}` - `fromunr` is additionally ran after exiting the unrolled loop (such that you can have separate accumulators for the unrolled bit, and convert them to single-vector acccumulators for the tail).
Tuples can be used in the iterated variable list for various things:
```
p:*T - regular pointer
tup{VT,p:*T} - loadBatch/storeBatch vector data
tup{'b',p:P} - b_getBatch (bits.singeli)
tup{'b',VT,p:P} - loadBatchBit (bits.singeli)
tup{'g',VT,p:*T} - gives a generator, used as g{} to loadBatch and g{newValue} to storeBatch
tup{'g',p:*T} - the above, but without load support
'm' - get the mask object - either maskNone or some maskAfter{n}
```
Stores via those will implicitly do a masked store when required.
Loads will load past the end, so `M in 'm'` must be used to mask off elements if they're used for something other than the above stores.
For `muLoop`, `M in 'm'` still gives a single generator, but it's only ever `maskAfter{n}` when there's only one index.
Example usage for a loop that adds `u16` and bit boolean elements to a `u32` accumulator, early-exiting on overflow:
```
fn acc_u32_u16_bit(r:*u32, x:*u16, bits:*u64, len:u64) : u1 = { # @maskedLoop
def bulk = 8
def VT = [8]u32
@maskedLoop{bulk}(
r in tup{VT, r},
x in tup{VT, x},
b in tup{'b', VT, bits},
M in 'm'
over i to len
) {
# x, r, and b now have type VT
r1:= r + x - b # subtracting b because it's all 1s for a one
if (homAny{M{r1 < r}}) return{0} # detect overflow
r = r1 # will be mask-stored if needed
}
1
}
fn acc_u32_u16_bit(r:*u32, x:*u16, bits:*u64, len:u64) : u1 = { # @muLoop, 2x unrolled
def bulk = 8
def VT = [8]u32
@muLoop{bulk,2}(
r in tup{'g', VT, r},
x in tup{VT, x},
b in tup{'b', VT, bits},
M in 'm'
over i to len
) {
def r0 = r{}
def r1 = each{-, each{+,r0,x}, b}
if (homAny{M{tree_fold{|, each{<,r1,r0}}}}) return{0}
r{r1}
}
1
}
```
<!--
bitops.singeli
b_get
b_getBatch
b_getBatchLo
b_set
b_setBatch
loadBatchBit
loadu
loaduBit
loaduBitRaw
loaduBitTrunc
ones
spreadBits
bmi2.singeli
pdep
pext
clmul.singeli
clmul
-->

View File

@ -19,7 +19,7 @@ def storea{a:T, n, v & w256f{eltype{T},64}} = emit{void, '_mm256_store_pd', *f6
def store {a:T, n, v & w256f{eltype{T},32}} = emit{void, '_mm256_storeu_ps', *f32 ~~ (a+n), v}
def storea{a:T, n, v & w256f{eltype{T},32}} = emit{void, '_mm256_store_ps', *f32 ~~ (a+n), v}
def loadLow{ptr:P, w & w256{eltype{P}} & w<=128} = undefPromote{eltype{P}, loadLow{*v_half{eltype{P}} ~~ ptr, w}}
def loadLow{ptr:P, w & w256{eltype{P}} & w<=128} = undefPromote{eltype{P}, loadLow{*n_h{eltype{P}} ~~ ptr, w}}
def loadLow{ptr:P, w & w256{eltype{P}} & w==256} = load{ptr}
def storeLow{ptr:P, w, x:T & w256{T} & w<=128} = storeLow{ptr, w, half{x, 0}}
@ -91,9 +91,9 @@ def floor{a:[4]f64} = emit{[4]f64, '_mm256_floor_pd', a}
def ceil{a:[4]f64} = emit{[4]f64, '_mm256_ceil_pd', a}
# conversion
def half{x:T, i & w256{T} & knum{i}} = v_half{T} ~~ emit{[8]i16, '_mm256_extracti128_si256', v2i{x}, i}
def half{x:T, i==0 & w256{T}} = v_half{T} ~~ emit{[8]i16, '_mm256_castsi256_si128', v2i{x}}
def pair{a:T,b:T & width{T}==128} = v_dbl{T} ~~ emit{[8]i32, '_mm256_setr_m128i', a, b}
def half{x:T, i & w256{T} & knum{i}} = n_h{T} ~~ emit{[8]i16, '_mm256_extracti128_si256', v2i{x}, i}
def half{x:T, i==0 & w256{T}} = n_h{T} ~~ emit{[8]i16, '_mm256_castsi256_si128', v2i{x}}
def pair{a:T,b:T & width{T}==128} = n_d{T} ~~ emit{[8]i32, '_mm256_setr_m128i', a, b}
def widen{T==[4]f64, x:X & X==[4]i32} = emit{T, '_mm256_cvtepi32_pd', x}
def widen{T==[4]f64, x:X & X==[4]f32} = emit{T, '_mm256_cvtps_pd', x}

View File

@ -156,17 +156,17 @@ def widen{T==[ 4]u64, x:X & X==[4]u32} = emit{T, '_mm256_cvtepu32_epi64', x}; de
def narrow{T, x:X & w256i{X,32} & width{T}==8} = {
a:= packQ{x, x}
b:= packQ{a, a}
to_el{T, sel{[8]u32, b, make{[8]i32, 0,4,0,4,0,4,0,4}}}
re_el{T, sel{[8]u32, b, make{[8]i32, 0,4,0,4,0,4,0,4}}}
}
def narrow{T, x:X & w256i{X,32} & width{T}==16} = to_el{T, shuf{[4]u64, packQ{x, x}, 4b3120}}
def narrow{T, x:X & w256i{X,16} & width{T}== 8} = to_el{T, shuf{[4]u64, packQ{x, x}, 4b3120}}
def narrow{T, x:X & w256i{X,32} & width{T}==16} = re_el{T, shuf{[4]u64, packQ{x, x}, 4b3120}}
def narrow{T, x:X & w256i{X,16} & width{T}== 8} = re_el{T, shuf{[4]u64, packQ{x, x}, 4b3120}}
def narrow{T, x:X & w256f{X,64} & T<i32} = narrow{T, narrow{i32, x}}
def narrow{T, x:X & w256f{X,64} & T==i32} = emit{[4]i32, '_mm256_cvtpd_epi32', x}
def narrow{T, x:X & w256u{X,64} & T==u32} = to_el{T, sel{[8]i32, x, make{[8]i32, 2*iota{8}}}}
def narrow{T, x:X & w256u{X,64} & T==u16} = to_el{T, sel{[16]i8, narrow{u32,x}, make{[32]i8, (iota{32}>>1<<2) | (iota{32}&1)}}}
def narrow{T, x:X & w256u{X,64} & T== u8} = to_el{T, sel{[16]i8, narrow{u32,x}, make{[32]i8, 4*iota{32}}}}
def narrow{T, x:X & w256u{X,64} & T==u32} = re_el{T, sel{[8]i32, x, make{[8]i32, 2*iota{8}}}}
def narrow{T, x:X & w256u{X,64} & T==u16} = re_el{T, sel{[16]i8, narrow{u32,x}, make{[32]i8, (iota{32}>>1<<2) | (iota{32}&1)}}}
def narrow{T, x:X & w256u{X,64} & T== u8} = re_el{T, sel{[16]i8, narrow{u32,x}, make{[32]i8, 4*iota{32}}}}
def cvt2{T, x:X & T==i32 & X==[4]f64} = emit{[4]i32, '_mm256_cvtpd_epi32', x}

View File

@ -27,6 +27,8 @@ def load{p:P & isptr{P}} = load{p, 0}
def store{p:P, v:V & isptr{P} & eltype{P}==v} = store{p, 0, v}
def loadu{p:T & isunsigned{eltype{T}}} = emit{eltype{T}, merge{'loadu_u',fmtnat{elwidth{T}}}, p}
def storeu{p:T, v:eltype{T} & isunsigned{eltype{T}}} = emit{void, merge{'storeu_u',fmtnat{elwidth{T}}}, p, v}
def loadu{p:T & elwidth{T}==8} = load{p}
def storeu{p:T, v:eltype{T} & elwidth{T}==8} = store{p, v}
def reinterpret{T, x:X & T==X} = x
def exportN{f, ...ns} = each{{n} => export{n, f}, ns}
@ -126,35 +128,37 @@ def tern{c, t:T, f:T & anyInt{c}} = {
res
}
# reinterpret vector element type to signed/unsigned
def ty_s{w} = primtype{'i', w}
def ty_u{w} = primtype{'u', w}
def ty_u{T & isprim{T}} = ty_u{width{T}}
def ty_s{T & isprim{T}} = ty_s{width{T}}
def ty_u{T & isvec{T}} = [vcount{T}](ty_u{eltype{T}})
def ty_s{T & isvec{T}} = [vcount{T}](ty_s{eltype{T}})
def ty_u{x:T} = ty_u{T}~~x
def ty_s{x:T} = ty_s{T}~~x
# reinterpret vector as one with element type E
def to_el{E, T & isvec{T}} = [width{T}/width{E}]E
def to_el{E, x:T} = to_el{E,T} ~~ x
# change width of primitive type
def to_w{T, w} = primtype{quality{T}, w}
# double/halve primitive or element type width
def ty_dbl {T & isprim{T}} = to_w{T, width{T}*2}
def ty_half{T & isprim{T}} = to_w{T, width{T}/2}
def ty_dbl {T & isvec{T}} = to_el{ty_dbl {eltype{T}}, T}
def ty_half{T & isvec{T}} = to_el{ty_half{eltype{T}}, T}
# double/halve vector count
def v_dbl {T & isvec{T}} = [vcount{T}*2](eltype{T})
def v_half{T & isvec{T}} = [vcount{T}/2](eltype{T})
# double element width without changing count
def el_dbl {T & isvec{T}} = [vcount{T}](ty_dbl{eltype{T}})
def el_half{T & isvec{T}} = [vcount{T}](ty_half{eltype{T}})
def re_el{E, V} = [width{V}/width{E}]E
def re_el{E, x:V} = re_el{E,V} ~~ x
local def qualChange{q} = {
def f{w & knum{w}} = primtype{q, w}
def f{T & isprim{T}} = primtype{q, width{T}}
def f{T & isvec{T}} = re_el{f{eltype{T}}, T}
def f{x:T} = f{T}~~x
}
def ty_u = qualChange{'u'}
def ty_s = qualChange{'i'}
def ty_f = qualChange{'f'}
def w_n{T, w & isprim{T}} = primtype{quality{T}, w}
def w_d{T & isprim{T}} = to_w{T, width{T}*2} # double/halve primitive type width
def w_h{T & isprim{T}} = to_w{T, width{T}/2}
def n_d{T & isvec{T}} = [vcount{T}*2](eltype{T}) # double/halve vector count
def n_h{T & isvec{T}} = [vcount{T}/2](eltype{T})
def el_d{T & isvec{T}} = [vcount{T}](w_d{eltype{T}}) # double/halve element width, preserving count
def el_h{T & isvec{T}} = [vcount{T}](w_h{eltype{T}})
def el_m{T & isvec{T}} = re_el{w_d{eltype{T}}, T}; def el_m{x:T} = re_el{T}~~x # double/halve element width, preserving width
def el_s{T & isvec{T}} = re_el{w_h{eltype{T}}, T}; def el_s{x:T} = re_el{T}~~x
# test if vector has a specific width & element type
def lvec{T, n, w} = 0
@ -169,15 +173,15 @@ def maxvalue{T & issigned{T}} = (1<<(width{T}-1))-1
def {
abs,andAllZero,andnz,b_getBatch,clmul,cvt,extract,floor,fold_addw,half,
homAll,homAny,homBlend,homMask,homMaskStore,homMaskStoreF,loadBatchBit,
loadLow,make,mul12,mulHi,narrow,narrowPair,packHi,packLo,packQ,pair,pdep,
loadLow,make,mulw,mulHi,narrow,narrowPair,packHi,packLo,packQ,pair,pdep,
pext,popcRand,sel,shl,shr,shuf,shuf16Hi,shuf16Lo,shufHalves,storeLow,
topBlend,topMask,topMaskStore,topMaskStoreF,unord,unpackQ,vfold,widen,
zip,zipHi,zipLo
}
def mzip {a:T, b:T} = each{{v}=>ty_dbl{T}~~v, zip{a, b}}
def mzipLo{a:T, b:T} = ty_dbl{T} ~~ zipLo{a, b}
def mzipHi{a:T, b:T} = ty_dbl{T} ~~ zipHi{a, b}
def mzip {a:T, b:T} = each{{v}=>el_m{T}~~v, zip{a, b}}
def mzipLo{a:T, b:T} = el_m{T} ~~ zipLo{a, b}
def mzipHi{a:T, b:T} = el_m{T} ~~ zipHi{a, b}
def andnot{a, b & anyNum{a} & anyNum{b}} = a & ~b
oper &~ andnot infix none 35
def widen{T, x:X & T==X} = x
@ -281,10 +285,6 @@ def forUnroll{exp,unr}{vars,begin,end,iter} = {
}
}
}
def forXUnroll{unr}{vars,begin,end,iter} = {
@forUnroll{unr}(is from begin to end) each{{i} => iter{i, vars}, is}
}
def makeBranch{Ts, F} = {
def args = undef{Ts}

View File

@ -23,7 +23,7 @@ def for_vec_overlap{vl}{vars,begin==0,n,iter} = {
}
# Shift as u16, since x86 is missing 8-bit shifts
def shr16{v:V, n} = V~~(to_el{u16, v} >> n)
def shr16{v:V, n} = V~~(re_el{u16, v} >> n)
# Forward or backwards in-place max-scan
# Assumes a whole number of vectors and minimum 0
@ -69,7 +69,7 @@ if (hasarch{'AVX2'}) {
{i} => sel{H, v, i}
}
def getsel{v:V & lvec{V, 32, 8}} = {
def H = v_half{V}
def H = n_h{V}
vtop := V**(vcount{V}/2)
hs := each{bind{shuf, [4]u64, v}, tup{4b3232, 4b1010}}
{i} => homBlend{...each{{h}=>sel{H,h,i}, hs}, V~~i<vtop}
@ -82,7 +82,7 @@ def uninterleave{x:V & hasarch{'AVX2'}} = {
def vl = vcount{V}; def bytes = width{eltype{V}}/8
def i = 2*iota{vl/4}
def i2= join{table{+, bytes*merge{i,i+1}, iota{bytes}}}
t := V~~sel{[16]u8, to_el{u8,x}, make{[32]u8, merge{i2,i2}}}
t := V~~sel{[16]u8, re_el{u8,x}, make{[32]u8, merge{i2,i2}}}
shuf{[4]u64, t, 4b3120}
}
@ -157,7 +157,7 @@ def bins_vectab_i8{up, w, wn, x, xn, rp, t0, t, done & hasarch{'AVX2'}} = {
assert{wn < 128} # Total must fit in i8
def vl = 32
def T = i8
def V = [vl]T; def H = v_half{V}
def V = [vl]T; def H = n_h{V}
def U = [vl]u8
# Convert to bit table
@ -212,7 +212,7 @@ def bins_vectab_i8{up, w, wn, x, xn, rp, t0, t, done & hasarch{'AVX2'}} = {
def isel{u} = sel{H, u, i0}
ind = isel{ui}
if (nu > 16) {
b := V~~(to_el{u16, i0} << (7 - lb{vl/2}))
b := V~~(re_el{u16, i0} << (7 - lb{vl/2}))
ind = topBlend{ind, isel{ui1}, b}
if (nu > 32) ind = homBlend{topBlend{...each{isel,ui2}, b}, ind, i0 < V**vl}
}
@ -260,7 +260,7 @@ def bin_search_vec{T, up, w:*T, wn, x:*T, xn, rp, maxwn & hasarch{'AVX2'}} = {
tup{wv, wv2} = each{un, tr_half{wv, un{load{wg, 1}}}}
}
}
def ms{v}{h} = getsel{to_el{I, if (lanes) half{v,h} else tupsel{h,v}}}
def ms{v}{h} = getsel{re_el{I, if (lanes) half{v,h} else tupsel{h,v}}}
def selw = ms{wv}{0}; def selw1 = if (ex>=1) ms{wv}{1} else 'undef'
def selw2 = if (ex>=2) each{ms{wv2}, iota{2}} else 'undef'
# Offset at end
@ -276,7 +276,7 @@ def bin_search_vec{T, up, w:*T, wn, x:*T, xn, rp, maxwn & hasarch{'AVX2'}} = {
if (this) @for_vec_overlap{vl} (j to xn) {
xv:= load{*V~~(x+j), 0}
s := U**bb{iota{isub}} # Select sequential bytes within each U
def ltx{se,ind} = lt{xv, V~~se{to_el{I,ind}}}
def ltx{se,ind} = lt{xv, V~~se{re_el{I,ind}}}
@unroll (j to klog) {
m := s | tupsel{klog-1-j,bits}
s = homBlend{m, s, ltx{selw, m}}
@ -295,7 +295,7 @@ def bin_search_vec{T, up, w:*T, wn, x:*T, xn, rp, maxwn & hasarch{'AVX2'}} = {
r -= off
rn := if (T==i8) r
else if (T==i16) half{narrow{u8, r}, 0}
else extract{to_el{i64, narrow{u8, r}}, 0}
else extract{re_el{i64, narrow{u8, r}}, 0}
store{*type{rn}~~(*i8~~rp+j), 0, rn}
}
}

View File

@ -75,7 +75,7 @@ def spreadBits{T==[16]u8, a:u16 & hasarch{'X86_64'}} = {
def spreadBits{T, a & vcount{T} <= elwidth{T} & quality{eltype{T}}=='u'} = {
b:= make{T, 1<<iota{vcount{T}}}
b == (b & T ~~ to_el{type{a}, T}**a) # not just T**a so that if a is read from RAM, it can use the single instruction for broadcasting from RAM; the extra bits don't matter
b == (b & T ~~ re_el{type{a}, T}**a) # not just T**a so that if a is read from RAM, it can use the single instruction for broadcasting from RAM; the extra bits don't matter
}
def loadBatchBit{T, x:*u64, n:(Size)} = { # vector with type T with each element being either all 0s or 1s

View File

@ -51,7 +51,7 @@ def arithChk2{F, M, w:T, x:T & same{F,__mul} & isvec{T} & i8==eltype{T} & hasarc
tup{packQ{rp}, tup{'homAny', M{packQ{bad}}}}
} else { # unmasked check; can do check in a simpler way
def bad = each{{v} => ty_s{v ^ (v>>15)}, rp}
def RU = to_el{u16,T}
def RU = re_el{u16,T}
tup{packQ{rp}, tup{'~andAllZero', RU~~tree_fold{|, bad}, RU**0xff80}}
}
}
@ -61,8 +61,8 @@ def arithChk2{F, M, w:T, x:T & same{F,__mul} & isvec{T} & i16==eltype{T} & hasar
tup{rl, tup{'anyne', rh, rl>>15}}
}
def arithChk2{F, M, w:T, x:T & same{F,__mul} & isvec{T} & i32==eltype{T} & hasarch{'X86_64'}} = {
max:= to_el{f32, (ty_u{T})**0x4efffffe}
def cf32{x:X} = emit{to_el{f32,X}, tern{T==[8]i32, '_mm256_cvtepi32_ps', '_mm_cvtepi32_ps'}, x}
max:= re_el{f32, (ty_u{T})**0x4efffffe}
def cf32{x:X} = emit{re_el{f32,X}, tern{T==[8]i32, '_mm256_cvtepi32_ps', '_mm_cvtepi32_ps'}, x}
f32mul:= cf32{w} * cf32{x}
tup{w*x, tup{'homAny', M{abs{f32mul} >= max}}}
# TODO fallback to the below if the above fails
@ -70,7 +70,7 @@ def arithChk2{F, M, w:T, x:T & same{F,__mul} & isvec{T} & i32==eltype{T} & hasar
# def wp = unpackQ{w, T**0}
# def xp = unpackQ{x, T**0}
# def rp = each{mul32, wp, xp}
# def T2 = to_el{i64, T}
# def T2 = re_el{i64, T}
# def bad = each{{v} => {
# ((T2~~v + T2**0x80000000) ^ T2**(cast{i64,1}<<63)) > T2**cast_i{i64, (cast{u64,1}<<63) | 0xFFFFFFFF}
# }, rp}
@ -78,7 +78,7 @@ def arithChk2{F, M, w:T, x:T & same{F,__mul} & isvec{T} & i32==eltype{T} & hasar
}
def arithChk2{F, M, w:T, x:T & same{F,__mul} & isvec{T} & hasarch{'AARCH64'}} = {
def r12 = mul12{w, x}
def r12 = mulw{w, x}
rl:= packLo{r12}
rh:= packHi{r12}
tup{rl, tup{'homAny', M{rh != (rl >> (elwidth{T}-1))}}}
@ -110,7 +110,7 @@ def runner{u, R, F} = {
def run{F, M, w:VW, x:VX & c & R==u32 & (same{F,__add} | same{F,__sub})} = { # 'a'+1, 'a'-1
r:= F{ty_u{w}, ty_u{x}}
tup{to_el{R, VW}~~r, tup{'homAny', M{r > type{r}**1114111}}}
tup{re_el{R, VW}~~r, tup{'homAny', M{r > type{r}**1114111}}}
}
run
}

View File

@ -48,7 +48,7 @@ fn equal{W, X}(w:*void, x:*void, l:u64, d:u64) : u1 = {
} else { # bitarr ≡ i8/i16/i32arr
def T = [bulk]X
def sh{c} = c << (width{X}-1)
def sh{c & X==u8} = T ~~ (to_el{u16,c}<<7)
def sh{c & X==u8} = T ~~ (re_el{u16,c}<<7)
def mask{x:X & hasarch{'X86_64'}} = topMask{x}
def mask{x:X & hasarch{'AARCH64'}} = homMask{andnz{x, ~T**0}}

View File

@ -48,7 +48,7 @@ def loadLowBatch{T, ptr:P, w, n & eltype{P}==eltype{T}} = loadLow{*T ~~ (ptr + n
def storeBatch{ptr:P, n, x:T, M} = {
def rpos = ptr + n*vcount{T}
def E0 = eltype{P}
def TF = to_el{E0, T}
def TF = re_el{E0, T}
xu:= narrow{E0, x}
if (M{0}) homMaskStoreF{*TF~~rpos, M{TF, 'to homogeneous bits'}, undefPromote{TF, xu}}
@ -60,7 +60,7 @@ def loadBatch{ptr:P, n, T} = {
def rpos = ptr + n*vcount{T}
def E0 = eltype{P}
widen{T, loadLow{*to_el{E0, T} ~~ rpos, vcount{T}*width{E0}}}
widen{T, loadLow{*re_el{E0, T} ~~ rpos, vcount{T}*width{E0}}}
}
def loadBatch {ptr:P, ns, T & istup{ns}} = each{{n } => loadBatch {ptr, n, T }, ns}
@ -73,13 +73,6 @@ def hCast{T,p} = assert{show{'expected pointer with element',T,'or void but got
def hCast{T,p:P & same{T,eltype{P}}} = p
def hCast{T,p:P & same{P,*void}} = *T~~p
# masked loop arguments
# p:*T - regular pointer, loaded by the batch index(es)
# tup{VT,p:*T} - loadBatch/storeBatch vector data
# tup{'b',p:P} - b_getBatch
# tup{'b',VT,p:P} - loadBatchBit
# tup{'g',VT,p:*T} - gives a generator supporting g{} for loadBatch and g{newValue} for storeBatch
# tup{'g',p:*T} - the above, but without load support
def mlExec{i, iter, vars0, bulk, M} = {
def vproc2{T,p:P} = tptr{{i} => loadBatch{p, i, T}, {i,x} => storeBatch{p, i, x, M}}
def vproc2{S=='b', p:P} = tptr{{i} => b_getBatch{bulk, hCast{u64,p}, i}, '!'}

View File

@ -42,16 +42,15 @@ def __shl{a:T,b:S & nveci{T} & nveci{S,elwidth{T}}} = emit{T, ntyp{'vshl', T},
def adds{a:T,b:T & nveci{T}} = emit{T, ntyp{'vqadd', T}, a, b}
def subs{a:T,b:T & nveci{T}} = emit{T, ntyp{'vqsub', T}, a, b}
def addp{a:T,b:T & nvec{T}} = emit{T, ntyp{'vpadd', T}, a, b}
def addw{a:T,b:T & w64i{T}} = emit{el_dbl{T}, ntyp{'vaddl', T}, a, b}
def subw{a:T,b:T & w64i{T}} = emit{el_dbl{T}, ntyp{'vsubl', T}, a, b}
def mulw{a:T,b:T & w64i{T}} = emit{el_dbl{T}, ntyp{'vmull', T}, a, b}
def mulw2{a:T,b:T & w128i{T}} = emit{ty_dbl{T}, ntyp0{'vmull_high', T}, a, b}
def mul12{a:T,b:T & w128{T}} = tup{mulw{half{a,0}, half{b,0}}, mulw2{a,b}}
def addwLo{a:T,b:T & w64i{T}} = emit{el_d{T}, ntyp{'vaddl', T}, a, b}
def subwLo{a:T,b:T & w64i{T}} = emit{el_d{T}, ntyp{'vsubl', T}, a, b}
def mulwLo{a:T,b:T & w64i{T}} = emit{el_d{T}, ntyp{'vmull', T}, a, b}
def mulwHi{a:T,b:T & w128i{T}} = emit{el_m{T}, ntyp0{'vmull_high', T}, a, b}
def mulw {a:T,b:T & w128{T}} = tup{mulwLo{half{a,0}, half{b,0}}, mulwHi{a,b}}
def __shl{a:T, s & nveci{T} & knum{s} & s>0 & s<elwidth{T}} = emit{T, ntyp{'vshl', '_n', T}, a, s}; def __shl{a:T,s==0 & nveci{T}} = a
def __shr{a:T, s & nveci{T} & knum{s} & s>0 & s<elwidth{T}} = emit{T, ntyp{'vshr', '_n', T}, a, s}; def __shr{a:T,s==0 & nveci{T}} = a
def shrn{a:T, s & w128i{T} & elwidth{T}>8} = { def H=el_half{T}; emit{H, ntyp0{'vshrn_n', T}, a, s} } # a>>s, narrowed
def shrn{a:T, s & w128i{T} & elwidth{T}>8} = { def H=el_h{T}; emit{H, ntyp0{'vshrn_n', T}, a, s} } # a>>s, narrowed
def shrm{a:T, s, d:T & nvecu{T}} = emit{T, ntyp{'vsri', '_n', T}, d, a, s} # (a>>s) | (d & (mask of new zeroes))
def shlm{a:T, s, d:T & nvecu{T}} = emit{T, ntyp{'vsli', '_n', T}, d, a, s} # (a<<s) | (d & (mask of new zeroes))
@ -59,14 +58,14 @@ def bitBlend{f:T, t:T, m:M & nvec{T} & nvecu{M,elwidth{T}} & width{T}==width{M}}
def homBlend{f:T, t:T, m:M & nvec{M}} = bitBlend{f, t, m}
def __neg{a:T & (nvecs{T}|nvecf{T})} = emit{T, ntyp{'vneg', T}, a}
def __not{a:T & nvecu{T}} = T~~emit{to_el{u8,T}, ntyp{'vmvn', to_el{u8,T}}, a}
def __not{a:T & nvecu{T}} = T~~emit{re_el{u8,T}, ntyp{'vmvn', re_el{u8,T}}, a}
def sqrt{a:T & nvecf{T}} = emit{T, ntyp{'vsqrt', T}, a}
def floor{a:T & nvecf{T}} = emit{T, ntyp{'vrndm', T}, a}
def ceil{a:T & nvecf{T}} = emit{T, ntyp{'vrndp', T}, a}
def abs{a:T & (nvecs{T}|nvecf{T})} = emit{T, ntyp{'vabs', T}, a}
def absu{a:T & nveci{T}} = ty_u{abs{a}}
def addpw { x:T & nveci{T} & elwidth{T}<=32 } = emit{ty_dbl{T}, ntyp{'vpaddl', T}, x} # add pairwise widening
def addpwa{a:D, x:T & nveci{T} & elwidth{T}<=32 & D==ty_dbl{T}} = emit{D, ntyp{'vpadal', T}, a, x} # add pairwise widening + accumulate
def addpw { x:T & nveci{T} & elwidth{T}<=32 } = emit{el_m{T}, ntyp{'vpaddl', T}, x} # add pairwise widening
def addpwa{a:D, x:T & nveci{T} & elwidth{T}<=32 & D==el_m{T}} = emit{D, ntyp{'vpadal', T}, a, x} # add pairwise widening + accumulate
def mla{a:T, x:T, y:T & nvec{T}} = emit{T, ntyp{'vmla', T}, a, x, y} # a + x*y
def mls{a:T, x:T, y:T & nvec{T}} = emit{T, ntyp{'vmls', T}, a, x, y} # a - x*y
def rbit{x:T & nvecu{T,8}} = emit{T, ntyp{'vrbit', T}, x}
@ -86,19 +85,19 @@ def __le{a:T,b:T & nvec{T}} = emit{ty_u{T}, ntyp{'vcle', T}, a, b}
def __ne{a:T,b:T & nvec{T}} = ~(a==b)
def fold_add {a:T & nvec{T}} = emit{eltype{T}, ntyp{'vaddv', T}, a}
def fold_addw{a:T & nveci{T}} = emit{ty_dbl{eltype{T}}, ntyp{'vaddlv', T}, a}
def fold_addw{a:T & nveci{T}} = emit{w_d{eltype{T}}, ntyp{'vaddlv', T}, a}
def fold_min {a:T & nvec{T} & ~nveci{T,64}} = emit{eltype{T}, ntyp{'vminv', T}, a}
def fold_max {a:T & nvec{T} & ~nveci{T,64}} = emit{eltype{T}, ntyp{'vmaxv', T}, a}
def vfold{F, x:T & nvec{T} & ~nveci{T,64} & same{F, min}} = fold_min{x}
def vfold{F, x:T & nvec{T} & ~nveci{T,64} & same{F, max}} = fold_max{x}
def vfold{F, x:T & nvec{T} & same{F, +}} = fold_add{x}
def storeLow{ptr:P, w, x:T & nvec{T} & w<=64} = { def E=ty_u{w}; storeu{*E~~ptr, extract{to_el{E,T}~~x, 0}} }
def storeLow{ptr:P, w, x:T & nvec{T} & w<=64} = { def E=ty_u{w}; storeu{*E~~ptr, extract{re_el{E,T}~~x, 0}} }
def storeLow{ptr:P, w, x:T & nvec{T} & w==width{T}} = store{*T~~ptr, 0, x}
def loadLow{ptr:P, w & w<=64} = { # a broadcast load
def T=eltype{P}
def L=to_el{ty_u{w}, T}
def L=re_el{ty_u{w}, T}
T ~~ emit{L, ntyp{'vld1', '_dup', L}, *ty_u{w}~~ptr}
}
def loadLow{ptr:P, w & w==elwidth{P}} = load{ptr}
@ -107,9 +106,9 @@ def loadLow{ptr:P, w & w==elwidth{P}} = load{ptr}
def undefPromote{T, x:X & w64{X} & w128{T} & eltype{T}==eltype{X}} = emit{T, ntyp{'vcombine', X}, x, x} # arm_neon.h doesn't actually provide a way to do this in a 0-instruction way. ¯\_(ツ)_/¯
def half{x:T, n==0 & w128{T}} = emit{v_half{T}, ntyp0{'vget_low', T}, x}
def half{x:T, n==1 & w128{T}} = emit{v_half{T}, ntyp0{'vget_high', T}, x}
def pair{a:T, b:T & w64{T}} = emit{v_dbl{T}, ntyp0{'vcombine', T}, a, b}
def half{x:T, n==0 & w128{T}} = emit{n_h{T}, ntyp0{'vget_low', T}, x}
def half{x:T, n==1 & w128{T}} = emit{n_h{T}, ntyp0{'vget_high', T}, x}
def pair{a:T, b:T & w64{T}} = emit{n_d{T}, ntyp0{'vcombine', T}, a, b}
def extract{x:T,n & nvec{T} & knum{n}} = emit{eltype{T}, ntyp{'vget', '_lane', T}, x, n}
def copyLane{dst:D, di, src:S, si & w64{D} & nvec{S} & eltype{D}==eltype{S}} = emit{D, ntyp{'vcopy_lane', S}, dst, di, src, si}
def copyLane{dst:D, di, src:S, si & w128{D} & nvec{S} & eltype{D}==eltype{S}} = emit{D, ntyp{'vcopyq_lane', S}, dst, di, src, si}
@ -119,15 +118,15 @@ def zipLo{a:T, b:T & nvec{T}} = emit{T, ntyp{'vzip1', T}, a, b}
def zipHi{a:T, b:T & nvec{T}} = emit{T, ntyp{'vzip2', T}, a, b}
def zip{a:T, b:T & nvec{T}} = tup{zipLo{a,b}, zipHi{a,b}}
def packLo{x:T, y:T & nvec{T}} = { def H=ty_half{T}; emit{H, ntyp{'vuzp1', H}, H~~x, H~~y} }
def packHi{x:T, y:T & nvec{T}} = { def H=ty_half{T}; emit{H, ntyp{'vuzp2', H}, H~~x, H~~y} }
def packLo{x:T, y:T & nvec{T}} = { def H=el_s{T}; emit{H, ntyp{'vuzp1', H}, H~~x, H~~y} }
def packHi{x:T, y:T & nvec{T}} = { def H=el_s{T}; emit{H, ntyp{'vuzp2', H}, H~~x, H~~y} }
def packLo{{x, y}} = packLo{x, y}
def packHi{{x, y}} = packHi{x, y}
def trn1{x:T, y:T & nvec{T}} = emit{T, ntyp{'vtrn1', T}, x, y}
def trn2{x:T, y:T & nvec{T}} = emit{T, ntyp{'vtrn2', T}, x, y}
def sel{L, x:T, i:I & lvec{L,16,8} & w128{T} & nvec{I, 8}} = to_el{eltype{T}, emit{I, ntyp{'vqtbl1',I}, to_el{eltype{I},x}, i}}
def sel{L, x:T, i:I & lvec{L,16,8} & w128{T} & nvec{I, 8}} = re_el{eltype{T}, emit{I, ntyp{'vqtbl1',I}, re_el{eltype{I},x}, i}}
@ -138,23 +137,23 @@ def cvt{T==i64, x:X & nvecf{X,64}} = emit{[vcount{X}]T, ntyp{'vcvt', '_s64', X},
def cvt{T==u64, x:X & nvecf{X,64}} = emit{[vcount{X}]T, ntyp{'vcvt', '_u64', X}, x}
def widen{T, x:X & w64{X} & eqqi{eltype{T},eltype{X}} & elwidth{T}==elwidth{X}*2} = emit{T, ntyp{'vmovl', X}, x}
def widen{T, x:X & w64{X} & eqqi{eltype{T},eltype{X}} & elwidth{T}> elwidth{X}*2} = widen{T, widen{ty_half{T}, x}}
def widen{T, x:X & w64{X} & eqqi{eltype{T},eltype{X}} & elwidth{T}> elwidth{X}*2} = widen{T, widen{el_s{T}, x}}
def widen{T, x:X & w64{X} & isfloat{eltype{T}}!=isfloat{eltype{X}} & elwidth{T}>elwidth{X}} = cvt{eltype{T}, widen{[vcount{T}](to_w{eltype{X},elwidth{T}}), x}}
def widen{T, x:X & w128{X} & vcount{X}>vcount{T}} = widen{T, half{x,0}}
def narrow{T, x:X & w128{X} & eqqi{T,eltype{X}} & width{T}*2< elwidth{X}} = narrow{T, undefPromote{ty_half{X}, narrow{ty_half{eltype{X}}, x}}}
def narrow{T, x:X & w128{X} & eqqi{T,eltype{X}} & width{T}*2==elwidth{X}} = emit{el_half{X}, ntyp0{'vmovn', X}, x}
def narrow{T, x:X & w128{X} & eqqi{T,eltype{X}} & width{T}*2< elwidth{X}} = narrow{T, undefPromote{el_s{X}, narrow{w_h{eltype{X}}, x}}}
def narrow{T, x:X & w128{X} & eqqi{T,eltype{X}} & width{T}*2==elwidth{X}} = emit{el_h{X}, ntyp0{'vmovn', X}, x}
def narrow{T, x:X & w128{X} & isfloat{T}!=isfloat{eltype{X}} & width{T}<elwidth{X}} = narrow{T, cvt{to_w{T, elwidth{X}}, x}}
def narrowUpper{lowRes:L, x:X & w64i{L} & w128{X} & el_dbl{L}==X} = emit{[vcount{L}*2](eltype{L}), ntyp0{'vmovn_high', X}, lowRes, x}
def narrowPair{a:T, b:T} = narrowUpper{narrow{ty_half{eltype{T}}, a}, b}
def narrowUpper{lowRes:L, x:X & w64i{L} & w128{X} & el_d{L}==X} = emit{[vcount{L}*2](eltype{L}), ntyp0{'vmovn_high', X}, lowRes, x}
def narrowPair{a:T, b:T} = narrowUpper{narrow{w_h{eltype{T}}, a}, b}
def narrowPair{a:T, b:T & isint{eltype{T}}} = packLo{a, b}
def widenUpper{x:T & w128i{T}} = emit{ty_dbl{T}, ntyp0{'vmovl_high', T}, x}
def widen{x:T & w128{T}} = tup{widen{ty_dbl{T}, x}, widenUpper{x}}
def widenUpper{x:T & w128i{T}} = emit{el_m{T}, ntyp0{'vmovl_high', T}, x}
def widen{x:T & w128{T}} = tup{widen{el_m{T}, x}, widenUpper{x}}
def bitAny{x:T} = fold_max{to_el{u32, x}}!=0
def bitAll{x:T} = fold_min{to_el{u32, x}}==0xffff_ffff
def bitAny{x:T} = fold_max{re_el{u32, x}}!=0
def bitAll{x:T} = fold_min{re_el{u32, x}}==0xffff_ffff
def topAny{x:T & nvec{T}} = fold_min{ty_s{x}}<0
def topAll{x:T & nvec{T}} = fold_max{ty_s{x}}<0
@ -200,7 +199,7 @@ def andAllZero{x:T, y:T & nveci{T}} = ~bitAny{x&y}
def homMaskX{a:T & eltype{T}!=u64} = {
def h = elwidth{T}/2
tup{h, truncBits{vcount{T}*h, extract{[1]u64~~shrn{ty_dbl{T}~~a, h}, 0}}}
tup{h, truncBits{vcount{T}*h, extract{[1]u64~~shrn{el_m{T}~~a, h}, 0}}}
}

View File

@ -194,7 +194,7 @@ def simd_plus_scan{X, b, R}{x:*X, c:(R), r:*R, len:u64} = {
def bulk = arch_defvw/b
def wd = (X!=R) & (width{X}<32) # whether to widen the working copy one size
def WE = tern{wd, ty_dbl{X}, X} # working copy element type
def WE = tern{wd, w_d{X}, X} # working copy element type
# maxFastA: max absolute value for accumulator
# maxFastE: max absolute value for vector elements (not used if ~wd)

View File

@ -56,7 +56,7 @@ fn searchOne{A, E}(x:*void, e0:A, len:u64) : u64 = {
search{E, x, len, {c:VT} => c == VT**e}
}
def isNegZero{x:T} = to_el{u64,x} == to_el{u64, T ** -f64~~0}
def isNegZero{x:T} = re_el{u64,x} == re_el{u64, T ** -f64~~0}
fn searchNormalizable{}(x:*f64, len:u64) : u64 = {
search{f64, x, len, {c:VT} => isNegZero{c} | (c!=c)}
}

View File

@ -59,7 +59,7 @@ def perm_select{ri, rd, TI, w, r, wl, xl, selx} = {
@maskedLoop{ri}(cw0 in tup{VI,w}, M in 'm' over i to wl) {
cw:= wrapChk{cw0, VI,xlf, M}
is:= (if (ext>1) i<<lb{ext}; else i)
def part{o} = widen{[8]i32, to_el{i8, shuf{[4]u64, cw, 4b3210+o}}}
def part{o} = widen{[8]i32, re_el{i8, shuf{[4]u64, cw, 4b3210+o}}}
def se{o} = storeExp{r, is+o, selx{part{o}}, M, ext, rd, wl}
each{se, iota{ext}}
}

View File

@ -112,7 +112,7 @@ def getter{c, V, x} = {
# These can only change between loop iterations, provided the
# given x for i32 is a multiple of the loop step
def topper{T, U, k, x} = {
def make_top{S} = to_el{S,U}**(if (T<i32) 0 else cast_i{S, x>>width{S}})
def make_top{S} = re_el{S,U}**(if (T<i32) 0 else cast_i{S, x>>width{S}})
top := each{make_top, replicate{{S}=>S<T, tup{i8,i16}}}
def i_off = if (T<i32) 0 else { assert{x%k==0}; cast_i{u64, x/k} }
# Increment top vector when i*k passes width of bottom vector

View File

@ -24,7 +24,7 @@ def anySNaN{M, x:T & T==[2]u64 & hasarch{'X86_64'} & ~hasarch{'SSE4.2'}} = {
}
def anyNonChar{M, x:T & isvec{T} & eltype{T}==u64} = homAny{M{~inRangeLen{x, cbqn_c32Tag{}<<48, 1<<48}}}
def anyNonChar{M, x:T & isvec{T} & hasarch{'X86_64'}} = {
def H = to_el{u32, T}
def H = re_el{u32, T}
def ne = H~~x != H**cast_i{u32, cbqn_c32Tag{}<<16}
topAny{M{T~~ne}}
}

View File

@ -170,7 +170,7 @@ def topMask{x:T & w128{T, 16}} = topMask{packs{[8]i16~~x, [8]i16**0}}
def topMask{x:T & w128{T, 32}} = emit{u8, '_mm_movemask_ps', v2f{x}}
def topMask{x:T & w128{T, 64}} = emit{u8, '_mm_movemask_pd', v2d{x}}
def homMask{x:T & w128{T}} = topMask{x}
def homMaskX{a:T & elwidth{T}==16} = tup{2, homMask{to_el{u8,a}}}
def homMaskX{a:T & elwidth{T}==16} = tup{2, homMask{re_el{u8,a}}}
def homAny{x:T & w128i{T}} = homMask{[16]u8 ~~ x} != 0
def homAll{x:T & w128i{T}} = homMask{[16]u8 ~~ x} == 0xffff

View File

@ -13,8 +13,9 @@
#define _mm_storeu_si32 custom_storeu_si32
#endif
static void storeu_u64(u64* p, u64 v) { memcpy(p, &v, 8); }
static u64 loadu_u64(u64* p) { u64 v; memcpy(&v, p, 8); return v; }
static void storeu_u64(u64* p, u64 v) { memcpy(p, &v, 8); } static u64 loadu_u64(u64* p) { u64 v; memcpy(&v, p, 8); return v; }
static void storeu_u32(u32* p, u32 v) { memcpy(p, &v, 4); } static u32 loadu_u32(u32* p) { u32 v; memcpy(&v, p, 4); return v; }
static void storeu_u16(u16* p, u16 v) { memcpy(p, &v, 2); } static u16 loadu_u16(u16* p) { u16 v; memcpy(&v, p, 2); return v; }
#define BCALL(N, X) N(b(X))
#define interp_f64(X) b(X).f
@ -27,11 +28,7 @@
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wunused-variable"
#ifdef SINGELI_DIR
#include SINGELI_FILE1(../../build/obj2/SINGELI_DIR/SINGELI_FILE.c)
#else
#include SINGELI_FILE1(../singeli/gen/SINGELI_FILE.c)
#endif
#pragma GCC diagnostic pop
#undef SINGELI_FILE