dzaima
|
e2b4dbcf20
|
use any_hom / all_hom for (any|all)_accumulator
|
2025-03-21 06:01:27 +02:00 |
|
dzaima
|
05d87dd7df
|
move groupstat to fancy accumulator
|
2025-03-21 06:01:26 +02:00 |
|
dzaima
|
97260e0051
|
special-cased x86 SSE2 i8 min/max accumulator
|
2025-03-21 04:32:26 +02:00 |
|
dzaima
|
f503880cc3
|
use fancy accumulator for getRange
|
2025-03-21 04:32:26 +02:00 |
|
dzaima
|
e1b62b95b6
|
fancy Singeli accumulator; use for integer sum
|
2025-03-21 04:32:26 +02:00 |
|
dzaima
|
3b1239d499
|
support requesting a flush every k iterations in for_mu
|
2025-03-21 04:32:26 +02:00 |
|
dzaima
|
9c2ea18e22
|
use x86 sad for vfold
not used by anything, but it's here
|
2025-03-21 04:32:26 +02:00 |
|
dzaima
|
6c6f6e4013
|
add x86 sad & madd intrinsics
|
2025-03-21 04:32:26 +02:00 |
|
dzaima
|
9876d1d622
|
reduce Singeli-generated C noise on constant broadcasts
|
2025-03-21 04:32:26 +02:00 |
|
dzaima
|
4a5a1fa35b
|
a couple more baseline utilities
assert_hom to be expanded upon later
|
2025-03-21 01:05:53 +02:00 |
|
dzaima
|
87be577ff7
|
actually use defined is in for_mu
|
2025-03-20 20:27:26 +02:00 |
|
dzaima
|
0c2fb1a267
|
a bit of cleanup
still from the long-ago days when symbols couldn't be assignment targets directly
|
2025-03-19 01:05:22 +02:00 |
|
dzaima
|
577b09a062
|
introduce x86.singeli
|
2025-03-19 01:05:15 +02:00 |
|
Marshall Lochbaum
|
233c88120d
|
Singeli function for initial group statistics pass since auto-vectorization isn't great
|
2025-03-13 14:21:38 -04:00 |
|
dzaima
|
d48194c3c8
|
fix comment
|
2025-03-12 17:20:18 +02:00 |
|
dzaima
|
7a412523dd
|
better SSE2 float getRange
|
2025-03-12 16:38:20 +02:00 |
|
dzaima
|
a013eb094e
|
select_fn doesn't require SIMD anymore
it's not used too much without SIMD extensions available, but it is used a bit
|
2025-03-12 04:20:53 +02:00 |
|
dzaima
|
d2ca600e02
|
actually make try_first_hom return things
|
2025-03-12 00:20:49 +02:00 |
|
dzaima
|
85ef1038b8
|
fix usz=32 build
|
2025-03-11 23:41:11 +02:00 |
|
Marshall Lochbaum
|
b69a2c2469
|
Move swtab outside of its function; it was being compiled as a dynamic array
|
2025-03-09 08:44:42 -04:00 |
|
Marshall Lochbaum
|
e898a81d68
|
Use a possibly-overlapping word and constant-length loop in k/bool, like ⊣`
|
2025-03-09 08:44:42 -04:00 |
|
Marshall Lochbaum
|
302c2f926f
|
AVX2 k/bool for odd 64<k<256 using shift-based masks
|
2025-03-09 08:44:42 -04:00 |
|
dzaima
|
bd942894bf
|
fix missing usage of unaligned store
|
2025-03-06 06:01:44 +02:00 |
|
dzaima
|
d5facc33ad
|
some Singeli cleanup
shuf_ind's staying as such I guess; make it more clear when debug.singeli is left included
|
2025-03-06 05:32:22 +02:00 |
|
dzaima
|
267ddc4987
|
clarifying comments
|
2025-03-06 04:02:27 +02:00 |
|
Marshall Lochbaum
|
a11ed3d79a
|
Mask out values past the end for strided +` overflow checks
|
2025-03-04 07:39:28 -05:00 |
|
Marshall Lochbaum
|
46300bcb51
|
Use scalar case for small-stride +` on <SSSE3 to avoid incorrectly overlapping vectors
|
2025-03-03 08:15:47 -05:00 |
|
Marshall Lochbaum
|
4072bde806
|
Strided scan with AVX2 [8]i32 permute if possible
|
2025-03-03 08:14:17 -05:00 |
|
Marshall Lochbaum
|
180c79e751
|
Strided +`, widening and re-trying on overflow
|
2025-03-03 06:54:40 -05:00 |
|
Marshall Lochbaum
|
87a7d066c8
|
Strided ⌊⌈ scans with shuffles
|
2025-03-03 06:54:40 -05:00 |
|
Marshall Lochbaum
|
360e4e8320
|
Save a word instead of writing to avoid stall for scan with 64<stride<128
|
2025-03-03 06:54:40 -05:00 |
|
Marshall Lochbaum
|
f3e0ea7531
|
Use CLMUL-based ≠` for power-of-two stride up to 64
|
2025-03-03 06:54:40 -05:00 |
|
Marshall Lochbaum
|
ee6b91be8a
|
Basic strided ∧∨≠= scan implementations
|
2025-03-03 06:54:40 -05:00 |
|
Marshall Lochbaum
|
b03b26656d
|
Fix usz=32 build
|
2025-03-03 06:54:40 -05:00 |
|
dzaima
|
445007b38c
|
unify toLast and broadcast_last
|
2025-03-02 19:31:03 +02:00 |
|
dzaima
|
bdee96a995
|
remove old blend definitions
|
2025-02-27 05:49:16 +02:00 |
|
Marshall Lochbaum
|
eda1cd8b50
|
More tuning based on dzaima's measurements
|
2025-02-26 22:01:37 -05:00 |
|
Marshall Lochbaum
|
a520ff70a1
|
Fix some NEON performance issues for row scans
|
2025-02-26 21:52:49 -05:00 |
|
Marshall Lochbaum
|
422e63bc12
|
Non-CLMUL SIMD ≠`
|
2025-02-26 08:21:22 -05:00 |
|
Marshall Lochbaum
|
d85d69e759
|
Improve baseline for ⊣`˘ to beat non-AVX2 on length ≥64
|
2025-02-25 20:26:51 -05:00 |
|
Marshall Lochbaum
|
709712f03c
|
Tuning; drop the scalar loose_mask branch for ∧∨ as it never seems faster
|
2025-02-25 19:33:26 -05:00 |
|
Marshall Lochbaum
|
b5ec2f5fa8
|
Extend loose_mask system to SSE and NEON (not always fast)
|
2025-02-25 19:28:51 -05:00 |
|
Marshall Lochbaum
|
fb413b1966
|
Enable vector max- min-, and wrapping plus-scan on NEON
Changes the results on NaNs, but they weren't consistent before
|
2025-02-24 22:08:49 -05:00 |
|
Marshall Lochbaum
|
317871db55
|
Extend unaligned <64-bit row scans from AVX2 to SSE, NEON
|
2025-02-24 21:47:46 -05:00 |
|
Marshall Lochbaum
|
b131914cbe
|
Factor out some common code
|
2025-02-24 21:13:19 -05:00 |
|
Marshall Lochbaum
|
0973d9b499
|
Medium-row ⊣˘ and ≠˘ with mask-aware carry
|
2025-02-24 21:13:19 -05:00 |
|
Marshall Lochbaum
|
1fef51a39e
|
AVX2 ∧ and ∨ with intermediate row sizes (64≤l<160)
|
2025-02-23 19:19:46 -05:00 |
|
Marshall Lochbaum
|
94e6b6cb3a
|
Better formula for within-word masked ∧ and ∨
|
2025-02-23 08:27:47 -05:00 |
|
dzaima
|
8c1040bd9f
|
cleaner vr?(add|sub)hn NEON defs
|
2025-02-08 18:37:46 +02:00 |
|
dzaima
|
09f2980ebf
|
more BMI and NEON defs
|
2025-02-08 17:36:44 +02:00 |
|