Commit Graph

3058 Commits

Author SHA1 Message Date
dzaima
60b848f1b7 don't require strict subset type for trunc 2025-03-22 19:09:51 +02:00
dzaima
c7632c23a2 actually, can define the x86 sadbw-using vfold for i8
it's unsigned, but that doesn't matter when just taking the low 8 bits
2025-03-22 18:21:09 +02:00
dzaima
ae29c4ad45 fix usz=32 build 2025-03-21 18:08:12 +02:00
dzaima
609f3418d8 use offset loads instead of vec_merge_shift_right for groupstat 2025-03-21 06:01:27 +02:00
dzaima
e2b4dbcf20 use any_hom / all_hom for (any|all)_accumulator 2025-03-21 06:01:27 +02:00
dzaima
05d87dd7df move groupstat to fancy accumulator 2025-03-21 06:01:26 +02:00
dzaima
97260e0051 special-cased x86 SSE2 i8 min/max accumulator 2025-03-21 04:32:26 +02:00
dzaima
f503880cc3 use fancy accumulator for getRange 2025-03-21 04:32:26 +02:00
dzaima
e1b62b95b6 fancy Singeli accumulator; use for integer sum 2025-03-21 04:32:26 +02:00
dzaima
3b1239d499 support requesting a flush every k iterations in for_mu 2025-03-21 04:32:26 +02:00
dzaima
9c2ea18e22 use x86 sad for vfold
not used by anything, but it's here
2025-03-21 04:32:26 +02:00
dzaima
6c6f6e4013 add x86 sad & madd intrinsics 2025-03-21 04:32:26 +02:00
dzaima
9876d1d622 reduce Singeli-generated C noise on constant broadcasts 2025-03-21 04:32:26 +02:00
dzaima
4a5a1fa35b a couple more baseline utilities
assert_hom to be expanded upon later
2025-03-21 01:05:53 +02:00
dzaima
87be577ff7 actually use defined is in for_mu 2025-03-20 20:27:26 +02:00
dzaima
0c2fb1a267 a bit of cleanup
still from the long-ago days when symbols couldn't be assignment targets directly
2025-03-19 01:05:22 +02:00
dzaima
577b09a062 introduce x86.singeli 2025-03-19 01:05:15 +02:00
dzaima
108379f121 fix UB shift 2025-03-16 23:47:01 +02:00
dzaima
f1021580ad silence warning on non-Singeli builds 2025-03-16 20:22:17 +02:00
dzaima
9c870ddab9 add more info in heap dump 2025-03-16 03:52:23 +02:00
dzaima
82ebd37b3f group statistics test 2025-03-13 23:36:02 +02:00
Marshall Lochbaum
233c88120d Singeli function for initial group statistics pass since auto-vectorization isn't great 2025-03-13 14:21:38 -04:00
dzaima
72c4eca6c0 fix float ⌾(l⊸⊏) doing wrong comparisons 2025-03-12 17:23:17 +02:00
dzaima
d48194c3c8 fix comment 2025-03-12 17:20:18 +02:00
dzaima
7a412523dd better SSE2 float getRange 2025-03-12 16:38:20 +02:00
dzaima
62b095be89 oneElt⊏𝕩 improvements & fixes 2025-03-12 06:29:26 +02:00
dzaima
3ae7d134d2 more uniform F¨ fill behavior 2025-03-12 05:52:52 +02:00
dzaima
a013eb094e select_fn doesn't require SIMD anymore
it's not used too much without SIMD extensions available, but it is used a bit
2025-03-12 04:20:53 +02:00
dzaima
bfce527096 fix 𝕗⌾(-2⋆63)𝕩 2025-03-12 03:01:11 +02:00
dzaima
e33b3b95ee relax documented RangeFn behavior to align with NEON impl 2025-03-12 02:10:05 +02:00
dzaima
a6202c8ca6 fix error message of ⌾(oob⊸⊏) 2025-03-12 01:41:52 +02:00
dzaima
0848faf8c7 don't leak memory in TEST_RANGE 2025-03-12 01:14:24 +02:00
dzaima
5f5bb37a92 fix builtin function comparison on ALL_R1 2025-03-12 01:03:00 +02:00
dzaima
d2ca600e02 actually make try_first_hom return things 2025-03-12 00:20:49 +02:00
dzaima
85ef1038b8 fix usz=32 build 2025-03-11 23:41:11 +02:00
Marshall Lochbaum
dcecc70c25 More careful fill handling with characters in insert_scal 2025-03-10 21:58:15 -04:00
dzaima
97e0210c41 remove unnecessary F˜˘ acceptance from F˘
scan_rows_bit only checks frtid, which is 0 for such
2025-03-11 00:17:11 +02:00
dzaima
bcf32838f4 some more ˝ tests, use COPY_TO
not much of a difference, but prettier
2025-03-11 00:15:45 +02:00
Marshall Lochbaum
f1972fc408 Faster stride-1 check 2025-03-10 12:57:25 -04:00
Marshall Lochbaum
e42273bfe8 Generic stride-1 𝔽`˘ implementation 2025-03-10 12:50:54 -04:00
Marshall Lochbaum
bbe4549aff Scalar implementation of 𝔽˝ for generic arithmetic 𝔽 and short rows 2025-03-10 12:48:06 -04:00
Marshall Lochbaum
b69a2c2469 Move swtab outside of its function; it was being compiled as a dynamic array 2025-03-09 08:44:42 -04:00
Marshall Lochbaum
e898a81d68 Use a possibly-overlapping word and constant-length loop in k/bool, like ⊣` 2025-03-09 08:44:42 -04:00
Marshall Lochbaum
302c2f926f AVX2 k/bool for odd 64<k<256 using shift-based masks 2025-03-09 08:44:42 -04:00
dzaima
bd942894bf fix missing usage of unaligned store 2025-03-06 06:01:44 +02:00
dzaima
d5facc33ad some Singeli cleanup
shuf_ind's staying as such I guess; make it more clear when debug.singeli is left included
2025-03-06 05:32:22 +02:00
dzaima
267ddc4987 clarifying comments 2025-03-06 04:02:27 +02:00
dzaima
79fc8d5116 use direct copy on known-different-eltype arrays 2025-03-04 23:18:30 +02:00
Marshall Lochbaum
d9a80ee83b Test bound and length instead of rank in scan where possible 2025-03-04 10:59:40 -05:00
Marshall Lochbaum
a11ed3d79a Mask out values past the end for strided +` overflow checks 2025-03-04 07:39:28 -05:00
dzaima
9fdeb5379a fix old gcc 2025-03-04 04:04:04 +02:00
Marshall Lochbaum
010e6a10d4 Implementation comments for new high-rank scans 2025-03-03 19:56:53 -05:00
Marshall Lochbaum
46300bcb51 Use scalar case for small-stride +` on <SSSE3 to avoid incorrectly overlapping vectors 2025-03-03 08:15:47 -05:00
Marshall Lochbaum
4072bde806 Strided scan with AVX2 [8]i32 permute if possible 2025-03-03 08:14:17 -05:00
Marshall Lochbaum
180c79e751 Strided +`, widening and re-trying on overflow 2025-03-03 06:54:40 -05:00
Marshall Lochbaum
87a7d066c8 Strided ⌊⌈ scans with shuffles 2025-03-03 06:54:40 -05:00
Marshall Lochbaum
360e4e8320 Save a word instead of writing to avoid stall for scan with 64<stride<128 2025-03-03 06:54:40 -05:00
Marshall Lochbaum
f3e0ea7531 Use CLMUL-based ≠` for power-of-two stride up to 64 2025-03-03 06:54:40 -05:00
Marshall Lochbaum
ee6b91be8a Basic strided ∧∨≠= scan implementations 2025-03-03 06:54:40 -05:00
Marshall Lochbaum
90d0f3015a Use rank-1 scan special code whenever cell size is 1 2025-03-03 06:54:40 -05:00
Marshall Lochbaum
b03b26656d Fix usz=32 build 2025-03-03 06:54:40 -05:00
dzaima
d3b4c3cc26 flush stdout on •Show and •Out 2025-03-03 00:53:55 +02:00
dzaima
05a9cd736b more spaces around names between name and 𝕨/𝕩 in error messages 2025-03-02 19:57:38 +02:00
dzaima
bdf9a52625 more proper error messages for /⁼𝕩 2025-03-02 19:50:01 +02:00
dzaima
17ec5f0a8a add 𝕩 to error messages of <⁼𝕩, ⋈⁼𝕩, ≍⁼𝕩 2025-03-02 19:32:49 +02:00
dzaima
445007b38c unify toLast and broadcast_last 2025-03-02 19:31:03 +02:00
dzaima
9826c4ce0e add 𝕨/𝕩 in dyadic ⌽/⌽⁼ error message 2025-03-02 19:29:49 +02:00
dzaima
bdee96a995 remove old blend definitions 2025-02-27 05:49:16 +02:00
Marshall Lochbaum
eda1cd8b50 More tuning based on dzaima's measurements 2025-02-26 22:01:37 -05:00
Marshall Lochbaum
a520ff70a1 Fix some NEON performance issues for row scans 2025-02-26 21:52:49 -05:00
Marshall Lochbaum
a775520927 Update scan implementation comments 2025-02-26 08:21:22 -05:00
Marshall Lochbaum
422e63bc12 Non-CLMUL SIMD ≠` 2025-02-26 08:21:22 -05:00
Marshall Lochbaum
d85d69e759 Improve baseline for ⊣`˘ to beat non-AVX2 on length ≥64 2025-02-25 20:26:51 -05:00
Marshall Lochbaum
709712f03c Tuning; drop the scalar loose_mask branch for ∧∨ as it never seems faster 2025-02-25 19:33:26 -05:00
Marshall Lochbaum
b5ec2f5fa8 Extend loose_mask system to SSE and NEON (not always fast) 2025-02-25 19:28:51 -05:00
Marshall Lochbaum
fb413b1966 Enable vector max- min-, and wrapping plus-scan on NEON
Changes the results on NaNs, but they weren't consistent before
2025-02-24 22:08:49 -05:00
Marshall Lochbaum
317871db55 Extend unaligned <64-bit row scans from AVX2 to SSE, NEON 2025-02-24 21:47:46 -05:00
Marshall Lochbaum
b131914cbe Factor out some common code 2025-02-24 21:13:19 -05:00
Marshall Lochbaum
0973d9b499 Medium-row ⊣˘ and ≠˘ with mask-aware carry 2025-02-24 21:13:19 -05:00
Marshall Lochbaum
1fef51a39e AVX2 ∧ and ∨ with intermediate row sizes (64≤l<160) 2025-02-23 19:19:46 -05:00
Marshall Lochbaum
94e6b6cb3a Better formula for within-word masked ∧ and ∨ 2025-02-23 08:27:47 -05:00
Marshall Lochbaum
242a61db11 Faster <`, adding instructions to cut dependency chains 2025-02-18 16:29:49 -05:00
dzaima
8c1040bd9f cleaner vr?(add|sub)hn NEON defs 2025-02-08 18:37:46 +02:00
dzaima
09f2980ebf more BMI and NEON defs 2025-02-08 17:36:44 +02:00
dzaima
bda9245ac3 another unnecessary tree_fold 2025-02-08 17:36:44 +02:00
dzaima
0d7a7652e3 better NEON all_hom{x:[16]u8} 2025-02-08 17:36:44 +02:00
dzaima
c792d47069 use Singeli a==b instead of a=(b) 2025-02-08 17:36:44 +02:00
dzaima
b12575722a better NEON any_hom
but clang undoes this sometimes :/ https://github.com/llvm/llvm-project/issues/125611
2025-02-08 17:36:44 +02:00
dzaima
72cf36334c make (any|all_(hom|top) accept mask or vl or multiple vectors
makes the previous commit superfluous, but still keeping it separate so that this one changes nothing
2025-02-08 17:36:44 +02:00
dzaima
341f3757e7 make all_hom{mask_none, vec} use simple path 2025-02-08 17:36:44 +02:00
dzaima
2cb15c936d get rid of b(x) macro 2025-02-08 17:36:44 +02:00
dzaima
74483f3859 more spaces between 𝕨/𝕩 and names, re-remove •SH args
sh_c1 calls sh_c2, so the arity isn't function-specific
2025-02-07 07:57:51 +02:00
tankorsmash
b945e48d29 add spacing around system functions; add missing args for SH; fix wrong args for bit._cast 2025-02-06 20:17:18 -05:00
dzaima
e3a4132722 clean up error messages
incl. correcting •_timed's message being wrong
2025-02-06 12:32:19 +02:00
tankorsmash
3362d46854 add best effort guesses for error messages in core/stuff and opt/comp.c 2025-02-05 20:02:44 -05:00
tankorsmash
5790d126cd add best effort guesses for many more builtin error messages 2025-02-05 19:56:08 -05:00
tankorsmash
63ee20b6fe add best effort guesses for many builtin error messages 2025-02-05 19:40:27 -05:00
tankorsmash
dd5d17ca83 add best effort guesses for builtins/cmp,fns,fold.c
I'm not sure about modifiers so I left them without arguments
2025-02-05 19:32:49 -05:00
tankorsmash
559b3f0d6e add best effort guesses for w and x for builtins/cells.c 2025-02-04 19:00:15 -05:00
tankorsmash
dce39a1b05 add w and x for builtins/arithm.c 2025-02-04 18:57:25 -05:00