Commit Graph

736 Commits

Author SHA1 Message Date
dzaima
87be577ff7 actually use defined is in for_mu 2025-03-20 20:27:26 +02:00
dzaima
0c2fb1a267 a bit of cleanup
still from the long-ago days when symbols couldn't be assignment targets directly
2025-03-19 01:05:22 +02:00
dzaima
577b09a062 introduce x86.singeli 2025-03-19 01:05:15 +02:00
Marshall Lochbaum
233c88120d Singeli function for initial group statistics pass since auto-vectorization isn't great 2025-03-13 14:21:38 -04:00
dzaima
d48194c3c8 fix comment 2025-03-12 17:20:18 +02:00
dzaima
7a412523dd better SSE2 float getRange 2025-03-12 16:38:20 +02:00
dzaima
a013eb094e select_fn doesn't require SIMD anymore
it's not used too much without SIMD extensions available, but it is used a bit
2025-03-12 04:20:53 +02:00
dzaima
d2ca600e02 actually make try_first_hom return things 2025-03-12 00:20:49 +02:00
dzaima
85ef1038b8 fix usz=32 build 2025-03-11 23:41:11 +02:00
Marshall Lochbaum
b69a2c2469 Move swtab outside of its function; it was being compiled as a dynamic array 2025-03-09 08:44:42 -04:00
Marshall Lochbaum
e898a81d68 Use a possibly-overlapping word and constant-length loop in k/bool, like ⊣` 2025-03-09 08:44:42 -04:00
Marshall Lochbaum
302c2f926f AVX2 k/bool for odd 64<k<256 using shift-based masks 2025-03-09 08:44:42 -04:00
dzaima
bd942894bf fix missing usage of unaligned store 2025-03-06 06:01:44 +02:00
dzaima
d5facc33ad some Singeli cleanup
shuf_ind's staying as such I guess; make it more clear when debug.singeli is left included
2025-03-06 05:32:22 +02:00
dzaima
267ddc4987 clarifying comments 2025-03-06 04:02:27 +02:00
Marshall Lochbaum
a11ed3d79a Mask out values past the end for strided +` overflow checks 2025-03-04 07:39:28 -05:00
Marshall Lochbaum
46300bcb51 Use scalar case for small-stride +` on <SSSE3 to avoid incorrectly overlapping vectors 2025-03-03 08:15:47 -05:00
Marshall Lochbaum
4072bde806 Strided scan with AVX2 [8]i32 permute if possible 2025-03-03 08:14:17 -05:00
Marshall Lochbaum
180c79e751 Strided +`, widening and re-trying on overflow 2025-03-03 06:54:40 -05:00
Marshall Lochbaum
87a7d066c8 Strided ⌊⌈ scans with shuffles 2025-03-03 06:54:40 -05:00
Marshall Lochbaum
360e4e8320 Save a word instead of writing to avoid stall for scan with 64<stride<128 2025-03-03 06:54:40 -05:00
Marshall Lochbaum
f3e0ea7531 Use CLMUL-based ≠` for power-of-two stride up to 64 2025-03-03 06:54:40 -05:00
Marshall Lochbaum
ee6b91be8a Basic strided ∧∨≠= scan implementations 2025-03-03 06:54:40 -05:00
Marshall Lochbaum
b03b26656d Fix usz=32 build 2025-03-03 06:54:40 -05:00
dzaima
445007b38c unify toLast and broadcast_last 2025-03-02 19:31:03 +02:00
dzaima
bdee96a995 remove old blend definitions 2025-02-27 05:49:16 +02:00
Marshall Lochbaum
eda1cd8b50 More tuning based on dzaima's measurements 2025-02-26 22:01:37 -05:00
Marshall Lochbaum
a520ff70a1 Fix some NEON performance issues for row scans 2025-02-26 21:52:49 -05:00
Marshall Lochbaum
422e63bc12 Non-CLMUL SIMD ≠` 2025-02-26 08:21:22 -05:00
Marshall Lochbaum
d85d69e759 Improve baseline for ⊣`˘ to beat non-AVX2 on length ≥64 2025-02-25 20:26:51 -05:00
Marshall Lochbaum
709712f03c Tuning; drop the scalar loose_mask branch for ∧∨ as it never seems faster 2025-02-25 19:33:26 -05:00
Marshall Lochbaum
b5ec2f5fa8 Extend loose_mask system to SSE and NEON (not always fast) 2025-02-25 19:28:51 -05:00
Marshall Lochbaum
fb413b1966 Enable vector max- min-, and wrapping plus-scan on NEON
Changes the results on NaNs, but they weren't consistent before
2025-02-24 22:08:49 -05:00
Marshall Lochbaum
317871db55 Extend unaligned <64-bit row scans from AVX2 to SSE, NEON 2025-02-24 21:47:46 -05:00
Marshall Lochbaum
b131914cbe Factor out some common code 2025-02-24 21:13:19 -05:00
Marshall Lochbaum
0973d9b499 Medium-row ⊣˘ and ≠˘ with mask-aware carry 2025-02-24 21:13:19 -05:00
Marshall Lochbaum
1fef51a39e AVX2 ∧ and ∨ with intermediate row sizes (64≤l<160) 2025-02-23 19:19:46 -05:00
Marshall Lochbaum
94e6b6cb3a Better formula for within-word masked ∧ and ∨ 2025-02-23 08:27:47 -05:00
dzaima
8c1040bd9f cleaner vr?(add|sub)hn NEON defs 2025-02-08 18:37:46 +02:00
dzaima
09f2980ebf more BMI and NEON defs 2025-02-08 17:36:44 +02:00
dzaima
bda9245ac3 another unnecessary tree_fold 2025-02-08 17:36:44 +02:00
dzaima
0d7a7652e3 better NEON all_hom{x:[16]u8} 2025-02-08 17:36:44 +02:00
dzaima
c792d47069 use Singeli a==b instead of a=(b) 2025-02-08 17:36:44 +02:00
dzaima
b12575722a better NEON any_hom
but clang undoes this sometimes :/ https://github.com/llvm/llvm-project/issues/125611
2025-02-08 17:36:44 +02:00
dzaima
72cf36334c make (any|all_(hom|top) accept mask or vl or multiple vectors
makes the previous commit superfluous, but still keeping it separate so that this one changes nothing
2025-02-08 17:36:44 +02:00
dzaima
341f3757e7 make all_hom{mask_none, vec} use simple path 2025-02-08 17:36:44 +02:00
dzaima
d17c6ebe8e actually fix AVX-512 more 2025-02-03 19:58:15 +02:00
dzaima
0bc491e931 fix AVX-512 more 2025-02-03 19:56:04 +02:00
dzaima
3a8f2a601a better NEON widening add/sub/mul definitions 2025-02-03 19:45:21 +02:00
dzaima
44d87fef78 fix AVX-512 build 2025-02-03 19:31:18 +02:00
dzaima
8c7c0227a5 first_hom, try_first_hom, popc_hom 2025-02-03 17:32:21 +02:00
dzaima
c64cdb2c61 make store_(blended|masked)_(hom|top) accept element pointer 2025-02-03 04:20:33 +02:00
dzaima
cd373e2ae6 remove (load|store)_low 2025-02-03 03:53:11 +02:00
dzaima
61565bfa17 define load_low/store_low in terms of vl-based load/store 2025-02-03 03:37:38 +02:00
dzaima
848813331a element-pointer-based vector loads & stores 2025-02-03 03:17:23 +02:00
dzaima
91800ddd1e more direct mask_none all-bits-zeroes check 2025-02-03 03:05:58 +02:00
dzaima
25faa71995 better NEON hom_to_int_ext 2025-02-03 02:37:07 +02:00
dzaima
afc4aa81fa more complete NEON widening multiply/add/subtract 2025-02-03 02:37:07 +02:00
dzaima
49dd6e394a mask.singeli loop renames 2025-02-03 02:13:39 +02:00
dzaima
92f40ddbe2 mask.singeli def renames
more bits of renames
2025-02-03 02:13:39 +02:00
dzaima
b6578e43a1 renames of the rest of base.singeli 2025-02-03 02:13:39 +02:00
dzaima
189ffa7d1f bitops.singeli renames 2025-02-03 02:12:58 +02:00
dzaima
f409ee8729 fix generic arch 2025-02-03 02:11:00 +02:00
dzaima
a6266b2168 remove unnecessary defs 2025-02-03 02:11:00 +02:00
dzaima
490285f1ca andAllZero → and_bit_none 2025-02-03 02:11:00 +02:00
dzaima
10ec04a76e (load|store)Low → (load|store)_low 2025-02-03 02:11:00 +02:00
dzaima
ad79ef71cf more Singeli mask op renames 2025-02-03 02:11:00 +02:00
dzaima
6f8b08bb1f oneVal/oneType/allSame → one_val/one_type/all_same 2025-02-03 02:11:00 +02:00
dzaima
a3e33366df exportT → export_tab
also remove exportN as it's unused
2025-02-03 02:11:00 +02:00
dzaima
70e2694f88 more Singeli NEON cleanup 2025-02-03 02:10:47 +02:00
dzaima
b899180166 any_num → any_int where applicable 2025-02-02 23:08:41 +02:00
dzaima
936300a492 any(Int|Num) → any_(int|num) 2025-02-02 23:08:41 +02:00
dzaima
0357d57171 various Singeli NEON cleanup things 2025-02-02 23:08:41 +02:00
dzaima
f3e7f4032c get rid of custom NEON rev
was non-compatibly overridden in places anyways
2025-02-02 23:08:41 +02:00
dzaima
6c7a444cd7 more consistent NEON naming 2025-02-02 23:08:41 +02:00
dzaima
e3b30e5db7 (hom|top)Mask → (hom|top)_to_int 2025-02-02 23:08:41 +02:00
dzaima
e9e1574d28 rearrange initial Singeli base defs 2025-02-02 23:08:41 +02:00
dzaima
426c1b4bf8 (bit|top|hom)(Any|All) → (any|all)_(bit|top|hom) 2025-02-02 23:08:41 +02:00
dzaima
c4afe7b329 (bit|top|hom)Blend → blend_(bit|top|hom) 2025-02-02 23:08:02 +02:00
dzaima
d506691690 fix 𝕨⍋𝕩 issues around NaNs 2025-02-01 20:55:36 +02:00
dzaima
204ec21a7c more compact vector width check definitions 2025-01-23 03:21:55 +02:00
dzaima
5f19f15556 NEON (hom|bit)(Any|All) improvements 2025-01-23 03:20:52 +02:00
dzaima
7509d0c238 don't shadow types 2024-11-30 20:26:54 +02:00
dzaima
79cffc5011 don't use deprecated syntax 2024-11-22 12:48:59 +02:00
dzaima
92641f26ba squeeze fns haven't been AVX2-specific for a while 2024-11-22 12:48:59 +02:00
dzaima
7df2f9ba4f a couple renames & simplifications 2024-11-22 00:44:03 +02:00
Marshall Lochbaum
33ae8744f5 Fast skip ahead for blocks of equal values in i8 and i16 /⁼ 2024-11-20 09:35:22 -05:00
Marshall Lochbaum
7baddcabc2 Better sum versus runs test for /⁼: account for per-element cost of run-based 2024-11-17 15:29:36 -05:00
Marshall Lochbaum
0bbb335893 Do general i8 and i16 /⁼ counts to i16 buffer, plus overflow list 2024-11-17 15:29:36 -05:00
Marshall Lochbaum
e6940e73d0 Fast /⁼ of sorted arguments using semi-sparse representation 2024-11-17 15:29:36 -05:00
Marshall Lochbaum
11117fcc67 Version of count_i32_i32 without large blocks 2024-11-17 15:29:36 -05:00
Marshall Lochbaum
3b103aadd0 Run-based i32 /⁼ (leaving out sum-based) 2024-11-17 15:29:36 -05:00
Marshall Lochbaum
e24d0bac63 Extend SIMD counting to i16 in addition to i8 2024-11-17 15:29:36 -05:00
Marshall Lochbaum
e681f3c09a Run-based 1-byte /⁼ implementation 2024-11-17 15:29:36 -05:00
dzaima
b72fe465a2 fix /⁼1e4⥊1‿2 on NEON 2024-11-14 22:45:43 +02:00
Marshall Lochbaum
db16b8fa6e Fix dropped high bit in ⊣˝˘ due to signed rather than unsigned saturating pack 2024-11-06 14:52:18 -05:00
Marshall Lochbaum
67d3fe3f81 Missing SSSE3 requirement for some uninterleave implementations 2024-11-06 14:51:16 -05:00
Marshall Lochbaum
e75b63831f Clean up dispatching for the various transpose kernel methods 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
2d75c6c535 And manually vectorize uninterleave 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
32ad2e9953 Manually vectorized interleave function 2024-11-05 21:48:54 -05:00