dzaima
|
267ddc4987
|
clarifying comments
|
2025-03-06 04:02:27 +02:00 |
|
Marshall Lochbaum
|
a11ed3d79a
|
Mask out values past the end for strided +` overflow checks
|
2025-03-04 07:39:28 -05:00 |
|
Marshall Lochbaum
|
46300bcb51
|
Use scalar case for small-stride +` on <SSSE3 to avoid incorrectly overlapping vectors
|
2025-03-03 08:15:47 -05:00 |
|
Marshall Lochbaum
|
4072bde806
|
Strided scan with AVX2 [8]i32 permute if possible
|
2025-03-03 08:14:17 -05:00 |
|
Marshall Lochbaum
|
180c79e751
|
Strided +`, widening and re-trying on overflow
|
2025-03-03 06:54:40 -05:00 |
|
Marshall Lochbaum
|
87a7d066c8
|
Strided ⌊⌈ scans with shuffles
|
2025-03-03 06:54:40 -05:00 |
|
Marshall Lochbaum
|
360e4e8320
|
Save a word instead of writing to avoid stall for scan with 64<stride<128
|
2025-03-03 06:54:40 -05:00 |
|
Marshall Lochbaum
|
f3e0ea7531
|
Use CLMUL-based ≠` for power-of-two stride up to 64
|
2025-03-03 06:54:40 -05:00 |
|
Marshall Lochbaum
|
ee6b91be8a
|
Basic strided ∧∨≠= scan implementations
|
2025-03-03 06:54:40 -05:00 |
|
Marshall Lochbaum
|
b03b26656d
|
Fix usz=32 build
|
2025-03-03 06:54:40 -05:00 |
|
dzaima
|
445007b38c
|
unify toLast and broadcast_last
|
2025-03-02 19:31:03 +02:00 |
|
dzaima
|
bdee96a995
|
remove old blend definitions
|
2025-02-27 05:49:16 +02:00 |
|
Marshall Lochbaum
|
eda1cd8b50
|
More tuning based on dzaima's measurements
|
2025-02-26 22:01:37 -05:00 |
|
Marshall Lochbaum
|
a520ff70a1
|
Fix some NEON performance issues for row scans
|
2025-02-26 21:52:49 -05:00 |
|
Marshall Lochbaum
|
422e63bc12
|
Non-CLMUL SIMD ≠`
|
2025-02-26 08:21:22 -05:00 |
|
Marshall Lochbaum
|
d85d69e759
|
Improve baseline for ⊣`˘ to beat non-AVX2 on length ≥64
|
2025-02-25 20:26:51 -05:00 |
|
Marshall Lochbaum
|
709712f03c
|
Tuning; drop the scalar loose_mask branch for ∧∨ as it never seems faster
|
2025-02-25 19:33:26 -05:00 |
|
Marshall Lochbaum
|
b5ec2f5fa8
|
Extend loose_mask system to SSE and NEON (not always fast)
|
2025-02-25 19:28:51 -05:00 |
|
Marshall Lochbaum
|
fb413b1966
|
Enable vector max- min-, and wrapping plus-scan on NEON
Changes the results on NaNs, but they weren't consistent before
|
2025-02-24 22:08:49 -05:00 |
|
Marshall Lochbaum
|
317871db55
|
Extend unaligned <64-bit row scans from AVX2 to SSE, NEON
|
2025-02-24 21:47:46 -05:00 |
|
Marshall Lochbaum
|
b131914cbe
|
Factor out some common code
|
2025-02-24 21:13:19 -05:00 |
|
Marshall Lochbaum
|
0973d9b499
|
Medium-row ⊣˘ and ≠˘ with mask-aware carry
|
2025-02-24 21:13:19 -05:00 |
|
Marshall Lochbaum
|
1fef51a39e
|
AVX2 ∧ and ∨ with intermediate row sizes (64≤l<160)
|
2025-02-23 19:19:46 -05:00 |
|
Marshall Lochbaum
|
94e6b6cb3a
|
Better formula for within-word masked ∧ and ∨
|
2025-02-23 08:27:47 -05:00 |
|
dzaima
|
8c1040bd9f
|
cleaner vr?(add|sub)hn NEON defs
|
2025-02-08 18:37:46 +02:00 |
|
dzaima
|
09f2980ebf
|
more BMI and NEON defs
|
2025-02-08 17:36:44 +02:00 |
|
dzaima
|
bda9245ac3
|
another unnecessary tree_fold
|
2025-02-08 17:36:44 +02:00 |
|
dzaima
|
0d7a7652e3
|
better NEON all_hom{x:[16]u8}
|
2025-02-08 17:36:44 +02:00 |
|
dzaima
|
c792d47069
|
use Singeli a==b instead of a=(b)
|
2025-02-08 17:36:44 +02:00 |
|
dzaima
|
b12575722a
|
better NEON any_hom
but clang undoes this sometimes :/ https://github.com/llvm/llvm-project/issues/125611
|
2025-02-08 17:36:44 +02:00 |
|
dzaima
|
72cf36334c
|
make (any|all_(hom|top) accept mask or vl or multiple vectors
makes the previous commit superfluous, but still keeping it separate so that this one changes nothing
|
2025-02-08 17:36:44 +02:00 |
|
dzaima
|
341f3757e7
|
make all_hom{mask_none, vec} use simple path
|
2025-02-08 17:36:44 +02:00 |
|
dzaima
|
d17c6ebe8e
|
actually fix AVX-512 more
|
2025-02-03 19:58:15 +02:00 |
|
dzaima
|
0bc491e931
|
fix AVX-512 more
|
2025-02-03 19:56:04 +02:00 |
|
dzaima
|
3a8f2a601a
|
better NEON widening add/sub/mul definitions
|
2025-02-03 19:45:21 +02:00 |
|
dzaima
|
44d87fef78
|
fix AVX-512 build
|
2025-02-03 19:31:18 +02:00 |
|
dzaima
|
8c7c0227a5
|
first_hom, try_first_hom, popc_hom
|
2025-02-03 17:32:21 +02:00 |
|
dzaima
|
c64cdb2c61
|
make store_(blended|masked)_(hom|top) accept element pointer
|
2025-02-03 04:20:33 +02:00 |
|
dzaima
|
cd373e2ae6
|
remove (load|store)_low
|
2025-02-03 03:53:11 +02:00 |
|
dzaima
|
61565bfa17
|
define load_low/store_low in terms of vl-based load/store
|
2025-02-03 03:37:38 +02:00 |
|
dzaima
|
848813331a
|
element-pointer-based vector loads & stores
|
2025-02-03 03:17:23 +02:00 |
|
dzaima
|
91800ddd1e
|
more direct mask_none all-bits-zeroes check
|
2025-02-03 03:05:58 +02:00 |
|
dzaima
|
25faa71995
|
better NEON hom_to_int_ext
|
2025-02-03 02:37:07 +02:00 |
|
dzaima
|
afc4aa81fa
|
more complete NEON widening multiply/add/subtract
|
2025-02-03 02:37:07 +02:00 |
|
dzaima
|
49dd6e394a
|
mask.singeli loop renames
|
2025-02-03 02:13:39 +02:00 |
|
dzaima
|
92f40ddbe2
|
mask.singeli def renames
more bits of renames
|
2025-02-03 02:13:39 +02:00 |
|
dzaima
|
b6578e43a1
|
renames of the rest of base.singeli
|
2025-02-03 02:13:39 +02:00 |
|
dzaima
|
189ffa7d1f
|
bitops.singeli renames
|
2025-02-03 02:12:58 +02:00 |
|
dzaima
|
f409ee8729
|
fix generic arch
|
2025-02-03 02:11:00 +02:00 |
|
dzaima
|
a6266b2168
|
remove unnecessary defs
|
2025-02-03 02:11:00 +02:00 |
|
dzaima
|
490285f1ca
|
andAllZero → and_bit_none
|
2025-02-03 02:11:00 +02:00 |
|
dzaima
|
10ec04a76e
|
(load|store)Low → (load|store)_low
|
2025-02-03 02:11:00 +02:00 |
|
dzaima
|
ad79ef71cf
|
more Singeli mask op renames
|
2025-02-03 02:11:00 +02:00 |
|
dzaima
|
6f8b08bb1f
|
oneVal/oneType/allSame → one_val/one_type/all_same
|
2025-02-03 02:11:00 +02:00 |
|
dzaima
|
a3e33366df
|
exportT → export_tab
also remove exportN as it's unused
|
2025-02-03 02:11:00 +02:00 |
|
dzaima
|
70e2694f88
|
more Singeli NEON cleanup
|
2025-02-03 02:10:47 +02:00 |
|
dzaima
|
b899180166
|
any_num → any_int where applicable
|
2025-02-02 23:08:41 +02:00 |
|
dzaima
|
936300a492
|
any(Int|Num) → any_(int|num)
|
2025-02-02 23:08:41 +02:00 |
|
dzaima
|
0357d57171
|
various Singeli NEON cleanup things
|
2025-02-02 23:08:41 +02:00 |
|
dzaima
|
f3e7f4032c
|
get rid of custom NEON rev
was non-compatibly overridden in places anyways
|
2025-02-02 23:08:41 +02:00 |
|
dzaima
|
6c7a444cd7
|
more consistent NEON naming
|
2025-02-02 23:08:41 +02:00 |
|
dzaima
|
e3b30e5db7
|
(hom|top)Mask → (hom|top)_to_int
|
2025-02-02 23:08:41 +02:00 |
|
dzaima
|
e9e1574d28
|
rearrange initial Singeli base defs
|
2025-02-02 23:08:41 +02:00 |
|
dzaima
|
426c1b4bf8
|
(bit|top|hom)(Any|All) → (any|all)_(bit|top|hom)
|
2025-02-02 23:08:41 +02:00 |
|
dzaima
|
c4afe7b329
|
(bit|top|hom)Blend → blend_(bit|top|hom)
|
2025-02-02 23:08:02 +02:00 |
|
dzaima
|
d506691690
|
fix 𝕨⍋𝕩 issues around NaNs
|
2025-02-01 20:55:36 +02:00 |
|
dzaima
|
204ec21a7c
|
more compact vector width check definitions
|
2025-01-23 03:21:55 +02:00 |
|
dzaima
|
5f19f15556
|
NEON (hom|bit)(Any|All) improvements
|
2025-01-23 03:20:52 +02:00 |
|
dzaima
|
7509d0c238
|
don't shadow types
|
2024-11-30 20:26:54 +02:00 |
|
dzaima
|
79cffc5011
|
don't use deprecated syntax
|
2024-11-22 12:48:59 +02:00 |
|
dzaima
|
92641f26ba
|
squeeze fns haven't been AVX2-specific for a while
|
2024-11-22 12:48:59 +02:00 |
|
dzaima
|
7df2f9ba4f
|
a couple renames & simplifications
|
2024-11-22 00:44:03 +02:00 |
|
Marshall Lochbaum
|
33ae8744f5
|
Fast skip ahead for blocks of equal values in i8 and i16 /⁼
|
2024-11-20 09:35:22 -05:00 |
|
Marshall Lochbaum
|
7baddcabc2
|
Better sum versus runs test for /⁼: account for per-element cost of run-based
|
2024-11-17 15:29:36 -05:00 |
|
Marshall Lochbaum
|
0bbb335893
|
Do general i8 and i16 /⁼ counts to i16 buffer, plus overflow list
|
2024-11-17 15:29:36 -05:00 |
|
Marshall Lochbaum
|
e6940e73d0
|
Fast /⁼ of sorted arguments using semi-sparse representation
|
2024-11-17 15:29:36 -05:00 |
|
Marshall Lochbaum
|
11117fcc67
|
Version of count_i32_i32 without large blocks
|
2024-11-17 15:29:36 -05:00 |
|
Marshall Lochbaum
|
3b103aadd0
|
Run-based i32 /⁼ (leaving out sum-based)
|
2024-11-17 15:29:36 -05:00 |
|
Marshall Lochbaum
|
e24d0bac63
|
Extend SIMD counting to i16 in addition to i8
|
2024-11-17 15:29:36 -05:00 |
|
Marshall Lochbaum
|
e681f3c09a
|
Run-based 1-byte /⁼ implementation
|
2024-11-17 15:29:36 -05:00 |
|
dzaima
|
b72fe465a2
|
fix /⁼1e4⥊1‿2 on NEON
|
2024-11-14 22:45:43 +02:00 |
|
Marshall Lochbaum
|
db16b8fa6e
|
Fix dropped high bit in ⊣˝˘ due to signed rather than unsigned saturating pack
|
2024-11-06 14:52:18 -05:00 |
|
Marshall Lochbaum
|
67d3fe3f81
|
Missing SSSE3 requirement for some uninterleave implementations
|
2024-11-06 14:51:16 -05:00 |
|
Marshall Lochbaum
|
e75b63831f
|
Clean up dispatching for the various transpose kernel methods
|
2024-11-05 21:48:54 -05:00 |
|
Marshall Lochbaum
|
2d75c6c535
|
And manually vectorize uninterleave
|
2024-11-05 21:48:54 -05:00 |
|
Marshall Lochbaum
|
32ad2e9953
|
Manually vectorized interleave function
|
2024-11-05 21:48:54 -05:00 |
|
Marshall Lochbaum
|
980e4b2248
|
Short-height version of 16x16 i8 transpose kernel, using overlapped writes
|
2024-11-05 21:48:54 -05:00 |
|
Marshall Lochbaum
|
abb96cb18a
|
Fixed-height transposes, more or less reverse of fixed-width
|
2024-11-05 21:48:54 -05:00 |
|
Marshall Lochbaum
|
0ab2e485ca
|
Width-4 transpose with shuffles
|
2024-11-05 21:48:54 -05:00 |
|
Marshall Lochbaum
|
8bc37098dc
|
Modular kernel transpose for odd*2 (so, just 6)
|
2024-11-05 21:48:54 -05:00 |
|
Marshall Lochbaum
|
091f08c6cc
|
Extend modular permutation transpose to any element size
|
2024-11-05 21:48:54 -05:00 |
|
Marshall Lochbaum
|
780bfdfa0b
|
Kernel transpose with overreading and skipped writes, for short rows
|
2024-11-05 21:48:54 -05:00 |
|
Marshall Lochbaum
|
b6e418ed5b
|
Fix modular permutation example in comment
|
2024-11-05 21:48:54 -05:00 |
|
Marshall Lochbaum
|
0bdc43cc0f
|
Even number handling for modular permutation ⊣˝˘
|
2024-11-05 21:48:54 -05:00 |
|
Marshall Lochbaum
|
4c55eab740
|
Fast ⊣˝˘ for power-of-two widths, 4- and 8-byte elements
|
2024-11-05 21:48:54 -05:00 |
|
Marshall Lochbaum
|
8d9d946ec5
|
Fast ⊣˝˘ with pack for power-of-two widths, 1- and 2-byte elements
|
2024-11-05 21:48:54 -05:00 |
|
Marshall Lochbaum
|
aefd2a4abd
|
Fast 1-byte small odd width transpose with blends and modular permutation
|
2024-11-05 21:48:54 -05:00 |
|
Marshall Lochbaum
|
5bdee886cc
|
Extend kernel transpose from AVX2 only to SSE2, NEON
|
2024-11-05 21:48:54 -05:00 |
|
Marshall Lochbaum
|
6e4859e17f
|
Extend select_rows_byte to any architecture with (shuffle and) blend
|
2024-11-05 21:48:54 -05:00 |
|
Marshall Lochbaum
|
4669418f1c
|
Extend modular permutation ⊣˝˘ to multi-byte element types
|
2024-11-05 21:48:54 -05:00 |
|