Commit Graph

666 Commits

Author SHA1 Message Date
Marshall Lochbaum
27af1c0558 Use Singeli select include for x86 blend, byteshift, zip 2024-10-21 18:53:03 -04:00
Marshall Lochbaum
6140290813 Use Singeli select includes for shuffling 2024-10-14 10:24:07 -04:00
dzaima
3d4a31ae6f access function/modifier runtime ID through macro 2024-10-10 02:38:17 +03:00
dzaima
85c5fb659d more clang-friendly multi-input homMask 2024-10-07 02:49:58 +03:00
dzaima
cf19280c54 fix reliance on signed wrapping in replicate 2024-10-03 20:33:02 +03:00
dzaima
27b7c0746c full interleave_cells
needed to make all rank-1 inputs work on non-Singeli builds, but beneficial in general anyway
2024-09-12 06:17:32 +03:00
dzaima
41785cb4bf Singeli const¨⌾(m⊸/)b 2024-09-10 01:45:56 +03:00
dzaima
02573fa4eb fatal{msg} 2024-09-10 01:45:18 +03:00
dzaima
b62e7cd820 fix generic arch build 2024-09-02 00:04:32 +03:00
Marshall Lochbaum
56cb10d6e5 Min/max scan that can skip unused argument vectors sometimes 2024-08-20 15:26:07 -04:00
dzaima
d185d32d9e remove debug include 2024-08-20 17:22:10 +03:00
dzaima
a654bd88d2 fix aarch64 build 2024-08-17 20:13:38 +03:00
dzaima
3b2b9bba1a AVX-512 bitwiden 2024-08-14 23:44:08 +03:00
dzaima
67626d1e49 a couple more AVX-512 defs 2024-08-14 23:44:08 +03:00
dzaima
6b72139d75 include avx512 in base 2024-08-14 22:13:24 +03:00
dzaima
60380e6270 fix out-of-bounds indexing in factors 2024-08-14 22:10:45 +03:00
Marshall Lochbaum
b2566c8f3a NEON k/bool support 2024-08-14 14:51:01 -04:00
Marshall Lochbaum
b801c7c186 Move to dedicated k/bool loops for k=5,7 and at least SSSE3 2024-08-14 14:51:01 -04:00
Marshall Lochbaum
b2758d355c k/bool cleanup and implementation comments 2024-08-14 14:51:01 -04:00
Marshall Lochbaum
bf69705c82 Factor (8×k)/bool as bit-replicate, then byte-replicate 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
fc1265b3f2 Enable (non-boolean) constrep on SSSE3 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
048529740b SSE2 k/bool where k divides 8 implementations 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
ee27717c97 SSE2 odd/bool implementation using [4]u32 shuffles 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
36e9ca5814 Extend odd/bool SSSE3 code to support AVX2 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
2be022921e Combine generic and SSSE3 general-case odd/bool, so SSSE3 uses mask unpacking 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
3e5dbdbf8d Dedicated generic k/bool for divisors of 8; factor; remove pdep emulation 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
13ec029d9f Get odd k/bool carry by shifting in the unified register, not after masking 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
1621c7d07c Generic-arch k/bool algorithm for odd k with mask unpacking 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
7d7d36b354 Extend SSSE3 k/bool to k<32 using pairwise swaps 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
696a23af9e Mask off last k/bool vector properly 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
1a4cada0cb AVX2 support in rep_const_bool_ssse3_div8 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
b4d84041bc Extend odd k/bool SSSE3 algorithm to k<16, factor even k≤32 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
a439a0a430 SSSE3 k/bool for 2≤k≤8, k≠6 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
a4b6d8d827 SSSE3 3/bool, faster than BMI2 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
7f6c401eb3 Generic-architecture k/bool for small 8<k 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
0602927b17 Move pdep-based k/bool code to Singeli 2024-08-14 10:30:26 -04:00
dzaima
fc800ee191 remove debug include 2024-08-14 15:06:43 +03:00
dzaima
9b515c3fad reverse pointer arguments of copy.singeli exports 2024-08-14 05:18:13 +03:00
dzaima
f6c52ef3b3 disable Singeli bitwiden on SSE2 2024-08-14 05:18:13 +03:00
dzaima
f356ba0f36 more definitions in base.singeli 2024-08-14 05:18:13 +03:00
dzaima
3fc92eceaa include util/tup in base.singeli 2024-08-14 05:18:13 +03:00
dzaima
eccbac37ab aarch64 8→n bitnarrow 2024-08-14 05:18:13 +03:00
dzaima
c72ed51149 Singeli n→8 bitwiden 2024-08-14 05:18:13 +03:00
dzaima
f36cefc9ac more Singeli utils 2024-08-14 05:18:13 +03:00
dzaima
5748833060 fast inds⊸⊏˘bits for ≤8-bit input & output cells 2024-08-14 05:18:13 +03:00
dzaima
081b928cb4 fix zip_halves being weird 2024-07-28 05:10:46 +03:00
dzaima
ffc250eb0e clearer aarch64 zip_halves mzip usage 2024-07-28 04:02:54 +03:00
dzaima
bd3789963c these are actually good 2024-07-28 02:52:58 +03:00
dzaima
6f25e76e5d tune NEON inds⊸⊏˘f64arr 2024-07-28 02:47:12 +03:00
dzaima
93fca7c628 fix overflow in index widening 2024-07-26 22:17:49 +03:00
dzaima
7cc1031f70 use outside-of-loop index widening on aarch64
slower as-is due to reduced supported ranges, but that's not an inherent restriction
2024-07-26 22:17:49 +03:00
dzaima
69ca524251 widen inds outside select-cells loops 2024-07-26 22:17:49 +03:00
dzaima
90646563a0 don't use zip method for select-cells 2024-07-26 22:16:23 +03:00
dzaima
4a874710c7 allow unhandled types for select_rows 2024-07-26 22:16:23 +03:00
dzaima
478c389c4b fast inds⊸⊏˘ mat 2024-07-26 22:16:23 +03:00
dzaima
f7dd900b3a faster 𝕨⊏𝕩, incl. aarch64 support 2024-07-26 00:01:54 +03:00
dzaima
d6483a93d3 x86 unzip & unzip128 2024-07-26 00:01:54 +03:00
dzaima
2ed06e0db6 get_range Singeli binding 2024-07-26 00:01:54 +03:00
dzaima
2ca488dd66 aarch64 bool128 select 2024-07-26 00:01:54 +03:00
dzaima
0d5e77766a build select.singeli everywhere 2024-07-26 00:01:54 +03:00
dzaima
c274c35b1e Singeli @for_blocks 2024-07-26 00:01:54 +03:00
dzaima
079d3ffe91 better aarch64 homMask{[16]u8,[16]u8} 2024-07-26 00:01:54 +03:00
dzaima
0b69fa84cd NEON comparisons to 0 2024-07-26 00:01:54 +03:00
dzaima
c0573d602c don't error on non-vector types checked by genchk 2024-07-26 00:01:54 +03:00
dzaima
2dd3798228 more Singeli cleanup 2024-07-15 00:56:02 +03:00
dzaima
2436c45f1d Singeli ceil_log2 for compile-time numbers 2024-07-15 00:52:00 +03:00
dzaima
66a8529d88 remove redundant parentheses in Singeli destructuring 2024-07-15 00:52:00 +03:00
dzaima
f79e4c6d0a make /(m?zip(128)?|pack)(Lo|Hi)/ not have Lo/Hi as part of name
instead being an optional 3rg arg
2024-07-15 00:52:00 +03:00
dzaima
a671ec9462 aarch64 bittab 2024-07-13 00:10:34 +03:00
dzaima
7aab7ceb8b cleanup 2024-07-10 03:44:05 +03:00
dzaima
2bc1ba9f4a fix aarch64 squeeze 2024-07-10 03:43:58 +03:00
dzaima
52d1e61fe0 don't bother early-exiting on masked iteration 2024-07-10 03:24:10 +03:00
dzaima
2fdd72f1c4 faster <SSE4.1 integer accumulating in squeeze 2024-07-10 03:18:17 +03:00
dzaima
a45925b0c6 fix SSE2 squeeze 2024-07-10 03:18:17 +03:00
dzaima
f941245b34 avoid needing two final folds on x86 2024-07-10 03:18:17 +03:00
dzaima
40a5c40bda faster squeeze 2024-07-10 03:18:17 +03:00
dzaima
d66be091be shufps Singeli mapping 2024-07-10 00:55:43 +03:00
dzaima
e96948cb64 Singeli zip128 2024-07-10 00:55:43 +03:00
dzaima
6824b9a747 fix another unaligned load 2024-06-26 06:09:30 +03:00
dzaima
7474e51642 rename mask_of in spaced.singeli
otherwise confusable with maskOf in mask.singeli
2024-06-26 05:39:48 +03:00
Marshall Lochbaum
ace6105bf6 Fixes for ∧˝˘ and ∨˝˘ on long rows 2024-06-25 22:30:56 -04:00
Marshall Lochbaum
355efd1eb4 Insert-cells and scan implementation comments, and minor tweaks 2024-06-19 18:36:53 -04:00
Marshall Lochbaum
4b0f105a7f Implement short-row num⊏˘bool (including ⊣˝˘ ⊢˝˘) with fold code 2024-06-18 14:05:16 -04:00
Marshall Lochbaum
1e6c7057e8 Extend boolean F`˘ special code to any frame and cell rank 2024-06-18 07:46:31 -04:00
Marshall Lochbaum
5e3cc6de81 Refactoring, simplification, comments 2024-06-18 07:46:31 -04:00
Marshall Lochbaum
a8b036ad08 Implement ≠˝˘ and =˝˘ like ∧∨ on width<64 boolean 2024-06-18 07:46:31 -04:00
Marshall Lochbaum
f0f130c42e Use a lookup table for evenly-spaced masks, getting division results from those 2024-06-18 07:46:31 -04:00
Marshall Lochbaum
e6f1e04de2 Fast generic and pext-based ∧˝˘ and ∨˝˘ on width<64 2024-06-18 07:46:31 -04:00
Marshall Lochbaum
40bf3bfd1c AVX2-based boolean short-row scans 2024-06-18 07:46:31 -04:00
Marshall Lochbaum
efde979fe9 Fast boolean ⊣`˘ 2024-06-18 07:46:31 -04:00
Marshall Lochbaum
8331a05547 Boolean ≠`˘ implementation 2024-06-18 07:46:31 -04:00
Marshall Lochbaum
5591d0f4f0 Boolean row-wise scans ∧˘ and ∨˘ 2024-06-18 07:46:31 -04:00
Marshall Lochbaum
7b4468c394 Move logical fold-rows functions to Singeli 2024-06-18 07:46:31 -04:00
Marshall Lochbaum
2590222988 Generic-architecture implementations of fold.singeli 2024-06-18 07:46:31 -04:00
dzaima
563c30c596 allSame Singeli helper, make NEON homMask accept signed types 2024-06-16 14:00:40 +03:00
dzaima
d85f628d72 accept tuple argument in homMask 2024-06-14 04:28:53 +03:00
dzaima
d161f93a38 more Singeli refactoring
makes some conditions more strict
removes wrong el_m & el_s definitions
2024-05-26 04:25:54 +03:00
Marshall Lochbaum
56944b8985 Minor Singeli simplifications 2024-05-25 18:31:47 -04:00
Marshall Lochbaum
90863851b7 Use new Singeli type destructuring (no change to generated C) 2024-05-25 18:31:47 -04:00
dzaima
44c8da7dc7 add checks for l>0 on equal() impls
some Singeli `fn equal` paths already assumed such, this just makes the requirement properly checked for, benefited from, and documented
2024-05-20 01:03:57 +03:00
dzaima
4f898f38d2 various Singeli usage improvements and fixes 2024-05-11 05:57:30 +03:00
Marshall Lochbaum
b2e3a5ff74 Move from & to if/and for Singeli conditions 2024-05-10 10:26:28 -04:00
Marshall Lochbaum
4d6612cb16 Replace tuplen{} and tupsel{} with length{} and select{} 2024-05-10 09:49:58 -04:00
Marshall Lochbaum
12d7629f9c Replace top-level if with if_inline 2024-05-10 08:36:24 -04:00
Marshall Lochbaum
526d9bbebe Wrap non-destructuring type parameters in parens 2024-05-10 08:30:08 -04:00
dzaima
606e07ff14 fix vqtbl1 NEON argument type 2024-02-27 01:42:56 +02:00
dzaima
c078f62b1f add GLOBAL, INIT_GLOBAL, or const to globals 2024-02-01 21:13:39 +02:00
dzaima
6e39b59e73 use tuple destructuring in mask.singeli 2024-01-28 02:38:16 +02:00
dzaima
83afa4ba99 rename functions in cmp.singeli 2024-01-11 23:36:03 +02:00
Marshall Lochbaum
f62981e370 Make Singeli if conditions have type u1 2024-01-10 17:10:53 -05:00
Marshall Lochbaum
7511420077 Fix first element of 1-byte Deduplicate result in some cases with 256 uniques 2024-01-07 21:27:12 -05:00
Marshall Lochbaum
5f3a4769be Rename constrep.singeli to replicate.singeli; not just constant now 2023-12-31 15:30:20 -05:00
Marshall Lochbaum
e50ae40f8b Port replicate-by-scan code from C to Singeli 2023-12-31 15:26:58 -05:00
Marshall Lochbaum
771496a25c Pass slow-pdep build option to Singeli to suppress BMI2 bool/bool 2023-12-31 10:05:51 -05:00
Marshall Lochbaum
9604844d0b Don't need ..._ before & in Singeli any more 2023-12-30 21:45:24 -05:00
Marshall Lochbaum
eb34213f0f Fast ≍˘ for matching-type type-sized cells 2023-12-30 21:45:24 -05:00
Marshall Lochbaum
58f4f15f8f Move more functionality to avx512.singeli 2023-12-30 10:43:40 -05:00
Marshall Lochbaum
92db3f15d0 Implement ≠` with AVX-512 clmul and GFNI 2023-12-30 10:43:06 -05:00
dzaima
9dc44ed1a9 make singeli/arch.bqn the source of SINGELI_AVX2
+ removes noting of lack of AVX2, which isn't shown from the makefile anyway
2023-12-30 02:09:56 +02:00
dzaima
18d2fa5f1c rename Singeli select function
in anticipation to Singeli update which adds select as a builtin
2023-12-29 23:35:22 +02:00
dzaima
f11bebf8f4 fix usz=64 build 2023-12-28 19:17:57 +02:00
Marshall Lochbaum
dcb031d142 Make sure value in store{} has an appropriate type 2023-12-27 16:44:53 -05:00
dzaima
840c40748f fix f64 scans on SSE2 2023-12-15 19:18:23 +02:00
dzaima
94b1d90995 SSE2 & NEON /⁼ 2023-12-14 18:08:56 +02:00
dzaima
ec4db6dbad tiny Singeli include changes 2023-12-10 15:55:57 +02:00
dzaima
3454fbe3ee defer negated ≤AVX2 comparison negation to GPR result 2023-12-05 17:37:37 +02:00
dzaima
f02c92ee6a fix 3 ∨ "abc" 2023-11-23 18:13:51 +02:00
dzaima
8814a7e66e use 2x unroll for i16→u1 copying to utilize two-vector homMask 2023-11-21 02:45:33 +02:00
dzaima
773429a959 recursive base-case of homMask 2023-11-21 02:40:50 +02:00
dzaima
08fa7d3d30 specialize x86-64 homMask of 2x[n]i16, use for comparison 2023-11-21 02:29:03 +02:00
dzaima
3ebfb24b0f unroll copying to bitarr
+ clean up comparison code as bulk homMask is supported everywhere
2023-11-21 00:18:18 +02:00
dzaima
fe985722c2 move multi-vector homMask to allow NEON to override it 2023-11-21 00:18:18 +02:00
dzaima
56cbe4aa03 mulHi → mulh for high half of element result
to not be confused with mulwHi or zipHi where it means providing the high batch of elements
2023-11-18 00:43:20 +02:00
dzaima
46e8ed1a3f fix SSE4.2 usz=64 build 2023-11-13 23:28:30 +02:00
dzaima
b4e53abbe4 some comments 2023-11-13 16:38:02 +02:00
Marshall Lochbaum
c94d51dcaf Fixes for 64-bit usz 2023-11-13 08:24:51 -05:00
dzaima
4fb6ce602f add a require{} for CRC32 2023-11-13 12:56:48 +02:00
Marshall Lochbaum
09572de49a Credible resizing policy for dyadic search hashes 2023-11-12 22:07:29 -05:00
Marshall Lochbaum
ee7a11a279 Unify self-search hash thresholds across primitives; doesn't seem to hurt performance 2023-11-12 14:35:25 -05:00
Marshall Lochbaum
5957487b70 Faster iteration over non-empty entries for hash table resizing 2023-11-11 20:24:20 -05:00
Marshall Lochbaum
b404c7c1d1 Eliminate a loop when resizing ordered hash 2023-11-11 20:24:20 -05:00
Marshall Lochbaum
3a50cda5d3 Specialize Index-of tables on result type 2023-11-10 17:12:37 -05:00
Marshall Lochbaum
b782693d15 Insert maxh to a special end bucket for ⊒ and reverse ⊐ (fixes ⊒ maxh bugs) 2023-11-10 12:04:55 -05:00
Marshall Lochbaum
e1b0c53c92 Fix various sentinel bugs 2023-11-09 20:24:46 -05:00
Marshall Lochbaum
2c625127d5 Reverse hashed Index-of 2023-11-09 10:39:07 -05:00
Marshall Lochbaum
f1485ca98c Turn hash function definition structure inside-out 2023-11-09 09:19:47 -05:00
Marshall Lochbaum
aded9d4782 Shortcutting for reverse Member-of hash 2023-11-08 17:34:33 -05:00
Marshall Lochbaum
8e9c04523a Fix overflow bug in hashed Member-of max_off calculation 2023-11-08 10:28:14 -05:00
Marshall Lochbaum
c75fdbc277 Reverse Member-of hash lookup 2023-11-07 22:07:17 -05:00
Marshall Lochbaum
8a2190b092 64-bit hashes with float normalization 2023-11-07 14:54:16 -05:00