Commit Graph

696 Commits

Author SHA1 Message Date
dzaima
9c68e67544 fix compilation error when targeting AVX1-only 2023-10-07 13:22:27 +03:00
Marshall Lochbaum
6330d61a88 Some rearrangement and minor simplifications 2023-09-12 10:24:00 -04:00
Marshall Lochbaum
72da47d48a Use standard for loops 2023-09-12 09:12:57 -04:00
Marshall Lochbaum
40ed55bc43 Use built-in NEON include for basic vector instructions 2023-09-11 21:17:38 -04:00
Marshall Lochbaum
7b9f7ba331 Move x86 vector and NEON includes into base.singeli 2023-09-11 20:25:25 -04:00
Marshall Lochbaum
082f800b3e Combine x86 vector extensions and use iintrinsic/basic when possible 2023-09-11 20:11:16 -04:00
Marshall Lochbaum
c85ca66dae Unify Scan and Bins min-scan code 2023-08-26 09:27:59 -04:00
Marshall Lochbaum
0a30fb309d SSE2 min- and max-scans 2023-08-26 07:26:33 -04:00
Marshall Lochbaum
a040a14744 Unify power-of-two shift pattern for scans 2023-08-26 07:26:33 -04:00
Marshall Lochbaum
d16ba6c3b1 Move xor-scan to scan.singeli; remove neq.singeli 2023-08-26 07:26:33 -04:00
Marshall Lochbaum
e261e80168 SSE min/max and pluswrap scans 2023-08-26 07:26:33 -04:00
Marshall Lochbaum
f9a4a5b68c Remove duplicated SSE4.1 min/max bindings 2023-08-26 07:26:33 -04:00
Marshall Lochbaum
00c454d37a Add generic-architecture versions of Singeli scans 2023-08-26 07:26:33 -04:00
dzaima
34d4fd6433 more explicit loadu/storeu 2023-08-20 00:18:09 +03:00
dzaima
4d19654e3b fix store types 2023-08-16 19:01:04 +03:00
dzaima
acca5c0c7c fix completely wrong dyTable lookup 2023-08-15 14:45:27 +03:00
Marshall Lochbaum
4e9745688b Use generic boolean Compress as the sparse case with Singeli 2023-08-07 13:47:58 -04:00
Marshall Lochbaum
bbad98bce1 In AVX2 boolean Compress, jump from 8 to 32 with 32-bit shifts 2023-08-07 11:41:53 -04:00
Marshall Lochbaum
120df26814 mullo works for both signed and unsigned 2023-08-07 11:23:38 -04:00
Marshall Lochbaum
d039562efe Fix disabled BMI2 boolean Compress; cleaner SIMD width system 2023-08-06 08:09:08 -04:00
Marshall Lochbaum
9046dd6b53 Handle boolean Compress SIMD tail with conditional write, not scalar loop 2023-08-06 08:03:14 -04:00
Marshall Lochbaum
07ace41d6c Vectorized version of the clmul boolean Compress 2023-08-06 07:36:05 -04:00
Marshall Lochbaum
371aa6f2ef AVX2 boolean Compress using variable shifts 2023-08-05 21:50:22 -04:00
Marshall Lochbaum
66c0fe041c Get pext and popcount together, since generic pext emulation includes popc 2023-08-05 20:38:54 -04:00
Marshall Lochbaum
056b18edd2 Move pext-based compress loop from C to Singeli 2023-08-05 20:38:54 -04:00
Marshall Lochbaum
e09d385df5 Simplify pairwise pext implementation; drop 12-bit as no longer competitive 2023-08-05 20:38:54 -04:00
dzaima
0b6c5189d8 move arithdDispatch declarations to a header
+ rename some things to make more sense
2023-07-31 20:26:05 +03:00
dzaima
0d7de8b190 define ux in Singeli, replacing Size; return ux from popc/clz/ctz 2023-07-26 17:42:23 +03:00
dzaima
f7a0eb94a3 don't use AVX2 buffer copying for a 16-byte buffer 2023-07-22 21:34:20 +03:00
dzaima
1451395ef5 don't do fancy char stuff for unchecked arith 2023-07-22 21:13:49 +03:00
dzaima
959614c785 use partial application 2023-07-22 18:19:31 +03:00
dzaima
033f3de6b9 remove unnecessary lambda wrappers
also use fmtnat
2023-07-22 18:15:07 +03:00
dzaima
446b2d09ed Singeli renames & docs 2023-07-22 17:37:20 +03:00
dzaima
6e96600878 move unaligned load & store to base.singeli 2023-07-20 00:21:09 +03:00
dzaima
0486adccb5 fix signedness of abs{x} 2023-07-20 00:19:54 +03:00
dzaima
2243fa76c1 better base case list 2023-07-19 21:56:35 +03:00
dzaima
eb46ec8d9c rename __mulhi & __mul32 2023-07-19 21:54:12 +03:00
dzaima
700d69e7dd unpack → zip & mzip; enable slash.singeli SIMD parts for NEON 2023-07-19 21:39:22 +03:00
dzaima
ba2e859dce use existing unpackLo in neq.singeli 2023-07-19 20:56:08 +03:00
dzaima
19b35e1074 fix size comment 2023-07-19 20:11:14 +03:00
dzaima
9945711b5e AVX2 i32 & i64 slash{} via widening load 2023-07-19 20:00:46 +03:00
dzaima
ff9bb258c2 more comments in slash.singeli 2023-07-19 18:50:01 +03:00
dzaima
48111d4873 extract iota{} for AVX-512 2023-07-19 17:24:20 +03:00
dzaima
a0797318ca use has_simd & arch_defvw 2023-07-19 17:03:41 +03:00
dzaima
03c21112c2 make NEON builds work 2023-07-19 16:58:35 +03:00
Marshall Lochbaum
77188b8e73 Add clmul.singeli 2023-07-19 09:38:37 -04:00
Marshall Lochbaum
c4364aefa6 SSSE3 4-byte compress with half-width table 2023-07-19 08:15:53 -04:00
Marshall Lochbaum
ba837ba01b Remove table-less where/compress methods, as they're not competitive 2023-07-19 07:22:55 -04:00
Marshall Lochbaum
64d65ae837 Never use generic dense compress methods if use_table: where is faster 2023-07-18 21:35:12 -04:00
Marshall Lochbaum
b0e0f210c6 2- and 4-byte Where with unpack and table 2023-07-18 21:10:39 -04:00
Marshall Lochbaum
93e1262864 Avoid using popcount primitive in Singeli if POPCNT isn't present 2023-07-18 16:52:08 -04:00
Marshall Lochbaum
6df283c379 Always increment by full popcount instead of two halves when splitting byte 2023-07-17 17:52:59 -04:00
Marshall Lochbaum
2f6a8e55a1 Always use vectors for for_special_buffered buffer copy 2023-07-17 16:52:10 -04:00
Marshall Lochbaum
8add95b7ab Remove BMI2 where/compress methods 2023-07-17 16:20:30 -04:00
Marshall Lochbaum
f005169518 1- and 2-byte compress with table and SSSE3 2023-07-17 16:19:05 -04:00
Marshall Lochbaum
68978f7e10 Table-based 1- and 2-byte Where 2023-07-17 16:05:06 -04:00
Marshall Lochbaum
fc187afdf2 Extend 1-byte SSE2 Where to 2-byte and 4-byte with unpacked writes 2023-07-17 11:28:04 -04:00
Marshall Lochbaum
8b297ae2dc Dedicated 32-bit Singeli where, taking index offset 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
3bd8d1de68 Extend all Singeli Compress methods to do Where 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
4415869496 Generic Singeli where implementation 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
c484f8ee8d Avoid UB from shifting by 64 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
da5f1faa07 Merge BMI2 slash{} cases, with some simplifications 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
b244ba98f5 Unify slash1{} and slash2{} 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
f5121f6806 Thresholds for compress methods (versus sparse) defined in Singeli 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
824bb04b4d Plain SSE2 1-byte compress 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
f852d12a5c Generic Singeli compress implementation 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
f36426031a Slightly faster table-less compress method 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
84a928863b AVX-512 compress implementations 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
80e93c41c9 Use new undefined{type,len} instead of alloca 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
13b3bcd74b Table-less AVX2 1-byte compress 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
d956ba921b Non-overwriting 1-byte and 2-byte compress/where 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
8f4b1966cb Non-overwriting AVX2 4-byte and 8-byte compress, using a buffer 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
9010f648bb Unify permutevar-based compress implementations 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
0921fc2c62 AVX2 compress implementations using permutevar and lookup tables 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
03d8919307 Add has='pclmul' option and avoid using broadcast 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
871a81ca5f BMI2 pext not-emulation in Singeli 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
1314859a80 clmul-based pext emulation 2023-07-16 21:56:33 -04:00
Marshall Lochbaum
f315a6d3ea Emulate pext instruction in Singeli 2023-07-16 21:56:33 -04:00
dzaima
3946f2cc87 use new Singeli builtins 2023-07-17 02:23:50 +03:00
dzaima
0601a86f56 don't special-case 2/𝕩 on NEON 2023-07-11 02:40:05 +03:00
dzaima
cb9f4dbbc1 remove debug include
:|
2023-07-10 19:19:55 +03:00
dzaima
cd7807fba0 NEON constrep 2023-07-10 17:11:15 +03:00
dzaima
bf89d2d561 a little size reduction on write_indices 2023-07-10 16:29:21 +03:00
Marshall Lochbaum
f6d1f9fcab Move utilities from bins to appropriate files 2023-07-10 07:14:44 -04:00
Marshall Lochbaum
492e97e2ca Right argument narrowing for numeric Bins 2023-07-10 07:14:44 -04:00
Marshall Lochbaum
1fa00c9c6f Update bins implementation comments 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
46c6d47055 Permutevar instead of shuffle for 4-byte vector binary search 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
fc57e0012d Shuffle-based 4-byte vector binary search 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
af5ba0a2d2 Switch from Eytzinger to linear search on extra lanes 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
64ae8f9afd Slightly better blend pattern for >16-byte bit table 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
c8d20fbf26 And 2-byte vector binary search on 4 lanes 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
fe92f91ca1 2-byte vector binary search on 2 lanes 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
d19df2693a Merge 1-byte table code with normal bins_lookup 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
f36dc7adba Use max-scan instead of plus-scan for 1-byte Bins table 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
8054597004 Handle up to 64 unique values in bit-table 1-byte Bins 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
b62bd02bc0 Precomputed midpoint bits in vector binary search
Clang already optimized to this but it's more explicit
2023-07-10 07:13:21 -04:00
Marshall Lochbaum
7f5ba961c2 Use default bins_lookup if there's no AVX2 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
1080236433 2-byte vector binary searches 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
d665b90bbf Slightly faster binary search pattern with a blend 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
a711eb72eb Table-based 2-byte Bins, using max-scan 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
724f685a57 Cleanup 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
2c9e07f33d 1-byte Bins up to 32 unique eleemnts by unique lookup then index 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
81da9c586f Table-based 1-byte Bins with >1-byte result 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
0346ff7c2d Use type of ≠𝕨 for Singeli Bins result and allocate outside Singeli 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
e0621f99f8 Unroll vector binary searches for different search depths 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
ed9e8b4057 Table-based 1-byte Bins implementations, including AVX2 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
7161689196 Fix bin_search_vec over-writing 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
302d637129 Allocate binary search result in Singeli, using i8 for vectors 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
70249d8b64 In bins, promote to smallest common int type instead of i32 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
ff9919dde2 AVX2 binary search on one lane of i8 (unused for now) 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
ee84f9fe1f 4-way branchless binary search unrolling 2023-07-10 07:13:21 -04:00
Marshall Lochbaum
464a53463d Generic Singeli version of 32-bit binary search 2023-07-10 07:13:21 -04:00
dzaima
5c7934387a update Singeli submodule, match{} → same{} 2023-06-29 15:19:57 +03:00
dzaima
a67a19dd0b move pair{{a,b}} expanding to base.singeli 2023-06-29 01:31:58 +03:00
dzaima
4ee8dfbbda update Singeli submodule, use destructuring parameters 2023-06-28 00:59:28 +03:00
dzaima
7618dce2f3 fix return value of overflowed dyadic arith 2023-06-23 12:10:42 +03:00
dzaima
4dbe72ae03 properly indicate estimating Singeli instructions 2023-06-23 11:51:34 +03:00
dzaima
8f8380749b generic Singeli builds 2023-06-16 11:12:15 +03:00
dzaima
5032c9a3ab remove unnecessary 'over _ to' 2023-06-15 19:12:20 +03:00
dzaima
90e7537225 move maskedLoopPositive to @-based 2023-06-15 19:09:53 +03:00
dzaima
dee29a1a02 block → iter 2023-06-15 18:11:57 +03:00
dzaima
9da567cd57 switch to new Singeli block execution 2023-06-15 18:08:57 +03:00
dzaima
2da457dafa fix el_half 2023-06-15 00:32:19 +03:00
dzaima
4260ef93b3 rename away underscores in __adds and __subs
plus add NEON addw, subw, and mulw (which is renamed from mul1)
2023-06-12 00:52:09 +03:00
dzaima
2a52dff482 @-forms of maskedLoop and muLoop 2023-06-12 00:52:09 +03:00
dzaima
fe8892ce6c rearrange shrm/shlm args, slightly better NEON homMask{[16]u8} 2023-06-12 00:52:09 +03:00
dzaima
8542ba071d homMaskX, merging multi-arg homMask 2023-06-07 18:14:18 +03:00
dzaima
c9e7dc9976 remove a debug include 2023-05-20 00:01:35 +03:00
dzaima
f6703df0b2 range calculation functions 2023-05-19 14:01:32 +03:00
dzaima
8bfe2b6b5b fix unrolled integer |𝕩 (not used currently) 2023-05-19 13:15:51 +03:00
dzaima
616fd48785 base.singeli additions 2023-05-19 13:15:51 +03:00
dzaima
e4d222173d a bunch of fixes for generic architecture & wasm 2023-05-16 00:07:48 +03:00
Marshall Lochbaum
53fb8db06f SSSE3 support in bit-table code 2023-05-11 21:05:24 -04:00
Marshall Lochbaum
464dd27a37 SIMD table intialization for Index-of 2023-05-11 21:05:24 -04:00
Marshall Lochbaum
844b537593 Singeli 1-byte table implementations for generic architecture 2023-05-11 21:05:23 -04:00
Marshall Lochbaum
6c81fb3149 Make AVX2 deduplicate branchless 2023-05-11 21:04:58 -04:00
Marshall Lochbaum
1700d84740 AVX2 1-byte Deduplicate implementation 2023-05-11 21:04:58 -04:00
Marshall Lochbaum
9658104f3e Unify AVX2 bit-table code between Member-of and Mark Firsts 2023-05-11 21:04:58 -04:00
Marshall Lochbaum
ba43dae908 Heuristic filtering when AVX2 Mark Firsts sees multiple new values 2023-05-11 21:04:58 -04:00
Marshall Lochbaum
9c7c6aabc5 AVX2 Mark Firsts implementation 2023-05-11 21:04:56 -04:00
Marshall Lochbaum
2fafe11d7c Fix fill_bittab tail masking for t=0 and simplify 2023-05-11 20:59:53 -04:00
Marshall Lochbaum
e45231154c Reverse lookup initialization for 1-byte Member-of 2023-05-11 20:59:53 -04:00
Marshall Lochbaum
0232f91582 AVX2 bit table Member-of 2023-05-11 20:59:52 -04:00
dzaima
03a28e4e34 fix float normalization from a •bit._cast result mutating the original 2023-05-11 22:39:59 +03:00
dzaima
300d217ee8 move SIMD constrep functions to a table 2023-05-09 00:14:14 +03:00
dzaima
56d1453185 use destructuring more in Singeli 2023-05-06 13:18:00 +03:00
dzaima
c567dff565 fix leading-axis operations on empty inputs 2023-04-30 19:15:46 +03:00
dzaima
1e11cf93c1 faster leading axis arithmetic 2023-04-30 17:27:07 +03:00
dzaima
aa5ed7cc7a BB2B & BBB2B → FC1 & FC2 2023-04-29 15:31:31 +03:00
dzaima
34a4d48d95 make singeli assertions checked in debug mode 2023-04-28 15:10:08 +03:00
dzaima
187eea1197 use memcpy for non-tiny equal-size copying 2023-04-26 20:19:05 +03:00
dzaima
01b17ea0d9 remove now-unnecessary workaround for float constant 2023-04-21 15:46:54 +03:00
dzaima
a622a6761b use vtestz/vptestz instead of movemask for mask-any checks 2023-04-15 15:10:58 +03:00
dzaima
712707d27b integer avx2 bitwise operation definitions
mainly for reduced casts in C output
2023-04-15 15:00:57 +03:00
dzaima
722107e470 move makeOptBranch to base.singeli, use in search 2023-04-15 12:32:14 +03:00
dzaima
147d4530c1 slightly better overflow check for i8×i8→i8 2023-04-15 01:53:04 +03:00
dzaima
bbb9e7f0ed don't overflow-check widening array-scalar cases when not needed 2023-04-15 01:19:09 +03:00
dzaima
1db746ea77 Singeli compile-time cdiv & low-stack @collect 2023-04-09 15:03:44 +03:00
dzaima
b66f628cff tiny improvements to plus-scan 2023-04-09 12:09:52 +03:00
dzaima
52dc05f228 SIMD +` 2023-04-08 20:04:23 +03:00
dzaima
6a0385b44b better Singeli abs & absu definitions 2023-04-08 19:23:55 +03:00
dzaima
c3fe3c5b42 more Singeli utilities, more Scan fixing 2023-04-07 21:09:46 +03:00
dzaima
b0d8bcb428 add argument passing to makeOptBranch 2023-04-06 22:41:20 +03:00
dzaima
897c46df76 aarch64 fold.singeli 2023-04-06 20:17:09 +03:00
dzaima
b3f3190028 SSE2 fold.singeli 2023-04-06 19:49:14 +03:00
dzaima
1d6b33d528 restrict fold_addw to integer input 2023-04-06 19:18:43 +03:00
dzaima
d13060f725 move fold to vfold 2023-04-06 19:05:38 +03:00
Marshall Lochbaum
9d7d330a03 Use AVX2 counting for 1-byte counting sort 2023-04-06 11:32:34 -04:00
Marshall Lochbaum
0e5b98c491 Return max from AVX2 counting function 2023-04-06 11:32:34 -04:00
Marshall Lochbaum
4b18466ae2 Clean up and simplify count.singeli; allow longer final block 2023-04-06 11:32:06 -04:00
Marshall Lochbaum
06b4f06e64 Move fold{} from squeeze.singeli to vecfold.singeli 2023-04-06 11:32:06 -04:00
Marshall Lochbaum
2ddcc14852 Use minimum in addition to maximum for AVX2 counting 2023-04-06 11:32:06 -04:00
Marshall Lochbaum
e6a3e846d9 Sum counts in groups of 4 when possible 2023-04-06 11:32:06 -04:00
Marshall Lochbaum
b655dd4771 AVX2 counting function for small-range 1-byte /⁼ 2023-04-06 11:32:04 -04:00
dzaima
cba6ef505a add usz option to build/build 2023-04-06 17:47:31 +03:00
dzaima
5492eae8c5 unrolling for single-element search 2023-04-06 15:42:10 +03:00
dzaima
876914d053 Singeli single-element search 2023-04-06 14:56:49 +03:00
dzaima
8b05cd243f fix SSE2 unsigned comparisons 2023-04-05 13:47:16 +03:00
dzaima
7d1fcf2c3a move &~ to base.singeli 2023-04-02 16:33:24 +03:00
dzaima
100a188545 fix aarch64 singeli 2023-04-02 15:39:42 +03:00
dzaima
00a5ddcb89 SSE2 squeeze 2023-04-02 15:04:00 +03:00
dzaima
e0a265472c SSE2 extract{x,0} for [4]i32 and [2]i64 2023-04-02 12:10:18 +03:00
dzaima
9209ca5d6d SSE2 copy.singeli 2023-04-02 01:53:16 +03:00
dzaima
5018cf9b2e reduce unnecessary casts in C output 2023-04-01 23:04:55 +03:00
dzaima
9826201343 move neon.singeli type checks to w64/w128 2023-04-01 22:53:38 +03:00
dzaima
a62c21733e SSE2 equal.singeli 2023-04-01 22:08:47 +03:00
dzaima
96ee73f644 enable Singeli monadic arith & comparisons on SSE2 2023-04-01 20:40:29 +03:00
dzaima
3a45eac1aa separate SSE2, implement SSE2 dyarith & bitsel
also separate maskstore to homogeneous & top bit mask variants, and make blend require equal element widths
2023-04-01 20:40:23 +03:00
dzaima
dc2ac37e0a arch-independent Singeli builds 2023-03-31 16:45:29 +03:00
Marshall Lochbaum
814a677676 Make transpose movement functions strided 2023-03-29 12:37:15 -04:00
Marshall Lochbaum
b829076271 Put CPU-size transpose functions in a table 2023-03-29 12:37:15 -04:00
Marshall Lochbaum
410d51501d Check for AVX2 in Singeli transpose so it can build on non-x86 architectures 2023-03-29 12:37:15 -04:00
Marshall Lochbaum
bf04b3aab0 Move w128i and similar from sse3 and avx.singeli to base 2023-03-28 16:11:46 -04:00
Marshall Lochbaum
ebbddf9dcf Move height/width 2 transposes to Singeli 2023-03-28 16:11:46 -04:00
dzaima
21b9ad5c3c don't include debug.singeli 2023-03-27 17:53:01 +03:00
dzaima
fa8437ca25 don't use _mm256_loadu2_m128i
doesn't exist on old gcc
2023-03-23 18:21:34 +02:00
Marshall Lochbaum
699faea69c Dedicated width-overflow loop for SIMD transpose 2023-03-22 17:34:37 -04:00
Marshall Lochbaum
5a2bc15f2a Overlap SIMD transpose on width too 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
61eefe0776 Fix writing past end in i16 transpose with overlapped halves 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
8ff40cb2e5 Overlap SIMD transpose to handle uneven heights (not widths yet) 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
fad7f3aa8b Simplify 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
dd141add3f Merge i16 SIMD transpose with the rest of them 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
9673771187 Combine code for the different transpose kernel geometries 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
e40feaa81c Make transpose kernel code type-generic 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
923c485cc2 Use packed-halves transpose kernel for 8x8 i16 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
2a8e6e9c8a Separate most transpose kernel logic from loops and move i8 case into main 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
7d800bf349 SIMD transpose on 1-byte elements 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
5fccf4cda3 Refactoring; check for small transpose width/height in Singeli, not C 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
0e2e315e51 Allow an 8x8 row for 2-byte transpose 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
239c765998 SIMD transpose on 2-byte elements 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
f6c6e72661 More transpose kernel refactoring 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
c0aaa6f615 SIMD transpose on 8-byte elements 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
a0e85db702 Refactor vtranspose with more general code 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
ca44d41daa Faster i32 transpose when height is a multiple of 16 by writing cache lines 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
bfb2519aed Remove transpose_rec (was unused): the whole cache-oblivious thing seems to be useless 2023-03-22 15:14:08 -04:00
dzaima
db8a4d3af0 don't use def as a variable name 2023-03-12 00:11:30 +02:00
dzaima
1178e5df56 new Singeli syntax 2023-03-11 23:17:30 +02:00
dzaima
397f8745ce go back to regular completions if no system commands match 2023-02-27 23:51:35 +02:00
dzaima
6d03071ae6 SIMD i32 transpose 2023-02-26 01:19:54 +02:00
dzaima
c32285dba9 Singeli copy from unaligned bitarr 2023-02-18 11:52:45 +02:00
dzaima
8e720a2faa use loadLow & storeLow in neq.singeli 2023-02-17 23:41:26 +02:00
dzaima
1e6b612e13 NEON copy.singeli 2023-02-17 22:10:21 +02:00
dzaima
089e2a38c6 update Singeli things 2023-02-16 21:31:19 +02:00
dzaima
a84ca6c5d6 strict prototypes 2023-02-03 15:33:31 +02:00
dzaima
6baa96c1e3 NEON squeeze 2023-01-25 02:22:53 +02:00
dzaima
88410c02e5 prep for NEON squeeze 2023-01-24 22:21:38 +02:00
dzaima
92c855920b unroll int×u1 2023-01-23 21:43:39 +02:00
dzaima
721dd8889b monadic arith 2023-01-23 01:22:40 +02:00
dzaima
7284eb7010 more manual unrolling in dyarith.singeli 2023-01-22 22:49:44 +02:00
dzaima
162adbca68 more Singeli utilities 2023-01-22 18:04:52 +02:00
dzaima
86c2aa7e2c unroll non-overflowing dyarith loops for aarch64 2023-01-22 04:22:21 +02:00
dzaima
afe6707654 dyarith NEON support 2023-01-22 03:27:41 +02:00
dzaima
4945a90ff0 valgrind changes 2023-01-18 02:05:24 +02:00
dzaima
5de565f1b0 use 'debug/printf' for Singeli printf 2023-01-17 01:01:13 +02:00
dzaima
6c89982bbc unrolled Singeli loop option 2023-01-15 22:51:31 +02:00
dzaima
6d79ce9fb1 use arch-independent names for things suppported on both x86-64 & NEON 2022-12-24 15:26:05 +02:00
dzaima
d38316e670 BQN-based build system 2022-12-23 23:09:55 +02:00
dzaima
d05621f8f4 generic vector width calculation for NEON-supported files 2022-12-20 00:02:46 +02:00
dzaima
b91e72567d update Singeli submodule, conditional NEON includes 2022-12-19 23:31:44 +02:00
dzaima
30a6de0a31 add unrolling to aarch64 cmp.singeli 2022-12-18 22:45:42 +02:00
dzaima
f9c919d5ce getmask → homMask/topMask 2022-12-18 22:27:42 +02:00
dzaima
a07b94b889 more general equal.singeli, remove trailing semicolons 2022-12-18 21:55:20 +02:00
dzaima
2a32f52f8d blend renaming 2022-12-18 02:49:42 +02:00
dzaima
09b3279958 unified Singeli including mechanism 2022-12-18 01:54:07 +02:00
dzaima
9768526ddc move last bit of equal.singeli to loadBatch 2022-12-18 01:53:56 +02:00
dzaima
40f36fa366 Singeli-generated table for bits.singeli 2022-12-17 17:51:56 +02:00
dzaima
2c73199ed4 use Singeli-generated function tables for comparison 2022-12-16 19:04:58 +02:00
dzaima
f49dd5e8cf move cmp.singeli NaN check outside for non-x86_64 2022-12-16 17:39:51 +02:00
dzaima
6e3cb9a425 andIsZero→andAllZero; NEON andnz, blendF, sel, bitops.singeli 2022-12-16 00:44:32 +02:00
dzaima
cbd891b768 NEON widen, narrow, undefPromote; make undefPromote more restrictive 2022-12-15 23:19:00 +02:00
dzaima
6fac426f90 add v_dbl & v_half, NEON half{x,n} 2022-12-15 21:57:34 +02:00
dzaima
791e7d122f ucvt→narrow 2022-12-15 21:44:48 +02:00
dzaima
2b7291dbfb cvt{E,T,v} → widen{T,to_el{E,v}} 2022-12-15 20:31:00 +02:00
dzaima
9a9fade18a NEON loadLow, storeLow, andIsZero, maskstore, pack 2022-12-15 16:14:00 +02:00
dzaima
6f2d6bfb0b remove offset argument from maskstore; don't use w256 in mask.singeli 2022-12-15 16:13:45 +02:00
dzaima
77edc734e8 move storeLow & loadLow to sse3 & avx; fix NEON __not 2022-12-15 00:39:44 +02:00
dzaima
9fea5f1e13 neg(Any|All) → top(Any|All), msk(Any|All) → hom(Any|All) 2022-12-13 21:46:50 +02:00
dzaima
336da996b3 any/all vector check renaming 2022-12-13 02:03:17 +02:00
dzaima
7e13bf7f45 more neon.singeli 2022-12-12 02:59:11 +02:00
dzaima
444268f051 width{eltype{T}} → elwidth{T} 2022-12-12 00:07:50 +02:00
dzaima
2a1611ac81 start of neon.singeli 2022-12-11 23:59:07 +02:00
dzaima
3a3f213a92 assign broadcast{T,v} to T**v 2022-12-07 23:53:19 +02:00
dzaima
ce45d487a3 use mutating assignment 2022-12-07 23:24:26 +02:00
dzaima
07d4567a52 use spread arguments 2022-12-07 23:10:17 +02:00
dzaima
bf71fdb4f8 singeli now formats generators itself
or, well, has for a long time
2022-12-07 22:41:17 +02:00
dzaima
6ca5050f4d reduce gcc warnings 2022-12-03 23:56:03 +02:00
Marshall Lochbaum
534c92f38a Select from <=128 booleans with shuffles 2022-11-29 22:14:57 -05:00
Marshall Lochbaum
fdfeb67e8f 2-register permutevar8x32 2022-11-29 20:27:45 -05:00
Marshall Lochbaum
41f464795f Selection with permutevar8x32 2022-11-29 18:45:22 -05:00
Marshall Lochbaum
dc6e49b289 Shuffle selection using >1 data register 2022-11-28 21:51:36 -05:00
Marshall Lochbaum
b651ca73d3 Move base{} utility to base.singeli 2022-11-28 13:55:19 -05:00
Marshall Lochbaum
43fba7cd78 Extend shuffle selection to >1 byte data 2022-11-28 13:55:19 -05:00
Marshall Lochbaum
6639c17b81 1-byte select with <=16 elements using shuffle 2022-11-27 21:03:58 -05:00
Marshall Lochbaum
e09d555eba Refactoring and renaming in select.singeli 2022-11-27 09:18:21 -05:00
Marshall Lochbaum
7cb8cf5772 Enable selection with AVX2 gather for i8 to i32 indices and 1- to 8-byte values 2022-11-27 09:18:21 -05:00
dzaima
c0e06833c0 move B squeeze non-integer case out of twice-generated generator 2022-11-27 15:45:33 +02:00
dzaima
b0fcc017ce clean up squeeze.singeli 2022-11-27 15:45:33 +02:00
dzaima
0f3f91105e use anynePositive for squeeze.singeli 2022-11-27 15:45:33 +02:00
dzaima
15f6dd5d48 simplify 16-bit anynePositive 2022-11-27 15:45:33 +02:00
dzaima
5f85305f43 GPR mask testing 2022-11-27 15:45:33 +02:00
dzaima
efd79881cb use single mask array for all type widths 2022-11-27 15:45:33 +02:00
dzaima
cf5ce07f51 simplify singeli xor/and/or definitions 2022-11-24 00:21:19 +02:00
Marshall Lochbaum
7715cbc158 Add •math.Sum with Singeli f64 implementation 2022-11-23 16:51:30 -05:00
Marshall Lochbaum
52608ba6ce Singeli min/max-fold 2022-11-21 11:47:50 -05:00
dzaima
24195780df reorder headers to allow OpenBSD hack to function 2022-11-20 18:22:51 +02:00
Marshall Lochbaum
5985e92834 Initialized min/max-scan 2022-11-17 07:59:00 -05:00
Marshall Lochbaum
1924aee61c f64 min/max scan 2022-11-16 21:14:32 -05:00
Marshall Lochbaum
136c1afacc Macro-ize integer min/max scan and add sorted flag 2022-11-16 20:49:57 -05:00
dzaima
a9737c22ea extract bit array negation to function 2022-11-14 19:47:53 +02:00
dzaima
6467530823 use ty_u 2022-11-08 16:56:43 +02:00
Marshall Lochbaum
666a41ab12 1- and 2-byte Singeli boolean +` 2022-11-07 17:54:11 -05:00
dzaima
c1a77a24fb fast (2⋆n)|ints 2022-11-03 15:35:41 +02:00
dzaima
bed2708537 move non-Singeli comparisons to function lookup 2022-10-27 19:27:15 +03:00
dzaima
8d6172126c use void* instead of u8* for singeli function arguments 2022-10-25 17:27:08 +03:00
dzaima
83af05d274 fix building on gcc≤10 2022-10-10 13:53:56 +03:00
Marshall Lochbaum
ce74e36a19 Fix lots of missing casts for function calls 2022-09-30 20:20:38 -04:00
Marshall Lochbaum
47cdf02877 Build Singeli files with a filename prefix 2022-09-30 11:35:25 -04:00
Marshall Lochbaum
cad566d547 Not-equals scan with carry-less multiply 2022-09-29 20:39:46 -04:00
Marshall Lochbaum
acc200222d Move constant Replicate Singeli code to its own file 2022-09-29 20:33:09 -04:00
Marshall Lochbaum
582afe33c9 Merge 3 to 7 replicate float with other types: shuffle instead of permute 2022-09-29 20:12:45 -04:00
Marshall Lochbaum
29886f355a Combine small-replication shuffle vector tables across types 2022-09-29 19:58:14 -04:00
Marshall Lochbaum
4836f70e0d Simplify rep_const_broadcast, removing unused kv==0 case 2022-09-28 14:14:39 -04:00
Marshall Lochbaum
d28a74b572 Move 4-shuffle data to a table 2022-09-28 13:35:08 -04:00
Marshall Lochbaum
9947de0146 Performance improvements when compiling slash.singeli 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
db3e0a6324 Move all the algorithm decision logic into main rep_const and simplify 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
c430922a7e Use shared functions and shuffle array for sizes 3 to 7 in constant Replicate 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
68aeb0f7cb Add large-constant replicate method always using 4 shuffles 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
7069a60453 Handle all constant Replicate in Singeli 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
7535d768dd Singeli shuffle-based code for constant replicates <=7 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
851f50635e Some simplifications in base.singeli 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
6ed3c18389 Implement SIMD wrapping plus-scan for Replicate 2022-09-21 19:40:04 -04:00
dzaima
2358436b89 more efficient spreadBits{[8]u32, u8} 2022-09-08 21:51:25 +03:00
dzaima
d3a1fee437 singeli bit_sel 2022-09-08 21:39:51 +03:00
dzaima
96ca9092ba attempt at better bit_sel dispatch 2022-09-08 00:37:44 +03:00
dzaima
59e947cc20 more consistent type checking & conversion functions 2022-09-07 23:56:03 +03:00
dzaima
1c279a0a20 fast bitarr×intarr 2022-09-06 22:47:16 +03:00
dzaima
a90e7de997 simpler atom-array arith on length 0 inputs 2022-09-06 18:24:49 +03:00
dzaima
311aa74303 simpler bitsel for ∨ 2022-09-06 17:31:42 +03:00
dzaima
ce1c81219e manual wideness check 2022-09-06 17:23:57 +03:00
dzaima
bffe16d6e9 fix overflowed & float functions 2022-09-05 23:23:51 +03:00
dzaima
c0b2164dc8 new array-atom arith 2022-09-05 22:52:53 +03:00
dzaima
76141f6fe6 reduce lifetime of e in dyArith_AA 2022-09-01 22:45:30 +03:00
dzaima
1b8bd0707e make dyarith array-array dispatch its own function 2022-09-01 22:22:13 +03:00
dzaima
5cfa89ad5a separate functions for comparison scalar & array cases 2022-09-01 15:06:34 +03:00
dzaima
88f73d2585 fix comment & add note 2022-08-31 22:22:03 +03:00
dzaima
ce1032f80d second instruction shouldn't have a swap requirement anymore 2022-08-29 23:09:47 +03:00
dzaima
8ea2352a7b clear out swap table after first instruction generation 2022-08-29 23:08:38 +03:00
dzaima
70ee775a33 fix i16arr+i8arr overflow 2022-08-29 22:59:12 +03:00
dzaima
39a0a894e2 correct char-char→signed int overflow checking 2022-08-29 22:48:46 +03:00
dzaima
78e5799147 merge dyarith.singeli and dyarith2.singeli 2022-08-29 19:18:20 +03:00
dzaima
777c5cc1a5 rename things 2022-08-29 19:05:34 +03:00
dzaima
ce5ece34e1 new arith 2022-08-28 23:28:06 +03:00
dzaima
1694bfb7d2 s?p?rnk(x) → S?P?RNK(x) 2022-08-20 08:36:59 +03:00
dzaima
e6b4f85662 x->ia → IA(x) / PIA(x) 2022-08-20 08:30:03 +03:00
Marshall Lochbaum
56a1d9ec4c Singeli integer ⌊ and ⌈ 2022-08-15 16:01:17 -04:00
dzaima
046284b96f random cleanup 2022-07-22 03:59:44 +03:00
dzaima
041a87ff98 make valgrind accept /𝕩 and 𝕨/𝕩 impls 2022-07-10 01:51:15 +03:00
dzaima
0684fe5fcb some fixes for ubsan 2022-07-10 01:43:21 +03:00
dzaima
74eaa3853f optimize squeeze result choosing 2022-06-30 03:51:00 +03:00
dzaima
859be5efb2 singeli chr_squeeze 2022-06-30 01:47:57 +03:00
dzaima
75bfa5d905 singeli num_squeeze 2022-06-30 01:47:50 +03:00
dzaima
8c73112d6c pull latest singeli 2022-05-07 15:28:30 +03:00
dzaima
1babdf247f unified int & char cases for bitarr/x 2022-04-30 21:49:47 +03:00
dzaima
de0352caf3 more usage of non-zero length for loops 2022-04-25 18:03:15 +03:00
dzaima
95455241a4 don't do useless tail case 2022-04-25 16:08:03 +03:00
dzaima
ab868a55f3 singeli copy 2022-04-25 03:06:11 +03:00
dzaima
37da98d871 singeli code cleanup 2022-04-21 22:05:47 +03:00
dzaima
7cd19c2840 use ~~ more 2022-04-21 19:35:35 +03:00
Marshall Lochbaum
3aea6aa9d0 Boolean prefix sum using AVX2 shuffling instead of pdep 2022-04-21 11:43:25 -04:00
dzaima
4c03520a07 use maskedLoop in dyarith 2022-04-21 18:33:36 +03:00