Commit Graph

295 Commits

Author SHA1 Message Date
dzaima
01b17ea0d9 remove now-unnecessary workaround for float constant 2023-04-21 15:46:54 +03:00
dzaima
a622a6761b use vtestz/vptestz instead of movemask for mask-any checks 2023-04-15 15:10:58 +03:00
dzaima
712707d27b integer avx2 bitwise operation definitions
mainly for reduced casts in C output
2023-04-15 15:00:57 +03:00
dzaima
722107e470 move makeOptBranch to base.singeli, use in search 2023-04-15 12:32:14 +03:00
dzaima
147d4530c1 slightly better overflow check for i8×i8→i8 2023-04-15 01:53:04 +03:00
dzaima
bbb9e7f0ed don't overflow-check widening array-scalar cases when not needed 2023-04-15 01:19:09 +03:00
dzaima
1db746ea77 Singeli compile-time cdiv & low-stack @collect 2023-04-09 15:03:44 +03:00
dzaima
b66f628cff tiny improvements to plus-scan 2023-04-09 12:09:52 +03:00
dzaima
52dc05f228 SIMD +` 2023-04-08 20:04:23 +03:00
dzaima
6a0385b44b better Singeli abs & absu definitions 2023-04-08 19:23:55 +03:00
dzaima
c3fe3c5b42 more Singeli utilities, more Scan fixing 2023-04-07 21:09:46 +03:00
dzaima
b0d8bcb428 add argument passing to makeOptBranch 2023-04-06 22:41:20 +03:00
dzaima
897c46df76 aarch64 fold.singeli 2023-04-06 20:17:09 +03:00
dzaima
b3f3190028 SSE2 fold.singeli 2023-04-06 19:49:14 +03:00
dzaima
1d6b33d528 restrict fold_addw to integer input 2023-04-06 19:18:43 +03:00
dzaima
d13060f725 move fold to vfold 2023-04-06 19:05:38 +03:00
Marshall Lochbaum
9d7d330a03 Use AVX2 counting for 1-byte counting sort 2023-04-06 11:32:34 -04:00
Marshall Lochbaum
0e5b98c491 Return max from AVX2 counting function 2023-04-06 11:32:34 -04:00
Marshall Lochbaum
4b18466ae2 Clean up and simplify count.singeli; allow longer final block 2023-04-06 11:32:06 -04:00
Marshall Lochbaum
06b4f06e64 Move fold{} from squeeze.singeli to vecfold.singeli 2023-04-06 11:32:06 -04:00
Marshall Lochbaum
2ddcc14852 Use minimum in addition to maximum for AVX2 counting 2023-04-06 11:32:06 -04:00
Marshall Lochbaum
e6a3e846d9 Sum counts in groups of 4 when possible 2023-04-06 11:32:06 -04:00
Marshall Lochbaum
b655dd4771 AVX2 counting function for small-range 1-byte /⁼ 2023-04-06 11:32:04 -04:00
dzaima
cba6ef505a add usz option to build/build 2023-04-06 17:47:31 +03:00
dzaima
5492eae8c5 unrolling for single-element search 2023-04-06 15:42:10 +03:00
dzaima
876914d053 Singeli single-element search 2023-04-06 14:56:49 +03:00
dzaima
8b05cd243f fix SSE2 unsigned comparisons 2023-04-05 13:47:16 +03:00
dzaima
7d1fcf2c3a move &~ to base.singeli 2023-04-02 16:33:24 +03:00
dzaima
100a188545 fix aarch64 singeli 2023-04-02 15:39:42 +03:00
dzaima
00a5ddcb89 SSE2 squeeze 2023-04-02 15:04:00 +03:00
dzaima
e0a265472c SSE2 extract{x,0} for [4]i32 and [2]i64 2023-04-02 12:10:18 +03:00
dzaima
9209ca5d6d SSE2 copy.singeli 2023-04-02 01:53:16 +03:00
dzaima
5018cf9b2e reduce unnecessary casts in C output 2023-04-01 23:04:55 +03:00
dzaima
9826201343 move neon.singeli type checks to w64/w128 2023-04-01 22:53:38 +03:00
dzaima
a62c21733e SSE2 equal.singeli 2023-04-01 22:08:47 +03:00
dzaima
96ee73f644 enable Singeli monadic arith & comparisons on SSE2 2023-04-01 20:40:29 +03:00
dzaima
3a45eac1aa separate SSE2, implement SSE2 dyarith & bitsel
also separate maskstore to homogeneous & top bit mask variants, and make blend require equal element widths
2023-04-01 20:40:23 +03:00
dzaima
dc2ac37e0a arch-independent Singeli builds 2023-03-31 16:45:29 +03:00
Marshall Lochbaum
814a677676 Make transpose movement functions strided 2023-03-29 12:37:15 -04:00
Marshall Lochbaum
b829076271 Put CPU-size transpose functions in a table 2023-03-29 12:37:15 -04:00
Marshall Lochbaum
410d51501d Check for AVX2 in Singeli transpose so it can build on non-x86 architectures 2023-03-29 12:37:15 -04:00
Marshall Lochbaum
bf04b3aab0 Move w128i and similar from sse3 and avx.singeli to base 2023-03-28 16:11:46 -04:00
Marshall Lochbaum
ebbddf9dcf Move height/width 2 transposes to Singeli 2023-03-28 16:11:46 -04:00
dzaima
21b9ad5c3c don't include debug.singeli 2023-03-27 17:53:01 +03:00
dzaima
fa8437ca25 don't use _mm256_loadu2_m128i
doesn't exist on old gcc
2023-03-23 18:21:34 +02:00
Marshall Lochbaum
699faea69c Dedicated width-overflow loop for SIMD transpose 2023-03-22 17:34:37 -04:00
Marshall Lochbaum
5a2bc15f2a Overlap SIMD transpose on width too 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
61eefe0776 Fix writing past end in i16 transpose with overlapped halves 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
8ff40cb2e5 Overlap SIMD transpose to handle uneven heights (not widths yet) 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
fad7f3aa8b Simplify 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
dd141add3f Merge i16 SIMD transpose with the rest of them 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
9673771187 Combine code for the different transpose kernel geometries 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
e40feaa81c Make transpose kernel code type-generic 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
923c485cc2 Use packed-halves transpose kernel for 8x8 i16 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
2a8e6e9c8a Separate most transpose kernel logic from loops and move i8 case into main 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
7d800bf349 SIMD transpose on 1-byte elements 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
5fccf4cda3 Refactoring; check for small transpose width/height in Singeli, not C 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
0e2e315e51 Allow an 8x8 row for 2-byte transpose 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
239c765998 SIMD transpose on 2-byte elements 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
f6c6e72661 More transpose kernel refactoring 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
c0aaa6f615 SIMD transpose on 8-byte elements 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
a0e85db702 Refactor vtranspose with more general code 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
ca44d41daa Faster i32 transpose when height is a multiple of 16 by writing cache lines 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
bfb2519aed Remove transpose_rec (was unused): the whole cache-oblivious thing seems to be useless 2023-03-22 15:14:08 -04:00
dzaima
db8a4d3af0 don't use def as a variable name 2023-03-12 00:11:30 +02:00
dzaima
1178e5df56 new Singeli syntax 2023-03-11 23:17:30 +02:00
dzaima
397f8745ce go back to regular completions if no system commands match 2023-02-27 23:51:35 +02:00
dzaima
6d03071ae6 SIMD i32 transpose 2023-02-26 01:19:54 +02:00
dzaima
c32285dba9 Singeli copy from unaligned bitarr 2023-02-18 11:52:45 +02:00
dzaima
8e720a2faa use loadLow & storeLow in neq.singeli 2023-02-17 23:41:26 +02:00
dzaima
1e6b612e13 NEON copy.singeli 2023-02-17 22:10:21 +02:00
dzaima
089e2a38c6 update Singeli things 2023-02-16 21:31:19 +02:00
dzaima
a84ca6c5d6 strict prototypes 2023-02-03 15:33:31 +02:00
dzaima
6baa96c1e3 NEON squeeze 2023-01-25 02:22:53 +02:00
dzaima
88410c02e5 prep for NEON squeeze 2023-01-24 22:21:38 +02:00
dzaima
92c855920b unroll int×u1 2023-01-23 21:43:39 +02:00
dzaima
721dd8889b monadic arith 2023-01-23 01:22:40 +02:00
dzaima
7284eb7010 more manual unrolling in dyarith.singeli 2023-01-22 22:49:44 +02:00
dzaima
162adbca68 more Singeli utilities 2023-01-22 18:04:52 +02:00
dzaima
86c2aa7e2c unroll non-overflowing dyarith loops for aarch64 2023-01-22 04:22:21 +02:00
dzaima
afe6707654 dyarith NEON support 2023-01-22 03:27:41 +02:00
dzaima
4945a90ff0 valgrind changes 2023-01-18 02:05:24 +02:00
dzaima
5de565f1b0 use 'debug/printf' for Singeli printf 2023-01-17 01:01:13 +02:00
dzaima
6c89982bbc unrolled Singeli loop option 2023-01-15 22:51:31 +02:00
dzaima
6d79ce9fb1 use arch-independent names for things suppported on both x86-64 & NEON 2022-12-24 15:26:05 +02:00
dzaima
d38316e670 BQN-based build system 2022-12-23 23:09:55 +02:00
dzaima
d05621f8f4 generic vector width calculation for NEON-supported files 2022-12-20 00:02:46 +02:00
dzaima
b91e72567d update Singeli submodule, conditional NEON includes 2022-12-19 23:31:44 +02:00
dzaima
30a6de0a31 add unrolling to aarch64 cmp.singeli 2022-12-18 22:45:42 +02:00
dzaima
f9c919d5ce getmask → homMask/topMask 2022-12-18 22:27:42 +02:00
dzaima
a07b94b889 more general equal.singeli, remove trailing semicolons 2022-12-18 21:55:20 +02:00
dzaima
2a32f52f8d blend renaming 2022-12-18 02:49:42 +02:00
dzaima
09b3279958 unified Singeli including mechanism 2022-12-18 01:54:07 +02:00
dzaima
9768526ddc move last bit of equal.singeli to loadBatch 2022-12-18 01:53:56 +02:00
dzaima
40f36fa366 Singeli-generated table for bits.singeli 2022-12-17 17:51:56 +02:00
dzaima
2c73199ed4 use Singeli-generated function tables for comparison 2022-12-16 19:04:58 +02:00
dzaima
f49dd5e8cf move cmp.singeli NaN check outside for non-x86_64 2022-12-16 17:39:51 +02:00
dzaima
6e3cb9a425 andIsZero→andAllZero; NEON andnz, blendF, sel, bitops.singeli 2022-12-16 00:44:32 +02:00
dzaima
cbd891b768 NEON widen, narrow, undefPromote; make undefPromote more restrictive 2022-12-15 23:19:00 +02:00
dzaima
6fac426f90 add v_dbl & v_half, NEON half{x,n} 2022-12-15 21:57:34 +02:00