Commit Graph

307 Commits

Author SHA1 Message Date
Marshall Lochbaum
814a677676 Make transpose movement functions strided 2023-03-29 12:37:15 -04:00
Marshall Lochbaum
b829076271 Put CPU-size transpose functions in a table 2023-03-29 12:37:15 -04:00
Marshall Lochbaum
410d51501d Check for AVX2 in Singeli transpose so it can build on non-x86 architectures 2023-03-29 12:37:15 -04:00
Marshall Lochbaum
bf04b3aab0 Move w128i and similar from sse3 and avx.singeli to base 2023-03-28 16:11:46 -04:00
Marshall Lochbaum
ebbddf9dcf Move height/width 2 transposes to Singeli 2023-03-28 16:11:46 -04:00
dzaima
21b9ad5c3c don't include debug.singeli 2023-03-27 17:53:01 +03:00
dzaima
fa8437ca25 don't use _mm256_loadu2_m128i
doesn't exist on old gcc
2023-03-23 18:21:34 +02:00
Marshall Lochbaum
699faea69c Dedicated width-overflow loop for SIMD transpose 2023-03-22 17:34:37 -04:00
Marshall Lochbaum
5a2bc15f2a Overlap SIMD transpose on width too 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
61eefe0776 Fix writing past end in i16 transpose with overlapped halves 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
8ff40cb2e5 Overlap SIMD transpose to handle uneven heights (not widths yet) 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
fad7f3aa8b Simplify 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
dd141add3f Merge i16 SIMD transpose with the rest of them 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
9673771187 Combine code for the different transpose kernel geometries 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
e40feaa81c Make transpose kernel code type-generic 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
923c485cc2 Use packed-halves transpose kernel for 8x8 i16 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
2a8e6e9c8a Separate most transpose kernel logic from loops and move i8 case into main 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
7d800bf349 SIMD transpose on 1-byte elements 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
5fccf4cda3 Refactoring; check for small transpose width/height in Singeli, not C 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
0e2e315e51 Allow an 8x8 row for 2-byte transpose 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
239c765998 SIMD transpose on 2-byte elements 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
f6c6e72661 More transpose kernel refactoring 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
c0aaa6f615 SIMD transpose on 8-byte elements 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
a0e85db702 Refactor vtranspose with more general code 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
ca44d41daa Faster i32 transpose when height is a multiple of 16 by writing cache lines 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
bfb2519aed Remove transpose_rec (was unused): the whole cache-oblivious thing seems to be useless 2023-03-22 15:14:08 -04:00
dzaima
db8a4d3af0 don't use def as a variable name 2023-03-12 00:11:30 +02:00
dzaima
1178e5df56 new Singeli syntax 2023-03-11 23:17:30 +02:00
dzaima
397f8745ce go back to regular completions if no system commands match 2023-02-27 23:51:35 +02:00
dzaima
6d03071ae6 SIMD i32 transpose 2023-02-26 01:19:54 +02:00
dzaima
c32285dba9 Singeli copy from unaligned bitarr 2023-02-18 11:52:45 +02:00
dzaima
8e720a2faa use loadLow & storeLow in neq.singeli 2023-02-17 23:41:26 +02:00
dzaima
1e6b612e13 NEON copy.singeli 2023-02-17 22:10:21 +02:00
dzaima
089e2a38c6 update Singeli things 2023-02-16 21:31:19 +02:00
dzaima
a84ca6c5d6 strict prototypes 2023-02-03 15:33:31 +02:00
dzaima
6baa96c1e3 NEON squeeze 2023-01-25 02:22:53 +02:00
dzaima
88410c02e5 prep for NEON squeeze 2023-01-24 22:21:38 +02:00
dzaima
92c855920b unroll int×u1 2023-01-23 21:43:39 +02:00
dzaima
721dd8889b monadic arith 2023-01-23 01:22:40 +02:00
dzaima
7284eb7010 more manual unrolling in dyarith.singeli 2023-01-22 22:49:44 +02:00
dzaima
162adbca68 more Singeli utilities 2023-01-22 18:04:52 +02:00
dzaima
86c2aa7e2c unroll non-overflowing dyarith loops for aarch64 2023-01-22 04:22:21 +02:00
dzaima
afe6707654 dyarith NEON support 2023-01-22 03:27:41 +02:00
dzaima
4945a90ff0 valgrind changes 2023-01-18 02:05:24 +02:00
dzaima
5de565f1b0 use 'debug/printf' for Singeli printf 2023-01-17 01:01:13 +02:00
dzaima
6c89982bbc unrolled Singeli loop option 2023-01-15 22:51:31 +02:00
dzaima
6d79ce9fb1 use arch-independent names for things suppported on both x86-64 & NEON 2022-12-24 15:26:05 +02:00
dzaima
d38316e670 BQN-based build system 2022-12-23 23:09:55 +02:00
dzaima
d05621f8f4 generic vector width calculation for NEON-supported files 2022-12-20 00:02:46 +02:00
dzaima
b91e72567d update Singeli submodule, conditional NEON includes 2022-12-19 23:31:44 +02:00
dzaima
30a6de0a31 add unrolling to aarch64 cmp.singeli 2022-12-18 22:45:42 +02:00
dzaima
f9c919d5ce getmask → homMask/topMask 2022-12-18 22:27:42 +02:00
dzaima
a07b94b889 more general equal.singeli, remove trailing semicolons 2022-12-18 21:55:20 +02:00
dzaima
2a32f52f8d blend renaming 2022-12-18 02:49:42 +02:00
dzaima
09b3279958 unified Singeli including mechanism 2022-12-18 01:54:07 +02:00
dzaima
9768526ddc move last bit of equal.singeli to loadBatch 2022-12-18 01:53:56 +02:00
dzaima
40f36fa366 Singeli-generated table for bits.singeli 2022-12-17 17:51:56 +02:00
dzaima
2c73199ed4 use Singeli-generated function tables for comparison 2022-12-16 19:04:58 +02:00
dzaima
f49dd5e8cf move cmp.singeli NaN check outside for non-x86_64 2022-12-16 17:39:51 +02:00
dzaima
6e3cb9a425 andIsZero→andAllZero; NEON andnz, blendF, sel, bitops.singeli 2022-12-16 00:44:32 +02:00
dzaima
cbd891b768 NEON widen, narrow, undefPromote; make undefPromote more restrictive 2022-12-15 23:19:00 +02:00
dzaima
6fac426f90 add v_dbl & v_half, NEON half{x,n} 2022-12-15 21:57:34 +02:00
dzaima
791e7d122f ucvt→narrow 2022-12-15 21:44:48 +02:00
dzaima
2b7291dbfb cvt{E,T,v} → widen{T,to_el{E,v}} 2022-12-15 20:31:00 +02:00
dzaima
9a9fade18a NEON loadLow, storeLow, andIsZero, maskstore, pack 2022-12-15 16:14:00 +02:00
dzaima
6f2d6bfb0b remove offset argument from maskstore; don't use w256 in mask.singeli 2022-12-15 16:13:45 +02:00
dzaima
77edc734e8 move storeLow & loadLow to sse3 & avx; fix NEON __not 2022-12-15 00:39:44 +02:00
dzaima
9fea5f1e13 neg(Any|All) → top(Any|All), msk(Any|All) → hom(Any|All) 2022-12-13 21:46:50 +02:00
dzaima
336da996b3 any/all vector check renaming 2022-12-13 02:03:17 +02:00
dzaima
7e13bf7f45 more neon.singeli 2022-12-12 02:59:11 +02:00
dzaima
444268f051 width{eltype{T}} → elwidth{T} 2022-12-12 00:07:50 +02:00
dzaima
2a1611ac81 start of neon.singeli 2022-12-11 23:59:07 +02:00
dzaima
3a3f213a92 assign broadcast{T,v} to T**v 2022-12-07 23:53:19 +02:00
dzaima
ce45d487a3 use mutating assignment 2022-12-07 23:24:26 +02:00
dzaima
07d4567a52 use spread arguments 2022-12-07 23:10:17 +02:00
dzaima
bf71fdb4f8 singeli now formats generators itself
or, well, has for a long time
2022-12-07 22:41:17 +02:00
dzaima
6ca5050f4d reduce gcc warnings 2022-12-03 23:56:03 +02:00
Marshall Lochbaum
534c92f38a Select from <=128 booleans with shuffles 2022-11-29 22:14:57 -05:00
Marshall Lochbaum
fdfeb67e8f 2-register permutevar8x32 2022-11-29 20:27:45 -05:00
Marshall Lochbaum
41f464795f Selection with permutevar8x32 2022-11-29 18:45:22 -05:00
Marshall Lochbaum
dc6e49b289 Shuffle selection using >1 data register 2022-11-28 21:51:36 -05:00
Marshall Lochbaum
b651ca73d3 Move base{} utility to base.singeli 2022-11-28 13:55:19 -05:00
Marshall Lochbaum
43fba7cd78 Extend shuffle selection to >1 byte data 2022-11-28 13:55:19 -05:00
Marshall Lochbaum
6639c17b81 1-byte select with <=16 elements using shuffle 2022-11-27 21:03:58 -05:00
Marshall Lochbaum
e09d555eba Refactoring and renaming in select.singeli 2022-11-27 09:18:21 -05:00
Marshall Lochbaum
7cb8cf5772 Enable selection with AVX2 gather for i8 to i32 indices and 1- to 8-byte values 2022-11-27 09:18:21 -05:00
dzaima
c0e06833c0 move B squeeze non-integer case out of twice-generated generator 2022-11-27 15:45:33 +02:00
dzaima
b0fcc017ce clean up squeeze.singeli 2022-11-27 15:45:33 +02:00
dzaima
0f3f91105e use anynePositive for squeeze.singeli 2022-11-27 15:45:33 +02:00
dzaima
15f6dd5d48 simplify 16-bit anynePositive 2022-11-27 15:45:33 +02:00
dzaima
5f85305f43 GPR mask testing 2022-11-27 15:45:33 +02:00
dzaima
efd79881cb use single mask array for all type widths 2022-11-27 15:45:33 +02:00
dzaima
cf5ce07f51 simplify singeli xor/and/or definitions 2022-11-24 00:21:19 +02:00
Marshall Lochbaum
7715cbc158 Add •math.Sum with Singeli f64 implementation 2022-11-23 16:51:30 -05:00
Marshall Lochbaum
52608ba6ce Singeli min/max-fold 2022-11-21 11:47:50 -05:00
dzaima
24195780df reorder headers to allow OpenBSD hack to function 2022-11-20 18:22:51 +02:00
Marshall Lochbaum
5985e92834 Initialized min/max-scan 2022-11-17 07:59:00 -05:00
Marshall Lochbaum
1924aee61c f64 min/max scan 2022-11-16 21:14:32 -05:00
Marshall Lochbaum
136c1afacc Macro-ize integer min/max scan and add sorted flag 2022-11-16 20:49:57 -05:00
dzaima
a9737c22ea extract bit array negation to function 2022-11-14 19:47:53 +02:00
dzaima
6467530823 use ty_u 2022-11-08 16:56:43 +02:00
Marshall Lochbaum
666a41ab12 1- and 2-byte Singeli boolean +` 2022-11-07 17:54:11 -05:00
dzaima
c1a77a24fb fast (2⋆n)|ints 2022-11-03 15:35:41 +02:00
dzaima
bed2708537 move non-Singeli comparisons to function lookup 2022-10-27 19:27:15 +03:00
dzaima
8d6172126c use void* instead of u8* for singeli function arguments 2022-10-25 17:27:08 +03:00
dzaima
83af05d274 fix building on gcc≤10 2022-10-10 13:53:56 +03:00
Marshall Lochbaum
ce74e36a19 Fix lots of missing casts for function calls 2022-09-30 20:20:38 -04:00
Marshall Lochbaum
47cdf02877 Build Singeli files with a filename prefix 2022-09-30 11:35:25 -04:00
Marshall Lochbaum
cad566d547 Not-equals scan with carry-less multiply 2022-09-29 20:39:46 -04:00
Marshall Lochbaum
acc200222d Move constant Replicate Singeli code to its own file 2022-09-29 20:33:09 -04:00
Marshall Lochbaum
582afe33c9 Merge 3 to 7 replicate float with other types: shuffle instead of permute 2022-09-29 20:12:45 -04:00
Marshall Lochbaum
29886f355a Combine small-replication shuffle vector tables across types 2022-09-29 19:58:14 -04:00
Marshall Lochbaum
4836f70e0d Simplify rep_const_broadcast, removing unused kv==0 case 2022-09-28 14:14:39 -04:00
Marshall Lochbaum
d28a74b572 Move 4-shuffle data to a table 2022-09-28 13:35:08 -04:00
Marshall Lochbaum
9947de0146 Performance improvements when compiling slash.singeli 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
db3e0a6324 Move all the algorithm decision logic into main rep_const and simplify 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
c430922a7e Use shared functions and shuffle array for sizes 3 to 7 in constant Replicate 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
68aeb0f7cb Add large-constant replicate method always using 4 shuffles 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
7069a60453 Handle all constant Replicate in Singeli 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
7535d768dd Singeli shuffle-based code for constant replicates <=7 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
851f50635e Some simplifications in base.singeli 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
6ed3c18389 Implement SIMD wrapping plus-scan for Replicate 2022-09-21 19:40:04 -04:00
dzaima
2358436b89 more efficient spreadBits{[8]u32, u8} 2022-09-08 21:51:25 +03:00
dzaima
d3a1fee437 singeli bit_sel 2022-09-08 21:39:51 +03:00
dzaima
96ca9092ba attempt at better bit_sel dispatch 2022-09-08 00:37:44 +03:00
dzaima
59e947cc20 more consistent type checking & conversion functions 2022-09-07 23:56:03 +03:00
dzaima
1c279a0a20 fast bitarr×intarr 2022-09-06 22:47:16 +03:00
dzaima
a90e7de997 simpler atom-array arith on length 0 inputs 2022-09-06 18:24:49 +03:00
dzaima
311aa74303 simpler bitsel for ∨ 2022-09-06 17:31:42 +03:00
dzaima
ce1c81219e manual wideness check 2022-09-06 17:23:57 +03:00
dzaima
bffe16d6e9 fix overflowed & float functions 2022-09-05 23:23:51 +03:00
dzaima
c0b2164dc8 new array-atom arith 2022-09-05 22:52:53 +03:00
dzaima
76141f6fe6 reduce lifetime of e in dyArith_AA 2022-09-01 22:45:30 +03:00
dzaima
1b8bd0707e make dyarith array-array dispatch its own function 2022-09-01 22:22:13 +03:00
dzaima
5cfa89ad5a separate functions for comparison scalar & array cases 2022-09-01 15:06:34 +03:00
dzaima
88f73d2585 fix comment & add note 2022-08-31 22:22:03 +03:00
dzaima
ce1032f80d second instruction shouldn't have a swap requirement anymore 2022-08-29 23:09:47 +03:00
dzaima
8ea2352a7b clear out swap table after first instruction generation 2022-08-29 23:08:38 +03:00
dzaima
70ee775a33 fix i16arr+i8arr overflow 2022-08-29 22:59:12 +03:00
dzaima
39a0a894e2 correct char-char→signed int overflow checking 2022-08-29 22:48:46 +03:00
dzaima
78e5799147 merge dyarith.singeli and dyarith2.singeli 2022-08-29 19:18:20 +03:00
dzaima
777c5cc1a5 rename things 2022-08-29 19:05:34 +03:00
dzaima
ce5ece34e1 new arith 2022-08-28 23:28:06 +03:00
dzaima
1694bfb7d2 s?p?rnk(x) → S?P?RNK(x) 2022-08-20 08:36:59 +03:00
dzaima
e6b4f85662 x->ia → IA(x) / PIA(x) 2022-08-20 08:30:03 +03:00
Marshall Lochbaum
56a1d9ec4c Singeli integer ⌊ and ⌈ 2022-08-15 16:01:17 -04:00
dzaima
046284b96f random cleanup 2022-07-22 03:59:44 +03:00
dzaima
041a87ff98 make valgrind accept /𝕩 and 𝕨/𝕩 impls 2022-07-10 01:51:15 +03:00
dzaima
0684fe5fcb some fixes for ubsan 2022-07-10 01:43:21 +03:00
dzaima
74eaa3853f optimize squeeze result choosing 2022-06-30 03:51:00 +03:00