Commit Graph

461 Commits

Author SHA1 Message Date
dzaima
a62c21733e SSE2 equal.singeli 2023-04-01 22:08:47 +03:00
dzaima
96ee73f644 enable Singeli monadic arith & comparisons on SSE2 2023-04-01 20:40:29 +03:00
dzaima
3a45eac1aa separate SSE2, implement SSE2 dyarith & bitsel
also separate maskstore to homogeneous & top bit mask variants, and make blend require equal element widths
2023-04-01 20:40:23 +03:00
dzaima
dc2ac37e0a arch-independent Singeli builds 2023-03-31 16:45:29 +03:00
Marshall Lochbaum
814a677676 Make transpose movement functions strided 2023-03-29 12:37:15 -04:00
Marshall Lochbaum
b829076271 Put CPU-size transpose functions in a table 2023-03-29 12:37:15 -04:00
Marshall Lochbaum
410d51501d Check for AVX2 in Singeli transpose so it can build on non-x86 architectures 2023-03-29 12:37:15 -04:00
Marshall Lochbaum
bf04b3aab0 Move w128i and similar from sse3 and avx.singeli to base 2023-03-28 16:11:46 -04:00
Marshall Lochbaum
ebbddf9dcf Move height/width 2 transposes to Singeli 2023-03-28 16:11:46 -04:00
dzaima
21b9ad5c3c don't include debug.singeli 2023-03-27 17:53:01 +03:00
dzaima
fa8437ca25 don't use _mm256_loadu2_m128i
doesn't exist on old gcc
2023-03-23 18:21:34 +02:00
Marshall Lochbaum
699faea69c Dedicated width-overflow loop for SIMD transpose 2023-03-22 17:34:37 -04:00
Marshall Lochbaum
5a2bc15f2a Overlap SIMD transpose on width too 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
61eefe0776 Fix writing past end in i16 transpose with overlapped halves 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
8ff40cb2e5 Overlap SIMD transpose to handle uneven heights (not widths yet) 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
fad7f3aa8b Simplify 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
dd141add3f Merge i16 SIMD transpose with the rest of them 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
9673771187 Combine code for the different transpose kernel geometries 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
e40feaa81c Make transpose kernel code type-generic 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
923c485cc2 Use packed-halves transpose kernel for 8x8 i16 2023-03-22 15:14:09 -04:00
Marshall Lochbaum
2a8e6e9c8a Separate most transpose kernel logic from loops and move i8 case into main 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
7d800bf349 SIMD transpose on 1-byte elements 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
5fccf4cda3 Refactoring; check for small transpose width/height in Singeli, not C 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
0e2e315e51 Allow an 8x8 row for 2-byte transpose 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
239c765998 SIMD transpose on 2-byte elements 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
f6c6e72661 More transpose kernel refactoring 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
c0aaa6f615 SIMD transpose on 8-byte elements 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
a0e85db702 Refactor vtranspose with more general code 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
ca44d41daa Faster i32 transpose when height is a multiple of 16 by writing cache lines 2023-03-22 15:14:08 -04:00
Marshall Lochbaum
bfb2519aed Remove transpose_rec (was unused): the whole cache-oblivious thing seems to be useless 2023-03-22 15:14:08 -04:00
dzaima
db8a4d3af0 don't use def as a variable name 2023-03-12 00:11:30 +02:00
dzaima
1178e5df56 new Singeli syntax 2023-03-11 23:17:30 +02:00
dzaima
397f8745ce go back to regular completions if no system commands match 2023-02-27 23:51:35 +02:00
dzaima
6d03071ae6 SIMD i32 transpose 2023-02-26 01:19:54 +02:00
dzaima
c32285dba9 Singeli copy from unaligned bitarr 2023-02-18 11:52:45 +02:00
dzaima
8e720a2faa use loadLow & storeLow in neq.singeli 2023-02-17 23:41:26 +02:00
dzaima
1e6b612e13 NEON copy.singeli 2023-02-17 22:10:21 +02:00
dzaima
089e2a38c6 update Singeli things 2023-02-16 21:31:19 +02:00
dzaima
a84ca6c5d6 strict prototypes 2023-02-03 15:33:31 +02:00
dzaima
6baa96c1e3 NEON squeeze 2023-01-25 02:22:53 +02:00
dzaima
88410c02e5 prep for NEON squeeze 2023-01-24 22:21:38 +02:00
dzaima
92c855920b unroll int×u1 2023-01-23 21:43:39 +02:00
dzaima
721dd8889b monadic arith 2023-01-23 01:22:40 +02:00
dzaima
7284eb7010 more manual unrolling in dyarith.singeli 2023-01-22 22:49:44 +02:00
dzaima
162adbca68 more Singeli utilities 2023-01-22 18:04:52 +02:00
dzaima
86c2aa7e2c unroll non-overflowing dyarith loops for aarch64 2023-01-22 04:22:21 +02:00
dzaima
afe6707654 dyarith NEON support 2023-01-22 03:27:41 +02:00
dzaima
4945a90ff0 valgrind changes 2023-01-18 02:05:24 +02:00
dzaima
5de565f1b0 use 'debug/printf' for Singeli printf 2023-01-17 01:01:13 +02:00
dzaima
6c89982bbc unrolled Singeli loop option 2023-01-15 22:51:31 +02:00
dzaima
6d79ce9fb1 use arch-independent names for things suppported on both x86-64 & NEON 2022-12-24 15:26:05 +02:00
dzaima
d38316e670 BQN-based build system 2022-12-23 23:09:55 +02:00
dzaima
d05621f8f4 generic vector width calculation for NEON-supported files 2022-12-20 00:02:46 +02:00
dzaima
b91e72567d update Singeli submodule, conditional NEON includes 2022-12-19 23:31:44 +02:00
dzaima
30a6de0a31 add unrolling to aarch64 cmp.singeli 2022-12-18 22:45:42 +02:00
dzaima
f9c919d5ce getmask → homMask/topMask 2022-12-18 22:27:42 +02:00
dzaima
a07b94b889 more general equal.singeli, remove trailing semicolons 2022-12-18 21:55:20 +02:00
dzaima
2a32f52f8d blend renaming 2022-12-18 02:49:42 +02:00
dzaima
09b3279958 unified Singeli including mechanism 2022-12-18 01:54:07 +02:00
dzaima
9768526ddc move last bit of equal.singeli to loadBatch 2022-12-18 01:53:56 +02:00
dzaima
40f36fa366 Singeli-generated table for bits.singeli 2022-12-17 17:51:56 +02:00
dzaima
2c73199ed4 use Singeli-generated function tables for comparison 2022-12-16 19:04:58 +02:00
dzaima
f49dd5e8cf move cmp.singeli NaN check outside for non-x86_64 2022-12-16 17:39:51 +02:00
dzaima
6e3cb9a425 andIsZero→andAllZero; NEON andnz, blendF, sel, bitops.singeli 2022-12-16 00:44:32 +02:00
dzaima
cbd891b768 NEON widen, narrow, undefPromote; make undefPromote more restrictive 2022-12-15 23:19:00 +02:00
dzaima
6fac426f90 add v_dbl & v_half, NEON half{x,n} 2022-12-15 21:57:34 +02:00
dzaima
791e7d122f ucvt→narrow 2022-12-15 21:44:48 +02:00
dzaima
2b7291dbfb cvt{E,T,v} → widen{T,to_el{E,v}} 2022-12-15 20:31:00 +02:00
dzaima
9a9fade18a NEON loadLow, storeLow, andIsZero, maskstore, pack 2022-12-15 16:14:00 +02:00
dzaima
6f2d6bfb0b remove offset argument from maskstore; don't use w256 in mask.singeli 2022-12-15 16:13:45 +02:00
dzaima
77edc734e8 move storeLow & loadLow to sse3 & avx; fix NEON __not 2022-12-15 00:39:44 +02:00
dzaima
9fea5f1e13 neg(Any|All) → top(Any|All), msk(Any|All) → hom(Any|All) 2022-12-13 21:46:50 +02:00
dzaima
336da996b3 any/all vector check renaming 2022-12-13 02:03:17 +02:00
dzaima
7e13bf7f45 more neon.singeli 2022-12-12 02:59:11 +02:00
dzaima
444268f051 width{eltype{T}} → elwidth{T} 2022-12-12 00:07:50 +02:00
dzaima
2a1611ac81 start of neon.singeli 2022-12-11 23:59:07 +02:00
dzaima
3a3f213a92 assign broadcast{T,v} to T**v 2022-12-07 23:53:19 +02:00
dzaima
ce45d487a3 use mutating assignment 2022-12-07 23:24:26 +02:00
dzaima
07d4567a52 use spread arguments 2022-12-07 23:10:17 +02:00
dzaima
bf71fdb4f8 singeli now formats generators itself
or, well, has for a long time
2022-12-07 22:41:17 +02:00
dzaima
6ca5050f4d reduce gcc warnings 2022-12-03 23:56:03 +02:00
Marshall Lochbaum
534c92f38a Select from <=128 booleans with shuffles 2022-11-29 22:14:57 -05:00
Marshall Lochbaum
fdfeb67e8f 2-register permutevar8x32 2022-11-29 20:27:45 -05:00
Marshall Lochbaum
41f464795f Selection with permutevar8x32 2022-11-29 18:45:22 -05:00
Marshall Lochbaum
dc6e49b289 Shuffle selection using >1 data register 2022-11-28 21:51:36 -05:00
Marshall Lochbaum
b651ca73d3 Move base{} utility to base.singeli 2022-11-28 13:55:19 -05:00
Marshall Lochbaum
43fba7cd78 Extend shuffle selection to >1 byte data 2022-11-28 13:55:19 -05:00
Marshall Lochbaum
6639c17b81 1-byte select with <=16 elements using shuffle 2022-11-27 21:03:58 -05:00
Marshall Lochbaum
e09d555eba Refactoring and renaming in select.singeli 2022-11-27 09:18:21 -05:00
Marshall Lochbaum
7cb8cf5772 Enable selection with AVX2 gather for i8 to i32 indices and 1- to 8-byte values 2022-11-27 09:18:21 -05:00
dzaima
c0e06833c0 move B squeeze non-integer case out of twice-generated generator 2022-11-27 15:45:33 +02:00
dzaima
b0fcc017ce clean up squeeze.singeli 2022-11-27 15:45:33 +02:00
dzaima
0f3f91105e use anynePositive for squeeze.singeli 2022-11-27 15:45:33 +02:00
dzaima
15f6dd5d48 simplify 16-bit anynePositive 2022-11-27 15:45:33 +02:00
dzaima
5f85305f43 GPR mask testing 2022-11-27 15:45:33 +02:00
dzaima
efd79881cb use single mask array for all type widths 2022-11-27 15:45:33 +02:00
dzaima
cf5ce07f51 simplify singeli xor/and/or definitions 2022-11-24 00:21:19 +02:00
Marshall Lochbaum
7715cbc158 Add •math.Sum with Singeli f64 implementation 2022-11-23 16:51:30 -05:00
Marshall Lochbaum
52608ba6ce Singeli min/max-fold 2022-11-21 11:47:50 -05:00
dzaima
24195780df reorder headers to allow OpenBSD hack to function 2022-11-20 18:22:51 +02:00
Marshall Lochbaum
5985e92834 Initialized min/max-scan 2022-11-17 07:59:00 -05:00
Marshall Lochbaum
1924aee61c f64 min/max scan 2022-11-16 21:14:32 -05:00
Marshall Lochbaum
136c1afacc Macro-ize integer min/max scan and add sorted flag 2022-11-16 20:49:57 -05:00
dzaima
a9737c22ea extract bit array negation to function 2022-11-14 19:47:53 +02:00
dzaima
6467530823 use ty_u 2022-11-08 16:56:43 +02:00
Marshall Lochbaum
666a41ab12 1- and 2-byte Singeli boolean +` 2022-11-07 17:54:11 -05:00
dzaima
c1a77a24fb fast (2⋆n)|ints 2022-11-03 15:35:41 +02:00
dzaima
bed2708537 move non-Singeli comparisons to function lookup 2022-10-27 19:27:15 +03:00
dzaima
8d6172126c use void* instead of u8* for singeli function arguments 2022-10-25 17:27:08 +03:00
dzaima
83af05d274 fix building on gcc≤10 2022-10-10 13:53:56 +03:00
Marshall Lochbaum
ce74e36a19 Fix lots of missing casts for function calls 2022-09-30 20:20:38 -04:00
Marshall Lochbaum
47cdf02877 Build Singeli files with a filename prefix 2022-09-30 11:35:25 -04:00
Marshall Lochbaum
cad566d547 Not-equals scan with carry-less multiply 2022-09-29 20:39:46 -04:00
Marshall Lochbaum
acc200222d Move constant Replicate Singeli code to its own file 2022-09-29 20:33:09 -04:00
Marshall Lochbaum
582afe33c9 Merge 3 to 7 replicate float with other types: shuffle instead of permute 2022-09-29 20:12:45 -04:00
Marshall Lochbaum
29886f355a Combine small-replication shuffle vector tables across types 2022-09-29 19:58:14 -04:00
Marshall Lochbaum
4836f70e0d Simplify rep_const_broadcast, removing unused kv==0 case 2022-09-28 14:14:39 -04:00
Marshall Lochbaum
d28a74b572 Move 4-shuffle data to a table 2022-09-28 13:35:08 -04:00
Marshall Lochbaum
9947de0146 Performance improvements when compiling slash.singeli 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
db3e0a6324 Move all the algorithm decision logic into main rep_const and simplify 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
c430922a7e Use shared functions and shuffle array for sizes 3 to 7 in constant Replicate 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
68aeb0f7cb Add large-constant replicate method always using 4 shuffles 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
7069a60453 Handle all constant Replicate in Singeli 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
7535d768dd Singeli shuffle-based code for constant replicates <=7 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
851f50635e Some simplifications in base.singeli 2022-09-27 22:00:59 -04:00
Marshall Lochbaum
6ed3c18389 Implement SIMD wrapping plus-scan for Replicate 2022-09-21 19:40:04 -04:00
dzaima
2358436b89 more efficient spreadBits{[8]u32, u8} 2022-09-08 21:51:25 +03:00
dzaima
d3a1fee437 singeli bit_sel 2022-09-08 21:39:51 +03:00
dzaima
96ca9092ba attempt at better bit_sel dispatch 2022-09-08 00:37:44 +03:00
dzaima
59e947cc20 more consistent type checking & conversion functions 2022-09-07 23:56:03 +03:00
dzaima
1c279a0a20 fast bitarr×intarr 2022-09-06 22:47:16 +03:00
dzaima
a90e7de997 simpler atom-array arith on length 0 inputs 2022-09-06 18:24:49 +03:00
dzaima
311aa74303 simpler bitsel for ∨ 2022-09-06 17:31:42 +03:00
dzaima
ce1c81219e manual wideness check 2022-09-06 17:23:57 +03:00
dzaima
bffe16d6e9 fix overflowed & float functions 2022-09-05 23:23:51 +03:00
dzaima
c0b2164dc8 new array-atom arith 2022-09-05 22:52:53 +03:00
dzaima
76141f6fe6 reduce lifetime of e in dyArith_AA 2022-09-01 22:45:30 +03:00
dzaima
1b8bd0707e make dyarith array-array dispatch its own function 2022-09-01 22:22:13 +03:00
dzaima
5cfa89ad5a separate functions for comparison scalar & array cases 2022-09-01 15:06:34 +03:00
dzaima
88f73d2585 fix comment & add note 2022-08-31 22:22:03 +03:00
dzaima
ce1032f80d second instruction shouldn't have a swap requirement anymore 2022-08-29 23:09:47 +03:00
dzaima
8ea2352a7b clear out swap table after first instruction generation 2022-08-29 23:08:38 +03:00
dzaima
70ee775a33 fix i16arr+i8arr overflow 2022-08-29 22:59:12 +03:00
dzaima
39a0a894e2 correct char-char→signed int overflow checking 2022-08-29 22:48:46 +03:00
dzaima
78e5799147 merge dyarith.singeli and dyarith2.singeli 2022-08-29 19:18:20 +03:00
dzaima
777c5cc1a5 rename things 2022-08-29 19:05:34 +03:00
dzaima
ce5ece34e1 new arith 2022-08-28 23:28:06 +03:00
dzaima
1694bfb7d2 s?p?rnk(x) → S?P?RNK(x) 2022-08-20 08:36:59 +03:00
dzaima
e6b4f85662 x->ia → IA(x) / PIA(x) 2022-08-20 08:30:03 +03:00
Marshall Lochbaum
56a1d9ec4c Singeli integer ⌊ and ⌈ 2022-08-15 16:01:17 -04:00
dzaima
046284b96f random cleanup 2022-07-22 03:59:44 +03:00
dzaima
041a87ff98 make valgrind accept /𝕩 and 𝕨/𝕩 impls 2022-07-10 01:51:15 +03:00
dzaima
0684fe5fcb some fixes for ubsan 2022-07-10 01:43:21 +03:00
dzaima
74eaa3853f optimize squeeze result choosing 2022-06-30 03:51:00 +03:00
dzaima
859be5efb2 singeli chr_squeeze 2022-06-30 01:47:57 +03:00
dzaima
75bfa5d905 singeli num_squeeze 2022-06-30 01:47:50 +03:00
dzaima
8c73112d6c pull latest singeli 2022-05-07 15:28:30 +03:00
dzaima
1babdf247f unified int & char cases for bitarr/x 2022-04-30 21:49:47 +03:00
dzaima
de0352caf3 more usage of non-zero length for loops 2022-04-25 18:03:15 +03:00
dzaima
95455241a4 don't do useless tail case 2022-04-25 16:08:03 +03:00
dzaima
ab868a55f3 singeli copy 2022-04-25 03:06:11 +03:00
dzaima
37da98d871 singeli code cleanup 2022-04-21 22:05:47 +03:00
dzaima
7cd19c2840 use ~~ more 2022-04-21 19:35:35 +03:00
Marshall Lochbaum
3aea6aa9d0 Boolean prefix sum using AVX2 shuffling instead of pdep 2022-04-21 11:43:25 -04:00
dzaima
4c03520a07 use maskedLoop in dyarith 2022-04-21 18:33:36 +03:00
dzaima
113f72e86d singeli bitarr conversion to wider arrays 2022-04-21 18:27:20 +03:00
dzaima
b1fd85621a actually use maskstore 2022-04-21 17:48:05 +03:00
dzaima
6f5414b2a5 ∨`bitarr 2022-04-21 15:12:23 +03:00
dzaima
c3504ec42b proper includes 2022-04-21 04:13:14 +03:00
dzaima
cd431d3338 singeli +` 2022-04-21 03:51:37 +03:00
dzaima
11d95b67d5 better error messages for no inverse header 2022-04-15 23:20:44 +03:00
dzaima
a4a72a78c4 singeli select (disabled) 2022-04-12 16:45:43 +03:00
dzaima
422dc80283 decG round 4 2022-04-07 17:19:36 +03:00
dzaima
04dbf9a9a4 add another sse3 include 2022-04-07 11:40:03 +03:00
dzaima
4a316c9398 don't include setjmp.h when CATCH_ERRORS=0 2022-04-06 18:52:54 +03:00
dzaima
1609e8b651 singeli cleanup 2022-04-04 16:28:13 +03:00
dzaima
e031b069fb test special float values 2022-04-02 20:00:34 +03:00
dzaima
b329ddee1d unaligned 128-bit vector loads 2022-04-02 17:57:49 +03:00
dzaima
40dc184063 use singeli for c16lut generation 2022-04-02 17:19:20 +03:00
dzaima
49ff5a04ff singeli ≡ 2022-04-02 16:58:35 +03:00
dzaima
b6dc3cc239 /bitarr → i8arr & i16arr 2022-04-01 23:30:57 +03:00
dzaima
37bf551e54 BMI2 bitarr/i8arr & bitarr/i16arr 2022-04-01 22:41:32 +03:00
dzaima
54a9000d88 more general any-not-equal check 2022-03-28 02:54:13 +03:00
dzaima
09e48854dd skip 16-bit movmsk for addition & subtraction 2022-03-28 01:56:46 +03:00
dzaima
60d3ba8c64 singeli collect 2022-03-20 00:30:06 +02:00
dzaima
4de990a73b bit of singeli cleanup, JIT comment 2022-03-18 15:24:13 +02:00
dzaima
47c8c1e5de singeli: iota, better make 2022-03-13 18:46:57 +02:00
dzaima
d771f07b30 more random singeli additions 2022-03-13 15:40:59 +02:00
dzaima
5cd19eafe8 vectorized dyadic arith tail element handling 2022-02-27 18:46:35 +02:00
dzaima
b1483e14e1 random singeli functions 2022-02-25 23:46:18 +02:00
dzaima
369831827a better singeli type restriction checks 2022-02-25 21:44:38 +02:00
dzaima
7af80e9e3c singeli cast changes 2022-02-25 18:22:26 +02:00
dzaima
0d977b73cd clean up some singeli 2022-01-25 19:54:18 +02:00
dzaima
3fb793526c )erase; try to not oom on printing errors 2022-01-19 01:02:07 +02:00
dzaima
39d1fc0eb3 fix precedence 2022-01-16 04:50:09 +02:00
dzaima
a6d4c8bb32 more singeli definitions 2022-01-12 18:29:27 +02:00
dzaima
1770251bad singeli + & - & × 2021-12-18 22:30:00 +02:00
dzaima
e7662b0f6a separate array & scalar cases in arith builtins 2021-12-13 00:16:23 +02:00
dzaima
dba145a14b rename folder 2021-10-28 17:50:10 +03:00
dzaima
dfa04f4922 properly free 𝕩 when needed 2021-10-26 16:15:33 +03:00
dzaima
8eb3989d84 separate avx file 2021-10-25 16:18:03 +03:00
dzaima
071aebd8ea god dammit c compilers, why 2021-10-25 15:45:28 +03:00
dzaima
798f6e9e17 move length zero check outside singeli 2021-10-25 14:00:01 +03:00
dzaima
4e17e28d4a get rid of function pointer cast 2021-10-25 13:40:40 +03:00
dzaima
4717023167 cmp C cleanup, assert length!=0 before while 2021-10-25 12:27:21 +03:00
dzaima
f8798affc4 full array-scalar comparison error checking in singeli 2021-10-25 01:59:25 +03:00
dzaima
3e372620a2 array-scalar comparison function lookup table 2021-10-24 21:12:39 +03:00
dzaima
d42dd3dcf9 merged typed types 2021-10-22 17:11:54 +03:00
dzaima
4ca34e3f8e function lookup table 2021-10-22 09:53:40 +03:00
dzaima
5d76148c51 lazy singeli bit comparison fns 2021-10-22 00:58:34 +03:00
dzaima
71c454235c singeli 2021-10-22 00:17:44 +03:00