Commit Graph

588 Commits

Author SHA1 Message Date
Marshall Lochbaum
2d75c6c535 And manually vectorize uninterleave 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
32ad2e9953 Manually vectorized interleave function 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
980e4b2248 Short-height version of 16x16 i8 transpose kernel, using overlapped writes 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
abb96cb18a Fixed-height transposes, more or less reverse of fixed-width 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
0ab2e485ca Width-4 transpose with shuffles 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
8bc37098dc Modular kernel transpose for odd*2 (so, just 6) 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
091f08c6cc Extend modular permutation transpose to any element size 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
780bfdfa0b Kernel transpose with overreading and skipped writes, for short rows 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
b6e418ed5b Fix modular permutation example in comment 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
0bdc43cc0f Even number handling for modular permutation ⊣˝˘ 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
4c55eab740 Fast ⊣˝˘ for power-of-two widths, 4- and 8-byte elements 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
8d9d946ec5 Fast ⊣˝˘ with pack for power-of-two widths, 1- and 2-byte elements 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
aefd2a4abd Fast 1-byte small odd width transpose with blends and modular permutation 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
5bdee886cc Extend kernel transpose from AVX2 only to SSE2, NEON 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
6e4859e17f Extend select_rows_byte to any architecture with (shuffle and) blend 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
4669418f1c Extend modular permutation ⊣˝˘ to multi-byte element types 2024-11-05 21:48:54 -05:00
Marshall Lochbaum
9997c52e4c Initial 1-byte ⊣˝˘ with blends and modular permutation 2024-11-05 21:48:54 -05:00
dzaima
bd6effe7ab restore dropped comment 2024-10-23 21:37:56 +03:00
Marshall Lochbaum
7ac7297617 More simplifications to Singeli vector utilities 2024-10-21 21:44:21 -04:00
Marshall Lochbaum
8099945068 Use vec_select from Singeli select includes 2024-10-21 18:53:03 -04:00
Marshall Lochbaum
25724b6b3e More NEON functionality moved to neon_intrin/select 2024-10-21 18:53:03 -04:00
Marshall Lochbaum
fa43ffd882 Alias vec_shuffle to shuf 2024-10-21 18:53:03 -04:00
Marshall Lochbaum
27af1c0558 Use Singeli select include for x86 blend, byteshift, zip 2024-10-21 18:53:03 -04:00
Marshall Lochbaum
6140290813 Use Singeli select includes for shuffling 2024-10-14 10:24:07 -04:00
dzaima
3d4a31ae6f access function/modifier runtime ID through macro 2024-10-10 02:38:17 +03:00
dzaima
85c5fb659d more clang-friendly multi-input homMask 2024-10-07 02:49:58 +03:00
dzaima
cf19280c54 fix reliance on signed wrapping in replicate 2024-10-03 20:33:02 +03:00
dzaima
27b7c0746c full interleave_cells
needed to make all rank-1 inputs work on non-Singeli builds, but beneficial in general anyway
2024-09-12 06:17:32 +03:00
dzaima
41785cb4bf Singeli const¨⌾(m⊸/)b 2024-09-10 01:45:56 +03:00
dzaima
02573fa4eb fatal{msg} 2024-09-10 01:45:18 +03:00
dzaima
b62e7cd820 fix generic arch build 2024-09-02 00:04:32 +03:00
Marshall Lochbaum
56cb10d6e5 Min/max scan that can skip unused argument vectors sometimes 2024-08-20 15:26:07 -04:00
dzaima
d185d32d9e remove debug include 2024-08-20 17:22:10 +03:00
dzaima
a654bd88d2 fix aarch64 build 2024-08-17 20:13:38 +03:00
dzaima
3b2b9bba1a AVX-512 bitwiden 2024-08-14 23:44:08 +03:00
dzaima
67626d1e49 a couple more AVX-512 defs 2024-08-14 23:44:08 +03:00
dzaima
6b72139d75 include avx512 in base 2024-08-14 22:13:24 +03:00
dzaima
60380e6270 fix out-of-bounds indexing in factors 2024-08-14 22:10:45 +03:00
Marshall Lochbaum
b2566c8f3a NEON k/bool support 2024-08-14 14:51:01 -04:00
Marshall Lochbaum
b801c7c186 Move to dedicated k/bool loops for k=5,7 and at least SSSE3 2024-08-14 14:51:01 -04:00
Marshall Lochbaum
b2758d355c k/bool cleanup and implementation comments 2024-08-14 14:51:01 -04:00
Marshall Lochbaum
bf69705c82 Factor (8×k)/bool as bit-replicate, then byte-replicate 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
fc1265b3f2 Enable (non-boolean) constrep on SSSE3 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
048529740b SSE2 k/bool where k divides 8 implementations 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
ee27717c97 SSE2 odd/bool implementation using [4]u32 shuffles 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
36e9ca5814 Extend odd/bool SSSE3 code to support AVX2 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
2be022921e Combine generic and SSSE3 general-case odd/bool, so SSSE3 uses mask unpacking 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
3e5dbdbf8d Dedicated generic k/bool for divisors of 8; factor; remove pdep emulation 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
13ec029d9f Get odd k/bool carry by shifting in the unified register, not after masking 2024-08-14 10:30:26 -04:00
Marshall Lochbaum
1621c7d07c Generic-arch k/bool algorithm for odd k with mask unpacking 2024-08-14 10:30:26 -04:00