Marshall Lochbaum
|
e261e80168
|
SSE min/max and pluswrap scans
|
2023-08-26 07:26:33 -04:00 |
|
Marshall Lochbaum
|
f9a4a5b68c
|
Remove duplicated SSE4.1 min/max bindings
|
2023-08-26 07:26:33 -04:00 |
|
Marshall Lochbaum
|
00c454d37a
|
Add generic-architecture versions of Singeli scans
|
2023-08-26 07:26:33 -04:00 |
|
dzaima
|
34d4fd6433
|
more explicit loadu/storeu
|
2023-08-20 00:18:09 +03:00 |
|
dzaima
|
4d19654e3b
|
fix store types
|
2023-08-16 19:01:04 +03:00 |
|
dzaima
|
acca5c0c7c
|
fix completely wrong dyTable lookup
|
2023-08-15 14:45:27 +03:00 |
|
Marshall Lochbaum
|
4e9745688b
|
Use generic boolean Compress as the sparse case with Singeli
|
2023-08-07 13:47:58 -04:00 |
|
Marshall Lochbaum
|
bbad98bce1
|
In AVX2 boolean Compress, jump from 8 to 32 with 32-bit shifts
|
2023-08-07 11:41:53 -04:00 |
|
Marshall Lochbaum
|
120df26814
|
mullo works for both signed and unsigned
|
2023-08-07 11:23:38 -04:00 |
|
Marshall Lochbaum
|
d039562efe
|
Fix disabled BMI2 boolean Compress; cleaner SIMD width system
|
2023-08-06 08:09:08 -04:00 |
|
Marshall Lochbaum
|
9046dd6b53
|
Handle boolean Compress SIMD tail with conditional write, not scalar loop
|
2023-08-06 08:03:14 -04:00 |
|
Marshall Lochbaum
|
07ace41d6c
|
Vectorized version of the clmul boolean Compress
|
2023-08-06 07:36:05 -04:00 |
|
Marshall Lochbaum
|
371aa6f2ef
|
AVX2 boolean Compress using variable shifts
|
2023-08-05 21:50:22 -04:00 |
|
Marshall Lochbaum
|
66c0fe041c
|
Get pext and popcount together, since generic pext emulation includes popc
|
2023-08-05 20:38:54 -04:00 |
|
Marshall Lochbaum
|
056b18edd2
|
Move pext-based compress loop from C to Singeli
|
2023-08-05 20:38:54 -04:00 |
|
Marshall Lochbaum
|
e09d385df5
|
Simplify pairwise pext implementation; drop 12-bit as no longer competitive
|
2023-08-05 20:38:54 -04:00 |
|
dzaima
|
0b6c5189d8
|
move arithdDispatch declarations to a header
+ rename some things to make more sense
|
2023-07-31 20:26:05 +03:00 |
|
dzaima
|
0d7de8b190
|
define ux in Singeli, replacing Size; return ux from popc/clz/ctz
|
2023-07-26 17:42:23 +03:00 |
|
dzaima
|
f7a0eb94a3
|
don't use AVX2 buffer copying for a 16-byte buffer
|
2023-07-22 21:34:20 +03:00 |
|
dzaima
|
1451395ef5
|
don't do fancy char stuff for unchecked arith
|
2023-07-22 21:13:49 +03:00 |
|
dzaima
|
959614c785
|
use partial application
|
2023-07-22 18:19:31 +03:00 |
|
dzaima
|
033f3de6b9
|
remove unnecessary lambda wrappers
also use fmtnat
|
2023-07-22 18:15:07 +03:00 |
|
dzaima
|
446b2d09ed
|
Singeli renames & docs
|
2023-07-22 17:37:20 +03:00 |
|
dzaima
|
6e96600878
|
move unaligned load & store to base.singeli
|
2023-07-20 00:21:09 +03:00 |
|
dzaima
|
0486adccb5
|
fix signedness of abs{x}
|
2023-07-20 00:19:54 +03:00 |
|
dzaima
|
2243fa76c1
|
better base case list
|
2023-07-19 21:56:35 +03:00 |
|
dzaima
|
eb46ec8d9c
|
rename __mulhi & __mul32
|
2023-07-19 21:54:12 +03:00 |
|
dzaima
|
700d69e7dd
|
unpack → zip & mzip; enable slash.singeli SIMD parts for NEON
|
2023-07-19 21:39:22 +03:00 |
|
dzaima
|
ba2e859dce
|
use existing unpackLo in neq.singeli
|
2023-07-19 20:56:08 +03:00 |
|
dzaima
|
19b35e1074
|
fix size comment
|
2023-07-19 20:11:14 +03:00 |
|
dzaima
|
9945711b5e
|
AVX2 i32 & i64 slash{} via widening load
|
2023-07-19 20:00:46 +03:00 |
|
dzaima
|
ff9bb258c2
|
more comments in slash.singeli
|
2023-07-19 18:50:01 +03:00 |
|
dzaima
|
48111d4873
|
extract iota{} for AVX-512
|
2023-07-19 17:24:20 +03:00 |
|
dzaima
|
a0797318ca
|
use has_simd & arch_defvw
|
2023-07-19 17:03:41 +03:00 |
|
dzaima
|
03c21112c2
|
make NEON builds work
|
2023-07-19 16:58:35 +03:00 |
|
Marshall Lochbaum
|
77188b8e73
|
Add clmul.singeli
|
2023-07-19 09:38:37 -04:00 |
|
Marshall Lochbaum
|
c4364aefa6
|
SSSE3 4-byte compress with half-width table
|
2023-07-19 08:15:53 -04:00 |
|
Marshall Lochbaum
|
ba837ba01b
|
Remove table-less where/compress methods, as they're not competitive
|
2023-07-19 07:22:55 -04:00 |
|
Marshall Lochbaum
|
64d65ae837
|
Never use generic dense compress methods if use_table: where is faster
|
2023-07-18 21:35:12 -04:00 |
|
Marshall Lochbaum
|
b0e0f210c6
|
2- and 4-byte Where with unpack and table
|
2023-07-18 21:10:39 -04:00 |
|
Marshall Lochbaum
|
93e1262864
|
Avoid using popcount primitive in Singeli if POPCNT isn't present
|
2023-07-18 16:52:08 -04:00 |
|
Marshall Lochbaum
|
6df283c379
|
Always increment by full popcount instead of two halves when splitting byte
|
2023-07-17 17:52:59 -04:00 |
|
Marshall Lochbaum
|
2f6a8e55a1
|
Always use vectors for for_special_buffered buffer copy
|
2023-07-17 16:52:10 -04:00 |
|
Marshall Lochbaum
|
8add95b7ab
|
Remove BMI2 where/compress methods
|
2023-07-17 16:20:30 -04:00 |
|
Marshall Lochbaum
|
f005169518
|
1- and 2-byte compress with table and SSSE3
|
2023-07-17 16:19:05 -04:00 |
|
Marshall Lochbaum
|
68978f7e10
|
Table-based 1- and 2-byte Where
|
2023-07-17 16:05:06 -04:00 |
|
Marshall Lochbaum
|
fc187afdf2
|
Extend 1-byte SSE2 Where to 2-byte and 4-byte with unpacked writes
|
2023-07-17 11:28:04 -04:00 |
|
Marshall Lochbaum
|
8b297ae2dc
|
Dedicated 32-bit Singeli where, taking index offset
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
3bd8d1de68
|
Extend all Singeli Compress methods to do Where
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
4415869496
|
Generic Singeli where implementation
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
c484f8ee8d
|
Avoid UB from shifting by 64
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
da5f1faa07
|
Merge BMI2 slash{} cases, with some simplifications
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
b244ba98f5
|
Unify slash1{} and slash2{}
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
f5121f6806
|
Thresholds for compress methods (versus sparse) defined in Singeli
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
824bb04b4d
|
Plain SSE2 1-byte compress
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
f852d12a5c
|
Generic Singeli compress implementation
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
f36426031a
|
Slightly faster table-less compress method
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
84a928863b
|
AVX-512 compress implementations
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
80e93c41c9
|
Use new undefined{type,len} instead of alloca
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
13b3bcd74b
|
Table-less AVX2 1-byte compress
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
d956ba921b
|
Non-overwriting 1-byte and 2-byte compress/where
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
8f4b1966cb
|
Non-overwriting AVX2 4-byte and 8-byte compress, using a buffer
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
9010f648bb
|
Unify permutevar-based compress implementations
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
0921fc2c62
|
AVX2 compress implementations using permutevar and lookup tables
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
03d8919307
|
Add has='pclmul' option and avoid using broadcast
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
871a81ca5f
|
BMI2 pext not-emulation in Singeli
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
1314859a80
|
clmul-based pext emulation
|
2023-07-16 21:56:33 -04:00 |
|
Marshall Lochbaum
|
f315a6d3ea
|
Emulate pext instruction in Singeli
|
2023-07-16 21:56:33 -04:00 |
|
dzaima
|
3946f2cc87
|
use new Singeli builtins
|
2023-07-17 02:23:50 +03:00 |
|
dzaima
|
0601a86f56
|
don't special-case 2/𝕩 on NEON
|
2023-07-11 02:40:05 +03:00 |
|
dzaima
|
cb9f4dbbc1
|
remove debug include
:|
|
2023-07-10 19:19:55 +03:00 |
|
dzaima
|
cd7807fba0
|
NEON constrep
|
2023-07-10 17:11:15 +03:00 |
|
dzaima
|
bf89d2d561
|
a little size reduction on write_indices
|
2023-07-10 16:29:21 +03:00 |
|
Marshall Lochbaum
|
f6d1f9fcab
|
Move utilities from bins to appropriate files
|
2023-07-10 07:14:44 -04:00 |
|
Marshall Lochbaum
|
492e97e2ca
|
Right argument narrowing for numeric Bins
|
2023-07-10 07:14:44 -04:00 |
|
Marshall Lochbaum
|
1fa00c9c6f
|
Update bins implementation comments
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
46c6d47055
|
Permutevar instead of shuffle for 4-byte vector binary search
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
fc57e0012d
|
Shuffle-based 4-byte vector binary search
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
af5ba0a2d2
|
Switch from Eytzinger to linear search on extra lanes
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
64ae8f9afd
|
Slightly better blend pattern for >16-byte bit table
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
c8d20fbf26
|
And 2-byte vector binary search on 4 lanes
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
fe92f91ca1
|
2-byte vector binary search on 2 lanes
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
d19df2693a
|
Merge 1-byte table code with normal bins_lookup
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
f36dc7adba
|
Use max-scan instead of plus-scan for 1-byte Bins table
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
8054597004
|
Handle up to 64 unique values in bit-table 1-byte Bins
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
b62bd02bc0
|
Precomputed midpoint bits in vector binary search
Clang already optimized to this but it's more explicit
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
7f5ba961c2
|
Use default bins_lookup if there's no AVX2
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
1080236433
|
2-byte vector binary searches
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
d665b90bbf
|
Slightly faster binary search pattern with a blend
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
a711eb72eb
|
Table-based 2-byte Bins, using max-scan
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
724f685a57
|
Cleanup
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
2c9e07f33d
|
1-byte Bins up to 32 unique eleemnts by unique lookup then index
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
81da9c586f
|
Table-based 1-byte Bins with >1-byte result
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
0346ff7c2d
|
Use type of ≠𝕨 for Singeli Bins result and allocate outside Singeli
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
e0621f99f8
|
Unroll vector binary searches for different search depths
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
ed9e8b4057
|
Table-based 1-byte Bins implementations, including AVX2
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
7161689196
|
Fix bin_search_vec over-writing
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
302d637129
|
Allocate binary search result in Singeli, using i8 for vectors
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
70249d8b64
|
In bins, promote to smallest common int type instead of i32
|
2023-07-10 07:13:21 -04:00 |
|
Marshall Lochbaum
|
ff9919dde2
|
AVX2 binary search on one lane of i8 (unused for now)
|
2023-07-10 07:13:21 -04:00 |
|