trout

tuning is going well (it isnt). its super slow so i made a version that
1. doesn't do quiescence search, only eval
2. 30x less positions.
the error did end up lowering and the gradient descent seemed to converge, but the performance was awful (giving negative values to pawns in some positions). gonna have to actually do it properly probably but that'll take hours.

need to do something about this performance. almost 3 minutes for one step of gradient descent with a 30x smaller dataset is terrible

fixed the parallelism issue, it was because of the tunable parameters being in a STVector and the output of the quiescence search being in ST. haskell couldn't spark with par properly i guess because of the risk of the STVector being edited

attempted to use Control.Parallel (par and ilk) to evaluate quiescence in parallel but no dice. its barely using more than a core (same as normal). this is not normal something weird's happening

fixed the strictness bug. still very slow but it doesn't oom when using tunedEval now (which i'm not doing anyway). might start using massiv for multithreading but that honestly feels like the wrong solution, single threaded should be ok judging from other people's attempt even though this is an embarrassingly parallel problem

continue work on the tuner. having performance problems, not sure how to fix. only a couple run-throughs of the data per minute i think which is really bad.

started parsing pgns for texel-tuner. not sure if they are correct yet. started with string in parsec but it was slow and bad so i switched to text in megaparsec (even though base project uses parsec for uci... maybe switch?)

It's still not blazing fast or anything but its ok. concerningly i found a mystery game in my output trove where one of the recent engine versions crashed. it was only one in a few dozen thousand so that's not great.

also forgot to devlog so here's a new update: validated the parser-ish! played through all the moves and none broke. had to debug for a while because i'm dumb. 14 seconds all told

set up the texel-tuner stack thing. surprisingly annoying. haskell-language-server apparently doesn't play nice with packages: .., and using a cabal hie.yaml to make hls not use stack breaks because cabal isn't aware of the packages: .. in the stack.yaml (only package.yaml is mirrored to trout.cabal). Solution is to create a separate cabal.project just for hls even though i'm not even using cabal.

tested a bunch of stuff...
ended up keeping mobility tuning (tapered mobility) and changing the lmr (late move reductions, searching moves the engine immediately thinks is worse less) reduction formula from hardcoded 1-2 depths depending on type of move to logarithm-based formula based on depth and move index.

Tried to improve eval a little but it just takes too long to sprt test. going to write a texel tuner to 1. tune the piece-square tables and 2. make tuning eval easier when adding more terms

finally got some elo out of razoring with quadratic depth scaling. also messed with reverse futility pruning but not much there.

still not entirely clear on relationship between futility, reverse futility, and razoring? i mean yeah they're slightly different but it feels like they should overlap at some point.

from my logs i think I also tried big delta pruning (don't know what the actual name is, whole nodes in quiescence), aspiration window adjustments, both fail.

spent lotta time fiddling with random stuff. got some elo out of delta pruning but that's kinda it

tried 7 bitboard storage format, fail. was significnatly slower.
did end up trying tt move ordering in quiescence search but it was maybe a little worse? definitely not better.
added all pieces to mobility calculation, small elo boost. it is a lot slower now because it's doing a lot of movegen in eval but i tried removing mobility completely and it was real bad (see screenshot)

big dubs checking for transposition table cut in quiescence search. 40+ elo increase. might try transposition table move ordering next, but effectiveness might be low because of the capture restriction on quiescence moves. dunno how else i could improve quiescence move ordering because it already has SEE which is pretty powerful and a lot of other ordering is based on quiet moves only.

could try inserting to tt in quiescence? that sounds like a bad idea though

Back into the chess engine grind.!?.
Added promotion support to quiescence search. like a little over 20 elo jump?
nothing really to show besides test data so here's a screenshot that i forgot to post

made it so the website doesn't crash and burn when the user's network is slow

made mobile actually properly responsive (the viewport thing is weird...) shouldn't have taken this long but websites be websiting. you know how it is. Also the board coordinates get messed up i guess because the board is dynamically sized? have to look into that. its not a big deal because they're so hard to see in the first place

added pgn button for exporting game. I thought it would be a lot harder but chess.js just has a pgn export built in so it was fine

added promotion, finished end animations, cleaned up bugs.... it's essentially done now i think. any further that's not a bugfix would probably be for v2, like a pgn export of the game/game history (I do like the vibe currently, very simple like minesweeper or something)

added en passant! promotion is next

https://osrepnay.github.io/trout-web/ added github actions workflow with only like 4 attempts! yay me! also added end screen for losing. also realized chessground doesn't do en passant either.

lowk forgot to devlog but ui!! its very close to done now, basically the only things left is promotion, which chessground doesn't do.... dammit chessground...

web workers added! (and iterative deepening and even more primitive time management.) also fiddled with the css for responsiveness

basic web interface done. engine moves pretty fast and is not horrible but it's on the main thread so it freezes up a little bit every time. i'll need to switch to web workers later. also apparently chessground just doesn't do promotion? that's nice.

Decided to start working on wasm-based web interface in case lichess is insufficient (and also for fun). after much struggle managed to get wasm building and built into the webpage, and the engine seems to be working? time to do everything else

super stuck now. tried adding improving to lmr and nmp (horrible), more lmr and nmp fiddling (nothing), more fiddling with mobility (nonlinear) (nothing), transposition table replacement strategy (nothing). Got 20 elo from blocked pawns penalty (i guess this is mobility in a way?) but that's kinda it. i think i'll ship, im burning too much time and cpu time on this

fix issue where engine just doesn't want to checkmate. added checkmate distance penalties, but the issue was actually with reverse futility pruning (static eval of -500cp is significantly higher than beta of -99999999cp, so clearly this is fail high!). added requirement for non-pv node, probably should add more. this is turning out to be pretty dangerous optimization

got 60 elo from fixing SEE and tweaking LMR. did i mention that? SEE was broken? it was broken. it didn't actually flip the color after the first capture

kept fiddling with king safety, mobility, didn't really get anywhere. also there was a bug in static exchange evaluation which i fixed but doesn't really seem to change elo significantly? quite puzzling.

feel like im starting to hit a wall here a bit. implemented king tropism which was a big fail, king virtual mobility was a little better and got 20 elo out of it. also did futility pruning, similar elo increase. next maybe keep tuning eval, add internal iterative deepening?

fixed move ordering, that was like 60 elo. previously quiet moves and equal captures were both scored 0 below killer, now equal captures are above killers and quiets are under killers. Also, futility pruning added, small elo increase.

my tuning attempts constantly fail.... why do i somehow always stumble into a local maximum instantly....

did like 20 things with null move pruning and couldn't get any elo out of it
fixed endgame disabling (no elo), enabled it on pv searches (negative elo), disabled pv checking altogether (no elo), put it after tt cut (negative elo/??? why???) made it set pv flag to false because null window (no elo), increased reduction amount (no elo)

also added insufficient material checking to draw detection (30 elo!)
i tricked you. i lied. that gave no elo as well

i saw kevin of pzchessbot added check extensions. i added check extensions. 40 elo

don't know whether to return to tuning old parameters or add new stuff or just ship at this point. theoretically the old parameters are all untuned but i can never improve the untuned versions and it's driving me insane

added mobility to eval function
I tried fiddling with it but then lost the original, good version so i spent several hours trying to figure out where the original was
not my most productive day

late move reduction! sizeable improvement
spent a long time fiddling with settings and for some reason the one that was really good was:
1. 1 ply reductions
2. research on fail-high (i think this is a given for lmr but i'm not completely sure)
3. disable on depth < 2
i'll try messing with move ordering, number of moves tried before reducing (rn is 3), variable reduction (i tried a little bit it didn't work very well for some reason) later

Experimented with killer move heuristic from previous plys (failure) and more history heuristic fiddling (failure)
maybe late move reductions now?
also I found a bug in the 50 move rule that accidentally made it the 25 move rule. oops
and the chess engine doesn't really know how to checkmate in K v RK, I would assume PeSTO would prioritize king on the edge more but it doesn't really

get some more elo out of smarter history pruning
history gravity (i think? it's slightly different than the cpwiki version which looks wrong) and history penalties/maluses for uncutting quiet moves
i also tried adding mobility to eval but that just killed elo for some reason like -100
i don't know if it's because of terrible performance (pst-only eval is already a significant fraction of runtime) or just too much/too little fraction of eval score dedicated on mobility
might revisit and retune some parameters from previous optimizations (e.g. null move pruning R, aspiration window settings)

got history heuristic working (i made it bugged again when i first wrote it)

initial history heuristic big failure
probably need to do more fiddling, maybe reset/decay between searches, open up history cap?
also apparently history may make killers redundant??

cleaned up tt short circuit/cut code, it's much less nested and tangly now

made a logo (it's very good. it's the banner)

re-add aspiration windows! disabled a long long time ago because of lack of evidence of goodness, but with better (?) settings (starting 25 centipawns, 4x multiplier instead of 50 centipawns, 2x multiplier like previous) and better move ordering managed to squeeze out 20 elo
still some room for tuning on the aspiration window settings however

i think next is history heuristic or smarter eval
might be able to ship soon, the engine should be pretty competent now

i dont know what happened since before (basically no change before) but tt cut (returning early in case of tt hit thatt is sufficient depth) massively increases elo (~50)

Fiddle some more with killer heuristic
got ~20 more elo out of it by implementing 3 slots per ply (fifo with duplicate ignoring) and only allowing non-captures to be killers (i think this prevents interference with SEE ordering?)
next: check if nmp check detection, tt cuts (again) do anything
possibly move on to history heuristic

moved killer moves to below winning captures (by static exchange evaluation) in move sorting
convincing elo improvement
i'll probably try adding multiple slots like the wiki says and see if that can get any more out of it

implemented what i think is killer heuristic move sorting?
its very basic though and possibly incorrect and not very good

humongo move ordering bug
static exchange eval wasn't being applied when there was no transposition table result
100+ elo increase???!?
i should rename this engine in honor of a more pathetic insectoid animal out of shame

various fiddling
1. retry a bunch of aspiration window configurations, couldn't get anything reasonable (all under 10 elo increase and even dropped elo sometimes). might need smarter aspiration-aware pvs like cpwiki says that drops searching immediately if in a pv node and aspirating or just give up completely (i dont think aspiration windows+pvs is universal?)
2. attempt wasm compilation with the new ghc backend. some hiccups, like it not supporting threaded runtime so it can't properly implement the uci interface (can't cancel searches mid-search), but it's also like 4x slower on a depth 7 search when i run with wasmtime. the engine is bad enough i dont need wasm to make it seem even worse. probably just gonna throw up a lichess bot acct, i have a server to use

next: possibly killer moves, smarter eval, late moves pruning, futility pruning?

switched some ints out for int16 to increase tt size and make room for return of noderesult
this allows aspiration windows for a whole ! 7 ! elo ! increase!!!!!
idk if i keep it's so small

short devlog, but it was pvs. it was subtly broken which killed null move pruning
for fun here's a sprt test between pre- principal variation search and null move pruning and post-
61 total elo

fix pvs
apparently I wasn't re-searching failed null windows with isPV flag set
time will tell if this is the source of my null move pruning troubles

squeezed some elo out of null move pruning but it's still not great

attempt very basic null move pruning
actually loses elo (speeds up root search like crazy), gonna see if adding safeties for zugzwang helps

minor reorg (split board into its own module)
test delta pruning in quiescence but it didnt seem to do much (tried delta of queen and queen+pawn)

fixed the stupid move selection thing. everything is fast and good now
pvs has been added
in total i think around +50+ elo? from move selection and pvs combined

oh no oh no oh no oh no oh no
my move selection function is just......... dropping moves??
they're gone
moves are disappearing

static exchange evaluation (filtering during quiescence search + move ordering)
~10elo increase?
mvv-lva removed completely
still don't know what's up with pvs/mtd-f/aspiration windows/dual bucket tt hurting strength

experimented with aspiration windows, similarly middling results. seems to be a very small loss from stock

i dont get it man. pvs and mtd(f) are both getting destroyed by search without any fanciness (not even aspiration windows)
losing my damn marbles

added mtd-f (https://www.chessprogramming.org/MTD(f)) to search
why didn't I do pvs?
there was less code.........
i'll probably compare pvs and mtd-f later. apparently mtd-f is less friendly to some search techniques and is definitely less tested/used

very simple mvv-lva for additional move ordering
very slight performance increase when searching root

Added PV logging to info
trying to find weird behavior where it can't find checkmate at higher depths
also illegal moves in pv log...

quiescence tweaks
most importantly make capture-only movegen, slightly faster (1.8 -> 1.5sec)
duplication seems a little ridiculous but there's no like super obvious solution that's still performant...

initial, slow, quiescence search implementation
no transposition table integration yet
no capture-only movegen yet
slightly incorrect with node result

finally mostly eliminated tt perf issue...
apparently it was a cache problem? i love cache problems
the boxed vectors were killing performance
added storable instances for everything and switched to storable vectors
below is criterion benchmark snippets

Commit to mutable transposition table.
reduced overhead made everything overall faster (~half a second on benchmark), but the weird transposition table size discrepancy remains
maybe it's just normal cpu cache whatever? i hope not

Continued fiddling + benchmarking with the mutable transposition table
Still don't understand why its so slow (and why speed varies by size significantly??)

various performance testing + trial of mutable array-backed transposition tables
somehow appears to be slower/worse than before?? I don't know why.....

simplify bestmove by using transposition table result
this is slightly slower (~~3.8 -> ~~3.9 sec on depth 6) because the transposition table is really really not great. might be forced to switch to st table because its so bad

add simplistic draw test (no checkmate or stalemate test yet..)

Actually add draw support (also I lied there was stalemate support, just unlabeled)
Pending testing, but this marks the first time any chess engine of mine has been actually fully conformant to the rules! yay!

Finish fleshing out game in preparation for draw support
Also apparently I never added stalemate feature? weird

zobrist was broken too, test was using cached value accidentally (effectively x==x) instead of recalculating correct zobrist

Apparently getPiece has been broken for ????
It didn't respect the piece bitboard format and returned the wrong piece every time I think
Added a test to check in case it happens again...

In the process of trying to make transposition table work with the draw rules. Very annoying. Tried to make transposition table faster by not checking for actual equality (only going off of hash), and apparently we already have a hash collision? very not cool.
Also another discovery: IntMap is significantly slower (in this use-case) than HashMap with Int keys. Why??????????????

try to make transposition table more implementation agnostic