Osmium -- Chess Engine Used AI

continued to try and fix TT for better hit rates, added a bunch of statistical stuff to try and figure out the problem, which is likely that the move pruning is too aggressive, albeit collisions are also part of it. This also for some reason killed the nodes per sec from ~~2M to ~~1M, which i'll try and fix in the next update
Need to continue working on UCI too because I paused work on that to work on TT and stuff
Also started working on Chess2, a side project which is why you may see more gaps between updates for this engine, once again a bit burnt out

worked more on the transposition table more, trying to improve hitrates, did get it up to around 17% at one point but it's still kinda bad for depth 11 even though the TT was allocated 256MB ram (albeit from testing changing the allocation size from 16~256 didn't make too much difference), much improvements needed

Did some performance profiling to try and do more optimization, and added a Transposition table lookup/hit counter, and it turns out the TT hit rate is only 10.9% which is kinda low
:sob:
gonna try and improve that now

The chess engine now has a name!
also continued to work on UCI

Created a new version of magic number generators in C++ -- unlike C# like the engine, because it's faster -- to try and solve the problem of magics, especially for rooks, being realllllly slow. Well, exaggerated, but during profiling rook magic calls alone take 5% of the processing time which is way more than I thought, and most of it is only during quiesce, which is concerning, even though quiesce should have less nodes than the major branching by a lot, most of the magic call are from quiesce, might look into capture move generation logic later
Time:
3 hours from this from the magic gen, 2 and a half from the chess engine

started work on UCI support, decided I wanted to make this version UCI, and do some testing, also found out out of the blue that my pruning algorithm was backwards, since the depths I use for the PVS (negamax) uses reduction in depth, so each iterative depth it starts by calling it with depth(n) and when it gets to depth(0) it calls the quiesce, but for pruning those depth values aren't helpful, so now I include a ply counter variable, but adding it has had some consequences and has broken some things, which will need to be debugged

Also added in time tracked from other project files that was clones of the engine to help me debug it at points during development

fixed filtermoves, did an overhaul on pawn move generation because when profiled pawn moves were a bit slow, now uses bit tables in movegen. Started debugging on special move gen for the quiesce, because currently there are some bugs with it that sometimes makes white pawns go to 8th rank without promoting, and then try to generate a move further, made the engine print out all the fen it encounters in quiesce before the crash, at the depth i knew the crash was going to happen at, so i can scan thru some of them and try and figure out why

added some debug methods in a new class to help debug the new move filtering, fixed one issue where the king was walking into an area controlled by opposing sliders (bishops, rooks, queens) because the method for checking if the square is attacked was using magics for slider attacks, using the current board state, meaning that the king acted as a blocker, when it shouldn't have, now trying to figure out why checkmate doesn't render the whole move list empty like it should

After a week of burnout trying to debug the engine and some new features, a progress update:
fixed many bugs in the new filtermoves system, but there are still bugs because the node count for each depth is different when using the old play move, check legality, undo than the new system
Added a new incrementally updated material delta tracker after a few iterations of prototyping, to speed up material eval
lots of debugging in general, and getting close to 1.5M NPS

turns out, the magic number generator that I coded is faulty, and the numbers it outputted actually allowed illegal moves, such as Be3 on starting position, I never realized this until I came across an unrelated bricking bug a few days also as mentioned in the devlogs that started when I added the new move filtering, I have reverted to the old magic numbers, and will continue to work on debugging the bricking problem. I have made a copy of my last working code using List before to compare, and that's how I found out that it was the magic numbers giving wrong slider movegen. I will try to figure out why the new movefiltering bricks the whole engine though

debugging the new filtermoves system, still working on it, got a bit of work done because am on a road trip, so lotsa time to try and figure this out on my laptop
there's also some illegal move being generated which is weird

started working on a new move-filtering class, using bitboard-based filtering instead of simulating each possible move and deleting the ones that aren't legal.

some refactoring, first 1M NPS!

honestly kinda burnt out from trying to refactor to maximize nodes per second, not much progress today, hopefully will be a lot more progress tmr

refactoring and rewriting methods, as well as removing unnecessary features to improve nodes per second, went from 400k->900k today, very good progress

tuning tuning tuning... added some debug statements to tell me how effective the pruning is at doing their job, so depth is quite reduced for now. Will continue to work on tuning upgrades, maybe better move ordering, and then, focusing on squeezing out every ounce of raw nodes-per-second I can get, currently around 400-500kNPS which is ok but I would much rather be at least 1M NPS, which would allow for much better searching at high depths

FINALLY got the new system to work, and also made some optimizations, such as a complete overhaul to the flagCheckAndMate method that is run on all legal moves, as a significant portion of time is spent there before, FIRST DEPTH 15 SEARCH

Still debugging the new Span version, it's somehow pretty inefficient, and also, still not doing the same thing as before. Will continue to debug until it has the same behavior as before, but I think I'm almost there
Almost at 24 hour mark!

AGHHHHHHHHHHHH chess engine still broken from trying to refactor everything to use Span instead of List asjdfjsdfjsaldfjasl;dfjak nothing works anymoreeeeeeeeeeeee
To be fair the change is basically updating the architecture of the whole program... bugs are expected...
Current status: engine breaks after depth 3, and just bricks itself, there's something running, because it doesn't let me type, but it doesn't end whatever process it was doing... never ending...

Refactoring, refactoring, refactoring...
Struggling to refactor all of my method that uses List to use Span, which I've found out to be a lot more efficient as there's a not a large amount of repeated memory allocation, will continue and hopefully finish up tmr

Trying to refactor movegen to be more efficient, as for a bitboard engine, despite the progress I've made, it's still slow in nodes per second, currently got it up to 400k nodes per second, which is decent, but not good. Will continue refactoring for efficiency tmr.
Screenshot is of the newly refactored flagcheckandmate that improves efficiency

TUNING... A LOT OF TUNING.... Lots, and lots, and lots, and lots of tuning
mostly just tuning the pruning parameters, as well as the aspiration window size, and futility margin
AT LEAST IT SEARCHES TO DEPTH 13 PRETTY GOOD NOW WOOOOOOOOOOOOOOO
ALSO FIRST DEPTH 15!!!!!!!!!!111!!11!!!1!!1 (albeit a huge node explosion after 14, it appears that there's always going to be a node explosion at some point, just how long you can delay it, and it eventually might get to the point that too much actual accuracy will be sacrificed that it won't be worth pruning that much
There's a chance I could be sacrificing a lot of accuracy by pruning so much, will do more testing later

added some more advanced eval, such as king safety, pawn struct, and tuned and fiddled around with some pruning parameters, getting real good now, might do some live testing soon against stockfish and see how things go

added some phased piece evaluation, will be working on king safety and pawn structure and such later

tried to add some more features to make it go faster or more efficient, only slowed it down or broke the engine... enough coding for the day

Added evaluation output, including formatting for mate in x by either color

FIXED NULL MOVE PRUNING BUGS BOIIIIII and also added more features, like aspiration window, futility pruning, and late move reductions, it's now a lot faster, (by cutting a lot of positions searched)

trying to debug null move pruning, currently very broken, returns an invalid move after searching past depth 5

Did a bunch of refactoring, added a small method that allows for future multithreading support, currently working on null move pruning for more search efficiency