SoulSearch Used AI
A high-performance search engine built from scratch in pure Go. Features web crawling, custom indexing, PageRank algorithm, and a modern web interface.
now working, though search results need to be polished as well as the need to remove the html from it
Followers
ari
Check their projects out: Musicnerds, Black-Scholes Option Pricer, Be A Barista!
Ship Your Project
Get ready!
Once you ship this you can't edit the description of the project, but you'll be able to add more devlogs and re-ship it as you add new features!
Timeline
hooked up the frontend and backend (i also made it so the backend serves the frontend) im gonna ship it now wish me luck!
trying to build upto pagerank alg and started scoring mechanics (will be pushed soon!) i added politness to my crawler too since it was sort of rude by not heeding to robot.txt. Can't wait to finish current stuff to implement new stuff but also feels sad not working on it more.
I built a multi-threaded web crawler in Go that:
Recursively crawls web pages
Extracts all links
Avoids re-visiting URLs
Saves progress to disk (results.json)
Can resume from where it left off
Gracefully handles shutdown (Ctrl+C)
Periodically checkpoints progress
has some erros that are still being resolved
Frontend is Finished!
Web Crawler: Multi-threaded crawler with polite crawling delays
Custom Indexer: Term frequency analysis with stop word filtering
PageRank Algorithm: Link-based authority scoring
Advanced Scoring: Combines TF-IDF, PageRank, and position-based scoring
Modern Web UI: Responsive interface with real-time search
REST API: JSON API for programmatic access
Pure Go: No external dependencies, built with standard library only
