Convincing some frc people to learn rust through this library. They hate using Rust because of the borrow checker but they said that xrp-rs isnt that bad

Finally got embedding search working and running nomic-embed-text:v1.5. I will disable this feature for now because my current host (hackclub's nest program) doesn’t have enough performance to be viable.

Spent a few days refactoring and cleaning up the codebase. I was originally embarrassed to upload this to GitHub, but I think its presentable enough. There are a couple of miscellaneous additions and optimizations. Also, the vibecoded stuff is nearly gone, and the codebase is approaching nearly 2k lines of code

For a while I have been improving the network protocol documentation, and I think its mostly finished now. There may still be some very minor errors, but I am very happy where it is now and there are not many things I can improve. If someone wanted to port it into a more popular language like python, just the readme is enough.

stayed up till 12am doing more data analysis. today (technically yesterday), I found every project that is leaking secrets in their .env file. Surprisingly, only about 0.5% of projects leak secrets there. Its mostly database keys but I found som cool ones like a gemini key and an aws key. tomorrow, I will properly report it to staff because everyone is sleeping rn

today I did some analysis on the current dataset of projects and found the most popular websites that people put as their repo link. to nobody's surprise, github is by far the most popular, then codeberg, then gitlab, and finally a bunch of self hosters.

Spent a while tuning the core algorithms, it's still really simple right now because I don’t have all the data. Next week I will do another scrape to go from 6k to 9k projects and get a lot more devlogs. Ranking right now is hard because theres almost data enough to do better algorithms. I will try my best to make it hard for people to beat the future algorithms and rank high even if the project is mid.

Now links to searches can be shared! It was hard to link someone the results of a search, now the query is embedded in the url

Went away for a robotics competition and was expecting to see the website down. I guess the nest team managed to scale up the servers and fix a lot of the issues. Huge props to them!

Wow! Not even a day later and nest is broken again. Eventually I need to switch to a proper host for this because this is getting annoying. A $5/month plan should be more than sufficient; heck, maybe I'll try one of those serverless things.

nest has frequent downtime, so I made a setup script that will hopefully start the website automatically

Got it hosted! Try it out at https://searxing.hackclub.app/ it’s fairly fast and contains over 6k projects. There are still some improvements I can make to the website, but it's nearly complete. Searching is pretty good, and feel free to use the results as inspiration for your own projects.

Doing just a little bit of scraping. Also, I made a custom library to help me with parsing the HTML schema, and after a bit of debugging, it is surprisingly good. Spent some time preventing other assets from loading to not overwhelm Hack Club's servers. They might get mad if I used up a few gigabytes of bandwidth. It took only like 1h 30m to scrape all projects, and it fits in a very small JSON file. Now I need to learn how to write a good search engine and host the website. Yay progress!

Spent way too long deciding on what type of database to use, so I ended up trying a lot of them. Instead of doing the correct thing and watching a YouTube video, reading the docs, or even reading a blog post, I decided to blindly try ones that I have heard of, even if they didn't match my use case. In the end, I ended up with terrible implementations of all of them, like an SQL DB that copies everything to Python for ranking, a Redis DB that is somehow relational, etc., etc. Anyways, instead of actually reading about which to use, I will write my own in Rust that will exactly match my use case. Remember, coding for 6 hours can save you 30 minutes of reading documentation.

The robotics team needs the robot for an outreach event, so I cannot try my code on the actual hardware for a while. Luckily, I recorded the communication with Wireshark a while back, so I can still continue testing some parts of my code. Right now, I tested everything that I can, and I need the robot for the rest. Hopefully I can get it back soon.

Fixed all bugs with packets and network tables. Here is a quick demo of a simple pid loop in AdvantageScope.

Very clean web scraper API. I might also use this to yoink some projects from other hackclub run sites. Figuring out how to get devlogs took some work, but all good now.

Searching mostly works, and the scraper got all the projects into one large JSON. I need to figure out how to implement some kind of ranking algorithm and what to do about the json size being 90% base64 images

Did an initial trial with FRC team logos; it's very slow, and some are miscategorized images. This is just an initial trial run to see how well it turns out, and I am very impressed with the results.

Im going to use ai only on the frontend because I suck at webdev and I want this tool to actually be useful. Projects are manually copied from journey.hackclub.com and I was heavily inspired by their style. Currently nothing is interactive but give me a week and I can make this something amazing.

Removed most of the slop that came with trying to ship this quickly for #journey. Still the same robot, but now the library is mostly done, and I can actually spend time making the robot move with fancy algorithms. I also broke it with the fall