Hey! - Crawler
Hey! is a small search engine I'm making.
This is its web crawler.
Followers
No followers yet
Ship Your Project
Get ready!
Once you ship this you can't edit the description of the project, but you'll be able to add more devlogs and re-ship it as you add new features!
Timeline
Heyo! I made the crawler a LOT faster. The downside is, it's using ALL my CPU, and it's slowing down because my CPU's lagging. I've tried running it on Github Actions, but the file was WAY to large. Also, it's so large my screen recording feature doesn't work ;-;
I might have to find a way around this bug. I ordered a Raspberry Pi off SOM, so that should fix the issue.
Bob
about 2 months ago
Alright, after working a bit more on the crawler, I have an idea (google pls hire me)
So, basically, my crawler now outputs information like this. It has a lot of whitespaces, which means that the data would require a bit more Kilobytes. however, not only does this easily seperate the text in the URL, it also seperates text that could be searched in different ways, much more easily (eg. GooglePlay and Google Play and accidentally typing G oogle P lay are the same thing).
