som-data Used AI
Scrape and visualize Summer of Making data in an easy-to-use web app. This app can list projects and users and allows sorting and filtering by different criteria.
Ship Your Project
Get ready!
Once you ship this you can't edit the description of the project, but you'll be able to add more devlogs and re-ship it as you add new features!
Timeline
I completely revamped the UI and fixed some bugs to make the user experience better. Also, I completely rewrote the scraper to be more reliable and scrape more data such as user bios and profile pictures. Finally, I changed the font from Roboto to Montserrat, which made the UI look much more appealing.
I refactored the backend to make the API much easier to consume. Rather than sending the entire 2.5MB database, clients can search for certain data and can receive up to 200 results. Also, the project now also leverages the Summer of Hacking API to allow searching by Slack user URLs.
I spent a lot of time tinkering with raw SQL to get it to work just right. I learned that when developing a moderate sized project, you should probably try to do things right from the start or you'll spend lots of hours cleaning up.
This project uses Node.js, Express, and Prisma ORM for the backend and plain HTML/JS/CSS for the frontend.
I created and deployed the web app after almost 6 hours of straight development time. Most of the time was spent developing the frontend UI and the scraper itself. Currently, the following features are supported:
- Request an update (re-scrape for available projects)
- Search for projects. Sort by minutes, devlog count, or random. Filter by author name or title
- Search for users. Sort by minutes, devlog count, or random. Filter by name
Scrapes are infrequent and the app does not scrape a page twice to avoid causing unnecessary load to the HackClub servers. For this reason, users, projects, and thumbnails are cached locally on the server.
Currently, all filtering and sorting is done on the frontend, so the entire ~3 MB scraped data collection is sent to the browser. I will fix this in a later update to increase the efficiency of the frontend.
AI was only used for problem-solving. All code and assets were created by me.
