Summer of Making Monitor

Summer of Making Monitor

11 devlogs
9h 55m
•  Ship certified
Created by mahad

A shop monitor for the Summer of Making shop!

Timeline

Ship 3

1 payout of shell 63.0 shells

mahad

7 months ago

mahad Covers 2 devlogs and 2h 34m
Earned sticker

I added Stickerlode scraping! The bot will now detect new stickers every day. Also cleaned up some code to make it even easier to add scrapers in the future.

Update attachment

I migrated the monitor to a new DOM parsing library called Linkedom. Here's my postmortem:
- SOM Monitor uses a library called Happy DOM to parse the pages
- The latest version of Happy DOM does not work on the latest version of Bun (see capricorn86/happy-dom#1762)
- So commit 43e1aeb downgraded it to 17.1.8 from 18.0.1, which doesn't have this issue
- However, 17.1.8 has a memory leak

This change has been great. At initial launch, the LinkeDOM version uses ~200MB less RAM than the happy-dom version. And unlike the happy-dom library, LinkeDOM doesn't suffer from memory leak bugs - the old version used to use up to 20GB (!!!) of RAM after just a few hours.

Update attachment

Ship 2

1 payout of shell 39.0 shells

mahad

8 months ago

mahad Covers 1 devlog and 1h 51m

So many cool updates!
- Now scrapes the Black Market
- Made the bot way more stable and has a lot less bugs
- Switched over to @channel pings
- The bot is now really modular, so updating scrapers is a lot easier
- New API! https://summer.skyfall.dev/api/shop

Update attachment

Ship 1

1 payout of shell 57.0 shells

mahad

10 months ago

mahad Covers 8 devlogs and 5h 28m

Fixed regions (now it shows prices for all countries!)

Update attachment

Fixed bugs, TS config. The shop code updated overnight and I fixed the bot so it can read prices properly.

Update attachment
  • Handles limited items (e.g. the Amp credits)
  • Ignores whitespace changes (the updates for these are pretty useless since no one wants to be pinged because someone removed a space from an item description)
  • Make the bot run every minute for faster alerts
  • Make the checks on the scraped data tighter in order to not return bad data
  • Made a sweet banner!
Update attachment

Made notification summaries in the Activity tab more helpful (thanks @Jeremy!)

(They used to just say @shop-watchers for every single update :pf:)

Update attachment

Cleaned up code, started working on improving channel ping notification messages in your Activity tab.

Update attachment

Observability is important. Slack said that my API requests were incorrect, and @GGORG mentioned how it would only send a maximum of one update, even when there were more than one!

To fix this, I added Sentry (https://sentry.io) to my project and now log all Slack API requests and items. The end result is that in case the SOM devs change up the shop code (the bot uses HTML scraping), I'll be notified and can fix the bot.

Update attachment
  • Added usergroup pings

  • Fixed a bug where more than one update wouldn't be posted at a time

Update attachment

Added update/new item alerts!

Update attachment