u-crawler

I added linux arm support to the prebuilt pyinstaller binaries so u-crawler can run on my raspberry pi :D

I also had to set the github actions runner to run bookworm instead of the latest debian version. Otherwise, older versions of debian distros wouldn’t work (see the attached image) as they are packaged with an older version of glibc (the c library that pyinstaller uses)

I fixed some errors that might have caused issues on other devices. For example, I changed the selenium headless flag to the older one that still works, disabled disable gpu and remade the spec file to be more efficient and only include modules that I actually need. The attached screenshot of it working on my device, and I also tested it on 2 other laptops so it should work now

I added github actions support so i can build using pyinstaller for MacOS, Windows and Linux!

I added pyinstaller support which builds u-crawler into an exe that is easy to run, with no need to install dependencies or to even have python!

Finished readme

I added error catching and logging in programs.py. Now I just need to write a readme

I implemented error catching and logging in categories.py

I implemented a robots.txt checker to make sure that it doesn't scrape forbidden pages

Now my scraper uses the results of categories.json to crawl every program inside of that category. Instead of simply using requests and beautiful soup like I did for the category scraper, I had to use selenium to launch a headless browser, because the data for the programs is rendered with JavaScript. In the the screenshot below, you can see the format of the results with categories.json and the programs in each category as their own file and with some of the code on the right.

I finished the code that gets all of the areas of interest on the home page and puts it in a json file