faucet - a c++ http server

I lied, I made another devlog, and another change (final this time fr).
I was checking to make sure the folders inside folders worked correctly in dir listing, and found out it was missing the empty table cells which basically made the dir listing look ugly since the line that would usually be there was not there. I fixed that, and then thought I might as well add the custom icons as well. I got the icons (as always) from lucide.dev, and made a helper function to return the correct class name for the file type. Right now it has film (mp4, mkv, etc), image, textfile, codexml (html, css, other code files), music and activity (.log) icons. That's pretty much it, I think it looks really cool, and it took a lot less time than I expected (most of the time was spent just looking up the icons and putting the svgs in the code), so I hope this is actually the final devlog before the first ship.

Final devlog for the first ship most likely. Added support for the Range header, and added images to the example site. For the example, I just added a few example images copied over from the previous devlogs, nothing special. For the range/content-range support, I had to add those because before, the server would just send the whole video file, and that would take a while to load, and once it did load you could not scroll through the video, you could only play it start to end. With the range header support, it sends the video data partially as needed as the browser requests via the Range header, and video/audio/etc scrolling seems to work fine now. I think pretty much everything is done now, and ready to be shipped, as there doesn't seem to be any false positives with the trust score system, server doesn't just crash randomly, everything is good. I did have in mind adding more icons for the dir listing (such as a video icon for mp4/webm, audio symbol for mp3/wav/ogg/etc, and other icons, but I'll leave that for another time.

Readme changes. Added badges (cause badges are the 2nd coolest thing next to <br>), changed some wording, added the new missing features, updated the .env example section, updated env variables section which describes what each variable does, and also added info on the trust score system, with basic info such as factors, honepot paths, and threshold. Readme should be good to go now, looks like it renders fine on the HTML version as well thanks to the marked js renderer. I think basically everything's done for now, could be ready for the first ship now, but I still want to let the server run for a while and do stress testing and such to confirm it doesn't just collapse first.

Made a new banner and logo. For the logo, I just searched for faucet on flaticon, copied the png, added a gradient and that's it. For the banner, I went with a similar style as the original purple banner, except with the new font. Nothing much to explain but I definitely think I did a better job on the new banner, and example site compared to the original purple design.

Worked quite a bit on improving the example page. I decided to get rid of the ugly purple, and replace it with blue instead. Added <hr>'s in between h2's and the p's in sections cause <hr> is the coolest element ever. Improved wording in some sections, updated info, and added the trust score section. Also added a few more links to the hero section, including a link to the HTML readme view, which gets the raw readme from github, and renders it using marked, though the view is not as good as it would be on github. I still have to make a new banner, and also probably a logo as well since I need a favicon.ico for the example page.

Styling update done, and percent-decoding added. Updated the default error page handler to use the new theme, same as in the dir listing. I fixed the 404 error for files with spaces in the name by adding percent-decoding. Before, e.g. files/test%20filename.txt would be looked up literally on the server, but now with percent decoding, it accounts for the spaces and looks up the file correctly pretty much. For the next changes, one idea I had was to add a custom player/page for certain files, e.g. a audio player html page for mp3 files and a video player for mp4 files, but I decided that wouldn't be the best idea since those should be just sent as is with the correct content type instead of sending it as a html page, so now I pretty much have no good ideas. Considering doing the first ship at this point, but I just want to make sure everything is clean and polished, I definitely still have to update the readme since it's outdated and missing some info, and I'll probably update the banner and the example page as well since looking at it now I did not go with the best color choice, might keep the purple but blue might look better. I'd also have to update the example page with more info, since rn it's pretty lacking and could use more detail, and I'll probably add another page which just gets the readme from github, and uses a js markdown renderer (marked maybe) to render the contents, instead of manually updating it on each readme change. But overall it seems like almost everything seems to be ready for the first ship, so that's it.

Started work on improving the overall quality of the default/server pages (dir listing and error pages). For now, I just improved the dir listing styling. I decided to go with a darker theme overall instead of the pure white background, improved the table styling to seperate the table headers and entries, added icons to indicate home/folders/files, thought about adding more content/extension specific icons but I'll probably do that later, went with a more modern font (segoe ui by default, and other fallback fonts if cant use that), and that's pretty much it. Next thing I'll work on is fixing the 404 for files that have a space in the name for now.

Simple changes to the trust score system again, and pretty much finalizing it. I started off by adding a check for if the user agent is unknown (not in trusted/suspicious user agents list), and if so, reduces trust score by 10. The next thing I did was adjusting the RPM trust score since I felt it was somewhat too lenient but I might be doing overkill, but for now that isn't a problem. Also added a honeypot hits per 3 minutes system, which lowers the trust score of the client based on how many honeypot urls/paths they've tried to access within the last 3 minutes, e.g. if >= 2, lower by 10, if >= 7, lower by 65 cause no regular human tries to access these paths (considering they do not exist). After that I made a few changes to the header checks, starting off by adding a Accept-Encoding header check, if it exists AND is legitimate (includes gzip and such), it raises the score by 5, and opposite for if it's not in the headers. Also improved on the sec-ch-ua-platform header check, before it would just check if that header exists and thats it, now it checks if the string in the header is in the legitimate platforms list, and only raises score then pretty much. Adjusted the 404pm scores to be more lenient since I don't want to false-positive regular users. And finally merged the trustscore branch into main, since it's pretty much functional now, and seems to work fine from testing. Tiniest change ever, but added git switch main to the nest.yml github action cause why not. The next things I plan to work on now then is improving the styling of the error pages and dir listing, cause I'd rather have it look modern than another regular boring http server, and I might also have to fix files with spaces in the name not working properly in the dir listing, since if you try to access those files, despite being listed, it'll just return a 404.

Small update, I let the server run for a while and I managed to catch a considerably malicious request series. The client was using python-httpx which was not in the malicious user agents list so I'll have to add that, and was going through /.git/, to /phpinfo, only being blocked at 5 trust score after 13 requests (which all returned 404 until the 13th request), while starting from 30 trust score. So with that, I had a few ideas. The first idea is to add a honeypots per minute system along with the 404's per minute tracker, e.g. if the client tries to access 4 or more honeypot links per minute (may make it per 3 minutes or such instead of just a minute), it'll penalise the trust score just like the 404 system, except it being harsher. Second idea is to potentially reduce the amount of 404's per minute required by the client to get a trust score drop, since 20 404's might be too lenient. And the third idea is adding a IP range/reputation check, since AbuseIPDB reports the ip as 100% malicious with 2,296 reports. That's pretty much it, I'll try to look into more ways to improve the trust score system since that's my main focus right now, and I want it to be as accurate as possible while not flagging legitimate requests. Also made a small change to the logic before, suspiciousAgents and trustedAgents no longer get double counted (if the useragent is something like curl wget or mozilla chrome for example), if a suspicious agent is detected, it adds -10 to the score, and no other findings are counted including from trustedagents, and if a trusted agent is found, no other findings are counted as well. And for the final idea, I should probably also add a deduction for unknown user agents, since right now if the user agent is not in the sus/trusted agents list, no trust score change is made, which is not the best.

Decided to remove the cache system in the trust score evaluation because it was pretty much unreliable, probably was slower for performance, and unnecessary if i think about it now. I instead replaced it with a lowest score per minute system, which is self-explanatory, say request 1 has a score of 50, request 2 has a score of 60, for request 2 the score used will be 50 since that was the lowest score in the last minute. I also added a honeypotPaths.txt file, which you can use to add your own honeypot paths, if enabled. If one is not found, the server will just use the default hardcoded honeypot paths. Also forgot to include a image of the 403(1) trust score block page in the previous devlog, so adding i'm adding it here.

Trust score evaluation is pretty much functional now, using the factors as described in the previous devlog (headers, 404 per minute count, honeypot paths, etc). Added a few basic trust score prints for now which shows the trust score per request, so I can gather data and make changes based on that via the site hosted on Nest. The workflow pretty much looks like this: Client makes request -> main.cpp checks if client in block list, if not requests trust score -> evaluateTrust.cpp evaluates trust, if client in cache it gets the previous trust score from there if the user-agent is same, and also before sending cached result, it check the 404 per minute count, which is not checked on the first uncached request cause ofc it would be only one 404 logged pretty much -> trust score is sent to main.cpp, if trust score under threshold, add to blocked clients list for user set duration, if not, continue with other checks such as ratelimit and 404 and etc. Implementing this was lowkey gonna tweak me out since the code is spaghetti at this point imo, still manageable though. Anyways, I'm gonna let the trustscore branch run on the nest server for a while, and see if everything looks ok or needs adjustment.

Been working on the evaluateTrust functionality for a while now, still not complete but thought I'd make a devlog anyways. So far I've just made the base functionality but it's not implemented anywhere. By default it checks for headers such as user-agent and sec-ch-ua-platform or if it's empty, checks how many requests per minute the client has done, and user toggleable honeypot paths such as /wp-login.php (might have to make a config file for this cause rn the honeypot paths are hardcoded), and the next thing I'm working on is checking 404's per minute, and adding that to the trust score evaluation as well. It also stores the trust score in the cache for user-set amount of time, but if the user-agent differs from the one in cache, the trust score will be re-evaluated. Tbh idk if caching or not would have a impact on performance but whatever. Pretty much good progress so far, but still needs to be implemented in the right places.

Got faucet working properly on nest, the issue was that I was using reverse_proxy in the Caddyfile, since I assumed that's required to redirect all traffic from the domain to the server, but I just had to run nest caddy add with the --proxy localhost:port argument. IPs are now correct and don't just show localhost. I'm thinking about adding a trust score factor as well to the server as a toggleable option, based on things such as headers, ip range, requests per second, maybe if the client tried to access robots.txt or such, etc. But overall, I think the server is basically ready for first ship but I'd rather polish up stuff and make sure everything looks nice rather than ship it fast but unpolished. Also considering a styling redo for the default error pages, and file listing, similar to the style of Apaxy possibly.

I had to add support for X-Real-IP/X-Forwarded-For since ratelimiting would work incorrectly without that, and also added toggling using that via TRUST_XREALIP via .env, as I'm hosting the example site on Nest, with Caddy as the proxy. The code seems to work fine, but it's a real pain, since for some reason, either Caddy or the code is wrong, and the headers aren't getting correctly added, so Faucet just sees all the requests origins as 127.0.0.1/localhost. I'm confident the issue is with Caddy so I'll try to figure it out cause this is just tiring.

Made a proper readme, added a MIT license, and made a banner. The readme should cover all the important information, such as basic features, installation/setup using the github releases binary (which i'll upload/setup later) or building it yourself, env variables info, and that's pretty much it. Added the MIT license since it's the most straightforward license, I considered GPLv3 but MIT seemed better for this. I made the banner in paint.net in like 15 minutes, I just took a screenshot of the site, added a radial blur, added random lines and just turned that into a road somewhat, shadow outline for the description text, and that's pretty much it, just tried doing random stuff until I came up with the final banner. I'll probably add a few basic badges to the readme as well later, but right now I want to get the server setup on Nest, and have github actions compile the binary on new commits.

got mp4's/large file transfers working, the issue was with SIGPIPE. Pretty much if the peer has closed the connection and you try to write, linux will raise a SIGPIPE, which by default (I think) terminates the process. Ignoring SIGPIPE fixes the issue, and mp4's/large files now load fine on browsers.

Finalizing things for the first ship. I decided to get rid of the old example page, and completely redo it with relevant info and links, and also renamed the example public folder to example/, along with a warning in loadConfig if the folder being served is named example, and added webp to contentTypes.h. Also added example files to the example directory listing. I'll continue to work on the readme next (was making a banner before, took a screenshot of the original example site but it looked ugly so that's why I redid the example page), and also might have to look into if it's a WSL issue or code issue when trying to load/view a mp4 file. But most of the things work fine and well, so that's great.

I probably spent too much time on this, but I added a header manager middleware. Right now it just validates the headers, and if a inconsistency is found it'll return invalid, and then the server can resort to a fallback basic header. It also adds a few basic headers, that being Server: faucet/1.0, X-Content-Type-Options: nosniff, and Referrer-Policy: no-referrer. I decided to make the headermanager since I might add more config options for headers later on for security, and it just helps with validation and not repeating header parts that will be included always, and flexibility for the future.

Worked mostly on improving the console output QoL. Started off by improving the overall style/quality of outputs and by adding HTTP version, method, path and user-agent info to the current request log, and also made it more concise. After that I worked on adding console logging to file, which I did by making logRequest.cpp, and just formatting the strings in main and passing it over there along with config data, and then logRequest handles output to console and logging to file if enabled, along with rotation if LOGMAXFILES is exceeded, to .log.bak.

Added a very basic HTTP auth option & check. You simply just uncomment and setup the user:pass in the .env file, and it'll return 401 to unauthenticated users. HTTP Basic Auth expects base64, so the server handles that by converting the env creds to base64, and checking the incoming requests authorization: field for the base64 creds. If the check fails, the server returns the 401 page.

I decided to add a very basic page for errors+404, instead of currently just returning the error in the header. It's pretty much straightforward, say you get a 404, 429 or a 418, it'll send back the basic page, and contact email/info if the config has one set, along with a link to return to root/home. If a custom 404 page is set, it'll take priority of course, but if one is not set/could not be opened, it'll return the basic page. Anyways for the next changes, I don't really have anything in mind right now, I might go for adding HTTP authentication support with WWW-Authenticate, but I'd also have to probably refactor/add middleware for how responses are sent, or there would just be a lot of repetition and that'd be a headache later.

A few changes, I might ship the project soon since I'm running out of ideas to add that I could realistically do, but I'll look into that later. For the dir listing, I added file sizes, and last modified dates to the table. For DRY, I decided to move the content guessing types to a header file since it was being repeated in the main.cpp file, and also just better for maintaining, added a few more types as well. And finally, added a pretty basic requests/second ratelimit, per IP. loadConfig defaults it to 10 requests/second, customizable via .env of course, and it'll return 429 to any IPs that make more requests than that. Also fixed a issue, if a client closes a connection early, the server would crash since the connection was not closed safely pretty much, this is now fixed.

Forgot to make a devlog on this earlier, but I got directory listing implemented and working. Right now it's pretty basic and simply displays the files/folders inside the folder, but I plan on adding file sizes, last modified dates, and basic svg icons, and overall just polishing the styling.

Worked on a few things before getting started on directory listing since I'd rather deal with those now than later.
Added custom document root dir option to .env so you can choose a different folder other than the default public/.
Added a custom 404 page option, if one provided it'll return 404 with the given html page, if not, it'll return a plain 404 error.
Also added timestamps to the request logs.
Hoping to get started on directory listing next, maybe I'll work on some other stuff before though.

Small changes, I decided to only move the config loading logic to a separate file for now, as I don't see a need to split anything else (and also moved source code to src/ + added makefile), and moved site root dir to public/ by default, planning on adding a config option for changing this though. I might also work on directory listing next, something similar to what Apache does.

The pages load properly now on firefox, adding proper headers with content-type, content-length, and etc fixed the problem.

Added looping accepts so the server doesn't exit after one request.
Added args to the main loop, right now the only accepted arg is --port for choosing another port other than the default 8080, and a .env loader as well, which also can be used to set the port, but can be overriden via args.
Added graceful exits via signals, so when you press ctrl-c, it'll exit gracefully, and fixed the ip shown in the request received message along with that, and finally added basic content types for js, css, etc so browsers can properly load these.

I'll probably move the site root files to a public/ folder instead of those being in the root of the project as the next change, and I might as well consider splitting the code into multiple files for readability.

I got the server to work with curl somewhat for now, but attempting to view the page via a browser will just return The connection was reset, so that's something I'll have to work on. Also added basic error checks, but overall this still needs a lot of work to be functional.

I decided to challenge myself and also learn c++ by making a http server. I originally had the idea of what if I made the xbox 360 run a http server, but since I don't have access to a windows pc or visual studio 2010 ultimate right now, I just decided to make a regular http server. This will also be my first c/c++ project despite using c before.