Heliodex

The site has been released to Mercury's beta testers! We now have maybe 25-30 accounts on site, all creating places, assets, forum & status posts, comments, groups etc, and discovering plenty of interesting bugs!

Avatars now have their images displayed correctly as well, as we have RCCService set up correctly, which the attached image shows some examples of in user avatars on a locally hosted version. This handles rendering of avatars, uploaded models, clothing accessories, and perhaps place thumbnails in the future as well. RCCService had some trouble running performantly on the site's VPS, so was moved to running on my own dedicated server, running under Wine on OpenSUSE. This was a pretty interesting process as well, as RCC requires a display server to run correctly, so can't run properly in a standard SSH environment. It ends up working really well and providing decently fast renders too! (I'm not sure if this is using hardware acceleration)

A small list of bugs to be fixed are as follows:
- Renders for shirt assets currently don't show the shirt correctly
- Creating randomised invite keys mistakenly sees the creation form as not fully submitted
- The result of the aforementioned form error (400) causes a new error (500) due to the form error function not working correctly
- Groups with a / (or other URL-unsafe characters) in their name get an invalid link as a result of the group name being in the URL (fixed, though not yet deployed)

The site was inaccessible for a short time due to issues with hosting. It is currently on a Windows VPS, and we're considering moving to the same dedicated server that RCCService is running on.

The demo site now includes a page on how programs are stored and the environment in which they are executed on Coputer. The page also covers some of the design choices behind parts of the Execution system and why they were made. A few optimisations and further tests have been added to the VM too.

At some stage I'd like to write a tutorial on how one could deploy their own programs to Coputer, though this is pretty far away at the moment given that there's barely any documentation (aside from autogenerated documentation from code comments), and that 3 separate executables (plus their Luau dependencies) are required to develop and deploy a program. I don't have any particular way in mind on how to simplify this given the currently fragmented nature of the system.

Another thing I'd like to improve in future is connections between Communication system. When setting up a Communication system it has no knowledge about its previous peers apart from those in its configuration file, so it adds them to its known peers and tries to contact them. Later, when a program execution is requested, if its Execution system does not have the program, it waits for a response from all of its peers to see if any of them have it. Since the network node core has no knowledge of connection state, it doesn't know when or if requests to its peers have timed out/will no longer retry, which means that the system will hang forever in the event of one of its peers failing to respond.

The simplest way to solve this will probably be to have some way of notifying the node of when it can't connect to its peers.

After bundling Coputer's code formatter into a VSCode extension, it seems to actually work! The only areas remaining to finish for this are things like variable type annotations, as the output from luau-ast is already typechecked, so the definitions it gives for each variable may or may not actually be there as an annotation. The extension also provides basic language support in other areas like syntax highlighting and comments, because these were fairly easy to add.

The extension executes the formatter binary whenever a document format is requested, so there's 3 options to make this work:
1. Distribute multiple binaries for different OSs, and show a configuration option in settings to select the binary path (current)
2. Download and install the correct binary for the user and manage it on the fly (this could be done alongside a setting like in option 1 to override it if necessary)
3. Compile the formatter to WebAssembly to allow it to work in browser VSCode (possible performance hit, and luau-ast would have to be built from source or replaced with a custom parser)
..or some combination of all 3.

These Luau AST experiments have been pretty fun, I'll probably merge ast into the main branch of the repository soon.

More asset-related improvements have been made, including assets now loading from Roblox's API, via Open Cloud, from Mercury user-uploaded assets, and from administrator-managed privileged assets.

The attached image shows the code changed for serving assets. We had a few issues with this, with a few particular types of asset: scripts, meshes, images, and XML.
- Script assets are already signed at compile time, so don't need to be modified, and meshes don't need to be either. These are text formats, so will be fine if they need to be re-encoded.
- XML assets need to have Roblox URLs in them replaced with the Mercury equivalents.
- Images must go through unchanged, as re-encoding a binary image format as text (UTF-8) will corrupt it.
Basically this all means we have to guess what type the asset is when loading it. If it begins with <roblox, it's probably XML, and this works well enough for our use cases.
Loadscripts are now instead signed at runtime, as parts of the script need to be replaced on-the-fly (for example, the API URLs that a hosted game server needs to access, or a user's username when joining a game).

The change of place and asset IDs to integers instead of strings caused some unforseen complications, and areas of code for loading icons/thumbnails/loadscripts etc all had to be changed to account for this, as well as the Economy service. All of these changes are still on Mercury Core's assets branch for the time being.

A number of issues with loading corescripts were encountered, so we spent a lot of time designing potential new solutions for loading them, even discussing bundling them all into the game client. However, we eventually found the issue was due to bad signatures at the top of each file: they must sign the entire file, but also including the file's asset ID. Fixing this, all corescripts now load correctly into the client, allowing us to get the first real hosted games running for the upcoming Mercury 3 beta release!

A number of improvements have been made to the way assets are loaded on Mercury. Firstly, assets loaded with hardcoded IDs, IDs entirely controlled by scripts served to the client, and these assets' dependencies (all of which we're calling privileged assets) will now be archived alongside the corescripts in the tp-link-extender/2013 repository, instead of needing to be uploaded through the site, which incurs complications with the database and is tedious whenever it needs to be reset.

The Luau script for compiling the corescripts has been rewritten as a Go tool for bundling these alongside libraries, loadscripts, and archived privileged assets. Signing of corescripts and loadscripts is now done at compile time instead of when they're served, simplifying site code and reducing load.

Modifications have also been made to the client's DLL hook, allowing assets to be correctly loaded from the endpoint at xtcy.dev. In the near future we'll need to be able to back up and handle more categories of assets, including those loaded and cached from Roblox's API, so we're still mapping out potential solutions for these.

On the repository's ast branch, I'm working on a basic code formatter for Luau, to aid development of Coputer programs. This system is now nearing completion, though still has some fairly major edge cases to be fixed. Mainly these are issues with indentation, niche features of Luau with strange syntax (generic type packs, function<T, U...>(), for example), or numbers (especially Infinity and -0).

Another bug I recently encountered is with Unicode string handling – the luau-ast tool often gives back responses as strangely-formatted JSON5, and non-ASCII or control characters used in string literals are jumbled into an odd escape format. I've been working on un-jumbling these so that they can be better formatted back into source code, and I'm once again reminded of the difficulty of converting between different Unicode formats outside of the ability of the language's standard library/built-in types.

2 large pull requests have been merged into Mercury Core: improved service process management (#400, https://github.com/tp-link-extender/MercuryCore/issues/400), and better runtime type checking (#401, https://github.com/tp-link-extender/MercuryCore/pull/401).

The first pull request improves the ease of use for users to start up the Database and Economy services by spawning them directly and as needed when the site is running. This means there's no longer any need to use complex and inconsistent container managers or cumbersome management of terminals for each service. I tried out several possible solutions to this problem and this one seemed the lowest friction and most ergonomic.

The second pull request fixes runtime schema validation errors for forms across the site, as well as Typescript type errors caused by trying to get around these errors. I ended up replacing the Zod library with ArkType for validation, as it reduces some incompatibilities with other dependencies.

To add more details on how the Coputer network works, I've added some diagrams to the System page of the demonstration site. At the moment, assets like these images are stored outside of the network and loaded from the internet (these diagrams are just from GitHub's CDN). It would be possible but cumbersome to generate these from a route in the program, though this solution doesn't scale – if the file was some binary data, it would have to be embedded into a Luau script.

In future I'd like to have a better system allowing for embedding files and having them stored alongside a program's code. The code is already distributed in a bundled format, which is effectively a concatenation of multiple gzipped files, so adding them to this shouldn't be much of an issue. More difficult is the question of how to serve them to a user, eg. accessible through a specific path in a request, available with parameters alongside the program type, or returned through a function added to the API/standard library.

In further experiments on the ast branch of the repository, I've implemented a system to turn the AST back into source code. This means that after parsing Luau source code with luau-ast and organising it into a tree of nodes, we can rebuild functionally identical source code from it.

This might seem pointless at first, but it means we can do some optimisations on the code while turning it back into text – this is essentially how code formatters work. I've previously built a Luau code formatter, but it wasn't as robust as it could be if built on top of a system like this, so that's another area of the system I could work towards to make development for Coputer easier.

Some optimisations have been made to the Coputer VM – I'm starting with low-level functions like vector mathematics and bit32 operations because they're the smallest and easiest to test. In some cases, like floating-point operations, my custom functions were actually a lot slower than standard library ones!

I'll keep benchmarking higher-level functions until we can actually make an impact on performance, as at the moment Coputer's main VM is around 10x slower than the Luau reference implementation for most operations. The main reason the system is still fast enough is because of aggressive caching and memoisation on function results and arguments wherever possible, though better VM performance is always nice.

More work has been done on the upcoming documentation site for Mercury. This will encompass more in-depth setup guides than currently in the README, installation guides for tools needed for development and production, and common usage patterns in integration with larger projects.
The documentation site is built with Astro + Starlight. As I had previously used Astro I found the documentation to be pleasant and easy to build, and trivial to style with the Mercury Core colour scheme and design. Implementing the documentation directly into Mercury Core was also considered, though this would mean that users would have to set up the system correctly before being able to see the guides on how to set it up.

The first Coputer webpage now features some information about how the network works, detailing the path taken when a program is requested to be run by the network.
Some program bundling bugs that were uncovered while updating the site have also been fixed, and the public gateway at https://coputer.heliodex.cf has been updated to list programs for each user correctly. I'm actually pretty surprised the system has been running for a few weeks now without crashing.

Alongside the system for working with the output of luau-ast, the ast branch now also contains a working Luau lexer, passing all test cases of the reference implementation, and a work-in-progress parser. The parser seems like it will be the more difficult side of the system to implement, though I'll be able to reuse the existing AST node structure I was testing previously.

The ast branch of the repository contains an experimental system for working with the syntax of the Luau language used to write Coputer programs, parsing the JSON output of the luau-ast tool into a typed and more easily workable hierarchy. A set of tests are also provided alongside the system with inputs and expected outputs.

Hopefully in future this means we'll be able to provide advanced tooling for Luau, specially tailored to use in Coputer, to improve development experience.

More tests have been added to the VM, which is the thing that actually runs Coputer programs. Test coverage has increased from 91.4% to 92.7% if that's a metric worth caring about, and bugs with string pattern matching and table sorting uncovered while testing have been fixed.

Coputer serves its first ever program as a webpage!

Its URL is https://first-h2lk2orlh-cctfxch0x-5yd7n93mg-zxmn64hy4-wfe6s2l9z.coputer.heliodex.cf. The domain might initially look strange: the part before the first - is the program name, then the rest is my public key so you can tell I published the program.

The program is accessed through a 'gateway' hosted at coputer.heliodex.cf, which asks the network for a program and returns it as a web response. I will go into more detail on how this process works later.