Devlog: Modules are implemented! ...So now what?

     So over the last week, I finished implementing the module things, at least this rough version of them. I read up on classloading and all that magic, and I'm fairly sure that the stuff works. I'm going to need to write some sort of tool to generate NBT fires (maybe using maven?) and then write up a simple enough API to have this hacky module shown working. I think I'm going to do that next week, because this week has me in a spot I didn't think I would hit this early into development. The main reason why I put so much work into the module system is because it's the main feature of almost every other feature the RenderEngine will have, and the biggest feature that a lot of other things rely on is the rendering components. And the big thing that I really wanted to push is the modularity of the RenderEngine, including which rendering backend to use. And that's where the issue lies:

    The two main rendering backends I want to support, OpenGL and Vulkan, have vastly different workflows.

    In order to support both, I need to come up with an API that is flexible enough to make it not a complete dumpster fire to work with. OpenGL is easy, it's just a large state machine that has decades of documentation and tutorials and all kinds of everything written for it. Vulkan is the opposite, with it being fairly new and most definitely not a state machine. Their APIs aren't even remotely similar (in most cases at least), so to have me support both would mean that I would have to write a LOT of boilerplate code just to get a high enough API ready for the RenderEngine to give to people using it that it makes my brain do very unhappy things. And I'm not a guru when it comes to any of this shit, this is just a hobby project and I don't want to spend years fucking with it to get it to a point where I'm where I want to be by the end of this year.

    So what do I do? The obvious choice would be to support one or the other solely, writing the RenderEngine's higher level APIs around the lower levels calls and workflows needed to support it. This, in turn, will lock the graphics library to whatever is chosen, and I'm going to have to live with that choice because I don't want to try and support both. So, what do I choose?

    The latest release of OpenGL was back in 2017 with version 4.6. It's stable and has a lot of really good features, and when paired with OpenCL, it makes a fairly buff graphics stack that does a lot. But, it's been radio silent on anything dealing with new versions. Even the latest blog post on the OpenGL website was from 2021. It honestly seems like it's in maintenance mode, and might be heading towards being sunset sooner or later. Like I said before, it's heavily documented and there's a lot on the subject matter on how it all works and tips and tricks, and a metric boatload of libraries on how to use it, but should I commit myself to a library that will become obsolete a lot sooner than I want it to? Do I want to commit myself to developing a personal project that has literally a handful of other, better designed and far superior engines use?

    Or do I want to commit to Vulkan? I'm going to alienate a lot of possible platforms because most GPUs before 2017 don't have Vulkan support, but the workflow is a lot more inline with how Java as a programming language operates. There's going to be a lot of learning experiences and hurtles, but the RenderEngine will be one of the first Vulkan-only game engines in Java, even if it's only used for my game developments. Plus, it's in active development, so I don't have to worry about needing to refactor the RenderEngine in any major way to force something else in, unless there's major Vulkan API changes that require it, like I would need to do to move from OpenGL to Vulkan down the road. Also assuming that I won't make the RenderEngine's rendering API library independent, which as of this blog post isn't a high probability.

    So it sounds like I'm heavily leaning into using Vulkan, I hear you say. And honestly, yeah, I think I'm going to. But that has it's own, very major issue: My current development computer (mid 2015 Macbook Pro Retina) doesn't support Vulkan, so I would need to buy a new development computer to actually do Vulkan development. And decent laptops that support a newest version of Vulkan is ~$600. So it's going to be a decent chunk of change to do major development. And I do have a desktop, however that needs a new GPU, and those are god awful expensive as well. so no matter what, if I want to do Vulkan, I need to pay for a new something to do so. Which sucks.

    So I'm going to leave it off here this week, and I'm probably going to work on some more engine features and test suites over the next week as I mull over what I want to do. As always, I appreciate all of you taking a moment out of your day to read my rambles, and I hope you come back next week.

Read More

Devlog: Module metadata, and revisiting old code

    So the majority of my week was working on writing the module system for the RenderEngine, since most of the core functionality relies on it. And with that, I looked up some of my old code to use for it, and oh boy. First, a little bit of explanation on the things I did, and then the trip down memory lane.

    So, modules. I looked up a few projects that I knew of that had a loading system that I knew of to get an idea on how to implement it. Bukkit was my first go to, but I forgot that was taken down by DMCA because of shenanigans. Forge, as I remembered back in the day, was full of black magic and I honestly didn't want to burn out my brain trying to unwind how it worked. I looked over a few random repos on github when I remembered that Spigot was a thing. Spigot is basically what Bukkit used to be, but without the DMCA. And, most importantly, it has a fairly simple to understand plugin loading system that I can study.

    Spigot's plugins use a metadata file to describe itself and point to the class extends the class that Spigot needs to use to load and control the plugin, using YAML, which depending on who you ask is Yet Another Markup Language or YAML ain't markup language. It's such a simple, yet genius, way to do it. And it works well for Spigot because it uses YAML for a lot of what it does, mostly for permissions to things. For us, as humans, YAML is great to make easy to read and edit data files; For us, as programmers, snakeyaml (the parser most commonly used for parsing YAML files in Java) is a pain in the ass to use unless you have a spec that you want concrete and won't change. Which I don't have, since I'm not sure what I need to do with my modules or how they need to fully act with everything yet. So I wanted to find something that was easy enough to modify during development without also needing to rewrite the parsing logic a metric shit-ton of times.

    Firstly I looked at XML, but it's bloaty. It's super human friendly, but like YAML, it's not very programmer friendly. Same with JSON (JavaScript Object Notation). And I was dreading writing something custom and one-off just for this. I just about made my decision to stick with YAML when I had a thought: Why don't I use NBT?

    Now, NBT is short for Named Binary Tag. It's the data format use for various Minecraft files. It's not super human friendly because it's written as bytes to the file, but there are enough third party open source apps that can parse and make these files human readable and editable that it's a moot point. It's also super programmer friendly, because all you really need to do is make a Compound, add other tags to it, and stuff it into a file and it's done. Literally

NBTCompound compound = new NBTCompound("root");

compound.add(new NBTInt("health", 100));

compound.add(new NBTString("name", "Billy Bob Joe"));

NBTFileOutputStream.write("file.nbt", compound);

and you had an NBT file. AND back in 2015 I wrote my own parser library for it following the Minecraft spec. I though it was the shit. I was so proud of myself for it and I posted it on my Github as part of my libHACK library.

Oh boy, now in 2023, I never knew how wrong I was.

    I started looking over the code and I started having those moments of "da fuck was I thinking?" over way to much of it. There were so many bugs in just the way the NBT tags deal with parenting that I'm surprised that it worked. For instance:

• The parent tag would get a new child tag added to it. Sounds normal so far.
• The parent tag would then check to see if it had a tag with the same name as the new child tag in it's underlying Map, get it, AND REMOVE ITSELF AS IT'S PARENT AND CAST IT TO THE SIDE. We now have an orphaned tag because it had the same name as a new tag. (Bug)
• The parent tag then checks to see if the new child tag has a parent, and if so, check to see if that parent is itself. If it does have a parent that isn't itself, it removes that parent reference from the child (Pretty sure there is a bug in this logic as well in some weird edge cases)
• The parent tag then sets itself as the parent of the child tag.

    So far, we have at least one bug, maybe two, but it doesn't end there. Because of my brilliance all of these APIs were public, due to them being in an Interface. Interfaces in Java can't have protected (either package or class) or private methods, and because I was all into putting everything into an interface because Fuck Yeah Interfaces! at that point of time, I had to have most of the same logic in the abstract implementation of the NBT tags, since it could be called anywhere in the code that had access to NBT tags. And since NBT Containers also implement the Abstract NBT implementation, there are some logic bugs where parent's can recursively become parent's of themselves, stuck in a loop of removing and adding itself. Yeah...

    Needless to say, I worked on making my old code not suck by making proper interfaces and using package protected and private methods to reduce the amount of stuff that's seen in other parts of the code where it's not needed. I'm also starting early and making JUnit tests for all of this because I don't want to be 9 months into development and realize there's a logic bug in this code that causes all kinds of fun things that are hard to chase down. Like I probably should have done in 2015, but I was dumb back then.

    I also made the decision to not follow the Minecraft spec for NBT, and instead use my own. So those third party apps wouldn't work out of the box for my files, but I should be able to find one of them that's easy enough to hack to work for mine that it's not a huge deal for me. But having my own spec allows me to do a lot of things that the Minecraft spec doesn't easily allow for, like nesting Compounds inside of other Compounds (since the Minecraft spec doesn't have a way to signal the end of a Tag, or the end of the file). I'm not entirely sure if this is still the case now, but it was back in 2015 so that's what I'm going on.

    Once I finish all of the Tag classes I need, and the JUnit tests, I'm going to work on the module loader and hopefully get a bare bones test working by the end of the month. That seems like a good goal to shoot for. Right? Right.

Read More

Hello, World

    Hello, My name is Jacob, aka HACKhalo2. I made it a point in 2023 to get a working prototype of a project that I started in 2012 and that I've been wanting to pick back up since 2015, called the RenderEngine. It was a really bad attempt at making a game engine in Java using LWJGL, and had interface hell to plug into the rendering system to do anything. I was so proud of it, but looking over what I had now, it makes me cringe somewhat hard. So, I'm redoing it from the ground up.

    Now, for those who are like "but there are so many game engines out there already, like Unreal or Unity or Godot or..." blah blah blah. I know. There are two well known Java game engines as well, jMonkeyEngine and LibGDX, that I could easily use if I wanted to make a game. I want to build out a personal game engine that can be used for games down the line. I wanna have something that's neat and that I'm ultimately proud of and want to show to people and show off in gamejams.

    So, what am I planning? Well, I want to expand on the idea I tinkered with with the original RenderEngine, but in a different way: I want to have the RenderEngine itself be a library-agnostic API for games, with modules that will be loaded at runtime to supply the logic. So, it doesn't matter if you use OpenGL or Vulkan or Metal or whatever, it'll just work. Same with modules to load resources, like audio or models or textures: everything will be more or less a stable API to use with the game logic side of things.

    Now, this does have some caveats. It's going to be difficult to make everything fit under a single API (especially with Metal, since that's Apple's bullshit. It's not super high on the support list because of that). But the sheer flexibility to be able to update modules to boost performance without needing to touch game code to do so feels so good. And being able to release a general purpose OpenGL or Vulkan Module that can be expanded with vendor-specific things done by users, and then brought upstream to help everyone? Glorious.

    It's definitely a goal to aim for. I honestly don't have any idea if it is possible at all. But then again, It'll be a fun learning experience figuring everything out. And for the first mini-goal: Figuring out how loading modules will work. Bukkit, back when I used to code Minecraft server plugins, obviously had a plugin loader that allowed it to change and extend the Minecraft server. But it was taken down by a DCMA years ago, so I can't really poke around the source to figure out how it works. Minecraft Forge also has a loading system, but for mods. I think I remember looking into it years ago, and the memories it brings up are super... black magicky voodoo stuff happens with that stuff to make it work. But, ClassLoaders are what they both used, so that's what I'm going to start tinkering with.

    I'm planning on doing one update a week until the end of the year, where I hope to have a rough-but-working prototype of the RenderEngine2 to show off. I also might do some updates randomly based on things I do outside this project here as well, like cool little things I do for my day job, or some voodoo I manage to make to get something to work, or just random other weird stuff.

    So, this is HACK, signing off until next week. If you want to shoot me a coffee, I got my Ko-fi button over there somewhere that you can use. Peace.

Read More