[WIP] Quickbeam

[WaeV]

Latest update: 10/28/2013

Python scripting is working flawlessly; the next step is to connect the Python backend to the buttons-and-menus sort of interface we're all used to. However, Python scripting will remain a feature of the editor.

Live editing (including swapping of model references!) is demoed below, using only Python scripting.

h-BZ6J_Ha4M

Howdy, y'all. It's been awhile. I stopped in once before, and I remember some of you from the old HaloMods. I thought you might be interested in what I've been up to lately. As of yet I've only been posting updates to OpenCarnage.net (where many of the ModHalo.net regulars migrated to).

So- my quest to make a Halo map editor. What began as an innocuous set of Python scripts eventually led to me hacking around with C# and some native dll calls. About three months ago I succeeded in embedding a running instance of Halo into my own project, and that's when things really started to get kicked off.

http://i.imgur.com/dSc1kl.jpg


Spurred onward by my initial successes (honestly, the embedding wasn't too hard; I found out how to do it on StackOverflow, although it does look mighty cool), I decided to start putting more effort into a little project called Quickbeam.


VoilÃ*, an early version of Quickbeam! All it could do is display tag names. Well, actually I could read a good bit of data, but writing back wasn't supported and plugins weren't implemented.

http://i.imgur.com/SnAQn.png


After working on that for a while, I found this thread (http://www.modacity.net/forums/showthread.php?11175-Halo-Structures) (woo, thanks Modacity) on Halo Structures. I also managed to find a download of Conure's beta Memory Editor. After decompiling his app (which wasn't too hard, on account of it being a .Net app) I found some really useful offsets and began work on adding memory-editing to Quickbeam.

Here's a screenshot of the first live reference-swap I made (back in the EmbedTest project, as you can see). I sprang for the classic "plasma pistol shoots tank shells" swap.

http://i.imgur.com/Z8j2zh.jpg

I'd also like to give a shoutout to Ryx from ModHalo/OpenCarnage - he was a huge help when it came to Halo's rendering problem. Apparently Halo only renders audio and video when the window is selected, otherwise it freezes output. Thanks to Ryx's fix, video now continues to render even if the window is deselected. No word on a fix for audio output (I spent a while looking, but it's on the backburner for the moment).


I got stuck for a long while at this point, wanting to add plugin support and real editing capability, but I didn't know WPF well enough. Just a few days ago, however, I decompiled one of my favorite Halo editors "Lethargy" (apparently Skeezix and I are the only two people who use that editor). I cleaned up the source code output, managed to get it compiling, and I'm now using Lethargy's code as a sort of guide. I like WPF and think it's technically superior, but for now the editing controls will be written in WinForms for that reason.

I've been making pretty rapid progress on the GUI since then. Here's a screenshot from early yesterday. You can see that I still have the WPF read-only tag tree on the left. The big area with the gross white-blue gradient is WinForms.

http://i.imgur.com/doTkN.png


And here's Quickbeam just a few hours later. The big purple 800x600 rectangle is where the embedded Halo ought to end up. Also note the WinForms tag tree (you can tell because it has the same visual style as HMT, Lethargy, etc.). The next step is HMT-style plugin support (finally!) I'm not sure if I'll bother with entity plugins straight away. It might be simple to add that, not sure.

http://i.imgur.com/UnSMX.png


I was actually pleasantly surprised to see that the WinForms 'button1' over there has a dark theme to match the WPF area of the application. I thought I'd have to manually restyle it like I will for the tag tree, but it looks like I got buttons for free!



Reading from and writing to memory already works. Adding the plugin-based editing controls is really the only thing standing between this and a proper live-memory-editing Halo mod tool.

Oh yeah, and it's all on GitHub: https://github.com/ChadSki/Quickbeam
The cool stuff is in the 'HaloCore' backend, if you're interested.

[Btcc22]

Saw this on OpenCarnage a while back; nice to see progress. :)

Just one question though. Wouldn't it be easier for you to develop this if you injected a DLL to get direct access to Halo's address space rather than going back and forth with Read/WriteMemory/OpenProcess etc? Seems like it's going to be cumbersome after a while if you're going to be working on this for some time.

[WaeV]

If by 'cumbersome' you mean worth a lot of developer time, then no, I don't think so. I built Quickbeam to be pretty generic regarding the way it reads and writes data from the mapfile. Basically, there's something I call a 'ByteAccess' which covers a certain area of bytes and provides read/write access. I've already implemented FileByteAccess and MemoryByteAccess, so there's really no work to be done. As far as the editor is concerned, it's all the same thing.

That's an interesting suggestion, though. Would an injected dll allow direct addressing of Halo's memory space? The one area where ReadProcessMemory falls short is with null-terminated strings of arbitrary length. I have to read x bytes, check if I found null, and grab more bytes if I haven't.

[Btcc22]

Would an injected dll allow direct addressing of Halo's memory space?

Sure would - that's one of the major benefits. You can just plonk the structs in, point at the address and work with the memory directly.

You can actually avoid having to inject with Halo or having to modify any of the original files (I pity strings.dll) but the downside is that you wouldn't be able to selectively disable the DLL, as you'd probably want with this sort of application.

[WaeV]

(I pity strings.dll)
Lol, why's that?


Interesting. I've started googling around and managed to find some interesting links. Many debuggers have ways of attaching to processes, for instance. Some people mention needing to start the to-be-read application as a child process, which I'm already doing when I embed Halo. Cool stuff. Do you have any links in particular you could point me to? This is completely new info to me.

[Btcc22]

Lol, why's that?

It seems to be the primary method of loading custom DLLs when it comes to Halo mods. Great fun if you have multiple mods that require a new strings.dll.

I don't think you'd want to go down the route of debuggers in this case. They're pretty complex beasts and the memory manipulation functionality provided by some tend to use exactly the same APIs as you're already using.

This article (http://web.archive.org/web/20101224064236/http://codingthewheel.com/archives/how-to-inject-a-managed-assembly-dll) has some good info on getting a managed DLL loaded into a native process though. Disclaimer: I haven't tried it.

[Patrickssj6]

I remember this...and since I have a pretty shitty memory, this has got to mean something :P

[WaeV]

This article (http://web.archive.org/web/20101224064236/http://codingthewheel.com/archives/how-to-inject-a-managed-assembly-dll) has some good info on getting a managed DLL loaded into a native process though. Disclaimer: I haven't tried it.
That article is excellent, thanks. I think that will come in handy even if I'm not starting up Quickbeam from within Halo... I've been planning on integrating the Io scripting language into the editor, but it's written in C and I was wondering of a good way to go about linking everything together.

[WaeV]

Update! -- Swap models at runtime!
Hey guys, how's it been? Checking in with scriptable runtime editing of metadata and reference-swapping.

Currently, most everything is implemented in Python. Edits can be made to mapfiles on disk, or in Halo's memory while the game is running (see video). Multiple maps can be open at a time, and I am looking into cross-map importing and exporting of tags.

h-BZ6J_Ha4M

[Limited]

Yeah so this is damn impressive - great work :O

[WaeV]

New update!

I have switched from CPython3 to IronPython to make C# interop easier. Thanks to that switch, the edit boxes (traditional HMT/Eschaton-type controls) are seamlessly integrated with Python scripting -- whenever scripts make changes, the GUI updates to reflect this.

Ki3-RjgCzZI




http://i.imgur.com/mmEWo93.png

Here is an overview of Quickbeam's (planned) architecture. So far, many of the pieces are in place, with the exception of add-ons.

Binary Access
At the very bottom is the layer that makes reading maps from memory feel exactly the same as reading from disk. Most users need not think about this layer; they can just take it for granted.

Plugins
One step up is the plugins layer. Whereas user-contributed plugins have been popular for older programs like HMT and Eschaton, Quickbeam's plugins will be moderated by me. This is because script compatibility depends on everyone having the same plugins, and we've pretty much mapped Halo out anyways. I can update plugins after the fact if they do need to change, and you can always substitute your own if you really want to.

Scripting
In the middle is the celebrated (by me, anyways) scripting layer! This should hopefully be intuitive to most modders. Scripting is done via Python, and I intend to publish a few tutorials and examples. Besides being cool, the scripting layer will make it easier for me to develop new features. Additionally, it will open the door for...

Add-ons
This feature is less fleshed-out as the rest, but I want users to be able to create and publish their own special-purpose buttons and controls. This could be as simple as a button for automating some tedious task using scripts, or something more complex. The design of this feature is hazy at the moment, and will not be present in the initial releases of Quickbeam

Classic Editor Interface
Finally, at the very top is the HMT/Eschaton-like editing controls we all know and love. Except these ones won't be glitchy or laggy (I hope).

[Rentafence]

Pretty cool stuff. I also like that you spent time making the interface attractive.

[WaeV]

In order to actually release something that much faster, I've spent the last couple months polishing up the scripting layer. Halolib 1.0 is now available for use! http://opencarnage.net/index.php?/topic/5878-introduction-to-memory-editing-with-python-and-halolib/ It works a lot like the scripting demos shown above (from 2013, I know). However, the code is a lot cleaner. It's written in straight Python 3 and requires no non-standard libraries. Currently it only memory-mods Halo PC 1.10. In theory it can memory-mod CE as well as dedicated servers, but I need to put the time into discovering the correct offsets for everything. Edit: why the hell are linebreaks not working?