[Survey] Rate the following options...

[Kornman00]

On a scale of 1 to 10, 1 being the lowest, 10 being the most awesomest-fucking-blows-my-mind-and-make-me-a-sammich, or N/A if it is not applicable to you, rate the following abilities\utilities based on if you could have them and how often you'd actually use them. Also order them from the most "must" have to the least "must have".



Override Tag/Map paths for HEK

Via a single XML file located with your HEK tools which specifies all tag/map profiles then allows you to specify the 'default' profile to use (ie, so you can switch from a stock tags folder to a halo3 based tag set)

Having a dedicated "Yelo" folder in your maps folder for all yelo related map/data-files

Keeps all your Yelo related data clumped together. Makes it so the stock (read: non-Yelo) game can never, ever load any of the files.

Tag reports

Reports based on a input tag. Figures out what tags it references, then also what other tags reference the dependencies

Auto tag archiving

Archive an entire tag's dependency tree based on an input tag

Auto tag renaming

Renaming a specific tag, then having that change be reflected to all tags in the working tag set

Auto tag relocating

Moving a tag or directory of tags to a different location within the working tag set then having that change be reflected in the entire tag set

Scenario Resources

a la Halo 2

[CodeBrain]

tbh I would say 10 for everything. XD

All those features sound really fucking nice!

[Dwood]

On a scale of 1 to 10, 1 being the lowest, 10 being the most awesomest- -blows-my-mind-and-make-me-a-sammich, or N/A if it is not applicable to you, rate the following abilities\utilities based on if you could have them and how often you'd actually use them. Also order them from the most "must" have to the least "must have".



Override Tag/Map paths for HEK

Via a single XML file located with your HEK tools which specifies all tag/map profiles then allows you to specify the 'default' profile to use (ie, so you can switch from a stock tags folder to a halo3 based tag set)


7- - Would be nice, but overcome by auto-renaming and relocating HOWEVER this would be the most practical as most if not all tag sets are a mess.

Having a dedicated "Yelo" folder in your maps folder for all yelo related map/data-files

Keeps all your Yelo related data clumped together. Makes it so the stock (read: non-Yelo) game can never, ever load any of the files.



1 - I've never encountered that as an issue

Tag reports

Reports based on a input tag. Figures out what tags it references, then also what other tags reference the dependencies


6

Auto tag archiving

Archive an entire tag's dependency tree based on an input tag


5 - would be nice but with renaming and relocating, we can archive it all ourselves

Auto tag renaming

Renaming a specific tag, then having that change be reflected to all tags in the working tag set


9 - Would be very nice to have, but i'd like an example of a command name or something if it's via OS Tool

Auto tag relocating

Moving a tag or directory of tags to a different location within the working tag set then having that change be reflected in the entire tag set


10 - This, and tag renaming would be grrrreat! If we could relocate and rename, It I could set another tag folder and use it any time I want to do some replacement/overrides.

Scenario Resources

a la Halo 2


1


Text in bold

[Arteen]

3. Tag reports
5. Auto tag renaming
6. Auto tag relocating


5 These would be great to have, and I assume they use much of the same code. It beats the regular method of renaming a tag and seeing what breaks. I rated them low though, since it's something I can easily live without and probably won't get much use out of it.


4. Auto tag archiving
1. Override Tag/Map paths for HEK


3 Both sound great, but I would have gotten much more usefulness out of them years ago. I don't think I will have much use for it now.

2. Having a dedicated "Yelo" folder in your maps folder for all yelo related map/data-files


N/A I don't use Yelo


7. Scenario resources


N/A since I don't even know what you mean by this.

[DarkHalo003]

1: N/A

2: 10 (I'd like to use Yelo)

3: 10 (PLEASE, it'd make debugging so much easier)

4: IDGI

5: 10 (THIS would be convenient)

6: 10 (THIS would be even more convenient)

7: N/A

[Kornman00]

These would be great to have, and I assume they use much of the same code. It beats the regular method of renaming a tag and seeing what breaks. I rated them low though, since it's something I can easily live without and probably won't get much use out of it.
Well they use the same back-end systems, yeah. I haven't stress tested these specific items (expect for tag reports) yet and thus wanted to get an idea on how big of a priority, if any, I should give them. Basically you'd have to create an initial database of your tag set then perform the operations you want with the tool. Any changes made to tag dependencies after the tool is used would require the database to be updated.

Since all my external tools have always been made to be non-intrusive to the existing tag formats I have to work around the existing standards and formats Bungie used for Halo 1/2/etc that really aren't optimized for massive operations or tracking. Having to load an entire tag just to find all of its child dependencies isn't a very cheap operation, especially when doing it on an entire tag set.

I ran a test a week or so ago with a debug build of my tool and loading a tag index for tutorial.scenario and another one for globals.globals then to spit out a tag database tag for both of them takes about 30 seconds (all of their dependencies are loaded too). Both are being loaded by two different threads at the same time (reference my computer specs), off the same HDD. If I was only using a single tag index for both tags then there wouldn't be any tags loaded twice (ie, weapons etc placed in the scenario then referenced in the global's cheat_weapons) and if I was running an optimized build I'd figure it would take 20 seconds or less.



Scenario resources could be useful if you wanted to export all the scenery (or other object types, ai, etc data) to a specific resource tag (ie, scenario_scenery_resource). Why would you want to export such data? It could act as a simple versioning system for your scenario system. You wouldn't have to save an entire scenario which may have finalized vehicle data (which wouldn't need to be versioned as it is already final) and have 10+ scenarios in your map's folder which you then have to go and merge by hand later on. Less duplicated data and easier resource management. I know there is a "child scenarios" block but I really don't know if that thing is even supported by the tools and it still doesn't solve resource management in a fashionable method.

[CodeBrain]

7. Scenario resources



N/A since I don't even know what you mean by this.



Halo 2 uses scenario resources, which group tags together, instead of adding them seperately, like Halo CE. Examples would be an ai_resource tag would contain all the ai used in a particular level as one tag. the same can be said for other resources, like scripts, bipeds, vehicles, and weapons.

Of course I can be terribly wrong and someone can clarify better than me <_>

edit: got damn kornman you said it before me

[il Duce Primo]

I would kill for the auto tag tools. It would allow me to organize the CMT tagset which in its current state is extremly messy and it makes it difficult to work with. It also has never been practicle to sit there and rename and relocate most of the tags so this tool would be amazing.

[Shock120]

1. I would like to see "Override Tag/Map paths for HEK", make use of Windows 7 library feature.

2. Convenience. 10.

3. I'd find reports useful, but I'd be too lazy to look at it.

4. 5. 6.
are useful for those who don't know where tags should go, I would say 10 on these 3.

7. bit by bit is ok.

[Kornman00]

I'm not making the tools any more platform specific than they are already or adding special platform functionality to a closed-program. Path overriding would only be done via an XML file in the same directory as the tool being used.

[Spartan094]

1. N/A
2. N/A
3. 7
4. N/A
5. 8
6. 10
7. 1

[paladin]

Is this why youve been in h2v for ever and not messaging me back

[Ifafudafi]

Override Tag/Map paths for HEK
8
Having a dedicated "Yelo" folder in your maps folder for all yelo related map/data-files
N/A
Tag reports
7
Auto tag archiving
5
Auto tag renaming
10 OH MY GOD YES DO THIS
Auto tag relocating
10 SEE ABOVE
Scenario Resources
N/A
Also Shift+Enter is a pretty cool keyboard combination, works great for lists, hth

[Kornman00]

For some of the automated tag tools I can either fine tune it for speed or less memory. Yes, I know it would be nice to have something of a balance of the two but the codebase I'm working in is too old to do such updates on how things operate.

If you (the users who want automated tag tools) had to choose between faster operations vs. less memory consumption, which would you prefer? As it stands it will eat up memory (nothing my machine can't handle at least) in favor of faster operations as I've been engineering it so that it works best/fastest on a completely analyzed tag set (which requires the tool to go over your ENTIRE tags folder, loading each tag to find references unless the tag itself doesn't have any tag reference fields). From there you would want to perform all your operations (moving, renaming etc). After that you'd be free to work again on your tag set. Most of these operations I don't see many people doing on a daily basis (unless they're just horrible HEK users) so waiting around a minute (depending on how many scenarios and extra content you have) for an analysis to be performed on a tag set shouldn't be that bad overall.

[Dwood]

Less memory consumption please. I can wait a minute or two.

[Ifafudafi]

Exactly how much memory are we talking? If it's not any more than it takes to run Halo itself then I think there's no reason not to go for speed, but if it's eating up, like, 5GB, then go for efficiency

[Kornman00]

It's all .NET based (so there is a small amount of managed overhead) and most of the memory hording comes from keeping a real-time link on each object (ie, a tag_reference) which references a loaded tag instance. IE, if I said load scenario X and all of its dependencies, then it would have a collection of links from each tag definition (ie, a sound) to all the tag_references (in a vehicle, another in a looping_sound, etc) in all the currently loaded tags which reference that tag definition (again ie, a sound). This way performing operations (ie, renaming) are nearly instantaneous since it already has direct tabs, or at least a very short path it needs to travel, for all objects which need to be updated due to those operations.

Another way is to just work off the cached tag database which is created when the tag set is scanned. This database has every tag in the tag set, then also weak links to all of its child dependencies and to all tags which reference it. I give the database the name of the tag I'm wanting to rename, then I get a collection of links to the entries in the database which represent the tags referencing my tag file I want to rename. I can then rename the tag file, then use that collection of links to find the name of every single offending tag which contains a reference to the now renamed tag, load them, then tell every tag_reference in it the new value.

The first method is great for large operations, which is what I see it being mostly used for initially. The second method is great for small one-off operations every now and then, which you would start to see after a few months.


Can these two meet half way? Yes, but I'm not devoting the time to fully develop these old tools (the codebase itself has been iterated on for going on 5+ years now) for these old Halo games which have a very, very weak tag file format for dealing with these operations which I have to work around. These are all just "extra" things I'm considering ironing out for public consumption. There are quite a few optimizations in a couple different areas I can think of today which would speed these operations up but they would require some major changes to how I handle and manage tags, which I'm not going to do in this old codebase.

Loading tutorial and globals, plus every single child tag in a debug environment under Visual Studio takes up a good deal of memory, but no where even near 1GB. This totals around 4k+ tags loaded in memory IIRC. To give a comparison, H2CE's tags folder has ~10,500 tags (with I would say maybe around 500 to 1000 of them being non-tag files, autosaves, or dead tags). If I were to load the H2CE tag set for operating on, I'd say it would probably start nearing 1GB in real-time memory being used (there would then be fluxes due to temp memory allocated and deallocated by the actual operations, ie renaming, when performed).

Also: the codebase can be compiled and ran under x64 platforms. I haven't tested to see if this even gives a gain in speed, but I'd be willing to bet there would be around a 60% increase in memory usage if using a x64 based binary.

[Dwood]

So in this case, x86//32 bit is better?

[Kornman00]

I just ran a debug build test for a load (nothing more, nothing less) of globals and tutorial, plus all their child tags. Both tags have their own tag index and each index is operating on a different thread. During this time the tool is also creating those memory links I was talking about earlier which would allow near instantaneous updates to tag references.



SCENARIO LOAD: Time taken: 00:00:04.9352823
GLOBALS LOAD: Time taken: 00:00:09.5915486
(Note: some of the tags may have still been warm in the file system cache since I had performed some tests about 30 minutes earlier. To get the most accurate worst-case times to load I'd have to run the tests when no tags are stored in the file system cache)

It then dumps the tags which appear in both tag indexes (basically, which tags are shared between scenario and its children, and globals and its children).

1570


e: I can't say which binary build would be better as I don't know if a x64 build would result in better optimized code. I do know that memory would be significantly increased due to the CPU's word size being double that of x86's. But then again, the x86 code is running under SysWoW since my machine is x64, who knows what sorts of overhead this means in both native and managed framework terms.

[Kornman00]

Adding the ui tag_collection tags to the same thread which loads the globals tag adds about 8 extra seconds to the processing time. So loading globals/ui plus the tutorial takes roughly 17 seconds when threaded into two threads.

To give you an example of a tag report:

(file manager is chopping off parts of the file names)
1571 (scenario)
1572 (globals)

[bleach]

I know I am late and I'm sorry but since I just started modding, I figured I would just give my opinion.


Override Tag/Map paths for HEK: 9
Having a dedicated "Yelo" folder in your maps folder for all yelo related map/data-files: N/A
Tag reports: 8
Auto tag archiving: 10
Auto tag renaming: 10
Auto tag relocating: 10
Scenario Resources: N/A

Also, I favor faster operations over less memory consumption - but I know most members might veer towards efficiency or a combination.
Is it possible to release 2 versions - 1 for efficiency and the other for faster operations?

[S3anyBoy]

I know there is a "child scenarios" block but I really don't know if that thing is even supported by the tools and it still doesn't solve resource management in a fashionable method.

When I was working on my Firefight maps I setup the AI and scripts in a scenario with nothing else referenced and referenced it as the child scenario in each scenario, that way I easily had all the scripts and AI placed and then just had to worry about map specific thing (like weapons, vehicles, etc) I don't see many other people using that reference, even though it can be extremely helpful.

[CrAsHOvErRide]

C# my guess? Though I have faith in your superior programming skillz...maybe show code snippets so we can look for optimizations?

[Kornman00]

Seeing as how there is at least 5 major components (then the actual tags themselves) at work I don't think posting snippets would give any light to optimizations which I haven't already seen. Actual rewrites of some of these system's implementations (as most were made in 2006/2007) would yield better performances but I'm not giving the code base that much attention anymore.

Here's one unit test's implementation (for extracting from a game's cache to file)


static void CacheExtractionMethod(object param)
{
var args = param as CacheFileOutputInfoArgs;

using (var handler = new CacheHandler<Blam.Halo2.CacheFile>(args.Game, args.MapPath))
{
handler.Read();
var cache = handler.CacheInterface;

var ti = cache.TagIndexManager as Blam.Halo2.InternalCacheTagIndex;
ti.ExtractionInitialize();
Assert.IsNotNull(ti.kVertexBuffers);
{
string test_results_tags_path = EngineGetTestResultsPath(args.Game) + @"tags\";

// extract with dependents, database and overwrite existing tag files
var ex_args = new Blam.CacheExtractionArguments(test_results_tags_pa th,
true, true, true, kExtractionDontUseTags);
Blam.CacheIndex.Item tag_item;
Assert.IsTrue(cache.TryAndFind(@"objects\characters\masterchief\masterchief", Blam.Halo2.TagGroups.hlmt, out tag_item));
{
var cei = ti.ExtractionBegin(tag_item.Datum, ex_args);
Assert.IsTrue(ti.Extract(cei));
ti.ExtractionEnd();
}
}
ti.ExtractionDispose();
}
}

[CrAsHOvErRide]

Seems bulletproof. Without changing some major components there is not much to optimize there.