Kaetemi

The 3ds Max file format, not too much documentation to be found about it. There are some hints here and there about how it's built up, but there exists no central documentation on it.

Right now we are in the following situation. A few thousand of max files, created by a very old version of max (3.x), containing path references to textures and other max files that have been renamed and relocated or which simply no longer exist. Yes, we have a maxscript that can go through them all, and that manages to fix a large number of paths. However, there are a lot of paths that are stored as part as fields in plugins and material scripts that don't get noticed, and the performance of opening and closing this number of files from 3ds Max directly is horrible. The obvious solution? Figure out how we can read and save the max file with modified contents, without having to understand all of the actual data it contains. Fortunately, this is actually possible without too much work.

Some research online brings up the following blog post, relating to a change in the max file format in version 2010, which would make it easier to update asset paths: http://www.the-area.com/blogs/chris/reading_and_modifying_asset_file_paths_in_the_3ds_max_file. That's nice and all, but it's only from version 2010 on, and it very likely won't contain any assets referenced by path by old plugins and such.

So, starting at the beginning. The blog post I referred to above nicely hints us to the OLE structured file format. Since there exist a wide range of implementations for that, we can pretty much skip that, and accept that it's basically a filesystem in a file, so it's a file containing multiple file streams. A reliable open source implementation of this container format can be found in libgsf. When scanning a fairly recent max file, using the command gsf list, we can find the following streams inside this file:

f         52 VideoPostQueue
f     147230 Scene
f        366 FileAssetMetaData2
f       2198 DllDirectory
f      29605 Config
f       3438 ClassDirectory3
f        691 ClassData
f      29576 SummaryInformation
f       2320 DocumentSummaryInformation

The FileAssetMetaData2 is new in 3ds Max 2010.

One step further, we can start examining the contents of these streams. And it's usually easiest to start off with one of the more simple ones. VideoPostQueue seems small enough to figure out the overall logic of the file format, hoping that the rest is serialized in a similar way. Using the command gsf dump we can get a hex output of one of the streams, and using a simple text editor we can find how it's structured. Binary formats often contain 32 bit length values, which are usually easy to spot in small files, since they'll contain a large number of 00 values. It's basically a matter of finding possible 32bit length integers, and matching them together with various fixed length fields and other typical binary file contents, until something programatically logical turns up. Here's a manually parsed VideoPostQueue storage stream:

[
        50 00 (id: 0x0050)
        0a 00 00 00 (size: 10 - 6 = 4)
        [
                01 00 00 00 (value: 1)
        ]
]
[
        60 00 (id: 0x0060)
        2a 00 00 80 (size: 42 - 6 = 36) (note: negative bit = container)
        [
                10 00 (id: 0x0010)
                1e 00 00 00 (size: 30 - 6 = 24)
                [
                        07 00 00 00 (value: 7)
                        01 00 00 00 (value: 1)
                        00 00 00 00
                        00 00 00 00
                        20 12 00 00 (value: 4610)
                        00 00 00 00
                ]
                20 00 (id: 0x0020)
                06 00 00 00 (size: 6 - 6 = 0)
        ]
]

The storage streams in the max container file contain a fairly simple chunk based file format (and in fact similar in format to the fairly well known .3ds file format). Being based on chunks is what allows 3ds Max to open a file for which certain plugins are missing. It's basically a tree structured format where every entry has an identifier and a size, so when an identifier is unknown, or when it's contents are incompatible, it can simply be kept as is or discarded. The only exceptions in the file that don't use this structure are SummaryInformation and DocumentSummaryInformation, which are supposedly in a standard Windows format, and the new FileAssetMetaData2 section is formatted differently as well unfortunately.

In this format, the chunk header consists of a 2-byte unsigned integer which is the identifier, and a 4-byte unsigned integer, where the 31 least significant bits are the size and the msb is a flag that helpfully lets us know if the chunk itself contains more chunks, and thus is a container, or not. For very large files, where 31 bits is insufficient for the size, the entire size field is set to 0, and the header increases with an additional 64-bit unsigned integer field which is similarly structured as the 32-bit size field. The size field includes the size of the header.

       0 | 0f 20 (id)
                 00 00 00 00 (size missing)
                             17 fe 01 00 00 00 00 80 (size in 64 bits)

With this information it is possible to read a max file, modify the binary contents of chunks (most of them are fairly basic of format), and we should be able to re-save the max file with our modified data. The DllDirectory section, for example, parsed programatically starts like this:

CStorageContainer - items: 20
        [0x21C0] CStorageValue - bytes: 4
        786432216
        [0x2038] CStorageContainer - items: 2
                [0x2039] CStorageUCString - length: 39
                Viewport Manager for DirectX (Autodesk)
                [0x2037] CStorageUCString - length: 19
                ViewportManager.gup
        [0x2038] CStorageContainer - items: 2
                [0x2039] CStorageUCString - length: 49
                mental ray: Material Custom Attributes (Autodesk)
                [0x2037] CStorageUCString - length: 21
                mrMaterialAttribs.gup
        [0x2038] CStorageContainer - items: 2
                [0x2039] CStorageUCString - length: 37
                Custom Attribute Container (Autodesk)
                [0x2037] CStorageUCString - length: 23
                CustAttribContainer.dlo
...

Of course, it would be interesting if we could go further, and directly manipulate the parameters of our own plugins and scripts from our own tools back into the max files so that everything is centrally stored without any duplicate source data in the way. And that's exactly what I'll be doing next.

Beta version will be available for download in the near future.

SSE2 provides functionality for performing faster on aligned memory. By copying the first and last bytes of an unaligned memory destination using the conventional unaligned functionality, and copying everything in between as aligned, it is possible to make use of this performance improvement on large unaligned memory blocks as well.

In this graph the green lines are the conventional memcpy available in Microsoft Visual Studio 2008, the red lines are the SSE memcpy function available in Nevrax NeL, and the blue lines are the custom SSE2 function. The bright colored lines are the performance on alinged memory blocks, while the dark colored lines are tested on differently unaligned blocks of memory. Horizontally the copy function is tested on different sizes of memory, on the vertical axis the copy speed is displayed in MB/s.

As you can see, NeL's SSE memcpy performs very well on aligned memory, but gives horrible performance on unaligned memory, as it does not take the aligning of the memory blocks into account. The builtin memcpy function is fastest of all at copying blocks below 128 bytes, but also reaches it's speed limit there. The SSE2 memcpy takes larger sizes to get to it's maximum performance, but peaks above NeL's aligned SSE memcpy even for unaligned memory blocks.

Code is available below, ask before using.

The project to run NeLSound on top of Microsoft's new XAudio2 API is already nearing it's final stages of development. The driver currently implements all basic functionality required by NeLSound, as well as the some of the environment effects (commonly known as EAX), and has support for the OGG Vorbis music format. I am currently working on adding support for the ADPCM sample buffer format used by NeL to the driver, and I will soon work on the rest of the environment effects implementation as well. Also, the new owners of NeL have made available a new official website for NeL over at http://dev.ryzom.com/, and you can browse the code of the new sound driver at http://dev.ryzom.com/repositories/browse/nel/nel/src/sound/driver/xaudio2.

Hi, I'm Kaetemi, currently sort of working on bringing Snowballs back alive. Snowballs is the prototype game of the open source OpenNeL MMORPG engine. I recommend brining a visit to http://www.opennel.org/ for some more detailed information. As the title of this blog entry suggests, this bit of text is about the gameplay of Snowballs, and there will be an article about a few other parts of Snowballs here every few days, weeks or months, depending on how much time I have for this.

Here's the description of the current gameplay (or at least how it was intended to be) from the readme: "You're now a penguin on an artic snowfield. You have snowballs which you can throw at other people. When you throw a snowball, you have a few seconds during which you gather more snow before you can throw the next one. If you hit someone, your score increases, and the snowed opponent becomes immune to further snowing (he flashes on the screen) for 5s. He cannot retaliate while invulnerable." If you have already playded the game, you would probably have noticed already that this description isn't really a very accurate description of the actual game as it is now. In the code that we have the penguin isn't implemented, you can throw snowballs as fast as you like and you neither become immune nor flash on the screen when being hit by a snowball. In any case, that description would make an extremely boring game anyways.

Obviously if we want to improve the gameplay, we do have to look at what people playing the game actually enjoy doing. This seems to be mostly either exploring the terrain, hitting the left mouse button as fast as they can (preferably on a non-moving target who's most likely afk), or also just talking with each other. So what we need here is a gameplay which allows at least these few things.

In order to allow exploration, we need a continueous gameplay. This means that instead of resetting all player stats or whatever on every round or something like that, a more dynamic system should be in place. Similarly to how the game is implemented right now, anyone should just be able to join and leave the game at any time without any problems. A new player should also not have any extreme disadvantages over people who have been playing for a while. Not allowing any players to have any advantages over others at all would make the game extremely boring, though.

An appropriate way of solving this, is by having some sort of short-term advantages only. These could for example be implemented using either a time, or usage limit, but doing it in that style for every bonus is really boring, especially since other players have no other way than waiting for the advantage to go away on that player, and waiting is a really boring game. Instead, it must also be possible for a player to make another player lose his advantage over other players. A possible method is simply by letting x amount of bonuses go down when a player gets hit by a snowball, but since this isn't very effective when a player has something like an extremely rare overpowered defense shield bonus, or high speed snowball dodge skill, or something, other useful methods could be special bomb items (somewhat less rare than the extreme defense bonus) which instantly reset all advantages of all players in x radius around you (including yourself, obviously) without doing any other damage. Bonus stats and items or whatever can be given to players in different ways. Players could get higher hit points when they hit someone else, they could pick up bonus items or skills at random locations, there should be bonuses that are less common than others, there could even be bonuses that allow players to gain bonuses in an alternative way.

Now to attack another player, hitting the left mouse button really fast, and throwing loads of snowballs, is definitely more fun than just click, hold, wait and release, click, hold, wait and release, etc. One of the types of bonuses a player can gain, would be different attacks, trough which you'd then be able to switch with your mouse scroller. This includes things like throwing at your target at high speed with small target radius, large snowball bombs falling from the sky, snowballs being launched around you in a spiral, snowballs being launched to random positions in front of you at high speed, higher target radius the longer you hold your mouse button, loads of snowballs launched around you in all directions, etc, very much like games of the shoot 'em up genre (those 2d games where loads and loads of bullets are being shot at you the whole time).

There will also be some sort of scoreboard which shows the best players online. Points can be earned, obviously, by hitting someone with a snowball, or by picking up bonuses from the ground (gives some exploration value). A team system could be added, where you'd have to defend or take over a few different villages in the area. And for those who just want to talk to others, there will also be a bar, which is a safe zone where no snowballs can land; and after leaving it you'll have ten seconds to run as far as you can before you're back to being attacked by snowballs.