Kaetemi

To content | To menu | To search

Tag - autodesk

Entries feed - Comments feed

Sunday 19 August 2012

3ds Max File Format (Part 3: The department of redundancy department; Config)

By Kaetemi on Sunday 19 August 2012, 11:26 - Articles

Now we'll have a look at the Config stream. It begins like follows, and goes on forever with various integer fields and other binary blobs.

(StorageContainer) [15] { 
0 0x2090: (StorageRaw) { 
        Size: 4
        String: .... 
        Hex: 00 00 00 00 } 
1 0x20e0: (StorageContainer) [4] { 
        0 0x0100: (StorageRaw) { 
                Size: 12
                String: .. A........ 
                Hex: 00 00 20 41 0a 00 00 00 01 00 00 00 } 
        1 0x0400: (StorageRaw) { 
                Size: 8
                String: ........ 
                Hex: 07 00 00 00 01 00 00 00 }

As most of the contents seems fairly different from eachother, it's best to look from a distance to the main container.

(StorageContainer) [15] {
0 0x2090: (StorageRaw)
1 0x20e0: (StorageContainer) [4]
2 0x20a0: (StorageContainer) [2]
3 0x20a5: (StorageContainer) [2]
4 0x20a6: (StorageContainer) [1]
5 0x2190: (StorageContainer) [2]
6 0x20b0: (StorageContainer) [10]
7 0x2130: (StorageContainer) [3]
8 0x2080: (StorageContainer) [213]
9 0x20d0: (StorageContainer) [9]
10 0x2160: (StorageContainer) [5]
11 0x21a0: (StorageContainer) [82]
12 0x2180: (StorageContainer) [1]
13 0x2007: (StorageContainer) [1]
14 0x2008: (StorageContainer) [3] }

The first id seems to be unique, so we can assume that each of these containers has a specific set of information in it. Comparing between files of max versions, there are some less and some more of these identifiers, but the contents of them remains pretty much the same.

One container in this file particularly interests me, as it contains strings related to the NeL Material, and thus will likely be necessary to parse the Scene format where this is stored. More specifically, chunk 0x2180 contains stuff like the following:

9 0x0007: (StorageContainer) [3] { 
        0 0x0060: (StorageRaw) { 
                Size: 4
                String: .... 
                Hex: 02 00 00 00 } 
        1 0x0006: (StorageRaw) { 
                Size: 17
                String: ....bForceZWrite. 
                Hex: 0d 00 00 00 62 46 6f 72 63 65 5a 57 72 69 74 65 00 } 
        2 0x0007: (StorageContainer) [7] { 
                0 0x0060: (StorageRaw) { 
                        Size: 4
                        String: .... 
                        Hex: 06 00 00 00 } 
                1 0x0006: (StorageRaw) { 
                        Size: 9
                        String: ....type. 
                        Hex: 05 00 00 00 74 79 70 65 00 } 
                2 0x0006: (StorageRaw) { 
                        Size: 12
                        String: ....boolean. 
                        Hex: 08 00 00 00 62 6f 6f 6c 65 61 6e 00 } 
...

The block itself begins like:

12 0x2180: (StorageContainer) [1] { 
        0 0x0040: (StorageContainer) [2] { 
                0 0x0050: (StorageRaw) { 
                        Size: 12
                        String: ....._.d..+" 
                        Hex: 00 0c 00 00 ec 5f c7 64 b9 9e 2b 22 } 
                1 0x0007: (StorageContainer) [10] { 
                        0 0x0060: (StorageRaw) { 
                                Size: 4
                                String: .... 
                                Hex: 09 00 00 00 } 
                        1 0x0007: (StorageContainer) [15] { 
                                0 0x0060: (StorageRaw) { 
                                        Size: 4
                                        String: .... 
                                        Hex: 0e 00 00 00 } 
                                1 0x0006: (StorageRaw) { 
                                        Size: 9
                                        String: ....nlbp. 
                                        Hex: 05 00 00 00 6e 6c 62 70 00 } 
...

A max file that has the NeL Multi Bitmap script used in it as well, has 2 0x0040 entries in the 0x2180 container. We'll call the 0x2180 block ConfigScript, and the 0x0040 container ConfigScriptEntry from now on, as these seem to be related to how script parameters will be stored in the file. What's also interesting is that all the chunks with id 0x0007 in this entire block are containers, and all the 0x0060 blocks are integers. The 0x0050 block is the header block for the ConfigScriptEntry, and contains the same SuperClassID and ClassID from the NeL Material script as seen previously.

Here are a few 0x0007 blocks from a specific depth inside the tree structure:

2 0x0007: (StorageContainer) [5] { 
        0 0x0060: (StorageRaw) { 
                Size: 4
                String: .... 
                Hex: 04 00 00 00 } 
        1 0x0006: (StorageRaw) { 
                Size: 9
                String: ....type. 
                Hex: 05 00 00 00 74 79 70 65 00 } 
        2 0x0006: (StorageRaw) { 
                Size: 12
                String: ....boolean. 
                Hex: 08 00 00 00 62 6f 6f 6c 65 61 6e 00 } 
        3 0x0006: (StorageRaw) { 
                Size: 12
                String: ....default. 
                Hex: 08 00 00 00 64 65 66 61 75 6c 74 00 } 
        4 0x0001: (StorageRaw) { 
                Size: 4
                String: .... 
                Hex: 00 00 00 00 } } } 

2 0x0007: (StorageContainer) [7] { 
        0 0x0060: (StorageRaw) { 
                Size: 4
                String: .... 
                Hex: 06 00 00 00 } 
        1 0x0006: (StorageRaw) { 
                Size: 9
                String: ....type. 
                Hex: 05 00 00 00 74 79 70 65 00 } 
        2 0x0006: (StorageRaw) { 
                Size: 10
                String: ....float. 
                Hex: 06 00 00 00 66 6c 6f 61 74 00 } 
        3 0x0006: (StorageRaw) { 
                Size: 12
                String: ....default. 
                Hex: 08 00 00 00 64 65 66 61 75 6c 74 00 } 
        4 0x0004: (StorageRaw) { 
                Size: 4
                String: ..#< 
                Hex: 0a d7 23 3c } 
        5 0x0006: (StorageRaw) { 
                Size: 7
                String: ....ui. 
                Hex: 03 00 00 00 75 69 00 } 
        6 0x0007: (StorageContainer) [2] { 
                0 0x0060: (StorageRaw) { 
                        Size: 4
                        String: .... 
                        Hex: 01 00 00 00 } 
                1 0x0006: (StorageRaw) { 
                        Size: 17
                        String: ....cfBumpUSpeed. 
                        Hex: 0d 00 00 00 63 66 42 75 6d 70 55 53 70 65 65 64 00 } } } } 

2 0x0007: (StorageContainer) [5] { 
        0 0x0060: (StorageRaw) { 
                Size: 4
                String: .... 
                Hex: 04 00 00 00 } 
        1 0x0006: (StorageRaw) { 
                Size: 9
                String: ....type. 
                Hex: 05 00 00 00 74 79 70 65 00 } 
        2 0x0006: (StorageRaw) { 
                Size: 12
                String: ....integer. 
                Hex: 08 00 00 00 69 6e 74 65 67 65 72 00 } 
        3 0x0006: (StorageRaw) { 
                Size: 12
                String: ....default. 
                Hex: 08 00 00 00 64 65 66 61 75 6c 74 00 } 
        4 0x0003: (StorageRaw) { 
                Size: 4
                String: .... 
                Hex: 01 00 00 00 } } }

If you have an understanding of the file format at this point, and try to understand the contents of these blocks, you should notice how ridiculous this looks.

It's like making an xml file that goes

<name>Name</name><value>Number</value>
<name>Value</name><value>2</value>

instead of

<Number>2</Number>

thanks to abstraction layers being piled up on each other.

The 0x0060 entry contains an integer which states the number of chunks that follow in the container, and is thus basically the size header of an array. Chunks with id 0x0006 recognizably contain strings, prefixed with their size, which is already known by the chunk header, and followed by an unnecessary null value byte suffix. It gets even sillier. The blocks shown above are actually not arrays, but tables of two columns stored in an array stored in the chunk tree structure. The first value in the array is the name column, and the second value the value column.

These blocks are child chunks of containers, containing a chunk with a name of a data field, and describe the format of this data field. It is fairly straightforward, the value of the name 'type' is the type, the 'default' is the default, and so on. For the default value, and this actually goes for the entire format of the ConfigScript block, the id of the chunk is directly related to the type field, and defines the actual low level storage of the field. The type fields is very helpful in finding out the meanings of these chunk ids.

0x0001 is a boolean stored as 4 bytes, 0x0002 does not appear in my file, 0x0003 is a 32 bit possibly signed integer, 0x0004 is a float, 0x0005 is a string in the same format as the 0x0006 internal strings, 0x0007 is the previously covered array-in-a-container, 0x0008 is a color stored as a 3 floating point vector with value 255.f being the maximum. With this information, the file can be made much more readable. Here's a short excerpt (pun intended) from inside the ConfigScript block.

2 0x0007: (ConfigScriptMetaContainer) [42] { 
        0 0x0060: (CStorageValue) { 41 } 
        1 0x0006: (ConfigScriptMetaString) { main } 
        2 0x0003: (CStorageValue) { 1 } 
        3 0x0003: (CStorageValue) { 2 } 
        4 0x0007: (ConfigScriptMetaContainer) [3] { 
                0 0x0060: (CStorageValue) { 2 } 
                1 0x0006: (ConfigScriptMetaString) { rollout } 
                2 0x0006: (ConfigScriptMetaString) { NelParams } } 
        5 0x0007: (ConfigScriptMetaContainer) [3] { 
                0 0x0060: (CStorageValue) { 2 } 
                1 0x0006: (ConfigScriptMetaString) { bLightMap } 
                2 0x0007: (ConfigScriptMetaContainer) [5] { 
                        0 0x0060: (CStorageValue) { 4 } 
                        1 0x0006: (ConfigScriptMetaString) { type } 
                        2 0x0006: (ConfigScriptMetaString) { boolean } 
                        3 0x0006: (ConfigScriptMetaString) { default } 
                        4 0x0001: (CStorageValue) { 0 } } } 
        6 0x0007: (ConfigScriptMetaContainer) [3] { 
                0 0x0060: (CStorageValue) { 2 } 
                1 0x0006: (ConfigScriptMetaString) { bUnlighted } 
                2 0x0007: (ConfigScriptMetaContainer) [7] { 
                        0 0x0060: (CStorageValue) { 6 } 
                        1 0x0006: (ConfigScriptMetaString) { type } 
                        2 0x0006: (ConfigScriptMetaString) { boolean } 
                        3 0x0006: (ConfigScriptMetaString) { default } 
                        4 0x0001: (CStorageValue) { 0 } 
                        5 0x0006: (ConfigScriptMetaString) { ui } 
                        6 0x0007: (ConfigScriptMetaContainer) [2] { 
                                0 0x0060: (CStorageValue) { 1 } 
                                1 0x0006: (ConfigScriptMetaString) { cbUnlighted } } } } 
        7 0x0007: (ConfigScriptMetaContainer) [3] { 
                0 0x0060: (CStorageValue) { 2 } 
                1 0x0006: (ConfigScriptMetaString) { bStainedGlassWindow } 
                2 0x0007: (ConfigScriptMetaContainer) [7] { 
                        0 0x0060: (CStorageValue) { 6 } 
                        1 0x0006: (ConfigScriptMetaString) { type } 
                        2 0x0006: (ConfigScriptMetaString) { boolean } 
                        3 0x0006: (ConfigScriptMetaString) { default } 
                        4 0x0001: (CStorageValue) { 0 } 
                        5 0x0006: (ConfigScriptMetaString) { ui } 
                        6 0x0007: (ConfigScriptMetaContainer) [2] { 
                                0 0x0060: (CStorageValue) { 1 } 
                                1 0x0006: (ConfigScriptMetaString) { cbStainedGlassWindow } } } }

Most other blocks in this file seem to contain value sets where the type is fixed to the id, and the id is basically the name of the config value. Right now, there doesn't seem to be anything in there that interests me, so I won't bother with them too much, but here's an example of one simplified anyways.

2 0x20a0: (Config20a0) [2] { 
        0 0x0100: (CStorageValue) { 1 } 
        1 0x0110: (Config20a0Entry) [25] { 
                0 0x0100: (CStorageValue) { 220 } 
                1 0x0110: (CStorageValue) { 0 } 
                2 0x0120: (CStorageValue) { 1 } 
                3 0x0130: (CStorageValue) { 0 } 
                4 0x0140: (CStorageValue) { 0 } 
                5 0x0150: (CStorageValue) { 0 } 
                6 0x0160: (CStorageValue) { 1 } 
                7 0x0161: (CStorageValue) { 1 } 
                8 0x0170: (CStorageValue) { 1 } 
                9 0x0180: (CStorageValue) { 0 } 
                10 0x0190: (CStorageValue) { 0 } 
                11 0x0200: (CStorageValue) { 0 } 
                12 0x0210: (CStorageValue) { 0 } 
                13 0x0220: (CStorageValue) { 994352038 } 
                14 0x0230: (CStorageValue) { 1041059807 } 
                15 0x0240: (CStorageValue) { 266338296 } 
                16 0x0250: (CStorageValue) { 131008 } 
                17 0x0260: (CStorageValue) { 0 } 
                18 0x0270: (CStorageValue) { 1 } 
                19 0x0280: (CStorageValue) { 0 } 
                20 0x0310: (CStorageValue) { 0 } 
                21 0x0290: (CStorageValue) {  } 
                22 0x0390: (CStorageValue) { default } 
                23 0x0300: (StorageContainer) [1] { 
                        0 0x0100: (StorageRaw) { 
                                Size: 4
                                String: .... 
                                Hex: 00 00 00 00 } } 
                24 0x0330: (StorageRaw) { 
                        Size: 16
                        String: ................ } } }

Quite boring, right?

Next up is the long awaited Scene.

no trackback

3ds Max File Format (Part 2: The first inner structures; DllDirectory, ClassDirectory3)

By Kaetemi on Sunday 19 August 2012, 10:22 - Articles

Now that we understand the outer structure of the file, it's time to look closer to what's inside. The DllDirectory stream looks like a good starting point. After cleaning up a whole bunch of code to make it easier to plug in specialized handling code, a nice and readable output of this structure shows up as follows:

DllDirectory
(StorageContainer) [20] { 
0 0x21c0: (StorageRaw) { 
        Size: 4
        String: .... 
        Hex: d8 00 e0 2e } 
1 0x2038: (StorageContainer) [2] { 
        0 0x2039: (StorageRaw) { 
                Size: 78
                String: V.i.e.w.p.o.r.t. .M.a.n.a.g.e.r. .f.o.r. .D.i.r.e.c.t.X. .(.A.u.t.o.d.e.s.k.). } 
        1 0x2037: (StorageRaw) { 
                Size: 38
                String: V.i.e.w.p.o.r.t.M.a.n.a.g.e.r...g.u.p. } } 
2 0x2038: (StorageContainer) [2] { 
        0 0x2039: (StorageRaw) { 
                Size: 98
                String: m.e.n.t.a.l. .r.a.y.:. .M.a.t.e.r.i.a.l. .C.u.s.t.o.m. .A.t.t.r.i.b.u.t.e.s. .(.A.u.t.o.d.e.s.k.). } 
        1 0x2037: (StorageRaw) { 
                Size: 42
                String: m.r.M.a.t.e.r.i.a.l.A.t.t.r.i.b.s...g.u.p. } } 
...
19 0x2038: (StorageContainer) [2] { 
        0 0x2039: (StorageRaw) { 
                Size: 54
                String: B.i.p.e.d. .C.o.n.t.r.o.l.l.e.r. .(.A.u.t.o.d.e.s.k.). } 
        1 0x2037: (StorageRaw) { 
                Size: 18
                String: b.i.p.e.d...d.l.c. } } }

Thanks to the chunks, most of this is self-explanatory, and fairly easy to handle. The 0x21c0 chunk seems to be a header chunk for the DllDirectory, so we can call it DllHeader or something similar, and contains 4 bytes. This chunk is found in a version 2010 file, but doesn't seem to exist in files from version 3, so it's probably not crucial to handle and we can ignore it's contents until it seems we need it for something. The rest of the chunks in this container are all of id 0x2038, and are the entries in this list, so they are called DllEntry. Inside each of these, there is a 0x2039 chunk containing a description, and a 0x2037 chunk containing a name, both in UTF-16.

The meaning of chunk identifiers depends on the parent container chunk, so we have to code it like that as well. Each container type is set up with a handler to create the class that handles a chunk of a specified identifier. Parsing this block with some smarter code results in a data reading that looks as follows:

(DllDirectory) [20] { 
0 0x21c0: (CStorageValue) { 786432216 } 
1 0x2038: (DllEntry) [2] { 
        0 0x2039: (CStorageValue) { Viewport Manager for DirectX (Autodesk) } 
        1 0x2037: (CStorageValue) { ViewportManager.gup } } 
2 0x2038: (DllEntry) [2] { 
        0 0x2039: (CStorageValue) { mental ray: Material Custom Attributes (Autodesk) } 
        1 0x2037: (CStorageValue) { mrMaterialAttribs.gup } } 
...
19 0x2038: (DllEntry) [2] { 
        0 0x2039: (CStorageValue) { Biped Controller (Autodesk) } 
        1 0x2037: (CStorageValue) { biped.dlc } } }

Next, we do a similar thing for the ClassDirectory3 stream.

ClassDirectory3
(StorageContainer) [57] { 
0 0x2040: (StorageContainer) [2] { 
        0 0x2060: (StorageRaw) { 
                Size: 16
                String: ................ 
                Hex: ff ff ff ff 82 00 00 00 00 00 00 00 82 00 00 00 } 
        1 0x2042: (StorageRaw) { 
                Size: 22
                String: P.a.r.a.m.B.l.o.c.k.2. } } 
1 0x2040: (StorageContainer) [2] { 
        0 0x2060: (StorageRaw) { 
                Size: 16
                String: ....<).Z..B0`... 
                Hex: 00 00 00 00 3c 29 06 5a 1e 0c 42 30 60 11 00 00 } 
        1 0x2042: (StorageRaw) { 
                Size: 30
                String: V.i.e.w.p.o.r.t.M.a.n.a.g.e.r. } } 
...
8 0x2040: (StorageContainer) [2] { 
        0 0x2060: (StorageRaw) { 
                Size: 16
                String: ................ 
                Hex: 03 00 00 00 02 00 00 00 00 00 00 00 00 0c 00 00 } 
        1 0x2042: (StorageRaw) { 
                Size: 16
                String: S.t.a.n.d.a.r.d. } } 
...
13 0x2040: (StorageContainer) [2] { 
        0 0x2060: (StorageRaw) { 
                Size: 16
                String: ....._.d..+".... 
                Hex: fe ff ff ff ec 5f c7 64 b9 9e 2b 22 00 0c 00 00 } 
        1 0x2042: (StorageRaw) { 
                Size: 24
                String: N.e.L. .M.a.t.e.r.i.a.l. } } 
...
56 0x2040: (StorageContainer) [2] { 
        0 0x2060: (StorageRaw) { 
                Size: 16
                String: ...."".......... 
                Hex: ff ff ff ff 22 22 00 00 00 00 00 00 00 01 00 00 } 
        1 0x2042: (StorageRaw) { 
                Size: 10
                String: S.c.e.n.e. } } }

This container does not seem to have a header chunk, but again simply contains a whole bunch of entries of id 0x2040, containing a binary blob with id 0x2060 and a UTF-16 string with id 0x2042 that has a description. There's a block in here with some data that I can recognize and reference from our own code. The NeL Material, which is a MAXScript, has a class id of (0x64c75fec, 0x222b9eb9) which matches the middle 8 bytes of the 16 byte blob (read them backwards). The last four bytes in the blob match with the last four bytes in the Standard (material) class entry, and appear to be the SuperClassID. When we look closer at the first four bytes, this appears to be a signed integer, given that there's both ff ff ff as 00 00 00 numbers without too much inbetween. For the NeL Material, which is a script, this value is -2, cross-referencing with other max files with scripted classes reveals the same. Builtin types, such as Scene, have this number as -1. Classes that come from plugins, such as ViewPortManager, have a positive value. Even closer inspection reveals that this value matches with the index of the associated dll in the DllDirectory, ViewPortManager being part of ViewPortManager.gup, and Standard being part of mtl.dlt. It can be expected that the indices of the classes in this list will be needed later on as well. A smarter parsing output looks as follows:

(ClassDirectory3) [57] { 
0 0x2040: (ClassEntry) [2] { 
        0 0x2060: (ClassDirectoryHeader) { 
                DllIndex: -1
                ClassID: (0x00000000, 0x00000082)
                SuperClassID: 130 } 
        1 0x2042: (CStorageValue) { ParamBlock2 } } 
1 0x2040: (ClassEntry) [2] { 
        0 0x2060: (ClassDirectoryHeader) { 
                DllIndex: 0
                ClassID: (0x30420c1e, 0x5a06293c)
                SuperClassID: 4448 } 
        1 0x2042: (CStorageValue) { ViewportManager } } 
...
56 0x2040: (ClassEntry) [2] { 
        0 0x2060: (ClassDirectoryHeader) { 
                DllIndex: -1
                ClassID: (0x00000000, 0x00002222)
                SuperClassID: 256 } 
        1 0x2042: (CStorageValue) { Scene } } }

The ClassData stream is very similar, and seems to contain a global data storage for classes, or something in that style. It doesn't seem to have anything in it that interests me or seems crucial at this point, so I won't bother with it too much for now. It's fairly self-explanatory.

(ClassData) [7] { 
0 0x2100: (ClassDataEntry) [2] { 
        0 0x2110: (ClassDataHeader) { 
                ClassID: (0xbe7c7e52, 0x87d987f4)
                SuperClassID: 16 } 
        1 0x2120: (StorageRaw) { 
                Size: 0
                String:  
                Hex: } } 
...
4 0x2100: (ClassDataEntry) [2] { 
        0 0x2110: (ClassDataHeader) { 
                ClassID: (0x33b673a4, 0x44b50d1e)
                SuperClassID: 4128 } 
        1 0x2120: (StorageContainer) [14] { 
                0 0x0190: (StorageRaw) { 
                        Size: 48
                        String: [email protected]@...= 
                        Hex: 00 00 00 00 00 00 00 00 1f 1c c1 c3 01 00 00 00 cd cc cc 3d cd cc cc 3d 00 00 00 00 cf f7 7b 40 e1 7a 1d 42 01 00 00 00 00 00 a0 40 cd cc cc 3d } 
                1 0x019c: (StorageRaw) { 
                        Size: 72
                        String: [email protected]@[email protected][email protected]=..........HC 
                        Hex: 00 00 00 00 00 00 00 00 1f 1c c1 c3 01 00 00 00 00 00 80 3f 00 00 a0 40 00 00 00 00 cf f7 7b 40 e1 7a 1d 42 01 00 00 00 00 00 a0 40 cd cc cc 3d cd cc cc 3d 00 40 9c 45 cd cc cc 3d 01 00 00 00 01 00 00 00 00 00 48 43 } 
...

So far, this was easy. After this comes the real stuff.

no trackback

Friday 17 August 2012

3ds Max File Format (Part 1: The outer file format; OLE2)

By Kaetemi on Friday 17 August 2012, 20:45 - Articles

The 3ds Max file format, not too much documentation to be found about it. There are some hints here and there about how it's built up, but there exists no central documentation on it.

Right now we are in the following situation. A few thousand of max files, created by a very old version of max (3.x), containing path references to textures and other max files that have been renamed and relocated or which simply no longer exist. Yes, we have a maxscript that can go through them all, and that manages to fix a large number of paths. However, there are a lot of paths that are stored as part as fields in plugins and material scripts that don't get noticed, and the performance of opening and closing this number of files from 3ds Max directly is horrible. The obvious solution? Figure out how we can read and save the max file with modified contents, without having to understand all of the actual data it contains. Fortunately, this is actually possible without too much work.

Some research online brings up the following blog post, relating to a change in the max file format in version 2010, which would make it easier to update asset paths: http://www.the-area.com/blogs/chris/reading_and_modifying_asset_file_paths_in_the_3ds_max_file. That's nice and all, but it's only from version 2010 on, and it very likely won't contain any assets referenced by path by old plugins and such.

So, starting at the beginning. The blog post I referred to above nicely hints us to the OLE structured file format. Since there exist a wide range of implementations for that, we can pretty much skip that, and accept that it's basically a filesystem in a file, so it's a file containing multiple file streams. A reliable open source implementation of this container format can be found in libgsf. When scanning a fairly recent max file, using the command gsf list, we can find the following streams inside this file:

f         52 VideoPostQueue
f     147230 Scene
f        366 FileAssetMetaData2
f       2198 DllDirectory
f      29605 Config
f       3438 ClassDirectory3
f        691 ClassData
f      29576 SummaryInformation
f       2320 DocumentSummaryInformation

The FileAssetMetaData2 is new in 3ds Max 2010.

One step further, we can start examining the contents of these streams. And it's usually easiest to start off with one of the more simple ones. VideoPostQueue seems small enough to figure out the overall logic of the file format, hoping that the rest is serialized in a similar way. Using the command gsf dump we can get a hex output of one of the streams, and using a simple text editor we can find how it's structured. Binary formats often contain 32 bit length values, which are usually easy to spot in small files, since they'll contain a large number of 00 values. It's basically a matter of finding possible 32bit length integers, and matching them together with various fixed length fields and other typical binary file contents, until something programatically logical turns up. Here's a manually parsed VideoPostQueue storage stream:

[
        50 00 (id: 0x0050)
        0a 00 00 00 (size: 10 - 6 = 4)
        [
                01 00 00 00 (value: 1)
        ]
]
[
        60 00 (id: 0x0060)
        2a 00 00 80 (size: 42 - 6 = 36) (note: negative bit = container)
        [
                10 00 (id: 0x0010)
                1e 00 00 00 (size: 30 - 6 = 24)
                [
                        07 00 00 00 (value: 7)
                        01 00 00 00 (value: 1)
                        00 00 00 00
                        00 00 00 00
                        20 12 00 00 (value: 4610)
                        00 00 00 00
                ]
                20 00 (id: 0x0020)
                06 00 00 00 (size: 6 - 6 = 0)
        ]
]

The storage streams in the max container file contain a fairly simple chunk based file format (and in fact similar in format to the fairly well known .3ds file format). Being based on chunks is what allows 3ds Max to open a file for which certain plugins are missing. It's basically a tree structured format where every entry has an identifier and a size, so when an identifier is unknown, or when it's contents are incompatible, it can simply be kept as is or discarded. The only exceptions in the file that don't use this structure are SummaryInformation and DocumentSummaryInformation, which are supposedly in a standard Windows format, and the new FileAssetMetaData2 section is formatted differently as well unfortunately.

In this format, the chunk header consists of a 2-byte unsigned integer which is the identifier, and a 4-byte unsigned integer, where the 31 least significant bits are the size and the msb is a flag that helpfully lets us know if the chunk itself contains more chunks, and thus is a container, or not. For very large files, where 31 bits is insufficient for the size, the entire size field is set to 0, and the header increases with an additional 64-bit unsigned integer field which is similarly structured as the 32-bit size field. The size field includes the size of the header.

       0 | 0f 20 (id)
                 00 00 00 00 (size missing)
                             17 fe 01 00 00 00 00 80 (size in 64 bits)

With this information it is possible to read a max file, modify the binary contents of chunks (most of them are fairly basic of format), and we should be able to re-save the max file with our modified data. The DllDirectory section, for example, parsed programatically starts like this:

CStorageContainer - items: 20
        [0x21C0] CStorageValue - bytes: 4
        786432216
        [0x2038] CStorageContainer - items: 2
                [0x2039] CStorageUCString - length: 39
                Viewport Manager for DirectX (Autodesk)
                [0x2037] CStorageUCString - length: 19
                ViewportManager.gup
        [0x2038] CStorageContainer - items: 2
                [0x2039] CStorageUCString - length: 49
                mental ray: Material Custom Attributes (Autodesk)
                [0x2037] CStorageUCString - length: 21
                mrMaterialAttribs.gup
        [0x2038] CStorageContainer - items: 2
                [0x2039] CStorageUCString - length: 37
                Custom Attribute Container (Autodesk)
                [0x2037] CStorageUCString - length: 23
                CustAttribContainer.dlo
...

Of course, it would be interesting if we could go further, and directly manipulate the parameters of our own plugins and scripts from our own tools back into the max files so that everything is centrally stored without any duplicate source data in the way. And that's exactly what I'll be doing next.

2 comments no trackback

Wednesday 18 November 2009

Camera tracking using manually placed tracking points (Autodesk MatchMover)

By Kaetemi on Wednesday 18 November 2009, 17:27 - Tutorials

Originally written and published in Dutch on the DaeXperience forums.

While fully automated tracking may be tempting, there's a good chance that often manually placing and verifying all of the tracking points on your footage may turn out to be much less frustrating, as you do not have to mask out any moving objects, and you have full control and responsibility over all the tracks. Take roughly four hours trial and error for a reliable finished tracking of a short video cut.

Continue reading...

3 comments no trackback

Tuesday 10 November 2009

Particle Playground

By Kaetemi on Tuesday 10 November 2009, 02:00 - Video

Particle Playground created with 3ds Max 2010 for the Digital Arts and Entertainment 3D5 particles assignment.


"Particle Playground" made with 3ds Max (3D Particle Effects)

one comment no trackback

Wednesday 14 October 2009

Camera Tracking Test

By Kaetemi on Wednesday 14 October 2009, 18:35 - Video

First test of camera tracking as part of a Digital Arts and Entertainment Video Production assignment. Tracking was done with Autodesk MatchMover. The box and cylinder wires are just for show and have no real use whatsoever except for testing stuff.


"Camera Tracking Test" tracked in MatchMover (3D Camera Tracking)

no trackback