NeL Files and Serialisation
Version 1 (kervala, 09/29/2010 10:13 pm)
1 | 1 | kervala | h1. NeL Files and Serialisation |
---|---|---|---|
2 | 1 | kervala | |
3 | 1 | kervala | h2. Introduction |
4 | 1 | kervala | |
5 | 1 | kervala | This is really quite a difficult subject to write about - so this file is an introduction which describes the basic features and principles of our system. |
6 | 1 | kervala | |
7 | 1 | kervala | h2. How files work in NeL |
8 | 1 | kervala | |
9 | 1 | kervala | The NeL files are NOT designed to be man-readable. Interpretation and generation of file contents is performed by the objects that are to be read and written using a standardised mechanism. This mechanism was inspired by the system provided by Java. |
10 | 1 | kervala | |
11 | 1 | kervala | We use the term 'serialisable' to describe a class that can be read from/ written to a NeL data file. |
12 | 1 | kervala | Counter-intuitive as it may, at first, appear, each 'serialisable' class supplies a single method that is used for both reading and writing. |
13 | 1 | kervala | |
14 | 1 | kervala | Note that the files are encoded in little-endian and that the NeL library code deals with conversion of endian-ness for big-endian platforms. |
15 | 1 | kervala | |
16 | 1 | kervala | h2. Serialisation beyond files |
17 | 1 | kervala | |
18 | 1 | kervala | The serialisation system can be used for generating binary data buffers in memory (without writing the result to a file) or for packing and unpacking data for transfer over a LAN. |
19 | 1 | kervala | |
20 | 1 | kervala | h2. How it works |
21 | 1 | kervala | |
22 | 1 | kervala | Technically, we define a 'serialisable' class as a class that can be passed to @IStream::serial()@. |
23 | 1 | kervala | |
24 | 1 | kervala | In order for a class to be serialisable it is sufficient for it to include the following method: |
25 | 1 | kervala | |
26 | 1 | kervala | <pre><code class="CPP">void serial(IStream&)</code></pre> |
27 | 1 | kervala | |
28 | 1 | kervala | The fact that we use a template method definition means that a serialisable class does not have to be derived from any other class. |
29 | 1 | kervala | |
30 | 1 | kervala | All standard types are serialisable due to a non-template prototypes shown below. |
31 | 1 | kervala | STL containers of serialisable types are serialisable. |
32 | 1 | kervala | Pointers to non-polymorphic serialisable types are serialisable. |
33 | 1 | kervala | |
34 | 1 | kervala | The IStream class definition looks something like this: |
35 | 1 | kervala | |
36 | 1 | kervala | <pre><code class="CPP"> |
37 | 1 | kervala | class IStream |
38 | 1 | kervala | { |
39 | 1 | kervala | ... |
40 | 1 | kervala | void serial (int&); |
41 | 1 | kervala | void serial (float&); |
42 | 1 | kervala | ... |
43 | 1 | kervala | template <class T> void serial (T&t) |
44 | 1 | kervala | { |
45 | 1 | kervala | t.serial (*this); |
46 | 1 | kervala | } |
47 | 1 | kervala | }; |
48 | 1 | kervala | </code></pre> |
49 | 1 | kervala | |
50 | 1 | kervala | Example: |
51 | 1 | kervala | |
52 | 1 | kervala | To make the following class serialisable: |
53 | 1 | kervala | |
54 | 1 | kervala | <pre><code class="CPP"> |
55 | 1 | kervala | class CMyFirstClass |
56 | 1 | kervala | { |
57 | 1 | kervala | uint32 a, b; |
58 | 1 | kervala | }; |
59 | 1 | kervala | </code></pre> |
60 | 1 | kervala | |
61 | 1 | kervala | you would need to extend the class as follows: |
62 | 1 | kervala | |
63 | 1 | kervala | <pre><code class="CPP"> |
64 | 1 | kervala | class CMyFirstClass |
65 | 1 | kervala | { |
66 | 1 | kervala | uint23 a, b; |
67 | 1 | kervala | void serial (IStream &istream) |
68 | 1 | kervala | { |
69 | 1 | kervala | istream.serial(a); |
70 | 1 | kervala | istream.serial(b); |
71 | 1 | kervala | } |
72 | 1 | kervala | }; |
73 | 1 | kervala | </code></pre> |
74 | 1 | kervala | |
75 | 1 | kervala | The following example shows how to serialise a more complicated data structure: |
76 | 1 | kervala | |
77 | 1 | kervala | <pre><code class="CPP"> |
78 | 1 | kervala | class CMyFirstClass |
79 | 1 | kervala | { |
80 | 1 | kervala | void serial (IStream &); |
81 | 1 | kervala | }; |
82 | 1 | kervala | |
83 | 1 | kervala | class CMyclass |
84 | 1 | kervala | { |
85 | 1 | kervala | uint32 BaseType; |
86 | 1 | kervala | CMyFirstClass SerialisableClass |
87 | 1 | kervala | std::vector< myFirstClass> STLContainerOfSerialisableClass; |
88 | 1 | kervala | CMyFirstClass *PointerToSerialisableClass; |
89 | 1 | kervala | std::vector< myFirstClass*> STLContainerOfPointersToSerialisableClass; |
90 | 1 | kervala | |
91 | 1 | kervala | void serial (IStream &istream) |
92 | 1 | kervala | { |
93 | 1 | kervala | istream.serial(BaseType); |
94 | 1 | kervala | istream.serial(SerialisableClass); |
95 | 1 | kervala | istream.serialCont(STLContainerOfSerialisableClass); |
96 | 1 | kervala | istream.serialPtr(PointerToSerialisableClass); |
97 | 1 | kervala | istream.serialContPtr(STLContainerOfPointersToSerialisableClass); |
98 | 1 | kervala | } |
99 | 1 | kervala | }; |
100 | 1 | kervala | </code></pre> |
101 | 1 | kervala | |
102 | 1 | kervala | h2. Dealing with cross referenced or hierarchical data |
103 | 1 | kervala | |
104 | 1 | kervala | If an object contains a pointer to another object in memory then the serialPtr() method is used to read/write the referenced object. |
105 | 1 | kervala | |
106 | 1 | kervala | The NeL library code writes a value corresponding to the pointer to the serialised data, followed by the data that the pointer points to (In the case of a NULL pointer the value 0 is written without any following data). |
107 | 1 | kervala | |
108 | 1 | kervala | The NeL library code automatically deals with the cases where two or more objects reference the same object or there is a circular reference. Each time a pointer is de-referenced, for writing, NeL checks against a table of previous pointers; if the pointer value already exists in the table then no data is written. At read time the data structures are faithfully reconstructed. |
109 | 1 | kervala | |
110 | 1 | kervala | h2. Dealing with polymorphism within cross referenced data |
111 | 1 | kervala | |
112 | 1 | kervala | In a nut shell, in order to un-serialise a data record that one only has an interface type for, one needs to store an additional identifier with the data record that identifies its real type. The mechanism for doing this is best shown with an example: |
113 | 1 | kervala | |
114 | 1 | kervala | <pre><code class="CPP"> |
115 | 1 | kervala | class IBaseClass : public IStreamable |
116 | 1 | kervala | { |
117 | 1 | kervala | // This class is an interface. It is polymorphic. |
118 | 1 | kervala | virtual void foo () = 0; |
119 | 1 | kervala | |
120 | 1 | kervala | // It must declare its name |
121 | 1 | kervala | NLMISC_DECLARE_CLASS(MyClass); |
122 | 1 | kervala | }; |
123 | 1 | kervala | |
124 | 1 | kervala | class CClassToSerialise |
125 | 1 | kervala | { |
126 | 1 | kervala | IBaseClass *PointerToAPolymorphicClass; |
127 | 1 | kervala | |
128 | 1 | kervala | void serial (IStream& s) |
129 | 1 | kervala | { |
130 | 1 | kervala | s.serialPolyPtr (PointerToAPolymorphicClass); |
131 | 1 | kervala | } |
132 | 1 | kervala | }; |
133 | 1 | kervala | |
134 | 1 | kervala | void main () |
135 | 1 | kervala | { |
136 | 1 | kervala | ... |
137 | 1 | kervala | // The polymorphic class must be registered in the registry |
138 | 1 | kervala | NLMISC_REGISTER_CLASS (MyClass); |
139 | 1 | kervala | ... |
140 | 1 | kervala | } |
141 | 1 | kervala | </code></pre> |
142 | 1 | kervala | |
143 | 1 | kervala | h2. Dealing with file format evolution |
144 | 1 | kervala | |
145 | 1 | kervala | <pre><code class="CPP"> |
146 | 1 | kervala | void serial (IStream &s) |
147 | 1 | kervala | { |
148 | 1 | kervala | // At the beginning of the serial process, read/write the version number of the class implementation |
149 | 1 | kervala | |
150 | 1 | kervala | // In the following example - at read time 'version' contains the version read from the stream. At |
151 | 1 | kervala | // write time version code '3' is written to the stream and to the variable 'version'. |
152 | 1 | kervala | uint32 version = s.serialVersion (3); |
153 | 1 | kervala | |
154 | 1 | kervala | // Now switch the version |
155 | 1 | kervala | switch (version) |
156 | 1 | kervala | { |
157 | 1 | kervala | case 3: |
158 | 1 | kervala | // The last field added in the class |
159 | 1 | kervala | s.serial (LastField); |
160 | 1 | kervala | |
161 | 1 | kervala | // do some different stuff at read time and write time |
162 | 1 | kervala | if (s.isReading()) |
163 | 1 | kervala | { |
164 | 1 | kervala | // at read time |
165 | 1 | kervala | ... |
166 | 1 | kervala | } |
167 | 1 | kervala | else |
168 | 1 | kervala | { |
169 | 1 | kervala | // at write time |
170 | 1 | kervala | ... |
171 | 1 | kervala | } |
172 | 1 | kervala | |
173 | 1 | kervala | case 2: |
174 | 1 | kervala | // note that the code provided as of here allows for the reading of old versions of the class |
175 | 1 | kervala | s.serial (Toto); |
176 | 1 | kervala | |
177 | 1 | kervala | // in the case where the evolution from my version 1 implementation to my version 2 |
178 | 1 | kervala | // is not simply an extension of version 1 we need to break execution here |
179 | 1 | kervala | break; |
180 | 1 | kervala | |
181 | 1 | kervala | case 1: |
182 | 1 | kervala | s.serial (Foo); |
183 | 1 | kervala | case 0: |
184 | 1 | kervala | s.serial (Truc); |
185 | 1 | kervala | } |
186 | 1 | kervala | } |
187 | 1 | kervala | </code></pre> |
188 | 1 | kervala | |
189 | 1 | kervala | h2. NeL File Headers |
190 | 1 | kervala | |
191 | 1 | kervala | The objective of NeL file headers is to verify that a file is in the right format before attempting to interpret the contents. |
192 | 1 | kervala | |
193 | 1 | kervala | <pre><code class="CPP"> |
194 | 1 | kervala | // The NeL team use the following advise serialise a file this way: |
195 | 1 | kervala | void CFileRootClass::serial (IStream &s) |
196 | 1 | kervala | { |
197 | 1 | kervala | // First write / read-check the header |
198 | 1 | kervala | s.serialCheck ((uint32)'_LEN'); |
199 | 1 | kervala | s.serialCheck ((uint32)'HSEM'); |
200 | 1 | kervala | |
201 | 1 | kervala | // This code write / read-check the header 'NEL_MESH' at the beginning of the file. |
202 | 1 | kervala | // If the check fails, serialCheck throws the EInvalidDataStream exception. |
203 | 1 | kervala | } |
204 | 1 | kervala | </code></pre> |
205 | 1 | kervala | |
206 | 1 | kervala | Good examples to look at: |
207 | 1 | kervala | |
208 | 1 | kervala | <pre> |
209 | 1 | kervala | include/nel/misc/stream.h // Stream base classes |
210 | 1 | kervala | class CTileBank in src/3d/tile_bank.cpp // Good example of file format evolution |
211 | 1 | kervala | class CAnimation in src/3d/animation.cpp // Good example of polymorphism |
212 | 1 | kervala | </pre> |