NeL Files and Serialisation

Version 1 (kervala, 09/29/2010 10:13 pm)

1 1 kervala
h1. NeL Files and Serialisation
2 1 kervala
3 1 kervala
h2. Introduction
4 1 kervala
5 1 kervala
This is really quite a difficult subject to write about - so this file is an introduction which describes the basic features and principles of our system.  
6 1 kervala
7 1 kervala
h2. How files work in NeL
8 1 kervala
9 1 kervala
The NeL files are NOT designed to be man-readable. Interpretation and generation of file contents is performed by the objects that are to be read and written using a standardised mechanism. This mechanism was inspired by the system provided by Java.
10 1 kervala
11 1 kervala
We use the term 'serialisable' to describe a class that can be read from/ written to a NeL data file.
12 1 kervala
Counter-intuitive as it may, at first, appear, each 'serialisable' class supplies a single method that is used for both reading and writing.
13 1 kervala
14 1 kervala
Note that the files are encoded in little-endian and that the NeL library code deals with conversion of endian-ness for big-endian platforms.
15 1 kervala
16 1 kervala
h2. Serialisation beyond files
17 1 kervala
18 1 kervala
The serialisation system can be used for generating binary data buffers in memory (without writing the result to a file) or for packing and unpacking data for transfer over a LAN.
19 1 kervala
20 1 kervala
h2. How it works
21 1 kervala
22 1 kervala
Technically, we define a 'serialisable' class as a class that can be passed to @IStream::serial()@.
23 1 kervala
24 1 kervala
In order for a class to be serialisable it is sufficient for it to include the following method: 
25 1 kervala
26 1 kervala
<pre><code class="CPP">void serial(IStream&)</code></pre>
27 1 kervala
28 1 kervala
The fact that we use a template method definition means that a serialisable class does not have to be derived from any other class.
29 1 kervala
30 1 kervala
All standard types are serialisable due to a non-template prototypes shown below.
31 1 kervala
STL containers of serialisable types are serialisable.
32 1 kervala
Pointers to non-polymorphic serialisable types are serialisable.
33 1 kervala
34 1 kervala
The IStream class definition looks something like this:
35 1 kervala
36 1 kervala
<pre><code class="CPP">
37 1 kervala
class IStream
38 1 kervala
{
39 1 kervala
...
40 1 kervala
	void serial (int&);
41 1 kervala
	void serial (float&);
42 1 kervala
...
43 1 kervala
	template <class T> void serial (T&t)
44 1 kervala
	{
45 1 kervala
		t.serial (*this);
46 1 kervala
	}
47 1 kervala
};
48 1 kervala
</code></pre>
49 1 kervala
50 1 kervala
Example:
51 1 kervala
52 1 kervala
To make the following class serialisable:
53 1 kervala
54 1 kervala
<pre><code class="CPP">
55 1 kervala
class CMyFirstClass
56 1 kervala
{
57 1 kervala
	uint32 a, b;
58 1 kervala
};
59 1 kervala
</code></pre>
60 1 kervala
61 1 kervala
you would need to extend the class as follows:
62 1 kervala
63 1 kervala
<pre><code class="CPP">
64 1 kervala
class CMyFirstClass
65 1 kervala
{
66 1 kervala
	uint23 a, b;
67 1 kervala
	void serial (IStream &istream)
68 1 kervala
	{
69 1 kervala
		istream.serial(a);
70 1 kervala
		istream.serial(b);
71 1 kervala
	}
72 1 kervala
};
73 1 kervala
</code></pre>
74 1 kervala
75 1 kervala
The following example shows how to serialise a more complicated data structure:
76 1 kervala
77 1 kervala
<pre><code class="CPP">
78 1 kervala
class CMyFirstClass
79 1 kervala
{
80 1 kervala
	void serial (IStream &);
81 1 kervala
};
82 1 kervala
83 1 kervala
class CMyclass
84 1 kervala
{
85 1 kervala
	uint32					BaseType;
86 1 kervala
	CMyFirstClass				SerialisableClass
87 1 kervala
	std::vector< myFirstClass>		STLContainerOfSerialisableClass;
88 1 kervala
	CMyFirstClass				*PointerToSerialisableClass;
89 1 kervala
	std::vector< myFirstClass*>		STLContainerOfPointersToSerialisableClass;
90 1 kervala
91 1 kervala
	void serial (IStream &istream)
92 1 kervala
	{
93 1 kervala
		istream.serial(BaseType);
94 1 kervala
		istream.serial(SerialisableClass);
95 1 kervala
		istream.serialCont(STLContainerOfSerialisableClass);
96 1 kervala
		istream.serialPtr(PointerToSerialisableClass);
97 1 kervala
		istream.serialContPtr(STLContainerOfPointersToSerialisableClass);
98 1 kervala
	}
99 1 kervala
};
100 1 kervala
</code></pre>
101 1 kervala
102 1 kervala
h2. Dealing with cross referenced or hierarchical data
103 1 kervala
104 1 kervala
If an object contains a pointer to another object in memory then the serialPtr() method is used to read/write the referenced object.
105 1 kervala
106 1 kervala
The NeL library code writes a value corresponding to the pointer to the serialised data, followed by the data that the pointer points to (In the case of a NULL pointer the value 0 is written without any following data).
107 1 kervala
108 1 kervala
The NeL library code automatically deals with the cases where two or more objects reference the same object or there is a circular reference.  Each time a pointer is de-referenced, for writing, NeL checks against a table of previous pointers; if the pointer value already exists in the table then no data is written.  At read time the data structures are faithfully reconstructed.
109 1 kervala
110 1 kervala
h2. Dealing with polymorphism within cross referenced data
111 1 kervala
112 1 kervala
In a nut shell, in order to un-serialise a data record that one only has an interface type for, one needs to store an additional identifier with the data record that identifies its real type.  The mechanism for doing this is best shown with an example:
113 1 kervala
114 1 kervala
<pre><code class="CPP">
115 1 kervala
class IBaseClass : public IStreamable
116 1 kervala
{
117 1 kervala
	// This class is an interface. It is polymorphic.
118 1 kervala
	virtual void foo () = 0;
119 1 kervala
120 1 kervala
	// It must declare its name
121 1 kervala
	NLMISC_DECLARE_CLASS(MyClass);
122 1 kervala
};
123 1 kervala
124 1 kervala
class CClassToSerialise
125 1 kervala
{
126 1 kervala
	IBaseClass *PointerToAPolymorphicClass;
127 1 kervala
128 1 kervala
	void serial (IStream& s)
129 1 kervala
	{
130 1 kervala
		s.serialPolyPtr (PointerToAPolymorphicClass);
131 1 kervala
	}
132 1 kervala
};
133 1 kervala
134 1 kervala
void main ()
135 1 kervala
{
136 1 kervala
	...
137 1 kervala
	// The polymorphic class must be registered in the registry
138 1 kervala
	NLMISC_REGISTER_CLASS (MyClass);
139 1 kervala
	...
140 1 kervala
}
141 1 kervala
</code></pre>
142 1 kervala
143 1 kervala
h2. Dealing with file format evolution
144 1 kervala
145 1 kervala
<pre><code class="CPP">
146 1 kervala
void serial (IStream &s)
147 1 kervala
{
148 1 kervala
	// At the beginning of the serial process, read/write the version number of the class implementation
149 1 kervala
150 1 kervala
	// In the following example - at read time 'version' contains the version read from the stream. At
151 1 kervala
	// write time version code '3' is written to the stream and to the variable 'version'.
152 1 kervala
	uint32 version = s.serialVersion (3);
153 1 kervala
154 1 kervala
	// Now switch the version
155 1 kervala
	switch (version)
156 1 kervala
	{
157 1 kervala
	case 3:
158 1 kervala
		// The last field added in the class
159 1 kervala
		s.serial (LastField);
160 1 kervala
161 1 kervala
		// do some different stuff at read time and write time
162 1 kervala
		if (s.isReading())
163 1 kervala
		{
164 1 kervala
			// at read time
165 1 kervala
			...
166 1 kervala
		}
167 1 kervala
		else
168 1 kervala
		{
169 1 kervala
			// at write time
170 1 kervala
			...
171 1 kervala
		}
172 1 kervala
173 1 kervala
	case 2:
174 1 kervala
		// note that the code provided as of here allows for the reading of old versions of the class
175 1 kervala
		s.serial (Toto);
176 1 kervala
177 1 kervala
		// in the case where the evolution from my version 1 implementation to my version 2
178 1 kervala
		// is not simply an extension of version 1 we need to break execution here
179 1 kervala
		break;
180 1 kervala
181 1 kervala
	case 1:
182 1 kervala
		s.serial (Foo);
183 1 kervala
	case 0:
184 1 kervala
		s.serial (Truc);
185 1 kervala
	}
186 1 kervala
}
187 1 kervala
</code></pre>
188 1 kervala
189 1 kervala
h2. NeL File Headers
190 1 kervala
191 1 kervala
The objective of NeL file headers is to verify that a file is in the right format before attempting to interpret the contents.
192 1 kervala
193 1 kervala
<pre><code class="CPP">
194 1 kervala
// The NeL team use the following advise serialise a file this way:
195 1 kervala
void CFileRootClass::serial (IStream &s)
196 1 kervala
{
197 1 kervala
	// First write / read-check the header
198 1 kervala
	s.serialCheck ((uint32)'_LEN');
199 1 kervala
	s.serialCheck ((uint32)'HSEM');
200 1 kervala
201 1 kervala
	// This code write / read-check the header 'NEL_MESH' at the beginning of the file.
202 1 kervala
	// If the check fails, serialCheck throws the EInvalidDataStream exception.
203 1 kervala
}
204 1 kervala
</code></pre>
205 1 kervala
206 1 kervala
Good examples to look at:
207 1 kervala
	
208 1 kervala
<pre>
209 1 kervala
include/nel/misc/stream.h			// Stream base classes
210 1 kervala
class CTileBank in src/3d/tile_bank.cpp	// Good example of file format evolution
211 1 kervala
class CAnimation in src/3d/animation.cpp	// Good example of polymorphism
212 1 kervala
</pre>