Do you have any technical advice on linking/building/compiling with multicore? (solved)

Added by ragnar-gd over 7 years ago

Hi, keyboard-cowboys,

thanks to kervala, i now was able to build/compile not only interactively, but also from the command line with cmake and nmake, as is instructed on
http://dev.ryzom.com/wiki/ryzom/BuildForWindowsCmake

On my 4-core AMD, 2.4 GHz, energy-efficient, 45 W, xp pro 32 bit, 2 GB Ram, slc-ssd for OS and software, Raid5 for Data, that took 3h 40 min.
Watching the task-manager, i could see every core was only used once in a while, and i/o close to a minimum, let's say between 25% and 40% cpu-load.
Memory was alway closer to 512 MB than to 1GB.
As i went to bed some time after starting, no one was hurt, and the cpu remained cold, and as that is a passively cooled system: fine.

On the other hand, when using the vc compiler interactively, i was easily able to make my cpu glow, cpu-load on average closer to 75%, memory hitting 1.25 GB
I'm sure, by setting effective CPUs in vc to 8 could improve this even further, though not much.

The problem is, of course, that nmake or cmake, using command line, are not MP-enabled by default.

A little searching provided interesting links:
http://stackoverflow.com/questions/601970/how-do-i-utilise-all-the-cores-for-nmake
http://de.w3support.net/index.php?db=so&id=601970
http://msdn.microsoft.com/en-us/library/bb385193.aspx
http://msdn.microsoft.com/en-us/library/bb385193.aspx#guidelines

I think optimizing the commandline for the open-source pipeline is worth it, as the open-source philosophy "change little, release often" can only be achieved by highly automated builds and tests, which requires automaticly scheduled commandlines, of course.

But before i start using time on experiments: Did anyone of you optimize the commandline for MP, and if, how?


Replies (5)

RE: Do you have any technical advice on linking/building/compiling with multicore? - Added by kervala over 7 years ago

You can follow the part with "jom" :) That's a nmake clone created by Nokia for compiling with all cores (it supports the -j <cores> parameter and uses all cores by default).

RE: Do you have any technical advice on linking/building/compiling with multicore? - Added by ragnar-gd over 7 years ago

This justifies jom...
... but was that the reason to use jom in the first place?

RE: Do you have any technical advice on linking/building/compiling with multicore? - Added by kervala over 7 years ago

Hum, jom is marked "optional", I don't see what you mean.

RE: Do you have any technical advice on linking/building/compiling with multicore? - Added by ragnar-gd over 7 years ago

...well... put it like this:

To easily use multicore for building would have been a very good reason.
But your comment sounds as if that wasn't the case.

What for did you include (or use) jom anyway, if you could have used nmake in the first place?

RE: Do you have any technical advice on linking/building/compiling with multicore? (solved) - Added by ragnar-gd over 7 years ago

Ok, i installed and used JOM: ftp://ftp.qt.nokia.com/jom/jom100.zip
It was fairly easy to install, the instruction on the wiki says it all.
Instead of the recommended one, i used the newest build of JOM as well - ftp://ftp.qt.nokia.com/jom/ - and i added the link to the wiki.
JOM does in fact speed up bulding and compiling from the commandline decisively, compared to nmake.
It still is a bit slower that using the visual studio gui, but it still is worth the (neglectable) trouble, as you still save time.
I'd recommend the commandline for building of releases on windows.
I'd still recommend using visual studio/vc++ express for debugging, as in that case the compiler uses incremental building, which is still faster even with single core than full build with multi-core (at least it was for me).
Fair warning: I still can't say if there are caveats in the result, that is, in the binaries, as there are warnings galore on the commandline, nmake or not.
I just hope it works... ;)

(1-5/5)