In
Section "Device" of your xorg.conf file you should check that
Option DynamicTwinView "true" is set. In section screen you should set your resolution and RefreshRate by just adding the line
Option metamodes "1280x1024_75 +0+0; 1280x1024 +0+0" (modify to your needs - this line is for a 17" flat panel).
You'll notice that every app that uses xrandr extension to read and set resolution will have the values all messed up, but that's a known issue and it still works well. Set Ryzom to use the new RefreshRate by selecting it in ryzom_configurator_rd.exe.
If it's not already done otherwise, start nvidia-settings after every reboot to ensure the settings of it are loaded correctly.
This should fix your problem with the windows to mess up in Ryzom. 
Edit: Added it to the first post