How to Setup: DeSmuME 0.9.9 JIT, VBA and Camtasia Recorder for best quality
UPDATED, Read description!
For Camtasia I have found that the best quality you are going to get is with the Lagarith Lossless Codec, you can find more information about it here:
http://lags.leetcode.net/codec.html
However, to avoid desync you MUST use exactly 60 FPS. I used to use 50 and it desynchronized badly.
As for VBA, for best quality possible you will need a screen resolution of 800x600 or higher. 640x480 (which you don't see often these days) is too low.
Go to options, Pixel filter, 3x Magnification, HQ3x or HQ3xS. Then go to Options, Video and Select 3x Window Size.
For desmume you will need a screen resolution of at least 1280x720, preferably at least 1600x900. Here I was using 1366x768. Go to View, Magnification Filter, select HQ2x or HQ2xS. Scale your window down as far as it will go. If you are using a resolution smaller than 1600x900, you must also go to View, LCDs Layout, Select Vertical.
If your screen resolution is at least 2560x1440 (double 1280x720) you can set the LCDs Layout to vertical and select HQ4x as the magnification filter. This will give you a HUGE emulator with a size of 2048x768 for both screens.