I've already set that parameter in Register, but it still got a lot of mosaic when bitrate set to a low level. Clearly, it's not VBR because the quality of encoding could not be influenced by bitrate settings. The right way to control VBR quality is to change QP and throttling settings. So how to change encoder to a "true" VBR? Thanks.
I'm not sure I quite undestand your requirement. Target Bitrate does affect the quality of encoding. Probably the level you set is just too low for the resolution and complexity of particular stream. Can you provide the exact configuration for the case with unsatisfactory quality? Media SDK Tracer tool (it's in the package under \tools folder) will help you get the complete log of all parameters values. Could you please create one and share with us?
VBR mode implemented by Media SDK is the standard one. If your intention is to comtrol per-frame QP directly, such a mode is also supported by Media SDK encoder and is called Constant Quantization Parameter (look for CQP in mediasdk-man.pdf). But Direct Show sample enoder filter doesn't expose this method. You will need to extend the sample code yourself.