I observed 1-2% bitrate improvement when setting trellis (for I or IP frames, SD/HD, cqp, best quality, 12:2 GOP) using modified sample_encode. No improvement measured when trellis is enabled for IPB simultaneously. Could you describe the algorithm used in msdk, comparing to x264 one (http://akuvian.org/src/x264/trellis.txt) ?
Thanks for your comments !
We can not share the implementation details of the HW but on a high level this is how it works:
1. Trellis Quantization enables Rate-Distortion optimum quantization using a Trellis based algorithm. The algorithm allows the quantization decisions for all coefficients in a block to be made jointly to improve coding efficiency.
2. Quality gain from Trellis Quantization depends on bit rate, resolution and frame type. Higher quality gain on higher bit rate (lower QP).
3. Quality gain also depends on the number of B frames used. Media SDK API has options to disable Trellis for P/B frames if desired.