OpenGL video output quality in single-plane mode
= Description Hunting down the problem reported in bug #20799 (closed) I stumbled about yet another issue with the OpenGL video output module. Effectively reducing luminance resolution of single-plane 4:2:2 encoded videos by a factor of two.
= Steps to Reproduce Plug your single-plane input device such as my stk1160 and switch video output mode to OpenGL. Even using any VLC version before the bug reported in #20799 (closed) luminance resolution will be off by a factor of 2 compared to any other video output module.
= Further details Again talking about single-plane 4:2:2 encoded data, buffer data gets uploaded to a RGBA texture.
As the coresponding comment in fragment_shaders.c correctly states "Y1 U Y2 V fits in R G B A". Unfortunately the generated fragment shader never makes use of the Y2 value stored in the alpha component, reducing luminance resolution by a factor of 2.
IMHO it could be a good idea to just store the buffer handed over by hardware twice. Once to a 4 component RGBA texture and once to a 2 component say RG texture.
This way we could get both Y values from our RG texture. And corresponding Cb and Cr values from our RGBA texture. Meaning our fragment shader would look something like this.
uniform sampler2D Texture0;
uniform sampler2D Texture1;
smooth in vec2 TexCoord0;
void main() {
float y = texture2D(Texture0, TexCoord0).g * 255;
float cb = texture2D(Texture1, TexCoord0).r * 255;
float cr = texture2D(Texture1, TexCoord0).b * 255;
float r = y + 1.402 * (cr - 128);
float g = y - 0.34 * (cb - 128) - 0.71 * (cr - 128);
float b = y + 1.77 * (cb - 128);
gl_FragColor = vec4(r/255, g/255, b/255, 1);
I quick-hacked together a stand-alone program using this strategy to verify the mentionend algorithm shows expected results. But alas I got totally stuck in ifdef mazes, fragment shader code generation, etc. to integrate any of it in the current VLC source code.
Don't forget to contact me, if anybody with enough knowledge about the OpenGL video output module would be willing to give me some hint on how to go about integrating the mentioned strategy without breaking other things, or be in need of more information.