Endianness issue with RGB video output?
From: Alexander Gall
Date: 20 februari 2006 11:08:00 GMT+01:00
To: vlc@videolan.org
Subject: Possible endian bug in video_chrome/video_output
I have built the vlc svn snapshot 20060217 for Solaris 11 on SPARC.
It mostly works but the video output (x11 and opengl) is broken. With
x11, the display is monochrome black/red and with opengl it looks like
some colors are missing or swapped. The visual I'm using is 24
planes/32bpp, see attached output of xdpyinfo (screen [#0](https://code.videolan.org/videolan/vlc/-/issues/0)). Note that
the "image byte order" is big endian (MSBFirst).
I experimented a bit and found that the colours look ok with opengl
when I reverse the byte ordering of the 32 bit word in the macro
CONVERT_Y_PIXEL in modules/video_chroma/i420_rgb.h (see attached
patch). I have attached the output of vlc run with -vvv as well.
I can't say whether this is a genuine bug or whether vlc just fails to
detect the properties of my display correctly. The videou output and
chroma code never seems to check the image byte order of the display
(i.e. there's no reference to the symbols MSBFirst/LSBFirst), which
seems a bit surprising to me. In any case, I hope that my hack is
enough to track down the source of the problem.
BTW, the mpeg2dec utility that comes with the mpeg2dec library
displays video correctly with both x11 and opengl output using the
libmpeg2convert library. This library also has the advantage to
support the SPARC VIS instruction set. Maybe it would make sense for
vlc to use it as well.
name of display: :0.0
version number: 11.0
vendor string: Sun Microsystems, Inc.
vendor release number: 6620
maximum request size: 262140 bytes
motion buffer size: 256
bitmap unit, bit order, padding: 32, MSBFirst, 32
image byte order: MSBFirst
number of supported pixmap formats: 4
supported pixmap formats:
depth 1, bits_per_pixel 1, scanline_pad 32
depth 8, bits_per_pixel 8, scanline_pad 32
depth 16, bits_per_pixel 16, scanline_pad 32
depth 24, bits_per_pixel 32, scanline_pad 32
keycode range: minimum 8, maximum 254
focus: window 0x1480009, revert to PointerRoot
number of extensions: 34
AccessX
Adobe-DPS-Extension
DAMAGE
DOUBLE-BUFFER
DPMS
DPSExtension
Extended-Visual-Information
FBPM
GLX
LBX
MIT-SCREEN-SAVER
MIT-SHM
MIT-SUNDRY-NONSTANDARD
Multi-Buffering
RECORD
SECURITY
SHAPE
ST
SUN_ALLPLANES
SUN_DGA
SUN_OVL
SUN_SME
SYNC
SolarisIA
TOG-CUP
X-Resource
XC-APPGROUP
XC-MISC
XEVIE
XFIXES
XIE
XInputDeviceEvents
XInputExtension
XTEST
default screen number: 0
number of screens: 2
screen [#0](https://code.videolan.org/videolan/vlc/-/issues/0):
dimensions: 1280x1024 pixels (361x288 millimeters)
resolution: 90x90 dots per inch
depths (3): 1, 8, 24
root window id: 0x4a
depth of root window: 24 planes
number of colormaps: minimum 1, maximum 2
default colormap: 0x34
default number of colormap cells: 256
preallocated pixels: black 0, white 16777215
options: backing-store YES, save-unders YES
largest cursor: 32x32
current input event mask: 0xfa2033
KeyPressMask KeyReleaseMask EnterWindowMask
LeaveWindowMask ButtonMotionMask StructureNotifyMask
SubstructureNotifyMask SubstructureRedirectMask FocusChangeMask
PropertyChangeMask ColormapChangeMask
number of visuals: 17
default visual id: 0x27
visual:
visual id: 0x20
class: PseudoColor
depth: 8 planes
available colormap entries: 256
red, green, blue masks: 0x0, 0x0, 0x0
significant bits in color specification: 8 bits
visual:
visual id: 0x21
class: PseudoColor
depth: 8 planes
available colormap entries: 256
red, green, blue masks: 0x0, 0x0, 0x0
significant bits in color specification: 8 bits
visual:
visual id: 0x22
class: StaticColor
depth: 8 planes
available colormap entries: 256
red, green, blue masks: 0x7, 0x38, 0xc0
significant bits in color specification: 8 bits
visual:
visual id: 0x23
class: StaticGray
depth: 8 planes
available colormap entries: 256
red, green, blue masks: 0x0, 0x0, 0x0
significant bits in color specification: 8 bits
visual:
visual id: 0x24
class: GrayScale
depth: 8 planes
available colormap entries: 256
red, green, blue masks: 0x0, 0x0, 0x0
significant bits in color specification: 8 bits
visual:
visual id: 0x25
class: TrueColor
depth: 8 planes
available colormap entries: 8 per subfield
red, green, blue masks: 0x7, 0x38, 0xc0
significant bits in color specification: 8 bits
visual:
visual id: 0x26
class: StaticGray
depth: 8 planes
available colormap entries: 256
red, green, blue masks: 0x0, 0x0, 0x0
significant bits in color specification: 8 bits
visual:
visual id: 0x2f
class: PseudoColor
depth: 8 planes
available colormap entries: 255
red, green, blue masks: 0x0, 0x0, 0x0
significant bits in color specification: 8 bits
visual:
visual id: 0x30
class: PseudoColor
depth: 8 planes
available colormap entries: 255
red, green, blue masks: 0x0, 0x0, 0x0
significant bits in color specification: 8 bits
visual:
visual id: 0x27
class: TrueColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff, 0xff00, 0xff0000
significant bits in color specification: 8 bits
visual:
visual id: 0x28
class: TrueColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff, 0xff00, 0xff0000
significant bits in color specification: 8 bits
visual:
visual id: 0x29
class: DirectColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff, 0xff00, 0xff0000
significant bits in color specification: 8 bits
visual:
visual id: 0x2a
class: DirectColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff, 0xff00, 0xff0000
significant bits in color specification: 8 bits
visual:
visual id: 0x2b
class: TrueColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff, 0xff00, 0xff0000
significant bits in color specification: 8 bits
visual:
visual id: 0x2c
class: TrueColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff, 0xff00, 0xff0000
significant bits in color specification: 8 bits
visual:
visual id: 0x2d
class: TrueColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff, 0xff00, 0xff0000
significant bits in color specification: 8 bits
visual:
visual id: 0x2e
class: TrueColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff, 0xff00, 0xff0000
significant bits in color specification: 8 bits
screen [#1](https://code.videolan.org/videolan/vlc/-/issues/1):
dimensions: 1280x1024 pixels (361x289 millimeters)
resolution: 90x90 dots per inch
depths (2): 1, 24
root window id: 0x4c
depth of root window: 24 planes
number of colormaps: minimum 1, maximum 1
default colormap: 0x35
default number of colormap cells: 256
preallocated pixels: black 0, white 16777215
options: backing-store YES, save-unders YES
largest cursor: 64x64
current input event mask: 0xfa2033
KeyPressMask KeyReleaseMask EnterWindowMask
LeaveWindowMask ButtonMotionMask StructureNotifyMask
SubstructureNotifyMask SubstructureRedirectMask FocusChangeMask
PropertyChangeMask ColormapChangeMask
number of visuals: 1
default visual id: 0x36
visual:
visual id: 0x36
class: TrueColor
depth: 24 planes
available colormap entries: 256 per subfield
red, green, blue masks: 0xff0000, 0xff00, 0xff
significant bits in color specification: 8 bits
diff -Naur vlc-0.8.5-20060217.orig/modules/video_chroma/i420_rgb.h vlc-0.8.5-20060217/modules/video_chroma/i420_rgb.h
--- vlc-0.8.5-20060217.orig/modules/video_chroma/i420_rgb.h 2006-02-17 00:00:28.000000000 +0100
+++ vlc-0.8.5-20060217/modules/video_chroma/i420_rgb.h 2006-02-19 14:42:32.667710000 +0100
@@ -73,9 +73,9 @@
#define CONVERT_Y_PIXEL( BPP ) \
/* Only Y sample is present */ \
p_ybase = p_yuv + *p_y++; \
- *p_buffer++ = p_ybase[RED_OFFSET-((V_RED_COEF*128)>>SHIFT) + i_red] | \
+ *p_buffer++ = p_ybase[RED_OFFSET-((V_RED_COEF*128)>>SHIFT) + i_red] <<24 | \
p_ybase[GREEN_OFFSET-(((U_GREEN_COEF+V_GREEN_COEF)*128)>>SHIFT) \
- + i_green ] | p_ybase[BLUE_OFFSET-((U_BLUE_COEF*128)>>SHIFT) + i_blue];
+ + i_green ] <<8 | p_ybase[BLUE_OFFSET-((U_BLUE_COEF*128)>>SHIFT) + i_blue] >>8;
#define CONVERT_YUV_PIXEL( BPP ) \
/* Y, U and V samples are present */ \
VLC media player 0.8.5-svn Janus
[00000001] main vlc debug: opening config file /home/gall/.vlc/vlcrc
[00000001] main vlc debug: checking builtin modules
[00000001] main vlc debug: checking plugin modules
[00000001] main vlc debug: loading plugins cache file /home/gall/.vlc/cache/plugins-0404be.dat
[00000001] main vlc debug: recursively browsing `modules'
[00000001] main vlc debug: recursively browsing `/usr/local/lib/vlc'
[00000001] main vlc debug: recursively browsing `plugins'
[00000001] main vlc debug: module bank initialized, found 196 modules
[00000001] main vlc debug: opening config file /home/gall/.vlc/vlcrc
[00000001] main vlc debug: CPU has capabilities FPU
[00000001] main vlc debug: looking for memcpy module: 1 candidate
[00000001] main vlc debug: using memcpy module "memcpy"
[00000254] main playlist debug: waiting for thread completion
[00000254] main playlist debug: thread 2 (playlist) created at priority 0 (src/playlist/playlist.c:184)
[00000255] main private debug: waiting for thread completion
[00000255] main private debug: thread 3 (preparser) created at priority 0 (src/playlist/playlist.c:210)
[00000256] main interface debug: looking for interface module: 1 candidate
[00000256] main interface debug: using interface module "hotkeys"
[00000256] main interface debug: thread 4 (interface) created at priority 0 (src/interface/interface.c:229)
[00000258] main interface debug: looking for interface module: 5 candidates
[00000258] main interface debug: using interface module "screensaver"
[00000258] main interface debug: thread 5 (interface) created at priority 0 (src/interface/interface.c:229)
[00000260] main interface debug: looking for interface module: 4 candidates
[00000260] main interface debug: using interface module "wxwidgets"
[00000260] main interface debug: thread 6 (manager) created at priority 0 (src/interface/interface.c:214)
[00000260] wxwidgets interface debug: Using last windows config '(-1,0,0,1280,1024)(6,0,0,-1,150)'
[00000260] wxwidgets interface debug: id=6 p=(0,0) s=(-1,150)
[00000260] wxwidgets interface debug: last windows config is invalid (ignored)
[00000254] main playlist debug: adding playlist item `dvdsimple:///vol/dev/aliases/cdrom0' ( dvdsimple:///vol/dev/aliases/cdrom0 )
[00000254] main playlist debug: creating statistics handler
[00000254] main playlist debug: creating new input thread
[00000264] main input debug: waiting for thread completion
[00000264] main input debug: `dvdsimple:///vol/dev/aliases/cdrom0' gives access `dvdsimple' demux `' path `/vol/dev/aliases/cdrom0'
[00000264] main input debug: creating demux: access='dvdsimple' demux='' path='/vol/dev/aliases/cdrom0'
[00000265] main demuxer debug: looking for access_demux module: 1 candidate
[00000264] main input debug: thread 7 (input) created at priority 0 (src/input/input.c:260)
[00000265] dvdread demuxer debug: VMG opened
[00000265] dvdread demuxer debug: number of titles: 4
[00000265] dvdread demuxer debug: title 0 has 23 chapters
[00000265] dvdread demuxer debug: title 1 has 23 chapters
[00000265] dvdread demuxer debug: title 2 has 2 chapters
[00000265] dvdread demuxer debug: title 3 has 2 chapters
[00000265] dvdread demuxer debug: open VTS 4, for title 1
[00000265] dvdread demuxer debug: title 1 vts_title 1 pgc 1 pgn 1 start 0 end 1863688 blocks: 1863689
[00000264] main input debug: selecting program id=0
[00000265] dvdread demuxer debug: audio position 0
[00000265] dvdread demuxer debug: spu 1 0x80000200
[00000265] dvdread demuxer debug: spu 2 0x80010300
[00000265] main demuxer debug: using access_demux module "dvdread"
[00000267] main decoder debug: looking for decoder module: 23 candidates
[00000267] main decoder debug: using decoder module "libmpeg2"
[00000267] main decoder debug: thread 8 (decoder) created at priority 0 (src/input/decoder.c:159)
[00000272] main decoder debug: looking for decoder module: 23 candidates
[00000272] main decoder debug: using decoder module "a52"
[00000272] main decoder debug: thread 9 (decoder) created at priority 0 (src/input/decoder.c:159)
[00000264] main input debug: `dvdsimple:///vol/dev/aliases/cdrom0' successfully opened
[00000267] libmpeg2 decoder debug: 720x480 (display 720,480), aspect 768000, sar 32:27, 29.971 fps
[00000267] main decoder debug: no usable vout present, spawning one
[00000275] main video output debug: window size: 853x480
[00000275] main video output debug: looking for video output module: 5 candidates
[00000276] main private debug: Registering subpicture channel, ID: 2
[00000276] main private debug: Registering subpicture channel, ID: 3
[00000276] main private debug: Registering subpicture channel, ID: 4
[00000276] main private debug: Registering subpicture channel, ID: 5
[00000275] opengl video output debug: Texture size: 1024x512
[00000278] main private debug: looking for opengl provider module: 1 candidate
[00000278] glx private debug: Using GLX 1.3 API
[00000278] main private debug: window size: 853x480
[00000278] main private error: option glx-shm does not exist
[00000278] glx private debug: Window manager supports NetWM
[00000272] a52 decoder: A/52 channels:6 samplerate:48000 bitrate:384000
[00000272] main decoder debug: no aout present, spawning one
[00000280] main audio output debug: looking for audio output module: 2 candidates
[00000280] main audio output debug: using audio output module "aout_sdl"
[00000280] main audio output debug: output 's16b' 48000 Hz Stereo frame=1 samples/4 bytes
[00000280] main audio output debug: mixer 'fl32' 48000 Hz Stereo frame=1 samples/8 bytes
[00000280] main audio output debug: filter(s) 'fl32'->'s16b' 48000 Hz->48000 Hz Stereo->Stereo
[00000282] main private debug: looking for audio filter module: 23 candidates
[00000282] main private debug: using audio filter module "float32tos16"
[00000280] main audio output debug: found a filter for the whole conversion
[00000280] main audio output debug: looking for audio mixer module: 3 candidates
[00000280] main audio output debug: using audio mixer module "float32_mixer"
[00000280] main audio output debug: input 'a52 ' 48000 Hz 3F2R/LFE frame=1536 samples/1536 bytes
[00000280] main audio output debug: filter(s) 'a52 '->'fl32' 48000 Hz->48000 Hz 3F2R/LFE->Stereo
[00000300] main private debug: looking for audio filter module: 23 candidates
No accelerated IMDCT transform found
[00000300] main private debug: using audio filter module "a52tofloat32"
[00000280] main audio output debug: found a filter for the whole conversion
[00000280] main audio output debug: filter(s) 'fl32'->'fl32' 52800 Hz->48000 Hz Stereo->Stereo
[00000301] main private debug: looking for audio filter module: 23 candidates
[00000301] main private debug: using audio filter module "bandlimited_resampler"
[00000280] main audio output debug: found a filter for the whole conversion
[00000278] glx private debug: Window manager supports _NET_WM_STATE_FULLSCREEN
[00000278] glx private debug: Window manager supports _NET_WM_STATE_ABOVE
[00000278] glx private debug: Window manager supports _NET_WM_STATE_BELOW
[00000278] main private debug: using opengl provider module "glx"
[00000275] main video output debug: using video output module "opengl"
[00000275] main video output debug: waiting for thread completion
[00000275] main video output debug: got 1 direct buffer(s)
[00000275] main video output debug: picture in 720x480 (0,0,720x480), chroma I420, ar 16:9, sar 32:27
[00000275] main video output debug: picture user 720x480 (0,0,720x480), chroma I420, ar 16:9, sar 32:27
[00000275] main video output debug: picture out 720x480 (0,0,720x480), chroma RV32, ar 16:9, sar 32:27
[00000275] main video output debug: looking for chroma module: 5 candidates
[00000275] main video output debug: using chroma module "i420_rgb"
[00000275] main video output debug: indirect render, mapping render pictures 0-7 to system pictures 1-8
[00000275] main video output debug: thread 11 (video output) created at priority 0 (src/video_output/video_output.c:421)
[00000303] main private warning: dts != current_pts (617028)
[00000303] main private warning: backward_pts != current_pts (-33366)
[00000275] main video output warning: late picture skipped (310287)
[00000275] main video output warning: late picture skipped (72724)
[00000275] main video output warning: late picture skipped (23098)
[00000275] main video output warning: late picture skipped (-10165)
[00000275] main video output warning: late picture skipped (-60126)
[00000264] main input debug: control type=0
[00000264] main input debug: control: stopping input
[00000264] main input debug: closing input
[00000267] main decoder debug: removing module "libmpeg2"
[00000267] main decoder debug: thread 8 joined (src/input/decoder.c:191)
[00000267] main decoder debug: killing decoder fourcc `mpgv', 118 PES in FIFO
[00000272] main decoder debug: removing module "a52"
[00000272] main decoder debug: thread 9 joined (src/input/decoder.c:191)
[00000272] main decoder debug: killing decoder fourcc `a52 ', 0 PES in FIFO
[00000300] main private debug: removing module "a52tofloat32"
[00000301] main private debug: removing module "bandlimited_resampler"
[00000280] main audio output debug: removing module "aout_sdl"
[00000282] main private debug: removing module "float32tos16"
[00000280] main audio output debug: removing module "float32_mixer"
[00000264] main input debug: Program doesn't contain anymore ES
[00000265] main demuxer debug: removing module "dvdread"
[00000264] main input debug: thread 7 joined (src/input/input.c:402)
[00000254] main playlist debug: garbage collector destroying 1 vout
[00000275] main video output debug: removing module "i420_rgb"
[00000278] main private debug: removing module "glx"
[00000275] main video output debug: removing module "opengl"
[00000275] main video output debug: thread 11 joined (src/video_output/video_output.c:461)
(:25683): Gtk-CRITICAL **: file gtkwidget.c: line 5916: assertion `GTK_IS_WIDGET (widget)' failed
(:25683): GLib-GObject-CRITICAL **: file gobject.c: line 1222: assertion `G_IS_OBJECT (object)' failed
[00000001] main vlc debug: removing all interfaces
[00000260] main interface debug: thread 6 joined (src/interface/interface.c:256)
[00000260] main interface debug: removing module "wxwidgets"
[00000258] main interface debug: thread 5 joined (src/interface/interface.c:256)
[00000258] main interface debug: removing module "screensaver"
[00000256] main interface debug: thread 4 joined (src/interface/interface.c:256)
[00000256] main interface debug: removing module "hotkeys"
[00000001] main vlc debug: removing playlist handler
[00000255] main private debug: thread 3 joined (src/playlist/playlist.c:247)
[00000254] main playlist debug: thread 2 joined (src/playlist/playlist.c:248)
[00000254] main playlist: stopping playback
[00000254] main playlist debug: deleting playlist item `dvdsimple:///vol/dev/aliases/cdrom0'
[00000001] main vlc debug: removing all video outputs
[00000001] main vlc debug: removing all audio outputs
[00000001] main vlc debug: TIMER Playlist walk : Total 0.010 ms / 1 intvls (Avg 0.010 ms)
[00000001] main vlc debug: removing module "memcpy"
[00000001] main vlc debug: opening config file /home/gall/.vlc/vlcrc
[00000001] main vlc debug: saving plugins cache file /home/gall/.vlc/cache/plugins-0404be.dat
Activity
According to sam the problem most likely lies in the values that are used in this part of the code for his specific machine:
#define RGB2PIXEL( p_vout, i_r, i_g, i_b ) \ (((((uint32_t)i_r) >> p_vout->output.i_rrshift) \ << p_vout->output.i_lrshift) \ | ((((uint32_t)i_g) >> p_vout->output.i_rgshift) \ << p_vout->output.i_lgshift) \ | ((((uint32_t)i_b) >> p_vout->output.i_rbshift) \ << p_vout->output.i_lbshift))changed milestone to %0.8.5-final
I have i_rmask: 0x000000ff i_rrshift: 0 i_lrshift: 0 i_gmask: 0x0000ff00 i_rrshift: 0 i_lrshift: 8 i_bmask: 0x00ff0000 i_rrshift: 0 i_lrshift: 16 The mask values are wrong. They should be i_rmask: 0xff000000 i_gmask: 0x00ff0000 i_bmask: 0x0000ff00 I'm not sure where exactly these are set for opengl. If it's in modules/video_output/opengl.c, then the problem is obvious, because the values are just set statically, instead of read from the capabilities of the output device. Now, when I set the proper mask values by hand, the output is still not ok. This is because src/video_output/video_output.c:MaskToShift() produces the wrong result for a mask of 0xff000000. The line i_high = i_mask + i_low; /* higher bit of the mask */ results in an overflow and the value of rshift becomes 31369 (it should be 0). When I fix this as well, the output is finally correct. BTW, MaskToShift seems to expect that the mask can be smaller than 8 bits, in which case the lower order bits of a color value are discarded by RGB2PIXEL. I'm wondering whether this makes sense and if it actually occurs in practice. Finally, the same problem needs to be fixed for the xvideo and glx output modules as well.http://www.google.com/search?client=safari&rls=en&q=PIXELFORMATDESCRIPTOR&ie=UTF-8&oe=UTF-8
The PIXELFORMATDESCRIPTOR which is used to request/select an OpenGL pictureformat seems to be used for this.
added Status::fixed label
Please register or sign in to reply