Screen capture's subscreen region check flawed
The subscreen region check for screen capturing is flawed. I tested on Windows Vista with an ATI Radeon HD 5700 card. It seems that if the parameters screen-width and screen-height are not set then they have the default values zero causing the following not being executed in screen.c (http://git.videolan.org/?p=vlc.git;a=blob;f=modules/access/screen/screen.c) unless you specify both parameters:
147 if( p_sys->i_width > 0 && p_sys->i_height > 0 )
148 msg_Dbg( p_demux, "capturing subscreen top: %d, left: %d, "
149 "width: %d, height: %d",
150 p_sys->i_top,
151 p_sys->i_left,
152 p_sys->i_width,
153 p_sys->i_height );
And similarly the following check will also be skipped if either width or height is omitted:
167 if( p_sys->i_width > 0 && p_sys->i_height > 0 )
168 {
169 if( p_sys->i_left + p_sys->i_width > p_sys->fmt.video.i_width ||
170 p_sys->i_top + p_sys->i_height > p_sys->fmt.video.i_height )
171 {
172 msg_Err( p_demux, "subscreen region overflows the screen" );
173 free( p_sys );
174 return VLC_EGENERIC;
175 }
176 else
177 {
...
188 }
189 }
So in summary if one only specifies width OR height, one can bypass the check.
On my computer the screen size is reported to be 1280x1024 by the debug printout in line 162, which is the resolution of my primary display. There is a second and third display connected right to and below of the primary one. The secondary has also resolution 1280x1024 and the third one has 1920x1080. Thus again the subscreen region check will not be correct because it assumes p_sys->fmt.video.i_width/height to be the total width/height of all screens in line 169/170.
It also seems that the origin of the coordinate system is always in the upper-left corner of the primary display so the subscreen region overflow check is flawed again since it assumes that the extended display ranges from (0,0) to (total_width, total_height) which seems to be wrong on Windows. On Windows screens apparently can have negative offsets if they are located left of or above the primary screen. On Linux this seems to be different. Can anyone confirm this maybe with an NVIDIA card?
It seems more people ran into these problems: https://forum.videolan.org/viewtopic.php?f=14&t=97199