Fix a VLC crash when exiting and there are two or more video filters loaded
When using a commande line such as :
./vlc -vv -I dummy udp://@233.0.0.1:7400 --sout '#transcode{deinterlace,vcodec=mpgv,height=176,width=144}:std{access=file,mux=ts,dst=/dev/null}'
two video filters got loaded : deinterlace and swscale
When looking at UpdateBufferFunctions, at src/misc/filter_chain.c:446
[1] for( i = 0; i < p_chain->filters.i_count - 1; i++ )
{
p_filter = pp_filter[i];
if( p_filter->pf_vout_buffer_new != VideoBufferNew )
{
if( p_chain->pf_buffer_allocation_clear )
p_chain->pf_buffer_allocation_clear( p_filter );
p_filter->pf_vout_buffer_new = VideoBufferNew;
p_filter->pf_vout_buffer_del = NULL;
}
}
[2] if( p_chain->filters.i_count >= 1 )
{
[3] p_filter = pp_filter[i];
if( p_filter->pf_vout_buffer_new == VideoBufferNew )
{
p_filter->pf_vout_buffer_new = NULL;
if( p_chain->pf_buffer_allocation_init( p_filter,
p_chain->p_buffer_allocation_data ) != VLC_SUCCESS )
return VLC_EGENERIC;
}
}
When the first video filter (deinterlace) is loaded, filters.i_count == 1 and nothing is executed.
When the second video filter (swscale) is loaded, filters.i_count == 2, the for() at [1] loops one time and clears buffers from the deinterlace module. When the for() exits, 'i' == 1.
So the execution flow continues into the if() at [2] because filters.i_count == 2 but as 'i' == 1, the resulting 'p_filter', from [3], contains the second module : swscale. As its buffers has not been cleared, its pf_vout_buffer_new != VideoBufferNew and nothing happens. Thus, the buffers from the deinterlace module was cleared but not alloc()'ed again.
When exiting VLC, it crashes when freeing the "deinterlace" module :
(gdb) bt
[#0](https://code.videolan.org/videolan/vlc/-/issues/0) 0x00002b9d7a244a25 in free () from /lib/libc.so.6
[#1](https://code.videolan.org/videolan/vlc/-/issues/1) 0x00002aaaac263b63 in video_del_buffer (p_this=0x86c7e8, p_pic=0x85ca00) at transcode.c:2270
[#2](https://code.videolan.org/videolan/vlc/-/issues/2) 0x00002aaaac261137 in transcode_video_filter_allocation_clear (p_filter=0x86c7e8) at transcode.c:1470
[#3](https://code.videolan.org/videolan/vlc/-/issues/3) 0x00002b9d79f812a8 in filter_chain_DeleteFilterInternal (p_chain=0x8b8450, p_filter=0x86c7e8) at misc/filter_chain.c:282
[#4](https://code.videolan.org/videolan/vlc/-/issues/4) 0x00002b9d79f80add in filter_chain_Delete (p_chain=0x8b8450) at misc/filter_chain.c:96
[#5](https://code.videolan.org/videolan/vlc/-/issues/5) 0x00002aaaac26257e in transcode_video_close (p_stream=0x80d8b8, id=0x83edf0) at transcode.c:1830
[#6](https://code.videolan.org/videolan/vlc/-/issues/6) 0x00002aaaac25facd in Del (p_stream=0x80d8b8, id=0x83edf0) at transcode.c:960
Crash log :
[00000440] main packetizer debug: removing module "packetizer_mpegvideo"
[00000440] main packetizer debug: thread ended
[00000440] main packetizer debug: thread 1166059856 joined (input/decoder.c:248)
[00000440] main packetizer debug: killing decoder fourcc `mpgv', 0 PES in FIFO
[00000397] main stream output debug: removing a sout input (sout_input:0x873200)
[00000475] main encoder debug: TIMER encoding video frame : 0.533 ms - Total 17.965 ms / 33 intvls (Avg 0.544 ms)
[00000474] main decoder debug: removing module "libmpeg2"
[00000475] main encoder debug: removing module "avcodec"
[00000398] main stream out debug: Filter 'deinterlace' (0x86c7e8) removed from chain
zsh: segmentation fault (core dumped) ./vlc -vv -I dummy udp://@233.0.0.1:7400 --sout
The attached patch fixes the issue.
--
Aurélien Nephtali