VLC won't exit on closing window (with VA-API acceleration)
System: Linux box 5.6.19-159.current #1 (closed) SMP PREEMPT Fri Oct 16 17:49:06 UTC 2020 x86_64 GNU/Linux
OS: Solus Fortitude 4.1
VLC version: 3.0.11
mesa library: 20.1.9 (may be important as relates to hardware acceleration)
Reproducible: always
When VLC is playing a video and you CLOSE THE WINDOW (not select file->quit menu entry) the video player stops, the window closes, but the VLC process remains running and "stuck". You cannot start another instance of VLC and you can't kill it unless you use "kill -9". The process is not a zombie, it's just stuck waiting for something.
If you have the system tray icon enabled, it will remain visible and clicking on it will re-display the VLC window, but you still won't be able to exit (in this case even selecting File-Quit does not work -- this works if you have NOT PLAYED any video).
I narrowed this down to the output decoder being used. In "Tools->Preferences->Input/Codecs" (using Simple mode) the option "Hardware-accelerated decoding" is set to "Automatic" by default. Selecting "VA-API" (with or without DRM) or setting it to "disabled" works around the issue: VLC will then exit normally whenever you close the window. Selecting "VDPAU" makes the bug reappear. I assume that on my system "Automatic" selects "VDPAU".
To give some more details, I ran "strace vlc" to log system calls.
When using VDPAU I see the following:
futex(0x207a108, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x207a0b8, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT PIPE TERM], NULL, 8) = 0
rt_sigaction(SIGHUP, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigtimedwait([HUP INT QUIT TERM], libva info: VA-API version 1.8.0
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_8
libva info: va_openDriver() returns 0
mesa: for the --simplifycfg-sink-common option: may only occur zero or one times!
mesa: for the --global-isel-abort option: may only occur zero or one times!
mesa: for the --amdgpu-atomic-optimizations option: may only occur zero or one times!
mesa: for the --simplifycfg-sink-common option: may only occur zero or one times!
mesa: for the --global-isel-abort option: may only occur zero or one times!
mesa: for the --amdgpu-atomic-optimizations option: may only occur zero or one times!
[ at this point the video is playing and ]
[ I press the X on the window for to close it]
{si_signo=SIGTERM, si_code=SI_TKILL, si_pid=8038, si_uid=1000}, NULL, 8) = 15 (SIGTERM)
rt_sigaction(SIGINT, {sa_handler=SIG_IGN, sa_mask=[INT], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7fd8816a4a40}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGALRM, {sa_handler=0x4016e0, sa_mask=[ALRM], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7fd8816a4a40}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [INT ALRM], NULL, 8) = 0
alarm(3) = 0
getpid() = 8038
tgkill(8038, 8038, SIGTERM) = 0
futex(0x207a108, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x207a0b8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7fd864f349d0, FUTEX_WAIT, 8042, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
rt_sigaction(SIGINT, {sa_handler=SIG_DFL, sa_mask=[INT], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7fd8816a4a40}, {sa_handler=SIG_IGN, sa_mask=[INT], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7fd8816a4a40}, 8) = 0
rt_sigreturn({mask=[HUP QUIT PIPE TERM]}) = 202
futex(0x7fd864f349d0, FUTEX_WAIT, 8042, NULL
^Cstrace: Process 8038 detached
<detached ...>
It appears to me that the signal is received but some lock is not being released and VLC gets stuck.
Running with that option set to VA-API uses another output mode which works fine:
rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT PIPE TERM], NULL, 8) = 0
rt_sigaction(SIGHUP, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigtimedwait([HUP INT QUIT TERM], [00007f52a0001800] gl gl: Initialized libplacebo v2.72.0 (API v72)
libva info: VA-API version 1.8.0
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_8
libva info: va_openDriver() returns 0
mesa: for the --simplifycfg-sink-common option: may only occur zero or one times!
mesa: for the --global-isel-abort option: may only occur zero or one times!
mesa: for the --amdgpu-atomic-optimizations option: may only occur zero or one times!
[00007f52a0001800] glconv_vaapi_x11 gl error: vaDeriveImage: operation failed
[00007f52ac0076c0] main video output error: video output creation failed
[00007f52b248d120] main decoder error: failed to create video output
[00007f52a0809c10] gl gl: Initialized libplacebo v2.72.0 (API v72)
mesa: for the --simplifycfg-sink-common option: may only occur zero or one times!
mesa: for the --global-isel-abort option: may only occur zero or one times!
mesa: for the --amdgpu-atomic-optimizations option: may only occur zero or one times!
[00007f51a0045890] gl gl: Initialized libplacebo v2.72.0 (API v72)
[ at this point the video is playing and ]
[ I press the X on the window for to close it]
{si_signo=SIGTERM, si_code=SI_TKILL, si_pid=30820, si_uid=1000}, NULL, 8) = 15 (SIGTERM)
rt_sigaction(SIGINT, {sa_handler=SIG_IGN, sa_mask=[INT], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f53a8562a40}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGALRM, {sa_handler=0x4016e0, sa_mask=[ALRM], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f53a8562a40}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [INT ALRM], NULL, 8) = 0
alarm(3) = 0
getpid() = 30820
tgkill(30820, 30820, SIGTERM) = 0
futex(0xc61ee8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc61e98, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f538bdf29d0, FUTEX_WAIT, 30825, NULL) = 0
write(13, "\1\0\0\0\0\0\0\0", 8) = 8
futex(0x7f538bbf09d0, FUTEX_WAIT, 30829, NULL) = 0
getpid() = 30820
tgkill(30820, 30826, SIGRTMIN) = 0
futex(0x7f538bcf19d0, FUTEX_WAIT, 30826, NULL) = 0
close(10) = 0
close(9) = 0
close(8) = 0
[[ truncated, it proceeds to exit normally ]]
This makes VLC quite unusable and the recommendation in Solus forums is to not use it: Discussion in Solus forums
Note that in Solus forums two more bugs are referenced which may or may not be related (they give very little information other than "vlc won't exit"):
Both of the above reference version 3.0.x and one references Linux so you may want to associate them with this one.