libvlc crashes when a vaapi driver is not found
This bug was originally reported by one Kaffeine user, at KDE bugzilla:
https://bugs.kde.org/show_bug.cgi?id=388020
I was able to reproduce myself by forcing libva to seek on a wrong directory:
$ LIBVA_DRIVERS_PATH=/foo kaffeine
18-12-17 22:45:12.710 [Info ] kaffeine.dvb: Using built-in dvb device manager
18-12-17 22:45:13.230 [Warning ] QObject::connect: No such slot DvbTab::hideCursor()
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /foo/i965_drv_video.so
libva info: va_openDriver() returns -1
[00007f6da4001e80] glconv_vaapi_x11 gl error: vaInitialize: unknown libva error
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /foo/i965_drv_video.so
libva info: va_openDriver() returns -1
[00007f6da4001e80] glconv_vaapi_drm gl error: vaInitialize: unknown libva error
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /foo/i965_drv_video.so
libva info: va_openDriver() returns -1
[00007f6da4001e80] glconv_vaapi_drm gl error: vaInitialize: unknown libva error
Segmentation fault (core dumped)
It can also be reproduced with VLC:
$ LIBVA_DRIVERS_PATH=/foo vlc
VLC media player 3.0.0-rc1 Vetinari (revision 3.0.0-git-134-gadec1d1874)
[000055f3622beae0] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
[000055f3622be350] main playlist: playlist is empty
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /foo/i965_drv_video.so
libva info: va_openDriver() returns -1
[00007f2eb8001f70] glconv_vaapi_x11 gl error: vaInitialize: unknown libva error
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /foo/i965_drv_video.so
libva info: va_openDriver() returns -1
[00007f2eb8001f70] glconv_vaapi_drm gl error: vaInitialize: unknown libva error
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /foo/i965_drv_video.so
libva info: va_openDriver() returns -1
[00007f2eb8001f70] glconv_vaapi_drm gl error: vaInitialize: unknown libva error
Segmentation fault (core dumped)
It crashes with vlc-3.0.0-0.43.git20171208.fc27.x86_64 (a Fedora 27 VLC snapshot). This is the backtrace:
[#0](https://code.videolan.org/videolan/vlc/-/issues/0) 0x00007fff7e92c4c6 in vlc_vaapi_InitializeInstanceDRM () at /usr/lib64/vlc/plugins/video_output/libglconv_vaapi_drm_plugin.so
[#1](https://code.videolan.org/videolan/vlc/-/issues/1) 0x00007fff7e92bfa2 in Open () at /usr/lib64/vlc/plugins/video_output/libglconv_vaapi_drm_plugin.so
[#2](https://code.videolan.org/videolan/vlc/-/issues/2) 0x00007ffff0905da3 in module_load.isra () at /lib64/libvlccore.so.9
[#3](https://code.videolan.org/videolan/vlc/-/issues/3) 0x00007ffff090618f in vlc_module_load () at /lib64/libvlccore.so.9
[#4](https://code.videolan.org/videolan/vlc/-/issues/4) 0x00007fff855212f3 in opengl_init_program () at /usr/lib64/vlc/plugins/video_output/libgl_plugin.so
[#5](https://code.videolan.org/videolan/vlc/-/issues/5) 0x00007fff855224c2 in vout_display_opengl_New () at /usr/lib64/vlc/plugins/video_output/libgl_plugin.so
[#6](https://code.videolan.org/videolan/vlc/-/issues/6) 0x00007fff85527412 in Open () at /usr/lib64/vlc/plugins/video_output/libgl_plugin.so
[#7](https://code.videolan.org/videolan/vlc/-/issues/7) 0x00007ffff0905da3 in module_load.isra () at /lib64/libvlccore.so.9
[#8](https://code.videolan.org/videolan/vlc/-/issues/8) 0x00007ffff090618f in vlc_module_load () at /lib64/libvlccore.so.9
[#9](https://code.videolan.org/videolan/vlc/-/issues/9) 0x00007ffff094214f in DisplayNew () at /lib64/libvlccore.so.9
[#10](https://code.videolan.org/videolan/vlc/-/issues/10) 0x00007ffff0943493 in vout_NewDisplay () at /lib64/libvlccore.so.9
[#11](https://code.videolan.org/videolan/vlc/-/issues/11) 0x00007ffff094ff40 in vout_OpenWrapper () at /lib64/libvlccore.so.9
[#12](https://code.videolan.org/videolan/vlc/-/issues/12) 0x00007ffff0946283 in ThreadStart () at /lib64/libvlccore.so.9
[#13](https://code.videolan.org/videolan/vlc/-/issues/13) 0x00007ffff0947c61 in Thread () at /lib64/libvlccore.so.9
[#14](https://code.videolan.org/videolan/vlc/-/issues/14) 0x00007ffff1356609 in start_thread () at /lib64/libpthread.so.0
[#15](https://code.videolan.org/videolan/vlc/-/issues/15) 0x00007ffff1687e6f in clone () at /lib64/libc.so.6
