VP9 video freezes on first frame, process won't exit [INVESTIGATION NEEDED]

I cannot get any VP9 files to play in the vlc-2.2.0-win32.exe release. All I see is the first frame and no movement while the cursor advances along the timeline and the sound (if present) plays. Even after the window is closed, the process does not exit and uses significant CPU. If the file had audio, occasional audio glitches are heard until the process is manually ended.

Attached is a message log and VP8/VP9 example files.

Tested on Windows 8.1 x64

VP8 file, works fine

Added attachment:

• out8.webm

VP9 file, causes bug

Added attachment:

• out9.webm

Message log from trying to play out9.webm

Added attachment:

• vp9log.txt

Symbolic threaded stack trace required for lockups. Cannot reproduce with supplied video.

added Status::needs more info label

closed

OK, I will investigate

changed title from VP9 video freezes on first frame, process won't exit to VP9 video freezes on first frame, process won't exit [INVESTIGATION NEEDED]

added Component::Demuxers: MKV Type::task labels and removed Status::needs more info label

changed milestone to %2.2.1

reopened

Added attachment:

• vlc-vp9-backtrace.txt

Backtrace attached for version vlc-2.2.0-20150228-0203

Looks like a busy loop inside libavcodec...

Added attachment:

• vlc-vp9-backtrace-full.txt

assigned to @jbk

vp9_decode_frame() returns 0, but it should return the packet size. It works for me with these changes:

diff --git a/contrib/src/ffmpeg/rules.mak b/contrib/src/ffmpeg/rules.mak
index 69c6ae4..d94015a 100644
--- a/contrib/src/ffmpeg/rules.mak
+++ b/contrib/src/ffmpeg/rules.mak
@@ -178,6 +178,7 @@ ffmpeg: ffmpeg-$(HASH).tar.xz .sum-ffmpeg
 	mkdir -p $@-$(HASH)
 ifdef USE_FFMPEG
 	$(XZCAT) "$<" | (cd $@-$(HASH) && tar xv --strip-components=1)
+	$(APPLY) $(SRC)/ffmpeg/vp9-loop.patch
 else
 	$(XZCAT) "$<" | (cd $@-$(HASH) && tar xv --strip-components=1 --exclude=README)
 	$(APPLY) $(SRC)/ffmpeg/0001-svq1-fix-av_fast_malloc-call.patch
diff --git a/contrib/src/ffmpeg/vp9-loop.patch b/contrib/src/ffmpeg/vp9-loop.patch
new file mode 100644
index 0000000..3bbee33
--- /dev/null
+++ b/contrib/src/ffmpeg/vp9-loop.patch
@@ -0,0 +1,20 @@
+--- a/libavcodec/vp9.c	2015-02-19 18:40:49.000000000 +0100
++++ b/libavcodec/vp9.c	2015-03-02 19:00:43.010555400 +0100
+@@ -3760,7 +3760,7 @@
+         if ((res = av_frame_ref(frame, s->refs[ref].f)) < 0)
+             return res;
+         *got_frame = 1;
+-        return 0;
++        return pkt->size;
+     }
+     data += res;
+     size -= res;
+@@ -3984,7 +3984,7 @@
+         *got_frame = 1;
+     }
+ 
+-    return 0;
++    return pkt->size;
+ }
+ 
+ static void vp9_decode_flush(AVCodecContext *ctx)

This patch doesn't apply to the current ffmpeg code. I had a look at the current code and vp9_decode_frame() is called by vp9_decode_packet() which handles the size properly. vp9_decode_frame() returning 0 means all the data have been used.

Edit: I was compiling the default avcodec with is libav. But actual builds are built with ffmpeg.

ffmpeg has indeed vp9_decode_frame() included in vp9_decode_packet() and returns the wrong value.

Here's a patch for the current FFmpeg master

Added attachment:

• 0001-fix-VP9-packet-decoder-returning-0-instead-of-the-us.patch

Nice job.