colors messed up in DXVA2 AVC decoding on GeForce 9600M GS

DXVA2 GPU decoding in vlc-1.1.0-rc-20100510-0202-win32.exe does not work correctly on my system. When it is enabled, playback of any AVC video stream results in severely messed up colors (screenshot: http://i39.tinypic.com/2jee3o5.jpg).

System info:

LG R510 Notebook (screen resolution 1280x800) Windows 7 x64 Intel Core2Duo P7350, 2.0 GHz (SSE, SSE2, SSE3, SSSE3, SSE4.1) NVIDIA GeForce 9600M GS (driver version: 197.16)

relevant line from DxVAChecker: ModeH264_VLD_NoFGT DXVA2 NV12 720x480 / 1280x720 / 1920x1080

FYI, the same files play fine with DXVA in MediaPlayerClassic HomeCinema, but I don't know if that player is using DXVA1 or 2.

added Severity::normal Type::bug Version::master git + 1 deleted label

changed milestone to %Bugs paradize

Does it also occur in the previous nightly builds ?

assigned to @fenrir and unassigned @Courmisch

added Component::Core: Video output label and removed 1 deleted label

I have just tested the following older nightlies without encountering the problem when GPU decoding is enabled: vlc-1.1.0-pre2-20100415-0203-win32.zip vlc-1.1.0-pre4-20100504-0202-win32.zip

Original author: FlashGekko

commit b0c7f05854bfae340ee33b9e1e815df715a5c3cb
Author: Laurent Aimar <fenrir@videolan.org>
Date:   Fri May 14 22:14:13 2010 +0200

    Fixed dxva2 regression when using NV12.

added Status::fixed label

closed