In real.c:
---
if( fmt.i_codec == VLC_FOURCC('c','o','o','k')
|| fmt.i_codec == VLC_FOURCC('a','t','r','c') )
{
tk->i_subpackets =
i_subpacket_h * i_frame_size / tk->i_subpacket_size;
tk->p_subpackets =
calloc( tk->i_subpackets, sizeof(block_t *) );
}
else if( fmt.i_codec == VLC_FOURCC('2','8','_','8') )
{
tk->i_subpackets =
i_subpacket_h * i_frame_size / tk->i_coded_frame_size;
tk->p_subpackets =
calloc( tk->i_subpackets, sizeof(block_t *) );
}
---
i_subpacket_h, i_frame_size, and tk->i_coded_frame_size all come directly from the file. if they're large enough, calloc will return null, leading to a null dereference later.
There are other similar allocations in the file, but those are all allocating a 8 or 16 bit number, which is less likely to cause null to be returned.