m3u8/HLS relative URL resolution

added Severity::normal Type::bug Version::master git + 1 deleted label

changed milestone to %Bugs paradize

Responsible m3u8 file looks like: #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:168 #EXTINF:10, 1240/1240168.ts #EXTINF:10, 1240/1240169.ts #EXTINF:10, 1240/1240170.ts #EXTINF:10, 1240/1240171.ts #EXTINF:10, 1240/1240172.ts #EXTINF:10, 1240/1240173.ts #EXTINF:10, 1240/1240174.ts #EXTINF:10, 1240/1240175.ts #EXTINF:10, 1240/1240176.ts #EXTINF:10, 1240/1240177.ts #EXTINF:10, 1240/1240178.ts #EXTINF:10, 1240/1240179.ts #EXTINF:10, 1240/1240180.ts

Original author: ServerAlex

Could you try a niglty build ?

I tried vlc-1.2.0-git-20101021-0003-win32.7z today.. It doesn't even play that file format. VLC 1.1.4 does.

VLC kann das Eingabeformat nicht erkennen.:
Das Format von 'http://devimages.apple.com/iphone/samples/bipbop/gear1/prog_index.m3u8' konnte nicht festgestellt werden. Sehen Sie für Details im Fehlerprotokoll nach.

Url tested with nightly: http://devimages.apple.com/iphone/samples/bipbop/gear1/prog_index.m3u8 This URL doesn't use relative paths in the m3u8 file, so it does play fine with VLC 1.1.4.

As soon as the m3u8 file uses relative paths (e.g. 1240/1240171.ts in the m3u8 example above) it produces an invalid URL by url-encoding the / character.

Original author: ServerAlex

I had a look at the GIT now.. not sure why the nightly build isn't playing the file.

BUT: The error exists there, too.

modules/demux/playlist/m3u.c (l.220): psz_mrl = ProcessMRL( psz_parse, p_demux->p_sys->psz_prefix ); modules/demux/playlist/playlist.c (l.216): char *postfix = encode_URI_component( psz_mrl );

There is even a comment in this file, which exactly describes the issue: modules/demux/playlist/playlist.c (l.217): /* FIXME: postfix may not be encoded correctly (esp. slashes) */

Please fix. Thanks!

Original author: ServerAlex

Please patch, thanks.

unassigned @Courmisch

added Component::Core: Playlist + 1 deleted label and removed 1 deleted label

In nightly build VLC media player 1.2.0-git-20110224-1439 Twoflower there is still a problem getting the translation of the m3u8 to ts URL's right

stream_filter_httplive info: HTTP Live Streaming (http://video.danmon.dk:80/iPhone/iphone-src/riders.mp4-20110207-141516.m3u8) stream_filter_httplive debug: parse_M3U8 #EXTM3U #EXT-X-TARGETDURATION:10 #EXTINF:10, /iPhone/iPhone-src/Riders.mp4-20110207-141516/1-17118.ts #EXTINF:10, /iPhone/iPhone-src/Riders.mp4-20110207-141516/2-26106.ts #EXTINF:6, /iPhone/iPhone-src/Riders.mp4-20110207-141516/3-16073-END.ts #EXT-X-ENDLIST stream_filter_httplive info: VOD Playlist HLS protocol version: 1 stream_filter_httplive info: video on demand (vod) mode stream_filter_httplive info: Choose segment 0/3 main debug: creating access 'http' location='http:80//video.danmon.dk:80/iPhone/iphone-src//iPhone/iPhone-src/Riders.mp4-20110207-141516/1-17118.ts', path='\http:80\video.danmon.dk:80\iPhone\iphone-src\iPhone\iPhone-src\Riders.mp4-20110207-141516\1-17118.ts' main debug: looking for access module: 2 candidates access_http debug: http: server='http' port=80 file='//video.danmon.dk:80/iPhone/iphone-src//iPhone/iPhone-src/Riders.mp4-20110207-141516/1-17118.ts' main debug: net: connecting to http port 80 main error: cannot resolve http port 80 : V�rten kendes ikke. access_http error: cannot connect to http:80 main debug: net: connecting to http port 80 main error: cannot resolve http port 80 : V�rten kendes ikke. access_mms error: cannot connect to http:80 main debug: no access module matching "http" could be loaded main debug: TIMER module_need() : 4605.462 ms - Total 4605.462 ms / 1 intvls (Avg 4605.462 ms) main debug: waitpipe: object killed main error: no suitable access module for http://http:80//video.danmon.dk:80/iPhone/iphone-src//iPhone/iPhone-src/Riders.mp4-20110207-141516/1-17118.ts' stream_filter_httplive error: fetching first segment failed. main debug: no stream_filter module matching "any" could be loaded main debug: TIMER module_need() : 4606.797 ms - Total 4606.797 ms / 1 intvls (Avg 4606.797 ms) main debug: looking for stream_filter module: 1 candidate main debug: using stream_filter module "stream_filter_record" main debug: TIMER module_need() : 0.331 ms - Total 0.331 ms / 1 intvls (Avg 0.331 ms) main debug: creating demux: access='http' demux='' location='http://video.danmon.dk:80/iPhone/iphone-src/riders.mp4-20110207-141516.m3u8' file='\\http:\\video.danmon.dk:80\iPhone\iphone-src\riders.mp4-20110207-141516.m3u8' main debug: looking for demux module: 55 candidates playlist debug: not enough data es error: cannot peek es error: cannot peek es error: cannot peek es error: cannot peek playlist debug: not enough data playlist debug: not enough data ps error: cannot peek mpgv error: cannot peek mjpeg warning: no more data mjpeg error: cannot peek lua debug: Trying Lua scripts in C:\Documents and Settings\gen-lj.DANMON\Application Data\vlc\lua\playlist lua debug: Trying Lua scripts in C:\Programmer\VideoLAN\VLC\lua\playlist lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\anevia_streams.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\anevia_xml.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\appletrailers.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\bbc_co_uk.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\break.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\canalplus.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\cue.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\dailymotion.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\france2.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\googlevideo.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\jamendo.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\joox.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\katsomo.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\koreus.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\lelombrik.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\megavideo.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\metacafe.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\metachannels.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\mpora.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\pinkbike.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\rockbox_fm_presets.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\youtube.luac lua debug: Trying Lua playlist script C:\Programmer\VideoLAN\VLC\lua\playlist\youtube_homepage.luac avcodec debug: trying url: \\http:\\video.danmon.dk:80\iPhone\iphone-src\riders.mp4-20110207-141516.m3u8 avcodec warning: cannot peek vobsub debug: could not read vobsub IDX file ps error: cannot peek main debug: no demux module matching "m3u8" could be loaded main debug: TIMER module_need() : 57.768 ms - Total 57.768 ms / 1 intvls (Avg 57.768 ms) main error: no suitable demux module for http/://video.danmon.dk/m3ugen/iphone-src/riders.mp4' main debug: removing module "stream_filter_record" main debug: removing module "access_http" main debug: waitpipe: object killed main debug: dead input main debug: thread ended main debug: changing item without a request (current 4/5) main debug: nothing to play qt4 debug: IM: Deleting the input main debug: Destroying the input for 'http://video.danmon.dk/m3ugen/iphone-src/riders.mp4' main debug: TIMER input launching for 'http://video.danmon.dk/m3ugen/iphone-src/riders.mp4' : 5395.106 ms - Total 5395.106 ms / 1 intvls (Avg 5395.106 ms)

added Component::Demuxers: Playlist label and removed Component::Core: Playlist label

changed title from m3u8 playback errors to m3u8/HLS relative URL resolution

More confirmation that resolving relative URLs in M3U8 (HLS) playlists isn't working perfectly...

1-2-0-git-201107220-0002

When presented with the following playlist from our server at http://barque:8081/test

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=4526000
/test?variant=0
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=4000000
/test?variant=1

VLC requests:

http://barque:8081//test?variant=0

note additional slash in path part. We can probably work round that but just for completeness...

Good news, though, is that the %-escaping problem that was in 1.1.10 has been fixed, and it is now using HTTP/1.1 - excellent.

Cheers

Paul Clark Packet Ship

The hosts of relative URIs are wrong too in case the host of the meta playlist is different from the host of the bit-rate specific playlist. The URIs used for fetching the segments are always built from the host of the meta playlist, which is wrong.

Eg. Livestream's iPhone streams work like this. http://iphone.livestream.com/

For a particular example: http://iphone.livestream.com/fox8news/ Which contains a video tag referring to this meta playlist: http://xfox8newsx.api.channel.livestream.com/3.0/playlist.m3u8

The above meta playlist contains URLs like this:

#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=198000
http://ichunk.livestream.com/212.67/livestreamiphone/fox8news-jbates000020111017-053733_198/playlist.m3u8?wowzasessionid=1986120311
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=52
http://ichunk.livestream.com/212.67/livestreamiphone/fox8news-jbates000020111017-053733_198/playlist.m3u8?wowzaaudioonly&wowzasessionid=1986120311

And these segment playlists contain relative URIs like this:

#EXTM3U
#EXT-X-ALLOW-CACHE:NO
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:689
#EXTINF:2,
media_689.ts?wowzasessionid=1986120311
#EXTINF:8,
media_690.ts?wowzasessionid=1986120311
#EXTINF:4,
media_691.ts?wowzasessionid=1986120311

The relative_URI() function uses the host of the meta playlist, thus in the example above it'll try to fetch the first segment from this URI: http://xfox8newsx.api.channel.livestream.com/212.67/livestreamiphone/fox8news-jbates000020111017-053733_198/media_689.ts?wowzasessionid=1986120311

The path is correctly taken from the segment playlist, but the host is taken from the meta playlist. The correct URI would be: http://ichunk.livestream.com/212.67/livestreamiphone/fox8news-jbates000020111017-053733_198/media_689.ts?wowzasessionid=1986120311

A possible patch for relative_URI() that worked for me is:

static char *relative_URI(stream_t *s, const char *uri, const char *path, const char *host)
{
    stream_sys_t *p_sys = s->p_sys;

    char *p = strchr(uri, ':');
    if (p != NULL)
        return NULL;

    if (p_sys->m3u8.psz_path == NULL)
        return NULL;

    char *psz_path = strdup(p_sys->m3u8.psz_path);
    if (psz_path == NULL) return NULL;
    p = strrchr(psz_path, '/');
    if (p) *p = '\0';

    char *psz_uri = NULL;
    if (p_sys->m3u8.psz_password || p_sys->m3u8.psz_username)
    {
        if (asprintf(&psz_uri, "%s://%s:%s@%s:%d%s/%s", p_sys->m3u8.psz_protocol,
                     p_sys->m3u8.psz_username, p_sys->m3u8.psz_password,
                     host ? host : p_sys->m3u8.psz_host, p_sys->m3u8.i_port,
                     path ? path : psz_path, uri) < 0)
            goto fail;
    }
    else
    {
        if (asprintf(&psz_uri, "%s://%s:%d%s/%s", p_sys->m3u8.psz_protocol,
                     host ? host : p_sys->m3u8.psz_host, p_sys->m3u8.i_port,
                     path ? path : psz_path, uri) < 0)
           goto fail;
    }
    free(psz_path);
    return psz_uri;

fail:
    free(psz_path);
    return NULL;
}

And parse_AddSegment():

static int parse_AddSegment(stream_t *s, hls_stream_t *hls, const int duration, const char *uri)
{
    assert(hls);
    assert(uri);

    /* Store segment information */
    char *psz_path = NULL;
    if (hls->url.psz_path != NULL)
    {
        psz_path = strdup(hls->url.psz_path);
        if (psz_path == NULL)
            return VLC_ENOMEM;
        char *p = strrchr(psz_path, '/');
        if (p) *p = '\0';
    }
    // Use the playlist's host if available.
    char *psz_host = NULL;
    if (hls->url.psz_host != NULL)
    {
        psz_host = strdup(hls->url.psz_host);
        if (psz_host == NULL)
            return VLC_ENOMEM;
    }
    char *psz_uri = relative_URI(s, uri, psz_path, psz_host);
    free(psz_path);

    vlc_mutex_lock(&hls->lock);
    segment_t *segment = segment_New(hls, duration, psz_uri ? psz_uri : uri);
    if (segment)
        segment->sequence = hls->sequence + vlc_array_count(hls->segments) - 1;
    vlc_mutex_unlock(&hls->lock);

    free(psz_uri);
    return segment ? VLC_SUCCESS : VLC_ENOMEM;
}

relative_URI() is called in parse_StreamInformation() too. I didn't dive into it's details (what it does, how it works), just added the extra host argument to the call as a NULL:

static int parse_StreamInformation(stream_t *s, vlc_array_t **hls_stream,
                                   hls_stream_t **hls, char *p_read, const char *uri)
{
...
    char *psz_uri = relative_URI(s, uri, NULL, NULL);
...
}

The above behaviour is present in vlc-snapshot-20111016 daily snapshot (of the 1.2.0 dev tree).

P.S.: and actually the problem is not just with the host ... the port, protocol, username, password are all the same. relative_URI() uses for these the values from the meta playlist and not the value from the segment's playlist (if such values are present in the segment's playlist's URI). The fix for all these is pretty much plausible based on the fix for the host problem.

Another problem is that playback stops after the initial segments (from the first download of the segment playlist). In the previous example you can see that in a single playlist there're three segments (of course this can change between playlist providers) and each segment is 10s long. Trying to play it back with VLC, only the first 30s (these first three segments) are played and then it stops. I didn't go into the cause, just wanted something to be played back and the 30s are OK for now (at least for my purposes). But for a regular user 30s are obviously not enough.

(P.S.: I understand that this is -probably- a completely different problem/bug. Feel free to open a new ticket if you like.)

When I use the link mentioned above http://xfox8newsx.api.channel.livestream.com/3.0/playlist.m3u8, than in the server seems to return on one occasion

#EXTM3U
#EXT-X-ALLOW-CACHE:NO
#EXT-X-TARGETDURATION:12
#EXT-X-MEDIA-SEQUENCE:1258
#EXTINF:10,
media_1258.ts?wowzasessionid=1174766038
#EXTINF:9,
media_1259.ts?wowzasessionid=1174766038
#EXTINF:11,
media_1260.ts?wowzasessionid=1174766038

and with the next request it returns:

#EXTM3U
#EXT-X-ALLOW-CACHE:NO
#EXT-X-TARGETDURATION:12
#EXT-X-MEDIA-SEQUENCE:1258
#EXTINF:10,
media_1258.aac?wowzasessionid=1174766038&wowzaaudioonly
#EXTINF:9,
media_1259.aac?wowzasessionid=1174766038&wowzaaudioonly
#EXTINF:11,
media_1260.aac?wowzasessionid=1174766038&wowzaaudioonly

Can someone explain to me why the additional "&wowzaaudioonly" appears???

commit 5de5d154 Author: Jean-Paul Saman jean-paul.saman@m2x.nl Date: Wed Jan 18 16:48:52 2012 +0100

stream_filter/httplive.c: Rewrite relativeURL (fix [#4268](https://code.videolan.org/videolan/vlc/-/issues/4268))

The relativeURL function missed various checks to determine the correct
location for the next download. Checking not only the meta playlist locations,
but also the HLS playlist locations itself should fix [#4268](https://code.videolan.org/videolan/vlc/-/issues/4268).

Original author: jean-paul.saman@m2x.nl

added Status::fixed label

closed

changed milestone to %2.0.0 bugs

m3u8/HLS relative URL resolution

Child items 0

Activity