Here are just some notes reading patch (I did no tests at all)
MacOS X:
Since dladdr() is a GNU (non POSIX) extension, isn't '#define _GNU_SOURCE' required?
It looks like -ldl is needed to link dladdr() (maybe not a problem for VLC/autotools and probably already required)
Does this line will actually concatenate char* strings? : dirname( info.dli_fname ) + "/modules"
If "init" symbol is detected inside "VLC Plugin" file and if --plugin-path accept several pathes like PATH env var does, sound ok for MacOS X part
Windows:
Is GetModuleFileNameA() compatible with extended characters? (if detected path contains Chinese characters for example)
If it is not, maybe it will just not work (appending a wrong path to the --plugin-path).
I fail to see where "addr" symbol is defined for Windows part. Even nm tool can't find it in VLC folder.
Where is it defined?
Is "addr" symbol choice motivated, or are there better symbols to use? (the same for MacOS X)
dirname() seems to be POSIX. Will is work on Windows?
If "addr" symbol is actually defined in either libvlc*.dll or npvlc.dll, it sound ok. Otherwise I don't know which module/module path will be returned for this symbol.
Idea is good, especially for MacOS X in which moz-plugin embeds most of VLC stuff. If char* addition is fixed, i find no problem in this part of code.
About dladdr, I was looking in my linux man pages, different from Mac OS system. ok
Question: Will dlfcn.h be included at the time dladdr will be used? Maybe not.
About Windows part, I don't know how it will be used because if VLC install dir is not already well detected (usage of libvlc_new), execution will not reach this point. But never mind, things are suddenly more blur for me on this side and the concept is good as for Mac OS.
About HMODULE cast, I never used such windows stuff but in case there are difficulties with it, google return me usages like "HMODULE(mbi.AllocationBase)" and "reinterpret_cast(mbi.AllocationBase)" but it is out of my skills.
About GetModuleFileName, sorry to bother you on such out-of-my-skill windows API but without the trailing 'A' it works with LPCSTR (TCHAR[] looking the example, 2-bytes per char iirc) strings and you are using char*. Maybe '..A' was better for now (handling non extended charecters with a TODO)? I don't know.
Also, MSDN says the result name can be prefixed with a weird "\?".. :-(
Actually I don't have a macosx and a windows image to check the source, the patch is more or less a rough base hoping a windows coder and a macosx dev will test and fix it =\
I have almost no experience in windows so I just took example code and I stitched it together, probably there is even a better way to get the same result in windows.
Probably using the A variant could work better hopefully.
Picking up multiple module paths in vlc is not safe. If you need the dll path, then you probably cannot rely on the default path being from the same application instance.
Is there a way to know that the plugin is started from an xpi package? If so, then I would prefer to use that as a trigger to add the dll path instead of the default path.
using std::string tmp might be convenient, but it mixes c and c++ code, which I find ugly(tm) ;) I'd prefer that you'd stay with c-code there.
Right now the dll path and the registry reported path must be the same, if the code works the registry reported path could be removed. The same applies for macosx.