0
0
Fork 0
mirror of https://github.com/yt-dlp/yt-dlp.git synced 2024-11-21 02:05:12 +00:00

Improve plugin architecture (#5553)

to make plugins easier to develop and use:
* Plugins are now loaded as namespace packages.
* Plugins can be loaded in any distribution of yt-dlp (binary, pip, source, etc.).
* Plugin packages can be installed and managed via pip, or dropped into any of the documented locations.
* Users do not need to edit any code files to install plugins.
* Backwards-compatible with previous plugin architecture.

As a side-effect, yt-dlp will now search in a few more locations for config files.

Closes https://github.com/yt-dlp/yt-dlp/issues/1389

Authored by: flashdagger, coletdjnz, pukkandan, Grub4K
Co-authored-by: Marcel <flashdagger@googlemail.com>
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
This commit is contained in:
Matthew 2023-01-01 04:29:22 +00:00 committed by GitHub
parent 2fb0f85868
commit 8e40b9d1ec
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
20 changed files with 455 additions and 126 deletions

8
.gitignore vendored
View file

@ -120,9 +120,5 @@ yt-dlp.zip
*/extractor/lazy_extractors.py */extractor/lazy_extractors.py
# Plugins # Plugins
ytdlp_plugins/extractor/* ytdlp_plugins/*
!ytdlp_plugins/extractor/__init__.py yt-dlp-plugins/*
!ytdlp_plugins/extractor/sample.py
ytdlp_plugins/postprocessor/*
!ytdlp_plugins/postprocessor/__init__.py
!ytdlp_plugins/postprocessor/sample.py

View file

@ -61,6 +61,8 @@
* [Modifying metadata examples](#modifying-metadata-examples) * [Modifying metadata examples](#modifying-metadata-examples)
* [EXTRACTOR ARGUMENTS](#extractor-arguments) * [EXTRACTOR ARGUMENTS](#extractor-arguments)
* [PLUGINS](#plugins) * [PLUGINS](#plugins)
* [Installing Plugins](#installing-plugins)
* [Developing Plugins](#developing-plugins)
* [EMBEDDING YT-DLP](#embedding-yt-dlp) * [EMBEDDING YT-DLP](#embedding-yt-dlp)
* [Embedding examples](#embedding-examples) * [Embedding examples](#embedding-examples)
* [DEPRECATED OPTIONS](#deprecated-options) * [DEPRECATED OPTIONS](#deprecated-options)
@ -1110,15 +1112,20 @@ # CONFIGURATION
* If `-P` is not given, the current directory is searched * If `-P` is not given, the current directory is searched
1. **User Configuration**: 1. **User Configuration**:
* `${XDG_CONFIG_HOME}/yt-dlp/config` (recommended on Linux/macOS) * `${XDG_CONFIG_HOME}/yt-dlp/config` (recommended on Linux/macOS)
* `${XDG_CONFIG_HOME}/yt-dlp/config.txt`
* `${XDG_CONFIG_HOME}/yt-dlp.conf` * `${XDG_CONFIG_HOME}/yt-dlp.conf`
* `${APPDATA}/yt-dlp/config` (recommended on Windows) * `${APPDATA}/yt-dlp/config` (recommended on Windows)
* `${APPDATA}/yt-dlp/config.txt` * `${APPDATA}/yt-dlp/config.txt`
* `~/yt-dlp.conf` * `~/yt-dlp.conf`
* `~/yt-dlp.conf.txt` * `~/yt-dlp.conf.txt`
* `~/.yt-dlp/config`
* `~/.yt-dlp/config.txt`
See also: [Notes about environment variables](#notes-about-environment-variables) See also: [Notes about environment variables](#notes-about-environment-variables)
1. **System Configuration**: 1. **System Configuration**:
* `/etc/yt-dlp.conf` * `/etc/yt-dlp.conf`
* `/etc/yt-dlp/config`
* `/etc/yt-dlp/config.txt`
E.g. with the following configuration file yt-dlp will always extract the audio, not copy the mtime, use a proxy and save all videos under `YouTube` directory in your home directory: E.g. with the following configuration file yt-dlp will always extract the audio, not copy the mtime, use a proxy and save all videos under `YouTube` directory in your home directory:
``` ```
@ -1789,19 +1796,68 @@ #### twitter
# PLUGINS # PLUGINS
Plugins are loaded from `<root-dir>/ytdlp_plugins/<type>/__init__.py`; where `<root-dir>` is the directory of the binary (`<root-dir>/yt-dlp`), or the root directory of the module if you are running directly from source-code (`<root dir>/yt_dlp/__main__.py`). Plugins are currently not supported for the `pip` version Note that **all** plugins are imported even if not invoked, and that **there are no checks** performed on plugin code. **Use plugins at your own risk and only if you trust the code!**
Plugins can be of `<type>`s `extractor` or `postprocessor`. Extractor plugins do not need to be enabled from the CLI and are automatically invoked when the input URL is suitable for it. Postprocessor plugins can be invoked using `--use-postprocessor NAME`. Plugins can be of `<type>`s `extractor` or `postprocessor`.
- Extractor plugins do not need to be enabled from the CLI and are automatically invoked when the input URL is suitable for it.
- Extractor plugins take priority over builtin extractors.
- Postprocessor plugins can be invoked using `--use-postprocessor NAME`.
See [ytdlp_plugins](ytdlp_plugins) for example plugins.
Note that **all** plugins are imported even if not invoked, and that **there are no checks** performed on plugin code. Use plugins at your own risk and only if you trust the code Plugins are loaded from the namespace packages `yt_dlp_plugins.extractor` and `yt_dlp_plugins.postprocessor`.
If you are a plugin author, add [ytdlp-plugins](https://github.com/topics/ytdlp-plugins) as a topic to your repository for discoverability In other words, the file structure on the disk looks something like:
yt_dlp_plugins/
extractor/
myplugin.py
postprocessor/
myplugin.py
yt-dlp looks for these `yt_dlp_plugins` namespace folders in many locations (see below) and loads in plugins from **all** of them.
See the [wiki for some known plugins](https://github.com/yt-dlp/yt-dlp/wiki/Plugins) See the [wiki for some known plugins](https://github.com/yt-dlp/yt-dlp/wiki/Plugins)
## Installing Plugins
Plugins can be installed using various methods and locations.
1. **Configuration directories**:
Plugin packages (containing a `yt_dlp_plugins` namespace folder) can be dropped into the following standard [configuration locations](#configuration):
* **User Plugins**
* `${XDG_CONFIG_HOME}/yt-dlp/plugins/<package name>/yt_dlp_plugins/` (recommended on Linux/macOS)
* `${XDG_CONFIG_HOME}/yt-dlp-plugins/<package name>/yt_dlp_plugins/`
* `${APPDATA}/yt-dlp/plugins/<package name>/yt_dlp_plugins/` (recommended on Windows)
* `~/.yt-dlp/plugins/<package name>/yt_dlp_plugins/`
* `~/yt-dlp-plugins/<package name>/yt_dlp_plugins/`
* **System Plugins**
* `/etc/yt-dlp/plugins/<package name>/yt_dlp_plugins/`
* `/etc/yt-dlp-plugins/<package name>/yt_dlp_plugins/`
2. **Executable location**: Plugin packages can similarly be installed in a `yt-dlp-plugins` directory under the executable location:
* Binary: where `<root-dir>/yt-dlp.exe`, `<root-dir>/yt-dlp-plugins/<package name>/yt_dlp_plugins/`
* Source: where `<root-dir>/yt_dlp/__main__.py`, `<root-dir>/yt-dlp-plugins/<package name>/yt_dlp_plugins/`
3. **pip and other locations in `PYTHONPATH`**
* Plugin packages can be installed and managed using `pip`. See [ytdlp-sample-plugins](https://github.com/yt-dlp/yt-dlp-sample-plugins) for an example.
* Note: plugin files between plugin packages installed with pip must have unique filenames
* Any path in `PYTHONPATH` is searched in for the `yt_dlp_plugins` namespace folder.
* Note: This does not apply for Pyinstaller/py2exe builds.
.zip, .egg and .whl archives containing a `yt_dlp_plugins` namespace folder in their root are also supported. These can be placed in the same locations `yt_dlp_plugins` namespace folders can be found.
- e.g. `${XDG_CONFIG_HOME}/yt-dlp/plugins/mypluginpkg.zip` where `mypluginpkg.zip` contains `yt_dlp_plugins/<type>/myplugin.py`
Run yt-dlp with `--verbose`/`-v` to check if the plugin has been loaded.
## Developing Plugins
See [ytdlp-sample-plugins](https://github.com/yt-dlp/yt-dlp-sample-plugins) for a sample plugin package with instructions on how to set up an environment for plugin development.
All public classes with a name ending in `IE` are imported from each file. This respects underscore prefix (e.g. `_MyBasePluginIE` is private) and `__all__`. Modules can similarly be excluded by prefixing the module name with an underscore (e.g. `_myplugin.py`)
If you are a plugin author, add [yt-dlp-plugins](https://github.com/topics/yt-dlp-plugins) as a topic to your repository for discoverability
See the [Developer Instructions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) on how to write and test an extractor.
# EMBEDDING YT-DLP # EMBEDDING YT-DLP

View file

@ -40,8 +40,12 @@ def main():
_ALL_CLASSES = get_all_ies() # Must be before import _ALL_CLASSES = get_all_ies() # Must be before import
import yt_dlp.plugins
from yt_dlp.extractor.common import InfoExtractor, SearchInfoExtractor from yt_dlp.extractor.common import InfoExtractor, SearchInfoExtractor
# Filter out plugins
_ALL_CLASSES = [cls for cls in _ALL_CLASSES if not cls.__module__.startswith(f'{yt_dlp.plugins.PACKAGE_NAME}.')]
DummyInfoExtractor = type('InfoExtractor', (InfoExtractor,), {'IE_NAME': NO_ATTR}) DummyInfoExtractor = type('InfoExtractor', (InfoExtractor,), {'IE_NAME': NO_ATTR})
module_src = '\n'.join(( module_src = '\n'.join((
MODULE_TEMPLATE, MODULE_TEMPLATE,

73
test/test_plugins.py Normal file
View file

@ -0,0 +1,73 @@
import importlib
import os
import shutil
import sys
import unittest
from pathlib import Path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
TEST_DATA_DIR = Path(os.path.dirname(os.path.abspath(__file__)), 'testdata')
sys.path.append(str(TEST_DATA_DIR))
importlib.invalidate_caches()
from yt_dlp.plugins import PACKAGE_NAME, directories, load_plugins
class TestPlugins(unittest.TestCase):
TEST_PLUGIN_DIR = TEST_DATA_DIR / PACKAGE_NAME
def test_directories_containing_plugins(self):
self.assertIn(self.TEST_PLUGIN_DIR, map(Path, directories()))
def test_extractor_classes(self):
for module_name in tuple(sys.modules):
if module_name.startswith(f'{PACKAGE_NAME}.extractor'):
del sys.modules[module_name]
plugins_ie = load_plugins('extractor', 'IE')
self.assertIn(f'{PACKAGE_NAME}.extractor.normal', sys.modules.keys())
self.assertIn('NormalPluginIE', plugins_ie.keys())
# don't load modules with underscore prefix
self.assertFalse(
f'{PACKAGE_NAME}.extractor._ignore' in sys.modules.keys(),
'loaded module beginning with underscore')
self.assertNotIn('IgnorePluginIE', plugins_ie.keys())
# Don't load extractors with underscore prefix
self.assertNotIn('_IgnoreUnderscorePluginIE', plugins_ie.keys())
# Don't load extractors not specified in __all__ (if supplied)
self.assertNotIn('IgnoreNotInAllPluginIE', plugins_ie.keys())
self.assertIn('InAllPluginIE', plugins_ie.keys())
def test_postprocessor_classes(self):
plugins_pp = load_plugins('postprocessor', 'PP')
self.assertIn('NormalPluginPP', plugins_pp.keys())
def test_importing_zipped_module(self):
zip_path = TEST_DATA_DIR / 'zipped_plugins.zip'
shutil.make_archive(str(zip_path)[:-4], 'zip', str(zip_path)[:-4])
sys.path.append(str(zip_path)) # add zip to search paths
importlib.invalidate_caches() # reset the import caches
try:
for plugin_type in ('extractor', 'postprocessor'):
package = importlib.import_module(f'{PACKAGE_NAME}.{plugin_type}')
self.assertIn(zip_path / PACKAGE_NAME / plugin_type, map(Path, package.__path__))
plugins_ie = load_plugins('extractor', 'IE')
self.assertIn('ZippedPluginIE', plugins_ie.keys())
plugins_pp = load_plugins('postprocessor', 'PP')
self.assertIn('ZippedPluginPP', plugins_pp.keys())
finally:
sys.path.remove(str(zip_path))
os.remove(zip_path)
importlib.invalidate_caches() # reset the import caches
if __name__ == '__main__':
unittest.main()

View file

@ -0,0 +1,5 @@
from yt_dlp.extractor.common import InfoExtractor
class IgnorePluginIE(InfoExtractor):
pass

View file

@ -0,0 +1,12 @@
from yt_dlp.extractor.common import InfoExtractor
class IgnoreNotInAllPluginIE(InfoExtractor):
pass
class InAllPluginIE(InfoExtractor):
pass
__all__ = ['InAllPluginIE']

View file

@ -0,0 +1,9 @@
from yt_dlp.extractor.common import InfoExtractor
class NormalPluginIE(InfoExtractor):
pass
class _IgnoreUnderscorePluginIE(InfoExtractor):
pass

View file

@ -0,0 +1,5 @@
from yt_dlp.postprocessor.common import PostProcessor
class NormalPluginPP(PostProcessor):
pass

View file

@ -0,0 +1,5 @@
from yt_dlp.extractor.common import InfoExtractor
class ZippedPluginIE(InfoExtractor):
pass

View file

@ -0,0 +1,5 @@
from yt_dlp.postprocessor.common import PostProcessor
class ZippedPluginPP(PostProcessor):
pass

View file

@ -32,6 +32,7 @@
from .extractor.common import UnsupportedURLIE from .extractor.common import UnsupportedURLIE
from .extractor.openload import PhantomJSwrapper from .extractor.openload import PhantomJSwrapper
from .minicurses import format_text from .minicurses import format_text
from .plugins import directories as plugin_directories
from .postprocessor import _PLUGIN_CLASSES as plugin_postprocessors from .postprocessor import _PLUGIN_CLASSES as plugin_postprocessors
from .postprocessor import ( from .postprocessor import (
EmbedThumbnailPP, EmbedThumbnailPP,
@ -3773,10 +3774,6 @@ def get_encoding(stream):
write_debug('Lazy loading extractors is forcibly disabled') write_debug('Lazy loading extractors is forcibly disabled')
else: else:
write_debug('Lazy loading extractors is disabled') write_debug('Lazy loading extractors is disabled')
if plugin_extractors or plugin_postprocessors:
write_debug('Plugins: %s' % [
'%s%s' % (klass.__name__, '' if klass.__name__ == name else f' as {name}')
for name, klass in itertools.chain(plugin_extractors.items(), plugin_postprocessors.items())])
if self.params['compat_opts']: if self.params['compat_opts']:
write_debug('Compatibility options: %s' % ', '.join(self.params['compat_opts'])) write_debug('Compatibility options: %s' % ', '.join(self.params['compat_opts']))
@ -3810,6 +3807,16 @@ def get_encoding(stream):
proxy_map.update(handler.proxies) proxy_map.update(handler.proxies)
write_debug(f'Proxy map: {proxy_map}') write_debug(f'Proxy map: {proxy_map}')
for plugin_type, plugins in {'Extractor': plugin_extractors, 'Post-Processor': plugin_postprocessors}.items():
if not plugins:
continue
write_debug(f'{plugin_type} Plugins: %s' % (', '.join(sorted(('%s%s' % (
klass.__name__, '' if klass.__name__ == name else f' as {name}')
for name, klass in plugins.items())))))
plugin_dirs = plugin_directories()
if plugin_dirs:
write_debug(f'Plugin directories: {plugin_dirs}')
# Not implemented # Not implemented
if False and self.params.get('call_home'): if False and self.params.get('call_home'):
ipaddr = self.urlopen('https://yt-dl.org/ip').read().decode() ipaddr = self.urlopen('https://yt-dl.org/ip').read().decode()

View file

@ -1,10 +1,10 @@
import contextlib import contextlib
import os import os
from ..utils import load_plugins from ..plugins import load_plugins
# NB: Must be before other imports so that plugins can be correctly injected # NB: Must be before other imports so that plugins can be correctly injected
_PLUGIN_CLASSES = load_plugins('extractor', 'IE', {}) _PLUGIN_CLASSES = load_plugins('extractor', 'IE')
_LAZY_LOADER = False _LAZY_LOADER = False
if not os.environ.get('YTDLP_NO_LAZY_EXTRACTORS'): if not os.environ.get('YTDLP_NO_LAZY_EXTRACTORS'):

View file

@ -29,6 +29,8 @@
expand_path, expand_path,
format_field, format_field,
get_executable_path, get_executable_path,
get_system_config_dirs,
get_user_config_dirs,
join_nonempty, join_nonempty,
orderedSet_from_options, orderedSet_from_options,
remove_end, remove_end,
@ -42,62 +44,67 @@ def parseOpts(overrideArguments=None, ignore_config_files='if_override'):
if ignore_config_files == 'if_override': if ignore_config_files == 'if_override':
ignore_config_files = overrideArguments is not None ignore_config_files = overrideArguments is not None
def _readUserConf(package_name, default=[]): def _load_from_config_dirs(config_dirs):
# .config for config_dir in config_dirs:
conf_file_path = os.path.join(config_dir, 'config')
conf = Config.read_file(conf_file_path, default=None)
if conf is None:
conf_file_path += '.txt'
conf = Config.read_file(conf_file_path, default=None)
if conf is not None:
return conf, conf_file_path
return None, None
def _read_user_conf(package_name, default=None):
# .config/package_name.conf
xdg_config_home = os.getenv('XDG_CONFIG_HOME') or compat_expanduser('~/.config') xdg_config_home = os.getenv('XDG_CONFIG_HOME') or compat_expanduser('~/.config')
userConfFile = os.path.join(xdg_config_home, package_name, 'config') user_conf_file = os.path.join(xdg_config_home, '%s.conf' % package_name)
if not os.path.isfile(userConfFile): user_conf = Config.read_file(user_conf_file, default=None)
userConfFile = os.path.join(xdg_config_home, '%s.conf' % package_name) if user_conf is not None:
userConf = Config.read_file(userConfFile, default=None) return user_conf, user_conf_file
if userConf is not None:
return userConf, userConfFile
# appdata # home (~/package_name.conf or ~/package_name.conf.txt)
appdata_dir = os.getenv('appdata') user_conf_file = os.path.join(compat_expanduser('~'), '%s.conf' % package_name)
if appdata_dir: user_conf = Config.read_file(user_conf_file, default=None)
userConfFile = os.path.join(appdata_dir, package_name, 'config') if user_conf is None:
userConf = Config.read_file(userConfFile, default=None) user_conf_file += '.txt'
if userConf is None: user_conf = Config.read_file(user_conf_file, default=None)
userConfFile += '.txt' if user_conf is not None:
userConf = Config.read_file(userConfFile, default=None) return user_conf, user_conf_file
if userConf is not None:
return userConf, userConfFile
# home # Package config directories (e.g. ~/.config/package_name/package_name.txt)
userConfFile = os.path.join(compat_expanduser('~'), '%s.conf' % package_name) user_conf, user_conf_file = _load_from_config_dirs(get_user_config_dirs(package_name))
userConf = Config.read_file(userConfFile, default=None) if user_conf is not None:
if userConf is None: return user_conf, user_conf_file
userConfFile += '.txt' return default if default is not None else [], None
userConf = Config.read_file(userConfFile, default=None)
if userConf is not None:
return userConf, userConfFile
return default, None def _read_system_conf(package_name, default=None):
system_conf, system_conf_file = _load_from_config_dirs(get_system_config_dirs(package_name))
if system_conf is not None:
return system_conf, system_conf_file
return default if default is not None else [], None
def add_config(label, path, user=False): def add_config(label, path=None, func=None):
""" Adds config and returns whether to continue """ """ Adds config and returns whether to continue """
if root.parse_known_args()[0].ignoreconfig: if root.parse_known_args()[0].ignoreconfig:
return False return False
# Multiple package names can be given here elif func:
# E.g. ('yt-dlp', 'youtube-dlc', 'youtube-dl') will look for assert path is None
# the configuration file of any of these three packages args, current_path = func('yt-dlp')
for package in ('yt-dlp',): else:
if user: current_path = os.path.join(path, 'yt-dlp.conf')
args, current_path = _readUserConf(package, default=None) args = Config.read_file(current_path, default=None)
else: if args is not None:
current_path = os.path.join(path, '%s.conf' % package) root.append_config(args, current_path, label=label)
args = Config.read_file(current_path, default=None) return True
if args is not None:
root.append_config(args, current_path, label=label)
return True
return True return True
def load_configs(): def load_configs():
yield not ignore_config_files yield not ignore_config_files
yield add_config('Portable', get_executable_path()) yield add_config('Portable', get_executable_path())
yield add_config('Home', expand_path(root.parse_known_args()[0].paths.get('home', '')).strip()) yield add_config('Home', expand_path(root.parse_known_args()[0].paths.get('home', '')).strip())
yield add_config('User', None, user=True) yield add_config('User', func=_read_user_conf)
yield add_config('System', '/etc') yield add_config('System', func=_read_system_conf)
opts = optparse.Values({'verbose': True, 'print_help': False}) opts = optparse.Values({'verbose': True, 'print_help': False})
try: try:

171
yt_dlp/plugins.py Normal file
View file

@ -0,0 +1,171 @@
import contextlib
import importlib
import importlib.abc
import importlib.machinery
import importlib.util
import inspect
import itertools
import os
import pkgutil
import sys
import traceback
import zipimport
from pathlib import Path
from zipfile import ZipFile
from .compat import functools # isort: split
from .compat import compat_expanduser
from .utils import (
get_executable_path,
get_system_config_dirs,
get_user_config_dirs,
write_string,
)
PACKAGE_NAME = 'yt_dlp_plugins'
COMPAT_PACKAGE_NAME = 'ytdlp_plugins'
class PluginLoader(importlib.abc.Loader):
"""Dummy loader for virtual namespace packages"""
def exec_module(self, module):
return None
@functools.cache
def dirs_in_zip(archive):
with ZipFile(archive) as zip:
return set(itertools.chain.from_iterable(
Path(file).parents for file in zip.namelist()))
class PluginFinder(importlib.abc.MetaPathFinder):
"""
This class provides one or multiple namespace packages.
It searches in sys.path and yt-dlp config folders for
the existing subdirectories from which the modules can be imported
"""
def __init__(self, *packages):
self._zip_content_cache = {}
self.packages = set(itertools.chain.from_iterable(
itertools.accumulate(name.split('.'), lambda a, b: '.'.join((a, b)))
for name in packages))
def search_locations(self, fullname):
candidate_locations = []
def _get_package_paths(*root_paths, containing_folder='plugins'):
for config_dir in map(Path, root_paths):
plugin_dir = config_dir / containing_folder
if not plugin_dir.is_dir():
continue
yield from plugin_dir.iterdir()
# Load from yt-dlp config folders
candidate_locations.extend(_get_package_paths(
*get_user_config_dirs('yt-dlp'), *get_system_config_dirs('yt-dlp'),
containing_folder='plugins'))
# Load from yt-dlp-plugins folders
candidate_locations.extend(_get_package_paths(
get_executable_path(),
compat_expanduser('~'),
'/etc',
os.getenv('XDG_CONFIG_HOME') or compat_expanduser('~/.config'),
containing_folder='yt-dlp-plugins'))
candidate_locations.extend(map(Path, sys.path)) # PYTHONPATH
parts = Path(*fullname.split('.'))
locations = set()
for path in dict.fromkeys(candidate_locations):
candidate = path / parts
if candidate.is_dir():
locations.add(str(candidate))
elif path.name and any(path.with_suffix(suffix).is_file() for suffix in {'.zip', '.egg', '.whl'}):
with contextlib.suppress(FileNotFoundError):
if parts in dirs_in_zip(path):
locations.add(str(candidate))
return locations
def find_spec(self, fullname, path=None, target=None):
if fullname not in self.packages:
return None
search_locations = self.search_locations(fullname)
if not search_locations:
return None
spec = importlib.machinery.ModuleSpec(fullname, PluginLoader(), is_package=True)
spec.submodule_search_locations = search_locations
return spec
def invalidate_caches(self):
dirs_in_zip.cache_clear()
for package in self.packages:
if package in sys.modules:
del sys.modules[package]
def directories():
spec = importlib.util.find_spec(PACKAGE_NAME)
return spec.submodule_search_locations if spec else []
def iter_modules(subpackage):
fullname = f'{PACKAGE_NAME}.{subpackage}'
with contextlib.suppress(ModuleNotFoundError):
pkg = importlib.import_module(fullname)
yield from pkgutil.iter_modules(path=pkg.__path__, prefix=f'{fullname}.')
def load_module(module, module_name, suffix):
return inspect.getmembers(module, lambda obj: (
inspect.isclass(obj)
and obj.__name__.endswith(suffix)
and obj.__module__.startswith(module_name)
and not obj.__name__.startswith('_')
and obj.__name__ in getattr(module, '__all__', [obj.__name__])))
def load_plugins(name, suffix):
classes = {}
for finder, module_name, _ in iter_modules(name):
if any(x.startswith('_') for x in module_name.split('.')):
continue
try:
if sys.version_info < (3, 10) and isinstance(finder, zipimport.zipimporter):
# zipimporter.load_module() is deprecated in 3.10 and removed in 3.12
# The exec_module branch below is the replacement for >= 3.10
# See: https://docs.python.org/3/library/zipimport.html#zipimport.zipimporter.exec_module
module = finder.load_module(module_name)
else:
spec = finder.find_spec(module_name)
module = importlib.util.module_from_spec(spec)
sys.modules[module_name] = module
spec.loader.exec_module(module)
except Exception:
write_string(f'Error while importing module {module_name!r}\n{traceback.format_exc(limit=-1)}')
continue
classes.update(load_module(module, module_name, suffix))
# Compat: old plugin system using __init__.py
# Note: plugins imported this way do not show up in directories()
# nor are considered part of the yt_dlp_plugins namespace package
with contextlib.suppress(FileNotFoundError):
spec = importlib.util.spec_from_file_location(
name, Path(get_executable_path(), COMPAT_PACKAGE_NAME, name, '__init__.py'))
plugins = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = plugins
spec.loader.exec_module(plugins)
classes.update(load_module(plugins, spec.name, suffix))
return classes
sys.meta_path.insert(0, PluginFinder(f'{PACKAGE_NAME}.extractor', f'{PACKAGE_NAME}.postprocessor'))
__all__ = ['directories', 'load_plugins', 'PACKAGE_NAME', 'COMPAT_PACKAGE_NAME']

View file

@ -33,14 +33,15 @@
from .sponskrub import SponSkrubPP from .sponskrub import SponSkrubPP
from .sponsorblock import SponsorBlockPP from .sponsorblock import SponsorBlockPP
from .xattrpp import XAttrMetadataPP from .xattrpp import XAttrMetadataPP
from ..utils import load_plugins from ..plugins import load_plugins
_PLUGIN_CLASSES = load_plugins('postprocessor', 'PP', globals()) _PLUGIN_CLASSES = load_plugins('postprocessor', 'PP')
def get_postprocessor(key): def get_postprocessor(key):
return globals()[key + 'PP'] return globals()[key + 'PP']
globals().update(_PLUGIN_CLASSES)
__all__ = [name for name in globals().keys() if name.endswith('PP')] __all__ = [name for name in globals().keys() if name.endswith('PP')]
__all__.extend(('PostProcessor', 'FFmpegPostProcessor')) __all__.extend(('PostProcessor', 'FFmpegPostProcessor'))

View file

@ -18,7 +18,6 @@
import html.parser import html.parser
import http.client import http.client
import http.cookiejar import http.cookiejar
import importlib.util
import inspect import inspect
import io import io
import itertools import itertools
@ -5372,22 +5371,37 @@ def get_executable_path():
return os.path.dirname(os.path.abspath(_get_variant_and_executable_path()[1])) return os.path.dirname(os.path.abspath(_get_variant_and_executable_path()[1]))
def load_plugins(name, suffix, namespace): def get_user_config_dirs(package_name):
classes = {} locations = set()
with contextlib.suppress(FileNotFoundError):
plugins_spec = importlib.util.spec_from_file_location( # .config (e.g. ~/.config/package_name)
name, os.path.join(get_executable_path(), 'ytdlp_plugins', name, '__init__.py')) xdg_config_home = os.getenv('XDG_CONFIG_HOME') or compat_expanduser('~/.config')
plugins = importlib.util.module_from_spec(plugins_spec) config_dir = os.path.join(xdg_config_home, package_name)
sys.modules[plugins_spec.name] = plugins if os.path.isdir(config_dir):
plugins_spec.loader.exec_module(plugins) locations.add(config_dir)
for name in dir(plugins):
if name in namespace: # appdata (%APPDATA%/package_name)
continue appdata_dir = os.getenv('appdata')
if not name.endswith(suffix): if appdata_dir:
continue config_dir = os.path.join(appdata_dir, package_name)
klass = getattr(plugins, name) if os.path.isdir(config_dir):
classes[name] = namespace[name] = klass locations.add(config_dir)
return classes
# home (~/.package_name)
user_config_directory = os.path.join(compat_expanduser('~'), '.%s' % package_name)
if os.path.isdir(user_config_directory):
locations.add(user_config_directory)
return locations
def get_system_config_dirs(package_name):
locations = set()
# /etc/package_name
system_config_directory = os.path.join('/etc', package_name)
if os.path.isdir(system_config_directory):
locations.add(system_config_directory)
return locations
def traverse_obj( def traverse_obj(
@ -6367,3 +6381,10 @@ def calculate_preference(self, format):
# Deprecated # Deprecated
has_certifi = bool(certifi) has_certifi = bool(certifi)
has_websockets = bool(websockets) has_websockets = bool(websockets)
def load_plugins(name, suffix, namespace):
from .plugins import load_plugins
ret = load_plugins(name, suffix)
namespace.update(ret)
return ret

View file

@ -1,4 +0,0 @@
# flake8: noqa: F401
# The imported name must end in "IE"
from .sample import SamplePluginIE

View file

@ -1,14 +0,0 @@
# ⚠ Don't use relative imports
from yt_dlp.extractor.common import InfoExtractor
# Instructions on making extractors can be found at:
# 🔗 https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#adding-support-for-a-new-site
class SamplePluginIE(InfoExtractor):
_WORKING = False
IE_DESC = False
_VALID_URL = r'^sampleplugin:'
def _real_extract(self, url):
self.to_screen('URL "%s" successfully captured' % url)

View file

@ -1,4 +0,0 @@
# flake8: noqa: F401
# The imported name must end in "PP" and is the name to be used in --use-postprocessor
from .sample import SamplePluginPP

View file

@ -1,26 +0,0 @@
# ⚠ Don't use relative imports
from yt_dlp.postprocessor.common import PostProcessor
# See the docstring of yt_dlp.postprocessor.common.PostProcessor
class SamplePluginPP(PostProcessor):
def __init__(self, downloader=None, **kwargs):
# ⚠ Only kwargs can be passed from the CLI, and all argument values will be string
# Also, "downloader", "when" and "key" are reserved names
super().__init__(downloader)
self._kwargs = kwargs
# See docstring of yt_dlp.postprocessor.common.PostProcessor.run
def run(self, info):
if info.get('_type', 'video') != 'video': # PP was called for playlist
self.to_screen(f'Post-processing playlist {info.get("id")!r} with {self._kwargs}')
elif info.get('filepath'): # PP was called after download (default)
filepath = info.get('filepath')
self.to_screen(f'Post-processed {filepath!r} with {self._kwargs}')
elif info.get('requested_downloads'): # PP was called after_video
filepaths = [f.get('filepath') for f in info.get('requested_downloads')]
self.to_screen(f'Post-processed {filepaths!r} with {self._kwargs}')
else: # PP was called before actual download
filepath = info.get('_filename')
self.to_screen(f'Pre-processed {filepath!r} with {self._kwargs}')
return [], info # return list_of_files_to_delete, info_dict