The high priority CUDA stream causes libOBS to be at a lower priority than the tracking, which is not what we want. Instead we want tracking to be incomplete in those cases, rather than slowing down encoding and other things.
Geometry updates are also now done once per frame instead of one per tracking update, which should improve the smoothness without affecting performance too much. Additionally all tracking info is now in the 0..1 range, which drastically simplifies some math - especially with texture coordinates.
To deal with tracking and updates being asynchronous, a very simple approximation of movement velocity has been added. This is mostly wrong, but it can bridge the gap where tracking updates are slower, as the values are all filtered anyway.
Adds a new CMake option "ENABLE_PROFILING" which enables all CPU and GPU performance profiling available in StreamFX for tracking what's actually causing things to be slow.
Asynchronous rendering allows the GPU to perform work while the CPU performs other work, and is significantly faster than lockstep immediate rendering. By reusing existing render targets we can see a performance improvement of up to 500%, while still doing the same things.
The new libOBS API allows us to directly access the underlying API instead of having to mess around in memory. By using it we can avoid crashing in case the compiler for it is different, or in case the actual back end structure changes.
Additionally the mostly unimplemented and unused options have also been removed, which streamlines the use of this class even further and reduces both shader and code complexity.
Finally by optimizing the use of the internal render target we can achieve a speed up of up to 3000% over the old way, allowing for many more mipmapped filters.
Q_INIT_RESOURCE and Q_CLEANUP_RESOURCE can't be called from within a namespace and instead have to be in outside of the namespace, so by moving them into small inline functions we can fulfill this restriction.
Related: #192#155
Fixes rendering at unexpected sizes by first rendering to a render target and then rendering the contents of that render target to the frame buffer instead. This also prevent rendering twice or more, which might cause severe FPS impact.
Implements support for various new UI features that weren't possible up until now, such as an 'About StreamFX' window with a thank you to everyone that supported the project up until now.
Through converting the code to a threaded asynchronous approach, the libOBS video renderer no longer has to wait on our tracking code to run, and we can enjoy a little bit of extra calculation time before we actually have to do anything.
However due to the remaining synchronization with the Direct3D11/OpenGL context, it is not entirely safe to spend a full frame tracking as libOBS will then start skipped/dropping frames. Even though the priority of the stream is now increased, this still means that we can't just sit around and have to quickly finish all work.
Related #150
Load additional functions from CUDA and add new enumerations to support them:
* cuDevicePrimaryCtxSetFlags allows us to sched scheduling mode for the GPU.
* cuCtxgetStreamPriorityRange allows us to check which priority levels are supported.
* cuStreamCreateWithPriority allows us to create streams with non-default priority.
The scheduler mode is now set to yield so that other threads can do work when we hit an eventual stalling problem. Streams can also now be created with higher priority and different flags, if necessary. In most cases this should allow CUDA resources to execute even while the GPU is under heavy load.
Previously sources had to manually implement migration code, which resulted in unresolvable regression issues due to the lack of version and commit tagging. With the new migration code, all sources automatically have this version and commit tagging at all times, and as such can now have a temporary regression fixed without the user needing to change any values manually.
As OBS Studio locks some mutexes in a different order depending on what actions are being done, using modified_properties for GPU work causes things to freeze in place. Instead have users manually click the refresh button when they changed files in order to prevent this freeze from happening.
Fixes: #118
With this, GCC 8 and above should now be able to compile the project both in obs-studio and as a standalone install. Some features are currently still not fully supported and require extra work, but the majority of things are supported and work out of the box. Exact feature parity can be looked up here on the wiki: https://github.com/Xaymar/obs-StreamFX/wiki/Platform-Feature-Parity
Related: #119#98#30
This header includes all common data between headers used in the plugin. This should improve cross-platform compiling support whenever possible, as all platform-dependent common includes and defines can be done here.
Ever wished you had a professional camera operator to highlight and follow the action, ensuring the audience never misses a beat? Thanks to NVIDIA, you can now do this at home for free! The new NVIDIA AR SDK unlocks augmented reality features, including motion tracking for faces.
This allows me to provide you with an automated zoom and cropping solution for your video camera to transform your streams into a slick, polished broadcast, where you’ll always be the star of the show. Don’t forget - everything is customizable so the possibilities are endless. You can even recreate that Futurama squinting meme if you wanted to (with some scripting)!
The filter requires compatible Nvidia RTX hardware and the Nvidia AR SDK Runtime to be installed ahead of time. This filter is considered "stable" and shouldn't change much from version to version.
Due to the 'nvcuda' library being part of the driver, it falls in a clause of the GPL which allows us to load and interface with system drivers. Since we can't rely on Nvidias headers here (incompatible license), most of this was pulled from FFmpeg and other things were found out via testing.
'Time.x' gets inaccurate if OBS Studio is running for more than two hours, therefore we have to do something to fix it. By allowing the shader code to control when things loop using 'Time.y' (0..1) and 'Time.z' (the number of times 'Time.y' wrapped back to 0), a much more stable animation can be achieved.
These allow you to apply any kind of filtering to a any source, using just standard HLSL. Just like transitions, one extra parameter is set called 'InputA'.
Fixes#95
With this, the first proper shader effect is now possible. By using the four new automated shader parameters 'InputA', 'InputB', 'TransitionTime' and 'TransitionSize' you can write your own transition in HLSL.
Fixes#96
Due to render logic required for transitions, some of the render logic is split into an additional function called 'prepare_render'. Additionally the storage for some temporary objects has been removed as it these objects usually do not outlive their rendering time anyway.
Related: #96#95#94#5
This fixes#116 which was caused by a refactor in commit efb6b0b9be. This bug was left undiscovered until users started upgrading from the last stable version to the current pre-release.
For an unknown reason, OBS Studio v25.x now causes a freeze to happen at this location. This should hopefully work around that issue by ensuring that we are in a location that does not cause an unusual lock order.
On highly parallel systems (> 4 Threads) audio had a chance of being played back out of order, causing it to jitter. This queue should help eliminate the issue entirely.
Fixes#111
Previously a wrong blend state caused a slight discoloration on transparent sources, which was caused by assuming them to always be fully solid without transparency. By instead relying on OBS to do the rendering we do not have to deal with blend states as much and instead can simply enjoy the result.
Fixes#104
* Remove Minimum Bitrate as it is not supported by the encoder.
* Add several default states (-1 or Default) to options.
* Possibly fix bitrate lock with CQP and CQ mode.
* Fix log output for bitrate and B-Frames.
The new logic drastically simplifies Source Mirror and reduces the attack surface for bugs introduced by humans. Additionally the new layout detection should help with improved audio mirroring which hopefully will not crash libobs as often.
Fixes#61.
This thread pool can take large or small tasks and as such alleviates the burden of having a thread per source. Particularly for large setups, this drastically reduces the number of threads running in the background waiting for work.
Fixes support for OBS_SOURCE_CUSTOM_DRAW sources and refactors the class onto better isolated and wrapped classes to deal with specific tasks. This drastically improves stability without causing code complexity to increase, and makes the code vastly easier to read too.
Related: #99
Scaling shouldn't be part of the Source and instead should be done as a filter. Not only does supporting it drastically increase code complexity, it also doesn't add anything that is really necessary as you can do everything it did better in an actual transform.
Caching wasn't actually used except for scaling and was mostly broken too, causing flickering.
Adds support for specifying Minimum Bitrate directly in the UI instead of requiring custom settings to do so. Additionally Adaptive I/B-Frames are now only shown if Look-Ahead is a value greater than 0 frames.
Quality Minimum can also now be left at a default value of -1, the Quality group is no longer toggleable and Quality Target moved into the group. Settings options on the context is now searching children too (if there are any).
Finally, some C++17 formatting was done.
Fixes#101
Scaling is now fully supported for Floats and Integers, which allows much higher precision inputs, or upscaling to a different range. Complex functions for scaling are not supported as those would be a scripting thing and should be kept as that (OBS Studio has built in Lua scripting).
Additionally, enumerations are now correctly loaded with data.
Related #5
Allow for overriding type and size of an element, opening the path for `int#[]`, `float#[]`, `int#x#`, `float#x#`, `bool#x#`, `vector<type, #>` and `matrix<type, #, #>`. Also allows for specifying the exact type of texture instead of hoping the user gets it right, as well as samplers.
Parameters are also now created if they are invisible, which means that the properties() function must not be called, but they must still be used like any other. This is due to a problem with default values not being applied all the time, and sometimes just vanishing.
The code also now throws exceptions with reasonable text, which should be caught by the gfx::shader implementation and refuse a load of the effect. No other state should be modified at that point, so care must be taken that up until the moment the complete initialization is done no other state is modified.
Thanks to the workaround in obs::tools, gfx::shader::shader now supports dynamically rebuilding the properties with new properties without crashing OBS Studio. This effectively allows you to have an up to date view of the current parameters for the shader technique.
Additionally with file watching, live development of shaders is possible at very little cost. Currently only file times and size is looked at every 333ms, but in the future it is possible to also watch for file renames and more.
The current implementation of obs_properties_remove_by_name corrupts the obs_properties_t object whenever it is called on the first or last property in the list. This leads to rapid unscheduled disassembly, and therefore must be fixed in order for this function to be used.
Fixed by upstream PR https://github.com/obsproject/obs-studio/pull/2257.
Removes the stutter when re-opening the properties dialog which was caused by recompiling the same exact shader every time this happened. Also paves the way for simple file watching.
Getting the resolution of a source is very expensive in libOBS, as libOBS does not cache it and instead always calls into the filters and sources to determine the actual source. This also leads to potential lockups due to the filter list mutex being locked for the target source.
Therefore instead of calling it multiple times, cache the result of the call, if that result is even necessary. This reduces the need to synchronize lightly parallelized work (UI and libOBS code) and helps against the potential race condition in libOBS.
Prevents the use of get() and reset() where not actually needed and forces us to actually implement all of the methods needed to interface with the object, leading to cleaner and safer code.
These two wrap the underlying gs_epass and gs_effect_technique objects, to allow direct and improved access to them without relying on the libobs API to provide this access for us. Additionally these make it safe for us to use them instead of relying on C-like code to deal with it.
This drastically improves stability and prevents all exceptions from leaking into libobs C code, which prevents crashes and unexpected freezes from exception handlers further down the stack.
Additionally minor work was done to further improve the quality and user experience for the filter.
Caching the output of a source is only necessary for really expensive to render sources, so it is disabled by default now. Thanks to that, most Source Mirrors are now "free" instead of requiring two context switches and a texture, while those really expensive can be manually set to cache.
The scaling mode is also set to disabled instead of point when rescaling is off to further improve performance. The previous method would incorrectly cause an extra texture to be used.
Additionally we now have support for debug markers for graphics debugging, allowing us to exactly tell apart improvements in rendering cost for this source.
This class and template should be used to reduce the code clutter from repeatedly doing the same thing. It requires OBS v24.0 or newer since the get_properties2 and get_defaults2 API were fully implemented with it.
This is a massive improvement to stability and safety when using the plugin, as all exceptions should now no longer be leaked into C controlled code, which can't actually handle exceptions at all.
Warnings fixed:
* Potentially throwing exception during library load.
* Possibly throwing function passed to C.
* Statement does nothing.
* Variable is initialized but not referenced.
* Variable overloads variable in parent scope.
It is extremely problematic to throw C++ exceptions into C code, especially because C code usually does not handle C++ exceptions at all. Therefore we have to prevent any exception from leaving the function and define it as noexcept.
The util::event code suffers from the problem that it could call into a class that no longer exists, corrupting memory or even crashing. By tracking lifetime using std::weak_ptr<void> this can be avoided and the dead listeners can be automatically removed.
This fixes the case where the effect vanishes, but we still hold an invalid pointer to a parameter of that effect. With the new code, the effect will not vanish as long as an effect pointer exists.
However, all effects must be created either through std::shared_ptr or std::make_shared. If they were not made through one of those means, the code will crash on the ->shared_from_this() call.
Adds a new property to control the alignment of the source within the calculated boundary when rescaling the source. Also fixes the permanently left aligned mirror at the same time.
Similar to the one used in enc-amf, allows looking up string containing other strings, thus drastically simplifying the necessary time to build translated strings, while also caching them for future use.
When copying or re-ordering filters, an occasional crash can be observed due to the rendering not being possible yet. With this, it will simply revert back to the original source instead of crashing.
Fixes the bug that rendering an outline would remove all other effects not directly under the outline, while also improving the rendering speed drastically by only initializing and clearing the rendertarget once instead of for each effect.
This filter allows the use of another source as a mask, allowing complex filter graphs and trippy effects, such as creating a text source with three animated videos, each using a different color channel as the mask.
Fixes#18.
When a source was renamed, obs::source_tracker did not use the new name and instead relied on the old name. This caused source lists with this feature to slowly degrade over time as sources were renamed, until libobs was restarted.
With this fix the renamed sources are now properly tracked, updating as expected.
Related Issue: #64
Clang on Windows and Clang on Linux behave differently, and of course GCC on Windows (MinGW) and GCC on Linux do too. This is the point where using either compiler on either platform should successfully compile the project without any issues.
Clang and GCC have a ton of warnings however, which definitely need to be fixed in the near future. Some of them are great warnings, like old C style casts, others are non-sense like suggest brackets.
Fixes#47Fixes#60
Automatically updates the long description (hover text) if the Type or Subtype field is changed, allowing for more contextual information about what the selected information does.
This refactors the SDF Effects to use a normal blend function instead of doing the blend in the effect itself, improving quality and reducing problematic sampling issues. In addition to this, the effect files have been cleaned up slightly and renamed to their proper names. Glow and Stroke are now supported, which solves both #2 and #4 in one go.
The caching optimization has also now been implemented, reducing the number of renders for this filter to 1 for each tick.
This fixes a dead lock due to attempting to lock the same filter mutex twice, causing libOBS to completely lock up on every mutex. This extends the fix for 4fc25cfd8d.
Additionally this switches to a proper queue system for audio, so we no longer cause audio stuttering or audio skipping.
To further distance the code from having to do too much manually, the graphics context is now available as a managed class. All places that previously used obs_enter_graphics and obs_leave_graphics are now using the new gs::context class instead.
libOBS will simply forget to unlock a mutex if you set the boundary size to zero, resulting in the filters dialog for that source simply getting stuck. Why video rendering can affect the audio filter dialogs is a question for another time, but it's best to just accept it and move on.
Previously this would not do anything correctly, as the call to obs_source_get_flags does not return anything for most sources. Instead obs_source_get_output_flags should be used to verify that we actually have a video or audio capable source to work with.
obs::source_tracker allows us to do more over more versions of OBS Studio, so we are able to enumerate any and all Sources, Scenes and Transitions that weren't created private. If we were to rely on libOBS for this, we would need to limit ourselves to version 23 or higher, but with this we can go very far back - as far back as the global event system was fully supported.
Audio capture previously needed an additional class, however there is no need for this if this can be done together with the actual obs::source object. So obs::source now has a new audio_data event that can be listened to, which handles capturing audio data if requested.
Additionally all other events are now selectively listened to, instead of permanently listening to them. This should drastically reduce the overhead of using obs::source.
Dual Filtering (or Dual Kawase) is an approximation of Gaussian Blur that can reach much higher Blur sizes at a much lower cost. However it is locked to a 2^n size, which means that currently it isn't possible to use it for blur sizes like 19, 24 and 31.
The Blur works by using the linear sampling of a GPU, combined with down- and upsampling and carefully placed sampling points. This means that there is no need for a linear optimized version of this Blur.
Related: #45, #6
While Gaussian Blur is not a Blur type that really benefits much from linear sampling, it can be used for a slight quality and gpu usage reduction. However for Area and Directional Blur there is a better alternative: Dual Filtering Blur. And as with all other currently implement Linear versions of Blur, only Area and Directional Blur are supported.
This type of Gaussian Blur also has the loading hitch that exists in normal Gaussian Blur.
Related: #45, #6
Gaussian Blur is another Blur that now supports the new system, increasing the maxium Blur size to 128 and adding support for Rotational and Zoom blur. Various optimizations were done to the actual shader code which further reduced the GPU usage.
Currently the Gaussian curve is recalculated when the blur is first created, which can lead to a short hitch due to it having to search for the correct kernels. This is currently unavoidable and expected behavior until a better solution is found.
Related: #45, #6
Box Blur is a prime candidate for Linear optimizations, and as such reducing the total necessary samples by about half. However due to the reduction in samples, only Area and Directional Blur are supported.
Related: #45, #6
Box Blur is the first to be on the new system and has received all the new features and optimizations available. The maximum Blur size has been increased to 128, Rotational and Zoom Blur are supported and a small optimization has been done to the shader.
Related: #45, #6
This is the new code base for blur effects, which is much more effective at being re-usable than the old code base. It allows for much better extendable behavior and uses an interface and factory pattern, instead of hardcoding supported features.
Small performance optimization as this removes the extra texture render that would be done by libobs which we don't actually need as we already have a texture to render to for it.
Rendering happens with cached Geometry that is only updated when necessary, such as when the user changes settings. This optimization was necessary to reduce CPU and GPU usage as we can simply re-use the same geomtry instead of recalculating it.
However, the actual source size was never checked. That meant that when a source changed size through any means, the filter would not update the geomtry accordingly, and the result was a squished or stretched image.
With this, the source size is now checked each tick.
Fixes#48
The number of files in the source directory was a little bit much and just made file naming more complex than it had to be. Therefore all files were moved into subdirectories where it matters.
Filters now reside in source/filters/, Sources in source/sources/, OBS Wrappers in obs/, OBS GS Wrappers in obs/gs/, Transitions will reside in source/transitions, Graphics Helpers will be in gfx/.
MipGenerator.Strength was renamed to MipGenerator.Intensity, which better fits the actual intention of the property. Additionally all the mipmap generator options are now correctly named.
Gaussian blur used to have a hard edge which was caused by the filter width being wider than the actual sampling range, which is what is determined by the size option. In order to fix this, we have to figure out the proper filter width for a sampling range, so that we avoid any and all hard edges.
Additionally removes the old texture method entirely.
Fixes#44
Source Mirror would incorrectly crash if an Audio only source was selected, as it expected a video frame to be present of any size. This fixes the problem by first testing if there is any video information to render and then just catching any exception that happens during rendering.
The normal OBS behavior is to cache files that are loaded from disk, however this is not the behavior we want at all, as it can miss file updates as it never checks the disk for an updated file.
So instead of relying on OBS, this implements the file reading and effect creation without the cache. If a cache is necessary, it would not be the job of this class.
This allows increasing or decreasing the accuracy of the Signed Distance Field at runtime without requiring the initial source to be resized. Use case for this would be small text where the higher quality would only be noticable on the shadow and not the text itself.
Sources were rendering with a black border around them if they had a soft fade, which is due to how the shadow calculated the image sample. With the new shader code this is now fixed and the source looks like it should be. Additionally this removes the bug where enabling any shadow would cause only the texels to draw that were above the threshold.
Additionally this adds support for inverse gradients (min > max) and negative gradients for outer shadows instead of only positive gradients. This technically allows for cleaner shadows.
The nature of this Signed Distance Field based Filter is not to only draw shadows, but to also draw an outline or even glow around a source. Therefore the name "Inner/Outer Shadow (SDF)" does not fit well with the intent of the actual Filter.
Allows for enumerating any kind of source, as long as it has a name and is not private. As obs_enum_scenes is a 23.x and above function, this class should be used instead to remain backwards compatible.
Fixes the m_source_texture = nullptr bug, which happened due to m_tick and m_last_tick both being 0, as no video_tick is called when the source is first created. Instead a boolean flag is now used that is reset to false on each video_tick.
This replaces the use of gs_texture_t* with gs::texture, further improving stability. Additionally some of the useless checks were replaced or reduced.
The filters are always the same for every language and should not be translated, instead only the file type should be translated. This way bad translations will not affect the options that the user can select.
The Transform filter will now no longer render the child source more than once per tick(), resulting in an overall speed up for heavy sources. This also applies to mipmapping and shape rendering. Any other calls to video_render will instead just use the cached texture.
Additionally the crash on exit has been fixed which was caused by strings.hpp containing static const char*s and using these directly in obs calls. Instead we now use #define for those property names
Rendering will now use a cached version of the mirrored source in order to reduce rendering cost drastically for heavy sources or scenes. Additionally rescaling now uses the internal scene instead of being a custom implementation, allowing for some new things.
Rescaling now has a new option: Bounds Type! This option allows control over just how a source is scaled, by default it is set to Stretch, but there are other options that can keep the aspect ratio of the mirrored source in tact.
Additionally the ScalingMethod struct has been replaced with obs_scale_type, which means that Bilinear Low Resolution is now no longer an option.
Implements #35
Initialization happens in the constructor again, as using a delayed initialization only caused update() to be slower.
Additionally refactored some variable names to make more sense.
load() is guaranteed to happen after all sources have been created and no earlier, and we can use update() to guarantee that we don't need to duplicate our initialization code in case we go back to initializing earlier than load().
This fixes the multi-reference issue that caused Source Mirror to keep a permanent reference to a source that should have been released.
The obs::audio_capture was also updated to now use the util::event code for improved callback support. There are a bunch of new additional checks in the acquire_input function that prevent bad references from happening, and is also now self-cleaning through the use of obs::source
By using util::event a single obs::audio_capture can now have multiple callbacks for the same data, without requiring additional obs::audio_capture to be present. Additionally it allows us to stop listening on an obs callback if there are no actual listeners, and start listening as soon as there is a listener.
Move and Copy constructors are currently not supported by this class and thus have been marked deleted. Internally also now using obs::source to keep track of the used source.
There was no proper implementation for all copy and move constructors and operators before, which meant that occasionally the default behavior took over. This is not what we want, as the default behavior doesn't deal well with reference counted pointers.
Related: #32, #33
While Linux was not an original goal of the project, it should still be supported out of the box. Therefore a number of changes are contained in this changeset:
- All C++ .h files were renamed to .hpp.
- All C includes (<....h>) were replaced with C++ includes (<c...>) and missing includes were added.
- std::memset and std::memcpy was replaced with memset and memcpy.
- All anonymous structs were removed where necessary.
- `extern "C"` was removed where it wasn't needed.
- #pragma warning was placed behind an #ifdef _MSC_VER
- The macros for `min`, `max` and `clamp` were removed as they were not used.
- `-fpedantic` was added to the GCC flags for bitmask support.
- `gs::rendertarget_op` is now declared before use.
- std::exception(const char*) was replaced with std::runtime_error(const char*).
- Added aligned_alloc and aligned_free macros for GCC and MSVC support.
- Replaced use of `sprintf_s` with `snprintf`.
- Placed additional guards around windows only code.
Additonally some changes were made that do not affect Linux:
- `NOMINMAX` and `NOINOUT` were removed.
Fixes: #27Fixes: #13
Directional Blur (also known as Motion Blur in some art software) enables full control over the direction the passes are applied in, which can further be combined with the scaling option. With this very simple blur effects can have a massively bigger influence.
Step Scaling allows for improved control over individual Blur passes and can be combined with Directional Blur to imitate certain effects. It also allows smaller Blur sizes to cover a larger area, although with a drastic loss in final quality. For example, if someone is streaming at 720p and wants to have their dynamic 2160p background blurry to a very high degree, this option would allow that at next to no quality loss in the final stream. The actual values that need to be used vary for each source and encoder however, so they will have to be chosen carefully.
Translucent sources previously rendered with wrong alpha, which caused sources to look different than they should. With the new code, this is no longer the case and instead now renders the source correctly.
Additionally the modernized code should reduce any problems with rendering by reducing the amount of duplicated manually written code and instead using classes.
This speeds up Gaussian Blur and Linear Gaussian Blur drastically reduces time spent reading textures and instead uses existing registers - maximizing time spent reading the actual image texture.
See Also: #21 Blur Quality
Similar to Linear Box Blur, this version of Gaussian Blur reduces the total number of sample by up to n. This results in a total sample count (per pass) of O(n+1) for even radii and O(n+2) for odd radii. The quality sacrificed to do this is higher this time, though careful adjustment of the halfTexelDelta value can bring it much closer to normal Gaussian Blur. The current offset however had no noticable effects on visual quality.
See Also: #21 Blur Quality
Linear Box Blur abuses the fact that with Linear Sampling we can sample up to four adjacent texels at the same time and get a correct result for Box Blur back. Using this the total number of sample for Box Blur is reduced by n, making the total either n+1 (Even Radius) or n (Odd Radius).
Additionally all blur effect files have been merged into a single blur.effect file to reduce the time required to change a single parameter name. New blur effects should be added as a new technique instead of as a new effect file.
See Also: #21 Blur Quality
Adds Inner/Outer Shadows for dynamic sources based on signed distance field generation. This is fast, but does add a bit of latency when it comes to updates - which means that moving objects will leave a trail before the generator has a chance to update.
Fixes#3
