Diffusion Studio is a browser-based framework for programmatic video editing. It enables developers to automate complex editing workflows, build AI-powered video editors and create videos at scale.

npm i @diffusionstudio/core

Visit https://docs.diffusion.studio to view the full documentation.

💻 Fully client-side
📦 Fully extensible with Pixi.js
🩸 WebGPU/WebGL API support
🏎️ WebCodecs API support

Let's take a look at an example:

import * as core from '@diffusionstudio/core';

const source = await core.VideoSource // convenience function for fetch -> blob -> file
  .from('https://diffusion-studio-public.s3.eu-central-1.amazonaws.com/videos/big_buck_bunny_1080p_30fps.mp4');

// create a video clip and trim it
const video = new core.VideoClip(source) // compatible with the File API
  .subclip(0, 160); // The base unit is frames at 30 FPS

// create a text clip and add styles
const text = new core.TextClip({ 
  text: 'Bunny - Our Brave Hero', 
  position: 'center', 
  stop: 80, 
  stroke: { color: '#000000' } 
});

const composition = new core.Composition(); // 1920x1080

// this is how to compose your clips
await composition.add(video);  // convenience function for 
await composition.add(text);   // clip -> track -> composition

// export video using webcodecs at 25 FPS
new core.WebcodecsEncoder(composition, { fps: 25 })
  .export(); // use resolution = 2 to render at 4k 

This may look familiar to some. That is because the API is heavily inspired by Moviepy and Swift UI. It models the structure of popular video editing applications such as Adobe Premiere or CapCut. The current state can be visualized as follows:

Whereas each track contains zero or more clips of a single type in ascending chronological order. Clips within a track cannot overlap with other clips, similar to Adobe Premiere etc.

A track will be created implicitly with composition.add(clip) however you can also create them manually like this:

const track = composition.createTrack('text');
await track.add(text0);
await track.add(text1);
await track.add(text2);
...

You can find more examples here., or give them a whirl on: https://examples.diffusion.studio

Remotion acts as a React-based video creation tool, enabling you to render the entire DOM tree as well as the full suite of browser visualization features, such as HTML, CSS, Canvas, etc.. This makes Remotion ideal for beginners looking to create videos with code. However, it is limited to react and relies heavily on the CPU, which can be less efficient compared to GPU backed rendering.

In contrast, Motion Canvas uses a Canvas 2D implementation for rendering. It is intended as a standalone editor for creating production-quality animations. In addition, Motion Canvas uses an imperative API. Instead of rendering markup based on timestamps, elements are procedurally added to the timeline. This approach is perfect for creating animations with code (the intended purpose). However, it usually demands static workflows with little variability, making it difficult to build dynamic applications.

Diffusion Studio combines the strengths of both Remotion and Motion Canvas by offering a declarative (yet framework-agnostic) API like Remotion, while also being GPU-backed like Motion Canvas. Diffusion Studio is optimized for video processing performance, utilizing the latest and greatest technologies (WebGPU and WebCodecs). Its API is specifically designed for building video editing apps and to automate complex video workflows.

Note: Diffusion Studio eliminates the need to pay for rendering server infrastructure, since all processing is performed client-side!

• Video/Audio trim and offset
• Tracks & Layering
• Splitting clips
• Html & Image rendering
• Text with multiple styles
• Web & Local Fonts
• Filters
• Keyframe animations
  • Numbers, Degrees and Colors
  • Easing (easeIn, easeOut etc.)
  • Extrapolation 'clamp' | 'extend'
• Realtime playback
• Hardware accelerated encoding via WebCodecs
• Dynamic render resolution and framerate

✅ Supported ⏰ Not yet supported ❌ Not planned 🔬 Not tested

Contributions to Diffusion Studio are welcome and highly appreciated. Simply fork this respository and run:

npm install

Before checking in a pull request please verify that all unit tests are still green by running:

npm run test

This project has been started in March 2023 with the mission of creating the "video processing toolkit for the area of AI". During an extensive research period, we quickly decided to fully embrace WebGPU, which offers a substantial performance improvement over its predecessor WebGL and technologies alike. The following implementations were evaluated:

• C++ w/ Python bindings - inefficient to develop.
• Rust - early ecosystem (might come back here).
• Typescript - efficient to develop, great performance when gpu based.

They all support WebGPU, however, in the case of Typescript, WebGPU is currently only available in Chromium-based browsers, which is why a WebGL fallback is mandatory.