We replaced ffmpeg.wasm with WebCodecs. Here’s what actually changed.

The first version of this site’s video tools ran ffmpeg.wasm. It worked, and the numbers were indefensible: a cold visit downloaded and compiled tens of megabytes of WebAssembly before a single frame was decoded, and then decoded that frame on the CPU, single-threaded, while the machine’s dedicated H.264 hardware sat idle three inches away. We rebuilt on WebCodecs. This is what that trade actually costs and buys, because most write-ups of it are either vendor copy or benchmarks with no methodology.

What ffmpeg.wasm is, precisely

It is FFmpeg — the real C codebase, libavcodec and all — compiled to WebAssembly with Emscripten and driven from JavaScript through a shim that mimics a command line and a POSIX filesystem. That framing matters because it explains every one of its properties. You get FFmpeg’s complete codec and container coverage and its entire filter graph, and you get them as portable software with no privileged access to anything.

Cost one: the payload

The core wasm binary is in the region of 25–30 MB, and it is not a lazy dependency — nothing runs until it has been fetched, and the browser then has to compile it. On a fast connection with a warm cache this is a couple of seconds. On a phone on mobile data it is the entire user experience. You can trim custom builds down considerably by disabling codecs you do not need, which requires maintaining an Emscripten toolchain in your build pipeline, and most projects do not.

Cost two: threads, and the header tax

The default build is single-threaded. Multi-threaded FFmpeg needs pthreads, pthreads in WebAssembly need SharedArrayBuffer, and SharedArrayBuffer has been gated behind cross-origin isolation since Spectre. To get it you serve two headers on the document:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

Which then breaks your page. require-corp means every cross-origin subresource must explicitly opt in by sending Cross-Origin-Resource-Policy, or it simply fails to load. Your fonts, your CDN images, your analytics snippet, the YouTube embed in the marketing section, the Stripe or Intercom script — all of them, unless the third party happens to send that header, which most do not. Teams discover this after shipping, usually via a support ticket about a missing logo.

The escape hatch is Cross-Origin-Embedder-Policy: credentialless. It still gives you cross-origin isolation and SharedArrayBuffer, but instead of demanding opt-in it loads cross-origin subresources without credentials — no cookies sent, so nothing user-identifying can leak into your isolated context. Public assets work again. Anything that depends on a cookie to authenticate does not, and iframe content has additional requirements. Support landed in Chromium first and Firefox after; Safari has been slower, so feature-detect rather than assume.

Cost three: it cannot touch the media engine

This is the structural one and no amount of tuning fixes it. Your laptop and your phone contain fixed-function hardware for H.264 and HEVC — VideoToolbox, Media Foundation, VA-API underneath. A WebAssembly module has no path to any of it. It cannot ask for it and the browser cannot offer it. ffmpeg.wasm is doing motion compensation and entropy decoding in portable software on the same cores that are running your UI.

“It is not slow because it is badly written. It is slow because it is the only video decoder on the machine that is not allowed to use the video decoder.”

What WebCodecs gives you instead

WebCodecs exposes the browser’s own codec implementations as JavaScript objects: VideoDecoder, VideoEncoder, AudioDecoder, AudioEncoder, with EncodedVideoChunk going in and VideoFrame coming out. These are the same implementations a <video> element uses, hardware path included. You can ask for it explicitly:

const support = await VideoEncoder.isConfigSupported({
  codec: 'avc1.42001f',
  width: 1920, height: 1080,
  bitrate: 4_000_000,
  hardwareAcceleration: 'prefer-hardware',
});

There is no download and no compile step, because the decoder was already in the browser. A VideoFrame can be drawn straight to a canvas or uploaded as a WebGL texture without a round trip through ArrayBuffer, which is what makes real-time preview possible at all. The widely reported figure for H.264 work is somewhere in the region of 3–10× faster than ffmpeg.wasm; we are citing that as the reported range rather than as our own measurement, and the honest answer is that it depends enormously on codec, resolution, and whether the specific profile is hardware-accelerated on the specific machine. The qualitative claim is safe: it is the difference between using the media engine and not using it.

What WebCodecs takes away

Three things, and they are not small.

• Codec coverage collapses. FFmpeg decodes essentially everything ever shipped. WebCodecs decodes what that browser on that OS happens to support — realistically H.264, VP8, VP9, AV1, and HEVC where the platform licenses it. Hand it ProRes, DNxHD, VC-1, an old DivX file or a broadcast MPEG-2 transport stream and you get nothing.
• There is no filter graph. At all. No scale, no delogo, no crop, no overlay. WebCodecs decodes and encodes; anything between those two steps is your code. Every pixel operation becomes a canvas, WebGL or WebGPU shader you write and maintain yourself.
• There is no muxer or demuxer. This surprises people most. WebCodecs handles elementary streams and knows nothing about MP4 or WebM. Feeding it a file means parsing the container to pull out encoded samples, and writing the output means assembling a valid MP4 — moov atoms, sample tables, timescales, edit lists. You do not write that yourself; you use a library. Mediabunny is the one we settled on: TypeScript, reads and writes MP4, WebM/Matroska and the common audio containers, and drives WebCodecs for you rather than wrapping a second codec implementation.

Support is no longer the blocker it was in 2022. Chromium shipped WebCodecs in 2021, Safari from 16.4, and Firefox more recently — with the caveat that Firefox’s decoder support arrived ahead of its encoder support, so check VideoEncoder separately from VideoDecoder and keep a real fallback for the case where a specific config is unsupported. isConfigSupported is cheap; call it before you build a pipeline, not after.

When you should still reach for ffmpeg.wasm

The dividing line is whether your bottleneck is pixels. If the job is decode → touch every pixel → encode, WebCodecs wins by a margin that changes what is buildable: it is the difference between a tool that processes a clip while you watch and a tool that shows a progress bar and asks for patience. If the job is parsing, remuxing, or a codec nobody has licensed since 2009, use FFmpeg.

Practically, that architecture is what let us ship a delogo tool where you can drag the box and see the result on the current frame immediately — the interpolation is our own shader over a decoded VideoFrame, because WebCodecs gave us no filter to call — and a size-targeted compressor that can re-encode a whole clip locally in the time an upload would have taken. Neither of those is a plausible product on a 30 MB software decoder. Both of them are ordinary on the hardware path.

If you want the FFmpeg side of the same problem, finding delogo coordinates by hand covers what those tools are doing under the surface.