The Job Nobody Wanted
When I joined the editor team the task on my plate was filter download. Filters live on a server, the app fetches them on demand, and someone has to make that fast and correct. It is plumbing, and I treated it as plumbing for about a month, until a bug report arrived that I could not reason about: a filter that downloaded fine, installed fine, and rendered as a flat grey rectangle on one device and perfectly on another.
To fix that I had to understand what a filter is once it leaves my code, which meant understanding what the editing engine does with a frame, which meant understanding what a frame is. This post is that chain of understanding, written in the order I wish someone had given it to me. It ends back at the download path, because the pipeline turned out to dictate most of the download design, and because the last decision, naming a filter by identifier rather than by file path, is what made cloud drafts possible a year later.
The Pipeline in Five Stages
Strip away every feature and a video editor does one thing: it takes a file apart into pictures, changes the pictures, and puts them back into a file.
The media layer abstracts the platform decoder and encoder; the effect engine never sees a file, only a frame and a set of parameters. Export is the same pipeline as preview, without the real-time deadline.
Demux. A video file is a container. The container holds separate compressed streams, usually one video and one audio, plus an index that says where in the file each chunk of each stream lives. Demuxing reads the container and hands the video stream's compressed samples to the decoder in order. Nothing is a picture yet.
Decode. The decoder turns compressed samples into frames. On iOS this is VideoToolbox, on Android MediaCodec, and in both cases the media layer prefers the hardware path: the device has silicon dedicated to H.264 and HEVC, and software decoding of 4K is both slow and a battery problem. The software path exists as a fallback for codecs or profiles the hardware does not support, and it exists quietly, because a user who can open a file does not care which path opened it. The media layer's job at this stage is exactly one abstraction: give me frame N of this file, with the platform decoder behind it.
Frame. The unit everything else works in. The next section is about what it actually is.
Render. The effect engine takes a frame and a description of what should happen to it, filters, text, transitions, keyframed parameters, and produces a new frame. It runs on the GPU, once per frame, and it is the only stage that knows what a "filter" is. The media layer could not apply a filter if it wanted to; the effect engine could not open a file.
Encode and mux. The reverse of the first two stages. Rendered frames are compressed by the hardware encoder and written into a new container with the audio. Export is this pipeline run from start to finish without a display; preview is the same pipeline run against the clock.
The clock is the number that shapes everything. A 30 fps source gives the pipeline 33.3 milliseconds per frame. A 60 Hz display refreshes every 16.7 milliseconds. During preview the decode, render, and present of one frame have to fit inside the source's frame interval or playback stutters; during export the deadline disappears and the pipeline runs as fast as the encoder allows. Same stages, same code, different constraint.
What a Frame Is and What Render Means
I had used the word frame for years without being able to say what one was in memory. Here is the version that finally stuck.
A display is not continuous. It shows a fixed grid of pixels and replaces the whole grid at its refresh rate. A video is not continuous either: a 24 fps clip is 24 still pictures per second, which is above the threshold where human vision stops seeing individual pictures. So a frame is a picture: a grid of pixels at the video's resolution, each pixel a color.
The size of that grid is worth computing once, because it explains most of the engineering around it. A 1080p frame is 1920 by 1080 pixels, which is 2,073,600 pixels. In RGBA with one byte per channel that is 8.3 MB for one frame, and 30 of them per second is 249 MB/s. That is why decoded frames never leave the GPU if the pipeline can help it, why the decoder emits them in a planar YUV layout (NV12 at 1.5 bytes per pixel, 3.1 MB for the same frame) rather than RGBA, and why "just copy the frame to the CPU and look at it" is the sentence that ends a performance review.
Render is what happens to that grid before it reaches the screen or the encoder. The engine allocates a framebuffer on the GPU, a contiguous block of memory with one slot per pixel, and runs a program per pixel that computes the slot's final color from the input frame and the effect parameters. When every slot is written the framebuffer is complete, and it is handed off: to the display, which reads it on the next refresh, or to the encoder. Preview double-buffers this, rendering into one framebuffer while the display reads the previous one, so the two never contend.
Everything an editor calls an effect is a program in that per-pixel step. Which brings us back to filters.
A Filter Is a Lookup Table
A color filter does not know anything about the picture. It maps input colors to output colors, every pixel independently, the same way for every frame. That makes it a pure function from one RGB triple to another, and a pure function on a finite domain can be stored as a table.
The table is a cube. Sample the input color space at 64 points per axis and you get 64 × 64 × 64 = 262,144 entries, each holding the output color for that input. That is the 3D LUT, and it is the entire filter. There is no shader per filter, no code shipped with it, nothing to compile: the engine has one LUT shader, and a filter is data fed into it.
The same cube renders identically on every platform because the lookup is arithmetic, not code. Intensity is a blend applied after the lookup, which is what a keyframe animates.
Two details of the storage matter to the download path.
The cube is shipped as an ordinary image. Sixty-four slices of 64 × 64, laid out eight by eight, is a 512 × 512 PNG. It is lossless, it is a format every platform decodes, and it compresses well because neighboring entries have neighboring colors. The grey-rectangle bug from the first section was a filter whose PNG had been re-encoded by a tool that applied a color profile on save: the table was subtly wrong on devices that honored the profile and correct on devices that ignored it. The fix was in my code, not the engine's. The download path now verifies the image's dimensions and bit depth before it accepts a package, and the manifest carries a hash of the decoded pixels, not of the file.
The lookup interpolates. An 8-bit input has 256 values per channel and the cube has 64, so the shader lands between entries and interpolates trilinearly among the eight neighbors. That is why 64 is enough: the function a filter represents is smooth, and interpolation reconstructs it. It is also why the lookup is deterministic across platforms, which is the property the cloud-draft work later depended on. Every GPU does the same arithmetic on the same table.
The last parameter is intensity. The shader computes mix(original, lut(original), intensity), and intensity is a scalar the engine receives per frame. At 1.0 the filter is fully applied; at 0.0 the frame passes through. That one scalar is the hook for the next section.
The Download Path
With the pipeline understood, the download design mostly writes itself, because every requirement is a consequence of something above.
A filter is installed atomically or not at all. The registry answers only one question, from id and version to an installed package, and both the engine and the draft loader ask it.
A filter is a package. One zip: the LUT image, a thumbnail for the picker, and a manifest with the filter's identifier, version, display name, default intensity, the hash of the decoded table, and the minimum engine version that can apply it. The manifest is the contract; the image is payload.
The catalog is small and the packages are lazy. On launch the app fetches a catalog: every filter's id, version, hash, and download URL, a few kilobytes. Packages are fetched only when a user opens the picker category that contains them, or when a draft references them. The top entries of the catalog are prefetched on Wi-Fi so the default category is instant. The catalog is also where the engine-version gate is applied: a filter the installed engine cannot render is not shown, which is cheaper than downloading it and failing.
Installation is atomic. The package downloads to a temporary file, the archive is expanded into a temporary directory, the manifest is parsed, the image is decoded and checked against the manifest's dimensions and pixel hash, and only then is the directory renamed into the cache under <id>/<version>/. A rename is atomic on the file system; a half-expanded package can never be observed. A filter that fails any check is deleted and reported, not kept. This was the direct lesson of the grey rectangle, generalized: a corrupt filter is strictly worse than a missing one, because a missing one produces a retry and a corrupt one produces a bug report with no reproduction.
Requests are deduplicated and bounded. Two callers asking for the same package while it is in flight share one download. In-flight downloads are capped at a small number, three on our build, because the picker is a scroll view and an unbounded scroll generates an unbounded request queue. Failures are classified the same way the sync layer classifies them: network failures retry with backoff, a missing or mismatched package is permanent and surfaces as a placeholder in the picker.
The cache evicts by use, with a pinned set. The cache is bounded and evicts least-recently-used packages, the same policy I wrote about for an image cache years ago. The exception is the pinned set: any filter referenced by a draft on this device is never evicted, because evicting it would turn opening a draft into a network request. The pinned set is computed from the drafts, not maintained by hand, so it cannot drift.
The registry resolves identifiers. Everything above exists to serve one function: given a filter id and version, return an installed package or nil. The engine calls it when it builds a render. The draft loader calls it when it opens a document. Nothing else in the app knows where filters live on disk.
Keyframes Close the Loop
The intensity scalar from the LUT section is what makes a filter animatable. A keyframe on filter intensity is two or more points on the timeline, each with a value, and an easing between them. At 0.0 seconds the intensity is 0, at 2.0 seconds it is 1, and the frames between get a value in between.
The question I got wrong at first was who computes the in-between value. The intuitive answer is the effect engine, because the engine is where the filter lives. The right answer is the timeline, on the host side, before the engine is called. For each frame the host evaluates the keyframe curve at the frame's timestamp, linear or eased, producing a single number, and passes that number to the engine alongside the frame. The engine receives a scalar and a frame; it does not know a keyframe exists.
The reason is the same determinism argument as the LUT itself. Curve evaluation is arithmetic, and if it lives in one place that every platform shares, the same document produces the same intensity at the same timestamp on every device. If each platform's engine evaluated curves its own way, two devices would render the same draft differently, and the user would call that corruption. The cloud-draft post lists render consistency as one of the three things to over-invest in. This is where that investment was made.
Why a Draft Names a Filter by Identifier
The decision that paid off longest was the smallest. A clip in the draft document does not record the filter's file path. It records the filter's id and version:
{
"type": "clip",
"filter": { "id": "fx.warm_film", "version": 3, "intensity": 0.8 }
}
The path would have been the obvious choice, and it would have been device-local, which means it would have meant nothing on the user's other phone. The identifier means the same thing everywhere, and resolving it is the registry's one function. When a draft opens and the registry returns nil, the draft loader asks the download path for that id and version, waits, and retries. The user sees a clip with a loading placeholder for a second and then the filter. On the device where the draft was made, the pinned set guarantees the registry never returns nil at all.
A year later the cloud-draft project needed every reference in a document to be meaningful on a device that had never seen it. Assets got content hashes for that reason. Filters already had identifiers, and the download path was already the thing that resolved them. The cold-open flow on a new device, fetch the document, resolve what it references, render, was the same flow that had been running locally since the beginning, with a network hop added at the registry's miss.
Takeaway
The pipeline has five stages and I owned the one that is not even on the diagram: getting a 512 × 512 image onto the device. Understanding the stages did not change what the download path had to do. It changed why, and the whys turned into the design. Frames are large, so filters are data and not code. Lookups are arithmetic, so a filter renders the same everywhere, so a package can be verified by hashing what it decodes to. The engine takes a scalar per frame, so keyframes are the host's job. A draft outlives a device, so it names a filter by what it is and not where it is.
I still think filter download is plumbing. The lesson is that plumbing has to know what flows through it.