The Seam Between a Hero `<video>` and the Photo Below Never Lines Up? Video Frames Are a Function of Time, Not Scroll

On a client site, the hero was a six-second looping coast video. Directly below it sat a single still photo that was supposed to continue seamlessly from the video's last visible frame, as if the moving image froze into a portrait under the fold. Great on paper. In practice, the join never lined up. There was always a seam: a broken line right at the boundary between the <video> and the photo beneath it. Sometimes the horizon was off by a few pixels, sometimes a wave got cut mid-motion. Never the same twice.

The frustrating part was that the photo below was static and perfect, and the video itself was smooth. The only thing that was wrong was where the two met, and that meeting point changed every single time the page loaded.

The symptoms

My first instinct was wrong, as usual. I assumed this was a matter of matching frames: if I could force the video to stop on the exact frame as the block boundary entered the viewport, I could just cut the photo from that frame. I tried pause() at a certain scroll point, I tried pinning currentTime, I tried requestVideoFrameCallback to read the active frame. All of it was fragile: currentTime on a <video> element is not accurate to the frame, seeking jumps to the nearest keyframe, and the result still drifted.

On top of that, <video> brings its own family of bugs. There is a brief black flash on the first render (DirectComposition on Chromium), a seek artifact each time the loop returns to the start, and autoplay policies that make the video refuse to play without muted. I spent time patching these one by one before realizing I was fixing the wrong thing.

The root cause

The turning point came when I stopped and wrote down what I was actually asking for. A playing <video> is a function of TIME. The visible frame depends on exactly when the user scrolls to the block boundary, and that is nondeterministic: one visitor arrives at second 2.3, another at second 4.8. The frame at the boundary is random. If the boundary frame is random, I can never cut a matching continuation photo, because I do not know which frame I am supposed to join.

The problem was not inaccurate seeking. The problem was that I was using a source parameterized by time when I needed one parameterized by scroll position. As long as the source was time, the seam could never be controlled.

The fix

The fix: drop <video> entirely and replace it with a sequence of .webp frames drawn onto a <canvas>, where the visible frame is a function of scroll position, not time. If the frame is deterministic against scroll, I can force the block boundary to always land on the same frame, then cut the continuation photo from that frame exactly once.

First step, extract the frames. But a raw loop from the video looks broken when it wraps, so I built a loop that crossfades into itself using xfade in ffmpeg:

ffmpeg -i coast.mp4 -filter_complex "[0]split[a][b];[a]trim=2:8,setpts=PTS-STARTPTS[main];[b]trim=0:2,setpts=PTS-STARTPTS[head];[main][head]xfade=transition=fade:duration=2:offset=4,fps=20,scale=1920:1080:flags=lanczos" -c:v libwebp -q:v 68 -start_number 1 frames/frame-%04d.webp

The markup is simple: one <canvas> that carries its frame list. A canvas is a replaced element, so object-fit: cover stays valid just like on a normal image.

<canvas class="coast-canvas" data-frames="..." data-frame-count="120"></canvas>

While idle, a GSAP loop drives a frame counter, and on each update I draw the matching frame. I skip clearRect because every frame covers the whole canvas:

gsap.to(ani, {
  frame: "+=" + 120,
  duration: 6,
  ease: "none",
  repeat: -1,
  onUpdate: render,
});
 
function render() {
  const i = Math.round(ani.frame) % 120;
  ctx.drawImage(frames[i], 0, 0, canvas.width, canvas.height);
}

The heart of the trick is a handoff zone just before the block edge. A ScrollTrigger spans from blockH - vh - 1.0*vh to blockH - vh - 0.05*vh. When that zone enters, I kill the idle loop, record the current frame as c, then scrub the frame along scroll progress by a distance of k:

const c = ani.frame % 120;
const k = (120 - c) + 120; // always ends on a multiple of 120
 
scrub.onUpdate = (self) => {
  ani.frame = c + self.progress * k;
  render();
};

Why k = (120 - c) + 120? Because 120 - c carries the counter from a random position c up to the next multiple of 120, and + 120 adds one full turn so the motion is not too short when c happens to be large. A multiple of 120, modulo 120, equals frame 0. So no matter what random frame the user arrives on, the block edge is GUARANTEED to land on frame 0. Once the edge frame is always frame 0, I cut the poster strip from frame 1, place it directly under the canvas, and the join is seamless. The seam disappears not because I guessed the right frame, but because I forced the right frame.

An unexpected bonus: once <video> is gone, its whole family of bugs goes with it. No more DirectComposition black flash, no seek artifact, no autoplay drama, because there is no video element at all.

A few production details. I render two asset sets: desktop at 360 frames, 1920x1080, 60fps (around 83MB), and a light set for mobile or weak devices at 180 frames, 960x540, 30fps (around 13MB). The choice runs through matchMedia under 991px, or deviceMemory <= 4, or hardwareConcurrency <= 4. The canvas context is created with alpha: false, a 1:1 CSS-pixel buffer, and render only fires when the rounded index changes so it never redraws the same frame. The loop auto-pauses when the hero leaves the screen, and a preloader gates the handoff until every frame has loaded. A 15fps bake felt choppy; 60fps only became affordable precisely because the loop is only six seconds, so 360 frames stays reasonable.

Checklist

• If the join between a <video> and the element below it is never consistent, remember the video frame is a function of time, not scroll position.
• Do not waste time matching currentTime or seeking; <video> seeking jumps to a keyframe and is not accurate to the frame.
• Replace the hero video with a frame sequence on <canvas> when you need frames that are deterministic against scroll.
• Build a smooth loop with ffmpeg xfade so the start-to-end join does not look broken.
• Use the simple math k = (120 - c) + 120 so the scrub always lands on frame 0 at the block edge, then cut the poster from frame 1.
• Ship responsive assets (fewer frames and smaller resolution for weak devices) and gate the handoff behind a preloader.