Editing a video with AI is not one button, it is six steps, and in every one of them part of the work runs automatically while another part stays a decision. One, review: a speech model turns your raw footage into a timestamped transcript, you decide which take is the good one. Two, clean up: the tool reliably finds silence by level, filler words only depending on language and product, and content mistakes not at all; you decide which pause survives. Three, the rough cut: scene detection and clip suggestions come from the machine, order and structure stay subject to your approval. Four, subtitles: generated automatically, always proofread. Five, audio: loudness can be measured and normalised to a target value, and this is the step that runs automatically with the most confidence. Six, export: format, bitrate and aspect ratio follow the platform, and every file gets checked afterwards. Step two is the one that saves the most time and most often goes too far: a video with every breath removed sounds rushed, no matter how cleanly the automation did its job.
I no longer edit my own videos. At my end this workflow runs through a fixed role in my AI workforce, on the command line with ffmpeg, a locally running Whisper model and Remotion, and I check the result. Everything I collect on AI at work lives on the AI page; how I decide what gets handed over at all is in Which Tasks You Can Hand Off to AI. Which program you use for this workflow is a separate question I went through with prices and limits in AI Video Editing Software Compared. This article is about the workflow itself.
The workflow in six steps
The right-hand column is the actual work. Tools can now suggest or execute parts of it, including order and length, but in this workflow those decisions remain subject to human approval because a suggestion does not know what matters. The left-hand column has very good tools that free up time for that decision.
Step 1: review and sort the footage
Before the first cut you make a working copy and leave the original untouched. It sounds like a detail and it is the only mistake on this list that cannot be repaired.
Automatic: a timestamped transcript. Whisper does this locally: the command line writes an SRT file directly (--output_format srt), and --language German tells the model the language instead of letting it guess. Alongside it you read resolution, duration and frame rate per file. In a timestamped transcript you can search, and searching beats scrubbing through four hours of footage by orders of magnitude.
Your decision: which take is the good one. A machine can count how often you tripped over a sentence, it cannot hear which attempt landed. If you have three versions of the same passage, you mark the one you want in the transcript, not in the timeline.
One decision belongs here up front: browser services process your raw footage on their servers. If clients, patients or staff appear in it, uploading is a deliberate choice. How I handle that is in AI and Privacy: What the AI Gets to See.
Step 2: cut stumbles, pauses and filler words
This is where the time savings are, and this is where the limit is.
Automatic, and good at it. The underlying idea is simple: find the places where the level drops below a threshold for a minimum duration. In ffmpeg the silencedetect filter does this, by default with a noise threshold of -60 dB and a minimum duration of two seconds. For speech you set the threshold higher, meaning less negative, and the duration shorter, for example silencedetect=n=-30dB:d=0.5. Out comes a list with the start, end and length of every silence, and that list is the template for the cut. One caveat: the related silenceremove filter works on the audio track only. For video you take the list and cut picture and sound together.
The programs call the same thing different names. Premiere Pro filters the transcript for "Pauses" and deletes them one by one or all at once, either as "Extract", which closes the gap, or as "Lift", which leaves it; the shortest pause you can filter for is a tenth of a second. Descript has two separate functions: "Remove filler words" for the ums and uhs, which the maker states detects filler words in English transcripts only, and "Shorten word gaps", which trims every gap over a chosen length down to a target, 200 milliseconds for instance.
And this is exactly where it goes too far. "Shorten every gap over 400 milliseconds to 200" is a sentence you execute in two seconds and that produces a video nobody watches to the end. Because one pause is not the same as another. There is the searching pause, where somebody is hunting for the word: that one can go. There is the breath at the end of a sentence: that one carries the rhythm. And there is the setup pause before the important sentence, which exists precisely because the point follows it. Pull them all to the same value and you get an even audio track with no emphasis at all, and the effect is paradoxical: the video does not feel shorter, it feels harder to sit through.
The same goes for filler words. An "uh" before a number is often the moment somebody is thinking; cut it out and the number sounds rehearsed. Descript separates two similarly named functions: in its filler-word list, "Ignore" removes the audio while leaving the word struck through in the script; the general inline "Ignore" strikes text through without deleting the media. Check which function you are using before applying it across the transcript.
The working rule: the automation finds, the human decides. Let the tool flag every candidate and go through them in one pass instead of deleting them wholesale. And check the cuts on the picture, not only on the sound: a clean audio cut where the head jumps in frame is worse than the pause you just removed.
Step 3: lay down the rough cut
The rough cut sets the structure, not the polish. It answers three questions: what does it open with, in what order does the rest come, where does it end.
Automatic: detecting scene changes and suggesting clips. The reel automators go further and pull the passages that make good short clips out of a long video by themselves. That is the one place in the whole workflow where an AI genuinely makes editorial cuts.
Your decision: everything to do with intent. The machine can tell where a thought ends, not whether it matters. The strongest sentence in an interview is often buried in a subordinate clause. In practice: you mark the passages that stay in the transcript and build the order from there. Transitions and overlays come afterwards. Polishing before the structure exists means polishing footage that gets cut later.
Step 4: subtitles
Subtitles are their own step here, not a by-product. The transcript from step one is the basis, the rest is correction: every automation mishears proper nouns and technical terms, writes numbers differently from how a subtitle should write them, and breaks lines wherever there happens to be room. Google puts it in its own help page: always review automatic captions and correct them. How to generate them, which file format is for what, and when to burn in rather than attach, is laid out step by step in Auto-Generate Subtitles; which tool can do what for vertical video is in Subtitle Tools for Reels Compared. The rule of thumb here: subtitles come after the rough cut, never before, otherwise you correct text for passages you are about to delete.
Step 5: level the audio
The step almost everyone skips, and the only one with a documented target value.
Automatic: measuring and adjusting loudness. The broadcast reference EBU R 128 uses -23.0 LUFS for programme loudness and -1 dBTP as the production true-peak limit. ffmpeg's loudnorm follows the measurement logic but its defaults are different: -24.0 LUFS integrated loudness, a loudness range of 7.0 and -2.0 dBTP. If you want the broadcast target, set it explicitly, for example loudnorm=I=-23:TP=-1. If you hand the filter the measured source values, it can normalise linearly instead of intervening dynamically.
Your decision: the target and the balance. Social platforms do not share one universal EBU target, so choose a consistent, clipping-free value for your channel and document it. Then decide how loud the music sits under the speech, what happens to the one loud laugh, and whether two recordings from two rooms sound like the same video.
Step 6: export, and the check afterwards
Export is not a button, it is a decision about how many versions exist.
For YouTube, Google recommends MP4 with the moov atom at the start of the file, H.264 in the High profile, AAC-LC audio at 48 kHz, and for SDR material a bitrate of around 8 Mbps at 1080p and 35 to 45 Mbps at 4K, or 12 and 53 to 68 Mbps at high frame rates. You keep the frame rate you shot at instead of converting. YouTube Shorts are vertical, up to three minutes long and uploaded at a maximum of 1080p. For Instagram Reels, Meta lists 9:16, MP4 or MOV, up to 15 minutes and a maximum of 4 GB, plus one figure that matters more for the edit than any bitrate: at least 14 percent at the top, 35 percent at the bottom and 6 percent on each side should stay free of text, logos and design elements, otherwise the interface covers them. So your subtitles do not belong in the lower middle of the frame, even though every tool puts them there by default.
And then you check. The most common export fault is not a sound or picture problem but a black frame at a cut point that you never notice playing it back inside the program. ffmpeg finds that with blackdetect, which by default only triggers at two seconds of black and at a ratio of 98 percent black pixels; for a reel you lower the minimum duration considerably. Add to that: watch the first and last second, check the audio track is not silent, hold the file size against your expectation. Four minutes that save you a complaint.
When this workflow happens every week
Six steps for one video is doable. Six steps for three videos a week is a job. I did not add another piece of software for that, I gave the task to a role: Eddi is my AI employee for video editing. He gets raw footage and a script and hands back a finished video: pauses out, subtitles on, overlays placed, 9:16 format, black-frame check passed.
He uses the same tools as above, ffmpeg, a locally running Whisper model and Remotion, all on the command line, no editing suite and no mouse. The difference is that the checks are built in rather than optional. For every raw file he records resolution, duration and a checksum before he touches it, and he never overwrites raw footage. After every cut he compares two transcripts against each other to see whether a word got lost in the trimming. After every encode a black-frame scan runs, before every full render he looks at test stills. That routine does not come from a manual, it comes from a complaint of mine: after a delivery with black frames, the scan has been hard-wired ever since.
What he does not do is equally clear. He does not shoot, without raw footage there is no video, and publishing stays my gate. How a role like that comes about is in Hiring an AI Employee: The Process, how the work is divided up at my end in My AI Workforce. And if you would rather hand the editing out of the house entirely: what a service provider needs from you and how it gets billed is in Hiring a Video Editor: Process, Pricing.
Frequently asked questions
Can AI edit a video entirely on its own?
Partly. Reel automators can select passages from a long video, and newer tools can suggest or execute order, length and style. In this workflow, transcript, silence detection, subtitles and loudness normalisation are automated groundwork, while what remains in the final cut stays subject to human approval. The cut is a statement about what matters, not just an arithmetic problem but the craft you learn before any program.
How do I remove pauses automatically?
Through silence detection. In ffmpeg, silencedetect finds every stretch below a level threshold, for speech roughly n=-30dB:d=0.5; Premiere Pro filters the transcript for pauses and deletes them singly or in bulk, and Descript trims every gap over a chosen length to a target value. Work through the hits one by one instead of deleting them wholesale, or you lose the breaths along with them.
How short may a pause be in the finished video?
There is no standard figure for that, there is a distinction. Searching pauses, where somebody is hunting for a word, can go almost entirely. Breaths at the end of a sentence and setup pauses before an important point stay, otherwise the video sounds rushed. That is why a blanket target length for every gap is the step that turns saved time into an unwatchable video.
Do I need a script for AI video editing to work?
Not strictly, but it helps a great deal. A script, or at least a list of bullet points, gives you a text to check the automatically generated transcript against. That comparison catches exactly the faults that otherwise surface in the finished video: misheard proper nouns in the subtitles, and sentences that lost a word during trimming.
Which format do I use for social media?
Vertical 9:16 for reels, TikTok and Shorts, landscape for YouTube. Instagram takes MP4 or MOV for reels up to 15 minutes and 4 GB, YouTube Shorts are vertical and up to three minutes at a maximum of 1080p. Plan the margins in: Meta recommends leaving 14 percent at the top, 35 percent at the bottom and 6 percent on each side free of text and logos.
Where to go from here
Take your next video and change one thing deliberately, step two: let the silence be found automatically, but decide for every single pause yourself whether it may go, and note how many you kept. That number is your own lower limit, and from the next video on you know how far you can let the automation run.
If what comes out of it is that your bottleneck is not the tools but the repetition of the same six steps, the next step is a fixed role rather than another subscription. The ready-made templates for that are in my community.
