Object removal

On this page 7

Object removal is the task of erasing a selected person, item or blemish from an image or video and filling the vacated region with plausible background, so that the result looks as if the object was never there. Among AI applications in artificial intelligence, it is one of the most widely used consumer features, built into phone galleries, photo editors and video tools.

Modern object removal combines two steps: segmentation, which finds the exact pixels that belong to the object, and inpainting, which synthesizes new pixels for the hole. The quality of the result depends on how well the model understands scene structure, lighting and texture, not merely on how well it copies nearby pixels.

How AI object removal works

AI object removal works as a pipeline of mask creation and generative fill. The user marks the object with a brush, a tap or a text prompt, a segmentation model refines that hint into a pixel-accurate mask, and an inpainting model generates content for the masked area that agrees with the surrounding scene.

The mask is usually dilated by a few pixels so that shadows, reflections and soft edges are removed together with the object. Leaving a thin halo of the original object is the most common cause of visible artifacts, which is why good tools also detect cast shadows and reflections as part of the same object.

The following table summarizes the stages of a typical pipeline and what each one contributes.

StageTypical techniqueWhat it contributes
Object selectionTap, brush stroke, box or text promptTells the system what to remove
SegmentationPromptable segmentation modelPrecise mask including fine edges
Mask expansionDilation and shadow detectionRemoves halos, shadows and reflections
InpaintingDiffusion or transformer-based generatorSynthesizes the missing background
BlendingColor matching and feathered compositingHides the seam between old and new pixels
Stages of an object removal pipeline

Inpainting methods behind object removal

The core of object removal is the inpainting method, and three generations of techniques are still in use. Each one makes a different trade-off between speed, realism and faithfulness to the original photograph.

Patch-based synthesis copies texture from elsewhere in the same image and works well for grass, sand or brick, but it cannot invent structure it has never seen. Feed-forward neural inpainting, typically a convolutional network trained on masked images, runs in a fraction of a second and handles large flat areas reliably. Diffusion-based inpainting generates content step by step from noise and produces the most convincing results for complex scenes, at a higher computational cost.

ApproachStrengthWeaknessTypical speed
Patch-based synthesisFaithful local textureFails on missing structureMilliseconds to seconds
Feed-forward networkFast and stable on large masksBlurry detail on complex scenesUnder one second
Diffusion modelRealistic structure and lightingMay invent unwanted detailsSeconds, often on a server
Comparison of inpainting approaches

Diffusion models introduce a specific risk: because they are generative, they sometimes add objects that were not requested, such as an extra person, a sign or a strange texture. Many products therefore constrain the generator with a neutral background prompt or offer several variants for the user to choose.

Object removal in images and video

Object removal in video is considerably harder than in still images because the fill must stay consistent across frames. A fill that looks perfect in one frame but changes slightly in the next produces visible flicker or shimmering, which viewers notice immediately.

Video systems therefore track the object through the clip, propagate the mask, and borrow real background pixels from other frames where the object has moved away. Only the remaining unseen regions are generated, which keeps the result anchored to real footage rather than invented content.

AspectStill imagesVideo
Main difficultyPlausible structure in one frameTemporal consistency across frames
Source of fillGenerated or copied from the same imageMostly borrowed from neighboring frames
Mask handlingSingle maskTracked mask for every frame
Typical artifactSmudged texture or wrong geometryFlicker, ghosting and edge wobble
Still image and video object removal compared

Choosing the right object removal approach

The right object removal approach depends on object size, background complexity and the intended use of the image. A small dust spot on a plain sky needs nothing more than a healing brush, while removing a car from a street scene requires a generative model that understands perspective.

SituationRecommended approachReason
Small blemish on uniform texturePatch-based healingFast and exact, no invented content
Tourist in front of a landmarkDiffusion inpainting with variantsNeeds plausible architecture behind the person
Power line across a skyLine-aware mask plus fast inpaintingThin structures need precise masks
Moving object in a video clipVideo inpainting with mask trackingAvoids flicker between frames
Product photo for a catalogGenerative fill plus manual retouchCommercial images require inspection
Decision guide for object removal

For professional work, the decisive recommendation is to keep the original file and work non-destructively, on a separate layer, so that every fill can be reviewed, regenerated or reverted. Large masks covering more than about a third of the frame usually give unreliable results with any method.

Limitations, risks and ethical questions

Object removal has clear technical limits and ethical risks. The generated fill is a plausible guess, not a recovery of what was really behind the object, so it must never be treated as factual information about the scene.

  • Hallucinated content: the model may invent text, faces or objects that never existed.
  • Geometry errors: straight lines, tiled floors and repeating patterns often bend or break at the seam.
  • Lighting mismatch: the fill may ignore shadows or color casts from nearby light sources.
  • Evidence manipulation: removing people or items from news, legal or insurance images can mislead and is often prohibited.
  • Disclosure duties: editorial standards and some platform rules require labeling substantially altered images, and content provenance standards such as C2PA record edits in metadata.

A common misconception is that AI can reveal what was hidden behind an object. It cannot: it produces statistically likely pixels. For journalism, forensics and documentation, the ethical boundary is that removal edits must be disclosed or avoided entirely.

Explained without jargon

Object removal is a digital eraser for photos and videos. You point at something you do not want, such as a stranger walking through your holiday picture, and the software makes it disappear. It does not leave an empty hole. Instead, it paints in new background that matches the rest of the picture.

The software has learned from millions of images what walls, skies and streets usually look like, so it can make a convincing guess about what should fill the gap. The key point is that it is a guess: the result looks real, but it does not show what was truly there.

A typical use case

A real estate photographer shoots a bright living room for a property listing, but the frame includes a power cord across the floor, a reflection of a tripod in a glass cabinet and a parked delivery van visible through the window. Reshooting would require returning to the property, which is not practical.

In the editor, the photographer taps each object, checks that the masks include the cord’s shadow and the reflection’s edges, and runs a diffusion-based fill on a separate layer. The cord and tripod reflection disappear cleanly in seconds. The van needs three attempts before the generated street looks consistent with the rest of the window view, and one variant that invented a nonexistent tree is discarded.

The finished image is delivered in about ten minutes instead of a second visit. Because the edits remove only distracting temporary items and do not change the property itself, the photograph remains an honest representation of the home, which listing rules typically require.