One of the most important parts of creating realistic lighting for 3D animation is compositing. Compositing, which is the last phase of the visual effects pipeline, brings together all the multiple visual elements from different sources into a unified image. This phase could be a complex combination of footage, images, and other elements or simply two images laid over one another.

Shooting on a blue or green screen is an example of compositing. The green screen footage lets you place a new environment in the background by keying out the green and then compositing with a background or matte painting.  

Making sequences as close to the real thing as possible is a process and compositing is key to creating realistic 3D environments.  

Compositing Techniques for VFX

Two of the most popular compositing techniques are rotoscoping and matte paintings:


Rotoscoping started out as an animation technique that involves tracking over film footage to create life-like character movements. But Walt Disney developed an offshoot technique that utilizes footage of real actors as reference materials for Disney’s many animated movies. Disney’s hybrid rotoscoping technique was first seen on Pinocchio. This technique is also used to remove wire rigs, which is another technique developed by Disney.

Rotoscoping for VFX is different from traditional rotoscoping. The process creates a matte or mask for an element so that the element could be extracted out and placed on a different background, mask out colors, and so on. Essentially, rotoscoping allows an artist to remove safety rigs, wires, and any unwanted objects or elements in a sequence.

Using a set of tools within a compositing software, a roto artist would create a new alpha channel for a certain part of an image. Because footage taken by a camera has no alpha data, a roto artist has to create the alpha channel manually by tracking over the elements in a sequence. The artist would create different shapes around the element and animate these shapes to match the movement frame-by-frame.

Rotoscoping could take a few hours to several days to complete. Just to avoid a roto nightmare, it’s important to compose and shoot sequences carefully.

Photo Credit:

Matte Paintings and Digital Mattes

Matte painting is a technique that’s used to create a representation of a scene that’s otherwise impossible to deliver in real life. This technique is designed to trick the eye into seeing a photo-realistic landscape. It is one of the oldest VFX techniques in the business and was first utilized by photographers before being applied in movies.

Back in the day (the late 1800s) painted glass panes were used to create physical mattes to create VFX all in-camera. Using glass panels to create matte paintings will require the camera to stay still.

Frank Williams developed and patented a new process that places actors in black backgrounds to create high contrast negatives. The high contrast negatives are turned into white silhouettes of the actors. Dubbed as the Williams Process, this technique was the basis for the modern-day blue and green screen compositing.

These days, however, different computer programs are used to create highly detailed 3D worlds and landscapes. Photoshop is one of the most widely used software programs for matte painting. This technique makes an excellent base plate for creating a photo-real image.

Compositing Tips

Study the Sequence

It’s important to check the kind of footage that you are working with before applying any type of 3D effects to enhance the shot. It’s never a good idea to apply special effects without thinking things through; you have to strategize carefully to achieve the most realistic results.

So review the sequence, study the camera movements, check the lighting then figure out what elements in the video to retain, what to remove, how to enhance the lighting, etc. You want the effects to be seamless and convincing. Figure out the process carefully especially if the footage is not easy to work with.

Save Custom Presets

Custom presets save a lot of time because you can implement the same effects instead of working from scratch. Most applications let you save out custom presets, which give you an excellent starting point to build from. You could still modify the presets as you implement them into different elements of your composite and ultimately save a lot of time.

Save Your Work

Can you imagine losing a file that you’ve been working on for hours, days, even weeks when your computer or application crashes? We can’t that’s why we’ve made it a point to enable the auto-save feature each and every time. Make a habit out of saving your work as often as possible. Avoid the heartbreak of redoing a task you’ve been working for hours because of a glitch by enabling the autosave function.

Label the Layers

Work can get confusing if you are working on different layers within a composite. Things become even more hectic if you are juggling different projects or working with extremely complex composites; some tasks become overlooked, work becomes less efficient, etc. That’s why it’s so important to work in steps so you don’t end up forgetting things and being all over the place.

It’s a no-brainer but start your work by organizing the layers by the label. Make a habit out of naming your layers appropriately as you create them so it’s much easier to keep track of your work.

And in case the composites have to be passed off down the pipeline, it’s much easier for the next person who’ll work on the layers to apply whatever effects that the layers need because the footage has been labeled properly.

Render Each Light Separately

When it comes to integrating 3D elements into your footage, it’s hard to predict what kind of changes are needed to be made so consider rendering out the lights separately. This gives you more control over the final outcome because you can modify the lights individually while also saving time.

The same thing applies for any elements of a 3D render such as ambient occlusion or illumination pass.

Rendering and re-rendering is a time-intensive process so this tip will save you a lot of time. Rendering each light individually allows you to amend your work without affecting any of the other passes.

Banish the Sharpness

Forgetting to blur the elements when integrating CG into live-action is perhaps one of the most common mistakes that artists make. Perfect sharpness makes the CG element stand out like a sore thumb and viewers could tell that the special effect is not part of the actual footage. Imperfections are a part of real-life and if reality is what you’re after then imperfections are okay. Besides, the imperfections help the CG blend into the footage so the effects look seamless or even more life-like.   


If you are working on a big project, one that involves many people from different departments, then it’s important to collaborate with your fellow artists to achieve the best possible results. Giving and taking feedback, being a team player and establishing strong communication within the team will improve everyone’s work. If you are working on your own then a strong communication between you and your client is paramount to achieving the desired results.

Get your free guide: Three Expert Insider Secrets for a Successful Demo Reel

You have Successfully Subscribed!

Pin It on Pinterest

Share This