There are many possible techniques to use for cleaning up a shot, ranging in difficulty from extremely easy, to mind-numbingly complex. How hard it is depends on how complex the background behind the object being removed is, and what might be occluding the object being removed in various parts of the shot.

For example, if there is a large unfortunate piece of rig that happens to be in front of a complex and defined tree-branch blowing wildly in the wind, occluded in the foreground by a healthy wisp of smoke, cue the nightmare scenario. Basically the aim of cleanup work is to re-create the background behind the object needing to be removed, such that a person can’t tell there was ever anything there.

Here are some of the techniques used to do this.
2D or 3D tracking of still “cleanup” images into shots: this works well for background objects that are not deforming, for example, the sides of buildings, trucks, rocks, and other hard things. This technique does not work as well for soft moving things like people, clothes, energetic trees, and water. Another thing that confounds this technique is interactive lighting changes. If there is a flickering light on the side of a building, using a still image to clean up something on the wall of said building will look out of place, unless a keyframed color correction is applied to match the lighting changes.

Cloning one area of an image to another area, in order to cover something up: This works well for shots where the background of the object needing to be removed has a moving texture. For example, for something like ripples in water, still image “patching” will not work because the ripples in the water have to move. Since the texture of the water is ideally relatively consistent in its pattern of ripples, cloning from one area of the frame to the other might not be noticeable. However, if the background’s pattern is non-repeating or complex, this technique might easily be foiled.

Clone Painting: This technique is varied and quite effective with the right tool in skilled hands. It is similar to wielding the “rubber stamp” tool in Photoshop, except that it must be kept in mind at all times that one is working with a moving shot. One can clone areas from adjacent frames to replace the background over a moving wire. In order to do this effectively, the plate has to be motion tracked and stabilized to the object being manipulated, so that the object being removed doesn’t change position from frame to frame. Clone painting from the same frame using an offset to remove something on a moving object also can work well. When cloning with an offset on consecutive frames, one has to be very careful in order to avoid motion artifacts that result from the cloning happening slightly differently on each frame. A first inclination might be to just clone out an object on each frame and be done with, but when you watch it back in motion, horrible boiling artifacts will occur over the object that is removed so perfectly when looking at each frame individually. Generally, offset cloning is easier to get away with on edges, and objects in motion, and harder to get away with on static objects that have subtle gradients.

Here are a couple of video tutorials on how to accomplish some of the things discussed, using The Foundry’s Nuke 6.0.

Watch in HD: 2D Tracking and Cloning from Jed Smith on Vimeo.

A simple tutorial in Nuke on how to clone from one area of a moving image to another, using 2D tracking and stabilization, and basic compositing. Uses a shot from The Hotdog Cycle, produced by The Last Quest in Seattle.
The Hotdog Cycle Trailer

Watch in HD: Temporal Clone Painting, Patching, and Grain Manipulation from Jed Smith on Vimeo.

A demonstration of a method of cleanup using clone-painting from adjacent frames on a stabilized plate, in NukeX 6.0.
The shot used is from the animation “High Strung”, produced by Tommy Thompson at The Evergreen State College.
Tommy Thompson’s Production Blog
A Short Documentary About The Project.

This post then, we will consider a step in the right direction, but not by any means achievement of this goal. Namely, I mean that this 2nd Rotoscoping Video Tutorial that follows is exceedingly rough, rambly, random, unrehearsed, raw, borderline-reprehensible, and reeking of underflowed thought-speech-buffer. If you have 30 spare minutes of your time, however, you can get a 1st person experience of not only one of the many things that I have been up to of late, but some information about what rotoscoping is, and how a novice student performs one of the things essential to feature film visual effects.

This is a tutorial primarily centered around the rotoscoping features of the software called Silhouette Roto. A couple of notes: For this screen recording I am using iShowU to capture my screen and my system audio, a decent microphone to record myself, and a software called Mouseposé to show you what keys I am pressing, and what mouse buttons I am clicking. When I press a key, or combination of keys, they will pop up in overlay at the bottom of the screen. When I press the left mouse button, the cursor will be outlined in Blue. When I press the middle mouse button, the cursor will be outlined in Yellow. Similarly, the right button is Red. This should let you know what I’m doing without the need to explain everything.

[Edit -- several months later]: You might also note that the technique of rotoscoping that I am employing in this is somewhat of a “straight-ahead” method, using animation terminology. That is, I am refining the roto shape one frame at a time in one direction. A much better method is to use a “nonlinear” approach, in which one places initial keyframes on key points (extremes) of motion, and then refines the shapes iteratively. This is a better technique because it results in less noticeable motion artifacts such as jittering of points and other inconsistencies of motion.

Download MP4 Video, 1152×720 (148MB)

Here is a full resolution frame-capture of a video stream shot with the HV20 (the image is a link to the full resolution image). All of the images posted here are compressed with jpeg 80% quality, and should have most of their original attributes preserved. Note that the pulldown has been removed with After Effects, and that this was originally a frame comprised of two interlaced fields. Note that this picture is HUGE (1920×1080), so if you have a regular sized monitor, expect to scroll around to look at it completely.

If you look closely at the image, you can discern fragments of interlacing artifacts left behind from the pulldown removal. This happens because the mpeg2 compression of the interlaced image results in fields that are imperfect with macroblocks and other imperfections. This confuses After Effects’ algorithms, leaving behind chunks of image that still have interlaced areas. Effectively this just looks like bits of the image have weird blocks of ‘liney’ areas. Below is a 4:1 crop of the above original frame, doubled in size.

You can notice these artifacts on the edge of the finger. In After Effects, there are plugins to process a frame to remove interlacing artifacts. With one of those applied, the image looks a little cleaner.

There are still noticeable artifacts from the mpeg2 compression. You can make out macroblocking and chroma abnormalities on the thumb and surrounding areas. If you were to look at the individual color channels, this would be more noticeable. Usually the compression artifacts are worse in the blue channel. There is a plugin that comes with the Magic Bullet Suite called the “Deartifactor” that takes a whack at removing compression artifacts such at as these.

Deartifacted:

Non-Deartifacted – Blue Channel Only:

Deartifacted – Blue Channel Only:

Sometimes it is hard to see what the deartifactor is doing in areas of detail. Here is another comparison in another less detailed area of the image.

Non-Deartifacted – Blue Channel Only:

Deartifacted – Blue Channel Only:

There is also a filter in After Effects called Remove Grain that was once sold as Grain Surgery by Visual Infinity, but is now included for free with After Effects. This plugin is quite amazingly good at removing grain while preserving detail in the image. While the compression artifacting is not quite grain, it does do a fair amount to improve the apparent image clarity, and while it does soften the image some, it can be adjusted for good results.

Remove Grain filter applied:

Here is the final processed image, in full resolution.

Most likely the effective qualitative differences in the image when viewed full-resolution are relatively minor. However, when you get into things like doing dramatic color correction or other processing, or attempting to extract a matte from an image using color keying, it is absolutely essential to have the best quality image that you can to work with. Of course it is not really a great idea to try to use HDV for something requiring excellent chroma key matte extraction, but for 800 dollars, this camera produces some amazing images, and you would be hard pressed to find a better solution for less than several thousand. There is also the possibility of capturing 4:2:2 uncompressed video before the MPEG2 compression stage from the HDMI output of this camera, if you have a Black Magic Intensity card, or an equivalent HDMI capture device, which could theoretically be plugged into a laptop. There has been some experimentation in this area, but little noticeable benefit is likely to be achieved.

As free alternatives for the post-processing of HDV, there is a plethora of possibility with AviSynth and various other freeware windows applications. There is quite a bit of activity around the area of using AviSynth for post-processing of HV20 footage.
The Farnsworth plus Sillyman Process
Automated 24p pulldown workflow using DGIndex and VirtualDubMod
Morgan MJPEG2000 as an alternative HD Intermediate Codec
AVISynth and VDub Templates / Code

To end this post, here are some other full-frame image-captures from the HV20, with no processing applied to them other than pulldown removal and deinterlacing.

An indoor image recorded in low light, showing the characteristic of the camera in high-gain mode, which can be avoided if operated properly.

An outdoor image from the camera on a relatively bright day.

Interface
xplorer2 – The single best file manager I have ever used, and I have used alot. I wish there were anything that remotely approached an equivalent on the osX side of things. (lite version = freeware, more advanced version is shareware)
Launchy / Colibri – Keyboard launching utilities are the single greatest enhancement in interface technology since … (freeware)
TaskSwitchXP – A superior replacement for the Alt-Tab task switcher built into windows
Stickies – Sticky note program, freeware
EssentialPIM – Calendar / ToDo / Personal Organization utility, freeware

MultiMedia
Foobar2000 – Audio player, encoder, ripper, tagger, cataloger, with many more advanced features (freeware)
VLC – Monolithic multimedia player with Matroska support (freeware)
Mediaplayer Classic – DirectShow multimedia player. Best when combined with the Combined Community Codec Pack (freeware)
XnView – Photo viewer / editor (freeware)
wScite / Notepad++ – Text editors for programming and just plain text editing (freeware)

Network
Firefox / Opera – Web browsers (freeware)
Pidgin (formerly Gaim) / Miranda – Instant messaging multi-protocol clients (freeware)
X-Chat 2 – IRC client (freeware)
uTorrent – Torrent client (freeware)
StrongDC++ – Direct Connect client (freeware)
eMule – eDonkey client (freeware)

Security/System
Sygate Personal Firewall Pro – Firewall
NOD32 Antivirus – Antivirus software

As a resource for anyone that might be interested, here is a small section about how to teach yourself how to use After Effects.

How to get help and teach yourself
The most effective way to learn After Effects is to experiment with it and learn on your own. In accordance with this goal, here are some useful resources.

• After Effects Help File
• Everything you need to know to operate the program is contained in this well-written, intuitively organized, and easily searchable help file. Just press F1 while running After Effects, or click Help / After Effects Help.
• Online Tutorials
• http://www.creativecow.net/
  CreativeCow.net has a great deal of superb and useful tutorials. This link is to a list of all of their written After Effects Tutorials.
• http://www.creativecow.net/aepodcast/
  CreativeCow’s new series of excellent video podcast After Effects tutorials.
• http://wikivid.com/index.php/After_Effects
  a compilation of links to introductory After Effects video tutorials.
• http://nututorials.com/subtutorials/AfterEffects/23001/1/10/0
  categorized After Effects tutorials.
  
• http://www.stephenschleicher.com/
• http://aftereffects.digitalmedianet.com/
• http://www.editorsguild.com/
  an introductory tour of after effects, fortified with example projects
• http://www.toolfarm.com/
• http://www.xtreme-is-back.com/
• http://www.pixel2life.com/
• http://www.ayatoweb.com/
• http://www.videocopilot.net/
  a very energetic after effects instructional video creator…
• http://msp.sfsu.edu/Instructors/rey/aepage/aeportal.html
  a portal to all things AE…
• http://media-motion.tv/
  an excellent and well-trafficked After Effects list-serve.

• Total Training/Lynda.com Training Videos
• These are two great software training video resources, containing ridiculously thorough video training in all things After Effects, conducted by articulate professionals. Talk to the computer center about their subscription to Lynda.com available for the use of interested students.

• Books
• There are many books out there about After Effects, but one of the best is Creating Motion Graphics with After Effects – Volume 1: The Essentials by Trish and Chris Meyer. Instead of just telling you how to perform functions in After Effects, it attempts to teach you techniques for being creative inside the program, teaching you in the context of real-world design problems.

Attached to this post is the After Effects tutorial handout that I wrote for the last workshop I did with Ruth Hayes for SOS: Media, which might be of use to anyone looking for a basic text-mediated introductory reference for After Effects.

Suppress Control

All Cocks Plaster

These videos were created in Adobe After Effects 7.0 utilizing the Expressions, which are just a way to control keyframed parameters in After Effects with a script in Javascript. You can create little ‘expressions’ by using a collection of custom ‘variables’ based on elements of the after effects environment, such as timecode, layer values, etc. Often expressions are used as a way of linking values of parameters together, to make something animate in defined relationship with something else. However there are a myriad of creative possibilities for using expressions in After Effects. Here is what I did to bring about “Suppress Control”.

timeToFrames(t = time, fps = 23.976);
t = random(t,thisComp.duration);

This expression is on the Time Remap property of my main video layer. Time remapping is just a parameter that lets you alter the playback rate and direction of a layer dynamically with keyframes or expressions. All this script is saying is

1). convert the timecode of the current position into its frame-number, based on the given frames per second, and store this value in the variable ‘t’.
2). let ‘t’ then equal a random number between the current value of ‘t’, and the length of the current layer.
This effectively makes each frame in the layer a random frame between the beginning and the end. This has a very chaotic direct-animation type aesthetic to it, because it is a frame-based manipulation, and has no slow changes. In “Suppress Control” I did this for 3 different stacked layers, and then created another expression on the top two that turned the transparency on or off every x frames. Each of these 3 layers was then combined with the layer below it using blending modes. This created an additional chaotic abstraction to the image, because each image is now not only a random frame of the entire video, and not only one of 3 random layers, but each frame is an aleatoric combination of each of the 3 layers (or just 1 or 2, depending on if all the layers are turned on or not), blended into each other to create a strange abstract image. Here is the expression that blinks a layer on and off every X frames.
fr = 23.976;
FrameNum = 2;
Scale = 50
trig = Math.cos(time*Math.PI*fr*(1/FrameNum));
if ( trig > trig/2 ) { trig = Math.ceil(trig) } else { trig = Math.floor(trig) };
trig = Math.round(trig);
trig = trig*Scale + Scale;

1). What is the framerate? (or, how many times per second should this happen)
2). How many frames should each alternation last?
3). How large should the variation in value be?
4). A trigonometry function to alternate between +1 and -1 every x amount of time defined by the framerate and FrameNum.
5). If the result of this function is greater than half of it, round up; else, round down.
6). Scales the value to +Scale, -Scale, and then shifts it up to be between 0 and 2*ScaleI

If applied to a transparency parameter, it effectively makes a layer flash on or off every x amount of frames.

Another cool expression, which is used on the train layer in All Cocks Plaster is as follows:

layTime = time – (inPoint-framesToTime(112));
random(layTime, layTime + framesToTime(10));

This is a very simple expression also applied to the time-remapping attribute of a layer. All it does is, for each frame, choose a new frame between the current and 10 frames ahead of it. So the motion continues on its path, but is randomly jumpy, because it is jumping around randomly in a little box of 10 frames that is moving forwards.

As you can see, the chroma screens are underexposed, as are the subjects, and as a result the image is somewhat noisy, which doesn’t help matters with the already artifacted DV source. Also, there are lighting inconsistencies in the chroma screen. Also, like an idiot, I decided it would be a good idea to backlight the subjects with a blue-gel; I thought this would make my planned key more believable. I thought that because the blue light was from a source other than spill from the bluescreen, it would be okay. It’s not okay. Blue is blue! That must be kept in mind in the future.

Anyway, using the magic of the Keylight plugin for After Effects (read this awesome tutorial, anyone who might be interested in seeing how it can function), and some creative masking, I was able to pull a decent key.

Pixels should not be this large or this blocky!

This is slightly better… I achieved this by tweaking the settings of keylight and adding a subtle matte-choker.

This is the key without the mask on the circuit board. As you can see, the blue spilled onto the board from my backlight (arr!) is causing the keying plugin to make the board transparent, something that we don’t want to happen.

Since the board is stationary within the shot for a large period of time, I duplicated the layer and applied a mask to the board, effectively making the key only affect areas outside of the circuit board area. The results look alright, but I will be tweaking it more, possibly using the technique of blurring an adjustment layer and applying a transfer mode of “color”, in order to blur the chroma slightly without blurring the luminance, and thus counteract the DV artifacts somewhat, as outlined in the keylight tutorial.

Today I enthusiastically enjoyed the DVX100A proficiency with Ben, Alex, Dre, and the almighty Dave Cramton. Tomorrow, I will pick up my work order for it, and break open that pack of 5 new Panasonic Master Quality DV tapes, leaving the solitary Sony DVCAM tape that I bought solitary… and unused. (take that, DVCAM!). I will be doing tests on Friday to see if the 24p advanced mode can deliver the realistic progressive psuedo-filmlike motion and blur that I’m looking for, as well as doing a test pulldown, edit, reverse pulldown, and author to progressive-scan DVD. I will of course post the results here.

In other news, I have been drooling all over my monitor, reading about the Panasonic AG-HVX200, the HD successer to the DVX100, which records in 4:2:2 DVCPRO on P2 cards at a data rate of 100MB/s, and is capable of recording real variable framerates, much like the Varicam, except about $60,000 cheaper at $6,000. I’m excited by the fact that, in 6 or 7 years, this technology will be somewhat outdated, and I will probably be able to buy one for under 2 thousand dollars.