I know I didn't cut out the hair as cleanly as I could have, but I get the feeling there's a bigger issue I'm overlooking. The image feels very "cut out" (read: fake) to me. Pretty sure it's something to do with the lighting or sharpness? Something I should be doing with the edges? I know the banding of the red is one contributor, though it feels cut out in other areas too.
Getting the lighting and other issues (color temp, perspective, etc) right is crucial. However, your background is so dark and nondescript, you don't have to worry much about lighting differences. Feathering the edges of the higher layer can help the object look more natural, but that's not a major problem here, IMO.
The big problem with the image is the flat lighting allowing the hair to disappear into the background. A hair light would provide some depth to the subject.
Another major problem with the image is that the hairline on the forehead doesn't look at all natural and realistic. While the hair along the llama's right temple is fairly sharp, the hairline over the forehead is indistinct and possibly blurred.
allison mindy wrote: I don't know much about photoshop, but I feel like one of the problems may be that the lighting of your subject does not match the lighting of the background you composited him into.
In CS5 or CS 6 the content aware, refine edge works well. It is easier if there is contrast between your subject and the background. The greater the difference in color between your subject and background the easier the process , which is why when shoot for composites shooting on gray or white tend to work best.
However, the smart refine feature works well and I have been able extract subjects with backgrounds that didn't have much difference. I have tried the refine edge twice, and have received decent results using that process.
The one thing that stands out, is the subjects forearms and hands, right where they make contact with the table ... there should be some shadow there, you have a touch of shadow where the forearms contact the end cards and that's about all.
Try burning down those edges with 5 or 10% opacity.
It would help if you also showed the original, before any masking so we can see what the hair looks like around the edges.
EDIT: One reason it might look cut-out is because the background looks kind of cheesy and fake. Drop in something real like a theater curtain, and vary background the lighting as I've done below.
The edges of the shirt are easy enough to mask cleanly. It's the hair that often gets botched. To make that look more real, use the smudge tool (very tiny) to fuzz up the edges of the hair a little. Work on a blank layer.
Here I made a deliberately fake, clean mask around the hair to show the fuzzing-up with the smudge tool.
The arms are the giveaway for me - they look like they're floating. If they were really resting on the table, their shape would be different at th epart where they make contact with the table. Adding some shadow under there might help.
You have an underexposed background and a completely different exposure for the foreground image. To get the subject to match, it should be much darker, and shadows would be wrapping his skin from behind him.
The biggest disconnect I have with the photo is actually the scaling:
You've clearly shot him with a wide-angle lens. A wide-angle lens exaggerates the distance between elements of the image, which makes his arms look super long. Then you've dropped in a background and middle ground that is more consistent with a telephoto lens (compressing the distance between elements of the image). That background with that foreground is an impossibility unless you were in a room with a GIANT (what appears to be) booth and a GIANT curtain. "GIANT" = way larger than you would see on a Broadway stage and that would need curtain folds with a massive distance between the peaks of its sine wave shape.
Given your focal length choice for the subject, the only way you could possibly have the background to be taken up entirely by the curtain is if he were absolutely right in front of it, in which case it would be a lot more in focus. But even then, I believe more of the booth back would appear in the frame. And even then, you'd be left asking, why is there a card dealer setup right in front of a curtain?
Many thanks for all the replies. Burning under the arms, changing lens size in Blender to match that of the camera, and a few tweaks to the hair have improved things greatly. Still need to study up on some hair tutorials though. Hmm... Either that or I could work with bald models!