Vertus Fluid Mask works great for me to get hair fine precision cuts...however the background is best if its not a complex background to start with.
If I shoot a model in the studio against a light battleship gray background...or outdoors in front of the clear blue sky or another solid color...then the extractions can come out just great...and I can usually see every single strand of hair without any artifacts on it.
Photoshop itself does only BASIC hair cutting for me...it can handle smooth, uncomplicated edged hair pretty well. Doing fine extractions in Photoshop... even with intricate techniques and much trickery and patience... often leaves me with halos, matting, clumps or haze around the fine edges. For me I have had to purchase outside products (Like Vertus and Topaz) to get good cutting end results.
My best cuts are made with knowing I will want to cut the image afterwards...so that way... I shoot the image from the beginning to optimize the possibility of getting a great extraction. There are 4 or 5 other factors that seem to enhance the likelyhood of me getting a good end product cut.
What are some of the success factors for me? #1. I use a Solid color background that Contrasts well with the Brightness and the Color of the hair. The background is best when it is a very simple smooth background without any texture or shadows. I dont use hot hair lights on images I intend to cut...it produces different color strands in the same area of the head... and the multiple luminance and color values confuses the software. The model needs to be separated from the background by 3-6 feet to reduce visual complexity with the background details and shadows...the more visual separation I have between the model and the background the easier it seems to cut for me. Models with black hair are great to cut, dark brown hair is usually great, red is good...BUT...blond is terrible to cut out cleanly because some strands are semi transparent and translucent and refract the background color through the strand of hair. This creates different colored hair strands...and strands that contain the background color in them sometimes. Light blond hair is very challenging to cut out unless it is shot well from the beginning and is optimized with the background. Flash images seem to cut better than the variableness of available light many times...the edges of the whispy hairs always have to be clear and clean to the naked eye if you want to be able to cut them well and clean. If they are soft or blurry, or poorly exposed, or are being hit by a shadow or a complex background...then cutting can become troublesome. Available light work, catching images as you feel them...often produces images that are taken against complex backgrounds with less than perfect light and colors to create the visual contrast you would like to have.
So, If I watch my process well, and If I shoot the model smartly.. and If I optimize each step of the work flow...THEN...I end up with perfectly cut out single strand pieces of whispy beautiful hair...if I miss one important factor...then I can end up having a bad hair day myself when I get back to the studio!!