Virtual studios are widely used for television programs or movies. However, in the traditional virtual studio, a director is necessary to control the switcher when using typical program director systems and these systems are mostly based on the bluescreen technology. In this study, we developed a program director system that can be used in natural scenes and can automatically remove the background. It can also carry out instructions and interact with virtual objects through the user’s gestures directly.
In the proposed system, Kinect sensor is used to assist the professional studio camera. Therefore, we need to align the depth information of Kinect sensor with another high-resolution color camera. Our work contains three parts. The first part is camera calibration and registration. The second part is background removal. The third one is hand gesture recognition. Zhang's calibration method was used to calibrate and register two cameras. To segment the actor foreground, we utilized the depth information, human skeleton data provided by Kinect sensor and processed the color image captured by the high-resolution camera to get a more detailed contour of the foreground. In addition, three-dimensional coordinates of hand joints of the human skeleton were recorded and preprocessed for hand gesture recognition. Eight types of dynamic hand gesture were trained using the multi-class SVM classifier.
According to our experimental results, the proposed algorithm can achieve effective background removal in natural scenes instead of in front of a green or blue screen. Moreover, the immersive interaction and occlusion effect of the human foreground and virtual objects were achieved with the depth information. Our system can provide more freedom for the actor to interact with virtual 3-D objects and trigger instructions using gestures.