KR20010084470A - An apparatus and method for extracting of camera motion in virtual studio - Google Patents

Abstract

PURPOSE: An apparatus and method for sampling a motion of a camera in a virtual studio is provided to sample a motion of a camera by adding a blue pentagon pattern having the same color tone as a background on a blue screen. CONSTITUTION: A blue screen(201) includes various forms of a pentagon pattern to sample a camera parameter. An image camera(202) is installed in a virtual studio and a takes an image of a subject of the blue screen(201) and a subject in front thereof. An image processor(203) detects only pentagon of a blue screen to recognize a corresponding pattern from a color image inputted from the image camera(202). A corresponding pentagon pattern recognizer(204) searches a pattern corresponding to a pentagon pattern detected by the image processor(203) from the blue screen. A camera calibration section(205) samples a camera parameter using a corresponding point searched by the corresponding pentagon pattern recognizer(204).

Description

An apparatus and method for extracting camera motion in a virtual studio

The present invention relates to an apparatus and method for extracting camera movement in a virtual studio used for synthesizing a three-dimensional virtual reality and a subject image in a studio.

Recently, 3D graphics and virtual reality using high-speed computers have been used for television broadcasting, so that it is possible to produce images that could not be expressed only with a camera. Among them, the chromating technique is an image synthesis technique used for news, weather forecasting, and election broadcasting. It is a technology that synthesizes in real time by replacing the blue part of an image shot in a studio with a blue screen as a background or computer graphics.

So far, chromating has synthesized a studio image and a 2D background image at a fixed view with the camera fixed. Therefore, when the camera moves, the fixed 2D background image and the synthesized studio image become unnatural. In addition, 3D graphics and virtual reality that need to change the viewpoint of the camera could not be used as a background image. As a way to improve this, the technology of virtual studio was presented.

Virtual studio is a broadcasting technology for synthesizing 3D graphics generated by following the camera's point of view and projection images of virtual reality in real time with studio images. Using this technology, actors and three-dimensional graphics are naturally synthesized in the studio, and television viewers can feel the stereoscopic feeling as if the actor is moving in a three-dimensional virtual space. Then, the position of the actor in the studio can be detected to create an effect in which the actor is covered in whole or in part by the virtual object.

1 is a conceptual diagram of a virtual studio, wherein a virtual studio image is a digital image taken by a camera and is divided into an actor part and a blue part. The blue part is replaced with a new background image. In conjunction with the camera's point of view in the studio, three-dimensional graphics or real images are projected in two dimensions in perspective and replaced with the front blue parts.

As such, in order to synthesize a natural image in a virtual studio, it is necessary to extract camera parameters representing the viewpoint of a camera in the studio and to detect an actor's position. Camera parameters refer to the position of the camera, the direction of the camera head and the size of the zoom. The camera's orientation is the angle of pan, tilt and roll, and the camera's position is the three-dimensional spatial coordinates of the camera in the virtual studio.

The parameter extraction method of the camera in the proposed virtual studio is divided into the method using a sensor and the method of pattern recognition.

The conventional method using a sensor is a method of measuring a parameter by mounting a mechanical sensor for measuring the amount of rotation or distance movement in the camera body and the lens [Reference: IMAGICA Virtual Studio Set, http: //www.imagica. co.jp/products/vset/index. htm .; Masaki Hayashi, Itsuji Nakamura and Kazuo Fukui, "Virtual Camera System using Panoramic Image," Journal of ITE, Vol. 49, no. 10, pp. 1315-1322, 1995.]

However, this method is sensitive to mechanical noise and requires the initialization of the sensor. In particular, lenses with nonlinear optical characteristics affect the measurement of the camera zoom size. That is, since there is a nonlinear relationship between the distance from the lens to the focus and the size of the zoom, precise measurement and initialization are required to compensate for this. This method also cannot be mounted on a portable camera because a mechanical device must be attached to the camera, which is a space constraint for the cameraman because a mechanical device that takes up a large space is required to extract the position of the camera.

The conventional pattern recognition method is divided into active pattern recognition and passive pattern recognition. Active pattern recognition draws a blue grid pattern with a similar color to the background on a blue screen in the studio and then shoots it with the actor. The grid pattern of the captured image changes in association with the movement of the camera and has two vanishing points. At this time, the coordinates of the two vanishing points are invariant to the position of the camera and change in conjunction with the camera direction and the size of the zoom [Ref: Mikael Tamir, "The Orad Virtual Set," International Broadcast Engineering, pp. 16-24, Mar . 1996 .; DigiWarpVirtual Studio System, http://www.for-a.co.jp/japa/products/digiwarp.htm] The greater the angle of the pan and tilt, the closer the vanishing point is to the center of the screen. However, this method cannot be used when the pan angle or tilt angle in the camera direction is close to zero and both vanishing points do not exist. In addition, this method needs to find the corresponding point of the blue screen and the virtual studio image to find the position of the camera, so the size of the grid must be varied. For example, in order to find a corresponding point, since multiple grid patterns must be included in one screen, application of the enlarged image is difficult. Finally, this method does not consider lens distortion and is difficult to apply to small cameras with severe lens distortion.

Manual pattern recognition attaches an infrared light emitting device on the camera body and moves the camera. In addition, infrared light shines differently on the blue screen as the camera moves. This change is extracted using the X-pecto Unit installed separately. However, this method requires a plurality of cameras installed in the X-fecto unit, an imaging unit for each camera, and a separate sensor for measuring the zoom size. GMD 3DK Virtual Studio, http://www.viswiz.gmd.de/DML/vst/vst.html; GMD, "Projects at DMP: An infrared based tracking system for free moving cameras in a virtual studio," http://imk.gmd.de/ docs / ww / dmp / proj1_2.mhtml, 1998.]

The present invention for solving the above-mentioned problems is to provide a device and method for including a blue pentagonal pattern having a color tone similar to the background on a general blue screen using the same to extract the movement (ie, parameters) of the camera Its purpose is.

1 is a conceptual diagram of a virtual studio

2 is a block diagram of a camera motion extraction apparatus according to the present invention

3 is a flowchart illustrating a camera motion extraction method according to the present invention.

Figure 4 is an illustration of a blue screen arranged a pentagonal pattern according to the present invention

5 is a shape diagram of a pentagonal pattern according to the present invention;

6 is a diagram for explaining a camera parameter finally output according to the present invention.

Explanation of symbols on the main parts of the drawings

101: virtual studio 102: camera parameters

103: virtual studio video 104: 3D virtual reality

105: composite video 202: image camera

203: Image processing unit 204: Corresponding pentagon pattern recognition unit

205: camera calibration unit 601: the position of the camera (X0)

602: camera position (Y0) 603: camera position (Z0)

604: camera direction (pan) 605: camera direction (tilt)

606: camera direction

Camera motion extraction apparatus of the present invention for achieving the above object, a blue screen including a pentagonal pattern of various forms used to extract the camera parameters; An image capturing means installed in a virtual studio and capturing an image of the blue screen and a subject in front of the screen; Pentagonal pattern detecting means for detecting only a pentagonal pattern of a blue screen so as to recognize a corresponding pattern in a color image inputted from the image capturing means; Corresponding point recognition means for finding a pattern corresponding to the pentagonal pattern detected by the pentagonal pattern detecting means on a blue screen; And camera parameter extraction means for extracting camera parameters (the three-dimensional spatial coordinates of the camera and the angles of the direction pan, tilt, and roll of the camera) using the corresponding points found by the corresponding point recognition means. It is characterized in that the configuration.

In addition, the camera motion extraction method of the present invention comprises the steps of photographing a blue screen and a subject image in front of the screen including a pentagonal pattern of various forms; Detecting only a pentagonal pattern of a blue screen to recognize a corresponding pattern in the captured image; Finding a pattern corresponding to the detected pentagonal pattern on a blue screen; And extracting camera parameters using the found corresponding point.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a block diagram of an apparatus for extracting a camera motion according to the present invention, which includes a blue screen 201 including various pentagonal patterns used to extract camera parameters, and a blue screen 201 installed in a virtual studio. And an image camera 202 for capturing an image of a subject in front of the screen, and an image processor 203 for detecting only a pentagonal pattern of a blue screen to recognize a corresponding pattern in a color image input from the image camera 202, A camera that extracts a camera parameter by using a corresponding pentagonal pattern recognition unit 204 for finding a pattern corresponding to the pentagonal pattern detected by the image processing unit 203 on a blue screen and a corresponding point found by the corresponding pentagonal pattern recognition unit 204. It consists of a calibration part 205.

The method of extracting camera movements in the virtual studio of the present invention using the camera motion extracting apparatus having such a configuration is performed through the process illustrated in FIG. 3.

First, the image camera 202 is a camera installed in a virtual studio and is a normal TV camera. It allows the camera operator to move freely and there is no special device for extracting the movement. The color image captured by the image camera 202 is composed of a blue screen 201 including various pentagonal patterns proposed in the present invention and a subject in front of the screen (S301). Here, the blue part except the subject is replaced by a new virtual reality by chromating later.

Subsequently, the image processor 203 detects only a pentagonal pattern of a blue screen so as to recognize a corresponding pattern in the color image (S302). The extracted pentagonal pattern consists of five vertices. The image processor 203 detects a pentagonal pattern through the following three steps in order to minimize the effects of real-time processing and lighting of the system. The first step is to use a horizontal boundary filter. At this time, in order to shorten the processing time, only the horizontal profile of a predetermined interval (for example, 16 pixel interval) in the vertical direction and the vertical profile of a predetermined interval (for example 16 pixel interval) in the horizontal direction are searched without searching all pixels of the image. . In the second step, each boundary pixel obtained by considering the directionality and the neighboring boundary pixel are connected. Finally, the vertex of the pentagon is determined by the intersection of the straight lines obtained through the pixel connection. Accordingly, elements obtained by the image processor 202 are vertices of pentagons included in the color image.

In this way, after detecting the vertices of the pentagonal patterns in the color image, the corresponding pentagonal pattern recognition unit 204 searches for a pattern corresponding to the pentagonal pattern existing in the color image on the blue screen (S303). At this time, the pentagon pattern camera of the blue screen is deformed by the movement. That is, the area of the pentagon, the length of the sides, and the angles of the pentagon are changed.

The reason why the pentagon is selected among the various patterns in the present invention is that the five inflection points constituting the pentagon have a characteristic amount that is invariant to the projection of the plane of the camera unlike other patterns. This invariant feature is used to find correspondence patterns and correspondence points in blue screen and virtual studio images. In addition, while a conventional method using a grid pattern finds a corresponding point using information of surrounding patterns, in the present invention, since only the pentagonal information is used, recognition of the corresponding point is easy even in an enlarged image.

4 shows a blue screen of the present invention to extract camera movement. As shown in FIG. 4, in the present invention, a blue screen is constructed by using seven kinds of pentagons shown in FIG. 5 among various pentagonal patterns.

The pentagonal pattern of FIG. 5 used in the blue screen 201 in the present invention has the following characteristics. First, the pentagon is the same width and width. Second, all pentagrams are the same size. These two conditions are for estimating the size of the pentagonal pattern in the image processor 203. The third angle of each pentagram is one of π / 4, π / 2, and 3π / 4. This is to prevent the pentagon from being mistaken as a rectangle or a triangle due to the distortion of the camera lens. Fourth, each pentagon is rotated by 0, π / 2, π, -π / 2 with respect to one point and reused. This is to make various pentagons. Fifth, each pentagon has one or two pairs of parallel segments. Of the five pentagons satisfying these five conditions, the pentagons shown in FIG. 5 were used in the present invention.

In the present invention, seven types of pentagons are reused by rotating 0, π / 2, π, and 3π / 2. Finally, 28 different pentagons are disposed on the blue screen 201. In FIG. 4, a blue screen including a pentagon of five rows and seven columns is illustrated.

In the present invention, a plurality of pentagons are arranged on the blue screen 201 in consideration of the fact that the actor who becomes the foreground of the object or the studio obscures the pattern as shown in FIG. 4 and the pentagonal pattern is arranged as follows. Has

That is, each column is arranged by rotating one kind of pentagon, and the last row pentagon coincides with the first row pentagon. This is based on the assumption that the uppermost pentagonal pattern is unlikely to appear at the bottom of the screen. In addition, in the present invention, the width and width of the gap match the width and width of the pentagonal pattern. This is because the greater the interval between patterns, the greater the probability that the pattern is covered by the actor, and the smaller the difficulty in pattern recognition.

On the other hand, since the rotated pentagon has the same invariant feature, it is impossible to recognize and separate the rotation using the invariant feature. Therefore, the present invention recognizes rotation using simple logic. For example, if the roll rotation of the camera is less than π / 4 and rotates the first pattern of FIG. 5 by 0, π / 2, π, 3π / 2, the direction of one segment of the rotated pattern is 0, π / 2, π, 3π / 2. As a result, the rotation of the pattern can be distinguished using the direction of one line segment of the rotated pattern. In reality, the roll rotation of the camera is extremely small in virtual studios and may not be considered.

Finally, the camera calibration unit 205 finally extracts the camera parameter using the corresponding point recognized by the corresponding pentagram pattern recognition unit 204 (S304). There are several ways to extract camera parameters from conventional counterparts [Ref .: Juyang Weng, Paul Cohen and Marc Herniou, "Camera Calibration with Distortion Models and Accuracy Evaluation," IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 14, No. 10, pp. 965-980, October 1992 .; S. Ganapathy, "Decomposition of transformation matrices for robot vision," In proceeding. IEEE International Conference Automation, Atlanta, pp. 130-139, Mar. 1984. et al.

However, the conventional methods require a lot of calculation and consideration of complex internal parameters. In addition, there is a difficulty in applying to the virtual studio since the relationship between the parameters in the continuous images is not used. In particular, when the virtual studio image and the blue screen are parallel, the focal length f and the vertical distance Zo from the blue screen to the camera cannot be measured at the same time. Therefore, in the present invention, a camera parameter is extracted from the corresponding point using a method well known in the art, and a two-step camera calibration suitable for a virtual studio is performed. That is, in the first step, the optical axis of the camera and the direction of the camera head are obtained using the corresponding point. In this case, two or more consecutive images are used to remove an error in the optical axis due to the error of the scale element and the corresponding point recognition. In the second step, only one parameter of f and Zo is selected and changed in the continuous image using the condition that the change of f and Zo do not occur at the same time in the virtual studio, and the other maintains the parameter value of the previous frame. Finally, the camera parameters are extracted.

Finally, the extracted camera parameters are as shown in FIG. 6. That is, the three-dimensional spatial coordinates 601, 602, 603 of the camera and the angles of the direction pans 604, the tilts 605, and the rolls 606 of the cameras.

As described above, the present invention includes a pentagonal pattern having a characteristic that is invariant to the projection conversion of a camera on a blue screen, processes the captured image to recognize the pentagon, and extracts camera parameters from corresponding points on the blue screen. As a result, it is possible to solve the problem of the noise of the sensor, the installation of a special device, and the problem of extracting the corresponding point, which are a problem in the conventional method, as well as extracting camera parameters easily and quickly.

The technical spirit of the present invention has been described above with reference to the accompanying drawings, but this is by way of example only and not intended to limit the present invention. In addition, it is obvious that any person skilled in the art can make various modifications and imitations without departing from the scope of the technical idea of the present invention.

Claims (8)

An apparatus for extracting camera motion (parameters) in a virtual studio by pattern recognition, A blue screen containing various types of pentagonal patterns used to extract camera parameters; An image capturing means installed in a virtual studio and capturing an image of the blue screen and a subject in front of the screen; Pentagonal pattern detecting means for detecting only a pentagonal pattern of a blue screen so as to recognize a corresponding pattern in a color image inputted from the image capturing means; Corresponding point recognition means for finding a pattern corresponding to the pentagonal pattern detected by the pentagonal pattern detecting means on a blue screen; And Camera parameter extraction means for extracting a camera parameter (the three-dimensional spatial coordinates of the camera and the angles of the pan, tilt and roll of the camera) using the corresponding points found by the corresponding point recognition means. Camera motion extraction apparatus in a virtual studio, characterized in that the. The method of claim 1, Pentagram pattern included in the blue screen, They are all the same size, Each pentagram has the same width and width, Each pentagonal cabinet is any one of π / 4, π / 2, 3π / 4, Each pentagram has one or two pairs of parallel line segments, wherein the camera motion extraction apparatus in the virtual studio. The method according to claim 1 or 2, The blue screen, Place pentagonal patterns into multiple matrices, Each column is rotated by one kind of pentagon, Camera motion extraction device in a virtual studio, characterized in that the last pentagons are configured to match the first pentagons. The method of claim 3, One type of pentagons arranged in each column of the blue screen is rotated at any one rotation angle of 0, π / 2, π, 3π / 2, and the camera motion extraction apparatus in the virtual studio. The method of claim 3, The blue screen, Camera motion extraction apparatus in a virtual studio, characterized in that the width and width of the interval formed by the pentagonal pattern to match the width and width of the pentagonal pattern. In the method of extracting camera motion (parameter) in the virtual studio by pattern recognition, Photographing a blue screen including various pentagonal patterns and a subject image in front of the screen; Detecting only a pentagonal pattern of a blue screen to recognize a corresponding pattern in the captured image; Finding a pattern corresponding to the detected pentagonal pattern on a blue screen; And In the virtual studio comprising the step of extracting the camera parameters (the three-dimensional spatial coordinates of the camera and the angle of the camera's direction pan, tilt and roll) by using the corresponding point Camera movement extraction method. The method of claim 6, Detecting only a pentagonal pattern in the captured image, Searching only a horizontal profile of a predetermined interval in the vertical direction and a vertical profile of the predetermined interval in the horizontal direction using a horizontal boundary filter; Connecting each boundary pixel obtained in consideration of directionality with a neighboring boundary pixel; And And determining a vertex of a pentagon as a vertex of a straight line obtained through pixel connection. The method of claim 6, Extracting the camera parameter, Obtaining the optical axis of the camera and the direction of the camera head using corresponding points in two or more consecutive images; Focal length (in the virtual studio)f) And the vertical distance from the blue screen to the camera (Zo) Selecting and changing only one of the parameters, and finally extracting the camera parameters by maintaining the values of the parameters of the previous frame.