JP3230481B2 - Television image synthesis method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to the synthesis of television images used for production of programs for television and movies, entertainment facilities, etc., and more particularly to the synthesis of synthesized images using a light source television camera for shadow photography. The present invention relates to a television image combining method for generating a shadow based on a subject serving as a foreground and adding the shadow to a combined screen.

[0002]

2. Description of the Related Art A person or other subject image is cut out and extracted from its background image in television program production, and the like.
Apart from that, a technique of photographing or fitting into a background image created by CG (computer graphics) is widely used. In addition, a chroma key device has been frequently used as a subject extraction device in the synthesis method, but the subject to be extracted and extracted is often a caster or the like. With the back (chroma key back) as the background,
A key signal indicating an image area of a subject is created by using a level difference caused by a color difference in an image signal, and is synthesized by fitting the key signal.

[0003] An example of the above-described conventional image synthesizing system will be described with reference to the drawings. FIG. 5 is a configuration diagram of a conventional image synthesis method. In the figure, 1 is a subject (person), 2 is a background (wall, floor) such as a blue background, 3 is a studio camera (studio color television camera), 4 is an extraction device (chroma key device), and 5 is a subject image signal. , 6 are subject key signals, 7 is an input terminal, 8 is a background image signal, 9 is a synthesizing device, 10 is an output terminal, 11 is a synthetic image signal, 12 is a synthetic image, and 16 is an image signal. The subject 1 is photographed by the studio camera 3 as a foreground in front of a background 2 such as a blue background. Then, the image signal 16 is output to the extraction device 4.
The subject image signal 5 of the area of the subject 1 extracted by a method such as chroma keying based on the color difference from the background 2 is output with its background portion replaced by a black signal and input to the synthesizing device 9. Is done. The key signal 6 indicating the area of the subject portion is also extracted and generated by the extracting device 4 and input to the synthesizing device 9. With this signal, the image signal of the subject 1 is inserted into the background image signal 8 for the composite image 12 created by photographing or CG input from the input terminal 7, synthesized by the synthesizing device 9, and then output. A composite image signal 11 is output from a terminal 10 and a composite image 12
Can be obtained.

When a three-dimensional space in which a subject 1 such as a person moves around is required as in a virtual studio, it is desirable that the walls and floor of the studio be a background 2 such as a blue background and uniform illumination. Non-uniform illumination can be dealt with to some extent by setting the parameters of the extraction operation of the extraction device 4 (chroma key device).
Phenomena such as the boundary part of being disturbed and becoming unnatural tend to occur,
In order to obtain a higher quality composite image 12, it is necessary to repeatedly adjust the illuminance and direction of the illumination and adjust the parameters of the extracting device 4, which requires a high level of skill and experience.
Further, as an important factor relating to the quality of the image composition, there is a possibility that the shadow of the subject 1 is expressed in the composite image 12 without a sense of incongruity. .

Next, in order to express the shadow of the subject 1 on the composite screen, A shadow area of the subject 1 formed on the floor of the studio (background 2 such as a blue background) by illumination is extracted and generated from the image signal 16 as a key signal of the shadow, and the image signal of the shadow area is converted into a black image signal. Convert and combine with the background image (not shown). As described above, in the shadow expression method, adjustment of the extraction device 4 (chroma key device) for extracting the shadow from the floor of the blue background (the background 2 such as the blue background) is more difficult than usual adjustment. Lighting and chroma key parameter adjustments need to be repeated and meticulous.

[0006] In the virtual studio, the position, roll (tilt), and pan of the virtual camera in the CG three-dimensional space are set in order to eliminate a sense of discomfort in the relationship between the subject 1 as the foreground of the composite image 12 and the three-dimensional CG image of the background. It is necessary to match and link shooting conditions such as tilt, tilt and zoom with the studio camera. Therefore, the reference position of the studio camera 3 and the subject 1 is determined, the position information of each point of the three-dimensional CG is determined, and information such as the movement, roll, pan, tilt, zoom, etc. of the studio camera 3 with respect to the reference position is determined by a sensor. And so on. This method includes: 1). A method of measuring the position information of the camera device for photographing the subject (studio camera 3) and the angle of view information such as roll, pan, tilt, zoom and the like by a sensor attached to the studio camera 3, or 2). The color of the blue back (background 2 such as blue back) is set to a plurality of colors blue, and an information pattern such as a checkerboard pattern is drawn, and the pattern information in the image signal 16 is recognized. ), There is a pattern information method for obtaining the position and angle of view information. Based on these information, the computer of the CG device calculates the movement of each pixel and makes the shooting conditions of the virtual camera the same as those of the studio camera 3 (FIG. Not shown). In the above,
The latter 2). The method based on the pattern information is an excellent method that can use the existing camera device as it is and can also use a handy camera.However, the method based on the background video signal in which multiple blue information patterns are drawn In addition to the extraction, it is extremely difficult to extract the shadow area by adjusting the chroma key parameters.

[0007]

As described above, in a composite image in a chroma key or a virtual studio, etc., it is an important factor related to image quality that the expression of the shadow of a subject which is a foreground is not unnatural. If this condition is not satisfied, the image of the subject will feel floating, lacking a feeling of contact with the ground, resulting in an unnatural image. In addition, normal chroma key parameter adjustment for separating and extracting a subject from a blue background background image requires adjustment by advanced technology and experience. In addition to this, it is necessary to extract a shadow image signal region. Requires more precise and fine adjustment and adjustment of background lighting. Therefore, the boundary portion of the shadow is easily disturbed, and it is difficult to obtain a high-quality image signal. In particular, it is extremely difficult to extract the shadow from the background of the blue background on which the information pattern such as the above-described lattice pattern is drawn. is there. Further, conventionally, the chroma key or virtual studio apparatus is very expensive and the operation cost per unit time is high. However, as described above, it is much more time and labor to express the shadow of the subject without discomfort. And the production cost of programs and the like is high. Further, the shadow extracted by the above-described method is a shadow projected on the floor or wall of the studio, which is the background of the subject, and furniture such as a desk, furniture, pillars and the like are added to a background image such as CG to be synthesized. If there is an object of, it will not match the shadow to be projected on it. Therefore, the present invention provides a method that can easily solve the above-mentioned problems.

[0008]

In view of the above, the present inventor has conducted intensive studies and generated an image signal of a subject image area as a shadow image signal by changing its position, shape, and the like by a coordinate transformation process. To obtain a high-quality composite image, and by setting the position of a virtual light source for projecting a shadow, solves the drawback that requires a great deal of time and labor due to fine adjustment of the illumination and parameters. As a result, a high-quality composite image could be obtained. The means are described below. (1) A means for separating and extracting an image signal of a subject to be a foreground of a composite image and a key signal indicating an image area of the subject from a background image of the subject, and combining the subject image and a different background image. In the image synthesizing method, a key signal based on a subject image area extracted from an output image signal of a light source television camera for shadow photographing is provided by a coordinate conversion means of an image signal, and a position, a direction, and an image of the studio television camera are extracted. A television image synthesizing method wherein a shadow of an object is generated and synthesized by performing coordinate conversion as a key signal adapted to an object image based on a corner. (2) The television image synthesizing method according to the above (1), wherein the television image synthesizing method is provided with a low-pass filter to dull the waveform of a key signal for synthesizing a shadow. (3) a means for changing the key signal level in an inclined manner;
The television image synthesizing method according to the above mode (1) or (2), wherein the transmittance of the shadow is inclinedly changed based on the ground point.

[0009]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention is characterized in that a television image synthesizing system is provided with a generating device having the same function as a special effect device having a coordinate conversion processing function,
By performing a two-dimensional or three-dimensional coordinate conversion process so that the image signal and the key signal of the region of the subject extracted by the extraction device from the background image such as the blue background have the target shadow position and shape. , A shadow image, and a high-quality composite image is obtained. The special effect device is generally used in a television system, and has a function of performing processing such as rotation, inversion, movement, reduction, enlargement, skew, perspective, and other defocus effects of an image by performing coordinate conversion of an image signal. Have.

First, a description will be given of the coordinate conversion means performed by the generating device 8 of the present invention. An arbitrary pixel position in the image is expressed in two dimensions (X, Y), and when both input and output are two-dimensional images, any pixel (x, y) of the input image is replaced with a pixel (x ′, y) of the output image. If mapped to '), In the above equation, f is the mapping position of an arbitrary pixel (x, y) and indicates the position of the pixel x on the output image, and g is the mapping position of an arbitrary pixel (x, y) and the pixel y of the output image. Indicates the position of the axis. In addition, coordinate conversion such as three-dimensional rotation, movement, reduction (enlargement), and skew is performed when an arbitrary pixel (x, y) of an input image is mapped to an output image (x ′, y ′). It is expressed as follows by a projective transformation matrix. Equation 1 shows a mathematical expression in which a two-dimensional image input is coordinate-transformed and mapped to a three-dimensional output image.

[0011]

(Equation 1)

The above is schematically shown below. FIG. 1 is a schematic diagram showing an input image according to the present invention as a three-dimensional output image. FIG. 1A shows an input image, and FIG. 1B shows an output image after conversion. In the figure, 17 is an input image, 20 is an output image, 23 is an image symbol, and 25 is an output image frame. As shown, an arbitrary pixel (x, y) point and the symbol A of the input image in the input image (a) are converted to the pixel (x ′,
y ′) and the symbol A. Therefore, 1
If the image is finely divided and the above equation is applied to each of the images, the input image can be mapped to an object having a three-dimensional shape such as a sphere or a cube. It can be generated as if it were projected on an object such as a desk, furniture, pillar, etc. inside.

Next, a composite screen according to the present invention will be described. FIG. 2 is an explanatory diagram of a composite screen according to the present invention.
(A) The figure is a state diagram in which the coordinate conversion parameters are set so as to be centered on the feet of the person. (B) The figure is a figure in which the center of the coordinate transformation is off the foot of the person when the person moves below the screen. In the figure, 18 is a person, 19 is a foot (ground point), 21 is a shadow of the person, and 22 is a moving direction of the person. Since the coordinate transformation for generating the shadow 21 is performed on the entire image, if the parameter is a fixed value, it cannot follow the movement of the subject, for example, the person 18. Composite screen (a)
In the figure, the coordinate conversion parameter is set to the ground point 1 of the person 18.
9, that is, a state in which the foot is centered and the shadow 21 is generated from the foot 19 (ground point) of the person 18, but if the parameter is fixed, the person 18 moves downward in the screen. 22, the center step 19 of the coordinate transformation is the step 1 of the person 18 as shown in the composite screen (b).
It is out of 9

A first method for solving the above problem. Person 18
A first method for causing the shadow 21 to follow the movement of the object is a foot 19 (ground point) of the subject, that is, when the subject is a person 18, the two-dimensional coordinates of the foot 19 in the image are represented by:
Detected from the position corresponding to the foot 19 in the key signal,
This method uses the coordinates as the center of conversion. In this case 2
Since the calculation is performed in three dimensions, a consistent shadow cannot be generated three-dimensionally, but the shadow can be easily set so that it looks natural. FIGS. 2C and 2D show examples of the above description, wherein FIG. 2C is a state diagram in which the coordinate conversion parameters are set centered on the foot of the person, and FIG. Respectively shows a state diagram in which the shadow follows the person of the subject even if the object moves downward. As shown in the figure, even if the subject person moves in the composite screen (c), the shadow follows the subject person as shown in the composite screen (d).

Next, various processing means for an image signal according to the present invention will be described. For the low-pass filter, the image signal is passed through the low-pass filter,
A defocus effect such as shifting the focus of the lens can be performed. If this is performed on the key signal, the waveform of the key signal becomes dull, and a soft shadow having a soft outline can be generated. Regarding the adjustment of the transmittance of the key signal level, the key signal indicates a level value of 0% to 100%.
At%, only the shadow image is output in the key signal area, and between 0% and 100%, the background image and the shadow image are mixed at a rate proportional to the level value, and the transparency between them is obtained. Output. As described above, the video signal in the shadow area to be generated is usually converted to a black signal, and the video signal of the subject is used as it is, and the background image, the shadow image, and the subject image are synthesized by the synthesis device in this order. By adding a shadow by adjusting the key signal level to an appropriate degree of transparency, a natural and high-quality composite image can be obtained. Regarding the means for changing the inclination of the key signal level, in the above case, the entire image area of the shadow is not made to have a uniform transparency. By changing the level of the key signal at an angle, a more natural shadow or reflection can be obtained.

A second method for solving the above-mentioned problem. The second method is to set a three-dimensional coordinate in a studio regarding a shadow, and to generate a shadow at a virtual position of a light source at which a shadow of the subject is to be generated. Is installed, and a shadow extracting device extracts a shadow subject image signal and a shadow key signal and inputs them to the generating device. The operation will be described below with reference to FIG.

FIG. 3 is an explanatory diagram of shadow generation by the light source camera according to the present invention. FIG. 3A is a composite image diagram in which a light source camera and a studio camera shoot in a studio and a person and a shadow are generated. (B) The figure is a photographed image of the light source camera. (C) The figure is the image figure of the coordinate transformed shadow. (D) The figure is a composite image of a person and a shadow. (E) shows a photographed image of a studio camera. In the above figures (a) to (e), reference numeral 24 denotes a light source camera, 26 denotes an image of a shadow taken by the light source camera, 27 denotes an image of a shadow transformed by coordinates, 28 denotes a composite image of a person and a shadow, and 29 denotes an image of a studio camera. , A indicates a point on the subject surface, B indicates a point on the floor surface, and C indicates a point on the subject image of the light source camera. (I)
In the figure, since the shadow 21 of the subject 18 is generated when the light emitted from the light source is blocked by the person 18, the light source camera 24 is installed at the position of the light source to photograph the person 18. For example, the subject image in the image is the same as the image area of the shadow 21 of the person 18 projected on the floor. The light source is assumed to be a point light source. For example, a light beam emitted from the light source and passing through a point A on the subject surface irradiates a point B on the floor surface. Assuming that a point on the subject image of the camera is C, points A, B, and C are on a straight line, and the same applies to other portions of the subject surface. (B) The photographed image 26 of the shadow by the light source camera is shown in FIG. The figure is a standing image of a shadow obtained by photographing the person 18 from the side.

Next, information on the position and angle of view of the light source camera 24 and the studio camera 3 is input to the generator.
An image area of the subject 18 extracted and generated based on the image of the light source camera 24 is subjected to coordinate conversion processing by the generation device, so that it can be converted as a shadow 21 adapted to an image captured by the studio camera 3. .
(C) The figure shows the shadow-converted image 27.
(E) FIG. 29 shows a photographed image diagram 29 of the studio camera,
And (d) FIG. 28 shows a composite image of a person and a shadow. In this method, calculation can be performed with a complete three-dimensional model based on a light source that generates the shadow 21. Therefore, a complete shadow can be generated three-dimensionally, and furniture such as a desk or the like existing in the background image can be generated. By inputting the data on the arrangement of objects (not shown) such as pillars and the like to the generation device and performing computer processing, the synthesizing process is performed as if the shadow of the subject as the foreground is projected on those objects. Can do it.

Next, an embodiment of the present invention will be described with reference to a block diagram. FIG. 4 is an example of a configuration diagram of an image synthesis method of a person and a shadow according to the present invention. In the figure, reference numeral 13 denotes a camera data detection device, 14 denotes a shadow extraction device, 15 denotes a shadow subject key signal, and 30 denotes a generation device. The present embodiment relates to a shadow, in which a subject 1 is photographed by a tagio camera 3 in front of a studio background 2 such as a blueback, and the image signal is converted to a black signal or the like by an extraction device 4 in a background portion. The image signal 5 of the subject 1 and the key signal 6 for the subject 1 are extracted by a chroma key method and input to the synthesizing device 9. In addition, the above-described light source camera (television camera for shadow shooting) installed at a position of a virtual light source that generates shadows
24, the shadow extraction device 14 extracts the shadow subject key signal 15 indicating the image area of the subject 1,
It is input to the generation device 30.

On the other hand, the studio camera 3 and the light source camera 2
4, the position and direction in three-dimensional coordinates in the studio,
The camera data of the angle-of-view information is detected by the camera data detection information 13 by a mechanical sensor mounted on the camera device or by means for recognizing and calculating pattern information drawn on the blue background 2 in the image signal. Is input to the generation device 30. In this case, if the position and the angle of view of the light source camera 24 are fixed, camera data relating to the light source camera 24 is unnecessary, and the data of the position and angle of view information may be input to the generation device 30. Also, by inputting data of the three-dimensional shape and the arrangement of objects such as furniture such as desks and pillars existing in the background image to be finally synthesized into the generating device 30 in advance and performing calculation processing by a computer, The effect can be as if a shadow is cast on the object. In the generating device 30, which coordinates in the three-dimensional model are reflected on arbitrary pixels in the image of the studio camera 3 from the position and angle of view information of the studio camera 3 and the input three-dimensional shape and arrangement data. , And then calculate which pixel of the light source camera 24 reflects the coordinates in the three-dimensional model. By performing such a coordinate conversion process, it is possible to completely follow the subject and generate a key signal of a three-dimensionally consistent shadow. Conversion processing is performed so as to obtain a shape, and a key signal for shadow is generated and input to the synthesizing device 9.

[0021]

According to the present invention, the following excellent effects are exhibited. 1. According to the invention of claim 1 of the present invention, a key signal based on a subject image area extracted from an output image signal of a light source television camera for shadow photographing is provided, comprising a coordinate transformation means of an image signal. By performing coordinate conversion as a key signal adapted to the subject image based on the position, direction, and angle of view to generate a shadow of the subject and synthesize the image, the shadow of the subject that follows the movement of the subject is generated. Can be combined. For this reason, the conventional method of extracting a shadow from an actual image signal tends to be unnatural because its boundary is disturbed, and requires a great deal of time and effort due to fine adjustment of illumination and parameters. According to the present invention, a high-quality composite image can be easily obtained. 2. According to the second aspect of the present invention, by providing a low-pass filter and dulling the waveform of a key signal for synthesizing a shadow, it is possible to perform a defocusing effect as if the focus of the lens was shifted. Can create soft and natural shadows. 3. According to the third aspect of the present invention, there is provided means for changing the key signal level in a gradient manner, and a more natural and high-quality composite image is obtained by adjusting the transmittance of the shadow in a gradient manner based on the ground point. be able to.

[Brief description of the drawings]

FIG. 1 is a schematic diagram showing an input image according to the present invention as a three-dimensional output image.

FIG. 2 is an explanatory diagram of a composite screen according to the present invention.

FIG. 3 is an explanatory diagram of shadow generation by the light source camera according to the present invention.

FIG. 4 is a configuration diagram of a screen combining method of a person and a shadow according to the present invention.

FIG. 5 is a configuration diagram of a conventional image synthesis method.

[Explanation of symbols]

1: subject 2: background 3: studio camera 4: extraction device 5: subject image signal 6: subject key signal 7: input terminal 8: background image signal 9: combining device 10: output terminal 11: combined image signal 12: combined image 13: Camera data detection device 14: Shadow extraction device 15: Key signal 16: Image signal 17: Input image 18: Person 19: Foot 20: Output image 21: Shadow of person 22: Moving direction of person 23: Image symbol 24: Light source camera 25: Output image frame 26: Captured image of shadow by light source camera 27: Image of shadow transformed by coordinates 28: Composite image of person and shadow 29: Captured image of studio camera 30: Generation device A: On the subject surface Point B: Floor point C: Point on subject image of light source camera

──────────────────────────────────────────────────続 き Continued on front page (58) Field surveyed (Int.Cl. ⁷ , DB name) H04N 5/272-5/275 H04N 9/75

Claims (3)

(57) [Claims] A means for separating and extracting, from a background image of the subject, an image signal of the subject as a foreground of the composite image and a key signal indicating the image area of the subject, and combining the subject image and a different background image. In the image synthesizing method provided with, a key signal based on a subject image area extracted from an output image signal of a light source television camera for shadow photographing is provided, and a position, a direction, and a position of a studio television camera are provided. A television image synthesizing method characterized by generating and synthesizing a shadow of a subject by performing coordinate conversion as a key signal adapted to a subject image based on an angle of view. 2. The television image synthesizing method according to claim 1, further comprising a low-pass filter for dulling the waveform of a key signal for synthesizing a shadow. 3. The television image composition according to claim 1, further comprising means for changing the key signal level in a gradient manner, wherein the transmittance of the shadow is varied in a gradient manner based on the ground point. method.