JPH07131817A - Method and device for image synthesis - Google Patents

Abstract

PURPOSE:To beautifully extract an original object are from an original image by dividing the area on the original image into the object area, a nonobject area, and a contour area by dividing the space into three. CONSTITUTION:A key signal generation part synthesizes a key signal by using a foreground image as an input signal and an (alpha) synthesizing part switches the foreground image and a background image by using this key signal to generate a composite image. The key signal generation part consists of a chroma key are division part 4, an contour decision part 6, and an (alpha) value generation part 6. The chroma key area division part 4 has a function for defining a closed curved surface and a function for deciding an area, and divides the color space where the pixels of the foreground image are arranged on the basis of three-dimensional color information into three by using two closed curved surfaces shaped optionally so that one include the other. The purpose of this division is to remove dust other than the contour part by deciding the contour by the contour decision part 5.

Description

Detailed Description of the Invention

[0001]

[Table of Contents] The present invention will be described in the following order. Field of Industrial Application Conventional Technology (FIG. 12) Problem to be Solved by the Invention Means for Solving the Problem (FIG. 2) Action Example (FIGS. 1 to 11) Effect of the Invention

[0002]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing method and an image synthesizing apparatus, and is particularly suitable for application to a special effect apparatus.

[0003]

2. Description of the Related Art Conventionally, there is a technique called "chroma key" as a method for cutting out an object used in the world of movies and television. This is a technique for synthesizing two images by extracting an original object region in a foreground image based on color information and fitting it into a foreground image, and is for realizing a beautiful and natural image synthesis.

Here, the principle of the conventional chroma key processing is shown in FIG.
2 is used for the explanation. First, all pixels in the foreground image 20 are mapped on an appropriate color space, an appropriate slice plane is determined, and the pixels are divided into groups to divide the region on the image 20 into colors. That is, the original object area (object area in the foreground image) and the original non-object area (non-object area in the foreground image) are combined in the object area 21 (object in the color space) based on the color information. Region) and the non-target region 22 (non-target region in color space).

Next, this information is applied to the original image 20 so that each pixel in the image 20 is converted into a target region 21 and a non-target region 22.
Classify into and. After that, a key signal indicating whether or not each pixel belongs to the target area 21 is generated. Switch 2
A composite image 26 is obtained by selecting one of the two image signals 24 and 25 input to the signal 3 according to the key signal.

As described above, since the chroma key processing divides the area without considering the shape of the object, it is very effective for extracting the object in the moving image and is widely used.

[0007]

The conventional chroma key processing has the following problems. That is, if the target object has a color close to the background (non-target object) or if the background (non-target object) has uneven color, a non-target area is created within the original target area, or conversely It is impossible to cleanly extract the original target area because dust such as a target area is formed in the target area.

Here, the dust is a pixel in which the object color and the background color are close to each other in the color space, and no matter how the slice plane is defined, it cannot be completely separated in the vicinity of the slice plane and remains. is there. Therefore, dust is also included in the contour area (contour area in the color space). Therefore, when the images are combined using the chroma key, it is necessary to remove dust from the combined image.

As a technique for solving such a problem, there is a method of removing a dust area by utilizing a brightness difference between a target area color and a non-target area color (background color).
No. 335279). However, this method is effective when the background color is constant or when only the image information of the background is known, but in other cases, the brightness values of the target area color and the non-target area color are There is a problem that it is difficult to divide the object area and the non-object area using only the difference.

The present invention has been made in consideration of the above points, and it is an object of the present invention to propose an image synthesizing apparatus and an image synthesizing method capable of extracting an original object region from an original image.

[0011]

In order to solve such a problem, according to the present invention, in an image synthesizing method for forming a synthetic image by extracting an original object region from an original image and fitting it into a foreground image, color information A first closed curved surface and a first closed curved surface, in which each pixel of the original image is arranged in a color space formed by arranging three-dimensionally and surrounding the color of the original object region in an arbitrary shape By dividing the color space into three by using the second closed curved surface having an arbitrary shape included by, the region on the original image is divided into a target region, a non-target region and a contour region. Then, for each pixel in the contour area, it is determined whether or not each pixel contributes to the contour of the target object, and each pixel in the contour area is determined as a target object area, a non-target object area, or a contour area. Embedded in a new target area, resulting in a new non-target area And to generate a key signal necessary for image synthesis using each information of the contour area.

Further, according to the present invention, in an image synthesizing apparatus for forming a synthetic image by extracting an original object region from an original image and fitting it into a foreground image, color information is arranged in a three-dimensional manner on a color space. Each pixel of the original image is arranged on the first closed surface, and it has an arbitrary shape enclosed by the first closed curved surface and the first closed curved surface which can surround the color of the original object region in an arbitrary shape. The second closed surface and the color space 3
Area dividing means 4 that divides the area on the original image into an object area, a non-object area, and a contour area by dividing the original image into pixels, and each pixel contributes to the contour of the object with respect to the pixels of the contour area. It is newly obtained by the contour discriminating means 5 and each of the contour discriminating means 5 which decides whether or not to do so and incorporates each pixel of the contour region into any one of the object region, the non-object region and the contour region. The key signal generating means 6 for generating a key signal necessary for image composition is provided by using each information of the target area, the non-target area, and the contour area.

[0013]

A first closed curved surface is provided in which each pixel of an original image is arranged in a color space formed by arranging color information in a three-dimensional manner and which can surround a color of an arbitrary shape and an original object region. By dividing the color space into three by using the second closed curved surface having an arbitrary shape included by the first closed curved surface, the region on the original image is divided into a target region and a non-target region. And a contour area. Next, with respect to the pixels in the contour area, it is determined whether or not each of the pixels contributes to the contour of the target object, and each pixel in the contour area is determined as an object area, a non-target object area, or a contour area. Built in. As a result, the area on the original image can be more accurately divided into the object area, the non-object area, and the contour area, so that the original object area can be extracted neatly from the original image.

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail below with reference to the drawings.

In FIG. 1, reference numeral 1 generally indicates the flow of chroma key processing. The key signal generation unit 2 generates a key signal using the foreground image as an input signal, and the α synthesis unit 3 uses this key signal to foreground the foreground image. An image and a background image are switched to generate a composite image.

As shown in FIG. 2, the key signal generating section 2 is composed of a chroma key area dividing section 4, a contour discriminating section 5, and an α value generating section 6. The chroma key area dividing unit 4 has a function of defining a closed curved surface and a function of determining an area, and includes a color space in which each pixel of the foreground image is arranged based on three-dimensional color information, one of which includes the other. The area on the image is divided into an object area, a non-object area, and a contour area by dividing the area into three using two closed curved surfaces of arbitrary shapes.

That is, the first closed curved surface that can surround the color of the original object region is defined in the color space, and the second closed surface that includes the first closed curved surface outside the first closed curved surface is defined. By creating the closed curved surface of, the area having the color belonging to the inside of the first closed curved surface is the object area, the area having the color belonging to the outside of the second closed curved surface is the non-object area, and the others are the contour areas. Can be divided into Because of the spatial resolution of the image, the pixel corresponding to the contour portion of the object has a mixture of the color component of the object and the color component of the background, so that the contour portion of the object is You can see that it is included.

In this embodiment, the purpose of dividing the area on the image into three areas is to remove dust other than the contour portion by discriminating the contour in the contour discriminating section 5, and not to the contour portion. Chromakey method that uses a polyhedron to remove the object color and divides the color space into three parts to improve the image quality of the composite outline (1992 issue, NICOGRAPH Proceedings, Yasushi Mishima, "Chromakey using polyhedron slices Software ").

2 which the chroma key area dividing section 4 has
Two functions will be described in detail. In this embodiment, a polyhedron is used instead of the closed curved surface. In the following drawings, those denoted by the same reference numerals have the same function.

FIG. 3 shows a data flow of the operation processing unit for defining the closed polyhedron. The input device 7 is a keyboard or a mouse capable of inputting commands for operating a polyhedron and numerical values such as vertex designations and coordinate values. The display device 8 is a display device for confirming the shape of the polyhedron. The vertex coordinate conversion unit 9 is a computing device that receives a command for operating a polyhedron from the input device 7, performs a computation, and reads / writes to / from a memory. Further, the internal point / external point discriminating unit 10 executes an operation for discriminating whether the designated coordinates are inside or outside the designated polyhedron.

There are two kinds of commands for operating the polyhedron, and by inputting the command with the input device 7, a polyhedron of any shape can be defined.
That is, according to the first command, the vertex coordinate conversion unit 9 reads the vertex data of the large and small polyhedra recorded in advance from the vertex coordinate source data memories A and B, and writes this vertex data in the vertex coordinate data memories A and B. . This 2
Two polyhedra are defined so that one contains the other.

By the following command, the vertex coordinate conversion unit 9
Specifies any one vertex of the polyhedron, inputs new coordinates, and updates the vertex data in the vertex coordinate data memories A and B to deform the polyhedron. Here, in the vertex coordinate conversion unit 9, before updating the vertex data, it is determined whether or not two large and small polyhedra intersect each other when the coordinates are converted.

Whether or not two large and small polyhedra intersect each other is determined by whether or not the changed vertex moves to the inside of the small polyhedron after coordinate conversion when the changed vertex is the vertex of the large polyhedron. If the changed vertex is a vertex of a small polyhedron, it is determined whether the changed vertex moves to the outside of the large polyhedron after coordinate conversion. The calculation for the intersection determination is performed by the internal point / external point determination unit 10 according to an instruction from the vertex coordinate conversion unit 9. If the result of the determination is that the two polyhedra intersect, the coordinates are not transformed.

After defining the closed polyhedron in this way, the region is determined. FIG. 4 shows the flow of the area determination processing. The display device 11 is a display device for confirming image data, and the area determination unit 12 is a calculation device for performing calculation and reading / writing from / to a memory.

The coordinate system of the color space in which the polyhedron exists depends on the format of the image data. In this embodiment, an example of data expressed in RGB space will be described. Here, the RGB space is a space in which colors are represented by the intensities of three colors of red (R), green (G), and blue (B). The image data memory 13 and the key signal data memory 14 store image data and key signal data, respectively, as shown in FIGS. 5 (A) and 5 (B). Here, α takes a value of 0 to 1.

The area determination unit 12 sequentially reads pixel values from the image data, determines which of the three spaces divided by the two polyhedrons the position of this pixel in the color space belongs to, and the result is determined. Is written in the address of the corresponding pixel in the key signal data memory 14. Here, the determination of which space belongs is performed by the following procedure.

The position of the pixel in the color space is the pixel value (R,
G, B). The inner point / outer point discriminating unit 10 discriminates twice, and the discrimination result is divided into three cases, namely, when a pixel exists inside a small polyhedron, when a pixel exists outside a large polyhedron, and in other cases. Divide. Based on these three cases, the α value is written in the corresponding pixel of the key signal data memory 14.

That is, α = 1 when the pixel exists inside the small polyhedron (exists in the object area), and α = 0 when the pixel exists outside the large polyhedron (exists in the non-object area). , If a pixel exists in a place other than these (exists in the contour area), write α = 0.5. If it is easier to create the shape of the polyhedron by placing the non-target area inside the small polyhedron and making the outside of the large polyhedron the target area, the pixels are inside the small polyhedron (non-target). Exists in the object area) α =
0, α = 1 when the pixel exists outside the large polyhedron (existing in the object area), and α = 0.5 when the pixel exists in other locations (existing in the contour area).

Next, the contour discriminating section 5 comprises a key signal data memory 14, a contour discriminating processing key signal data memory 15 and a contour discriminating processing calculating section 16, as shown in FIG. The contour discriminating unit 5 discriminates whether or not the pixel in the contour region contributes to the contour of the object and removes dust. Here, the removed dust is incorporated in the target area or the non-target area.

That is, as shown in FIG. 7, the key signal data memory 14 is first obtained by dividing the chroma key area.
The key signal data stored in the key signal data memory 15 for contour discrimination processing is copied. Next, the contour area of the copied key signal data is thinned.

Here, thinning means a process of reducing the area under the condition that the connection relation of the areas is not damaged. For example, based on a rule that paying attention to a certain pixel and its eight neighborhoods and erasing the pixel, if the continuity of the vertical, horizontal, and diagonal four-direction pixels passing through the pixel is lost, the pixel is not erased. Make a thin line. In this embodiment, when the pixel in the contour area does not satisfy the above rule, the pixel value is rewritten to a pixel value in the vicinity of the contour area. By repeating this operation, a contour area with a thickness of "1" can be created. FIG. 8 shows an example of thinning processing.

Next, only the contours that contribute to the contour of the object are detected from the thinned contour area. Here, in order to detect only the contours that contribute to the object in the thinned contour area, for all the pixels of the thinned contour area, the object area pixel and the non-object area are provided in the vicinity of 8 of the pixels. It is checked whether or not both the pixel and the pixel exist, and when the pixel satisfies the condition, the pixel is a pixel that contributes to the contour of the object. The pixel corresponding to this has a pixel value α =
Rewrite to 0.7. By this operation, the contour of the object is formed as the α value in the contour discrimination processing key signal data memory 15. FIG. 9 shows an example of contour detection processing.

The processing of pixel values in the above-described thinning processing and contour detection processing is performed by the contour discrimination processing calculation section 16.

After the contour detection, the pixels contributing to the detected contour of the object are copied to the original key signal data memory 14.

By performing the thinning process and the contour detecting process in the contour discriminating unit 5 as described above, dust can be easily removed as described below. That is, since the dust and the contour are discriminated based on topological information other than the color information, it is possible to easily discriminate the dust and the contour even in an image having color unevenness. Therefore, it is possible to easily deal with a general background (non-target area).

The dust removal process is executed in the following procedure. In the first process, the contour is formed in the original key signal data memory 14 by writing the pixel value of α = 0.7 of the contour discrimination processing key signal data memory 15 into the original key signal data memory 14. . In the second processing, on the key signal data memory 14, for 8 neighborhoods of the pixel value of α = 0.7, if the pixel value is α = 0.5, it is rewritten to α = 0.7. The third process repeats the second process until there are no corresponding pixels. In the fourth processing, for the pixel value of α = 0.5 on the key signal data memory 14, if there is a pixel of α = 0 or α = 1 in the 8 vicinity, the pixel value is rewritten. In the fifth process, the fourth process is repeated until there are no corresponding pixels. FIG. 10 shows an example of dust removal processing.

By the above operation, the three areas divided by the chroma key area dividing section 4 can be divided more accurately. That is, of the contour areas obtained by the first area division, the area to which the contour of the object belongs is α = 0.7, and the other areas are determined to be dust, and are incorporated in the surrounding object area or non-object area. Therefore, the area on the image can be more accurately divided into the object area (α = 1), the non-object area (α = 0) and the contour area (α = 0.7). Therefore, a key signal having three α values is created.

As shown in FIG. 11, the α value generation unit 6 uses the key signal data memory 14, the counter 17, and the α value generation processing key signal used as a work area for knowing the pixel distance from the object area. It is composed of a data memory A, an α value generation processing key signal data memory B used as a work area for knowing a pixel distance from a non-target area, and an α value generation processing operation unit 18 processing a pixel value. Has been done.

The α value generating unit 6 determines the α value of the contour area to an appropriate value in order to synthesize the foreground image and the background image cleanly. Appropriate algorithms for generating the α value can be selected to obtain the optimal synthesis.

As described above, the pixels belonging to the contour area include the color components of the object and the other color components, so that the α value is calculated by using α synthesis used in the α synthesis unit 3 described later. A natural composite image is generated by setting such that the pixel value of 1 is gradually changed to the pixel value of the background. The method of determining the α value for obtaining a natural composition will be described below.

For a pixel (x, y) having an outline area, it is assumed that the pixel distance d1 (x, y) from the object area and the pixel distance d2 (x, y) from the non-object area are separated.
(X, y) = d2 / (d1 + d2).

First, the original key signal data p (x, y) is read and the key signal data memories A and B for α generation processing are read.
Is written in the pixel pa (x, y), pb (x, y) of the pixel in accordance with the following rule (step (1)). That is, when p = 0, the key signal data memory A for α value generation processing is pa
= -1 is written, and if p << 0, then pa = 1-p is written. Further, in the α value generation processing key signal data memory B, pb = −1 is written when p = 1, and pb = p is written when p << 1.

Next, the counter 17 is set to "0" (step (2)), and the following processing is performed on both the key signal data memories A and B (step (3)). That is, when the pixel value is equal to the counter value in each pixel, the pixel values of the eight neighboring pixels that are integers smaller than “1” are counted
Rewrite to 1. After that, the counter 17 is incremented by "1" (step (4)). The above steps (3) to (4)
Is repeated until the relevant pixel is exhausted (step (5)).

Finally, in the original key signal data memory 14, α of the pixel which is the contour area (α (x, y) = 0.7)
The values are d1 (x, y) = pa (x, y), d2 (x, y)
= Pb (x, y), α (x, y) = d2 / (d1 + d
It is calculated and updated in 2) (step (6)).

By the above processing, it is possible to generate a key signal having an α value that gradually decreases from the object area to the non-object area.

Finally, the α synthesizing section 3 uses the key signal generated by the key signal generating section 2 to switch between the foreground image and the foreground image. That is, assuming that the key signal is obtained from the key signal generation unit 2 with a value of “0” to “1” for each pixel, a combined pixel is obtained using a method called α combination.

Here, the α synthesis is the key signal α (0 to 0
1), when the foreground pixel f (x, y) and the foreground pixel b (x, y) are given, the composite pixel d (x, y) is changed to d = af +
This is the method of obtaining by (1-a) b.

In actual chroma key composition, the object is often photographed in a single-color background such as blue back, and used as the foreground image. In such a case, it has been announced that the result is good if the pixel value of the single color background is set to b0 and the result is obtained by d = f + (1-a) (b-b0) (the above-mentioned "using polyhedral slices" Chroma key software "). In this embodiment, since the pixel value b0 of the single color background can be known from the above-mentioned non-object region, the α combination can be applied.

In the above configuration, when the original object area is extracted from the foreground image and fitted into the foreground image to form a natural composite image, first, the key signal generating section 2 uses the foreground image as an input signal and the chroma key area. The dividing unit 4 defines a closed polyhedron and performs area determination to divide the area on the image into three areas of an object area, a non-object area, and a contour area.

Next, in the contour discriminating section 5, the key signal data obtained in the chroma key area dividing section 4 is once copied,
The contour area of the copied key signal data is thinned to detect the contour. Pixels that contribute to the detected contour of the object are copied to the original key signal data to remove dust. As a result, the image can be more accurately divided into the three regions of the target region, the non-target region, and the contour region, and the key signal having three α values can be generated.

Thereafter, in the α value generation unit 6, in order to synthesize the original object area in the foreground image neatly with the foreground image,
The α value of the contour area is determined, and the α combining unit 3 combines the images by switching the foreground image and the background image using the key signal generated by the key signal generating unit 2.

According to the above configuration, the chroma key area dividing section 4 divides the area on the image into three areas of the object area, the non-object area and the contour area, and the contour discriminating section 5 performs thinning processing and By performing contour detection processing and distinguishing dust and contours based on topological information other than color information and removing dust, the area on the original image can be more accurately targeted areas and non-target areas. Since it can be divided into the object area and the contour area, the object area can be extracted neatly from the original image even when the non-object area has uneven color.

In the above embodiment, the case where the pixel value contributing to the contour of the object is set to α = 0.7 has been described. However, the present invention is not limited to this, and the pixel value contributing to the contour may be another value. You may

In the above embodiment, the case where the color space in which RGB is the color information is used has been described, but the present invention is not limited to this, and a color space in which hue, saturation and luminance are the color information is used. You may use.

[0055]

As described above, according to the present invention, each pixel of the original image is arranged in the color space, and the first closed surface capable of enclosing the color of the original object region in an arbitrary shape. And the second closed curved surface having an arbitrary shape enclosed by the first closed curved surface, the color space is divided into three, and the region on the original image is a target region and a non-target region. And a contour area. Next, with respect to the pixels in the contour area, it is determined whether or not each of the pixels contributes to the contour of the target object, and each pixel in the contour area is set as the target area, the non-target area, or the contour area. By incorporating it, the area on the original image can be more accurately divided into the object area, the non-object area, and the contour area, so that the original object area can be extracted neatly from the original image.

[Brief description of drawings]

FIG. 1 is a block diagram showing a flow of chroma key processing in an embodiment of the present invention.

FIG. 2 is a block diagram for explaining a key signal generation unit.

FIG. 3 is a block diagram for explaining a closed polyhedron operation process in a chroma key area dividing unit.

FIG. 4 is a block diagram for explaining a region determination process in a chroma key region dividing unit.

FIG. 5 is a schematic diagram used for explaining a configuration of image data and key signal data in the embodiment.

FIG. 6 is a block diagram for explaining a contour determination processing unit.

FIG. 7 is a block diagram for explaining a processing flow in a contour determination processing unit.

FIG. 8 is a schematic diagram for explaining a thinning process in a contour determination processing unit.

FIG. 9 is a schematic diagram for explaining a contour detection process in a contour determination processing unit.

FIG. 10 is a schematic diagram for explaining dust removal processing in a contour determination processing unit.

FIG. 11 is a block diagram for explaining an α value generation unit.

FIG. 12 is a block diagram for explaining the principle of conventional chroma key processing.

[Explanation of symbols]

1 ... Flow of chroma key processing, 2 ... Key signal generation unit,
3 ... α combiner, 4 ... Chromakey region divider, 5 ...
Contour discriminating unit, 6 ... α value generating unit, 7 ... input device, 8,
11 ... Display device, 9 ... Vertex coordinate conversion unit, 10 ... Internal point and external point discrimination unit, 12 ... Region determination unit, 13 ...
Image data memory, 14 ... Key signal data memory, 1
5 ... Key signal data memory for contour discrimination processing, 16 ...
Contour discrimination processing calculation unit, 17 ... Counter, 18 ... α value generation processing calculation unit, 20 ... Foreground image, 21 ... Target region, 22 ... Non-target region, 23 ... Switcher, 2
4, 25 ... Image input signal, 26 ... Composite image.

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Office reference number FI technical display location H04N 5/272

Claims (4)

[Claims] 1. An image synthesizing method for forming a synthetic image by extracting an original object region from an original image and fitting it into a foreground image, wherein the original information is placed in a color space in which color information is three-dimensionally arranged. A first closed curved surface, which has each pixel of the image and has an arbitrary shape and can surround the color of the original object region, and an arbitrary shape enclosed by the first closed curved surface. By dividing the color space into three using the closed curved surface of 2 and dividing the region on the original image into a target region, a non-target region and a contour region, with respect to the pixels of the contour region, It is determined whether or not each of the pixels contributes to the contour of the object, and the pixels of the contour area are incorporated into any of the object area, the non-object area, or the contour area, and the result is obtained. Information of the new target area, non-target area, and contour area An image synthesizing method characterized in that a key signal necessary for image synthesizing is generated by using the key signal. 2. In the pixel of the contour area, when the pixel does not meet a predetermined rule, the value of the pixel is rewritten to a pixel value in the vicinity of the contour area, and the processing is executed. The image synthesizing method according to claim 1, wherein only pixels that contribute to the contour of the object area are detected among the pixels of the contour area. 3. An image synthesizing apparatus for forming a synthetic image by extracting an original object region from an original image and fitting it into a foreground image, wherein the original image is placed on a color space in which color information is three-dimensionally arranged. Of the first closed curved surface having the arbitrary shape and surrounding the color of the original object region and the arbitrary shape enclosed by the first closed curved surface. An area dividing unit that divides the area on the original image into an object area, a non-object area, and a contour area by dividing the color space into three using the two closed curved surfaces. For a pixel, it is determined whether or not each pixel contributes to the contour of the object, and the contour that incorporates each pixel of the contour area into any of the object area, the non-object area, or the contour area Newly obtained by the discriminating means and the contour discriminating means. Object area,
An image synthesizing apparatus, comprising: a key signal generating means for generating a key signal necessary for image synthesizing by using each information of the non-object region and the contour region. 4. The thinning means for rewriting the value of the pixel to a pixel value in the vicinity of the non-contour area when the pixel does not meet a predetermined rule among the pixels in the contour area. 4. The image synthesizing apparatus according to claim 3, further comprising: a contour detecting unit that detects only pixels that contribute to the contour of the object region in the contour region thinned by the converting unit. .