JP2000151982A - Image compositor, image compositing method image generating device, image generating method and served medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently composite images. SOLUTION: A 1st bounding box including all non-zero matte values and a 2nd mounding box consisting of a mat value of only '1' are set to a matte image, and image composition processing is applied to a forefront images and a background image corresponding to each of three areas consisting of the area outside the 1st bounding box (outer area), the area of the 1st bounding box but outside the area of the 2nd bounding box (intermediate area) and the area of the 2nd bounding box (internal area).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing apparatus, an image synthesizing method, an image generating apparatus, an image generating method, and a providing medium, and more particularly, to an image synthesizing method using mat values. The present invention relates to an image synthesizing apparatus and an image synthesizing method, an image generating apparatus and an image generating method, and a providing medium.

[0002]

2. Description of the Related Art In recent years, with the progress and development of digital signal processing technology, for example, in the production of movies, television broadcast programs, etc.
Digital synthesis of images, such as combining images of characters taken separately from the background, has been increasingly performed.

In digitally synthesizing an image, a mat value (also called a key signal or α (alpha) value) set for each pixel constituting a foreground image (foreground image) to be synthesized is used. Can be

[0004] That is, the mat value represents the opacity of each pixel of the foreground image by a value in a predetermined range. Generally, the minimum mat value in the predetermined range is such that the pixels of the foreground image are completely transparent. The maximum mat value corresponds to a state where the pixels of the foreground image are completely opaque.

Here, in the case of performing image synthesis using mat values, the minimum value of the range of possible values of the mat value is generally normalized to 0, and the maximum value is normalized to 1, respectively. In the following, a description will be given assuming that the minimum value or the maximum value of the mat value is 0 or 1, respectively.

[0006] As described above, the mat value represents the opacity of each pixel of the foreground image, and therefore exists for each pixel constituting the foreground image. Sometimes called an image. The matte image may be given by itself,
It may be provided with the foreground image (as part of the foreground image).

Image synthesis using mat values is performed, for example, as follows. That is, the pixel values of the pixels of the foreground image and the background image (background image) are RGB (Red, Gre
en, Bule), and R,
The G and B components are represented as Fr, Fg and Fb, and the R, G and B components of the pixels of the background image are represented as Br, Bg and Bb. Further, the mat value for the pixel of the foreground image is represented by M, and the R, G, B components of the pixel of the combined image obtained by combining the foreground image and the background image are represented by Cr, Cg, Cb. In this case, the R, G, and B components of the pixels of the composite image are obtained by the following equations.

Cr = M × Fr + (1-M) × Br Cg = M × Fg + (1-M) × Bg Cb = M × Fb + (1-M) × Bb (1)

[0009] The image synthesis according to the equation (1) is described in, for example, T. Porter and T. Duff, "Compositing Digi.
tal Images ", Computer Graphics Vol.18, No.3 (1984),
The details are disclosed in pp. 253-259 and the like.

From equation (1), the matte value M represents the degree of contribution of the pixels of the foreground image to the composite image. For a pixel having a matte value M of 1 (pixels of the composite image), only the foreground image appear. For a pixel whose mat value M is less than 1 and larger than 0, the foreground image and the background are mixed in accordance with the mat value M, and for a pixel whose mat value M is 0, only the background image appears.

[0011]

The operation of the equation (1) includes multiplication and addition (subtraction).
Generally, it takes more time than the addition. In particular, when synthesizing a high-resolution image (an image having a large number of pixels), since the number of pixels is large, the number of multiplications becomes enormous, resulting in a reduction in processing speed.

[0012] In order to reduce such slowing of processing,
It is necessary to reduce the amount of calculation, and as a method of reducing the amount of calculation, for example, J. Foley, et al., Computer Grap
hics: Principles and Practice, Addison-Wesley, sec
Ond edition, 1990., etc., which limit a region to be calculated (calculation target region) using a so-called bounding box (concept of a circumscribed rectangle) is known.

When the amount of calculation in image synthesis is reduced by using a bounding box, as shown in FIG. 11, a rectangular area circumscribing a non-zero mat value (see FIG. Area enclosed by a dotted line), and only within the rectangular area,
What is necessary is just to perform the calculation of Formula (1). That is, although this rectangle is called a bounding box, since the mat value of an area outside the bounding box is 0,
In the composite image, the background image appears as it is. Therefore, the background image corresponding to the area outside the bounding box may be copied as it is to the composite image, and the calculation of Equation (1) may be performed only on the foreground image and the background image corresponding to the area inside the bounding box.

Here, in the matte image of FIG. 11 (similarly in FIG. 3 described later), an area without a shadow is an area having a matte value of 0 (hereinafter, appropriately referred to as a transparent area).
Is a region where the mat value is larger than 0 and smaller than 1 (hereinafter referred to as a semi-transparent region as appropriate), and a portion where the matte value is 1 Each region (hereinafter, referred to as a completely opaque region as appropriate) is shown.

When the composite image is generated separately from the foreground image and the background image, it is necessary to copy the background image corresponding to the area outside the bounding box to the composite image as described above. When a composite image is generated by overwriting the background image, no special processing is required for the background image corresponding to the area outside the bounding box.

The bounding box may have any shape as long as it includes all non-zero mat values, and need not be rectangular. However, since it is easy to determine the inside and outside of the bounding box, a rectangular bounding box is generally used. Is often done.

Furthermore, in FIG. 11, the bounding box is set so as to circumscribe the translucent area, but it is not always necessary to make contact.

Next, a conventional image synthesizing process using a bounding box will be described with reference to a flowchart of FIG. Here, it is assumed that the composite image is generated by overwriting the background image (therefore, in this case, as described above, it is necessary to perform particularly processing on the background image corresponding to the area outside the bounding box. No).

First, in step S81, a bounding box is set. That is, step S
At 81, the user designates, for example, the lower left vertex and the upper right vertex of the bounding box.

Here, as shown in FIG. 11, the lower left vertex of the matte image is set to the origin (0, 0), and the x-axis or the y-axis is taken from left to right or from bottom to upward. Using the coordinate system, the coordinates of the lower left or upper right vertex of the bounding box specified by the user are _expressed as (X _min1 , Y _min1 ) or (X _max1 , Y _max1 ), respectively (X _min1 <X _max1 , Y _min1 < Y _max1 ). Also, the number of horizontal or vertical pixels of the mat image is W or H, respectively.

When the points (X _min1 , Y _min1 ) and (X _max1 , Y _max1 ) are specified by the user, a rectangle is determined with each point as the lower left vertex or the upper right vertex. , The rectangle is set as the bounding box.

When the bounding box is set,
Hereinafter, the inside of the bounding box (X _min1 ≦ x
≦ X _max1 , Y _min1 ≦ y ≦ Y _max1 ) Only the foreground image and the background image are processed.

That is, in step S82, the pixel y
A variable y representing coordinates is initialized to Y _min1 and step S
Proceeding to 83, it is determined whether the variable y is _{equal to} or less than _Ymax1 . In step S83, the variable y is set to Y _max1
If it is determined to be the following, that is, if the variable y represents the y coordinate of the pixel in the bounding box, the process proceeds to step S84, and a one-line calculation subroutine (for example, a function or an object) to be described later is called. Thus, a one-line calculation process is performed. The one-line calculation subroutine is called using the variable y and the x-coordinates X _min1 and X _max1 of the left and right boundaries of the bounding box as arguments.

When the one-line calculation process is completed, step S8
Proceeding to 5, the variable y is incremented by 1 and the process returns to step S83. Then, in step S83, when it is determined that the variable y is not _smaller than or _{equal to Ymax1} , the process ends.

Next, the one-line calculation process of FIG. 12 will be described with reference to the flowchart of FIG. As described above, in step S84 of FIG. 12, the one-line calculation subroutine is called with the variable y and the x-coordinates X _min1 and X _max1 of the left and right boundaries of the bounding box as arguments, but in FIG. , X _mi
n1 or X _max1 is _designated as XL or XR, respectively.

In the one-line calculation process, first, in step S91, a variable x representing the x coordinate is initialized to XL, and the flow advances to step S92 to determine whether the variable x is equal to or smaller than XR. In step S92, when it is determined that the variable x is equal to or smaller than XR, that is, when the variable x represents the x coordinate of the pixel in the bounding box, the process proceeds to step S93, and the variable M is set to the coordinates ( Pixel) (x, y) matte value [x, y]
Is set. Here, the array variable matte [x,
[y] stores a mat value of the coordinates (x, y).

Then, the process proceeds to step S94, in which it is determined whether or not the mat value M is 0. If it is determined that the value is not 0, the process proceeds to step S95, in which it is determined whether or not the mat value M is 1. In step S95, when it is determined that the mat value M is 1, that is, the pixel at the coordinates (x, y) of the mat image is completely opaque in the bounding box (in FIG. 11, a dark shadow is added. If it is a pixel of a certain part), the process proceeds to step S96, and the pixel value B [x, y] of the pixel (x, y) of the background image is replaced with the pixel value F [x, y] of the pixel (x, y) of the foreground image. y] is copied and the process proceeds to step S98.

Here, R, R of pixel (x, y) of the background image
G and B components are array variables B [x, y].
r, B [x, y]. g, B [x, y]. b, the R, G, and B components of the pixel (x, y) of the foreground image are represented by array variables F [x, y]. r, F [x, y]. g, F
[X, y]. In step S96, the calculation represented by the following equation is performed.

B [x, y]. r = F [x, y]. r B [x, y]. g = F [x, y]. g B [x, y]. b = F [x, y]. b ... (2)

If it is determined in step S95 that the matte value M is not 1, that is, if the pixel at the coordinates (x, y) of the matte image is in the semi-transparent area in the bounding box (in FIG. Part marked with)
In step S97, the process proceeds to step S97, using the pixel (x, y) of the foreground image and the pixel (x, y) of the background image, and calculating the following equation corresponding to the equation (1). Is performed, and the process proceeds to step S98.

B [x, y]. r = M × F [x, y]. r + (1-M) * B [x, y]. r B [x, y]. g = M × F [x, y]. g + (1-M) * B [x, y]. g B [x, y]. b = M × F [x, y]. b + (1-M) * B [x, y]. b ... (3)

On the other hand, if it is determined in step S94 that the mat value M is 0, that is, if the pixel at the coordinates (x, y) of the mat image is in the transparent region (in FIG. If it is a pixel of a part (not attached), the process proceeds to step S98, the variable x is incremented by 1, the process returns to step S92, and the same processing is repeated thereafter.

Then, in step S92, the variable x
Is determined not to be equal to or smaller than XR, that is, when the processing is performed on all of the pixels constituting the line whose y coordinate is represented by the variable y and which are within the bounding box, the process returns.

As can be seen from equations (2) and (3), the composite image is a variable B in which the background image is set.
Set to [x, y].

As described above, when a bounding box is used, it is not necessary to perform addition or multiplication on pixels corresponding to an area outside the bounding box, so that the number of calculation targets can be reduced. However, pixels in the bounding box basically need to be subjected to the operation of the equation (1).

Therefore, in the one-line calculation process shown in FIG. 13, the mat value M is determined for each pixel in the bounding box (steps S94 and S9).
5) If the value is either 0 or 1, it is excluded from the calculation of Expression (1). In this case, the mat value M is determined for all the pixels in the bounding box. Must be done and is not efficient.

Here, for example, Japanese Patent Application Laid-Open No. 8-115413
Japanese Patent Application Laid-Open Publication No. H11-157556 discloses that in a graphic image synthesizing process in a page description language, when a mat value used for a region extraction operation indicates that it is completely opaque, an operation operation that should be performed is performed in a short time. There is disclosed a method of replacing the clipping operation with a clipping operation that can be performed. Since the clipping operation is equivalent to copying, the method can perform the processing at high speed.

However, in this method, the mat value is 0
Alternatively, it is assumed that only two values of 1 can be taken.

On the other hand, in the production of a movie or the like, in order to ensure the naturalness of the composite image (to reduce unnaturalness), not only two values of 0 and 1 but also an intermediate value between them are taken as mat values. Generally, what is called a so-called soft key is used.

Therefore, in producing a movie or the like, it is difficult to apply the method disclosed in Japanese Patent Application Laid-Open No. Hei 8-115413 from the viewpoint of the image quality of a composite image and the like.

The present invention has been made in view of such a situation, and it is an object of the present invention to efficiently combine images.

[0042]

According to a first aspect of the present invention, there is provided an image synthesizing apparatus, comprising: a first area including all mat values which are not a minimum value in a predetermined range; Setting means for setting a second area consisting only of a value mat value, an area outside the first area, an area other than the second area of the first area, and a third area.
Processing means for performing synthesis processing for image synthesis on the first and second images corresponding to each of the three regions.

In the image synthesizing method according to the present invention, the matte image includes only a first area including all non-minimum mat values in a predetermined range and a maximum mat value in a predetermined range. A setting step of setting a second area of the first area, an area outside the first area, and a second area of the first area.
And a processing step of performing synthesis processing for image synthesis on the first and second images corresponding to the three areas of the non-area and the second area, respectively.

According to a ninth aspect of the present invention, the provision medium includes only a first area including all non-minimum mat values in a predetermined range and a maximum mat value in a predetermined range. A setting step of setting a second area;
For the first and second images corresponding to the three areas of the area outside the first area, the area other than the second area of the first area, and the second area, respectively, Control information comprising a processing step of performing a combining process.

According to a tenth aspect of the present invention, for the mat image, only the first area including all non-minimum mat values in the predetermined range and the maximum mat value in the predetermined range are used. And setting means for setting the second area.

According to a fourteenth aspect of the present invention, in the image generation method according to the first aspect, only a first area including all non-minimum mat values in a predetermined range and a maximum mat value in a predetermined range are obtained. And a setting step of setting a second area.

According to a fifteenth aspect of the present invention, for a mat image, only a first area including all non-minimum mat values within a predetermined range and a maximum mat value within a predetermined range are provided. It is characterized by providing control information including a setting step of setting a second area.

A providing medium according to a sixteenth aspect of the present invention comprises a first area in a mat image that includes all mat values that are not the minimum value in a predetermined range, and a second area that includes only a mat value having a maximum value in a predetermined range. It is characterized in that a mat image is provided together with specific information for specifying an area of the image.

[0049] In the image synthesizing apparatus according to the first aspect, the setting means may include, for the mat image, a first area including all mat values which are not the minimum value in the predetermined range, and a maximum area in the predetermined range. And a second area consisting of only the mat value of
The first and second images corresponding to the three regions of the region other than the second region and the second region are subjected to a combining process for combining images.

In the image synthesizing method according to the present invention, only the first area including all the non-minimum mat values in the predetermined range and the maximum mat value in the predetermined range are provided for the mat image. And a first area corresponding to the three areas of the first area, the area outside the first area, the area other than the second area, and the second area. The second image is subjected to synthesis processing for image synthesis.

According to a ninth aspect of the present invention, in the provision medium, only the first area including all the non-minimum mat values in the predetermined range and the maximum mat value in the predetermined range are provided for the mat image. The first and second regions corresponding to the three regions of the region outside the first region, the region other than the second region of the first region, and the second region are set. The control information for causing the information processing apparatus to perform the process of performing the synthesis process for the image synthesis on the second image is provided.

In the image generating apparatus according to the tenth aspect, the setting means includes, for the mat image, a first area including all the non-minimum mat values in the predetermined range, and a maximum area in the predetermined range. And a second area consisting of only the mat value of the second value.

In the image generation method according to the present invention, for the mat image, only the first area including all the non-minimum mat values in the predetermined range and the maximum mat value in the predetermined range are provided. Is set.

[0054] In the providing medium according to claim 15,
A process of setting a first area including all non-minimum mat values in a predetermined range and a second area including only the maximum mat value in a predetermined range in a mat image is performed by an information processing. It is provided to provide control information to be performed by the device.

In the providing medium according to claim 16,
The matte image includes identification information for identifying a first area including all non-minimum mat values in a predetermined range and a second area including only the maximum matte value in a predetermined range. Images have been made available.

[0056]

FIG. 1 shows a configuration example of an embodiment of an image processing apparatus to which the present invention is applied. This image processing apparatus combines an foreground image (first image) and a background image (second image) in accordance with a mat image to generate a synthesized image, and a mat image used for the image synthesis. Function as any of the image generation apparatuses that generate the image data.

That is, the ROM (Read Only Memory) 1
For example, it stores a program for IPL (Initial Program Loading). CPU (Centoral Proce
The ssing unit 2 reads and executes an OS (Operating System) program stored in the external storage device 7, for example, and under the control of the OS, also as an application program stored in the external storage device 7. By executing the synthesizing processing program or the matte image generation program with area information, the image synthesis apparatus generates a matte image (matte image with area information) as described later. And an image generation device. RAM (Random
The Access Memory 3 temporarily stores a program to be executed by the CPU 2 and data necessary for processing by the CPU 2.

The input device 4 (operation means) is composed of, for example, a keyboard, a mouse, a combination of a tablet and a pen, and is operated to input necessary commands and data to the CPU 2. The output device 5 is, for example, C
It is composed of an RT (Cathode Ray Tube), a liquid crystal display, or the like, and displays necessary information under the control of the CPU 2. External I / F (Interface)
6 is, for example, a modem, a TA (Terminal Adapter),
It is made up of a network adapter and controls communication with the outside.

The external storage device 7 is, for example, an HD (Hard D)
isk), CD (Compact Disc), etc., and as described above, the OS program, the synthesis processing program,
In addition to storing a matte image generation program with area information, it stores a foreground image and a background image to be subjected to image synthesis by the synthesis processing program, a matte image used for the synthesis, and the like. In addition, the external storage device 7 is also configured to store a composite image obtained by image composition by a composition processing program, a mat image with region information generated by a mat image generation program with region information, and the like.

The above-mentioned blocks constituting the data processing device are interconnected via a bus.

In the image processing apparatus configured as described above, when the power of the apparatus is turned on, the CPU 2
The IPL program stored in the ROM 1 is executed,
Thus, the OS program stored in the external storage device 7 is read, expanded on the RAM 3 and executed. Then, for example, in response to the operation of the input device 4, the CP
U2 reads out the synthesis processing program and the matte image generation program with area information stored in the external storage device 7 under the control of the OS, and develops and executes the program on the RAM 3 to thereby execute the image synthesis and the matte with area information. Generate an image.

That is, when the image synthesis is performed, the foreground image, the background image, and the mat image stored in the external storage device 7 are read out and stored in the RAM 3. Various operations and the like are performed using the foreground image, the background image, and the matte image stored in the RAM 3 to generate a composite image in which the foreground image and the background image are synthesized in accordance with the matte image. You. This composite image
It is stored in the RAM 3 or supplied to the external storage device 7 and stored. Alternatively, the composite image is supplied to the output device 5 to be displayed, or supplied to the external I / F 6, and is transmitted, for example, via the Internet or another transmission medium (not shown), so as to be transmitted to another device. Etc. are provided.

When a mat image with area information is generated, the mat image stored in the external storage device 7 is read out, supplied to the RAM 3 and stored. Then, various operations and the like are performed on the mat image to generate a mat image with region information. This mat image with area information is stored in, for example, the RAM 3
Is transferred to and stored in the external storage device 7, or
The data is supplied to the external I / F 6 and transmitted to, for example, the Internet or another transmission medium to be provided to another device or the like. In addition, when the image synthesis is performed immediately after the generation of the area information-attached mat image, the area information-attached mat image is stored in the RAM 3.

Next, FIG. 2 shows an example of a functional configuration of the data processing device of FIG. 1 when functioning as an image synthesizing device for synthesizing a foreground image and a background image according to a matte image. Note that the configuration in FIG. 2 is realized by executing a synthesis processing program in the CPU 2 in FIG.

The foreground image storage unit 11 or the background image storage unit 12 is supplied with the foreground image or the background image stored in the external storage device 7, respectively. The unit 12 is configured to store the foreground image or the background image.
Further, here, a synthesized image obtained by synthesizing the foreground image and the background image is stored so as to overwrite the background image, for example. Therefore, the background image storage unit 12
The composite image is also stored.

The mat image storage unit 13 is supplied with a mat image stored in the external storage device 7, and the mat image storage unit 13 stores the mat image. The matte image can be generated by, for example, a method using a chroma key or the like.

Area setting section 14 (setting means) (output means)
Are a first bounding box (hereinafter, appropriately referred to as a first bounding box) including all non-zero mat values with respect to the mat image stored in the mat image storage unit 13 and a first mat. A bounding box consisting only of a value (hereinafter, appropriately referred to as a second bounding box) (second region) is set. That is, for example, as shown in FIG. 3, the region setting unit 14 circumscribes a semi-transparent region (a portion with a light shadow in FIG. 3) of the mat image stored in the mat image storage unit 13. Is set as the first bounding box, and a rectangular area whose two vertices are inscribed in the completely opaque area is set as the second bounding box.
Set as the bounding box of.

FIG. 3 shows the same mat image as that of FIG. 11 described above. In FIG. 3, FIG.
In the same coordinate system as in the case of, the coordinates of the lower left or upper right vertex of the first bounding box are _expressed as (X _min1 , Y _min1 ) or (X _max1 , Y _max1 ), respectively. Also, the coordinates of the lower left or upper right vertex of the second bounding box are represented as (X _min2 , Y _min2 ) or (X _max2 , Y _max2 ), respectively. Further, it is assumed that the number of horizontal or vertical pixels of the mat image is W or H pixels, respectively. It is also assumed that the number of horizontal or vertical pixels of the foreground image and the background image is also W or H pixels, respectively. Furthermore, the mat value of the coordinates (pixel) (x, y) is represented by an array variable (two-dimensional array variable) matte [x, y], and the pixel values of the foreground image and the background image are R, G ,
It is assumed that the three color components B are represented in a continuous packed pixel format. The pixel value of the pixel (x, y) of the foreground image or the background image is represented by an array variable F [x, y] or B [x, y] as a structure, respectively. The R, G, B components of (x, y) are F
[X, y]. r, F [x, y]. g, F [x, y]. b
And the R, G, B components of the pixel (x, y) of the background image are B
[X, y]. r, B [x, y]. g, B [x, y]. b
, Respectively. Where x = 0,
, W-1 and y = 0, 1,..., H-
It is one.

After setting the first and second bounding boxes, the area setting unit 14 obtains area information (specific information) for specifying each of the bounding boxes, and supplies the area information (specific information) to the rendering unit 15 together with the matte image. I have. As described above, the area setting unit 14 outputs a matte image including the area information of the first and second bounding boxes. This is a matte image with area information. This is realized by the CPU 2 executing a mat image generation program with area information. The matte image with area information is rendered by the rendering unit 1
5, it can be supplied to and stored in the external storage device 7 as described above, or can be provided to another device via the external I / F 6.

The rendering unit 15 (processing means)
, An area outside the first bounding box (hereinafter, appropriately referred to as an external area), an area of the first bounding box that is not the second bounding box (hereinafter, appropriately referred to as an intermediate area), and The foreground image and the background image corresponding to each of the three regions of the second bounding box region (hereinafter, appropriately referred to as an internal region) are subjected to a synthesis process for image synthesis, and the resultant synthesized image is set as a background image. The image is written in the image storage unit 12 so as to overwrite the background image stored therein.

Next, referring to the flowchart of FIG.
The processing of the image combining device in FIG. 2 will be described.

First, in step S1, the foreground image, the background image, or the matte image stored in the external storage device 7 is read, and the foreground image storage unit 11, the background image storage unit 12, or the matte image storage unit is read. 13 and stored therein, and then proceeds to step S2.

In step S2, the area setting section 14 reads out the matte image stored in the matte image storage section 13, and sets the first and second bounding boxes for the matte image. That is, for example,
The area setting unit 14 supplies the mat image to the output device 5,
Display as a grayscale image. The user specifies the first and second bounding boxes by operating the input device 4 while viewing the grayscale image, and the area setting unit 14 sets the first and second bounding boxes in accordance with the specification. Set the box.

Specifically, the user specifies, for example, the lower left vertex and the lower right vertex of the first and second bounding boxes. Alternatively, for example, the lower left vertex of the first and second bounding boxes and the horizontal and vertical lengths or the diagonal length are specified. Area setting unit 1
In 4, the first and second rectangular bounding boxes are set according to the user's designation.

When the first and second bounding boxes are set, the area setting unit 14 sets, as area information for specifying the first and second bounding boxes, for example, the respective lower left vertices (X _min1 , X _min1 , Y _min1 ), (X _min2 , Y _min2 )
And the upper right vertex (X _max1 , Y _max1 ), (X _max2 ,
Y _max2 ) is obtained and output to the rendering unit 15 together with the matte image. That is, the area setting unit 14 supplies the matte image with the area information to the rendering unit 15.

Upon receiving the area information-attached mat image, the rendering unit 15 initializes a variable y representing the y coordinate to Y _min1 in step S3, and proceeds to step S4 to determine whether the variable y is smaller than Y _min2 . Is determined.
If it is determined in step S4 that the variable y is smaller than Y _min2 , that is, if the variable y is _{equal to} or more than Y _{min1 and} less than Y _min2 , the process proceeds to step S5, and the one-line calculation subroutine for performing the one-line calculation process includes the variable y and a list of three variables of x coordinates X _min1 and X _{max1 on} the left and right borders of the first bounding box are called as arguments.

When the one-line calculation process in step S5 is completed, the process proceeds to step S6, where the variable y is incremented by one. Then, returning to step S4,
Hereinafter, the processing of steps S4 to S6 is repeated until it is determined in step S4 that the variable y is not smaller than Y _min2 .

That is, a line (row) whose y coordinate is _smaller than Y _min1 which is the y coordinate of the lower left vertex of the first bounding box is composed of only pixels in the external area (FIG. 3), , Only transparent areas (areas with a mat value of 0) are included. In this case, as described above, since the composite image is generated by overwriting the background image, the pixels constituting the line of such a transparent region are not included in the background image stored in the background image storage unit 12. The pixel value can be used as it is, and no special processing is required.

A line whose y coordinate is _{equal to} or more than Y _min1 which is the y coordinate of the lower left vertex of the first bounding box and less than Y _min2 which is the y coordinate of the lower left vertex of the second bounding box is formed. X
Pixels whose coordinates are _smaller than X _min1 which is the x coordinate of the lower left vertex of the first bounding box, and pixels whose coordinate is larger than X _max1 which is the x coordinate of the upper right vertex thereof are also pixels of the external region (FIG. 3). Again, for such a pixel, the pixel value of the background image stored in the background image storage unit 12 can be used as it is, so that no special processing is required.

Further, among the pixels constituting the line whose y coordinate is not less than Y _{min1 and} less than Y _min2 , the x coordinate is X coordinate of the lower left vertex of the first bounding box.
X _max1 that is the x coordinate of the upper right vertex that is not _less than _min1
The following pixels are pixels in the intermediate area (FIG. 3), and the intermediate area includes a translucent area (a matte value larger than 0 and 1).
(Smaller area) may be included, so it is necessary to calculate the equation (1). Therefore, in step S5, such a pixel (of the pixels constituting the line whose y coordinate is not less than Y _{min1 and} less than Y _min2 , the x coordinate is X
_min1 or X _max1 less) only intended for, line calculation process is performed.

Here, the flowchart of FIG. 5 shows details of the one-line calculation process in step S5 of FIG. As described above, the one-line calculation subroutine is called with a sequence of three variables as arguments. In the embodiment of FIG. 5, the sequence of the three variables is represented by variables y, XL, and X.
It corresponds to R.

In the one-line calculation process, first, in step S21, a variable x representing the x coordinate is initialized to XL, and the flow advances to step S22 to determine whether the variable x is equal to or smaller than XR. If it is determined in step S22 that the variable x is equal to or smaller than XR, the process proceeds to step S22.
23, and in the rendering unit 15, the variable M
In the mat image stored in the mat image storage unit 13, a mat value matte [x, y] of coordinates (pixels) (x, y) is set.

Then, the process proceeds to step S24, where the rendering unit 15 determines whether or not the mat value M is 0. If it is determined that the mat value M is not 0, the process proceeds to step S25.
To determine whether the mat value M is 1.
If it is determined in step S25 that the mat value M is 1, the process proceeds to step S26, where the rendering unit 1
At 5, the pixel value B [x, y] of the pixel (x, y) of the background image stored in the background image storage unit 12 is added to the pixel (x, y) of the foreground image stored in the foreground image storage unit 11. Is copied, and the process proceeds to step S28.
That is, in step S26, the calculation shown in the above equation (2) is performed, and the process proceeds to step S28.

If it is determined in step S25 that the mat value M is not 1, the process proceeds to step S27, where the rendering unit 15 causes the foreground image storage unit 11
Using the pixel (x, y) of the foreground image stored in the background image storage unit 12 and the pixel (x, y) of the background image stored in the background image storage unit 12, the calculation represented by the above-described equation (3) is performed. Then, the process proceeds to step S28.

On the other hand, if it is determined in step S24 that the mat value M is 0, the process proceeds to step S28, where the variable x is incremented by 1, and the process proceeds to step S2.
2, the same process is repeated.

Then, in step S22, the variable x
Is determined to be not less than or equal to XR.

Referring back to FIG. 4, if it is determined in step S4 that the variable y is not smaller than Y _{min2, the process proceeds} to step S4.
7, the rendering unit 15 sets the variable y to Y
_The variable y is initialized to _min2, and the process proceeds to step S8.
It is determined whether it is less than or equal to _max2 . If it is determined in step S8 that the variable y is equal to or _smaller than _Ymax2 , that is, if the variable y is equal to or _larger than _{Ymin2 and} equal to or _smaller than _Ymax2 , the process proceeds to step S9 to perform the one-line calculation process illustrated in FIG. The one-line calculation subroutine calculates the variable y, the x coordinate X _min1 of the left boundary of the first bounding box, and the second
X coordinate of the left boundary of the bounding box of X _min1 −
Called with 1 as an argument.

When the one-line calculation process in step S 9 is completed, the process proceeds to step S 10, where the rendering unit 15 determines the pixel values F [X _min2 , y] to F [X] of the foreground image stored in the foreground image storage unit 11. _max2 , y] is converted to the pixel value B [X of the background image stored in the background image storage unit 12.
_min2 , y] to B [ _Xmax2 , y], and then proceeds to step S11.

In step S11, the one-line calculation subroutine for performing the one-line calculation process shown in FIG. 5 includes a variable y, an x coordinate X _max2 + of the right boundary of the second bounding box.
1 and called with the x coordinate X _max1 of the right boundary of the first bounding box as arguments.

When the one-line calculation process in step S11 is completed, the process proceeds to step S12, where the rendering unit 15 increments the variable y by one. Then, the process returns to step S8, and thereafter, step S8
Then, the processing of steps S8 to S12 is repeated until it is determined that the variable y is not equal to or less than _Ymax2 .

That is, a line whose y coordinate is not less than Y _min2 which is the y coordinate of the lower left vertex of the second bounding box and not more than Y _max2 which is the y coordinate of the upper right vertex of the second bounding box is formed. Among the pixels, the x coordinate is the x of the lower left vertex of the first bounding box.
_Pixels smaller than X _min1 which is coordinates and pixels larger than X _max1 which is the x coordinate of the upper right vertex of the first bounding box are pixels in the external region (FIG. 3), and only the transparent region is included in the external region. Not included. Then, as described above, since the composite image is generated by overwriting the background image, the pixel value of the background image stored in the background image storage unit 12 can be used as it is for the pixels in the transparent area. Yes, no special processing is required.

Further, among the pixels forming the line whose y coordinate is not less than Y _{min2 and not} more than Y _max2 , the x coordinate is X _min1 which is the x coordinate of the lower left vertex of the first bounding box.
Pixels smaller than X _min2, which is the x coordinate of the lower left vertex of the second bounding box, and X _{max x2,} which is the x coordinate of the upper right vertex of the second bounding box
Pixels that are larger and are _smaller than or _{equal to} X _max1 that is the x coordinate of the upper right vertex of the first bounding box are pixels in the intermediate region (FIG. 3), and the intermediate region may include a translucent region. Therefore, it is necessary to be the target of the calculation of the expression (1). Therefore, in steps S9 and S11, one-line calculation processing is performed only on such pixels.

Further, among the pixels constituting the line whose y coordinate is not less than Y _{min2 and not} more than Y _max2 , the x coordinate thereof is the X coordinate of the lower left vertex of the second bounding box.
X _max2 that is the x coordinate of the vertex at the upper right that is not less than _min2
The following pixels are pixels in an internal area (FIG. 3), and the internal area includes only a completely opaque area (an area having a mat value of 1). In this case, as described above, since the composite image is generated by overwriting the background image, the pixels of the foreground image stored in the foreground image storage unit 11 need only be copied for the pixels in the completely opaque area. . Thus, in step S10, for such a pixel, the pixel value of the foreground image stored in the foreground image storage unit 11 is copied to the background image storage unit 12.

Then, in step S8, the variable y
Is not smaller than Y _max2 , step S
Proceeding to 13, the rendering unit 15 initializes the variable y representing the y coordinate to Y _max2 +1 and proceeds to step S14.
In step S14, the rendering unit 15 determines whether the variable y is _{equal to} or _smaller than Y _max1 . In step S14, when it is determined that the variable y is _smaller than Y _max1 , that is, when the variable y is _{equal to} or _larger than Y _{max2 and smaller} than Y _max1 , the process proceeds to step S15, where one line for performing the one-line calculation process shown in FIG. The calculation subroutine uses the variables y and X
_min1, called the X _max1 as an argument.

When the one-line calculation process in step S15 is completed, the process proceeds to step S16, where the variable y is set to 1
Is only incremented. Then, the process returns to step S14, and _thereafter , the processing of steps S14 to S16 is repeated until it is determined in step S14 that the variable y is not _{equal to} or less than Y _max1 .

That is, of the pixels forming the line whose y coordinate is not less than Y _{max2 and not} more than Y _max1, the pixel whose x coordinate is _smaller than X _{min1 and} the pixel larger than X _max1 are located in the outer region (FIG. 3). It is a pixel, and for such a pixel, the pixel value of the background image stored in the background image storage unit 12 can be used as it is.

[0097] In addition, y coordinates, of the pixels constituting the Y _max2 or Y _max1 following line, the x-coordinate, X _min1 or X _max1 following pixel is the pixel of the intermediate region (Fig. 3), Since the intermediate area may include a translucent area, the intermediate area needs to be the target of the calculation of Expression (1). Therefore, in step S15, a one-line calculation process is performed on only such pixels.

Then, in step S14, the variable y
_If it is determined that is not less than or _equal to Y _{max1, the process proceeds to} step S17, where the composite image stored in the background image storage unit 12 is written to the external storage device 7, and the process ends.

That is, a line whose y coordinate is larger than Y _max1 is composed of only pixels in the outer region (FIG. 3), and the outer region includes only a transparent region. In this case, as described above, since the composite image is generated by overwriting the background image, the pixels constituting the line of such a transparent region are not included in the background image stored in the background image storage unit 12. The pixel value can be used as it is, and no special processing is required.

Therefore, when the variable y representing the y coordinate is no longer less than Y _max1 , the background image storage unit 12 stores a synthesized image obtained by synthesizing the foreground image and the background image according to the matte image. In step S17, the composite image is written in the external storage device 7 (or
Displayed on the output device 5).

As described above, for the matte image, the first
And setting a second bounding box, a region outside the first bounding box (outer region), a region of the first bounding box that is not the second bounding box (intermediate region), and a second Regarding the foreground image and the background image corresponding to each of the three regions of the bounding box (inner region),
Since processing for image synthesis is performed, image synthesis can be performed efficiently.

That is, conventionally, the pixel in the first bounding box had to be subjected to the operation of the equation (1), whereas the image processing apparatus of FIG. For example, of the pixels in the first bounding box, only the pixels in the second bounding box excluding the pixels in the second bounding box may be subjected to the calculation of the equation (1).
For the pixels in the bounding box, only copying is necessary.

For copying the pixel values of the foreground image to the background image storage unit 12 in step S10 in FIG. 4, for example, a subroutine program for performing a memory block copy is prepared in the image processing apparatus in FIG. In such cases, it can be performed by calling the subroutine. That is, in general, the memory block copy can efficiently copy (collectively copy) data in units of a plurality of bytes, and copy the pixel values of the foreground image to the background image storage unit 12 in units of one pixel. Copying can be performed more efficiently than in such a case.

Next, as described above, in the image processing apparatus of FIG. 2, it is possible to cause the area setting section 14 to output a mat image with area information to the outside.
Indicates a data format of a mat image with area information output by the area setting unit 14. Note that the foreground image stored in the foreground image storage unit 11 can be included in the matte image with region information. FIG. 6 shows the data of the matte image with region information when such a foreground image is included. The format is shown. Here, in order to include the foreground image stored in the foreground image storage unit 11 in the matte image with area information, the area setting unit 14 stores the foreground image in the foreground image storage unit 11 as shown by a dotted line in FIG. The foreground image may be read.

In FIG. 6, the first line (the first line from the top)
Represents the number of horizontal or vertical pixels of the foreground image and the matte image, respectively. Further, in the second row (X _{mi n1,} Y _min1) or (X _max1, Y _max1), the first
Represents the lower left or upper right vertex of the bounding box of the second bounding box, respectively, and (X _min2 , Y _min2 ) or (X _max2 , Y _max2 ) on the third line represents the lower left or upper right vertex of the second bounding box, respectively. Represents. In the fourth and subsequent rows, pixels (x,
y), the R, G, B components R _{x, y} , G _{x, y} , B _{x, y} and the mat values M _{x, y} corresponding to the pixels are the same as the number of pixels of the foreground image, that is, W × H are arranged.

The external storage device 7 (FIG. 1) has
4 is generated and stored, and in step S1 of FIG. 4, the foreground image storage unit 11 may store the matte image with region information of FIG. it can. In this case, step S2
_Now , the coordinates (X _min1 , Y _min1 ), (X _max1 , Y _min ) in the matte image with area information stored in the foreground image storage unit 11
_{max1), (X min2, Y} mi n2), and (X _max2,
By reading Y _max2 ), the first and second bounding boxes are set.

The matte image with area information shown in FIG.
Although the foreground image is included, the matte image with area information is
It is possible to configure without including the foreground image. In this case, in step S1 in FIG. 4, such a mat image with area information that does not include the foreground image can be stored in the mat image storage unit 13. In step S2, the mat image storage unit 13 (X _min1 , Y _min1 ) in the matte image with area information stored in
_max1 , _Ymax1 ), ( _Xmin2 , _Ymin2 ), and ( _Xmax2 , _Ymax2 ) to read the first and second
Is set.

Next, in the above case, a first bounding box containing all non-zero mat values and a second bounding box consisting of only one mat value are determined based on the vertex input from the user. However, the first and second bounding boxes can be set based on the mat values forming the mat image. Further, in the above case, since the first and second bounding boxes are rectangular, the first bounding box contains 0
May be included, and the second bounding box may not include all of the matte values of 1. However, the first and second bounding boxes may have any shapes, The first bounding box can be configured not to include a matte value of zero, and the second bounding box can be configured to include all matte values of one. That is, as shown in FIG. 7, the first bounding box is configured to coincide with the outer periphery of the translucent region, and the second bounding box is configured to coincide with the outer periphery of the completely opaque region. It is possible.

In this case, as shown in FIG. 7, for example, four x-coordinates are introduced for each row of the matte image (the horizontal and vertical are composed of W × H pixels as described above). , First and second bounding boxes. That is, a row represented by a certain y coordinate (hereinafter, appropriately,
X-coordinate of the left end of the translucent area,
The x coordinate of the left edge of the completely opaque area, the x coordinate of the right edge of the completely opaque area, and the x coordinate of the right edge of the translucent area are introduced. Here, the x coordinate of the left end of the translucent area, the x coordinate of the left end of the completely opaque area, the x coordinate of the right end of the completely opaque area, or the x coordinate of the right end of the translucent area in the y-th row are represented by X _min1 [ _{y], X min2 [y]} , X ma x2 [y],
Or, it is expressed as X _max1 [y]. In this case, 0 ≦ x <X _min1
The range of [y] and X _max1 [y] <x ≦ W is the outer region, and X _min1 [y] ≦ x <X _min2 [y] and X
_{The range of max2} [y] <x ≦ _Xmax1 [y] is the middle area, and the range of _Xmin2 [y] ≦ x ≦ _Xmax2 [y] is the inner area.

In the y-th row, all three areas, that is, the external area, the intermediate area, and the internal area do not always exist. Therefore, for example, if there is no internal area, the x coordinate X defining the internal area
_{For min2} [y] and _Xmax2 [y], the relationship that is satisfied when an internal region exists ( _Xmin2 [y] ≦ X
_max2 [y]), that is, X _min2 [y]> X
By setting a value such that _max2 [y] (for example, _Xmin2 [y] = _Xmax1 [y] + 1, _Xmax2 [y] =
X _max1 [y]) to indicate that no internal area exists.

Next, referring to the flowchart of FIG.
As described above, the processing of the image synthesizing apparatus in FIG. 2 in the case where the first and second bounding boxes are set to perform image synthesis will be described.

First, in step S31, as in step S1 of FIG. 4, the foreground image, background image, or mat image stored in the external storage device 7 is read out, and the foreground image storage unit 11, Background image storage unit 1
2 or the matte image storage unit 13 is supplied and stored, respectively, and the process proceeds to step S32.

In step S32, the area setting section 14 reads out the matte image stored in the matte image storage section 13 and applies the first and second matte images to the matte image.
Is set. That is, here, the area setting unit 14, for each row of the stored mat image mat image storage unit 13, X _{mi n1} described in FIG. 7
[Y], X _min2 [y], X _max2 [y], and X
The first and second bounding boxes are set by finding _max1 [y]. The area setting unit 14 includes:
When the second bounding box is set, X _min1 [y], X _min2 [y], X set for each row of the matte image are used as area information for specifying the second bounding box.
_{ma x2} [y], and X _max1 the [y], together with the mat image, and outputs to the rendering unit 15. That is, the area setting unit 14 outputs the matte image with the area information to the rendering unit 1
5

Here, the details of the processing in step S32 for setting the first and second bounding boxes and generating a mat image with area information will be described later.

When the rendering unit 15 receives the matte image with the area information, it initializes a variable y representing the y coordinate to 0 in step S33, and proceeds to step S34, where the variable y is set to the number of vertical pixels of the matte image. It is determined whether it is smaller than H. In step S34, the variable y
Is smaller than H, that is, when the variable y is
If it is not less than 0 and less than H, the process proceeds to step S35, and FIG.
The one-line calculation subroutine that performs the one-line calculation process shown in
The variable y, and the x coordinate X _min1 [y] of the left boundary of the first bounding box in the y th row, and the x coordinate of the left boundary of the second bounding box in the y th row
It is called with an arrangement of three variables at the coordinate X _min2 [y] -1.

When the one-line calculation process in step S35 is completed, the process proceeds to step S36, in which the rendering unit 15 determines the pixel values F [X _min2 [y], y] through the foreground image stored in the foreground image storage unit 11. F [X
_max2 [y], y] is converted from the pixel values B [X _min2 [y], y] to B [X] of the background image stored in the background image storage unit 12.
_max2 [y], y] and proceed to step S37.

In step S37, the one-line calculation subroutine for performing the one-line calculation process shown in FIG. 5 includes a variable y and an x coordinate X on the right boundary of the second bounding box.
_{It is called with max2} [y] +1 and the x coordinate _Xmax1 [y] of the right boundary of the first bounding box as arguments.

When the one-line calculation process in step S37 is completed, the process proceeds to step S38, where the rendering unit 15 increments the variable y by one. Then, the process returns to step S34, and thereafter, returns to step S34.
Until it is determined at 34 that the variable y is not less than H, the processing of steps S34 to S38 is repeated.

That is, of the pixels constituting the line of the y-th row, the pixel whose x coordinate is smaller than X _min1 [y] which is the x coordinate of the left boundary of the first bounding box, and the first bounding Pixels larger than X _max1 [y], which is the x coordinate of the right boundary of the box, are pixels in the external region (FIG. 7), and here, the external region includes only a transparent region. And, as mentioned above,
Since the composite image is generated by overwriting the background image, the pixel values of the background image stored in the background image storage unit 12 can be used as they are for the pixels in the transparent area, and there is no need to perform any particular processing.

Further, among the pixels forming the line of the y-th row, the x-coordinate is _{equal to} or more than X _min1 [y] which is the x-coordinate of the left boundary of the first bounding box, and
A pixel that is less than X _min2 [y], which is the x coordinate of the left boundary of the bounding box, and is greater than X _max2 [y], which is the x coordinate of the right boundary of the second bounding box, and the first bounding box the X _{ma x1} [y] following pixel is x-coordinate of the right boundary of a pixel of the intermediate region (7), wherein the intermediate region, to include only the translucent area, equation (1) Must be calculated. Therefore, in steps S35 and S37, one-line calculation processing is performed only on such pixels.

Further, among the pixels constituting the line of the y-th row, the x coordinate is _{equal to} or more than X _min2 which is the x coordinate of the left boundary of the second bounding box and the x coordinate of the right boundary thereof. Pixels below a certain X _max2 are pixels in the internal area (FIG. 7), and the internal area includes only a completely opaque area. In this case, as described above, since the composite image is generated by overwriting the background image, the pixels of the foreground image stored in the foreground image storage unit 11 need only be copied for the pixels in the completely opaque area. .
Therefore, in step S36, for such a pixel, the pixel value of the foreground image stored in the foreground image storage unit 11 is copied to the background image storage unit 12.

Thereafter, in step S34, the variable y
Is not less than H, the process proceeds to step S39, where the composite image stored in the background image storage unit 12 is written to the external storage device 7, and the process ends.

In the embodiment of FIG. 8 as well, only the pixels in the first bounding box excluding the pixels in the second bounding box are expressed by the formula (1).
And the pixels in the second bounding box need only be copied, so that the images can be efficiently synthesized.

Next, referring to the flowchart of FIG.
The details of the process in step S32 in FIG. 8 will be described.

In step S32 of FIG. 8, each row (line) constituting the mat image is scanned, and the above-mentioned x coordinate X is determined based on the mat value of the pixels constituting the row.
_{min1 [y], X mi n2} [y], X max2 [y], and X
_max1 [y] is set.

That is, in step S41, the variable y representing the y coordinate is initialized to 0, and the flow advances to step S42 to determine whether or not the variable y is smaller than the number H of the vertical pixels of the matte image. In step S42, when it is determined that the variable y is less than H, that is, when the variable y is 0 or more and H
If it is less than the value, the process proceeds to step S43, a variable x representing the x coordinate is initialized to 0, and the process proceeds to step S44.

In step S44, it is determined whether the variable x is less than the number W of horizontal pixels of the mat image. In step S44, when it is determined that the variable x is less than the number of horizontal pixels W of the mat image, that is, when the variable x is
If it is 0 or more and less than W, the process proceeds to step S45, and it is determined whether or not the matte value matte [x, y] (matte value of coordinates (x, y)) is equal to 0. Step S4
If it is determined in 5 that the matte value matte [x, y] is equal to 0, the process proceeds to step S46, and the x coordinate is incremented by one. Then, step S44
Returning to step S44, it is determined that the x coordinate is not smaller than W, or
Until it is determined that [x, y] is not equal to 0, the processing of steps S44 to S46 is repeated.

In step S45, mat
If it is determined that te [x, y] is not equal to 0, that is, if the matte value of the y-th row is scanned from left to right and a non-zero matte value first appears, step S
47, the x-coordinate of the pixel having the non-zero matte value is the x-coordinate of the left boundary of the first bounding box.
A variable X _min1 [y] representing coordinates is set in step S
Go to 48.

In step S48, as in step S44, it is determined whether or not the x coordinate is less than W, and if it is determined that it is less than W, the process proceeds to step S4.
Proceeding to 9, it is determined whether or not the matte value matte [x, y] is larger than 0 and smaller than 1. In step S49, the matte value matte [x, y] is
If it is determined that the value is larger than 0 and smaller than 1, the process proceeds to step S50, and the x coordinate is incremented by one. Then, the process returns to step S48, and step S48
Until it is determined that the x coordinate is not less than W, or until the matte value matte [x, y] is determined to be equal to either 0 or 1 in step S49, step S4.
8 to S50 are repeated.

If it is determined in step S49 that the matte value matte [x, y] is equal to either 0 or 1, the process proceeds to step S51, where the matte value matte [x, y] is set to 1 Is determined to be equal to In step S51, matte [x,
If y] is determined to be equal to 1, that is, if the mat value of the y-th row is scanned from left to right and a mat value of 1 appears first, the process proceeds to step S52, and 1
Is a variable X representing the x coordinate of the left boundary of the second bounding box.
_min2 [y], and the process proceeds to step S53.

In step S53, as in step S44, it is determined whether the variable x is less than the number W of horizontal pixels of the matte image. If it is determined in step S53 that the variable x is less than the number W of horizontal pixels of the matte image, the process proceeds to step S54, and it is determined whether the matte value matte [x, y] is equal to one. In step S54, the matte value matte
If [x, y] is determined to be equal to 1, step S
Proceeding to 55, the x coordinate is incremented by one. Then, the process returns to step S53. Thereafter, steps S53 to S55 are repeated until it is determined in step S53 that the x coordinate is not less than W, or it is determined in step S54 that matte [x, y] is not equal to 1. The process is repeated.

Then, in step S54, mat
If it is determined that te [x, y] is not equal to 1, that is, the mat value of the y-th row is scanned from left to right, a mat value of 1 appears, and then a mat value other than 1 If the value first appears, the process proceeds to step S56, in which the x coordinate -1 of the pixel having the non-matte value is set to a variable X representing the x coordinate of the right boundary of the second bounding box.
_max2 [y] is set, and the process proceeds to step S57.

In step S57, as in step S44, it is determined whether the variable x is less than the number W of horizontal pixels of the mat image. If it is determined in step S57 that the variable x is smaller than the number W of horizontal pixels of the mat image, the process proceeds to step S58, and it is determined whether the mat value mute [x, y] is larger than 0. In step S58, the mat value matte
If [x, y] is determined to be larger than 0, the process proceeds to step S59, and the x coordinate is incremented by 1.
Then, the process returns to step S57, and thereafter, returns to step S57.
Are determined to be not less than W, or the process of steps S57 to S59 is repeated until it is determined in step S58 that matte [x, y] is not greater than 0.

Then, in step S58, mat
If it is determined that te [x, y] is not greater than 0,
That is, the mat value of the y-th row is scanned from left to right, a mat value of 1 appears, a mat value other than 1 appears, and thereafter, a mat value not larger than 0, that is, 0
If the matte value of the first bounding box first appears, the process proceeds to step S60, and the x coordinate -1 of the pixel having the matte value of 0 is a variable X _max1 [y] representing the x coordinate of the right boundary of the first bounding box. Is set, and the process proceeds to step S61.

In step 61, as in step S44, the variable x is set to the number of horizontal pixels W of the matte image.
It is determined whether it is less than. In step S61, when it is determined that the variable x is less than the number W of horizontal pixels of the mat image, the process proceeds to step S62, where the mat value m
It is determined whether atte [x, y] is equal to zero. In step S62, the matte value matte
If [x, y] is determined to be equal to 0, step S
Proceeding to 63, the x coordinate is incremented by one. Then, the process returns to step S61, and thereafter, steps S61 to S61 are performed until it is determined in step S61 that the x coordinate is not less than W, or until it is determined in step S62 that the matte value matte [x, y] is not equal to 0. The process of S63 is repeated.

If it is determined in step S62 that the matte value matte [x, y] is not equal to 0, the process returns to step S57, and the same processing is repeated. That is, when the y-th row crosses the translucent region a plurality of times, in step S60,
X coordinate of the right boundary of the last translucent area -1
Is set to the variable X _max1 [y].

In step S61, when it is determined that the x coordinate is not less than W, that is, when the mat value of the y-th row is scanned from left to right and reaches the rightmost pixel, Proceeding to step S64, the y coordinate is incremented by one. Then, the process returns to step S42, and thereafter, the same processing is repeated.

On the other hand, if it is determined in step S44 that the x coordinate is not less than W, that is, the mat value of the y-th row is scanned from left to right to detect a non-zero mat value. Cannot be obtained (therefore, when the matte value of the y-th row is all 0), the process proceeds to step S65, and the variable X _min1 [y] representing the x coordinate of the left boundary of the first bounding box cannot be obtained. A value, for example, W is set, and the process proceeds to step S66. In step S66, an _unacceptable value, for example, W, is set in a variable X _min2 [y] representing the x coordinate of the left boundary of the second bounding box, and the process proceeds to step S67.
In step S67, the variable X _max2 [y] representing the x coordinate of the right boundary of the second bounding box is set to the variable X
_When W is set to _min2 [y], a value that cannot be obtained, for example, W-1 is set, and the process proceeds to step S68. In step S68, the variable X _max1 [y] representing the x coordinate of the right boundary of the first bounding box is a value that cannot be obtained when W is set to the variable X _min1 [y], for example, W-1. Is set, the process proceeds to step S64, and thereafter, the same processing as in the case described above is performed.

In step S48, when it is determined that the x coordinate is not less than W, that is, after scanning the mat value of the y-th row from left to right, and a non-zero mat value appears. If the non-zero mat value continues to the right end of the y-th row, the process proceeds to step S66, and the same processing as in the above-described case is performed. Further, step S
In S53, when it is determined that the x coordinate is not less than W, that is, the mat value of the y-th row is scanned from left to right, and after one mat value appears, Continues to the right end of the y-th row, the process proceeds to step S67, and thereafter, the same processing as in the above-described case is performed. When it is determined in step S57 that the x coordinate is not less than W, that is, the mat value of the y-th row is scanned from left to right, and a mat value of 1 appears.
After a non-zero matte value less than one appears, the less than one zero
If the mat value does not continue to the right end of the y-th row, the process proceeds to step S68, and the same processing as in the above-described case is performed.

On the other hand, in step S51, matt
When it is determined that e [x, y] is not equal to 1, that is, the mat value in the y-th row is scanned from left to right, and a mat value larger than 0 and smaller than 1 appears. After that, if the matte value of 1 does not appear and the matte value of 0 appears, the process proceeds to step S69, and similarly to the case of step S60, the x-coordinate of the pixel having the matte value of 0 is obtained.
1 is x on the right border of the first bounding box
The variable X _max1 [y] representing the coordinates is set in step S
Go to 70.

In step S70, as in step S44, it is determined whether the variable x is less than the number W of horizontal pixels of the matte image. If it is determined in step S70 that the variable x is less than the number W of horizontal pixels of the matte image, the process proceeds to step S71, and it is determined whether or not the matte value matte [x, y] is equal to zero. In step S71, the matte value matte
If [x, y] is determined to be equal to 0, step S
Proceeding to 72, the x coordinate is incremented by one. Then, the process returns to step S70, and thereafter, steps S70 to S70 are performed until it is determined in step S70 that the x coordinate is not less than W, or until it is determined in step S71 that the matte value matte [x, y] is not equal to 0. The process of S72 is repeated.

If it is determined in step S71 that the matte value matte [x, y] is not equal to 0, that is, if a matte value other than 0 appears, the process proceeds to step S4.
8, the same process is repeated. That is, thereby, in step S52, the y-th row is scanned from left to right, and the x coordinate of the first matte value that appears is
Set to the variable X _min2 [y].

If it is determined in step S70 that the x coordinate is not smaller than W, the flow advances to step S73 to set the variable X _min2 [y] representing the x coordinate of the left boundary of the second bounding box to the first value. For example, a variable X _max1 [y] +1 representing the x coordinate of the right boundary of the second bounding box is set, and the x coordinate of the right boundary of the second bounding box is set to a value that cannot be taken in relation to the bounding box of A variable X _max2 [y] representing a value that cannot be taken in relation to the first bounding box, for example, a variable X _max1 [y] representing an x coordinate of a right boundary thereof.
Is set. Then, the process proceeds to step S64, and thereafter, the same processing as the above case is repeated.

FIG. 10 shows a data format of a mat image with area information obtained by performing the processing described with reference to FIG. The data format of the matte image with area information including the foreground image stored in the foreground image storage unit 11 is also shown in the embodiment of FIG. 10 as in the case of FIG.

In FIG. 10, W or H on the first row (first row from the top) indicates the number of horizontal or vertical pixels of the foreground image and the matte image, respectively. Further, X _min1 [y], X _min2 [y], X in the H row from the second row
_max2 [y] or _Xmax1 [y] is the y-th of the matte image.
The left border of the first bounding box, the left border of the second bounding box, the right border of the second bounding box, the first
Respectively represents the x coordinate of the right boundary of the bounding box. Then, in the subsequent W × H rows, the R, G, and B components R _{x, y} , G _{x, y of} the pixels (x, y) constituting the foreground image, as in the fourth and subsequent rows in FIG. , B _{x, y} ,
The corresponding mat values M _{x, y} are arranged.

The matte image with area information shown in FIG. 6 and FIG. 10 is particularly useful, for example, when image synthesis using the foreground image included in the matte image is performed a plurality of times.

Further, in the present embodiment, the first and second
Is bounded by a rectangle defined by the lower left and lower right vertices, and four x's set for each line.
Although the coordinates are represented by the coordinates X _min1 [y], X _min2 [y], X _max2 [y], and X _max1 [y], the expression form of the first and second bounding boxes is not particularly limited. Rather, if a point (pixel) on the matte image is given, it can be determined whether the point is located inside or outside the first and second bounding boxes. good. That is, as the expression format of the first and second bounding boxes,
For example, a quad-tree or the like can be used.

Further, in the present embodiment, the composite image is
Although the image is generated by overwriting the background image, the composite image can be generated separately from the background image.
However, in this case, it is necessary to copy the background image as a composite image in a portion where it is not necessary to particularly perform the processing on the background image.

Further, in this embodiment, as the foreground image and the background image, those having RGB components are used. However, as the foreground image and the background image, pixel values other than RGB, such as YIQ and the like, are used. It is possible to use what is expressed.

Further, in the present embodiment, the first and second rectangular bounding boxes defined by the two vertices are set based on the input of the two vertices from the user. It is also possible for the user to input three or more vertices, and to set the first and second bounding boxes of a shape other than a rectangle defined by the three or more vertices.

In this embodiment, in FIG.
A first bounding box circumscribing the translucent area,
And a second bounding box whose two vertices are inscribed in the completely transparent area is set, but the method of setting the first and second bounding boxes is not limited to this. That is, the first bounding box does not have to circumscribe the translucent area as long as it includes the translucent area and the completely opaque area. Also, if the second bounding box has at least a part of the completely opaque area,
It does not have to be inscribed in the completely opaque area.

Further, in the present embodiment, the CPU 2
Although the image synthesis and the generation of the matte image with the area information are performed by executing the computer program, these processes can also be performed by dedicated hardware.

In the present embodiment, an application program to be executed by the CPU 2 is stored in the external storage device 7.
Although the application program is stored and provided, the application program can be provided by being transmitted via, for example, the Internet or another transmission medium.

[0154]

According to the image synthesizing apparatus according to the first aspect, the image synthesizing method according to the eighth aspect, and the providing medium according to the ninth aspect, the minimum value of the predetermined range with respect to the mat image is provided. A first region containing all non-mat values,
A second area consisting only of the maximum matte value in the predetermined range is set, and the area outside the first area, the area other than the second area of the first area, and the second area The first and second images corresponding to each of the three regions are subjected to a synthesis process for image synthesis. Therefore, it is possible to efficiently combine images.

According to the image generating apparatus of the tenth aspect, the image generating method of the fourteenth aspect, and the providing medium of the fifteenth aspect, a mat having a non-minimum value within a predetermined range with respect to a matte image A first area including all the values and a second area including only the maximum mat value in a predetermined range are set. Further, according to the providing medium of the sixteenth aspect, in the mat image, the first area including all the non-minimum mat values in the predetermined range and the first area including only the maximum mat value in the predetermined range. A mat image is provided together with the identification information for identifying the second area. Therefore, it is possible to provide a mat image that can efficiently combine images.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration example of an embodiment of an image processing apparatus to which the present invention has been applied.

FIG. 2 is a block diagram illustrating a functional configuration example of the image processing apparatus in FIG. 1;

FIG. 3 is a diagram for explaining a method of setting first and second bounding boxes.

FIG. 4 is a flowchart illustrating a process of the image processing apparatus of FIG. 2;

FIG. 5 is a flowchart for explaining the processing of step S5 in FIG. 4 in more detail;

FIG. 6 is a diagram showing a format of a mat image with area information.

FIG. 7 is a diagram for explaining another setting method of the first and second bounding boxes.

FIG. 8 is a flowchart for explaining processing of the image processing apparatus of FIG. 2;

FIG. 9 is a flowchart for explaining the processing of step S32 in FIG. 8 in more detail;

FIG. 10 is a diagram showing a format of a mat image with area information.

FIG. 11 is a diagram for explaining conventional image synthesis using a bounding box.

FIG. 12 is a flowchart for explaining conventional image composition using a bounding box.

FIG. 13 is a flowchart for explaining the process of step S84 in FIG. 12 in more detail;

[Explanation of symbols]

1 ROM, 2 CPU, 3 RAM, 4 input device (operation means), 5 output device, 6 external I / F,
7 external storage device, 11 foreground image storage unit, 12
Background image storage, 13 matte image storage, 14
Area setting unit (setting means) (output means), 15 rendering unit (processing means)

 ──────────────────────────────────────────────────の Continued on the front page F term (reference) 5B057 CA01 CA08 CB01 CB08 CC03 CE08 5B080 DA07 DA08 FA03 FA17 5C023 AA16 BA02 BA13 CA03 CA08 DA02 DA03 DA08 5C076 AA01 AA12

Claims (16)

[Claims] 1. An image synthesizing apparatus for synthesizing a first image and a second image according to a matte image composed of a matte value representing the opacity of the first image by a value in a predetermined range. A setting for setting, for the mat image, a first area including all mat values that are not the minimum value in the predetermined range and a second area including only the maximum mat value in the predetermined range. Means, a region outside the first region, a region of the first region that is not the second region, and three of the second region.
Processing means for performing synthesis processing for image synthesis on the first and second images corresponding to each of the two regions. 2. The image processing apparatus according to claim 1, wherein the processing unit is configured to perform processing on the first and second images corresponding to an area outside the first area.
The image synthesizing apparatus according to claim 1, wherein a process of copying the second image is performed. 3. The processing means according to claim 1, wherein said first and second images corresponding to an area of said first area other than said second area are converted to said matte image. 2. The image synthesizing apparatus according to claim 1, wherein the image synthesizing apparatus performs a synthesizing process according to the value. 4. The apparatus according to claim 1, wherein the processing unit performs a process of copying the first image for the first and second images corresponding to the second area. Image synthesis device. 5. The image synthesizing apparatus according to claim 4, wherein said processing means copies a group of consecutive pixels of the first image corresponding to the second area at a time. 6. An operation unit operated when designating the first and second regions, wherein the setting unit is configured to operate the first and second regions in response to an operation of the operation unit. The image synthesizing apparatus according to claim 1, wherein 7. The image synthesizing apparatus according to claim 1, wherein the setting unit sets the first and second areas based on a mat value constituting the mat image. 8. An image synthesizing method for synthesizing a first image and a second image according to a matte image composed of a matte value representing the opacity of the first image by a value in a predetermined range. A setting for setting, for the mat image, a first area including all mat values that are not the minimum value in the predetermined range and a second area including only the maximum mat value in the predetermined range. A step outside of the first region, a region of the first region that is not the second region, and 3 of the second region.
Performing a combining process for combining the first and second images corresponding to each of the two regions. 9. A process of synthesizing a first image and a second image according to a matte image composed of a matte value representing the opacity of the first image with a value in a predetermined range,
A providing medium for providing control information to be performed by an information processing apparatus, wherein a first area including all non-minimum mat values of the predetermined range is provided for the mat image, A setting step of setting a second area consisting of only the maximum mat value of the first area; an area outside the first area; an area of the first area that is not the second area; 3 of 2 area
A control step of performing a synthesis process for image synthesis on the first and second images corresponding to each of the two regions. 10. A method according to claim 1, wherein said first image and said second image are combined with each other by a value in a predetermined range.
An image generating apparatus that generates a matte image including matte values representing the opacity of the image of the image, wherein a first area including all matte values that are not the minimum value in the predetermined range with respect to the matte image, Setting means for setting a second area consisting only of the maximum mat value in the predetermined range; and output means for outputting identification information for identifying the first and second areas together with the mat image. An image generating apparatus comprising: 11. An operation unit operated when designating the first and second regions, wherein the setting unit is configured to operate the first and second regions in response to an operation of the operation unit. The image generation apparatus according to claim 10, wherein: 12. The apparatus according to claim 10, wherein the setting unit sets the first and second areas based on a mat value forming the mat image. 13. The apparatus according to claim 10, wherein said output means outputs said specific information together with said matte image and said first image. 14. The first image processing apparatus according to claim 1, wherein the first image and the second image are combined with each other by a value in a predetermined range.
An image generation method for generating a matte image including matte values representing the opacity of an image, wherein: a first area including all non-minimum matte values in the predetermined range with respect to the matte image; A setting step of setting a second area consisting only of the maximum mat value in the predetermined range; and an output step of outputting identification information for identifying the first and second areas together with the mat image. An image generation method comprising: 15. The first image processing apparatus according to claim 1, wherein the first image and the second image are combined with each other by a value in a predetermined range.
A providing medium for providing control information for causing an information processing apparatus to perform a process of generating a mat image including a mat value representing the opacity of the image, wherein the matte image has a predetermined range of A setting step of setting a first area including all non-minimum mat values and a second area including only the maximum mat value in the predetermined range; and specifying the first and second areas. An output step of outputting specific information for performing the operation together with the mat image. 16. A method for synthesizing a first image and a second image, comprising the steps of:
A providing area for providing a matte image including a matte value representing the opacity of the image of the image, wherein a first area including all non-minimum matte values in the predetermined range in the matte image; A providing medium for providing the matte image together with identification information for identifying a second area consisting only of a maximum matte value of a range.