JP4121639B2 - Object image clipping method, apparatus, system, and medium on which program is recorded - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to object image segmentation in which an object is photographed using a specific background and the object is segmented from the captured image.
[0002]
[Prior art]
Conventionally, a chroma key technique is known as a method of image processing. This chroma key technique is a technique often used in broadcasting stations and the like, and is a technique for synthesizing a plurality of videos to create a new video.
[0003]
For example, an image of an announcer or the like is taken in front of the blue background, and this is taken as image 1. On the other hand, a screen of a weather map prepared separately is referred to as video 2. Then, the blue component of the video 1 is regarded as transparent and is superimposed on the video 2 to generate a video as if an announcer is standing in front of the weather map screen. There are also services that take photos of guests and provide photos that are synthesized with various background images. Chroma key technology is also used for this.
[0004]
On the other hand, there are increasing opportunities to use three-dimensional CG (computer graphics) as electronic catalogs for various products, graphic data on the Internet, and the like. And when producing this CG original image, there exists a method of estimating the three-dimensional shape of an object from a plurality of object image (silhouette) information and performing a three-dimensional reconstruction (shape from silhouette). By this method, the labor for creating the original image can be greatly reduced, and an original image having an accurate shape can be obtained.
[0005]
In this method, it is necessary to obtain silhouette information of an object from an image including the object. The chroma key technique is one effective means for obtaining silhouette information of an object. That is, for example, if an object is photographed in front of a blue background and a part other than the blue part is regarded as the silhouette of the object, correct silhouette information can be obtained as long as the object does not contain blue.
[0006]
In this way, the silhouette of an object can be cut out by using a background plate of a color different from that of the object by the chroma key technique.
[0007]
[Problems to be solved by the invention]
However, with this method using the chroma key technique, in the case of a colorful object, 1) it is very difficult to select the color of the background board. There was a problem that it was impossible to cut out. In particular, with regard to 2), for example, when transparent processing is performed on blue, an allowable value (for example, transparency between H = 220-260 in the HSV space) must be set to some extent. In many cases, there are colors that fall within the range, and the portion is determined to be the background portion, which makes it difficult to accurately cut out the object.
[0008]
The present invention has been made in view of the above problems, and relates to an object image segmentation that can reliably segment even a colorful object.
[0009]
[Means for Solving the Problems]
The present invention is an object image cutout method for shooting an object using a specific background, and cutting out the object from the shot image, which is obtained by shooting an object placed in front of a background of a first background color. A first cutout step of cutting out a portion other than the first background color as an object area from the captured image, and shooting the object in front of a background of a second background color different from the first background color. A second cutout step of cutting out a portion other than the second background color as an object region from the image, and a logical sum step of obtaining an object image by taking the logical sum of the object regions obtained in the first and second cutout steps. It is characterized by having.
[0010]
In this way, by taking the logical sum of the object regions cut out from the image of the object in the two background colors, it is possible to cut out a normal object image even if the object is colorful.
[0011]
Further, it is preferable that the first background color and the second background color used for the background color are colors having hues that are far from each other in the color space. More specifically, for example, a color that is 60 degrees or more apart in the hue space is preferable. In this way, by setting the two background colors to hues that are sufficiently distant from each other, an object region that is erroneously recognized as a background color region in one background color is always recognized as an object region in another background color. Is done. For this reason, the object image finally cut out after the logical sum is taken is correct.
[0012]
Further, the imaging of the object in the first and second clipping steps is performed a plurality of times with different directions with respect to the object, and the corresponding object region obtained in the first clipping step and the second clipping step is performed in the logical sum step. It is preferable to obtain a plurality of object images by obtaining a logical sum of them, and to obtain a three-dimensional image of the object based on the obtained plurality of object images. By such processing, it is possible to obtain a three-dimensional image of the object using the cut object image. For example, a three-dimensional image of the object can be obtained by placing an object on which a three-dimensional image is desired on the table, rotating the object, and photographing the object. Various CG images can be created based on the three-dimensional image.
[0013]
In addition, it is preferable that a pair of object regions before the first and second background colors is sequentially obtained by two imaging units having different viewing angles, and an object image of a moving object is cut out. If the difference in the viewing angle is such that the object image is not affected, a correct object image can be cut out by the object regions obtained by the two imaging units.
[0014]
Further, the background of the first background color and the background of the second background color are switched at high speed, and a pair of object regions before the first and second background colors that can ignore the movement of the object are sequentially obtained according to the movement of the object, It is preferable to cut out an object image of a moving object.
[0015]
The present invention also relates to an apparatus for performing the above-described object cutting method and a medium storing a program for this operation.
[0016]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention (hereinafter referred to as embodiments) will be described with reference to the drawings.
[0017]
FIG. 1 is a block diagram showing the overall configuration of an image processing apparatus including an object image cutout apparatus according to the present invention. Imaging data from the image input unit 10 composed of a CCD camera or the like is supplied to and stored in the storage unit 12. The storage unit 12 can be composed of various types such as a RAM, a hard disk, and a magneto-optical disk.
[0018]
An object cutout unit 14 is connected to the storage unit 12. The object cutout unit 14 excludes a specific color portion stored in the storage unit 12 and cuts out an object region. Then, the result is stored in the storage unit 12. Here, when the object region is cut out, the image data is converted into HSV data, and the portion in the range where the color is considered to be the same as the background color is excluded. For example, in the case of blue, the portion of H = 220-260 may be recognized as the background portion.
[0019]
In particular, according to the present invention, a final object image is cut out by cutting out a region (object region) of an object image for each pair of images in a background of two kinds of colors and taking the logical sum of the cut out object regions. Then, the data about the object image as a result of the cutout is stored in the storage unit 12.
[0020]
An image processing unit 16 is also connected to the storage unit 12. The image processing unit 16 performs various processes on the image data stored in the storage unit 12. In the present embodiment, a plurality of pieces of image data with different shooting angles are obtained for one object, a three-dimensional shape is constructed based on the plurality of two-dimensional images, and this is stored in the storage unit 12. Further, the image processing unit 16 generates three-dimensional CG data based on the generated three-dimensional shape data.
[0021]
An image output unit 18 is connected to the image processing unit 16, and the image processing unit 16 displays various created images. The image output unit 18 includes a display such as a CRT or LCD. The image output unit 18 also performs display based on various image data stored in the storage unit 12.
[0022]
Here, all of the above-described configurations may be configured as one computer system, but the camera may be configured as a separate unit and connected to the computer by communication. In this case, it is preferable to incorporate a memory card in the camera and input data from the memory card to the computer.
[0023]
The operation programs of the object cutout unit 14 and the image processing unit 16 are stored in the storage unit 12, and various operations of the object cutout unit 14 and the image processing unit 16 are achieved by operating the computer according to the operation program. Is done. The operation program can be loaded from a storage medium such as a CD-ROM, or can be obtained and loaded by communication.
[0024]
Next, extraction of an object image according to the present embodiment will be described with reference to FIG.
[0025]
First, an object is imaged by placing it in front of the background color A and the background color B, and a pair of image data is obtained in the image input unit 10. The object cutout unit 14 obtains a pair of cutout object areas by excluding the same color areas as the background color A and the background color B from the pair of image data. In this example, the color of the head of the object is the same as the background color A, and the color of the left leg of the object is the same as the background color B. Therefore, the object region cut out from the background color A image has a missing head, and the object region cut out from the background color B image has a left foot missing. However, by taking the logical sum of the two, a complete object region can be obtained.
[0026]
For the background A and the background B, it is preferable to employ colors such as red and blue that are in a relationship as far as possible in the HSV space (complementary color relationship). Thus, the object portion recognized as the same as the background color A is different from the background color B and is finally cut out as an object region. In other words, if two types of background boards with sufficiently different hues are used, even if the difference between the background and the object is not recognized with one background color, the difference between the background and the object is always recognized with the other background color. . Therefore, even if the object is colorful including various colors, the object can be reliably cut out.
[0027]
Next, a processing flow in the case where such object cutout is used for the composition of a photograph will be described with reference to FIG.
[0028]
First, an image with the background color A from the image input unit 10 is input to the object cutout unit 14 via the storage unit 12 (S11). Next, an image of the background color B from the image input unit 10 is input to the object cutout unit 14 via the storage unit 12 (S12). For example, a background plate of background color A and a background plate of background color B may be prepared, and these may be exchanged to photograph the object.
[0029]
Then, the mask image C of the object region is cut out by eliminating the color region of the background color A (S13). Next, the mask image D of the object region is cut out by eliminating the background color B color region (S14). Then, the final object image is cut out by taking the logical sum of the mask image C and the image D, and the final mask image E is obtained (S15).
[0030]
Furthermore, by superimposing the final mask image and the original image prepared in advance, a composite photograph in which the object image is superimposed on the original image is obtained (S16). In addition, if the color information (RGB information) of each pixel of the obtained final mask image E is determined based on the color information of one of the original two image data (image data of the background color A and the background color B). Good. If only silhouette information is required, the step S16 is not necessary.
[0031]
Here, when obtaining three-dimensional image data, the processes of S11 to S15 are repeated a plurality of times while changing the shooting direction of the object. For example, an object may be placed on a rotary table, and after completion of S15, the rotary table may be rotated by a predetermined angle, and then the process may return to S11 to obtain an image in the next shooting direction. Note that after performing S11 and S12, the rotary table may be rotated to change the shooting direction, and the process may return to S11 to repeat image capture. Thereby, a plurality of pairs of two image data in a fixed photographing direction can be obtained, and a mask image of a plurality of objects can be obtained by this processing.
[0032]
Then, three-dimensional image data of the object is obtained from the plurality of final mask images by a known method. In this way, if 3D image data of an object is obtained, any change can be easily made, and various CG screens can be easily generated, or an electronic catalog using stereoscopic images can be easily generated. Can do.
[0033]
Here, in the present invention, it is necessary to change the background color. In the above example, the background plate was replaced with the object placed. However, there are various other methods for changing the background color.
[0034]
FIG. 4 shows an example in which the projector 30 is used for changing the background color. An object 32 is arranged in front of the camera that is the image input unit 10. A background plate (screen) 34 for displaying a projected image from the back surface is disposed behind the screen. The projector 30 projects the background color A and the background color B alternately on the background plate 34. Thereby, image data of two background colors can be obtained without performing work such as replacement of the background plate.
[0035]
Next, a process for extracting an object region from this moving image when the object moves will be described.
[0036]
"Use of lenticular"
5 and 6 are diagrams showing an example of cutting out from a moving image. In this example, a background plate 40 on which stripes of two background colors A and B are formed is prepared. Then, a lenticular 42 is disposed on the front side of the background plate 40. The lenticular 42 includes a plurality of cylindrical lenses arranged side by side, and is formed of a cylindrical lens that matches the stripe of the background plate 40. Therefore, by changing the viewing angles of the two cameras A and B as the image input unit 10 by a predetermined angle, the camera A becomes the background color A and the camera B becomes the background color B (the visible background color may be opposite). . Therefore, in the two cameras A and B, a pair of images of the objects of the background colors A and B can be obtained, and the object image can be cut out by the same processing as described above. Since two images can be obtained simultaneously, a pair of images can be obtained without any problem even if the object is moving, and the object image can be cut out from the moving image.
[0037]
Note that the image of the object is slightly different between the cameras A and B due to the difference in viewing angle, but this difference can be ignored if it is far away from the object to some extent. In addition, since the area where the image of the object due to this difference exists is known based on the position of the object and the positions of the cameras A and B, predetermined image processing may be performed on this part so as not to have an adverse effect. .
[0038]
Further, if the background plate 40 or the lenticular 42 is moved, a pair of images with different background colors can be obtained in time series using a single camera. Therefore, by moving the background plate 40 or the lenticular 42 at a sufficiently high speed with respect to the movement of the object and obtaining images of different background colors at substantially the same position, it can be applied to a moving image.
[0039]
"Use of slit barrier"
FIG. 7 shows an example using the slit barrier 50. A slit barrier 50 having slits corresponding to the stripes is arranged in front of the background plate 52 on which the stripes of the two background colors A and B are formed as in the case of using the lenticular described above. By photographing this with the cameras A and B having different positions as in the above case, only the background color A of the background plate 40 can be seen from the camera A, and only the background color B can be seen from the camera B. (The visible background color may be the opposite.)
[0040]
By doing in this way, images of different background colors can be obtained in the cameras A and B, as in the case of using the lenticular. Further, either the background plate 52 or the slit barrier 50 can be moved to make one camera.
[0041]
Here, when this slit barrier 50 is used, the slit barrier 50 itself is photographed. Therefore, it is necessary to remove the image of the slit barrier 50. For the slit barrier 50, black or the like is used.
[0042]
For the removal of the slit barrier 50, a filter process such as a max-min filter can be used for the photographed image.
[0043]
This max-min filter applies a max filter to an image once and then applies a min filter.
[0044]
The max filter is a filter that employs a maximum value (for example, a maximum value of luminance) within a predetermined range (± n, ± m) around the target pixel (i, j). When the value of the pixel to be processed is v (i, j), the value is expressed as follows.
[0045]
[Expression 1]
v (i, j) = MAX (v (k, l))
k = i−n to i + n, l = j−m to j + m
With this max filter, the black pixels are replaced with the surrounding maximum luminance, and the black stripes are removed.
[0046]
The min filter is a filter that employs a minimum value (for example, a minimum value of luminance) within a predetermined range (± n, ± m) around the target pixel (i, j). When the value of the pixel to be processed is v (i, j), the value is expressed as follows.
[0047]
[Expression 2]
v (i, j) = MIN (v (k, l))
k = i−n to i + n, l = j−m to j + m
By this max filter, the pixel once replaced with the maximum surrounding luminance is replaced with the minimum luminance around it. Thereby, the black stripe is removed, and the background color can be made uniform. Note that m and n are normally 1 to 3, but it is necessary to correspond to the thickness of the stripe.
[0048]
Also, the stripe since known in advance what will image (such as by continuous in the direction of the stripe), it is preferable to perform the image processing for removal by considering the properties of the stripes.
[0049]
Further, if the camera is focused on the object but not the background, and the black stripe is not clearly visible, the max-min filter may not be necessary. Such a case exists particularly when the distance between the background and the object is large or when a lens having a shallow focal depth is used.
[0050]
"Using a projector"
In the example using the projector shown in FIG. 4, the projector 30 can obtain a pair of object images in the two background colors A and B by switching the background color projected onto the background plate 34 at high speed. If the background color is switched sufficiently faster than the movement of the object, the object image can be cut out in the same manner as described above.
[0051]
"Mechanical"
FIG. 8 shows a mechanical background color conversion mechanism. In the example of this figure, the background plate 60 is composed of a plurality of background plate elements 62. The background plate element 62 has different colors on the front surface and the back surface. Therefore, the color of the background plate 60 can be changed by rotating the background plate element 62 and setting which surface is the camera side. All the background plate elements 62 are simultaneously rotated at a high speed, the background plate 60 is changed from the background color A to the background color B, and images before and after that are obtained, so that a pair of images of the background colors A and B for the object is obtained. Can be obtained.
[0052]
"Other"
Although an example of changing the background color corresponding to a moving image has been described above, these configurations can also be applied to a stationary object. In this case, it is not necessary to switch the background color at high speed.
[0053]
In the present invention, an object is placed in front of two different background colors. Then, the color information of the object is determined based on the image data obtained in any background. Regardless of the background, there is no problem as long as the color of the object can be accurately captured by the camera, but the color information of the object is often influenced by the background color. In this case, the relationship between the colors of various objects photographed with the background colors A and B and the color when the background is an achromatic color such as gray is examined in advance, and the color information of the objects obtained with the two background colors. From this, it is preferable to calculate the color information of the desired object.
[0054]
【The invention's effect】
As described above, according to the present invention, normal object image segmentation is possible even if the object is colorful by taking the logical sum of the object regions (chroma key object segment segmentation) segmented from the object image in two background colors. It can be performed.
[Brief description of the drawings]
FIG. 1 is a block diagram showing an overall configuration of an image processing apparatus including an object image cutout apparatus according to the present invention.
FIG. 2 is a diagram for explaining object region segmentation by logical sum;
FIG. 3 is a flowchart illustrating processing.
FIG. 4 is a diagram showing an example using a projector.
FIG. 5 is a diagram showing an example using a lenticular.
FIG. 6 is a plan view showing an example using a lenticular.
FIG. 7 is a diagram showing an example using a slit barrier.
FIG. 8 is a diagram illustrating an example using a mechanical background plate.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 10 Image input part, 12 Memory | storage part, 14 Object cutout part, 16 Image processing part, 18 Image output part, 30 Projector, 34,40,52,60 Background board, 42 Lenticular, 50 Slit barrier, 62 Background board element.

Claims (3)

A background plate disposed behind the object and formed with stripes of a first background color and a second background color of different colors;
A lenticular disposed on the object side of the background plate;
First imaging means for imaging the first background color and the object separated by the lenticular;
Second imaging means for imaging the second background color and the object separated by the lenticular;
First cutout means for cutting out a part other than the first background color as an object region from an image picked up by the first image pickup means;
Second cutout means for cutting out a part other than the second background color as an object region from the image picked up by the second image pickup means;
An object image clipping device comprising: a logical sum means for obtaining an object image by taking a logical sum of the object regions obtained by the first and second clipping means. A background plate disposed behind the object and formed with stripes of a first background color and a second background color of different colors;
A slit barrier disposed on the object side of the background plate;
First imaging means for imaging the first background color and the object separated by the slit barrier;
Second imaging means for imaging the second background color and the object separated by the slit barrier;
First cutout means for cutting out a part other than the first background color as an object region from an image picked up by the first image pickup means;
Second cutout means for cutting out a part other than the second background color as an object region from the image picked up by the second image pickup means;
An object image clipping device comprising: a logical sum means for obtaining an object image by taking a logical sum of the object regions obtained by the first and second clipping means. A background plate disposed behind the object and formed with stripes of a first background color and a second background color of different colors;
A lenticular disposed on the object side of the background plate;
Means for moving either the background plate or the lenticular;
The first background color and the object separated by the lenticular are imaged to obtain a first image, and then the second background color and the object separated by the lenticular are imaged to obtain a second image. Imaging means to obtain;
First cutout means for cutting out a portion other than the first background color from the first image as an object region;
Second cutout means for cutting out a portion other than the second background color from the second image as an object region;
An object image clipping device comprising: a logical sum means for obtaining an object image by taking a logical sum of the object regions obtained by the first and second clipping means.

Images