JP4148240B2 - Image processing method, image processing apparatus, program, and recording medium - Google Patents

Description

  The present invention relates to a technique used to generate 3D CG data from an actual object, and more particularly to an image processing technique for separating a subject and a background from a photographed image obtained by photographing the subject.

  In recent years, with the spread of the Internet and the improvement of image processing capability, 3D model data has been used for presentation of products and the like. For example, by using the three-dimensional model data for product introduction, the customer can see the product from a desired angle and direction on the screen of a personal computer or the like without actually having the product in front of him.

  In order to represent a subject such as a product as 3D CG data, a geometry-based 3D representation method in which the surface of the product is represented by a plurality of polygons is generally used. However, this method requires a huge amount of polingons to faithfully express the details of the subject, and its data generation cost and data amount are enormous.

  Therefore, there is an image-based expression method in which a subject is photographed in advance from all directions, and a photograph taken from the direction in which the user wants to view the subject is presented and viewed three-dimensionally. QuickTimeVR is a typical application that uses photographs, so that the subject can be displayed as it is. However, in order to construct a virtual space by using data created by this method and composing it on another background image, it is necessary to accurately extract the background from the captured image.

One common technique for cutting out the background is the chroma key method. This method is a method in which the background is photographed with a predetermined single color and the same color as the background is cut out. However, with this method, there are problems that color unevenness occurs in the background due to the shooting environment or the like, and when a color close to the background color is used for the subject, it cannot be cut out well. In order to solve these problems, it is necessary to add an algorithm for searching the boundary with the subject.However, if the processing time is long and the number of subjects to be photographed is large, the processing takes a lot of time. There is a problem that there is. There are Patent Documents 1 to 3 as prior art documents related to the present application.
JP 2004-46776 A NR Pal and SK Pal, "A review on image segmentation techniques," Pattern Recognition, vol. 26, no. 9, pp. 1277-1294, 1993. Y. Deng, BS Manjunath, and H. Shin, "Color image segmentation," Proc. Of IEEE Conf. On Computer Vision and Pattern Recognition, vol. 2, pp. 446-451, 1999.

  As described above, when the number of subjects to be photographed is large, the number of images to be subjected to background removal increases, and the conventional segmentation technique has a problem that the processing takes an enormous amount of time.

  The present invention has been made in view of the above points, and an object of the present invention is to provide an image processing technique capable of performing background removal in a short time even when the number of images subjected to background removal is large.

  The above-described problem is an image processing method executed by an image processing apparatus that performs processing for removing an area other than the subject O from an image obtained by photographing the subject O, and two images obtained by photographing the subject O from two orthogonal directions. An approximate shape S circumscribing the subject O from the initial image input step for inputting the first photographed image Po1 and the second photographed image Po2, which are images, and the first photographed image Po1 and the second photographed image Po2. An approximate calculation step for obtaining the image, an image input step for inputting a third photographed image Pn obtained by photographing the subject O from a different direction from the first photographed image Po1 and the second photographed image Po2, and a third photographed image. A processing area calculation step for obtaining a processing area A that is an area including the subject O on the third photographed image Pn from the Pn and the schematic shape S, and a subject from the processing area A in the third photographed image Pn. It is solved by an image processing method characterized by having a background removal step of removing a region other than.

  In the rough calculation step, the first background-removed image Ps1 and the second background-removed image Ps2 are images obtained by removing portions other than the subject O from the first captured image Po1 and the second captured image Po2, respectively. An initial background removing step to be obtained; an area calculating step to obtain a first rectangular area S1 and a second rectangular area S2 circumscribing the subject O in the first background removed image Ps1 and the second background removed image Ps2; An approximate shape generating step for obtaining the approximate shape S from the first rectangular area S1 and the second rectangular area S2 obtained in the area calculating step can be provided.

  In the image processing method, instead of the rough calculation step, the first photographed image Po1 and the second photographed image Po2 are displayed, and subject areas on those images designated by the user are input and input. A rough calculation step for obtaining the rough shape S from the subject area thus obtained may be provided.

  Further, in place of the initial image input step, there is a step of inputting three or more photographed images obtained by photographing the subject O from different directions. In the approximate calculation step, the outline is based on the three or more photographed images. The shape S may be obtained.

  In addition, according to the present invention, there are provided an image processing apparatus suitable for carrying out the image processing method, a program for causing a computer to realize the functions of the image processing apparatus, and a computer-readable recording medium storing the program. .

  According to the present invention, by using the approximate shape S, the processing area A that is the area including the subject O is obtained on an arbitrary captured image Pn, and the background removal that removes the area other than the subject O only in the processing area A is performed. Since the process is executed, it is not necessary to perform the background removal process on the entire image, and the background removal process time per image can be shortened. Therefore, even when the number of images to be subjected to background removal is large, background removal can be performed in a short time.

  Embodiments of the present invention will be described below with reference to the drawings.

(System configuration)
FIG. 1 shows a system configuration example in the present embodiment. As shown in FIG. 1, this system includes a camera C that captures a subject and an image processing apparatus 10 that performs a process of removing a background other than the subject O from an image captured by the camera C.

  The image processing apparatus 10 also uses a rough calculation unit 11 that calculates a rough shape S of the subject O, a processing region calculation unit 12 that calculates a processing region A corresponding to the subject O using the rough shape S, and a processing region A. A background removal processing means 13 for removing the background of a desired photographed image of the subject O, and a storage means 14 for storing data of the approximate shape S and the like. The rough calculation unit 11 includes an initial background removal processing unit 111, a minimum area calculation unit 112, and a coordinate conversion unit 113.

  The image processing apparatus 10 can be realized by mounting a program for executing the processing described in the present embodiment on a general computer including a CPU, a storage device, and the like. Further, the program can be distributed by being stored in a recording medium such as a memory, or can be distributed via a network.

(Outline of processing)
Next, the outline of the processing operation of the image processing apparatus 10 will be described with reference to the flowchart shown in FIG.

  First, two images Po1 and Po2 obtained by photographing the subject O from the front and directly above are input to the image processing device 10, and the image processing device 10 obtains the approximate shape S of the subject using these images. The data of the general shape S is held (step 1: general calculation step).

  Then, an image Pn obtained by photographing the subject O from a desired direction is input to the image processing apparatus 10, and the image processing apparatus 10 perspectively projects the schematic shape S on the image Pn, and is an area where the schematic shape S is imprinted. A processing area A of a certain subject is obtained, and data of the processing area A is held (step 2: processing area calculation step).

  Subsequently, all portions that are not the processing area A from the image Pn are regarded as the background, the background is removed only in the processing area using a background removal algorithm such as a chroma key method, and the subject area is extracted (step 3: background removal processing step). .

  When extracting a subject area from a plurality of images photographed from a plurality of directions, the processes in steps 2 and 3 are executed for each image photographed from a plurality of directions using the schematic shape S obtained in step 1. That's fine.

  In the above method, once the approximate shape is obtained, the area where the subject exists in the image taken from an arbitrary direction can be narrowed down, so that it is not necessary to perform processing for background removal on the entire screen. The background removal processing time per image can be shortened. Therefore, it is possible to solve the conventional problem that enormous processing time is required when the number of images for background removal is large.

(Details of processing)
Next, the process of step 1 (rough calculation step) will be described in more detail with reference to FIG. Here, two cameras are used, the camera in front of the subject O is C1, and the camera directly above is C2. Further, the distance between the subject O and the camera C1 is d1, the distance between the subject O and the camera C2 is d2, an image photographed by the camera C1 is Po1, and an image photographed by the camera C2 is Po2. Note that the images Po1 and Po2 may be acquired by using one camera and changing the orientation of the subject.

  Then, background-removed images Ps1 and Ps2 are obtained by removing portions other than the subject from these images (step 11). Examples of the background removal method include a chroma key method, a method of removing a background from a plurality of images taken by changing the background (Patent Document 1, etc.), and a method of obtaining a subject boundary using information such as color ( Patent Documents 2, 3, etc.) can be used. In addition, when using the several image image | photographed by changing a background, the several image image | photographed by changing a background is used about each of background removal image Ps1 and Ps2.

  Next, a minimum area calculation process for obtaining the minimum rectangular areas S1 and S2 surrounding the subject portion of the photographed image from each of the background removed images Ps1 and Ps2 is performed (step 12). For example, when obtaining the upper side of the rectangle, it is checked whether there is a subject on one line at the top of the image. If there is no subject, the line is lowered by one pixel to check for the presence of the subject, and the line is lowered until a subject is found. By performing the same processing on the right side, the left side, and the lower side, the minimum area surrounding the subject can be obtained. The four vertices of the rectangular area S1 are V11 to 14, and the four vertices of the rectangular area S2 are V21 to 24.

  Subsequently, coordinate conversion processing is performed (step 13). First, the pixel coordinates (coordinates on the screen) of the four vertices V11 to V21 and V21 to 24 of the rectangular areas S1 and S2 are obtained from the rectangular areas S1 and S2 of the background removed images Ps1 and Ps2. The pixel coordinates of V11-14 are Vp11 = (xp11, yp11), Vp12 = (xp12, yp11), Vp13 = (xp12, yp12), Vp14 = (xp11, yp12), and the pixel coordinates of V21-24 are Vp21 = ( xp21, yp21), Vp22 = (xp22, yp21), Vp23 = (xp22, yp22), Vp24 = (xp21, yp22).

  Next, the four vertices V11 to V14 of the rectangular area S1 are converted into coordinate values in the camera coordinate system of the camera C1. At this time, it is assumed that the four vertices V11 to V14 of the rectangular area S1 are projected on the screen at a position d1 away from the camera C1 (position where the subject exists) (rectangles Vo11 to 14 in the schematic shape S in FIG. 3). reference).

  The coordinate values in the camera coordinate system for vertices V11 to V14 are Vc11 = (xc11, yc11, -d1), Vc12 = (xc12, yc11, -d1), Vc13 = (xc12, yc12, -d1), Vc14 = (xc11 , yc12, -d1).

  Further, when the object O is converted into an object coordinate system in which the center of the subject O is the origin, the direction from the object to the camera C1 is the Z axis, the X axis of the camera C1 is the X axis, and the Y axis of the camera C1 is the Y axis, The coordinates of the four vertices V11 to 14 are Vo11 = (xc11, yc11, 0), Vo12 = (xc12, yc11, 0), Vo13 = (xc12, yc12, 0), Vo14 = (xc11, yc12, 0) .

  Similarly to the rectangular area S1, when the four vertices V21 to 24 of the rectangular area S2 are converted into the coordinate values of the camera coordinate system of the camera C2, Vc21 = (xc21, yc21, -d2), Vc22 = (xc22, yc21, -d2 ), Vc23 = (xc22, yc22, -d2), Vc24 = (xc21, yc22, -d2). When these coordinates are converted to the above object coordinate system, Vo21 = (xc21, 0, -yc21), Vo22 = (xc22, 0, -yc21), Vo23 = (xc22, 0, -yc22), Vo24 = (xc21, 0, -yc22). Thus, as the coordinates Vs1 to Vs8 of the eight vertices of the object coordinate system of the general shape S shown in FIG. 3, Vs1 = (xc11, yc11, -yc21), Vs2 = (xc12, yc11, -yc21), Vs3 = (xc12 , yc12, -yc21), Vs4 = (xc11, yc12, -yc21), Vs5 = (xc11, yc11, -yc22), Vs6 = (xc12, yc11, -yc22), Vs7 = (xc12, yc12, -yc22) , Vs8 = (xc11, yc12, -yc22).

  The photographed images Po1 and Po2 of the subject O are presented to the operator, and the rectangular areas S1 and S2 are determined by manually specifying the four sides of the rectangular areas S1 and S2 through the graphical user interface, and the approximate shape S You may ask for. In this case, the image processing apparatus 10 includes means for presenting the captured images Po1 and Po2 of the subject O to the operator, a graphical user interface that receives input from the operator, and the vertices of the rectangular areas S1 and S2 specified by the operator. Means for calculating the approximate shape S from

  In addition, the shooting direction of the two images for obtaining the approximate shape S is not only a combination of the shooting direction from the front and directly above, but also an image shot from two other orthogonal shooting directions such as the front and right. The approximate shape S may be obtained.

  Further, by using images taken from other directions as well as two, a more accurate schematic shape S can be obtained, and the calculation area can be reduced. Further, when the approximate shape S is obtained using three or more photographed images, the photographing directions are not necessarily orthogonal to each other.

  Next, with reference to FIG. 4, the process of step 2 (processing area calculation step) of FIG. 2 will be described.

  First, the general shape S is projected onto the image Pn (step 21). That is, an image in which the image of the approximate shape S and the image Pn obtained when the approximate shape S at the position surrounding the subject O as shown in FIG. 3 is taken from the same direction as when the image Pn is taken is superimposed. Get. Subsequently, a roughly shaped portion (a portion surrounded by an outer frame) is extracted as the processing region A (step 22).

  In step 3 (background removal processing step) in FIG. 2, the background removal processing is performed by a chroma key method, a method of removing the background from a plurality of images taken by changing the background, or the like. For example, when the background removal method described in Patent Document 1 is used, after obtaining the approximate shape S, a processing area is obtained for each of a plurality of images, and processing for obtaining a transparency image or the like is performed within the processing area. By using the method of the present embodiment, it is only necessary to perform processing for obtaining a transparency image or the like only in the processing region, not in the entire image, so that the number of pixels to be processed is greatly reduced and the processing speed is improved.

  By using the method described in the present embodiment, the background can be quickly acquired even when the number of captured subjects is large as in the case of creating 3D CG data for capturing a subject from any direction and displaying it in a virtual space. Removal can be performed.

It is a system configuration figure in an embodiment of the invention. 3 is a flowchart illustrating an outline of processing operation of the image processing apparatus according to the embodiment of the present invention. It is a figure for demonstrating the process of a rough calculation step. It is a figure for demonstrating the process of a process area | region calculation step.

Explanation of symbols

C Camera O Subject 10 Image processing apparatus 11 Approximate calculation means 12 Processing area calculation means 13 Background removal processing means 14 Storage means

Claims (10)

An image processing method executed by an image processing apparatus that performs processing for removing a region other than the subject O from an image obtained by photographing the subject O,
An initial image input step of inputting a first photographed image Po1 and a second photographed image Po2 which are two photographed images photographed from two directions orthogonal to the subject O;
A rough calculation step for obtaining a rough shape S circumscribing the subject O from the first picked-up image Po1 and the second picked-up image Po2;
An image input step of inputting a third photographed image Pn obtained by photographing the subject O from a different direction from the first photographed image Po1 and the second photographed image Po2.
A processing area calculation step for obtaining a processing area A that is an area including the subject O on the third captured image Pn from the third captured image Pn and the schematic shape S;
A background removal step of removing a region other than the subject O from the processing region A in the third photographed image Pn. The rough calculation step includes:
An initial background removal step for obtaining a first background removal image Ps1 and a second background removal image Ps2 that are images obtained by removing portions other than the subject O from the first photographed image Po1 and the second photographed image Po2, respectively;
An area calculation step for obtaining a first rectangular area S1 and a second rectangular area S2 circumscribing the subject O in the first background removed image Ps1 and the second background removed image Ps2,
The image processing method according to claim 1, further comprising: a rough shape generation step of obtaining a rough shape S from the first rectangular region S1 and the second rectangular region S2 obtained in the region calculation step. Instead of the rough calculation step,
A first calculation image Po1 and a second shooting image Po2 are displayed, subject areas on those images designated by the user are input, and an approximate calculation step for obtaining the approximate shape S from the input subject area is included. The image processing method according to claim 1.   In place of the initial image input step, there is a step of inputting three or more photographed images obtained by photographing the subject O from different directions, and in the approximate calculation step, an approximate shape S based on the three or more photographed images. The image processing method according to any one of claims 1 to 3, wherein: An image processing apparatus that performs processing for removing a region other than the subject O from an image obtained by photographing the subject O,
Initial image input means for inputting a first photographed image Po1 and a second photographed image Po2 which are two photographed images photographed from two directions orthogonal to the subject O;
Rough calculation means for obtaining a rough shape S circumscribing the subject O from the first shot image Po1 and the second shot image Po2,
Image input means for inputting a third photographed image Pn obtained by photographing the subject O from a different direction from the first photographed image Po1 and the second photographed image Po2.
Processing area calculation means for obtaining a processing area A that is an area including the subject O on the third captured image Pn from the third captured image Pn and the schematic shape S;
An image processing apparatus comprising: a background removing unit that removes an area other than the subject O from the processing area A in the third photographed image Pn. The rough calculation means includes
Initial background removal means for obtaining a first background removal image Ps1 and a second background removal image Ps2, which are images obtained by removing portions other than the subject O from the first photographed image Po1 and the second photographed image Po2, respectively;
Area calculation means for obtaining the first rectangular area S1 and the second rectangular area S2 circumscribing the subject O in the first background removed image Ps1 and the second background removed image Ps2,
The image processing apparatus according to claim 5, further comprising: a rough shape generation unit that calculates a rough shape S from the first rectangular region S1 and the second rectangular region S2 obtained in the region calculation step. Instead of the rough calculation means,
The first photographed image Po1 and the second photographed image Po2 are displayed, subject areas on those images designated by the user are input, and approximate calculation means for obtaining the approximate shape S from the input subject areas is provided. The image processing apparatus according to claim 5.   In place of the initial image input means, there is provided means for inputting three or more photographed images obtained by photographing the subject O from different directions, and in the approximate calculation means, an approximate shape S based on the three or more photographed images. The image processing device according to claim 5, wherein the image processing device is calculated. A program for causing a computer to realize the function of an image processing apparatus that performs processing for removing a region other than the subject O from an image obtained by photographing the subject O.
An initial image input means for inputting a first photographed image Po1 and a second photographed image Po2 which are two photographed images photographed from two directions orthogonal to the subject O;
Rough calculation means for obtaining a rough shape S circumscribing the subject O from the first picked-up image Po1 and the second picked-up image Po2;
Image input means for inputting a third photographed image Pn obtained by photographing the subject O from a different direction from the first photographed image Po1 and the second photographed image Po2.
A processing region calculation means for obtaining a processing region A that is a region including the subject O on the third captured image Pn from the third captured image Pn and the schematic shape S;
Background removal means for removing a region other than the subject O from the processing region A in the third photographed image Pn;
Program to function as.   A computer-readable recording medium on which the program according to claim 9 is recorded.