JP6736422B2 - Image processing apparatus, image processing method and program - Google Patents

Description

The present invention relates to a technique for estimating a three-dimensional shape of a subject at high speed and with high accuracy.

BACKGROUND ART Conventionally, a visual volume intersection method has been known as a method for rapidly estimating a three-dimensional outline shape of a subject using images having parallax (parallax images) captured by a plurality of cameras from different viewpoints. There is. Since the visual volume intersection method uses only the contour information of the subject, it is possible to obtain a stable shape, but it is not possible to restore an originally non-subject region (false region) or to restore a concave subject surface. Problem. Various methods have been proposed for the purpose of overcoming this fundamental problem. For example, Non-Patent Document 1 proposes a method of visually determining whether or not a voxel is seen by each camera, and deleting voxels whose colors are not consistent from the color information of the camera determined based on the determination result. ing. Further, in Patent Document 1, for voxels existing on the surface of the shape estimated by the visual volume intersection method, consistency of color information, smoothness of the surface, and the like are expressed as an energy function to optimize energy. , A method for improving the precision of the shape has been proposed. The voxel means a unit cell divided in a three-dimensional space of x axis, y axis, and z axis.

JP2012-208759A

Kutulakos et al. "A Theory of Shape by Space Carving" International Journal of Computer Vision 2000

However, in the method described in Non-Patent Document 1, voxels are deleted based on the color consistency of the voxels alone, so there is a possibility that the voxels forming the original subject may be deleted by mistake. Further, the method described in Patent Document 1 has a problem that the amount of calculation becomes enormous because energy optimization is performed.

An image processing apparatus according to the present invention is based on a plurality of images acquired by imaging of a plurality of imaging devices, a generating unit that generates a candidate region corresponding to a three-dimensional shape of a subject and is composed of voxels, and the candidate. for a particular voxel constituting the region includes a processing means for performing processing including deletion, wherein the processing means is based on the density of the other voxels around the particular voxel, the particular voxel to determine whether to remove for, characterized in that.

An image processing apparatus according to the present invention is an image processing apparatus that estimates a three-dimensional shape of a subject from parallax images captured from a plurality of viewpoints, and an acquisition unit that acquires a silhouette image of the subject in the parallax image, Based on the silhouette image, a generation unit that generates a candidate region of the three-dimensional shape of the subject in the three-dimensional space to be processed, and the candidate region generated in the three-dimensional space are determined to be unnecessary according to a predetermined criterion. And a restoring means for restoring the voxels deleted by the deleting means to the position of the target grid point based on the peripheral voxel density of the target grid point in the candidate area. Is characterized by.

According to the present invention, the three-dimensional shape of a subject can be estimated at high speed, with high accuracy, and with a small amount of calculation.

It is the figure which showed an example of the camera arrangement for acquiring a parallax image. FIG. 3 is a diagram illustrating an example of a hardware configuration of an image processing apparatus according to the first embodiment. 3 is a functional block diagram of the image processing apparatus according to the first embodiment. FIG. 6 is a flowchart showing a flow of a series of image processing executed by the image processing apparatus according to the first embodiment. It is a figure which shows a mode that a to-be-photographed object is image|photographed from three cameras. (A) is an example of a parallax image, (b) is an example of a silhouette image. (A) shows each image image|photographed by three cameras, (b) is a figure which shows a mode that an arbitrary voxel is projected on each camera. It is a figure explaining the concept of voxel restoration processing. 6 is a flowchart showing details of voxel restoration processing according to the first embodiment. FIG. 6 is a diagram illustrating an effect of the invention according to the first embodiment. 6 is a functional block diagram of an image processing apparatus according to a second embodiment. FIG. 9 is a flowchart showing a flow of a series of image processing executed by the image processing apparatus according to the second embodiment.

[Example 1]
In the present embodiment, when the peripheral voxel density of the grid point of interest in the three-dimensional space including the subject is equal to or higher than a certain level, the voxel is added to the position of the grid point of interest in the candidate region of the subject shape to delete it. A method of restoring an excessively voxel will be described.

FIG. 1 is a diagram showing an example of a camera arrangement for acquiring images having parallax with each other (hereinafter, parallax images) captured from different viewpoints. FIG. 1 shows a situation in which ten cameras 101 arranged so as to surround a soccer field 104 photograph a player 102 and a ball 103 on the field 104. The image data captured by each camera 101 is sent to the image processing apparatus 200 as multi-viewpoint parallax image data and subjected to predetermined image processing. Here, a sports scene will be described as an example, but the method described in the present embodiment will be described with respect to a scene in which a plurality of cameras are arranged so as to surround the object to be a subject and the shape of the object is estimated. , Widely applicable.

In the following, using the voxels described above, the target three-dimensional space is divided into regular grids of size d[mm] and expressed discretely. The coordinates of each voxel are expressed using a lattice vector such as (0,0,0), (1,0,0), (3,0,1)... in the order of x,y,z coordinates. I shall. The actual physical coordinate value is obtained by multiplying the above integer vector by the size d of the regular lattice. As a specific value d, a value such as 5 mm is adopted.

(Structure of image processing device)
First, the configuration of the image processing apparatus 200 in this embodiment will be described. FIG. 2 is a diagram showing an example of the hardware configuration of the image processing apparatus 200. The image processing apparatus 200 includes a CPU 201, a RAM 202, a ROM 203, an HDD 204, an input I/F 205, and an output I/F 206. Then, the respective units constituting the image processing apparatus 200 are mutually connected by the system bus 207. The image processing apparatus 200 is also connected to the camera 101, the operation unit 210, and the external memory 211 via the input I/F 205. Further, it is connected to the external memory 211 and the display device 212 via the output I/F 206.

The CPU 201 uses the RAM 202 as a work memory to execute a program stored in the ROM 203, and centrally controls each unit of the image processing apparatus 200 via the system bus 207. As a result, various processes described later are realized. The HDD 204 is a large-capacity storage device that stores various data handled by the image processing apparatus 200, and may be, for example, an SSD or the like. The CPU 201 can write data to the HDD 204 and read data stored in the HDD 204 via the system bus 207.

The input I/F 205 is, for example, a serial bus I/F such as USB or IEEE1394, and input of data, commands, and the like from the external device to the image processing apparatus 200 is performed via the input I/F 205. The image processing apparatus 200 acquires data from the external memory 208 (for example, a storage medium such as a hard disk, a memory card, a CF card, an SD card, a USB memory) via the input I/F 205. Further, the image processing apparatus 200 acquires a user command input using the operation unit 210 via the input I/F 205. The operation unit 210 is an input device such as a mouse and a keyboard, and is used to input a user instruction to the processing device 200.

The output I/F 206 includes a serial bus I/F such as USB or IEEE1394 similar to the input I/F 205. In addition, a video output terminal such as DVI or HDMI (registered trademark) can be used. Output of data and the like from the image processing apparatus 200 to an external device is performed via this output I/F 206. The image processing apparatus 200 displays the image by outputting the processed image data and the like to the display device 209 (various image display devices such as a liquid crystal display) via the output I/F 206. It should be noted that although the constituent elements of the image processing apparatus 200 exist in addition to the above, they are not the main subject of the present invention, and thus the description thereof is omitted.

Next, a series of image processing performed by the image processing apparatus 200 will be described. FIG. 3 is a functional block diagram of the image processing apparatus 200 according to this embodiment. The image processing apparatus 200 includes an image data acquisition unit 301, a distortion correction unit 302, a candidate region generation unit 303, a voxel deletion unit 304, a voxel restoration unit 305, a camera parameter acquisition unit 306, and a threshold value setting unit 307. The CPU 201 reads the control program stored in the ROM 203, expands it in the RAM 202, and executes it to realize the functions of the above-mentioned units. Then, FIG. 4 is a flowchart showing a flow of a series of image processing executed by the image processing apparatus 200 of the present embodiment. The image processing apparatus 200 may be configured to include a dedicated processing circuit corresponding to each of the above units. The flow of image processing performed by the image processing apparatus 200 according to this embodiment will be described below.

In step 401, the image data acquisition unit 301 acquires the above-described parallax image and silhouette image data directly from a plurality of cameras via the input I/F 205 or from the HDD 204 or the external memory 211. Here, the silhouette image is a binary image in which an area where a subject is present is white (pixel value=255) and an area where no subject is present is black (pixel value=0) in each image forming the parallax image. .. FIG. 5 is a diagram showing a state in which a perfect sphere 510 as a subject and a sphere 520 with a part missing are photographed from three cameras 501 to 503. Images 601 to 603 in FIG. 6A are parallax images obtained by photographing the two spheres 510 and 520 with the three cameras 501 to 503, and FIG. 6B is a silhouette image thereof. 611 to 613 are shown respectively. It is assumed that the silhouette image is generated in advance based on the parallax image by using a method such as background extraction or subject cutout.

In step 402, the camera parameter acquisition unit 306 acquires predetermined parameters regarding the camera that has captured the parallax image, such as internal parameters, external parameters, and distortion parameters. Here, the internal parameter is information on the coordinate value of the image center and the focal length of the lens included in the camera. The external parameter is information indicating the position and orientation of each camera. Here, a method of describing the direction and position of each camera by a position vector and rotation matrix in the world coordinate system is adopted, but it may be described by another method. The distortion parameter is information indicating the distortion of the lens included in the camera. These camera parameters may be estimated by, for example, SFM (Structure from Motion) based on the parallax image data, or may be calculated by performing calibration using a chart or the like in advance.

In step 403, the distortion correction unit 302 corrects the distortion caused by the lens distortion in the parallax image and the silhouette image acquired in step 401 based on the distortion parameter of the camera parameters acquired in step 402 (distortion). Correction processing).

In step 404, the candidate area generation unit 303 generates a candidate shape area of the subject shape in the three-dimensional space based on the camera parameters acquired in step 402 and the silhouette image data acquired in step 401. Here, the subject shape candidate region represents a region of a convex polyhedron that is a candidate for the three-dimensional shape of the subject, in which the subject is expected to exist inside. Then, this candidate area is referred to in the voxel addition processing described later. In this embodiment, a candidate area is generated from the silhouette image data based on a method called a visual volume intersection method. In FIG. 5 described above, candidate regions generated from the silhouette images 611 to 613 shown in FIG. 6B are shown. In FIG. 5, fan-shaped regions 504 to 509 extending from the cameras 501 to 503 by two are cones whose vertical cross section is the silhouette (here, a circle) of the subject as viewed from above. Regions 504 to 509 are projected in space, and common regions (regions indicated by thick lines) 511, 512, 521, and 522 of polygons in which the cones overlap each other become the above-described candidate regions. ..

In step 405, the voxel deletion unit 304 performs processing (voxel deletion processing) of deleting unnecessary voxels in the candidate area generated in step 404. Details of the voxel deletion processing will be described later.

In step 406, the voxel restoration unit 305 restores the voxels that were excessively deleted by the voxel deletion process based on the threshold value set by the threshold value setting unit 307 and the information of the candidate area generated by the candidate area generation unit 303 ( Voxel restoration processing) is performed. Details of the voxel restoration processing will be described later.

The above is the contents of a series of image processing in the image processing apparatus 200 of the present embodiment.

<Characteristics of visual volume intersection method>
The visual volume intersection method does not use the correspondence relationship based on the color information of the subject. Therefore, there is an advantage that the shape can be estimated robustly and at high speed, but on the other hand, there are disadvantages that a false area is generated and only a convex shape can be estimated. In the above-described shooting scene shown in FIG. 5, a part of the right sphere of the two spheres that are the subject is partially missing. As described above, the cones 504 and 505 are projected in the space from the camera 501, the cones 506 and 507 are projected from the camera 502, and the cones 508 and 509 are projected in the space respectively. In this case, in addition to the convex polyhedrons 511 and 521 that circumscribe the two spheres that are subjects, relatively small convex polyhedrons 512 and 522 are generated as candidate regions. Since these four candidate regions are estimated as convex polyhedrons circumscribing the subject, in the case of a subject having a concave portion such as the sphere 520, the concave portion cannot be reproduced. Further, the convex polyhedrons 512 and 522 that do not originally exist as the subject are also estimated as false candidate regions. The present invention compensates for the drawbacks of the visual volume intersection method as described above.

<Voxel deletion processing>
Next, the voxel deletion processing in step 405 of the flow of FIG. 4 will be described. In this processing, whether or not to delete an arbitrary voxel is determined based on color consistency when an arbitrary voxel is projected onto a plurality of cameras having different viewpoints. FIG. 7 is a diagram for explaining the voxel deletion process. FIG. 7A shows images 601 to 602 taken by the cameras 501 to 503 shown in FIG. 6A, and FIG. 7B shows a state in which an arbitrary voxel 700 is projected on the cameras 501 to 503. FIG. In FIG. 7A, black rectangle points 701 to 703 on the images 601 to 603 indicate positions when the voxels 700 are projected in the images. Further, in FIG. 7B, line segments 711, 712, 713 indicate virtual projection planes of the cameras 501, 502, 503, respectively. The voxel deleting unit 304 determines the pixel value {I _1R , I _1G , I _1B } of the point 701 onto which the voxel 700 is projected, the pixel value {I _2R , I _2G , I _2B } of the point 702, and the pixel value {I of the point 703. _{Based on 3R} , I _3G , I _3B }, the similarity between pixel values is calculated. Here, NCC (Normalized Cross-Correlation) is used to evaluate the similarity, but another similarity index such as SSD (Sum of Squared Difference) or SAD (Sum of Absolute Difference) may be used. When there are N cameras for which it is determined that the voxel 700 is visible (visible determination), the average NCC is calculated using the following equation (1).

In the above formula (1), bold I is a three-dimensional vector having R, G, B channels as elements. Since different voxels have different cameras that can be seen, the number of cameras N also changes depending on the voxel. In the example of FIG. 7, since N=3, the sum is calculated for the three combinations in equation (1).

In NCC, when the pixel value is taken as a vector, the angle formed by the vector is obtained. In the above equation (1), the NCC average is obtained for the combination of N cameras. At this time, in the visual determination of the voxel 700, a depth map is created from the rough shape obtained by the visual volume intersection method, and a method performed based on this depth map, or the normal line of the rough shape and the optical axis direction of the camera are determined. There is a method based on the angle formed by the vector to be represented, and there is no particular limitation.

Then, if the average NCC value obtained by the above equation (1) for the voxel of interest is less than a predetermined threshold D _th (for example, 0.9), it is determined that there is no color consistency (that is, unnecessary voxels). Then, the relevant voxel is deleted. In addition, here, the similarity is calculated based only on the pixel values of the points 701 to 703, but the pixel values around the points 701 to 703 (for example, in a 3×3 block centered on the points 701 to 703, respectively) The degree of similarity may be obtained by using (pixel value).

Further, instead of the determination based on color consistency, voxel deletion may be performed using another method such as determining whether or not an unnecessary voxel is based on geometrical information of voxels. The above is the content of the voxel deletion processing.

<Voxel restoration processing>
Next, the voxel restoration processing in step 406 of the flow of FIG. 4 will be described. Before starting the description of the specific processing content, the principle of the voxel restoration processing will be described first.

As described above, the false area among the candidate areas is generated as a common area of the cones in the space where the subject does not exist. Therefore, the volume is usually considerably smaller than that of the subject. Further, since the voxel deleting process described above is likely to delete voxels in the false region, the total number of voxels in the false region after the voxel deleting process is further reduced. Therefore, it is expected that the peripheral voxel density of a specific grid point where the corresponding voxel does not exist in the three-dimensional space including the candidate region is relatively low. On the other hand, the subject area where the subject actually exists has a large original volume. Therefore, even if a part of the originally required voxels is deleted by the voxel deletion process, the peripheral voxel density of the specific grid point is considered to be higher than that in the false region. Therefore, in the present invention, according to the peripheral voxel density of the specific grid point in the candidate region, the voxels are not restored in the false region but the voxels are restored only in the subject region. Specifically, the voxel is restored to the position of the specific grid point only when the number of voxels existing in the predetermined range including the specific grid point in the candidate region is equal to or larger than the predetermined number. The predetermined number here is represented by R _th . As described above, as a result, the voxels are restored only in the subject region that has a large volume and is still densely filled to some extent after the voxel deletion processing. This is the same even when the false area is adjacent to the subject area.

Here, how to determine the predetermined number R _th in the threshold setting unit 307 will be described. For example, when the voxel deleting unit 304 deletes voxels based on the peripheral voxel density of a certain grid point, based on the number of grid points (3^3 -1=26) adjacent to the grid point in the three-dimensional space. Can determine the predetermined number R _th . As described above, the peripheral voxel density of the specific grid point in the false area is smaller than that in the subject area. Therefore, half (26/2=13) or 1/3 (26/3≈8) of the number of all adjacent grid points in the three-dimensional space is set as the predetermined number R _th . Further, in the intra-subject region, there is a high possibility that there are more voxels than the number of adjacent lattice points in the two-dimensional plane even after the voxel deletion processing is performed. Therefore, it is also possible to determine the predetermined number R _{th based} on the number of adjacent grid points in the two-dimensional plane. For example, since the number of adjacent grid points on the two-dimensional plane is 8 (3^2 -1), a number larger than 8 may be set as the predetermined number R _th . Further, even when referring to a grid point that is not adjacent to a specific grid point, the predetermined number R _th can be determined based on the two-dimensional or three-dimensional geometrical property. As described above, by setting the predetermined number R _th at the time of voxel restoration based on the geometrical property, it is possible to perform the voxel restoration process with high accuracy.

8A to 8C are diagrams illustrating the concept of the voxel restoration process. Here, for convenience of explanation, it is expressed in two dimensions, but in reality it is three dimensions. In FIG. 8A, a thick line hexagon 801 shows a horizontal section when a convex polyhedron circumscribing a sphere as a subject is viewed from above, and a thick line triangle 802 shows a horizontal section of a false region. Then, a thin line rectangle 803 indicates a horizontal cross section of the voxels included in the two convex polyhedrons. The actual volume of the false area is smaller than that of the subject area, but is exaggerated here. The inside of the convex polyhedron corresponding to each of the hexagon 801 and the triangle 802 is a candidate area. FIG. 8B shows a state in which some voxels are deleted by the voxel deletion process based on color matching. In FIG. 8B, the voxels in the triangle 802 representing the false area are deleted, while the voxels in the hexagon 801 representing the subject area are deleted too much. Here, for convenience of description, voxels in the false region are not completely deleted, but a method of further deleting unnecessary voxels in such a case will be described in the second embodiment. FIG. 8C shows a state in which voxels are added to the subject area by the voxel restoration processing. It can be seen from FIG. 8C that voxels once deleted in the false area are not added again, but voxels are added only in the subject area. The above is the principle of the voxel restoration processing in the present embodiment.

Next, a specific procedure of voxel restoration processing will be described. FIG. 9 is a flowchart showing details of the voxel restoration processing according to this embodiment.

In step 901, a three-dimensional space that surely includes a subject whose shape is to be estimated, which is a target of voxel restoration processing, is set as a processing target space. For example, in the scene of FIG. 1 described above, when a player or a ball on the field is the subject of shape estimation, a range in which the angle of view of the ten cameras 101 is common is set as the processing target space. Of course, a narrower range may be set as the processing target space, and for example, in FIG. 1, only a predetermined range centered on the center of the field may be set as the processing target space. Here, it is assumed that a rectangular parallelepiped including the subject (two spheres) in the parallax image in FIG. 5 is set as the processing target space.

In step 902, the target grid point is initialized (the initial position of the target grid point is set) in the set processing target space. Here, any one of the grid points when the above rectangular parallelepiped set as the processing target space is divided by a unit grid of the same size as the voxel, for example, (x,y,z)=(0,0 , 0) is set to the initial position of the grid point of interest.

In step 903, it is determined whether or not a voxel exists on the target grid point in the processing target space. If no voxel exists on the grid point of interest, the process proceeds to step 904. On the other hand, if there is a voxel on the grid point of interest, the process proceeds to step 908.

In step 904, it is determined whether the grid point of interest in the processing target space is included in the above-mentioned candidate area. If the grid point of interest is included in the candidate area, the process proceeds to step 905. On the other hand, if the grid point of interest is not included in the candidate area, the process proceeds to step 908.

In step 905, the number of voxels existing around the target grid point is counted, and it is determined whether or not the count number of peripheral voxels is equal to or _larger than a predetermined number R _th . The “peripheral” range is, for example, 26 grid points adjacent to the grid point of interest in the three-dimensional space, that is, 3×3×3=27 grid points in the three-dimensional space, and The removed ones are referred to as "surroundings". However, a wider range (for example, 4×4×4=63 grid points excluding the grid point of interest at the center from 64 grid points) may be set as the “periphery”. Further, the predetermined number R _th may be set in advance according to the above-mentioned “principle of voxel restoration processing” in consideration of the “peripheral” range, the size of the unit lattice in the processing target space, and the like. Here, it is assumed that 10 is set as the value of the predetermined number R _th . If the result of determination is that the count number of peripheral voxels is greater than or equal to the predetermined number R _th , processing proceeds to step 906. On the other hand, if the count number of peripheral voxels is less than the predetermined number R _th , the process proceeds to step 908.

In step 906, voxels are added to the positions in the candidate area corresponding to the grid point of interest. Then, in step 907, it is determined whether the processing has been completed for all grid points in the processing target space. If there are unprocessed grid points, the process proceeds to step 908. On the other hand, if the processing has been completed for all grid points, the process proceeds to step 909.

In step 908, the target grid point is updated. The grid point of interest is updated by changing its position coordinates in the order of x-axis→y-axis→z-axis. For example, (0,0,0)→(1,0,0)→(2,0,0)... When a new grid point of interest is set by the update, the process returns to step 903 and the processing for the new grid point of interest is continued.

In step 909, it is determined whether the above process has been performed a specified number of times. If the specified number of times of processing has not been performed, the number of times of processing is updated in step 910, and the process returns to step 902. On the other hand, if the process has been performed the specified number of times, this process ends. For example, if n=5 is set as the prescribed number of times n, the above processing is repeated 5 times.

The above is the content of the voxel restoration processing. The reason that the voxels in the subject area are properly restored in FIG. 8C is that the grid points to which voxels are added are limited to only the candidate areas in the processing of step 904. .. Further, the voxels in the false region are not restored because the number of voxels adjacent to the grid point of interest is counted in step 905.

<Effect of this embodiment>
FIG. 10 is a diagram for explaining the effect of the invention according to the present embodiment. FIG. 10A is the same diagram as FIG. 8B and shows a state in which voxels are deleted by the voxel deletion processing. In this state, not only the voxels in the candidate area 1002 which is a false area but also the necessary voxels in the candidate area 1001 representing the subject area have been deleted. FIG. 10B shows a state in which the dilation processing by a known morphological operation is applied to the state after the voxel deletion processing in FIG. 10A. In this case, a voxel is added if at least one voxel exists in the adjacent grid point of the grid point of interest, so voxels are added beyond the range in both candidate regions 1001 and 1002. As described above, in the case of the conventional expansion processing by the morphological operation, as a result of adding voxels beyond the range of the subject area, the shape of the subject is not accurately estimated, and further, the false area is expanded. .. FIG. 10C shows a state in which the method of this embodiment is applied to the state after the voxel deletion processing in FIG. 10A. In the case of the method of the present embodiment, voxels are not added to the false region, and voxels are added only to the subject region. As a result, the subject shape can be estimated more accurately.

<Modification>
As a modified example of the present embodiment, in the voxel deletion processing (step 405), the voxel (the voxel to be deleted) that is to be deleted by the judgment based on the color consistency (judgment using the threshold D _th ) is deleted again. An aspect of re-determining whether or not to perform using different criteria will be described. Specifically, the number of voxels around the voxel to be deleted is counted, and the count number is compared with a predetermined number set separately to determine whether or not to actually delete the voxel to be deleted. The predetermined number here is defined as S _th , and is distinguished from the predetermined number R _th in step 905 described above. Here, the value of the predetermined number S _th , which serves as a reference for actually deleting the voxel to be deleted, is set to 5, but any other value may be used.

First, the value of the average NCC obtained from the above equation (1) becomes less than the above-mentioned threshold value D _th (for example, 0.9. Hereinafter, this threshold value is referred to as D _th1 as the first threshold value) related to the similarity, and the voxel is reached. Suppose you decide to delete. Next, the number of voxels around the deletion target voxel is counted. The “peripheral” range here may be the same as the voxel restoration process (step 406). If the number of peripheral voxels is less than the predetermined number S _th (for example, 5), it is determined that the deletion target voxel is likely to belong to the false area, and the deletion is actually performed. On the other hand, when the number of peripheral voxels is equal to or _larger than the predetermined number S _th , it is determined that the deletion target voxel may belong to the subject region, and another threshold value (for example, 0.7. _Is represented by D _th2 as a second threshold), and the color consistency is determined again. Then, when the average NCC value obtained from the above equation (1) is less than the second threshold value D _{th2, the} deletion target voxel is actually deleted. On the other hand, when the average NCC value is equal to or _larger than the second threshold value D _th2 , it is determined that the deletion target voxel is likely to belong to the subject area, and the deletion target voxel is maintained without being actually deleted.

It is desirable that the value of the predetermined number S _th used in the re-determination in this modification be set to a value smaller than the predetermined number R _th in the voxel addition process. However, if it is possible to prevent the voxels in the deleted false region from being restored by the subsequent voxel addition process, the value of the predetermined number S _th can be freely set without being bound by the relationship with the predetermined number R _th of the voxel addition process. You may decide.

In the case of this modification, it is possible to prevent unnecessary deletion of voxels in the subject region in the voxel deletion processing.

As described above, according to the present embodiment, unnecessary voxels can be further deleted from the false area and the voxels that have been deleted too much can be restored from the object area by a simpler process.

[Example 2]
Next, a second embodiment will be described in which a voxel deletion process with different determination criteria is performed a plurality of times, and a voxel is added and a candidate region is updated each time to perform more accurate shape estimation. The description of the parts common to the first embodiment will be omitted or simplified, and the differences will be mainly described below.

FIG. 11 is a functional block diagram of the image processing apparatus 200 according to this embodiment. The image processing apparatus 200 of the present embodiment has a candidate area updating unit 1101 in addition to the units shown in FIG. Then, FIG. 12 is a flowchart showing a flow of a series of image processing executed by the image processing apparatus 200 of the present embodiment. The flow of image processing performed by the image processing apparatus 200 according to this embodiment will be described below.

Steps 1201 to 1204 are the same as steps 401 to 404 in the flow of FIG. 4 of the first embodiment. That is, parallax image data and silhouette image data are acquired in step 1201, and various parameters related to the camera 101 are acquired in subsequent step 402. Then, in step 1203, distortion correction processing based on the distortion parameter of the acquired camera parameters is performed, and in step 1204, a “candidate region” in which the subject is expected to exist inside is generated.

In step 1205, one voxel process of the plurality of voxel deletion processes to be applied is performed on the candidate region generated in step 1204. In the present embodiment, a case will be described as an example where a plurality of voxel deleting processes, in addition to the voxel deleting process based on the color consistency described in the first embodiment, a process of deleting a voxel forming a shadow of a subject is performed. To do. The subject part in the silhouette image may include a part of the subject's shadow, and this shadow needs to be removed in order to estimate the shape of the subject alone. Therefore, processing is performed to delete the voxels in the portion where the shadow of the subject such as the ground and the table is projected. For example, when the coordinate of the surface on which the shadow of the subject is projected is z=0, a process of deleting all voxels whose z coordinate value is equal to or less than a certain threshold value is performed. Either of these two types of voxel deletion processing may be executed first.

In step 1206, a voxel restoration process for restoring voxels that have been deleted too much in the voxel deletion process is performed. The content of the voxel restoration processing is the same as step 406 in the flow of FIG. 4 of the first embodiment. If the process of deleting the voxels that make up the shadow of the subject is executed first, it is necessary to add voxels that restore parts other than the shadow, such as the foot if the subject is a person, or the bottom if the subject is an object. Done.

In step 1207, it is determined whether all the voxel deleting processes have been executed. If there is unexecuted voxel deletion processing, the process proceeds to step 1208, and candidate area update processing is executed. That is, when the deletion process is performed a plurality of times, in the second and subsequent deletion processes, the candidate region that reflects the results of the deletion process and the restoration process performed immediately before is set as a new candidate region. For example, when the voxel deleting process executed immediately before is the deleting process of the voxels forming the shadow of the subject, a new candidate area in which the shadow of the subject is removed is set. Then, returning to step 1205, the next voxel deletion process is executed for the newly set candidate area. On the other hand, if all voxel deletion processing has been executed, this processing ends.

The above is the content of the series of image processing executed by the image processing apparatus 200 of the present embodiment. By executing a plurality of types of voxel deletion processing in this manner, it is possible to delete voxels in the false area 802 that have not been completely deleted by one kind of voxel deletion processing, as shown in FIG. 8C, for example. It will be possible.

Note that the plurality of voxel deletion processes executed in this embodiment are not limited to the above-mentioned two types, and three or more types of voxel deletion processes may be performed. Further, the iteration number n of the voxel addition process may be changed for each voxel deletion process. For example, with respect to the voxel deletion processing for determining the color consistency, it is possible to reproduce the concave object shape by setting a small number of iterations in the voxel addition processing.

In this embodiment, a plurality of types of voxel deletion processing are executed while updating the candidate area. By repeatedly deleting and adding voxels in this manner, the three-dimensional shape of the subject can be estimated with higher accuracy.

<Other embodiments>
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

200 image processing device 301 image data acquisition unit 303 candidate region generation unit 304 voxel deletion unit 305 voxel restoration unit

Claims (11)

Generating means for generating a candidate region corresponding to the three-dimensional shape of the subject and composed of voxels , based on the plurality of images acquired by the plurality of image capturing devices;
For a specific voxel forming the candidate area, processing means for performing processing including deletion,
Have
The processing means, based on the density of the other voxels around the particular voxel, determining whether to delete with respect to the particular voxel,
An image processing device characterized by the above. The image processing apparatus according to claim 1, wherein the processing unit further processes the specific voxel based on a predetermined criterion. The processing means, the particular voxel does not satisfy a predetermined criterion, and based on the fact that the density does not satisfy a predetermined condition, according to claim 2, wherein the deleting the particular voxel Image processing device. The processing means, the particular voxel is based on the fact that not satisfy the predetermined criteria, to delete the particular voxel, other conditions are satisfied density voxels of a predetermined surrounding of the deleted the particular voxel The image processing apparatus according to claim 2, wherein the deleted specific voxel is restored based on the above. The image processing apparatus according to claim 3, wherein the predetermined condition is that the number of other voxels existing in a predetermined range including the specific voxel is equal to or larger than a predetermined number. The image processing apparatus according to claim 5, wherein the predetermined number is determined based on the number of voxels adjacent to one voxel . Wherein the predetermined criteria, the image processing apparatus according to any one of claims 2 to 6, characterized in that means that there is a color consistency in each of the plurality of images corresponding to the particular voxel .. Wherein there is a consistent, according to claim 7, characterized in that refer to the evaluation value of the pixel values mutual similarity in each of the plurality of images corresponding to the particular voxel is above the threshold Image processing device. Wherein the predetermined criteria, the particular voxel, the image processing apparatus according to any one of claims 2 to 8, characterized in that means it is not a voxel corresponding to the shadow of the object. A generation step of generating a candidate region corresponding to the three-dimensional shape of the subject and composed of voxels , based on the plurality of images acquired by the plurality of imaging devices;
For a specific voxel that constitutes the candidate area, a processing step of performing processing including deletion,
Have
In the processing step, based on the density of other voxels around the specific voxel, it is determined whether to delete for the specific voxel ,
An image processing method characterized by the above. The computer program to function as the image processing apparatus according to any one of claims 1 to 9.

Images