JP2022099127A - Subject silhouette extract apparatus, method, and program - Google Patents

Abstract

To provide a subject silhouette extraction apparatus, method, and program capable of extracting a subject silhouette suitable for 3D model generation from a video image in consideration of an estimation error of camera parameters.SOLUTION: In a subject silhouette extraction apparatus 1, a video image acquisition unit 10 acquires a video image from at least one camera cam, a camera parameter error estimation unit 20 estimates an error of a camera parameter indicating each camera state for each camera based on a still image extracted from the video image, for example, a frame image, a silhouette extraction parameter calculation unit 30 calculates a parameter (silhouette extraction parameter) when extracting a silhouette from the video image based on the estimation result of the error of the camera parameter, and a silhouette calculation unit 40 extracts a silhouette image from the video image by silhouette calculation using the silhouette extraction parameter for each frame.SELECTED DRAWING: Figure 1

Description

The present invention relates to a subject silhouette extraction device, a method, and a program for extracting a silhouette with only the subject portion as the foreground from a moving image taken by a camera.

Free-viewpoint video technology is a technology that enables video viewing from any viewpoint, including viewpoints where cameras do not exist, by inputting video from multiple cameras. As a method for realizing a free-viewpoint image, a 3D model-based free-viewpoint image generation method based on the visual volume crossing method disclosed in Non-Patent Document 1 is known.

In the visual volume crossing method, as shown in Fig. 13, the intersection of the visual cones when N silhouette images taken at different camera positions are projected onto the 3D world coordinates is viewed based on the following equation (1). Volume (Visual Hull) This is a technology acquired as VH (K).

Here, the set K is a set of silhouette images of each camera, and V _k is a visual cone calculated based on the silhouette images obtained from the kth camera. In addition, normally, the part that is the common part of all N cameras is modeled, but even if the number of cameras to be modeled is changed, such as modeling when N-1 cameras are common. good.

At this time, it may have a function of converting a voxel model into a polygon model by using a method of converting a voxel model into a polygon model such as a marching cube method, and may have a function of outputting a 3D model as a polygon model. ..

Such a visual volume crossing method is used as a basic technique for realizing a full model free viewpoint (= a free viewpoint of a method that faithfully expresses the shape of a 3D model) disclosed in Non-Patent Document 2. ..

As a silhouette extraction method for obtaining an intersection used in the visual volume crossing method, a method based on the background subtraction method represented by Non-Patent Document 3 is known. The background subtraction method is a method called a background model in which a subject is extracted based on the difference between a model in which no subject exists and an input image. Further, in recent years, a deep learning-based subject silhouette extraction method disclosed in Non-Patent Document 4 has also appeared, and methods capable of extracting silhouettes with high accuracy have been proposed one after another.

Japanese Patent Application No. 2020-012676

Laurentini, A. "The visual hull concept for silhouette based image understanding.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 150-162, (1994). J. Kilner, J. Starck, A. Hilton and O. Grau, "Dual-Mode Deformable Models for Free-Viewpoint Video of Sports Events," Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007), Montreal, QC, 2007, pp. 177-184. C. Stauffer and W. E. L. Grimson, "Adaptive background mixture models for real-time tracking," 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 246-252, Vol. 2, (1999). Lim, Long Ang, and Hacer Yalim Keles. "Foreground Segmentation Using Convolutional Neural Networks for Multiscale Feature Encoding." Pattern Recognition Letters, (2018). J. Chen, et al, "Sports Camera Calibration via Synthetic Data," CVPR Workshop, 2019. Z. Zhang. A flexible new technique for camera calibration. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 22 (11): 1330-1334, 2000. Yuki Tsurusaki, Keisuke Nonaka, Ryosuke Watanabe, Sei Naito, "Study on Higher Accuracy of Camera Calibration Using Line Segment Detector,", 2020-AVM-108, 7, pp. 1-6, 2020. O. Barnich and M. Van Droogenbroeck, "ViBe: A Universal Background Subtraction Algorithm for Video Sequences," in IEEE Transactions on Image Processing, vol. 20, no. 6, pp. 1709-1724, June 2011.

In the silhouette extraction used for the visual volume crossing method, the silhouette obtained by the silhouette extraction techniques of Non-Patent Documents 3 and 4 is perfect (perfectness here is used for evaluation in Non-Patent Documents 3 and 4). Even if F-Measure is defined as showing 100%), defects can occur in the 3D model generated by the visual volume crossing method. One factor in this is the estimation error of the internal parameters and external parameters of the camera.

In order to generate a 3D model using the visual volume crossing method, the position and orientation of the camera are estimated correctly in advance, and the distortion caused by the lens contained in the image is removed to reduce the estimation error of the camera parameters. There is a need to. Although these studies have been improved in Non-Patent Document 5, etc., they are still under research, and it is difficult to accurately estimate the position and orientation of the camera automatically. As a result, as shown in FIG. 6, there is a problem that a defect occurs in the synthesized 3D model and a feeling of strangeness occurs during viewing.

An object of the present invention is to provide a subject silhouette extraction device, a method and a program capable of extracting a subject silhouette suitable for 3D model generation from a moving image in consideration of the estimation error of camera parameters by solving the above technical problems. It is in.

In order to achieve the above object, the present invention is characterized in that the subject silhouette extraction device for extracting the silhouette of the subject from the moving image taken by the camera has the following configuration.

(1) Equipped with a means for estimating the camera parameter error based on the moving image, a means for calculating the silhouette extraction parameter based on the estimation result of the camera parameter error, and a means for calculating the subject silhouette based on the silhouette extraction parameter. did.

(2) The contour of the subject silhouette is expanded as the camera parameter error increases.

(3) The degeneracy and expansion processes for the contour of the subject silhouette are repeated at least once in this order.

(4) The background subtraction threshold is calculated so that the larger the camera parameter error, the easier it is for each pixel to be identified in the foreground region.

(5) The larger the camera parameter error, the lower the update rate of the background model is calculated.

(6) The background difference threshold is calculated so that the larger the camera parameter error is, the easier it is for each pixel to be identified in the foreground region, and the update rate of the background model is calculated to a low value.

(1) Since the subject silhouette is calculated using the silhouette extraction parameters calculated based on the estimation result of the camera parameter error, defects that may occur due to the camera parameter error when generating a 3D model using the subject silhouette, etc. It becomes possible to suppress the deterioration of the quality of the camera.

(2) Since the contour of the subject silhouette expands as the camera parameter error increases, it becomes possible to provide a subject silhouette that can suppress quality deterioration such as chipping that may occur due to the camera parameter error.

(3) Since each process of degeneration and expansion is repeated at least once in the order for the contour of the subject silhouette, it is possible to provide a subject silhouette that can suppress quality deterioration such as chipping and noise that may occur due to camera parameter error. It will be like.

(4) The background difference threshold is calculated so that the larger the camera parameter error, the easier it is for each pixel to be identified in the foreground region, so the subject silhouette can be expanded. Therefore, it becomes possible to provide a subject silhouette that can suppress quality deterioration such as chipping that may occur due to a camera parameter error.

(5) The larger the camera parameter error, the lower the update rate of the background model is calculated, so each pixel can be easily identified in the foreground region. Therefore, it becomes possible to provide a subject silhouette that can suppress quality deterioration such as chipping that may occur due to a camera parameter error.

(6) The background subtraction threshold is calculated so that the larger the camera parameter error is, the easier it is for each pixel to be identified in the foreground region, and the update rate of the background model is calculated to a low value, which may occur due to the camera parameter error. Quality deterioration such as chipping can be suppressed.

It is a functional block diagram of the 3D model generation system including the subject silhouette extraction apparatus to which this invention is applied. It is a functional block diagram of the subject silhouette extraction apparatus which concerns on 1st Embodiment of this invention. It is a figure which showed schematically the estimation method of a camera parameter. It is a figure which showed the example which a reprojection error occurs. It is a figure which showed the example which the visual volume is calculated small due to the estimation error of a camera parameter. It is a figure which showed the example which a defect occurs in a 3D model due to the estimation error of a camera parameter. It is a figure which showed the example which the 3D model with few defects is formed by the contour expansion of a silhouette. It is a functional block diagram of the subject silhouette extraction apparatus which concerns on 2nd Embodiment of this invention. It is a functional block diagram of the subject silhouette extraction apparatus which concerns on 3rd Embodiment of this invention. It is a figure which showed schematically the method of extracting the silhouette by the background subtraction method using the background model. It is a functional block diagram of the subject silhouette extraction apparatus which concerns on 4th Embodiment of this invention. It is a functional block diagram of the subject silhouette extraction apparatus which concerns on 5th Embodiment of this invention. It is a figure which showed the generation method of the 3D model by the visual volume crossing method.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing a configuration of a main part of a 3D model generation system 100 including a subject silhouette extraction device 1 to which the present invention is applied.

The subject silhouette extraction device 1 mainly includes a moving image acquisition unit 10, a camera parameter error estimation unit 20, a silhouette extraction parameter calculation unit 30, and a silhouette calculation unit 40. In addition to the subject silhouette extraction device 1, the 3D model generation system 100 includes a 3D model generation unit 2 that generates a 3D model by a visual volume crossing method using a silhouette image.

Such a 3D model generation system 100 or its subject silhouette extraction device 1 provides an application (program) that realizes each function on at least one general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, and the like. It can be configured by implementing it. Alternatively, it can be configured as a dedicated machine or a single-purpose machine in which a part of the application is made into hardware or software.

The moving image acquisition unit 10 acquires a moving image from at least one camera cam. When moving images are acquired from a plurality of cameras, each camera is arranged so as to shoot the subject from different viewpoints. The moving image may be acquired directly from the camera, or may be acquired by reading the moving image file stored in the moving image database (DB) 3 or the like.

The camera parameter error estimation unit 20 estimates a camera parameter error (camera parameter error) indicating the state of each camera for each camera based on a still image extracted from a moving image, for example, a frame image.

The silhouette extraction parameter calculation unit 30 calculates a parameter (silhouette extraction parameter) for extracting a silhouette from a moving image based on the estimation result of the camera parameter error. The silhouette calculation unit 40 extracts a silhouette image from a moving image by silhouette calculation using the silhouette extraction parameter for each frame.

As described with reference to FIG. 13, the 3D model generation unit 2 generates a 3D model of the subject by the visual volume crossing method using the silhouette image.

FIG. 2 is a functional block diagram showing the configuration of the first embodiment of the subject silhouette extraction device 1, and the configurations unnecessary for the description of the present invention are not shown here. The present embodiment is characterized in that the silhouette of the subject acquired from the moving image by a known method is expanded based on the silhouette extraction parameter.

The camera parameter error estimation unit 20 includes a camera parameter estimation unit 21 and a reprojection error estimation unit 22. Camera parameters are used to transform a 3D point (X, Y, Z) on world coordinates into a 2D point (u, v) on the camera image, including internal and external parameters. The conversion formula in the pinhole camera model is represented by the matrix of the following formula (2).

r11 to r33 are rotation matrices indicating the direction of the camera, and t1 to t3 are parallel traveling matrices indicating the position of the camera, both of which are called external parameters of the camera. fx and fy are the focal lengths in pixel units that indicate the zoom condition, and cx and cy are the principal points of the image, both of which are called internal parameters of the camera. The internal parameters may also include distortion coefficients when modeling radial distortion, tangential distortion, etc. peculiar to the camera lens. s is a variable used for scaling to make it [u, v, 1].

The camera parameter estimation unit 21 estimates camera parameters by a known method. In this embodiment, as disclosed in Non-Patent Document 6, a large number of pairs of known 3D world coordinates of the world coordinate system and corresponding 2D coordinates on the image are collected so that their reprojection errors are reduced. A method for estimating the optimum parameters is adopted.

FIG. 3 is a diagram schematically showing a method of estimating camera parameters according to Non-Patent Document 6. If the subject to be photographed includes fields and courts in the stadium and the three-dimensional world coordinates (Xi, Yi, Zi) of the white line intersections are known from the standards, etc., as shown in Fig. (A), the same as these white line intersections. A large number of pixel pairs with the 2D positions (u _{c, i} , v _{c, i} ) of the corresponding white line intersections reflected in the 2D image of Fig. (B) are collected and each parameter is tentatively estimated. Then, using this parameter, the 3D coordinates (Xi, Yi, Zi) are projected onto the 2D image at the point (u'c _{, i} , v'c _{, i} ) and the reflected position (u _{c, i} , v _c) . _{, i} ) The camera parameters are estimated by optimizing each parameter so that the difference in distance (reprojection error) Rc, i becomes small.

i is the index of the pixel pair, and in the example of Fig. 3, the 3D position of the white line intersection of the stadium is known, so the corresponding point on the image [Fig. (B)] corresponding to that 3D position is manually set on the UI. (Xi, Yi, Zi) and points on 2D images (u _{c, i} , v _{c, i} ) by automatic estimation based on the selection of or the method disclosed in Non-Patent Document 7. ) And the corresponding pixel pair.

In the present embodiment, the camera parameter estimation unit 21 estimates the camera parameter in which the average value of the reprojection errors Rc and i of each pixel pair i becomes small, but the acquired moving image is distorted due to the influence of the distortion caused by the lens. In some cases, or when the estimated position of the point (uc, i, vc, i) on the 2D image contains an error, the reprojection error Rc, i does not become 0.

Figure 4 shows an example in which a reprojection error occurs. In this example, the detected positions of the white line intersections (uc, 11, vc, 11) deviate significantly from the actual white line intersection positions. Therefore, the actual white line intersections at the points (u'c, ₁₁ , v'c, ₁₁ ) where the 3D coordinates (X11, Y11, Z11) are projected onto the 2D image using the estimated camera parameters. It deviates significantly from the position of, and finally the reprojection error Rc, i also becomes large. This is an extreme example, but it is generally difficult to set the reprojection error Rc and i to 0.

If the reprojection error Rc and i are large, there is a high possibility that the estimation result of the camera parameters contains an error. Therefore, when free viewpoint production is performed under this situation, the visual volume becomes large as shown in FIG. It may be calculated small, resulting in defects in the 3D model, as shown in Figure 6.

In the present embodiment, the reprojection error estimation unit 22 calculates the average of the reprojection errors Rc and i calculated from all the pixel pairs on the camera c by the following equation (3), whereby each camera c is calculated by the following equation (3). It represents the camera parameter error Ec that it has. Here, I is the total number of corresponding pixel pairs of (Xi, Yi, Zi) and (uc, i, vc, i).

In the above equation (3), the camera parameter E _c is obtained as one value for each camera, but the camera parameter error E _c, (u, v) that differs for each pixel (u, v) on the camera is calculated. May be good. As a means of calculating different camera parameter errors for each pixel, in the above equation (3), the reprojection error R _{c, i} of the pixel pair closest to each pixel (u, v) is obtained as E _c (u, v). The method can be considered. Alternatively, for each pixel (u, v), the reprojection error R _{c, i} of each surrounding pixel pair is reflected more strongly as the reprojection error R _{c, i} of the pixel pair with a shorter distance, depending on the reciprocal of the distance. It may be a weighted sum.

As a result, the reprojection error according to the pixel (u, v) can be adopted, so that the error can be estimated more accurately when only a part of the image contains a large amount of distortion. Further, by obtaining the error E _c (u, v) for each pixel, different camera parameters can be set for each pixel (u, v) in the silhouette extraction parameter calculation unit 30 in the subsequent stage.

The calculation method of the camera parameter error Ec (u, v) is not limited to the method using the reprojection error. For example, based on the tendency that the radial distortion caused by the camera lens increases toward the outside of the angle of view of the camera, a method is adopted such that Ec (u, v) increases as the distance from the center of the image increases. You may.

The silhouette extraction parameter calculation unit 30 includes the expansion amount calculation unit 31, and calculates the expansion amount when expanding the silhouette extracted from the moving image based on the estimation result of the camera parameter error by the camera parameter error estimation unit 20.

In this embodiment, the contour of the foreground region (the region where the silhouette is white) of the silhouette extracted from the moving image is expanded by the typical method disclosed in Non-Patent Document 3 and Non-Patent Document 4. As a result, as shown in FIG. 7, it becomes possible to form a model with few defects when forming a 3D model.

By expanding the contour of the silhouette, there is a concern that the contour of the 3D model generated using the silhouette will also be expanded. However, when the 3D model of the subject is inflated, the discomfort during viewing becomes less noticeable than when a part of the subject is missing. In addition, regarding the expansion of the subject, as invented by the inventor of the present invention and disclosed in Patent Document 1, a method for reducing discomfort has already been proposed.

The expansion amount calculation unit 31 calculates the expansion amount d based on the camera parameter error E _c obtained by the above equation (3). In the present embodiment, the expansion amount d is set to a larger value as the silhouette is extracted from the camera having a larger camera parameter error E _c , and the silhouette is expanded more. As a result, it is possible to prevent the subject 3D model from being damaged due to the influence of the camera parameters. If the parameter error can be calculated not in the camera unit (Ec) but in the pixel unit Ec, i (u, v) for each camera, the expansion amount d may also be determined for each pixel (u, v).

The silhouette calculation unit 40 includes a silhouette extraction unit 41 and a silhouette contour expansion processing unit 42. The silhouette extraction unit 41 extracts the silhouette of the subject by applying an arbitrary method disclosed in Non-Patent Document 3 and Non-Patent Document 4 to the frame image. The silhouette contour expansion processing unit 42 performs contour expansion processing according to the expansion amount d as a post-processing for the silhouette extracted by the silhouette extraction unit 41.

The expansion in the silhouette contour expansion processing unit 42 is performed by expanding each pixel in the foreground region of the extracted silhouette to the peripheral (2d + 1) × (2d + 1) pixels. d is a parameter that represents the amount of expansion of the contour of the silhouette, and must be an integer greater than or equal to 0 (d = 0 means that expansion processing is not performed). The contour expansion of the silhouette is not limited to the method of expanding to the peripheral (2d + 1) × (2d + 1) pixels, and other expansion methods such as repeating expansion to 4 pixels in the vertical and horizontal directions may be adopted. ..

When the silhouette calculation unit 40 further includes the silhouette contour shrinkage processing unit 43 as in the second embodiment shown in FIG. 8, the silhouette extraction parameter calculation unit 30 may further include the degeneracy amount calculation unit 32. good.

In silhouette extraction, it is known that by performing a degeneracy process of a small size and then an expansion process of a large size, the foreground region of the silhouette can be expanded and minute noise can be eliminated during the degeneration process.

In the degeneration process, when even one non-foreground pixel is included in the peripheral (2e + 1) × (2e + 1) pixels of the foreground pixel of the silhouette, the pixel is near the outline of the foreground area. It is done by changing the state of the silhouette from the foreground to the background. e is the degenerate amount of the silhouette, which is an integer greater than or equal to 0 (where e = 0 means that the degenerate process is not performed).

The shrinkage amount calculation unit 32 can calculate the shrinkage amount e of d> e by multiplying the calculation result of the expansion amount d by a predetermined coefficient or by performing a predetermined function calculation with the expansion amount d as a variable. At this time, it is desirable that the degeneracy amount e is set to d> e from the viewpoint of noise reduction. However, if the degenerate amount e is too large, there is a concern that the region that should originally be the foreground will be used as noise in the background. Therefore, in the present embodiment, the degeneracy amount e = 2 is fixedly set, and the expansion amount calculation unit 31 applies the camera parameter error Ec and the degeneracy amount e = 2 to the following equation (4) to calculate the expansion amount d. ..

Here, round is a function that rounds off to the nearest whole number. Econst is a constant for adjusting the expansion amount d and is set manually. According to the above equation (4), the expansion amount d is adjusted for each pixel by the magnitude of the camera parameter error Ec (u, v) for each pixel of each camera.

In this embodiment, the degenerate amount e is a constant, but the degenerate amount d may be a constant and the degenerate amount e may be a variable. In this case, in the above equation (4), the expansion amount d may have a negative value depending on the setting of Econst, but if the expansion amount d becomes a negative value, processing may be performed with d = 0. ..

Further, in the present embodiment, the expansion amount d and the degeneration amount e are calculated so that d> e from the viewpoint of noise reduction, but the present invention is not limited to this, and d <. It may be e or d = e. In this embodiment, the degeneration treatment and the expansion treatment are repeated at least once in the order.

FIG. 9 is a functional block diagram showing the configuration of the third embodiment of the subject silhouette extraction device 1, and the silhouette calculation unit 40 performs silhouette calculation by the background subtraction method using a background model. The silhouette extraction parameter calculation unit 30 calculates the parameters used in the silhouette calculation by the background subtraction method based on the estimation result of the parameter error.

In the present embodiment, the background subtraction threshold calculation unit 33 and the background model update rate calculation unit 34 are provided in the silhouette extraction parameter calculation unit 30, and the threshold T (u, v) and the background model used for foreground / background determination are provided as silhouette extraction parameters. It is characterized by adopting the update rate U (u, v) of.

The background subtraction threshold calculation unit 33 calculates the threshold T (u, v) based on, for example, the following equation (5). Here, T _min and T _max are the minimum threshold value and the maximum threshold value when determining the threshold value, respectively. E _max is a constant for controlling the amount of parameter change due to the camera parameter error E _c (u, v). These values are manually determined in consideration of the target scene and the like.

The background model update rate calculation unit 34 calculates the update rate U (u, v) based on, for example, the following equation (6). Here, U _min and U _max are the minimum update rate and the maximum update rate when the update rate is changed, respectively, and these values are manually determined in consideration of the target scene and the like.

According to the present embodiment, according to the above equation (5), the background difference threshold T (u, v) is set smaller as the camera parameter error Ec (u, v) becomes larger, so that each pixel can be easily determined as the foreground. .. As a result, since many pixels are determined to be the foreground, it is possible to obtain an effect close to the contour expansion of the first and second embodiments.

Further, according to the above equation (6), the update rate U (u, v) of the background model is set smaller as the camera parameter error Ec (u, v) becomes larger, so that the background model can be less likely to be updated. As a result, it is possible to suppress the effect that the background model is updated and the contour is cut.

In the silhouette calculation unit 40, the foreground extraction processing unit 44 extracts the foreground from the moving image by the background subtraction method using the background model. When the background is modeled with a single Gaussian distribution, the average Gaussian distribution up to the F frame for constructing a background model for a pixel is μ _F (u, v), as schematically shown in Figure 10. ), When the standard deviation is given by σ _F (u, v), the formula for calculating background subtraction is the following equation (7).

In the present embodiment, the pixels (u, v) satisfying the above conditional expression (7) are determined to be the background. _IF (u, v) is the brightness value of each pixel of the acquired moving image, z is a parameter that adjusts how many times the standard deviation is judged as the background, and the threshold value T (u, v) is the above equation. Calculated in (5).

The color space of the image used for determining background subtraction may be grayscale or may be implemented in a color space such as RGB or YUV, but if there are multiple color channels, all channels are processed independently. However, if all the colors satisfy the conditions for background, it is determined that the background is used.

The method for constructing the background model is not limited to the method using the simple Gaussian distribution, and the method for constructing the background model using the mixed Gaussian distribution disclosed in Non-Patent Document 3 and the method for constructing the background model for each pixel position disclosed in Non-Patent Document 8. A method of constructing a background model by keeping a specific number of past pixel samples may be adopted.

In the method of holding a specific number of pixel samples, the foreground / background is determined based on how many pixels are similar to the input pixels in the held sample. By moving it up and down, the mechanism of the present invention can be realized. Also, from the viewpoint of the update rate, since the process of replacing the pixel sample held in a specific number with the pixel of the current frame is performed with a certain probability, it can be realized by increasing or decreasing this probability (= update rate).

The background model update unit 45 updates the Gaussian distribution mean μF (u, v) and standard deviation σF (u, v) of the background model at each frame by the following equations (8), (9), and (10).

In this way, in this embodiment, by gradually updating the background model for each frame, the background is dynamically updated for a scene where the background color changes little by little according to changes in sunshine, etc., and the accuracy is improved. It is possible to realize excellent silhouette extraction.

In the present embodiment, the camera parameter error Ec (u, v) has been described as being estimated in pixel units for each camera, but the present invention is not limited to this, and is estimated in camera units. Is also good. In this case, the determination threshold value and the update rate of the background model are also calculated for each camera.

Further, in the third embodiment described above, it has been described that the silhouette extraction parameters calculated based on the estimation error of the camera parameters are the background subtraction threshold T (u, v) and the background model update rate U (u, v). However, the present invention is not limited to this, and may be only the background subtraction threshold T (u, v) as in the fourth embodiment shown in FIG. 11, or the first aspect shown in FIG. 5 As in the embodiment, only the background model update rate U (u, v) may be used.

Further, when the first or second embodiment and the third to fifth embodiments are appropriately combined, the expansion amount d of the silhouette is calculated based on the estimation error of the camera parameters, and the silhouette is further extracted by the background subtraction method. The background subtraction threshold T (u, v) and the background model update rate U (u, v) may be calculated.

1 ... subject silhouette extractor, 2 ... 3D model generator, 3 ... moving image DB, 10 ... moving image acquisition unit, 20 ... camera parameter error estimation unit, 21 ... camera parameter estimation unit, 22 ... reprojection error estimation unit, 30 ... Silhouette extraction parameter calculation unit, 31 ... Expansion amount calculation unit, 32 ... Decrease amount calculation unit, 33 ... Background difference threshold calculation unit, 34 ... Background model update rate calculation unit, 40 ... Silhouette calculation unit, 41 ... Silhouette extraction unit , 42 ... Silhouette contour expansion processing unit, 43 ... Silhouette contour reduction processing unit, 44 ... Foreground extraction processing unit, 45 ... Background model update unit, 100 ... 3D model generation system

Claims (15)

In the subject silhouette extractor that extracts the silhouette of the subject from the moving image taken by the camera,
A means of estimating camera parameter error based on moving images,
A means to calculate silhouette extraction parameters based on the estimation result of camera parameter error,
A subject silhouette extraction device comprising a means for calculating a subject silhouette based on the silhouette extraction parameter. The means for estimating the camera parameter error is
A means of estimating camera parameters based on the coordinates of the corresponding pixel pair of the 3D space taken by the camera and its moving image,
A means for calculating the reprojection error of the camera parameter is provided.
The subject silhouette extraction device according to claim 1, wherein the reprojection error represents a camera parameter error. Further provided with means for optimizing the camera parameters to minimize the reprojection error.
The subject silhouette extraction device according to claim 2, wherein the camera parameter error is represented by the reprojection error of the camera parameter after the optimization. The subject silhouette extraction device according to any one of claims 1 to 3, wherein the means for estimating the camera parameter error is to estimate the camera parameter error for each camera. The means for estimating the camera parameter error is to calculate the reprojection error of each pixel pair for each camera.
The subject silhouette extraction device according to claim 4, wherein the camera parameter error is represented by the average value of the reprojection errors of each pixel pair for each camera. The subject silhouette extraction device according to any one of claims 1 to 3, wherein the means for estimating the camera parameter error is to estimate the camera parameter error for each pixel of each camera. The means for estimating the camera parameter error is to calculate the reprojection error of each pixel pair for each camera.
The subject silhouette extraction device according to claim 6, wherein the camera parameter error is represented by the reprojection error of the pixel pair having the closest distance for each pixel of each camera. The means for estimating the camera parameter error is to calculate the reprojection error of each pixel pair for each camera.
The subject silhouette extraction device according to claim 6, wherein the camera parameter error is represented by a function of the reprojection error of each pixel pair and the distance to the pixel pair for each pixel of each camera. The means for calculating the silhouette extraction parameter includes means for calculating the expansion amount of the subject silhouette based on the camera parameter error.
The means for calculating the subject silhouette includes contour expansion processing means for expanding the contour of the subject silhouette according to the expansion amount.
The subject silhouette extraction device according to any one of claims 1 to 8, wherein the means for calculating the expansion amount is calculated so that the expansion amount increases as the camera parameter error increases. The means for calculating the subject silhouette includes a contour shrinking processing means for shrinking the contour of the subject silhouette by a predetermined amount.
The subject silhouette extraction device according to claim 9, wherein each process of degeneration and expansion is repeated at least once in the order with respect to the contour of the subject silhouette. The means for calculating the subject silhouette is to calculate the subject silhouette by the background subtraction method.
The means for calculating the silhouette extraction parameter is provided with a means for calculating a threshold value for discriminating between the foreground area and the background area.
The subject silhouette extraction device according to any one of claims 1 to 10, wherein a threshold value is calculated so that the larger the camera parameter error is, the easier it is for each pixel to be identified in the foreground region. The means for calculating the subject silhouette is to calculate the subject silhouette by the background subtraction method.
The means for calculating the silhouette extraction parameter is provided with a means for calculating the update rate of the background model.
The subject silhouette extraction device according to any one of claims 1 to 10, wherein the larger the camera parameter error is, the lower the update rate is calculated. The means for calculating the subject silhouette is to calculate the subject silhouette by the background subtraction method.
The means for calculating the silhouette extraction parameters is
A means of calculating a threshold for distinguishing between the foreground area and the background area,
Equipped with a means to calculate the update rate of the background model,
The subject silhouette according to any one of claims 1 to 10, wherein a threshold value is calculated so that the larger the camera parameter error is, the easier it is for each pixel to be identified in the foreground region, and the update rate is calculated to a lower value. Extractor. In the subject silhouette extraction method, in which the computer extracts the silhouette of the subject from the moving image taken by the camera.
Estimate camera parameter error based on moving image
Calculate the silhouette extraction parameters based on the estimation result of the camera parameter error,
A method for extracting a subject silhouette, which comprises calculating a subject silhouette based on the silhouette extraction parameter. In the subject silhouette extraction program that extracts the silhouette of the subject from the moving image taken by the camera
The procedure for estimating the camera parameter error based on the moving image, and
The procedure to calculate the silhouette extraction parameters based on the estimation result of the camera parameter error, and
A subject silhouette extraction program that causes a computer to execute a procedure for calculating a subject silhouette based on the silhouette extraction parameters.

Images