JP7063837B2 - Area extraction device and program - Google Patents

Description

The present invention relates to a region extraction device and a program capable of appropriately detecting a region of an object even if there is occlusion for a multi-viewpoint image.

The subject contour (silhouette) is an extraction of the area of a person, animal, or other general object in the image as a binary mask image, and corresponds to the boundary of the object in which the silhouette boundary is shown. It becomes. In the contour extraction technique, the object in the image is separated from the background by separating the foreground / background. Contour extraction techniques are broad and can be classified into two types: those that use deep learning and those that model the background.

As a deep learning method, in Mask R-CNN (Mask R-CNN) of Non-Patent Document 1, it is common to perform instance separation, that is, to detect an object mask in an image and give an identification result of each object mask. Framework is provided. This mask R-CNN can be further generalized and used for other tasks such as person pose estimation, object detection in a rectangular bounding box, or keypoint detection.

In the background modeling method, the background is mathematically modeled by statistical processing, and the parameters of the model distribution are estimated from the distribution of pixel values in a small range. Non-Patent Document 2 utilizes the assumption that the distribution of pixel values in the background follows a normal distribution distributed with a small amplitude (variance) around a specific value (mean value).

He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017, October). Mask r-cnn. In Computer Vision (ICCV), 2017 IEEE International Conference on (pp. 2980-2988) .IEEE. Kenji Terabayashi, Kazunobu Umeda, Moro Alessandro, "Real-time adaptive threshold processing of background subtraction using person tracking information", Institute of Electrical Engineers of Japan General Industry Study Group, GID-09-17, pp.89-90 (2009). Laurentini A. The visual hull concept for silhouette-based image understanding. IEEE Transactions on pattern analysis and machine intelligence, 1994, 16 (2): 150-162. Lorensen, William E., and Harvey E. Cline. "Marching cubes: A high resolution 3D surface construction algorithm." ACM siggraph computer graphics. Vol. 21. No. 4. ACM, 1987.

However, the contour extraction of the above-mentioned conventional technique has the following problems.

The mask R-CNN of Non-Patent Document 1 is robust against noise and situations where objects are crowded with each other, but the mask shape cannot be obtained as a precise one, and only a coarse-shaped mask can be obtained. I couldn't. This coarsely shaped mask was unsuitable for application to free-viewpoint video technology, which uses contour masks to generate 3D models of objects in multi-viewpoint video. In addition, Mask R-CNN is vulnerable to occlusion (shielding) and generally could not properly separate objects with occlusion.

The background modeling method of Non-Patent Document 2 functions properly in an environment where the light source is appropriately controlled, but obtains sufficient accuracy in an environment where the light source environment changes dynamically such as outdoors. I couldn't. Furthermore, when there is a shadow area of the object, it cannot be detected correctly because it moves with the object.

In view of the above problems of the prior art, it is an object of the present invention to provide a region extraction device and a program capable of appropriately detecting a region of an object even if there is occlusion for a multi-viewpoint image.

In order to achieve the above object, the present invention is a region extraction device, which is a divided portion that applies region division to an image of each viewpoint of a multi-viewpoint image to obtain a first foreground mask of each viewpoint, and a divided portion of each viewpoint. A back-projection unit that obtains a three-dimensional model by applying the visual volume crossing method that relaxes the crossing determination to the first foreground mask, and the three-dimensional model is projected onto the image plane of each viewpoint of the multi-viewpoint image of each viewpoint. A background that takes the distance map into consideration for the projection unit that obtains the second foreground mask, the calculation unit that calculates the distance map from the second foreground mask of each viewpoint to the foreground, and the second foreground mask of each viewpoint. It is characterized by comprising an improved portion for obtaining a third foreground mask of each viewpoint as a result of region extraction from the multi-viewpoint image by applying the difference method to extract the foreground. Further, the program is characterized in that the computer functions as the area extraction device.

According to the present invention, the area of an object can be appropriately detected even if there is occlusion by using the visual volume crossing method in which the crossing determination is relaxed and the background subtraction method in consideration of the distance map.

It is a functional block of the area extraction apparatus which concerns on one Embodiment. It is a flowchart of the operation of the area extraction apparatus for the image which concerns on one Embodiment. It is a flowchart of the operation of the area extraction apparatus which concerns on one Embodiment with respect to the multi-viewpoint image of arbitrary time on the image. It is a figure which shows the schematic example of the camera arrangement which obtains a multi-viewpoint image. It is a figure for showing the schematic example of the area extraction as a rough foreground mask by a division part. The results of the visual volume crossing method of the existing method and the visual volume crossing method under the relaxation conditions in the present embodiment are shown in comparison. As a schematic example of the background image for the background subtraction method, it is a figure which shows the background image prepared for the image of FIG. It is a figure which shows the schematic example of exclusive OR (XOR) with respect to the example of this foreground mask and the previous foreground mask. It is a figure which shows the schematic example of the processing data of a region extraction apparatus. It is a figure which shows the hardware configuration in a general computer device.

FIG. 1 is a functional block of the area extraction device 10 according to the embodiment. The area extraction device 10 includes a division unit 1, a back projection unit 2, a projection unit 3, a calculation unit 4, an improvement unit 5, and a prediction unit 6.

FIG. 2 is a flowchart of the operation of the area extraction device 10 for the video according to the embodiment. The area extraction device 10 can read a multi-viewpoint video and extract a region of the multi-viewpoint image as a frame image at each time t. FIG. 2 processes each time frame of the video in this way. It represents the flow. Hereinafter, each step in FIG. 2 will be described.

When the flow of FIG. 2 is started with the current time t as the initial time t = 1 of the multi-viewpoint video, in step S10, the region extraction device 10 reads the multi-viewpoint image as the frame image of the current time t from the multi-viewpoint video as input. In addition, after acquiring the parameter of the current time t predicted at the immediately preceding time t-1 (statistical parameter for applying the background subtraction method in the improvement unit 5), the process proceeds to step S11.

In step S11, the area extraction device 10 performs the processing described in detail later in the division unit 1, the back projection unit 2, the projection unit 3, the calculation unit 4, and the improvement unit 5, so that the improvement unit 5 has multiple viewpoints at the current time t. After outputting the foreground mask of the image, proceed to step S12.

In step S12, the prediction unit 6 predicts the parameters (statistical parameters of the background subtraction method) at the next time t + 1 using the output results of the improvement unit 5 in the immediately preceding step S11, and then proceeds to step S13. move on. In step S13, the current time t is updated to the next time t + 1, and then the process returns to step S10, and the flow of FIG. 2 is repeated for each time t of the multi-viewpoint video in the same manner as described above.

The parameter of the current time t predicted at the immediately preceding time t-1 acquired in step S10 is the parameter predicted by the prediction unit 6 in step S12 (time t-1) immediately before reaching the step S10. When t = 1 (initial time), as will be described later in the description of the prediction unit 6, parameters that are calculated in advance from a predetermined background image for applying the background subtraction method may be used.

As shown in steps S10 to S12 of FIG. 2, the area extraction device 10 performs common processing on the multi-viewpoint image as a frame at each time t of the multi-viewpoint video. FIG. 3 is a flowchart of the operation of the region extraction device 10 according to the embodiment for this common process, that is, a multi-viewpoint image at an arbitrary time t on the video.

Hereinafter, the details of the processing contents of each functional unit of the area extraction device 10 will be described while explaining each step of FIG.

(0) Multi-viewpoint video as input data The multi-viewpoint video as input data to the area extraction device 10 is obtained by shooting a common scene with a plurality of (at least two) cameras arranged from different camera viewpoints. It is something that can be done. FIG. 4 is a diagram showing a situation in which 10 cameras C1 to C10 surround a common field F (for example, a field F where sports are performed) as a schematic example of a camera arrangement for obtaining a multi-viewpoint image. .. Time synchronization shall be performed for the images of each camera of the multi-viewpoint image. In addition, each camera shall be calibrated independently and the camera parameters shall be obtained.

As shown as lines L1 and L2 in FIG. 1, the multi-viewpoint image at each time in the multi-viewpoint video as input data is input to the division unit 1 and the improvement unit 5 of the region extraction device 10.

(1) Step S1 ... In step S1, the division unit 1 obtains a sparse foreground mask. In step S1, the division unit 1 divides the area of the input multi-viewpoint image at the current time t from each camera viewpoint. , Obtaining a foreground mask in a rough state, outputting this foreground mask to the back projection unit 2, and then proceeding to step S2.

Specifically, the divided portion 1 can obtain a rough foreground mask by using the deep learning (convolutional neural network) -based mask R-CNN of Non-Patent Document 1 described above. (It is known that the mask R-CNN can be detected robustly even if there is a shadow.) As the training data, for example, a COCO data set may be used to learn the network parameters. This learning data can detect more than 20 types of objects, but in this embodiment, it is set in advance to be extracted as the foreground in the input multi-viewpoint image (video) instead of detecting all of them. It is possible to extract only a predetermined type of object as a foreground mask. For example, when the multi-viewpoint image is a volleyball image as a sports image, only the player and the ball may be extracted. Also in the following, as an explanatory example, the case where the player and the ball are extracted in the volleyball video will be used.

FIG. 5 is a schematic example of region extraction as a rough foreground mask by the division portion 1, in which the player PL exists in the field F in the image P of a certain camera viewpoint, and the player PL is shielded by the horizontal band B of the volleyball net. It is shown that when occlusion occurs, the athlete PL is detected as a divided region such as regions R1 and R2, as shown as data D. The location BN shown in FIG. 5 will be referred to in the explanation of the improvement unit 5 described later.

In the division unit 1, when the foreground mask is extracted by the mask R-CNN, only those larger than the area size threshold value according to the target type to be extracted may be selected and used as the extraction result. For example, the area detected as a player is limited to the area larger than the first threshold value TH1, the area detected as a ball is limited to the area larger than the second threshold value TH2, and the area not satisfying the threshold condition is not included in the extraction result. You can do it. The threshold value may be set for each of the vertical width, the horizontal width, and the number of area pixels, or may be set for only a part of them.

(2) Step S2 ... The back projection unit 2 back-projects the foreground mask to obtain a 3D model under relaxation conditions. In step S2, the back projection unit 2 sets the foreground mask at each viewpoint of the multi-viewpoint image in the field F (actual). By back-projecting into the 3D space of (space) and applying the visual volume crossing method, a 3D model of the object is obtained, this 3D model is output to the projection unit 3, and then the process proceeds to step S3. ..

Here, the visual volume crossing method as an existing method is as disclosed in Non-Patent Document 3 and the like described above, and is expressed by the following equation (1). That is, a 3D model of the object can be obtained by taking the intersection of the visual volume (Visual Cone) of the area of the identified object in the foreground mask for each camera viewpoint.

Here, I is the total ID set of the masks of each camera, i ∈ I is the ID of the i-th camera mask, and V _i is the visual volume (Visual Cone) formed by the i-th camera mask (foreground). ). Spatial coordinates (x, y, z) (camera viewpoint) corresponding to the image coordinates (u, v) on the foreground mask by calculating the perspective projection matrix (camera matrix) using the camera parameters of the i-th camera. This visual volume V _i can be obtained by obtaining the spatial coordinates (x, y, z) on the ray extending from the pixel position (u, v). The product set of Eq. (1) is obtained in a voxel set in a three-dimensional space, and the product set VK (I) in the camera mask set I is obtained as a point of cloud in the space. By applying the marching cubes algorithm of Non-Patent Document 4, the visual volume can be obtained as a polygon mesh model in a three-dimensional space.

Also in this embodiment, the back projection unit 2 obtains a 3D model by performing the visual volume crossing method and polygon meshing, but at this time, the visual volume crossing method (and polygon meshing) of the existing method is applied as it is. Instead, the visual volume crossing method is applied under relaxed conditions. That is, while the equation (1) of the existing method obtained the intersection in the foreground masks of the images of all the camera viewpoints in the camera mask set I, in the present embodiment, for example, the following equation (1') The intersection judgment is relaxed as in. That is, when the number of camera viewpoints of the camera mask set I is N, not all of them (N), but r * N or more corresponding cameras of a fixed ratio r (0 <r <1). The spatial region through which the visual volume V _i of the viewpoint passes is adopted as the relaxed intersection VK (I). In equation (1'), S (I) is a subset of the camera mask set I, and the number of elements is r * N.

FIG. 6 shows the results of the visual volume crossing method of the existing method and the visual volume crossing method under relaxed conditions in the present embodiment in comparison. The upper side is the existing method, and when there are 10 cameras as shown in Fig. 4, the result is shown as the result R3 by passing all 10 visual volumes, and a part is enlarged to the right as the result R30. ing. As a result, in R30, it can be seen that the player PL3 of the obtained 3D model is also divided due to the presence of the player area divided by the occlusion as shown in the data D of FIG. .. On the other hand, the lower side is the result R4 and the enlarged result R40 of a part thereof under the relaxation condition (for example, the condition that at least 8 out of 10 visual volumes need to pass) in the present embodiment as a inverse proportion, and the existing ones. In the method, the player who has been divided is not divided as the player PL4, and the 3D model is obtained.

The foreground mask used by the back projection unit 2 to obtain the 3D model under relaxation conditions in step S2 is obtained in the immediately preceding step. When the immediately preceding step is step S1 (when step S2 is from step S1 to step S2), the rough foreground mask output by the dividing unit 1 in the immediately preceding step S1 is used as the input data of the back projection unit 2. Use as. On the other hand, if the immediately preceding step is step S7, which will be described later (when step S2 is when returning from step S7 to step S2), the foreground mask output by the improvement unit 5 in the immediately preceding step S7 (line in FIG. 1). The foreground mask of the "intermediate version" described later, which is shown as L3), is used as the input data of the back projection unit 2.

(3) Step S3 ... The projection unit 3 projects the 3D model onto each image of the multi-viewpoint image to obtain a foreground mask. In step S3, the projection unit 3 was obtained by the back projection unit 2 in the immediately preceding step S2. By projecting the 3D model onto each image plane of the multi-viewpoint image, a foreground mask at each viewpoint is obtained, and this foreground mask is output to the calculation unit 4 and the improvement unit 5, and then the process proceeds to step S4. The flow of this output is as shown in FIG. 1 as lines L3 and L4, respectively.

Since the 3D model is obtained as a polygon mesh model in the back projection unit 2, the relationship of the perspective projection matrix used in the back projection unit 2 is used for each polygon element (planar element such as a triangle). The projection unit 3 projects the multi-viewpoint image onto the image of each viewpoint (projection to obtain the corresponding image coordinates (u, v) from the spatial coordinates (x, y, z) of the 3D model), and the foreground mask as the projection result. Can be obtained.

Here, since the back projection unit 2 obtains a 3D model under relaxation conditions as shown in the schematic example in FIG. 6, the foreground mask obtained by projecting this 3D model on the projection unit 3 is shown in FIG. It is assumed that the division due to occlusion of the same object (division of the player PL into areas R1 and R2) as illustrated in the above is eliminated or alleviated.

For each polygon (face element) of the polygon mesh in the 3D model, the direction when turning around the side that surrounds the polygon (the turning direction when the face element is viewed face up on the 3D model) is defined. Only polygon elements that have the same rotation direction on the corresponding projected 2D (two-dimensional image) (polygons that are ostensibly in the image of the corresponding viewpoint) that rotate the surface element on 3D. It may be a projection target as. As a result, it may be omitted to project an invisible part (a part that does not need to be projected in the image of the corresponding viewpoint) on the back side of the 3D model.

(4) Step S4 ... The calculation unit 4 calculates the distance map using the foreground mask. In step S4, the calculation unit 4 calculates the distance map using the foreground mask obtained from the projection unit 3 in the immediately preceding step S3. , Output this distance map to the improvement section 5, and then proceed to step S5.

This distance map is obtained as a map of the shortest distance d (u, v) to a point belonging to the foreground mask obtained from the projection unit 3 at the pixel position (u, v) of the image plane of each viewpoint of the multi-viewpoint image. be able to. If the position (u, v) belongs to the foreground, d (u, v) = 0 may be set.

(5) Step S5 ... The improvement unit 5 improves the foreground mask using the distance map. In step S5, a variable threshold value (changes for each pixel position) using the distance map obtained from the calculation unit 4 in the immediately preceding step S4. By applying the background subtraction method (Uru threshold value), it is improved by extracting only the area determined to be the foreground by this background subtraction method from the foreground mask obtained from the projection unit 3 in the immediately preceding step S3. After obtaining the foreground mask, proceed to step S6.

That is, the improvement unit 5 excludes the portion determined to be the background from the pixel information by the background subtraction method from the foreground area by "shaving" the foreground area in the foreground mask obtained by projection by the projection unit 3. To obtain an improved foreground mask. Therefore, the following inclusion relationship is established. (Note that the inclusion relationship "⊂" includes cases where they are equal.)
Foreground mask obtained by improvement unit 5 ⊂ Foreground mask obtained by projection unit 3

Here, since pixel information is required to apply the background subtraction method, the improvement unit 5 refers to the input multi-viewpoint image as shown by the line L2 in FIG. 1 and obtains this pixel information. get.

The background subtraction method applied in the improved section 5 can determine the foreground / background by the following equations (2) and (3).

Equation (2) leaves pixel p = (u, v) (each pixel in the foreground mask obtained by the projection unit 3) as the foreground or deletes it as the background in a certain viewpoint image at the current time t of the multi-viewpoint image. It is an expression for determining whether to do so, and the pixel p such that M _{t, p} = 0 is used as the background, and the pixel p such that M _{t, p} = 1 is used as the foreground. (Under this definition, M _{t, p} is the foreground mask obtained by the improved part 5.) It _{, p} are the pixel values at this position p, and as shown by the line L2, the original multi-viewpoint image ( You can get this value by referring to the frame at time t).

μ _{t, p} and o't _{, p} are the mean and standard deviation in the background pixel distribution (Gaussian model) at the pixel position p in the viewpoint image targeted at the current time t of the multi-viewpoint image. Regarding the initial time t = 1, these averages and standard deviations are calculated for a predetermined model background image as prior knowledge, and the values at the subsequent time t ≧ 2 are described in the step described later. The value predicted by the prediction unit 6 in S9 may be used.

Dis _{t and p} are the values of the distance map obtained by the calculation unit 4 at the pixel position p in the viewpoint image targeted at the current time t of the multi-viewpoint video. In this embodiment, unlike the normal background subtraction method, the threshold value is determined by using the function value f (Dis _{t, p} ) corresponding to the distance map Dis _{t , p} (variable based on the distance map Dis t, p). Threshold judgment) is performed. As the function f, for example, as shown in Eq. (3), by using a function proportional to the distance map Dis _{t, p} by a constant k> 0, the larger the distance, the more the pixel value range determined to be the background. By enlarging, it becomes possible to appropriately extract the foreground according to the distance map obtained by the calculation unit 4 (the information of the foreground mask in the projection unit 3 is reflected). For the function f of the distances Dis _{t and p} , other increasing functions (non-decreasing functions) whose values are non-negative may be used in addition to the equation (3).

Model as prior knowledge for obtaining the mean and standard deviation of the pixel distribution model (Gaussian distribution model) at the initial time t = 1 for applying the background subtraction method The background image is extracted as the foreground in the multi-viewpoint video. It suffices to prepare something other than the target as a background image. At this time, although it is not the target to be extracted as the foreground, the target that can generate occlusion with respect to the foreground may not be included in the background image.

For example, when the player PL (or a ball (not shown) is extracted as the foreground from the multi-viewpoint image as shown in the image P of FIG. 5, the foreground of the volleyball net including the horizontal band B shown in FIG. The background BG (volleyball net that generates occlusion) as schematically shown in FIG. 7 by not including it in the background image because it is something other than the object to be extracted as and causes occlusion. It suffices to prepare in advance a background BG) composed only of the field F that does not include the above.

In this embodiment, by applying the background subtraction method based on the judgments of the equations (2) and (3), occlusion is generated even if occlusion occurs in the player PL set as the extraction target. As a result, only a partial part of the horizontal band B that overlaps with the player PL is extracted as the foreground, so that the occlusion of the player PL is eliminated. That is, the volleyball net including the horizontal band B as shown in FIG. 5 is all extracted as the foreground by the normal background subtraction method, but in the present embodiment, the equations (2) and (3) are used. By the judgment, only the part where the occlusion near the player PL to be extracted (the part near the player PL surrounded by the dotted ellipse in FIG. 5 BN) is extracted as the foreground together with the player PL, and as a result, the occlusion is performed. It is possible to extract the player P who has solved the problem.

(6) Step S6 ... Convergence determination of the improved foreground mask obtained in the improvement unit 5 In step S6, the improvement unit 5 determines whether or not the foreground mask obtained in the immediately preceding step S5 has converged. do. Specifically, assuming that the foreground mask obtained in the immediately preceding step S5 is the nth foreground mask FG (n) (nth in the loop processing structure by returning from step S7 to step S2 in FIG. 3), the previous time. The difference from the foreground mask FG (n-1) is the number of pixels of these exclusive ORs (XOR) set XOR (FG (n), FG (n-1)) | XOR (FG (n), FG ( It is evaluated as n-1)) |, and if the number of pixels is less than the predetermined threshold, it is considered to have converged and proceeds to step S8, and if it is equal to or more than the predetermined threshold, it is assumed that it has not converged and proceeds to step S7.

FIG. 8 shows a schematic example of the exclusive OR (XOR) for the example of the current foreground mask FG (n) and the previous example of the foreground mask FG (n-1).

When the number of loop processes in FIG. 3 is n = 1 (first time), that is, when the multi-viewpoint image at the current time t reaches this step S6 for the first time, the previous foreground mask FG (1) as a comparison target. -1) = FG (0) may be the foreground mask output by the projection unit 3 in step S3.

Alternatively, as another embodiment, when the number of loop processes in FIG. 3 is n = 1 (first time), a negative determination may always be made in step S6, and the process may always proceed to step S7.

When the number of loop processes n in FIG. 3 is n ≧ 2, the output of the improvement unit 5 in step S5 (nth loop process) immediately before this step S6 is output to the foreground mask FG (nth) of this time (nth). It may be set to n), and the output of the improvement unit 5 in step S5 in the loop processing n-1th time before that may be the foreground mask FG (n-1) in the previous (n-1) time.

(7) Step S7
In step S7, the convergence test was not obtained in step S6 immediately before, so as shown by line L5 in FIG. 1, the improvement unit 5 has changed the foreground mask FG (n) this time to the intermediate version instead of the final version. After outputting to the back projection unit 2 as if it is, the process returns to step S2.

(8) Step S8
In step S8, since the convergence test was obtained in step S6 immediately before, the improvement unit 5 made the final version (multi-viewpoint video) of this foreground mask FG (n) as shown by line L6 in FIG. Output as the final foreground extraction result in the frame image at the current time t), and then proceed to step S9.

(9) Step S9 ... Predicting the parameters of the background subtraction method at the next time t + 1 in the prediction unit 6. In step S9, notification that the improvement unit 5 has obtained the final result at the current time t as shown by the line L7. After receiving this, the prediction unit 7 predicts the parameters (mean and standard deviation) for the improvement unit 5 to apply the background subtraction method at the next time t + 1, and outputs it to the improvement unit 5. , The flow of FIG. 3 ends. Note that step S9 in FIG. 3 corresponds to step S12 in FIG.

Specifically, the prediction unit 7 may update the parameters by any existing method for updating the parameters of the background subtraction method on the video time series, and may be updated by, for example, the following equations (4) and (5). l _t is a predetermined weight for updating by the weighted sum, and may be set in the range of 0 <l _t <1. In addition, the parameters of the part that was used as the background at the current time t are updated by the following equations (4) and (5), and the background parameter that was used as the background at the time before that is used for the part that was used as the foreground. You can take over as it is.

FIG. 9 is a diagram showing a schematic example of the processing data of the region extraction device 10. In FIG. 9, the data D1, D2, D3, and D4 are separated, and the processing result for the image at the common time of a certain common viewpoint (the image of the volleyball game in which the player and the ball are detected) (FIG. 3). As the common n-th processing result in the loop processing), the foreground mask obtained by the division unit 1, the foreground mask obtained by the projection unit 3, the distance map obtained by the calculation unit 4, and the foreground mask obtained by the improvement unit 5, respectively. And. Further, on the left side of the data D1, D2, D3, D4, the data D10, D20, D30, D40 are shown as an enlarged view of the upper left region of the data D1, D2, D3, D4.

It can be confirmed that the occlusion defect of the player area seen in the data D10 has been repaired in the data D20, and that the extra background existing in the data D20 has disappeared in the data D40. ..

As described above, according to the embodiment of the present invention, the area of the object can be appropriately detected even if there is occlusion by using the visual volume crossing method in which the crossing determination is relaxed and the background subtraction method in consideration of the distance map. Can be done. The supplementary matters will be described below.

(1) The loop processing shown in FIG. 3 may not be performed. That is, step S6 may be omitted and the process immediately proceeds from step S5 to step S8, and the first output in the improvement unit 5 may be the final result.

(2) FIG. 10 is a diagram showing a hardware configuration in a general computer device 70, and the area extraction device 10 can be realized as one or more computer devices 70 having such a configuration. The computer device 70 is a CPU (central processing unit) 71 that executes a predetermined instruction, and a dedicated processor 72 (GPU (graphic calculation device)) that executes a part or all of the execution instructions of the CPU 71 on behalf of the CPU 71 or in cooperation with the CPU 71. And deep learning dedicated processor, etc.), RAM73 as the main storage device that provides a work area for the CPU71 and the dedicated processor 72, ROM74 as the auxiliary storage device, communication interface 75, display 76, camera 77, mouse, keyboard, touch panel, etc. It includes an input interface 78 that accepts user input, and a bus B for exchanging data between them.

Each part of the area extraction device 10 can be realized by a CPU 71 and / or a dedicated processor 72 that reads and executes a predetermined program corresponding to the function of each part from the ROM 74. Here, when the shooting-related processing is performed, the camera 77 further operates in conjunction with the display, and when the display-related processing is performed, the display 76 further operates in conjunction with the communication-related data transmission / reception. When the processing of is performed, the communication interface 75 further operates in conjunction with it.

For example, the input multi-viewpoint video may be acquired from the network via the communication interface 75. The final result obtained by the improvement unit 5 may be displayed on the display 76. When the area extraction device 10 is realized as a system by two or more computer devices 70, information necessary for each process may be transmitted and received via a network.

10 ... area extraction device, 1 ... division part, 2 ... back projection part, 3 ... projection part, 4 ... calculation part, 5 ... improvement part

Claims (7)

The division part that applies the area division to the image of each viewpoint of the multi-viewpoint image to obtain the first foreground mask of each viewpoint, and the division part.
A back projection unit for obtaining a three-dimensional model by applying the visual volume crossing method in which the crossing determination is relaxed to the first foreground mask of each viewpoint.
A projection unit that projects the three-dimensional model onto the image plane of each viewpoint of the multi-viewpoint image to obtain a second foreground mask for each viewpoint.
A calculation unit that calculates a distance map to the foreground from the second foreground mask of each viewpoint,
By applying the background subtraction method in consideration of the distance map to the second foreground mask of each viewpoint and extracting the foreground, the third foreground mask of each viewpoint as a result of region extraction from the multi-viewpoint image. With the improved part to get ,
In the back projection unit, the visual volume in which the intersection determination is relaxed by obtaining the three-dimensional model as a region where the intersection determination can be obtained in the first foreground mask of at least a part of the viewpoints of all the viewpoints of the multi-viewpoint image. Apply the crossing method,
In the improved section, by applying the background subtraction method, when it is determined that the difference in pixel values from the predefined background model is within a predetermined range at each pixel position, it is determined as the background.
In the determination, the larger the value of the distance of the distance map at the pixel position, the wider the predetermined range for the determination, so that the background subtraction method in consideration of the distance map is applied. Area extraction device. The region extraction device according to claim 1, wherein the division portion uses a convolutional neural network learned in advance to perform region division including identification of a region type. The region extraction device according to claim 2, wherein in the division portion, only a specific region type is used as the foreground to obtain the first foreground mask. The calculation unit is characterized in that the distance map is calculated by giving the shortest distance from the pixel position to the foreground in the second foreground mask as the value of the distance map at each pixel position. The region extraction device according to any one of 1 to 3 . The back-projection unit, the projection unit, the calculation unit, and the improvement unit perform each repetition process in this order, and the third foreground mask obtained by the processing of the improvement unit in the iteration process is viewed in the next back-projection unit. Used as an application target of the volume crossing method
The improved unit is characterized in that the third foreground mask obtained in the present time of the iterative process is compared with the third foreground mask obtained in the previous time, and the iterative process is completed when it is determined that there is no difference. The area extraction device according to any one of claims 1 to 4 . The fifth aspect of the present invention is characterized in that the improvement unit obtains an exclusive OR of the third foreground mask obtained in the present time of the iterative process and the third foreground mask obtained in the previous time, and makes the comparison. The area extraction device described. A program characterized in that the computer functions as the area extraction device according to any one of claims 1 to 6 .

Images