JP2023080290A - Information processing apparatus, control method of the same and program - Google Patents

Abstract

To provide an information processing apparatus capable of generating high quality 3D shape data and while suppressing missing of 3D shape data of an object.SOLUTION: The information processing apparatus generates three-dimensional shape data of an object on the basis of a plurality of pick-up images acquired by multiple cameras. The information processing apparatus includes: judgment means that, for each of predetermined elements that constitutes 3D space, determines whether the condition is met that the number of cameras in which the pixel or area corresponding to the specified element included in the area of the object in the pick-up images of the multiple cameras is equal to or less than the first threshold; and generating means that generates three-dimensional shape data of the object that includes predetermined elements that are not determined to meet the conditions by the judgment means.SELECTED DRAWING: Figure 4

Description

The present invention relates to an information processing device, a control method for an information processing device, and a program.

2. Description of the Related Art In recent years, attention has been paid to a technique of installing a plurality of cameras at different positions, synchronously capturing images from multiple viewpoints, and generating virtual viewpoint content using the captured images from the multiple viewpoints. According to technology that generates virtual viewpoint content from multiple viewpoint images, for example, highlight scenes of soccer or basketball can be viewed from various angles, giving the user a high sense of presence compared to normal images. be able to. The generation and browsing of virtual viewpoint content based on multi-viewpoint images is achieved by collecting images captured by multiple cameras in an image processing unit such as a server, and performing processing such as 3D model generation and rendering by the image processing unit. It can be realized by transmitting to the user terminal.

As a representative method for generating a detailed three-dimensional model, a calculation method called the visual volume intersection method has been proposed. Non-Patent Document 1 discloses extracting a region (silhouette) of a target object (foreground) from a multi-viewpoint image and generating a three-dimensional model (three-dimensional shape data) by the visual volume intersection method.

Laurenni A: "The Visual Hull Concept for Silhouette-Based Image Understanding", IEEE Transcriptions Pattern Analysis and Machine Intelligence, Vol.16, No.2, pp.150-162, Feb.1994

However, in the technique described in Non-Patent Document 1, when another object such as a structure overlaps the target object, or when the target object and the background are similar in color, one of the generated three-dimensional shape data There is a risk that parts may be missing.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for suppressing the lack of 3D shape data of an object and generating high-quality 3D shape data.

An information processing apparatus according to the present invention that achieves the above object includes:
An information processing device that generates three-dimensional shape data of an object based on a plurality of captured images captured by a plurality of cameras,
For each predetermined element constituting the three-dimensional space, the number of cameras including pixels or areas corresponding to the predetermined element in the area of the object in the captured image among the plurality of cameras is equal to or less than a first threshold. Determination means for determining whether or not the condition that there is is met;
generation means for generating three-dimensional shape data of the object including a predetermined element that is not determined by the determination means to match the condition;
characterized by comprising

According to the present invention, it is possible to suppress loss of 3D shape data of an object and generate high-quality 3D shape data.

FIG. 4 is a diagram showing an example of voxels forming a three-dimensional model according to one embodiment; 1 is a diagram showing a configuration example of a virtual viewpoint image generation system including a 3D model generation device according to an embodiment; FIG. FIG. 2 is a diagram showing an example of camera arrangement in a virtual viewpoint image generation system according to an embodiment; (a) A diagram showing an example of the functional configuration of the 3D model generation device according to the first embodiment, (b) A diagram showing an example of the hardware configuration of the 3D model generation device according to the first embodiment. 4 is a flowchart showing the procedure of processing performed by the 3D model generation device according to the first embodiment; FIG. 4 is a diagram showing an example of captured images captured by a plurality of cameras according to one embodiment of the present invention; FIG. 4 is a diagram showing an example of a structure mask image according to one embodiment of the present invention; FIG. 4 is a diagram showing an example of a foreground mask image according to one embodiment of the present invention; FIG. 4 is a diagram showing an example of an integrated mask image obtained by integrating a foreground mask image and a structure mask image according to one embodiment of the present invention; FIG. 4 is a diagram showing a voxel space to be used for generating a three-dimensional model of the stadium system according to the first embodiment; The figure which shows True Count/False Count which concerns on 1st Embodiment. FIG. 7 is a diagram showing an example of a three-dimensional model generated by applying the False Count threshold determination according to the first embodiment; FIG. 7 is a diagram showing an example of a three-dimensional model generated by applying the threshold determination of false count and the threshold determination of true count according to the first embodiment; The figure showing a three-dimensional model when omission arises. The figure which shows an example of the functional structure of the three-dimensional model generation apparatus which concerns on 2nd Embodiment. 9 is a flowchart showing the procedure of processing performed by a 3D model generation device according to the second embodiment; FIG. 11 is a diagram showing an example of camera arrangement and foreground in a virtual viewpoint image generation system according to a second embodiment; The figure which shows True/False Count which concerns on 2nd Embodiment. The figure showing the functional block of the three-dimensional model generation apparatus which concerns on 3rd Embodiment. The figure showing the processing flow of the three-dimensional model generation apparatus which concerns on 3rd Embodiment. FIG. 11 is a diagram showing True/False Counts with and without weight addition according to the third embodiment; FIG. 4 is a diagram showing an overview of three-dimensional model generation by the visual volume intersection method; The figure which shows an example of a functional structure of the three-dimensional model generation apparatus which concerns on 4th Embodiment. 10 is a flow chart showing the procedure of processing performed by a 3D model generation device according to the fourth embodiment; The figure which shows False Count/Structure which concerns on 4th Embodiment.

Hereinafter, embodiments will be described with reference to the drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

(First embodiment)
In the first embodiment, for each voxel (partial region) in a space to be determined for generating a 3D model (3D shape data) of a target object (foreground, object), It is determined whether or not the number of cameras whose positions (pixels or regions) corresponding to voxels are included in the foreground mask image representing the foreground region is equal to or less than a threshold. Then, an example of generating a foreground 3D model by removing the voxel (partial region) from the 3D model candidates when the number is equal to or less than the threshold will be described.

Specifically, for each of a plurality of partial regions that form a three-dimensional space, the number of cameras whose foreground region that indicates the region of the target object in the captured image among the plurality of cameras includes the partial region is the first threshold value. It is determined whether or not the following conditions are met. Then, a three-dimensional model of the target object including the partial regions that are not determined to meet the conditions is generated.

<Expression method of 3D model>
FIG. 1(a) shows a single cubic voxel. FIG. 1(b) shows a voxel set representing a target space for 3D model generation. As shown in FIG. 1(b), a voxel is a minute partial area that constitutes a three-dimensional space. FIG. 1(c) shows an example of generating a voxel set of a three-dimensional model of a quadrangular pyramid by removing voxels other than the quadrangular pyramid region from the set of voxels in the target space shown in FIG. 1(b). In the present invention, an example in which the three-dimensional space and the three-dimensional model are composed of cubic voxels will be described, but they may be composed of point groups and the like. The term "three-dimensional model" as used herein refers to data representing a three-dimensional shape.

<System configuration>
FIG. 2 is a block diagram showing a configuration example of a virtual viewpoint image generation system including the 3D model generation device according to the first embodiment. The virtual viewpoint image generation system 1 includes a camera array 11 including a plurality of cameras 10a-10z, a control device 12, a foreground/background separation device 13, a 3D model generation device 14, and a rendering device 15.

The camera array 11 is a photographing device group including a plurality of cameras 10a to 10z, photographs a subject from various angles, and outputs images to the foreground/background separation device 13 and the control device 12. FIG. The cameras 10a to 10z, the foreground/background separation device 13, and the control device 12 are assumed to be connected in a star topology. good. The camera array 11 is arranged around the stadium, for example, as shown in FIG. 3, and synchronously captures images from various angles toward a point of interest on the field common to all cameras. However, a plurality of gazing points may be set, such as a gazing point to which half of the cameras included in the camera array 11 are directed and another gazing point to which the remaining half is directed.

Here, the foreground is a predetermined target object (a subject for which a 3D model is to be generated based on a captured image) that can be viewed from any angle from a virtual viewpoint. refers to a person existing on the field of On the other hand, the background is an area other than the foreground, and in this embodiment, refers to the entire stadium (field, spectator seats, etc.). However, the foreground and background are not limited to these examples. Also, the virtual viewpoint images in this embodiment include not only images representing views from specifiable viewpoints, but also general images representing views from virtual viewpoints where no camera is installed.

The control device 12 calculates camera parameters indicating the positions and orientations of the cameras 10a to 10z from the images captured synchronously by the camera array 11, and outputs the calculated camera parameters to the three-dimensional model generation device . Here, the camera parameters are composed of extrinsic parameters and intrinsic parameters. The extrinsic parameters consist of a rotation matrix and a translation matrix, and represent the position and orientation of the camera. The internal parameters include information such as the focal length and optical center of the camera, and indicate the angle of view of the camera, the size of the imaging sensor, and the like.

The process of calculating camera parameters is called calibration. Camera parameters are, for example, a correspondence relationship between a point in a three-dimensional world coordinate system obtained using multiple images of a specific pattern such as a checkerboard photographed by a camera and the corresponding two-dimensional point. It can be obtained by using

The control device 12 calculates a structure mask image indicating a structure region that may overlap in front of the foreground among the images captured by the cameras 10a to 10z, and uses the information of the calculated structure mask image. Output. In this embodiment, the structure is a stationary object installed in the shooting target space. For example, a soccer goal is treated as a structure. It becomes a mask image. Note that the structural area that may overlap in front of the foreground means that there is a possibility that the foreground object will be blocked when photographed from at least one camera's photographing direction.

The foreground/background separation device 13 identifies, as the foreground, an area in which a person exists on the field and other background areas from images captured by a plurality of cameras input from the camera array 11, and identifies the foreground area. output the foreground mask image shown. As a method of identifying the foreground area, a method of identifying an area having a difference between a background image stored in advance and a captured image as the foreground area, a method of identifying an area of a moving object as the foreground area, and the like can be used.

Here, the mask image is a reference image representing a specific portion to be extracted from the captured image, and is a binary image represented by 0s and 1s. For example, the foreground mask image indicates an area in the captured image in which the foreground of, for example, a player exists, and has the same resolution as the captured image, with pixels indicating the foreground area being 1 and pixels other than the foreground being 0. is. However, the format of the mask image is not limited to this, and any information indicating the area of a specific object in the captured image may be used.

The 3D model generation device 14 has a function as an information processing device that generates a 3D model using a plurality of captured images captured by a plurality of cameras. First, the camera parameters and structure mask image information are received from the control device 12 , and the foreground mask image is received from the foreground background separation device 13 . Then, the three-dimensional model generation device 14 integrates the structure mask image and the foreground mask image to generate an integrated mask image indicating the integrated area. Furthermore, based on the number of cameras whose respective voxels in the space for which the 3D model of the foreground is to be generated is not included in the integrated mask image, and the number of cameras whose respective voxels are included in the foreground mask image, each voxel is removed. Based on the voxels remaining after the voxels determined to be removed are removed, a three-dimensional model of the foreground is generated by, for example, the visual volume intersection method, and is output to the rendering device 15 .

Now, with reference to FIG. 22, the basic principle of the visual volume intersection method will be described. As shown in FIG. 22A, when a target object C is photographed, a mask image Da representing a two-dimensional silhouette of the target object is obtained on a photographing surface S. As shown in FIG. A cone extending in three-dimensional space is conceivable so as to pass through each point on the contour of the mask image from the projection center Pa of the camera. This cone is called the "visual volume" of the object by the corresponding camera, and the visual volume Va is shown in FIG. 22(b). Furthermore, as shown in FIG. 22(c), a three-dimensional model of the target object can be obtained by obtaining a common area of a plurality of visual volumes, that is, intersection of the visual volumes.

The rendering device 15 receives the 3D model from the 3D model generation device 14 and an image showing the foreground from the foreground/background separation device 13 . In addition, the positional relationship between the image showing the foreground and the 3D model is determined from the camera parameters, and the foreground image corresponding to the 3D model is pasted to colorize the 3D model. Generate. Note that the virtual viewpoint image may include a background image. That is, the rendering device 15 may generate a virtual viewpoint image of the background and foreground viewed from the set viewpoint by setting a background model, a foreground model, and a viewpoint position in a three-dimensional space. .

<Functional configuration of 3D model generation device>
Next, with reference to FIG. 4A, the functional configuration of the three-dimensional model generation device according to this embodiment will be described. The 3D model generation device 14 includes a reception unit 100, a structure mask storage unit 101, a camera parameter storage unit 102, a mask integration unit 103, a coordinate conversion unit 104, a mask inside/outside determination unit 105, a threshold value setting unit 106, and a foreground model generation unit. 107 and an output unit 108 . The function of each processing unit is realized by executing a computer program read from the ROM 1002 or the RAM 1003 by the CPU 1001, which will be described later with reference to FIG. 4B.

The receiving unit 100 receives, from the control device 12 , camera parameters of each camera that constitutes the camera array 11 and a structure mask image that indicates the region of the structure. Further, the receiving unit 100 receives an image captured by each camera of the camera array 11 and a foreground mask image representing a foreground area in the image from the foreground/background separation device 13 each time the image is captured.

The structure mask storage unit 101 stores the structure mask image received by the reception unit 100 . The structure mask image is a fixed image according to the position of the camera.

The camera parameter holding unit 102 holds, as camera parameters, extrinsic parameters indicating the position and/or orientation of each camera photographed by the camera array 11 and intrinsic parameters indicating the focal length and/or image size.

The mask integration unit 103 integrates the foreground mask image received from the foreground/background separation device 13 and the structure mask image stored in the structure mask storage unit 101 each time the camera array 11 captures an image. Generate a mask image. The details of the method for integrating the foreground mask image and the structure mask image will be described later.

The coordinate transformation unit 104 calculates the position and angle of view of each captured image in the world coordinate system based on the camera parameters held in the camera parameter storage unit 102, and determines which shooting area in the three-dimensional space each captured image corresponds to. Converts to information that indicates or represents

If the number of cameras in which each voxel in the target voxel space is included in the foreground mask image is equal to or less than a threshold, the mask inside/outside determination unit 105 determines to remove the voxel. Also, when the number of cameras whose voxels in the target voxel space are not included in the integrated mask image is equal to or greater than another threshold, it is determined that the voxels are removed.

A threshold value setting unit 106 sets each threshold value for determining whether or not to remove a voxel by the mask inside/outside determination unit 105 . This threshold may be set according to a user's operation on the 3D model generation device 14 or may be automatically set by the threshold setting unit 106 . A foreground model generation unit 107 removes voxels determined to be removed by the mask inside/outside determination unit 105 from voxels in the target voxel space, and generates a three-dimensional model based on the remaining voxels. Generate. The output unit 108 outputs the 3D model generated by the foreground model generation unit 107 to the rendering device 15 .

<Hardware Configuration of 3D Model Generation Device>
Next, an example of the hardware configuration of the 3D model generation device according to this embodiment will be described with reference to FIG. 4(b). The 3D model generation device 14 has a CPU 1001 , a ROM 1002 , a RAM 1003 , a storage device 1004 and a bus 1005 and is connected to an input device 1006 and a display device 1007 .

The CPU 1001 controls various operations by the above functional blocks of the 3D model generation device 14 according to this embodiment. The contents of the control are instructed by programs on the ROM 1002 and RAM 1003, which will be described later. The CPU 1001 can also run a plurality of computer programs in parallel. The ROM 1002 stores computer programs and data that store control procedures by the CPU 1001 . A RAM 1003 stores a control program for processing by the CPU 1001 and provides a work area for various data when the CPU 1001 executes various controls. The functions of program codes stored in recording media such as the ROM 1002 and RAM 1003 are implemented by reading and executing them by the CPU 1001, but any type of recording medium is acceptable.

The storage device 1004 can store various data and the like. The storage device 1004 has recording media such as hard disks, floppy disks, optical disks, magnetic disks, magneto-optical disks, magnetic tapes, non-volatile memory cards, etc., and drives for recording information by driving the recording media. The stored computer programs and data are called up on the RAM 1003 when necessary according to instructions from the keyboard or the like or instructions from various computer programs.

A bus 1005 is a data bus or the like connected to each component, and is used to realize communication between the components and to exchange information at high speed. The input device 1006 provides various input environments by the user. A keyboard, a mouse, and the like are conceivable for providing various input operation environments, but a touch panel, a stylus pen, and the like may also be used. The display device 1007 is composed of a liquid crystal display or the like, and displays to the user the states of various input operations, calculation results corresponding thereto, and the like. In addition, the configuration described above is an example, and the configuration is not limited to the described configuration.

<Processing>
FIG. 5 is a flow chart showing the procedure of processing performed by the 3D model generation device according to the present embodiment.

In S<b>201 , the receiving unit 100 receives the structure mask image of each camera constituting the camera array 11 from the control device 12 . Here, an example of the photographed image and the structure mask image will be described. FIG. 6 shows an example of five captured images captured by five cameras that form part of the camera array 11 . Here, there is one person on the field, and a goal as a structure exists on the field. Because there is a part of the person is hidden. FIG. 7 shows a structure mask image corresponding to each photographed image shown in FIG. A goal area, which is a structure, is shown as 1 (white), and a non-structure area is shown as a binary image of 0 (black).

In S<b>202 , the receiving unit 100 receives the foreground mask image representing the foreground area from the foreground/background separation device 13 . An example of the foreground mask image will now be described. FIG. 8 shows a foreground mask image corresponding to each photographed image shown in FIG. Since the foreground/background separation device 13 extracts an area that changes with time as the foreground area, a portion of the person hidden behind the goal is extracted as shown in FIGS. 8(b), 8(c), and 8(d). are not extracted as foreground regions. Also, in FIG. 8E, a part of the person's foot that did not change over time is not extracted as the foreground area.

In S203, the mask integration unit 103 integrates the structure mask image and the foreground mask image received in S201 and S202 to generate an integrated mask image. FIG. 9 shows an example of an integrated mask image resulting from integrating the structure mask image shown in FIG. 7 and the foreground mask image shown in FIG. The integrated mask image is calculated by ORing the foreground mask image and the structure mask image represented by binary values.

In S204, the mask inside/outside determination unit 105 selects one unselected voxel from within the target voxel space.

In S205, the mask inside/outside determination unit 105 counts the number of cameras in which one selected voxel is not included in the mask area of the integrated mask image of each camera (hereinafter referred to as False Count).

In S206, the mask inside/outside determination unit 105 determines whether False Count is equal to or greater than a threshold. If the False Count is greater than or equal to the threshold, it can be determined that the selected one voxel is neither the foreground nor a structure, so the process proceeds to S207. This allows many voxels that are clearly non-foreground to be removed. On the other hand, if the False Count is less than the threshold, it can be determined that the selected one voxel is the foreground or a structure, so the process proceeds to S208.

In S207, the foreground model generation unit 107 removes one selected voxel from the target voxel space. In S208, the mask inside/outside determination unit 105 counts the number of cameras in which one selected voxel is included in the mask region of the foreground mask image of each camera (hereinafter referred to as True Count).

In S209, the mask inside/outside determination unit 105 determines whether or not the True Count is equal to or less than another threshold. If the True Count is equal to or less than another threshold, it can be determined that the selected voxel is a structure, so the process proceeds to S207 to remove the selected voxel from the target voxel space. On the other hand, if the True Count exceeds another threshold, the selected one voxel can be determined as the foreground and is not removed from the target voxel space.

In S210, the mask inside/outside determination unit 105 determines whether or not the processing has been completed for all voxels in the target voxel space. If the processing has been completed for all voxels, the process proceeds to S211. On the other hand, if the processing has not been completed for all voxels, the process returns to S204 to select the next voxel from among the unselected voxels, and perform the same processing thereafter.

In S211, the foreground model generation unit 107 generates a three-dimensional model of the foreground using the remaining voxels after performing the voxel removal determination for the target voxel space.

In S<b>212 , the output unit 108 outputs the foreground 3D model generated by the foreground model generation unit 107 to the rendering device 15 . The series of processes described above is performed for each frame captured by each camera.

Here, an example of generating a three-dimensional model will be described by taking as an example the virtual viewpoint image generating system that captures images of a stadium using 16 cameras shown in FIG. FIG. 10 is a diagram showing a voxel space for which a 3D model of the stadium system according to this embodiment is to be generated. A rectangular parallelepiped region indicated by a grid as a target region for three-dimensional model generation in the virtual viewpoint image generation system of FIG. 3 represents a target voxel space.

FIG. 11 shows the foreground in the virtual viewpoint image generation system shown in FIG. False Counts/True Counts of voxels and determination results for the head, goal, and other regions are shown. However, one camera failed to extract the foreground of the person's feet, and three cameras found that the person's head was hidden behind the goal structure. shall not be extracted.

In the determination of S206, if the False Count threshold is a fixed value of 10, voxels located in other regions are removed because the False Count is 16 and exceeds the threshold. As a result, a three-dimensional model composed of the foreground and the structure as shown in FIG. 12, for example, is generated. Here, FIG. 12 is a diagram showing an example of a three-dimensional model generated by applying the False Count threshold determination.

Furthermore, in the determination of S209, if the True Count threshold (other threshold) is a fixed value of 5, the voxels located in the goal area, which is a structure, are removed because the True Count is 0 and is equal to or less than the threshold. be. On the other hand, the voxels located in the human, human feet, and head regions have True Counts of 16, 15, and 13, respectively, which exceed the second threshold and are not removed.

That is, as shown in FIG. 11, the foreground (person), the partially undetected foreground (legs), and the foreground (head) hidden by the structure are determined to be voxels remaining, and the structure (goal) and non-foreground ( Other regions) will be determined to be voxel removed. Therefore, finally, a complete three-dimensional model of a person as shown in FIG. 13, for example, is generated from the set of voxels in the target space shown in FIG. Here, FIG. 13 is a diagram showing an example of a three-dimensional model generated by applying the False Count threshold determination and the True Count threshold determination.

On the other hand, FIG. 14 shows an example in which a three-dimensional model is generated by the visual volume intersection method using only the foreground mask image shown in FIG. Although FIG. 8(a) shows the whole person, in the photographed images shown in FIGS. 8(b), 8(c), and 8(d), part of the person's head is hidden by the goal of the structure. there is Furthermore, the person's feet are not extracted as the foreground in the photographed image shown in FIG. 8(e). Therefore, a part of the generated 3D model is also missing.

As described above, in the present embodiment, for each voxel in the target space for generating a three-dimensional model of the target object (foreground), the target voxel is included in the foreground mask image representing the foreground region. is equal to or less than a threshold value (threshold value of True Count), and if the number is equal to or less than the threshold value, the voxel is removed.

According to this embodiment, even if the foreground mask image showing the area of the target object (foreground) is missing, the missing of the 3D model of the target object (foreground) to be generated is avoided, and the quality of the 3D model is improved. can be made

In addition, when the foreground mask image and the structure mask image are integrated to generate an integrated mask image, and the number of cameras whose target voxels are not included in the integrated mask image is equal to or greater than a threshold value (False Count threshold) , to remove the voxel. As a result, many voxels that are clearly non-foreground can be removed, so that the speed of subsequent processing can be improved.

(Second embodiment)
In the present embodiment, an example will be described in which a three-dimensional model is generated by setting a threshold value based on the result of the inside/outside angle of view determination so that the foreground at a position away from the gaze point is not removed.

In the first embodiment, since it is not determined whether the voxel is within the shooting range (within the angle of view) of each camera, voxels that erroneously indicate the foreground are removed when they are outside the shooting range of many cameras. There is a possibility of doing so.

For example, in the virtual viewpoint image generation system of the stadium shown in FIG. Count becomes 3. At that time, if the threshold value of the True Count is 5, the voxel will be removed because it is less than the threshold value.

In this embodiment, by calculating the True Count threshold value based on the number of cameras that include the voxel within the imaging range (within the angle of view), even if the voxel is far from the gaze point, the voxel that erroneously indicates the foreground avoid removing the

<Functional configuration of 3D model generation device>
The functional configuration of the three-dimensional model generation device according to this embodiment will be described with reference to FIG. The 3D model generation device 14 according to this embodiment includes a receiving unit 100, a structure mask storage unit 101, a camera parameter storage unit 102, a mask integration unit 103, a coordinate conversion unit 104, a mask inside/outside determination unit 105, and a threshold value setting unit 106. , a foreground model generation unit 107 and an output unit 108 , an inside/outside angle of view determination unit 109 and a threshold calculation unit 110 are further provided. The function of each processing unit is realized by executing a computer program read from the ROM 1002 and the RAM 1003 by the CPU 1001 described with reference to FIG. 4B in the first embodiment. Also, since the configuration of the virtual viewpoint image generation system is the same as that of the first embodiment, description thereof will be omitted.

Since the functions of the receiving unit 100 to the output unit 108 of the 3D model generation device 14 are the same as those of the first embodiment, description thereof will be omitted.

The inside/outside angle of view determination unit 109 determines whether or not each voxel in the target voxel space is within the shooting range of each camera based on the camera parameters of each camera.

The threshold calculation unit 110 calculates a value obtained by multiplying the number of cameras determined to be within the shooting range by a predetermined ratio as the threshold of the True Count. For example, if the number of cameras that make a certain voxel within the shooting range is 5 and the predetermined ratio is 60%, the threshold value of True Count for that voxel is calculated as 3. The threshold calculated by the threshold calculation unit 110 is output to the threshold setting unit 106, and the threshold setting unit 106 sets the threshold input from the threshold setting unit 106 as the threshold of True Count.

Note that if the number of cameras that capture a voxel is less than a certain number, the accuracy of the generated 3D model will be low and processing will be unnecessary. A configuration may be adopted in which a threshold is set to a predetermined value for being less than.

<Processing>
FIG. 16 is a flowchart showing the procedure of processing performed by the 3D model generation device according to this embodiment. Since the processing of S301 to S304 is the same as the processing of S201 to S204 described with reference to FIG. 5 in the first embodiment, description thereof will be omitted.

In S305, the inside/outside angle of view determination unit 109 determines whether or not the one voxel selected in S304 is included within the angle of view of each camera, based on the camera parameters of each camera.

In S306, the mask inside/outside determination unit 105 determines the number of cameras in which the selected voxel is not included in the mask area of the integrated mask image of each camera and the selected voxel is included in the angle of view. (hereafter referred to as False Count).

Since the processing of S307-S309 is the same as the processing of S206-S208, the description thereof is omitted.

In S310, the threshold calculation unit 110 calculates the threshold of True Count based on the number of cameras including one selected voxel within the angle of view. The threshold setting unit 106 sets the threshold of True Count calculated by the threshold calculation unit 110 .

Since the processing of S311 to S314 is the same as the processing of S209 to S212, description thereof will be omitted. The above is the series of processing in FIG.

Here, FIG. 17 shows a virtual viewpoint image generation system in which a stadium including a foreground A close to the point of interest and a foreground B far from the point of interest is photographed by 16 cameras as in FIG. It is assumed that the foreground A is within the angle of view of all 16 cameras, and the foreground B is within the angle of view of only the three cameras 10k, 10l, and 10m in FIG.

FIG. 18 shows an example of False Count/True Count of voxels at the position of the foreground A close to the point of interest and voxels at the position of the foreground B far from the point of interest in the virtual viewpoint image generation system shown in FIG. show. The False Count threshold is a fixed value of 10, and the True Count threshold is 70% of the number of cameras including the voxel within the angle of view.

Since voxels located in the foreground A near the gaze point are included in the integrated mask image for all 16 cameras, there is no camera whose voxels are outside the integrated mask image. Therefore, the number of cameras whose voxels are outside the integrated mask image and within the angle of view is 0, and the False Count is 0.

Also, since the number of cameras including voxels located in the foreground A close to the gaze point within the angle of view is also 16, the threshold for the True Count is 11.2, which is 70% of the 16 cameras. Since the voxels located in the foreground A near the point of interest are within the foreground mask image for all cameras, the True Count is 16, and since the count value is equal to or greater than the threshold value (11.2), the voxels are not removed.

The voxel in the foreground B position far from the gaze point is outside the angle of view for 13 cameras (13 cameras excluding cameras 10k, 10l, and 10m), and is within the angle of view for 3 cameras (cameras 10k, 10l, and 10m). Become. Also, the voxels of the three cameras (cameras 10k, 10l, and 10m) are within the integrated mask image. Therefore, the number of cameras whose voxels are outside the integrated mask image and within the angle of view is zero, and the False Count is zero.

Also, since the number of cameras including voxels located in the foreground B far from the gaze point within the angle of view is three, the threshold value of the True Count is 2.1, which is 70% of the three cameras. A voxel located in the foreground B far from the gaze point is included in the foreground mask image by the three cameras, so the True Count is 3. Since the count value is equal to or greater than the threshold value (2.1), the voxel is not removed.

In this way, by setting the TrueCount threshold based on the number of cameras whose target voxels are included in the angle of view, the foreground, which is far from the point of interest and has a small number of cameras within the angle of view, is subject to tertiary The original model can be generated. Therefore, it is possible to generate a three-dimensional model in which omissions are suppressed even in the foreground far from the gaze point.

(Third Embodiment)
In this embodiment, the weight value is set based on the number of cameras whose target voxels are included in the structure mask image. Then, the value obtained by adding the number of cameras whose target voxels are included in the foreground mask image and the value obtained by multiplying the number of cameras whose target voxels are included in the structure mask image by the weight value is the True Count. If it is equal to or less than the threshold, it is determined to remove the voxel. An example of generating a three-dimensional model with no omissions even when the foreground is blocked by a structure using a large number of cameras will be described.

In the first and second embodiments, an example was described in which only cameras whose voxels are included in the foreground mask image are counted as the true count of each voxel. However, in that case, voxels in the foreground locations hidden by structures in many cameras may be removed without the TrueCount exceeding the threshold.

On the other hand, in this embodiment, if a target voxel is outside the foreground mask image but is included in the structure mask image, the voxel may be in the foreground. The loss of the foreground is avoided by adding the value obtained by multiplying the number of cameras determined to be included in the object mask image by the weight value to the True Count.

<Functional configuration of 3D model generation device>
The functional configuration of the three-dimensional model generation device according to this embodiment will be described with reference to FIG. The three-dimensional model generation device 14 according to this embodiment further includes a weight setting unit 111 in addition to the configuration of the three-dimensional model generation device of the second embodiment. The function of each processing unit is realized by executing a computer program read from the ROM 1002 and the RAM 1003 by the CPU 1001 described with reference to FIG. 4B in the first embodiment. Also, since the configuration of the virtual viewpoint image generation system is the same as that of the first embodiment, description thereof will be omitted.

The weight setting unit 111 sets, as a weight value per camera, a value to be added to the True Count when it is determined that the target voxel is within the structure mask image. This weight value is equivalent to the value indicating the likelihood of voxels located in the foreground, and in this embodiment the weight value per camera is set to 0.5. Then, a value obtained by multiplying the number of cameras for which the target voxel is determined to be within the structure mask image by a weight value of 0.5 per camera is added to the True Count.

<Processing>
FIG. 20 is a flowchart showing the procedure of processing performed by the 3D model generation device according to this embodiment.

The processing of S401-S404 is the same as the processing of S301-S304, and the processing of S405-S408 is the same as the processing of S306-S309. The processing of S409 is the same as the processing of S305, and the processing of S410 is the same as the processing of S310.

In S411, the mask inside/outside determination unit 105 counts the number of cameras in which one selected voxel is included in the mask area of the structure mask image of each camera.

In S412, the weight setting unit 111 multiplies the number of cameras included in the mask region of the structure mask image by a weight value of 0.5 per camera, and adds the value to the True Count calculated in S408. do. The processing of S413-S416 is the same as the processing of S311-S314. Thus, the series of processing in FIG. 20 ends.

Here, FIG. 21 shows an example of True Count without weight addition and an example of True Count with weight addition according to the present embodiment in a voxel located in a certain foreground region.

This voxel is within the angle of view of all 16 cameras, the number of cameras including the target voxel in the foreground mask image is 7, and the number of cameras including the target voxel in the structure mask image is 9. shall be a table. In this case, the number of cameras whose voxels are outside the integrated mask image is 0 (16 total cameras-7 cameras-9 cameras). Therefore, the number of cameras whose voxels are outside the integrated mask image and within the angle of view is 0, and the False Count is 0.

Without weight addition, the number of cameras including the target voxel in the foreground mask image is seven, so the True Count is seven. It is assumed that the True Count threshold is 70% of the number of cameras that include the target voxel within the angle of view. Then, since the threshold is 11.2 (16×0.7), True Count (7)<Threshold (11.2), and the True Count is less than or equal to the threshold, so the voxel is removed. .

On the other hand, with weight addition, the number of cameras including the target voxel in the foreground mask image is 7, so the True Count is similarly 7, and the weight value is added to this. Since the number of cameras including target voxels in the structure mask image is 9, and the weight value for each camera is 0.5, 9×0.5=4.5 is added as the weight value. . The True Count after adding the weight value is 11.5, and True Count (11.5)>Threshold (11.2), which exceeds the threshold, so the voxel is not removed as foreground.

In this embodiment, it is assumed that there is one structure, but if there are multiple different structures that may overlap with the foreground, different weight values are set for each type of structure mask image. , a value based on its weight value may be added to the True Count. For example, regarding the structure mask image of the electronic signboards installed to surround the playing field of the stadium, the electronic signboards are large and easily overlapped with the foreground, so there is a high possibility that the foreground will be included. Let the weight value be 0.5. For the goal structure mask image, the weight value per camera is set to 0.3. Since the electronic signboard is larger than the goal and does not have any gaps, it is highly likely that the signboard overlaps the foreground (person).

Also, different weight values may be set according to the voxel position, the scene, the size and shape of the mask area, the area of the stadium to be photographed, and the like.

As described above, in the present embodiment, threshold determination is performed after adding the weight True Count based on the number of cameras whose target voxels are included in the mask region of the structure mask image. As a result, even when the foreground is blocked by a structure with many cameras, it is possible to generate a 3D model with no omissions.

(Fourth embodiment)
In this embodiment, processing using the number of cameras included in the structure mask image instead of the number of cameras (True Count) included in the foreground mask image used in the first embodiment will be described.

In the first embodiment, for a three-dimensional model generated based on a foreground mask image and a structure mask image, the foreground mask image is updated each time so that the voxels constituting the three-dimensional model are included in the foreground mask image. processing may be complicated. In this embodiment, for a 3D model generated based on the foreground mask image and the structure mask image, the number of cameras included in the fixed structure mask image is counted to obtain a foreground 3D model that does not include the structure. Generate a model.

<Functional Configuration and Hardware Configuration of 3D Model Generating Device>
FIG. 23 is a diagram showing the configuration of a three-dimensional model generation device according to this embodiment. The configuration of the 3D model generation device 14 in this embodiment is substantially the same as in the first embodiment, and descriptions of blocks that perform the same processing will be omitted. The three-dimensional model generation device 14 according to the present embodiment includes an inside/outside mask determination unit 112 instead of the inside/outside mask determination unit 105 . The mask inside/outside determination unit 112 counts the number of cameras in which each voxel in the target voxel space is included inside or outside the mask of the integrated mask image and the structure mask image, and determines whether or not to remove the target voxel by threshold determination. or not, and output to the foreground model generation unit 107 . Also, the hardware configuration of the 3D model generation device 14 of the present embodiment is the same as that shown in FIG.

<Processing>
FIG. 24 is a flow chart showing the procedure of processing performed by the 3D model generation device 14 in this embodiment. S501 to S507 and S510 to S512 are the same as S201 to S207 and S210 to S212 described in the first embodiment with reference to FIG. will be mainly explained.

In S506, the mask inside/outside determination unit 112 determines whether or not the False Count is greater than or equal to the threshold. If the False Count is less than the threshold, it can be determined that the selected voxel is the foreground or a structure, so the process proceeds to S508.

In S508, the mask inside/outside determination unit 112 counts the number of cameras (hereinafter referred to as Structure Count) in which the pixel or area corresponding to one selected voxel is included in the mask area of the structure mask image of each camera. do.

In S509, the mask inside/outside determination unit 112 determines whether the Structure Count is greater than or equal to the threshold. If the Structure Count is equal to or greater than the threshold, it can be determined that the selected voxel is a structure, so the process proceeds to S507 to remove the selected voxel from the target voxel space. On the other hand, if the Structure Count is less than the threshold, the selected voxel can be determined as the foreground and is not removed from the target voxel space.

Here, an example of generating a three-dimensional model will be described by taking as an example the virtual viewpoint image generating system that captures images of a stadium using 16 cameras shown in FIG. FIG. 25 shows the foreground in the virtual viewpoint image generation system shown in FIG. False Counts/Structure Counts of voxels and determination results for the head, goal, and other regions of .

However, one camera failed to extract the foreground of the person's feet, and three cameras found that the person's head was hidden behind the goal structure. shall not be extracted.

In the determination of S504, if the False Count threshold is a fixed value of 10, voxels located in other regions excluding people, feet, heads, and goalposts of structures have a False Count of 16, exceeding the threshold. therefore removed. Here, FIG. 12 is a diagram showing an example of a three-dimensional model generated by applying the False Count threshold determination.

Furthermore, in the determination shown in S508, if the Structure Count threshold is the fixed value of 3, the voxels located in the goal region, which is a structure, have a Structure Count of 5, which is greater than or equal to the threshold, and therefore are removed. be. On the other hand, the voxels located in the regions of the person, the feet, and the head of the person have a Structure Count of 0, which is less than the threshold, and therefore are not removed. Therefore, a complete three-dimensional model of a person shown in FIG. 13 is generated.

By the above processing, it is possible to generate a 3D model without omission even when the foreground is blocked by the structure by threshold determination of the number of cameras (Structure Count) included in the structure mask.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

1: Virtual Viewpoint Video Generation System, 10: Camera, 11: Camera Array, 12: Control Device, 13: Foreground/Background Separation Device, 14: 3D Model Generation Device, 15: Rendering Device, 100: Receiver, 101: Structure object mask storage unit, 102: camera parameter storage unit, 103: mask integration unit, 104: coordinate conversion unit, 105: mask inside/outside determination unit, 106: threshold value setting unit, 107: foreground model generation unit, 108: output unit, 109 : view angle inside/outside determination unit 110: threshold calculation unit 111: weight setting unit 112: mask inside/outside determination unit

An information processing apparatus according to the present invention that achieves the above object includes:
a first acquisition means for acquiring first area information indicating an area of an object in a plurality of images obtained by photographing from a plurality of photographing directions;
a second acquisition means for acquiring second area information indicating an area of a structure in the plurality of images obtained by photographing from the plurality of photographing directions;
Three-dimensional shape data corresponding to the object is obtained based on the first area information indicating the area of the object obtained by the first obtaining means and the second area information indicating the area of the structure obtained by the second obtaining means. a generating means for generating,
The generating means generates the three-dimensional shape data corresponding to the object by removing the constituent elements corresponding to the structure from among the constituent elements that may constitute the three-dimensional shape data. .

Claims (20)

An information processing device that generates three-dimensional shape data of an object based on a plurality of captured images captured by a plurality of cameras,
For each predetermined element constituting the three-dimensional space, the number of cameras including pixels or areas corresponding to the predetermined element in the area of the object in the captured image among the plurality of cameras is equal to or less than a first threshold. Determination means for determining whether or not the condition that there is is met;
generation means for generating three-dimensional shape data of the object including a predetermined element that is not determined by the determination means to match the condition;
An information processing device comprising: further comprising integration means for generating an integrated area by integrating the area of the object and the structure area indicating the area of the structure;
When the number of cameras whose pixels or regions corresponding to the predetermined elements are not included in the integrated region is equal to or greater than a second threshold, the determining means determines the pixels or regions corresponding to the predetermined elements as objects. 2. The information processing apparatus according to claim 1, wherein it is determined to remove from the three-dimensional shape data candidates. When the number of cameras whose pixels or regions corresponding to the predetermined element are not included in the integrated region is less than the second threshold value, the determination means determines that the pixels or regions corresponding to the predetermined element are not included in the 3. The information processing apparatus according to claim 2, wherein it is determined whether or not the number of cameras included in the object area is equal to or less than the first threshold. The judging means determines that the number of cameras is the second, and the pixel or area corresponding to the predetermined element is not included in the integrated area and the pixel or area corresponding to the predetermined element is included in the angle of view. 4. The information processing apparatus according to claim 2 or 3, wherein, if the threshold is equal to or greater than the threshold of , it is determined that the pixel or region corresponding to the predetermined element is to be removed from the candidates for the three-dimensional shape data of the object. further comprising weight setting means for setting a weight value based on the number of cameras whose pixels or regions corresponding to the predetermined element are included in the structure region;
The determination means determines the number of cameras whose pixels or areas corresponding to the predetermined elements are included in the object area and the number of cameras whose pixels or areas corresponding to the predetermined elements are included in the structure area. Determining to remove the pixel or region corresponding to the predetermined element from the candidates for the three-dimensional shape data of the object when the value obtained by adding the value obtained by multiplying the weight value is equal to or less than the first threshold value. 5. The information processing apparatus according to any one of claims 2 to 4, characterized by: 6. The information processing apparatus according to claim 5, wherein said weight setting means sets said weight value according to the type or shape of a structure in said structure area. 7. The method according to any one of claims 1 to 6, further comprising threshold calculation means for calculating the first threshold based on the number of cameras whose angle of view includes pixels or areas corresponding to the predetermined element. The information processing device according to item 1. An information processing device that generates three-dimensional shape data of an object based on a plurality of captured images captured by a plurality of cameras,
For each predetermined element constituting the three-dimensional space, the number of cameras including pixels or areas corresponding to the predetermined element in the structure area in the photographed image among the plurality of cameras is equal to or greater than a first threshold. Determination means for determining whether or not the condition is met;
generation means for generating three-dimensional shape data of the object including a predetermined element that is not determined by the determination means to match the condition;
An information processing device comprising: further comprising integration means for generating an integrated area by integrating the object area and the structure area;
When the number of cameras whose pixels or regions corresponding to the predetermined elements are not included in the integrated region is equal to or greater than a second threshold, the determining means determines the pixels or regions corresponding to the predetermined elements as objects. 9. The information processing apparatus according to claim 8, wherein it is determined to remove from the three-dimensional shape data candidates. When the number of cameras whose pixels or regions corresponding to the predetermined element are not included in the integrated region is less than the second threshold value, the determination means determines that the pixels or regions corresponding to the predetermined element are not included in the 10. The information processing apparatus according to claim 9, wherein it is determined whether or not the number of cameras included in the structure area is equal to or greater than the first threshold. 11. The information processing apparatus according to any one of claims 1 to 10, wherein the object is a subject for which three-dimensional shape data is to be generated. 12. The information processing apparatus according to claim 1, further comprising output means for outputting the three-dimensional shape data of said object generated by said generating means to a rendering device. A control method for an information processing device that generates three-dimensional shape data of an object based on a plurality of captured images captured by a plurality of cameras, comprising:
For each predetermined element constituting the three-dimensional space, the number of cameras including pixels or areas corresponding to the predetermined element in the area of the object in the captured image among the plurality of cameras is equal to or less than a first threshold. A determination step of determining whether or not the condition that there is is met;
a generation step of generating three-dimensional shape data of the object including a predetermined element that is not determined to match the condition by the determination step;
A control method for an information processing device, comprising: further comprising an integration step of generating an integrated area by integrating the area of the object and a structure area indicating the area of the structure;
In the determination step, when the number of cameras whose pixels or regions corresponding to the predetermined element are not included in the integrated region is equal to or greater than a second threshold value, the pixels or regions corresponding to the predetermined element are defined as the object 14. The method of controlling an information processing apparatus according to claim 13, further comprising the step of determining to remove from the three-dimensional shape data candidates. In the determination step, if the number of cameras that do not include pixels or regions corresponding to the predetermined element in the integrated region is less than the second threshold value, pixels or regions corresponding to the predetermined element are not included in the 15. The method of controlling an information processing apparatus according to claim 14, further comprising determining whether or not the number of cameras included in the object area is equal to or less than the first threshold. In the determining step, the number of cameras for which the pixel or area corresponding to the predetermined element is not included in the integrated area and the pixel or area corresponding to the predetermined element is included in the angle of view is the second 16. The information processing apparatus according to claim 14 or 15, wherein the pixel or region corresponding to the predetermined element is determined to be removed from candidates for the three-dimensional shape data of the object when the control method. A control method for an information processing device that generates three-dimensional shape data of an object based on a plurality of captured images captured by a plurality of cameras, comprising:
For each predetermined element constituting the three-dimensional space, the number of cameras including pixels or areas corresponding to the predetermined element in the structure area in the photographed image among the plurality of cameras is equal to or greater than a first threshold. A determination step of determining whether or not the condition is met;
a generation step of generating three-dimensional shape data of the object including a predetermined element that is not determined to match the condition by the determination step;
A control method for an information processing device, comprising: further comprising an integration step of generating an integrated area by integrating the object area and the structure area;
In the determination step, when the number of cameras whose pixels or regions corresponding to the predetermined element are not included in the integrated region is equal to or greater than a second threshold value, the pixels or regions corresponding to the predetermined element are defined as the object 18. The control method for an information processing apparatus according to claim 17, wherein the control method of the information processing apparatus determines to remove from the three-dimensional shape data candidates. In the determination step, if the number of cameras that do not include pixels or regions corresponding to the predetermined element in the integrated region is less than the second threshold value, pixels or regions corresponding to the predetermined element are not included in the 18. The method of controlling an information processing apparatus according to claim 17, further comprising determining whether or not the number of cameras included in the structure area is equal to or greater than the first threshold. A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 12.

Images