JP2023026244A - Image generation apparatus, image generation method, and program - Google Patents

Abstract

To provide an image generation apparatus adapted to suppress a virtual visual-point image from lowering in quality.SOLUTION: An image generation apparatus is allowed to select one of three-dimensional shape information of a photographic subject generated based on silhouette information of the photographic subject obtained from a plurality of picked-up images obtained by imaging the photographic subject by a plurality of imaging devices and three-dimensional shape information of the photographic subject generated based on a posture of the photographic subject presumed from the plurality of picked-up images and generate a virtual visual-point image using the three-dimensional shape information selected.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to an image generation device, an image generation method, and a program.

A technique for generating virtual viewpoint images from virtual viewpoints designated by a user using a plurality of images (multi-viewpoint images) obtained by a plurality of imaging devices arranged at different positions has attracted attention. According to technology that generates virtual viewpoint content from multiple viewpoint images, for example, highlight scenes of soccer or basketball can be viewed from various angles, giving the user a high sense of realism compared to normal images. can do

Patent Document 1 describes a method for generating a virtual viewpoint image, in which a three-dimensional space forming a model is regarded as a point or voxel space, and the three-dimensional shape of the subject is acquired and colored from multiple viewpoint images. It describes how to be informed. In addition, Patent Document 2 describes a method for generating a virtual viewpoint image, in which a 3D shape is obtained and colored by adjusting a 3D object template model based on posture information of a subject obtained from multi-viewpoint images. method is described.

JP 2020-042666 A JP 2016-126425 A

The method of obtaining a three-dimensional shape based on silhouette information and the method of obtaining a three-dimensional shape based on posture information have different effects on the accuracy of three-dimensional shape estimation due to the imaging region and the state of the subject. A method based on silhouette information can generate a shape that conforms to the silhouette of a subject. There is a high possibility that it will not be possible to estimate correctly. Therefore, under these circumstances, the accuracy of 3D shape estimation is reduced. On the other hand, the method based on posture information can estimate the 3D shape with relatively high accuracy even in the above situation, but if the shape of the subject deviates from the 3D object template model, the 3D shape can be estimated correctly. It is highly likely that you won't be able to. Maintaining the accuracy of 3D shape estimation is important for maintaining the image quality of virtual viewpoint images.

According to one aspect of the present disclosure, there is provided a technique for suppressing deterioration in image quality of virtual viewpoint images.

An image generation device according to one aspect of the present invention has the following configuration. i.e.
a first generation means for generating three-dimensional shape information of a subject based on silhouette information of the subject obtained from a plurality of captured images of the subject captured by a plurality of imaging devices;
a second generation means for generating three-dimensional shape information of the subject based on the posture of the subject estimated from the plurality of captured images;
one of the three-dimensional shape information generated by the first generation means and the three-dimensional shape information generated by the second generation means for each subject included in the plurality of captured images, and displaying one of the three-dimensional shape information generated by the second generation means as a virtual viewpoint image; a selection means for selecting as three-dimensional shape information used to generate the
and generating means for generating a virtual viewpoint image using the type of three-dimensional shape information selected by the selecting means.

According to the present disclosure, it is possible to suppress deterioration in image quality of a virtual viewpoint image.

1 is a block diagram showing an example of the device configuration of an image processing system and an example of the functional configuration of an image generation device according to the first embodiment; FIG. FIG. 4 is a diagram for explaining three-dimensional shape information and posture information expressing the posture of a subject; FIG. 4 is a schematic diagram showing how an imaging region is imaged by a plurality of cameras; FIG. 2 is a block diagram showing a hardware configuration example of the image generation device according to the first embodiment; 4 is a flowchart showing virtual viewpoint image generation processing according to the first embodiment; FIG. 4 is a diagram for explaining selection of three-dimensional shape information by a selection unit; 10 is a flowchart showing processing for generating a virtual viewpoint image according to the second embodiment; FIG. 4 is a schematic diagram showing how an imaging region is imaged by a plurality of cameras;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

<First embodiment>
In the first embodiment, when generating a virtual viewpoint image, the virtual viewpoint image is generated while switching the shape estimation method to be used according to the number of imaging devices capable of imaging each region in the imaging region. An embodiment will be described. The image processing system of this embodiment includes a plurality of captured images (multi-viewpoint images) captured from different directions by a plurality of imaging devices, the state (position, orientation) of each imaging device, a specified virtual viewpoint (virtual viewpoint information), and the like. Based on this, a virtual viewpoint image representing the view from the virtual viewpoint is generated.

A plurality of imaging devices capture images of the imaging region from a plurality of directions. The imaging area is, for example, an area surrounded by a plane of a stadium where rugby or soccer is played and an arbitrary height. A plurality of imaging devices are installed in different positions and directions so as to surround such an imaging region, and perform imaging in synchronism. Note that the imaging devices do not have to be installed over the entire circumference of the imaging area. For example, a plurality of imaging devices may be installed only in a part of the imaging area due to restrictions on installation locations. In addition, there is no limit to the number of arranged imaging devices. For example, if the imaging area is a rugby stadium, several tens to hundreds of imaging devices may be installed around the stadium. Imaging devices with different angles of view, such as a telephoto camera and a wide-angle camera, may also be installed. For example, if a telephoto camera is used, the subject can be imaged with high resolution, so the resolution of the generated virtual viewpoint image is also improved. Also, for example, if a wide-angle camera is used, the range that can be captured by one camera is wide, so the number of cameras can be reduced. The imaging device is synchronized with one piece of time information in the real world, and imaging time information is added to each frame image of the captured moving image.

The state of the imaging device refers to the state of the imaging device, such as the position, orientation (orientation, imaging direction), focal length, optical center, distortion, and the like. The position and orientation (orientation, imaging direction) of the imaging device may be controlled by the imaging device itself, or may be controlled by a camera platform on which the imaging device is mounted. In the following description, the state of the imaging device will be described as a parameter of the imaging device (hereinafter referred to as a camera parameter). Camera parameters may include parameters for controlling another device such as a camera platform. Camera parameters relating to the position and orientation (orientation, imaging direction) of the imaging device are so-called external parameters, and camera parameters relating to the focal length, image center, and distortion of the imaging device are so-called internal parameters. The position and orientation of the imaging device are represented by a coordinate system having three axes orthogonal to one origin (hereinafter referred to as a world coordinate system).

A virtual viewpoint image is also called a free viewpoint image or an arbitrary viewpoint image. However, the virtual viewpoint image is not limited to the image corresponding to the viewpoint freely (arbitrarily) specified by the user. For example, the virtual viewpoint image includes an image corresponding to a viewpoint selected by the user from a plurality of candidates. Also, the designation of the virtual viewpoint may be performed by a user operation, or may be automatically performed based on the result of image analysis or the like. Also, in the following embodiments, the case where the virtual viewpoint image is a still image will be mainly described, but the virtual viewpoint image may be a moving image.

The virtual viewpoint information used to generate the virtual viewpoint image indicates the position and orientation of the virtual viewpoint. Specifically, the virtual viewpoint information includes parameters representing the three-dimensional position of the virtual viewpoint and parameters representing the orientation of the virtual viewpoint in the pan, tilt, and roll directions. Note that the content of the virtual viewpoint information is not limited to the above. For example, the parameters of the virtual viewpoint information may include parameters representing the size of the field of view (angle of view) of the virtual viewpoint. Also, the virtual viewpoint information may have parameters associated with multiple frames. In other words, the virtual viewpoint information may be information that has parameters corresponding to a plurality of frames that form a moving image of a virtual viewpoint image, and that indicates the position and orientation of the virtual viewpoint at each of a plurality of successive time points.

A virtual viewpoint image is generated, for example, by the following method. First, a plurality of captured images (multi-viewpoint images) are obtained by capturing images from different directions using a plurality of imaging devices. Next, a foreground image obtained by extracting a foreground area corresponding to a subject such as a person or a ball, and a background image obtained by extracting a background area other than the foreground area are obtained from the multi-viewpoint image. A foreground image and a background image have texture information (such as color information). A foreground model representing the three-dimensional shape of the subject and texture information for coloring the foreground model are generated based on the foreground image. Also, based on the background image, texture information for coloring the background model representing the three-dimensional shape of the background such as the stadium is generated. A virtual viewpoint image is generated by mapping texture information on the foreground model and the background model and performing rendering according to the virtual viewpoint indicated by the virtual viewpoint information. However, the method of generating a virtual viewpoint image is not limited to this, and various methods such as a method of generating a virtual viewpoint image by projective transformation of a captured image without using a foreground model or a background model can be used.

A foreground image is an image obtained by extracting a subject area (foreground area) from a captured image captured by an imaging device. A subject extracted as a foreground area (also called a foreground subject) generally refers to a dynamic subject (moving object) that moves (its position and shape can change) when images are captured from the same direction in time series. Point. For example, in a game, the subject is a person such as a player or a referee in the field where the game is played. Also, in the case of a ball game, in addition to a person, a ball or the like becomes a subject. In concerts and entertainment, singers, musicians, performers, moderators, and the like are subjects. Note that the foreground area can be interpreted as silhouette information of the foreground subject, and the foreground image is hereinafter referred to as foreground silhouette information in this embodiment.

A background image is an image of an area (background area) different from at least the foreground subject. Specifically, the background image is an image obtained by removing the foreground subject from the captured image. In addition, the background refers to an object to be imaged that is stationary or continues to be nearly stationary when imaged from the same direction in time series. Such imaging targets are, for example, stages of concerts and the like, stadiums where events such as competitions are held, structures such as goals used in ball games, fields, and the like. However, the background is at least a region different from the foreground subject, and the imaging target may include other objects in addition to the subject and background.

[Configuration of image processing system]
The configuration of the image processing system of the first embodiment will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of the device configuration of an image processing system and an example of the functional configuration of an image generation device 1 according to the first embodiment. The image processing system includes an image generation device 1 , an imaging system 2 , an operation device 3 and a display device 4 . An image generation device 1 is connected to an imaging system 2 , an operation device 3 and a display device 4 . The image generation device 1 acquires a captured image or foreground silhouette information and camera parameters from the imaging system 2 and acquires virtual viewpoint information from the operation device 3 . The image generation device 1 generates a virtual viewpoint image based on information obtained from the imaging system 2 and the operation device 3 . The generated virtual viewpoint image is output to the display device 4 . The imaging system 2 includes a plurality of imaging devices. Hereinafter, the imaging device included in the imaging system 2 will be referred to as a camera. Each camera of the imaging system 2 has an identification number for identifying the camera. The imaging system 2 may include functions other than imaging, such as a function of extracting foreground silhouette information from an image captured by a camera, and hardware (circuits, devices, etc.) for realizing these functions. The operation device 3 designates virtual viewpoint information for generating a virtual viewpoint image. The virtual viewpoint information is specified by a user (operator) using, for example, a joystick, jog dial, touch panel, keyboard, mouse, or the like. Note that designation of the virtual viewpoint information is not limited to user designation. For example, virtual viewpoint information may be automatically specified by recognizing a subject. The display device 4 acquires virtual viewpoint images from the image generation device 1 and outputs them using a display device such as a display.

Next, the functional configuration of the image generation device 1 will be described. The image generation device 1 has an information acquisition unit 101 , a first shape estimation unit 102 , a posture estimation unit 103 , a second shape estimation unit 104 , a selection unit 105 and an image generation unit 106 .

The information acquisition unit 101 acquires a plurality of captured images captured by a plurality of cameras of the imaging system 2 . The information acquisition unit 101 generates silhouette information of a foreground subject (foreground silhouette information) from the captured image acquired from the imaging system 2 . Note that the imaging system 2 may generate the foreground silhouette information. In that case, the information acquisition unit 101 acquires foreground silhouette information from the imaging system 2 . Furthermore, the information acquisition unit 101 acquires camera parameters of the imaging system 2 . Note that the information acquisition unit 101 may calculate camera parameters of the imaging system 2 . For example, the information acquisition unit 101 calculates corresponding points from images captured by a plurality of cameras, optimizes the corresponding points so that the error when projecting the corresponding points onto each camera is minimized, and calibrates each camera to obtain camera parameter Calculate Any existing method may be used to calibrate the camera. The camera parameters may be acquired synchronously with the captured image, may be acquired at the stage of preparation, or may be acquired asynchronously with the captured image as necessary. Furthermore, the information acquisition unit 101 acquires information on the number of cameras (hereinafter also referred to as effective cameras) capable of imaging each partial area in the imaging area. For example, based on the position, orientation, and angle of view information of each camera, for each of a plurality of partial areas that divide the imaging area, the number of cameras capable of photographing the partial area is determined. Note that the number of effective cameras for each partial area is provided to the selection unit 105 .

The first shape estimation unit 102 estimates three-dimensional shape information based on the foreground silhouette information of the foreground subject acquired by the information acquisition unit 101 and the camera parameters. Known methods such as the Visual Hull method and the Photo Hull method can be used for estimating the three-dimensional shape based on the foreground silhouette information. The three-dimensional shape information estimated by the first shape estimation unit 102 is hereinafter referred to as silhouette-based three-dimensional shape information.

A posture estimation unit 103 uses the captured image (or foreground silhouette information) acquired by the information acquisition unit 101 and camera parameters to estimate the posture of the foreground subject and generate posture information. The posture information is, for example, a bone model representing the skeleton of the target subject. For posture estimation, a known method such as a posture estimation method using deep learning can be used. Also, posture estimation section 103 acquires tracking information of the subject from the posture information. Subject tracking based on posture information is, for example, searching for a subject that minimizes the difference between each node of a certain subject's bone model and each node of bone models corresponding to all subjects one frame before. done.

Here, posture information according to this embodiment will be described with reference to FIG. FIG. 2 is a schematic diagram showing an example of a three-dimensional shape representing a subject and posture information. Although the three-dimensional shape and orientation information is information on a three-dimensional space, it is shown in a simplified two-dimensional image in FIG. 2 for explanation. A subject 201 is a subject on which a three-dimensional shape 202 (three-dimensional model) and posture information 203 are based. A three-dimensional shape 202 is represented by a point group, which is a set of points. A point group is a set of positional information (x, y, z) of points in a three-dimensional space and information indicating the size of the points. expressed. The posture information 203 is expressed as a bone model that is connected by lines connecting main nodes of the structure of the object and the nodes.

Returning to FIG. 1 , second shape estimation section 104 acquires the three-dimensional shape of the foreground subject based on the posture information of the foreground subject acquired by posture estimation section 103 . As a method of estimating three dimensions based on posture information, for example, a standard template model of the human body or a human body model that has been three-dimensionally scanned in advance is prepared, and a method of fitting the model to the posture information is used. obtain. The three-dimensional shape information estimated by the second shape estimation unit 104 is hereinafter referred to as posture-based three-dimensional shape information.

Here, how the accuracies of silhouette-based 3D shape estimation and attitude-based 3D shape estimation change with respect to the number of effective cameras for the subject will be described with reference to FIG. FIG. 3 is a schematic diagram showing how a plurality of cameras capture images of a subject existing in a certain imaging area. A plurality of cameras 302 (cameras 302a to 302f) are arranged so as to capture images of an imaging region 301 whose three-dimensional shape is to be estimated. A plurality of cameras 302 face the center of the imaging area 301 . A plurality of foreground silhouette information 303a to 303f are obtained from a plurality of captured images by a plurality of cameras 302. FIG. Also, in FIG. 3, a subject 304 exists in the center of the imaging area 301 and a subject 305 exists at the edge of the imaging area 301 . In silhouette-based 3D shape estimation, a 3D shape is obtained by projecting a point in 3D space onto each camera and determining whether the point is in the foreground. Therefore, silhouette-based three-dimensional shape estimation improves in accuracy as the number of cameras capable of capturing images of the subject increases. When silhouette-based three-dimensional shape estimation is performed in FIG. 3, there is a high possibility that the three-dimensional shape of the subject 304 can be obtained with high accuracy, while the three-dimensional shape accuracy of the subject 305 is highly likely to be low. On the other hand, posture-based 3D shape estimation obtains a 3D shape by deforming a model prepared in advance based on posture information, that is, a bone model representing the target skeleton. It is known that posture information can be obtained from a relatively small number of viewpoints, and the accuracy of 3D shape estimation based on posture information is almost constant regardless of the number of cameras capable of capturing images of the subject. Therefore, if three-dimensional shape estimation based on posture information is performed in FIG.

Next, how the accuracy of silhouette-based 3D shape estimation and the accuracy of attitude-based 3D shape estimation change according to the state of the subject will be described. Since the silhouette-based three-dimensional shape is estimated so that the silhouette of the subject observed by each camera remains unchanged, the three-dimensional shape can be estimated in line with the actual shape of the subject. On the other hand, pose-based 3D shape is estimated by fitting the underlying 3D shape to a bone model. For this reason, it is difficult to flexibly respond to changes in the shape of the subject. For example, if the appearance changes when the subject is holding a bat or ball, or wearing a hat, the estimation accuracy may decrease. .

Returning to FIG. 1, the selection unit 105 selects one of the silhouette-based three-dimensional shape information generated by the first shape estimation unit 102 and the pose-based three-dimensional shape information generated by the second shape estimation unit 104. is selected as the three-dimensional shape information used for generating the virtual viewpoint image. The selected three-dimensional shape information is output to image generator 106 . The selection unit 105 of the first embodiment selects two types, silhouette-based and posture-based, for each subject based on whether or not the number of cameras (effective cameras) capable of capturing a partial area in which each subject exists is equal to or greater than a certain number. 3D shape. The image generation unit 106 generates a virtual viewpoint image based on the captured image and camera parameters acquired by the information acquisition unit 101, the three-dimensional shape selected by the selection unit 105, and the virtual viewpoint information from the operation device 3. Generate.

FIG. 4 is a block diagram showing a hardware configuration example of the image generation device 1. As shown in FIG. The image generation device 1 has a CPU 401 , a ROM 402 , a RAM 403 , an auxiliary storage device 404 , a display section 405 , an operation section 406 , a communication I/F 407 , a GPU 408 and a bus 409 . The CPU 401 implements each function of the image generating apparatus 1 shown in FIG. 1 by controlling the entire image generating apparatus 1 using computer programs and data stored in the ROM 402 or RAM 403 . Note that the image generating apparatus 1 may have one or a plurality of dedicated hardware different from the CPU 401, and at least part of the processing by the CPU 401 may be executed by the dedicated hardware. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The ROM 402 stores programs that do not require modification. The RAM 403 temporarily stores programs and data supplied from the auxiliary storage device 404 and data externally supplied via the communication I/F 407 . The auxiliary storage device 404 is composed of, for example, a hard disk drive, and stores various data such as image data and audio data.

The GPU 408 is a processor dedicated to image processing, and can perform efficient computation by parallel processing more image data. Therefore, it is effective to use the GPU 408 when processing large-scale data, such as estimating the three-dimensional shape of a foreground object and generating a virtual viewpoint image. Therefore, in this embodiment, the GPU 408 can be used in addition to the CPU 401 for the processing by the first shape estimation unit 102, the second shape estimation unit 104, the image generation unit 106, and the like. However, such a configuration is not essential, and it is clear that the processes of the first shape estimation unit 102, the second shape estimation unit 104, and the image generation unit 106 may be realized by only one of the CPU 401 and the GPU 408.

The display unit 405 is composed of, for example, a liquid crystal display or an LED, and displays a GUI (Graphical User Interface) for the user to operate the image generating apparatus 1, and the like. The operation unit 406 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and inputs various instructions to the CPU 401 in response to user's operations. The CPU 401 operates as a display control unit that controls the display unit 405 and as an operation control unit that controls the operation unit 406 .

A communication I/F 407 is used for communication with an external device of the image generating device 1 . For example, when the image generating device 1 is connected to an external device by wire, a communication cable is connected to the communication I/F 407 . If the image generating device 1 has a function of wirelessly communicating with an external device, the communication I/F 407 has an antenna. A bus 409 connects each unit of the image generating apparatus 1 and transmits information.

In this embodiment, it is assumed that the display unit 405 and the operation unit 406 exist inside the image generating apparatus 1, but the present invention is not limited to this. At least one of the display unit 405 and the operation unit 406 may exist as a separate device outside the image generation device 1 . That is, the operation device 3 and the display device 4 shown in FIG.

[Generation processing of virtual viewpoint image]
Processing performed by the image generating apparatus 1 will be described using the flowchart shown in FIG.

In step S<b>501 , the information acquisition unit 101 acquires a plurality of captured images captured by a plurality of cameras and camera information of each camera from the imaging system 2 . Camera information includes, for example, camera parameters (position, orientation, angle of view, etc.) of each of a plurality of cameras.

The information acquisition unit 101 also acquires a plurality of captured images (multi-viewpoint images) from the imaging system 2 and generates foreground silhouette information. The foreground silhouette information can be generated by using a general method such as a background subtraction method that calculates the difference between a captured image of a subject and a background image captured before the start of a game when the subject does not exist. However, the method for generating foreground silhouette information is not limited to this. For example, the foreground silhouette information may be generated by extracting a recognized subject area using a method such as recognizing a human body. Note that the foreground silhouette information may be extracted by the imaging system 2, and the information acquisition unit 101 may acquire the extracted foreground silhouette information. In that case, the process of generating the foreground silhouette information of the subject in the information acquisition unit 101 can be omitted. Also, when the information acquisition unit 101 acquires a foreground image including texture information, foreground silhouette information can be generated by erasing the texture information. In this case, for example, if the foreground silhouette information is handled as 8-bit data, the pixel value of the area where the subject exists should be set to 255, and the pixel value of the other area should be set to 0. The acquired foreground silhouette information is output to first shape estimation section 102 and posture estimation section 103 . The information acquisition unit 101 also outputs the texture information of the foreground silhouette information to the image generation unit 106 .

Also, as described above, the information acquisition unit 101 may calculate camera parameters. Also, camera parameters do not have to be obtained/calculated each time a captured image is obtained, and may be obtained/calculated at least once before shape estimation. The acquired camera parameters are output to first shape estimation section 102 , posture estimation section 103 and image generation section 106 . Further, the information acquisition unit 101 acquires from the imaging system 2 information about the number of cameras (effective cameras) that can capture partial areas in the imaging area. The information acquisition unit 101 may calculate the number of effective cameras for each partial area in the imaging area based on the camera parameters acquired from the imaging system 2 . Note that the information on the number of effective cameras for each partial area in the imaging area does not need to be calculated each time a captured image is acquired, and can be calculated at least once before selecting the method for estimating the three-dimensional shape. good.

In step S502, the first shape estimation unit 102 generates a voxel set (silhouette-based three-dimensional shape information) that forms the silhouette shape of all foreground subjects based on the foreground silhouette information and camera parameters acquired in step S501. presume. As described above, a well-known method such as the Visual hull method or the Photo hull method can be used for estimating such a voxel set. The voxel size may be set in advance by the user using a GUI (Graphical User Interface), or may be set using a text file or the like. The first shape estimation unit 102 also calculates visibility information indicating whether each estimated point is visible from each camera.

In step S503, the posture estimation unit 103 estimates the postures of all foreground subjects and generates posture information based on the captured image or the foreground silhouette information acquired in step S501 and the camera parameters. For posture estimation, for example, a generally popular posture estimation method using deep learning can be used.

In step S504, the posture estimation unit 103 acquires subject tracking information. In this embodiment, the tracking information of a subject is obtained by searching for a subject that minimizes the difference between the orientation information of the subject of interest in the current frame and the orientation information of all the subjects one frame before. Here, the posture information is, for example, information indicating the position of each node of the bone model, and the difference in posture information is the sum of the absolute values of the differences in the positions of the nodes. The tracking of the subject may be estimated from the difference between the center of gravity of the bone model in the previous and subsequent frames, or may be performed by using person recognition using deep learning or the like. Posture estimation section 103 assigns identification information (subject ID) to each of the foreground subjects being tracked, and supplies it to first shape estimation section 102 and second shape estimation section 104 together with position information indicating the position of the foreground subject. . The first shape estimating unit 102 associates the silhouette-based three-dimensional shape information of each subject with the subject ID using the position information of the foreground subject.

In step S505, the second shape estimation unit 104 estimates the posture shape (posture-based three-dimensional shape information) of the foreground subject based on the posture information generated in step S503. For example, the subject is three-dimensionally scanned in advance to acquire a bone model as the three-dimensional model. The second shape estimation unit 104 obtains a three-dimensional shape by fitting the previously acquired bone model to the posture estimated in step S503. Note that the three-dimensional shape estimation method based on posture information is not limited to this. For example, the three-dimensional shape may be estimated by deforming a standard template model of the human body based on the estimated posture. The second shape estimation unit 104 uses the position information of the foreground object supplied from the posture estimation unit 103 to associate pose-based three-dimensional shape information of each object with the object ID.

In step S506, the selection unit 105 selects a subject of interest (hereinafter referred to as a subject of interest) from among all the subjects. In subsequent steps S507 to S510, the selection unit 105 selects one of the silhouette-based three-dimensional shape information and the orientation-based three-dimensional shape information as the three-dimensional shape information used to generate the virtual viewpoint image of the subject of interest. . In steps S507 to S509, it is determined whether or not conditions 1 to 3 are satisfied, and in step S510, a three-dimensional shape is selected according to these determination results.

In step S507, the selection unit 105 determines whether or not the number of cameras (effective cameras) capable of capturing an image of the partial area in which the subject of interest exists is less than a threshold (condition 1). The selection unit 105 first identifies a partial area where the subject of interest exists based on the position information of the foreground subject identified by the subject ID of the subject of interest. Then, the selection unit 105 acquires the number of effective cameras for the specified partial area from the information acquisition unit 101 and compares it with the threshold. It should be noted that the threshold used here is an index representing how much the accuracy of the silhouette shape is relied upon. Therefore, the threshold value is determined based on how many cameras are considered sufficient for silhouette shape accuracy. For example, if the silhouette shape of the subject seen from eight or more cameras is sufficient for accuracy, the threshold is set to eight.

In step S508, the selection unit 105 determines whether or not the three-dimensional shape based on the orientation information was used for the subject of interest one frame before (condition 2). In step S509, the selection unit 105 determines whether or not the subject of interest was present outside the angle of view of the virtual viewpoint one frame before (condition 3). The angle of view of the virtual viewpoint can be specified based on virtual viewpoint information acquired from the operation device 3 . In step S510, the selection unit 105 selects a three-dimensional shape to be used for the subject of interest in the current frame based on the determination results of conditions 1 to 3 in steps S507 to S509.

Here, the types of three-dimensional shape information (silhouette-based, posture-based) selected according to the combination of determination results of conditions 1 to 3 will be described with reference to FIG. FIG. 6 shows that either the silhouette-based 3D shape information or the pose-based 3D shape information is selected for all combinations of determination results as to whether or not conditions 1, 2, and 3 are satisfied. Represents Ruka. In FIG. 6, Y indicates that the conditions are satisfied, and N indicates that the conditions are not satisfied. Therefore, in Condition 1, Y indicates that the number of valid cameras is less than the threshold, and N indicates otherwise. In Condition 2, Y indicates that the three-dimensional shape information selected for the subject of interest one frame before was posture-based three-dimensional shape information, and N indicates that otherwise. In the condition 3, Y indicates that the middle object one frame before was outside the virtual viewpoint image (outside the angle of view of the virtual viewpoint), and N indicates otherwise. Basically, which three-dimensional shape information to select is determined according to condition 1, that is, whether or not the number of effective cameras photographing the area where the subject of interest exists is less than the threshold. However, if the determination is made based only on condition 1, an unnatural virtual viewpoint image may be generated. For example, when a certain subject is continuously projected in a virtual viewpoint image, when the subject of interest moves from an area where the number of effective cameras is greater than or equal to a threshold to an area where the number of effective cameras is less than the threshold, the type of 3D shape used at that moment is switched. . Therefore, an unnatural change occurs in the image of the subject of interest in the virtual viewpoint image. Therefore, in the present embodiment, conditions 2 and 3 are taken into consideration in order to prevent such unnatural changes from occurring.

Suppose that the number of effective cameras capable of capturing an image of the area where the subject of interest exists is less than the threshold, and the subject of interest one frame before has a silhouette-based three-dimensional shape. At this time, if posture-based 3D shape information is used for the subject of interest, the type of 3D shape will be switched. As a result, an unnatural discontinuity occurs in the image of the subject of interest in the virtual viewpoint image. Therefore, the continuity of the image of the target subject is maintained by selecting the three-dimensional shape information so that the type of the three-dimensional shape of the target subject does not change between the previous frame and the current frame according to condition 2. However, if the judgment is made based on conditions 1 and 2, the shape used in the first frame will always be used. Therefore, as condition 3, a judgment is used as to whether or not the subject of interest was present within the angle of view of the virtual viewpoint one frame before. If the subject of interest is outside the angle of view of the virtual viewpoint one frame before, the subject of interest is not shown in the virtual viewpoint image. Therefore, even if the subject of interest exists within the angle of view of the virtual viewpoint in the current frame and the type of three-dimensional shape to be used is different between the previous frame and the current frame, unnatural discontinuity occurs in the image of the subject of interest. sex does not occur. As a result, it is possible to switch between the pose-based three-dimensional shape and the silhouette-based three-dimensional shape without affecting the image of the subject of interest.

Therefore, the three-dimensional shape used for the subject of interest in Case 3 of FIG. 6 is a posture-based three-dimensional shape, and the three-dimensional shape used for the subject of interest in Case 4 is a silhouette-based three-dimensional shape. The explanation of the other Cases in FIG. 6 is the same, so the explanation is omitted.

Returning to FIG. 5, in step S511, it is checked whether all subjects have been processed. If all the objects have not been processed, the process returns to step S506 to select the three-dimensional shape to be used for generating the virtual viewpoint image with the next object as the object of interest. When it is determined that all subjects have been processed, the three-dimensional shape information of each foreground subject is output to the image generation unit 106 . In step S512, the image generation unit 106 generates a virtual viewpoint image based on the camera parameters from the information acquisition unit 101, the texture information of the subject, the three-dimensional shape information from the selection unit 105, and the virtual viewpoint information from the operation device 3. do. The generated virtual viewpoint image is output to the display device 4 .

A method of generating a virtual viewpoint image will be described. The image generation unit 106 executes processing for generating a foreground virtual viewpoint image (virtual viewpoint image of the subject region) and processing for generating a background virtual viewpoint image (virtual viewpoint image other than the subject region). Then, a virtual viewpoint image is generated by superimposing the foreground virtual viewpoint image on the generated background virtual viewpoint image. The generated virtual viewpoint image is transmitted to the display device 4 and output to the display device 4 .

A method of generating a foreground virtual viewpoint image of a virtual viewpoint image will be described. The foreground virtual viewpoint image is generated by assuming a voxel as a three-dimensional point with coordinates (Xw, Yw, Zw), calculating the color of the voxel, and rendering the colored voxel using an existing CG rendering method. can be Before calculating the color, the image generation unit 106 first generates a distance image in which the pixel value is the distance from the camera of the imaging system 2 to the surface of the three-dimensional shape of the subject. Next, in order to assign colors to voxels, the image generation unit 106 once transforms the 3D point (Xw, Yw, Zw) into the camera coordinate system in the camera that includes the 3D point (Xw, Yw, Zw) within the angle of view. The camera coordinate system is a three-dimensional coordinate system defined by the lens plane (Xc, Yc) and the lens optical axis (Zc) with the lens center of the camera as the origin. Then, the image generator 106 converts the three-dimensional point converted into the camera coordinate system into the camera image coordinate system, and calculates the distance d from the voxel to the camera and the coordinates (Xi, Yi) on the camera image. The camera image coordinate system is defined on a plane in front of the lens surface and separated by a certain distance. is a two-dimensional coordinate system such that The image generator 106 calculates the difference between the distance d and the pixel value (=distance to the surface) of the coordinates (Xi, Yi) of the distance image, and if the calculated difference is equal to or less than a preset threshold, Determine that the voxel is visible from the camera. When determined to be visible, the image generation unit 106 sets the pixel value of the coordinates (Xi, Yi) in the captured image of the imaging system 2 as the color of the voxel. If the voxel is determined to be visible in a plurality of cameras, the image generation unit 106 acquires pixel values from the texture information of the foreground silhouette information obtained from the captured images from each camera of the imaging system 2. is the color of the voxel. However, the method of calculating the color is not limited to this. For example, instead of using the average value, a method such as using the pixel value of the captured image acquired from the imaging system 2 closest to the virtual viewpoint may be used. By repeating the same process for all voxels, colors can be assigned to all voxels that constitute the three-dimensional shape information. Here, for each voxel that constitutes the shape, the cameras that are the targets of determination as to whether or not they are visible may be all the cameras that constitute the imaging system 2, but are not limited to this. For example, a camera whose camera information indicates that voxels are visible, or a camera used for shape estimation may be targeted. By doing so, the processing time for generating the virtual viewpoint image can be shortened.

Next, a method for generating a background virtual viewpoint image of a virtual viewpoint image will be described. The image generation unit 106 acquires three-dimensional shape information of a background such as a stadium in order to generate a background virtual viewpoint image. CG models of stadiums and structures that are created in advance and stored in the system are used as background three-dimensional shape information. The image generation unit 106 compares the normal vector of each surface that constitutes the CG model and the direction vector of each camera that constitutes the imaging system 2, and extracts the most facing camera by fitting each surface within the angle of view. Then, the image generation unit 106 projects the vertex coordinates of each surface onto the extracted image captured by the camera, generates a texture image to be pasted on each surface, and renders the background virtual viewpoint image by an existing texture mapping method. to generate A virtual viewpoint image is generated by superimposing the foreground virtual viewpoint image on the background virtual viewpoint image of the virtual viewpoint image thus obtained.

As described above, according to the first embodiment, it is possible to suppress deterioration in the accuracy of estimating the three-dimensional shape of the subject, and as a result, it is possible to improve the image quality of the virtual viewpoint image.

<Second embodiment>
In the first embodiment, the virtual viewpoint image is generated by switching the shape estimation method (three-dimensional shape information type) according to the number of effective cameras for each region in the imaging region. In the second embodiment, a virtual viewpoint image is generated while switching the shape estimation method according to the degree of density of subjects in each region in the imaging region. Note that the device configuration of the image processing system, the functional configuration of the image generation device, and the hardware configuration of the image generation device in the second embodiment are the same as those in the first embodiment (FIGS. 1 and 4).

How the precision of the three-dimensional shape based on the silhouette and the precision of the three-dimensional shape based on the posture change according to the density of subjects will be described with reference to FIG. FIG. 8 is a schematic diagram showing a state in which a plurality of subjects are in close proximity in a certain small area in the imaging area. A camera 802 captures an image of an imaging region 801 whose three-dimensional shape is to be estimated. Foreground silhouette information 803 is generated from the image captured by the camera 802 . FIG. 8 shows a state in which a plurality of subjects 804 are concentrated in a small area of an imaging area 801 . At this time, since the subjects overlap in the captured image from the camera 802 , the silhouette of each subject cannot be correctly obtained from the foreground silhouette information 803 . Therefore, the accuracy of the silhouette-based three-dimensional shape decreases as the objects are densely packed. On the other hand, it is known that pose estimation methods using deep learning can obtain pose information from captured images with relatively high accuracy regardless of the number of effective cameras and the degree of crowding of subjects. Therefore, even if the subjects are densely packed, the pose-based three-dimensional shape can maintain a certain degree of accuracy. Therefore, by selecting the type of three-dimensional shape according to the density of subjects, it can be expected to prevent the image quality of the virtual viewpoint image from deteriorating.

[Generation processing of virtual viewpoint image]
Processing performed by the image generating apparatus 1 according to the second embodiment will be described with reference to the flowchart shown in FIG. The same step numbers are assigned to steps that perform the same processing as in the first embodiment (FIG. 5), and detailed description thereof is omitted. This embodiment differs from the first embodiment in that condition 1 uses the degree of concentration of subjects in the area where the subject of interest exists.

In step S707, as condition 1, the selection unit 105 determines whether or not the partial area in which the subject of interest selected in step S506 exists is a subject dense area in which the degree of subject density exceeds a predetermined threshold. In the second embodiment, it is determined whether or not each partial area obtained by dividing the imaging area into n areas is a subject-dense area based on whether or not there are bone models in a number equal to or greater than a threshold. As described in the first embodiment, the posture estimation unit 103 assigns position information and subject IDs to all foreground subjects. Therefore, the partial area where the subject of interest exists can be specified based on the position information of the foreground subject selected as the subject of interest, for example. Also, the number of bone models existing in the specified partial area can be obtained, for example, by counting the number of foreground objects existing in the specified partial area based on the position information of all the foreground objects. Note that the number of divisions of the imaging region should be appropriately set according to the size (area or volume) of the imaging region. For example, the imaging area is divided so that the area of each divided partial area is about 1 m ² . Also, the threshold for the number of bone models determined to be an example subject-dense area is set to 3, for example. In addition to the number of bone models in partial areas obtained by dividing the imaging area into n parts, the density of subjects may be determined based on the number of bone models present within a predetermined distance from the subject of interest.

In step S710, the selection unit 105 determines the type of three-dimensional shape to be used for the subject of interest based on the determination results of conditions 1 to 3 in steps S707, S508, and S509. Note that the determination in step S710 is obtained by replacing condition 1 in FIG. 6 with the condition in step S707 ("Y" indicates the case where it is determined that the area is a subject-dense area).

As described above, according to the second embodiment, it is possible to suppress a decrease in accuracy in estimating the three-dimensional shape of each subject existing in an area where the subjects are concentrated. As a result, the image quality of the virtual viewpoint image can be improved.

In each of the embodiments described above, three-dimensional shape information is selected based on the determination results of conditions 1 to 3, but the present invention is not limited to this. For example, only condition 1 may be used to select three-dimensional shape information. Further, for example, instead of or in addition to the conditions 1 to 3, the three-dimensional shape estimated by the second shape estimation unit 104 and the three-dimensional shape estimated by the first shape estimation unit 102 If the difference is large, silhouette-based 3D shape information may be selected. As described above, when the deviation of the shape from the standard bone model is large, the estimation accuracy of pose-based three-dimensional shape information decreases. Therefore, the selection unit 105 quantifies the difference between the three-dimensional shape estimated by the first shape estimation unit 102 and the three-dimensional shape estimated by the second shape estimation unit 104, and the quantified difference exceeds the threshold. If exceeded, choose a silhouette-based 3D shape. As a result, it is possible to reduce the decrease in estimation accuracy due to the deviation of the shape of the subject of interest from the standard. Note that the quantification of the difference can be done, for example, by calculating the magnitude (volume) of the displacement between the two three-dimensional shapes.

Further, in each of the above-described embodiments, when the subject of interest in the previous frame is outside the angle of view of the virtual viewpoint, it is possible to change the type of the three-dimensional shape (the subject of interest in the previous frame is the image of the virtual viewpoint). If it is within the corner, the type of 3D shape information is maintained), but it is not limited to this. For example, it may be possible to change the type of three-dimensional shape when the size of the subject image drawn in the virtual viewpoint image of the previous frame is smaller than a predetermined size. In this case, the type of 3D shape information is maintained as long as the size of the subject image drawn in the virtual viewpoint image of the previous frame is equal to or greater than a predetermined size. If the subject drawn in the virtual viewpoint image is small, a change in the type of three-dimensional shape information does not cause a great sense of discomfort. Alternatively, it may be possible to change the type of three-dimensional shape information when the subject of interest drawn in the virtual viewpoint image of the previous frame is less than a predetermined proportion of the entire subject of interest. In this case, the type of the three-dimensional shape information is maintained while a predetermined ratio or more of the subject of interest is drawn in the virtual viewpoint image of the immediately preceding frame.

The present disclosure provides a program that implements one or more functions of the above-described embodiments to a system or device via a network or storage medium, and one or more processors in a computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

1: image generation device, 2: imaging device, 3: operation device, 4: display device, 101: camera information acquisition unit, 102: first shape estimation unit, 103: attitude estimation unit, 104: second shape estimation unit, 105: selection unit, 106: image generation unit

Claims (14)

a first generation means for generating three-dimensional shape information of a subject based on silhouette information of the subject obtained from a plurality of captured images of the subject captured by a plurality of imaging devices;
a second generation means for generating three-dimensional shape information of the subject based on the posture of the subject estimated from the plurality of captured images;
one of the three-dimensional shape information generated by the first generation means and the three-dimensional shape information generated by the second generation means for each subject included in the plurality of captured images, and displaying one of the three-dimensional shape information generated by the second generation means as a virtual viewpoint image; a selection means for selecting as three-dimensional shape information used to generate the
and image generating means for generating a virtual viewpoint image using the three-dimensional shape information selected by the selecting means. The selecting means performs the selection based on the number of imaging devices capable of imaging a partial region in which the subject exists, among a plurality of partial regions obtained by dividing the imaging region, among the plurality of imaging devices. 2. The image generating device according to claim 1, characterized by: The selection means selects the three-dimensional shape information generated by the first generation means when the number of imaging devices capable of imaging exceeds a threshold, and the number of imaging devices capable of imaging is equal to or less than the threshold. 3. The image generating apparatus according to claim 2, wherein the three-dimensional shape information generated by said second generating means is selected in case of an error. The selecting means performs the selection based on the degree of density of subjects in a partial area in which the subject exists among a plurality of partial areas obtained by dividing an imaging area among the plurality of imaging devices. 4. The image generation device according to any one of claims 1 to 3. The selecting means selects the three-dimensional shape information generated by the second generating means when the degree of congestion exceeds a threshold, and generates the three-dimensional shape information by the first generating means when the degree of congestion is equal to or less than the threshold. 5. The image generation device according to claim 4, wherein the three-dimensional shape information obtained by the processing is selected. 6. The image generating apparatus according to claim 1, wherein said selection means performs said selection based on the number of subjects existing within a predetermined distance range from said subject. . The selecting means selects the three-dimensional shape information generated by the second generating means when the number of subjects existing within the range exceeds the threshold, and the number of subjects existing within the range exceeds the threshold. 7. The image generating apparatus according to claim 6, wherein the three-dimensional shape information generated by said first generating means is selected in the following cases. The selection means quantifies the difference between the three-dimensional shape represented by the three-dimensional shape information generated by the first generation means and the three-dimensional shape represented by the three-dimensional shape information generated by the second generation means. 5. The image generation apparatus according to claim 1, wherein the selection is performed based on the size of the difference that has been quantified and quantified. The selection means selects the three-dimensional shape information generated by the first generation means when the magnitude of the difference exceeds the threshold, and selects the three-dimensional shape information generated by the first generation means when the magnitude of the difference is less than the threshold. 9. The image generating apparatus according to claim 8, wherein the three-dimensional shape information generated by the second generating means is selected. 10. The selecting means maintains the selection in the immediately preceding frame while the subject exists within the range of the virtual viewpoint image of the immediately preceding frame generated by the image generating means. The image generation device according to any one of . The selection means maintains the selection in the immediately preceding frame while the size of the subject image drawn in the immediately preceding frame virtual viewpoint image generated by the image generating means is equal to or larger than a predetermined size. 10. The image generating apparatus according to any one of claims 1 to 9, characterized by: 3. The selecting means maintains the selection in the immediately preceding frame while a predetermined proportion or more of the subject is drawn in the virtual viewpoint image of the immediately preceding frame generated by the image generating means. 10. The image generation device according to any one of 1 to 9. a first generation step of generating three-dimensional shape information of the subject based on silhouette information of the subject obtained from a plurality of captured images of the subject captured by a plurality of imaging devices;
a second generation step of generating three-dimensional shape information of the subject based on the posture of the subject estimated from the plurality of captured images;
one of the three-dimensional shape information generated in the first generating step and the three-dimensional shape information generated in the second generating step for each subject included in the plurality of captured images, and generating a virtual viewpoint image; A selection step of selecting as three-dimensional shape information used to generate the
and a generating step of generating a virtual viewpoint image using the three-dimensional shape information selected in the selecting step. A program for causing a computer to function as each means of the image generating apparatus according to any one of claims 1 to 12.

Images