JP2022103836A - Information processing device, information processing method, and program - Google Patents

Abstract

To generate an appropriate virtual viewpoint image.SOLUTION: Shape data of an object used for generating a virtual viewpoint image is generated using a plurality of captured images having different viewpoints, and it is determined whether the object related to the generated shape data corresponds to an unnecessary object. Then, a virtual viewpoint image in which the display of the object corresponding to the unnecessary object is suppressed is generated.SELECTED DRAWING: Figure 2

Description

The technique of the present disclosure relates to a technique of generating a virtual viewpoint image from an image taken by a plurality of image pickup devices.

Nowadays, multiple image pickup devices are installed at different positions to perform synchronous shooting from multiple viewpoints, and a virtual viewpoint image showing the view from a specified viewpoint (virtual viewpoint) using the multi-viewpoint image obtained by the shooting. The technology to generate is attracting attention. When generating a virtual viewpoint image, a three-dimensional model of a subject (foreground object) such as a person existing in the image capture area is calculated to create an image when the image capture area is viewed from the virtual viewpoint. There are various targets for generating virtual viewpoint images, and examples thereof include sporting events held in stadiums. Such sporting events are often filmed by broadcasting station staff (cameramen) for the purpose of broadcasting and distribution. Also, at large events, a special photography system called a wire cam may be used. The wire cam is a system for shooting by suspending an image pickup device from a plurality of wires extending over a space to be shot, and can shoot a dynamic bird's-eye view image in a field competition such as soccer or rugby.

As mentioned above, in a large-scale sporting event or the like, a cameraman or a wire cam of a broadcasting station moves in the shooting target space. As a result, there are cases in which they are reflected in the virtual viewpoint image as in the case of athletes and the like, which interferes with viewing. It is desirable that people and objects that interfere with viewing, which are not the original foreground objects, are not displayed on the virtual viewpoint image. According to Patent Document 1, when a shield is detected by the first camera, the image captured by the second camera is subjected to viewpoint conversion processing, and the image is combined with the image captured by the first camera to shield the image by the shield. A surveillance camera system has been proposed that allows the subject to be confirmed.

Japanese Unexamined Patent Publication No. 2011-77981

However, the technique of Patent Document 1 obtains an image in a state without an obstruction by performing viewpoint conversion processing on an image taken by a camera in which an obstruction is not detected among a plurality of cameras whose positions are fixed. .. Therefore, it is difficult to obtain a virtual viewpoint image by applying it as it is to the generation of a virtual viewpoint image and appropriately removing objects that hinder viewing.

Therefore, the technique of the present disclosure aims to generate an appropriate virtual viewpoint image.

The information processing apparatus according to the present disclosure includes a first generation means for generating shape data representing a three-dimensional shape of an object using a plurality of captured images having different viewpoints, and an object related to the shape data generated by the first generation means. However, the determination means for determining whether or not the object corresponds to an unnecessary object in the virtual viewpoint image based on at least the conditions relating to the shape of the object, and the display of the object determined to be an unnecessary object by the determination means are the determination. It is characterized by having a second generation means for generating a virtual viewpoint image, which is suppressed more than the display of an object that is not determined to be an unnecessary object by the means.

According to the technique of the present disclosure, an appropriate virtual viewpoint image can be generated.

A block diagram showing an example of the configuration of an image processing system that generates a virtual viewpoint image. The figure explaining the arrangement of the camera installed in the space to be photographed. A block diagram showing a hardware configuration of an information processing device. The figure block diagram which shows the software structure of the server which concerns on Embodiment 1. FIG. (A) and (b) are diagrams showing an example of a UI screen. A functional block diagram showing the internal configuration of the 3D model analysis unit. (A) to (l) are diagrams showing an example of a basic shape pattern of a three-dimensional model. A table showing an example of feature information of a three-dimensional model. The flowchart which shows the flow of the virtual viewpoint image generation processing which concerns on Embodiment 1. (A) and (b) are diagrams for explaining the effect of the first embodiment. The block diagram which shows the software structure of the server which concerns on Embodiment 2. The flowchart which shows the flow of the virtual viewpoint image generation processing which concerns on Embodiment 1.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. It should be noted that the following embodiments do not limit the present invention, and not all combinations of features described in the present embodiment are essential for the means for solving the present invention. The same configuration will be described with the same reference numerals.

[Embodiment 1]
In this embodiment, shape data representing the three-dimensional shape of an object (hereinafter referred to as "three-dimensional model") is analyzed, and the three-dimensional model of a specific object does not need to be displayed in a virtual viewpoint image. By adding information indicating that, and by making the three-dimensional model to which the information is added transparent, a virtual viewpoint image in which objects that hinder viewing are excluded is obtained. The virtual viewpoint image is an image generated by the end user and / or an appointed operator freely manipulating the position and posture of the virtual camera, and is also called a free viewpoint image or an arbitrary viewpoint image. Further, the virtual viewpoint image may be a moving image or a still image. Hereinafter, an image processing system that realizes the present embodiment will be described by taking the case where the virtual viewpoint image is a moving image as an example.

(Basic system configuration)
FIG. 1 is a block diagram showing an example of the configuration of an image processing system for generating a virtual viewpoint image according to the present embodiment. The image processing system 100 includes a plurality of image pickup modules 110, a switching hub 115, a server 116, a database (DB) 117, a control device 118, and a display device 119. A camera 111 and a camera adapter 112, which are image pickup devices, are connected to each of the plurality of image pickup modules 110 by internal wiring. Each image pickup module 110 transmits by a network cable. The switching hub (hereinafter referred to as “HUB”) 115 is a device that performs routing between network devices. Each of the image pickup modules 110 is connected to the HUB 115 by a network cable 113. Similarly, the server 116, the DB 117, and the control device 118 are also connected to the HUB 115 by the network cable 113, respectively. The control device 118 and the display device 119 are connected by a video cable 114. Each camera 111 takes pictures in synchronization with each other with high accuracy based on the synchronization signal. As shown in FIG. 2, in the present embodiment, a plurality of image pickup modules 110 (only 10 units are shown for convenience) are installed so as to surround the field of the stadium, which is the shooting target space. Further, a wire cam 120 for shooting a broadcast image, which is not used for generating a virtual viewpoint image, is installed in a state of being suspended from a wire stretched over the field.

The server 116 processes the captured image obtained by the image pickup module 110, generates and analyzes a three-dimensional model of an object, and colors the generated three-dimensional model (also called "texture pasting" or "texture mapping"). It is an information processing device that performs such things. The server 116 also has a time server function for generating a time synchronization signal for performing time synchronization of the system. In this embodiment, the object for which the three-dimensional model is generated is a moving object such as a player or a ball. In this case, the wire cam 120 that moves around the field and the cameraman who shoots broadcast video while moving around the field also formally correspond to moving objects and are the targets for generating a three-dimensional model. Is an issue in the disclosed technology. The database (hereinafter referred to as “DB”) 117 stores data such as captured images obtained by each image pickup module 110 and generated three-dimensional models, and stores the stored data in the server 116 and the control device 118. To provide to. The control device 118 is an information processing device that controls each image pickup module 110 and the server 116. The control device 118 is also used for setting a virtual camera (virtual viewpoint). The display device 119 displays a setting user interface screen (UI screen) for the user to specify a virtual viewpoint in the control device 118, displays a UI screen for viewing the generated virtual viewpoint image, and the like. The display device 119 is, for example, a television, a computer monitor, a liquid crystal display unit of a tablet or a smartphone, and the like, and the type of the device is not limited.

(Operation of image processing system)
Next, a rough operation in the image processing system 100 will be described. The captured image obtained by the image pickup module 110 is subjected to predetermined image processing such as foreground background separation and then transmitted to the next image pickup module 110. Similarly, in the next image pickup module 110, the photographed image obtained by the own module is combined with the photographed image received from the previous image pickup module 110, and further transmitted to the next image pickup module 110. By continuing such an operation, 100 sets of captured images (including foreground images) are transmitted to the server 116 via the HUB 115.

The server 116 generates a virtual viewpoint image by generating and rendering a three-dimensional model of an object based on captured image data with different viewpoints acquired from all the image pickup modules 110. Further, the server 116 transmits the time and synchronization signals to each imaging module 110. Each image pickup module 110 that has received the time and synchronization signal performs image pickup using the received time and synchronization signal, and performs frame synchronization of the captured image. That is, in each image pickup module 110, shooting is performed in frame units in synchronization with the same time. The format of the captured image data is not particularly limited. For example, the bit depth of each pixel (8 bits, 10 bits, etc.) and the format for expressing the color of each pixel (YUV444, YUV422, RGB, etc.) are not limited. There is no limitation on the image file format. For example, it is assumed that the format is a general PNG (Portable Network Graphics) or JPEG (Joint Photographic Experts Group).

When generating a virtual viewpoint image, first, the silhouette of each of the foreground objects is extracted from a plurality of captured images having different viewpoints. Next, using an image (foreground image) representing the silhouette of the extracted object, a three-dimensional model of the object is generated, for example, by the Visual Hull method (visual volume crossing method). Finally, the three-dimensional model of the generated object is colored by using the color information (texture information) included in the captured image of each physical camera 111. By the above processing, a virtual viewpoint image showing the appearance from an arbitrary virtual viewpoint can be obtained. When generating the virtual viewpoint image, a morphing method or a billboarding method may be used instead of the model-based method described above. The morphing method is a method of generating a virtual viewpoint image between cameras by interpolating images between adjacent cameras according to the positional relationship between the cameras. Further, the billboarding method is a method of generating a virtual viewpoint image by extracting a two-dimensional image of a subject of a captured image and changing the direction by a projection calculation process in a three-dimensional space according to the virtual viewpoint. Such a method of acquiring a virtual viewpoint image by synthesizing or projecting a plurality of images taken by a physical camera is called image-based rendering.

(Hardware configuration of information processing equipment)
Subsequently, the hardware configuration of the information processing device such as the server 116 and the control device 118 will be described with reference to FIG. The camera adapters 112a to 112j in the image pickup modules 110a to 110j basically have the same hardware configuration. The information processing device includes a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a communication I / F 205, an operation unit 206, and a bus 207.

The CPU 201 controls the entire information processing apparatus by using the programs and data stored in the ROM 202 and the RAM 203. It should be noted that one or a plurality of dedicated hardware different from the CPU 201 may be provided, and at least a part of the processing by the CPU 201 may be executed by the dedicated hardware. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The ROM 202 stores programs and the like that do not require changes. The RAM 203 temporarily stores programs and data provided from the auxiliary storage device 204, data provided from the outside via the communication I / F 205, and the like. The auxiliary storage device 204 is composed of, for example, an HDD or SSD, and stores various data and programs such as input data such as image data and audio data, tables referred to in various processes described later, and various application programs. The communication I / F 205 is used for communication with an external device. For example, when connected to an external device by wire, a communication cable is connected to the communication I / F 205, and when the communication I / F 205 has a function of wirelessly communicating with the external device, the communication I / F 205 includes an antenna. The operation unit 206 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and inputs various instructions to the CPU 201 in response to an operation by the user. The bus 207 connects each of the above parts to transmit data and signals.

In this embodiment, the server 116 generates both a three-dimensional model and a virtual viewpoint image, but the system configuration is not limited to this. For example, a server that generates a three-dimensional model and a server that generates a virtual viewpoint image may exist separately.

(Server software configuration)
FIG. 4 is a block diagram showing a software configuration of the server 116 according to the present embodiment. In the technique of the present disclosure, a three-dimensional model to be hidden is determined based on the characteristics of the three-dimensional model of the object that hinders viewing. Then, by making the three-dimensional model determined to be hidden transparent and displayed, the objects that hinder viewing are made invisible, and the quality of the virtual viewpoint image is improved. As shown in FIG. 4, the server 116 has a data input unit 401, a three-dimensional model generation unit 402, a three-dimensional model analysis unit 403, a non-display setting unit 404, and a virtual viewpoint image generation unit 405. Hereinafter, each part will be described in order.

The data input unit 401 acquires data of a plurality of captured images with different viewpoints captured by the imaging module 110 and stored in the DB 117. The acquired data of the plurality of captured images is sent to the three-dimensional model generation unit 402 and the virtual viewpoint image generation unit 405. Further, the data input unit 401 controls information including various parameters necessary for generating a virtual viewpoint image (hereinafter, referred to as "virtual viewpoint information") such as the position and orientation of the virtual viewpoint, the angle of view, and the focal length. Get from 118. The acquired virtual viewpoint information is sent to the virtual viewpoint image generation unit 405.

The three-dimensional model generation unit 402 generates a three-dimensional model of a moving object existing in each photographed image based on the photographed image data received from the data input unit 401. Hereinafter, a specific generation procedure will be briefly described. First, each captured image is subjected to foreground-background separation processing to generate a foreground image representing the silhouette of an object. Here, the background subtraction method is used as the foreground background separation method. In the background subtraction method, first, among captured images composed of a plurality of frames, frames that differ in time series are compared, and a portion having a small difference in pixel values is specified as a static pixel. Then, a background image is generated using the specified non-moving pixels. By comparing the background image generated in this way with the frame of interest of the captured image, the pixel having a large difference from the background in the frame is specified as the pixel to be the foreground, and the foreground image is generated. The above processing is performed for each of a plurality of captured images having different viewpoints. Next, a three-dimensional model is extracted by the visual volume crossing method using the foreground image corresponding to each captured image. In the visual volume crossing method, the target three-dimensional space is divided into small unit cubes (voxels), the pixel position when each voxel appears in multiple captured images is obtained by three-dimensional calculation, and each voxel corresponds to the pixel in the foreground. Judge whether or not. When it is determined to be a foreground pixel in the captured images of all the imaging modules, the voxel is identified as a voxel constituting an object in the target three-dimensional space. In this way, only the specified voxels are left, and other voxels are deleted. Then, the finally remaining voxel group becomes a three-dimensional model representing the three-dimensional shape of the object existing in the target three-dimensional space. The generated 3D model for each object is sent to the 3D model analysis unit 403 together with the captured image data on which it is based.

The three-dimensional model analysis unit 403 performs analysis processing on the three-dimensional model generated by the three-dimensional model generation unit 402. In this analysis process, first, it is determined whether or not it is a three-dimensional model of a specific object (hereinafter, referred to as “excluded object”) to be excluded from the virtual viewpoint image by referring to the exclusion condition described later. Then, for the three-dimensional model determined to correspond to the excluded object, the information indicating that the display in the virtual viewpoint image is unnecessary (for example, "1" when the display is unnecessary and "0" in other cases). The flag to which is assigned. Hereinafter referred to as "hidden flag".) Is added. The details of the three-dimensional model analysis unit 403 will be described later.

The non-display setting unit 404 makes settings necessary for realizing display control for preventing the excluded object from appearing on the virtual viewpoint image. Specifically, the user instructs the condition (exclusion condition) that specifies whether to enable / disable the operation mode (hidden mode) of whether to control the hiding of the excluded object and what kind of object is to be the excluded object. Set based on. FIG. 5A shows an example of a user interface screen (UI screen) for the user to set the non-display mode. A user who wants to prevent a specific object that interferes with viewing from being displayed on the virtual viewpoint image checks the radio button 501 for enabling the non-display mode and presses the OK button 503. Then, the screen transitions to the screen for setting the exclusion condition shown in FIG. 5 (b). On the UI screen after this transition, the user can specify detailed conditions for the excluded object (here, the color, shape, and position of the object in the three-dimensional space). Now, in FIG. 5B, the button 511 for specifying the position in the three-dimensional space is selected, the pop-up window 512 is displayed, and the value "3000 (mm)" that defines the height from the ground is input. It is in the state of being input to the field 513. In this case, an object having a height of 3 m or more from the ground is set as an excluded object, whereby the wire cam 120 moving around over the field is determined to be an excluded object in the analysis process. Further, as a condition of the position information in the height direction, an object at a position higher than the object to be displayed may be set as an excluded object.

Here, the only condition for the position is that the height (z coordinate) in the three-dimensional space is above a certain level, but for example, thresholds are set for each of the x-axis, y-axis, and z-axis, and the coordinate values in each axis are above the threshold. , Below the threshold value, and the difference from the threshold value is within a certain value. Further, a virtual viewpoint image corresponding to an arbitrary frame is displayed in the preview area 515, an object that hinders viewing is selected from the image by the user with a mouse or the like, and the features (color, shape, position, size) of the object are selected. Etc.) may be automatically set as an exclusion condition. Further, when the user selects an object, the features may be displayed in a list, and the user may be made to select a feature to be excluded from the list. The user who specifies various conditions for defining the excluded object in this way specifies the combination condition 516 which is a criterion for determining whether or not the object is finally targeted for non-display control. As the combination condition, a general logical operation such as a general AND condition (logical product) or an OR condition (logical sum) can be used. For example, when using the AND condition, "AND" may be input in the input field 517. When the AND condition is specified, the 3D model in which all the analysis results of each analysis unit 602 to 604 "satisfy the condition" is determined as the 3D model to be hidden in the virtual viewpoint image. .. Similarly, when the OR condition is used, the 3D model in which any of the analysis results of each analysis unit 602 to 604 "satisfies" is determined as the 3D model to be hidden in the virtual viewpoint image. To. Based on such a user instruction performed via the control device 118 or the like, conditions for enabling / disabling the hidden mode and specifying the excluded object are set.

The virtual viewpoint image generation unit 405 generates a virtual viewpoint image by using the captured image data and virtual viewpoint information input from the data input unit 401 and the three-dimensional model input from the three-dimensional model analysis unit 403. Specifically, by calculating how each 3D model looks in the 3D space at the angle of view specified by the virtual viewpoint information, and coloring it using the image taken by the corresponding real camera, Generates an image that represents the view from a virtual viewpoint. At that time, if the non-display mode is enabled, the display of the three-dimensional model of the excluded object is suppressed. Specifically, instead of performing normal coloring, a process of semi-transparency or complete transparency is performed using a method such as alpha blending that can control the transmittance. When the hidden mode is disabled, all three-dimensional models are subjected to normal coloring processing regardless of the content of the hidden flag, and an image showing the appearance from the virtual viewpoint is generated. The data of the virtual viewpoint image thus obtained is sent to the display device 119 via the control device 118 and is provided for viewing by the user.

It is also possible to share and realize the above five functional units by a plurality of information processing devices. For example, the output result of the three-dimensional model analysis unit 403 may be input to another information processing device having the function of the virtual viewpoint image generation unit 405.

(Details of 3D model analysis unit)
Subsequently, the non-display determination and the addition of feature information performed by the three-dimensional model analysis unit 403 will be described in detail. FIG. 6 is a functional block diagram showing the internal configuration of the three-dimensional model analysis unit 403. As shown in the figure, the three-dimensional model analysis unit 403 includes an exclusion condition holding unit 601, a color analysis unit 602, a position analysis unit 603, a shape analysis unit 604, a non-display determination unit 605, and a feature information addition unit 606.

The exclusion condition holding unit 601 holds the above-mentioned exclusion condition set by the non-display setting unit 404, and outputs the conditions according to the analysis items to the three analysis units 602 to 604. That is, among the exclusion conditions, the condition related to the position is output to the position analysis unit 602, the condition related to the color is output to the color analysis unit 603, and the condition related to the shape is output to the shape analysis unit 604. Further, the exclusion condition holding unit 601 holds the combination condition of the analysis results in each analysis unit 602 to 604 and outputs the combination condition to the non-display determination unit 505.

In the position analysis unit 602, the position of the 3D model provided by the 3D model generation unit 402 in the virtual viewpoint video space matches the condition regarding the position in the exclusion condition (height from the ground in the above example). Performs position analysis processing to determine whether or not. Specifically, when the height represented by the z-coordinate of the lower end of the bounding box of the target 3D model exceeds the height specified by the condition, the object of the target 3D model satisfies the condition regarding the position. Become. Here, the bounding box means a circumscribed rectangular cuboid of a group of voxels representing a three-dimensional shape of an object. The result of the position analysis process is given to the target three-dimensional model and sent to the non-display determination unit 605.

The color analysis unit 603 determines whether or not the color corresponding to each voxel constituting the three-dimensional model provided by the three-dimensional model generation unit 402 matches the color-related condition in the exclusion condition. I do. Now, it is assumed that each pixel of the captured image has a color represented by a color luminance value of the maximum value "4095" for each of RGB. As the condition regarding the color in this case, the condition that the color luminance value of each of RGB is "100" or less may be set. Under this condition, if the target 3D model has a color close to black, the object satisfies the condition of the excluded object. More specifically, for each voxel constituting the target 3D model, the captured image to be referred to in the color analysis process is determined based on the distance and angle to each actual camera (the line-of-sight direction of the actual camera), and the corresponding color is determined. Is given. Then, the given color is compared with the above conditions, and it is determined whether or not the color luminance value of each of RGB is equal to or less than the threshold value. At this time, it may be determined whether each of the colors of all voxels meets the above condition, or it may be determined whether the average value of the colors of all voxels meets the above condition. The result of the color analysis process is given to the target three-dimensional model and sent to the non-display determination unit 605.

The shape analysis unit 604 performs a shape analysis process for determining whether or not the shape of the three-dimensional model provided by the three-dimensional model generation unit 402 meets the conditions related to the shape in the exclusion conditions. 7 (a) to 7 (l) show an example of the basic shape pattern of the three-dimensional model. As a condition regarding the shape, for example, based on such a basic shape pattern, the shape of the target three-dimensional model does not correspond to any of the shape patterns of (j) humanoid, (k) ellipse, and (l) sphere. All you have to do is set the conditions. Under this condition, if the shape of the target 3D model has a shape other than a human shape, an ellipsoid, or a sphere, such as a cylinder shape, the object corresponds to an excluded object. The result of the shape analysis process is given to the target three-dimensional model and sent to the non-display determination unit 605.

The non-display determination unit 605 exists in the virtual viewpoint video space based on the results of each analysis process in the position analysis unit 602, the color analysis unit 603, and the shape analysis unit 604, and the combination conditions provided by the condition holding unit 601. Make a non-display judgment for the ternary model of each object. It should be noted that the determination is made based on conditions other than the above-mentioned combination conditions, such as hiding the target three-dimensional model that "does not satisfy the conditions" depending on how the threshold values are set in the various conditions used in each analysis unit 602 to 604. It is also possible to do. Each three-dimensional model that has completed such a determination process is output to the feature information adding unit 606 together with the determination result.

The feature information addition unit 606 adds feature information to each three-dimensional model received from the non-display determination unit 605. This feature information also includes the above-mentioned non-display flag determined based on the result of each non-display determination. FIG. 8 is a table showing an example of the feature information of the three-dimensional model. As described above, the three-dimensional model is composed of a plurality of voxels, and the bounding box (circular cuboid) of the entire voxel group representing the three-dimensional shape of one object is first defined. Then, for each three-dimensional model, information corresponding to each item such as its size, shape, and color used for coloring is stored. In this case, the size information is the volume value of the banding box and the number of voxels that actually exist. Further, as the shape information, for example, the information of the basic shape pattern most similar to the basic shape pattern shown in FIG. 7 described above is stored. Further, in the color information, the representative color indicating the most colors among the colors corresponding to each voxel and the average value of each of the RGB values are stored.

In the above example, the position, color, and shape of the three-dimensional model are used as analysis elements, but the analysis elements are not limited to these. For example, in addition to these, the size of the three-dimensional model may be used as an analysis element. In addition, when determining whether or not a certain condition is satisfied, in comparison with the threshold value, there are "greater than or equal to the threshold value", "below the threshold value", "greater than the threshold value", "less than the threshold value", and "difference from the reference value". You can use "within 〇〇", "difference from the standard value is △△ or more", etc.

(Flow of virtual viewpoint image generation process)
FIG. 9 is a flowchart showing the flow of the virtual viewpoint image generation process in the server 116 according to the present embodiment. The series of processes shown in this flowchart is realized by the CPU 201 of the server 116 reading a predetermined program from the ROM 202 or the auxiliary storage device 204 and executing the program.

Hereinafter, the flow of the virtual viewpoint image generation process according to the present embodiment will be described with reference to the flowchart of FIG. It is assumed that the flow of FIG. 9 starts its execution in response to the start signal of virtual viewpoint image generation from the control device 118. Then, it is assumed that the above-mentioned virtual viewpoint information and mode selection information indicating valid / invalid of the non-display mode are added to the start signal. Further, at the time of starting the execution, it is assumed that the captured image data that is the basis of the virtual viewpoint image has already been input to the data input unit 401. In the following description, the symbol "S" means a step.

In S901, the non-display setting unit 404 sets the enable / disable of the non-display mode based on the mode selection information input from the control device 118. In this case, when the non-display mode is enabled, the exclusion condition for specifying the target object (exclusion object) of the non-display control is also set.

In S902, the three-dimensional model generation unit 402 acquires an image of the frame to be processed based on the time information included in the virtual viewpoint information. Here, the time information is information on a start time and an end time that specifies a predetermined period in the captured image data. In the following S903, the three-dimensional model generation unit 402 generates a three-dimensional model of each object existing in the image of the processing target frame. The generated 3D model for each object is sent to the 3D model analysis unit 403.

In S904, the three-dimensional model analysis unit 403 performs the above-mentioned analysis processing on the three-dimensional model for each object generated in S903. That is, for each three-dimensional model, its characteristics are specified from the viewpoint of position, color, and shape, and whether the specified characteristics match the characteristics of the excluded object is determined by referring to the exclusion condition set in S901. It is judged. Then, feature information including a non-display flag according to the determination result is given to each three-dimensional model. In this way, the three-dimensional model to which the feature information is added according to the analysis result is sent to the virtual viewpoint image generation unit 405.

In S906, the processing is distributed according to the setting content of the non-display mode set in S901. That is, if the non-display mode setting is invalid (off), the process proceeds to S907, and if it is valid (on), the process proceeds to S908.

In S907 when the non-display mode is off, the virtual viewpoint image generation unit 405 generates a virtual viewpoint image in which all the objects existing in the processing target frame are displayed. Specifically, regardless of the value of the non-display flag, all the input three-dimensional models are normally colored to generate a virtual viewpoint image corresponding to the frame to be processed.

On the other hand, in S908 when the non-display mode is enabled, the virtual viewpoint image generation unit 405 generates a virtual viewpoint image in which the excluded objects are hidden in the processing target frame. Specifically, a three-dimensional model having a hidden flag value of "0" is normally colored, and a three-dimensional model having a hidden flag value of "1" is subjected to transparency processing to be processed. Generate a virtual viewpoint image corresponding to the frame.

In S909, it is determined whether or not to end the generation of the virtual viewpoint image. If the processing is completed for all the frames specified by the time information included in the virtual viewpoint information, this processing is terminated. On the other hand, if there is an unprocessed frame, the process returns to S902, the image of the next frame to be processed is acquired, and the processing is continued.

The above is the flow of the virtual viewpoint image generation process according to the present embodiment. FIGS. 10A and 10B are diagrams illustrating the effects of the present embodiment. FIG. 10A is an example of a virtual viewpoint image generated by the conventional method, and FIG. 10B is an example of a virtual viewpoint image generated by the method of the present embodiment. In the case of the virtual viewpoint image of the conventional method of FIG. 10A, the wire cam 120 is treated as a foreground object and is displayed as it is. On the other hand, in the case of the virtual viewpoint image by the method of the present embodiment of FIG. 10B, since the three-dimensional model of the wire cam 120 is transparently processed, the viewer cannot see it on the completed virtual viewpoint image. It has become like.

In the above example, the virtual viewpoint image is generated by sequentially using all the frames within the predetermined period specified by the time information as the processing target frames, but the present invention is not limited to this. For example, a virtual viewpoint image may be generated by using a general key frame method. In the key frame method, multiple reference frames (called "key frames") associated with any virtual viewpoint are set, and the path information of the virtual viewpoint is obtained by interpolating between the set multiple key frames. , Is a method of generating a virtual viewpoint image corresponding to a predetermined period. Even in the case of such a key frame method, the method of the present embodiment can be similarly applied.

Further, in the above example, for the sake of simplification of the explanation, the non-display mode is enabled / disabled before the loop processing for each frame, but the present invention is not limited to this. For example, the setting of the non-display mode may be changed during the generation of the virtual viewpoint image.

As described above, according to the present embodiment, based on the characteristics of the three-dimensional model of the object existing in the captured image, the three-dimensional model of the object satisfying a certain condition is transparentized at the time of coloring. As a result, the objects that hinder the viewing cannot be visually recognized on the virtual viewpoint image, so that the user can view the objects without worrying about the objects that hinder the viewing.

Although the wire cam has been described as an example, the object that hinders viewing is not limited to this. For example, the object that interferes with viewing may be a small aircraft such as a drone or a device in which a camera is attached to a flying object. Further, the fixed camera may be used as an object that hinders viewing.

[Embodiment 2]
In the first embodiment, when an object interferes with appreciation, the three-dimensional model is transparentized at the time of coloring to suppress the display in the virtual viewpoint image. Next, if it corresponds to an excluded object that interferes with viewing, the 3D model is deleted in advance, and rendering processing is performed using only the remaining 3D model to display the excluded object in the virtual viewpoint image. The mode of suppressing the above will be described as the second embodiment. The contents common to the first embodiment, such as the basic configuration of the image processing system, will be omitted or simplified, and the differences will be mainly described below.

(Server software configuration)
FIG. 11 is a block diagram showing a software configuration of the server 116 according to the present embodiment. In the present embodiment, by deleting the three-dimensional model determined to be hidden, a virtual viewpoint image in which the object that hinders viewing does not exist in the first place is generated. As shown in FIG. 11, in the case of the present embodiment as well, the five functional units (data input unit 401', three-dimensional model generation unit 402', three-dimensional model analysis unit 403', and non-display setting unit 404' are the same as in the first embodiment. 'And a virtual viewpoint image generation unit 405'). Hereinafter, each part will be described in order, focusing on the parts different from the first embodiment.

The data input unit 401'acquires captured image data from the DB 117, acquires virtual viewpoint information from the control device 118, and sends them to the three-dimensional model generation unit 402'.
The three-dimensional model generation unit 402'generates a three-dimensional model of a moving object based on the captured image data received from the data input unit 401'. Further, the three-dimensional model generation unit 402'performs a process of imparting color information to each voxel constituting the generated three-dimensional model. In this color information assignment, the captured image that is the source of the color information is determined based on the positional relationship such as the distance and angle between the boxel of interest and each actual camera (in the case of a plurality of images, weighting is performed as necessary). And so on), the RGB values of the corresponding colors in the determined captured image are given. The three-dimensional model to which the color information is given in voxel units (hereinafter, referred to as "colored three-dimensional model") is sent to the three-dimensional model analysis unit 403'.

The three-dimensional model analysis unit 403'performs an analysis process of whether or not the colored three-dimensional model generated by the three-dimensional model generation unit 402'is a three-dimensional model of the excluded object. At this time, it is sufficient to perform the analysis process only when the non-display mode is enabled. The content of the analysis process is basically as described in the first embodiment. However, in the color analysis process in the present embodiment, since the three-dimensional model already has the color information, the color analysis may be performed using the color information, which is different from the first embodiment. Further, in the case of the present embodiment in which the three-dimensional model of the excluded object is deleted, the hidden flag is not required, so that the hidden flag is not included in the feature information attached to each three-dimensional model. Then, the three-dimensional model analysis unit 403'deletes the colored three-dimensional model determined by the analysis process to meet the exclusion condition (= combination condition) without sending it to the virtual viewpoint image generation unit 405'. .. That is, only the colored three-dimensional model of the non-excluded object that remains without being deleted is sent to the virtual viewpoint image generation unit 405'.

The virtual viewpoint image generation unit 405'generates a virtual viewpoint image using the virtual viewpoint information input from the data input unit 401 and the colored three-dimensional model input from the three-dimensional model analysis unit 403'. That is, the virtual viewpoint image generation unit 405'performs coloring processing on all the input colored three-dimensional models using the color information given to each three-dimensional model to represent the appearance from the virtual viewpoint. Generate an image. In the case of the present embodiment, since the three-dimensional model of the excluded object that hinders viewing is not input to the virtual viewpoint image generation unit 405'in the first place, the control by the non-display flag is not performed.

As in the first embodiment, the above five functional units can be shared and realized by a plurality of information processing devices. For example, the output result of the three-dimensional model analysis unit 403'may be input to another information processing device having the function of the virtual viewpoint image generation unit 405'. In particular, in the present embodiment, the three-dimensional model of the excluded object is deleted and the display control by the non-display flag becomes unnecessary. Therefore, a general-purpose information processing device (virtual viewpoint image) that receives the three-dimensional model and generates a virtual viewpoint image. Easy to combine with a generator).

(Flow of virtual viewpoint image generation process)
FIG. 12 is a flowchart showing the flow of the virtual viewpoint image generation process in the server 116 according to the present embodiment. The series of processes shown in this flowchart is realized by the CPU 201 of the server 116 reading a predetermined program from the ROM 202 or the auxiliary storage device 204 and executing the program.

Hereinafter, the flow of the virtual viewpoint image generation process according to the present embodiment will be described with reference to the flowchart of FIG. It should be noted that the flow of FIG. 12 is assumed to start its execution in response to the start signal of virtual viewpoint image generation from the control device 118, as in the flow of FIG. 9 of the first embodiment. Then, it is assumed that the above-mentioned virtual viewpoint information and mode selection information indicating valid / invalid of the non-display mode are added to the start signal. Further, at the time of starting the execution, it is assumed that the captured image data that is the basis of the virtual viewpoint image has already been input to the data input unit 401'. Hereinafter, the points different from the flow of FIG. 9 of the first embodiment will be mainly described.

In S1201, the non-display setting unit 404'sets the enable / disable of the non-display mode based on the mode selection information input from the control device 118. Similar to S901 of the first embodiment, when the non-display mode is effectively set, the exclusion condition for specifying the target object (exclusion object) of the non-display control is also set. In the present embodiment, the valid / invalid information of the set non-display mode is sent to the three-dimensional model analysis unit 403'instead of the virtual viewpoint image generation unit 405'.

In S1202, the three-dimensional model generation unit 402'acquires an image of the frame to be processed based on the time information included in the virtual viewpoint information. In the following S1203, the three-dimensional model generation unit 402'generates a three-dimensional model to which color information is added to each object existing in the image of the frame to be processed. The generated colored three-dimensional model is sent to the three-dimensional model analysis unit 403'.

In S1204, the processing is distributed based on the valid / invalid information of the non-display mode input from the non-display setting unit 401'. If the hidden mode is invalid, the process proceeds to S1208. At this time, feature information not including the non-display flag is given to all the colored 3D models input from the 3D model generation unit 402', and the virtual viewpoint image generation unit is added via the 3D model analysis unit 403'. It will be sent to 405'.

On the other hand, if the non-display mode is valid, in S1205, the three-dimensional model analysis unit 403'performs the above-mentioned analysis processing for each of the colored three-dimensional models input from the three-dimensional model generation unit 402'. That is, for each colored three-dimensional model, its characteristics are specified from the viewpoint of position, color, and shape, and whether the specified characteristics match the characteristics of the excluded object is referred to the exclusion condition set in S1201. Is judged.

In S1206, the processing is distributed based on the result of the analysis processing. If an object determined to be an excluded object exists, the process proceeds to S1207, and if it does not exist, the process proceeds to S1208.

In S1207, the three-dimensional model analysis unit 403'deletes the three-dimensional model of the object determined to be an excluded object. Then, the feature information not including the non-display flag is given to the colored three-dimensional model determined not to be an excluded object (that is, not deleted), and is sent to the virtual viewpoint image generation unit 405.

In S1208, the virtual viewpoint image generation unit 405'generates a virtual viewpoint image corresponding to the frame to be processed by using all the input colored three-dimensional models. Specifically, each input colored three-dimensional model is normally colored using the given color information to generate a virtual viewpoint image showing the appearance from the virtual viewpoint.
In S1209, it is determined whether or not to end the generation of the virtual viewpoint image. If the processing is completed for all the frames specified by the time information included in the virtual viewpoint information, this processing is terminated. On the other hand, if there is an unprocessed frame, the process returns to S1202, the captured image of the next frame to be processed is acquired, and the processing is continued.

The above is the flow of the virtual viewpoint image generation process according to the present embodiment. In the above-described embodiment, the color information is added to each three-dimensional model and then the three-dimensional model of the excluded object is deleted, but it is not essential to add the color information in advance. For example, instead of giving color information, a deletion flag may be added, and a virtual viewpoint image may be generated using only a three-dimensional model in which the deletion flag is off. When coloring in this case, it is better to make an occlusion judgment by referring to the information in the bounding box of the three-dimensional model in which the deletion flag is on. This prevents the colors of the 3D model of the excluded object that should be deleted from being mistakenly used for coloring (pasting a texture) and resulting in an unnatural virtual viewpoint image.

As described above, according to the present embodiment, the virtual viewpoint image generation unit 405'can generate a virtual viewpoint image without being aware of the deleted three-dimensional model. As described above, even by the method of the second embodiment, the object that hinders the viewing can be appropriately removed from the virtual viewpoint image. Further, in the case of the second embodiment, since unnecessary objects are deleted in advance, when the information processing device that receives the three-dimensional model and generates the virtual viewpoint image is used as another device, the other device is based on the non-display flag. It is not necessary to have a control function. That is, since a general-purpose information processing device for generating a virtual viewpoint image is sufficient, the system configuration can be made simpler.

[Other embodiments]
The technique of the present disclosure supplies a program that implements one or more of the functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device program the program. It can also be realized by the process of reading and executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

116 Server 401 Data input unit 402 3D model generation unit 403 3D model analysis unit 404 Hidden setting unit 405 Virtual viewpoint image generation unit

Claims (14)

A first generation means for generating shape data representing a three-dimensional shape of an object using a plurality of captured images from different viewpoints.
A determination means for determining whether or not the object related to the shape data generated by the first generation means corresponds to an unnecessary object in the virtual viewpoint image, based on at least a condition related to the shape of the object.
A second generation means for generating a virtual viewpoint image in which the display of an object determined to be an unnecessary object by the determination means is suppressed more than the display of an object not determined to be an unnecessary object by the determination means.
An information processing device characterized by having. The determination means adds information indicating that the display in the virtual viewpoint image is unnecessary to the shape data of the object corresponding to the unnecessary object.
The second generation means performs transparency processing on the shape data to which the information indicating that the display in the virtual viewpoint image is unnecessary is added, and generates the virtual viewpoint image.
The information processing apparatus according to claim 1. The information processing apparatus according to claim 2, wherein the transparency process includes a process for making translucent. The determination means deletes the shape data of the object corresponding to the unnecessary object and deletes it.
The information processing apparatus according to claim 1, wherein the second generation means generates the virtual viewpoint image based on the shape data that has not been deleted. The first generation means generates shape data to which the color information of the object is added.
The second generation means performs a coloring process using the color information given to the shape data generated by the first generation means to generate the virtual viewpoint image.
The information processing apparatus according to claim 3. The determination means gives information instructing deletion to the shape data of the object corresponding to the unnecessary object, and gives the information.
The second generation means deletes the shape data to which the information instructing the deletion is added among the shape data, and generates the virtual viewpoint image using the shape data remaining without being deleted.
The information processing apparatus according to claim 5. The second generation means is characterized in that the occlusion determination is performed with reference to the information of the bounding box included in the shape data to which the information instructing the deletion is given, and the virtual viewpoint image is generated. The information processing apparatus according to 6. The determination means further comprises any one of claims 1 to 7, wherein the determination means makes the determination based on a condition relating to any one of a position, a size, and a color of an unnecessary object in three-dimensional space. The information processing device described in. The information processing apparatus according to any one of claims 1 to 8, further comprising a setting means for setting conditions used for the determination based on a user instruction. The setting means according to claim 9, wherein the setting means sets the characteristics of the designated object as the condition based on the user instruction to specify an unnecessary object from the objects included in the captured image. Information processing device. The information processing apparatus according to any one of claims 1 to 10, wherein the unnecessary object is a moving object. The information processing device according to claim 11, wherein the moving object is a photographing device suspended from a wire stretched over a space to be photographed in the photographed image. The first generation step to generate shape data representing the three-dimensional shape of an object using multiple captured images from different viewpoints,
A determination step of determining whether or not the object related to the shape data generated in the first generation step corresponds to an unnecessary object in the virtual viewpoint image, at least based on the conditions related to the shape of the object.
With the second generation step of generating a virtual viewpoint image, the display of the object determined to be an unnecessary object in the determination step is suppressed more than the display of the object not determined to be an unnecessary object in the determination step. ,
An information processing method characterized by having. A program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 12.

Images