JP2021018570A - Image processing apparatus, image processing system, image processing method and program - Google Patents

Abstract

To provide an image processing apparatus capable of determining a position where occlusion occurs in the shape data of the object while suppressing the increase of processing load.SOLUTION: A shape data, which is generated in an area around the area where shape data for rendering is generated, is acquired, and the visibility is judged whether or not it can be seen from each imaging apparatus on the shape data for rendering.SELECTED DRAWING: Figure 5

Description

The technique of the present disclosure relates to the generation of a virtual viewpoint image, and particularly to an occlusion measure when generating object shape data from a plurality of viewpoint images.

In recent years, multiple cameras (imaging devices) are installed at different positions to perform synchronous imaging from multiple viewpoints, and the multi-viewpoint images obtained by the imaging are used to represent the view from a non-existent virtual camera. Attention is being paid to techniques for generating images. According to this virtual viewpoint image, for example, the highlight scenes of soccer and basketball can be viewed and viewed from various angles, so that the user can be given a high sense of presence as compared with a normal captured image.

In the generation of the virtual viewpoint image, first, the images captured by a plurality of cameras are aggregated in a server or the like, and a three-dimensional model of an object (subject) in the image is generated. Then, the three-dimensional model is colored based on the designated virtual viewpoint, and further subjected to two-dimensional transformation such as projective transformation to obtain a two-dimensional virtual viewpoint image. Of the above generation processes, the viewpoint-dependent texture mapping method is often used for coloring the three-dimensional model (Non-Patent Document 1). In the viewpoint-dependent texture mapping, when determining the color of the point of interest on the object, the colors at the corresponding points on the captured images of the plurality of cameras are blended with weight. By doing so, the discontinuity of color change caused by the color difference between the captured images corresponding to each viewpoint, the three-dimensional model shape, the error of the camera parameter, etc. is suppressed, and a more natural texture expression is realized. There is.

In the above viewpoint-dependent texture mapping, the object in the foreground when viewed from a specific viewpoint hides a part of the object in the back, and the coloring of the object in the back to the 3D model becomes inaccurate. was there. In this regard, research is also being conducted in which it is determined from the three-dimensional model which camera's captured image the object is hidden, and coloring is performed based on the determination result (Non-Patent Document 2).

P.E.Debevec, C.J.Taylor, and J.Malik: "Modeling and Rendering Architecture from Photographs: A Hybrid Geometry-and Image-Based Approach," SIGGRAPH'96, pp.11-20, 1996. Gregory G.Slabaugh, Ronald W.Schafer, and Mat C.Hans: "Image-Based Photo Hulls for Fast and Photo-Realistic New View Synthesis," Real-Time Imaging, Vol.9, No.5, October 2003.

In the technique disclosed in Non-Patent Document 2, the location where occlusion occurs is determined by using the three-dimensional model itself generated from the multi-viewpoint image. Therefore, there is a problem that the occlusion caused by the object existing outside the three-dimensional space (model generation area) for generating the three-dimensional model in the imaging space cannot be discriminated. This problem can be dealt with by expanding the model generation area, but if the model generation area is expanded, the amount of calculation required to generate the 3D model will be enormous, and the processing load on the server etc. will increase significantly. It will be. Therefore, a simple expansion of the model generation area leads to deterioration of the processing efficiency of the system.

The technique of the present disclosure has been made in view of the above problems, and an object thereof is to determine the occurrence location of occlusion in the shape data of an object while suppressing an increase in processing load.

The image processing apparatus according to the present disclosure captures shape data representing the three-dimensional shape of an object included in at least one of a plurality of viewpoint images obtained by synchronous imaging with a plurality of imaging devices in the target three-dimensional space of the synchronous imaging. A generation means generated in a predetermined three-dimensional region, an acquisition means for acquiring shape data generated in a three-dimensional region different from the predetermined three-dimensional region in the target three-dimensional space for synchronous imaging, and the above-mentioned Using the shape data acquired by the acquisition means, the determination means for determining the visibility of the shape data generated by the generation means from the imaging device and the shape data generated by the generation means are displayed as visible. It has an output means for outputting together with the result of the sex determination, and the determination means determines whether or not the shape data generated by the generation means is blocked by the shape data acquired by the acquisition means. It is a feature.

According to the technique of the present disclosure, it is possible to accurately determine the occurrence location of occlusion in the shape data of an object while suppressing an increase in processing load.

A block diagram showing a configuration of an image processing system that generates a virtual viewpoint image according to the first embodiment. The figure which shows the arrangement of each camera which constitutes a camera array which concerns on Embodiment 1. (A) to (c) are diagrams for explaining a three-dimensional model in a voxel format. Diagram explaining the concept of visibility judgment processing Block diagram showing an example of the hardware configuration of an information processing device as a three-dimensional model generator A functional block diagram showing a software configuration of a three-dimensional model generator according to the first embodiment. (A) and (b) are diagrams for explaining the rendering area and the determination area adjacent thereto according to the first embodiment. (A) to (c) are diagrams for explaining the basic principle of the visual volume crossing method. (A) and (b) are diagrams for explaining the significance of visibility determination. (A) is a diagram showing the world coordinate system before conversion, and (b) is a diagram showing the camera coordinate system after conversion. Flow chart showing the flow of processing in the three-dimensional model generator according to the first embodiment A block diagram showing a configuration of an image processing system that generates a virtual viewpoint image according to the second embodiment. The figure which shows the arrangement of each camera which constitutes a camera array which concerns on Embodiment 2. Functional block diagram showing the software configuration of the three-dimensional model generator according to the second embodiment (A) to (c) are diagrams for explaining the rendering area and the determination area adjacent thereto according to the second embodiment. Flow chart showing the flow of processing in the three-dimensional model generator according to the second embodiment

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the following embodiments do not limit the present invention, and not all combinations of features described in the present embodiment are essential for the means for solving the present invention. The same configuration will be described with the same reference numerals.

[Embodiment 1]
In the present embodiment, a provisional three-dimensional model for visibility determination is generated in an area adjacent to the three-dimensional region for generating a three-dimensional model for rendering, and the visibility determination for the three-dimensional model for rendering is performed. Will be explained. In this embodiment, the case of moving images will be described as an example, but the same applies to still images.

<System configuration>
FIG. 1 is a block diagram showing a configuration of an image processing system for generating a virtual viewpoint image according to the present embodiment. The image processing system 1 includes a camera array 10 composed of a plurality of cameras (imaging devices) 10a-10p, a foreground extraction device group 12 composed of a plurality of foreground extraction devices 12a-12p, a control device 13, a three-dimensional model generation device 14, and rendering. It has a device 15.

The camera array 10 is composed of a plurality of cameras 10a-10p, and synchronously captures (photographs) an object (subject) from various angles. The image data captured by each camera (the group of which is the multi-viewpoint image data) is sent to the foreground extraction device group 12. In the present embodiment, each camera 10a-10p constituting the camera array 10 is arranged so as to surround the field in the stadium as shown in FIG. Then, each camera 10a-10p takes an image in synchronization with the time with the point T0 on the field as the gazing point.

Each foreground extraction device 12a-12p constituting the foreground extraction device group 12 is associated with each camera 10a-10p, and extracts a part of the object to be the foreground from the captured image of the camera corresponding to itself. To generate a mask image and a texture image. Here, the foreground refers to a dynamic object that can be seen from an arbitrary viewpoint in the imaging space, and in this embodiment, a person or a ball existing on the field is a typical example. Then, static objects other than the foreground, such as goals and spectators' seats on the field, become the background. The mask image is a binary silhouette image in which the foreground portion of the captured image is represented by white and the background portion is represented by black. The texture image is a multi-valued image obtained by cutting out a rectangular (extrinsic rectangular) portion including the foreground from the captured image. As a method of extracting the foreground from a captured image, for example, there is a background subtraction method. In the background subtraction method, a background image obtained by performing imaging in a state where there is no dynamic object as a foreground, for example, before the start of a game, is retained, and the background image and the captured image in a state where the dynamic object exists This is a method of detecting the difference between the two and extracting the part where the difference is above a certain level. Other methods such as the inter-frame difference method may be used to extract the foreground. The generated mask image and texture image data are sent to the three-dimensional model generation device 14.

The control device 13 acquires camera parameters of each camera 10a-10p and receives virtual viewpoint information via a UI (user interface) (not shown). Camera parameters include external parameters and internal parameters. The external parameters are composed of a rotation matrix and a parallel traveling matrix, and indicate the position and orientation of the camera. Internal parameters include the focal length of the camera, the optical center, etc., and indicate the angle of view of the camera, the size of the image sensor, and the like. The process of acquiring camera parameters is called calibration. In calibration, first, a plurality of images of a specific pattern such as a checkerboard are prepared. Then, the camera parameters can be obtained by calculating the correspondence between the points in the three-dimensional world coordinate system and the corresponding points on the two dimensions from these images. The camera parameters obtained by the calibration are sent to the 3D model generation device 14 and the rendering device 15. The virtual viewpoint information includes the position / posture, gazing point, movement path, etc. of the virtual viewpoint (virtual camera) set in the target three-dimensional space for synchronous imaging, and is specified by the user using, for example, a dedicated joystick. Or set automatically according to the shooting scene. The virtual viewpoint information set based on the user input or the like is sent to the rendering device 15.

The three-dimensional model generation device 14 generates a three-dimensional model (shape data representing a three-dimensional shape) of a dynamic object to be colored by the rendering device 15 based on the input mask image and camera parameters. In this embodiment, the voxel format will be described as an example of the data format of the three-dimensional model. In the voxel format, the three-dimensional shape of an object is expressed using a minute cube called a "voxel" as shown in FIG. 3 (a). FIG. 3B is a set of voxels representing the target three-dimensional space for generating a three-dimensional model. Then, FIG. 3 (c) shows a voxel as a component obtained by removing a voxel in a non-foreground portion in the target three-dimensional space from the voxel set shown in FIG. 3 (b) by the visual volume crossing method. It is a three-dimensional model of a pyramid. The data format of the three-dimensional model may be another format such as a point cloud format using a point cloud as a component expressing the shape or a polygon mesh format using polygons. The three-dimensional model generation device 14 performs a process (visibility determination process) of determining how each of the three-dimensional models to be rendered is viewed from each camera. In the following description, the three-dimensional model to be rendered is simply referred to as a "rendering model". FIG. 4 is a diagram for explaining the concept of the visibility determination process. In FIG. 4, the dashed line 401 indicates the angle of view of the camera c1 (any one camera in the camera array 10), and the alternate long and short dash lines 402 and 403 represent the line of sight from the camera c1. Now, the alternate long and short dash line 402 reaches the voxels v1 constituting the model m1 without being blocked by the model m2. This means that the voxel v1 can be seen from the camera c1. In this case, the voxel v1 is reflected in the captured image of the camera c1, that is, the determination result is that it is visible. On the other hand, the alternate long and short dash line 403 reaches only the model m2. The alternate long and short dash line 404 indicates the line of sight that would have arrived if the line of sight indicated by the alternate long and short dash line 403 had not been blocked by the model m2. Now, the alternate long and short dash line 404 has reached the voxels v2 constituting the model m1, which means that the voxels v2 could be seen from the camera c1 if the model m2 did not exist. In this case, since the voxel v2 does not appear in the captured image of the camera c1, the determination result is invisible. Such a determination is made for each voxel that constitutes the generated rendering model. By such visibility determination processing, whether or not each voxel constituting the rendering model is captured in the captured image within the angle of view of each camera 10a-10p without being obstructed by other objects (presence or absence of occlusion). ) Will be obtained. The generated rendering model and the result of the visibility determination processing are sent to the rendering device 15 together with the texture image data.

The rendering device 15 performs a coloring process on the rendering model received from the three-dimensional model generation device 14, and generates a virtual viewpoint image according to the virtual viewpoint information received from the control device 13. Specifically, first, the voxels to be focused on are determined from the voxels constituting the rendering model. Then, a camera whose visibility judgment result for the voxel of interest is "visible" is specified, the correspondence of the coordinates in the texture image group is obtained from the camera parameters, and the value obtained by blending the pixel values of the coordinates having the correspondence is obtained. , Perform the process of giving to the voxel of interest. In the blending of pixel values, only the texture image of the camera determined to be "visible" is used, and the weight is increased as the pixel value of the texture image of the camera closer to the voxel of interest is used. Further, the rendering model is perspectively projected and transformed based on an arbitrary viewpoint position specified by the virtual viewpoint information to generate a two-dimensional image on the target three-dimensional space.

The above is an outline of the configuration of the image processing system according to the present embodiment. For the connection between the foreground extraction device 12a-12p and the three-dimensional model generation device 14, any network topology such as a star type, a ring type, or a bus type may be adopted.

<Details of 3D model generator>
Subsequently, the processing in the three-dimensional model generation device 14 according to the present embodiment will be described in detail.

<Hardware configuration>
FIG. 5 is a block diagram showing an example of the hardware configuration of the information processing device as the three-dimensional model generation device 14. The hardware configuration of the foreground extraction device 12, the control device 13, and the rendering device 15 also has the same hardware configuration as the three-dimensional model generation device 14 described below. The three-dimensional model generation device 14 includes a CPU 101, a ROM 102, a RAM 103, an auxiliary storage device 104, a display unit 105, an operation unit 106, a communication I / F 107, and a bus 108.

The CPU 101 realizes each function of the three-dimensional model generation device 14 shown in FIG. 6, which will be described later, by controlling the entire device by using the computer programs and data stored in the ROM 102 and the RAM 103. It should be noted that one or a plurality of dedicated hardware different from the CPU 101 may be provided, and at least a part of the processing by the CPU 101 may be executed by the dedicated hardware. Examples of dedicated hardware include ASICs (application specific integrated circuits), FPGAs (field programmable gate arrays), and DSPs (digital signal processors). The ROM 102 stores a program or the like that does not need to be changed. The RAM 103 temporarily stores programs and data supplied from the auxiliary storage device 104, data supplied from the outside via the communication I / F 107, and the like. The auxiliary storage device 104 is composed of, for example, a hard disk drive or the like, and stores various data such as image data and audio data.

The display unit 105 is composed of, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) for the user to operate the three-dimensional model generation device 14. The operation unit 106 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, and the like, and inputs various instructions to the CPU 101 in response to an operation by the user. The CPU 101 operates as a display control unit that controls the display unit 105 and an operation control unit that controls the operation unit 106. The communication I / F 107 is used for communication with an external device connected via a network such as a LAN. For example, when connected to an external device by wire, a communication cable is connected to the communication I / F 107. Further, when the three-dimensional model generation device 14 has a function of wirelessly communicating with an external device, the communication I / F 107 includes an antenna. The bus 108 connects the above-mentioned parts to transmit information. The display unit 105 and the operation unit 106 may exist as external independent devices.

<Software configuration>
FIG. 6 is a functional block diagram showing a software configuration of the three-dimensional model generation device 14 of the present embodiment. The three-dimensional model generation device 14 includes an input unit 201, a model generation condition setting unit 202, a rendering model generation unit 203, a determination model generation unit 204, a model integration unit 205, a visibility determination unit 206, and an output unit 207. Each of these functional units is realized by the CPU 101 in the three-dimensional model generation device 14 described above expanding a predetermined program stored in the ROM 102 or the auxiliary storage device 104 into the RAM 103 and executing the program. The functions of each part will be described below.

The input unit 201 receives input of various data from an external device. Specifically, the camera parameters of each camera 10a-10p are received from the control device 13, and the texture image and mask image data corresponding to each of the cameras 10a-10p are received from the foreground extraction device group 12. The received camera parameter and mask image data are stored in the RAM 103 or the auxiliary storage device 104 for use in the visibility determination process and the three-dimensional model generation process. The received texture image data is provided to the rendering device 15 via the output unit 108.

The model generation condition setting unit 202 sets the generation conditions of the rendering model and the determination model based on user input via GUI (not shown). Here, the judgment model is a three-dimensional model representing the shape of an object existing outside the generation target area of the rendering model, and is a provisional three-dimensional model used only for the visibility judgment of the rendering model. Point to. The rendering model generation conditions include information such as the target space for generating the rendering model (three-dimensional area for drawing the foreground in the virtual viewpoint image) and the size of the unit voxels constituting the rendering model. Information such as the target space for generating the judgment model (a predetermined three-dimensional area adjacent to the three-dimensional area for generating the rendering model) and the size of the unit voxels constituting the judgment model are included in the generation conditions of the judgment model. Is included. Hereinafter, the three-dimensional area for generating the rendering model will be referred to as a "rendering area", and the three-dimensional area for generating the judgment model will be referred to as a "judgment area". The rendering area is set by the user by designating an arbitrary three-dimensional area via a GUI (not shown) or the like according to the shooting scene. For example, in the case of a soccer match, as shown in FIG. 7A, the long side direction is the length of one side of the field, the short side direction is the length of 2/3 of the field, and the height direction is 2 of the player's height. Set a three-dimensional area that is about doubled. The size of the unit voxel defines the fineness of the generated three-dimensional model, and a value such as 5 mm is set. Then, a determination area is set around the set rendering area. The rendering area shown in FIG. 7A is adjacent to the rendering area in which the long side direction, the short side direction, and the height direction are each expanded by several meters as shown in FIG. 7B, for example. The three-dimensional area is set as the judgment area. The size of the unit voxel in the determination area may be the same as the unit voxel in the rendering area, but it is desirable to set a size slightly larger (for example, 10 mm with respect to 5 mm). That is, it is desirable to determine the unit voxels so that the relationship is shown by the following inequality.
"Voxel size of rendering area" ≤ "Voxel size of judgment area"

Increasing the size of the unit voxel of the rendering model results in a coarser three-dimensional model, which leads to deterioration of image quality. On the other hand, since it is sufficient for the judgment model used only for the visibility judgment to grasp the approximate shape of the object that may block the rendering model, there is no problem even if it is a rough three-dimensional model. Further, by increasing the unit voxel size of the judgment model, the processing load can be reduced accordingly. The determination area shown in FIG. 7B is set so as to surround the entire rendering area, but the present invention is not limited to this. If necessary, a region adjacent only in a specific direction, such as only one side in the long side direction, may be used as a determination region. The rendering model and the determination model generation conditions (set values) set in this way are provided to the rendering model generation unit 203 and the determination model generation unit 204. The rendering model and the judgment model generation condition are not limited to the GUI setting, but may be set by the CUI (Character User Interface), or may be set by reading the file in which the condition is stored. It doesn't matter. Further, in the present embodiment, one model generation condition setting unit 202 sets the rendering model generation condition and the determination model generation condition, but they may be set in separate setting units.

The rendering model generation unit 203 generates a rendering model using the mask image and camera parameters input to the input unit 201 based on the rendering model generation conditions set by the model generation condition setting unit 202. Similarly, the judgment model generation unit 204 also generates a judgment model using the mask image and camera parameters input to the input unit 201 based on the judgment model generation conditions set by the model generation condition setting unit 202. To do. In the present embodiment, both the rendering model and the determination model generate a three-dimensional model represented by a voxel set by the visual volume crossing method.

Here, the visual volume crossing method will be described. 8 (a) to 8 (c) are diagrams for explaining the basic principle of the visual volume crossing method. From the image obtained by capturing an image of an object, a mask image showing the two-dimensional silhouette of the object can be obtained on the imaging surface (FIG. 8 (a)). Then, consider a cone that spreads in the three-dimensional space so as to pass each point on the contour of the mask image from the projection center of the camera (FIG. 8 (b)). This cone is called the "visual volume" of the object by the corresponding camera. Further, the three-dimensional shape of the object can be obtained by finding the common region of a plurality of visual volumes, that is, the intersection of the visual volumes (FIG. 8 (c)). The visual volume crossing method is an example of a three-dimensional model generation method, and the method is not limited to this.

The model integration unit 205 integrates the rendering model generated by the rendering model generation unit 203 and the determination model generated by the determination model generation unit 204. By this integration, the rendering model and the judgment model are arranged in a common three-dimensional space.

The visibility determination unit 206 determines the visibility of each voxel of the rendering model by using the rendering model and the determination model arranged in the same three-dimensional space. 9 (a) and 9 (b) are diagrams for explaining the significance of the visibility determination. Now, the model m1 is generated in the rendering area 901, and the determination model m2 is generated in the adjacent determination area 902. When viewed from the camera c1, the model m2 exists at a position that blocks the model m1. As a result, most of the voxels constituting the model m1 can be correctly determined to be invisible (invisible) from the camera c1. On the other hand, FIG. 9B is an example in which the determination area 902 is not set. As described above, when viewed from the camera c1, the model m2 should be present at a position that normally blocks the model m1. However, since the position of the model m2 is outside the rendering area 901, the three-dimensional model is not generated even if the object exists there. As a result, most of the voxels constituting the model m1 are treated as being visible even though they cannot be seen from the camera c1, and in the coloring process, weighting is performed according to the distance from the virtual viewpoint. In the present embodiment, the occurrence of such a problem is suppressed by generating a provisional three-dimensional model for visibility determination even in the area adjacent to the three-dimensional area for generating the three-dimensional model for rendering. In the visibility judgment, first, the coordinate system of the rendering model and each voxel constituting the judgment model is converted from the world coordinate system to the camera coordinate system based on the camera parameters. FIG. 10A shows the world coordinate system before conversion, and FIG. 10B shows the camera coordinate system with reference to the camera c1 after conversion. Then, the cameras 10a-10p are designated as the focus camera in order, and each voxel is checked as the focus box cell to see if there is another voxel whose x-coordinate and y-coordinate are the same and whose z-coordinate is smaller in the camera coordinate system. Then, when there is another voxel whose x-coordinate and y-coordinate are the same as the voxel of interest and whose z-coordinate is small, it is determined that the voxel of interest is invisible (invisible) from the camera of interest. In this way, the information of the visibility determination result obtained for each voxel for the rendering model is provided to the output unit 207.

The output unit 207 outputs the three-dimensional model for rendering, the visibility determination result for the three-dimensional model, and the texture image data to the rendering device 15.

<Processing flow in 3D model generator>
FIG. 11 is a flowchart showing a processing flow in the three-dimensional model generation device 14 according to the present embodiment. Before the start of execution of the flowchart of FIG. 11, it is assumed that the camera parameters are received from the control device 13 and stored in the RAM 103 or the like, and the generation conditions of the rendering model and the determination model have been set based on the user input. To do. Hereinafter, the flow of processing in the three-dimensional model generation device 14 will be described with reference to the flowchart of FIG. In the following description, the symbol "S" represents a step.

In S1101, the input unit 201 monitors the reception of input data (mask image data for each camera) necessary for generating the three-dimensional model. If the reception of the input data is detected, the process proceeds to S1102. In this embodiment, since it is assumed that the multi-viewpoint image data is a moving image, the processing after S1102 is executed in frame units.

In S1102, the rendering model generation unit 203 generates a three-dimensional model for rendering using the input data received in S1101 according to the rendering model generation conditions set by the model generation condition setting unit 202. .. In the following S1103, the judgment model generation unit 204 uses the input data received in S1101 according to the judgment model generation conditions set by the model generation condition setting unit 202, and uses the input data received in S1101 to provide a provisional three-dimensional shape for visibility judgment. Generate a model.

In S1104, the model integration unit 105 arranges the three-dimensional model for rendering generated in S1102 and the three-dimensional model for determination generated in S1103 in a common three-dimensional space. Then, in S1105, the visibility determination unit 206 is checked to see if each voxel constituting the rendering model arranged in the common three-dimensional space is blocked by another three-dimensional model including the determination model. Determine the visibility from the camera.

In S1106, the output unit 207 outputs the result of the visibility determination obtained in the rendering model S1105 generated in S1102 to the rendering device 15. At this time, the texture image data received by the input unit 201 is also output.

In S1107, it is determined whether or not the processing is completed for all the frames of the input data received in S1101. If there is an unprocessed frame, the process returns to S1102 and processing is continued for the next frame.

The above is the flow of processing in the three-dimensional model generation device 14 according to the present embodiment. In the flowchart of FIG. 11, although the output unit 207 outputs in frame units, a plurality of frames may be output together, or when the processing for all frames constituting the input data is completed. It may be output collectively.

<Modification example>
In the present embodiment, a predetermined peripheral area adjacent to the rendering area is set as the determination area, but the present invention is not limited to this. For example, an area where occlusion may occur in the rendering model due to an object outside the rendering area may be calculated and set as a determination area. In this calculation, information such as the shape and size of the foreground object, camera parameters including at least the position and orientation of each camera, and the shape and size of the rendering area are taken into consideration. FIG. 9C is a diagram illustrating a concept when calculating the determination area. In FIG. 9C, the position of the model m4 is a position that is a limit point where occlusion may occur with respect to the model m3 existing at the boundary of the rendering area 901. The position of this model m4 is obtained by calculation using the above information. By setting the determination area based on such a calculation result, more accurate visibility determination becomes possible.

As described above, according to the present embodiment, it is possible to accurately determine the occurrence location of occlusion in the three-dimensional model to be rendered while suppressing the increase in processing load. As a result, it is possible to appropriately color the three-dimensional model at the time of rendering, and a high-quality virtual viewpoint image can be obtained.

[Embodiment 2]
Next, when a plurality of rendering areas are set adjacent to each other on the field, a part of one rendering area is treated as the other judgment area, and the three-dimensional model is accommodated among the plurality of rendering areas. The mode of mutual interaction will be described as the second embodiment. The parts common to the first embodiment will be omitted or simplified, and the differences will be mainly described below.

<System configuration>
FIG. 12 is a block diagram showing a configuration of an image processing system that generates a virtual viewpoint image according to the present embodiment. The image processing system 1'has each element of the camera array 10', the foreground extraction device group 12', the control device 13', and the three-dimensional model generation device 14'with respect to the image processing system 1 shown in FIG. 1 of the first embodiment. Has been added. The functions of the additional elements 10', 12', 13', and 14'are the same as those of the elements 10, 12, 13, and 14 in FIG. 1 of the first embodiment.

The camera array 10 and each camera constituting the camera array 10'are arranged so as to surround the field in the stadium as shown in FIG. For convenience of illustration, the camera array 10'is arranged outside the camera array 10 and appears to be far from the field, but the actual distance to the field is substantially the same for the camera array 10 and the camera array 10'. In the case of the present embodiment, each camera 10a-10p constituting the camera array 10 has a point T1 on the field as a gazing point, and each camera 10a'-10p' constituting the camera array 10' has a point T2 on the field. As a viewpoint, imaging is performed by synchronizing the time.
The three-dimensional model generation device 14 that receives the input data from the foreground extraction device group 12 and the three-dimensional model generation device 14'that receives the input data from the foreground extraction device group 12'are connected to each other. Then, the three-dimensional model generation device 14 and the three-dimensional model generation device 14'provide a part of the rendering model generated by their own device as a three-dimensional model for visibility determination to the other device.

<Software configuration>
FIG. 14 is a functional block diagram showing software configurations of the three-dimensional model generation devices 14 and 14'of the present embodiment. The three-dimensional model generation devices 14 and 14'have an input unit 201, a model generation condition setting unit 202', a rendering model generation unit 203, a model integration unit 205', a visibility determination unit 206', and an output unit 207. Then, instead of the determination model generation unit 204, the determination model transmission / reception unit 1401 and the determination area setting unit 1402 are provided. Hereinafter, the functions of each unit will be described, but the same processing blocks as in the first embodiment (input unit 201, rendering model generation unit 203, visibility determination unit 206, and output unit 207) will be omitted.

The model generation condition setting unit 202'sets the generation conditions (rendering area and voxel size) when the rendering model generation unit 203 generates a three-dimensional model for rendering based on the user input via the GUI. .. In the present embodiment, since a part of the rendering three-dimensional model generated by the own device becomes the determination model in the other three-dimensional model generation device, the determination model generation unit 204 does not exist. Therefore, the model generation condition setting unit 202'of the present embodiment does not set the generation condition of the determination model.

The determination model transmission / reception unit 1401 transmits a part of the rendering model generated by its own device as a determination model based on an acquisition request from the other three-dimensional model generation device. In addition, the other 3D model generator is requested to acquire the judgment model, and a part of the rendering model generated by the other 3D model generator is received as the judgment model in the own device. To do. FIG. 15A shows a rendering area _1 corresponding to the gazing point T1 of the camera array 10 shown in FIG. 12 and a rendering area _2 corresponding to the gazing point T2 of the camera array 10'. Then, FIG. 15B shows a transmission area and a reception area for the three-dimensional model generation device 14 corresponding to the camera array 10. Here, the transmission area is a part of the rendering area _1 and refers to a target area when the rendering model generated by the own device is transmitted as the determination model. This transmission area is specified in the incidental information of the income request from the other device 14'. The reception area is an area (a part of the rendering area _2) in which the determination model received from the other device 14'is generated, and refers to the determination area _1 in the own device. This reception area is specified in the incidental information of the income request sent from the own device 14 to the other device 14'. Similarly, FIG. 15C shows a transmission area and a reception area for the 3D model generator 14'corresponding to the camera array 10'. The transmission area in this case is a part of the rendering area _2 and refers to the area to be the target when transmitting the rendering model generated by the own device. Further, the receiving area is an area (a part of the area in the rendering area _1) of the determination model received from the other device 14, and refers to the determination area _2 in the own device. As described above, in the case of the present embodiment, a part of the three-dimensional model generated by one three-dimensional model generator is used as a determination model in the visibility determination process in the other three-dimensional model generator. It will be.

The determination area setting unit 1402 sets the determination area (that is, the reception area) in the own device, which is defined in the above-mentioned incidental information of the income request, based on the user input. As described above, the area to be set at this time is a part of the other rendering area and is adjacent to the rendering area in the own device.

The model integration unit 205'integrates the rendering model generated by the rendering model generation unit 203 and the determination model received by the determination model transmission / reception unit 1401 from the other three-dimensional model generation device.

<Processing flow in 3D model generator>
FIG. 16 is a flowchart showing a processing flow in the three-dimensional model generation device 14/14'according to the present embodiment. Before the start of execution of the flowchart of FIG. 16, it is assumed that the camera parameters have been received from the control device 13 and have already been set in the RAM 103 or the like, and the generation conditions and transmission area of the rendering model have been set based on the user input. Further, it is assumed that the request for acquiring the determination model from the other three-dimensional model generator has already been received before the start of execution of this flow. Hereinafter, the flow of processing in the three-dimensional model generator 14/14'will be described with reference to the flowchart of FIG.

In S1601, the input unit 201 monitors the reception of input data (that is, data of the mask image and the texture image for each camera) necessary for generating the three-dimensional model, as in S1101 of the flow of FIG. If the reception of the input data is detected, the process proceeds to S1602. Since the present embodiment also assumes that the multi-viewpoint image data is a moving image, the processing after S1602 is executed in frame units.

In S1602, similarly to S1102 of the flow of FIG. 11, the rendering model generation unit 203 generates a three-dimensional model for rendering using the input data received in S1601. The generated rendering model is provided to the determination model transmission / reception unit 1401 in addition to the model integration unit 205'and the output unit 207.
In S1603, the determination model transmission / reception unit 1401 extracts the rendering model existing in the region specified by the acquisition request from the other three-dimensional model generation device among the rendering models generated in S1602. The rendering model extracted here becomes a determination model in the other three-dimensional model generator.

In S1604, the determination model transmission / reception unit 1401 transmits the rendering model extracted in S1603 to the other three-dimensional model generation device that is the transmission source of the acquisition request. In the following S1605, the determination model transmission / reception unit 1401 receives the determination model from the other three-dimensional model generator based on the acquisition request transmitted in advance.

Subsequent steps of S1606 to S1609 correspond to S1104 to S1107 in the flow of FIG. That is, the rendering model generated in S1602 and the judgment model received in S1605 are arranged in a common three-dimensional space (S1606), and the visibility judgment processing is executed for each camera for each voxel constituting the rendering model. (S1607). Then, the rendering model generated in S1602 and the result of the visibility determination obtained in S1607 are output to the rendering device 15 together with the texture image data (S1608), and this is repeated for all frames (S1609).

The above is the flow of processing in the three-dimensional model generator 14/14'related to the present embodiment. It should be noted that the output in the output unit 207 may be a plurality of frame units or an input data unit instead of the frame unit, as in the first embodiment. Further, in the present embodiment, an example in which both three-dimensional models are interchanged between two three-dimensional model devices has been described, but the present invention is not limited to this. For example, the camera array, the foreground extraction device group, and the like may be further increased to set three or more rendering areas, and the three-dimensional models generated by each of the three or more three-dimensional model generation devices may be interchanged.

Also in this embodiment, it is possible to accurately determine the occurrence location of occlusion in the three-dimensional model to be rendered while suppressing the increase in processing load. In particular, in the case of the present embodiment, since the three-dimensional model for rendering generated by the other three-dimensional model generator is used as the judgment model, the visibility judgment can be performed with higher accuracy and the quality is higher. Virtual viewpoint video can be obtained.

<Other Examples>
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

1 Image processing system 14 Three-dimensional model generation device 203 Rendering model generation unit 204 Judgment model generation unit 206 Visibility judgment unit 1401 Judgment model transmission / reception unit

Claims (14)

Shape data representing the three-dimensional shape of an object included in at least one of the multi-viewpoint images obtained by synchronous imaging with a plurality of imaging devices is generated in a predetermined three-dimensional region in the target three-dimensional space of the synchronous imaging. Generation means to be
An acquisition means for acquiring shape data generated in a three-dimensional region different from the predetermined three-dimensional region in the target three-dimensional space for synchronous imaging, and
Using the shape data acquired by the acquisition means, a determination means for determining the visibility of the shape data generated by the generation means from the image pickup apparatus, and
An output means that outputs the shape data generated by the generation means together with the result of the visibility determination, and
Have,
The image processing apparatus is characterized in that the determination means determines whether or not the shape data generated by the generation means is blocked by the shape data acquired by the acquisition means. The image processing apparatus according to claim 1, wherein a three-dimensional region different from the predetermined three-dimensional region is adjacent to the predetermined three-dimensional region. The acquisition means represents a three-dimensional shape of an object included in at least one of a plurality of viewpoint images obtained by synchronously imaging with the plurality of imaging devices in a three-dimensional region different from the predetermined three-dimensional region. The image processing apparatus according to claim 1 or 2, wherein the image processing apparatus is acquired by generating shape data. A three-dimensional region different from the predetermined three-dimensional region is determined by calculation based on the shape and size of the object, the position and orientation of each of the plurality of imaging devices, and the shape and size of the predetermined three-dimensional region. The image processing apparatus according to claim 3, wherein the image processing apparatus is characterized by the above. The image processing apparatus according to claim 3 or 4, wherein the unit of the component of the shape data acquired by the acquisition means is smaller than the unit of the component of the shape data generated by the generation means. The fifth aspect of claim 5 is characterized in that the determination means determines, in units of the components, whether or not the shape data generated by the generation means is blocked by the shape data acquired by the acquisition means. Image processing equipment. The determination means
The coordinate system of the shape data generated by the generation means and the shape data acquired by the acquisition means is converted from the world coordinate system to the image pickup device coordinate system.
When the position of the shape data acquired by the acquisition means is closer to the image pickup device of interest, it is determined that the shape data generated by the generation means is blocked by the shape data acquired by the acquisition means. The image processing apparatus according to claim 6, wherein the image processing apparatus is used. An external device having means for generating shape data representing a three-dimensional shape of an object to be captured in a multi-viewpoint image obtained by synchronous imaging with the plurality of imaging devices in a three-dimensional region different from the predetermined three-dimensional region. Connected with
The acquisition means acquires by receiving shape data generated in a three-dimensional region different from the predetermined three-dimensional region in the target three-dimensional space of the synchronous imaging from the external device.
The image processing apparatus according to claim 1 or 2. The image processing apparatus according to claim 8, wherein the unit of the component of the shape data acquired by the acquisition means is the same as the unit of the component of the shape data generated by the generation means. The ninth aspect of the present invention is characterized in that the determination means determines, in units of the components, whether or not the shape data generated by the generation means is blocked by the shape data acquired by the acquisition means. Image processing equipment. The determination means
The coordinate system of the shape data generated by the generation means and the shape data acquired by the acquisition means is converted from the world coordinate system to the image pickup device coordinate system.
When the position of the shape data acquired by the acquisition means is closer to the image pickup apparatus, it is determined that the shape data generated by the generation means is blocked by the shape data acquired by the acquisition means. The image processing apparatus according to claim 10. Shape data representing the three-dimensional shape of an object included in at least one of the multi-viewpoint images obtained by synchronous imaging with a plurality of imaging devices is generated in a predetermined three-dimensional region in the target three-dimensional space of the synchronous imaging. Generation means to be
An acquisition means for acquiring shape data generated in a three-dimensional region different from the predetermined three-dimensional region in the target three-dimensional space for synchronous imaging, and
Using the shape data acquired by the acquisition means, a determination means for determining the visibility of the shape data generated by the generation means from the image pickup apparatus, and
An output means that outputs the shape data generated by the generation means together with the result of the visibility determination, and
An acquisition means for acquiring information of a virtual viewpoint set in the three-dimensional space where the synchronous imaging was performed, and
A rendering means for coloring the shape data output by the output means based on the information of the virtual viewpoint acquired by the acquisition means and the result of the visibility determination output by the output means.
Have,
The image processing system is characterized in that the determination means determines whether or not the shape data generated by the generation means is blocked by the shape data acquired by the acquisition means. Shape data representing the three-dimensional shape of an object included in at least one of the multi-viewpoint images obtained by synchronous imaging with a plurality of imaging devices is generated in a predetermined three-dimensional region in the target three-dimensional space of the synchronous imaging. Generation steps and
An acquisition step of acquiring shape data generated in a three-dimensional region different from the predetermined three-dimensional region in the target three-dimensional space of the synchronous imaging, and
Using the shape data acquired in the acquisition step, a determination step of determining the visibility of the shape data generated in the generation step from the imaging device, and
An output step that outputs the shape data generated in the generation step together with the result of the visibility determination,
Including
The image processing method is characterized in that the determination step determines whether or not the shape data generated in the generation step is blocked by the shape data acquired in the acquisition step. A program for causing a computer to function as the image processing device according to any one of claims 1 to 11.

Images