JP7016899B2 - Image processing equipment, image processing system, image processing method and program - Google Patents

Description

The present invention relates to a technique for generating a virtual viewpoint image.

Recently, a technique of installing a plurality of cameras at different positions to shoot a subject from multiple viewpoints and generating a virtual viewpoint image or a three-dimensional model using the multiple viewpoint images obtained by the shooting has attracted attention. According to the technology for generating a virtual viewpoint image from a multi-viewpoint image as described above, for example, a highlight scene of soccer or basketball can be viewed from various angles, so that the user can see the highlight scene as compared with a normal image. It can give a high sense of presence.

In Patent Document 1, when images taken from a plurality of viewpoints are combined to generate a virtual viewpoint image, the image quality of the virtual viewpoint image is improved by reducing the rendering unit in the boundary region of the object in the image. Is described.

Japanese Unexamined Patent Publication No. 2013-223008

However, with the conventional technology, it may not be possible to generate a virtual viewpoint image according to a plurality of different requirements. For example, when only a high-quality virtual viewpoint image is generated, the processing time for generation may be long, and it is difficult to meet the requirements of users who want to see the virtual viewpoint image in real time even if the image quality is low. There is a risk of becoming. On the other hand, when only a low-quality virtual viewpoint image is generated, it may be difficult to meet the user's requirement that the virtual viewpoint image has high image quality rather than real-time performance.

The present invention has been made in view of the above problems, and an object of the present invention is to generate a virtual viewpoint image according to a plurality of different requirements.

In order to solve the above problems, the image processing system according to the present invention has, for example, the following configuration. That is, an image acquisition means for acquiring a plurality of images based on images taken from different positions by a plurality of photographing devices, a reception means for receiving an input according to an operation for designating a virtual viewpoint, and the image acquisition means. A generation means for generating a plurality of virtual viewpoint images based on the plurality of images and the input received by the reception means, the first virtual viewpoint image used for the operation, and the said. A second virtual viewpoint image corresponding to a virtual viewpoint designated based on an operation and having a larger image data size per frame of a moving image than the first virtual viewpoint image. The number of photographing devices having the generation means to generate and corresponding to a plurality of images used by the generation means to generate the second virtual viewpoint image is the number of the first virtual viewpoint image by the generation means. The number of photographing devices corresponding to the plurality of images used for generation is larger than the number of photographing devices corresponding to the plurality of images used for generation, and the plurality of photographing devices corresponding to the plurality of images used for generation of the second virtual viewpoint image by the generation means are produced by the generation means. A plurality of photographing devices corresponding to a plurality of images used for generating the first virtual viewpoint image are included.

According to the present invention, it is possible to generate a virtual viewpoint image according to a plurality of different requirements.

It is a figure for demonstrating the structure of the image processing system 10. It is a figure for demonstrating the hardware composition of the image processing apparatus 1. FIG. It is a flowchart for demonstrating one form of operation of image processing apparatus 1. It is a figure for demonstrating the composition of the display screen by a display device 3. It is a flowchart for demonstrating one form of operation of image processing apparatus 1. It is a flowchart for demonstrating one form of operation of image processing apparatus 1.

[System configuration]
Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, the configuration of the image processing system 10 that generates and outputs a virtual viewpoint image will be described with reference to FIG. 1. The image processing system 10 in the present embodiment includes an image processing device 1, a camera group 2, a display device 3, and a display device 4.

The virtual viewpoint image in the present embodiment is an image obtained when the subject is photographed from a virtual viewpoint. In other words, a virtual viewpoint image is an image that represents the appearance at a specified viewpoint. The virtual viewpoint (virtual viewpoint) may be specified by the user, or may be automatically specified based on the result of image analysis or the like. That is, the virtual viewpoint image includes an arbitrary viewpoint image (free viewpoint image) corresponding to the viewpoint arbitrarily designated by the user. Further, an image corresponding to a viewpoint designated by the user from a plurality of candidates and an image corresponding to the viewpoint automatically designated by the device are also included in the virtual viewpoint image. In the present embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the virtual viewpoint image may be a still image.

The camera group 2 includes a plurality of cameras, and each camera shoots a subject from a different direction. In the present embodiment, each of the plurality of cameras included in the camera group 2 is connected to the image processing device 1, and the captured image, the parameters of each camera, and the like are transmitted to the image processing device 1. However, the present invention is not limited to this, and a plurality of cameras included in the camera group 2 can communicate with each other, and any camera included in the camera group 2 can process images taken by the plurality of cameras, parameters of the plurality of cameras, and the like as an image processing device. It may be transmitted to 1. Further, even if any of the cameras included in the camera group 2 transmits an image based on the shooting by the camera group 2, such as an image generated based on the difference between the shot images taken by a plurality of cameras, instead of the shot image. good.

The display device 3 accepts the designation of the virtual viewpoint for generating the virtual viewpoint image, and transmits the information according to the designation to the image processing device 1. For example, the display device 3 has an input unit such as a joystick, a jog dial, a touch panel, a keyboard, and a mouse, and a user (operator) who designates a virtual viewpoint designates a virtual viewpoint by operating the input unit. The user in the present embodiment is an operator who operates the input unit of the display device 3 to specify a virtual viewpoint or a viewer who sees a virtual viewpoint image displayed by the display device 4, and particularly includes the operator and the viewer. If there is no distinction, simply describe as user. In this embodiment, the case where the viewer and the operator are different will be mainly described, but the present invention is not limited to this, and the viewer and the operator may be the same user. In the present embodiment, the information according to the designation of the virtual viewpoint transmitted from the display device 3 to the image processing device 1 is virtual viewpoint information indicating the position and direction of the virtual viewpoint. However, the present invention is not limited to this, and the information according to the designation of the virtual viewpoint may be information indicating the contents determined according to the virtual viewpoint such as the shape and orientation of the subject in the virtual viewpoint image, and the image processing device 1 may be such information. A virtual viewpoint image may be generated based on the information according to the designation of the viewpoint.

Further, the display device 3 displays the virtual viewpoint image generated and output by the image processing device 1 based on the image taken by the camera group 2 and the designation of the virtual viewpoint accepted by the display device 3. As a result, the operator can specify the virtual viewpoint while looking at the virtual viewpoint image displayed on the display device 3. In the present embodiment, the display device 3 that displays the virtual viewpoint image accepts the designation of the virtual viewpoint, but the present invention is not limited to this. For example, the device that accepts the designation of the virtual viewpoint and the display device that displays the virtual viewpoint image for causing the operator to specify the virtual viewpoint may be separate devices.

Further, the display device 3 gives a generation instruction to the image processing device 1 to start the generation of the virtual viewpoint image based on the operation by the operator. The generation instruction is not limited to this, and may be an instruction for reserving the image processing device 1 to generate the virtual viewpoint image so that the generation of the virtual viewpoint image is started at a predetermined time, for example. Further, for example, it may be an instruction for making a reservation so that the generation of the virtual viewpoint image is started when a predetermined event occurs. The device that gives an instruction to generate a virtual viewpoint image to the image processing device 1 may be a device different from the display device 3, or the user may directly input the generation instruction to the image processing device 1. ..

The display device 4 transfers the virtual viewpoint image generated by the image processing device 1 based on the designation of the virtual viewpoint by the operator using the display device 3 to a user (viewer) different from the operator who designates the virtual viewpoint. Display against. The image processing system 10 may have a plurality of display devices 4, and the plurality of display devices 4 may display different virtual viewpoint images. For example, the image processing system 10 includes a display device 4 that displays a virtual viewpoint image (live image) that is broadcast live, and a display device 4 that displays a virtual viewpoint image (non-live image) that is broadcast after recording. You may.

The image processing device 1 has a camera information acquisition unit 100, a virtual viewpoint information acquisition unit 110 (hereinafter referred to as a viewpoint acquisition unit 110), an image generation unit 120, and an output unit 130. The camera information acquisition unit 100 acquires an image based on shooting by the camera group 2, external parameters and internal parameters of each camera included in the camera group 2 from the camera group 2, and outputs the image to the image generation unit 120. The viewpoint acquisition unit 110 acquires information according to the designation of the virtual viewpoint by the operator from the display device 3 and outputs the information to the image generation unit 120. Further, the viewpoint acquisition unit 110 receives an instruction to generate a virtual viewpoint image by the display device 3. The image generation unit 120 is based on an image based on shooting acquired by the camera information acquisition unit 100, information according to the designation acquired by the viewpoint acquisition unit 110, and a generation instruction received by the viewpoint acquisition unit 110. , A virtual viewpoint image is generated and output to the output unit 130. The output unit 130 outputs the virtual viewpoint image generated by the image generation unit 120 to an external device such as the display device 3 or the display device 4.

In the present embodiment, the image processing device 1 generates a plurality of virtual viewpoint images having different image quality and outputs them to an output destination corresponding to each virtual viewpoint image. For example, a low-quality virtual viewpoint image with a short processing time related to generation is output to the display device 4 viewed by a viewer who desires a real-time (low delay) virtual viewpoint image. On the other hand, the display device 4 viewed by the viewer who desires the high-quality virtual viewpoint image outputs the high-quality virtual viewpoint image having a long processing time related to the generation. The delay in the present embodiment corresponds to the period from the shooting by the camera group 2 to the display of the virtual viewpoint image based on the shooting. However, the definition of delay is not limited to this, and for example, the time difference between the time in the real world and the time corresponding to the displayed image may be defined as the delay.

Subsequently, the hardware configuration of the image processing apparatus 1 will be described with reference to FIG. The image processing device 1 includes a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a display unit 205, an operation unit 206, a communication unit 207, and a bus 208. The CPU 201 controls the entire image processing device 1 by using computer programs and data stored in the ROM 202 and the RAM 203. The image processing device 1 may have a GPU (Graphics Processing Unit), and the GPU may perform at least a part of the processing by the CPU 201. The ROM 202 stores programs and parameters that do not need to be changed. The RAM 203 temporarily stores programs and data supplied from the auxiliary storage device 204, data supplied from the outside via the communication unit 207, and the like. The auxiliary storage device 204 is composed of, for example, a hard disk drive or the like, and stores content data such as still images and moving images.

The display unit 205 is composed of, for example, a liquid crystal display or the like, and displays a GUI (Graphical User Interface) for the user to operate the image processing device 1. The operation unit 206 is composed of, for example, a keyboard, a mouse, or the like, and inputs various instructions to the CPU 201 in response to an operation by the user. The communication unit 207 communicates with an external device such as the camera group 2, the display device 3, and the display device 4. For example, when the image processing device 1 is connected to an external device by wire, a LAN cable or the like is connected to the communication unit 207. When the image processing device 1 has a function of wirelessly communicating with an external device, the communication unit 207 includes an antenna. The bus 208 connects each part of the image processing device 1 to transmit information.

In the present embodiment, the display unit 205 and the operation unit 206 exist inside the image processing device 1, but the image processing device 1 does not have to include at least one of the display unit 205 and the operation unit 206. Further, at least one of the display unit 205 and the operation unit 206 exists as another device outside the image processing device 1, and the CPU 201 controls the display control unit 205 that controls the display unit 205 and the operation unit 206. It may operate as a control unit.

[Operation flow]
Next, one mode of operation of the image processing apparatus 1 will be described with reference to FIG. The process shown in FIG. 3 is started at the timing when the viewpoint acquisition unit 110 receives the instruction to generate the virtual viewpoint image, and is repeated periodically (for example, every frame when the virtual viewpoint image is a moving image). However, the start timing of the process shown in FIG. 3 is not limited to the above timing. The process shown in FIG. 3 is realized by the CPU 201 expanding the program stored in the ROM 202 into the RAM 203 and executing the program. It should be noted that at least a part of the processing shown in FIG. 3 may be realized by dedicated hardware different from the CPU 201.

In the flow shown in FIG. 3, S2010 and S2020 correspond to the process of acquiring information, and S2030-S2050 correspond to the process of generating and outputting a virtual viewpoint image (designation image) for the operator to specify a virtual viewpoint. do. Further, S2070-S2100 corresponds to a process of generating and outputting a live image. S2110-S2130 corresponds to a process of generating and outputting a non-live image. Hereinafter, the details of the processing in each step will be described.

In S2010, the camera information acquisition unit 100 acquires an image taken by each camera based on the image taken by the camera group 2, and external parameters and internal parameters of each camera. External parameters are information about the position and orientation of the camera, and internal parameters are information about the focal length and image center of the camera.

In S2020, the viewpoint acquisition unit 110 acquires virtual viewpoint information as information according to the designation of the virtual viewpoint by the operator. In the present embodiment, the virtual viewpoint information corresponds to the external parameters and the internal parameters of the virtual camera that captures the subject from the virtual viewpoint, and one virtual viewpoint information is required to generate one frame of the virtual viewpoint image.

In S2030, the image generation unit 120 estimates the three-dimensional shape of the object to be the subject based on the image captured by the camera group 2. The object to be a subject is, for example, a person or a moving object existing in the shooting range of the camera group 2. The image generation unit 120 extracts a portion (foreground area) corresponding to an object in the captured image by calculating the difference between the captured image acquired from the camera group 2 and the background image corresponding to each camera acquired in advance. Generate a silhouette image that has been created. Then, the image generation unit 120 estimates the three-dimensional shape of the object by using the silhouette image corresponding to each camera and the parameters of each camera. For example, the Visual Hull method is used to estimate the three-dimensional shape. As a result of this processing, a 3D point cloud (a set of points having three-dimensional coordinates) expressing the three-dimensional shape of the object to be the subject is obtained. The method of deriving the three-dimensional shape of the object from the image captured by the camera group 2 is not limited to this.

In S2040, the image generation unit 120 renders a 3D point cloud and a background 3D model based on the acquired virtual viewpoint information, and generates a virtual viewpoint image. The background 3D model is, for example, a CG model of a stadium where a camera group 2 is installed, and is created in advance and stored in the image processing system 10. In the virtual viewpoint image generated by the processing up to this point, the area corresponding to the object and the background area are each displayed in a predetermined color (for example, one color). The process of rendering a 3D point cloud or a background 3D model is known in the fields of games and movies, and a method for performing high-speed processing such as a method using a GPU is known. Therefore, the virtual viewpoint image generated by the processing up to S2040 can be generated at high speed according to the shooting by the camera group 2 and the designation of the virtual viewpoint by the operator.

In S2050, the output unit 130 outputs the virtual viewpoint image generated in S2040 by the image generation unit 120 to the display device 3 for allowing the operator to specify the virtual viewpoint. Here, the screen configuration of the display screen 30 displayed by the display device 3 will be described with reference to FIG. The display screen 30 is composed of an area 310, an area 320, and an area 330. For example, the virtual viewpoint image generated as a designated image is displayed in the area 310, the virtual viewpoint image generated as a live image is displayed in the area 320, and the virtual viewpoint image generated as a non-live image is displayed in the area 330. Will be done. That is, the virtual viewpoint image generated in S2040 and output in S2050 is displayed in the area 310. Then, the operator specifies the virtual viewpoint while looking at the screen of the area 310. The display device 3 may display at least a designated image, and may not display a live image or a non-live image.

In S2060, the image generation unit 120 determines whether or not to perform a process of generating a virtual viewpoint image having a higher image quality than the virtual viewpoint image generated in S2040. For example, if only a low-quality image for designating a virtual viewpoint is required, the process does not proceed to S2070 and the process ends. On the other hand, when a higher image quality image is required, the process proceeds to S2070 and the process is continued.

In S2070, the image generation unit 120 further improves the accuracy of the object shape model (3D point cloud) estimated in S2030 by using, for example, the PhotoHull method. Specifically, by projecting each point of the 3D point cloud onto the captured image of each camera and evaluating the degree of color matching in each captured image, whether or not that point is necessary for expressing the subject shape. To judge. For example, if the variance of the pixel value of the projection destination is larger than the threshold value for a certain point in the 3D point cloud, it is determined that the point is not correct as a point representing the shape of the subject, and the point is deleted from the 3D point cloud. .. This process is performed on all points in the 3D point cloud to realize high accuracy of the object shape model. The method for improving the accuracy of the object shape model is not limited to this.

In S2080, the image generation unit 120 colors the 3D point cloud refined in S2070 and projects it on the coordinates of the virtual viewpoint to generate a foreground image corresponding to the foreground region, and from the virtual viewpoint. Executes the process of generating the seen background image. Then, the image generation unit 120 generates a virtual viewpoint image as a live image by superimposing the foreground image on the generated background image.

Here, an example of a method of generating a foreground image (an image of an area corresponding to an object) of a virtual viewpoint image will be described. In order to generate a foreground image, a process of coloring a 3D point cloud is executed. The coloring process consists of point visibility determination and color calculation processing. In the determination of visibility, a camera capable of photographing each point can be specified from the positional relationship between each point in the 3D point cloud and a plurality of cameras included in the camera group 2. Next, for each point, the point is projected onto the captured image of the camera capable of capturing the point, and the color of the pixel at the projection destination is set as the color of the point. When a point can be photographed by a plurality of cameras, the color of the point is determined by projecting the point on the images captured by the plurality of cameras, acquiring the pixel value of the projection destination, and calculating the average of the pixel values. By rendering the 3D point cloud colored in this way by the existing CG rendering method, it is possible to generate a foreground image of the virtual viewpoint image.

Next, an example of a method of generating a background image of a virtual viewpoint image will be described. First, the vertices of the background 3D model (for example, the points corresponding to the edges of the stadium) are set. Then, these vertices are projected onto the coordinate system of the two cameras (the first camera and the second camera) close to the virtual viewpoint and the coordinate system of the virtual viewpoint. Further, using the corresponding point between the virtual viewpoint and the first camera and the corresponding point between the virtual viewpoint and the second camera, the first projection matrix between the virtual viewpoint and the first camera and the second between the virtual viewpoint and the second camera. 2 Projection matrix is calculated. Then, using the first projection matrix and the second projection matrix, each pixel of the background image is projected onto the captured image of the first camera and the captured image of the second camera, and the average of the two pixel values of the projection destination is calculated. As a result, the pixel value of the background image is determined. The pixel value of the background image may be determined from the images taken by three or more cameras by the same method.

By superimposing the foreground image on the background image of the virtual viewpoint image obtained in this way, a colored virtual viewpoint image can be generated. That is, the virtual viewpoint image generated in S2080 has a higher image quality in terms of the number of color gradations than the virtual viewpoint image generated in S2040. Conversely, the number of gradations of colors included in the virtual viewpoint image generated in S2040 is less than the number of gradations of colors included in the virtual viewpoint image generated in S2080. The method of adding color information to the virtual viewpoint image is not limited to this.

In S2090, the output unit 130 outputs the virtual viewpoint image generated in S2080 by the image generation unit 120 to the display device 3 and the display device 4 as a live image. The image output to the display device 3 is displayed in the area 320 and can be viewed by the operator, and the image output to the display device 4 can be viewed by the viewer.

In S2100, the image generation unit 120 determines whether or not to perform a process of generating a virtual viewpoint image having a higher image quality than the virtual viewpoint image generated in S2080. For example, when the virtual viewpoint image is provided to the viewer only by live broadcasting, the process does not proceed to S2110 and the process ends. On the other hand, when broadcasting a higher quality image to the viewer after recording, the process proceeds to S2110 and the process is continued.

In S2110, the image generation unit 120 further improves the accuracy of the shape model of the object generated in S2070. In the present embodiment, high accuracy is realized by deleting isolated points of the shape model. In the isolated point removal, first, for the voxel set (3D point cloud) calculated by PhotoHull, it is examined whether or not another voxel exists around each voxel. If there are no voxels around, the voxel is determined to be an isolated point and the voxel is removed from the voxel set. By executing the same processing as S2080 using the shape model with isolated points deleted in this way, a virtual viewpoint image with a higher accuracy in the shape of the object than the virtual viewpoint image generated in S2080 is generated. To.

In S2120, the image generation unit 120 applies a smoothing process to the boundary between the foreground area and the background area of the virtual viewpoint image generated in S2110, and corrects the image so that the boundary area is displayed smoothly.

In S2130, the output unit 130 outputs the virtual viewpoint image generated in S2120 by the image generation unit 120 to the display device 3 and the display device 4 as a non-live image. The non-live image output to the display device 3 is displayed in the area 330.

By the above processing, the image processing apparatus 1 generates a virtual viewpoint image as a designated image and a live image which is a virtual viewpoint image having higher image quality than the designated image based on a set of captured images and virtual viewpoint information. do. The image processing device 1 also generates a non-live image, which is a virtual viewpoint image having a higher image quality than the live image. Then, the image processing device 1 outputs the generated live image and non-live image to the display device 4 so that the live image is displayed before the non-live image is displayed. Further, the image processing device 1 outputs the generated designated image to the display device 3 so that the designated image is displayed on the display device 3 before the live image is displayed on the display device 4.

As a result, the display device 4 displays a low-quality designated image, a live image having a higher image quality than the designated image and being broadcast live, and a non-live image having a higher image quality than the live image and being broadcast after recording. It becomes possible to do. The display device 4 may display only one of the live image and the non-live image, and in that case, the image processing device 1 outputs a virtual viewpoint image suitable for the display device 4. Further, the display device 3 has three types of virtual viewpoint images: a low-quality virtual viewpoint image as a designated image, a medium-quality virtual viewpoint image as a live image, and a high-quality virtual viewpoint image as a non-live image. It becomes possible to display a viewpoint image. The display device 3 does not have to display at least one of a live image and a non-live image.

That is, the image processing device 1 outputs a designation image to the display device 3 for allowing the user to specify the virtual viewpoint. Then, the image processing device 1 is at least one of a live image and a non-live image having higher image quality than the designated image for the display device 4 for displaying the virtual viewpoint image generated based on the designation of the virtual viewpoint by the user. Is output. This makes it possible to meet the requirements of both an operator who wants to display a virtual viewpoint image with low delay in order to specify a virtual viewpoint and a viewer who wants to see a high-quality virtual viewpoint image.

In the above processing, a virtual viewpoint image is generated based on the image taken by the camera group 2 and the information according to the designation of the virtual viewpoint, and the image quality is higher based on the result of the processing for the generation. A virtual viewpoint image is generated. Therefore, the total amount of processing can be reduced as compared with the case where the low-quality virtual viewpoint image and the high-quality virtual viewpoint image are generated by independent processing. However, a low-quality virtual viewpoint image and a high-quality virtual viewpoint image may be generated by independent processing. Further, when the virtual viewpoint image is displayed on a display installed in a competition venue or a live venue or is broadcast live, and it is not necessary to broadcast after recording, the image processing device 1 generates a non-live image. Do not perform any processing to do so. As a result, the amount of processing for generating a high-quality non-live image can be reduced.

Next, another mode of operation of the image processing apparatus 1 will be described with reference to FIG. In the operation mode described above with reference to FIG. 3, a high-quality virtual viewpoint image is generated by additionally performing a new type of processing after generating a low-quality virtual viewpoint image. On the other hand, in the operation mode described below with reference to FIG. 5, the image quality of the virtual viewpoint image is improved by increasing the number of cameras used to generate the virtual viewpoint image. In the following description, the description of the same parts as those in FIG. 3 will be omitted.

The process shown in FIG. 5 is started at the timing when the viewpoint acquisition unit 110 receives the instruction to generate the virtual viewpoint image. However, the start timing of the process of FIG. 5 is not limited to this. In S2010 and S2020, the image processing device 1 acquires the captured image and the virtual viewpoint information by each camera of the camera group 2 by the same processing as described with reference to FIG.

In S4030, the image generation unit 120 sets the number of cameras corresponding to the captured images used for generating the virtual viewpoint image. Here, the image generation unit 120 sets the number of cameras so that the processing of S4050-S4070 is completed in a processing time equal to or less than a predetermined threshold value (for example, the time corresponding to one frame when the virtual viewpoint image is a moving image). .. For example, it is assumed that the processing of S4050-S4070 is executed in advance using the images taken by 100 cameras, and the processing time is 0.5 seconds. In this case, if you want to complete the processing of S4050-S4070 within 0.016 seconds corresponding to one frame of the virtual viewpoint image having a frame rate of 60 fps (frame per second), set the number of cameras to three. ..

If it is determined in S4080 to continue image generation after the virtual viewpoint image is output by the processing of S4050-S4070, the process returns to S4030 and the number of cameras to be used is reset. Here, the allowable processing time is lengthened and the number of cameras is increased accordingly so that the virtual viewpoint image having higher image quality than the previously output virtual viewpoint image is generated. For example, the number of cameras corresponding to the captured image to be used is set to 20 so that the processing of S4050-S4070 is completed in a processing time of 0.1 seconds or less.

In S4040, the image generation unit 120 selects a camera corresponding to the captured image used for generating the virtual viewpoint image from the camera group 2 according to the number of cameras set in S4030. For example, when selecting three cameras from 100 cameras, the camera closest to the virtual viewpoint, the 34th camera and the 67th camera counting from the cameras are selected.

Further, when the number of captured images to be used is increased after the virtual viewpoint image is generated once and the second processing is performed, the shape model estimated in the first processing is further improved in accuracy. A camera other than the one selected at the first time is selected. Specifically, when selecting 20 cameras from 100 cameras, first select the camera closest to the virtual viewpoint from the cameras not selected in the first process, and then select the camera closest to the virtual viewpoint at intervals of 5 cameras. Select a camera. At this time, the camera already selected in the first time is skipped and the next camera is selected. For example, when generating the highest quality virtual viewpoint image as a non-live image, all the cameras included in the camera group 2 are selected, and the processing of S4050-S4070 is executed using the captured image of each camera. do.

The method of selecting a camera corresponding to the captured image to be used is not limited to this. For example, a camera close to the virtual viewpoint may be preferentially selected. In this case, in the shape estimation of the object to be the subject, the accuracy of the shape estimation of the back region that cannot be seen from the virtual viewpoint is low, but the accuracy of the shape estimation of the front region that can be seen from the virtual viewpoint is improved. That is, it is possible to preferentially improve the image quality of a region of the virtual viewpoint image that is easily visible to the viewer.

In S4050, the image generation unit 120 executes the shape estimation process of the object by using the image taken by the camera selected in S4040. The processing here is, for example, a combination of the processing in S2030 (Visual Hull) and the processing in S2070 (PhotoHull) in FIG. The process of VisualHull includes the process of calculating the logical product of the visual volumes of a plurality of cameras corresponding to the plurality of captured images to be used. In addition, the PhotoHull process includes a process of projecting each point of the shape model onto a plurality of captured images to calculate the consistency of pixel values. Therefore, the smaller the number of cameras corresponding to the captured image to be used, the lower the accuracy of shape estimation and the shorter the processing time.

In S4060, the image generation unit 120 executes the rendering process. The process here is the same as the process in S2080 of FIG. 3, and includes a 3D point cloud coloring process and a background image generation process. Both the 3D point cloud coloring process and the background image generation process include a process of determining a color by calculation using pixel values of corresponding points of a plurality of captured images. Therefore, the smaller the number of cameras corresponding to the captured image to be used, the lower the rendering accuracy and the shorter the processing time.

In S4070, the output unit 130 outputs the virtual viewpoint image generated in S4060 by the image generation unit 120 to the display device 3 and the display device 4.

In S4080, the image generation unit 120 determines whether or not to perform a process of generating a virtual viewpoint image having a higher image quality than the virtual viewpoint image generated in S4060. For example, the virtual viewpoint image generated in S4060 is an image for allowing the operator to specify a virtual viewpoint, and when further generating a live image, return to S4030 and increase the number of cameras used to make the live image. Generate a virtual viewpoint image as. Further, when a non-live image is generated after the live image is further generated, the number of cameras is further increased to generate a virtual viewpoint image as a non-live image. That is, since the number of cameras corresponding to the captured image used to generate the virtual viewpoint image as the live image is larger than the number of cameras corresponding to the captured image used to generate the virtual viewpoint image as the designated image. Live images have higher image quality than designated images. Similarly, the number of cameras corresponding to the captured image used to generate the virtual viewpoint image as a non-live image is larger than the number of cameras corresponding to the captured image used to generate the virtual viewpoint image as a live image. , Non-live images have higher image quality than live images.

In S4080, when it is determined that it is not necessary to generate a virtual viewpoint image having a higher image quality than the already generated virtual viewpoint image, or when it is determined that a virtual viewpoint image having a higher image quality cannot be generated. , End the process.

Through the above processing, the image processing apparatus 1 can generate and output a plurality of virtual viewpoint images whose image quality is gradually improved at appropriate timings. For example, by limiting the number of cameras used for generating the virtual viewpoint image to the number of cameras that can complete the generation process within the set processing time, it is possible to generate the designated image with less delay. Further, when generating a live image or a non-live image, it is possible to generate a higher image quality image by increasing the number of cameras used and performing the generation process.

Next, another mode of operation of the image processing apparatus 1 will be described with reference to FIG. In the operation mode described above with reference to FIG. 5, the image quality of the virtual viewpoint image is improved by increasing the number of cameras used to generate the virtual viewpoint image. On the other hand, in the operation mode described below with reference to FIG. 6, the image quality of the virtual viewpoint image is improved by gradually increasing the resolution of the virtual viewpoint image. In the following description, the description of the same parts as those in FIGS. 3 and 5 will be omitted. In the operation mode described below, the number of pixels of the generated virtual viewpoint image is always 4K (3840 × 2160), and whether the pixel value is calculated for each large pixel block or each small pixel block. Controls the resolution of the virtual viewpoint image. However, the present invention is not limited to this, and the resolution may be controlled by changing the number of pixels of the generated virtual viewpoint image.

The process shown in FIG. 6 is started at the timing when the viewpoint acquisition unit 110 receives the instruction to generate the virtual viewpoint image. However, the start timing of the process of FIG. 6 is not limited to this. In S2010 and S2020, the image processing device 1 acquires the captured image and the virtual viewpoint information by each camera of the camera group 2 by the same processing as described with reference to FIG.

In S5030, the image generation unit 120 sets the resolution of the virtual viewpoint image to be generated. Here, the image generation unit 120 sets the resolution so that the processing of S5050 and S4070 is completed within the processing time of a predetermined threshold value or less. For example, it is assumed that the processing of S5050 and S4070 in the case of generating a virtual viewpoint image having a 4K resolution is executed in advance, and the processing time is 0.5 seconds. In this case, if it is desired to complete the processing of S5050 and S4070 within 0.016 seconds corresponding to one frame of the virtual viewpoint image having a frame rate of 60 fps, the resolution is 0.016 / 0.5 = 1 / of 4K. It is necessary to make it 31.25 times or less. Therefore, if the resolution of the virtual viewpoint image is set to 1/8 times the 4K resolution in each of the vertical and horizontal directions, the number of pixel blocks for which the pixel value should be calculated becomes 1/64, and the processing can be completed in less than 0.016 seconds.

If it is determined in S4080 to continue image generation after the virtual viewpoint image is output by the processing of S5050 and S4070, the process returns to S5030 and the resolution is reset. Here, the allowable processing time is lengthened and the resolution is increased accordingly so that the virtual viewpoint image having higher image quality than the previously output virtual viewpoint image is generated. For example, if the resolution is set to 1/4 of the 4K resolution in each of the vertical and horizontal directions, the processing of S5050 and S4070 is completed in a processing time of 0.1 seconds or less. In S5040, the image generation unit 120 determines the position of the pixel for which the pixel value should be calculated in the virtual viewpoint image according to the resolution set in S5030. For example, when the resolution of the virtual viewpoint image is set to 1/8 of the 4K resolution, the pixel value is calculated for each of 8 pixels in the vertical and horizontal directions. Then, the same pixel value as the pixel (x, y) is set for the pixel existing between the pixel (x, y) from which the pixel value is calculated and the pixel (x + 8, y + 8).

Further, when the virtual viewpoint image is generated once and then the resolution is increased to perform the second processing, the pixel for which the pixel value is calculated for the first time is skipped and the pixel value is calculated. For example, when the resolution is set to 1/4 of the 4K resolution, the pixel value of the pixel (x + 4, y + 4) is calculated, and the pixel existing between the pixel (x + 4, y + 4) and the pixel (x + 8, y + 8) , The same pixel value as the pixel (x + 4, y + 4) is set. By increasing the number of pixels for which the pixel value is calculated in this way, the resolution of the virtual viewpoint image can be increased to a maximum of 4K resolution.

In S5050, the image generation unit 120 calculates the pixel value of the pixel at the position determined in S5040 and performs the coloring process on the virtual viewpoint image. As a method for calculating the pixel value, for example, the method of Image-Based Visual Hull can be used. In this method, the pixel value is calculated for each pixel, so that the smaller the number of pixels for which the pixel value should be calculated, that is, the lower the resolution of the virtual viewpoint image, the shorter the processing time.

In S4070, the output unit 130 outputs the virtual viewpoint image generated in S5050 by the image generation unit 120 to the display device 3 and the display device 4.

In S4080, the image generation unit 120 determines whether or not to perform a process of generating a virtual viewpoint image having a higher image quality than the virtual viewpoint image generated in S5050. For example, the virtual viewpoint image generated in S5050 is an image for allowing the operator to specify a virtual viewpoint, and when a live image is generated, the process returns to S5030 to generate a virtual viewpoint image with a higher resolution. .. Further, when a non-live image is generated after the live image is generated, a virtual viewpoint image as a non-live image with a higher resolution is generated. That is, since the virtual viewpoint image as a live image has a higher resolution than the virtual viewpoint image as a designated image, the live image has a higher image quality than the designated image. Similarly, since the virtual viewpoint image as a non-live image has a higher resolution than the virtual viewpoint image as a live image, the non-live image has a higher image quality than the live image.

In S4080, when it is determined that it is not necessary to generate a virtual viewpoint image having a higher image quality than the already generated virtual viewpoint image, or when it is determined that a virtual viewpoint image having a higher image quality cannot be generated. , End the process.

Through the above processing, the image processing apparatus 1 can generate and output a plurality of virtual viewpoint images whose resolutions are gradually improved at appropriate timings. For example, by setting the resolution of the virtual viewpoint image to a resolution that allows the generation process to be completed within the set processing time, it is possible to generate a designated image with little delay. Further, when a live image or a non-live image is generated, a higher image quality image can be generated by setting a high resolution and performing the generation process.

As described above, the image processing device 1 generates a high-quality image (for example, a non-live image) by performing image processing for improving the image quality of the virtual viewpoint image. Further, the image processing apparatus 1 generates a low-quality image (for example, a live image) by a partial processing included in the image processing and executed in a processing time equal to or less than a predetermined threshold value. This makes it possible to generate and display both a virtual viewpoint image displayed with a delay of a predetermined time or less and a high-quality virtual viewpoint image.

In the description of FIG. 6, a generation parameter (resolution) for completing the generation process in a processing time equal to or less than a predetermined threshold value is estimated, and a virtual viewpoint image is generated with the estimated generation parameter. However, the present invention is not limited to this, and the image processing apparatus 1 may gradually improve the image quality of the virtual viewpoint image and output the generated virtual viewpoint image when the processing time reaches a predetermined threshold value. For example, when the processing time reaches a predetermined threshold, a virtual viewpoint image having a resolution of 1/8 of the 4K resolution has already been generated, and a virtual viewpoint image having a resolution of 1/4 of the 4K resolution has not been completed. In some cases, a virtual viewpoint image having a resolution of 1/8 may be output. Further, a virtual viewpoint image may be output in which the process of improving the resolution from 1/8 resolution to 1/4 resolution is performed halfway.

In the present embodiment, the image generation unit 120 of the image processing device 1 controls the generation of a virtual viewpoint image based on the image acquired by the camera information acquisition unit 100 and the virtual viewpoint information acquired by the viewpoint acquisition unit 110. The explanation has focused on the case of generating multiple virtual viewpoint images with different image quality. However, the present invention is not limited to this, and different devices may have a function of controlling the generation of the virtual viewpoint image and a function of actually generating the virtual viewpoint image.

For example, in the image processing system 10, there may be a generation device (not shown) that has the function of the image generation unit 120 and generates a virtual viewpoint image. Then, the image processing device 1 may control the generation of the virtual viewpoint image by the generation device based on the image acquired by the camera information acquisition unit 100 and the information acquired by the viewpoint acquisition unit 110. Specifically, the image processing device 1 transmits the captured image and the virtual viewpoint information to the generation device, and gives an instruction to control the generation of the virtual viewpoint image. The generator is a first virtual viewpoint image and a second virtual viewpoint image that should be displayed at an earlier timing than the first virtual viewpoint image is displayed, and the image quality is lower than that of the first virtual viewpoint image. The second virtual viewpoint image is generated based on the received captured image and the virtual viewpoint information. Here, the first virtual viewpoint image is, for example, a non-live viewpoint image, and the second virtual viewpoint image is, for example, a live image. However, the use of the first virtual viewpoint image and the second virtual viewpoint image is not limited to this. The image processing device 1 may be controlled so that the first virtual viewpoint image and the second virtual viewpoint image are generated by different generation devices. Further, the image processing device 1 may perform output control such as controlling the output destination and output timing of the virtual viewpoint image by the generation device.

Further, the generation device has the functions of the viewpoint acquisition unit 110 and the image generation unit 120, and the image processing device 1 controls the generation of the virtual viewpoint image by the generation device based on the image acquired by the camera information acquisition unit 100. You may. Here, the image acquired by the camera information acquisition unit 100 is an image based on shooting, such as a captured image captured by the camera group 2 or an image generated based on the difference between a plurality of captured images. Further, the generation device has the functions of the camera information acquisition unit 100 and the image generation unit 120, and the image processing device 1 controls the generation of the virtual viewpoint image by the generation device based on the image acquired by the viewpoint acquisition unit 110. You may. Here, the image acquired by the viewpoint acquisition unit 110 is information according to the designation of the virtual viewpoint, such as information indicating contents determined according to the virtual viewpoint such as the shape and orientation of the subject in the virtual viewpoint image and virtual viewpoint information. That is, the image processing device 1 acquires information related to the generation of the virtual viewpoint image including at least one of the image based on the shooting and the information corresponding to the designation of the virtual viewpoint, and generates the virtual viewpoint image based on the acquired information. May be controlled.

Further, for example, the generation device existing in the image processing system 10 has the functions of the camera information acquisition unit 100, the viewpoint acquisition unit 110, and the image generation unit 120, and the image processing device 1 has information related to the generation of a virtual viewpoint image. The generation of the virtual viewpoint image by the generator may be controlled based on the above. The information related to the generation of the virtual viewpoint image in this case includes, for example, at least one of a parameter relating to the image quality of the first virtual viewpoint image and a parameter relating to the image quality of the second virtual viewpoint image generated by the generator. Specific examples of parameters related to image quality include the number of cameras corresponding to the captured image used to generate the virtual viewpoint image, the resolution of the virtual viewpoint image, the time allowed as the processing time related to the generation of the virtual viewpoint image, and the like. The image processing device 1 controls the generation device based on the acquired parameters, for example, acquiring parameters related to these image quality based on the input by the operator and transmitting the parameters to the generation device. This allows the operator to generate a plurality of virtual viewpoint images having different desired image quality.

As described above, the image processing device 1 receives an instruction to generate a virtual viewpoint image based on an image based on shooting of a subject from different directions by a plurality of cameras and information according to a designation of a virtual viewpoint. Then, in the image processing device 1, the first virtual viewpoint image output to the first display device and the second virtual viewpoint image output to the second display device correspond to the designation of the image based on shooting and the virtual viewpoint. Control is performed according to the acceptance of the generation instruction so that the image is generated based on the information received. Here, the second virtual viewpoint image is a virtual viewpoint image having higher image quality than the first virtual viewpoint image. This makes it suitable for the timing to be displayed even when there are both a user who wants to see the virtual viewpoint image in real time and a user who prioritizes high image quality of the virtual viewpoint image over real time. A viewpoint image can be generated.

In the present embodiment, the case of controlling the color gradation, the resolution, and the number of cameras corresponding to the captured image used for generating the virtual viewpoint image as the image quality of the virtual viewpoint image has been described, but other parameters as the image quality have been described. May be controlled. Further, a plurality of parameters related to image quality may be controlled at the same time.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC or the like) that realizes one or more functions. Further, the program may be recorded and provided on a recording medium readable by a computer.

1 Image processing device 2 Camera group 100 Camera information acquisition unit 110 Virtual viewpoint information acquisition unit 120 Image generation unit

Claims (18)

Image acquisition means for acquiring multiple images based on images taken from different positions by multiple imaging devices,
A reception means that accepts input according to the operation for specifying a virtual viewpoint,
A first generation means used for the operation, which is a generation means for generating a plurality of virtual viewpoint images based on the plurality of images acquired by the image acquisition means and the input received by the reception means. A second virtual viewpoint image and a second virtual viewpoint image corresponding to the virtual viewpoint specified based on the operation, and the image data size per frame of the moving image is larger than that of the first virtual viewpoint image. It has a virtual viewpoint image and a generation means for generating it.
The number of photographing devices corresponding to the plurality of images used to generate the second virtual viewpoint image by the generation means corresponds to the plurality of images used to generate the first virtual viewpoint image by the generation means. More than the number of shooting devices,
The plurality of photographing devices corresponding to the plurality of images used to generate the second virtual viewpoint image by the generation means correspond to the plurality of images used to generate the first virtual viewpoint image by the generation means. An image processing system characterized by including a plurality of photographing devices. Image acquisition means for acquiring multiple images based on images taken from different positions by multiple imaging devices,
A reception means that accepts input according to the operation for specifying a virtual viewpoint,
A first generation means used for the operation, which is a generation means for generating a plurality of virtual viewpoint images based on the plurality of images acquired by the image acquisition means and the input received by the reception means. A second virtual viewpoint image and a second virtual viewpoint image corresponding to the virtual viewpoint specified based on the operation, and the image data size per frame of the moving image is larger than that of the first virtual viewpoint image. It has a virtual viewpoint image and a generation means for generating it.
The generation means performs image processing for increasing the image data size per frame of a moving image on a virtual viewpoint image generated based on the plurality of images and the input, thereby performing the second virtual viewpoint image. Is a partial process included in the process for generating the second virtual viewpoint image from the virtual viewpoint image, and the process is executed in a processing time equal to or less than a predetermined threshold. An image processing system characterized by generating a virtual viewpoint image of 1. In addition to the first virtual viewpoint image and the second virtual viewpoint image, the generation means has a third virtual viewpoint in which the image data size per frame of the moving image is larger than that of the second virtual viewpoint image. The image processing system according to claim 2, wherein an image is generated based on the plurality of images and the input. Image acquisition means for acquiring multiple images based on images taken from different positions by multiple imaging devices,
A reception means that accepts input according to the operation for specifying a virtual viewpoint,
A first generation means used for the operation, which is a generation means for generating a plurality of virtual viewpoint images based on the plurality of images acquired by the image acquisition means and the input received by the reception means. A second virtual viewpoint image and a second virtual viewpoint image corresponding to the virtual viewpoint specified based on the operation, and the image data size per frame of the moving image is larger than that of the first virtual viewpoint image. An image processing system characterized by generating a virtual viewpoint image and a third virtual viewpoint image having a larger image data size per frame of a moving image than the second virtual viewpoint image. The second virtual viewpoint image is a virtual viewpoint image that is broadcast live.
The image processing system according to claim 4, wherein the third virtual viewpoint image is a virtual viewpoint image that is broadcast after recording. The generation means performs image processing for increasing the image data size per frame of a moving image on a virtual viewpoint image generated based on the plurality of images and the input, thereby performing the second virtual viewpoint image. Is a partial process included in the process for generating the second virtual viewpoint image from the virtual viewpoint image, and the process is executed in a processing time equal to or less than a predetermined threshold. The image processing system according to claim 4 or 5, wherein the virtual viewpoint image of 1 is generated. Claim 2 is characterized in that the number of gradations of colors included in the second virtual viewpoint image generated by the generation means is larger than the number of gradations of colors included in the first virtual viewpoint image. The image processing system according to any one of 6 to 6. The number of photographing devices corresponding to the images used to generate the second virtual viewpoint image by the generation means is the number of photographing devices corresponding to the images used to generate the first virtual viewpoint image by the generation means. The image processing system according to any one of claims 2 to 7, wherein the number of images is greater than one. The second virtual viewpoint image according to any one of claims 1 to 8, wherein the second virtual viewpoint image is an image for being displayed to a viewer who sees the image generated in response to the operation. Image processing system. It has an output means for outputting the first virtual viewpoint image and the second virtual viewpoint image generated by the generation means.
Any of claims 1 to 9, wherein the timing at which the first virtual viewpoint image is output by the output means is earlier than the timing at which the second virtual viewpoint image is output by the output means. The image processing system according to item 1. The generation means so that the first virtual viewpoint image is displayed in the first display area and the second virtual viewpoint image is displayed in a second display area different from the first display area. The image processing system according to any one of claims 1 to 10, further comprising an output control means for controlling the output of the first virtual viewpoint image and the second virtual viewpoint image generated by the above. .. The generation means further uses at least one of the image data generated in the process of generating the first virtual viewpoint image from the image acquired by the image acquisition means and the first virtual viewpoint image. The image processing system according to any one of claims 1 to 11, wherein the second virtual viewpoint image is generated by performing the processing. The first virtual viewpoint image is an image showing the shape of an object captured by at least one of the plurality of photographing devices.
Any of claims 1 to 12, wherein the second virtual viewpoint image is an image representing the color of the object that does not appear in the first virtual viewpoint image in addition to the shape of the object. The image processing system according to item 1. The image processing system according to any one of claims 1 to 13, wherein the resolution of the second virtual viewpoint image is higher than the resolution of the first virtual viewpoint image. An image acquisition process that acquires multiple images based on images taken from different positions by multiple imaging devices, and
The reception process that accepts input according to the operation to specify the virtual viewpoint,
A first generation step used for the operation, which is a generation step of generating a plurality of virtual viewpoint images based on the plurality of images acquired in the image acquisition step and the input received in the reception step. A second virtual viewpoint image and a second virtual viewpoint image corresponding to the virtual viewpoint specified based on the operation, and the image data size per frame of the moving image is larger than that of the first virtual viewpoint image. It has a virtual viewpoint image and a generation process to generate it.
In the generation step, the number of photographing devices corresponding to the plurality of images used for generating the second virtual viewpoint image is the number of photographing devices corresponding to the plurality of images used for generating the first virtual viewpoint image. More than a number,
In the generation step, the plurality of photographing devices corresponding to the plurality of images used for generating the second virtual viewpoint image are a plurality of photographing devices corresponding to the plurality of images used for generating the first virtual viewpoint image. An image processing method comprising a device. An image acquisition process that acquires multiple images based on images taken from different positions by multiple imaging devices, and
The reception process that accepts input according to the operation to specify the virtual viewpoint,
A first generation step used for the operation, which is a generation step of generating a plurality of virtual viewpoint images based on the plurality of images acquired in the image acquisition step and the input received in the reception step. A second virtual viewpoint image and a second virtual viewpoint image corresponding to the virtual viewpoint specified based on the operation, and the image data size per frame of the moving image is larger than that of the first virtual viewpoint image. It has a virtual viewpoint image and a generation process to generate it.
In the generation step, the second virtual viewpoint image is performed by performing image processing for increasing the image data size per frame of the moving image for the virtual viewpoint image generated based on the plurality of images and the input. Is a partial process included in the process for generating the second virtual viewpoint image from the virtual viewpoint image, and the process is executed in a processing time equal to or less than a predetermined threshold. An image processing method characterized by generating a virtual viewpoint image of 1. An image acquisition process that acquires multiple images based on images taken from different positions by multiple imaging devices, and
The reception process that accepts input according to the operation to specify the virtual viewpoint,
A first generation step used for the operation, which is a generation step of generating a plurality of virtual viewpoint images based on the plurality of images acquired in the image acquisition step and the input received in the reception step. A second virtual viewpoint image and a second virtual viewpoint image corresponding to the virtual viewpoint specified based on the operation, and the image data size per frame of the moving image is larger than that of the first virtual viewpoint image. An image processing method comprising a generation step of generating a virtual viewpoint image and a third virtual viewpoint image having a larger image data size per frame of a moving image than the second virtual viewpoint image. .. A program for making a computer function as each means of the image processing system according to any one of claims 1 to 14.

Images