JP2022016929A - Information processing device, information processing method, and program - Google Patents

Abstract

To suppress the degradation of image quality in a virtual viewpoint image due to noise caused by errors in 3D shape estimation in a virtual viewpoint image.SOLUTION: An information processing device identifies a position of a virtual viewpoint corresponding to a virtual viewpoint image generated on the basis of the result of estimating a three-dimensional shape of an object by shooting a shooting area from different directions by a plurality of shooting devices. The information processing device also determines whether a predetermined condition related to the degradation of image quality due to the occurrence in the virtual viewpoint image of noise at the position where an error in the estimation of the three-dimensional shape is caused by the inability to shoot from at least any of the plurality of shooting devices has been met. The information processing device then controls the position of the virtual viewpoint to be closer to the position on an optical axis of a shooting device included in the plurality of shooting devices when it is determined that the predetermined condition is met.SELECTED DRAWING: Figure 6

Description

The present invention relates to a technique for generating a virtual viewpoint image using a plurality of photographing devices.

There is a technique of installing a plurality of imaging devices at different positions to perform synchronous imaging, and using a plurality of captured images obtained by the imaging to generate a virtual viewpoint image in which the viewpoint can be arbitrarily changed. Specifically, the three-dimensional shape of the object in the shooting area is estimated based on a plurality of shot images, and a virtual viewpoint image is generated by rendering using a three-dimensional model representing the estimated three-dimensional shape. To. According to the technology for generating a virtual viewpoint image, for example, a soccer or rugby image can be viewed from various viewpoints, so that a user can be given a high sense of presence as compared with a normal image.

In Patent Document 1, it is possible to specify an area within a shooting target area where a virtual viewpoint image can be generated at a resolution desired by the user based on the camera information of a plurality of cameras, and notify the user of the specific result. Are listed.

Japanese Unexamined Patent Publication No. 2019-145894

However, even when a virtual viewpoint image of the same area is generated, the virtual viewpoint generated depends on the situation of the objects existing in the area and the position from which the virtual viewpoint is set to be viewed. The image quality is different. For example, when generating a virtual viewpoint image of an area where multiple players are crowded in a rugby scrum, etc., the position that is shielded by the players and cannot be shot from some shooting devices is a three-dimensional shape based on the shot image. An error will occur in the estimation of. When a virtual viewpoint image is generated using the estimation result of such an inaccurate three-dimensional shape, it is conceivable that noise is generated in the virtual viewpoint image and the image quality is deteriorated depending on the position of the virtual viewpoint.

The present invention has been made in view of the above problems, and an object of the present invention is to suppress deterioration of the image quality of the virtual viewpoint image due to noise caused by an error in three-dimensional shape estimation in the virtual viewpoint image.

In order to solve the above-mentioned problems, the information processing apparatus according to the present invention has, for example, the following configuration. That is, the information processing device corresponds to a virtual viewpoint image generated based on the result of estimating the three-dimensional shape of an object in the shooting area by shooting the shooting area from a plurality of directions by a plurality of shooting devices. The virtual viewpoint image is a noise at a position where an error in estimating a three-dimensional shape occurs due to a specific means for specifying the position of a viewpoint and a position in the shooting area where shooting is not possible from at least one of the plurality of shooting devices. A determination means for determining whether or not a predetermined condition related to deterioration of image quality due to the occurrence of the above is satisfied, and a virtual viewpoint corresponding to the virtual viewpoint image when the determination means determines that the predetermined condition is satisfied. It has a control means for controlling the position of the above to be closer to a position on the optical axis of the photographing apparatus included in the plurality of photographing devices from the position specified by the specific means.

According to the present invention, by moving the virtual viewpoint, it is possible to generate a virtual viewpoint image by using a photographing device that captures a scene close to the scene seen from the virtual viewpoint. Therefore, three-dimensional shape estimation is performed in the virtual viewpoint image. Noise caused by errors is less likely to occur. As a result, deterioration of the image quality of the virtual viewpoint image can be suppressed.

It is a figure which shows the configuration example of an image processing system. It is a figure which shows the hardware configuration example of an information processing apparatus. It is a block diagram which shows the functional structure example of an information processing apparatus. It is a figure which shows the example of the arrangement of a virtual viewpoint and a photographing apparatus. It is a figure which shows the example of the arrangement of a telephoto camera and a wide-angle camera. It is a figure which shows the example of the screen displayed by the display device. It is a figure which shows the example of the shooting information about a shooting device. It is a flowchart which shows the example of the operation of an information processing apparatus. It is a figure which shows the relationship between the position of a virtual viewpoint and the optical axis of a photographing apparatus.

[System configuration]
FIG. 1 is a diagram showing a configuration example of the image processing system 100 according to the present embodiment. The image processing system 100 includes a plurality of photographing devices 101, an information processing device 102, a viewpoint input device 103, and a display device 104. The image processing system 100 is for generating a virtual viewpoint image representing a scene seen from a designated virtual viewpoint based on a plurality of shot images taken by a plurality of shooting devices 101 and a designated virtual viewpoint. It is a system. The virtual viewpoint image in the present embodiment is also called a free viewpoint image, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, for example, a viewpoint selected by the user from a plurality of candidates. Corresponding images and the like are also included in the virtual viewpoint image. Further, in the present embodiment, the case where the virtual viewpoint is specified by the user operation will be mainly described, but the virtual viewpoint may be automatically specified based on the result of image analysis or the like. Further, in the present embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the virtual viewpoint image may be a still image.

The plurality of photographing devices 101 are installed at different positions so as to surround the photographing area such as a field in the stadium, and a part or the whole of the photographing area is simultaneously photographed from different directions. The plurality of photographing devices 101 may not be installed over the entire circumference of the photographing area, and the photographing device 101 may be installed only in a part around the photographing area. Further, the number of photographing devices 101 is not limited to the example shown in FIG. 1. For example, when the photographing area is a rugby field, about 100 photographing devices 101 may be installed around the field.

FIG. 4 shows an example in which 18 photographing devices 101a to 101r are installed around the field in the stadium 400, and the virtual viewpoint 401 corresponding to the virtual viewpoint image is set on the field. In the example of FIG. 4, a virtual viewpoint image representing a scene seen from a virtual viewpoint 401 near the goal is generated by using a plurality of captured images acquired by a plurality of photographing devices 101 arranged around the field.

The plurality of photographing devices 101 may include cameras having different functions such as a telephoto camera and a wide-angle camera. FIG. 5A shows an example of camera arrangement in a photographing system having two types of cameras, a telephoto camera 502 and a wide-angle camera 503. The field 501 is a ground surface in a stadium where, for example, rugby is performed, and an object 500, which is a player as a subject, is present on the ground surface. The 12 telephoto cameras 502 constituting the telephoto camera group and the 12 wide-angle cameras 503 constituting the wide-angle camera group are arranged so as to surround the field 501.

FIG. 5B shows the shooting range of the telephoto camera 502. The area 505 surrounded by the dotted line indicates the shooting range of the telephoto camera 502. The telephoto camera 502 has a narrow shooting range because the angle of view is narrow, but has a characteristic that the resolution of the object 500 in the shot image is high. FIG. 5C shows the shooting range of the wide-angle camera 503. The area 507 surrounded by the dotted line indicates the shooting range of the wide-angle camera 503. Since the wide-angle camera 503 has a wide angle of view, the shooting range is wide, but the resolution of the object 500 in the shot image is low. Although the shape of the shooting range is shown as an ellipse in FIGS. 5 (b) and 5 (c) for convenience, the shape of the actual shooting range of each camera may be rectangular.

In the present embodiment, a case where a camera having an independent housing and capable of shooting from a single viewpoint is used as a shooting device 101 for shooting a shooting region from different positions will be described. However, the present invention is not limited to this, and two or more photographing devices 101 may be configured in the same housing. For example, a single camera having a plurality of lens groups and a plurality of sensors and capable of photographing from a plurality of viewpoints may be installed as a plurality of photographing devices 101. In the present embodiment, the shooting area to be shot by the plurality of shooting devices 101 is a field where a rugby game is played. However, the shooting area is not limited to this, and may be a stadium where other sports such as soccer are performed, or a stage where a concert or performance is performed.

The photographing device 101 is, for example, a digital video photographing device provided with a video signal interface represented by a serial digital interface (SDI). The photographing device 101 transmits to the information processing apparatus 102 video data in which time information represented by a time code is added to the video signal obtained by photographing.

The viewpoint input device 103 is an input device having a controller such as a joystick, and accepts a user operation for designating a virtual viewpoint corresponding to a virtual viewpoint image to be generated. Then, the viewpoint input device 103 outputs a signal corresponding to the received user operation to the information processing device 102.

The information processing device 102 acquires image data of a plurality of images (multiple viewpoint images) based on the images taken by the plurality of photographing devices 101. The image included in the multi-viewpoint image may be a captured image, or may be an image obtained by performing image processing such as extraction of a predetermined region on the captured image. Then, the information processing device 102 represents a scene seen from a designated virtual viewpoint based on the plurality of viewpoint images acquired from the plurality of photographing devices 101 and the viewpoint information acquired based on the input from the viewpoint input device 103. Generate a virtual viewpoint image. The generated virtual viewpoint image may display virtual contents that are not included in the captured image.

The viewpoint information used to generate the virtual viewpoint image is information indicating the position and direction (line-of-sight direction) of the virtual viewpoint. Specifically, the viewpoint information is a parameter including a parameter representing the three-dimensional position (coordinates of the X, Y, Z axes) of the virtual viewpoint and a parameter representing the orientation of the virtual viewpoint in the pan, tilt, and roll directions. It is a set. The content of the viewpoint information is not limited to the above. For example, the parameter set as the viewpoint information may include a parameter indicating the size (angle of view) of the field of view of the virtual viewpoint and a parameter relating to the resolution of the virtual viewpoint image. Further, the viewpoint information may have a plurality of parameter sets. For example, the viewpoint information may have a plurality of parameter sets corresponding to a plurality of frames constituting a moving image of the virtual viewpoint image, and may be information indicating the position and orientation of the virtual viewpoint at each of a plurality of consecutive time points. ..

The information processing apparatus 102 generates a virtual viewpoint image by, for example, the following method. First, a foreground image in which a foreground area corresponding to a predetermined object such as a person or a ball is extracted from a plurality of viewpoint images and a background image in which a background area other than the foreground area is extracted are acquired. In addition, a foreground model representing the three-dimensional shape of a predetermined object and texture data for coloring the foreground model are generated based on the foreground image, and the background model representing the three-dimensional shape of the background such as a stadium is colored. Texture data is generated based on the background image. Then, a virtual viewpoint image is generated by mapping the texture data to the foreground model and the background model and performing rendering according to the virtual viewpoint indicated by the viewpoint information. However, the method of generating a virtual viewpoint image is not limited to this, and various methods such as a method of generating a virtual viewpoint image by projective transformation of a captured image without using a three-dimensional model can be used.

The virtual viewpoint image generated by the information processing device 102 is output to the display device 104. The display device 104 displays the image output from the information processing device 102 on the display screen. The display device 104 is, for example, a liquid crystal display, an LED display, or the like.

The configuration of the image processing system 100 is not limited to the example shown in FIG. For example, any two or all of the information processing device 102, the viewpoint input device 103, and the display device 104 may be integrally configured. Further, the foreground image and the background image described above may be extracted by the information processing device 102 from the captured image, or may be extracted by another device (for example, the photographing device 101). When the photographing device 101 extracts the foreground image and the background image, each of the plurality of photographing devices 101 may extract both the foreground image and the background image. Alternatively, some of the photographing devices 101 may extract the foreground image and some other photographing devices 101 may extract the background image. Further, the plurality of photographing devices 101 may include a photographing device 101 that does not extract either a foreground image or a background image.

[Hardware configuration]
FIG. 2 is a diagram showing a hardware configuration example of the information processing apparatus 102. The configuration of the viewpoint input device 103 is also the same as that of the information processing device 102 described below. The information processing device 102 includes a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, a display unit 215, an operation unit 216, a communication I / F 217, and a bus 218.

The CPU 211 realizes each function of the information processing apparatus 102 by controlling the entire information processing apparatus 102 by using a computer program and data stored in the ROM 212 or the RAM 213. The information processing device 102 may have one or a plurality of dedicated hardware or GPU (Graphics Processing Unit) different from the CPU. Then, at least a part of the processing by the CPU may be performed by the GPU or dedicated hardware. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors).

The ROM 212 stores programs and the like that do not require changes. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, data supplied from the outside via the communication I / F 217, and the like. The auxiliary storage device 214 is composed of, for example, a hard disk drive or the like, and stores various data such as image data and volume data.

The display unit 215 is composed of, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) for the user to operate the information processing apparatus 102. The operation unit 216 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and inputs various instructions to the CPU 211 in response to an operation by the user. The CPU 211 operates as a display control unit that controls the display unit 215 and an operation control unit that controls the operation unit 216. The communication I / F 217 is used for communication with an external device of the information processing device 102. For example, when the information processing device 102 is connected to an external device by wire, a communication cable is connected to the communication I / F 217. When the information processing device 102 has a function of wirelessly communicating with an external device, the communication I / F 217 includes an antenna. The bus 218 connects each part of the information processing apparatus 102 to transmit information.

In the present embodiment, it is assumed that the display unit 215 and the operation unit 216 exist inside the information processing device 102, but at least one of the display unit 215 and the operation unit 216 exists as another device outside the information processing device 102. You may be doing it.

[Functional configuration of information processing device]
FIG. 3 is a block diagram showing a functional configuration example of the information processing apparatus 102. The information processing apparatus 102 includes an image acquisition unit 300, a separation unit 301, a storage unit 302, an image generation unit 303, an input unit 304, a calculation unit 305, a control unit 306, and a viewpoint acquisition unit 307. These components are realized by the CPU 211 executing arithmetic processing and various programs using a control program stored in the ROM 212 or the auxiliary storage device 214. However, at least a part of these components may be realized by one or a plurality of dedicated hardware different from the CPU 211.

The image acquisition unit 300 acquires image data of a plurality of captured images obtained by photographing by the plurality of photographing devices 101. The image acquisition unit 300 may acquire a captured image in real time during imaging by the imaging device 101, or may acquire the recorded captured image after the shooting of the event by the imaging device 101 is completed.

The separation unit 301 extracts images of specific objects such as athletes and referees from each of the captured images acquired from the imaging device 101. The method of extracting the image of a specific object from the captured image is not limited. For example, there is a background subtraction method as a method for extracting a specific object. The background subtraction method is a method of comparing the pixel values of a captured image and a background image corresponding to the captured image and extracting a specific object not included in the background image from the captured image. The background image is an image of a shooting area in which a specific object does not exist. As a method of acquiring a background image, for example, when the shooting area is a field of competition, the background image is acquired by shooting the field before the start of the competition. The separation unit 301 stores the background image and the extracted specific object image in the storage unit 302.

In addition to the background image and the specific object image, the storage unit 302 stores the photographing information regarding the plurality of photographing devices 101 input from the control unit 306. The shooting information includes information indicating the position of each of the photographing devices 101 in the three-dimensional space and information indicating the direction (shooting direction) of each of the photographing devices 101. The three-dimensional position of the photographing apparatus 101 is defined by three-dimensional coordinates indicating the position in each direction corresponding to the height, width, and depth of the three-dimensional space. The content of the shooting information is not limited to the above, and may include, for example, information on the resolution related to the shooting of the shooting device 101, information on the focal length corresponding to the wide angle of view of the shooting device, and the like. FIG. 7 shows an example of shooting information. In the example of FIG. 7, the X coordinate, the Y coordinate, and the Z coordinate indicating the three-dimensional position of each of the photographing devices 101, the Pan value, the Til value, and the Roll value indicating the photographing direction, and the Zoom value (focal length) indicating the angle of view are shown. Distance) is included in the shooting information.

The image generation unit 303 generates three-dimensional shape data showing the estimation result of the three-dimensional shape of the specific object based on the specific object image acquired from the storage unit 302 and the shooting information. As a method for generating three-dimensional shape data, for example, a visual volume crossing method can be used, but the method is not limited to this. Then, the image generation unit 303 generates a virtual viewpoint image according to the virtual viewpoint indicated by the viewpoint information input from the control unit 306 by using the three-dimensional shape data of the specific object, the specific object image, and the background image. ..

The input unit 304 accepts the input of data for acquiring the shooting information of the plurality of shooting devices 101. The calculation unit 305 acquires the shooting information of the plurality of shooting devices 101 based on the data input to the input unit 304. The data input to the input unit 304 is, for example, a plurality of captured images obtained by installing a marker in the photographing area and photographing with a plurality of photographing devices 101. The calculation unit 305 acquires shooting information about each of the shooting devices 101 by performing a camera calibration process based on these shot images. However, the user may input the shooting information into the information processing apparatus 102, or the input unit 304 may import the data file in which the shooting information is described from the outside.

The viewpoint acquisition unit 307 receives a signal input according to the user operation from the viewpoint input device 103. The signal corresponding to the user operation indicates, for example, a change direction and a change amount of the position or direction of the virtual viewpoint. Then, the viewpoint acquisition unit 307 generates viewpoint information indicating the position and direction of the virtual viewpoint based on the received input. The position and orientation of the virtual viewpoint indicated by the viewpoint information is represented by, for example, a coordinate system with the center of the stadium including the shooting area as the origin.

The control unit 306 outputs the shooting information of the shooting device 101 acquired from the calculation unit 305 to the storage unit 302, and outputs the viewpoint information of the virtual viewpoint acquired from the viewpoint acquisition unit 307 to the image generation unit 303. Further, the control unit 306 acquires the virtual viewpoint image generated by the image generation unit 303, outputs the acquired virtual viewpoint image to the display device 104, and displays the acquired virtual viewpoint image. The control unit 306 may superimpose an image based on the shooting information on the virtual viewpoint image acquired from the image generation unit 303 and output it, or superimpose an image of virtual content (for example, a virtual advertisement) that does not exist in the shooting area. And output.

Further, the control unit 306 receives an input to the information processing device 102 or the display device 104 according to the user operation, and controls to change the virtual viewpoint so that the image quality of the virtual viewpoint image is improved. This control will be described with reference to FIG. FIG. 6 shows an example of a screen displayed by the display device 104. The display screen 600 includes a display image 601 including a virtual viewpoint image generated by the information processing apparatus 102 and an operable button 602. When the button 602 is pressed by a user operation, the control unit 306 causes the display screen 600 to display an arrow 603 for notifying the user of the direction in which the virtual viewpoint should be moved so that the image quality of the virtual viewpoint image is improved.

The factors of deterioration of the image quality of the virtual viewpoint image in the present embodiment will be described. For example, consider a case where a virtual viewpoint image is generated using a plurality of captured images obtained by photographing a field in which a rugby game is played with a plurality of photographing devices 101. In a scene such as scrum in rugby, a plurality of athletes 604 are densely packed, and there is a position where at least one of the imaging devices 101 cannot photograph because the athletes are shielded by the athletes. Then, when the three-dimensional shape data is generated from the captured image, an error occurs in the estimation of the three-dimensional shape with respect to the position. On the other hand, if the virtual viewpoint is freely set, such a position that cannot be photographed may be included in the field of view from the virtual viewpoint. In this case, in the process of generating the virtual viewpoint image according to the virtual viewpoint, the inaccurate part of the three-dimensional shape is colored, so that jelly-like noise is generated at the position in the virtual viewpoint image that cannot be captured. 605 is generated, and the image quality of the virtual viewpoint image is deteriorated.

In order to suppress such noise 605 and improve the image quality of the virtual viewpoint image, the position shielded by the athlete and cannot be photographed by the photographing apparatus 101 may not be included in the field of view from the virtual viewpoint. Alternatively, it is sufficient to reduce such positions that cannot be photographed in the field of view from the virtual viewpoint. Therefore, the control unit 306 displays an arrow 603 indicating in which direction the virtual viewpoint should be moved in order to bring the virtual viewpoint closer to the position on the optical axis of any of the photographing devices 101 when the button 602 is pressed. Control to make it. The user looking at the display screen 600 operates the viewpoint input device 103 according to the display of the arrow 603 to move the virtual viewpoint to a position on the optical axis of the photographing device 101. If the virtual viewpoint is located on the optical axis of the photographing device 101, the range visible from the virtual viewpoint can be photographed from the photographing device 101 without being obscured by the object, so that the noise included in the virtual viewpoint image is reduced.

The optical axis of the photographing apparatus 101 in the present embodiment is a straight line passing through the center of the lens included in the photographing apparatus 101 and the center of the imaging sensor. In other words, the optical axis of the photographing device 101 is a straight line passing through the position of the photographing device 101 and the position reflected in the center of the photographed image acquired by the photographing device 101.

Further, if the shooting direction of the photographing device 101 and the line-of-sight direction from the virtual viewpoint are significantly different, a region not included in the photographing range of the photographing device 101 may be included in the field of view from the virtual viewpoint. Therefore, the control unit 306 may perform control for bringing the line-of-sight direction from the virtual viewpoint closer to the shooting direction of the shooting device 101. Specifically, the control unit 306 displays on the display screen 600 an arrow indicating which direction the line-of-sight direction should be changed in order to bring the line-of-sight direction from the virtual viewpoint closer to the shooting direction of the photographing device 101. May be good.

The display mode for notifying the user of the direction in which the virtual viewpoint should be moved or the direction in which the line-of-sight direction should be changed is not limited to the example of FIG. 6, and display may be performed by an index other than the arrow. Further, this index may be displayed outside the virtual viewpoint image. Further, the control by the control unit 306 for changing the virtual viewpoint so as to improve the image quality of the virtual viewpoint image is not limited to the image display control. For example, a voice prompting the user to change the virtual viewpoint may be output from the speaker. Further, for example, the control unit 306 may automatically move the virtual viewpoint to a position on the optical axis of the photographing device 101 in response to the button 602 being pressed. In this case, the control unit 306 may move the virtual viewpoint instantaneously or continuously. Further, the control unit 306 may gradually bring the virtual viewpoint closer to the position on the optical axis of the photographing device 101 only while the user is pressing the button 602. Similarly, the control unit 306 may automatically change the line-of-sight direction so that the line-of-sight direction corresponding to the virtual viewpoint image approaches the shooting direction of the photographing device 101.

The configuration of the information processing apparatus 102 is not limited to the example shown in FIG. For example, the virtual viewpoint image may be generated by the image acquisition unit 300, the separation unit 301, the viewpoint acquisition unit 307, and another device having the same function as the image generation unit 303. In that case, the information processing device 102 may acquire the data of the virtual viewpoint image generated by another device, perform predetermined processing, and output the data to the display device 104.

[Operation flow]
FIG. 8 is a flowchart for explaining the operation of the information processing apparatus 102. The series of processes shown in the flowchart of FIG. 8 is performed by the CPU 211 of the information processing apparatus 102 expanding the program code stored in the ROM 212 into the RAM 213 and executing the program code. Further, a part or all of the processing shown in FIG. 8 may be realized by one or a plurality of dedicated hardware. The process shown in FIG. 8 is started at the timing when shooting is performed by a plurality of shooting devices 101 and the information processing device 102 receives an instruction for displaying a virtual viewpoint image. However, the start timing of the process shown in FIG. 8 is not limited to the above. The processing flow shown in FIG. 8 is repeatedly executed, for example, for each moving image frame of the virtual viewpoint image.

In S800, the image acquisition unit 300 acquires a plurality of captured images based on the capture by the plurality of imaging devices 101. In S801, the viewpoint acquisition unit 307 acquires viewpoint information indicating the position and orientation of the virtual viewpoint. In S802, the input unit 304 and the calculation unit 305 acquire shooting information indicating the position and orientation of each of the plurality of shooting devices. In S803, the image generation unit 303 uses a plurality of captured images and captured information to generate a virtual viewpoint image according to the position and orientation of the virtual viewpoint indicated by the viewpoint information.

In S804, the control unit 306 determines whether or not a user operation instructing the suppression of deterioration of the image quality of the virtual viewpoint image has been performed. For example, when the jelly-like noise is conspicuous in the display image 601 of FIG. 6, the user presses the button 602 for reducing the noise. If the button 602 is not pressed, the process proceeds to S807, and the control unit 306 outputs the virtual viewpoint image generated in S803 to the display device 104. On the other hand, when the button 602 is pressed, the process proceeds to S805.

In S805, the control unit 306 identifies the optical axis closest to the position of the current virtual viewpoint among the optical axes of each of the plurality of photographing devices 101 based on the viewpoint information of the virtual viewpoint and the photographing information of the photographing device 101. .. For example, as shown in FIG. 9, the optical axis 1000 of the photographing apparatus 101i, which is the optical axis closest to the virtual viewpoint 401, is specified. The optical axis of each of the photographing devices 101 is calculated from the position and orientation of each of the photographing devices 101 indicated by the photographing information.

In S806, the control unit 306 superimposes a guide indicating in which direction the optical axis specified in S805 exists with respect to the current virtual viewpoint position specified from the viewpoint information on the virtual viewpoint image generated in S803. do. In S807, the control unit 306 outputs the virtual viewpoint image on which the guide is superimposed to the display device 104 and displays it. For example, in the example of FIG. 6, a virtual viewpoint image is displayed on which an arrow 603, which is a guide indicating that the optical axis of the photographing apparatus 101 exists to the right of the virtual viewpoint with respect to the direction of the virtual viewpoint, is superimposed.

When the virtual viewpoint image on which the guide is superimposed is displayed, the user performs an operation for moving the virtual viewpoint according to the guide. The virtual viewpoint image corresponding to the virtual viewpoint moved in response to this operation is generated by a rendering process using an image based on shooting by the shooting device 101 having an optical axis specified in S805 and three-dimensional shape data. That is, a virtual viewpoint image representing a scene seen from a virtual viewpoint is generated based on a photographed image representing a scene close to the scene seen from the virtual viewpoint. Therefore, the influence of the error of the three-dimensional shape estimation is less likely to appear in the virtual viewpoint image, the generation of jelly-like noise is suppressed, and the image quality of the virtual viewpoint image can be improved.

[Modification example]
In the above explanation, an example of specifying the optical axis closest to the virtual viewpoint and bringing the virtual viewpoint closer to the optical axis has been described, but the control of the virtual viewpoint for improving the image quality of the virtual viewpoint image is limited to this. Not done. For example, the control unit 306 identifies a plurality of optical axes within a predetermined distance from the position of the current virtual viewpoint, and can photograph the athlete 604 with high resolution from the specified plurality of optical axes (for example, a photographing device 101 (for example). You may select the optical axis of the telephoto camera). Then, the control unit 306 may perform a guide display so that the virtual viewpoint approaches the position on the selected optical axis.

Further, for example, the control unit 306 identifies an imaging device 101 having an imaging direction closest to the line-of-sight direction corresponding to the virtual viewpoint image, and guides and displays the virtual viewpoint so as to approach the position on the optical axis of the imaging device 101. May be done. Specifically, the inner product of the vector corresponding to the line-of-sight direction from the virtual viewpoint and the vector corresponding to the shooting direction of the shooting device 101 is calculated for each of the plurality of shooting devices 101, and the shooting device 101 having the largest inner product It may be specified.

Further, for example, the control unit 306 may specify the photographing device 101 at the position closest to the virtual viewpoint, and perform a guide display so that the virtual viewpoint approaches the position on the optical axis of the photographing device 101. Alternatively, the guide display may be performed so that the virtual viewpoint approaches the position of the photographing device 101 closest to the virtual viewpoint.

In any of the above examples, the control unit 306 may automatically move the virtual viewpoint in response to the button 602 being pressed, instead of the guide display or together with the guide display. That is, the control unit 306 has a position on the optical axis closest to the virtual viewpoint, a position on the optical axis of the photographing device 101 in the direction closest to the direction of the virtual viewpoint, and a position on the optical axis of the high-resolution photographing device 101. Alternatively, the virtual viewpoint may be moved to the position of the photographing device 101 closest to the virtual viewpoint. The merit of the guide display is that the user can select whether to prioritize the improvement of the image quality or the free setting of the virtual viewpoint. On the other hand, the merit of automatically moving the virtual viewpoint is that the user's operation becomes easy and the time for displaying the low-quality virtual viewpoint image can be shortened.

Further, in the above description, the information processing apparatus 102 determines that jelly-like noise is generated in the virtual viewpoint image in response to the button 602 being pressed by the user, and captures the virtual viewpoint 101. It was decided to perform control to bring it closer to the position on the optical axis of. However, the information processing apparatus 102 may always display the guide as shown by the arrow 603 in FIG. 6 without determining whether noise is generated (that is, without determining whether the button 602 is pressed). .. In this case, the display screen 600 may not include the button 602.

Further, the condition of whether or not the information processing device 102 controls to bring the virtual viewpoint closer to the position on the optical axis of the photographing device 101 is not limited to the above. For example, the information processing apparatus 102 may analyze the virtual viewpoint image generated in S803 to determine whether or not noise is included, instead of determining that the button is pressed in S804 of FIG. Then, the information processing apparatus 102 may perform control for bringing the virtual viewpoint closer to the position on the optical axis of the photographing apparatus 101 when the virtual viewpoint image contains noise.

Further, for example, the information processing apparatus 102 may determine whether the state of the object in the photographing area is a situation in which noise is likely to occur in the virtual viewpoint image, instead of the determination of pressing the button in S804 of FIG. Then, the information processing apparatus 102 may perform control for bringing the virtual viewpoint closer to the position on the optical axis of the photographing apparatus 101 when the virtual viewpoint image is likely to generate noise.

When it is determined that noise is likely to occur in the virtual viewpoint image, for example, the size of the area included in the view from the virtual viewpoint is larger than the threshold value from the three-dimensional shape data generated based on the captured image. This is the case when it is determined that the object exists. This threshold value is, for example, a value determined based on the size of a human being. That is, when there is an object larger than the size of a normal human being, a plurality of players who are densely packed like a scrum are collectively formed as one object, and as described above, at least one of the photographing devices 101 cannot photograph. Is likely to exist. Therefore, noise is likely to occur in the virtual viewpoint image in such a situation.

Further, as another example of the case where it is determined that the virtual viewpoint image is prone to noise, a plurality of three-dimensional shape data whose distances from each other are smaller than the threshold value in the region included in the field of view from the virtual viewpoint. This is the case when it indicates that the object exists. In this case as well, as in the case of the scrum, there is a high possibility that there is a position that is shielded by the object and cannot be photographed by the photographing apparatus 101, and noise is likely to occur in the virtual viewpoint image.

Further, for example, the information processing apparatus 102 may determine whether the noise of the virtual viewpoint image is easily visible instead of the determination of the button press in S804 of FIG. Then, the information processing apparatus 102 may perform control for bringing the virtual viewpoint closer to the position on the optical axis of the photographing apparatus 101 when the noise of the virtual viewpoint image is easily visually recognized. The case where it is determined that the noise of the virtual viewpoint image is easily visible is, for example, the case where the speed of change in the position or orientation of the virtual viewpoint is slower than the threshold value.

As in each of the above-mentioned examples, the information processing apparatus 102 captures the virtual viewpoint of the photographing device 101 only when noise is generated in the virtual viewpoint, noise is likely to be generated, or noise is easily visually recognized. Control may be performed to bring it closer to a position on the optical axis. As a result, the operation of the virtual viewpoint image by the user is restricted more than necessary, and the arrow 603 displayed on the display image 601 is suppressed from interfering with the viewing of the virtual viewpoint image, while the image quality of the virtual viewpoint image is deteriorated. Can be suppressed.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC or the like) that realizes one or more functions. Further, the program may be recorded and provided on a recording medium readable by a computer.

100 Image processing system 101 Imaging device 102 Information processing device 103 Viewpoint input device 104 Display device

Claims (16)

Specifying the position of the virtual viewpoint corresponding to the virtual viewpoint image generated based on the result of estimating the three-dimensional shape of the object in the shooting area by shooting the shooting area from a plurality of directions with a plurality of shooting devices. Means and
A predetermined position related to deterioration of image quality due to noise generated in the virtual viewpoint image at a position within the photographing region where an error in estimating a three-dimensional shape occurs due to the inability to photograph from at least one of the plurality of photographing devices. Judgment means for determining whether the condition of
When it is determined by the determination means that the predetermined condition is satisfied, the position of the virtual viewpoint corresponding to the virtual viewpoint image is determined by the specific means to be included in the plurality of photographing devices. An information processing device characterized by having a control means for controlling the position on the optical axis. The specific means identifies the line-of-sight direction from the virtual viewpoint corresponding to the virtual viewpoint image, and determines the direction of the line of sight.
When the determination means determines that the predetermined condition is satisfied, the control means sets the line-of-sight direction from the virtual viewpoint corresponding to the virtual viewpoint image from the line-of-sight direction specified by the specific means. The information processing device according to claim 1, wherein the information processing device is controlled so as to bring the image pickup device included in the image pickup device closer to the image pickup direction. Three-dimensional shape data representing the result of estimating the three-dimensional shape of an object in the photographing area from a virtual viewpoint image corresponding to a virtual viewpoint approaching a position on the optical axis of the photographing apparatus according to control by the control means. The information processing apparatus according to claim 1 or 2, wherein the information processing apparatus has a generation means generated by a rendering process using the image taken by the photographing apparatus and an image based on the photographing by the photographing apparatus. When the determination means determines that the predetermined condition is satisfied, the control means determines the position of the virtual viewpoint corresponding to the virtual viewpoint image by the specific means among the optical axes of the plurality of photographing devices. The information processing apparatus according to any one of claims 1 to 3, wherein control is performed to bring the image closer to the position on the optical axis closest to the specified position. The specific means identifies the line-of-sight direction from the virtual viewpoint corresponding to the virtual viewpoint image, and determines the direction of the line of sight.
When the determination means determines that the predetermined condition is satisfied, the control means determines the position of the virtual viewpoint corresponding to the virtual viewpoint image by the specific means among the plurality of photographing devices. The information processing apparatus according to any one of claims 1 to 3, wherein the information processing apparatus is controlled so as to approach a position on the optical axis of the photographing apparatus closest to the specified line-of-sight direction. The control according to any one of claims 1 to 5, wherein the control by the control means includes a control for moving the position of the virtual viewpoint corresponding to the virtual viewpoint image toward a position on the optical axis of the photographing apparatus. The information processing apparatus according to any one of the following items. The control by the control means includes a control for notifying the user to perform an operation of bringing the position of the virtual viewpoint corresponding to the virtual viewpoint image closer to the position on the optical axis of the photographing device. The information processing apparatus according to any one of claims 1 to 6. The control by the control means includes the control of displaying an image indicating the direction of the position on the optical axis of the photographing apparatus with respect to the position specified by the specific means, according to claims 1 to 7. The information processing apparatus according to any one of the following items. The information processing according to any one of claims 1 to 8, wherein the predetermined condition includes a plurality of objects whose distances from each other are shorter than a threshold value in the photographing region. Device. The information processing apparatus according to any one of claims 1 to 9, wherein the predetermined condition includes determining that an object larger than the threshold value exists in the photographing area. The information processing according to any one of claims 1 to 10, wherein the predetermined condition includes that the speed of change of the virtual viewpoint corresponding to the virtual viewpoint image is slower than the threshold value. Device. The information processing apparatus according to any one of claims 1 to 11, wherein the predetermined condition includes a user operation for instructing the suppression of deterioration of the image quality. A specific step of specifying the position of a virtual viewpoint corresponding to a virtual viewpoint image generated based on the result of estimating the three-dimensional shape of an object in the shooting area by shooting the shooting area from different directions with a plurality of shooting devices. When,
A predetermined position related to deterioration of image quality due to noise generated in the virtual viewpoint image at a position within the photographing region where an error in estimating a three-dimensional shape occurs due to the inability to photograph from at least one of the plurality of photographing devices. Judgment process to determine whether the conditions of
When it is determined in the determination step that the predetermined condition is satisfied, the position of the virtual viewpoint corresponding to the virtual viewpoint image is determined by the imaging device included in the plurality of imaging devices from the position specified in the specific step. An information processing method characterized by having a control process for controlling the position on the optical axis. In the specific step, the line-of-sight direction from the virtual viewpoint corresponding to the virtual viewpoint image is specified.
In the control step, when it is determined in the determination step that the predetermined condition is satisfied, the line-of-sight direction from the virtual viewpoint corresponding to the virtual viewpoint image is set to a plurality of line-of-sight directions from the line-of-sight direction specified in the specific step. 13. The information processing method according to claim 13, wherein control is performed to bring the image pickup device included in the image pickup apparatus closer to the image pickup direction. Three-dimensional shape data representing the result of estimating the three-dimensional shape of an object in the photographing area from a virtual viewpoint image corresponding to a virtual viewpoint approaching a position on the optical axis of the photographing apparatus according to the control in the control step. The information processing method according to claim 13, wherein the information processing method includes a generation step of generating the image and an image based on the image taken by the photographing apparatus by a rendering process. A program for making a computer function as each means of the information processing apparatus according to any one of claims 1 to 12.

Images