JP7042561B2 - Information processing equipment, information processing method - Google Patents

Description

The present invention relates to a technique relating to a virtual viewpoint image.

Recently, a technique of installing a plurality of image pickup devices at different positions to image a subject from multiple viewpoints and generating a virtual viewpoint image or a three-dimensional model using the multiple viewpoint images obtained by the imaging has attracted attention. .. According to the technique of generating a virtual viewpoint image from a plurality of viewpoint images as described above, it is possible to give a user a high sense of presence as compared with a normal image. In Patent Document 1, a plurality of cameras are arranged so as to surround the same range, an arbitrary virtual viewpoint is set using an image taken in the same range, and a virtual viewpoint corresponding to the set virtual viewpoint is set. Techniques for generating and displaying images are disclosed.

Japanese Unexamined Patent Publication No. 2014-215828

The image quality of the virtual viewpoint image varies greatly depending on the position and shape of the subject, the arrangement of the image pickup device, and the like. Therefore, depending on the position and orientation of the virtual viewpoint, an image with low image quality may be created or an image with high image quality may be created. In the past, no consideration has been given to a technique for allowing the user to understand the evaluation of image quality according to such a virtual viewpoint.

The present invention has been made in view of such a problem, and provides a technique for outputting an evaluation result of image quality of a virtual viewpoint image generated in a set virtual viewpoint.

The uniformity of the present invention describes the position of the virtual viewpoint, the direction of the line of sight from the virtual viewpoint, and the virtual viewpoint related to the generation of the virtual viewpoint image based on the plurality of captured images acquired by the plurality of imaging devices that capture images from different positions. A specific means of identifying the range of the field of view of
A means for providing a user with information indicating an evaluation result regarding the image quality of a virtual viewpoint image according to the position of the virtual viewpoint specified by the specific means, the line-of-sight direction from the virtual viewpoint, and the range of the field of view of the virtual viewpoint. It is characterized by being prepared.

According to the configuration of the present invention, it is possible to output the evaluation result of the image quality of the virtual viewpoint image generated in the set virtual viewpoint.

The figure which shows the outline of an information processing apparatus. The block diagram which shows the hardware configuration example of the main body apparatus 1. The block diagram which shows the functional composition example of the main body apparatus 1. The flowchart of the process performed by the main body apparatus 1. The figure explaining the display screen of the display device 1002. The block diagram which shows the functional composition example of the main body apparatus 1. The flowchart of the operation of the main body apparatus 1. The flowchart of the process performed by the evaluation unit 203.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In addition, the embodiment described below shows an example when the present invention is concretely implemented, and is one of the specific examples of the configuration described in the claims.

[First Embodiment]
In this embodiment, an example of an information processing device that predicts the image quality of a virtual viewpoint image at a virtual viewpoint at each position, visualizes the evaluation result for the predicted image quality, and presents it to the user will be described. First, an outline of the information processing apparatus according to the present embodiment will be described with reference to FIG. As shown in FIG. 1, the information processing device according to the present embodiment includes a main body device 1, a display device 1002, and a keyboard 1001a and a mouse 1001b as user interfaces.

The user can input various instructions to the main body device 1 by operating the keyboard 1001a and the mouse 1001b. The keyboard 1001a and the mouse 1001b are examples of user interfaces, and other types of interfaces may be used.

The main body device 1 performs various processes in response to instructions from the keyboard 1001a and the mouse 1001b. Further, the main body device 1 predicts the image quality of the virtual viewpoint image at each virtual viewpoint, visualizes the evaluation result for the predicted image quality, and performs various processes for presenting it to the user. The processing result by the main body device 1 is displayed on the display device 1002 as an image or characters. In the present embodiment, unless otherwise specified, the word "image" will be described as including the concepts of moving images and still images. That is, the apparatus of this embodiment can process both still images and moving images.

The display device 1002 is composed of a CRT, a liquid crystal screen, or the like, and can display the processing result of the main body device 1 with an image, characters, or the like. The display device 1002 may be a touch panel screen or a projection device that projects an image or characters on the wall surface. The contents displayed on the display device 1002 in FIG. 1 will be described later.

Next, a hardware configuration example of the main unit 1 will be described with reference to the block diagram of FIG. The configuration shown in FIG. 2 is only an example of a hardware configuration applicable to the main unit 1, and any hardware configuration can be used as long as each process described later can be executed by the main unit 1. May be adopted.

The CPU 11 executes various processes using computer programs and data stored in the ROM 12 and the RAM 13. As a result, the CPU 11 controls the operation of the main body device 1 as a whole, and also executes or controls each process described later as what the main body device 1 performs.

The ROM 12 stores computer programs (for example, BIOS programs and boot programs) and data that do not require rewriting. The RAM 13 has an area for storing computer programs and data loaded from the ROM 12 and the storage device 17, and a work area used by the CPU 11 to execute various processes. As described above, the RAM 13 can appropriately provide various areas. The display device 1002 is connected to the display I / F15. The keyboard 1001a and the mouse 1001b are connected to the input I / F16.

The storage device 17 is a large-capacity information storage device typified by a hard disk drive device. The storage device 17 stores an OS (operating system) and computer programs and data for causing the CPU 11 to execute or control each process described later as performed by the main body device 1. The computer programs and data stored in the storage device 17 are appropriately loaded into the RAM 13 according to the control by the CPU 11, and are processed by the CPU 11. The CPU 11, ROM 12, RAM 13, display I / F 15, input I / F 16, and storage device 17 are all connected to the bus 18.

Next, an example of the functional configuration of the main body device 1 will be described with reference to the block diagram of FIG. Each functional unit shown in FIG. 3 may be implemented by hardware, but in this embodiment, it is implemented by software (computer program). In the following, the functional unit shown in FIG. 3 may be described as the main body of the process, but in reality, the function of the functional unit is realized by the CPU 11 executing the computer program corresponding to the functional unit. ..

The first acquisition unit 101 acquires the environmental information stored in the storage device 17. Here, the environmental information will be described. For example, in a stadium where soccer, basketball, etc. are played, a plurality of image pickup devices that image the stadium from different directions are installed, and a virtual viewpoint image from a virtual viewpoint is generated based on the images captured by the image pickup device. Think about the system. For such a system, it is important to simulate the good / bad image quality of the virtual viewpoint image of the virtual viewpoint of what position and posture before operating the system. In this embodiment, in order to perform such a simulation, a plurality of virtual players (three-dimensional models imitating the three-dimensional shape of the player) are arranged in a virtual stadium (three-dimensional model imitating the three-dimensional shape of the stadium). A virtual viewpoint is provided at the position of. Then, a virtual viewpoint image is generated for each virtual viewpoint image, and the evaluation value of the image quality of the generated virtual viewpoint image is visualized and presented to the user.

In order to realize such a simulation, the environmental information includes the camera parameters of each image pickup device arranged in the stadium, the information defining the virtual stadium, and the information defining the virtual player. ing. The environmental information is information indicating the shooting target and the shooting environment performed to generate the virtual viewpoint image.

The information defining the virtual stadium includes information defining the three-dimensional shape and texture of the virtual stadium (for example, three-dimensional CAD data of the stadium). The information that defines the virtual player includes information that defines the three-dimensional shape and texture of the virtual player, information that defines the movement of the arms and legs of the virtual player, and the movement (position and posture) of the virtual player in the virtual stadium. Contains information to specify.

For the camera parameters of each image pickup device placed in the stadium, parameters such as the placement position / orientation, focal length, and angle of view of the image pickup device in the actual stadium are converted into parameters in the coordinate system to which the virtual stadium belongs. Is included.

The camera parameters of each image pickup device included in the environmental information are not limited to the camera parameters of the image pickup device actually arranged, and may be virtually created on the GUI. For example, as shown in FIG. 1, the camera parameters of the image pickup device can be set by operating the GUI 1003 (graphical user interface) displayed on the screen of the display device 1002. In this GUI 1003, a map that gives a bird's-eye view of the virtual stadium generated based on "information defining the virtual stadium" is displayed. The user can operate the keyboard 1001a and the mouse 1001b to arrange the image pickup device in an arbitrary position and an arbitrary posture in the virtual stadium. In FIG. 1, nine image pickup devices are arranged. For example, every time the user operates the keyboard 1001a or the mouse 1001b to input the camera parameters of the image pickup device, the image 1005 of the image pickup device of the arrangement position posture indicated by the input camera parameters is additionally displayed on the map. Then, when the user inputs the instruction to complete the setting of the camera parameter of the image pickup device by operating the keyboard 1001a or the mouse 1001b, the camera parameter of each image pickup device input by the user is included in the environmental information and stored in the storage device 17. To. The camera parameter setting method and acquisition method of the image pickup device are not limited to the specific setting method and the specific acquisition method. Further, the user may change the shape of the virtual stadium or the like in the GUI 1003 of FIG.

The second acquisition unit 102 randomly generates a plurality of camera parameters of the virtual viewpoint for evaluating the image quality (position / posture, focal length, angle of view, etc. of the virtual viewpoint in the virtual stadium). The method of generating the camera parameters of the virtual viewpoint is not limited to such a method. For example, each virtual viewpoint may be arranged two-dimensionally at equal intervals in a part or the whole area of the virtual stadium. Further, a plurality of virtual viewpoint camera parameters may be generated in advance and acquired by the second acquisition unit 102. The camera parameter of the virtual viewpoint may be any parameter as long as it can generate an image from the virtual viewpoint, and in addition to the position and orientation of the virtual viewpoint, the angle of view, zoom, focal length, and resolution. It may include parameters such as. Further, the camera parameters of the virtual viewpoint may define the position of the viewpoint and / or the direction of the line of sight for generating the virtual viewpoint image.

The evaluation unit 103 uses the environment information acquired by the first acquisition unit 101 and the camera parameters of the virtual viewpoint acquired by the second acquisition unit 102 for each virtual viewpoint set, and the evaluation unit 103 uses the virtual viewpoint in the virtual viewpoint. Evaluate the image quality. The output unit 104 outputs the evaluation result for each virtual viewpoint by the evaluation unit 103 to the display device 1002 as an image or characters.

Next, the process performed by the main unit 1 to predict the image quality of the virtual viewpoint image at the virtual viewpoint at each position, visualize the evaluation result for the predicted image quality, and present it to the user will be described according to the flowchart of FIG. do.

In step S1010, the first acquisition unit 101 reads the environment information stored in advance in the storage device 17 into the RAM 13. The environmental information may be received from an external device via the network and stored in the RAM 13, and the acquisition form is not limited to the specific acquisition form.

In step S1020, the second acquisition unit 102 acquires the camera parameters for each virtual viewpoint. The camera parameters for each virtual viewpoint may be acquired in the RAM 13 by generating them by some method as described above, or the ones generated in advance and stored in the storage device 17 may be loaded in the RAM 13. However, the acquisition form is not limited to a specific acquisition form.

Then, by performing the processing in step S1040 for each virtual viewpoint, the evaluation value of the image quality of the virtual viewpoint image for each virtual viewpoint is obtained. Hereinafter, the process of obtaining the evaluation value of the image quality of the virtual viewpoint image in the virtual viewpoint of interest will be described.

First, the evaluation unit 103 generates an image (correct image) of a virtual space (a space in which a virtual stadium and a virtual player are arranged) that can be seen from a viewpoint having the camera parameters, using the camera parameters of the virtual viewpoint of interest. For example, an image of a virtual space that can be seen from a viewpoint having a position and orientation of the virtual viewpoint of interest and having a focal length and an angle of view of the virtual viewpoint of interest is generated.

Further, the evaluation unit 103 uses the camera parameters of each image pickup device included in the environmental information to generate an image of the virtual space that can be seen from the viewpoint having the camera parameters (that is, by the image pickup device having the camera parameters). Image of virtual space). Then, the evaluation unit 103 uses the image of the virtual space (which may be a part) generated for each image pickup device to generate an image of the virtual space that can be seen from the virtual viewpoint of interest, that is, a virtual viewpoint image at the virtual viewpoint of interest.

Then, the evaluation unit 103 obtains an evaluation value related to the image quality of the virtual viewpoint image based on the correct answer image and the virtual viewpoint image. The method of obtaining the evaluation value related to the image quality of the virtual viewpoint image based on the correct answer image and the virtual viewpoint image is not limited to a specific method. For example, the evaluation unit 103 obtains the PSNR (peak signal to noise ratio) calculated from the difference between the correct answer image and the virtual viewpoint image as an evaluation value indicating the image quality index of the virtual viewpoint image with respect to the correct answer image.

By performing such processing for each of the plurality of scenes, the evaluation unit 103 obtains an evaluation value for the image quality of the virtual viewpoint image of the virtual viewpoint of interest for each of the plurality of scenes. For example, the evaluation unit 103 moves the position and posture, arms and legs of the virtual player for each scene, generates a virtual viewpoint image and a correct answer image of the virtual viewpoint of interest for each scene, and determines the evaluation value for each scene by the above method. I ask for it. Then, the evaluation unit 103 sets the average value of the evaluation values for each scene in the virtual viewpoint of interest as the final evaluation value in the virtual viewpoint of interest. The number of scenes may be 1.

Then, when the process of step S1040 is performed for all the virtual viewpoints, the process proceeds to step S1050. In step S1050, the output unit 104 visualizes the evaluation value for each virtual viewpoint and displays it on the display screen of the display device 1002. The method of visualizing the evaluation value for each virtual viewpoint is not limited to a specific method.

For example, as shown in FIG. 1, by setting the color of the pixel at the position of each virtual viewpoint on the map overlooking the virtual stadium to the color corresponding to the evaluation value for the virtual viewpoint, the evaluation value 2 is displayed on the map. Draw the dimensional maps on top of each other. When the color according to the evaluation value for the virtual viewpoint is "blue as the evaluation value is higher, red as the evaluation value is lower", the maximum value of the evaluation value is blue, the minimum value is red, and the maximum value and the minimum value are set. For the evaluation value in between, a color obtained by interpolating the maximum value color (blue) and the minimum value color (red) at the ratio R of the following equation (1) is added.

R = (evaluation value-minimum value) / (maximum value-minimum value) (1)
Regarding the color of the pixel located between the virtual viewpoint A and the virtual viewpoint B, for example, the color interpolated using the color corresponding to the evaluation value of the virtual viewpoint A and the color corresponding to the evaluation value of the virtual viewpoint B. do. In FIG. 1, a virtual viewpoint area 1006 having a relatively high evaluation value and a virtual viewpoint area 1007 having a relatively low evaluation value are visualized. Looking at the display screen of such a display device 1002, it can be seen that the user should consider creating a camera path of a virtual viewpoint inside the area 1006, so that the work efficiency of determining the camera path is improved. In addition, instead of or in addition to the color, the α value may be controlled according to the evaluation value.

<Modification example>
In the first embodiment, the virtual player is placed in the virtual stadium, but other virtual objects may be placed in addition to or in place of the virtual player, and virtual objects other than the virtual stadium are not placed. May be. Further, as described above, various user interfaces may be used, and for example, a pen tablet or a scanner may be used. Further, the user interface may be integrated with the display device 1002.

Further, as the display device 1002, a printer, a smartphone, a VR (Virtual Reality) system having an HMD (Head Mounted Display), or an AR (Augmented Reality) system may be used. These devices are an example of devices capable of visualizing the evaluation value of the image quality of the virtual viewpoint image in the virtual viewpoint.

Further, the above environmental information may include any information as long as the object (virtual stadium or virtual player in the above example) necessary for creating an image used by the evaluation unit 103 and the camera parameters of the image pickup apparatus can be acquired. Further, the virtual stadium and the virtual player may be computer graphics (CG) or may be a three-dimensional video created by measuring a real object. It is desirable that the content is the same as the image target, but it may be different. It is desirable that the content has motion information, but it does not have to have it, and it may be a set of a plurality of scenes. Further, among the elements included in the camera parameters, an indefinite element may be a variable element, and when the evaluation unit 103 evaluates, a plurality of types of values may be set for the variable element and each may be evaluated.

Further, the second acquisition unit 102 may acquire the camera parameters of the virtual viewpoint necessary for outputting the evaluation result by the output unit 104. The camera parameters of the virtual viewpoint include the position and orientation of the virtual viewpoint, the focal length, the angle of view, etc., but only some of the parameters may be used, and the parameters such as the zoom value and ROI at the time of imaging by the actual imaging device may be used. It may be included. In addition, the camera parameters of each virtual viewpoint may be generated so that the virtual viewpoint gazes at a specific position such as the gazing point, or the camera parameters of each virtual viewpoint are generated from the information such as the camera path of the virtual viewpoint generated in the past. You may try to get it.

Further, the evaluation unit 103 may be able to calculate an evaluation value for the image quality of the virtual viewpoint image in the virtual viewpoint. The evaluation value is not limited to PSNR, and may be any evaluation value used for image quality evaluation such as MSE (Mean Square Error) and SSIM (Structure Similarity). The evaluation value of the virtual viewpoint does not have to be limited to the average value of the evaluation values for each scene, but may be statistics such as the maximum value, the minimum value, and the variance value, or may be a weighted sum according to the importance of the scene. There may be. Further, the evaluation value of the image quality of the virtual viewpoint image in the virtual viewpoint is not limited to being obtained by comparing the virtual viewpoint image and the correct answer image. For example, the position in the virtual viewpoint image in the current frame where the feature points in the virtual viewpoint image in the frame before the current frame (previous frame) move is predicted from the movement of the image pickup device or the virtual player in the previous frame. .. This predicted position is called a predicted position. Then, the difference (prediction error) between the position of the feature point actually detected from the virtual viewpoint image in the current frame and the predicted position of the feature point is obtained, and the statistic (average or variance) of the prediction error obtained for each feature point is obtained. Is the evaluation value.

Further, the environmental information may include an image captured by each image pickup device actually arranged in the stadium. In such a case, the virtual viewpoint image in the virtual viewpoint may be generated based on the captured image included in the environmental information. Further, the captured image may be used instead of the correct image.

Further, as described above, the method for generating a virtual viewpoint image is not limited to the method described in Non-Patent Document 1. For example, it may be a model-based method that reproduces the appearance from an arbitrary viewpoint by restoring the three-dimensional shape of the subject, or an image-based method that reproduces the appearance from an arbitrary viewpoint by deforming or synthesizing the image. It may be a method.

In addition, the accuracy of the three-dimensional shapes of the virtual stadium and the virtual player (collectively referred to as virtual objects) generated in the process of generating the virtual viewpoint image is evaluated, and the arrangement of the image pickup device is evaluated according to the evaluation. You may do so. That is, the three-dimensional shape of the virtual object viewed from the virtual viewpoint and the three-dimensional shape of the virtual object restored from the image of the virtual space of the image pickup device (corresponding image pickup device) used to generate the virtual viewpoint image of the virtual viewpoint. And, the difference in shape is obtained. The larger the difference in the obtained shape, the lower the evaluation for the corresponding image pickup device, and the smaller the difference in the obtained shape, the higher the evaluation for the corresponding image pickup device. The "shape difference" is, for example, associating the nearest points or planes between the three-dimensional shape of the virtual object viewed from the virtual viewpoint and the three-dimensional shape of the restored virtual object, and the correspondence between the correspondences. It may be a statistic (shape statistic) such as an average value of distances. In addition, considering the color given to the three-dimensional shape, the difference in the color of the corresponding point and the corresponding surface between the three-dimensional shape of the virtual object seen from the virtual viewpoint and the three-dimensional shape of the restored virtual object. Statistics such as average values (color statistics) may be further added. At that time, the weighted sum of the shape statistic and the color statistic is used as the evaluation value. The larger the evaluation value, the lower the evaluation for the corresponding image pickup device, and the smaller the evaluation value, the lower the evaluation for the corresponding image pickup device. Is given a higher rating. Further, the number of unobserved surface voxels in the three-dimensional shape of the restored virtual object may be used as the evaluation value. In addition, a statistic (mean, dispersion, etc.) of the difference between the movement of the 3D feature point predicted from the movement of the image pickup device or the virtual player and the movement of the 3D feature point detected from the above-mentioned restored 3D shape. May be used as the evaluation value. Further, the number of image pickup devices in which the subject reflected in the virtual viewpoint is in the field of view, the statistic thereof, and the like may be used as the evaluation value.

The output form of the evaluation result for each virtual viewpoint by the output unit 104 is not limited to a specific output form. For example, by operating the keyboard 1001a or the mouse 1001b, an image of a virtual space (a virtual space including a virtual stadium and a virtual player) from a viewpoint that can be moved in the virtual stadium is generated and displayed on the display device 1002. .. At this time, the position of each virtual viewpoint or the area of the specified size centered on the position is drawn with the color and transparency according to the evaluation value for the virtual viewpoint.

In addition, the position and orientation of each virtual viewpoint is converted into the position and orientation in the actual stadium, and the position of the virtual viewpoint that enters the field of view of the HMD or the area of the specified size centered on the position is evaluated as the evaluation value for the virtual viewpoint. You may draw with the color and transparency according to. As a result, the user who wears the HMD on his / her head can set the actual stadium at the position of the virtual viewpoint or the area of the specified size centered on the position with the color and transparency according to the evaluation value for the virtual viewpoint. It can be observed in the drawn state. This is the same regardless of whether the HMD is an optical see-through method or a video see-through method. Further, instead of the HMD, a device such as a smartphone may be used.

Further, the output unit 104 may output text information in which the camera parameters of each virtual viewpoint and the evaluation value for the virtual viewpoint are associated with each other to the storage device 17 or an external device.

Further, the method of determining the color according to the evaluation value is not limited to the method shown in the above equation (1). Further, the color corresponding to the maximum value of the evaluation value and the color corresponding to the minimum value of the evaluation value are not limited to the colors described in the first embodiment. For example, the section from the maximum value to the minimum value that the evaluation value can take is divided into a plurality of subsections, and the color corresponding to each subsection is assigned. Then, when determining the color corresponding to the virtual viewpoint of interest, the partial section to which the evaluation value of the virtual viewpoint of interest belongs is specified, and the color assigned to the specified partial section is set as the color for the virtual viewpoint of interest. .. It is not necessary to assign a color depending on the subspace. Further, instead of changing the color according to the evaluation value, a pattern such as shading may be changed according to the evaluation value. Further, contour lines of evaluation values may be drawn. Further, it may be indicated by a score, a character, a symbol or an image for making the evaluation result distinguishable.

As described above, according to the present embodiment, by visualizing the evaluation value for the image quality of the virtual viewpoint image, the operator does not select a virtual viewpoint having a low image quality, so that the efficiency of the work of determining the camera path of the virtual viewpoint is improved. It improves and the work time is shortened.

[Second Embodiment]
In each of the following embodiments and modifications including the present embodiment, the differences from the first embodiment will be mainly described, and unless otherwise specified, the same as the first embodiment. In the present embodiment, an index corresponding to the image quality of the virtual viewpoint image in the camera path set for the virtual stadium is presented to the user. Since such an index is presented when setting the camera path, the work efficiency of determining the camera path is improved.

An example of the functional configuration of the main unit 1 according to the present embodiment will be described with reference to the block diagram of FIG. The input unit 205 inputs a camera path indicating the movement locus of the virtual viewpoint in the virtual stadium. The camera path includes a "set of camera parameters and time" for each virtual viewpoint on the camera path. The camera path can be set on the GUI 2003, for example, as shown in FIG. In the GUI 2003 of FIG. 5, an image of a virtual stadium seen from a viewpoint set by a user operating a keyboard 1001a or a mouse 1001b is displayed. The user operates the keyboard 1001a and the mouse 1001b to set the camera path 2009 indicating the movement locus of the virtual viewpoint. The input unit 205 inputs the camera path of the virtual viewpoint set in this way. The method of acquiring the camera path of the virtual viewpoint by the input unit 205 is not limited to such an acquisition method. For example, the input unit 205 may acquire a camera path created in advance from the storage device 17 or an external device.

The evaluation unit 203 obtains the evaluation result for the camera path in addition to the evaluation result for each virtual viewpoint. The output unit 204 outputs the evaluation result for the camera path to the display device 1002 as an image or characters in addition to the evaluation result for each virtual viewpoint by the evaluation unit 203.

Next, the operation of the main unit 1 according to the present embodiment will be described with reference to the flowchart of FIG. 7. In FIG. 7, the same processing steps as those shown in FIG. 4 are assigned the same step numbers, and the description of the processing steps will be omitted.

In step S2050, the input unit 205 inputs an unselected one of each set in the camera path of the virtual viewpoint as a selection set. Here, the virtual viewpoint corresponding to the selection set is the selection virtual viewpoint, and the time included in the selection set is t.

In step S2060, the evaluation unit 203 selects the evaluation value obtained for the virtual viewpoint located in the vicinity of the selected virtual viewpoint from the "evaluation values of each virtual viewpoint obtained for the scene at time t" obtained in step S1040. Obtained as the evaluation value of the virtual viewpoint. Alternatively, the evaluation unit 203 interpolates the evaluation value of the selected virtual viewpoint by using the evaluation value obtained for the virtual viewpoint located in the vicinity of the selected virtual viewpoint.

In step S2070, the output unit 204 visualizes the evaluation value for each virtual viewpoint and displays it on the display screen of the display device 1002, and visualizes the evaluation value of the selected virtual viewpoint and displays it on the display screen of the display device 1002.

For example, as shown in FIG. 5, when an image of a virtual stadium viewed from a certain viewpoint is displayed on the GUI 2003, the pixel color at the pixel position corresponding to the position of each virtual viewpoint is used as the first embodiment. Determine and draw in the same way. In FIG. 5, a virtual viewpoint region 2006 having a relatively high evaluation value and a virtual viewpoint region 2007 having a relatively low evaluation value are visualized.

Further, in the present embodiment, an object (a quadrangular pyramid in FIG. 5) is arranged at a position corresponding to each set on the camera path 2009 with the position and orientation indicated by the camera parameters in the set, and a line is drawn between the objects. Tie with. When a quadrangular frustum is used as an object, for example, a quadrangular pyramid representing the field of view of the virtual viewpoint is drawn three-dimensionally at the position of the virtual viewpoint according to the direction of the virtual viewpoint. Further, the quadrangular pyramid is drawn in a color corresponding to the evaluation value obtained for the virtual viewpoint corresponding to the quadrangular frustum. For the straight line connecting the quadrangular pyramids, for example, the color of the evaluation value of the virtual viewpoint on the one end side is graduated from the color of the evaluation value of the virtual viewpoint on the other end side from one end side to the other end side. Draw like this.

Further, an index corresponding to the evaluation value for each virtual viewpoint on the camera path 2009 or the virtual viewpoint designated on the screen by the user by operating the keyboard 1001a or the mouse 1001b may be displayed as the index 2010. The index 2010 is, for example, “◯” if the evaluation value is θ1 or more, “×” if it is less than θ2 (θ2 <θ1), and “Δ” if it is less than θ1 and θ2 or more.

Then, in step S2080, the input unit 205 determines whether or not all the sets in the camera path of the virtual viewpoint are selected as the selection set. As a result of this determination, when all the sets are selected as the selection set, the processing according to the flowchart of FIG. 7 ends. On the other hand, if there is still a set that has not been selected as the selection set, the process returns to step S2050 via step S2080.

Looking at the display screen of such a display device 1002, it can be seen that the user should consider creating a camera path of a virtual viewpoint inside the area 2006, so that the work efficiency of determining the camera path is improved.

In the present embodiment, the time is used as an index for specifying the scene corresponding to each virtual viewpoint on the camera path, so that the time does not have to be an exact time and is an appropriate index. It may be.

<Modification example>
As described above, the input method of the camera path of the virtual viewpoint is not limited to the specific input method. For example, the input is not limited to the keyboard 1001a and the mouse 1001b, and may be input using a 3D mouse, a joystick, a game controller, a camera, a three-dimensional sensor, or the like.

Further, the time included in the camera parameters of each virtual viewpoint on the camera path 2009 may be synchronized with the imaging of the virtual space by the imaging device. However, the camera parameters of a plurality of virtual viewpoints may be input with the time paused, or the camera parameters of the virtual viewpoint may be input with the fast forward or rewind. Further, the virtual viewpoint that can be input by the user for creating the camera path may be limited according to the evaluation value of the virtual viewpoint. For example, when a user inputs a virtual viewpoint, an evaluation value for the virtual viewpoint is obtained, and if the obtained evaluation value is equal to or less than a threshold value, the input is canceled and the input virtual viewpoint cannot be used in the camera path. You may do so. Further, for example, it is not necessary to accept the input of the virtual viewpoint of the evaluation value below the threshold value.

Further, as a method of visualizing the evaluation value by the output unit 204, it may be displayed two-dimensionally as in the first embodiment or may be displayed three-dimensionally as shown in FIG. Further, the method of presenting the index is not limited to the symbol as shown in the index 2010 of FIG. 5, and other figures, images, texts, and voices may be used. Further, when the input evaluation value of the virtual viewpoint is equal to or less than the threshold value, a warning may be output by sound or an image. Further, the conditions for presenting the index are not limited to the above conditions. For example, the index may be presented only when the evaluation value is lower than the threshold value, or the index may be presented according to the evaluation value. That is, the output form of the information by the output unit 204 is not limited to a specific output form, and may be output by text or voice.

Further, in addition to the second embodiment, the camera path input in the present embodiment and the data necessary for generating the virtual viewpoint image may be input to the virtual viewpoint image generation system (not shown) to generate the virtual viewpoint image. good. This also applies to the following embodiments and modifications.

[Third Embodiment]
In the present embodiment, the evaluation value for each virtual viewpoint on the camera path is obtained by a method different from that of the second embodiment. For example, in order to select either mode A in which the evaluation result for the camera path is obtained by using the method described in the second embodiment, or mode B in which the evaluation result for the camera path is obtained by using the method according to the present embodiment. Screen is displayed on the display device 1002. Then, the user operates the screen using the keyboard 1001a or the mouse 1001b to obtain the evaluation result for the camera path by the method corresponding to the selected mode. When the mode A is selected, the evaluation unit 203 performs the operation described in the second embodiment, but when the mode B is selected, the evaluation unit 203 performs the process according to the flowchart of FIG.

In step S3020, the evaluation unit 203 acquires the actual position of each player as dynamic information from the images captured by the respective image pickup devices actually provided in the stadium. Each image pickup device continuously captures images in synchronization with each other, and captures a scene at each time such as an image captured at time t1, an image captured at time t2, an image captured at time t3, and so on. The evaluation unit 203 acquires the position of each player at time t1 as dynamic information from the captured image captured by each image pickup device at time t1, for example, by the method described in Non-Patent Document 2. Similarly, the evaluation unit 203 acquires the position of each player at each time as dynamic information such as time t2, t3, ....

In step S3050, the input unit 205 inputs an unselected one of each set in the camera path of the virtual viewpoint as a selection set. Here, the virtual viewpoint corresponding to the selection set is the selection virtual viewpoint, and the time included in the selection set is t.

In step S3060, the evaluation unit 203 first obtains an evaluation value of the player for each player based on the number of image pickup devices including the player in the angle of view at time t. In the case of the player of interest, the number N of the image pickup devices including the position of the player of interest at the time t in the angle of view is counted, and the ratio of the counted number of units N to the number of all image pickup devices is taken as the evaluation value of the player of interest. Regarding whether or not the player of interest is included in the angle of view of the image pickup device, for example, the position of the player of interest is obtained by projecting the position of the player of interest onto the image pickup surface of the image pickup device based on the camera parameters of the image pickup device. If the obtained projection position is included in the angle of view of the image pickup device, it is determined that the player of interest is included in the angle of view of the image pickup device. Then, the evaluation unit 203 obtains the average value of the evaluation values of the players in the angle of view of the selected virtual viewpoint as the evaluation value for the selected virtual viewpoint.

<Modification example>
As dynamic information, it suffices if the position of each player synchronized with the image pickup by the image pickup device can be acquired. The time of the captured data may be set to match the time control of the camera path of the virtual viewpoint of the virtual viewpoint image, may be every frame, may be skipped, or may be the time of rewinding.

Further, the method of calculating the position of each player in the stadium is not limited to the above calculation method, as long as the position of the player can be acquired. For example, the three-dimensional position of the player may be calculated by detecting and associating the feature points from the captured images captured by a plurality of imaging devices, or the player may be measured by a three-dimensional sensor or the like to acquire the three-dimensional position. You may. Further, a sensor such as a GPS or an inertial sensor may be attached to the actual player, and the measurement data acquired from the sensor may be acquired as the position of the player. Further, the dynamic information is not limited to the position of the player, and may include the posture of the player, or may be a three-dimensional shape, a three-dimensional point cloud, or RGB-D data. As described above, the method of acquiring dynamic information is not limited to a specific acquisition method.

Further, the method of acquiring the evaluation value by the evaluation unit 203 is not limited to the above method, and it is sufficient that the evaluation value for the predicted image quality of the virtual viewpoint image can be acquired based on the dynamic information. For example, instead of projecting the position of the player to make a visual determination, a three-dimensional shape representing the player may be projected onto an image plane of an image pickup device or a virtual viewpoint, and the visual determination may be made in consideration of occlusion between players. Further, if the player is imaged in a large size, the resolution of the player in the virtual viewpoint image is increased and the image quality is improved, so that the evaluation may be performed based on the number of pixels taken by the player. For example, a function that monotonically increases with respect to the number of shot pixels of the player may be used as an evaluation function, or may be used by weighted addition with a plurality of evaluation functions such as the above-mentioned evaluation function for the number of units. Further, a virtual player is placed at the position of the player based on the dynamic information, and the evaluation value for each virtual viewpoint on the camera path is obtained based on the virtual viewpoint image and the correct answer image as in the first embodiment. May be. The virtual player is not limited to CG, and may use a three-dimensional shape or an image measured in the past. As described above, according to the present embodiment, since the index regarding the image quality of the virtual viewpoint image is presented at the time of inputting the camera path, the work efficiency of determining the camera path is improved.

In the above embodiment and modification, the image quality of the virtual viewpoint image in the stadium where the player exists is evaluated, but the field is not limited to the stadium, and the field is not limited to the player. It may be another subject. In addition, the above-mentioned embodiments and modifications may be used in combination in part or in whole, or may be selectively used.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101: 1st acquisition unit 102: 2nd acquisition unit 103: Evaluation unit 104: Output unit

Claims (17)

Specify the position of the virtual viewpoint, the line-of-sight direction from the virtual viewpoint, and the range of the field of view of the virtual viewpoint related to the generation of the virtual viewpoint image based on the plurality of captured images acquired by the plurality of image pickup devices that capture images from different positions. Specific means and
A means for providing a user with information indicating an evaluation result regarding the image quality of a virtual viewpoint image according to the position of the virtual viewpoint specified by the specific means, the line-of-sight direction from the virtual viewpoint, and the range of the field of view of the virtual viewpoint. An information processing device characterized by being provided. It has a reception means that accepts inputs according to user operations that specify the position of the virtual viewpoint, the direction of the line of sight from the virtual viewpoint, and the range of the field of view of the virtual viewpoint .
The specific means is characterized in that the position of the virtual viewpoint designated by the user operation, the line-of-sight direction from the virtual viewpoint, and the range of the field of view of the virtual viewpoint are specified based on the input received by the reception means. The information processing apparatus according to claim 1. An acquisition means for acquiring a virtual viewpoint image according to the position of the virtual viewpoint specified by the specific means, the line-of-sight direction from the virtual viewpoint, and the range of the field of view of the virtual viewpoint.
It has an evaluation means for evaluating the image quality of the virtual viewpoint image acquired by the acquisition means.
The information processing apparatus according to claim 1 or 2, wherein the providing means provides information indicating an evaluation result by the evaluation means to a user . The information processing apparatus according to any one of claims 1 to 3, wherein the evaluation result regarding the image quality of the virtual viewpoint image includes an evaluation result regarding the resolution of a subject included in the virtual viewpoint image. The evaluation result regarding the image quality of the virtual viewpoint image is a plurality of virtual viewpoint images according to the position of the virtual viewpoint specified by the specific means, the line-of-sight direction from the virtual viewpoint, and the range of the field of view of the virtual viewpoint. The virtual viewpoint image generated based on the captured image and other images corresponding to the position of the virtual viewpoint, the line-of-sight direction from the virtual viewpoint, and the range of the field of view of the virtual viewpoint of the virtual viewpoint image. The information processing apparatus according to any one of claims 1 to 4, wherein the information processing apparatus includes an evaluation result relating to a difference between the image quality and another image as a reference for evaluating the image quality . Claim 1 is characterized in that the providing means provides a user with an evaluation result regarding the image quality of the virtual viewpoint image by outputting at least one of characters, symbols, figures, colors, image patterns, and sounds. The information processing apparatus according to any one of 5 to 5. The information provided by the providing means is an evaluation result based on the evaluation of the image quality of the virtual viewpoint image according to the position of the virtual viewpoint in each of the plurality of scenes, the line-of-sight direction from the virtual viewpoint, and the range of the field of view of the virtual viewpoint. The information processing apparatus according to any one of claims 1 to 6 , wherein the information processing apparatus is characterized by the above. The information provided by the providing means is virtual according to the position of the virtual viewpoint, the line-of-sight direction from the virtual viewpoint, and the range of the field of view of the virtual viewpoint at the positions corresponding to each of the plurality of virtual viewpoints on the display screen. The information processing apparatus according to any one of claims 1 to 7 , wherein the information processing apparatus includes information in which a color corresponding to an evaluation result regarding the image quality of a viewpoint image is displayed. The information processing according to any one of claims 1 to 8 , wherein the evaluation result regarding the image quality of the virtual viewpoint image includes an evaluation result based on a statistic of a prediction error of a feature point in the virtual viewpoint image. Device. The specific means specifies the positions of a plurality of virtual viewpoints on the movement path of the virtual viewpoints, the line-of-sight directions from the plurality of virtual viewpoints, and the range of the visual fields of the plurality of virtual viewpoints.
The providing means evaluates the image quality of each of the plurality of virtual viewpoint images according to the position of the plurality of virtual viewpoints, the line-of-sight direction from the plurality of virtual viewpoints, and the range of the visual fields of the plurality of virtual viewpoints. The information processing apparatus according to any one of claims 1 to 9 , wherein the information to be shown is provided to the user . The evaluation results regarding the image quality of each of the plurality of virtual viewpoint images according to the positions of the plurality of virtual viewpoints on the movement path, the line-of-sight directions from the plurality of virtual viewpoints, and the range of the visual fields of the plurality of virtual viewpoints are the plurality . The information processing apparatus according to claim 10 , wherein the virtual viewpoint of the above is acquired based on the evaluation result of the acquired image. The information provided by the providing means includes information in which an image in which an object of a color corresponding to an evaluation result corresponding to the virtual viewpoint is arranged at a position of a plurality of virtual viewpoints on the movement path is displayed. The information processing apparatus according to claim 10 or 11 . One of claims 10 to 12 , wherein the information provided by the providing means includes an image in which an index corresponding to an evaluation result corresponding to a plurality of virtual viewpoints on the moving path is arranged. The information processing device described in. Depending on the evaluation result regarding the image quality of the virtual viewpoint image, the generation of the virtual viewpoint image according to the position of the virtual viewpoint specified by the specific means, the line-of-sight direction from the virtual viewpoint, and the range of the field of view of the virtual viewpoint is restricted. The information processing apparatus according to any one of claims 1 to 13 , further comprising a processing means for performing the processing. The evaluation result regarding the image quality of the virtual viewpoint image is acquired based on the evaluation value of the subject included in the visual field of the virtual viewpoint specified by the specific means.
The information processing apparatus according to any one of claims 1 to 14 , wherein the evaluation value of the subject is a value based on the number of image pickup devices including the subject in the field of view. It is an information processing method performed by an information processing device.
Specify the position of the virtual viewpoint, the line-of-sight direction from the virtual viewpoint, and the range of the field of view of the virtual viewpoint related to the generation of the virtual viewpoint image based on the plurality of captured images acquired by the plurality of image pickup devices that capture images from different positions. Specific process and
A providing process of providing to the user information indicating the evaluation result regarding the image quality of the virtual viewpoint image according to the position of the virtual viewpoint specified in the specific step, the line-of-sight direction from the virtual viewpoint, and the range of the field of view of the virtual viewpoint. An information processing method characterized by being prepared. A computer program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 15 .

Images