JP7471784B2 - Providing device, control method, and program - Google Patents

Description

The present invention relates to a technology for generating virtual viewpoint images.

In recent years, a technology that has attracted attention is one in which multiple imaging devices are installed in different positions, capture images synchronously, and a virtual viewpoint image is generated using the multiple captured images based on the images captured. With this technology, an image seen from a viewpoint (virtual viewpoint) within an imaging area is generated by specifying the position and direction of the viewpoint. Furthermore, the image quality of the virtual viewpoint image generated at this time is affected by the accuracy of the color information of the subject obtained from the captured images, the resolution of the virtual viewpoint image, and other factors.

Patent document 1 describes the display of an area in a three-dimensional space captured by multiple imaging devices where a high-resolution virtual viewpoint image can be generated.

US Patent Application Publication No. 2013/0321575

However, even if the technology described in Patent Document 1 is used to set a virtual viewpoint in an area where a high-resolution virtual viewpoint image can be generated, the image quality of the virtual viewpoint image may deteriorate depending on the direction of the set virtual viewpoint. This is because if an image is not captured from a direction equivalent to the virtual viewpoint direction, color information of the subject corresponding to the direction of the virtual viewpoint is not obtained. Therefore, in order to generate a high-quality virtual viewpoint image, it is necessary to set an appropriate virtual viewpoint direction.

The present invention was made in consideration of the above problems. Its purpose is to make it possible to set a virtual viewpoint in an appropriate direction.

The providing device of the present invention is characterized in that it has an acquisition means for acquiring information indicating the imaging directions of multiple imaging devices for acquiring multiple captured images used to generate a virtual viewpoint image, an output means for outputting information regarding an evaluation result obtained by evaluating, when the direction of the virtual viewpoint used to generate the virtual viewpoint image is set to a predetermined direction, the direction based on the imaging direction indicated by the information acquired by the acquisition means , a determination means for determining the direction of the virtual viewpoint based on an operation by a user, and a control means for causing a display means to display a screen notifying the user of the evaluation result output by the output means based on the direction of the virtual viewpoint determined by the determination means and the predetermined direction .

The present invention makes it possible to set a virtual viewpoint in an appropriate direction.

FIG. 1 is a diagram showing an example in which a plurality of imaging devices are arranged. FIG. 2 is a diagram for explaining a hardware configuration of a providing device 200. 2 is a diagram for explaining the functional configuration of an image processing system 1000 including a providing device 200. [0023] FIG. 11 is a flowchart illustrating a process performed by a providing device 200. 13 is a flowchart for explaining an evaluation process performed by an evaluation unit 303. FIG. 13 is a diagram showing an example of a graph generated by an evaluation unit 303. 2 is a diagram for explaining the functional configuration of an image processing system 1100 including a providing device 230. FIG. 13 is a flowchart illustrating a process performed by a providing device 230. 7 is a diagram for explaining an example of a control process performed by a control unit 703. FIG. 2 is a diagram for explaining the functional configuration of an image processing system 1200 including a providing device 240. FIG. 13 is a flowchart illustrating a process performed by a providing device 240. FIG. 11 is a diagram for explaining a modified example of the first embodiment.

The following describes an embodiment of the present invention with reference to the drawings. Note that the components described in the following embodiment are examples of the implementation of the present invention, and do not limit the scope of the present invention to these.

First Embodiment
In this embodiment, a providing device 200 that provides information on the image quality of a virtual viewpoint image according to the direction of the virtual viewpoint will be described. Here, the virtual viewpoint image is an image that represents the view from a specified virtual viewpoint based on multiple images based on imaging by multiple imaging devices and a specified arbitrary viewpoint (virtual viewpoint). The multiple imaging devices can be arranged to surround an imaging area, for example, as in the imaging device 100 shown in FIG. 1. The imaging area is, for example, a stadium where sports such as soccer and karate are held, or a stage where a concert or a play is held. The multiple imaging devices are installed at different positions so as to surround such an imaging area, and capture images synchronously. Note that the multiple imaging devices do not need to be installed around the entire circumference of the imaging area, and may be installed only in a partial direction of the imaging area depending on restrictions on the installation location, etc. The number of imaging devices is not limited to the example shown in the figure, and for example, when the imaging area is a soccer stadium, about 30 imaging devices may be installed around the stadium. In addition, imaging devices with different functions, such as a telephoto camera and a wide-angle camera, may be installed. In addition, the virtual viewpoint image in this embodiment is also called a free viewpoint image, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, and for example, an image corresponding to a viewpoint selected by the user from a plurality of candidates is also included in the virtual viewpoint image. The virtual viewpoint image in this embodiment may be a moving image or a still image. That is, the following embodiments are applicable to both moving images and still images.

Figure 2 shows a hardware configuration diagram of the providing device 200. The providing device 200 has a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, a communication I/F 215, and a bus 216. The CPU 211 realizes each function of the providing device 200 shown in Figure 1 by controlling the entire providing device 200 using computer programs and data stored in the ROM 212 and the RAM 213. Note that the providing device 200 may have one or more dedicated hardware pieces different from the CPU 211, and at least a part of the processing by the CPU 211 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

The ROM 212 stores programs that do not require modification. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, and data supplied from the outside via the communication I/F 217. The auxiliary storage device 214 is composed of, for example, a hard disk drive, and stores various data such as image data and audio data. The communication I/F 215 is used for communication with devices external to the providing device 200. For example, when the providing device 200 is connected to an external device via a wire, a communication cable is connected to the communication I/F 215. When the providing device 200 has a function for wireless communication with an external device, the communication I/F 215 is equipped with an antenna. The bus 216 connects each part of the providing device 200 to transmit information.

The providing device 200 in this embodiment may be connected to, for example, a plurality of imaging devices 100, an output device 217, an input device 218, and the like. The output device 217 is composed of, for example, a liquid crystal display or an LED, and displays a GUI (Graphical User Interface) for the user to operate the providing device 200. The input device 218 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, and the like, and inputs various instructions to the CPU 211 in response to operations by the user. In this embodiment, the output device 217 and the input device 218 are connected to the outside of the providing device 200, but at least one of the output device 217 and the input device 218 may be present inside the providing device 200 as an output unit or an input unit. In addition, the auxiliary storage device 214 in this embodiment is included inside the providing device 200, but may be configured to be connected to the outside of the providing device 200, or may be configured to be provided both inside and outside.

Figure 3 is a diagram for explaining the configuration of an image processing system 1000 including a providing device 200 in this embodiment. The image processing system 1000 has a plurality of imaging devices 100, a providing device 200, and a display device 304. The providing device 200 also has a storage unit 310, a viewpoint information acquisition unit 301, an evaluation information acquisition unit 302, and an evaluation unit 303. Each component will be explained below.

The multiple imaging devices 100 capture the imaging area from multiple directions. The storage unit 310 is a processing unit corresponding to the auxiliary storage device 214 in FIG. 2. The storage unit 310 in this embodiment stores viewpoint information and evaluation information. The viewpoint information is data acquired from the multiple imaging devices 100 and includes information about the multiple imaging devices 100. The viewpoint information may include parameters indicating, for example, the three-dimensional position, imaging direction, field of view size (angle of view), and resolution of each of the multiple imaging devices 100 as information about the imaging device 100. The viewpoint information is calculated based on the state of the multiple imaging devices 100 according to a known camera calibration method or the like. As an example of camera calibration, the viewpoint information is calculated by associating feature points in multiple captured images obtained by capturing images of the multiple imaging devices 100 with each other and performing geometric calculations. Note that the information included in the viewpoint information is not limited to the above. For example, information other than the above information, such as information indicating the total number of the multiple imaging devices 100, may be included. In addition, only arbitrary information from the above information may be included. In addition, the viewpoint information may have multiple parameter sets. For example, the viewpoint information may have a plurality of parameter sets corresponding to a plurality of frames constituting a video obtained by imaging with the imaging device 100. In this case, the parameter set is information indicating the position and imaging direction of the imaging device 100 at each of a plurality of consecutive points in time. The evaluation information is data including information regarding the settings of the evaluation process performed by the evaluation unit 303 described later. Details of the evaluation information will be described later.

The viewpoint information acquisition unit 301 acquires viewpoint information stored in the storage unit 310 and transmits it to the evaluation unit 303. The evaluation information acquisition unit 302 acquires evaluation information stored in the storage unit 310 and transmits it to the evaluation unit 303. The evaluation unit 303 performs evaluation processing and causes the display device 304 to display information based on the results of the processing. The evaluation processing is processing for acquiring information that allows recognition of the image quality of a virtual viewpoint image related to a virtual viewpoint that views a specific position from a specified direction at that position. The display device 304 is a device that corresponds to the output device 217 in FIG. 2 and is, for example, a display.

The processing performed by the providing device 200 in this embodiment will be described with reference to FIG. 4. The CPU 211 reads out and executes a program stored in the ROM 212 or the auxiliary storage device 214 to perform the following processing. In the following description, processing steps are simply denoted as S. The processing is started when the providing device 200 transmits a request to transmit viewpoint information.

In S401, the viewpoint information acquisition unit 301 acquires viewpoint information from the storage unit 310 and transmits it to the evaluation unit 303. In S402, the evaluation information acquisition unit 302 acquires evaluation information from the storage unit 310 and transmits it to the evaluation unit 303. In S403, the evaluation unit 303 performs evaluation processing using the transmitted viewpoint information and evaluation information. Details of the evaluation processing will be described later with reference to FIG. 5. In S404, the evaluation unit 303 causes the display device 304 to display information based on the results of the processing. When the display device 304 displays the information, the processing ends.

Next, the evaluation process performed by the evaluation unit 303 will be described. The following description will be given with reference to FIG. 5. FIG. 5 is a flowchart for explaining the evaluation process performed by the evaluation unit 303. The flowchart in FIG. 5 is a detailed flowchart of S403 in FIG. 4.

In S501, the evaluation unit 303 acquires information on the evaluation position and evaluation direction included in the evaluation information transmitted from the evaluation information acquisition unit 302. The information on the evaluation position included in the evaluation information (hereinafter referred to as evaluation position information) is information indicating the three-dimensional coordinates of a specific position in the imaging area captured by the multiple imaging devices 100. Furthermore, the information on the evaluation direction (hereinafter referred to as evaluation direction information) is information indicating the direction from which the above evaluation position is evaluated. The evaluation position information and the evaluation direction information may each include information on multiple evaluation positions and multiple evaluation directions. In the following description, it is assumed that, as an example of information included in the evaluation position information, information indicating three coordinates of (x, y, z) = (-3000.0, 2000.0, 0), (0.0, 2000.0, 0.0), and (3000.0, 2000.0, 0) is included. Note that the origin of the x, y, and z axes is set at any position within the imaging area. In the following description, an example of the information included in the evaluation direction information is information indicating direction vectors that end at the evaluation position on an xy plane that includes the evaluation position and are set radially around the evaluation position at 10 degree intervals. That is, based on the evaluation information, the evaluation unit 303 identifies multiple evaluation directions (directional vectors) that point toward the evaluation position, and evaluates the image quality of the virtual viewpoint image for each of the multiple evaluation directions when the evaluation direction is set as the direction of the virtual viewpoint.

The evaluation position information and evaluation direction information in this embodiment are set in advance by the user and stored in the storage unit 310 as evaluation information. However, the configuration of the providing device 200 is not limited to the above. For example, by configuring the providing device 200 to determine at least one of the evaluation position information and the evaluation direction information based on the input operation of the input device 218 by the user, the user can determine an arbitrary evaluation position and evaluation direction by performing the input operation. For example, the providing device 200 can be configured to automatically set evaluation position information indicating a plurality of evaluation positions arranged at equal intervals in the imaging area, and evaluation direction information indicating a plurality of directional vectors arranged radially at equal intervals with respect to the evaluation positions. This configuration reduces the burden on the user for determining the evaluation position and evaluation direction. In addition, in this configuration, when the evaluation position information and evaluation direction information are automatically set, the providing device 200 may be configured to determine the intervals of the arrangement of the evaluation positions and evaluation directions according to the granularity at which the position and direction of the virtual viewpoint can be set. In addition, the evaluation positions and evaluation directions do not necessarily need to be set at equal intervals. The providing device 200 can set arbitrary positions and directions by the user setting the evaluation information in advance. Also, only one evaluation position may be set, or only one evaluation direction may be set for one evaluation position. In other words, the providing device 200 can set one or more arbitrary evaluation positions and evaluation directions based on the evaluation information or an input operation by the user, etc.

In S502, the evaluation unit 303 determines whether or not there is an evaluation position for which the processes from S503 to S505 have not been performed for the multiple evaluation positions indicated by the evaluation position information. If there is an evaluation position for which the processes have not been performed, the evaluation unit 303 proceeds to S503. If there is no evaluation position for which the processes have not been performed, the evaluation unit 303 proceeds to S507.

In S503, the evaluation unit 303 sets an evaluation position for which processing has not been performed as a processing target. In S504, for multiple evaluation directions corresponding to the evaluation position to be processed, it is determined whether or not there is an evaluation direction for which processing has not been performed in S505 and S506. If there is an evaluation direction for which processing has not been performed, the evaluation unit 303 proceeds to S505. If there is no evaluation direction for which processing has not been performed, the evaluation unit 303 performs the processing from S502 onwards again.

In S505, the evaluation unit 303 identifies an imaging device 100 from among the multiple imaging devices 100 based on the evaluation direction. First, the evaluation unit 303 excludes imaging devices 100 whose three-dimensional coordinates of the evaluation position are not included in the angle of view from candidates for imaging devices 100 to be identified based on parameters indicating the position and angle of view of the imaging device 100 included in the viewpoint information transmitted from the viewpoint information acquisition unit 301. The evaluation unit 303 acquires the value of the angle between the evaluation direction and the imaging direction of the imaging device 100 for each of the remaining multiple imaging devices 100 based on the viewpoint information. The angle at this time is an angle of 180 degrees or less between the direction vector corresponding to the evaluation direction and the direction vector corresponding to the imaging direction of the imaging device 100. In this embodiment, as an example, the angle between the xy in-plane component of the evaluation direction and the xy in-plane component of the imaging direction is used for the evaluation process. The evaluation unit 303 also identifies the imaging device 100 whose acquired angle value is the smallest.

In S506, the evaluation unit 303 stores information indicating the value of the angle and information indicating the spatial resolution of the imaging device 100 for the imaging device 100 identified in S505 in the storage unit 310. Here, the spatial resolution in this embodiment is an index obtained based on the resolution performance of the imaging device 100. Specifically, the spatial resolution of the imaging device 100 indicates how many pixels are used to represent a unit cube (voxel) of a predetermined size (for example, 1 mm square) virtually installed at the evaluation position when the projected voxel is projected onto the projection surface of the imaging device 100. The greater the number of pixels, i.e., the higher the spatial resolution, the more finely the imaging device 100 can represent the space. The evaluation unit 303 stores the information indicating the angle and spatial resolution corresponding to the identified imaging device 100 in the storage unit 310 in association with the evaluation direction, and then returns the process to S503. Thereafter, the processes from S503 to S506 are repeated until the processes are performed for all evaluation positions.

After processing all the evaluation positions, in S506, the evaluation unit 303 generates a graph to be used for setting the virtual viewpoint based on information about the image capture device 100 corresponding to each evaluation position. FIG. 6 is a diagram showing an example of a graph generated by the evaluation unit 303. The graph shown in FIG. 6 is a graph generated based on the result of evaluating the image quality of the virtual viewpoint image according to a plurality of evaluation directions corresponding to the evaluation position. Graph 601 is a graph corresponding to the evaluation position (x, y, z) = (-3000.0, 2000.0, 0). Also, arrow 602 is an example of a direction vector indicating the evaluation direction at the evaluation position. The evaluation unit 303 colors each direction vector according to the angle it makes with the identified imaging direction of the imaging device 100. For example, the color is changed in four stages depending on whether the angle between the direction vector and the imaging direction of the imaging device 100 is less than 10 degrees, 10 degrees or more and less than 30 degrees, 30 degrees or more and less than 45 degrees, or 45 degrees or more. In this way, the evaluation unit 303 evaluates the image quality corresponding to the evaluation direction depending on whether the angle between the evaluation direction and the imaging direction of the imaging device 100 is less than the above threshold. The threshold and color for color coding are not limited to the above. Also, the color coding is not limited to the classification by different colors, and may be expressed by light shades as shown in FIG. 6. The evaluation unit 303 generates a graph as shown in FIG. 6 based on the color coding result. In the graph of FIG. 6, a sector centered on the evaluation position is associated with each evaluation direction, and coloring is performed based on the above color coding result. Also, the size of the radius of the sector is associated with the high or low spatial resolution of the imaging device 100 specified by the evaluation direction corresponding to the sector. In generating a virtual viewpoint image, the smaller the angle between the virtual viewpoint direction and the imaging direction of the imaging device 100, the smaller the deviation between the image seen from the virtual viewpoint and the texture used when generating the virtual viewpoint image, so that the image quality of the generated virtual viewpoint image is expected to be high. Also, the higher the spatial resolution of the imaging device 100, the finer the generated virtual viewpoint image is expressed, so that the image quality is expected to be high. Now, for the graph 603, it is assumed that the direction vector corresponding to the direction of the virtual viewpoint 604 is colored green, and the direction vector corresponding to the direction of the virtual viewpoint 605 is colored yellow. At this time, at the evaluation position (x, y, z) = (3000.0, 2000.0, 0), it is estimated that the image quality of the virtual viewpoint image when the evaluation position is viewed from the virtual viewpoint 605 is higher than that of the virtual viewpoint 604, based on the color and radius of the sector shown by the graph 603. Therefore, by displaying the graph generated in S506 on the display device 304, the user can estimate how to set the direction of the virtual viewpoint to generate a virtual viewpoint image with good image quality while looking at the graph. As a result, when setting a virtual viewpoint that can obtain a virtual viewpoint image with good image quality, the user can set an appropriate direction of the virtual viewpoint using the graph. Note that the providing device 200 is assumed to perform the above series of evaluation processes only once, but it is also possible to improve the accuracy of the evaluation process by configuring the evaluation process to be performed again when conditions change, such as when some of the multiple imaging devices 100 fail.

The above is a description of the evaluation process. The expressions of the angle and spatial resolution are not limited to the above example. For example, the angle and spatial resolution may be expressed by a combination of any expression method, such as the expression method using color, brightness, the radial size of the graph, and the height in the z-axis direction of the graph. The evaluation unit 303 may generate a graph so that the graph becomes smaller as the spatial resolution increases. The evaluation unit 303 may generate a graph by associating each numerical value with the saturation or brightness of the color, without classifying the angle as in the above example. The graph shown in FIG. 6 is drawn for all of the multiple directional vectors, but is not limited to this. For example, a directional vector with a small angle and high spatial resolution is selected as a representative directional vector, and only the representative directional vector is drawn, thereby reducing the processing related to drawing the directional vector. The graph displayed on the display device 304 is not limited to the orthogonal projection as in FIG. 6, but may be a graph projected from an arbitrary virtual viewpoint. In this way, various methods are possible for expressing a graph. Furthermore, in the above embodiment, the evaluation unit 303 evaluates image quality based on the imaging direction and resolution of multiple imaging devices 100, but the evaluation unit 303 can evaluate image quality by using at least information indicating the imaging direction of the imaging device 100.

In addition, in the above embodiment, the angle between the evaluation direction and the imaging direction of the imaging device 100 is the angle between the xy in-plane component of the evaluation direction and the xy in-plane component of the imaging direction, but this is not limited to this. By considering the angle between the spatial vector corresponding to the evaluation direction and the spatial vector corresponding to the imaging direction as the angle, the providing device 200 can evaluate the image quality for the evaluation direction corresponding to the three-dimensional space. In addition, as a method of expressing the graph at this time, a hemisphere centered on the evaluation position may be generated, and a graph may be generated that also takes into account the height of the virtual viewpoint in the z-axis direction.

In the above embodiment, the evaluation unit 303 of the providing device 200 generates a graph, but the present invention is not limited to this. For example, the evaluation unit 303 may transmit information acquired based on the processes from S502 to S505 to the display device 304, and the display device 304 may generate a graph based on the transmitted information. The display device 304 may separately acquire virtual viewpoint information and a virtual viewpoint image and superimpose the graph to display the graph, thereby making it possible to provide the user with a virtual viewpoint image and a graph in association with each other. The display device 304 may also be configured to superimpose the coordinates of an evaluation position in the imaging area and a graph corresponding to the evaluation position, and display the graph in association with the virtual viewpoint image or the image overlooking the imaging area. In this case, when there are multiple evaluation positions, the display device 304 displays a screen in which multiple graphs are arranged at the respective coordinates of the same virtual viewpoint image or the image overlooking the imaging area. With this configuration, the user can easily recognize which position in the imaging area should be set as a virtual viewpoint from which direction.

As described above, the providing device 200 in this embodiment performs evaluation processing based on viewpoint information and evaluation information to generate a graph. By looking at the graph displayed on the display device 304, the user can easily recognize how to set the direction of the virtual viewpoint that includes a specific position (evaluation position) in the field of view in order to generate a virtual viewpoint image with good image quality. As a result, when setting a virtual viewpoint, the user can use the above graph to set the direction of the virtual viewpoint that can generate a high-quality virtual viewpoint image. Note that, although an example of generating a graph as information for setting the direction of the virtual viewpoint by the providing device 200 in this embodiment has been described, the information generated by the providing device may be other than a graph. The providing device 200 may be configured to generate a table in which the evaluation direction and the above-mentioned angle are associated, for example.

(Modification of the first embodiment)
As a modification of the first embodiment, another method for evaluating image quality will be described. As an example, a case where image quality corresponding to the evaluation position 12 is evaluated will be described with reference to FIG. 12. The evaluation unit 303 identifies an image capture device 100 that includes the evaluation position 12 in its angle of view from among a plurality of image capture devices 100 based on viewpoint information. The evaluation unit 303 calculates a line segment 13 connecting the position of the identified image capture device 100 and the evaluation position 12. Furthermore, the evaluation unit 303 calculates a sector 14 with a central angle θ centered on the calculated line segment 13 for a circle centered on the evaluation position. Note that the central angle θ at this time is an arbitrary value and is assumed to be determined in advance. When there are a plurality of identified image capture devices 100, the sector 14 is calculated for each image capture device 100 in the same manner. In the example shown in FIG. 12, one sector 14 is illustrated as a representative. After calculating all sectors 14, the evaluation unit 303 identifies an area 15 that is not included in all sectors. The area 15 is an area indicating that there is no image capture device 100 that captures the evaluation position 12 from the direction of the area. Therefore, it is estimated that the image quality of the virtual viewpoint image corresponding to the virtual viewpoint viewing the evaluation position 12 from the direction of the area 15 is low. The evaluation unit 303 causes the display device 304 to display a screen display such as that shown in Fig. 12. This allows the user to easily estimate from which direction the virtual viewpoint should be set to view the evaluation position 12 in order to obtain a virtual viewpoint image with good image quality.

In the above modified example, the evaluation direction does not necessarily need to be specified, and the image quality can be evaluated based on the evaluation position and the viewpoint information of the multiple imaging devices 100. In addition, the evaluation unit 303 can be configured to calculate how many overlapping sectors 14 (hereinafter referred to as the overlapping number) of a specific region after calculating the sectors 14 corresponding to each imaging device 100, and to color-code the region according to the overlapping number. For example, since the above region 15 is not included in all sectors, the overlapping number of region 15 is 0. In this case, the evaluation unit 303 can color-code the region whose overlapping number is equal to or less than a predetermined threshold in the same way as the region 15, or can color-code the region with a different color for each overlapping number. By configuring in this way, it is possible to evaluate the image quality in more detail. In addition, the evaluation unit 303 can be configured to obtain multiple levels of image quality by setting multiple central angles θ with different values. In addition, the screen display shown in FIG. 12 is an example, and is not limited to this. Various expression methods described in the first embodiment can be used.

Second Embodiment
In the above-described first embodiment, an example has been described in which a graph is generated from the results based on the evaluation process and displayed on the display device 304, thereby providing the user with information to be used as a reference when setting a virtual viewpoint. In this embodiment, a providing device 230 that provides information when the user sets a virtual viewpoint will be described. Note that the hardware configuration of the providing device 230 is assumed to be the same as that of the providing device 200 in the first embodiment. In the following description, the same reference numerals will be used to designate the same functional configurations and processing flows as those in the first embodiment, and description thereof will be omitted.

Figure 7 is a diagram for explaining the functional configuration of an image processing system 1100 including a providing device 230 in this embodiment. In addition to the configuration of the image processing system 1000, the image processing system 1100 includes an operation device 701 and a speaker 704. Furthermore, the providing device 230 included in the image processing system 1100 has a virtual viewpoint determination unit 702 and a control unit 703 in addition to each processing unit of the providing device 200.

The operation device 701 corresponds to the input device 218 in FIG. 2 and is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc. The user can operate the operation device 701 to perform input for setting the position, direction, and field of view (angle of view) of the virtual viewpoint. In the following explanation, information including parameters indicating the position, direction, and angle of view of the virtual viewpoint is referred to as virtual viewpoint information. In addition to setting virtual viewpoint information, the user can operate the operation device 701 to perform various inputs such as operation commands to the providing device 230.

The virtual viewpoint determination unit 702 determines virtual viewpoint information based on an input operation of the operation device 701 by the user. Note that the operation device 701 may determine virtual viewpoint information and transmit it to the virtual viewpoint determination unit 702. Alternatively, the virtual viewpoint determination unit 702 may acquire virtual viewpoint information stored in advance in the auxiliary storage device 214 or the like. The virtual viewpoint determination unit 702 transmits the acquired virtual viewpoint information to the control unit 703. The control unit 703 acquires information based on the result of the evaluation process from the evaluation unit 303, performs a control process described below, and outputs information obtained based on the result of the control process to the display device 304 and the speaker 704. The speaker 704 corresponds to the output device 217 in FIG. 2. The speaker 704 provides information to the user by outputting sound or the like based on the information output from the control unit 703.

Next, the processing performed by the providing device 230 will be described with reference to FIG. 8. FIG. 8 is a flowchart for explaining the processing executed by the providing device 230. Note that the processing in S401, S402, S403, and S404 is the same as in the first embodiment, and therefore the description will be omitted.

In S801, the control unit 703 acquires information based on the result of the evaluation process executed in S403 from the evaluation unit 303. In S802, the virtual viewpoint determination unit 702 determines virtual viewpoint information based on an input operation of the operation device 701 by the user. In addition, the virtual viewpoint determination unit 702 transmits the determined virtual viewpoint information to the control unit 703. In S803, the control unit 703 performs control processing based on the acquired information.

Here, the control process will be described. FIG. 9 is a diagram for explaining an example of the control process performed by the control unit 703. The control unit 703 calculates the positional relationship between the virtual viewpoint and the evaluation position from the virtual viewpoint information transmitted from the virtual viewpoint determination unit 702 and the information generated based on the result of the evaluation process performed by the evaluation unit 303. In the example of FIG. 9, the control unit 703 calculates rays 901 and 902 connecting the evaluation position within the angle of view of the virtual viewpoint 40 and the position of the virtual viewpoint 40, and selects a direction vector that is most similar to the ray for each evaluation position from among multiple direction vectors. The control unit 703 refers to the evaluation result of the image quality corresponding to the selected direction vector, and determines a control value based on the evaluation result. The control value is a value determined for the control unit 703 to control the display device 304 and the speaker 704. In addition, if the evaluation result of the image quality does not meet a predetermined standard, the control value becomes larger as it deviates from the standard. For example, consider a case where the predetermined criterion is that the angle between the direction vector and the imaging direction of the imaging device 100 is less than 30 degrees or the spatial resolution is a predetermined value (e.g., 0.5 pixels) or more. In this case, if the angle is 30 degrees or more and the spatial resolution is less than 0.5 pixels for the image quality corresponding to the selected direction vector, the control value increases as the angle increases and the spatial resolution decreases. On the other hand, if the image quality corresponding to the selected direction vector satisfies the above criterion, the control value becomes 0. Note that the content of the criterion is not limited to the above. The control value is determined by preparing a table in which the angle and spatial resolution are associated with the control value in advance. However, the present invention is not limited to this method, and the control value may be calculated using a function of the angle and spatial resolution.

In S804, the control unit 703 determines whether the control value is 0 or not. If it is determined that the control value is 0, the process ends. If it is determined that the control value is not 0, the process proceeds to S805. In S805, the control unit 703 transmits a signal based on the control value to the speaker 704, and causes the speaker 704 to output a sound according to the control value. The speaker 704 outputs, for example, a sound with a different volume and type according to the control value. Alternatively, for a sound that is sounded at a fixed cycle, a sound whose cycle varies according to the control value may be output.

In S806, when the virtual viewpoint information is determined based on the user's input operation, the control unit 703 controls the display device 304 to display a screen display indicating a warning of image quality degradation. The display device 304 displays icons, text, etc. according to the display control by the control unit 703. At this time, the display device 304 displays the icons, text, etc. with different sizes, colors, or types depending on the control values. The control unit 703 may also be configured to change parameters such as the size, brightness, color, or transmittance of the filter for filter processing that darkens the entire screen, and perform filter processing on the display of the display device 304. The display device 304 may also be configured to display the above-mentioned screen display by superimposing it on the virtual viewpoint image corresponding to the determined virtual viewpoint information.

In S807, the control unit 703 transmits a signal to the operation device 701 to correct the virtual viewpoint information according to the control value. For example, for the virtual viewpoint 41 in FIG. 9, it is assumed that the image quality of the direction vector corresponding to the ray 903 does not meet the standard. In this case, the control unit 703 outputs a signal to the operation device 701 to correct the position, direction, or angle of view of the virtual viewpoint so that the direction vector is not included in the angle of view of the virtual viewpoint 41. Based on the signal transmitted from the control unit 703, the operation device 701 can perform an operation to correct the virtual viewpoint information without user operation. When S807 ends, the process proceeds to S404.

Note that the processes of S805, S806, S807, and S404 do not necessarily have to all be executed, and only optional processes may be executed. The processes of S805, S806, S807, and S404 may also be executed in any order. A control value may be determined for each process from S805 to S807. In this embodiment, the display device on which the graph is displayed in S404 and the display device controlled in S806 are the same, but the display devices may be different devices. In this embodiment, the operation device 701 is connected to the outside of the providing device 230, but may be included inside the providing device 230.

The providing device 230 described above provides information used to set a virtual viewpoint by controlling the speaker 704, the display device 304, and the operation device 701. According to the providing device 230 of this embodiment, when setting a virtual viewpoint, the user can obtain a virtual viewpoint image with good image quality by setting the virtual viewpoint so that no sound is generated or no display is displayed notifying the user of a decrease in image quality. In addition, by correcting the virtual viewpoint information, the risk of the image quality of the virtual viewpoint image decreasing can be reduced.

Third Embodiment
In the above-described first embodiment, a method for evaluating image quality corresponding to each evaluation direction at an evaluation position specified based on evaluation information by referring to viewpoint information has been described. In this embodiment, another example in which the evaluation unit 303 performs evaluation processing will be described. Note that the hardware configuration of the providing device 240 described in this embodiment is assumed to be the same as that of the providing device 200 in the first embodiment. In the following description, the same reference numerals will be used for functional configurations and processing flows that are the same as those in the first and second embodiments, and descriptions thereof will be omitted.

Figure 10 is a diagram for explaining the functional configuration of an image processing system 1200 including a providing device 240 in this embodiment. In addition to each processing unit of the providing device 200, the providing device 240 has an imaging information acquisition unit 1001 and a shape data acquisition unit 1002.

The imaging information acquisition unit 1001 acquires the imaging information stored in the storage unit 310 and transmits it to the evaluation unit 303. Here, the imaging information is data including information about the captured image obtained by the multiple imaging devices 100 capturing the imaging area. The information about the captured image is information related to the quality of the captured image, and includes, for example, a numerical value indicating the shaking of the imaging device 100, the blur of the captured image, exposure, white balance, and the signal-to-noise ratio in the captured image. The shaking of the imaging device 100, the blur of the captured image, exposure, and white balance are acquired based on an analysis by an external image analysis device, an external sensor, the settings of the imaging device 100, and information included in the metadata of the captured image. In addition, the captured image may contain noise if the sensitivity of the imaging device 100 is not set appropriately. The signal-to-noise ratio in the captured image is acquired by calculating the ratio of noise in the captured image when the captured image captured with an appropriate sensitivity setting is used as a reference image. In addition, the imaging device 100 may not be able to capture an evaluation position because the evaluation position is blocked by a structure (e.g., a soccer goal) in the imaging area. In addition, the imaging device 100 may not be able to properly capture an imaging target because, for example, spectators, cheering flags, birds, insects, etc. are included in the angle of view of the imaging device 100, and the imaging target is blocked. In order to indicate the above-mentioned situation, the imaging information includes information indicating that the imaging target captured by the imaging device 100 is blocked as information regarding the captured image. The information indicating that the imaging device 100 is blocked is obtained by referring to the coordinates of the structure stored in advance in the auxiliary storage device 214, etc., or by performing image analysis on the captured image to determine whether or not an object that is not the imaging target is included. Note that the information regarding the captured image is not limited to the above example, and may include, for example, only any number of pieces of information, or may include information other than the above.

The shape data acquisition unit 1002 acquires the shape data stored in the storage unit 310 and transmits it to the evaluation unit 303. Here, the shape data is data indicating the shape of the subject captured by the multiple image capture devices 100. The shape data in this embodiment is, for example, mesh polygon data or point cloud data. The mesh polygon data and point cloud data are generated based on the multiple captured images by using a known three-dimensional restoration method (for example, Structure from Motion (SfM)). In addition, data indicating the shape of the subject is linked to information indicating the position of the subject in the captured area. Note that the data included in the shape data is not limited to this. For example, data based on image-based rendering may be included. The shape data is generated in advance using the captured images and stored in the storage unit 310.

The processing performed by the providing device 240 will be described below with reference to FIG. 11. Note that the determination of the start and end of processing, as well as S401, S402, and S404, are the same as in the first embodiment, and therefore will not be described. In S1101, the imaging information acquisition unit 1001 acquires imaging information from the storage unit 310 and transmits it to the evaluation unit 303. In S1102, the shape data acquisition unit 1002 acquires shape data from the storage unit 310 and transmits it to the evaluation unit 303.

In S1103, the evaluation unit 303 calculates the evaluation position based on the shape data. Here, an example of a method for calculating the evaluation position will be described. The evaluation unit 303 performs clustering based on the shape data and the position information linked to the shape data, and divides the objects into groups of objects whose three-dimensional positions are close to each other. The evaluation unit 303 calculates the center of gravity of each group based on the shape data of the objects included in the divided clusters, and sets the coordinates of the center of gravity as the evaluation position. Note that the method for calculating the evaluation position is not limited to this. For example, the evaluation unit 303 can also calculate the evaluation position based only on the position information linked to the shape data. In addition, the evaluation position corresponding to each group is not limited to the center of gravity of the group, and may be set in the vicinity of each group. In addition, for example, the evaluation unit 303 can set the position of the object as the evaluation position without performing clustering. By performing the above processing, the evaluation unit 303 can set the evaluation position according to the distribution of multiple objects in the imaging area.

In S1104, the evaluation unit 303 performs evaluation processing for each evaluation position based on the acquired information. In addition to the processing shown in FIG. 5, the evaluation processing in this embodiment performs evaluation of image quality based on imaging information. The evaluation unit 303 identifies the imaging device 100 based on the evaluation direction, and refers to the imaging information corresponding to the imaging device 100. The evaluation unit 303 obtains an imaging quality value by performing weighted addition of each numerical value included in the imaging information. The weighting coefficient used in the weighted addition is set to an arbitrary value in advance. Furthermore, if there is information indicating that the imaging device 100 is blocked by a structure or the like, the evaluation unit 303 may prevent the imaging device 100 from being identified.

As described above, the evaluation unit 303 performs the above process in addition to the evaluation process described in the first embodiment, and evaluates the image quality corresponding to each evaluation position. The evaluation unit 303 also generates a graph based on the evaluation results and displays it on the display device 304. The generated graph is a graph in which three types of values, the angle, the spatial resolution, and the weighted sum based on the imaging information, are each associated with one of the graph's color, brightness, radial size, and height in the z-axis direction. Alternatively, a weighted average may be calculated from the three types of values, and then graphed using the above-mentioned expression method after being reduced to one or two types of values.

As described above, by adding image quality evaluation processing based on imaging information to the evaluation processing performed by the evaluation unit 303, a more detailed evaluation of image quality becomes possible. By looking at the graph provided by the providing device 240, the user can estimate a more appropriate virtual viewpoint direction for a virtual viewpoint viewing a specific position. Furthermore, by the evaluation unit 303 setting an evaluation position based on shape data, it is possible to evaluate image quality in the vicinity of the subject. Note that, although the present embodiment has been described with respect to a case where both imaging information and shape data are used in the evaluation processing, a configuration in which only one of imaging information and shape data is used may also be used.

Other Embodiments
The providing device in the above-described embodiment acquires viewpoint information as information indicating the imaging directions of a plurality of imaging devices for acquiring a plurality of captured images used to generate a virtual viewpoint image. In addition, the providing device outputs information on an evaluation result obtained by evaluating a predetermined direction (evaluation direction) of the virtual viewpoint based on the information indicating the imaging direction when the direction of the virtual viewpoint is set to the predetermined direction. According to the above embodiment, it is possible to set a virtual viewpoint in an appropriate direction.

In the above embodiment, the providing device is configured to perform the evaluation process on the evaluation direction, but the present invention is not limited to this. The providing device may be configured to obtain the evaluation results evaluated by other devices based on the viewpoint information, and output information related to the evaluation results.

In the above embodiment, the information provided by the providing device is mainly used when the user sets a virtual viewpoint, but the present invention is not limited to this. For example, it is also possible to automatically set the virtual viewpoint based on the information provided by the providing device. The embodiment in this case will be described with reference to the functional configuration shown in FIG. 7. The evaluation unit 303 causes the display device 304 to display a screen display that allows the user to specify an evaluation position in the imaging area. The user operates the operation device 701 while looking at the display screen of the display device 304, and specifies a position in the imaging area where the user wants to see a virtual viewpoint image (for example, in front of a soccer goal, etc.). The operation device 701 transmits information (position information) indicating the position specified by the input operation by the user to the evaluation unit 303. The evaluation unit 303 performs an evaluation process on the position indicated by the acquired position information, and transmits the obtained processing result to the virtual viewpoint determination unit 702. The virtual viewpoint determination unit 702 automatically determines a virtual viewpoint that faces the direction of the specified position and is capable of generating a high-quality virtual viewpoint image based on the acquired result of the evaluation process. At this time, when a plurality of positions are designated by the user, the virtual viewpoint determination unit 702 determines a moving path of the virtual viewpoint that faces the direction of the designated plurality of positions in order. At this time, as an initial value of the moving path, the virtual viewpoint determination unit 702 selects a path connecting the designated plurality of positions. The evaluation unit 303 performs an evaluation process for each of the designated plurality of positions. Based on the result of the evaluation process, the virtual viewpoint determination unit 702 determines whether or not a virtual viewpoint image corresponding to a virtual viewpoint passing through the path selected as the initial value satisfies a predetermined image quality. If it is determined that the image does not satisfy the predetermined image quality, the virtual viewpoint determination unit 702 corrects the position or direction of the virtual viewpoint so that the virtual viewpoint is in the direction of a virtual viewpoint that can generate a virtual viewpoint image with higher image quality. As described above, the virtual viewpoint determination unit 702 can determine a moving path of a virtual viewpoint that can generate a virtual viewpoint image with high image quality or has a small fluctuation in image quality based on the result of the evaluation process. According to the embodiment described above, for any evaluation position, an appropriate virtual viewpoint for viewing the evaluation position is automatically set, thereby improving convenience. The evaluation unit 303 can also be configured to display the evaluation position identified based on the evaluation information or shape data, etc., on the display device 304 as a candidate position designated by the user.

Furthermore, the above-mentioned embodiments can be implemented by combining any of the above forms. Furthermore, each of the functional configurations described in the above-mentioned embodiments may be configured to be connected as an independent processing device.

The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

200 Providing device 301 Viewpoint information acquiring unit 303 Evaluation unit

Claims (22)

an acquisition means for acquiring information indicating image capturing directions of a plurality of image capturing devices for acquiring a plurality of captured images used to generate a virtual viewpoint image;
an output means for outputting information on an evaluation result obtained by evaluating a direction of a virtual viewpoint used for generating the virtual viewpoint image, when the direction of the virtual viewpoint is set to a predetermined direction, based on an imaging direction indicated by the information acquired by the acquisition means ; and
A determination means for determining a direction of a virtual viewpoint based on an operation by a user;
a control means for causing a display means to display a screen notifying the user of the evaluation result output by the output means, based on the direction of the virtual viewpoint determined by the determination means and the predetermined direction;
A providing device comprising: The providing device according to claim 1, characterized in that the evaluation result is obtained by evaluating the specified direction based on the angle between the specified direction and the imaging direction indicated by the information acquired by the acquisition means. The providing device according to claim 2, characterized in that the evaluation result is obtained by evaluating the specified direction based on whether the angle formed is less than a specified threshold value. The provision device according to claim 2 or 3, characterized in that the information on the evaluation result is information that enables the predetermined direction and the angle corresponding to that direction to be recognized. the acquiring means further acquires information indicating resolutions of the plurality of image capture devices,
The providing device according to any one of claims 1 to 4, characterized in that the output means outputs information regarding the evaluation result obtained by evaluating the specified direction based on the resolution indicated by the information acquired by the acquisition means. The providing device according to claim 5, characterized in that the information on the evaluation result is information that enables the specified direction and the resolution of the imaging device specified based on the direction to be recognized. The acquiring means further acquires information regarding the plurality of captured images,
7. The providing device according to claim 2, wherein the output means evaluates the predetermined direction further based on information about the plurality of captured images acquired by the acquisition means. The providing device according to claim 7, characterized in that the information about the captured image includes at least one of the following: shaking of the imaging device, blur of the captured image, exposure, white balance, signal-to-noise ratio in the captured image, and information indicating that the subject captured by the imaging device is occluded. The provision device according to any one of claims 1 to 8, characterized in that the output means outputs information about the evaluation results to a display means. The provision device according to claim 9, characterized in that the output means outputs to the display means information for displaying a display screen on which multiple evaluation results corresponding to each of multiple predetermined directions can be recognized. The output means outputs information regarding the evaluation result to the control means;
11. The providing device according to claim 1, wherein the control means performs control according to the direction of the virtual viewpoint determined by the determination means, based on information related to the evaluation result. The provision device according to claim 11, characterized in that, when the direction of the virtual viewpoint determined by the determination means is close to the predetermined direction, the control means causes the display means to display a display screen for notifying the user of the evaluation result corresponding to the predetermined direction. The provision device according to claim 11 or 12, characterized in that, when the direction of the virtual viewpoint determined by the determination means is close to the predetermined direction, the control means causes the output means to output a sound for notifying the user of the evaluation result corresponding to the predetermined direction. The providing device according to any one of claims 11 to 13, characterized in that the control means controls an operation means operated by the user to change the direction of the virtual viewpoint determined by the determination means, when the direction of the virtual viewpoint determined by the determination means is approximate to the specified direction and an angle between the specified direction and the imaging direction indicated by the information acquired by the acquisition means is not less than a specified threshold. an acquisition means for acquiring information indicating image capturing directions of a plurality of image capturing devices for acquiring a plurality of captured images used to generate a virtual viewpoint image;
an output means for outputting information on an evaluation result obtained by evaluating a direction of a virtual viewpoint used for generating the virtual viewpoint image, when the direction of the virtual viewpoint is set to a predetermined direction, based on an imaging direction indicated by the information acquired by the acquisition means; and
and determining a direction of a virtual viewpoint based on information regarding the evaluation result. The provision device according to any one of claims 1 to 15, characterized in that the predetermined direction is a direction toward a specific position in an imaging area captured by the multiple imaging devices. The providing device according to claim 16, characterized in that the specific position is identified based on the position of the subject in the imaging area captured by the multiple imaging devices. The providing device according to claim 17, characterized in that the specific position is identified based on the distribution of multiple subjects in an imaging area captured by the multiple imaging devices. The provision device according to claim 16, characterized in that the specific position is identified based on the operation of an operating means by a user. an acquisition step of acquiring information indicating imaging directions of a plurality of imaging devices for acquiring a plurality of captured images used to generate a virtual viewpoint image;
an output step of outputting information on an evaluation result obtained by evaluating a direction of a virtual viewpoint used for generating the virtual viewpoint image, when the direction of the virtual viewpoint is set to a predetermined direction, based on an imaging direction indicated by the information acquired in the acquisition step ;
A determination step of determining a direction of a virtual viewpoint based on an operation by a user;
a control step of displaying on a display means a screen notifying the user of the evaluation result outputted in the output step, based on the direction of the virtual viewpoint determined in the determination step and the predetermined direction;
A method for controlling a providing device, comprising: an acquisition step of acquiring information indicating imaging directions of a plurality of imaging devices for acquiring a plurality of captured images used to generate a virtual viewpoint image;
an output step of outputting information on an evaluation result obtained by evaluating a direction of a virtual viewpoint used for generating the virtual viewpoint image, when the direction of the virtual viewpoint is set to a predetermined direction, based on an imaging direction indicated by the information acquired in the acquisition step ;
a determination means for determining a direction of a virtual viewpoint based on information regarding the evaluation result;
A method for controlling a providing device, comprising: A computer program for causing a computer to function as each of the means possessed by the providing device according to any one of claims 1 to 19.

Images