JP2021027548A - Information processing device, information processing method, and program - Google Patents

Abstract

To facilitate a task of generating a virtual viewpoint image having given quality.SOLUTION: An information processing device for estimating the quality of virtual viewpoint images generated from a plurality of pieces of image data captured by a plurality of imaging devices includes a quality estimation unit 260 that identifies an imaging device capable of capturing image data in a recording time zone in each of a plurality of recording time zones, and estimates the quality of a virtual viewpoint image in each of the recording time zones on the basis of the identified imaging device, and a display unit 270 that displays each of the recording time zones and the estimation result of the quality of the virtual viewpoint image in association with each other.SELECTED DRAWING: Figure 3

Description

The present invention relates to a system that generates a virtual viewpoint image.

Recently, a technique of installing a plurality of cameras at different positions to perform synchronous shooting from multiple viewpoints and generating a virtual viewpoint image using the multiple viewpoint images obtained by the shooting has attracted attention (see Patent Document 1).

Japanese Unexamined Patent Publication No. 2015-204512

In order to generate a high-quality virtual viewpoint image, image data obtained by shooting the same scene from various directions at the same time zone is required. On the other hand, when shooting with a plurality of cameras, it is conceivable that some cameras may not be able to shoot normally at a specific time zone due to the influence of a failure or an obstacle. However, if the user confirms the image data of each time zone taken by each camera and manually confirms in which time zone it is possible to generate a high-quality virtual viewpoint image, it is troublesome for the user. And it takes time to confirm.

The present invention has been made in view of such a problem, and an object of the present invention is to facilitate an operation for generating a virtual viewpoint image of a predetermined quality.

In one embodiment of the present invention, the information processing device for estimating the quality of the virtual viewpoint image generated from the plurality of image data captured by the plurality of imaging devices is used for the recording in each of the plurality of recording time zones. A specific means for identifying an imaging device capable of capturing image data in a time zone, an estimation means for estimating the quality of the virtual viewpoint image in each recording time zone based on the specified imaging device, and each of the above. It has a display means for displaying the recording time zone and the estimation result of the quality of the virtual viewpoint image in association with each other.

According to the present invention, it is possible to facilitate the work for generating a virtual viewpoint image of a predetermined quality.

It is a figure which shows the installation example of a plurality of cameras. It is a figure which shows the configuration example of a system. It is a figure explaining the recording list generation process. It is a flowchart of a recording list generation process. It is a figure which shows the example of the recording table. It is a flowchart of quality estimation processing. It is a figure which shows the example of the quality evaluation result. It is a figure which shows the example of the recording table which includes the evaluation value. It is a figure explaining the digest image generation processing. It is a flowchart of the digest image generation processing. It is a figure which shows an example of the digest image. It is a figure which shows the hardware configuration example.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The following embodiments do not limit the inventions described in the claims. Moreover, not all combinations of features described in the following embodiments are essential to the present invention.

(First Embodiment)
In the following, the configuration of this embodiment will be described by taking live distribution at a stadium (stadium), generation of replay images, and the like as examples.

FIG. 1 shows an installation example of a plurality of cameras (imaging devices) in the present embodiment. The plurality of cameras 100-1 to 100-10 are installed so as to surround the ground of the stadium and face the direction of one gazing point 130. From the images captured by each of the plurality of cameras 100-1 to 100-10, a virtual viewpoint image of a scene generated in the vicinity of the gazing point 130 is generated. In the present embodiment, when the virtual viewpoint image is generated, the image quality is evaluated in advance and the evaluation result is displayed. In FIG. 1, the gaze point 130 is set to one for the sake of simplification of the explanation, but usually, a plurality of gaze points are set and each camera faces one of the gaze points. .. Further, the number of cameras is not limited to 10 as shown in the figure, and any number of cameras may be set. In the present embodiment, the case where the optical axis of the camera intersects the gazing point will be mainly described, but the case where the camera is facing the gazing point is not limited to this, and the gazing range of the camera includes the gazing point. It suffices if it is.

FIG. 2 shows a configuration example of the system according to the present embodiment. As described above, the plurality of cameras 100-1 to 100-10 are installed so as to surround the ground. Further, each of the plurality of cameras 100-1 to 100-10 is connected to the camera adapters 210-1 to 210-10.

The internal configuration of the camera adapter 210-1 will be described as an example. The camera adapter 210-1 includes a camera control unit 211, a time management unit 212, a recording unit 213, and a communication unit 214. The camera control unit 211 controls the shutter speed, aperture, focal length, etc. of the camera 100-1. The time management unit 212 manages the time information. The recording unit 213 records the image data taken by the camera 100-1. The communication unit 214 connects the camera adapter 210-1 to the network, receives a control command, and transmits an image signal via the network. The camera adapters 210-1 to 210-10 can have the same internal configuration. Further, in the following, the branch number of the code of the camera adapter 210-1 to 210-10 is omitted, and the term is simply referred to as the camera adapter 210. Similarly, the cameras 100-1 to 100-10 also omit the branch number of the code and are also simply referred to as the camera 100.

The recording unit 213 receives a recording start / end instruction from the control unit 230, which will be described later, via the network, and records the image data from the camera 100. For example, the recording unit 213 records the time during which the play is being performed on the side with the gaze point of the ground, and does not record the time during which the play is being performed on the side without the gaze point of the ground. Can be controlled as such. The recorded information recorded by the recording unit 213 includes the following data. That is, it includes image data taken by the camera 100, recording time information from the time management unit 212, and shooting information such as a sequence number of the image data and a camera identifier (camera ID).

The hub 220 connects each device in the system with a network. That is, the hub 220 connects each camera adapter 210 and the information processing device 200 via a network. The control unit 230 controls the entire system. The virtual viewpoint image generation unit 240 generates and outputs a virtual viewpoint image 250 from image data taken by a plurality of cameras 100. The quality estimation unit 260 estimates and evaluates the image quality of the virtual viewpoint image 250 generated by the virtual viewpoint image generation unit 240. The display unit 270 displays the image quality estimated by the quality estimation unit 260. The synchronization unit 280 is connected to each camera adapter 210 via a network, and synchronizes the time managed by the time management unit 212 in each camera adapter 210 with high accuracy. Then, by sending the time information from the synchronized time management unit 212 to the camera 100, each camera 100 synchronizes the shooting timing with high accuracy to perform shooting.

The virtual viewpoint image generation unit 240 acquires image data from each camera adapter 210 via a network, and generates a virtual viewpoint image 250 according to the position and orientation of the virtual viewpoint. Specifically, the virtual viewpoint image generation unit 240 executes foreground background separation processing based on the acquired image data, generates a 3D model from the foreground, and displays the color of the 3D model when viewed from the virtual viewpoint. Render.

The system configuration is not limited to the above-described configuration, and the camera 100 may be directly connected to the control unit 230 or the virtual viewpoint image generation unit 240. Further, in the present embodiment, the network connection is described in a star type, but the present invention is not limited to this, and a daisy chain connection of a plurality of cameras may be used. Further, the recording unit 213 is not limited to each camera adapter 210, and may be provided in the virtual viewpoint image generation unit 240. Further, each functional unit of the information processing device 200 may be mounted on different devices instead of a single device.

FIG. 3 is a diagram illustrating a quality evaluation of a virtual viewpoint image by a recording list and a display method thereof according to the present embodiment. In the following, a process of estimating how much image quality is secured depending on the shooting conditions when generating a virtual viewpoint image is estimated by the quality estimation unit 260, and displaying a recording list including the estimation result on the display unit 270 will be described. To do.

In FIG. 3, each recording information is photographed by each camera 100-1 to 100-10 and recorded in each recording unit 213-1 to 213-10 in each camera adapter 210-1 to 210-10. As described above, the recording information includes shooting information such as recording time information, sequence number, and camera identifier (camera ID) together with image data. The quality estimation unit 260 collects recorded information from each camera adapter 210-1 to 210-10 via a network.

The quality estimation unit 260 includes a list generation processing unit 310 and a quality estimation processing unit 320. The list generation processing unit 310 generates recording time zone information of each camera from the recording time information collected from each camera adapter 210-1 to 210-10, and the sequence number of the image data taken by each camera in each recording time zone. Generate a recording list showing. In the present embodiment, each camera performs synchronous shooting in a plurality of time zones according to a recording start / end instruction from the control unit 230, and each recording information in the plurality of time zones is recorded in the recording unit of each camera adapter 210. It is recorded in 213. That is, each camera basically does not always shoot. Further, the sequence number of the image data may be sequentially assigned to the image data recorded in each of the plurality of time zones for each camera. The quality estimation processing unit 320 estimates the image quality of the virtual viewpoint image that can be generated in each recording time zone based on the recording list generated by the list generation processing unit 310. The display unit 270 displays the recording list 330 including the image quality of the virtual viewpoint image estimated by the quality estimation unit 260.

FIG. 4 shows a flowchart of the recording list generation process in the present embodiment. In the following, a processing procedure in which the list generation processing unit 310 generates a recording list representing the correspondence between the camera identifier (camera ID) and the recording time zone information will be described. The series of processes shown in the flowchart is performed by the CPU of the information processing device 200 expanding the control program stored in the ROM or the HDD into the RAM and executing it. Alternatively, some or all the functions of the steps in the flowchart may be realized by hardware such as an ASIC or an electronic circuit. Further, the symbol "S" in the description of the flowchart means a "step" in the flowchart. The same applies to other flowcharts.

The recording list generation process is performed on the image data taken by the cameras 100 facing the same gazing point.

In S401, the list generation processing unit 310 clears and initializes the recording list, and prepares to generate a new recording list.

In S402, the list generation processing unit 310 acquires the recording start time and the recording start time from the recording time information recorded in each camera adapter 210-1 to 210-10, and generates the recording time zone information of each camera. If all the camera adapters 210-1 to 210-10 are operating normally, the same recording time zone information is generated for all the cameras. On the other hand, for example, if one or more camera adapters do not operate normally and there is a time zone in which image data is not recorded in the camera adapter, the recording time information recorded in the camera adapter is different. Recording time zone information is generated.

In S403, the list generation processing unit 310 registers the recording time zone information of each generated camera in the recording list in order for each camera. Specifically, in S4031, the list generation processing unit 310 determines whether or not the generated recording time zone information is registered in the recording list. If the recording time zone information is not registered, the process proceeds to S4032, and the recording time zone information is registered in the recording list. That is, since all the recording time zone information is not registered in the recording list for the camera to be processed first, the process proceeds from S4031 to S4032, and all the recording time zone information of the camera is registered in the recording list. On the other hand, for the second and subsequent cameras, the recording time zone information already registered in the recording list is not newly registered (YES in S4031), and the recording time zone information not registered in the recording list is added to the recording list. It is newly registered (NO of S4031). That is, if there is a time zone in the camera to be processed that could not be taken by the camera processed before that, the time zone is registered in the recording list. By repeating this process for all cameras, a recording list including all recording time zones can be generated.

In S404, the list generation processing unit 310 acquires the number T of the recording time zones registered in the recording list.

In S405, the list generation processing unit 310 acquires the total number M of the cameras registered in the recording list. The total number of cameras M can be obtained from, for example, the number of camera identifiers included in the shooting information, the number of connected camera adapters 210, and the like. In this embodiment, the total number M is 10.

In S406, the list generation processing unit 310 generates a recording list of T (row) × M (column) from the number T of the recording time zones acquired in S404 and the total number M of the cameras acquired in S405.

In S407, the list generation processing unit 310 registers the sequence number of the image data taken by each camera for each recording time zone of the recording list (S4071).

As described above, the list generation processing unit 310 generates a recording list in which the recording time zone of each camera and the sequence number of the image data of each camera are registered.

FIG. 5 shows an example of a recording list according to the present embodiment. The vertical axis represents the recording time zone, and the horizontal axis represents the camera number (camera identifier). In the example of FIG. 5, the number of recording time zones is 9 (T1 to T9), and the number of cameras is 10 (cam01 to cam10). In the recording list, the sequence number of the image data for each camera is registered for each recording time zone. The column of the recording time zone in which the image data did not exist remains blank (x mark).

FIG. 6 shows a flowchart of the quality estimation process in the present embodiment. In the following, the image quality estimation process when the quality estimation processing unit 320 generates a virtual viewpoint image from the image data of each camera in each recording time zone by using the recording list generated as described above will be described. To do. Here, it is estimated that the higher the number of cameras that shoot the same scene in the same time zone, the higher the quality of the virtual viewpoint image because the image data taken from various angles can be obtained. Also, even if the number of cameras that shot the same scene in the same time zone is the same, the farther the cameras are, the more image data taken from various angles can be obtained, so the quality of the virtual viewpoint image will be higher. presume.

In S601, the quality estimation processing unit 320 refers to the recording list and acquires the number N (T) of cameras recorded in the recording time zone T. Here, the recording time zone T refers to, for example, one of the recording time zones T1 to T9 shown in the recording list of FIG. For example, the quality estimation processing unit 320 selects in order from the recording time zone T1 as the processing target, and obtains the number N (T) of cameras in which the sequence number of the image data is registered.

In S602, the quality estimation processing unit 320 obtains a difference value of the camera numbers as the inter-camera distance of the cameras recorded in the recording time zone T. The camera numbers are, for example, the camera identifiers cam01 to cam10 shown in the recording list of FIG. 5 as numbers 1 to 10, respectively. Since the camera numbers are assigned in the order of arrangement with respect to a predetermined camera, they indicate the relative positional relationship between the cameras. Assuming that the camera numbers are {C1, C2, ... Cn}, the inter-camera distance {D1, D2, ... Dn} can be obtained by Dk = C (k + 1) -Ck. k takes a value from 1 to n. n represents the number of cameras used for recording in the recording time zone T. The inter-camera distance Dn of the camera number Cn is set to M + C1-Cn by using M, which is the total number of cameras.

In S603, the quality estimation processing unit 320 obtains the variance value D (T) of the inter-camera distance. The variance value D (T) is a virtual viewpoint generated depending on whether the number of cameras that could not be recorded in the recording time zone T is the same, whether they are in one place or scattered positions. Since the image quality of the image is different, it is used to take this into consideration. When the cameras that could not be recorded are dispersed, the dispersion value D (T) becomes small because a large value does not appear as the distance between the cameras. On the other hand, the dispersion value D (T) becomes large because a large value appears as the distance between the cameras when the cameras that could not be recorded are gathered in one place.

In S604, the quality estimation processing unit 320 obtains a quality estimation value of the virtual viewpoint image generated from the image data in the recording time zone T. Here, the quality estimate is N (T) −α × D (T) (α is a constant). The larger the number of recorded cameras, the larger the N (T) value. Further, even if the number of recorded cameras is the same, the larger the number of cameras that could not be recorded is gathered in one place, the larger the α × D (T) value, and as a result, the smaller the quality estimation value.

In S605, the quality estimation processing unit 320 evaluates the quality estimation value of the virtual viewpoint image obtained above in five stages by comparing with a predetermined threshold value, and sets this as the quality evaluation value of the virtual viewpoint image. That is, the evaluation value is derived based on the quality estimation value. As a value representing the degree of dispersion of the arrangement of the cameras, a value other than the above-mentioned dispersion value D (T) of the distance between cameras may be used. For example, even if a value corresponding to the angle difference in the direction of each camera viewed from a specific position in the imaging region is used as a value indicating the degree of dispersion of the camera arrangement, the quality of the virtual viewpoint image can be estimated in the same manner. Can be done.

FIG. 7 shows an example of the quality evaluation value of the virtual viewpoint image in this embodiment. Here, the quality evaluation value is indicated by a five-stage evaluation value of A to E. As shown in FIG. 7A, when all the cameras are recording, the quality evaluation value is A. As the number of cameras that cannot record increases, or the dispersion value of cameras that cannot record increases (that is, they are gathered in one place), the quality estimation value decreases, and the quality evaluation value also becomes B. It gradually decreases from to E (see FIGS. 7 (b) to 7 (e)).

As described above, in the present embodiment, by performing the above-mentioned quality evaluation processing for each recording time zone, the quality evaluation of the virtual viewpoint image is performed in all the recording time zones of the recording list.

FIG. 8 shows an example of the display result displayed on the display unit 270. As shown in the figure, the display unit 270 acquires the recording list generated by the list generation processing unit 310 and the quality evaluation value of the virtual viewpoint image estimated and evaluated by the quality estimation processing unit 320, and displays the list. As a result, the user can intuitively judge the image quality when the virtual viewpoint image is generated in each recording time zone.

In this embodiment, the quality evaluation value is shown in five stages of A to E, but the present invention is not limited to this. For example, the quality evaluation value may be represented by the color of the border of the table or the background color of the table. Further, as shown in FIG. 8, the sequence number of the image data of each camera may not be shown, and the recording time zone and the quality evaluation value may be simply displayed. That is, it is sufficient that each recording time zone and the quality estimation result can be displayed in association with each other.

Further, in the present embodiment, the quality evaluation value is obtained from the quality estimated value and displayed, but the quality estimated value may be displayed as it is.

Further, the list generation processing unit 310 collects the recording time zone information from the camera adapter 210 and generates a recording list showing the correspondence between the camera identifier and the recording time zone information, but the present invention is not limited to this. For example, when the camera adapter 210 analyzes the image data during shooting and detects an obstacle such as vibration or a bird, it may record information that the image data in that time zone cannot be used to generate a virtual viewpoint image. .. Then, the quality estimation processing unit 320 may obtain a quality evaluation value based on the information. That is, when an obstacle such as vibration or a bird is detected from the image data, the camera may evaluate it as if the image data could not be captured during that time period.

As described above, according to the present embodiment, it is possible to easily confirm the image data for generating the virtual viewpoint image.
(Second Embodiment)
In the present embodiment, an example of displaying the quality evaluation result of the virtual viewpoint image in a specific recording time zone together with the image data taken by a specific camera among the plurality of cameras will be described.

FIG. 9 is a diagram illustrating a method of displaying a quality evaluation result of a virtual viewpoint image using a digest image in the present embodiment. In the present embodiment, the display unit 270 displays the border of the digest image of the specific camera with a color corresponding to the quality evaluation value (display result 910). The digest image is an image including still images taken out at predetermined time intervals from image data in a specific recording time zone. For example, the still images taken out at predetermined time intervals may be reduced, and the reduced still images may be imitated on a film and connected.

FIG. 10 shows a flowchart of the display process according to the present embodiment. In the present embodiment, the processing by the quality estimation unit 260 is the same as that in the first embodiment, but since the operation of the display unit 270 is different, this will be described below.

In S1010, the display unit 270 acquires the image data of the recording time zones S1 to S2 from the recording unit 213 in the camera adapter 210 selected by the user. Here, the recording time zones S1 to S2 correspond to the recording time zones T3 to T5 in the recording list of FIG. 5 generated by the list generation processing unit 310.

In S1020, the display unit 270 extracts a still image from the acquired image data in the recording time zones S1 to S2 at predetermined time intervals and reduces the image, and connects the reduced still images to generate a digest image imitated on a film.

In S1030, the display unit 270 performs the following processing (S1031 and S1032) on each still image included in the generated digest image.

In S1031, the display unit 270 acquires the quality evaluation value of the corresponding recording time zone from the quality estimation processing unit 320 for each still image included in the generated digest image. Then, in S1032, the display unit 270 attaches a color corresponding to the acquired quality evaluation value to the frame of each still image. As a result, as shown in FIG. 11, it is possible to display a digest image in which a color corresponding to the quality evaluation value is added to the frame of each still image.

As described above, by displaying the digest image according to the present embodiment, the relationship between the captured scene and the image quality when the virtual viewpoint image is generated can be understood at a glance.

FIG. 12 shows a hardware configuration example of the information processing apparatus 200 according to the present embodiment. The hardware configuration of the camera adapter 210 is the same.

The information processing device 200 includes a CPU 1211, a ROM 1212, a RAM 1213, an auxiliary storage device 1214, a display device 1215, an operation unit 1216, a communication I / F 1217, and a bus 1218.

The CPU 1211 realizes each function by controlling the entire information processing apparatus 200 by using computer programs and data stored in the ROM 1212 and the RAM 1213. The information processing device 200 may have one or more dedicated hardware different from the CPU 1211, and the dedicated hardware may execute at least a part of the processing by the CPU 1211. Examples of dedicated hardware include ASICs (application specific integrated circuits), FPGAs (field programmable gate arrays), and DSPs (digital signal processors). The ROM 1212 stores programs and the like that do not require changes. The RAM 1213 temporarily stores programs and data supplied from the auxiliary storage device 1214, data supplied from the outside via the communication I / F 1217, and the like. The auxiliary storage device 1214 is composed of, for example, a hard disk drive or the like, and stores various data such as image data and audio data.

The display device 1215 is composed of, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) for the user to operate the information processing device 200. The operation unit 1216 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, and the like, and inputs various instructions to the CPU 1211 in response to an operation by the user. The CPU 1211 operates as a display control unit that controls the display device 1215 and an operation control unit that controls the operation unit 1216.

The communication I / F 1217 is used for communication with an external device of the information processing device 200. For example, when the information processing device 200 is connected to an external device by wire, a communication cable is connected to the communication I / F 1217. When the information processing device 200 has a function of wirelessly communicating with an external device, the communication I / F 1217 includes an antenna. The bus 1218 connects each part of the information processing device 200 to transmit information.

In the present embodiment, it is assumed that the display device 1215 and the operation unit 1216 exist inside the information processing device 200, but at least one of the display device 1215 and the operation unit 1216 is a separate device outside the information processing device 200. It may exist.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100-1 to 100-10 Camera 210-1 to 210-10 Camera adapter 211 Camera control unit 212 Time management unit 213 Recording unit 214 Communication unit 260 Quality estimation unit 270 Display unit 310 List generation processing unit 320 Quality estimation processing unit

Claims (10)

An information processing device for estimating the quality of a virtual viewpoint image generated from a plurality of image data captured by a plurality of image pickup devices.
Specific means for identifying an imaging device capable of capturing image data in the recording time zone in each of the plurality of recording time zones, and
An estimation means for estimating the quality of the virtual viewpoint image in each recording time zone based on the specified imaging device, and
An information processing apparatus comprising: a display means for displaying each recording time zone and an estimation result of quality of the virtual viewpoint image in association with each other. The information processing device according to claim 1, wherein the estimation means estimates the quality of the virtual viewpoint image in each recording time zone based on the number of the specified imaging devices. The information processing device according to claim 2, wherein the estimation means estimates the quality of the virtual viewpoint image in each recording time zone based on the positional relationship of the specified imaging device. The display means displays a list with the recording time zone as one axis and the identifier of the imaging device as the other axis, and the image data captured by each imaging device in each recording time zone and the virtual viewpoint image. The information processing apparatus according to any one of claims 1 to 3, wherein the information processing apparatus is related to the estimation result of the quality of the above. The estimation means compares the estimation result of the quality of the virtual viewpoint image with a predetermined threshold value, and derives a grade evaluation value.
The information processing device according to any one of claims 1 to 4, wherein the display means represents the derived grade evaluation value. When the vibration is detected by analyzing the image data, it is assumed that the estimation means cannot capture the image data during the recording time zone in which the image data in which the vibration is detected is captured. The information processing apparatus according to any one of claims 1 to 5, wherein the quality of the virtual viewpoint image is estimated. When an obstacle is detected by analyzing the image data, the estimation means cannot capture the image data during the recording time zone in which the image data in which the obstacle is detected is captured. The information processing apparatus according to any one of claims 1 to 6, wherein the quality of the virtual viewpoint image is estimated assuming that the image is of a certain value. The display means generates a digest image including still images taken out from specific image data at predetermined time intervals, and displays the digest image so as to represent the estimation result of the quality of the recording time zone corresponding to each still image. The information processing apparatus according to any one of claims 1 to 7, wherein the information processing apparatus is displayed. A program for causing a computer to function as the information processing device according to any one of claims 1 to 8. It is an information processing method for estimating the quality of a virtual viewpoint image generated from a plurality of image data captured by a plurality of imaging devices.
In each of the plurality of recording time zones, a specific step for identifying an imaging device that could not capture image data in the recording time zone, and a specific step.
An estimation step for estimating the quality of the virtual viewpoint image in each recording time zone based on the identified imaging device, and
An information processing method including a display step for displaying each recording time zone in association with an estimation result of the quality of the virtual viewpoint image.

Images