JP7458713B2 - Image processing device, image processing system, control method, and program - Google Patents

Description

The present invention relates to a technique for generating virtual viewpoint images.

In recent years, technology has attracted attention in which multiple imaging devices are installed in different positions to take synchronized images from multiple directions, and the multi-viewpoint images obtained from the imaging devices are used to generate a virtual viewpoint image based on an arbitrary viewpoint. ing.

Patent Document 1 describes an image generating device that generates a virtual viewpoint image based on captured images captured by multiple imaging devices, and data such as a foreground texture image, a background texture image, and a three-dimensional model that are generated based on the captured images. Hereinafter, data for generating a virtual viewpoint image (including at least any one of captured images captured by multiple imaging devices, a foreground texture image, a background texture image, and a three-dimensional model that are generated based on the captured images) will be referred to as material data.

Patent No. 06429829

Consider an image processing system in which an image processing device generates material data, outputs it to an image generation device, and generates a virtual viewpoint image based on the material data acquired by the image generation device. At this time, if the image generation device cannot recognize what kind of processing has been performed to generate the acquired material data, a problem may arise in the generation process or quality of the virtual viewpoint image. For example, assume that the image generation device has a function of correcting acquired material data in order to improve the quality of the virtual viewpoint image to be generated. However, even if the image processing device has already performed similar corrections in the process of generating material data, the image generation device cannot simply acquire the material data and the correction processing has already been performed on the material data. I can't recognize what happened. As a result, by performing the same correction on the image generation device side, there is a risk that the correction processing will be duplicated, resulting in an increase in processing time, etc. Further, even if the image processing device has not performed correction on material data that requires correction, the image generation device cannot recognize that the correction processing has not been performed. As a result, the image generation device may not be able to perform appropriate correction on the material data, and the quality of the virtual viewpoint image may deteriorate.

The present invention has been made in view of the above problems. The purpose is to make it possible to easily refer to what kind of processing was performed on the material data when the material data was generated.

An image processing device according to the present invention is an image processing device that generates material data for generating a virtual viewpoint image based on a plurality of captured images captured by a plurality of imaging devices, the image processing device generating the material data for generating the material data. It has a data generation means, and an output means for correlating and outputting the material data generated by the material data generation means and the generation information related to the generation process of the material data, and the generation information includes the material data generated by the material data generation means. The output means includes information indicating whether a process for generating or a process for correcting the material data has been performed on the material data , and the output means generates a virtual viewpoint image based on the material data and the generation information. The image generation device generates the material data without performing additional processing on the material data, when a process of generating the material data or a process of correcting the material data is performed on the material data. or when the process of correcting the material data is not performed on the material data, outputting the material data and the generated information in association with each other to the image generation device that performs additional processing on the material data. It is characterized by

According to the present invention, it becomes possible to easily refer to what kind of processing has been performed on the material data when the material data is generated.

Diagram for explaining the functional configuration of part of the image processing system 100 Diagram for explaining an example of the structure of a material data file Diagram for explaining an example of the structure of a material data file Diagram for explaining an example of the structure of a material data file Diagram for explaining an example of the structure of a material data file FIG. 1 is a diagram for explaining an example of the configuration of a material data file; FIG. 1 is a diagram for explaining an example of the configuration of a material data file; Flowchart for explaining processing executed by a part of the image processing system 100 Diagram for explaining the functional configuration of an image generation device 120 included in the image processing system 100 Flowchart for explaining processing executed by image generation device 120 Diagram for explaining an example of information displayed on the display unit 904 A diagram for explaining the hardware configuration of an image processing device 110 and an image generation device 120 included in the image processing system 100. A diagram for explaining an example of the arrangement of a plurality of imaging devices 101

This embodiment will be described below with reference to the drawings.

(First embodiment)
The hardware configuration of the image processing device 110 and the image generating device 120 of the image processing system 100 in this embodiment will be described with reference to Fig. 12. The image generating device 110 has a CPU (Central Processing Unit) 1211, a ROM (Read Only Memory) 1212, a RAM (Random Access Memory) 1213, an auxiliary storage device 1214, a display unit 1215, an operation unit 1216, a communication I/F 1217, and a bus 1218.

The CPU 1211 controls the entire image processing apparatus 110 using computer programs and data stored in the ROM 1212 or RAM 1213. Note that the image processing device 110 may include one or more dedicated hardware different from the CPU 1211, and the dedicated hardware may execute at least part of the processing by the CPU 1211. Examples of specialized hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gateways), and DSPs (Digital Signal Processors). The ROM 1212 stores programs that do not require modification. The RAM 1213 temporarily stores programs and data supplied from the auxiliary storage device 1214, data supplied from the outside via the communication I/F 1217, and the like. The auxiliary storage device 1214 is configured with, for example, a hard disk drive, and stores various data such as image data and audio data.

The display unit 1215 is configured with, for example, a liquid crystal display or an LED, and displays a GUI (graphical user interface) or the like for the user to operate the image processing device 110. The operation unit 1216 includes, for example, a keyboard, a mouse, a joystick, a touch panel, and the like, and inputs various instructions to the CPU 1211 in response to user operations. The CPU 1211 operates as a display control unit that controls the display unit 1215 and an operation control unit that controls the operation unit 1216. The communication I/F 1217 is used for communication with an external device of the image processing device 110. For example, when the image processing device 110 is connected to an external device by wire, a communication cable is connected to the communication I/F 1217. When the image processing device 110 has a function of wirelessly communicating with an external device, the communication I/F 1217 includes an antenna. A bus 1218 connects each part of the image processing device 110 and transmits information.

In this embodiment, the display section 1215 and the operation section 1216 are assumed to exist inside the image processing apparatus 110, but at least one of the display section 1215 and the operation section 1216 is installed outside the image processing apparatus 110 as a separate device. May exist.

The above is a description of the hardware configuration of the image processing device 110. The CPU 1221, ROM 1222, RAM 1223, auxiliary storage device 1224, display unit 1225, operation unit 1226, communication I/F 1227, and bus 1228 in the image generation device 120 are also used for the CPU 1211, ROM 1212, RAM 1213, and auxiliary memory in the image processing device 110, respectively. It has the same functions as the storage device 1214, display section 1215, operation section 1216, communication I/F 1217, and bus 1218.

FIG. 1 is a diagram for explaining the functional configuration of a part of the image processing system 100. Each processing section will be explained below.

The imaging device 101 images an imaging area and transmits the acquired image to the material data generation unit 102. The imaging area is, for example, a stadium where a competition such as soccer or karate is held, or a stage where a concert or play is held. Further, the image processing system 100 includes a plurality of imaging devices 101. FIG. 13 is an example in which a plurality of imaging devices 101 are arranged. The plurality of imaging devices 101 are arranged at different positions surrounding the imaging area, and perform imaging in synchronization with each other. Further, the imaging device 101 is connected to an image processing device 110 via a network or the like. Note that the plurality of imaging devices 101 do not need to be installed all around the imaging area, and may be installed only in a part of the imaging area depending on restrictions on the installation location. Further, the number of imaging devices 101 is not limited to the example shown in FIG. 13; for example, when the imaging area is a soccer stadium, about 30 imaging devices 101 may be installed around the stadium. . Furthermore, imaging devices with different functions, such as a telephoto camera and a wide-angle camera, may be installed.

The material data generation unit 102 acquires captured images from the plurality of imaging devices 101, and generates material data for generating a virtual viewpoint image based on the captured images. Here, the virtual viewpoint image is an image representing the view from an arbitrary viewpoint (virtual viewpoint) in the imaging region. The virtual viewpoint image in this embodiment is also called a free viewpoint video, but is not limited to an image corresponding to a virtual viewpoint freely (arbitrarily) designated by the user, and is, for example, a virtual viewpoint image selected by the user from a plurality of candidates. Images corresponding to the viewpoint are also included in the virtual viewpoint image. Further, material data is data for generating a virtual viewpoint image, and refers to a captured image captured by the imaging device 101 and data generated based on the captured image. The material data generated based on the captured image is, for example, data such as three-dimensional model data of the subject in the captured area and texture data for coloring the three-dimensional model. The type of material data is not limited to the above data. For example, camera parameters of the imaging device 101 that captured the captured image may be included in the material data. The camera parameters are, for example, parameters indicating the position, orientation, focal length, etc. of the imaging device 101. As described above, the material data is data necessary for generating a virtual viewpoint image, and its type is not limited. The method of generating material data will be described below.

The material data generating unit 102 generates a foreground image by extracting a foreground region from the multiple captured images. The foreground is a subject whose position in the captured image changes (for example, a ball in a stadium, a player, etc.). Subjects other than the foreground in the captured image are used as the background. For example, the foreground can be extracted by taking the difference between a captured image of an image captured in a state where there is no subject corresponding to the foreground and a captured image of an image captured in a state where there is a subject corresponding to the foreground. The method of extracting the foreground is not limited to the above example. The material data generating unit 102 generates a three-dimensional model of the foreground using the extracted foreground image. For example, a method such as a volume intersection method, which is a publicly known technology, is used to generate the three-dimensional model. However, a method other than the volume intersection method may be used to generate the three-dimensional model. The material data generating unit 102 transmits the multiple captured images obtained, and material data such as a foreground image and a three-dimensional model generated based on the captured images to the storage unit 104.

Note that although this embodiment describes an example in which a three-dimensional model generation method is used as a method for configuring the foreground in a virtual viewpoint image, the method for configuring the foreground is not limited to this. For example, a morphing processing method can be used to configure the foreground in a virtual viewpoint image by interpolating and composing a subject viewed from a viewpoint located between adjacent imaging devices 101 according to the positional relationship of the adjacent captured images 101. may be used. For example, a billboarding method may be used in which the foreground in the virtual viewpoint image is configured by projectively transforming the foreground image onto a three-dimensional space. When using the morphing processing method, billboarding method, or other methods, the type of material data required is different from the three-dimensional model generation method. In this case as well, the material data generation unit 102 generates necessary material data and transmits it to the storage unit 104.

In addition, the material data generation unit 102 performs correction processing to improve the quality of the virtual viewpoint image during the material data generation process. Below, an example of the correction processing performed by the material data generation unit 102 and the method of each correction processing are described.

The distortion correction process is a correction process that corrects a captured image that is distorted due to the optical characteristics of a lens included in the imaging apparatus 101. The material data generation unit 102 acquires in advance parameters indicating the optical characteristics of the lens included in the imaging device 101, and corrects the distortion by performing arithmetic processing on the distorted captured image according to the parameters. do.

Noise reduction processing is a correction process that reduces noise that occurs in a captured image. Noise in a captured image is likely to occur, for example, when the imaging area is dark and the sensitivity of the imaging sensor of the imaging device 101 is increased to capture the image. The material data generation unit 102 reduces noise by performing filtering processing using a bilateral filter or the like on the captured image in which noise has occurred.

The calibration process is a correction process that corrects camera parameters using image processing based on captured images captured by the plurality of imaging devices 101. Since the virtual viewpoint image is generated based on a plurality of captured images, the higher the accuracy of the camera parameters indicating the positional relationship of the plurality of imaging devices 101, the higher the accuracy of the generated virtual viewpoint image. The material data generation unit 102 performs a feature point extraction process on a plurality of captured images acquired from a plurality of imaging devices 101 to specify a common feature point in each captured image. The designated common feature point is a common coordinate in world coordinates. The material data generation unit 102 uses common coordinates to grasp the positional relationship of the plurality of imaging devices 101 that have captured the plurality of captured images, and corrects camera parameters based on the positional relationship. The calibration process described above is performed when acquiring the positional relationship of the imaging device 101 at the start of imaging of the imaging area. Further, this is performed for the purpose of correcting camera parameters when the position or orientation of the imaging device 101 changes due to vibration or the like during operation of the image processing system 100.

The vibration correction process is a process for correcting positional deviations between captured images caused by vibrations of the imaging device 101. When the imaging device 101 vibrates during imaging, the position or orientation of the imaging device 101 may shift. As a result, the imaging position shifts, so that when the imaging device 101 continuously captures images, the captured images are shifted in position. If a positional shift occurs between captured images, the quality of material data generated based on the captured images may deteriorate, or when a video of a virtual viewpoint image is generated using the captured images, the video image may vibrate. There is a fear. For example, the material data generation unit 102 obtains the amount of vibration based on a gyro sensor included in the imaging device 101, or refers to continuously captured images, and calculates the amount of vibration based on the deviation of feature points between the captured images. It is possible to calculate. The material data generation unit 102 corrects positional deviations between the captured images by performing image processing on the captured images based on the acquired or calculated amount of vibration. Although the above is an example of a process of correcting a captured image in which a positional shift has occurred, the imaging apparatus 101 may have a function to prevent positional shift between captured images. An example of a function for preventing positional deviation between captured images is an optical image stabilization function. If the imaging device 101 has a function to prevent positional deviation between captured images, the imaging device 101 notifies the material data generation unit 102 that vibration correction processing has been performed during imaging.

As explained above, the material data generation unit 102 performs material data correction processing in the material data generation process. In the above description, correction processing of a captured image and camera parameters was described as an example, but the material data corrected by the material data generation unit 102 is not limited to this. For example, when the material data generation unit 102 generates a foreground image, an area that should normally be separated as a background may be included in the foreground image as noise. At this time, the material data generation 102 can perform correction processing to remove noise other than the foreground. Note that the types of correction processing and correction methods described above are merely examples, and the material data generation unit 102 may perform other correction processing or use other correction methods. Other correction processing includes, for example, color correction processing for correcting color information. Furthermore, since the correction process is not an essential process, the material data generation unit 102 may generate material data without performing the correction process.

The generation information acquisition unit 103 acquires from the material data generation unit 102 information related to generation processing when the material data generation unit 102 generates material data. Information related to the generation process (hereinafter referred to as generation information) is information that can specify the process executed by the image processing apparatus 110. The generation information includes, for example, information indicating the type of processing performed by the material data generation unit 102 in the material data generation processing, information indicating whether the processing has been performed, and the like. The type of processing performed by the material data generation unit 102 is information indicating a foreground extraction process, a three-dimensional model generation process, a correction process such as a distortion correction process or a noise reduction process, or a method used for the above process. . The above processing is image processing performed to finally generate a virtual viewpoint image, and the image generation device 120 described below may be configured to perform the above processing. However, it is necessary to ensure consistency between the image processing device 110 and the image generation device 120 as to what processing has or has not been performed. By acquiring the generation information, the image generation device 120 can refer to the generation information and perform processing that should be performed when generating the virtual viewpoint image. The generation information acquisition unit 103 transmits the acquired generation information to the storage unit 104.

The storage unit 104 stores the material data and generation information acquired from the material data generation unit 102 and the generation information acquisition unit 103. It is assumed that the storage unit 104 is a storage device. The storage unit 104 includes, for example, a Hard Disk Drive (HDD), a Solid State Drive (SSD), or a flash memory. The storage unit 104 may be a single device or a collection of multiple devices. Furthermore, the storage unit 104 may be configured of multiple types of devices, or may be a device on a cloud virtualized on a network.

The association unit 105 performs a process of associating the material data stored in the storage unit 104 with information related to the generation process of the material data (generation information), and outputs the associated data to the storage unit 104. As an example of association, a method of generating a material data file in which generation information is added as metadata to material data will be described.

2 is a diagram for explaining an example of the configuration of a material data file generated by the association unit 105. The material data 201 includes material data such as a foreground image and a three-dimensional model generated by the material data generation unit 102, and camera parameters. The metadata 202 attached to the material data 201 includes generation information 203 related to the generation process of the material data included in the material data 201. The generation information 203 includes processing information 204 indicating the type of processing performed when generating the material data, and parameters 205 related to the processing. The processing information 204 includes information indicating the generation method or correction process of the material data. The parameters 205 include processing parameters used when performing the corresponding processing. In addition, if a processing is not performed, the parameters 205 corresponding to the processing information 204 indicating the processing include parameters indicating that the processing was not performed. In the example shown in FIG. 2, it can be seen that processing A and processing C were performed by the material data generation unit 102, and processing B was not performed. The parameters included in the parameters 205 may be codes associated with a specific state, such as being "0" when no processing was performed.

Another example of the structure of the material data file generated by the association unit 105 will be described using FIG. 3. The material data 301 includes material data. Further, the metadata 302 includes generation information 303 related to generation processing of the material data included in the material data 301. The generation information 303 includes processing information 304 indicating the content of the process performed when generating the material data, and pointer information 305 indicating the location where the parameter 306 related to the process is stored. A parameter used in the corresponding process or a parameter indicating that the process was not performed is stored at the location indicated by the pointer information 305. The configuration shown in FIG. 3 has the advantage that the code representing the generated information 303 has a fixed length, making it easier to design the structure of material data.

Another example of the structure of the material data file generated by the association unit 105 will be explained using FIG. 4. The functions of the material data 401, metadata 402, generation information 403, and processing information 404 and parameters 405 included in the generation information 403 in FIG. This is similar to the information 204 and parameters 205. In addition to processing information 404 and parameters 405, generation information 403 includes standard information 406. Standard information 406 is information that supplements the content of the process indicated by process information 404. For example, since there are various algorithms for a single process, such as noise reduction processing, adding information indicating which algorithm was used to perform the correction process makes it possible to understand the details of the generation process in more detail. become. It is assumed that standards are defined for commonly used processes. Therefore, when the process indicated by the process information 404 is a process based on a standard, the standard information 406 includes information indicating the standard. Furthermore, if the process indicated by the process information 404 is a system-specific process that is not defined by a general standard, the standard information 406 includes information indicating that the process is based on a method for which no standard is defined. is included. At this time, if a system-specific regulation is defined, the standard information 406 may include information indicating the system-specific regulation. Note that the parameter 405 may have the same configuration as the pointer information 305 in FIG. 3.

Using FIG. 5, another example of the configuration of the material data file generated by the association unit 105 will be described. Hereinafter, the term frame will be used as a unit indicating a virtual viewpoint image. That is, one virtual viewpoint image corresponds to one frame. By combining a plurality of frames, a video of a virtual viewpoint image can also be generated. Since material data is often used in a lump for each data for generating one frame of a virtual viewpoint image, the material data and the generation information are also associated in units of frames, making it easier to handle the material data. FIG. 5 shows an example of the configuration of a material data file when material data is generated according to a frame rate of 60 fps (frames per second). fps is a unit of frame rate, which indicates how many frames of a captured image are captured per second. Therefore, one frame of material data is material data corresponding to 1/60 seconds. The material data 501 includes one frame of material data. The metadata 502 includes generation information 503 related to the generation process of the material data included in the material data 501. The configuration of the generation information 503 is the same as that of the generation information 203 in FIG. 2, the generation information 303 in FIG. 3, or the generation information 403 in FIG. 4. The metadata 502 also includes time information 504. The time information 504 is information indicating which frame of the time the material data included in the material data 501 corresponds to. In the example of FIG. 5, T=1/60 seconds, and as shown in the imaging timing 505, the captured images are captured at the timings of T=1, T=2, . . . Since the material data included in the material data 501 corresponds to T=0, the time information 504 is information indicating "T=0". As an example of the format of the time information 504, there is a time code standardized by the Society of Motion Picture and Television Engineers (SMPTE). However, the format is not limited to this. The configuration of the material data files corresponding to T=1, T=2, . . . is also the same. The configuration shown in FIG. 5 has the advantage that it is easy to refer to the generation information associated with each piece of material data, even when using material data corresponding to only a specific frame, or when using material data corresponding to multiple consecutive frames together.

Another example of the structure of the material data file generated by the association unit 105 will be explained using FIG. 6. In the example of FIG. 6, it is assumed that the material data of the frames corresponding to the shooting timings T=0 to T=N-1 are generated by the same generation process. At this time, the material data corresponding to T=0 to T=N-1 are associated with the same generation information. Therefore, it is possible to create a configuration in which frames with the same generation information are grouped together into a frame set, and the generation information is associated with the frame set. The frame set 601 includes material data of frames corresponding to T=0 to T=N-1. Each material data 602 is given metadata 603. The metadata 603 includes time information similar to the time information 504 in FIG. 5, so that it can be distinguished to which shooting timing a frame included in a frame set corresponds. Metadata 604 is also added to the frame set 601. The metadata 604 includes generation information 605. The configuration of generation information 605 is similar to the configuration of generation information 203 in FIG. 2, generation information 303 in FIG. 3, or generation information 403 in FIG. 4. By associating the generation information 605 with the dataset 601, it can be seen that the material data included in the dataset 601 was generated by a common process indicated by the generation information 605. Also, assume that the generated information changes at T=N. The timing at which the generation information changes is, for example, the timing at which the content of the material data generation process is changed, or the timing at which the parameters related to the process are changed. Next, the generation information remains the same from T=N to T=M-1 until the generation information changes at T=M. Therefore, the frame set 606 includes material data of frames corresponding to T=N to T=M-1, and the metadata 607 added to the frame set 606 includes generated information 608 after the change. Frames corresponding to subsequent shooting timings also have a similar configuration. By configuring as shown in FIG. 6, it becomes easy to use material data corresponding to common generation processing. Furthermore, by adding generation information to the frame set, it is no longer necessary to add generation information to each frame, which has the effect of reducing the amount of data in the material data file.

Another example of the structure of the material data file generated by the association unit 105 will be described using FIG. 7. The material data file shown in FIG. 7 has a configuration in which a certain number of frames (N frames) are collectively made into a frame set, and generation information is associated with the frame set. Frame set 701 includes N frames corresponding to T=0 to T=N-1. Metadata 703 is added to the material data 702 of each frame. Metadata 703 includes time information similar to time information 504 in FIG. 5. Metadata 704 is added to the frame set 701, and the metadata 704 includes time range information 705 and generation information 706. The configuration of the generation information 706 is similar to the configuration of the generation information 203 in FIG. 2, the generation information 303 in FIG. 3, or the generation information 403 in FIG. 4. The time range information 705 is information indicating at which imaging timing the corresponding generation information 706 is generated. For example, if the generated information is the same between T=n and T=m, which is the shooting timing between T=0 and T=N-1, the time range information 705 is T=n to T=m. This information represents the time range of . Further, the generation information 706 corresponding to the time range information 705 at this time includes generation information related to material data corresponding to T=n to T=m. Similarly, for frames after T=N, a frame set is constructed every N frames, and metadata is added. By configuring as shown in FIG. 7, it becomes easy to use material data according to a certain time range.

In addition to the configuration of the material data file described above, a frame set may be configured in which frames are grouped together for each scene in a virtual viewpoint image generated based on the material data, and generation information may be associated with the frame set. Scenes in the virtual viewpoint image include, for example, a goal scene in a virtual viewpoint image related to a sports match, the start of a match, or a change of players. The type of scene is not limited to this. Since it is assumed that different scenes are requested by users who use virtual viewpoint images, the configuration of the frame set may be changed depending on the scenes requested by the users.

By generating the material data file as described above, there is an effect that the material data can be applied to other image processing devices or other image generation devices that conform to a common file format. Furthermore, it has the effect of facilitating handling when handling part of the material data or handling them all at once. Note that the material data and the generated information do not necessarily need to be the same file. For example, the information included in the metadata in FIGS. 2 to 7 may be generated as a file separate from the material data. Further, the method of associating the material data and the generated information performed by the associating unit 105 is not limited to the above method, as long as the material data and the generated information are linked and can be referenced. The associated material data and generated information are output to the storage unit 104.

Figure 8 is a flowchart for explaining the processing executed by a part of the image processing system 100 shown in Figure 1. The processing shown in Figure 8 is performed by the CPU 1211 executing a program. Hereinafter, processing steps are simply denoted as S. When the imaging device 101 starts imaging, the processing starts. In S801, the imaging device 101 captures an imaging area. In S802, the material data generation unit 102 acquires captured images from the multiple imaging devices 101. In S803, the material data generation unit 102 performs a material data generation process and stores the generated material data in the storage unit 104. The generation process performed by the material data generation unit 102 at this time is at least one of a process of generating material data based on the captured image and a correction process performed in the process of generating the material data. In S804, the generation information acquisition unit 103 acquires generation information related to the generation process performed by the material data generation unit 102 from the material data generation unit 102, and stores the acquired generation information in the storage unit 104. In S805, the associating unit 105 performs processing to associate the material data stored in the storage unit 104 with the generation information related to the material data. As described above, the association method is not limited to a specific method. In S806, the associating unit 105 outputs the associated material data and generation information to the storage unit 104. This ends the processing.

Note that in this embodiment, a case has been described in which material data and generation information associated with the material data are output to the storage unit 104, but they may be directly output to the image generation device 120, which will be described later, without going through the storage unit 104. It may also be a configuration. Further, in this embodiment, the storage unit 104 and the association unit 105 are included inside the image processing device 110, but one or both of the storage unit 104 and the association unit 105 are connected to the outside of the image processing device 110. It may be configured as follows.

Next, the functional configuration of the image generation device 120 included in the image processing system 100 will be described. FIG. 9 is a diagram for explaining the functional configuration of the image generation device 120 included in the image processing system 100. Each processing section will be explained below.

The generation information extraction unit 901 acquires generation information associated with the material data stored in the storage unit 104. For example, if material data and generation information are stored as one material data file, the generation information extraction unit 901 performs a process of extracting data related to generation information from the material data file. The generation information extraction unit 901 transmits the extracted generation information to the image generation unit 903.

The virtual viewpoint designation unit 902 determines virtual viewpoint information based on operations by the user. Here, the virtual viewpoint information is information indicating the position and orientation of an arbitrary viewpoint (virtual viewpoint) in the imaging area. The virtual viewpoint information is a parameter set including a parameter indicating the three-dimensional position of the virtual viewpoint and a parameter indicating the orientation of the virtual viewpoint in the pan, tilt, and roll directions. The contents of the virtual viewpoint information are not limited to the above. For example, the parameter set as the virtual viewpoint information may include a parameter indicating the size of the field of view (angle of view) of the virtual viewpoint. The virtual viewpoint designation unit 902 accepts input from an input device connected internally or externally, such as a touch panel, a button, or a keyboard. The virtual viewpoint designation unit 902 determines information such as the position and orientation of the virtual viewpoint determined based on operations of the input device by the user, and virtual viewpoint information such as the time corresponding to the virtual viewpoint image to be generated, and transmits the virtual viewpoint information to the image generation unit 903. At this time, the virtual viewpoint image generated based on the virtual viewpoint information determined by the input operation by the user is displayed on the display unit 904. The user can perform input operations while watching the virtual viewpoint image displayed on the display unit 904.

The image generation unit 903 generates a virtual viewpoint image based on the material data stored in the storage unit 104 and the virtual viewpoint information transmitted from the virtual viewpoint designation unit 902. At this time, it is determined whether or not to perform additional processing based on the generation information transmitted from the generation information extraction unit 901. If it is determined that additional processing is necessary, the image generation unit 903 performs the additional processing. For example, it is assumed that the generation information includes information indicating that foreground extraction processing has not been performed in the generation processing of the material data. In this case, the image generation unit 903 determines that additional processing is necessary and performs foreground extraction processing as additional processing on the material data associated with the generation information. Also, for example, it is assumed that the generation information includes information indicating that distortion correction processing has been performed. In this case, the image generation unit 903 determines that it is not necessary to perform additional distortion correction processing.

If there is no generation information related to material data, the image generation unit 903 cannot know what kind of processing the material data has undergone to be generated. The problems that arise at this time will be explained. For example, in the present embodiment, the image processing device 110 performs foreground extraction processing, and the image generation device 120 generates a virtual viewpoint image based on the material data that has been subjected to the foreground extraction processing. However, as another embodiment, when the foreground extraction process is performed on the side of the device that generates the virtual viewpoint image, the device that generates the material data does not need to perform the foreground extraction process. There is no problem if the material data generation processing is consistent between the device that generates the material data and the device that generates the virtual viewpoint image, but this does not necessarily mean that the device is consistent. Therefore, for example, if the image generation unit 903 cannot know that foreground extraction has not been performed, the image generation unit 109 uses material data that has not been subjected to foreground extraction as foreground extracted data, and generates a virtual viewpoint image. There is a possibility that generation will not be possible. For example, if the image generation unit 903 cannot know that the distortion correction process is being performed on the material data, the image generation unit 109 may perform distortion correction processing on the material data as an additional process. In this case, there is a risk that the distortion correction processing will be repeated, resulting in an increase in the overall processing time for generating the virtual viewpoint image, or a decrease in the accuracy of the generated virtual viewpoint image. Furthermore, although this embodiment describes an example in which material data is generated according to a three-dimensional model generation method, there may be cases where the image generation device does not support the three-dimensional model generation method. In this case as well, without the generation information, the image generation unit 903 cannot know the method by which the material data was generated. As a result, the image generation unit 903 may not be able to generate a virtual viewpoint image. By associating the generation information with the material data, the image generation unit 903 can refer to the generation information, and the above problem is solved. Furthermore, for non-essential processing such as correction processing, the image processing unit 903 can also refer to the generated information and determine whether or not to perform additional processing, taking into account processing time, processing load, etc. .

An example of the virtual viewpoint image generation process performed by the image generation unit 903 will be described. The image generation unit 903 calculates the coordinates of the subject in the virtual viewpoint image based on the virtual viewpoint information transmitted from the virtual viewpoint designation unit 902. The image generation unit 903 also calculates the coordinates of the subject in the actual imaging area based on the camera parameters of the imaging device 101. The image generation unit 903 selects a color or blends a plurality of colors based on the calculated coordinates of the subject and the colors of the subject imaged by the plurality of imaging devices 101, thereby determining the color of the subject in the virtual viewpoint image. Decide on the color. The image generation unit 903 generates a virtual viewpoint image by coloring the three-dimensional model acquired from the storage unit 104 with the color of the subject on the determined virtual viewpoint image. The generated virtual viewpoint image is displayed on the display unit 904. Note that the method for generating a virtual viewpoint image explained above is an example when a three-dimensional model generation method is used, but the method for generating a virtual viewpoint image when using a method other than the three-dimensional model generation method is as follows. Not as long.

FIG. 10 is a flowchart for explaining the processing executed by the image generation device 120. When the CPU 1221 executes the program, the processing shown in FIG. 10 is performed. When the virtual viewpoint specifying unit 1202 detects an input operation or the like by the user to determine virtual viewpoint information, processing is started. In S<b>1001 , the virtual viewpoint specifying unit 902 determines virtual viewpoint information based on the user's input operation, etc., and transmits it to the image generating unit 903 . In S1002, the image generation unit 903 acquires necessary material data from the storage unit 104 based on the virtual viewpoint information acquired from the virtual viewpoint designation unit 902. In S1003, the generation information extraction unit 901 acquires generation information associated with the material data acquired by the image generation unit 903, and transmits it to the image generation unit 903. In S1004, the image generation unit 903 determines whether additional processing is necessary based on the acquired generation information. If it is determined that additional processing is necessary, the process advances to S1005. If it is determined that additional processing is not necessary, the process advances to S1006. In S1005, the image generation unit 903 performs additional processing based on the generation information. In S1006, the image generation unit 903 generates a virtual viewpoint image using the acquired material data and outputs it to the display unit 904. In S1007, the display unit 904 displays the virtual viewpoint image output by the image generation unit 903. In S1008, the image generation processing device 120 determines whether to end the virtual viewpoint image generation processing based on the user's input operation or the like. If it is determined that the virtual viewpoint image generation process is to be finished, the process is finished. If it is determined that the process does not end, the process from S1001 onwards is executed again.

Note that in this embodiment, the display unit 904 is included inside the image generation device 120, but the display unit 904 may be connected to the outside of the image generation device 120.

As described above, by associating the generation information related to the process of generating material data with the material data, it becomes possible to know the details of the process at the time of generating the material data. Further, by referring to the generation information, the image generation device can perform additional processing necessary when generating a virtual viewpoint image using material data.

Second Embodiment
In this embodiment, an image processing system 100 that provides a user with information about material data and a virtual viewpoint image generated based on the material data will be described. Note that the hardware configuration and functional configuration of the image processing system 100 in this embodiment are assumed to be similar to those in the first embodiment. In this embodiment, only parts of the processing that differ from those in the first embodiment will be described.

Based on the generation information acquired from the generation information extraction unit 902, the image generation unit 903 causes the display unit 904 to display information regarding the material data and the virtual viewpoint image generated based on the material data. FIG. 11 is a diagram for explaining an example of information displayed by the display unit 904. The display unit 904 displays thumbnails 1101 of virtual viewpoint images generated by the image generation unit 903, quality information 1103 to 1105 of virtual point-in-time images, and generation information 1106 related to material data generation processing. The image generation unit 903 generates a virtual viewpoint image based on the material data and the generation information associated with the material data, and causes the display unit 904 to display the generated virtual viewpoint image as a thumbnail 1101. At this time, if the image generation unit 903 is unable to generate a virtual viewpoint image for a reason such as not being compatible with the format of the material data, information 1102 indicating that the virtual viewpoint image could not be generated is displayed on the display unit 904. . Furthermore, the image generation unit 903 causes the display unit 904 to display generation information regarding the material data and information regarding the quality of the virtual viewpoint image determined based on the generation information. For example, in FIG. 11, quality information 1103 indicates that the quality of the generated virtual viewpoint image is good. The quality of the virtual viewpoint image is determined based on whether or not correction processing was performed when the material data was generated, whether redundant processing was performed, or whether the parameters used in the correction processing were appropriate. Further, the generated information 1106 corresponding to the quality information 1103 is displayed as "distortion correction provided, noise reduction strong, voxel size 5 mm". According to the generation information, the user is informed that the corresponding virtual viewpoint image has been subjected to distortion correction processing and noise reduction processing (noise reduction) when the material data was generated, and that the resolution (voxel size) of the three-dimensional model is 5 mm. You can know that. The resolution (voxel size) here is expressed by the length of one side of a voxel when real space is divided into cubes (voxels) of unit volume. That is, when the resolution (voxel size) is 5 mm, the three-dimensional model is constructed by a set of voxels each side of which is 5 mm. However, the way the resolution is expressed is not limited to the above. For example, quality information 1104 indicates that a virtual viewpoint image has been generated, but the quality is lower than a predetermined standard. Quality information 1105 indicates that the image generation unit 903 was unable to generate a virtual viewpoint image. Note that when the image generation unit 903 performs additional processing, the displayed generated information 1106 may display generated information that reflects the results of the additional processing. For example, if the image generation unit 903 performs distortion correction processing as additional processing because the generation information extracted by the generation information extraction unit 901 is “no distortion correction processing”, the display unit 904 displays the corresponding generation information 1106 as “No distortion correction processing”. It may also be displayed as "Distortion correction available".

As described above, the image generation unit 903 displays material data and information regarding the virtual viewpoint image generated based on the material data on the display unit 904, so that the user can easily determine the virtual viewpoint image to be generated. It will be possible to provide supporting information. The user can generate a desired virtual viewpoint image while referring to the displayed information.

(Other embodiments)
In the above embodiment, a system in which the image processing device 110 and the image generation device 120 are separate devices has been described, but the present invention is not limited to this. Image processing device 110 and image generation device 120 may be included in the same device. Further, a part of the processing unit included in the image processing device 110 or the image generation device 120 may be connected as a separate device. Furthermore, the functions of the image processing device 110 may be incorporated into each of the plurality of imaging devices 101 to perform the same processing as each processing unit in the image processing device 110.

Further, in the above-described embodiment, an example was described in which the generated information includes information indicating the type of processing and information indicating whether the image processing device 110 has performed the processing, but the information included in the generated information is Not limited to. For example, if the image generation device 120 is set in advance for processing to be performed between the image processing device 110 and the image generation device 120, the generation information may include only information indicating the type of processing executed by the image processing device 110. May contain. By referring to the generation information, the image generation device 120 can determine that processing types not included in the generation information are not being executed in the image processing device 110, and can perform appropriate additional processing. Further, in the above example, the generated information may include only information indicating the type of processing that the image processing device 110 did not execute. The image generation device 120 can determine that processing types not included in the generation information have been executed in the image processing device 110, and can perform appropriate additional processing. As in the above example, the generation information may be any information that allows identification of the processing performed on the material data.

The present invention provides a system or device with a program that implements one or more of the functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

As described above, according to the present invention, by associating material data with generation information related to the generation process of the material data, it is possible to determine what kind of image processing was performed or not performed when the material data was generated. You will be able to know if there are any. Further, by referring to the generation information associated with the material data, the image generation device can perform appropriate processing when generating a virtual viewpoint image using the material data.

102 Material data generation unit 103 Generation information acquisition unit 105 Association unit 110 Image processing device

Claims (16)

An image processing device that generates material data for generating a virtual viewpoint image based on a plurality of captured images captured by a plurality of imaging devices,
Material data generation means for generating the material data;
an output means that associates and outputs the material data generated by the material data generation means and generation information related to the generation process of the material data;
The generation information includes information indicating whether a process of generating the material data or a process of correcting the material data has been performed on the material data,
The output means is an image generation device that generates a virtual viewpoint image based on the material data and the generation information, and the output means performs a process of generating the material data or a process of correcting the material data on the material data. If the material data has been processed, no additional processing is performed on the material data, and if no processing for generating the material data or processing for correcting the material data has been performed on the material data, additional processing is performed on the material data. An image processing apparatus characterized in that the material data and the generation information are outputted in association with each other to the image generation apparatus that executes the process. The image processing device according to claim 1, characterized in that the material data includes the plurality of captured images and data generated based on the captured images. 3. The image processing apparatus according to claim 1, wherein the process of generating the material data includes at least one of a foreground extraction process and a three-dimensional model generation process. The image processing device according to claim 1 or 2, characterized in that the process for correcting the material data includes at least one of distortion correction, noise reduction, calibration, and vibration correction. 5. The image processing apparatus according to claim 1, wherein the generation information includes information regarding processing parameters used in generation processing of the material data. 6. The image processing apparatus according to claim 5, wherein the generation information includes information indicating a location where processing parameters used for generation processing of the material data are stored. 6. The image processing apparatus according to claim 1, wherein the generation information includes information indicating a processing method used to generate the material data. 8. The image processing apparatus according to claim 1, wherein the output means outputs the material data and the generated information in association with each other for each predetermined group. 9. The image processing apparatus according to claim 8, wherein the predetermined group is a group of material data, each frame, the same generation process, or a certain number of frames. The image processing apparatus according to claim 8 or 9, wherein the output means outputs a material data file generated from the material data and the generation information. The image processing device according to any one of claims 1 to 10, characterized in that the additional process is a process that has not been performed on the material data, and is a process for generating the material data or a process for correcting the material data. An acquisition means for acquiring the material data and generation information associated with the material data;
the image generating device having an image generating means for generating a virtual viewpoint image based on the material data and the generation information acquired by the acquisition means;
An image processing system comprising the image processing device according to any one of claims 1 to 11. The image generating means includes:
13. The image processing system according to claim 12, wherein it is determined whether or not to perform the additional processing based on the generation information acquired by the acquisition means. 14. The image processing system according to claim 12, wherein the image generating means controls the generated information to be displayed on a display means. A computer program for causing a computer to function as each of the means possessed by the image processing device according to any one of claims 1 to 11. A method for controlling an image processing device that generates material data for generating a virtual viewpoint image based on a plurality of captured images captured by a plurality of imaging devices, the method comprising:
a material data generation step of generating the material data;
an output step of correlating and outputting the material data generated by the material data generation step and generation information related to the generation process of the material data;
The generation information includes information indicating whether a process of generating the material data or a process of correcting the material data has been performed on the material data,
The output step is an image generation device that generates a virtual viewpoint image based on the material data and the generation information, and includes performing a process of generating the material data or a process of correcting the material data on the material data. If the material data has been processed, no additional processing is performed on the material data, and if no processing for generating the material data or processing for correcting the material data has been performed on the material data, additional processing is performed on the material data. A control method characterized in that the material data and the generation information are outputted in association with each other to the image generation device that executes the above.

Images