JP2022070746A - Virtual viewpoint image generation system, image processing device, image generation device, control methods therefor, and program - Google Patents

Abstract

To provide a technique for generating an appropriate image quality deterioration of a virtual viewpoint image according to a change in lighting condition.SOLUTION: A virtual viewpoint image generation system generates a virtual viewpoint image on the basis of three-dimensional shape information of a foreground, which is a subject based on a plurality of captured images acquired by a plurality of imaging devices, color information of the foreground, three-dimensional shape information of a background in an imaging space of the plurality of imaging devices, unlike the foreground, color information of the background, and a designated virtual viewpoint. The virtual viewpoint image generation system determines whether or not a lighting condition has changed beyond a predetermined criterion on the basis of at least one of the plurality of captured images, and adjusts the color information of at least one of the foreground and the background so that lighting conditions of the color information of the foreground and the background used for generating the virtual viewpoint image are brought closer, when it is determined that the lighting condition has changed.SELECTED DRAWING: Figure 1A

Description

The present invention relates to a virtual viewpoint image generation system, an image processing device, an image generation device, a control method thereof, and a program.

Recently, a technique called virtual viewpoint image, which enables viewing of images from various viewpoints and directions, is attracting attention. According to the virtual viewpoint image technology, for example, the highlight scenes of soccer and basketball can be viewed from various angles, so that the user can be given a higher sense of presence than a normal image. Such a virtual viewpoint image is generated from a plurality of captured images (multi-viewpoint captured images) obtained by capturing images from multiple directions at the same timing using a plurality of cameras installed so as to surround the subject. ..

Patent Document 1 describes a method for generating a virtual viewpoint image as described above. According to this generation method, each of the plurality of captured images obtained from a plurality of cameras having different positions and postures (viewpoints) is separated into a foreground image representing the subject and a background image other than the subject, and each of the plurality of foreground images. The silhouette image of the subject is extracted from. The extracted plurality of silhouette images are from different viewpoints, and the three-dimensional shape of the foreground image (subject) is derived by the visual volume crossing method or the like. Then, an image of an arbitrary viewpoint (virtual viewpoint image) is generated by rendering the color information obtained from the captured image on the three-dimensional shape observed from the viewpoint instructed by the operator.

Japanese Unexamined Patent Publication No. 2018-194985

In the technique disclosed in Patent Document 1, a three-dimensional shape is derived from a silhouette image, and a virtual viewpoint image is generated. Generally, in rendering a virtual viewpoint image, color information captured in advance and a background model are used for the background, and only the subject in the foreground is rendered based on the captured data, thereby reducing the processing load of the system. .. By the way, when the lighting conditions suddenly change due to a flash or the like, the texture image showing the color information of the foreground of the subject may be captured in an overexposed state. In such a case, a texture image that is overexposed under the influence of the flash is used only for the foreground, which is the subject, and a texture image before the lighting conditions change is used for the background, which is an unnatural virtual viewpoint. An image will be generated. That is, there is a problem that only the foreground, which is the subject, is displayed under the influence of the flash, the presence of the virtual viewpoint image is lost, and the viewer feels uncomfortable.

The present invention provides a technique for generating an appropriate virtual viewpoint image according to changes in lighting conditions.

The virtual viewpoint image generation system according to one aspect of the present invention has the following configurations. That is,
The three-dimensional shape information of the foreground, which is the subject based on the plurality of captured images acquired by the plurality of imaging devices, the color information of the foreground, and the background 3 in the imaging space of the plurality of imaging devices, which is different from the foreground. A generation means for generating a virtual viewpoint image based on the three-dimensional shape information, the background color information, and the designated virtual viewpoint.
A determination means for determining whether or not a change exceeding a predetermined reference has occurred in the lighting conditions based on at least one of the plurality of captured images.
When it is determined by the determination means that the lighting conditions have changed, the foreground color information and the background are brought closer to each other so that the lighting conditions of the foreground color information and the background color information used by the generation means are brought closer to each other. It has an adjusting means for adjusting at least one of the color information of the above.

According to the present invention, it is possible to generate an appropriate virtual viewpoint image according to a change in lighting conditions.

The block diagram which shows the functional composition example of an image processing apparatus. The block diagram which shows the hardware configuration example of an image processing apparatus. The figure which shows the example of the virtual viewpoint image. The figure explaining the process of a flash information acquisition part. A flowchart showing the processing of the flash information acquisition unit. The figure which shows the example of the luminance of image data. The flowchart which shows the processing of the flash information processing part by 1st Embodiment. The figure which shows the example of the 3D coordinates which shows the position of a flash. The flowchart which shows the processing of the flash information processing part by 2nd Embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are given the same reference numbers, and duplicate explanations are omitted.

<First Embodiment>
FIG. 1A shows a configuration example of the virtual viewpoint image generation system 1 according to the first embodiment. The virtual viewpoint image generation system 1 includes an image processing device 100, an operation device 110, a display device 120, and an image pickup device 130. The operation device 110, the display device 120, and the image pickup device 130 are connected to the image processing device 100. FIG. 1A also shows an example of a functional configuration of the image processing apparatus 100. The operation device 110 receives instructions such as the position / orientation of the virtual viewpoint camera from the operator, and supplies the received instructions to the image processing device 100 (virtual viewpoint image generation unit 105). The operating device 110 includes, for example, a lever and a switch operated by the user to specify the position / orientation of the virtual viewpoint camera. The display device 120 has a display device typified by a liquid crystal display or the like, and displays a virtual viewpoint image generated by the image processing device 100 (virtual viewpoint image generation unit 105). The operator can specify the position / posture of the virtual camera from the operation device 110 while looking at the virtual viewpoint image displayed on the display device 120.

The image processing device 100 generates a virtual viewpoint image based on the images captured by the plurality of image pickup devices 130 and the position of the virtual viewpoint camera input from the operation device 110. The image processing device 100 has an image pickup image processing device 10 provided corresponding to each image pickup device 130, and an image generation device 20 connected to the image pickup image processing device 10. The captured image processing device 10 generates a silhouette image and flash information from the captured image acquired from the image pickup device 130, and outputs the silhouette image and the flash information to the image generation device 20. The image generation device 20 generates a virtual viewpoint image at the position / orientation of the virtual viewpoint camera instructed by the operation device 110 based on the silhouette image and the flash information input from the captured image processing device 10.

The image pickup apparatus 130 is a digital video camera provided with an image signal interface represented by a serial digital interface (SDI). The image pickup device 130 supplies image data to the image pickup image processing device 10 via the image signal interface. A plurality of image pickup devices 130 are installed so as to surround the periphery of the subject (object), and the plurality of image pickup devices 130 perform image pickup substantially at the same time. Each image pickup device 130 is connected to the image pickup image processing device 10. The image pickup image processing device 10 may be provided in the image pickup device 130.

The captured image processing device 10 includes an image input unit 101, a silhouette image derivation unit 102, and a flash information acquisition unit 103 as functional configurations. The image input unit 101 receives image data (captured image) from the image pickup device 130 and supplies the image data to the silhouette image derivation unit 102 and the flash information acquisition unit 103.

The silhouette image derivation unit 102 derives a silhouette image which is shape information of a subject (foreground) by a method typified by the background subtraction method or the like. The silhouette image is a black-and-white image in which the inside of the outline of the subject is filled. The silhouette image has information on the shape of the subject, which is a binary representation of whether each pixel is inside or outside the contour of the subject. The silhouette image derivation unit 102 outputs the derived silhouette image and the texture image which is the image data corresponding to the silhouette to the image generation device 20 (three-dimensional shape derivation unit 104).

The flash information acquisition unit 103 determines the presence or absence of flash light emission from the image data (captured image) acquired from the image input unit 101. The flash information acquisition unit 103 is an example of a configuration for determining whether or not a change exceeding a predetermined reference has occurred in the lighting conditions based on at least one of a plurality of captured images. The flash information acquisition unit 103 determines that the flash is emitted when the lighting conditions change beyond a predetermined reference. Further, the flash information acquisition unit 103 generates information (for example, a histogram of luminance) indicating the intensity of the flash. The flash information acquisition unit 103 outputs flash information including information indicating the presence / absence of flash emission and information indicating the intensity of the flash to the image generation device 20 (flash information processing unit 106). The processing of the flash information acquisition unit 103 will be described later using the flowchart of FIG.

It should be noted that the present embodiment shows an embodiment in which the same number of image pickup image processing devices 10 as the number of image pickup devices 130 are installed, but the present embodiment is not limited to this. For example, one image pickup image processing device 10 may be provided for a predetermined number of image pickup devices 130 of two or more. Further, the function of the image pickup image processing device 10 (silhouette image derivation unit 102, flash information acquisition unit 103) may be provided in the image pickup device 130. Alternatively, the function of the captured image processing device 10 may be included in the image generation device 20. However, in that case, since the plurality of image pickup devices 130 and the image processing device 100 are connected by a network and the images captured by the image pickup device 130 are transmitted on the network, the load on the network increases.

Next, the functional configuration of the image generation device 20 will be described. The 3D shape derivation unit 104 derives 3D shape information (3D shape data) representing the 3D shape of the subject from a plurality of silhouette images input from the plurality of silhouette image derivation units 102, and obtains this as a virtual viewpoint image. It is supplied to the generation unit 105. As a method for the 3D shape deriving unit 104 to derive 3D shape information from a silhouette image, a generally used visual volume crossing method (for example, the shape from silhouette method) can be used. In the visual volume crossing method, silhouette images from a plurality of imaging units are back-projected into a three-dimensional space, and three-dimensional shape information is obtained by obtaining an intersecting portion of each visual volume. Further, the 3D shape derivation unit 104 generates a texture corresponding to the 3D shape information based on the texture (color information) received from the silhouette image derivation unit 102, and supplies the texture to the virtual viewpoint image generation unit 105.

The virtual viewpoint image generation unit 105 observes from the position and orientation of the virtual viewpoint camera designated by the operation device 110 based on the three-dimensional shape information and texture of the subject (foreground) and the three-dimensional shape information and texture of the background. Generate a viewpoint image. However, in the present embodiment, the background texture is provided by the flash information processing unit 106. The virtual viewpoint image generation unit 105 receives the 3D shape information and texture of the subject (foreground) from the 3D shape derivation unit 104. Further, the virtual viewpoint image generation unit 105 has a background model (three-dimensional shape method of the background) previously stored in the background data storage unit 107, and a corrected background texture provided by the flash information processing unit 106 described later. To get. The virtual viewpoint image generation unit 105 performs rendering processing from the acquired three-dimensional shape information and texture information of the foreground and background, and generates a virtual viewpoint image. The virtual viewpoint image generation unit 105 outputs the generated virtual viewpoint image to the display device 120.

The flash information processing unit 106 adjusts (hereinafter, also referred to as correction) the texture of the background acquired from the background data storage unit 107 based on the flash information acquired from the flash information acquisition unit 103. More specifically, when the flash information indicates that the flash is emitted, the flash information processing unit 106 adjusts the background texture stored in the background data storage unit 107 based on the information indicating the intensity of the flash. .. Hereinafter, this adjustment process may be referred to as a flash reproduction process. The adjusted (corrected) texture is provided to the virtual viewpoint image generation unit 105. Further, when the flash information indicates that there is no flash emission, the flash information processing unit 106 provides the virtual viewpoint image generation unit 105 with the background texture stored in the background data storage unit 107 as it is. Further detailed processing of the flash information processing unit 106 will be described later using the flowchart of FIG. The background data storage unit 107 stores the three-dimensional shape method of the background and the texture of the background. The three-dimensional shape information of the background and the texture of the background are obtained based on the captured images obtained by the plurality of imaging devices 130 capturing the imaging space while the subject (wrestler in this example) does not exist in advance.

An example of the hardware configuration of the image pickup image processing device 10 and the image generation device 20 that realize the above-mentioned functions will be described with reference to FIG. 1B. Although the configuration of the image generation device 20 will be described below, the captured image processing device 10 also has the same configuration. The image generation device 20 includes a CPU 161, a ROM 162, a RAM 163, an auxiliary storage device 164, a display unit 165, an operation unit 166, a communication I / F 167, and a bus 168. The display unit 165 and the operation unit 166 may be omitted when the display device 120 and the operation device 110 are connected. Further, in the captured image processing device 10, the auxiliary storage device 164, the display unit 165, and the operation unit 166 can be omitted.

The CPU 161 realizes each function of the image generation device 20 shown in FIG. 1A by controlling the entire image generation device 20 by using computer programs and data stored in the ROM 162 and the RAM 163. The image generation device 20 may have one or more dedicated hardware different from the CPU 161 and the dedicated hardware may execute at least a part of the processing by the CPU 161. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The ROM 162 stores programs and the like that do not require changes. The RAM 163 temporarily stores programs and data supplied from the auxiliary storage device 164, data supplied from the outside via the communication I / F 167, and the like. The auxiliary storage device 164 is composed of, for example, a hard disk drive or the like, and stores various data such as image data and audio data.

The display unit 165 is composed of, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) for the user to operate the image generation device 20. The operation unit 166 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and inputs various instructions to the CPU 161 in response to an operation by the user. The CPU 161 operates as a display control unit that controls the display unit 165 and an operation control unit that controls the operation unit 166.

The communication I / F 167 is used for communication with an external device of the image generation device 20. For example, when the image generation device 20 is connected to an external device by wire, a communication cable is connected to the communication I / F 167. When the image generation device 20 has a function of wirelessly communicating with an external device, the communication I / F 167 includes an antenna. The bus 168 connects each part of the image generation device 20 to transmit information.

In the present embodiment, it is assumed that the display unit 165 and the operation unit 166 exist inside the image generation device 20, but at least one of the display unit 165 and the operation unit 166 exists as another device outside the image generation device 20. You may be doing it.

Next, with reference to FIG. 2, the virtual viewpoint image generation process by the image processing device 100 will be described. FIG. 2 (g) shows a virtual viewpoint image 270 generated and output by the image processing device 100 (image generation device 20) of the present embodiment. Hereinafter, the process of generating the virtual viewpoint image 270 of FIG. 2 (g) by the image processing apparatus 100 will be described. The virtual viewpoint image 270 shown in FIG. 2 (g) shows a scene of sumo wrestling.

First, in FIG. 2A, the captured image 210 is an image obtained by the imaging device 130 taking images of the wrestlers 211 and 212, which are the subjects, from the periphery of the ring 213. Further, in FIG. 2A, the cameraman 215 photographs the state of the efforts on the ring 213 in the audience seats outside the ring 213. The cameraman 215 uses a flash to take an image. Since the image pickup apparatus 130 takes an image so as to have an appropriate exposure in a state where the flash image is not taken, the images of the wrestlers 211 and 212, which are the subjects, are overexposed due to the flash being fired.

As shown in FIG. 2B, the silhouette image deriving unit 102 derives the silhouette images 221 and 222 of the wrestlers 211 and 212, which are the subjects, from the captured image 210. Further, as shown in FIG. 2C, the silhouette image deriving unit 102 derives the texture images 231 and 232 corresponding to the silhouette images 221 and 222 from the captured image 210. Since the texture image is generated from the image data of FIG. 2A that is affected by the flash, the texture images 231 and 232 are affected by the flash. For example, overexposure occurs in at least a part of the texture images 231 and 232. The texture image is an image showing the color information of the subject.

Next, as shown in FIG. 2 (d), the three-dimensional shape deriving unit 104 derives the three-dimensional shape information 241 and 242 from the silhouette images 221 and 222 as shown in FIG. 2 (b). A plurality of silhouette images derived from each of the plurality of captured images obtained by the plurality of imaging devices 130 arranged around the subject substantially simultaneously imaging are input to the three-dimensional shape deriving unit 104. The three-dimensional shape derivation unit 104 derives the three-dimensional shape information 241,242 shown in FIG. 2D by using the visual volume crossing method or the like based on the plurality of silhouette images.

Next, the virtual viewpoint image generation unit 105 generates a virtual viewpoint image. In the present embodiment, the background model and the background texture are stored in the background data storage unit 107 in advance. FIG. 2 (e) shows the background texture 253 of the ring stored in advance. The background texture 253 of the ring 213 stored in advance in the background data storage unit 107 is not affected by the flash. When rendering is performed using the texture images 231 and 232 of the wrestlers of the subject affected by the flash and the background texture 253 not affected by the flash, only the foreground is an unnatural virtual viewpoint affected by the flash. It becomes an image. Therefore, in the image processing device 100 (image generation device 20) of the present embodiment, the flash information processing unit 106 corrects the background texture 253 stored in advance and reproduces the influence of the flash on the background texture.

The flash information acquisition unit 103 detects the presence or absence of flash light emission from the captured image, and notifies the flash information processing unit 106 of the detection result. The flash information processing unit 106 corrects the background texture by reproducing the flash for the frame in which the flash emission is detected. The background texture 263 in FIG. 2 (f) represents a texture in which the influence of the flash is reproduced on the background texture 253. The virtual viewpoint image generation unit 105 renders using the foreground texture images 231 and 232, the three-dimensional shape information 241, 242, the corrected background texture 263, and the background model, and is shown in FIG. 2 (g). The virtual viewpoint image 270 shown is generated. In this way, both the foreground image and the background image become images affected by the flash, so that a more natural virtual viewpoint image can be obtained.

Next, a process in which the flash information acquisition unit 103 acquires flash information and a process in which the flash information processing unit 106 corrects the background texture based on the flash information will be described with reference to FIG.

3 (a), (c), and (e) are images taken by the image pickup apparatus 130. 3A, 3C, and 3E are captured images of continuous frames, which are captured images at time T = 100, T = 101, and T = 102, respectively. Further, the histograms derived by the flash information acquisition unit 103 for each frame are shown in FIGS. 3 (b), (d), and (f). In the histograms of FIGS. 3B, 3D, and 3F, the horizontal axis shows the brightness of pixels and the vertical axis shows the number of pixels. In the present embodiment, the area of the luminance value is 8 bits (0 to 255).

The captured image 310 of FIG. 3A shows a scene in which the wrestlers 311 and 312, who are the subjects, are working on the ring 313. It is assumed that the cameraman 315 is taking an image of the effort in the audience seats outside the ring 313. At time T = 100, the camera flash of the cameraman 315 is not firing. FIG. 3B shows a histogram 340 of the captured image 310 of FIG. 3A. In the histogram 340, it can be seen that the mode is located at 120 near the center and the exposure is appropriate.

Next, it is assumed that the flash of the camera possessed by the cameraman 315 fires at the time T = 101. In the frame (captured image 320) of FIG. 3C, overexposure occurs in the wrestlers 311 and 312 and the ring 313 due to the influence of the flash. In this case, as shown in FIG. 3D, in the histogram 360 of the captured image 320, it can be seen that the mode is located at 240, which is close to the upper limit of the luminance, and is in an overexposed state.

Next, at time T = 102, the flash of the camera possessed by the cameraman 315 has finished firing. In this case, the captured image 330 is obtained as shown in FIG. 3 (e), and in the histogram 380, the mode returns to 100 near the center as shown in FIG. 3 (f), and the appropriate exposure is obtained. It has become.

The flash information acquisition unit 103 derives an average value of the brightness of each pixel from the captured image obtained from the image pickup device 130, and determines the presence or absence of flash light emission based on the derived average value. For example, the flash information acquisition unit 103 fires the flash when the difference between the average value of the brightness values of the 10 frames before the current frame to be determined and the brightness value of the current frame is equal to or more than the threshold value. Judge. When the flash light emission is detected, the flash information acquisition unit 103 sends the flash information including the histogram of the luminance value obtained from the captured image at that time to the flash information processing unit 106 in addition to the information notifying that the flash light emission is present. The histogram of the luminance value is an example of information indicating the intensity of the flash. The flash information acquisition unit 103 of the present embodiment has shown an example of comparison with the average value of the previous 10 frames of the current frame, but it is clear that the number of frames is not limited to 10 and may be another number of frames. be. When the flash light emission is not detected, the flash information acquisition unit 103 sends flash information including information notifying that there is no flash light emission to the flash information processing unit 106.

FIG. 5 shows an example of the luminance value (average luminance value) of each frame calculated by the flash information acquisition unit 103. The flash information acquisition unit 103 starts calculating the average luminance in the frame (average luminance value in the frame) from time T = 0. The flash information acquisition unit 103 calculates the in-frame average luminance value of T = 0 as 101. Further, the flash information acquisition unit 103 calculates the average value of the luminance for a plurality of frames before the current frame (hereinafter, the average luminance value between frames). The flash information acquisition unit 103 of the present embodiment calculates the average luminance for 10 frames (T = n-1 to n-10) before the current frame (T = n), and calculates the average luminance value between frames. .. When T = 0, there is no frame before that, so the average luminance of the current frame is used as the average luminance value between frames.

Although the description from time T = 1 to 99 is omitted, the flash information acquisition unit 103 calculates the in-frame average luminance value of the current frame, and further calculates the inter-frame average luminance value from the 10 frames before the current frame. .. At time T = 100, the flash information acquisition unit 103 obtains an in-frame average luminance value = 101 and an inter-frame average luminance value = 100. At time T = 101, the flash fires as shown in FIG. 3 (c), and in the histogram 360 as shown in FIG. 3 (d), the mode is 240, which is close to the upper limit of luminance. It is located and is overexposed. In this state, the flash information acquisition unit 103 obtains an in-frame average luminance value = 200 and an inter-frame average luminance value = 100. As described above, in the captured image affected by the flash, the difference between the in-frame average brightness value and the inter-frame average brightness value becomes large. The flash information acquisition unit 103 determines the presence or absence of flash light emission by utilizing such a phenomenon.

FIG. 4 is a flowchart showing the processing of the flash information acquisition unit 103. Hereinafter, the processing of the flash information acquisition unit 103 will be described with reference to the flowchart shown in FIG. 4 and examples of the images and luminance values shown in FIGS. 3 and 5. In addition, in order to simplify the explanation, the explanation from time T = 0 to 99 is omitted, and the explanation starts from the process of time T = 100. Even at times T = 0 to 99, the flash information acquisition unit 103 is performing the process shown in FIG.

When the image processing device 100 is activated and the image pickup operation by the image pickup device 130 is started, in step S401, the flash information acquisition unit 103 acquires the captured image from the image pickup device 130. At time T = 100, the captured image 310 as shown in FIG. 3A is acquired from the imaging device 130. Next, in step S402, the flash information acquisition unit 103 derives the in-frame average luminance value from the captured image acquired in step S401. At time T = 100, as shown in FIGS. 3 (b) and 5, the in-frame average luminance value = 101 is acquired. Next, in step S403, the flash information acquisition unit 103 derives the inter-frame average luminance value. As described above, in the present embodiment, the average value of the in-frame average luminance values of the 10 frames from the frame immediately before the current frame to 10 frames before is calculated as the interframe average luminance value. Here, from T = 90 to T = 99, which is 10 frames before the time T = 100, frames similar to the time T = 100 continue, and as shown in FIG. 5, the flash information acquisition unit 103 has 10 frames. It is assumed that the average brightness between frames for each frame is derived as 100.

Next, in step S404, the flash information acquisition unit 103 determines whether or not a change exceeding a predetermined reference has occurred in the lighting conditions. In this example, it is determined whether or not the change in brightness in the captured image is equal to or greater than the threshold value. If the change in luminance is equal to or greater than the threshold value, it is determined that the flash has fired, and the process proceeds to step S405. If the change in luminance is less than the threshold value, it is determined that the flash is not emitting light, and the process proceeds to step S407. More specifically, the flash information acquisition unit 103 compares the in-frame average luminance value of the current frame calculated in step S402 with the inter-frame luminance value calculated in step S403 up to the previous frame. The flash information acquisition unit 103 determines that the change in luminance in the captured image is equal to or greater than the threshold when the difference between the average luminance value in the frame and the average luminance value between frames is equal to or greater than a predetermined threshold value. For example, when the difference between the average luminance value in the frame and the average luminance value between frames is 50 or more, it is determined that the change in luminance is large, and it is determined that the flash fires. Of course, the threshold is not limited to this example. At time T = 100, the average in-frame luminance value of the current frame is 101, the average luminance value between frames up to the previous frame is 100, and the difference between the two is 1. Since the difference is 50 or less, it is determined that the flash is not emitting light, and the process proceeds to step S407. In step S407, the flash information acquisition unit 103 sends flash information indicating “no flash emission” to the flash information processing unit 106.

Next, in step S406, the flash information acquisition unit 103 determines whether or not the processing of all frames has been completed. When it is determined in step S406 that the processing of all frames is completed, or when the end of imaging by the image pickup apparatus 130 is instructed, this processing ends (YES in step S406). In the case of this example, at time T = 100, the processing of all frames has not been completed yet (NO in step S406), and the processing transitions to step S401.

Subsequently, in step S401, the flash information acquisition unit 103 acquires an image captured at time T = 101 from the image pickup device 130. The flash information acquisition unit 103 acquires the captured image 320 (FIG. 3 (c)) via the image input unit 101 at time T = 101. In step S402, the flash information acquisition unit 103 derives the in-frame average luminance value from the captured image 320. As described above, in FIG. 3 (c), the flash is emitted, and in the histogram 360 (FIG. 3 (d)), the mode is located at 240, which is close to the upper limit of the luminance, and is in an overexposed state. ing. In this example, for the captured image 320 at time T = 101, the in-frame average luminance value = 200 of the current frame is derived. Next, in step S403, the flash information acquisition unit 103 derives the inter-frame average luminance value. As described above, the average value of the luminance values from the frame immediately before the current frame to 10 frames before is derived. In this example, the inter-frame average luminance value = 100 from T = 91 to T = 100 10 frames before the time T = 101 is derived.

Next, in step S404, the flash information acquisition unit 103 compares the in-frame average luminance value acquired in step S402 with the interframe average luminance value acquired in step S403. When the time T = 101, the average luminance value in the frame is 200 and the average luminance value between frames is 100, so that the difference between the two is 100. Since this is larger than the threshold value (= 50), the process proceeds to step S405. In step S405, the flash information acquisition unit 103 determines that the flash is emitting light, and sends flash information including information indicating that the flash is emitted and information (histogram) indicating the intensity of the flash to the flash information processing unit 106.

Next, in step S406, the flash information acquisition unit 103 determines whether or not the processing of all frames has been completed. At time T = 101, the processing of all frames has not been completed yet, so the processing proceeds to step S401.

In step S401, the flash information acquisition unit 103 acquires an image captured at time T = 102 from the image pickup device 130. The flash information acquisition unit 103 acquires an image captured image 330 as shown in FIG. 3 (e) from the image pickup apparatus 130 at time T = 102. Then, in step S402, the flash information acquisition unit 103 derives the in-frame average luminance value from the captured image 330. As described above, at T = 102, which is the shooting time of the captured image 330, the flash emitted in the previous frame is no longer emitted, and the average luminance value in the frame is as shown in FIG. 3 (f). It is 100. Next, in step S403, the flash information acquisition unit 103 derives the average luminance value between frames. The average (= 110) of the luminance values from the frame of T = 92 10 frames before the time T = 102 to the frame of T = 101 is derived as the interframe luminance value.

Next, in step S404, the flash information acquisition unit 103 compares the in-frame average luminance value of the current frame derived in step S402 with the interframe average luminance value up to the previous frame derived in step S403. In this example, the average luminance value in the frame is 100 and the average luminance value between frames is 110 for the captured image 330 at time T = 102, and the difference between the two is 10 (<threshold value = 50). Therefore, the process proceeds to step S407, and the flash information acquisition unit 103 determines that the flash is not emitting light, and transmits the flash information indicating "no flash emission" to the flash information processing unit 106.

In this way, the flash information acquisition unit 103 can determine the presence or absence of flash light emission from the image data, derive a histogram as flash information, and transmit it to the flash information processing unit 106 by the above-mentioned processing. In the above example, the flash emission period is one frame period of T = 101, but the flash emission period may span a plurality of consecutive frames. By using the difference between the average luminance value in the frame and the average luminance value between frames, it is possible to detect a plurality of consecutive frames in which flash emission exists. For example, in FIG. 5, even if the in-frame average luminance value at T = 101 and T = 102 is 200, the average luminance values calculated at T = 101 and T = 102 are 100 and 110. Therefore, even at T = 102, the difference between the average luminance value in the frame and the average luminance value between frames is 90> threshold value (50), and the flash emission is detected.

Next, the operation of the flash information processing unit 106 that has received the flash information from the flash information acquisition unit 103 will be described. FIG. 6 is a flowchart showing the processing of the flash information processing unit 106. In the following, for the sake of simplicity, the description of the processing related to the captured image at time T = 0 to 99 will be omitted, and the processing of the captured image after time T = 100 will be described. The processing described below is also performed on the captured images from time T = 0 to 99.

In step S601, the flash information processing unit 106 acquires flash information from the flash information acquisition unit 103. Next, in step S602, the flash information processing unit 106 confirms the presence or absence of flash light emission from the flash information received in step S601. For the captured image at time T = 100, "no flash emission" is notified by the flash information. When "no flash emission" is confirmed, the process proceeds to step S604. In step S604, the flash information processing unit 106 sends the background texture stored in the background data storage unit 107 to the virtual viewpoint image generation unit 105 as it is. In this way, when it is determined that the flash is not emitting light, the flash information processing unit 106 does not perform the flash reproduction process, but provides the background texture to the virtual viewpoint image generation unit 105 as it is.

As described above, when the flash does not emit light, the virtual viewpoint image generation unit 105 renders the virtual viewpoint image 280 (FIG. 2 (h)) that has not been subjected to the flash reproduction process. The texture (background texture) of the ring 283 stored in advance is used for rendering the ring 283 of FIG. 2 (h) rendered by the virtual viewpoint image generation unit 105.

In step S605, the flash information processing unit 106 determines whether or not the processing of all the frames is completed. Since the processing of all frames is not completed at time T = 100, the processing proceeds to step S601.

Subsequently, the flash information processing unit 106 processes the captured image 320 at time T = 101. In step S601, the flash information processing unit 106 acquires flash information from the flash information acquisition unit 103. Next, in step S602, the flash information processing unit 106 confirms the presence or absence of flash light emission from the received flash information. The flash information at time T = 101 notifies that the flash is firing. Therefore, the process transitions to step S603.

In step S603, the flash information processing unit 106 adjusts (corrects) the background texture so as to approach the luminance characteristics of the captured image when a change is detected (when the presence or absence of flash emission is detected). It should be noted that the background texture does not have to be adjusted to perfectly match the luminance characteristics of the captured image. It suffices if it is possible to reduce the discomfort between the foreground and the background due to the flash on the virtual viewpoint image. In the present embodiment, the flash information processing unit 106 performs flash reproduction processing on the background texture based on the information (brightness histogram) indicating the intensity of the flash attached to the flash information. As described above, when the flash fires, the flash information acquisition unit 103 adds a histogram to the flash information and sends it to the flash information processing unit 106. The flash information processing unit 106 corrects the brightness of the background texture 253 of the ring stored in the background data storage unit 107 based on the histogram included in the flash information. For example, a histogram matching correction can be applied to this correction. Histogram matching correction is an image processing method that makes the brightness and contrast of an image the same by matching the shapes of the histograms. The flash information processing unit 106 corrects the brightness of the background texture 253 of the ring stored in advance based on the transmitted flash information (histogram), thereby correcting the corrected background shown in FIG. 2 (f). Derivation of texture 263. In FIG. 2 (f), since the histogram matching correction is performed based on the histogram of the captured image when the flash fires, the ring when the flash fires can be reproduced.

The virtual viewpoint image generation unit 105 renders a virtual viewpoint image using the background texture 263 (FIG. 2 (f)) of the ring corrected by the flash reproduction process, and the virtual viewpoint image 270 (FIG. 2 (g)). To generate. By this process, as described above, it is possible to suppress the generation of an unnatural virtual viewpoint image in which only the foreground is affected by the flash. Next, the processing proceeds to step S605, and the flash information processing unit 106 determines whether or not the processing of all the frames is completed. At time T = 101, the processing of all frames has not been completed yet, so the processing proceeds to step S601. Although the description of the process at time T = 102 is omitted, since the flash does not emit light, the flash reproduction process is not performed as at time T = 100.

By the above processing, it is possible to suppress the generation of an unnatural virtual viewpoint image in which only the foreground is affected by the flash, and the provision of a virtual viewpoint image that does not impair the viewer's sense of presence and does not give a sense of discomfort. Is possible.

In this embodiment, an example of correcting the background texture is shown, but the present invention is not limited to this. In order to provide a virtual viewpoint image that does not give a sense of discomfort, the lighting conditions of the foreground and background textures used to generate the virtual viewpoint image should be brought closer to each other when the lighting conditions of the imaging space change beyond a predetermined standard. In addition, at least one of those textures needs to be corrected. Therefore, for example, a process of correcting the texture of the foreground to reduce the influence of the flash may be performed. For example, the foreground texture emitted by the flash may be corrected based on the histogram of the foreground texture of the foreground frame in which the flash is not emitted. However, the flash information acquisition unit 103 transmits the histogram of the foreground texture when it detects that there is no flash emission to the flash information processing unit 106. The histogram of the foreground texture is transmitted to the flash information processing unit 106, for example, every time no flash emission is detected, or at an arbitrary timing in a series of imaging (for example, the timing when no flash emission is first detected). Can be done. For example, the flash information indicating no flash emission transmitted in step S407 of FIG. 4 includes a histogram of the luminance value of the captured image. In this case, the histogram included in the flash information transmitted in step S405 can be omitted.

Further, in the configuration shown in FIG. 1A, since a plurality of flash information acquisition units 103 are provided corresponding to the plurality of image pickup devices 130, a plurality of flash information is transmitted to the flash information processing unit 106. Therefore, in step S602 and step S603, it is necessary to select the flash information to be used. For example, the flash information processing unit 106 selects the flash information to be used from the relationship between the position of the virtual camera used for generating the virtual viewpoint image and the position of the image pickup apparatus 130 among the plurality of flash information. For example, the flash information processing unit 106 executes the processes of steps S602 and S603 by using the flash information from the flash information acquisition unit 103 corresponding to the image pickup device 130 closest to the position of the virtual camera. Alternatively, the flash information may be transmitted only from the selected flash information acquisition unit 103, or the system may be provided with one flash information acquisition unit 103 and the flash information may be transmitted using an image captured by a specific image pickup device. May be generated.

The above-mentioned selection of flash information is an example of correction based on the determination of a single flash information acquisition unit 103, but the determination of the presence or absence of flash emission of a plurality of flash information acquisition units 103 may be used. For example, the execution of the flash reproduction process may be decided by a majority vote based on the result of the presence / absence of the flash emission of the plurality of flash information acquisition units 103. Further, when making a majority decision, weighting may be made based on the positional relationship between the position of the virtual camera and the positional relationship of the image pickup device. For example, the flash information of the image pickup device close to the position of the virtual camera may be given a larger weight to make a judgment by majority vote. Alternatively, when a predetermined number or more (1 or more) of the acquired flash information indicates that the flash is emitted, it may be determined that the flash is emitted in the image pickup. Further, the flash information may include the in-frame average luminance value, and the flash information having the highest in-frame average luminance value may be used in the adjustment process of step S603.

<Second Embodiment>
In the first embodiment, an example is shown in which a change exceeding the standard of lighting conditions (for example, flash emission) is detected to correct the texture of the background. In the second embodiment, a configuration will be described in which the position of a light source that causes a change in lighting conditions is specified, the light source is arranged in a virtual viewpoint image, and the background texture is corrected. The following is an example of deriving the light emitting position of the flash as a light source in the three-dimensional space and reproducing the flash light emission when rendering the virtual viewpoint image.

In the second embodiment, the flash is reproduced by deriving the light emitting position of the flash in the three-dimensional space and adding a light source corresponding to the flash at the time of rendering. Therefore, it is desirable to perform the processing described later using a texture that is not affected by the flash for the foreground. In the second embodiment, the texture of the foreground is corrected by the histogram matching correction shown in the first embodiment to reduce or eliminate the influence of the flash on the foreground texture, and then the processing described later is performed. The configuration of the virtual viewpoint image generation system 1 of the second embodiment is the same as that of the first embodiment (FIG. 1A). However, in the second embodiment, in order to reduce the influence of the flash on the foreground texture, the foreground texture transmitted by the silhouette image deriving unit 102 is provided to the flash information processing unit 106.

In the second embodiment, the flash information acquisition unit 103 specifies the light emission position (image coordinates) of the flash in addition to detecting the presence or absence of the flash light emission. Further, the flash information processing unit 106 receives the image coordinates (flash emission position) derived by each of the plurality of flash information acquisition units 103. The flash information processing unit 106 derives the three-dimensional position of the light emission of the flash from a plurality of image coordinates, and notifies the virtual viewpoint image generation unit 105 of the three-dimensional position. The virtual viewpoint image generation unit 105 renders the virtual viewpoint image by installing a new light source at the three-dimensional position of the derived flash. By adding a light source that reproduces the flash to the frame emitted by the flash and rendering, it is possible to give the influence of the flash to the foreground and / or the background.

FIG. 7 shows the image coordinates of the flash derived by the flash information acquisition unit 103 of the second embodiment and the three-dimensional position of the light emission of the flash derived by the flash information processing unit 106.

FIG. 7A shows an example of an captured image (image data) acquired by the imaging apparatus 130 of the second embodiment. FIG. 7A shows only a cameraman who owns a camera having a flash 701, but as in the first embodiment, a plurality of image pickup devices 130 are arranged around the subject, and a wrestler or a wrestler or a wrestler The image of the ring is taken. Further, FIG. 7A shows images 711 to 715 taken from five image pickup devices 130 among all the image pickup devices 130 arranged. The captured images 711 to 715 are images captured when the flash 701 emits light, and the flash light is captured in each of the captured images.

Next, FIG. 7B shows an example of the image coordinates of the flash acquired by the flash information acquisition unit 103 from the image captured by the image pickup apparatus 130. When the flash fires, the brightness of the flash position becomes sharply higher than that of the previous frame. The flash information acquisition unit 103 calculates the brightness change of each pixel of the previous frame and the current frame. For example, the binary image 720 of FIG. 7B is derived by setting the coordinates whose brightness change is equal to or greater than the threshold value to white and the coordinates whose luminance change is equal to or lower than the threshold value to black. In the binary image 720, the pixel whose brightness suddenly changes due to the flash emission is represented by the white pixel 721, which is data indicating the image coordinates of the flash emission position. The flash information acquisition unit 103 sends the binary image 720 to the flash information processing unit 106 as flash emission position information.

The flash information processing unit 106 uses a plurality of binary images 720 showing the flash emission positions provided by the plurality of flash information acquisition units 103, and obtains a three-dimensional position of the flash emission by a visual volume crossing method or the like. FIG. 7C shows an example of the three-dimensional position 730 of the flash derived by the flash information processing unit 106. Although the binary image is used as the data indicating the flash coordinates in the image coordinates, the present invention is not limited to this, and for example, the image coordinates of the pixels may be used. In this case, the flash information processing unit 106 derives the three-dimensional position 730 of the flash emission from a plurality of image coordinates indicating the emission position of the flash by using the stereo imaging method. The three-dimensional position information indicating the three-dimensional position 730 of the derived flash light emission is provided to the virtual viewpoint image generation unit 105.

The virtual viewpoint image generation unit 105 adds a light source based on the three-dimensional position information of the flash provided by the flash information processing unit 106, and renders for the virtual viewpoint image. As the light source, the virtual viewpoint image generation unit 105 may use, for example, a point light that radiates light from one point, or a spotlight that can specify the direction (angle designation) of the light ray from one point. In the case of a point light light source, the light source is set according to the three-dimensional position information of the flash emission output by the flash information acquisition unit 103. If you want to reproduce the flash light more faithfully, a spotlight light source is used. However, when a spotlight light source is used, it is necessary to derive an angle indicating the direction of the light beam from the flash in addition to the light emission position (three-dimensional position information) of the flash.

FIG. 7D shows an example of the angle of the spotlight light source. The three-dimensional position 730 in FIG. 7D shows the light emitting position of the flash derived by the above-mentioned processing. The direction (angle) of the spotlight light source can be set in the direction from the position of the light source to a predetermined position in the imaging space. For example, assuming that the camera owned by the cameraman is aimed at the wrestler or the ring, which is the subject, the three-dimensional coordinates indicating the part of the subject may be determined in advance. For example, the angle of the spotlight light source is a vector connecting the three-dimensional position 740 (three-dimensional coordinates) in the center of the bale, which is the subject, and the three-dimensional position 730 (three-dimensional coordinates) of the flash emission. In this way, the flash information processing unit 106 can derive the angle of the spotlight light source and add the spotlight light source.

The above process will be described with a flowchart. First, the operation of the flash information acquisition unit 103 of the second embodiment will be described by diverting the flowchart of FIG. The operation of the flash information acquisition unit 103 of the second embodiment is the same as that of the first embodiment except for the contents of the flash information transmitted in steps S405 and S407. In the second embodiment, the flash information acquisition unit 103 transmits, in step S407, information indicating no flash emission and flash information including a histogram of the luminance value of the captured image. Further, in step S405, the flash information acquisition unit 103 transmits flash information including information indicating the presence or absence of flash light emission and information indicating the light emission position of the flash.

FIG. 8 is a flowchart illustrating the operation of the flash information processing unit 106 according to the second embodiment. In step S801, the flash information processing unit 106 receives the flash information from the flash information acquisition unit 103 and the foreground texture from the silhouette image derivation unit 102. Next, in step S802, the flash information processing unit 106 confirms the presence or absence of flash light emission from the flash information received in step S801. If it is determined that there is no flash emission, the process proceeds to step S806, and the flash information processing unit 106 holds the histogram included in the flash information as a histogram at the time of non-flash emission. After that, in step S807, the flash information processing unit 106 provides the foreground texture received from the silhouette image derivation unit 102 to the virtual viewpoint image generation unit 105 as it is.

On the other hand, if it is determined in step S802 that there is flash emission, the process proceeds to step S803. In step S803, the flash information processing unit 106 corrects the foreground texture received in step S801 by using the histogram at the time of non-emission saved in step S806, and generates a foreground texture in which the influence of the flash is reduced or eliminated. .. In step S804, the flash information processing unit 106 calculates the three-dimensional position of the light emitting position of the flash included in the flash information. In step S805, the flash information processing unit 106 provides the virtual viewpoint image generation unit 105 with the three-dimensional position calculated in step S804. Then, in step S807, the foreground texture corrected in step S803 is provided to the virtual viewpoint image generation unit 105. In step 8608, the flash information processing unit 106 determines whether or not the processing of all the frames is completed. If the processing of all frames is completed, this processing ends, and if the processing of all frames is not completed, the processing proceeds to step S801.

The virtual viewpoint image generation unit 105 is based on the light emitting position provided by the flash information processing unit 106, the foreground texture, the background data stored in the background data storage unit 107, and the position of the virtual camera designated by the operation device 110. , Generate a virtual viewpoint image. At this time, the brightness of the flash (light source) may be predetermined with reference to the brightness of a general flash, or the brightness histogram acquired by the flash information acquisition unit 103 from the captured image at the time of flash emission is used. The brightness of the light source may be set.

As described above, according to the second embodiment, it is possible to derive the light emitting position of the flash in the three-dimensional space and reproduce the flash at the time of rendering. As a result, it is possible to suppress the generation of an unnatural virtual viewpoint image, and it is possible to provide a virtual viewpoint image that does not give a sense of discomfort to the viewer without impairing the sense of presence.

According to each of the above-described embodiments, a natural virtual viewpoint image is generated even when the imaging environment (illumination conditions) is suddenly changed by a flash or the like. In addition, since the generation of an unnatural virtual viewpoint image that blinks white under the influence of the flash only in the foreground, which is the subject, is suppressed, a virtual viewpoint image that does not impair the viewer's sense of presence and does not give a sense of discomfort is provided. It becomes possible.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The present invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the present invention. Therefore, in order to publicize the scope of the present invention, the following claims are attached.

1: Virtual viewpoint image generation system, 100: Image processing device, 101: Image input unit, 102: Silhouette image derivation unit, 103: Flash information acquisition unit, 104: Three-dimensional shape derivation unit, 105: Virtual viewpoint image generation unit, 106: Flash information processing unit, 110: Operation device, 120: Display device, 130: Image pickup device

Claims (20)

The three-dimensional shape information of the foreground, which is the subject based on the plurality of captured images acquired by the plurality of imaging devices, the color information of the foreground, and the background 3 in the imaging space of the plurality of imaging devices, which is different from the foreground. A generation means for generating a virtual viewpoint image based on the three-dimensional shape information, the background color information, and the designated virtual viewpoint.
A determination means for determining whether or not a change exceeding a predetermined reference has occurred in the lighting conditions based on at least one of the plurality of captured images.
When it is determined by the determination means that the lighting conditions have changed, the foreground color information and the background are brought closer to each other so that the lighting conditions of the foreground color information and the background color information used by the generation means are brought closer to each other. A virtual viewpoint image generation system characterized by having an adjusting means for adjusting at least one of the color information of the above. The virtual according to claim 1, wherein the determination means determines whether or not a change has occurred in the lighting conditions based on a change in the average luminance value of the captured image acquired by the plurality of imaging devices. Viewpoint image generation system. The determination means is the illumination when the difference between the average luminance value of the captured image of the current frame to be determined and the average luminance value of the captured images of a predetermined number of frames before the current frame exceeds the threshold value. The virtual viewpoint image generation system according to claim 2, wherein it is determined that a change has occurred in the conditions. The determination means determines that the lighting condition has changed based on the number of the captured images determined that the lighting condition has changed among the plurality of captured images acquired by the plurality of imaging devices. The virtual viewpoint image generation system according to any one of claims 1 to 3, which is characterized. The determination means is characterized in that it determines whether or not the lighting conditions have changed by using an image taken from an image pickup device closest to the virtual viewpoint among the plurality of image pickup devices. The virtual viewpoint image generation system according to any one of 1 to 3. The virtual viewpoint image according to any one of claims 1 to 5, wherein the adjusting means adjusts the color information of the background so as to approach the characteristics of the captured image when the change is detected. Generation system. The virtual viewpoint image generation system according to claim 6, wherein the adjusting means uses a histogram of the brightness of the captured image when the change is detected by the determination means as the characteristic. The adjusting means according to claim 1 to 7, wherein the adjusting means adjusts the color information of the foreground so that the characteristic of the color information of the foreground approaches the characteristic of the captured image when the change is not detected. The virtual viewpoint image generation system according to any one of the items. The virtual viewpoint image generation system according to claim 8, wherein the adjusting means uses a histogram of the brightness of the captured image when the change is not detected by the determination means as the characteristic. Further provided with specific means for identifying the position of the light source in the three-dimensional space that causes the change in the lighting condition,
The virtual viewpoint according to any one of claims 1 to 9, wherein the generation means arranges a light source specified by the specific means in the three-dimensional space to render the virtual viewpoint image. Image generation system. The virtual viewpoint image generation system according to claim 10, wherein the adjusting means adjusts the color information of the foreground to the color information corresponding to the lighting condition before the change occurs. The virtual viewpoint image generation system according to claim 10, wherein the generation means renders the virtual viewpoint image using the light source as a point light light source. The virtual viewpoint image generation system according to claim 10, wherein the generation means renders the virtual viewpoint image using the light source as a spotlight light source. The virtual viewpoint image generation system according to claim 13, wherein the generation means sets the direction of the spotlight light source in a direction from the position of the light source toward a predetermined position in the imaging space. An acquisition means for acquiring three-dimensional shape information of the foreground, which is a subject, and color information of the foreground from an image captured by an image pickup device.
Based on the captured image, a determination means for determining whether or not a change exceeding a predetermined reference has occurred in the lighting conditions, and a determination means.
It has a three-dimensional shape of the foreground acquired by the acquisition means, color information of the foreground, and a transmission means for transmitting the determination result of the determination means to an external device.
The image processing device is characterized in that, when it is determined by the determination means that a change has occurred, the transmission means further transmits information representing the characteristics of the luminance of the captured image. The three-dimensional shape information of the foreground, which is the subject based on the plurality of captured images acquired by the plurality of imaging devices, the color information of the foreground, and the background 3 in the imaging space of the plurality of imaging devices, which is different from the foreground. A generation means for generating a virtual viewpoint image based on the three-dimensional shape information, the background color information, and the designated virtual viewpoint.
When it is notified that the lighting conditions of the imaging space have changed beyond a predetermined reference, the foreground color information used by the generation means and the background color information are brought closer to each other. An image generation device comprising: an adjusting means for adjusting at least one of the color information of the background and the color information of the background. The three-dimensional shape information of the foreground, which is the subject based on the plurality of captured images acquired by the plurality of imaging devices, the color information of the foreground, and the background 3 in the imaging space of the plurality of imaging devices, which is different from the foreground. A generation process for generating a virtual viewpoint image based on the three-dimensional shape information, the background color information, and the designated virtual viewpoint.
A determination step of determining whether or not a change exceeding a predetermined reference has occurred in the lighting conditions based on at least one of the plurality of captured images.
When it is determined by the determination step that the lighting conditions have changed, the foreground color information and the foreground color information and the above so as to bring the lighting conditions of the foreground color information and the background color information used in the generation step closer to each other. A control method for a virtual viewpoint image generation system, which comprises an adjustment step of adjusting at least one of background color information. The acquisition process of acquiring the three-dimensional shape information of the foreground, which is the subject, and the color information of the foreground from the image captured by the image pickup device.
A determination step for determining whether or not a change exceeding a predetermined standard has occurred in the lighting conditions based on the captured image, and a determination step.
It has a three-dimensional shape of the foreground acquired by the acquisition step, color information of the foreground, and a transmission step of transmitting the determination result of the determination step to an external device.
A control method for an image processing apparatus, characterized in that, in the transmission step, when it is determined that a change has occurred in the determination step, information representing the characteristics of the luminance of the captured image is further transmitted. The three-dimensional shape information of the foreground, which is the subject based on the plurality of captured images acquired by the plurality of imaging devices, the color information of the foreground, and the background 3 in the imaging space of the plurality of imaging devices, which is different from the foreground. A generation process for generating a virtual viewpoint image based on the three-dimensional shape information, the background color information, and the designated virtual viewpoint.
When it is notified that the lighting conditions of the imaging space have changed beyond a predetermined reference, the lighting conditions of the foreground color information and the background color information used in the generation step are brought close to each other. A control method for an image generator, comprising: an adjustment step of adjusting at least one of a foreground color information and the background color information. A program for causing a computer to execute each step of the control method according to any one of claims 17 to 19.

Images