WO2023026543A1

WO2023026543A1 - Information processing device, information processing method, and program

Info

Publication number: WO2023026543A1
Application number: PCT/JP2022/011543
Authority: WO
Inventors: 久之館野
Original assignee: ソニーグループ株式会社
Priority date: 2021-08-27
Filing date: 2022-03-15
Publication date: 2023-03-02

Abstract

An information processing device according to the present invention comprises: an acquisition unit that acquires an infrared (IR) image, which is a captured image obtained by irradiating a subject with infrared light, including a visible light component and an IR component; an extraction unit that extracts information of the IR component from the IR image; and an image processing unit that, on the basis of the information on the IR component, subjects the captured image of the subject to image processing pertaining to luminance or brightness.

Description

Information processing device, information processing method, and program

The present disclosure relates to an information processing device, an information processing method, and a program.

Opportunities for individuals to use cameras are increasing. For example, in recent years, the demand for remote work, video conferences, and video calls has increased, and opportunities to take close-up shots of faces have increased. Also, in recent years, there has been an increase in demand for personal video production, such as production of videos to be posted on video sharing sites.

JP 2016-6627 A

　In order to improve the quality of video recording, it is desirable to use lighting. However, the use of lighting may cause discomfort to the user. For example, with lighting, the user must put up with the glare while speaking to the camera.

Therefore, the present disclosure proposes an information processing device, an information processing method, and a program that enable high-quality video shooting.

It should be noted that the above problem or object is only one of the multiple problems or objects that can be solved or achieved by the multiple embodiments disclosed herein.

In order to solve the above problems, an information processing apparatus according to one embodiment of the present disclosure acquires an IR image that is a captured image obtained by irradiating an object with infrared light and that includes a visible light component and an IR component. an acquisition unit that extracts IR component information from the IR image; and an image processing unit that performs image processing related to brightness or brightness of the captured image of the target based on the IR component information. .

FIG. 11 is a diagram showing a state in which visible light is used for moving image shooting; FIG. 10 is a diagram showing how a moving image is captured using an IR light; It is a figure which shows the outline|summary of the image processing of this embodiment. 1 is a diagram illustrating a configuration example of a server according to an embodiment of the present disclosure; FIG. It is a figure which shows an example of an infrared illumination part. It is a figure which shows the frequency characteristic of the filter with which an imaging part is provided. FIG. 4 is a diagram for explaining basic method 1; 4 is a flowchart showing image output processing for realizing basic method 1; FIG. 10 is a diagram for explaining basic technique 2; 9 is a flowchart showing image output processing for realizing basic method 2; It is a figure for demonstrating an expansive method. It is a figure for demonstrating an expansive method. 10 is a flowchart showing image output processing for realizing an advanced method; It is a flowchart which shows an estimation process. 1 is a diagram showing an example of a photographing studio of the live-action volumetric photographing system of this embodiment; FIG. FIG. 3 is a diagram showing a processing example of the information processing device 10 in the live-action volumetric imaging system; FIG. 3 is a diagram showing a state in which a plurality of visible light lights and a plurality of IR lights are arranged omnidirectionally;

Below, embodiments of the present disclosure will be described in detail based on the drawings. In addition, in each of the following embodiments, the same parts are denoted by the same reference numerals, thereby omitting redundant explanations.

Each of one or more embodiments (including examples and modifications) described below can be implemented independently. On the other hand, at least some of the embodiments described below may be implemented in combination with at least some of the other embodiments as appropriate. These multiple embodiments may include novel features that differ from each other. Therefore, these multiple embodiments can contribute to solving different purposes or problems, and can produce different effects.

Also, the present disclosure will be described according to the order of items shown below.
1. overview
2. Configuration of information processing apparatus 3 . Operation of Information Processing Apparatus 3-1. Basic method 1
3-2. Basic method 2
3-3. Advanced method 4. Application to live-action volumetric 4-1. Issue 4-2. Example 4-3. Other examples5. Modification 5-1. Application to products and services 5-2. Other Modifications6. Conclusion

<<1. Overview＞＞
In recent years, there have been increasing opportunities for individuals to use cameras. In particular, in recent years, opportunities to use remote work, video conferences, video phones, etc. have increased, and opportunities to take close-up photographs of users have increased.

　In order to improve the quality of video recording, it is desirable to use lighting. FIG. 1 is a diagram showing how a visible light is used for moving image shooting. In the example of FIG. 1, a ring-shaped visible light (LED (Light Emitting Diode) light in the example of FIG. 1) is used for illumination. The impression of the image changes depending on how the lighting is applied, so it is difficult to decide how to apply the lighting. Although it is desirable to illuminate the user as evenly as possible from the same direction as the camera, it is difficult to install such lighting. In addition, when the user wears glasses, there is a problem that the light is reflected on the glasses when the user is exposed to the light from the front.

Even with equipment such as lighting, users have to put up with dazzling lighting while talking to the camera. For example, even though the user just wants to make a video call casually, he has to invest in the equipment, have a hard time setting it up, and have to endure and keep lighting his face. This is extremely painful for the user.

Therefore, in this embodiment, a camera capable of simultaneously acquiring visible light and infrared light is combined with an IR (Infrared) light that outputs invisible infrared light. FIG. 2 is a diagram showing how moving images are captured using an IR light. Then, the information processing apparatus of the present embodiment performs image processing on the captured image based on the infrared light irradiation information as if the subject were irradiated with visible light. This realizes simple relighting that is effective for close-up scenes.

FIG. 3 is a diagram showing an overview of image processing according to this embodiment. An information processing apparatus obtains an IR image obtained by irradiating an object (eg, a user and surrounding objects) with infrared light. An IR image is a captured image containing a visible light component and an IR component obtained by irradiating an object with infrared light. Then, the information processing device performs image processing related to brightness or brightness on the captured image of the target based on the information of the IR component extracted from the IR image. As a result, the user can shoot moving images with stable lighting without painful or complicated lighting adjustments.

For infrared light irradiation, it is desirable to use an IR ring light in which infrared light emitting elements are arranged in a ring around the lens. In addition, a polarizing filter may be used to prevent reflection of light on the glasses.

The outline of the present embodiment has been described above, and the information processing apparatus 10 of the present embodiment will be described in detail below.

<<2. Configuration of Information Processing Device >>
First, the configuration of the information processing device 10 will be described.

The information processing device 10 is a computer used by the user for video shooting. The information processing device 10 is typically a personal computer, but is not limited to a personal computer. For example, the information processing device 10 may be a mobile terminal such as a mobile phone, a smart device (smartphone or tablet), a PDA (Personal Digital Assistant), or a notebook PC. Also, the information processing device 10 may be a wearable device such as a smart watch.

The information processing apparatus 10 may also be an xR device such as an AR (Augmented Reality) device, a VR (Virtual Reality) device, or an MR (Mixed Reality) device. At this time, the xR device may be a glasses-type device such as AR glasses or MR glasses, or a head-mounted device such as a VR head-mounted display.

The information processing device 10 may also be a portable IoT (Internet of Things) device. Also, the information processing apparatus 10 may be a motorcycle, a mobile relay vehicle, or the like equipped with a communication device such as an FPU (Field Pickup Unit). Further, the information processing device 10 may be a server device such as a PC server, a midrange server, or a mainframe server. In addition, the information processing apparatus 10 can employ any form of computer.

FIG. 4 is a diagram showing a configuration example of the information processing device 10 according to the embodiment of the present disclosure. The information processing apparatus 10 includes a communication section 11 , a storage section 12 , a control section 13 , an output section 14 , an infrared illumination section 15 , a synchronization signal generation section 16 and an imaging section 17 . Note that the configuration shown in FIG. 4 is a functional configuration, and the hardware configuration may differ from this. Also, the functions of the information processing apparatus 10 may be distributed and implemented in a plurality of physically separated configurations.

The communication unit 11 is a communication interface for communicating with other devices. For example, the communication unit 11 is a LAN (Local Area Network) interface such as a NIC (Network Interface Card). Also, the communication unit 11 may be a device connection interface such as USB (Universal Serial Bus). The communication unit 11 may be a wired interface or a wireless interface. The communication unit 11 communicates with an external device under the control of the control unit 13 .

The storage unit 12 is a data readable/writable storage device such as a DRAM (Dynamic Random Access Memory), an SRAM (Static Random Access Memory), a flash memory, a hard disk, or the like. The storage unit 12 functions as storage means of the information processing device 10 . For example, the storage unit 12 functions as a frame buffer for moving images captured by the imaging unit 17 .

The control unit 13 is a controller that controls each unit of the information processing device 10 . The control unit 13 is implemented by a processor such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), or the like. For example, the control unit 13 is implemented by the processor executing various programs stored in the storage device inside the information processing apparatus 10 using a RAM (Random Access Memory) or the like as a work area. The control unit 13 may be realized by an integrated circuit such as ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). CPUs, MPUs, GPUs, ASICs, and FPGAs can all be considered controllers.

The control unit 13 includes an acquisition unit 131 , an extraction unit 132 , an image processing unit 133 , an output control unit 134 , a learning unit 135 and an estimation unit 136 . Each block (obtaining unit 131 to estimating unit 136) constituting the control unit 13 is a functional block indicating the function of the control unit 13. FIG. These functional blocks may be software blocks or hardware blocks. For example, each of the functional blocks described above may be one software module realized by software (including microprograms), or may be one circuit block on a semiconductor chip (die). Of course, each functional block may be one processor or one integrated circuit. The control unit 13 may be configured in functional units different from the functional blocks described above. The configuration method of the functional blocks is arbitrary.

It should be noted that the control unit 13 may be configured in functional units different from the functional blocks described above. Also, some or all of the blocks (acquisition unit 131 to estimation unit 136) that make up the control unit 13 may be operated by another device. The operation of each block constituting the control unit 13 will be described later.

The output unit 14 is a device that performs various outputs such as sound, light, vibration, and images to the outside. The output unit 14 performs various outputs to the user under the control of the control unit 13 . Note that the output unit 14 includes a display device (display unit) that displays various types of information. The display device is, for example, a liquid crystal display or an organic EL display. Note that the output unit 14 may be a touch panel display device. In this case, the output section 14 also functions as an input section.

The infrared illumination unit 15 is an IR light (IR illumination light source) that outputs invisible infrared light. The upper limit of the wavelength of light that can be perceived by the human eye is 760-830 nm. Wavelengths such as 850 nm or 940 nm are the major IR illumination sources on the market. Therefore, the infrared illuminator 15 is typically an IR light that outputs infrared light with a wavelength of 850 nm or 940 nm. However, the infrared illuminator 15 is not limited to an IR light that outputs infrared light with a wavelength of 850 nm or 940 nm. The infrared illuminator 15 may be capable of outputting infrared light of other wavelengths. FIG. 5 is a diagram showing an example of the infrared illuminator 15. As shown in FIG. It is desirable that the infrared illuminator 15 be a ring light in order to clearly project the face. In the example of FIG. 5, the infrared illumination unit 15 is an IR light in which IR light emitting elements are arranged in a ring shape around a lens.

The synchronizing signal generating unit 16 is a synchronizing signal generator that generates a synchronizing signal for synchronizing the blinking period of the infrared illumination unit 15 and the frame period of the video (moving image) captured by the imaging unit 17 . The synchronizing signal generator 16 outputs a synchronizing signal under the control of the control unit 13 .

The imaging unit 17 is a conversion unit that converts an optical image into an electrical signal. The imaging unit 17 includes, for example, an image sensor and a signal processing circuit that processes analog pixel signals output from the image sensor, and converts light entering from the lens into digital data (image data). An image captured by the imaging unit 17 is not limited to a video (moving image), and may be a still image. Note that the imaging unit can be rephrased as a camera.

The imaging unit 17 of this embodiment is a camera (hereinafter also referred to as an IR camera) that can simultaneously acquire visible light and infrared light (IR light). An IR camera can be realized by removing the IR cut filter normally included in commercially available cameras. However, in order to eliminate the influence (=noise) of infrared rays of wavelengths other than IR light on the sensor, it is desirable that the imaging unit 17 be provided with an IR cut filter and a bandpass filter having characteristics as shown in FIG. . FIG. 6 is a diagram showing frequency characteristics of a filter included in the imaging unit 17. As shown in FIG. In the example of FIG. 6, the imaging unit 17 is configured to detect infrared light with a wavelength of 850 nm. However, if the infrared illumination unit 15 is a light source that outputs infrared light with a wavelength of 940 nm, the imaging unit 17 may be configured to detect infrared light with a wavelength of 940 nm.

<<3. Operation of Information Processing Apparatus >>
The configuration of the information processing apparatus 10 has been described above. Next, the operation of the information processing apparatus 10 having such a configuration will be described.

<3-1. Basic method 1>
FIG. 7 is a diagram for explaining basic method 1. FIG. An outline of basic method 1 will be described below with reference to FIG.

The information processing device 10 operates the infrared illumination unit 15 and the imaging unit 17 according to the user's operation. At this time, the information processing apparatus 10 blinks the infrared illuminator 15 while synchronizing with the image (moving image) captured by the imaging unit 17 . For example, the information processing device 10 blinks the infrared illumination unit 15 while synchronizing with the frame period of the video (moving image) captured by the imaging unit 17 . Thereby, the information processing apparatus 10 can acquire a visible light image and an IR image in a time division manner in synchronization with the blinking cycle of infrared light. More specifically, the information processing apparatus 10 can acquire the image of the frame when the infrared light is not irradiated as the visible light image and the image of the frame when the infrared light is irradiated as the IR image. In the example of FIG. 7, the frame when the IR light is OFF is the visible light image, and the frame when the IR light is OFF is the IR image. Here, an IR image is a captured image containing a visible light component and an IR component obtained by irradiating an object (the user and surrounding objects in the example of FIG. 7) with infrared light.

Then, the information processing device 10 extracts IR component information (hereinafter referred to as IR component information) from the IR image. This IR component information indicates from which direction the infrared light from the infrared illumination unit 15 hits the object. At this time, the information processing device 10 may acquire the difference between the visible light image and the IR image as IR component information. In the example of FIG. 7, the information processing apparatus 10 calculates the difference between two consecutive frames of images (a visible light image and an IR image) starting from a frame at a timing when infrared light is not irradiated (IR light OFF frame). Obtained as ingredient information. By using the difference between the visible light image and the IR image as the IR component information, the information processing device 10 can detect the IR component of light that always exists, such as the light of a room light (for example, a fluorescent lamp), from the IR component information. However, it is possible to purely extract information on the influence of infrared light on the target due to lighting of the infrared illumination unit 15 as IR component information.

Then, the information processing device 10 performs image processing related to brightness or brightness on the captured image based on the IR component information. For example, based on the IR component information, the information processing device 10 performs image processing on the next frame image (visible light image) of the two continuous frames (visible light image and IR image) used to extract the IR component information. . In the example of FIG. 7, the information processing apparatus 10 rewrites the luminance (L) information in the HSL color space of the visible light image based on the IR component information. More specifically, the information processing device 10 converts the visible light image from RGB to HSL, and maps the intensity of the IR component to the luminance (L) of the visible light image in the HSL color space. Here the mapping may be a complete replacement or blending with the original luminance. In the above example, the color space used in the visible light image (input image) is RGB, but the color space used in the visible light image (input image) is not limited to RGB. For example, the color space used in the visible light image (input image) may be a color space other than RGB, such as YUV. The YUV color space is a color space that expresses colors with luminance (Y) and color difference components (U, V). In addition, the color space used for the visible light image (input image) can be appropriately changed according to the color space of the image output by the camera. Note that if the color space used in the visible light image (input image) is a color space having an axis capable of mapping IR components such as brightness and lightness, this color space conversion step can be omitted.

In addition, in the case of a scene with motion, a gap occurs between frames with simple mapping. In this case, the information processing device 10 may blur the edge of the IR component. For example, the information processing apparatus 10 performs edge blurring processing on the difference image as IR component information, and performs image processing on the next frame image of two continuous frames based on the edge-blurred difference image. . Thereby, the information processing apparatus 10 can generate an image with little discomfort even in a scene with motion.

Further, the information processing apparatus 10 may correct the IR component information based on motion prediction between frames, and perform image processing on the next frame image of two continuous frames based on the corrected IR component information. . For example, the information processing device 10 may acquire the optical flow between adjacent frames of the visible light image, transform the IR component, and map it. This also makes it possible to generate an image with little sense of incongruity.

Then, the information processing device 10 converts the image from HSL to RGB and outputs it to the output unit 14 .

In the example of FIG. 7, the information processing device 10 rewrites the luminance (L) information of the captured image in the HSL color space based on the IR component information. However, the information processing apparatus 10 may rewrite the brightness (V) information of the captured image in the HSV color space based on the IR component information. The information processing device 10 may rewrite the luminance (Y) information of the captured image in the YCoCg color space based on the IR component information. Of course, the information processing device 10 may rewrite the values in the RGB color space based on the IR component information. The color space used by the information processing apparatus 10 for image processing is not limited to the color space described above. The color space of the final output image is not limited to RGB depending on the application, and may be YUV, for example.

The outline of the basic method 1 has been described above, and the image output processing for realizing the basic method 1 will be described below. FIG. 8 is a flowchart showing image output processing for realizing basic method 1. FIG. The following processing is executed by the control unit 13 of the information processing device 10 . The control unit 13 starts image output processing when the user starts imaging (for example, a video conference).

First, the control unit 13 activates the imaging unit 17 (step S101). As described above, the imaging unit 17 is an IR camera that can simultaneously acquire visible light and infrared light (IR light). Then, the control unit 13 blinks the infrared illumination unit 15 while synchronizing with the frame cycle of the video (moving image) captured by the imaging unit 17 (step S102). As described above, the infrared illuminator 15 is an IR light that outputs invisible infrared light.

Subsequently, the acquisition unit 131 of the information processing device 10 acquires the image captured by the imaging unit 17 . Since the infrared light is blinking in synchronization with the frame period of the video, the acquisition unit 131 alternately acquires the visible light image and the IR image (step S103). Here, the IR image is a captured image including not only the IR component under the influence of the infrared light emitted by the infrared illuminator 15 but also the visible light component.

The extraction unit 132 of the information processing device 10 extracts IR component information from the IR image (step S104). Specifically, the extraction unit 132 acquires the difference between the visible light image and the IR image as IR component information. In basic method 1, the extraction unit 132 acquires, as IR component information, the difference between two consecutive frames of images (a visible light image and an IR image) starting from the frame at which the IR light is turned off.

Subsequently, the image processing unit 133 of the information processing device 10 performs image processing related to brightness or brightness of the captured image based on the IR component information (step S105). For example, based on the IR component information, the information processing device 10 performs image processing on the next frame image (visible light image) of the two continuous frames (visible light image and IR image) used to extract the IR component information. . For example, the image processing unit 133 rewrites the luminance information of the visible light image based on the IR component information.

Then, the output control unit 134 of the information processing device 10 outputs the captured image subjected to the image processing to the output unit 14 (step S106).

After that, the control unit 13 of the information processing device 10 determines whether or not the shooting has ended (step S107). If the shooting has not ended (step S107: No), the control unit 13 returns the process to step S103. If the shooting has ended (step S107: Yes), the control unit 13 stops the operations of the imaging unit 17 and the infrared illumination unit 15 (step S108). When the operations of the imaging unit 17 and the infrared illumination unit 15 are stopped, the control unit 13 ends the image output processing.

According to this method, the information processing apparatus 10 performs image processing so that it appears as if visible light is hitting an object (for example, a user) based on irradiation information (that is, IR component information) of invisible infrared light. It is carried out. As a result, the user can shoot moving images with stable lighting without being dazzled.

<3-2. Basic method 2>
In basic method 1, the information processing apparatus 10 performs image processing on the next frame image (visible light image) of two continuous frames (visible light image and IR image). However, in the case of a scene with motion, this method may result in an unnatural image after image processing. Therefore, in basic method 2, the frame used for generating the difference image is used as the frame to be subjected to image processing, so that even in a scene with motion, the image does not look unnatural.

FIG. 9 is a diagram for explaining basic method 2. The outline of basic method 2 will be described below with reference to FIG.

The information processing device 10 operates the infrared illumination unit 15 and the imaging unit 17 according to the user's operation. At this time, the information processing apparatus 10 blinks the infrared illuminator 15 while synchronizing with the image (moving image) captured by the imaging unit 17 . For example, the information processing device 10 blinks the infrared illumination unit 15 while synchronizing with the frame period of the video (moving image) captured by the imaging unit 17 . Thereby, the information processing apparatus 10 can acquire a visible light image and an IR image in a time division manner in synchronization with the blinking cycle of infrared light.

Then, the information processing device 10 extracts IR component information from the IR image. At this time, the information processing device 10 acquires the difference between the visible light image and the IR image as IR component information. In the example of FIG. 9, the information processing apparatus 10 converts the difference between two consecutive frames of images (visible light image and IR image) starting from the frame at the timing when the infrared light is irradiated (IR light ON frame) to the IR component. obtained as information.

Then, the information processing device 10 performs image processing related to brightness or brightness on the captured image based on the IR component information. For example, based on the IR component information, the information processing device 10 performs image processing on the last frame image (visible light image) of two consecutive frames (IR image and visible light image) used to extract the IR component information. conduct. In the example of FIG. 9, the information processing apparatus 10 rewrites the luminance (L) information in the HSL color space of the visible light image based on the IR component information. Here, the HSL color space is a color space that expresses colors with three components of hue (Hue), saturation (Saturation), and brightness (Lightness).

Note that in the example of FIG. 9, the information processing device 10 rewrites the information of the luminance (L) in the HSL color space of the captured image based on the IR component information. However, the information processing apparatus 10 may rewrite the brightness (V) information of the captured image in the HSV color space based on the IR component information. Here, the HSV color space is a color space that expresses colors with three components of hue (Hue), saturation (Saturation/Chroma), and brightness (Value/Brightness). The information processing device 10 may rewrite the luminance (Y) information of the captured image in the YCoCg color space based on the IR component information. The YCoCg color space is a color space that expresses colors by luminance (Y) and color difference components (Co (darkness of orange) and Cg (darkness of green)). Of course, the information processing device 10 may rewrite the values in the RGB color space based on the IR component information. The color space used by the information processing apparatus 10 for image processing is not limited to the color space described above.

The outline of the basic method 2 has been described above, and the image output processing for realizing the basic method 2 will be described below. FIG. 10 is a flowchart showing image output processing for realizing basic method 2. FIG. The following processing is executed by the control unit 13 of the information processing device 10 . The control unit 13 starts image output processing when the user starts imaging (for example, a video conference).

First, the control unit 13 activates the imaging unit 17 (step S201). Then, the control unit 13 blinks the infrared illumination unit 15 while synchronizing with the frame period of the video (moving image) captured by the imaging unit 17 (step S202).

Subsequently, the acquisition unit 131 of the information processing device 10 acquires the image captured by the imaging unit 17 . Since the infrared light is blinking in synchronization with the frame cycle of the video, the acquisition unit 131 alternately acquires the visible light image and the IR image (step S203).

The extraction unit 132 of the information processing device 10 extracts IR component information from the IR image (step S204). Specifically, the extraction unit 132 acquires the difference between the visible light image and the IR image as IR component information. In basic method 2, the extraction unit 132 acquires, as IR component information, the difference between two consecutive frames of images (an IR image and a visible light image) starting from the frame at which the IR light is ON.

Subsequently, the image processing unit 133 of the information processing device 10 performs image processing related to luminance or brightness of the captured image based on the IR component information (step S205). For example, based on the IR component information, the information processing device 10 performs image processing on the last frame image (visible light image) of the two consecutive frames (IR image and visible light image) used to extract the IR component information. conduct. For example, the image processing unit 133 rewrites the luminance information of the visible light image based on the IR component information.

Then, the output control unit 134 of the information processing device 10 outputs the captured image subjected to the image processing to the output unit 14 (step S206).

After that, the control unit 13 of the information processing device 10 determines whether or not the shooting has ended (step S207). If the shooting has not ended (step S207: No), the control unit 13 returns the process to step S203. If the shooting has ended (step S207: Yes), the control unit 13 stops the operations of the imaging unit 17 and the infrared illumination unit 15 (step S208). When the operations of the imaging unit 17 and the infrared illumination unit 15 are stopped, the control unit 13 ends the image output processing.

According to this method, the information processing apparatus 10 performs image processing on one of the frames used to generate the IR component information (difference image). The time lag with the target image is small. Therefore, the user can obtain an image with less discomfort.

<3-3. Advanced method>
In basic methods 1 and 2, the information processing device 10 performs image processing on the captured image based on the IR component information. However, the information processing apparatus 10 generates a learning model by learning based on the image before image processing and the image after image processing, and uses the generated learning model to estimate the image after image processing from the captured image. may Accordingly, the information processing apparatus 10 can acquire an image as if the user were illuminated without irradiating the user with infrared light.

　Figures 11 and 12 are diagrams for explaining the advanced method. FIG. 11 is a diagram showing processing up to completion of learning of the learning model, and FIG. 12 is a diagram showing processing after completion of learning of the learning model. An outline of the advanced method will be described below with reference to FIGS. 11 and 12. FIG.

First, referring to FIG. 11, the processing up to the completion of learning of the learning model will be described. The information processing apparatus 10 operates the infrared illumination section 15 and the imaging section 17 according to the user's operation. At this time, the information processing apparatus 10 blinks the infrared illuminator 15 while synchronizing with the image (moving image) captured by the imaging unit 17 . Thereby, the information processing apparatus 10 can acquire a visible light image and an IR image in a time division manner in synchronization with the blinking cycle of infrared light. Then, the information processing device 10 extracts IR component information from the IR image. Then, the information processing apparatus 10 performs image processing regarding brightness or brightness on the captured image based on the IR component information. Then, the information processing device 10 outputs the image after image processing to the output unit 14 .

In parallel with image processing, the information processing device 10 learns a learning model based on the image before image processing and the image after image processing. A learning model is, for example, a model for learning the relationship between an image before image processing and an image after image processing. The information processing apparatus 10 learns the learning model so as to minimize the difference between the image before image processing and the image after image processing.

A learning model is, for example, a machine learning model such as a neural network model. A neural network model is composed of layers called an input layer containing a plurality of nodes, an intermediate layer (or hidden layer), and an output layer, and each node is connected via edges. Each layer has a function called activation function, and each edge is weighted. A learning model has one or more intermediate layers (or hidden layers). When the learning model is a neural network model, learning the learning model means, for example, setting the number of intermediate layers (or hidden layers), the number of nodes in each layer, or the weight of each edge.

Here, the neural network model may be a model based on deep learning. In this case, the neural network model may be a model called DNN (Deep Neural Network). Also, the neural network model may be a model called a CNN (Convolution Neural Network), RNN (Recurrent Neural Network), or LSTM (Long Short-Term Memory). Of course, neural network models are not limited to these forms of models.

Also, learning models are not limited to neural network models. For example, the learning model may be a model based on reinforcement learning. In reinforcement learning, actions (settings) that maximize value are learned through trial and error. Alternatively, the learning model may be a logistic regression model.

The learning model may consist of multiple models. For example, a learning model may consist of multiple neural network models. More specifically, the learning model may consist of multiple neural network models selected from, for example, CNN, RNN, and LSTM. When a learning model is composed of multiple neural network models, these multiple neural network models may be in a dependent relationship or in a parallel relationship.

The information processing device 10 stores, in the storage unit 12, character strings, numerical values, and the like that indicate the model structure and connection coefficients as information that constitutes the learning model.

The learning model uses a pair of data of an image before image processing (captured image such as a visible light image) and an image after image processing as learning data, and acquires an image before image processing (for example, a captured image such as a visible light image). It may be a model that has learned to output an image after image processing (hereinafter referred to as an estimated image) when a captured image is input. In this case, the first learning model includes an input layer for inputting a captured image, an output layer for outputting an estimated image, and a layer other than the output layer which is one of the layers from the input layer to the output layer. 1 element and a second element whose value is calculated based on the weight of the first element and the first element. As the first element, an operation is performed based on the first element and the weight of the first element (that is, the connection coefficient), so that the estimated image is output from the output layer according to the captured image input to the input layer. , may be a model for making a computer work.

Here, it is assumed that the learning model is realized by a neural network with one or more hidden layers, such as DNN. In this case, the first element included in the learning model corresponds to any node of the input layer or intermediate layer. Also, the second element corresponds to the next node, which is a node to which the value is transmitted from the node corresponding to the first element. Also, the weight of the first element corresponds to the connection coefficient, which is the weight considered for the value transmitted from the node corresponding to the first element to the node corresponding to the second element.

Also, assume that the learning model is realized by a regression model indicated by "y=a1*x1+a2*x2+...+ai*xi". In this case, the first element included in the learning model corresponds to input data (xi) such as x1 and x2. Also, the weight of the first element corresponds to the coefficient ai corresponding to xi. Here, the regression model can be viewed as a simple perceptron with an input layer and an output layer. When each model is regarded as a simple perceptron, the first element can be regarded as a node of the input layer, and the second element can be regarded as a node of the output layer.

The information processing device 10 uses a model having an arbitrary structure, such as a neural network or a regression model, to calculate information to be output. Specifically, the learning model is set with coefficients so that an estimated image is output when a captured image (for example, a visible light image before image processing) is input. For example, the information processing apparatus 10 sets the coefficient based on the degree of similarity between the image after image processing and the value obtained by inputting the captured image (visible light image before image processing) into the learning model. The information processing apparatus 10 uses such a learning model to generate an estimated image from the captured image.

In the above example, as an example of a learning model, a model that outputs an estimated image when a captured image is input is shown. However, the learning model according to the embodiment may be a model that is generated based on results obtained by repeatedly inputting and outputting data to the learning model.

Also, when the information processing apparatus 10 performs learning or generation of output information using GAN (Generative Adversarial Networks), the learning model may be a model that constitutes part of the GAN.

The learning device that learns the learning model (for example, the learning model) may be the information processing device 10, or may be another information processing device. For example, assume that the information processing apparatus 10 learns a learning model. In this case, the information processing apparatus 10 learns the learning model and stores the learned learning model in the storage unit 12 . More specifically, the information processing apparatus 10 sets the connection coefficient of the learning model so that the learning model outputs return information when drafter information is input to the learning model.

For example, the information processing apparatus 10 inputs a captured image to a node in the input layer of the learning model, propagates the data to the output layer of the learning model by following each intermediate layer, and outputs an estimated image. Then, the information processing apparatus 10 corrects the connection coefficients of the learning model based on the difference between the estimated image actually output by the learning model and the actual image after image processing. For example, the information processing apparatus 10 may correct the connection coefficients using a technique such as back propagation. At this time, the information processing apparatus 10 may correct the connection coefficient based on the cosine similarity between the vector representing the first measured data and the vector representing the value actually output by the learning model.

Note that the information processing device 10 may learn the learning model using any learning algorithm. For example, the information processing device 10 may learn a learning model using learning algorithms such as neural networks, support vector machines, clustering, and reinforcement learning.

Next, referring to FIG. 12, the processing after the learning of the learning model is completed will be described. When the learning of the learning model is completed (for example, when a predetermined period of time has elapsed since the start of shooting or the start of learning), the information processing apparatus 10 starts generating an estimated image. At this time, the information processing apparatus 10 uses the generated learning model to estimate an image after image processing of the captured image from the newly acquired captured image.

Then, the information processing apparatus 10 switches the image output to the output unit 14 from the image generated by the image processing to the image estimated using the learning model (hereinafter referred to as the estimated image).

The information processing device 10 may stop outputting infrared light from the infrared illumination unit 15 at the timing when the image output to the output unit 14 is switched from the image generated by the image processing to the estimated image. As a result, half of the frames captured by the information processing apparatus 10 are visible light images before the learning of the learning model is completed, but all the frames are visible light images after the learning of the learning model is completed. The information processing apparatus 10 then generates an estimated image of the visible light image using the learning model, and outputs the generated estimated image to the output unit 14 . As a result, the information processing apparatus 10 can double the frame rate of the video output to the output unit 14 before the completion of learning.

The outline of the advanced method has been explained above, and the image output processing for realizing the advanced method will be explained below. FIG. 13 is a flow chart showing image output processing for realizing the advanced method. The following processing is executed by the control unit 13 of the information processing device 10 . The control unit 13 starts image output processing when the user starts imaging (for example, a video conference).

First, the control unit 13 activates the imaging unit 17 (step S301). Then, the control unit 13 blinks the infrared illumination unit 15 while synchronizing with the frame period of the video (moving image) captured by the imaging unit 17 (step S302).

Subsequently, the acquisition unit 131 of the information processing device 10 acquires the image captured by the imaging unit 17 . Since the infrared light is blinking in synchronization with the frame cycle of the video, the acquisition unit 131 alternately acquires the visible light image and the IR image (step S303).

The extraction unit 132 of the information processing device 10 extracts IR component information from the IR image (step S304). Specifically, the extraction unit 132 acquires the difference between the visible light image and the IR image as IR component information. Then, the image processing unit 133 of the information processing device 10 performs image processing related to luminance or lightness of the captured image based on the IR component information (step S305). Then, the output control unit 134 of the information processing device 10 outputs the processed image to the output unit 14 (step S306).

Subsequently, the learning unit 135 of the information processing device 10 performs learning of the learning model based on the image before image processing and the image after image processing (step S307).

After that, the control unit 13 of the information processing device 10 determines whether or not the shooting has ended (step S308). If the shooting has ended (step S308: Yes), the control unit 13 advances the process to step S311. If the shooting has not ended (step S308: No), the control unit 13 determines whether learning of the learning model has been completed (step S309). If learning has not been completed (step S309: No), the control unit 13 returns the process to step S303. If the learning has been completed (step S309: Yes), the control unit 13 starts the estimation process (step S310). FIG. 14 is a flow chart showing the estimation process.

First, the control unit 13 of the information processing device 10 stops the operation of the infrared illumination unit 15 (step S401). Then, the acquisition unit 131 of the information processing device 10 acquires the captured image (that is, the visible light image) (step S402). Then, the estimating unit 136 of the information processing apparatus 10 inputs the captured image to the learning model, thereby estimating the image after the image processing of the captured image (step S403). Then, the output control unit 134 of the information processing device 10 outputs the estimated image to the output unit 14 (step S404).

After that, the control unit 13 of the information processing device 10 determines whether or not the shooting has ended (step S405). If the shooting has not ended (step S405: No), the control unit 13 returns the process to step S402. If the shooting has ended (step S405: Yes), the control unit 13 returns the processing to the flow of FIG. 13 and stops the operations of the imaging unit 17 and the infrared illumination unit 15 (step S311). When the operations of the imaging unit 17 and the infrared illumination unit 15 are stopped, the control unit 13 ends the image output processing.

According to this method, after the learning of the learning model is completed, the information processing apparatus 10 can acquire an image as if the user were lighting without irradiating the user with infrared light. Also, before learning is completed, the IR image is not output to the output unit 14, so the frame rate is halved. can.

<<4. Application to live-action volumetric >>
The method of this embodiment can be applied to actual volumetric photography. Here, the live-action volumetric is a technique that acquires three-dimensional information of a subject (for example, a person) in a studio or the like and converts it into 3DCG as it is.

<4-1. Issue>
In the live-action volumetric imaging system, the information processing apparatus 10 surrounds and photographs a subject with multiple cameras. Then, the information processing apparatus 10 converts the subject into three-dimensional data from the image data to generate content. Then, the information processing apparatus 10 renders the content from a free viewpoint based on the user's operation.

Currently, the information processing apparatus 10 shoots a subject mainly in a studio with a plurality of fixed lighting fixtures on the ceiling in order to realize volumetric photography. At this time, if the subject is uniformly illuminated with bright illumination, the quality of the texture and modeling is improved, but the unevenness is reduced, resulting in an unnatural CG-like image. On the other hand, if the subject is shot with biased lighting, the shadows and unevenness will increase, but the texture and modeling quality will deteriorate. It is difficult to add additional lighting even if there is no part of the subject that is illuminated by the shape of the subject and the contrast with the green screen is low.

<4-2. Example>
Therefore, in this embodiment, a visible light camera capable of photographing infrared rays and an IR light capable of individually controlling lighting, extinguishing, and irradiation direction are additionally arranged in a conventional live-action volumetric imaging system. Then, the information processing apparatus 10 performs image processing (for example, correction or enhancement of shadows) on the visible light image based on the IR component information. As a result, the information processing apparatus 10 can generate an image with unevenness while maintaining texture and modeling quality.

FIG. 15 is a diagram showing an example of a photography studio of the live-action volumetric photography system of this embodiment. In addition to a plurality of visible light lights 20, a plurality of IR lights (infrared illuminator 15 shown in FIG. 15) are arranged in the photography studio. A plurality of IR cameras 30 are arranged in the photography studio. The IR camera 30 is a camera that can simultaneously acquire visible light and infrared light. The configuration of the IR camera 30 is similar to that of the imaging section 17 .

FIG. 16 is a diagram showing a processing example of the information processing device 10 in the live-action volumetric imaging system. The information processing device 10 acquires a multi-viewpoint image composed of visible light images from a plurality of directions of a subject and an IR image of the subject from a plurality of directions. Here, a multi-viewpoint image is an image for generating a 3D model of a subject. The information processing device 10 corrects the multi-viewpoint image based on IR component information (shadow information) extracted from the IR image. IR component information can be used as auxiliary information for foreground-background separation. A specific correction method (image processing method) may be a method similar to the methods shown in the basic methods 1 and 2 and the advanced method described above.

This makes it possible to dynamically change the lighting conditions from a specific direction (relighting), which was not possible when shooting with only visible light. Also, by illuminating the subject with IR light, it is possible to enhance the separation of the foreground and background and improve the accuracy without affecting the imaging with visible light.

<4-3. Other examples>
The imaging environment of the object may be a combination of a high-speed imaging camera capable of simultaneously acquiring visible light and infrared light, and visible light and IR light arranged omnidirectionally. FIG. 17 is a diagram showing a state in which a plurality of visible light lights 20 and a plurality of IR lights (infrared illuminators 15) are omnidirectionally arranged.

Then, the information processing device 10 simultaneously acquires shading/reflectance (albedo) from an arbitrary light source position while shooting with visible light. Since the imaging frame ratio of IR image:visible light image is not limited to 1:1, shading/albedo from multiple light source positions can be acquired simultaneously depending on camera performance. Since it uses infrared light, it does not affect visible light image capturing. If only the shading in the IR monochrome image is acquired, the frame rate can be increased to the limit independently of the visible light camera. Further, the information processing apparatus 10 can add photo-realistic shadows later based on the albedo.

<<5. Modification>>
The above-described embodiment is an example, and various modifications and applications are possible.

<5-1. Application to products and services>
For example, the technology according to the present disclosure can be applied to various products and services.

(1) Production of content For example, the user may produce new video content by synthesizing the 3D model of the subject generated in this embodiment and the 3D data managed by another server. Further, for example, when there is background data acquired by an imaging device such as Lidar, the user can combine the 3D model of the subject generated in the present embodiment with the background data, so that the user can see the subject as background data. It is also possible to create content that makes you feel as if you are in the indicated location. The video content may be 3D video content, or may be 2D video content converted to 2D. Note that the 3D model of the subject generated in the present embodiment includes, for example, a 3D model generated by a 3D model generation unit and a 3D model reconstructed by a rendering unit.

(2) Experience in virtual space For example, the information processing device 10 arranges a subject (for example, a performer) generated in the present embodiment in a virtual space where the user communicates as an avatar. be able to. In this case, the user becomes an avatar and can view the photographed subject in the virtual space.

(3) Application to communication with a remote location For example, by transmitting a 3D model of an object generated by the image processing unit 133 from the communication unit 11 to a remote location, a remote user can A 3D model of the subject can be viewed. For example, the information processing apparatus 10 can transmit the 3D model of the subject in real time, so that the subject and the remote user can communicate in real time. For example, it can be assumed that the subject is a teacher and the user is a student, or that the subject is a doctor and the user is a patient.

(4) Others For example, the information processing apparatus 10 can also generate a free-viewpoint video of sports or the like based on the 3D models of a plurality of subjects generated in the present embodiment. Also, an individual can distribute himself/herself, which is a 3D model generated in this embodiment, to a distribution platform. As such, the content of the embodiments described herein can be applied to a variety of technologies and services.

<5-2. Other modified examples>
The information processing apparatus 10 of this embodiment may be implemented by a dedicated computer system or may be implemented by a general-purpose computer system.

For example, a communication program for executing the above operations is distributed by storing it in a computer-readable recording medium such as an optical disk, semiconductor memory, magnetic tape, or flexible disk. Then, for example, the control device is configured by installing the program in a computer and executing the above-described processing. At this time, the control device may be a device (for example, a personal computer) external to the information processing device 10 . Also, the control device may be a device inside the information processing device 10 (for example, the control unit 13).

Also, the above communication program may be stored in a disk device provided in a server device on a network such as the Internet, so that it can be downloaded to a computer. Also, the functions described above may be realized through cooperation between an OS (Operating System) and application software. In this case, the parts other than the OS may be stored in a medium and distributed, or the parts other than the OS may be stored in a server device so that they can be downloaded to a computer.

Further, among the processes described in the above embodiments, all or part of the processes described as being automatically performed can be manually performed, or the processes described as being performed manually can be performed manually. All or part of this can also be done automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the illustrated one, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured. Note that this distribution/integration configuration may be performed dynamically.

In addition, the above-described embodiments can be appropriately combined in areas where the processing contents are not inconsistent. Also, the order of the steps shown in the flowcharts of the above-described embodiments can be changed as appropriate. Further, for example, each step of one flowchart may be executed by one device, or may be executed by a plurality of devices. Furthermore, when one step includes a plurality of processes, the plurality of processes may be executed by one device, or may be shared by a plurality of devices. In other words, a plurality of processes included in one step can also be executed as processes of a plurality of steps. Conversely, the processing described as multiple steps can also be collectively executed as one step.

Further, for example, a computer-executed program may be configured such that the processing of the steps described in the program is executed in chronological order according to the order described in this specification, in parallel, or when calls are executed. It may also be executed individually at necessary timings such as when it is interrupted. That is, as long as there is no contradiction, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of the steps describing this program may be executed in parallel with the processing of other programs, or may be executed in combination with the processing of other programs.

Also, for example, the present embodiment can be applied to any configuration that constitutes a device or system, such as a processor as a system LSI (Large Scale Integration), a module using a plurality of processors, a unit using a plurality of modules, etc. Furthermore, it can also be implemented as a set or the like (that is, a configuration of a part of the device) to which other functions are added.

In addition, in this embodiment, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a single device housing a plurality of modules in one housing, are both systems. .

Also, for example, multiple technologies related to this technology can be implemented independently as long as there is no contradiction. Of course, it is also possible to use any number of the present techniques in combination. For example, part or all of the present technology described in any embodiment can be combined with part or all of the present technology described in other embodiments. Also, part or all of any of the techniques described above may be implemented in conjunction with other techniques not described above.

Also, for example, this embodiment can take a configuration of cloud computing in which one function is shared by a plurality of devices via a network and processed jointly.

<<6. Conclusion>>
As described above, according to the present embodiment, the information processing apparatus 10 extracts IR component information from an IR image obtained by irradiating an object (for example, a user and surrounding objects) with infrared light, Based on the extracted IR component information, image processing relating to brightness or brightness is performed on the captured image of the target. Infrared light is invisible to the human eye. The user can obtain an image as if it were illuminated with visible light without being dazzled.

The embodiments of the present disclosure have been described above, but the technical scope of the present disclosure is not limited to the embodiments described above, and various modifications can be made without departing from the gist of the present disclosure. be. Moreover, you may combine the component over different embodiment and modifications suitably.

Also, the effects of each embodiment described in this specification are merely examples and are not limited, and other effects may be provided.

Note that the present technology can also take the following configuration.
(1)
an acquisition unit that acquires an IR image that is a captured image obtained by irradiating an object with infrared light and that includes a visible light component and an IR component;
an extraction unit that extracts IR component information from the IR image;
an image processing unit that performs image processing related to brightness or brightness of the captured image of the target based on the information of the IR component;
Information processing device.
(2)
The acquisition unit acquires a visible light image of the target in addition to the IR image,
The extraction unit acquires a difference between the visible light image and the IR image as information on the IR component.
The information processing device according to (1) above.
(3)
the infrared light is blinking,
The acquisition unit acquires the visible light image and the IR image in a time division manner in synchronization with the blinking cycle of the infrared light.
The information processing device according to (2) above.
(4)
The infrared light blinks in synchronization with the frame period of the video,
The acquisition unit acquires the image of the frame at the timing when the infrared light is not irradiated as the visible light image, and acquires the image of the frame at the timing when the infrared light is irradiated as the IR image.
The information processing apparatus according to (2) or (3) above.
(5)
The extracting unit acquires, as the IR component information, a difference between images of two consecutive frames starting from a frame at which the infrared light is not irradiated,
The image processing unit performs image processing related to brightness or brightness of an image of the next frame of the two consecutive frames based on the information of the IR component.
The information processing device according to (4) above.
(6)
the IR component information is a difference image of the two consecutive frames;
The image processing unit performs a process of blurring the edges of the difference image, and performs image processing related to brightness or brightness of the image of the next frame of the consecutive two frames based on the difference image with the edges blurred.
The information processing device according to (5) above.
(7)
The image processing unit corrects the IR component information based on inter-frame motion prediction, and determines the brightness or brightness of the next frame image of the two continuous frames based on the corrected IR component information. perform image processing,
The information processing device according to (5) above.
(8)
The extracting unit acquires, as the IR component information, a difference between images of two consecutive frames starting from a frame at which the infrared light is irradiated,
The image processing unit performs image processing related to brightness or brightness of the image of the last frame of the two consecutive frames based on the information of the IR component.
The information processing device according to (4) above.
(9)
The image processing unit rewrites luminance information in the HSL color space of the captured image based on the IR component information.
The information processing apparatus according to any one of (1) to (8) above.
(10)
The image processing unit rewrites lightness information in the HSV color space of the captured image based on the IR component information.
The information processing apparatus according to any one of (1) to (8) above.
(11)
An output control unit that outputs an image generated by the image processing to an output unit,
The information processing apparatus according to any one of (1) to (10) above.
(12)
a learning unit that performs learning of a learning model based on the image before the image processing and the image after the image processing;
an estimating unit that estimates an image after the image processing of the newly acquired captured image using the learning model;
After a predetermined period of time, the output control unit switches the image output to the output unit from the image generated by the image processing to the estimated image estimated using the learning model.
The information processing device according to (11) above.
(13)
The output control unit controls the infrared irradiation unit so that the infrared light blinks in synchronization with a video frame cycle while the image generated by the image processing is output to the output unit,
While the infrared light is blinking, the acquisition unit acquires an image of the frame at the timing when the infrared light is not irradiated as a visible light image, and an image of the frame at the timing when the infrared light is irradiated. obtained as said IR image,
The output control unit controls the infrared irradiation unit to stop outputting the infrared light at a timing when the image output to the output unit is switched from the image generated by the image processing to the estimated image,
After the output of the infrared light is stopped, the acquisition unit acquires images of all frames as the visible light image,
The estimating unit uses the learning model to estimate an image after the image processing of the visible light image.
The information processing device according to (12) above.
(14)
The acquisition unit comprises multi-viewpoint images, which are images for generating a 3D model of a subject and which are composed of visible light images from a plurality of directions of the subject, IR images of the subject from a plurality of directions, and get
The image processing unit corrects the multi-viewpoint image based on the information of the IR component extracted from the IR image.
The information processing device according to (1) above.
(15)
Acquiring an IR image that is a captured image obtained by irradiating an object with infrared light and that includes a visible light component and an IR component,
extracting IR component information from the IR image;
performing image processing on the brightness or brightness of the captured image of the target based on the IR component information;
Information processing methods.
(16)
the computer,
an acquisition unit that acquires an IR image that is a captured image obtained by irradiating an object with infrared light and that includes a visible light component and an IR component;
an extraction unit that extracts IR component information from the IR image;
an image processing unit that performs image processing related to brightness or brightness of the captured image of the target based on the information of the IR component;
A program to function as

REFERENCE SIGNS LIST 10 information processing device 11 communication unit 12 storage unit 13 control unit 14 output unit 15 infrared illumination unit 16 synchronization signal generation unit 17 imaging unit 20 visible light 30 IR camera 131 acquisition unit 132 extraction unit 133 image processing unit 134 output control unit 135 Learning unit 136 Guessing unit

Claims

an acquisition unit that acquires an IR image that is a captured image obtained by irradiating an object with infrared light and that includes a visible light component and an IR component;
an extraction unit that extracts IR component information from the IR image;
an image processing unit that performs image processing related to brightness or brightness of the captured image of the target based on the information of the IR component;
Information processing device.
The acquisition unit acquires a visible light image of the target in addition to the IR image,
The extraction unit acquires a difference between the visible light image and the IR image as information on the IR component.
The information processing device according to claim 1 .
the infrared light is blinking,
The acquisition unit acquires the visible light image and the IR image in a time division manner in synchronization with the blinking cycle of the infrared light.
The information processing apparatus according to claim 2.
The infrared light blinks in synchronization with the frame period of the video,
The acquisition unit acquires the image of the frame at the timing when the infrared light is not irradiated as the visible light image, and acquires the image of the frame at the timing when the infrared light is irradiated as the IR image.
The information processing apparatus according to claim 2.
The extracting unit acquires, as the IR component information, a difference between images of two consecutive frames starting from a frame at which the infrared light is not irradiated,
The image processing unit performs image processing related to brightness or brightness of an image of the next frame of the two consecutive frames based on the information of the IR component.
The information processing apparatus according to claim 4.
the IR component information is a difference image of the two consecutive frames;
The image processing unit performs a process of blurring the edges of the difference image, and performs image processing related to brightness or brightness of the image of the next frame of the consecutive two frames based on the difference image with the edges blurred.
The information processing device according to claim 5 .
The image processing unit corrects the IR component information based on inter-frame motion prediction, and determines the brightness or brightness of the next frame image of the two continuous frames based on the corrected IR component information. perform image processing,
The information processing device according to claim 5 .
The extracting unit acquires, as the IR component information, a difference between images of two consecutive frames starting from a frame at which the infrared light is irradiated,
The image processing unit performs image processing related to brightness or brightness of the image of the last frame of the two consecutive frames based on the information of the IR component.
The information processing apparatus according to claim 4.
The image processing unit rewrites luminance information in the HSL color space of the captured image based on the IR component information.
The information processing device according to claim 1 .
The image processing unit rewrites lightness information in the HSV color space of the captured image based on the IR component information.
The information processing device according to claim 1 .
An output control unit that outputs an image generated by the image processing to an output unit,
The information processing device according to claim 1 .
a learning unit that performs learning of a learning model based on the image before the image processing and the image after the image processing;
an estimating unit that estimates an image after the image processing of the newly acquired captured image using the learning model;
After a predetermined period of time, the output control unit switches the image output to the output unit from the image generated by the image processing to the estimated image estimated using the learning model.
The information processing device according to claim 11 .
The output control unit controls the infrared irradiation unit so that the infrared light blinks in synchronization with a video frame cycle while the image generated by the image processing is output to the output unit,
While the infrared light is blinking, the acquisition unit acquires an image of the frame at the timing when the infrared light is not irradiated as a visible light image, and an image of the frame at the timing when the infrared light is irradiated. obtained as said IR image,
The output control unit controls the infrared irradiation unit to stop outputting the infrared light at a timing when the image output to the output unit is switched from the image generated by the image processing to the estimated image,
After the output of the infrared light is stopped, the acquisition unit acquires images of all frames as the visible light image,
The estimating unit uses the learning model to estimate an image after the image processing of the visible light image.
The information processing apparatus according to claim 12.
The acquisition unit comprises multi-viewpoint images, which are images for generating a 3D model of a subject and which are composed of visible light images from a plurality of directions of the subject, IR images of the subject from a plurality of directions, and get
The image processing unit corrects the multi-viewpoint image based on the information of the IR component extracted from the IR image.
The information processing device according to claim 1 .
Acquiring an IR image that is a captured image obtained by irradiating an object with infrared light and that includes a visible light component and an IR component,
extracting IR component information from the IR image;
performing image processing on the brightness or brightness of the captured image of the target based on the IR component information;
Information processing methods.
the computer,
an acquisition unit that acquires an IR image that is a captured image obtained by irradiating an object with infrared light and that includes a visible light component and an IR component;
an extraction unit that extracts IR component information from the IR image;
an image processing unit that performs image processing related to brightness or brightness of the captured image of the target based on the information of the IR component;
A program to function as