JP2024021344A - Image processing device, control method and program - Google Patents

Abstract

The present invention performs image correction according to the irradiation mode of a light source. [Solution] An image processing device captures a first captured image of a subject with light in a first wavelength range and a second captured image of light in a second wavelength range including a wavelength longer than the first wavelength range. an acquisition means for acquiring a second captured image, and a setting means for setting a target area to be corrected based on the first captured image, based on the difference between the first captured image and the second captured image. , a first correction means for generating a first corrected image by correcting an image of the target area of the first captured image; a first corrected image generated by the first correction means; and a first captured image; a first irradiation mode indicating the distribution of intensity with which a subject in a target area is irradiated with light in a first wavelength range; and control means for controlling the application rate of the first corrected image in the composition of the composition means based on the difference in the second irradiation mode indicating the distribution of the intensity with which the object in the target region is irradiated by the light of the region. [Selection diagram] Figure 4

Description

The present invention relates to an image processing device, a control method, and a program, and particularly relates to an image correction technique for a human skin region.

Taking advantage of the fact that melanin pigmented areas of a person's skin, such as spots and freckles, are difficult to show up in infrared images, the system creates an image in which the areas are corrected based on the difference image between the visible light image and the infrared image. There is a technology for generating this (Patent Document 1).

JP 2019-106045 Publication

By the way, some imaging devices employ a light source that does not contain infrared light, such as an LED, as an illumination device. When photographing a person with such an imaging device, the irradiation mode of visible light and infrared light may differ, and even if a difference image between a visible light image and an infrared light image is used as in Patent Document 1, However, in some cases, it may not be possible to appropriately identify the site where melanin pigment is deposited. That is, in the conventional technology, depending on the irradiation mode of the light source, it may not be possible to generate an image in which the human skin area is suitably corrected.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an image processing device, a control method, and a program that perform image correction according to the irradiation mode of a light source.

In order to achieve the above-mentioned object, the image processing device of the present invention provides a first captured image of a subject that captures light in a first wavelength range, and a second image that includes a wavelength longer than the first wavelength range. an acquisition means for acquiring a second captured image capturing light in a wavelength range of , and a setting means for setting a target area to be corrected based on the first captured image; a first correction unit that generates a first corrected image in which an image of the target area of the first captured image is corrected based on a difference with the captured image; and a first correction generated by the first correction unit. a synthesis means for generating a corrected image for output by synthesizing the image and the first captured image; Based on the difference between the irradiation mode and the second irradiation mode indicating the distribution of the intensity with which the object in the target area is irradiated with light in the second wavelength range, the application rate of the first corrected image in the synthesis of the synthesis means is controlled. It is characterized by having a control means for.

With such a configuration, according to the present invention, it is possible to perform image correction according to the irradiation mode of the light source.

A block diagram illustrating the functional configuration of an imaging device according to an embodiment and a modification of the present invention A block diagram illustrating a functional configuration related to beautiful skin correction processing according to embodiments and modified examples of the present invention Diagrams for explaining how face areas are divided according to embodiments and modified examples of the present invention A block diagram illustrating the detailed configuration of the correction unit 204 according to the embodiment and modification of the present invention Diagram for explaining the difference between visible light and infrared light irradiation modes according to embodiments and modified examples of the present invention Diagram for explaining application rates according to embodiments and modified examples of the present invention A flowchart illustrating a beautiful skin correction process executed in the image processing unit 102 according to the embodiment and modification of the present invention. A diagram showing a configuration example of an image sensor according to an embodiment and a modification of the present invention

[Embodiment]
Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

An embodiment described below describes an example in which the present invention is applied to an imaging device capable of capturing visible light images and infrared light images, which is an example of an image processing device. However, the present invention provides a first captured image that captures light in a first wavelength range, and a second captured image that captures light in a second wavelength range that includes a wavelength longer than the first wavelength range. It is applicable to any equipment that can be acquired. Furthermore, in this specification, the term "irradiation mode (of light)" refers to the distribution of intensity with which a subject is irradiated with light, and simply refers to how the subject is illuminated by light or a light source.

《Functional configuration of imaging device》
FIG. 1 is a block diagram showing the functional configuration of an imaging device 100 according to this embodiment.

The imaging unit 101 images a subject and outputs a captured image. The imaging unit 101 includes an imaging optical system, an image sensor, and an A/D conversion processing unit. The imaging unit 101 performs a shooting operation, for example, when the operation input unit 108 accepts a shooting instruction from a user. More specifically, the imaging unit 101 performs photography by photoelectrically converting an optical image of a subject formed on an image sensor by an imaging optical system, and outputs a captured image.

In this embodiment, the imaging unit 101 is configured to be able to output two types of captured images: a visible light image and an infrared light image. The output of a visible light image and an infrared light image from the imaging unit 101 can be realized, for example, by configuring an image sensor in the manner shown in FIG. 8. In FIG. 8, the pixel (photoelectric conversion element) array in the image sensor is shown in a grid, and the characters shown in each grid indicate the filters applied to the grid. Here, R, G, and B indicate filters that pass red, green, and blue visible light, respectively, and IR indicates a filter that passes infrared light or a filter that cuts visible light. Further, W indicates a filter that cuts infrared light.

FIG. 8(a) shows an embodiment including two types of image sensors capable of outputting visible light images and infrared light images, and shows a visible light filter and an infrared light filter applied to each image sensor. ing. On the other hand, FIG. 8(b) shows a mode in which one type of image sensor can output a visible light image and an infrared light image, and the filter applied to the image sensor has 2×2 pixels for each color. A filter is applied that transmits visible light and infrared light. FIG. 8(c) shows a mode in which a visible light image and an infrared light image can be output by changing sequentially applied filters in one type of image sensor. Filters are applied alternately.

In this embodiment, in order to facilitate understanding of the invention, it is assumed that the visible light image and the infrared light image are captured in the same imaging range, and that the same subject is captured in the visible light image and the infrared light image. The images shall appear in the same position. That is, it is assumed that a subject appearing in a pixel at an arbitrary pixel position in a visible light image appears in a pixel at the same pixel position in an infrared light image.

The image processing unit 102 performs various image processing on the input image. The targets of image processing by the image processing unit 102 include, in addition to the captured image output by the imaging unit 101, images read out from the recording unit 106 (described later) and images acquired from an external device via the communication I/F 107. can be included. The image processing unit 102 performs development processing such as white balance processing, demosaic processing, and gamma processing on the input captured image. In addition, the image processing unit 102 executes skin beautification correction processing, which will be described later, based on the visible light image and the infrared light image.

The system control unit 103 controls the operation of each block included in the imaging apparatus 100. The system control unit 103 includes a CPU, ROM, and RAM (not shown). The operation of each block is controlled by the CPU reading out the operation program for each block stored in the ROM, loading it into the RAM, and executing it.

The display control unit 104 is an output interface for display video signals provided in the imaging device 100, and controls the display of the display unit 105. The imaging device 100 of this embodiment includes a display unit 105 that may be a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display control unit 104 outputs captured images, images generated by the image processing unit 102, and video signals of various graphical user interfaces to the display unit 105. The display unit 105 may be a display device built into the imaging device 100, or may be an external display device detachably connected to the imaging device 100.

The recording unit 106 is a recording device included in the imaging device 100, and mainly records images (image files) generated by the image processing unit 102. The recording unit 106 can include a regular device such as a memory card made of a semiconductor memory or a hard disk made of a magneto-optical disk. The recording unit 106 may be configured to be detachable or may be built into the imaging device 100.

The communication I/F 107 is an interface that controls information communication with an external device included in the imaging device 100. The communication I/F 107 may be connected to an external device via a network (not shown), or may be directly connected by wire or wirelessly. The imaging operation and image processing performed by the imaging device 100 of this embodiment may be performed by operating the operation input unit 108, or may be performed by an external device connected via the communication I/F 107. It may also be executed based on a request. Therefore, the communication I/F 107 receives control input from an external device and transmits an image generated by the image processing unit 102 to the external device.

The operation input unit 108 is a user interface included in the imaging device 100. The operation input unit 108 can include, for example, operation members such as buttons, a touch panel sensor, and the like. When the operation input unit 108 detects that an operation input has been made, it outputs a control signal corresponding to the operation input to the system control unit 103.

The bus 109 is a communication path used for information communication between blocks included in the imaging apparatus 100. In the example of FIG. 1, an imaging unit 101, an image processing unit 102, a system control unit 103, a display control unit 104, a recording unit 106, and a communication I/F 107 are configured to be able to communicate with each other via a bus 109.

《Overview of skin beautification processing》
Next, an outline of the skin-beautifying correction process executed by the image processing unit 102 of this embodiment will be explained. The skin beautifying process executed by the image processing unit 102 is a correction process for making stains, freckles, pores, etc. appearing on the skin area of the human image included in the visible light image less noticeable.

Here, if the skin is in a beautiful state, the skin area in the visible light image will be in a flat state with the skin color spread evenly except for edges and the like. On the other hand, if there are spots, freckles, pores, etc. on the skin, the skin area in the visible light image will have "color unevenness" in which colors other than the skin are mixed. Therefore, such color unevenness appears in the AC component when the visible light image is frequency-resolved. However, in the AC component of the visible light image, brightness changes that are not caused by spots, freckles, pores, etc., such as those caused by bumps on the skin (hereinafter referred to as brightness changes inherent to the skin) also appear.

On the other hand, infrared light images have the characteristic that spots, freckles, pores, etc. are less likely to appear. Therefore, from the AC component of the infrared light image, it is possible to obtain the original brightness change of the skin, with brightness changes caused by spots, freckles, pores, etc. removed (or reduced). Therefore, by calculating the difference in AC components between visible light images and infrared light images taken of a subject at the same time, it is possible to obtain luminance changes caused by spots, freckles, pores, etc. That is, based on the difference between the AC components of the visible light image and the infrared light image, it is possible to specify areas where spots, freckles, pores, etc. are present. Therefore, the AC component of the visible light image is corrected based on the difference between the AC component of the visible light image and the infrared light image to reduce the effects of spots, freckles, pores, etc. By adding it to the components, a visible light image of a suitable skin condition can be obtained.

However, when the modes of irradiation of visible light and infrared light on a subject (person) are different, even if the difference between the AC components of the visible light image and the infrared light image is taken, the brightness changes due to spots, freckles, pores, etc. It is not possible to specify the minute. For example, if a light source that emits only visible light is placed to locally illuminate a person's skin area, the difference between the AC components of the visible light image and the infrared light image includes the skin's original brightness change. In addition to this, a change in brightness due to the difference in the irradiation mode between visible light and infrared light appears. For example, the same applies when a person's skin area is locally irradiated with visible light indoors or outdoors at night (where no sunlight is irradiated). As a result, when correction is performed on an area identified based on the difference between the AC components of a visible light image and an infrared light image, for example, areas where spots, freckles, pores, etc. The corrected image that is displayed is not in a suitable manner.

Therefore, the image processing unit 102 of the present embodiment changes the processing content of the skin beautification process based on the difference between the visible light irradiation mode and the infrared light irradiation mode, so that the skin area is suitably corrected. Obtain a corrected image. Specifically, the image processing unit 102 generates an image in which the skin area of the visible light image is corrected based on the difference in AC components between the visible light image and the infrared light image, and a visible light image without simply referring to the infrared light image. A corrected image is obtained by combining the optical image with the image in which the skin area has been corrected at a ratio that corresponds to the difference in the irradiation mode.

<Functional configuration related to skin beautification processing>
FIG. 2 is a block diagram showing a functional configuration related to skin beautification processing performed by the image processing unit 102 of this embodiment. As mentioned above, the input for the skin beautification processing is the visible light image 210 and the infrared light image 220 obtained by photographing the subject at the same time, and after processing, one corrected image (visible light image ) 230 is output. More specifically, the visible light image 210 is input to the target area detection section 201 and the first irradiation detection section 202, and the infrared light image 220 is input to the second irradiation detection section 203.

The target area detection unit 201 detects a target area to be corrected in the skin beautification process from the input visible light image 210. In this embodiment, the target area of the skin-beautifying correction process is an area in a person's face area where the skin appears (a skin area of the person's face). Although a detailed explanation will be omitted, the target area detection unit 201 refers to the detection result of face detection processing performed separately on the visible light image 210, and detects the skin color in the face area indicated by the detection result. Set the area (including areas such as spots, freckles, pores, etc.) as the target area. Then, the visible light image 210 outputs target area information indicating the detection result of the target area and the visible light image 210 to the subsequent correction unit 204.

The first irradiation detection unit 202 generates visible light irradiation information indicating the visible light irradiation mode based on the input visible light image 210. Specifically, the first irradiation detection unit 202 divides the face area 300 (target area) in the visible light image 210 into blocks as shown in FIG. A value indicating the light irradiation state is derived. In this embodiment, it is assumed that the value indicating the visible light irradiation state is derived by normalizing the average brightness value within a block by the average brightness value of the entire face area. That is, the value BY _norm (i, j) indicating the visible light irradiation state of the block indicated by the block coordinates (i, j) is the average luminance BY (i, j) in the block and the value in the face area 300. Using the average luminance BY _ave ,
It is derived as When the first irradiation detection unit 202 derives values indicating the visible light irradiation state for all blocks, it outputs visible light irradiation information that integrates these values to the correction unit 204.

The second irradiation detection unit 203 generates infrared light irradiation information indicating an infrared light irradiation mode based on the input infrared light image 220. The processing performed by the second irradiation detection unit 203 is similar to the processing performed by the first irradiation detection unit 202. That is, the second irradiation detection unit 203 divides the face area in the infrared light image 220 into blocks, integrates values indicating the infrared light irradiation state derived for each block, and generates infrared light irradiation information. . When the second irradiation detection unit 203 generates the infrared light irradiation information, it outputs the infrared light irradiation information and the infrared light image 220 to the correction unit 204.

Here, in this embodiment, both the first irradiation detection unit 202 and the second irradiation detection unit 203 normalize the average luminance value within a block by the average luminance value of the entire face area. Therefore, the generated visible light irradiation information and infrared light irradiation information are free from the influence of the difference in spectral reflectance between visible light and infrared light.

The correction unit 204 performs skin beautification processing on the visible light image 210 based on the input visible light image 210, infrared light image 220, target area information, visible light irradiation information, and infrared light irradiation information, and outputs the result. A corrected image 230 is generated.

<Detailed configuration of correction unit 204>
Here, the detailed configuration of the correction unit 204 will be explained with further reference to FIG. 4. As described above, the inputs of the correction unit 204 are the visible light image 210, the infrared light image 220, the target area information 410, the visible light irradiation information 420, and the infrared light irradiation information 430. The visible light image 210 is input to the first AC component extraction section 401, the first correction section 404, and the second correction section 405, and the infrared light image 220 is input to the second AC component extraction section 402. Further, the target area information 410 is input to the first correction unit 404 and the second correction unit 405, and the visible light irradiation information 420 and the infrared light irradiation information 430 are input to the derivation unit 406.

The first AC component extraction unit 401 extracts an AC component image (visible light AC image) from the input visible light image 210. Specifically, the first AC component extraction unit 401 extracts and outputs a visible light AC image by applying a bandpass filter process to the visible light image 210 to extract a predetermined frequency band.

The second AC component extraction unit 402 extracts an AC component image (infrared light AC image) from the input infrared light image 220. Similarly to the first AC component extraction unit 401, the second AC component extraction unit 402 extracts and outputs an infrared light AC image by applying band pass filter processing to the infrared light image 220.

The AC component comparison unit 403 indicates the difference between the visible light AC image and the infrared light AC image based on the visible light AC image and the infrared light AC image, and derives a correction coefficient for correction of the visible light image in the first correction unit 404. A correction coefficient is derived for each coordinate of the image. In this embodiment, the AC component comparison unit 403 calculates the correction coefficient k(x,y) for the coordinates (x,y) from the value Y_AC(x,y) at the same coordinates of the visible light AC image and the infrared light AC image. Using the same coordinate value IR_AC (x, y) of
Derive it as Here, when the correction coefficient k (x, y) is 0, there is no difference between the visible light image 210 and the infrared light image 220, that is, the correction coefficient k (x, y) is set to the pixel at the coordinates (x, y) of the visible light image 210. Indicates that dark spots, freckles, pores, etc. are not visible. On the other hand, if the correction coefficient k(x, y) is negative, there is a difference between the visible light image 210 and the infrared light image 220, that is, the pixel at the coordinates (x, y) of the visible light image 210 Indicates that wrinkles, freckles, pores, etc. are visible. Further, the smaller the value of the correction coefficient k(x, y) (the larger the absolute value), the more deviated the color appears from the normal skin color. The reason why the correction coefficient k(x, y) is derived so that it is negative for pixels where spots, freckles, pores, etc. appear is that in the correction in the first correction unit 404, the AC of the visible light image 210 This is to reduce (weaken) the components.

The first correction unit 404 generates an image (first image) in which spots, freckles, pores, etc. appearing in the target area of the visible light image 210 are reduced based on the difference in AC components between the visible light image 210 and the infrared light image 220. corrected image). More specifically, the first correction unit 404 corrects the AC component of the pixels in the target area of the visible light image 210 based on the correction coefficient k(x,y) derived by the AC component comparison unit 403, and By adding it to the DC component, the pixel value of the target area after correction is derived. The first correction unit 404 then generates a first corrected image by changing the pixel value of the target area in the visible light image 210 to the obtained corrected pixel value of the target area.

First, the first correction unit 404 calculates a correction amount k_AC(x,y) for reducing spots, freckles, pores, etc. for each coordinate (x,y) of the target area of the visible light AC image.
Derive it as Further, the DC component Y_DC(x,y) of the visible light image 210 is
It can be derived as Therefore, the first correction unit 404 uses the DC component Y_DC(x,y) of the target area of the visible light image 210 and the correction amount k_AC(x,y).
The pixel value Y _after1 (x, y) of the coordinates (x, y) of the target area after correction is derived. Then, the first correction unit 404 generates a first corrected image by changing the pixel value of the target area of the visible light image 210 to Y _after1 (x, y).

On the other hand, the second correction unit 405 performs a correction process different from that of the first correction unit 404 on the target area of the visible light image 210 to reduce stains, freckles, pores, etc. (corrected image). The correction process performed by the second correction unit 405 is not based on the difference between the visible light image 210 and the infrared light image 220, but is performed with reference only to the visible light image 210. That is, the correction process performed by the second correction unit 405 is a noise reduction process that reduces the amplitude of the AC component in the target area.

The second correction unit 405 applies filtering processing, such as a bilateral filter, to the target region of the visible light image 210 to reduce the amplitude of the AC component in the region while preserving edges. At this time, the scale of the applied filter may be adjusted to match the area of spots, freckles, pores, etc., so that the larger the size of the facial area, the larger the filter. Then, the second correction unit 405 generates a second corrected image by replacing pixels in the target area of the visible light image 210 with pixels for each filtering process. In addition, the noise reduction processing performed by the second correction unit 405 can include various types of filtering processing using an average value filter, a low-pass filter, and the like.

The deriving unit 406 calculates the first corrected image generated by the first correcting unit 404 and the second corrected image generated by the second correcting unit 405 based on the visible light irradiation information and the infrared light irradiation information. Derive the composite ratio. In the present embodiment, the derivation unit 406 derives the application rate α of the first corrected image in the compositing process performed by the compositing unit 407, which will be described later.

First, the derivation unit 406 derives the average value of the sum of absolute differences using the difference between the visible light irradiation information and the infrared light irradiation information for each block. The average value S _ave of the sum of absolute differences is a value BY norm (i, j) indicating the visible light irradiation state of the block indicated by the block _coordinates (i, j) and a value BIR _norm indicating the infrared light irradiation state. Using (i, j),
It is derived as Here, m is the total number of blocks included in the face area.

By taking the difference between blocks at the same block position in this way, we can exclude the influence of variations in signal values caused by the facial structure and estimate the difference in the irradiation mode of visible light and infrared light in the target area. be able to. Therefore, the larger the difference in the irradiation mode between visible light and infrared light, the higher the value of _Save appears. For example, as shown in FIG. 5A, if there is a difference in the irradiation mode of visible light and infrared light, a value corresponding to the difference appears in _Save . In other words, when S _ave is large, it is possible that spots, freckles, pores, etc. are not reduced appropriately in the first corrected image due to the difference in the irradiation mode of visible light and infrared light. Therefore, the derivation unit 406 sets the application rate α of the first corrected image so that the larger S _ave is, the smaller the synthesis ratio of the first corrected image is, and the smaller S _ave is, the larger the synthesis ratio of the first corrected image is. Determine.

The application rate α may be determined, for example, by referring to an LUT as shown in FIG. 6, which shows the relationship between _Save and the application rate α. In the example of FIG. 6, S _ave and application rate α are shown as a linear relationship; however, the larger S _ave is, the smaller the application rate α of the first corrected image is, and the smaller S _ave is, the smaller the application rate α of the first corrected image is. The present invention is not limited to this, as long as the application rate α tends to be large. After determining the application rate α, the derivation unit 406 transmits information on the application rate α to the synthesis unit 407.

The synthesizing unit 407 synthesizes the first corrected image and the second corrected image based on the application rate α determined by the deriving unit 406 to generate a corrected image 230 for output. Specifically, the combining unit 407 combines the pixel value at the coordinates (x, y) of the corrected image 230 with the pixel value Y _after1 (x, y) at the same coordinates of the first corrected image and the second corrected image. Using the pixel value Y _after2 (x, y) of the same coordinates,
Derive it as Therefore, if there is a difference in the irradiation mode of visible light and infrared light, the corrected image 230 will be a combination of the first corrected image and the second corrected image, so that stains, freckles, pores, etc. is output with the remaining correction amount reduced. In other words, stains, freckles, pores, etc. that could not be properly corrected using the difference between the visible light AC image and the infrared light AC image can be corrected by combining the second corrected image that is simply corrected using only the visible light image 210. The image 230 makes it less noticeable. On the other hand, if there is no difference in the irradiation mode of visible light and infrared light, the corrected image 230 becomes the first corrected image itself.

《Beautiful skin correction processing》
Next, the skin beauty correction process performed by the imaging device 100 of this embodiment having such a configuration will be described in detail with reference to the flowchart of FIG. 7. The processing corresponding to the flowchart is realized by the CPU of the system control unit 103 reading a corresponding processing program stored in, for example, a ROM, loading it into a RAM, and executing it to control the operation of the image processing unit 102. can do. This beautiful skin correction process will be explained assuming that it is started, for example, when a photographing operation is performed in a state where the beautiful skin correction mode is set as the imaging mode in the imaging apparatus 100.

In S701, the target area detection unit 201 detects a target area from the visible light image 210 obtained by photographing. When the target area detection unit 201 detects the target area, it outputs target area information 410.

In S<b>702 , the first irradiation detection unit 202 generates visible light irradiation information 420 based on the image of the target area of the visible light image 210 . In parallel, the first irradiation detection unit 202 generates infrared light irradiation information 430 based on the image of the target area of the infrared light image 220 in S703.

In S704, the first AC component extraction unit 401 extracts an AC component from the visible light image 210 to generate a visible light AC image. Further, in S705, the second AC component extraction unit 402 extracts an AC component from the infrared light image 220 to generate an infrared light AC image.

In S706, the AC component comparison unit 403 derives a correction coefficient for correction processing in the first correction unit 404 based on the visible light AC image and the infrared light AC image.

In S707, the first correction unit 404 generates a first corrected image based on the correction coefficient derived in S706. More specifically, the first correction unit 404 generates an image of the AC component of the target area after correction by multiplying the absolute value of the pixel value of the target area of the visible light AC image by a correction coefficient. The first correction unit 404 also generates an image of the DC component of the visible light image 210 by subtracting the visible light AC image from the visible light image 210, and adds the AC component image of the target area after correction to this. This generates the first corrected image.

In parallel, the second correction unit 405 generates a second corrected image by applying a predetermined filtering process to the target area of the visible light image 210 in S708.

In S709, the derivation unit 406 derives the application rate α of the first corrected image based on the visible light irradiation information 420 and the infrared light irradiation information 430.

In S710, the synthesis unit 407 synthesizes the first corrected image and the second corrected image based on the application rate α derived in S709, generates the corrected image 230, and completes the main beautiful skin correction process.

In this manner, the image processing device of this embodiment performs image correction according to the irradiation mode of the light source, and generates a beautiful skin corrected image in which spots, freckles, pores, etc. are reduced in a suitable manner. be able to.

[Modification 1]
In the above-described embodiment, the target area is an area indicating the skin color in the face area, but the implementation of the present invention is not limited to this. For example, the target area may be an area irradiated with visible light or infrared light. In this case, a difference in pixel values between an image irradiated with infrared light and an image not irradiated with infrared light may be derived, and a region with a large difference may be extracted as a target region.

[Modification 2]
In the embodiment described above, when deriving the visible light irradiation information and the infrared light irradiation information, the average brightness value in each block is normalized by the average brightness value of the entire face area. However, the present invention The implementation is not limited to this.

For example, each piece of irradiation information may be configured as a luminance histogram of only divided regions whose variance value is within a threshold value by deriving the variance value of brightness in each block. In this case as well, the irradiation information can indicate the distribution of the irradiation mode within the face area, and by taking the difference between visible light and infrared light, the difference in the irradiation mode can be derived.

Alternatively, each piece of irradiation information may be configured based on, for example, the mode of a brightness histogram for each block. In this case, the value BYmode _norm (i, j) indicating the visible light irradiation state of the block indicated by the block coordinates (i, j) is the mode value BYmode (i, j) within the block and the value BYmode norm (i, j) within the face area. Using the average BYmode _ave of the mode of brightness,
It can also be derived as

[Modification 3]
In the embodiment described above, the method of deriving the correction coefficient k (x, y) for correcting the visible light image in the first correction unit 404 using the visible light AC image and the infrared light AC image as is has been described. However, implementation of the present invention is not limited to this. For example, the AC component comparison unit 403 may derive the correction coefficient k(x,y) after matching the brightness of the visible light AC image and the infrared light AC image. The brightness adjustment may be performed using, for example, the average value of the brightness signal. In this case, the correction coefficient k(x, y) is calculated using the average value Y _ave of the luminance signals of the entire visible light image 210 and the average value IR _ave of the luminance signals of the entire infrared light image 220.
It can be derived by In addition, brightness matching may be performed using an average value around the pixel of interest (x, y).

[Modification 4]
In the embodiments and modified examples described above, the correction coefficient k(x,y) for correcting the visible light image in the first correction unit 404 is set to the ratio of the pixel values of the visible light AC image and the infrared light AC image. Although the description has been given as deriving based on the above, the implementation of the present invention is not limited to this. For example, areas where features that are not spots, freckles, pores, etc., but may be facial features, can have high contrast even in an infrared light AC image. Therefore, the correction coefficient k(x, y) may be controlled to be 0 for areas in the infrared light AC image that exhibit pixel values equal to or greater than the threshold value. By doing so, it is possible to prevent the first correction unit 404 from performing correction on a region where the AC component on the infrared light image 220 side has a characteristic.

[Modification 5]
In the embodiment described above, when deriving the application rate α of the first corrected image, a mode was explained in which the average value S _ave of the sum of absolute differences between blocks of visible light irradiation information and infrared light irradiation information is derived. The implementation of the present invention is not limited to this. In the facial region, there are organs such as eyeballs that exhibit a color that is different from the rest of the skin or have a different tendency to diffuse irradiated light. Therefore, the average value of the sum of absolute differences referred to when deriving the application rate α may be derived excluding the region where such an organ appears.

In the example of the embodiment shown in FIG. 5(a), the area including the eyeball part in the face area is also included, but in this modified example, as shown in FIG. 5(b), the organ area including the eyeball is included. are excluded when deriving the average value of the sum of absolute differences. In this case, the average value S _{ave_norm} of the sum of absolute differences obtained for n blocks excluding the organ area among the m blocks included in the face area is the sum of absolute differences S ave_norm of the blocks excluding the organ area. Using _norm ,
It can be derived as However, n≦m.

Furthermore, in the above-described embodiment and this modified example, it has been explained that the difference in the irradiation mode is evaluated using the average value of the sum of absolute differences between the visible light irradiation information and the infrared light irradiation information. The value is not limited to this, and for example, the average of the sum of squared differences may be used.

[Modification 6]
In the above-described embodiment, a mode has been described in which the corrected image 230 is generated by combining the first corrected image and the second corrected image, but the implementation of the present invention is not limited to this. According to the implementation of the present invention, the generation of the second corrected image is not an essential configuration, and the synthesis in the combining unit 407 may be performed based on the first corrected image and the visible light image 210. . That is, the pixel value Y _after (x, y) at the coordinates (x, y) of the corrected image 230 is the same as the pixel value Y _after1 (x, y) at the same coordinates of the first corrected image and the same coordinates of the visible light image 210. Using the coordinate pixel value Y _in2 (x, y),
It may be derived as follows.

[Modification 7]
In the above-described embodiments and modified examples, the first corrected image is obtained by acquiring a visible light image and an infrared light image and correcting the visible light image based on the difference in their AC components. However, the implementation of the present invention is not limited to this. The effects of spots, freckles, pores, etc. are generally less noticeable in captured images with wavelengths of 600 nm or more. It can also be used in place of an infrared image.

[Summary of embodiments and modifications]
The disclosure of this specification includes the following image processing device, control method, and program.
(Item 1)
For the subject, a first captured image is captured of light in a first wavelength range, and a second captured image is captured of light in a second wavelength range including a wavelength longer than the first wavelength range. an acquisition means to
Setting means for setting a target area to be corrected based on the first captured image; and setting means for setting a target area to be corrected in the first captured image based on a difference between the first captured image and the second captured image. a first correction means that generates a first corrected image by correcting the image of the region;
a combining unit that generates a corrected image for output by combining the first corrected image generated by the first correcting unit and the first captured image;
a first irradiation mode showing a distribution of intensity with which a subject in the target area is irradiated with light in the first wavelength range; and a distribution of intensity with which a subject in the target area is irradiated with light in the second wavelength range. A control means for controlling an application rate of the first corrected image in the synthesis of the synthesis means based on a difference in a second irradiation mode showing the distribution;
An image processing device comprising:
(Item 2)
The first correction means adjusts the AC component of the image of the target area of the first captured image based on the difference between the AC component of the first captured image and the AC component of the second captured image. The image processing device according to item 1, wherein the first corrected image is generated by reducing the amount of image.
(Item 3)
Item 1 or 2, wherein the first correction means does not correct the first captured image for a region in which the AC component of the second captured image shows a pixel value equal to or higher than a predetermined threshold. 2. The image processing device according to 2.
(Item 4)
The synthesizing means includes a second correcting means that generates a second corrected image by correcting the image of the target area of the first captured image without referring to the second captured image, 4. The image processing apparatus according to any one of items 1 to 3, wherein the output corrected image is generated by combining the corrected image with the second corrected image.
(Item 5)
5. The image processing device according to item 4, wherein the second correction means reduces the AC component of the first captured image without referring to the second captured image.
(Item 6)
The control means specifies a difference between the first irradiation mode and the second irradiation mode for the subject in the target area based on a difference in brightness between the first captured image and the second captured image. The image processing device according to any one of items 1 to 5, characterized in that:
(Item 7)
The control means generates an average luminance distribution for each divided area in the target area for each of the first captured image and the second captured image, and The image processing device according to item 6, characterized in that a difference in distribution of average luminance of the captured image is specified as a difference between the first irradiation mode and the second irradiation mode.
(Item 8)
The control means generates a brightness histogram for each divided area in the target area for each of the first captured image and the second captured image, and 7. The image processing apparatus according to item 6, wherein a difference in a histogram of brightness of the image is specified as a difference between the first irradiation mode and the second irradiation mode.
(Item 9)
The control means specifies the mode of brightness for each divided area in the target area for each of the first captured image and the second captured image, and 7. The image processing apparatus according to item 6, wherein the difference in the mode of luminance of the captured image is specified as the difference between the first irradiation mode and the second irradiation mode.
(Item 10)
The control means matches the brightness of the first captured image and the second captured image, and then controls the light in the first wavelength range and the light in the second wavelength range to the subject in the target area. 10. The image processing apparatus according to any one of items 6 to 9, characterized in that the image processing apparatus specifies a difference in irradiation mode between the image processing apparatus and the image processing apparatus.
(Item 11)
The target area is a skin area of a person's face,
The control means controls the first irradiation mode and the second irradiation mode based on a difference in brightness between divided regions in the target region excluding a region of a predetermined organ constituting the face. The image processing device according to any one of items 6 to 10, characterized in that a difference in aspect is specified.
(Item 12)
According to any one of items 1 to 10, the target area is an area irradiated with at least one of light in the first wavelength range and light in the second wavelength range. image processing device.
(Item 13)
The image processing device according to any one of items 1 to 12, wherein the light in the second wavelength range is light with a wavelength of 600 nm or more.
(Item 14)
The light in the first wavelength range is visible light,
14. The image processing device according to item 13, wherein the light in the second wavelength range is infrared light.
(Item 15)
a first imaging means for imaging light in the first wavelength range;
a second imaging means for imaging light in the second wavelength range;
15. The image processing device according to any one of items 1 to 14, further comprising:
(Item 16)
For the subject, a first captured image is captured of light in a first wavelength range, and a second captured image is captured of light in a second wavelength range including a wavelength longer than the first wavelength range. an acquisition process to
A setting step of setting a target area to be corrected based on the first captured image; and a setting step of setting a target area to be corrected based on the first captured image. a first correction step of generating a first corrected image in which the image of the region is corrected;
a combining step of generating a corrected image for output by combining the first corrected image generated in the first correcting step and the first captured image;
a first irradiation mode showing a distribution of intensity with which a subject in the target area is irradiated with light in the first wavelength range; and a distribution of intensity with which a subject in the target area is irradiated with light in the second wavelength range. a control step of controlling an application rate of the first corrected image in the synthesis of the synthesis step based on a difference in a second irradiation mode showing the distribution;
A method for controlling an image processing device, comprising:
(Item 17)
A program for causing a computer to function as each means of the image processing apparatus described in any one of items 1 to 14.

[Other embodiments]
The present invention provides a system or device with a program that implements one or more of the functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are hereby appended to disclose the scope of the invention.

100: Imaging device, 101: Imaging unit, 102: Image processing unit, 103: System control unit, 201: Target area detection unit, 202: First irradiation detection unit 202, 203: Second irradiation detection unit, 204: Correction unit , 401: first AC component extraction section, 402: second AC component extraction section, 403: AC component comparison section, 404: first correction section, 405: second correction section, 406: derivation section, 407: synthesis section

Claims (17)

For the subject, a first captured image is captured of light in a first wavelength range, and a second captured image is captured of light in a second wavelength range including a wavelength longer than the first wavelength range. an acquisition means to
Setting means for setting a target area to be corrected based on the first captured image; and setting means for setting a target area to be corrected in the first captured image based on a difference between the first captured image and the second captured image. a first correction means that generates a first corrected image by correcting the image of the region;
a combining unit that generates a corrected image for output by combining the first corrected image generated by the first correcting unit and the first captured image;
a first irradiation mode showing a distribution of intensity with which a subject in the target area is irradiated with light in the first wavelength range; and a distribution of intensity with which a subject in the target area is irradiated with light in the second wavelength range; A control means for controlling an application rate of the first corrected image in the synthesis of the synthesis means based on a difference in a second irradiation mode showing the distribution;
An image processing device comprising: The first correction means adjusts the AC component of the image of the target area of the first captured image based on the difference between the AC component of the first captured image and the AC component of the second captured image. The image processing apparatus according to claim 1, wherein the first corrected image is generated by reducing the amount of the image. 1 . The first correction means does not correct the first captured image for a region in which an AC component of the second captured image exhibits a pixel value equal to or greater than a predetermined threshold value. The image processing device described in . The synthesizing means includes a second correcting means that generates a second corrected image by correcting the image of the target area of the first captured image without referring to the second captured image, The image processing apparatus according to claim 1, wherein the output corrected image is generated by combining the corrected image with the second corrected image. 5. The image processing apparatus according to claim 4, wherein the second correction means reduces the AC component of the first captured image without referring to the second captured image. The control means specifies a difference between the first irradiation mode and the second irradiation mode for the subject in the target area based on a difference in brightness between the first captured image and the second captured image. The image processing device according to claim 1, characterized in that: The control means generates an average luminance distribution for each divided area in the target area for each of the first captured image and the second captured image, and 7. The image processing apparatus according to claim 6, wherein a difference in distribution of average luminance of the captured image is specified as a difference between the first irradiation mode and the second irradiation mode. The control means generates a brightness histogram for each divided area in the target area for each of the first captured image and the second captured image, and 7. The image processing apparatus according to claim 6, wherein a difference in a histogram of brightness of the image is specified as a difference between the first irradiation mode and the second irradiation mode. The control means specifies the mode of brightness for each divided area in the target area for each of the first captured image and the second captured image, and 7. The image processing apparatus according to claim 6, wherein the difference in the mode of brightness of the captured images is specified as the difference between the first irradiation mode and the second irradiation mode. The control means matches the brightness of the first captured image and the second captured image, and then controls the light in the first wavelength range and the light in the second wavelength range to the subject in the target area. 7. The image processing apparatus according to claim 6, wherein the image processing apparatus specifies a difference in irradiation mode from the irradiation mode. The target area is a skin area of a person's face,
The control means controls the first irradiation mode and the second irradiation mode based on a difference in brightness between divided regions in the target region excluding a region of a predetermined organ constituting the face. The image processing apparatus according to claim 6, characterized in that a difference in aspect is specified. The image processing apparatus according to claim 1, wherein the target area is an area that is irradiated with at least one of light in the first wavelength range and light in the second wavelength range. The image processing apparatus according to claim 1, wherein the light in the second wavelength range is light with a wavelength of 600 nm or more. The light in the first wavelength range is visible light,
The image processing apparatus according to claim 13, wherein the light in the second wavelength range is infrared light. a first imaging means for imaging light in the first wavelength range;
a second imaging means for imaging light in the second wavelength range;
The image processing apparatus according to claim 1, further comprising: an image processing apparatus; For the subject, a first captured image is captured of light in a first wavelength range, and a second captured image is captured of light in a second wavelength range including a wavelength longer than the first wavelength range. an acquisition process to
A setting step of setting a target area to be corrected based on the first captured image; and a setting step of setting a target area to be corrected based on the first captured image. a first correction step of generating a first corrected image in which the image of the region is corrected;
a combining step of generating a corrected image for output by combining the first corrected image generated in the first correcting step and the first captured image;
a first irradiation mode showing a distribution of intensity with which a subject in the target area is irradiated with light in the first wavelength range; and a distribution of intensity with which a subject in the target area is irradiated with light in the second wavelength range; a control step of controlling an application rate of the first corrected image in the synthesis of the synthesis step based on a difference in a second irradiation mode showing the distribution;
A method for controlling an image processing device, comprising: A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 14.

Images