JP2020182163A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To obtain high contrast correction effect without side effects in accordance with a scene.SOLUTION: A first luminance gain map is generated from a luminance signal, and a second luminance gain map is generated from a luminance reduction signal obtained by reducing the luminance signal. On the other hand, a first infrared gain map is generated from an infrared signal, and a second infrared gain map is generated from an infrared reduction signal obtained by reducing the infrared signal. Then, on the basis of subject information, a first composite ratio between the first luminance gain map and the second luminance gain map and a second composite ratio between the first infrared gain map and the second infrared gain map are calculated. Then, the first and second luminance gain maps are composed at first composite ratio to generates a gain map for performing gain processing, and the first and second infrared gain maps are composed at first composite ratio to generates a gain map for performing gain processing.SELECTED DRAWING: Figure 2

Description

The present invention relates in particular to image processing devices, image processing methods and programs suitable for use in locally improving image contrast.

Conventionally, there is known a technique called local tone mapping that locally improves the contrast of an image by applying a different gain to each part. As such a technique, in Patent Document 1, a plurality of stages of pixel values are generated by repeating resolution conversion with respect to the pixel values of the input image, and the difference between the pixel values of the input image and the pixel values of each stage is used. Based on this, a technique for correcting the pixel value of an input image for each step is disclosed. According to this technique, it is possible to generate an image in which the pixel value of the input image is emphasized by using the high frequency component corresponding to each resolution of the image data of the resolution of each stage having the corrected pixel value. It is said that it can be done.

Further, Patent Document 2 describes a pixel that works to generate a digital image and another pixel that mainly works to measure the relative sharpness between at least two colors on at least one region of the digital image. A technique for improving the sharpness of an image by utilizing and is disclosed.

Japanese Unexamined Patent Publication No. 2014-68330 Japanese Unexamined Patent Publication No. 2015-19378 Japanese Unexamined Patent Publication No. 2006-237737 JP-A-2018-129609 Japanese Unexamined Patent Publication No. 2014-154108

However, since the process described in Patent Document 1 does not take the wavelength of the input image into consideration, a sufficient contrast improving effect can be obtained in the case of a subject having a small signal in the visible light region, such as a leaf of a coniferous tree. It may not be possible to give. In addition, when applied to a skin-beautifying treatment that reduces the contrast of only a specific area such as a spot on human skin, the skin-beautifying treatment is applied to a subject other than the spot, which may cause a side effect of blurring the part other than the spot. ..

The process described in Patent Document 2 is that when the difference in relative sharpness of at least two colors of the pixels working to generate a digital image is small, that is, when the difference in the focal positions of the two colors is small, the sharpness of the image is small. The degree cannot be improved.

In view of the above-mentioned problems, it is an object of the present invention to make it possible to obtain an appropriate and high contrast correction effect according to the scene.

The image processing apparatus according to the present invention includes an acquisition means for acquiring a brightness signal and an infrared signal related to image data, a brightness reduction signal obtained by generating a first brightness gain map from the brightness signal, and reducing the brightness signal. A first generation means for generating a second brightness gain map from, and a second red from an infrared reduced signal obtained by generating a first infrared gain map from the infrared signal and reducing the infrared signal. A second generation means for generating an outer gain map, a first composite ratio of the first brightness gain map and the second brightness gain map based on the subject information of the image data, and the first Based on the calculation means for calculating the second composite ratio of the infrared gain map of the above and the second infrared gain map, and the first and second composite ratios calculated by the calculation means, the image data is obtained. On the other hand, it is characterized by including a third generation means for generating a gain map for performing gain processing.

According to the present invention, an appropriate and high contrast correction effect can be obtained depending on the scene.

It is a block diagram which shows the internal structure example of the image processing apparatus in Embodiment. It is a block diagram which shows the detailed configuration example of a local contrast adjustment part. It is a flowchart which shows an example of the detailed procedure of the gain map correction processing. It is a figure for demonstrating a gain map.

Hereinafter, preferred embodiments of the present invention will be described.

(Basic configuration of image processing device 100)
FIG. 1 is a block diagram showing an example of the internal configuration of the image processing device 100 according to the present embodiment. In this embodiment, an example of applying to a digital camera as an example of an image processing device will be described.
In the image processing device 100, the subject light is imaged on the image sensor 102 by an optical system 101 such as a diaphragm and a lens, and is photoelectrically converted into an electric signal to be output from the image sensor 102. The image sensor 102 is, for example, a single-plate color image sensor provided with an infrared signal filter in addition to a general primary color filter. The primary color filter consists of three types of color filters having transmission main wavelength bands in the vicinity of 650 nm, 550 nm, and 450 nm, respectively, and colors corresponding to the R (red), G (green), and B (blue) bands, respectively. Shoot the plane. The infrared signal filter has a transmission main wavelength band in the vicinity of 850 nm and captures a color plane corresponding to the IR (infrared) band.

In the single-plate color image sensor, the color filter and the infrared signal filter are spatially arranged in a mosaic pattern for each pixel, and each pixel obtains the intensity in a single color plane. Therefore, the image sensor 102 Will output a color mosaic image. The A / D conversion unit 103 converts the electric signal obtained by the image sensor 102 into digital image data and outputs it to the development processing unit 104. In the present embodiment, 12-bit image data is generated for each pixel at this point.

The development processing unit 104 performs a series of development processing such as pixel interpolation processing, luminance signal processing, and color signal processing on the image data. In the present embodiment, R, G, B, and IR information for each pixel of the image data is acquired by the pixel interpolation processing. As a method for acquiring R, G, B, and IR information from an image sensor having an infrared light receiving pixel, for example, a known technique as described in Patent Document 3 is used. The development processing unit 104 converts the color spaces of R, G, and B into the color spaces of 8-bit luminance (Y) data and color difference (U, V) data, and outputs them as YUV data. Further, IR is converted into 8 bits and output as an infrared signal.

The infrared information acquisition unit 112 acquires infrared information (infrared signal) for each pixel of the image data. In the present embodiment, the infrared information acquisition unit 112 acquires the infrared signal of the subject output from the development processing unit 104 as infrared information. However, the present embodiment is not limited to this, and for example, an infrared element is provided separately from the image pickup element for acquiring a captured image, and the infrared element is used, for example, an A / D conversion unit. The infrared signal may be acquired without using the image data output from 103.

The local contrast adjustment unit 105 performs a local contrast adjustment process described later on the image data output from the development processing unit 104. The detailed configuration of the local contrast adjustment unit 105 will be described later.

The signal processing unit 106 performs resizing processing or the like on the image data that has undergone the local contrast adjustment processing, and supplies the image data to the output unit 107. The output unit 107 outputs image data to an output interface such as HDMI (registered trademark), records it on a recording medium such as a semiconductor memory card, and outputs it to a display device (not shown) of the image processing device 100. To do.

The UI unit 109 is an input device such as a touch panel provided on a switch, a button, or a display device (not shown), and an external operation such as an instruction by a user is sent to the image processing device 100 via the UI unit 109. Entered.

The control unit 110 controls each unit via the bus 108, and also performs necessary arithmetic processing as appropriate. Further, the control unit 110 receives an input signal via the UI unit 109 to perform an calculation and controls each unit.

The memory 111 stores image data used in each processing unit and shooting information such as aperture value, shutter speed, shooting mode, ISO sensitivity, white balance gain value, and color gamut settings such as s-RGB. The stored shooting information is appropriately read out according to the instruction of the control unit 110 and used as subject information. The components shown in FIG. 1 are communicably connected to each other via the bus 108.

(Details of local contrast adjustment processing)
Hereinafter, a detailed configuration for performing the local contrast adjustment processing in the image processing apparatus 100 according to the present embodiment will be described with reference to FIG. In the present embodiment, the gain map generated by using the image data and the infrared gain map generated by using the infrared signal are corrected according to the subject information to give a high contrast correction effect to the image data without any side effect. Generate a gain map. Here, the gain map refers to an image in which the gain to be applied is determined according to the position of the screen, as shown in FIG.

The local contrast adjustment unit 105 of the present embodiment performs gain processing on the input image data and outputs the processed image data. The image data input / output by the local contrast adjustment process of the present embodiment is composed of a luminance signal.

FIG. 2 is a block diagram showing a detailed configuration example of the local contrast adjustment unit 105.
The reduced image generation unit 202 performs a reduction process on the input signal (luminance signal) to generate a brightness reduction signal of a low frequency component. The infrared reduced image generation unit 206 performs reduction processing on the infrared signal input from the infrared information acquisition unit 112 to generate an infrared reduced signal having a low frequency component. As the method of reduction processing, a general method such as reduction processing using the bilinear method is used.

The first gain conversion unit 201 generates the first luminance gain map by applying the first luminance characteristic to the luminance signal. The second gain conversion unit 203 generates a second luminance gain map by applying the first luminance characteristic to the luminance reduction signal. The first infrared gain conversion unit 205 generates the first infrared gain map by applying the first gradation characteristic to the infrared signal. The second infrared gain conversion unit 207 generates a second infrared gain map by applying the first gradation characteristic to the infrared reduced signal. Regarding the method of generating the gain map from the input signal, as described in Patent Document 4, the first floor in which the monotonically increasing part is uniformly converted to the monotonically decreasing characteristic with respect to the characteristic originally desired to be applied by the gradation processing. A known method of applying tonal characteristics is used.

The hierarchical gain map compositing unit 204 generates a luminance composite gain map by compositing the first luminance gain map and the second luminance gain map. The infrared layered gain map synthesizing unit 208 generates an infrared composite gain map by synthesizing the first infrared gain map and the second infrared gain map. Regarding the method of generating the composite gain map, for example, as described in Patent Document 5, the gain signal of the gain map having a large image size and the gain signal of the gain map having a small image size are weighted and added according to the signal difference of the gain. Use a known method.

The contrast correction determination unit 211 calculates the composite ratio of the first luminance gain map and the second luminance gain map using the subject information input from the control unit 110, and outputs the composite ratio to the hierarchical gain map composite unit 204. Further, the contrast correction determination unit 211 also calculates the composition ratio of the first infrared gain map and the second infrared gain map, and outputs the composition ratio to the infrared layer gain map composition unit 208. Further, the contrast correction determination unit 211 calculates the gain map correction ratio and outputs it to the gain map correction unit 209.

The gain map correction unit 209 inputs the gain map correction ratio from the contrast correction determination unit 211, the infrared composite gain map from the infrared hierarchical gain map composite unit 208, and the luminance composite gain map from the hierarchical gain map composite unit 204. Then, the gain map correction unit 209 corrects the luminance composite gain map by using the gain map correction ratio and the infrared composite gain map.
The gain processing unit 210 performs gain processing on the input image data using the corrected brightness composite gain map, and outputs the image data.

Next, the gain map correction process of the present embodiment will be described with reference to the flowchart of FIG.
FIG. 3 is a flowchart showing an example of a detailed procedure of the gain map correction process. Each operation of this flowchart is performed by each unit according to the instruction of the control unit 110 or the control unit 110.
In step S301, the hierarchical gain map synthesizing unit 204 inputs the first luminance gain map and the second luminance gain map from the first gain conversion unit 201 and the second gain conversion unit 203, respectively.

In step S302, the infrared layer gain map synthesizing unit 208 converts the first infrared gain map and the second infrared gain map into the first infrared gain conversion unit 205 and the second infrared gain conversion unit, respectively. Input from 207.
In step S303, the contrast correction determination unit 211 inputs subject information from the control unit 110. In the present embodiment, for example, information on the shooting mode of the camera at the time of shooting is used as the subject information.

In step S304, the contrast correction determination unit 211 determines the necessity of skin-beautifying treatment using the subject information. For example, when the shooting mode of the camera is a mode for shooting a person such as a portrait mode, it is determined that skin-beautifying treatment is necessary, and the process proceeds to step S305. On the other hand, if the skin-beautifying treatment is unnecessary, the process proceeds to step S306.

In step S305, first, the contrast correction determination unit 211 generates information necessary for generating a gain map for skin beautification treatment. The purpose of the skin-beautifying treatment in the present embodiment is to reduce stains on human skin. Here, the stain has two characteristics. Hereinafter, the procedure for generating the gain map in step S305 will be described based on the two characteristics related to the stain.

The first feature is that the spots that should be reduced by the skin-beautifying treatment in the portrait image contain a lot of low-frequency components. If the face of a person occupying the shooting angle of view is made smaller, the stains on the skin of the person may be contained in a few pixels of the image sensor, and the stains may be composed of only high-frequency components. However, in this case, since the face and the stains with respect to the angle of view are small, there is no possibility that the stains are anxious, and it is not necessary to perform the stain reduction treatment in the first place.

Therefore, in order to reduce the stain by adjusting the local contrast, a second luminance gain map generated from the luminance reduced image may be used. That is, the contrast correction determination unit 211 calculates a composite ratio in which the addition ratio of the first luminance gain map is 0 and the addition ratio of the second luminance gain map is skin beautification strength × α. The skin-beautifying intensity in this embodiment is determined by the shooting mode as 3 (weak) in the boat rate mode and 5 (strong) in the skin-beautifying mode or the self-shooting mode. For example, the user can set it from the UI unit 109. You may. On the other hand, the coefficient α is an adjustment coefficient for adjusting the skin-beautifying process according to the characteristics of the image signal input from the image sensor 102 and the contents of the developing processing unit 104.

In step S305, the hierarchical gain map compositing unit 204 further inputs a compositing ratio from the contrast correction determination unit 211, generates a luminance composite gain map according to the input compositing ratio, and outputs it to the gain map correction unit 209. Further, the contrast correction determination unit 211 outputs a control signal that inverts the sign of the luminance composite gain map to the gain map correction unit 209.

The second feature is that an infrared signal with a long wavelength makes it difficult to image a stain near the surface of the skin, so that the infrared signal contains almost no stain component. Since the low-frequency components of the luminance signal include signals other than stains, if local contrast correction is performed using a gain map in which the sign of the luminance composite gain map is inverted, the eyes, nose, mouth, etc., along with the stains, etc. The contrast of the signal indicating the facial features of the subject is also lost. On the other hand, in the low frequency component of the infrared signal, there is only a signal showing the facial features of the subject excluding stains.

Therefore, in the present embodiment, the contrast of the signal indicating the facial features of the subject, which is lowered by the gain map obtained by reversing the luminance composite gain map, is improved by using the infrared signal. Specifically, the contrast correction determination unit 211 calculates a composite ratio in which the addition ratio of the first infrared gain map is 0 and the addition ratio of the second infrared gain map is skin beautification strength × β. Here, the coefficient β is an adjustment coefficient for adjusting the skin-beautifying treatment according to the characteristics of the image signal input from the image sensor 102, the characteristics of the infrared signal, and the contents of the developing processing unit 104.

In step S305, the infrared layer gain map synthesizing unit 208 further inputs the synthesizing ratio from the contrast correction determination unit 211, generates an infrared synthesizing gain map according to the input synthesizing ratio, and outputs it to the gain map correction unit 209. To do. The contrast correction determination unit 211 outputs a control signal for adding the infrared composite gain map to the luminance composite gain map to the gain map correction unit 209. The gain map correction unit 209 generates a gain map in which the luminance composite gain map is corrected by the infrared composite gain map according to the control signal.

By performing the gain processing using the gain map generated in step S305 as described above, it is possible to reduce skin spots without lowering the contrast of signals indicating the characteristics of the subject such as eyes, nose, and mouth. it can.

In step S306, the contrast correction determination unit 211 determines the necessity of coniferous tree processing using the subject information. For example, when the shooting mode of the camera is a mode for landscape shooting such as landscape mode, it is determined that coniferous tree processing is necessary, and the process proceeds to step S307. On the other hand, if the coniferous tree treatment is not required, the process proceeds to step S308.

In step S307, first, the contrast correction determination unit 211 generates information necessary for generating a gain map for conifer processing. The purpose of the softwood treatment in the present embodiment is to improve the resolution of the softwood. Here, the softwood has two characteristics, the first is that the softwood contains a large amount of high-frequency components, and the second is that the output luminance signal is small because the softwood is dark green, but red. Since many external components are reflected, the output infrared signal is large.

Therefore, in order to improve the resolution of coniferous trees by adjusting the local contrast, the first infrared gain map generated from the infrared signal may be used. That is, the contrast correction determination unit 211 calculates the composition ratio in which the addition ratio of the first infrared gain map is γ and the addition ratio of the second infrared gain map is 0. Here, the coefficient γ is an adjustment coefficient for adjusting the softwood processing according to the characteristics of the image signal input from the image sensor 102, the characteristics of the infrared signal, and the contents of the developing processing unit 104.

In step S307, the infrared layer gain map synthesizing unit 208 further inputs a synthesizing ratio from the contrast correction determination unit 211, generates an infrared synthesizing gain map according to the input synthesizing ratio, and outputs it to the gain map correction unit 209. To do.

In addition, since the mode for landscape photography is selected, it is necessary to improve the contrast of high-frequency components for subjects other than coniferous trees. Therefore, the first luminance gain map generated from the luminance signal may be used. That is, the contrast correction determination unit 211 calculates the composition ratio in which the addition ratio of the first luminance gain map is θ and the addition ratio of the second luminance gain map is 0. Here, the coefficient θ is an adjustment coefficient for adjusting the local contrast processing for the landscape according to the characteristics of the image signal input from the image sensor 102 and the contents of the developing processing unit 104.

In step S307, the hierarchical gain map compositing unit 204 further inputs a compositing ratio from the contrast correction determination unit 211, generates a luminance composite gain map according to the input compositing ratio, and outputs it to the gain map correction unit 209. The contrast correction determination unit 211 outputs a control signal for adding the infrared composite gain map to the luminance composite gain map to the gain map correction unit 209. The gain map correction unit 209 generates a gain map in which the luminance composite gain map is corrected by the infrared composite gain map according to the control signal.

By performing the gain processing using the gain map generated in step S307 as described above, the resolution of the coniferous tree can be improved.

In step S308, the mode is other than the mode for portrait photography and the mode for landscape photography, and neither skin beautification treatment nor coniferous tree treatment is required. Therefore, the contrast correction determination unit 211 generates information necessary for generating a general-purpose gain map suitable for a wide range of scenes. In this case, since it is not necessary to use infrared information, the contrast correction determination unit 211 sets the composition ratio in which the addition ratio of the first infrared gain map and the addition ratio of the second infrared gain map are both 0. Generate. On the other hand, an arbitrary value is calculated for the composite ratio of the luminance layer gain map.

The hierarchical gain map compositing unit 204 inputs a compositing ratio from the contrast correction determination unit 211, generates a luminance compositing gain map according to the input compositing ratio, and outputs it to the gain map correction unit 209. The contrast correction determination unit 211 outputs a control signal that does not add the infrared composite gain map to the gain map correction unit 209. The gain map correction unit 209 adopts the luminance composite gain map as it is as a gain map according to the control signal.

By performing the gain processing using the gain map generated in step S308 as described above, local contrast correction suitable for a wide range of scenes can be performed.

In step S309, the gain map correction unit 209 outputs the generated gain map to the gain processing unit 210.

As described above, according to the present embodiment, the luminance signal and the infrared signal are controlled for each frequency band according to the subject information, and the local contrast is adjusted. As a result, a contrast correction effect that cannot be obtained by the contrast correction process using only the luminance signal can be obtained. In addition, it is possible to perform contrast correction with less side effects than contrast correction processing using only the luminance signal.
In the present embodiment, a method of generating and synthesizing a gain map is used as a method of adjusting the local contrast, but it is not always necessary to use the gain map if the gradation correction amount for each region can be specified.

Although each configuration of the local contrast adjustment unit 105 has been described in this embodiment, the operation of each block may be realized by using a dedicated circuit or by software. That is, a part or all of each operation of the local contrast adjusting unit 105 may be implemented by software processing. Further, with respect to the other blocks in the image processing apparatus 100 shown in FIG. 1, a part or all of them may be similarly implemented by software processing.

(Other embodiments)
In the above-described embodiment, the digital camera has been described as an example of the image processing device. On the other hand, an image processing device having an image pickup system such as a scanner may be used, and the embodiment is not particularly limited as long as it is an image processing device capable of processing image data. That is, the image processing device may be an information processing device such as a personal computer, or an image forming device such as a portable information terminal or a printer.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

104 Development processing unit 112 Infrared information acquisition unit 201 First gain conversion unit 203 Second gain conversion unit 205 First infrared gain conversion unit 207 Second infrared gain conversion unit 209 Gain map correction unit 211 Contrast correction Judgment unit

Claims (9)

An acquisition means for acquiring a luminance signal and an infrared signal related to image data,
A first generation means for generating a first luminance gain map from the luminance signal and a second luminance gain map from the luminance reduction signal obtained by reducing the luminance signal.
A second generation means for generating a first infrared gain map from the infrared signal and a second infrared gain map from an infrared reduced signal obtained by reducing the infrared signal.
Based on the subject information of the image data, the first composite ratio of the first brightness gain map and the second brightness gain map, and the first infrared gain map and the second infrared gain A calculation method for calculating the second composite ratio with the map,
A third generation means for generating a gain map for performing gain processing on the image data based on the first and second composite ratios calculated by the calculation means.
An image processing device characterized by comprising. The image processing apparatus according to claim 1, wherein the subject information is information related to a shooting mode of the image data. When the shooting mode is a mode intended for portrait photography, the calculation means calculates the first composite ratio so as to increase the composite ratio of the second luminance gain map, and also calculates the second composite ratio. The second composite ratio was calculated so as to increase the composite ratio of the infrared gain map.
The third generation means according to claim 2, wherein the gain map for performing the gain processing is generated by inverting the sign of the gain map synthesized at the first synthesis ratio. Image processing device. When the shooting mode is a mode for landscape shooting, the calculation means calculates the first composite ratio so as to increase the composite ratio of the first luminance gain map, and also calculates the first composite ratio. The image processing apparatus according to claim 2, wherein the second composition ratio is calculated so as to increase the composition ratio of the infrared gain map. When the shooting mode is a mode other than the mode for capturing a person and the mode for shooting a landscape, the third generation means is based on the first composite ratio calculated by the calculation means. The image processing apparatus according to claim 2, wherein a gain map obtained by combining the first luminance gain map and the second luminance gain map is generated as a gain map for performing the gain processing. The image processing apparatus according to any one of claims 1 to 5, wherein the acquisition means acquires the infrared signal via an image sensor provided with an infrared signal filter. The image processing apparatus according to any one of claims 1 to 5, wherein the acquisition means acquires the infrared signal via an infrared element. The acquisition process for acquiring the luminance signal and infrared signal related to the image data,
A first generation step of generating a first luminance gain map from the luminance signal and generating a second luminance gain map from the luminance reduced signal obtained by reducing the luminance signal.
A second generation step of generating a first infrared gain map from the infrared signal and generating a second infrared gain map from an infrared reduced signal obtained by reducing the infrared signal.
Based on the subject information of the image data, the first composite ratio of the first brightness gain map and the second brightness gain map, and the first infrared gain map and the second infrared gain The calculation process to calculate the second composite ratio with the map,
A third generation step of generating a gain map for performing gain processing on the image data based on the first and second composite ratios calculated in the calculation step, and a third generation step.
An image processing method characterized by comprising. The acquisition process for acquiring the luminance signal and infrared signal related to the image data,
A first generation step of generating a first luminance gain map from the luminance signal and generating a second luminance gain map from the luminance reduced signal obtained by reducing the luminance signal.
A second generation step of generating a first infrared gain map from the infrared signal and generating a second infrared gain map from an infrared reduced signal obtained by reducing the infrared signal.
Based on the subject information of the image data, the first composite ratio of the first brightness gain map and the second brightness gain map, and the first infrared gain map and the second infrared gain The calculation process to calculate the second composite ratio with the map,
A third generation step of generating a gain map for performing gain processing on the image data based on the first and second composite ratios calculated in the calculation step, and a third generation step.
A program that lets your computer run.

Images