JP6701299B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing device and an image processing method, and particularly to a technique for correcting the brightness of an image.

  BACKGROUND ART Conventionally, there is known a technique (rewriting) for correcting brightness by irradiating a subject in an image with light from a virtual light source (Patent Document 1). As a result, it is possible to lighten a dark area such as a shadow generated by ambient light, and for example, it is possible to correct an image so that a subject existing in a blackened area can be identified.

  For example, when performing relighting on a face area, a brightness area that is lower than the average brightness of the entire face area is extracted as a shadow area, and the brightness of the shadow area is increased so that the brightness of other areas is not affected. The shadow of the face area can be suppressed.

JP, 2010-135996, A

  In the method of Patent Document 1, the brightness of the shadow area is increased by increasing the gain. However, when the gain-up is used, not only the signal component but also the noise component is amplified, and the S/N of the corrected area is reduced. In Patent Document 1, a noise removal filter is applied in order to suppress this decrease in S/N. However, since the noise removal process is a smoothing process, the detail of the image may be reduced.

  The present invention has been made in view of the problems of the related art as described above, and provides an image processing apparatus and an image processing method capable of correcting the brightness of pixels while suppressing noise amplification and reduction in detail of an object. With the goal.

The above-described object is to correct the brightness of the first area by applying the effect of irradiating the first area of the target image with virtual light, and the correction means corrects the brightness of the first area. Determining means for determining in the target image a second area to be referred to at the time, and the correcting means, when correcting the luminance of the first area, uses a correction signal based on the color information of the second area. Is used to correct the color signal of the first area so that the color information of the first area approaches the color information of the second area.

  According to the present invention having such a configuration, it is possible to provide an image processing apparatus and an image processing method capable of correcting the brightness of pixels while suppressing noise amplification and reduction in detail of an object.

It is a block diagram which shows the structure of the digital camera in this invention. It is a block diagram which shows the structure of the image processing part in this invention. It is a figure which shows the flowchart which shows the process of the ambient light reflection characteristic extraction part in the 1st Embodiment of this invention. It is a figure which shows the to-be-photographed object example in embodiment of this invention. It is a flowchart which shows the process of virtual light source addition in the 1st Embodiment of this invention. It is a block diagram which shows the structure of the relighting process part in the 1st Embodiment of this invention. It is a figure which shows the relationship between the to-be-photographed object and virtual light source in this invention. It is a flowchart which shows the process of virtual light source addition in the 2nd Embodiment of this invention. It is a block diagram which shows the structure of the relighting process part in the 2nd Embodiment of this invention.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the embodiments described below, an example in which the image processing device according to the present invention is applied to a digital camera will be described. Note that a digital camera means an electronic device having a photographing function using a photoelectric conversion element, and includes any electronic device having a camera or usable, such as a mobile phone, a game machine, or a personal computer. Further, the photographing function is not essential to the present invention, and the image processing apparatus according to the present invention can be applied to any electronic device capable of image processing.

● (First embodiment)
FIG. 1 is a block diagram showing a configuration example of a digital camera 100 according to the first embodiment of the present invention.
In FIG. 1, a lens group 101 is a zoom lens including a focus lens. A shutter 102 having a diaphragm function is provided between the lens group 101 and the image pickup unit 103. The image pickup unit 103 has an image pickup device represented by a CCD/CMOS image sensor that converts an optical image formed on the image pickup surface by the lens group 101 into an electric signal in units of pixels. The A/D converter 104 converts the analog signal output by the image capturing unit 103 into a digital signal (image data).

  The image processing unit 105 performs various image processing such as color interpolation (demosaic), white balance adjustment, γ correction, contour enhancement, noise reduction, and color correction on the image data output from the A/D converter 104. The image memory 106 temporarily stores image data. The memory control unit 107 controls reading and writing of the image memory 106. The D/A converter 108 converts the image data into an analog signal. The display unit 109 has a display device such as an LCD or an organic EL display, and displays various GUIs, live view images, images read from the recording medium 112 and reproduced, and the like. The codec unit 110 encodes the image data stored in the image memory 106 by a predetermined method for recording on a recording medium, or decodes the encoded image data included in the image file for display, for example. Or

  The interface (I/F) 111 mechanically and electrically connects a removable recording medium 112 such as a semiconductor memory card or a card-type hard disk to the digital camera 100. The system controller 50 may be a programmable processor such as a CPU or MPU. The system control unit 50 realizes the function of the digital camera 100 by executing a program recorded in, for example, the nonvolatile memory 121 or a built-in nonvolatile memory to control necessary blocks and circuits. The relighting processing unit 114 performs relighting processing on the captured image.

  The operation unit 120 is a collective description of buttons and switches for the user to input various instructions to the digital camera 100.

The non-volatile memory 121 may be an electrically erasable/recordable EEPROM, for example. The nonvolatile memory 121 stores various setting values, GUI data, and a program to be executed by the system control unit 50 when the system control unit 50 is an MPU or a CPU.
The system memory 122 is used to expand constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 121, and the like.

Next, the operation of the digital camera 100 during shooting will be described.
For example, the image capturing unit 103 photoelectrically converts a subject image formed on the image capturing surface by the lens group 101 when the shutter 102 is open, and outputs the subject image to the A/D converter 104 as an analog image signal. The A/D converter 104 converts the analog image signal output from the imaging unit 103 into a digital image signal (image data) and outputs the digital image signal to the image processing unit 105.

  The image processing unit 105 performs various types of color interpolation (demosaic), γ correction, edge enhancement, noise reduction, color correction, etc. on the image data from the A/D converter 104 or the image data from the memory control unit 107. Perform image processing.

  Further, in the image processing unit 105, a predetermined calculation process regarding brightness, contrast, etc. is performed using the image data obtained by photographing, and the system control unit 50 performs distance measurement control and exposure control based on the obtained calculation result. To do. As described above, the digital camera 100 according to the present embodiment performs AF (auto focus) processing and AE (auto exposure) processing of the TTL (through-the-lens) method. The image processing unit 105 also performs automatic white balance (AWB) adjustment using image data obtained by shooting.

  The image data output from the image processing unit 105 is written in the image memory 106 via the memory control unit 107. The image memory 106 stores the image data output from the imaging unit 103 and the image data to be displayed on the display unit 109.

  Further, the D/A converter 108 converts the image display data stored in the image memory 106 into an analog signal and supplies the analog signal to the display unit 109. The display unit 109 displays on a display device such as an LCD according to the analog signal from the D/A converter 108.

  The codec section 110 encodes the image data recorded in the image memory 106 based on a standard such as JPEG or MPEG. The system control unit 50 adds a predetermined header or the like to the encoded image data to form an image file, and records the image file on the recording medium 112 via the interface 111.

  In a current digital camera, it is general that a moving image is shot in a shooting standby state, and the shot moving image is continuously displayed on the display unit 109 so that the display unit 109 functions as an electronic viewfinder (EVF). is there. In this case, the shutter 102 is opened, and the so-called electronic shutter of the image capturing unit 103 is used to capture images at, for example, 30 frames/sec.

  Then, when the shutter button included in the operation unit 120 is half-pressed, the above-described AF and AE control is performed, and when the shutter button included in the operation unit 120 is fully pressed, the still image shooting for recording is executed by the main shooting and is recorded in the recording medium 112. It Further, when the moving image shooting is instructed by the moving image shooting button or the like, the moving image recording on the recording medium 112 is started.

FIG. 2 is a block diagram showing a functional configuration example of the image processing unit 105.
The image data output from the A/D converter 104 in FIG. 1 is input to the luminance/color signal generation unit 200. The image data has a value corresponding to one of the color components forming the color filter provided in the image sensor. When a generally used primary color filter of Bayer array is used, the image data is composed of R pixel data, G pixel data, and B pixel data.

  The luminance/color signal generation unit 200 performs demosaic processing on such image data, generates color signals R, G, B for each pixel, and further generates a luminance signal Y from the color signals. The luminance/color signal generation unit 200 outputs the generated color signals R, G, and B to the white balance (WB) amplification unit 203 and the luminance signal Y to the contour enhancement processing unit 201.

  The contour enhancement processing unit 201 performs contour enhancement processing on the luminance signal Y and outputs it to the luminance gamma processing unit 202. The luminance gamma processing unit 202 performs gamma correction on the luminance signal Y and outputs the corrected luminance signal Y to the image memory 106.

  The WB amplification unit 203 applies a gain to the color signals R, G, B based on the white balance gain value calculated by the system control unit 50 by the process described below, and adjusts the white balance. The color conversion processing unit 204 converts the color signals R, G, B into a desired color balance by a matrix operation or the like. The color gamma processing unit 205 performs gamma correction on the color signals R, G, B. The color difference signal generation unit 206 generates color difference signals RY and BY from the color signals R, G, and B and outputs them to the image memory 106.

  The luminance signal Y and the color difference signals RY and BY output to the image memory 106 are encoded by the codec unit 110 and finally recorded on the recording medium 112.

  The color signals R, G, B output from the color conversion processing unit 204 are also input to the light source characteristic extraction unit 207. The light source characteristic extraction unit 207 extracts the reflection characteristic information of the subject under the environment light source as the light source characteristic information and records it in the system memory 122.

The operation of the light source characteristic extraction unit 207 will be described with reference to the flowchart of FIG. The extraction of the light source characteristic is executed for each pixel.
In S301, the light source characteristic extraction unit 207 generates a luminance (L) signal, a hue (H) signal, and a saturation (S) signal from the color signals R, G, and B of the processing target pixel.

  In S302, the light source characteristic extraction unit 207 determines whether the processing target pixel is included in a predetermined characteristic region, for example, the face region detected by the face detection unit 113 (401 in FIG. 4A), and the pixel is included. If not included, the process ends, and if included, the process proceeds to S303.

  In S303, the light source characteristic extraction unit 207 determines whether or not the color of the pixel to be processed corresponds to the specific color. Here, since the predetermined characteristic region is the face region, it is determined whether or not it is included in the range of the color space that defines the predetermined skin color range. FIG. 4B shows an example of the skin color region 410 defined in the CrCb (or UV) color space. Note that in the determination in S303, the HS coordinate system may be used, or another color space coordinate system that can be converted from the color signals R, G, B as shown in FIG. 4B. ..

  When it is determined in S303 that the color of the processing target pixel is included in the predetermined range of the color space, the light source characteristic extraction unit 207 advances the processing to S304, and when it is determined that the color is not included, the processing of the target pixel is performed. To finish. In the image example of FIG. 4A, the light source characteristic extraction unit 207 advances the processing to S304 when the pixels 403 and 404 included in the skin color area 410 are processed, and processes the pixels 402 not included in the skin color area 410. If so, the process ends.

  In S304, the light source characteristic extraction unit 207 determines whether or not the brightness signal value of the pixel to be processed corresponds to a predetermined high brightness value, based on whether the brightness signal value is equal to or more than a predetermined value. S304 is a process of extracting a subject region that is strongly exposed to ambient light. In the image example of FIG. 4A, when the pixel 403 that is a high-luminance pixel is the processing target, the light source characteristic extraction unit 207 advances the processing to step S<b>305, and the pixel 404 with low luminance due to the effect of being shaded or the like. If is a processing target, the processing ends.

In S305, the light source characteristic extraction unit 207 integrates, for each color component, the color signals R, G, and B of the processing target pixel that satisfies all the conditions of S302 to S304. The integration result is (Rt, Gt, Bt).
In S306, the light source characteristic extraction unit 207 counts (+1) the number of integrated pixels. Let N be the number of counts.

  The above is the processing of the light source characteristic extraction unit 207. By this processing, it is possible to extract the information of the pixel area as shown in FIG. 4A 403 which is included in a specific part of the subject, has a specific color, and is exposed to ambient light and has high brightness.

  Note that, here, a case has been described in which, for example, pixels included in the entire captured image or a predetermined part thereof are sequentially processed. However, the light source characteristic extraction process may be performed with pixels included in a specific subject region such as a face region as a processing target. In this case, the determination in S302 is unnecessary.

  Next, the relighting process for the image output from the image processing unit 105 will be described. In the present embodiment, the relighting processing is executed by the relighting processing unit 114 using the control parameter calculated by the system control unit 50. Therefore, first, the calculation and setting process of the control parameter used in the relighting process will be described with reference to the flowchart of FIG.

  In step S501, the system control unit 50 receives a relighting processing instruction from the user through the operation unit 120. Specifically, the system control unit 50 accepts a user operation on the operation unit 120 that selects execution of relighting processing from a menu (not shown). Further, the system control unit 50 accepts, for example, a relighting process parameter specified on the menu screen together with the relighting process execution instruction.

  In the present embodiment, the position of the virtual light source and the intensity (α) of the light source are input by the user as parameters for relighting in the present embodiment (a method of selecting from preset options may be used). Assuming that the virtual light source illuminates the subject surface vertically, if the position of the virtual light source is determined, the position where the light of the virtual light source strikes most (center irradiation position) is also determined.

  An example of the position of the virtual light source and the center irradiation position is schematically shown in FIG. FIG. 7A shows a state in which the virtual light source 701 has an irradiation range 703 and a central irradiation position 702. The virtual light source only affects the irradiation range 703.

  Returning to FIG. 5A, in S502, the system control unit 50 calculates the highlight reflection characteristics (Rs, Gs, Bs) of the specific color. The highlight reflection characteristic of a specific color is an RGB average value of a highlight portion (high-luminance portion) of a reflected light of ambient light at the time of shooting, which is reflected by a specific color portion of a subject. In the present embodiment, the light source characteristic extraction unit 207 obtains an integral value (Rt, Gt, Bt) of a pixel having a flesh color and high brightness in the subject area. This is because the diffuse reflection component that reflects the object color is dominant in the low luminance part, while the highlight (high luminance) part includes both the diffuse reflection component and the specular reflection component that reflects the environment light source color. It depends on going out.

  The system control unit 50 divides the integrated value (Rt, Gt, Bt) calculated by the light source characteristic extraction unit 207 by the count number N calculated in S306, and the color of the highlight part where the ambient light strongly hits the skin color area. (Rs, Gs, Bs) is calculated as the flesh color highlight reflection characteristic.

In step S<b>503, the system control unit 50 generates the color of the virtual light source to be emitted as a gain (coefficient) for converting and expressing the color of the ambient light at the time of shooting, and sets it in the relighting processing unit 114. The color gain (Rv-Gain, Bv-Gain) of the virtual light source is arbitrarily set by the user, and when the virtual light source and the environmental light source have the same color,
Rv-Gain=1
Bv-Gain=1
Becomes

  In step S504, the system control unit 50 instructs the relighting processing unit 114 to execute the relighting process.

Next, an example of the configuration and operation of the relighting processing unit 114 will be described.
FIG. 6 is a block diagram showing the configuration of the relighting processing unit 114.
In FIG. 6, an RGB signal conversion unit 601 converts the luminance and color difference signals (Y, BY, RY) input from the image memory 106 into color signals (R, G, B). The degamma processing unit 602 performs degamma processing (reverse processing of the luminance gamma processing unit 202 and the color gamma processing unit 205).

The virtual light source processing unit 603 adds the lighting effect of the virtual light source to the image. The smoothing processing unit 604 smoothes the color signals (R, G, B) output from the degamma processing unit 602 and outputs the smoothed color signals to the virtual light source processing unit 603. The hue/saturation conversion unit 605 converts the RGB signal output from the degamma processing unit 602 into a hue and saturation signal (H, S).
The gamma processing unit 606 performs gamma correction on the color signals (R, G, B) output by the virtual light source processing unit 603. The luminance color difference signal generation unit 607 converts the gamma-corrected color signals (R, G, B) into luminance and color difference signals (Y, BY, RY) and outputs them to the image memory 106.

Next, the operation of the relighting processing unit 114 will be described.
The relighting processing unit 114 reads out and inputs the luminance/color difference signals (Y, BY, RY) recorded in the image memory 106. The RGB signal conversion unit 601 converts the input luminance/color difference signals (Y, BY, RY) into RGB signals and outputs them to the degamma processing unit 602.

  The degamma processing unit 602 performs a calculation of a characteristic opposite to the gamma characteristic multiplied by the gamma processing unit of the image processing unit 105 and converts it into linear data. The RGB signal after the linear conversion is output to the virtual light source processing unit 603, the smoothing processing unit 604, and the hue/saturation conversion unit 605. In addition, the degamma processing unit 602 may be arranged in front of the RGB signal conversion unit 601 to perform the degamma processing. For example, the reverse processing of the luminance gamma processing unit 202 is applied to the Y signal, and the reverse processing of the color gamma processing unit 205 is applied to the color difference signals RY and BY after returning to the RGB signals. , And the color difference signals R-Y and B-Y are converted again, and then input to the RGB signal conversion unit 601.

  The smoothing processing unit 604 replaces each color component of the RGB format color signal (Rt, Gt, Bt) after the degamma processing of the pixel to be processed with the average value of the color signal values of the peripheral pixels. For example, an average value of 9 pixels included in an area of 3×3 pixels centered on the processing target pixel is calculated for each color component, and is set as a color component value of the target pixel after smoothing. Note that this method is an example of smoothing, and any other method can be used. The smoothing processing unit 604 outputs the smoothed color signals (Ra, Ga, Ba) to the virtual light source processing unit 603. The color signals (Ra, Ga, Ba) obtained by the smoothing process represent the reflection characteristics of ambient light in the vicinity of the pixel to be processed.

  The hue/saturation conversion unit 605 converts the input RGB color signal (Rt, Gt, Bt) after degamma processing into a hue (H) and a saturation (S), and converts the converted hue (H) and The saturation (S) signal is output to the virtual light source processing unit 603.

The details of the virtual light source processing performed by the virtual light source processing unit 603 will be described with reference to the flowchart of FIG. The virtual light source process is a process of adding the relighting effect by the virtual light source to the input image.
The virtual light source processing unit 603 executes the virtual light source processing shown in FIG. 5B for each pixel included in the input image.

  In step S510, the virtual light source processing unit 603 determines whether the pixel to be processed is located within the irradiation range of the virtual light source. In the example illustrated in FIG. 7A, the virtual light source processing unit 603 determines whether the processing target pixel is located within the irradiation range 703 of the virtual light source 701 at the central irradiation position 702. This determination is performed, for example, by obtaining the irradiation range 703 on the image using a predetermined function with the position of the central irradiation position 702 and the position of the virtual light source 701 as parameters, and comparing it with the image coordinates of the processing target pixel. You can The position of the virtual light source 701 may be fixed or variable, or a different function may be used depending on the type of virtual light source. Further, the user may be allowed to change the spreading method of the luminous flux.

  The virtual light source processing unit 603 advances the process to S511 when it is determined that the processing target pixel is located within the irradiation range of the virtual light source, and when it is determined that the processing target pixel is not located within the irradiation range of the virtual light source. The process ends without adding the relighting effect.

  In step S511, the virtual light source processing unit 603 determines whether the color of the pixel to be processed corresponds to the specific color. Here, the specific color is the human skin color, and the virtual light source processing unit 603 determines whether or not the region of the color space corresponding to the skin color, which is stored in advance, includes the color of the pixel to be processed. And The determination can be performed by converting the pixel value into a color space coordinate system that defines a specific color.

  As described above, in the present embodiment, the skin color area 410 is defined in the CrCb (or UV) color space shown in FIG. 4B, but it may be defined in another color space coordinate system. Here, since the determination is performed using the hue signal (H) and the saturation signal (S) of the same phase generated by the hue/saturation conversion unit 605, the virtual light source processing unit 603 uses the hue and saturation signal of the processing target pixel. Whether or not is included in the skin color area is determined. If it is determined that the color of the pixel to be processed is included in the flesh color area (corresponding to the specific color), the virtual light source processing unit 603 advances the process to S512, and if it is determined that the color is not included in the flesh color area, the process is performed. Proceed to S513.

  In S512 and S513, the virtual light source processing unit 603 determines the color (Rv, Gv, Bv) in which the pixel to be processed reflects the light of the virtual light source (virtual light). This reflected color also reflects the sensor spectral characteristics of the digital camera 100 and the characteristics of image processing.

In step S512, the virtual light source processing unit 603 calculates the virtual light reflection color (Rv, Gv, Bv) reflected by the specific color (skin color) subject of virtual light. For the virtual light reflection color (Rv, Gv, Bv), the virtual light source color gain (Rv-Gain, Bv-Gain) set in S502 is calculated as follows for the skin color highlight reflection characteristic (Rs, Gs, Bs) calculated in S512. It can be calculated by multiplying like.
Rv=Rv-Gain*Rs
Gv = Gs
Bv=Bv-Gain*Bs

On the other hand, in step S513, the virtual light source processing unit 603 calculates the virtual light reflection color (Rv, Gv, Bv) in which the virtual light is reflected by the subject that is not the specific color. The virtual light reflection color (Rv, Gv, Bv) is obtained by adding the virtual light source color gain (Rv-Gain, Bv) set in S502 to the color signal (Ra, Ga, Ba) calculated by the smoothing processing unit 604 (FIG. 6). -Gain) can be calculated by multiplying as follows.
Rv=Rv-Gain*Ra
Gv = Ga
Bv=Bv-Gain*Ba

  In step S514, the virtual light source processing unit 603 calculates the distance D between the pixel to be processed and the virtual light source. In the present embodiment, as shown in FIG. 7B, the distance between the central irradiation position 702 of the virtual light source 701 and the processing target pixel 704 is calculated as the distance D, which is a value in units of pixel pitch, for example. Good.

  In step S515, the virtual light source processing unit 603 adds the relighting effect to the processing target pixel. Specifically, the virtual light reflection color (Rv, Gv, Bv) calculated in S512 or S513 is weighted based on the virtual light source irradiation gain (α) and the distance D calculated in S514, and the processing target pixel (Rt, Gt, Bt).

In the present embodiment, each component of the processing target pixel (Rout, Gout, Bout) illuminated by the virtual light source is calculated by the following formula.
Rout=Rt+α×(1/D^2)×Rv
Gout = Gt + α x (1/D^2) x Gv
Bout=Bt+α×(1/D^2)×Bv
However, the virtual light source irradiation gain is 0<α<1.
The relighting effect can be added by performing the above processing for each pixel.

  Returning to FIG. 6, the processing target pixels (Rout, Gout, Bout) output from the virtual light source processing unit 603 are input to the gamma processing unit 606. The gamma processing unit 606 applies gamma correction to the processing target pixel (Rout, Gout, Bout) and outputs the pixel (R'out, G'out, B'out) having the corrected value. The luminance color difference signal generation unit 607 generates and outputs a luminance signal (Y) and a color difference signal (RY, BY) from the processing target pixel (R'out, G'out, B'out).

  The system control unit 50 accumulates the luminance and color difference signals output by the relighting processing unit 114 in the image memory 106 under the control of the memory control unit 107, and then compresses and encodes them using the codec unit 110. Then, the system control unit 50 converts the compression-encoded image data into a predetermined file format, and then records the image data on the recording medium 112 via the I/F 111.

  As described above, in the present embodiment, the actual reflection characteristic (reflection color) of ambient light is estimated by performing the smoothing process on the processing target pixel and a plurality of pixels existing in the vicinity in the image space or the color space. To do. Then, the reflection color (Rv, Gv, Bv) of the virtual light is estimated based on the estimated reflection characteristic (reflection color) of the ambient light. Then, the estimated reflection color (Rv, Gv, Bv) of the virtual light is added to the processing target pixel (Rt, Gt, Bt) as a correction value of brightness to add the relighting effect. Therefore, it is possible to add the relighting effect while suppressing the increase of the noise component, as compared with the case where the gain is applied to the value of the processing target pixel (Rt, Gt, Bt) to amplify the value.

  In addition, for a color region in which it is not particularly preferable that the hue changes due to the relighting effect, addition is performed based on the reflection characteristics (Rs, Gs, Bs) of the highlight portion of the color. Therefore, when the virtual light source has the same color as the ambient light source, the effect of additionally illuminating the ambient light can be obtained, and the hue of the relighted area can be prevented from changing.

  Usually, a dark part such as a shadow may have a hue different from that of the highlight region due to the dark part characteristic of the image sensor. When adding a relighting effect to brighten such a dark part, applying a positive gain to the pixel value of the dark part and simply amplifying it makes the originally bright part and the bright part due to the relighting effect originally A hue shift occurs when the colors should be the same. However, in the present embodiment, when a pixel of a certain color is made brighter, it is possible to make it brighter while maintaining the hue by adding pixel values by using the reflection characteristics obtained from the highlight portion of the same color.

  In this embodiment, the highlight part of the skin color is extracted and the reflection color of the virtual light source is predicted from the extracted skin color, but the extracted area is not limited to the skin color part. Any area may be used as long as it is configured to predict the reflection of virtual light from the area where the ambient light hits differently. For example, a color other than the skin color may be added as a condition. In addition to the skin color, a black area of hair may be added.

  Further, in the present embodiment, in order to facilitate understanding of the invention, the case where the flesh color is the characteristic color is illustrated, but an arbitrary characteristic color of the subject can be used. The characteristic color may be a color that occupies a large proportion in the subject, or may be a color according to the type of subject. Alternatively, the user may specify the characteristic color. The reflection characteristic of ambient light can be obtained for any characteristic color. Furthermore, the number of characteristic colors is not limited to one, and the reflection characteristics of ambient light may be obtained for each of a plurality of characteristic colors and the above-described processing may be performed.

  Further, the reflected color of virtual light estimated from the highlight portion of the same color is more accurate when estimated in a region related to the processing target pixel to which the relighting process is applied. Therefore, for example, if the pixel to be processed is a skin color, a value estimated from a highlight part in the skin color region to which the pixel to be processed belongs or a highlight part of the same color included in the region of the same subject is used. can do. Therefore, when a plurality of regions of a specific color exist independently in the image, the reflected color of the virtual light is predicted and held from the highlight portion of each region, and the virtual color is used according to the coordinates of the pixel to be processed. It may be configured to select the reflection color of light. In this case, the value calculated closest can be used, or the value calculated in the same subject region can be used.

  Since the relighting effect of brightening the pixel is assumed here, the virtual light source irradiation gain α is set to a positive value, but a relighting effect of darkening the pixel may be added. In this case, the light source characteristic extraction unit 207 extracts the reflection characteristic from the dark portion instead of the strongly reflecting portion, and sets the virtual light source irradiation gain α to a negative value.

  Further, in the present embodiment, in order to facilitate the description and understanding, it has been described that all pixels included in the circular region centered on the central irradiation position are affected by the virtual light source, but pixels satisfying a specific condition Only the virtual light source may be affected. For example, the face detection unit 113 may extract the face area of the person, and only the face area in the irradiation range 703 may be affected by the virtual light source. Alternatively, using the distance information of the subject such as the defocus amount measured by the digital camera 100, only the pixels corresponding to the subject existing within the predetermined distance within the irradiation range 703 may be influenced by the virtual light source. ..

  The distance D used for weighting the added value is not limited to the distance on the image from the irradiation center position to the addition target pixel described in the present embodiment, and may be any other distance. For example, the three-dimensional positions of the digital camera and the subject may be acquired and the three-dimensional distance between the two may be used for calculation.

  Further, although the example in which the effect of the virtual light source is added by the weight inversely proportional to the square of the distance D has been described, the relationship between the distance D and the weight is not limited to this. For example, the weight may be inversely proportional to the distance D, or the weight may change in a Gaussian distribution.

● (Second embodiment)
Next, a second embodiment of the present invention will be described. In the first embodiment, the reflection color of virtual light is estimated from one captured image, but in the second embodiment, it is estimated from a plurality of captured images. Therefore, only the internal configuration of the relighting processing unit 114 and the operation of the system control unit 50, which are different from those of the first embodiment, will be described below.

  FIG. 8A is a flowchart showing the operation of the system control unit 50 in the present embodiment at the time of shooting. In FIG. 8A, the system control unit 50 determines the shooting mode in S801. It is assumed that the user can operate the operation unit 120 to switch between a shooting mode that requires relighting and a normal shooting mode. The system control unit 50 advances the processing to S802 if a shooting mode premised on relighting is set, and to S806 if another shooting mode (normal mode here) is set. In step S806, the system control unit 50 shoots one image according to the normal mode and ends the process.

  In step S802, the system control unit 50 first captures an overexposed image. For example, the image is captured under a high exposure condition such that the shadow portion of the subject has a proper exposure like the pixel 404 in FIG. Therefore, the brighter region 403 may be saturated (whiteout).

In step S<b>803, the system control unit 50 records the captured overexposure image in the image memory 106.
In step S804, the system control unit 50 shoots with proper exposure. The proper exposure here is an exposure in which the main subject (for example, the bright area 403 of the subject's face in FIG. 4A) has appropriate brightness. Note that the processing of S802 to S804 may be performed after photographing with proper exposure. Also, as is usually taken into consideration when generating an HDR composite image, two shots are performed in a short period such as continuous shooting so that the subject does not move or the brightness of the subject does not change significantly.

  In step S805, the system control unit 50 sequentially supplies the two captured images with different exposures to the image processing unit 105 to generate a so-called HDR composite image. This composite image corresponds to information indicating the reflection characteristic of ambient light. Since the functional configuration and the basic processing operation of the image processing unit 105 are the same as those in the first embodiment, redundant description will be omitted, and the processing operation of the light source characteristic extraction unit 207 that is characteristic of this embodiment will be focused. Explained.

  For example, the system control unit 50 supplies the second captured image to the image processing unit 105 without recording it in the system memory 122, and then supplies the first captured image recorded in the system memory 122 to the image processing unit 105. To do. In the relighting mode, the light source characteristic extraction unit 207 combines a plurality of supplied images to generate one combined image.

  Specifically, the light source characteristic extraction unit 207 determines that a dark portion such as a region where the pixel 404 in FIG. It replaces and produces|generates one synthetic image. At this time, if the pixel used for the replacement is overexposed, it is used after being converted to the brightness corresponding to the proper exposure. The composite image obtained in this way is an image with few dark areas due to shadows. The image processing unit 105 records the composite image in the image memory 106.

  It should be noted that, in S802 to S804, a plurality of images with different degrees of overexposure may be taken, and an appropriately exposed pixel of the plurality of overexposure images may be used when generating the composite image.

  Next, the configuration and operation of the relighting processing unit 114' in this embodiment will be described. FIG. 9 shows a functional configuration example of the relighting processing unit 114' in this embodiment. The same functional blocks as those in the first embodiment are designated by the same reference numerals as those in FIG. 6, and description thereof will be omitted. The operation performed by the relighting processing unit 114' following the control parameter calculation and setting processing described with reference to FIG. 5A will be described.

  An image captured with proper exposure is input to the RGB signal conversion unit 601 and a combined image generated by the light source characteristic extraction unit 207 is input to the relighting processing unit 114 ′ and the smoothing processing unit 902, respectively. The smoothing processing unit 902 performs the smoothing process on the pixel having the same image coordinates as the processing target pixel which the virtual light source processing unit 901 receives from the degamma processing unit 602 and processes the pixel signal (Rc, Gc) after the smoothing process. , Bc) is output to the virtual light source processing unit 901. The contents of the smoothing process may be the same as in the first embodiment.

  The virtual light source processing unit 901 adds a relighting effect to the captured image signal captured by the degamma processing unit 602 and captured with proper exposure. The operation of the virtual light source processing unit 901 will be described with reference to the flowchart shown in FIG.

  In S810, the virtual light source processing unit 901 determines whether the pixel to be processed is located within the irradiation range of the virtual light source, as in the process of S510 in the first embodiment. Then, the virtual light source processing unit 901 advances the process to S811 when it is determined that the processing target pixel is located within the irradiation range of the virtual light source, and determines that the processing target pixel is not located within the irradiation range of the virtual light source. In this case, the process ends without adding the relighting effect.

In step S811, the virtual light source processing unit 901 calculates the virtual light reflection color (Rv, Gv, Bv). The virtual light reflection color (Rv, Gv, Bv) is obtained by adding the virtual light source color gain (Rv-Gain, Bv-) set in S502 to the color signal (Rc, Gc, Bc) calculated from the composite image by the smoothing processing unit 902. It can be calculated by multiplying Gain) as follows.
Rv=Rv-Gain*Rc
Gv = Gc
Bv=Bv-Gain*Bc

  In step S812, the virtual light source processing unit 901 calculates the distance D between the virtual light source and the pixel to be processed. The process may be the same as the process of S514 in the first embodiment.

  In step S813, the virtual light source processing unit 901 adds the relighting effect to the processing target pixel. Specifically, the virtual light reflection color (Rv, Gv, Bv) calculated in S811 is weighted based on the virtual light source irradiation gain (α) and the distance D calculated in S812, and the processing target pixel (Rt, Gt, Bt).

Also in the present embodiment, each component of the processing target pixel (Rout, Gout, Bout) illuminated by the virtual light source is calculated by the following formula.
Rout=Rt+α×(1/D^2)×Rv
Gout = Gt + α x (1/D^2) x Gv
Bout=Bt+α×(1/D^2)×Bv
However, the virtual light source irradiation gain is 0<α<1.
The relighting effect can be added by performing the above processing for each pixel.

  The virtual light source processing unit 901 applies the gamma processing to the pixel signal to which the relighting effect has been added, the gamma processing unit 606 performs the gamma processing, and the luminance/color difference signal generation unit 607 converts and outputs the luminance/color difference signal.

  As described above, in the present embodiment, the reflection color (Rv, Gv, Bv) of the virtual light is calculated using the image in which the pixels of the dark part of the proper exposure image are combined using the pixels of the overexposure image. .. Therefore, when adding a relighting effect to dark pixels in a properly exposed image, the color after adding the relighting effect is the same as the color of the bright part from the original even if the reflected color of virtual light estimated from the highlight part of the same color is not used. The shift can be suppressed.

  Further, as in the first embodiment, it is possible to add a relighting effect without increasing noise.

  It should be noted that, in the present embodiment, it has been described that the overexposure shooting is performed only when the shooting mode premised on the relighting is set, but the images of the proper exposure and the overexposure are shot regardless of the shooting mode. Then, the data may be recorded on a recording medium.

  It should be noted that the image to be subjected to the relighting process is searched for an image taken by overexposure, and if there is, the method of the second embodiment is used, and if not, the method of the first embodiment is executed. It may be configured as follows. In this case, for example, when the relighting processing unit 114 is realized by software or by using a dynamically reconfigurable programmable logic array, both methods can be dynamically executed. Of course, the relighting processing units 114 and 114' may be provided independently.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads the program. This is the process to be executed.

  50... System control section, 101... Optical system, 102... Shutter, 103... Imaging section, 104... A/D converter, 105... Image processing section, 106... Image memory, 107... Memory control section, 109... Display section, 110... Codec unit, 111... Recording I/F, 112... Recording medium, 113... Face detection unit, 114... Relighting processing unit, 120... Operation unit, 121... Nonvolatile memory, 122... System memory

Claims (17)

Correction means for correcting the luminance of the first area by applying the effect of irradiating the first area of the target image with virtual light ;
Determining means for determining a second area in the target image to be referred to when the correcting means corrects the luminance of the first area;
The correction means corrects the color signal of the first area by using a correction signal based on the color information of the second area when correcting the luminance of the first area, and The image processing device is characterized in that the color information of the area is corrected so as to approach the color information of the second area. It said correcting means, by adding the correction signal as an effect of radiating the virtual light, the image processing apparatus according to claim 1, characterized in that for correcting the luminance of the first area.   The image processing apparatus according to claim 2, further comprising setting means for setting the intensity of the virtual light.   The image processing apparatus according to claim 3, wherein the setting unit sets the intensity of the virtual light based on a user operation.   The image processing apparatus according to claim 2, further comprising setting means for setting the position of the light source of the virtual light.   The image processing apparatus according to claim 5, wherein the setting unit sets the position of the light source of the virtual light based on a user operation.   The image processing apparatus according to claim 2, further comprising a specifying unit that specifies an irradiation position of the virtual light in the target image. The specifying means specifies a central irradiation position of the virtual light,
The image processing apparatus according to claim 7, wherein the correction unit calculates the correction signal based on the first area and the center irradiation position.   The image processing apparatus according to claim 2, further comprising a specifying unit that specifies an irradiation range of the virtual light in the target image.   The image processing apparatus according to claim 9, wherein the correction unit does not set an area that is not included in the irradiation range of the virtual light as the correction target.   The image processing apparatus according to claim 1, wherein the correction signal is calculated for each pixel of the target image.   The image processing apparatus according to claim 1, wherein the first area and the second area are areas determined to have a specific color.   The image processing apparatus according to claim 12, wherein the second area is an area that has been determined to have the specific color and has a brightness equal to or higher than a predetermined value.   The image processing apparatus according to claim 12, wherein the specific color is a skin color.   The image processing apparatus according to claim 1, wherein the second area is included in an area determined to be a face area. A correction step of correcting the brightness of the first area by applying the effect of irradiating the first area of the target image with virtual light ;
A determining step of determining, in the target image, a second area to be referred to when correcting the brightness of the first area in the correcting step,
In the correction step, when correcting the luminance of the first area, the color signal of the first area is corrected using a correction signal based on the color information of the second area, thereby The image processing method, wherein the color information of the area is corrected so as to approach the color information of the second area.   A program for causing a computer to function as each unit included in the image processing apparatus according to claim 1.

Images