JP2017103685A - Image processing device and method, and imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce a shadow with higher accuracy by adaptively determining a non-shadow region to be referred to in order to reduce the shadow.SOLUTION: An image processing device includes extraction means (50) for extracting a shadow region of a subject occurring in a first image by using lighting means from the first image photographed by using the lighting means, a reference region determination processing part (503) for setting a reference region to be referred to in order to correct an image signal of the shadow region in the first image so as not to include the shadow region and a region satisfying a predetermined condition, and a reference pixel conversion processing part (504) for correcting the image signal of the shadow region by using information of the reference region.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an image processing apparatus and method, and an imaging apparatus.

  Conventionally, a technique for correcting a shadow of a subject has been disclosed. Specifically, a technique for detecting a shadow area using luminance and hue information of an image and removing the detected shadow area is disclosed.

  In Patent Document 1, a shadow area is detected, and an arbitrary non-shadow area adjacent to the shadow area is selected as a reference area. Then, by adding the luminance and hue differences between the shadow area and the non-shadow reference area to the shadow area, an image from which the shadow is removed can be obtained.

JP 2010-237976 A

  However, Patent Document 1 does not disclose how to select a non-shadow reference region used for shadow removal. Moreover, it cannot be said that all the regions adjacent to the shadow region are suitable as the reference region, and it is necessary to adaptively select an appropriate reference region.

  The present invention has been made in view of the above problems, and an object of the present invention is to adaptively determine a non-shadow region to be referred to in order to reduce shadows, thereby reducing shadows with higher accuracy.

  In order to achieve the above object, the image processing apparatus of the present invention extracts a shadow region of a subject generated in the first image by using the illumination unit, from a first image taken using the illumination unit. A setting unit configured to set, in the first image, a reference region to be referred to for correcting the image signal of the shadow region and the reference region to be corrected so as not to include the shadow region and a region satisfying a predetermined condition. And correction means for correcting the image signal of the shadow area using the information of the reference area.

  According to the present invention, it is possible to reduce the shadow with higher accuracy by adaptively determining the non-shadow region to be referred to in order to reduce the shadow.

1 is a block diagram illustrating a configuration of a digital camera according to an embodiment of the present invention. The block diagram which shows the structure of the image process part in embodiment. 6 is a flowchart illustrating photographing processing and shadow reduction processing according to the first embodiment. The figure which shows an example of the picked-up image and difference image in 1st Embodiment. The block diagram which shows the structure of the shadow reduction process part in 1st Embodiment. FIG. 6 is a diagram for explaining a reference image region determination method according to the first embodiment. FIG. 10 is a diagram illustrating a method for determining a reference image area in the second embodiment.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. In the present embodiment, an example in which the image processing apparatus is applied to a digital camera will be described.

<First Embodiment>
The first embodiment of the present invention will be described below with reference to FIGS. FIG. 1 is a block diagram illustrating a configuration example of a digital camera 100 according to the first embodiment.

  In the digital camera 100 shown in FIG. 1, light incident through a lens group 101 (imaging optical system) including a zoom lens and a focus lens and a shutter 102 having a diaphragm function is photoelectrically converted in an imaging unit 103. The imaging unit 103 is configured by a CCD, a CMOS element, or the like, and an electric signal obtained by photoelectric conversion is output to the A / D converter 104 as an image signal. The A / D converter 104 converts the analog image signal output from the imaging unit 103 into a digital image signal (image data) and outputs the digital image signal to the image processing unit 105.

  The image processing unit 105 performs color conversion processing such as white balance, γ processing, image data from the A / D converter 104 or image data read from the image memory 106 via the memory control unit 107. Various image processing such as contour enhancement processing and color correction processing is performed. The image data output from the image processing unit 105 is written into the image memory 106 via the memory control unit 107. The image memory 106 stores image data output from the image processing unit 105 and image data to be displayed on the display unit 109.

  The face detection unit 113 detects a face area where a human face exists from the captured image. The image processing unit 105 performs a predetermined evaluation value calculation process using the face detection result of the face detection unit 113 and the captured image data, and the system control unit 50 performs exposure control and focus adjustment based on the obtained evaluation value. Take control. Thus, TTL (through-the-lens) AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and the like are performed.

  Further, the D / A converter 108 converts the display digital image data stored in the image memory 106 into an analog signal and supplies the analog signal to the display unit 109. The display unit 109 performs display according to an analog signal from the D / A converter 108 on a display such as an LCD.

  The codec unit 110 compresses and decodes the image data stored in the image memory 106 based on a standard such as JPEG or MPEG. The system control unit 50 stores the encoded image data in a recording medium 112 such as a memory card or a hard disk via an interface (I / F) 111.

  Further, the image data read from the recording medium 112 via the I / F 111 is decoded and expanded by the codec unit 110 and stored in the image memory 106. Then, by displaying the image data stored in the image memory 106 on the display unit 109 via the memory control unit 107 and the D / A converter 108, the image can be reproduced and displayed.

  The shadow reduction processing unit 114 reduces or removes the shadow of the captured image. The shadow reduction processing performed by the shadow reduction processing unit 114 will be described in detail later.

  The system control unit 50 controls the entire system of the digital camera 100. The nonvolatile memory 121 is configured by a memory such as an EEPROM, and stores programs, parameters, and the like necessary for processing of the system control unit 50. The system memory 122 is used for developing constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 121, and the like. The system control unit 50 implements each process of the present embodiment to be described later by executing a program recorded in the nonvolatile memory 121. In addition, the program here is a program for executing various flowcharts described later in the present embodiment. At this time, constants, variables for operation of the system control unit 50, programs read from the nonvolatile memory 121, and the like are expanded in the system memory 122.

  The operation unit 120 receives an operation by a user. The flash 123 is a lighting device, and may be a lighting device built in the digital camera 100 or a lighting device detachable from the digital camera 100. The distance measuring sensor 124 measures the distance to the subject and outputs distance information corresponding to the pixel unit of the photographic pixel as a two-dimensional distance map image. The method for measuring the distance to the subject is not limited to the method using the distance measuring sensor 124. For example, a method of measuring the distance to the subject based on image data obtained using the imaging unit 103 may be used.

  FIG. 2 is a block diagram illustrating a configuration of the image processing unit 105. Hereinafter, the processing in the image processing unit 105 will be described with reference to FIG. In the present embodiment, it is assumed that the imaging unit 103 is covered with a Bayer array color filter. Accordingly, one of R, G, and B image signals is output from each pixel of the imaging unit 103.

  The image signal input from the A / D converter 104 of FIG. 1 to the image processing unit 105 is first input to the synchronization processing unit 200 of FIG. The synchronization processing unit 200 performs synchronization processing on the input R, G, and B image signals, and generates a color signal RGB for each pixel. The WB amplification unit 201 adjusts the white balance by applying a gain to the generated color signal RGB of each pixel based on the white balance gain value calculated by the system control unit 50 by a known process. The color signal RGB whose white balance is adjusted by the WB amplification unit 201 is input to the luminance / color signal generation unit 202. The luminance / color signal generation unit 202 generates a luminance signal Y from the color signals RGB, and outputs the generated luminance signal Y to the contour enhancement processing unit 203 and the color signal RGB to the color conversion processing unit 205.

  The contour enhancement processing unit 203 performs contour enhancement processing on the luminance signal Y and outputs the result to the luminance gamma processing unit 204. The luminance gamma processing unit 204 performs gamma correction on the luminance signal Y and outputs the luminance signal Y to the image memory 106 via the memory control unit 107.

  On the other hand, the color conversion processing unit 205 converts the color signal RGB into a desired color balance by performing, for example, a matrix operation. The color gamma processing unit 206 performs gamma correction on the color signals RGB. The color difference signal generation unit 207 generates color difference signals RY and BY from the RGB signals.

  The luminance signal Y and the color difference signals RY and BY (image data) generated in this way are temporarily stored in the image memory 106, compressed and encoded by the codec unit 110, and recorded on the recording medium 112. .

  Next, processing in the first embodiment of the digital camera 100 having the above configuration will be described. Note that the digital camera 100 according to the first embodiment includes at least two types of shooting modes, that is, a normal shooting mode and a shadow reduction shooting mode in which a shadow caused by the flash 123 can be reduced or removed.

  FIG. 3A is a flowchart for explaining the process of the system control unit 50 in the photographing process. First, in S301, it is determined whether or not it is a photographing mode using the flash 123 (flash emission mode). If flash photography is to be performed, the process proceeds to S302. If the photography mode is not to perform flash photography (flash non-flash mode), the process proceeds to S305 and normal flash non-flash photography is performed.

  In S302, it is determined whether or not the shadow reduction shooting mode is selected by an operation on the operation unit 120 by the user. If the shadow reduction shooting mode has been selected, the process proceeds to S303. If the shadow reduction shooting mode has not been selected, the process proceeds to S305 to perform normal flash shooting.

  In step S303, first, the flash 123 is set in a non-light-emitting state, and exposure control is performed. Specifically, with respect to the main subject (such as a person) region, the exposure at the time of non-light emission is controlled so that an image having the same brightness as that obtained by flash photography can be obtained. Note that the brightness at the time of flash photography affects the light emission amount of the flash 123 and the reflectance of the subject area, and therefore the brightness at the time of flash photography assumed in S303 differs from the brightness at the time of actual flash photography. There is. However, the brightness at the time of flash photography assumed in S303 is the target brightness for flash photography. For this reason, in step S303, the exposure during non-light emission is controlled so that an image having a brightness equivalent to the target brightness in flash photography can be obtained for the main subject (such as a person) region. At this time, if the reflectance of the main subject (person or the like) area or the flash amount at the time of flash shooting is set by the user operating the operation unit 120 or the like, the flash is calculated from the flash amount and the reflectivity. You may ask for the brightness at the time of photography.

  In step S304, flash non-flash photography is performed. An example of an image obtained by non-flash photography is shown in FIG. Then, an image obtained by flash non-flash shooting (hereinafter referred to as “non-flash image”) is recorded in the image memory 106. Thereafter, the process proceeds to S305, and the flash 123 is emitted in the flash emission mode to perform the main photographing.

  In the main photographing performed in S305, as described above, when NO is determined in S302, photographing is performed with the flash 123 being emitted, and in the case of NO in S301, photographing is performed without emitting the flash 123. Done.

  In S306, the image taken in S305 is recorded on the recording medium 112. An example of an image when the flash 123 emits light (hereinafter referred to as a “light-emitting image”) is shown in FIG. If the shadow reduction shooting mode is selected and a non-light-emitting image is recorded in S304, the non-light-emitting image and the light-emitting image are recorded in association with each other on the recording medium 112, and the shooting process ends.

  Next, the shadow reduction process will be described with reference to the flowchart of FIG. After the shooting shown in FIG. 3A, when an image for shadow reduction is selected by a user operation, the process of FIG. 3B is performed on the selected image data.

  In S310 of FIG. 3B, the difference between the image data of the selected image and the non-light-emitting image recorded in association with each other is calculated. When calculating the difference, the amount of positional deviation between the two images is calculated, the positional deviation is corrected, and then the difference is calculated. FIG. 4C shows an example of a difference image between the non-luminescent image shown in FIG. 4A and the luminescent image shown in FIG. Since the flash 123 emits and does not emit light and the exposure is controlled so that the subject area has the same brightness, there is no significant difference in the main subject part between the emitted image and the non-emitted image. . On the other hand, as shown in 401, a large difference occurs in the shadowed area generated by the flash light.

  However, since the flash light does not reach the subject that is far from the flash, it becomes brighter as much as it is taken with the exposure increased during non-flash photography, and a difference occurs. Therefore, it is possible to adopt a configuration in which a difference for shadow detection is obtained by limiting to a distance within a predetermined distance range.

  A region where the difference between the non-light-emitting image and the light-emitting image acquired as described above is equal to or greater than a threshold (for example, the region 401 in FIG. 4) is used as a shadow region in the first embodiment.

  Returning to FIG. 3, in S <b> 311, the system control unit 50 determines whether a shadow area by the flash 123 exists. Specifically, it is determined whether or not there are areas having a difference equal to or greater than the threshold calculated in S310. If there are more than a predetermined number of pixels having a difference greater than or equal to the threshold value, it is determined that a shadow is generated by the light emission of the flash 123, and the process proceeds to S312. On the other hand, if it is determined that there is no shadow area, the process ends.

  In S312, a shadow reduction ratio (shadow reduction ratio) input by a user operation on the operation unit 120 is set. The shadow reduction ratio is a numerical value indicating how much the shadow is reduced. When the shadow reduction ratio is 100%, the shadow is completely removed.

  In S313, the image data (light emission image) recorded on the recording medium 112 is input to the shadow reduction processing unit 114, and shadow reduction processing is performed. Details of the shadow reduction process will be described later. In S314, the image with the shadow reduced is recorded on the recording medium 112, and the process ends.

  Next, the shadow reduction process performed by the shadow reduction processing unit 114 in S313 will be described. FIG. 5 is a block diagram showing the internal configuration of the shadow reduction processing unit 114.

  The image data (luminance signal Y and color difference signals RY, BY) input to the shadow reduction processing unit 114 is input to the LPF 501 as shown in FIG. 5 and subjected to low-pass LPF processing. On the other hand, it is also input to the HPF 502 and subjected to high-pass HPF processing. The low frequency signal processed by the LPF 501 is output to the reference area determination processing unit 503.

  In the reference area determination processing unit 503, in addition to the low-frequency signal of the light emission image from the LPF 501, the distance map image (distance information) generated by the distance measuring sensor 124 and the shadow area information calculated in S310 of FIG. Entered. The reference area determination processing unit 503 determines whether the image data input from the LPF 501 is image data of a shadow area pixel. If the image data is a shadow area pixel, the reference area determination processing unit 503 determines whether the shadow area pixel is similar to the shadow pixel to be reduced. A reference region used for reduction is detected.

  Here, the processing performed in the reference area determination processing unit 503 will be specifically described with reference to FIG. The reference area determination processing unit 503 refers to the shadow area information and sequentially processes the image signals of the pixels in the shadow area among the input image signals.

  In FIG. 6A, a shadow pixel 601 indicates one of the pixels in the shadow area to be processed, and a reference pixel search range 602 is set for the shadow pixel 601 in the light emission image. FIG. 6B is an enlarged view of a part of FIG. When the shadow pixel 601 is a pixel to be processed, a search range 602 based on the shadow pixel 601 is set, and a pixel similar to the shadow pixel 601 is detected with respect to the pixels within the search range 602. At this time, pattern matching is performed not in the comparison of only one pixel but in a region including peripheral pixels. The reference pixel 603 is an example of the detected reference pixel.

  At this time, the pattern matching is not simply performed using only luminance information, but uses distance information and shadow region information. Since the shadow region is not suitable as a reference object, pattern matching is not performed even within the search range. Further, the distance information is referred to, and the subject region that is closer to the camera than the shadow region is excluded from the pattern matching target because it is not suitable as the reference target of the shadow region (reference region). The reason for processing in this way will be described with reference to FIG.

  FIG. 6C shows an example of a distance map image generated based on the distance information signal. Here, the distance map is expressed as white as the distance of the subject is closer to the camera and black as the distance is longer.

  In the example of FIG. 6, a subject near the shadow region and in front of the shadow region is likely to be the subject that caused the shadow, and is suitable for reference to reduce the shadow. Absent. Therefore, it is excluded from pattern matching targets.

  As described above, the pattern matching process is performed on the shadow area and the pixels excluding those that are closer than the shadow area. Note that, since pattern matching is performed by matching shadows and non-shadow areas, matching is performed with luminance normalized.

  As described above, the reference region determination processing unit 503 determines a reference pixel similar to the pixel to be processed, and sends information on the determined reference pixel and an image signal to the reference pixel conversion processing unit 504.

  The reference pixel conversion processing unit 504 performs a process of combining the shadow pixel and the reference pixel. The composition ratio at this time is determined by the shadow reduction ratio set in S312 of FIG. That is, when the shadow reduction ratio is 100%, the shadow pixel is completely replaced with the reference pixel. When the shadow reduction ratio is 50%, the shadow pixel and the reference pixel are combined at a ratio of 1: 1.

  The signal obtained by converting the shadow pixel using the reference pixel (the shadow is reduced) is output to the synthesis processing unit 505, and the synthesis processing unit 505 outputs the high-frequency signal of the original image signal to the signal obtained by converting the shadow pixel. Are added and an image of the composite result is output.

  As described above, in the first embodiment, when selecting a pixel to be referred to in order to reduce the shadow area, the shadow area and the area closer to the distance than the shadow area (pixels satisfying a predetermined condition) are selected. It was set as the structure controlled so that it may not be referred. As a result, the reference pixel or the reference region for correcting the shadow can be accurately selected, and the accuracy of shadow reduction is improved.

  In the first embodiment, the subject whose distance is shorter than the shadow region is controlled to be excluded from the reference object for shadow reduction. However, the subject causing the shadow (for example, the person in FIG. 6) is excluded. If possible, other controls may be used. For example, template matching is performed using contour information (shape information) of the shadow region, and a region (for example, a human region in FIG. 6) having a high degree of matching between the shadow region and the contour is specified. Then, a method of selecting a reference pixel from a shadow area and an area excluding an area having a high degree of coincidence with the shadow area may be used.

  It is also possible to extract a main subject area such as the person area of FIG. 6 and exclude the area from the selection of the reference pixel. For example, processing such as excluding a subject area including a face from selection of a reference area can be performed using face detection.

  Further, when the hue of a shadow pixel to be processed differs from a hue exceeding a predetermined threshold, the pixel may be controlled by adding information such as a hue so as not to be a reference pixel.

  Further, in the first embodiment, the configuration in which the shadow pixel and the subject pixel causing the shadow are excluded when selecting the reference pixel has been described. However, a configuration in which the pixel is not completely excluded may be employed. . For example, instead of completely excluding shadows and the subject that causes the shadows, it is possible to adopt a configuration in which control is performed so that the matching result is low so that it is difficult to be selected as a reference pixel.

  In the first embodiment, the case where the reference pixel is one pixel has been described as an example. However, the reference pixel may be not only one pixel but also a plurality of pixels. For example, it is possible to adopt a configuration in which a plurality of reference areas having a high degree of coincidence with shadow pixels are detected, and processing for replacing the shadow pixels to be processed with the average value is performed. As a result, the influence of noise can be suppressed as compared with the case of only one pixel.

  It is also possible to adopt a configuration in which the unit of the shadow pixel and the reference pixel is processed in units of small areas instead of pixels. For example, it is possible to adopt a configuration in which a shadow region and a reference region are matched in units of small regions of 3 × 3 pixels and replaced with the average value of the reference regions in units of regions.

  In the first embodiment, the process of combining the reference pixel and the shadow pixel has been described. However, any process may be performed as long as it is a process for reducing the shadow of the shadow pixel using the reference pixel. For example, it is possible to calculate a conversion gain that brings a shadow pixel closer to the characteristics of the reference pixel and apply the conversion gain to the shadow pixel. Specifically, the luminance and color ratios of the shadow pixel and the reference pixel are calculated, and the luminance and color gain for bringing the shadow pixel close to the reference pixel are calculated. Then, the shadow pixel is multiplied by the luminance and the color gain. Note that the conversion gain may be calculated not for each pixel but for the entire screen or small area.

<Second Embodiment>
Next, a second embodiment of the present invention will be described. In the second embodiment, the reference pixel search range setting method for shadow reduction, that is, the processing in the reference region determination processing unit 503 is different from that in the first embodiment. Note that the configuration of the digital camera in the second embodiment is the same as that described with reference to FIGS. 1, 2, and 5 in the first embodiment, and thus the description thereof is omitted here. The flow of the imaging process and the shadow reduction process is the same as that shown in the flowchart of FIG.

  In the first embodiment, as shown in FIG. 6A, for the shadow pixel 601 to be processed, a reference pixel search range is set around the shadow pixel. However, when the periphery of the shadow is surrounded by a subject that causes the shadow, there is a case where there is no effective area for reference. For example, in FIG. 7A, a pixel 701 is a shadow pixel to be processed, and an area 702 indicates a reference pixel search range. In this case, since the search range is almost the shadow area and the subject area causing the shadow, the reference pixel cannot be detected.

  Therefore, in the second embodiment, the search range is adaptively set based on the shadow area and the subject area information causing the shadow. Hereinafter, a specific control method will be described. First, as shown in FIG. 7B, candidate areas 702 to 704 for a plurality of search ranges are set. Then, for each of the search range candidate areas 702 to 704, it is determined whether the pixels at the four corners of the rectangular search candidate area are shadow areas or subject areas that cause shadows. The search range closest to the shadow pixel is selected as the search range, not two or more shadow pixels or pixels in the subject area causing the shadow.

  In the example of FIG. 7B, the candidate area 702 is a shadow area or a pixel in the subject area in which all the pixels at the four corners of the area cause shadows. The candidate area 703 is a pixel in the subject area in which three pixels excluding the upper left pixel are shadow pixels or shadows. The candidate area 704 is not a pixel in all four corners, or a pixel in the subject area that causes a shadow. Therefore, in the example of FIG. 7B, the candidate area 704 is set as the search range.

  As described above, in the second embodiment, in addition to the first embodiment, the search range of the reference pixel is adaptively selected using information on the shadow pixel or the subject area causing the shadow. The configuration. As a result, it is possible to reduce a state in which no referenceable pixel is included in the search range, and it is possible to detect the reference pixel with higher accuracy.

  In the second embodiment, three reference pixel search range candidates are set and the optimum search range is selected from the three search pixel candidates. However, the method of selecting the reference pixel search range is limited to this method. Not what you want. Any method may be used as long as the search range of the reference pixel is determined using the shadow pixel or the pixel of the subject area causing the shadow.

  For example, it is possible to set more search range candidates for reference pixels in the horizontal and vertical directions with respect to the shadow pixels, and to select from among them. It is also possible to take a method of examining the search range little by little until a shadow pixel and an area that does not include the pixel of the subject area causing the shadow are formed at the four corners.

  It is also possible to use a method of determining the search range using the distance information without using the shadow region and the subject region causing the shadow. For example, the search range may be set in an area close to the shadow area. Referring to FIG. 7B as an example, an average distance is calculated for each of the candidate regions 702 to 704 using the distance information, and a search range whose average distance is closest to the shadow distance to be processed is selected. In the example of FIG. 7B, since the candidate areas 702 and 703 include a subject in front of the shadow, the average value of the distance is in front of the shadow. On the other hand, since the candidate area 704 is an area having the same distance as the shadow, the candidate area 704 is determined as the search range.

  In the first and second embodiments, the configuration in which the shadow reduction processing unit 114 is provided inside the camera and the shadow reduction processing is performed in the camera has been described. However, the place where the shadow reduction processing is performed is limited to the camera. Not what you want. For example, it is possible to adopt a configuration in which a shadow reduction processing unit 114 is provided in another device such as a personal computer and the shadow reduction processing is performed in the device.

  Further, the present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus execute the program. It can also be realized by a process of reading and executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  50: System control unit, 103: Imaging unit, 105: Image processing unit, 112: Recording medium, 113: Face detection unit, 114: Shadow reduction processing unit, 123: Flash, 124: Distance sensor, 503: Determination of reference area Processing unit, 504: Reference pixel conversion processing unit, 505: Composition processing unit

Claims (16)

An extraction means for extracting a shadow region of a subject generated in the first image by using the illumination means from a first image photographed using the illumination means;
Setting means for setting, in the first image, a reference area to be referred to for correcting the image signal of the shadow area so as not to include the shadow area and an area satisfying a predetermined condition;
An image processing apparatus comprising: a correction unit that corrects an image signal of the shadow region using the information of the reference region. An extraction means for extracting a shadow region of a subject generated in the first image by using the illumination means from a first image photographed using the illumination means;
Setting means for setting, in the first image, a reference area to be referred to for correcting the image signal of the shadow area;
Correction means for correcting the image signal of the shadow region using the information of the reference region;
The image processing apparatus according to claim 1, wherein the setting unit controls the shadow region and a region satisfying a predetermined condition in the reference region so as not to be used for correction by the correction unit.   3. The image according to claim 1, wherein the extraction unit extracts the shadow region from a difference between the first image and a second image captured without using the illumination unit. 4. Processing equipment.   The area that satisfies the predetermined condition includes an area of a subject whose distance from the imaging device that captured the first image is in front of the shadow area, an area of the subject that generated the shadow area, and a main subject The region and the hue include at least one of regions of the image signal different from the hue of the image signal of the shadow region exceeding a predetermined range. The image processing apparatus according to item.   5. The method according to claim 1, further comprising selection means for selecting a reference pixel to be referred to in order to correct an image signal of the shadow area from the reference area. Image processing apparatus.   The image processing apparatus according to claim 5, wherein the selection unit selects a plurality of reference pixels for an image signal of one pixel in the shadow region.   The image processing apparatus according to claim 5, wherein the selection unit selects a plurality of reference pixels for each predetermined number of pixels.   The image processing according to any one of claims 1 to 7, wherein the setting unit sets a plurality of candidate areas, and sets one of the plurality of candidate areas as the reference area. apparatus.   The said setting means sets the candidate area | region where there are few pixels which satisfy | fill the said predetermined conditions as said reference area among the pixels of the four corners of the said some candidate area of a rectangle, The said area | region is characterized by the above-mentioned. Image processing apparatus.   The setting means sets the candidate area closest to the distance to the shadow area as the reference area, among the plurality of candidate areas, the distance from the imaging device that captured the first image. The image processing apparatus according to claim 8.   The setting means shifts the rectangular candidate area in the first image, and when the pixels at the four corners of the candidate area become positions not corresponding to the predetermined condition, the candidate at the position The image processing apparatus according to claim 1, wherein an area is set as the reference area. An image processing apparatus according to any one of claims 1 to 11,
An image pickup apparatus comprising: an image pickup unit for taking the first image. An extraction step in which an extraction unit extracts a shadow region of a subject generated in the image by using the illumination unit from an image captured using the illumination unit;
A setting step in which a setting unit sets a reference area to be referred to for correcting an image signal of the shadow area in the image so as not to include the shadow area and an area satisfying a predetermined condition;
And a correction step of correcting the image signal of the shadow area using the information of the reference area. An extraction step in which an extraction unit extracts a shadow region of a subject generated in the image by using the illumination unit from an image captured using the illumination unit;
A setting step in which a setting unit sets a reference area to be referred to for correcting the image signal of the shadow area in the image;
A control step in which the setting means controls the shadow region and a region satisfying a predetermined condition among the reference regions so as not to be used for the correction;
And a correction step of correcting the image signal of the shadow region using information on the reference region controlled in the control step.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 11.   A computer-readable storage medium storing the program according to claim 15.