JP6991750B2 - Image processing equipment, image processing methods, and programs - Google Patents

Description

The present invention relates to an image processing apparatus, an image processing method, and a program for processing captured images.

Conventionally, there is known a technique for correcting a high-luminance region of a subject in a captured image. This makes it possible to correct the shine of the face caused by the ambient light and obtain a preferable image.
For example, Patent Document 1 discloses a technique of detecting a predetermined portion of a subject and correcting a shiny region based on the detection result. Specifically, Patent Document 1 describes a technique of detecting predetermined parts such as the face, eyes, and mouth of a person as a subject, and determining the degree of correction of each pixel based on the position and color of the predetermined parts. Is disclosed. As a result, according to the technique described in Patent Document 1, the high brightness and brightness are maintained for the parts that do not need to reduce the brightness and the brightness, and only the parts where the skin shine that needs to be lowered is generated. However, it is possible to correct it.

Japanese Unexamined Patent Publication No. 2005-327709

However, in the conventional correction method described above, since the shine correction is performed based on the result of face detection or the organ detection for detecting the part of the face, for example, if the face detection or the organ detection fails, the shine correction cannot be performed. .. For this reason, when continuous shooting such as continuous shooting is performed, in the conventional correction method, the correction may vary depending on the continuously shot images. For example, if an image in which face detection or the like is successful and the shine correction is performed correctly and an image in which face detection or the like fails and the shine correction cannot be performed correctly are mixed in each of the continuously shot images, each image is used. There will be variations in shine correction.

Therefore, it is an object of the present invention to make it possible to reduce the variation in correction when continuous shooting is performed.

The present invention generates a detection means for detecting an image area of a predetermined subject from a captured image and a correction amount for the shiny area of the image area of the predetermined subject, and the generated correction amount is used to make the shiny area shiny. The correction means includes a correction means for performing correction, and the correction means is a shot image in which the correction amount generated for the shot images obtained in continuous shooting is taken before the shot image at the time of the continuous shooting. It is characterized in that it does not exceed the generated correction amount.

According to the present invention, it is possible to reduce the variation in correction when continuous shooting is performed.

It is a figure which showed the flow of the process in each configuration which concerns on a shine correction process. It is a block diagram which shows the schematic structure of one application example of an embodiment. It is a flowchart of the shine correction in the 1st Embodiment. It is a figure used to explain the reliability. It is a conceptual diagram used for the explanation of the block integral at the time of shine correction. It is a figure which shows the correction characteristic for each luminance at the time of shine correction. It is an image diagram used for the explanation of the shine correction area. It is a flowchart of the shine correction in the second embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
FIG. 1 is a processing flow showing a processing flow in each configuration related to the shine correction processing of the image processing apparatus of the present embodiment. FIG. 2 is a block diagram showing a schematic configuration of a digital camera 217, which is an application example of the image processing apparatus of the present embodiment. It is assumed that the shine correction processing according to the image processing apparatus of the present embodiment shown in the processing flow of FIG. 1 is performed by, for example, the image processing unit 208 in the digital camera 217 of FIG. The image processing device of the present embodiment is also applied to various information terminals such as digital video cameras, smartphones and tablet terminals equipped with camera functions, personal computers, and various cameras such as industrial cameras, in-vehicle cameras, and medical cameras. It is possible.

In the digital camera 217 shown in FIG. 1, the lens 200 is a zoom lens including a focus lens, and the incident light L is condensed to form an optical image of a subject or the like on the image pickup surface of the image pickup element 203. The aperture 201 adjusts the amount of incident light through the lens 200, and the shutter 202 adjusts the exposure time of the image pickup device 203. The focus lens and the like of the lens 200 are driven by the distance measuring control unit 211 under the control of the system control unit 216. The adjustment of the aperture value of the aperture 201 and the shutter speed of the shutter 202 is performed by the exposure control unit 210 under the control of the system control unit 216.

The photometric sensor 213 is a sensor for measuring the average brightness within the shooting angle of view, and outputs the information of the measured brightness value to the system control unit 216. The system control unit 216 calculates an appropriate exposure amount based on the luminance value measured by the photometric sensor 213, and sends the exposure amount information to the exposure control unit 210. The exposure control unit 210 performs exposure control by setting the aperture value of the aperture 201 and the shutter speed of the shutter 202 based on the exposure amount.
The distance measuring sensor 212 is a sensor that detects the distance from the camera to the subject, and outputs the information of the detected distance to the system control unit 216. The system control unit 216 sends the distance information detected by the distance measurement sensor 212 to the distance measurement control unit 211. The distance measurement control unit 211 drives the focus lens of the lens 200 to focus on the subject or the like based on the distance information.

The image pickup element 203 converts an optical image formed on the image pickup surface via the lens 200 into an electric signal by photoelectric conversion, and outputs an analog signal. The image pickup device 203 includes a CCD image sensor or a CMOS image sensor. Further, on the image pickup surface of the image pickup element 203, color filters corresponding to each of the three primary colors R (red), G (green), and B (blue) are provided for each pixel in a so-called Bayer array. .. Therefore, the image sensor 203 outputs an analog signal composed of color signals of each RGB pixel corresponding to the Bayer arrangement. The A / D converter 204 converts the RGB analog signal output from the image sensor 203 into RGB digital data. The timing generation unit 205 supplies a clock signal and a control signal to the image sensor 203 and the A / D converter 204 to control their operations. The timing generation unit 205 is controlled by the memory control unit 206 and the system control unit 216.

The image processing unit 208 performs various image processing on the image data output from the A / D converter 204 or the image data read from the memory 207 by the memory control unit 206. Specifically, the image processing unit 208 performs white balance processing, color interpolation processing for converting an RGB Bayer array signal into an RGB3 plane signal divided for each RGB color component, gamma correction processing, saturation correction, hue correction, and the like. Performs various image processing.

The image processing unit 208 also performs the shine correction processing according to the present embodiment. The shiny correction process is a process for correcting a shiny region when a high-intensity shiny region exists in the image region of a predetermined subject in the captured image. When, for example, a person is photographed as a predetermined subject, the shine correction process is performed on a high-brightness shine region in the face of the person. When correcting the shiny area of the face, the image processing unit 208 has a face detection process for detecting the face area from the captured image, and an organ detection process for detecting each organ such as eyes and mouth, which are a plurality of elements constituting the face. Also do. Then, the image processing unit 208 calculates a region that can be the target of the shine correction (hereinafter referred to as a shine correction region) based on the detection results of those face regions and each organ, and calculates the correction amount of the shine correction. Then, the shine correction process is performed. Details of the detection of the face area, the detection process of the organ, the calculation process of the shine correction area and the correction amount, and the like will be described later.

In addition to controlling the writing and reading of the memory 207, the memory control unit 206 also controls the A / D converter 204, the timing generation unit 205, and the image processing unit 208 under the instruction of the system control unit 216. As a result, the digital image data A / D converted by the A / D converter 204 is written to the memory 207 via the image processing unit 208 or directly via the memory control unit 206.

The memory 207 is a memory for storing image data of still images and moving images taken by the image sensor 203, and has a sufficient storage capacity for storing, for example, a predetermined number of still images and moving images of a predetermined time. It is equipped with. The memory 207 can also be used as a work area for the system control unit 216 and the image processing unit 208. The memory 207 is also used as a buffer memory for temporarily recording each image data during continuous shooting by so-called continuous shooting.

The external storage device 209 is a detachable external recording medium such as a memory card of various standards. The external storage device 209 also includes an interface between the digital camera 217 main body and the memory card. The image data temporarily recorded in the memory 207 is finally recorded in the memory card of the external storage device 209. The image data recorded on the memory card may be so-called RAW data captured or compression-encoded according to a predetermined standard.

The release button 214 is a switch capable of detecting a user's pressing operation of the first stroke on the release button 214 and a pressing operation of the second stroke having a pressing amount larger than that of the first stroke. The release button 214 outputs a SW1 signal that is turned on when the pressing operation of the first stroke is detected to the system control unit 216, and outputs a SW2 signal that is turned on when the pressing operation of the second stroke is detected. Output to 216. When the SW1 signal is input, the system control unit 216 starts each operation for light measurement and distance measurement described above, and when a SW2 signal is further input, starts each operation for exposure and imaging described above. Let me. Further, when the user continues to press the release button 214 for the second stroke, the system control unit 216 controls the digital camera 217 to the continuous shooting state. In the continuous shooting state, the system control unit 216 controls each unit so as to perform continuous shooting while the pressing operation of the second stroke of the release button 214 continues.

Although not shown in FIG. 2, the digital camera 217 also has a display device such as an LCD (liquid crystal display). The displayed image displays a captured image, an image read from the external storage device 209, a so-called live view image, a user interface image, and the like.
The operation unit 215 includes a touch panel arranged on the screen of the display device and buttons and switches arranged on the main body of the digital camera 217, detects an operation by the user, and transmits the detected operation information to the system control unit 216. Output.

The system control unit 216 controls the entire operation of the digital camera 217 including the above-mentioned operation at the time of shooting. The system control unit 216 controls each function of the entire digital camera 217 by expanding the program stored in the non-volatile memory to the work area of the memory 207 and executing the program including the CPU, MPU, and non-volatile memory. .. The non-volatile memory includes a non-volatile semiconductor memory such as an EEPROM that stores programs and various parameters. When the image processing according to the present embodiment is executed by a CPU or the like, the program stored in the non-volatile memory includes the image processing program according to the present embodiment.

[First Embodiment]
Hereinafter, the first embodiment will be described. In the first embodiment, a person (face of a person) is photographed as a predetermined subject by continuous shooting such as continuous shooting, and a shine correction process is performed on a high-intensity shine area of the face of the person. Take the case where it is done as an example. Further, in the present embodiment, even when the face or each part cannot be detected due to the movement of the subject during continuous shooting and the shine correction cannot be performed, or when the shine correction area changes in the face due to opening and closing of the mouth or the like. By the following processing, it is possible to prevent variations in the shine correction in each captured image.

Hereinafter, with reference to the processing flow shown in FIG. 1, the configuration of the digital camera 217 shown in FIG. 2, and the flowchart of the shine correction processing shown in FIG. 3, the shine correction at the time of continuous shooting in the first embodiment is performed. The processing will be described in detail. The processing shown in the flowchart of FIG. 3 is performed by the image processing unit 208 of FIG. 2, and the image processing unit 208 includes each circuit and the like shown in the processing flow of FIG. That is, as shown in FIG. 1, the image processing unit 208 has a face detection circuit 101, an organ detection circuit 102, a block integration circuit 103, a brightness value calculation circuit 104, a correction characteristic determination circuit 105, a correction area calculation circuit 106, and a correction color calculation. It has a circuit 107 and arithmetic units 110 to 113.

It should be noted that the processing of the flowchart of FIG. 3 may be executed not only when it is realized by the hardware configuration as in each circuit of FIG. 1, but also by the software configuration, and a part is a software configuration and the rest is hardware. It may be realized by the configuration. When realized by a software configuration, for example, the CPU of the system control unit 216 executes a program stored in the non-volatile memory to realize the processing of the flowchart of FIG. The program related to this software configuration may be read from, for example, a detachable semiconductor memory or downloaded from a network such as the Internet (not shown), as well as when it is prepared in advance in the non-volatile memory. In the following description of the flowchart of FIG. 3, steps S301 to S312 are abbreviated as S301 to S312, respectively. These things are the same in other flowcharts described later.

When the digital camera 217 starts shooting, the image processing unit 208 according to the present embodiment starts the processing of the flowchart of FIG. 3 for the shine correction processing for the shot image.
When the process of the flowchart of FIG. 3 starts, first, as the process of S301, the face detection circuit 101 of FIG. 1 performs a face detection process of detecting a face region from the RAW image 100 which is a photographed image. It is assumed that the input captured image has undergone white balance processing and color interpolation processing for converting the RGB Bayer array into an RGB3 plain image. Since the white balance processing and the color interpolation processing are performed using known techniques, detailed description thereof will be omitted.

The face detection circuit 101 outputs information such as the number of faces detected from the captured image, the coordinates in the image showing the face area, and the reliability of each detected face. As a method of detecting a face from a photographed image, for example, a method using learning represented by a neutral network, a method of extracting a part having a characteristic physical shape such as eyes and nose from a photographed image by template matching processing, etc. It has been known. In addition, a method of detecting a face by detecting an image feature such as a skin color or an eye shape and performing an analysis using a statistical method is also known. The face detection circuit 101 detects a face region from the captured image by using these methods, and outputs information such as the number of detected faces, the coordinates of the face region, and the reliability of the detected face.

Next, as the process of S302 in the flowchart of FIG. 3, the organ detection circuit 102 of FIG. 1 has the mouth, nose, eyes, and the like, which are elements constituting the face detected by the face detection circuit 101 from the captured image. Each organ is detected, and detailed information such as those organs and the angle (orientation) of the face is acquired. As a method for detecting a predetermined part such as an organ in a face, a method using a template matching process or a method using a neural network is known as in the method of face detection. The organ detection circuit 102 detects each organ, the angle of the face, and the like by using these methods, and outputs the information. The organ detection circuit 102 also outputs information on the reliability of each detected organ and the reliability of the face based on the detected angle of the face.

4 (a) to 4 (d) are explanatory views of an example of the reliability of the detected face. FIG. 4A shows an example of the reliability 401 according to the size of the face. In the reliability 401 of FIG. 4A, the reliability is set to 100% when the face size is within the predetermined size range SR, and the reliability is increased as the face size becomes smaller or larger than the predetermined size range SR. The degree decreases. FIG. 4B shows an example of the reliability 402 when the face direction (angle) is oriented in the left-right direction with respect to the front surface. In the reliability 402 of FIG. 4B, the reliability is 100% when the angle (direction) of the face is less than 60 degrees in the left-right direction from 0 degrees (front), and the reliability increases as the angle increases when the angle is 60 degrees or more and less than 90 degrees in the left-right direction. The degree gradually decreases, and the reliability is 0% when the degree is 90 degrees or more in the left-right direction. FIG. 4C shows an example of the reliability 403 when the face is oriented vertically from the front. In the reliability 403 of FIG. 4C, the reliability is 100% when the angle of the face is less than 30 degrees in the vertical direction from 0 degrees (front), and the reliability gradually increases as the angle increases when the angle is 30 degrees or more and less than 45 degrees in the vertical direction. The reliability is 0% when the temperature is 45 degrees or more in the vertical direction. The reliability 404 in FIG. 4D shows an example of the reliability 404 based on the position of the face. In the reliability 404 of FIG. 4D, the reliability decreases as the position of the face approaches the edge of the screen (screen frame), while the reliability is 100% when the face is within a certain distance from the center of the screen. Will be done.

In addition, the information indicating the reliability of each organ detected by the organ detection circuit 102 is based on the shape, inclination, color, position of the organ in the face and its arrangement relationship of each detected organ, and the eyes and mouth. It is acquired as information indicating the certainty of each organ such as. As an example, if an organ detected as an eye from the facial area is located in the face where the eye should be placed, and its shape and color are the shape and color of the eye, then the probability of being an eye is high. The reliability that indicates is obtained. The reliability is not limited to this example, and may be obtained by other methods as long as the certainty of each organ can be expressed.

Next, as the process of S303 of FIG. 3, the block integration circuit 103 of FIG. 1 divides the face region detected by the face detection circuit 101 into a plurality of blocks in the captured image, and integrates each block. Calculate the value (hereinafter referred to as the block integral value). Specifically, the block integration circuit 103 calculates a value obtained by integrating the R, G, and B values of each block as the block integration value. FIG. 5 is a diagram showing an example of the detected face region 510 and each block 502 obtained by dividing the face region 510 into a plurality of parts. The size of each block 502 from which the block integral value is acquired may be predetermined or may be varied according to the size of the face region. Although FIG. 5 shows an example in which the face region 510 is divided into blocks to calculate the block integral value, the block division of the entire captured image 500 and the block division of the main region 501 excluding the outer peripheral portion of the captured image are described. And the calculation of each block integral value may be performed.

Next, as the process of S304 in FIG. 3, the correction characteristic determination circuit 105 of FIG. 1 is used in the shine correction process in the shine correction area of the face region based on the block integral value calculated by the block integration circuit 103. Determines the shine correction characteristics. The correction characteristic determination circuit 105 determines, as the shine correction characteristic, the shine correction characteristic for determining the intensity of the shine correction corresponding to the brightness of the captured image (hereinafter referred to as the input brightness). Then, the correction characteristic determination circuit 105 sends a correction characteristic parameter representing the determined shiny correction characteristic to the calculator 110.

6 (a) and 6 (b) are diagrams showing an example of the shine correction characteristic 601 and the shine correction characteristic 602 that determine the correction gain as the shine correction strength corresponding to the input luminance. For these shiny correction characteristics 601, 602, the correction gain is set to 0 between the input luminance 0 and the luminance A, and the correction gain is set so that the correction gain is gradually increased as the luminance increases after the input luminance A. It is a correction characteristic. However, the shine correction characteristic 601 has a larger slope (correction gain increase rate) after the input luminance A than the shine correction characteristic 602. That is, in the case of the shine correction characteristic 601 of FIG. 6A, the correction gain for controlling the shine correction amount is greatly increased after the input brightness A, which is effective for the overexposed area (shiny area) having high brightness. It is possible to perform shine correction. On the other hand, in the case of the shine correction characteristic 602 of FIG. 6B, the correction gain for controlling the shine correction amount is gradually increased after the input luminance A, so that the correction strength for the overexposure region (shiny region) is increased. , It is possible to perform a shiny correction that has been weakened to some extent. The correction gain C is a correction gain corresponding to the maximum luminance value that the input luminance can take. These shiny correction characteristics 601, 602 are examples, and the shiny correction characteristics may be variable as well as predetermined cases. As an example, it is possible to change the shine correction characteristic by changing the relationship between the input luminance and the correction gain.

The correction characteristic determination circuit 105 determines the shine correction characteristic used in the shine correction process based on the average brightness in the face area detected from the captured image, the degree of overexposure, the reliability of the detected face and organ, and the like. decide.
Specifically, the correction characteristic determination circuit 105 first calculates the average luminance in the face region based on the block integral value, and determines the average luminance in the face region in FIGS. 6 (a) and 6 (b). It is set as the input luminance A of the rising portion where the correction gain of is larger than 0.

Further, the correction characteristic determination circuit 105 determines the correction gain C corresponding to the maximum possible luminance value of the input luminance based on the degree of overexposure in the face region and the reliability of each of the detected faces and organs. Specifically, the correction characteristic determination circuit 105 compares each block integral value of R, G, and B in the face region with a predetermined threshold value, and counts the number of blocks whose block integral value exceeds the threshold value. Then, the correction characteristic determination circuit 105 sets the correction gain C to a larger value as the number of blocks exceeding the threshold value increases.

Further, the correction characteristic determination circuit 105 sets the value of the correction gain C by using each reliability of the detected face and organ. Specifically, the correction characteristic determination circuit 105 sets the correction gain C to a smaller value as the minimum value in each reliability of the face and the organ is lower. Further, for example, when the minimum value in each reliability of the face and the organ is 0%, the correction characteristic determination circuit 105 also sets the value of the correction gain C to 0. The case where the reliability of each face or organ is 0% is, for example, the case where the detection of the face or organ fails. That is, in the case of the present embodiment, when the face or any of the organs of the eyes and the mouth cannot be detected from the captured image, the correction characteristic determination circuit 105 sets the correction gain C to 0. On the other hand, when the reliability of each of the face and the organ is 100%, the correction characteristic determination circuit 105 sets the value of the correction gain C based only on the number of blocks whose block integral value exceeds the threshold value.

Next, as the processing of S305, the correction area calculation circuit 106 calculates a region (shininess correction region) that can be the target of the shine correction processing in the face region. This shiny correction region is set based on the coordinate information of the face region detected by the face detection circuit 101 and the coordinate information of each organ detected by the organ detection circuit 102. If the face region or organs such as eyes and mouth cannot be detected, the correction region calculation circuit 106 does not set the shine correction region. Then, the correction area calculation circuit 106 sends the correction area parameter representing the set shiny correction area to the calculator 111.

7 (a) to 7 (d) are diagrams showing an example of a region set as a shiny correction region. 7 (a) and 7 (d) show the face area 700 of the face facing the front, FIG. 7 (b) shows the face area 700 of the face facing diagonally sideways (leftward), and FIG. 7 (c) shows the face area 700 of the face. The facial area 700 of the face with the mouth 711 open is shown.

The correction area calculation circuit 106 determines the size of the ellipse from, for example, the distance between the coordinates of both eyes 710, and further determines the center position of the ellipse from the position of both eyes 710, based on the coordinate information of the face area and each organ. Then, the correction area calculation circuit 106 calculates the area indicated by the ellipse in the face area 700 as the shine correction area. In the case of the face region 700 facing the front illustrated in FIG. 7A, the shine correction region 701 is calculated by the correction region calculation circuit 106.

Further, the correction area calculation circuit 106 calculates the angle of the face (direction of the face) based on the positional relationship of each organ, and determines whether or not the face is displaced from the front based on the angle. Then, when the correction area calculation circuit 106 determines that the face is deviated from the front, the correction area calculation circuit 106 calculates the direction and angle of the deviation, and corresponds to the deviating angle in the direction opposite to the deviating direction. Shift the center of the ellipse by the amount. Further, the correction area calculation circuit 106 at this time corrects the size of the ellipse (that is, the size of the shiny correction area) according to the angle of deviation. In the case of the face region 700 facing diagonally sideways as illustrated in FIG. 7 (b), the shiny correction region 702 is calculated by the correction region calculation circuit 106. In this way, by shifting the center of the shiny correction area and correcting the size according to the orientation of the face, it is possible to avoid the harmful effect (miscorrection of shiny) caused by the shiny correction area 702 protruding from the face area 700. It will be possible.

Further, the correction area calculation circuit 106 determines whether the mouth 711 of the facial organs is open or closed. Specifically, in the example of FIG. 7C, the correction area calculation circuit 106 calculates the distance between the coordinates 703 of the upper lip of the mouth 711 and the coordinates 704 of the lower lip, and the value of the distance is a predetermined value. In the above case (when separated), it is determined that the mouth 711 is open. On the other hand, the correction area calculation circuit 106 determines that the mouth 711 is closed when the distance is less than a predetermined value. When the correction region calculation circuit 106 determines that the mouth 711 is open, the correction region calculation circuit 106 narrows the shine correction region 705 so as not to cover the coordinates 703 of the upper lip, as shown in FIG. 7 (c). By narrowing the shine correction area 705 so that the coordinates 703 of the upper lip are not covered when the mouth 711 is open in this way, for example, the shine correction process can be performed on white teeth that tend to have high brightness in the open mouth 711. Don't do it.

Further, as shown in FIG. 7D, the correction region calculation circuit 106 sets a correction exclusion region 706 for the eyes 710, which tend to have high luminance in the shine correction region 701. The correction exclusion area 706 is set as an area in the shine correction area 701 where the shine correction is not performed.

Next, as the process of S306 of FIG. 3, the correction color calculation circuit 107 of FIG. 2 calculates a color correction value for the color to which the shine correction is performed, that is, the skin color in the case of a face. Here, in the case of a face, the shiny correction process is performed so as to bring the corrected color closer to the skin color by subtracting the corrected color from the saturated image data of high brightness. That is, it is necessary that the color after subtracting the correction color value from the saturation value of the overexposed shiny region is the skin color, and therefore, the difference value between the saturation value and the skin color value is the correction color. It becomes a value.

For example, when the values of R, G, and B are each represented by 8 bits, the average value of the skin color of the face area is
(R, G, B) = (230,200,170)
On the other hand, the saturation value can be expressed by
(R, G, B) = (255, 255, 255)
Will be.
Therefore, when the average skin color of the face area is subtracted from the saturation value, the value of the corrected color is
(R, G, B) = (20,55,85)
Will be.

The correction color calculation circuit 107 calculates the average brightness of the face region detected by the face detection circuit 101, and calculates the tint from the RGB color information of the block close to the average brightness in the face region. Calculate the average skin color. Then, the correction color calculation circuit 107 determines the color correction value by calculating the value of the correction color using the skin color average value, and sends the correction color parameter representing the color correction color to the calculator 112.

Next, as the process of S307, the image processing unit 208 determines whether or not the image shooting by continuous shooting is performed. When the image processing unit 208 determines that continuous shooting has not been performed, or when it is determined in continuous shooting that the captured image is the first frame of continuous shooting, the image processing unit 208 proceeds to S309. On the other hand, when the image processing unit 208 determines in the continuous shooting that the captured image is the second and subsequent frames of continuous shooting, the image processing unit 208 proceeds to processing in S308.

Proceeding to S308, the image processing unit 208 compares the shiny correction characteristic and the shiny correction area determined in the previous frame during continuous shooting with the shiny correction characteristic and the shiny correction area calculated in the latest frame. Then, the image processing unit 208 determines the correction amount (correction gain) and the shine correction area to be applied to the captured image of the latest frame based on the result of the comparison.

For example, when the correction gain C (maximum correction amount) of the shine correction characteristic calculated in the latest frame is larger than the correction gain C of the shine correction characteristic determined in the previous frame, the image processing unit 208 adapts to the latest frame. Set the correction amount to the same value (correction gain) as the correction amount of the previous frame. That is, the image processing unit 208 sets the correction gain determined according to the input luminance in the previous frame as it is for the latest frame so as not to exceed the correction gain determined in the previous frame. On the other hand, when the correction gain C of the shine correction characteristic calculated in the latest frame is the same as or smaller than the correction gain C of the shine correction characteristic of the previous frame, the image processing unit 208 performs the correction amount (correction) calculated in the latest frame. Gain) is set as the correction amount applied to the latest frame. That is, in this case as well, the correction gain of the latest frame does not exceed the correction gain determined in the previous frame. In this way, by preventing the correction amount applied to the latest frame from exceeding the correction amount of the previous frame, if the face or organ cannot be detected during continuous shooting, or the reliability of the detected face or organ can be improved. Even if it becomes low, it is possible to reduce the variation in the shine correction for each continuous shooting frame.

Further, in S308, when the area of the shine correction area calculated in the latest frame is wider than the area of the shine correction area determined in the previous frame, the image processing unit 208 sets the shine correction area adapted to the latest frame in front. Set to the same size as the shine correction area of the frame. That is, the image processing unit 208 sets the shiny correction area determined in the previous frame as it is for the latest frame so that the area of the shiny correction area determined in the previous frame is not widened (or fixed). do. On the other hand, when the area of the shiny correction area calculated in the latest frame is the same as or smaller than the area of the shiny correction area of the previous frame, the image processing unit 208 uses the shiny correction area calculated in the latest frame as the latest frame. Set as a shiny correction area to be applied to. That is, also in this case, the area of the shine correction area in the latest frame is not wider than the area of the shine correction area set in the previous frame. If the face cannot be detected in the previous frame and the shiny correction area is not calculated, the image processing unit 208 does not set the shiny correction area for the latest frame. In this way, by making the area of the shine correction area applied to the latest frame not wider (or fixed) than the area of the shine correction area determined in the previous frame, for example, the mouth opens and closes during continuous shooting. Even if it is done, it is possible to reduce the variation in the shine correction for each continuous shooting frame.

Next, as the processing of S309, the image processing unit 208 stores parameters such as the shine correction characteristic and the shine correction area determined in the latest frame in, for example, the memory 207 for use in comparison with the next frame during continuous shooting. ..

Next, as the processing of S310, the image processing unit 208 performs the shine correction processing according to the input luminance value for the shine correction region of the captured image. Therefore, the luminance value calculation circuit 104 of FIG. 1 converts the RGB data of each pixel of the input RAW image 100 into the luminance value Y data to create the luminance value image 108. Then, the luminance value image 108 is sent to the arithmetic unit 110. The conversion from the RGB data to the data of the luminance value Y is performed, for example, by the calculation of the equation (1).
Y = R × 0.3 + G × 0.6 + B × 0.1 Equation (1)

The calculator 110 calculates each correction gain according to the brightness value of each pixel of the luminance value image 108 based on the shine correction characteristic indicated by the above-mentioned correction characteristic parameter, and calculates each correction gain for each pixel. Send to 111. The arithmetic unit 111 extracts each correction gain corresponding to the shine correction area indicated by the above-mentioned correction area parameter from each correction gain calculated based on the luminance value image 108, and sets each correction gain in the shine correction area. It is sent to the arithmetic unit 112. The extraction of each correction gain corresponding to the shine correction region is performed by setting the correction gain outside the shine correction region to 0. The calculator 112 multiplies the skin color correction value indicated by the above-mentioned correction color parameter by each correction gain in the shine correction region, and sends information on each correction color after the correction gain multiplication to the calculator 113. The arithmetic unit 113 subtracts each correction color after the above-mentioned correction gain multiplication from the above-mentioned image data of the RGB3 plane of the RAW image 100. As a result, the arithmetic unit 113 generates an RGB image after the shiny correction is performed on the shiny region in the shiny correction region.

Next, as the processing of S311, the image processing unit 208 performs development processing such as gamma correction processing, saturation correction, and hue correction on the image after the shine correction processing is performed as described above. After that, as the process of S312, the image processing unit 208 determines whether or not the photographing is completed. In the case of the present embodiment, when the SW2 signal of the release button 214 of FIG. 2 is in the ON state, information indicating that shooting is being continued is supplied from the system control unit 216, whereby image processing is performed in S312. Unit 208 determines that the shooting has not been completed. Then, when it is determined that the shooting is not completed, the image processing unit 208 returns the processing to S301 and performs a series of processing after S301. On the other hand, when the SW2 signal of the release button 214 in FIG. 2 is turned off, information to the effect that shooting is finished is supplied from the system control unit 216, whereby the image processing unit 208 determines in S312 that shooting has been completed. , The processing of the flowchart of FIG. 3 is terminated.

As described above, the image processing apparatus of the first embodiment has the latest frame so as not to exceed the correction amount and the shine correction area determined in the previous frame when continuous shooting by continuous shooting is performed. The correction amount is set. Further, in the image processing apparatus of the first embodiment, when continuous shooting is performed by continuous shooting, the area of the latest shiny correction area is set so as not to be wider than the area of the shiny correction area determined in the previous frame. To. As a result, according to the image processing apparatus of the first embodiment, even if face detection or organ detection cannot be performed during continuous shooting, or even if the correction target area changes due to opening and closing of the mouth, etc., the shine correction for each shot image is performed. It is possible to reduce the variation in processing.

[Second Embodiment]
Next, the second embodiment will be described. Since the processing flow of the shine correction processing in the second embodiment is the same as that in FIG. 1 and the configuration of the digital camera 217 is the same as that in FIG. 2, the description of the same processing as shown in the drawings is omitted. FIG. 8 shows a flowchart of the shiny correction process according to the second embodiment. Hereinafter, the shine correction process in the image processing unit 208 of the second embodiment will be described with reference to FIGS. 1, 2, and 8. In the flowchart of FIG. 8, the processes from S801 to S806 are the same as the processes from S301 to S306 in FIG. 3, and therefore the description thereof will be omitted. In the flowchart of FIG. 8, after the calculation processing of the shiny correction color in S806, the image processing unit 208 advances the processing to S807.

Proceeding to S807, the image processing unit 208 has the shining correction characteristics calculated by S804, the shining correction area calculated by S805, and the shining correction color calculated by S806 for each shot image taken by continuous shooting. The information is recorded in the memory 207. That is, in the memory 207, a series of each shot image after the start of continuous shooting, and information on the shiny correction characteristic, the shiny correction area, and the shiny correction color calculated for each of the shot images are recorded.

Next, as the processing of S808, the image processing unit 208 determines whether or not the shooting is completed in the same manner as in S312 of FIG. 3, and if the shooting is not completed, returns the processing to S801 and the shooting is completed. If it is determined, the process proceeds to S809.

Proceeding to S809, the image processing unit 208 has a correction gain C (maximum correction amount) from the shine correction characteristics corresponding to each series of shot images recorded in the memory 207 by continuous shooting. Select the shine correction characteristic that minimizes.
Further, the image processing unit 208 selects the shiny correction area having the smallest (narrowest) shiny correction area among the shiny correction areas corresponding to each captured image recorded in the memory 207.

Next, as the processing of S810, the image processing unit 208 uses the shine correction characteristic with the minimum correction gain C selected in S809 and the shine correction area with the minimum area, and all recorded in the memory 207. Performs shine correction processing on the captured images (all continuous shot images). The image processing unit 208 uses the correction color calculated for each captured image and recorded in the memory 207 as the shine correction color used for the shine correction processing. The shine correction process is the same as that of S310 of the first embodiment. Further, in S810, the image processing unit 208 performs development processing such as gamma correction processing, saturation correction, and hue correction on the image after the shine correction processing is performed, as in the above-mentioned S311.

As described above, the image processing apparatus of the second embodiment applies the same shine correction characteristics and the same area of shine correction region to all the series of shot images obtained by continuous shooting of continuous shooting. As a result, according to the image processing apparatus of the second embodiment, there is a variation in the shine correction processing for each captured image due to a change in the correction target area due to a failure of face detection or organ detection that occurs during continuous imaging, opening and closing of the mouth, and the like. Can be reduced.
In the second embodiment, the operation during continuous shooting by the digital camera 217 has been described as an example, but for a plurality of continuously shot images selected by the user, for example, using RAW development application software or the like. The same method as described above may be used when performing the shine correction process.

<Other embodiments>
Although the preferred embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and various modifications and changes can be made within the scope of the gist thereof. In the above-described embodiment, an example of the shine correction process for the skin color region of the face image is given as an image region of a predetermined subject, but the present invention is not limited to the shine correction process for the skin color region of the face image. As an example, it can be applied to a case where a high-luminance region (shiny region) of an automobile body is corrected for an image in which an automobile or the like is continuously shot as a predetermined subject.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The above-mentioned embodiments are merely examples of embodiment in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from its technical idea or its main features.

101: Face detection circuit, 102: Organ detection circuit, 103: Block integration circuit, 104: Luminance value calculation circuit, 105: Correction characteristic determination circuit, 106: Correction area calculation circuit, 107: Correction color calculation circuit, 208: Image processing Department

Claims (12)

A detection means for detecting an image area of a predetermined subject from a captured image,
It has a correction means for generating a correction amount for a shiny region of an image region of a predetermined subject and performing a shiny correction on the shiny region using the generated correction amount.
The correction means ensures that the correction amount generated for the captured images obtained by continuous shooting does not exceed the correction amount generated for the captured image captured before the captured image during the continuous shooting. An image processing device characterized by this. The correction means has a determination means for determining a correction region that can be the target of the shine correction.
In the determination means, the area of the correction region determined for the captured images obtained by the continuous imaging is narrower than the area of the correction region determined for the captured image captured before the captured image during the continuous imaging. If this is the case, the image processing according to claim 1, wherein the area of the correction area for the captured image after the captured image whose area is narrowed is not wider than the narrowed area. Device. The image processing apparatus according to claim 2, wherein the determination means fixes the area of the correction region for the captured image after the captured image whose area is narrowed by the narrowed area. The detection means detects a plurality of elements constituting the predetermined subject, and detects the plurality of elements.
2. The image processing device described. A detection means for detecting an image area of a predetermined subject from a captured image,
It has a correction means for generating a correction amount for a shiny region of an image region of a predetermined subject and performing a shiny correction on the shiny region using the generated correction amount.
The correction means is characterized in that the shiny correction is performed on all the shot images in the series by using the smallest correction amount among the correction amounts generated for each shot image in the series obtained by continuous shooting. Image processing device. The correction means has a determination means for determining a correction region that can be the target of the shine correction.
The determination means sets the correction region having the smallest area among the areas of the correction regions determined for each of the series of captured images as the area of the correction region applied to all the captured images of the series. The image processing apparatus according to claim 5. Any of claims 1 to 6, wherein the detection means extracts a feature amount of the image region of the predetermined subject and calculates the correction amount for the shiny region based on the extracted feature amount. The image processing apparatus according to item 1. The detecting means includes a face detecting means for detecting an area of a human face as an image area of the predetermined subject, and an organ detecting means for detecting a plurality of organs constituting the face.
The image processing apparatus according to claim 2 or 6 , wherein the determination means determines the correction region based on the detected face and the organ. The image processing apparatus according to claim 8, wherein the correction means calculates the correction amount based on the reliability of the detected face and organ. A detection process that detects an image area of a predetermined subject from a captured image,
It has a correction step of generating a correction amount for a shiny region of an image region of a predetermined subject and performing a shiny correction on the shiny region using the generated correction amount.
In the correction step, the correction amount generated for the captured images obtained by continuous shooting does not exceed the correction amount generated for the captured image captured before the captured image during the continuous shooting. An image processing method of an image processing apparatus, characterized in that. A detection process that detects an image area of a predetermined subject from a captured image,
It has a correction step of generating a correction amount for a shiny region of an image region of a predetermined subject and performing a shiny correction on the shiny region using the generated correction amount.
The correction step is characterized in that the shiny correction is performed on all the shot images in the series by using the smallest correction amount among the correction amounts generated for each shot image in the series obtained by continuous shooting. The image processing method of the image processing device. A program for making a computer function as each means of the image processing apparatus according to any one of claims 1 to 9.

Images