JP2013017097A - Image processor and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform preferable correction even if a past frame and a present frame are photographed by different conditions when a correction amount of the past frame is referred to in calculation of the correction amount of the present frame.SOLUTION: An image processor includes: a photographing time information acquiring section for acquiring photographing time information during image photographing; a correction amount calculating section for calculating a correction amount of the photographed image from an image feature amount in the photographed image; a correction section for correcting the correction amount on the basis of the photographing time information; a correction processing section for applying the corrected correction amount to the photographed image and performing a correction processing; a correction amount recording section for recording the correction amount before correction by the correction section; and a correction amount acquiring section for acquiring the correction amount before a past image is corrected when a present image is corrected. The correction section corrects the correction amount on the basis of the correction amount before the past image is corrected when the present image is corrected and on the photographing time information when the present image is photographed. The correction processing section applies the corrected correction amount to the present photographed image to perform the correction processing.

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus that corrects an input image.

  As a technique for performing gradation correction of an image, a technique has been proposed in which a face is detected from an image and face information obtained therefrom is used for calculating a correction amount. For example, Patent Document 1 describes a configuration in which when a face with low luminance is included in a certain shooting scene, gradation correction is performed so that the face has appropriate brightness.

  In such a method, when a face can be detected from an image, a preferable correction amount can be calculated. However, when a series of steps from shooting to various processing and saving is considered as a single shot frame, if face detection fails because the face turns sideways when shooting a certain frame, Processing using face information cannot be performed.

  For this reason, for example, in Patent Document 2, the luminance information of the subject in the past frame is compared with the luminance information of the subject in the current frame, and the correction value of the past frame is used according to the amount of change in luminance. Information can now be used for frames.

JP 2010-045457 A JP 2002-247590 A JP 2010-183460 A

  However, in the method of applying the past gradation correction value to the current photographed image, when the two frames of the past frame and the current frame are photographed with different sensitivities, the correction amount is not preferable. There is.

  In gradation correction, noise may be emphasized in dark part correction in which a signal is lifted digitally. At this time, when dark part correction is performed on an image photographed with low sensitivity, there is little noise at the stage before dark part correction, and thus it is often acceptable even if noise is slightly enhanced by dark part correction. However, when dark part correction is performed on an image shot with high sensitivity, there is a lot of noise before dark part correction, and noise is further emphasized by dark part correction from there, which may result in an unacceptable image. is there. Due to such a problem, a technique (hereinafter referred to as “high sensitivity limiter”) has been used in which the degree of correction by dark portion correction is weakened as the image is shot with high sensitivity. When the correction value of the past frame as described above is used while suppressing the correction amount at such high sensitivity, the following problem occurs.

  First, as shown in FIG. 4A, when the past frame 401 has been shot with higher sensitivity than the current frame 406, the correction amount 404 calculated for the past frame 401 is corrected so as not to emphasize too much noise. A correction amount is suppressed by applying a high sensitivity limiter to the curve 403. If this is applied as it is as the correction curve 408 of the current frame 406, it is preferable to make it brighter in nature, and the image should be an image with insufficient correction such as the image 409, although the image should have no problem with the amount of noise. .

  On the other hand, as shown in FIG. 5A, when the past frame 420 was shot with a lower sensitivity than the current frame 425, the correction curve 423 calculated for the past frame 420 shows noise at the stage before dark portion correction. It is assumed that there are few. For this reason, the correction curve 423 is substantially the same as the correction curve 422 in that the correction amount is not suppressed. If this is applied as it is to the current frame 425 as the correction curve 427, the current frame 425 is photographed with high sensitivity, and therefore there is a lot of noise in the stage before dark part correction, and a larger correction is applied thereto, as shown in an image 428. In other words, an image with too much noise is emphasized.

  In addition, some gradation corrections suppress whiteout in highlight portions. For example, in Patent Document 3, when the amount of whiteout is large, an image is taken slightly darker, and the brightness reduced at the time of shooting is reduced later. It is compensated by adjusting the tone.

  However, in the method of applying the correction value for suppressing past whiteout to the current photographed image, when two frames of the past frame and the current frame are shot with different whiteout suppression amounts. The following problems occur.

  First, as shown in FIG. 11, when whiteout suppression is performed in the past frame 401 and whiteout suppression is not performed in the current frame 408, the correction amount 406 calculated in the past frame 401 is reduced by the whiteout suppression. A correction amount 405 for compensating for this is included. If this is applied to the image 409 of the current frame 408 as it is, the current frame will be corrected excessively even though the brightness has not been reduced by whiteout suppression, so an excess like the image 411 will occur. The image will be corrected.

  On the other hand, as shown in FIG. 13, when whiteout suppression is not performed in the past frame 425 and whiteout suppression is performed in the current frame 431, the correction amount 429 calculated in the past frame 425 corrects the brightness of the face. This is equivalent to the correction amount 427 to be performed. If this is applied to the current frame 431 as it is, although the current frame 431 has been reduced by whiteout suppression, no correction is made to compensate for it. For this reason, the image becomes an undercorrected image like the image 435, and in some cases, the image becomes darker than the original brightness.

  The present invention has been made in view of the above problems, and when referring to the correction amount of the past frame in calculating the correction amount of the current frame, even when the past frame and the current frame are photographed under different conditions. An object is to enable preferable correction.

  An image processing apparatus according to the present invention is an image processing apparatus that performs correction processing on a photographed image, and includes a photographing time information acquisition unit that obtains photographing time information at the time of photographing an image, and an image feature amount in the photographed image A correction amount calculating means for calculating a correction amount of the captured image; a correcting means for correcting the correction amount based on the shooting time information; and applying the correction amount corrected by the correcting means to the captured image. Correction processing means for performing the correction processing, correction amount recording means for recording the correction amount before correction by the correction means, and the correction at the time of taking a past image when performing the correction processing on the current image Correction amount acquisition means for acquiring the correction amount before correction recorded in the amount recording means, and the correction means performs correction processing on the current image when the correction amount is acquired. The correction of the current image based on the correction amount of the past image acquired before correction and the shooting time information of the current image acquired by the shooting time information acquisition unit. The correction processing means performs the correction processing by applying the corrected correction amount to a current photographed image.

  According to the present invention, when referring to the correction amount of the past frame in calculating the correction amount of the current frame, preferable correction can be performed even when the past frame and the current frame are shot under different conditions. .

1 is a block configuration diagram of an imaging apparatus common to embodiments of the present invention. 1 is a block diagram of an image processing apparatus common to embodiments of the present invention. 5 is a flowchart illustrating dark part correction processing according to the first embodiment. The figure which shows the mode of the inheritance of the correction value of the past frame by 1st Embodiment. The figure which shows the mode of the inheritance of the correction value of the past frame by 1st Embodiment. The figure which shows the example of the dark part correction | amendment by 1st Embodiment. The figure which shows the example of the high sensitivity limiter by 1st Embodiment. 9 is a flowchart showing dark part correction processing according to the second embodiment. The flowchart which shows the dark part correction | amendment process by the modification of 2nd Embodiment. 10 is a flowchart illustrating tone correction processing according to the third embodiment. The figure which shows the mode of the inheritance of the correction value of the past frame by 3rd Embodiment. The figure which shows the mode of the inheritance of the correction value of the past frame by 3rd Embodiment. The figure which shows the mode of the inheritance of the correction value of the past frame by 3rd Embodiment. The figure which shows the mode of the inheritance of the correction value of the past frame by 3rd Embodiment. FIG. 10 is a diagram illustrating an example of gradation correction according to a third embodiment. The figure which shows the example of the whiteout suppression compensation by 3rd Embodiment. The figure which shows the mode of the whiteout suppression amount calculation by 4th Embodiment. 10 is a flowchart illustrating tone correction processing according to a fourth embodiment.

  Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

(First embodiment)
In this embodiment, when calculating the correction amount, values before and after the application of the high sensitivity limiter are calculated, and information before the application is recorded, so that an image for calculating a more appropriate correction amount in subsequent shooting is calculated. The processing apparatus will be described.

  FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus 100 that is an example for realizing the image processing apparatus according to the first embodiment. The imaging device includes a device such as a digital camera or a digital video camera that captures a subject and obtains image data. In FIG. 1, an optical system 101 includes a lens, a shutter, and a diaphragm, and forms an image of light from a subject on an image sensor 102 under the control of a CPU 103. The image sensor 102 converts light imaged through the optical system 101 into an image signal. The CPU 103 performs various calculations and controls each part of the imaging apparatus 100 according to the input signal and program.

  The primary storage device 104 stores temporary data and is used for the work of the CPU 103. The secondary storage device 105 stores a program (firmware) for controlling the imaging device 100 and various setting information. The storage medium 106 stores image data and the like imaged by the optical system 101, converted into an electrical signal of the image sensor 102, and processed by the CPU 103. Note that the storage medium 106 can be removed after shooting, and data can be read out by being attached to a personal computer (hereinafter referred to as a PC). That is, the imaging apparatus 100 only needs to have a function of accessing the storage medium 106 and can read and write data from and to the storage medium 106. A display unit 107 displays a viewfinder image at the time of shooting, displays a shot image, and displays characters for interactive operation. The operation unit 108 is for accepting a user operation. For the operation unit 108, for example, a button, a lever, a touch panel, or the like can be used.

  The communication device 109 is connected to an external device and transmits / receives control commands and data. For example, PTP (Picture Transfer Protocol) is used as a protocol for establishing a connection and performing data communication. Note that the communication device 109 may perform communication through a wired connection such as a USB (Universal Serial Bus) cable. Further, communication may be performed through a wireless connection such as a wireless LAN. Further, it may be directly connected to an external device, or may be connected to the external device via a server or a network such as the Internet.

  FIG. 2 is a block diagram illustrating a configuration of an information processing apparatus 200 that is another example that realizes the image processing apparatus according to the first embodiment. In FIG. 2, the display unit 201 displays captured images and characters for interactive operation. The operation unit 202 is for receiving a user operation. For example, a keyboard or a pointing device can be used as the operation unit 202. The CPU 203 performs various operations and controls each part of the information processing apparatus 200 in accordance with the input signal and program. The primary storage device 204 stores temporary data and is used for the work of the CPU 203. The secondary storage device 205 stores a program for performing image processing and various setting information. The communication device 206 is connected to an external device and transmits / receives control commands and data. The form of the communication device 206 can be selected in the same manner as the communication device 109.

  FIG. 3 is a diagram showing a flow of dark portion correction processing of an image in the present embodiment. The first embodiment will be described below with reference to the correction amount of the past frame and calculating a correction amount more suitable for the current frame with reference to FIG. In the following, the luminance dark portion correction processing for the captured image in the imaging apparatus 100 will be described, but it is obvious that the information processing apparatus 200 can be similarly applied to general image data (not limited to the captured image).

  First, when an image is shot through the optical system 101 and the image sensor 102 in step S301, the CPU 103 acquires shooting time information (shooting time information acquisition) in step S302. This shooting time information includes, for example, shooting sensitivity. Note that an image stored in the storage medium 106 may be read out and subjected to the processing. Further, when the processing is performed by the information processing device 200, an image stored in the secondary storage device 205 or an image acquired via the communication device 206 may be the target of the processing.

  Next, in step S303, the CPU 103 acquires an image feature amount. This image feature amount is used for calculating the correction amount in the current frame, and includes, for example, luminance information of a face included in the image.

  Next, in step S304, the CPU 103 calculates a correction curve for the current frame. In this correction curve calculation, for example, as shown in FIG. 6A, the correction curve is calculated by drawing a curve having control points such that the face luminance obtained in step S303 becomes the target luminance. Is possible.

  When the correction curve for the current frame is calculated in this way, the CPU 103 acquires a correction curve for the previous frame (acquisition of correction amount) in step S305. The correction curve for the past frame referred to here is a correction curve calculated from a photograph taken before the currently processed frame, and is recorded in step S308 described later.

  Next, in step S306, the CPU 103 acquires image information of past frames. The image information of the past frame here is image information obtained from the image taken before the frame currently being processed, and is recorded in step S310 described later. As such image information of the past frame, for example, luminance information can be recorded and used in later step S307.

  In step S307, the CPU 103 takes over the correction values of the past frames. In this takeover, for example, the amount of change in luminance between the past frame and the current frame is calculated using the luminance of the past frame obtained in step S306. If the amount of change is within a certain value, the past obtained in step S305 is calculated. Processing such as taking over frame correction values is possible. If the amount of change exceeds a certain value, the correction amount of the current frame obtained in step S304 and the correction amount of the past frame obtained in step S305 are mixed. It is also possible to perform processing such as increasing the ratio.

  Usually, a high sensitivity limiter is applied to the correction amount obtained in this way based on the sensitivity information at the time of shooting obtained in step S302, or a high sensitivity limiter is previously applied when calculating a correction curve in step S304. Or calculating a correction amount in consideration. However, in this embodiment, first, in step S308, the correction curve is recorded (correction amount recording) before the high sensitivity limiter is applied (correction amount correction). This is used as past frame information in subsequent frames. For this reason, the correction curve recorded here is a correction value that does not depend on the sensitivity at the time of shooting. The correction curve recorded here does not necessarily need to record the relationship between all input luminances and output luminances. For example, as shown in FIG. 6B, the capacity necessary for recording can be reduced by recording the relationship between the thinned input luminance and output luminance. In such a case, it is possible to reproduce the one close to the original correction curve as shown in FIG. 6B by performing a linear interpolation in step S305.

  When the correction curve is recorded in this way, the CPU 103 applies a high sensitivity limiter in step S309. In the application of the high sensitivity limiter, for example, a method may be considered in which a maximum correction amount at a specific luminance is determined for each sensitivity in advance, and the entire correction curve is suppressed when the maximum correction amount is exceeded. FIG. 7 shows an example of such processing.

  First, the maximum correction amount is calculated based on the sensitivity information at the time of shooting obtained in step S302. For this purpose, for example, as shown in FIG. 7A, the maximum value of the corrected luminance at a specific luminance is determined in advance for each sensitivity, and the maximum value of the corrected luminance at the specific luminance according to the actual sensitivity. Is obtained by calculating. For example, the maximum value of luminance after correction of luminance a at a certain sensitivity y is α, similarly β for luminance b and γ for luminance c. A plot of these relationships is the point indicated by a circle in FIG. The actual correction amount is limited so as not to exceed this point. Note that the sensitivities x, y, and z shown in FIG. 7 (a) do not necessarily have to be set at all sensitivities that can be set, and possess design values for discrete sensitivities and calculate by interpolation between them. It is also possible. Based on the maximum correction amount obtained in this way, a high sensitivity limiter is applied in step S309.

  In FIG. 7B, f (Y) represents the correction curve obtained in step S304, and g (Y) represents a straight line (g (Y) = Y) where input luminance = output luminance. In such a case, since the correction amounts f (a), f (b), and f (c) exceed the maximum correction amount indicated by a circle in the drawing, it is necessary to limit the correction amount. In this limiter, for example, the correction amount f ′ (Y) after the high sensitivity limiter is obtained by the following equation.

f ′ (Y) = {f (Y) −g (Y)} × MIN [{α−g (a)} / {f (a) −g (a)}, {β−g (b)} / {F (b) -g (b)}, {γ-g (c)} / {f (c) -g (c)}] + g (Y)
The correction curve f ′ (Y) thus obtained is applied to the actual image in step S311 to correct the dark portion of the image.

  At that time, the CPU 103 records necessary image information in step S310 in order to perform the past correction value transfer in step S307 in the subsequent photographing. Here, for example, if the luminance information of the image is used for the determination as described above in step S307, the luminance information of the current frame is recorded.

  The above is the first embodiment of the present invention. In order to clarify the information to be referred to in the figure, the correction curve and the image information are expressed as recorded in different recording devices, but need not be stored in different storage devices.

  The effect of performing such processing will be described with reference to FIGS. FIG. 4 shows how the correction amount of the past frame is taken over when the past frame and the current frame are shot with different sensitivities.

  First, FIGS. 4A and 4B will be described. FIGS. 4A and 4B show an example in which past frames 401 and 410 are shot with high sensitivity and current frames 406 and 415 are shot with low sensitivity. Conventionally, first, an ideal correction amount 403 of the frame is calculated in the past frame 401. Since the previous frame 401 has been shot with high sensitivity, if this correction amount 403 is applied as it is, noise will be overemphasized, so the high sensitivity limiter will be applied stronger, and the correction amount that is actually applied is 404 become. When the correction amount 404 applied in the past frame 401 is inherited when the correction amount of the past frame is taken over, the correction amount is applied to the image 407 of the current frame 406 as it is. Therefore, in the current frame 406, there is little noise before the dark portion correction, and although there is no problem even if the original correction amount is applied, the limited correction amount 408 is actually applied. As a result, an image with insufficient correction such as an image 409 is obtained. On the other hand, in the present embodiment, as shown in FIG. 4B, the correction curve 412 before applying the high sensitivity limiter in the past frame 410 is taken over, and the high sensitivity limiter corresponding to the sensitivity of the current frame 415 is applied. At this time, since the frame 415 is currently photographed with low sensitivity, a correction amount 418 that does not require a high sensitivity limiter is obtained. As a result, the current frame 415 can perform more appropriate correction using the face information detected in the past frame 410.

  Next, FIGS. 5A and 5B will be described. 5A and 5B show an example in which the past frames 420 and 429 are photographed with low sensitivity and the current frames 425 and 434 are photographed with high sensitivity. Conventionally, first, an ideal correction amount 422 of the frame is calculated in the past frame 420. Since the previous frame was shot with low sensitivity, the noise enhancement level is within the allowable range even if this correction amount 422 is used as it is, so that the correction amount 423 that hardly requires the high sensitivity limiter is obtained. If the correction amount 423 applied in the past frame 420 is inherited as it is when the correction amount of the past frame 420 is taken over, the correction curve 427 is applied to the current frame 425. However, since the frame 425 is currently photographed with high sensitivity, the noise of the image 426 before dark portion correction is large, and the dark portion correction 427 further enhances the noise. As a result, an image having a problem in image quality such as the image 428 is obtained. On the other hand, in the present embodiment, as shown in FIG. 5B, the correction curve 431 before applying the high sensitivity limiter in the past frame 429 is taken over, and the high sensitivity limiter corresponding to the sensitivity of the current frame 434 is applied. At this time, since the current frame 434 is photographed with high sensitivity, a correction amount 437 in which the high sensitivity limiter is applied strongly is obtained. As a result, the current frame 434 can perform correction that does not emphasize noise excessively while using the face information detected in the past frame 429.

  In this way, by recording the correction amount before applying the high sensitivity limiter, when referring to the correction amount of the past frame in calculating the correction amount of the current frame, the past frame and the current frame are under different conditions. It is possible to calculate a preferable correction amount even when the image is taken.

(Second Embodiment)
In 1st Embodiment, the effect in 1st Embodiment was shown using FIG. 4, FIG. Here, in FIG. 5A, since the noise is excessively emphasized in the image 428 in which the correction amount 423 after applying the high sensitivity limiter in the past frame 420 is directly transferred to the current frame 425, it is problematic to provide it. is there. However, in FIG. 4A, an image 409 in which the correction amount 404 after applying the high sensitivity limiter in the past frame 401 is transferred to the current frame 406 as it is is insufficiently corrected, but there is no problem as a noise amount. Here, attention is paid to the advantages of the image 409 and the image 419 in FIG. An image 419 to which the correction amount 412 before the high sensitivity limiter is applied in the past frame 410 in FIG. 4B is applied and the correction amount 418 to which the high sensitivity limiter 418 is applied according to the sensitivity of the current frame 415 is applied. The correction amount as a sheet is appropriate. In addition, the noise sensation is comparable to the image 414 of the past frame 410. However, since the image 419 is more strongly corrected, the brightness of the entire image is different from the image 414. On the other hand, in the image 409 of FIG. 4A, although the correction degree is slightly insufficient, it is as bright as the image 405.

  In the case of the image 419 in FIG. 4B, it can be said that the image is preferable when viewed as one image, but may not be preferable when viewed as one image in a continuous image. In other words, even if a similar scene is shot with the same exposure, the brightness may be different.

  Therefore, in this embodiment, when the sensitivity of the past frame and the current frame is different, the processing is different depending on whether the sensitivity changes from low sensitivity to high sensitivity or from high sensitivity to low sensitivity.

  First, an example of processing in this embodiment will be described with reference to FIG. In FIG. 8, the basic processing flow is the same as in FIG. 3, but in FIG. 8, after applying the high sensitivity limiter, the correction curve after applying the high sensitivity limiter is recorded in step S710a. Thus, by recording the correction value of the past frame before application of the high sensitivity limiter and the correction value after application of the high sensitivity limiter, it is possible to select a correction value to be taken over in step S707a. That is, as shown in the image 419 in FIG. 4B, in the case where the noise amount of the current frame and the past frame is set to the same level and priority is given to sufficiently correcting the current frame, the high sensitivity limiter application of the past frame is applied. Takes over the previous correction amount. On the other hand, when priority is given to setting the brightness of the past frame and the current frame to the same level as in the image 409 of FIG. 4A, the correction amount of “after” application of the high sensitivity limiter of the past frame is inherited. .

In addition, as a selection criterion whether to give priority to giving the same amount of noise and sufficient correction of the current frame or giving priority to the same brightness of the past frame and the current frame, for example, as follows There are many things.
(1) Continuous shooting or single shooting For example, when there is a sensitivity change during continuous shooting, it is often preferable that the obtained image always has the same brightness. Conversely, when there is a change in sensitivity during single-shooting, it is often preferable to obtain an image with the same noise sensation and ideally corrected. Therefore, it is preferable to take over the correction amount after applying the high-sensitivity limiter for the previous frame in continuous shooting, and take over the correction amount before applying the high-sensitivity limiter for the previous frame in single shooting. Seem.
(2) Unconscious sensitivity change or conscious sensitivity change For example, in a shooting mode in which the sensitivity is automatically controlled, when the user unconsciously changes the sensitivity and shoots, the resulting image has the same brightness. Often it is preferred. On the contrary, when the user consciously changes the sensitivity and shoots, it is often preferable to obtain an image with the same noise feeling and ideal correction. Therefore, in the case of shooting with automatic sensitivity control, the correction amount after applying the high sensitivity limiter of the previous frame is inherited, and in the case of shooting with sensitivity manual control, the correction amount of applying the high sensitivity limiter of the past frame is set to “before”. It seems to be preferable to take over.

  In the description using FIG. 8, an example in which the correction curves before and after the application of the high sensitivity limiter are recorded is shown. However, the present invention is not necessarily limited to this. For example, as shown in FIG. This is also possible by referring to the frame sensitivity information. For example, as described above, when priority is given to applying the ideal correction with the same level of noise in the past frame and the current frame, a high sensitivity limiter corresponding to the sensitivity of the “current frame” is applied. A correction curve 418 is created from the correction curve 417 of (b). On the other hand, when priority is given to setting the brightness of the past frame and the current frame to the same level, a correction curve 417 in FIG. 4B is applied by applying a high sensitivity limiter corresponding to the sensitivity of the “past frame”. From this, a curve corresponding to the correction curve 408 in FIG.

  As described above, in the present embodiment, when the sensitivity is different between the past frame and the current frame, more preferable correction is made by changing the processing depending on whether the sensitivity changes from high sensitivity to low sensitivity or low sensitivity to high sensitivity. Can be performed.

  In the present embodiment, it is assumed that a face is detected in a previous frame and the face information detected in the past frame is used in the current frame when no face is detected in the current frame. The same idea can be applied when a face is detected in both frames. For example, when the sensitivity changes from high sensitivity to low sensitivity during continuous shooting, it is possible to suppress variations in brightness by taking over the correction amount after “after” application of the high sensitivity limiter. On the other hand, when the sensitivity is changed from high sensitivity to low sensitivity during a single shot, the same amount of noise is applied and ideal correction is applied by taking over the correction amount of “Before” when the high sensitivity limiter is applied. The obtained image can be obtained.

(Third embodiment)
In the present embodiment, values before and after the application of the whiteout suppression compensation are calculated at the time of calculating the correction amount, and information before the application is recorded, so that a more appropriate correction amount is calculated in subsequent shooting. An image processing apparatus will be described. Note that the configurations of the imaging apparatus and the image processing apparatus in the present embodiment are the same as the configurations of the imaging apparatus 100 and the image processing apparatus 200 illustrated in FIG. 1 described in the first embodiment, and thus description thereof is omitted.

  FIG. 10 is a diagram showing the flow of image gradation correction processing in this embodiment. Hereinafter, a third embodiment in which a correction amount more suitable for the current frame is calculated with reference to the correction amount of the past frame will be described with reference to FIG. In the following, luminance gradation correction processing for a captured image in the imaging apparatus 100 will be described. However, it is obvious that the information processing apparatus 200 can be similarly applied to general image data (not limited to a captured image).

  First, in step S1001, an image is taken through the optical system 101 and the image sensor 102. At this time, if the captured image includes a high-brightness area that seems to be overexposed, in order to suppress overexposure, the exposure is reduced and the image is taken slightly dark (exposure correction). Here, with respect to the ratio of decreasing the exposure at the time of shooting, for example, it is possible to take a method of increasing the ratio of decreasing the exposure as the area of the region that appears to be over-exposed in the image is larger. For this, for example, information of a photometric sensor (not shown) provided in the vicinity of the optical viewfinder can be used. Note that an image stored in the storage medium 106 may be read out and subjected to the processing. Further, when the processing is performed by the information processing device 200, an image stored in the secondary storage device 205 or an image acquired via the communication device 206 may be the target of the processing.

  Next, in step S1002, the CPU 103 acquires shooting time information. The information at the time of photographing includes, for example, information on whether or not whiteout suppression has been performed at the time of photographing and information on the amount of whiteout suppression.

  Next, in step S1003, the CPU 103 acquires an image feature amount. This image feature amount is used for calculating the correction amount in the current frame, and includes, for example, luminance information of a face included in the image.

  Next, in step S1004, the CPU 103 calculates a correction curve for the current frame. In this correction curve calculation, for example, as shown in FIG. 15A, the correction curve is calculated by drawing a curve having control points such that the face luminance obtained in step S1003 becomes the target luminance. Is possible. At this time, the face luminance detected from the image is acquired from an image photographed darkly by whiteout suppression, and thus has a lower luminance than the original brightness. Therefore, here, the face luminance is calculated in consideration of whiteout suppression compensation.

  In the whiteout suppression compensation, a dark image is taken to suppress whiteout, and correction is performed to compensate for the reduced brightness. At this time, if the entire brightness is lifted in the same manner, the white area will fly again to the area where whiteout is suppressed. For example, a correction with a small amount of lifting the high brightness area as in the correction curves 602, 603, and 604 in FIG. Compensates for whiteout suppression using a curve. Here, a case where the amount of whiteout suppression at the time of shooting in step S1001 is y and a whiteout suppression compensation curve 603 is used is taken as an example. For example, when the luminance of the face detected from the image is the luminance a shown in the correction curve 603, the luminance is actually corrected to the luminance b by whiteout suppression compensation. Therefore, it is preferable to use the luminance b as the luminance of the face referred to in step S1004. Based on b calculated in this way, as shown in FIG. 15A, a correction curve for face correction is calculated by drawing a curve having control points such that the face luminance b becomes the target luminance. Do.

  When the correction curve for the current frame is thus calculated, the CPU 103 acquires the correction curve for the previous frame in step S1005. The correction curve for the past frame mentioned here is a correction curve calculated from a photograph taken before the currently processed frame, and is recorded in step S1008 described later.

  In step S1006, the CPU 103 acquires image information of past frames. The image information of the past frame here is image information obtained from the image taken before the frame currently being processed, and is recorded in step S1010 described later. As the image information of such past frames, for example, luminance information can be recorded and used in the subsequent step S1007.

  In step S1007, the CPU 103 takes over the correction values of past frames. In this takeover, for example, using the luminance of the past frame obtained in step S1006, the amount of change in luminance between the past frame and the current frame is calculated. If the amount of change is within a certain value, the past obtained in step S1005 is calculated. Processing such as taking over frame correction values is possible. When the amount of change exceeds a certain value, the correction amount for the current frame obtained in step S1004 and the correction amount for the previous frame obtained in step S1005 are mixed. It is also possible to perform processing such as increasing the ratio.

  Normally, whiteout suppression compensation is applied to the correction amount obtained in this way based on the whiteout suppression amount at the time of shooting obtained in step S1002. However, in the present embodiment, first, in step S1008, a correction curve before applying the whiteout suppression compensation is recorded. This is used as past frame information in subsequent frames. For this reason, the correction curve recorded here is a correction value that does not depend on the amount of whiteout suppression during shooting. The correction curve recorded here does not necessarily need to record the relationship between all input luminances and output luminances. For example, as shown in FIG. 15B, it is possible to reduce the capacity required for recording by recording the relationship between the thinned input luminance and output luminance. In such a case, in step S1005, a process close to the original correction curve as shown in FIG. 15B can be reproduced by performing a linear interpolation on the thinned portion.

  When the correction curve is recorded in this way, the CPU 103 applies whiteout suppression compensation in step S1009. In application of the whiteout suppression compensation, for example, a correction curve for whiteout suppression compensation corresponding to the amount of whiteout suppression is determined in advance, and the correction curve obtained in step S1004 and the correction curve for whiteout suppression compensation are synthesized. Can be considered. FIG. 16 shows an example of such processing.

  First, a whiteout suppression compensation curve is acquired based on the information on the whiteout suppression amount at the time of shooting obtained in step S1002. For this purpose, for example, a whiteout suppression compensation curve corresponding to the whiteout suppression amount, such as 602, 603, and 604 in FIG. 16, is determined in advance, and the whiteout suppression compensation curve is set according to the actual whiteout suppression amount. Obtained by choosing. For example, a whiteout suppression compensation curve at a certain whiteout suppression amount y is as indicated by 603. First, a certain luminance a in the photographed image is shifted to luminance b by whiteout suppression compensation. At this time, if the correction curve obtained in step S1004 has a shape like 601, the luminance b described above is further shifted to luminance c by the correction curve 601. That is, when the correction curve 601 and the correction curve 603 are combined, a correction curve in which a certain luminance a shifts to the luminance c is obtained. By scanning this method for each input luminance, a correction curve 605 obtained by combining the correction curve 601 and the whiteout suppression compensation curve 603 can be finally obtained.

  The correction curve thus obtained is applied to the actual image in step S1011 to perform image gradation correction and whiteout suppression compensation.

  At that time, the CPU 103 records necessary image information in step S1010 in order to take over the past correction value in step S1007 in the subsequent photographing. Here, for example, if the luminance information of the image is used for the determination as described above in step S1007, the luminance information of the current frame is recorded.

  The above is the third embodiment of the present invention. In order to clarify the information to be referred to in the figure, the correction curve and the image information are expressed as recorded in different recording devices, but need not be stored in different storage devices.

  The effect of performing such processing will be described with reference to FIGS. FIGS. 11 to 14 show how the correction amount of the past frame is taken over when the past frame and the current frame are shot with different whiteout suppression amounts.

  First, FIGS. 11 and 12 will be described. FIGS. 11 and 12 show an example in which the past frames 1101 and 1112 are captured without whiteout suppression and the current frames 1108 and 1119 are captured without whiteout suppression. Conventionally, first, a correction amount 1104 for correcting the face of the frame to an appropriate brightness in the past frame 1101 is calculated. Since the past frame 1101 is photographed slightly dark in order to suppress overexposure, a correction amount 1105 for correcting the reduced brightness is further applied. As a result, the correction amount actually applied becomes 1106. When the correction amount 1106 applied in the past frame 1101 is inherited when the correction amount of the past frame 1101 is taken over, the correction amount is applied to the image 1109 of the current frame 1108 as it is. For this reason, although the current frame 1108 was photographed at a normal brightness without whiteout suppression, a correction amount 1110 that was actually compensated for whiteout suppression in the past frame 1101 would be applied. Become. As a result, an overcorrected image such as an image 1111 is obtained. On the other hand, in the present embodiment, as shown in FIG. 12, the correction curve 1115 before applying the whiteout suppression compensation in the past frame 1112 is taken over, and the whiteout suppression compensation curve 1122 corresponding to the whiteout suppression amount of the current frame 1119 is obtained. Apply. At this time, since the current frame 1119 is not subjected to whiteout suppression, a correction amount 1123 for performing only face correction is obtained. As a result, the current frame 1119 can perform more appropriate correction using the face information detected in the past frame 1112.

  Next, FIGS. 13 and 14 will be described. FIGS. 13 and 14 show examples in which past frames 1125 and 1136 are photographed without whiteout suppression and current frames 1131 and 1422 are photographed with whiteout suppression. Conventionally, first, in the past frame 1125, a correction amount 1127 for correcting the face of the frame to an appropriate brightness is calculated. In the past frame 1125, since the whiteout is not suppressed and the image is captured at the original brightness, the brightness is not reduced, and the whiteout suppression compensation is a straight line 1128 that directly passes through the input. As a result, the correction amount actually applied becomes 1129. When the correction amount of the past frame 1125 is taken over and the correction amount 1129 applied in the past frame 1125 is taken over, the correction amount is applied to the current frame 1131 as it is. Therefore, although the current frame 1131 is suppressed from whiteout and is photographed slightly dark, a correction amount 1134 is actually applied in which the brightness reduced by the whiteout suppression is not compensated. As a result, an image with insufficient correction such as an image 1135 is obtained. On the other hand, when taking over the correction curve 1138 before applying the whiteout suppression compensation in the past frame 1136 and applying the whiteout suppression compensation curve 1146 according to the whiteout suppression amount of the current frame 1142 as in the present embodiment, A correction amount 1147 that compensates for the reduced brightness is also obtained. This makes it possible to perform more appropriate correction using the face information detected in the past frame in the current frame.

  In this way, by recording the correction amount before applying the whiteout suppression compensation, when referring to the correction amount of the past frame in calculating the correction amount of the current frame, the condition that the past frame differs from the current frame It is possible to calculate a preferable correction amount even in the case of shooting with

(Fourth embodiment)
In the third embodiment, the method of increasing the ratio of decreasing the exposure as the area of the region that is likely to be overexposed in the image when determining the overexposure suppression amount is exemplified. However, for example, if the exposure is too low when there is a large area in the image that is likely to be overexposed, the brightness is reduced as shown by 602, 603, and 604 in FIG. The amount of correction necessary for correction increases. In addition, when a dark face is included in the image, a correction curve for face correction such as 601 in FIG. 16 is also required for this amount of whiteout suppression compensation. The correction curve represented by 605 in FIG. 16 in which the suppression compensation curve is synthesized becomes large. However, since the tone correction for raising the dark part is a process for digitally raising the image signal, if the correction amount is too large, the noise contained in the image will be overemphasized. In order to avoid such a phenomenon, for example, a correction amount such as face correction is estimated in advance, and if the correction amount is large, the whiteout suppression amount is reduced. Conversely, if the correction amount is small, the whiteout suppression is performed. One method is to increase the amount.

  However, as shown in the third embodiment, there is a possibility that such processing may not be successful when the correction amount in the past frame is taken over. This phenomenon will be described with reference to FIG.

  FIG. 17 shows a state in which a face is detected in advance at the time of shooting, and an amount of whiteout suppression is determined after calculating a correction amount assumed from the brightness of the face. Here, it is assumed that a dark face is detected in the past frame, and a dark face cannot be detected because the face is turned sideways in the current frame. First, a dark face is detected from the image 701 in the past frame. For this reason, a correction amount 704 is assumed as a correction amount for face correction at the time of shooting a past frame. For this reason, the amount of whiteout suppression is estimated to be small in anticipation that a large correction will be applied later in the past frame, and the whiteout suppression compensation curve is also small, such as 705. Accordingly, the correction actually applied in the past frame becomes a correction curve 706, and correction can be performed in a range in which noise is not overemphasized. Next, in the current frame, the face cannot be detected because the face is turned sideways in the image 711. For this reason, it is assumed that face correction is not performed like the correction curve 707 at the time of shooting the current frame. For this reason, the amount of whiteout suppression is estimated to be large in anticipation that a large correction will not be applied later at this time. However, when the past correction amount is taken over, the face correction amount 704 of the past frame is taken over at the current frame. Therefore, a large correction amount 708 due to face correction and a large correction amount 709 due to whiteout suppression compensation are combined, and a correction amount 710 that is excessively large is eventually applied. As a result, noise may be excessively emphasized in the image 713. In this embodiment, such a problem is dealt with.

  FIG. 18 shows the flow of processing in this embodiment. Each component performs substantially the same operation as the corresponding component in the third embodiment shown in FIG. Here, unlike the third embodiment, actual photographing is performed in step S807 after the past correction value handover.

  First, with respect to the processing after the acquisition of shooting time information in step S801, the processing is performed in accordance with the shooting operation in the third embodiment, but in the case of this embodiment, for example, a shutter button (not shown) by the user. It is also possible to start the processing in response to the counter-pressing operation of). In addition, it is necessary to perform comparison with past luminance information in taking over the past correction value, but this processing is performed before photographing is performed. In this regard, for example, information from a photometric sensor (not shown) provided in the vicinity of the optical viewfinder can be used. Alternatively, when the display unit 107 functions as an EVF during standby and a display image continuously captured at a predetermined rate is always displayed, detection can be performed from the display image. The whiteout suppression amount is calculated based on the correction amount obtained in advance in this way.

  By performing such processing, since a correction amount that also includes the correction amount of the past frame is calculated in advance at the time of shooting in step S807, it is possible to perform shooting with a more suitable whiteout suppression amount. .

  As described above, in the present embodiment, by recording the correction amount before applying the whiteout suppression compensation, when referring to the correction amount of the past frame in calculating the correction amount of the current frame, more preferable whiteout suppression is performed. It is possible to shoot the current frame by amount.

  Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. A part of the above-described embodiments may be appropriately combined.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (5)

An image processing apparatus that performs correction processing on a captured image,
Shooting time information acquisition means for acquiring shooting time information at the time of shooting an image;
Correction amount calculating means for calculating a correction amount of the captured image from an image feature amount in the captured image;
Correction means for correcting the correction amount based on the shooting time information;
Correction processing means for performing the correction processing by applying the correction amount corrected by the correction means to a captured image;
Correction amount recording means for recording the correction amount before correction by the correction means;
Correction amount acquisition means for acquiring the correction amount before correction recorded in the correction amount recording means at the time of shooting the past image when performing the correction processing for the current image,
The correction means, when performing correction processing on the current image, the correction amount before correction of the past image acquired by the correction amount acquisition means, and the current acquired by the shooting time information acquisition means The correction amount is corrected for the current image based on the shooting time information at the time of shooting the image, and the correction processing unit applies the corrected correction amount to the current shot image to perform the correction process. An image processing apparatus that performs the processing.   The image processing apparatus according to claim 1, wherein the shooting time information referred to by the correction unit is a shooting sensitivity.   The image processing apparatus according to claim 1, wherein the correction process is a gradation correction process.   A whiteout amount calculating unit that calculates a whiteout amount from the photographed image; and an exposure correction unit that corrects an exposure at the time of photographing in order to reduce the whiteout amount based on the size of the whiteout amount. The image processing apparatus according to claim 1, further comprising:   An imaging apparatus comprising the image processing apparatus according to claim 1.