JP2013135391A - Image processor, digital camera, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a moving image free from blown out highlights.SOLUTION: An image processor comprises: blown out highlight determination means which determines whether blown out highlights which whiten a part or the whole of a frame image occur or not, for each of plural continuous frame images constituting a moving image; and image processing means which performs, on a frame image determined to have the blown out highlights occurring therein by the blown out highlight determination means, processing different from that for the other frame images.

Description

  The present invention relates to an image processing apparatus, a digital camera, and an image processing program.

  2. Description of the Related Art There are known photometers having fast photoresponsiveness, a wide dynamic range performance from low luminance to high luminance, and digital cameras equipped with an AE (Automatic exposure) function (see, for example, Patent Document 1).

JP 2010-045294 A

  By the way, when a flash is emitted from another camera during shooting of a moving image, a so-called “whiteout” phenomenon occurs in which the entire image or a part of the frame being exposed turns white. However, even in the above-described conventional digital camera, the AE function cannot respond to the flash of the flash, and cannot prevent overexposure.

(1) An image processing apparatus according to the present invention includes an overexposure determining unit that determines, for each of a plurality of continuous frame images constituting a moving image, whether there is an overexposure that makes part or the whole of the frame image white. And an image processing means for performing a process different from that of the other frame images on the frame image determined to have a whiteout by the whiteout determining means.
(2) The digital camera according to any one of claims 1 to 7, wherein the digital camera performs image processing on an image capturing unit that captures an image and a moving image including a plurality of continuous frame images captured by the image capturing unit. The image processing apparatus according to one aspect, and a recording control unit that records a moving image after image processing by the image processing apparatus on a recording medium.
(3) An image processing program according to the present invention includes: The computer is caused to execute an image processing procedure for performing processing different from that for other frame images on the frame image determined to have undergone the whiteout by the whiteout determination procedure.

  According to the present invention, it is possible to generate a moving image without overexposure.

The figure which shows the structure of the digital camera of one embodiment The flowchart which shows the moving image shooting program of one embodiment

  An embodiment in which the present invention is applied to a compact digital camera will be described. Note that the present invention is not limited to a compact digital camera, and can be applied to all types of cameras that shoot moving images, such as single-lens reflex digital cameras and video cameras.

  FIG. 1 is a diagram illustrating a configuration of a digital camera according to an embodiment. In FIG. 1, illustration and description of camera devices, circuits, and apparatuses that are not directly related to the present invention are omitted. The camera 1 includes a photographing lens 2, an image sensor 3, an AFE (Analog Front End) circuit 4, an overexposure determination circuit 5, a DFE (Digital Front End) circuit 6, an image processing circuit 7, a TG (Timing Generator) circuit 8, AF ( Auto Focus) circuit 9, lens drive circuit 10, flag memory 11, image memory 12, monitor 13, interface 14, memory card 15, operation member 16, controller 17 and the like.

  Although the photographing lens 2 is illustrated as a single lens, it is composed of a plurality of lens groups including a zooming lens and a focusing lens, and forms a subject image on the imaging surface of the imaging device 3. The image pickup device 3 is composed of a CMOS image sensor or the like in which a plurality of pixels are two-dimensionally arranged on the light receiving surface, and photoelectrically converts a subject image formed by the photographing lens 2 and outputs an analog image signal.

  The AFE circuit 4 performs analog processing such as correlated double sampling and gain adjustment on the analog image signal, and converts the image signal after the analog processing into digital image data. The whiteout determination circuit 5 determines whether or not “whiteout” has occurred for each frame image (details will be described later). The DFE circuit 6 performs signal processing before image processing by the image processing circuit 7 in the subsequent stage, that is, pre-stage correction processing such as clamping processing, AGC processing (automatic gain adjustment), and shading processing on the image signal from the AFE circuit 4. This is a circuit applied to a signal.

  The image processing circuit 7 performs processing related to the image, that is, white balance processing, gamma correction, color interpolation, contour enhancement, vignette correction, compression / expansion, etc., on the image signal after processing by the processing circuit of the previous stage, that is, the DFE 6. It is a processing circuit. The TG circuit 8 controls the timing of various processes such as charge accumulation, readout of accumulated signals, A / D conversion, and image processing with respect to the image sensor 3, the AFE circuit 4, the DFE circuit 6, and the image processing circuit 7.

  The AF circuit 9 is a circuit that detects a focusing lens position at which the contrast of image signals output from the imaging pixels R, G, and B peaks in a plurality of focus detection areas set in the shooting screen. The lens driving circuit 10 drives the focusing lens of the photographing lens 2 to adjust the focus, and also drives the zooming lens to change the focal length. The flag memory 11 is a memory that stores a flag in association with a frame in which it is determined that overexposure has occurred. The image memory 12 is a memory that temporarily stores frame image data.

  The monitor 13 is an LCD that displays a through image captured by the image sensor 3, a captured image, and various types of information. The interface 14 writes image data and various information data to the memory card 15 and reads image data and various information data from the memory card 15. The operation member 16 includes various switches such as a release half-press and full-press switch, a zoom switch, a mode switch, a menu switch, a cross key switch, and an OK switch.

  The controller 17 includes peripheral components such as a CPU 17a and a memory 17b, and controls various devices, circuits, and devices of the digital camera 1 described above to control various types of the digital camera 1 such as imaging, exposure, focus adjustment (AF), and image display. In addition to executing the operation, a moving image shooting program described later is executed to shoot a moving image. The memory 17b stores various data and setting values of the digital camera 1 in addition to the moving image shooting program described later.

  FIG. 2 is a flowchart showing a moving image shooting program according to an embodiment. The moving image shooting operation of the embodiment will be described with reference to this flowchart. When the moving image shooting mode is selected by the shooting mode switch of the operation member 16 and the release full-press switch of the operation member 16 is turned on, the CPU 17a of the controller 17 starts executing the moving image shooting program shown in FIG.

  In step 1, imaging is started at a predetermined frame rate, for example, 30 FPS. In the subsequent step 2, the image data of the frame imaged from the image sensor 3 to the AFE circuit 4 is read out, subjected to various analog processes and AD conversion by the AFE circuit 4, and then temporarily stored in the image memory 12. In step 3, overexposure determination processing is performed on the image data of the frame immediately after imaging temporarily stored in the image memory 12 by the overexposure determination circuit 5.

  In this specification, under a subject with very different light and darkness, a portion that is exposed to strong light (highlight) is extremely overexposed and loses tone information (gradation) to become pure white. Is called “white jump”. Causes of overexposure in captured images include the case where a flash is fired by another camera during imaging and the subject is exposed to strong light, or light from a light source such as the sun is directly incident on the taking lens. Or flare. While the flash from another camera's flash is as short as tens of microseconds, the light and flare from the light source last much longer than the flash from a flash.

  The conventional AE (Automatic exposure) function cannot respond to the flash of a flash that generates strong light in a short time, and cannot prevent overexposure. On the other hand, there is a possibility that the AE function responds to a light or flare from the light source at the beginning, but the AE function responds after a certain amount of time has elapsed, and the subsequent overexposure can be suppressed.

《White skip determination processing》
The image data for each pixel constituting the frame image is compared with a predetermined threshold value, and the number of pixels having image data larger than the threshold value and close to saturation is a predetermined ratio with respect to the number of pixels on the entire screen. In the case where there is the above, it is determined that overexposure has occurred in this frame image. According to this whiteout determination method, absolute whiteout determination can be performed for each frame image regardless of the previous and next frame images.

  In the above-described overexposure determination method, an example in which image data for each pixel is compared with a predetermined threshold value is shown. However, pixels at the same position in a frame image captured before the frame image to be determined The image data may be used as a threshold value. Then, the image data is compared with the image data of the pixel at the same position in the previous frame for each pixel of the frame image captured this time, and a difference is obtained for each pixel. When the number of pixels in which the difference for each pixel exceeds a predetermined amount is greater than or equal to a predetermined ratio with respect to the number of pixels on the entire screen, it is determined that the current frame image is overexposed. According to this whiteout determination method, whiteout determination is performed based on a relative difference from the immediately preceding frame image. For example, the image is taken with knowledge that a part of the screen has a light source such as the sun. Even in this case, it is possible to avoid overexposure determination for a series of consecutive frame images.

  By the way, when the shutter speed at the time of shooting is high (when the exposure time is short), slit exposure is performed, so that a horizontal stripe-like whiteout may occur in the frame image due to flash light. In such a case, the absolute whiteout determination method described above has a small number of pixels exceeding the threshold value, and the relative whiteout determination method described above has a small number of pixels exceeding a predetermined difference. There is a possibility that it is not determined that overexposure occurs. Therefore, in the horizontal band-shaped area corresponding to the slit exposure width in the screen, the number of pixels exceeding the threshold by the above-described absolute overexposure determination method is a predetermined ratio with respect to the total number of pixels in the area. In the above case, or when the number of pixels exceeding a predetermined difference in the relative overexposure determination method is equal to or greater than a predetermined ratio with respect to the total number of pixels in the area, overexposure is performed on the frame image. Is determined to have occurred. According to this whiteout determination method, accurate whiteout determination can be performed even when the shutter speed during shooting is high and slit exposure is performed.

  Further, when the main subject such as a person is separated from the background, only the main subject on the frame image may be overexposed by the flash of another camera. In such a case, there is a possibility that it is not determined that overexposure has occurred for the same reason as in the case of the slit exposure. Therefore, in the focus detection area focused by the AF circuit 9 and the surrounding area among the plurality of focus detection areas in the photographing screen, the above-described absolute whiteout determination method or relative whiteout determination method. Use to check for overexposure. Alternatively, in a camera equipped with a scene recognition function, overexception is determined by the above-described absolute overexposure determination method or relative overexposure determination method in the main subject recognized by the scene recognition and the surrounding area. Do. According to such an overexposure determination method, an accurate overexposure determination can be performed even in a shooting scene in which only the main subject is overexposed.

  In step 4 of FIG. 2 after performing the overexposure determination by the above-described overexposure determination method, it is confirmed whether or not the occurrence of overexposure is determined. If it is determined that overexposure has occurred, the process proceeds to step 5 where the overexposure flag for the overexposure frame image is stored in the flag memory 11 provided separately from the image memory 12 that temporarily stores the frame image data. To do. On the other hand, if it is determined that no whiteout has occurred, step 5 is skipped.

  In step 6, it is determined whether or not the release full-press switch of the operation member 16 is turned off and the moving image shooting is finished. If the moving image shooting has not been completed, the process returns to step 1 to capture the next frame image. On the other hand, when the moving image shooting is completed, the process proceeds to step 7 to correct overexposed frames.

《Whiteout frame correction》
When performing the above-described digital processing and various image processing in the DFE circuit 6 and the image processing circuit 7 on the frame image after the whiteout determination processing, the frame image in which the whiteout flag is set is not adopted as the frame image. The image data of the frame image before the whiteout frame image is adopted as the image data of the whiteout frame image. As a result, the frame images of the same image data as in the previous time are continuous. For example, when shooting a moving image at a frame rate of about 30 FPS, even if two frames of the same image are continuously played back, You don't feel unnatural, and you can watch videos without overexposure.

  In the above-described method for correcting a whiteout frame image, an example in which the previous frame image is used without using a whiteout frame image has been shown. However, when the frame rate is lowered, the viewer feels unnaturalness of image change. become. Therefore, if it is determined that the overexposure has occurred in the second frame image at the center among the first, second, and third consecutive frame images, the second frame image data The second frame image data may be obtained by interpolation from the frame image data for each of the first and second pixels, and may be recorded as the second frame image data. According to this whiteout frame image correction method, even in the case of a moving image shot at a low frame rate, the image change becomes smooth, and it is possible to prevent the viewer from feeling unnatural.

  If it is determined that overexposure occurs in a partial area of the frame image, the image data in that area is replaced with the image data in the same area in the previous frame image, or the frames before and after A correction method may be used in which image data in a whiteout region is obtained from image data by interpolation.

  Further, when the occurrence of overexposure is determined continuously for a plurality of frames, the use of the above-described correction method makes the image change unnatural during reproduction noticeable. In such a case, the image data of the entire overexposed frame or overexposed area may be uniformly reduced to a predetermined ratio (for example, 80%). Alternatively, the correction for overexposure is not performed until the AE function responds, and after the response of the AE function, it is possible to wait for the overexposure to be eliminated by the AE function.

  In step 8 after correcting the overexposed frame image, the DFE circuit 6 and the image processing circuit 7 perform the above-described digital processing and various types of image processing. At this time, the above-described digital processing and various types of image processing are performed on the image data generated by the correction for the overexposed frame. In step 9, a series of frame image data of the moving image is recorded on the memory card 15 via the interface 14, and the execution of the moving image shooting program is terminated.

  In the above-described embodiment, the example of performing the process based on the overexposure determination and the determination result in the digital camera has been described. However, a program that causes the computer to execute the above-described overexposure determination and the process based on the determination result. The program may be provided to the user via the Internet or recorded on a recording medium, and the program may be executed on the user's personal computer. In that case, a moving image may be read from a camera or a recording medium, or a moving image may be acquired via the Internet.

  In the above-described embodiments and their modifications, all combinations of the embodiments or the embodiments and the modifications are possible.

  According to the above-described embodiment and its modifications, the following operational effects can be achieved. First, for each of a plurality of consecutive frame images constituting a moving image, a whiteout determination is performed to determine whether there is a whiteout in which a part or the whole of the frame image is whitened. Since the determined frame image is processed differently from the other frame images, it is possible to identify the frame image in which overexposure has occurred and take appropriate measures.

  According to the above-described embodiment and its modification, the image data for each pixel constituting the frame image is compared with a predetermined threshold value, and the number of pixels having image data larger than this threshold value is the frame image. When there is a predetermined ratio or more with respect to the total number of pixels, it is determined that the overexposure has occurred in the frame image. It is possible to perform typical overexposure determination.

  According to the above-described embodiment and its modification, the image data for each pixel constituting the frame image is compared with the image data for each pixel of the frame image captured before the frame image, and the difference for each pixel. If the number of pixels whose pixel-by-pixel difference exceeds a predetermined amount is greater than or equal to a predetermined ratio with respect to the number of pixels in the entire frame image, it is determined that overexposure occurs in the frame image. Therefore, for example, even when shooting is performed by knowing that there is a light source such as the sun in a part of the screen, it is possible to avoid overexposure determination in a series of continuous frame images.

  According to the above-described embodiment and its modification, the presence / absence of overexposure is determined in the horizontal band-like region of the frame image. Therefore, even when the shutter speed at the time of shooting is high and slit exposure is performed accurately Can be determined.

  According to the above-described embodiment and its modification, the overexposed flag is added in association with the frame image that is determined to have overexposed. The frame image can be identified and appropriate measures can be taken.

  According to the above-described embodiment and its modification, the image data of the frame image is obtained from the frame image captured before the frame image with respect to the frame image determined to be overexposed. Since it has been replaced with image data, it is possible to allow viewers to watch videos without overexposure.

  According to the above-described embodiment and its modification, for the frame image determined to be overexposed, the image data of the frame image is converted into the frame image captured before and after the frame image. Since the image data is used for the interpolation, the change in the image is smooth even in the case of a moving image shot at a low frame rate, so that the viewer does not feel unnaturalness.

DESCRIPTION OF SYMBOLS 3; Image pick-up element, 5; Overexposure determination circuit, 7; Image processing circuit, 11; Flag memory, 12: Image memory, 15; Memory card, 17; Controller, 17a;

Claims (9)

For each of a plurality of continuous frame images constituting a moving image, a whiteout determination unit that determines the presence or absence of whiteout in which part or the whole of the frame image becomes white,
An image processing apparatus comprising: an image processing unit that performs a process different from that of another frame image on a frame image that is determined to have a whiteout by the whiteout determination unit. The image processing apparatus according to claim 1.
The overexposure determining means compares the image data for each pixel constituting the frame image with a predetermined threshold value, and the number of pixels having image data larger than the threshold value is the number of pixels in the entire frame image. On the other hand, when there is a predetermined ratio or more, it is determined that overexposure occurs in the frame image. The image processing apparatus according to claim 1.
The overexposure determining means compares the image data for each pixel constituting the frame image with the image data for each pixel of the frame image captured before the frame image to obtain a difference for each pixel, and An image processing characterized in that when the number of pixels whose difference exceeds a predetermined amount is equal to or greater than a predetermined ratio with respect to the number of pixels of the entire frame image, it is determined that overexposure has occurred in the frame image apparatus. The image processing apparatus according to claim 2 or 3,
2. The image processing apparatus according to claim 1, wherein the overexposure determining unit determines the presence or absence of the overexposure in a horizontal band-like region of a frame image. In the image processing device according to any one of claims 1 to 4,
The image processing device is characterized by adding overexposed information to a frame image determined to have overexposed by the overexposed determining unit. In the image processing device according to any one of claims 1 to 4,
The image processing unit is configured to output image data of the frame image with respect to a frame image that is determined to have a whiteout by the whiteout determination unit, and an image of a frame image that is captured before the frame image. An image processing apparatus characterized by being replaced with data. In the image processing device according to any one of claims 1 to 4,
The image processing means, for the frame image that is determined to have a whiteout by the whiteout determination means, the image data of the frame image, the image of the frame image captured before and after the frame image An image processing apparatus characterized by obtaining by interpolation using data. An imaging means for capturing an image;
The image processing apparatus according to any one of claims 1 to 7, wherein image processing is performed on a moving image including a plurality of continuous frame images captured by the imaging unit.
A digital camera comprising recording control means for recording a moving image after image processing by the image processing apparatus on a recording medium. For each of a plurality of continuous frame images constituting a moving image, a whiteout determination procedure for determining the presence or absence of whiteout in which part or all of the image becomes white,
An image processing program for causing a computer to execute an image processing procedure for performing a process different from that of another frame image on a frame image determined to have undergone a whiteout by the whiteout determination procedure.

Images