JP2011044846A - Image processor and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a corrected image with little blur and noise from a plurality of images. <P>SOLUTION: A target corrected image is generated by combining images (third intermediate generation images) based on a short exposure image of short exposure time and an appropriate exposure image of long exposure time. An edge intensity value calculation part 55 calculates edge intensity at each pixel position of an edge evaluation image (the short exposure image, for instance) as an edge intensity value. An area edge amount calculation part 71 calculates an area edge amount by dividing the entire image area of the edge evaluation image into a plurality of partial image areas and integrating the edge intensity value inside the partial image area for the respective partial image areas. A second composition part 56b sets the composition rate of the short exposure image to the target correction image for each pixel position. The composition rate at the position of the pixel under consideration is increased as the edge intensity value for the position of the pixel under consideration becomes larger, and is increased as the area edge amount for the partial image area to which the position of the pixel under consideration belongs becomes larger. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that performs image processing on an image, and an imaging apparatus such as a digital still camera or a digital video camera.

  In recent years, techniques for suppressing image blur caused by camera shake and subject blur have been actively developed. Among these types of techniques, there are some that perform optical shake correction, but in order to perform optical shake correction, a sensor that detects the shake and a mechanism that optically corrects the shake are required. The mounting of these sensors and mechanisms is disadvantageous in terms of cost and device miniaturization. Therefore, various techniques for correcting blur by image processing after shooting have been proposed.

  For example, in a conventional method, a first image with a high resolution (resolution) but a lot of noise is captured by exposure with a short exposure time, and a second image with a low resolution (resolution) but low noise by exposure with a long exposure time. Shoot. Then, based on the image data of the first and second images, a positional deviation between the two images is detected, and after correcting for the positional deviation, correction is performed by combining the first and second images. An image is generated (for example, see Patent Documents 1 and 2 below).

  At this time, in the method of Patent Document 1, the difference value of the pixel signal between the first and second images is calculated for each pixel. Then, in order to generate a corrected image with less blur and noise, for a pixel having a difference value larger than the threshold value, the pixel is determined to be an edge, and the composition ratio of the first image is relatively increased. For a pixel whose value is less than or equal to the threshold value, the pixel is determined as a non-edge, and the composition ratio of the second image is made relatively large. If the composition ratio of the first image is increased, blurring is suppressed and sharpness is increased. Conversely, if the composition ratio of the second image is increased, noise is suppressed.

  In the method of Patent Document 2, an edge portion is extracted from the difference between pixel signals between the first and second images, and the distance from the edge portion is detected for each pixel. Then, the composition ratio of the first and second images is set for each pixel according to the detection distance.

JP 2007-324770 A JP 2008-236739 A

  In the above-described conventional method, edge / non-edge classification determination is performed for each pixel or distance determination from the edge portion is performed for each pixel, and a composition ratio is set for each pixel based on the determination result. That is, the composition ratio is set for each pixel of interest based on only the determination result for the pixel of interest, regardless of the overall image characteristics of the image region including the pixel of interest.

  However, for example, when the target pixel is located in an image region representing a large flat portion, it seems that priority is given to noise reduction in the image region, which contributes to the overall image quality improvement. Is located in the image area where the flat part and the edge part are mixed, it seems that priority is given to the sharpness in the image area to contribute to the improvement of the overall image quality (similar contents are shown in the drawing). And will be described in detail in the following embodiments). In addition, image data of various types of objects may exist on the captured image, but the composition ratio can be set regardless of the type of object (subject) appearing on the first and second images. It may not be optimal (for example, depending on the type of object, it may be better to prioritize sharpness or not).

  As described above, the conventional method in which the composition ratio is set for each pixel of interest based only on the determination result for the pixel of interest cannot cope with these problems.

Accordingly, an object of the present invention is to provide an image processing apparatus and an imaging apparatus that contribute to the optimization of blur and noise suppression in an image obtained by combining a plurality of images.
The purpose is to provide.

  A first image processing apparatus according to the present invention uses a first image obtained by photographing as a first composition target image, and a second image obtained by photographing with an exposure time longer than an exposure time of the first image, Alternatively, a fourth image that is a synthesized image of the third image and the second image obtained by reducing noise in the first image is set as a second synthesis target image, and the first and second synthesis target images are used. An image processing apparatus that generates an output image by combining the first image to be combined or the entire image area of the third image into a plurality of partial image areas, and for each partial image area, A feature amount extraction unit that extracts an image feature amount of the partial image region based on the image data of the image region is provided, and a combination ratio when combining the first and second combination target images is set for each partial image region. Image of the partial image area And adjusting based on the symptoms weight.

  Accordingly, the synthesis ratio for the partial image area is set in consideration of the overall image characteristics of the partial image area. As a result, for a partial image area where it is better to give priority to suppression of blurring (improvement of sharpness), a part where it is better to give priority to suppression of noise while increasing the composition ratio of the first synthesis target image For the image region, it is possible to increase the composition ratio of the second composition target image.

  Specifically, for example, in the first image processing apparatus, the feature amount extraction unit extracts, for each partial image region, the magnitude of edge strength in the partial image region as the image feature amount.

  Further, for example, in the first image processing apparatus, the contribution degree of the first synthesis target image to the output image is relatively large for a partial image region having a relatively large edge intensity as the image feature amount. The contribution of the first compositing target image to the output image is relatively small with respect to a partial image region having a relatively small edge strength as the image feature amount. The composition ratio may be adjusted.

  Thereby, when the edge portion and the flat portion are mixed in the image of the focused partial image region, the edge strength in the partial image region is increased, and the contribution degree of the first synthesis target image to the output image is increased. As a result, a reduction in the sharpness of the edge portion in the focused partial image region is avoided. In addition, when the image of the focused partial image region is an image of a large flat portion that is continuous, the edge strength in the partial image region is reduced, and the contribution degree of the first synthesis target image to the output image is reduced. That is, priority is given to noise reduction in flat portions where noise is conspicuous.

  More specifically, for example, the first image processing apparatus further includes an edge strength deriving unit that derives an edge strength at each pixel position of the first composition target image or the third image, and the composition ratio is determined based on the pixel position. And the composition ratio with respect to the target pixel position is set based on the edge strength at the target pixel position and the image feature amount of the partial image region to which the target pixel position belongs.

  Also, for example, in the first image processing apparatus, the composition ratio for each partial image area derived based on the image feature amount is corrected so that the amount of change in the composition ratio between adjacent partial image areas is suppressed. Then, the output image may be generated by combining the first and second combining target images using the corrected combining ratio.

  Thereby, it is possible to suppress a sharp change in the texture of the image (image sharpness or the like) at the boundary between adjacent partial image regions.

  The second image processing apparatus according to the present invention uses the first image obtained by photographing as the first composition target image, and the second image obtained by photographing with an exposure time longer than the exposure time of the first image, Alternatively, a fourth image that is a synthesized image of the third image and the second image obtained by reducing noise in the first image is set as a second synthesis target image, and the first and second synthesis target images are used. An image processing apparatus that generates an output image by combining the detection target image, which is an image obtained by photographing one of the first to fourth images or before or after photographing the first image. Each of the first and second synthesis target images includes a specific object detection unit that detects a position and a size of an image area where the image data of the specific object exists from the detection target image based on image data. Image area detected The specific area corresponding to the position and size is classified into a non-specific area other than the specific area, and the composition ratio when the first and second composition target images are synthesized is made different between the specific area and the non-specific area. .

  As a result, when it is considered better to give priority to the suppression (sharpness improvement) of blurring of the specific area according to the type of the specific object, the composition ratio of the first composition target image to the specific area is set to the non-specific area. It is possible to increase the composition ratio of the second synthesis target image with respect to the specific area to be higher than that of the non-specific area when it is considered that priority should be given to suppression of noise in the specific area. Become.

  Specifically, for example, in the second image processing apparatus, the image area in which the image data of the specific object exists is a face area in which image data of a human face exists, and the specific area corresponding to the face area For the region, the contribution of the first composition target image to the output image is relatively small, and for the non-specific region, the contribution of the first composition target image to the output image The synthesis ratio may be adjusted so that the degree becomes relatively large.

  As a result, in the face area of the output image, details are not conspicuous (for example, skin roughness is inconspicuous), and an effect of producing beautiful skin can be obtained.

  Alternatively, for example, in the second image processing apparatus, the contribution ratio of the first synthesis target image to the output image is relatively large for the specific area, and for the non-specific area. The composition ratio may be adjusted so that the degree of contribution of the first composition target image to the output image is relatively small.

  For example, in the second image processing apparatus, the specific area is classified into a first specific area that is relatively far from the non-specific area and a second specific area that is relatively close to the non-specific area. Then, the composition ratio for each region is adjusted so that the contribution degree of the first composition target image to the output image increases or decreases in the order of the first specific region, the second specific region, and the non-specific region. May be.

  A sharp change in the texture of the image (image sharpness or the like) at the boundary between the specific area and the non-specific area is suppressed.

  An imaging apparatus according to the present invention includes an imaging unit that acquires an image by shooting, and any one of the image processing apparatuses described above. The first and second images for the image processing apparatus are acquired by the imaging unit.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus and imaging device which contribute to the suppression suppression optimization of the blur and noise in the image obtained by synthesize | combining a some image can be provided.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. It is an internal block diagram of the imaging part of FIG. It is a figure which shows the example of the short exposure image image | photographed with the imaging device of FIG. 1, and a proper exposure image. It is a figure which shows the two-dimensional coordinate system and two-dimensional image of a space area. It is an internal block diagram of the image correction part which concerns on 1st Embodiment of this invention. It is a figure which shows the 2nd intermediate generation image obtained by reducing the noise of the short exposure image of Fig.3 (a). It is a figure which shows the difference image between the appropriate exposure image (1st intermediate generation image) after alignment, and the short exposure image (2nd intermediate generation image) after a noise reduction process. It is a figure which shows the relationship between the difference value obtained by the difference value calculation part of FIG. 5, and the mixing rate of the pixel signal of a 1st and 2nd intermediate generation image. It is a figure which shows the 3rd intermediate generation image obtained by synthesize | combining the short exposure image (2nd intermediate generation image) after the appropriate exposure image (1st intermediate generation image) after a position alignment, and a noise reduction process. It is a figure which shows the differential filter used in the edge strength value calculation part of FIG. It is a figure which shows the edge image obtained by performing an edge extraction process with respect to the short exposure image (2nd intermediate generation image) after a noise reduction process. FIG. 6 is a diagram illustrating a relationship between the edge intensity value obtained by the edge intensity value calculation unit in FIG. 5 and the pixel signal mixing ratio of the short exposure image and the third intermediate generation image according to the first embodiment of the present invention. is there. It is a figure which shows the target correction image obtained by synthesize | combining a short exposure image and a 3rd intermediate generation image. It is a modified internal block diagram of the image correction unit according to the first embodiment of the present invention. It is a figure which shows the example of imaging | photography object of the imaging device of FIG. FIG. 16 is an enlarged view of a part of a target correction image obtained by applying the method according to the first embodiment of the present invention to the shooting result of the shooting target in FIG. 15. It is an internal block diagram of the 2nd synthetic | combination part which concerns on 2nd Embodiment of this invention. It is a figure which shows a mode that the whole image area | region of a two-dimensional image is divided | segmented into several partial image area | region concerning 2nd Embodiment of this invention. It is a figure which concerns on 2nd Embodiment of this invention and shows the relationship (three types of relationship) of an edge strength value and the mixing rate in a 2nd synthetic | combination calculating part. FIG. 18 is a diagram illustrating a modified example of an input image to the second synthesis unit in FIG. 17 according to the second embodiment of the present invention. It is a figure which concerns on 3rd Embodiment of this invention and shows four partial image areas mutually adjacent. It is a figure which shows the numerical example of the area | region edge amount calculated with respect to the partial image area | region of FIG. It is a figure which shows a mode that the mixing rate of the 2nd synthetic | combination calculating part which concerns on 3rd Embodiment of this invention is correct | amended in comparison with 2nd Embodiment. It is an internal block diagram of the 2nd synthetic | combination part which concerns on 4th Embodiment of this invention. It is the figure which showed a mode that all the image areas of a two-dimensional image were divided | segmented into a specific area and a non-specific area | region by the specific area detection part of FIG. It is a figure showing signs that a face field is extracted from a detection object picture concerning a 4th embodiment of the present invention. It is a figure which concerns on 4th Embodiment of this invention and shows the relationship (two types of relationship) of the edge strength value and the mixing rate in a 2nd synthetic | combination calculating part. It is a schematic internal block diagram of the tracking process part which concerns on 4th Embodiment of this invention. It is a figure which shows the picked-up image sequence assumed in 4th Embodiment of this invention. FIG. 25 is a diagram illustrating a modification of the input image to the second synthesis unit in FIG. 24 according to the fourth embodiment of the present invention. FIG. 16 is a diagram illustrating a manner in which a face area is divided into a face center area and a face peripheral area according to the fifth embodiment of the present invention. It is a figure which concerns on 5th Embodiment of this invention and shows the relationship between an edge strength value and the mixing rate in a 2nd synthetic | combination calculating part. FIG. 20 is a diagram illustrating a manner in which a plurality of pixel positions in a face area are referred to in order to determine a mixing ratio in a second synthesis calculation unit according to the fifth embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle. The first to fifth embodiments will be described later. First, matters that are common to the embodiments or items that are referred to in the embodiments will be described.

  FIG. 1 is an overall block diagram of an imaging apparatus 1 according to an embodiment of the present invention. The imaging device 1 is a digital still camera capable of capturing and recording still images, or a digital video camera capable of capturing and recording still images and moving images. Note that shooting and imaging are synonymous.

  The imaging device 1 includes an imaging unit 11, an AFE (Analog Front End) 12, a main control unit 13, an internal memory 14, a display unit 15, a recording medium 16, and an operation unit 17. The main control unit 13 is provided with an image correction unit 20. The operation unit 17 is provided with a shutter button 17a. The display unit 15 may be regarded as a display device provided outside the imaging device 1.

  FIG. 2 shows an internal configuration diagram of the imaging unit 11. The imaging unit 11 drives and controls the optical system 35, the diaphragm 32, the imaging element 33 such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and the optical system 35 or the diaphragm 32. And a driver 34. The optical system 35 is formed from a plurality of lenses including the zoom lens 30 and the focus lens 31. The zoom lens 30 and the focus lens 31 are movable in the optical axis direction. The driver 34 drives and controls the positions of the zoom lens 30 and the focus lens 31 and the opening degree of the diaphragm 32 based on the drive control signal from the main control unit 13, so that the focal length (angle of view) and the imaging unit 11 are controlled. The focal position and the amount of light incident on the image sensor 33 are controlled.

  The image sensor 33 photoelectrically converts an optical image representing a subject incident through the optical system 35 and the diaphragm 32 and outputs an electrical signal obtained by the photoelectric conversion to the AFE 12. More specifically, the imaging device 33 includes a plurality of light receiving pixels arranged two-dimensionally in a matrix, and each light receiving pixel stores a signal charge having a charge amount corresponding to the exposure time (that is, the length of the exposure period). . An analog signal from each light receiving pixel having a magnitude proportional to the amount of stored signal charge is sequentially output to the AFE 12 in accordance with a drive pulse generated in the imaging device 1. In the following description, “exposure” means exposure of the image sensor 33.

  The AFE 12 amplifies the analog signal output from the imaging unit 11 (image sensor 33), and converts the amplified analog signal into a digital signal. The AFE 12 sequentially outputs this digital signal to the main control unit 13.

  The main control unit 13 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and has a function as a video signal processing unit. Based on the output signal of the AFE 12, the main control unit 13 generates a video signal representing an image captured by the imaging unit 11 (hereinafter also referred to as “captured image”). The main control unit 13 also has a function of controlling the display content of the display unit 15 and performs control necessary for display on the display unit 15. The main control unit 13 outputs an exposure time control signal for controlling the exposure time of the image sensor 33 to the imaging unit 11. The function of the image correction unit 20 provided in the main control unit 13 will be described later.

  The internal memory 14 is formed by SDRAM (Synchronous Dynamic Random Access Memory) or the like, and temporarily stores various data generated in the imaging device 1. The display unit 15 is a display device including a liquid crystal display panel and the like, and displays a photographed image, an image recorded on the recording medium 16, and the like under the control of the main control unit 13. The recording medium 16 is a non-volatile memory such as an SD (Secure Digital) memory card, and records captured images and the like under the control of the main control unit 13. The main control unit 13 is provided with a compression processing unit that compresses image data of an arbitrary image and an expansion processing unit that expands compressed image data, and the compressed image data can be recorded on the recording medium 16. .

  The operation unit 17 receives an operation from the outside. The content of the operation on the operation unit 17 is transmitted to the main control unit 13. The shutter button 17a is a button for instructing photographing and recording of an image.

  The shutter button 17a is formed so that it can be pressed in two stages. When the photographer lightly presses the shutter button 17a, the shutter button 17a is half pressed, and when the shutter button 17a is further pressed from this state, the shutter button 17a is fully pressed.

  The photographed image can include a camera shake caused by camera shake. The image correction unit 20 has a function of generating an image with less blur by image processing using image data represented by an output signal of the imaging unit 11 without optically correcting camera shake. An image with less blur that should be generated by the image correction unit 20 is hereinafter referred to as a “target corrected image”.

The target correction image is generated based on the proper exposure image and the short exposure image. The proper exposure image refers to an image obtained from the image sensor 33 by photographing with exposure with the proper exposure time T _OP , and the short exposure image is the image sensor 33 by photographing with exposure having an exposure time T _SH shorter than the proper exposure time T _OP. Refers to the image obtained from

The sensitivity at the time of acquisition of the short exposure image is adjusted so that the brightness of the photographed image is comparable between the short exposure image and the appropriate exposure image. That is, the sensitivity at the time of acquisition of the short exposure image is appropriate exposure based on the ratio of the appropriate exposure time T _OP and the exposure time T _SH so that the brightness of the photographed image is comparable between the short exposure image and the appropriate exposure image. It is larger than the sensitivity at the time of image acquisition. The sensitivity is, for example, ISO sensitivity. The ISO sensitivity means sensitivity defined by ISO (International Organization for Standardization), and the brightness (luminance level) of a captured image can be adjusted by adjusting the ISO sensitivity. Actually, the amplification factor of the signal amplification in the AFE 12 is determined according to the ISO sensitivity. The amplification degree is proportional to the ISO sensitivity. If the ISO sensitivity is doubled, the degree of amplification is also doubled, whereby the luminance value (that is, brightness) of each pixel of the photographed image is also doubled (however, saturation in amplification is ignored). The luminance value of a certain pixel refers to the value of the luminance signal of that pixel.

  Since the short exposure image is taken with a relatively short exposure time, the short exposure image includes relatively few blurs derived from camera shake and subject blur. Therefore, the edge is clearly depicted in the short exposure image. However, since the sensitivity at the time of shooting is high, the short-exposure image contains a relatively large amount of noise.

  On the other hand, since the proper exposure image is taken with a relatively long exposure time, the noise included in the proper exposure image is relatively small. However, since the exposure time is long, the proper exposure image easily includes camera shake and camera shake.

  3A and 3B show examples of a short exposure image and a proper exposure image. Images 310 and 311 in FIGS. 3A and 3B are examples of a short exposure image and a proper exposure image, respectively. The short exposure image 310 and the proper exposure image 311 are obtained by photographing a state in which a person SUB as a subject of interest stands in front of a mountain that is a background subject.

  Compared to the proper exposure image 311, the short exposure image 310 contains a relatively large noise (corresponding to black spots in FIG. 3A) although the edge is clearly depicted. On the other hand, compared with the short exposure image 310, the proper exposure image 311 contains less noise, but the person SUB is greatly blurred on the proper exposure image 311. In FIGS. 3A and 3B, it is assumed that the person SUB is moving while the short exposure image 310 and the proper exposure image 311 are captured, whereby the person SUB on the short exposure image 310 is assumed. The person SUB on the proper exposure image 311 is located on the right side as compared to the position, and subject blurring has occurred on the person SUB on the proper exposure image 311.

  Also, as shown in FIG. 4, a two-dimensional coordinate system XY of a spatial domain where an arbitrary two-dimensional image 300 is arranged is defined. The image 300 is, for example, the above-described captured image, proper exposure image, short exposure image, target correction image, or first to third intermediate generation images described later. The X axis and the Y axis are axes along the horizontal direction and the vertical direction of the two-dimensional image 300. The two-dimensional image 300 is formed by arranging a plurality of pixels in a matrix in each of the horizontal direction and the vertical direction, and the position of a pixel 301 that is any pixel on the two-dimensional image 300 is (x, y ). In this specification, the position of a pixel is also simply referred to as a pixel position. x and y are coordinate values of the pixel 301 in the X-axis and Y-axis directions, respectively. In the two-dimensional coordinate system XY, when the position of a certain pixel is shifted to the right by one pixel, the coordinate value of the pixel in the X-axis direction increases by 1, and when the position of a certain pixel is shifted downward by one pixel, The coordinate value of the pixel in the Y-axis direction increases by 1. Therefore, when the position of the pixel 301 is (x, y), the positions of the pixels adjacent to the right side, the left side, the lower side, and the upper side of the pixel 301 are (x + 1, y) and (x-1, y), respectively. , (X, y + 1) and (x, y-1).

  Hereinafter, first to fifth embodiments will be described as embodiments for explaining the configuration and operation of the imaging apparatus 1 (particularly, the image correction unit 20). As long as there is no contradiction, the matters described in one embodiment can be applied to other embodiments. In particular, since the premise technique of the technique shown in the second to fifth embodiments is shown in the first embodiment, the matters described in the first embodiment are the second to fifth implementations as long as there is no contradiction. Also applies to form.

<< First Embodiment >>
First, a first embodiment of the present invention will be described. In the first embodiment, as the shutter button 17a is pressed, a short exposure image and a proper exposure image are acquired as follows.

When it is confirmed that the shutter button 17a is in the half-pressed state, the main control unit 13 executes the pre-exposure, determines a proper exposure time T _OP from the output signal level of AFE12 obtained by pre-exposure To do. Pre-exposure refers to exposure performed prior to exposure of a short exposure image and a proper exposure image. Thereafter, when it is confirmed that the shutter button 17a is fully pressed, a short exposure image and a proper exposure image are taken continuously. A proper exposure image may be taken after taking a short exposure image, or a short exposure image may be taken after taking a proper exposure image, but they are continuous so that the exposure interval between both images is as short as possible. Taken on. As described above, the proper exposure image is taken at the proper exposure time T _OP , and the short exposure image is taken at the exposure time T _SH shorter than the proper exposure time T _OP . For example, the exposure time T _SH is set to be equal to or less than the camera shake limit exposure time corresponding to the reciprocal of the focal length of the optical system 35. T _SH may be set to T _OP / 4 or the like.

  FIG. 5 is an internal block diagram of the image correction unit 20a according to the first embodiment. In the first embodiment, an image correction unit 20a is used as the image correction unit 20 in FIG. The image data of the short exposure image and the proper exposure image captured as described above are input to the image correction unit 20a. Image data refers to data representing the color and brightness of an image.

  The image correction unit 20a includes a registration unit 51 that detects a positional deviation between the short exposure image and the proper exposure image and performs registration between the two images, and a noise reduction unit that reduces noise included in the short exposure image. 52, a difference value calculation unit 53 that calculates a difference value between each pixel position by taking the difference between the proper exposure image after alignment and the short exposure image after noise reduction, and a combination ratio based on the difference value A first composition calculator 54 that synthesizes a properly exposed image after alignment and a short-exposure image after noise reduction, and an edge strength value calculator that extracts an edge from the short-exposure image after noise reduction and calculates an edge strength value And a second synthesis calculation unit 56 that generates a target correction image by synthesizing the short-exposure image and the synthesized image by the first synthesis calculation unit 54 at a synthesis ratio based on the edge intensity value. The difference value calculation unit 53 and the first combination calculation unit 54 form a first combination unit 61, and the edge strength value calculation unit 55 and the second combination calculation unit 56 form a second combination unit 62.

  The operation of each part in the image correction unit 20a will be described in detail. It should be noted that simply referring to a short-exposure image refers to a short-exposure image that has not been subjected to noise reduction processing by the noise reduction unit 52 (the same applies to other embodiments described later). The short exposure image 310 illustrated in FIG. 3A is a short exposure image that has not been subjected to noise reduction by the noise reduction unit 52.

  The alignment unit 51 detects misalignment between the short exposure image and the proper exposure image based on the image data of the short exposure image and the proper exposure image, and based on the detected misalignment, the short exposure image and the proper exposure image. Align between them. As described above, since the short-exposure image and the proper exposure image are continuously captured, the capturing areas at the time of capturing both images are substantially the same, but the capturing time of both images is not completely the same, so both images As a result, the position on the short exposure image where the image data of the same subject exists and the position on the proper exposure image may be slightly shifted.

  The alignment unit 51 detects this positional deviation and corrects the positional deviation by performing coordinate conversion on the coordinate values of each pixel of the appropriate exposure image so that the detected positional deviation is canceled. As a result, if the error is ignored, the position on the short exposure image where the image data of the same subject exists and the position on the appropriate exposure image after coordinate conversion completely coincide with each other. The appropriate exposure image after the misalignment correction by the alignment unit 51 (in other words, the appropriate exposure image after alignment) is referred to as a first intermediate generation image.

  As a method of detecting a positional shift by the alignment unit 51, any method capable of detecting a positional shift between two images from image data of two images can be used. For example, the positional deviation can be detected using a representative point matching method, a block matching method, or a gradient method. The positional deviation may be detected using a camera shake detection sensor (for example, an angular velocity sensor that detects an angular velocity of the housing of the imaging device 1) that detects the movement of the housing of the imaging device 1.

  The noise reduction part 52 reduces the noise contained in a short exposure image by performing a noise reduction process with respect to a short exposure image. The noise reduction processing in the noise reduction unit 52 can be realized by arbitrary spatial filtering suitable for noise reduction. In the spatial filtering in the noise reduction unit 52, it is desirable to use a spatial filter that preserves as many edges as possible. For example, spatial filtering using a median filter is employed. Note that the noise reduction processing in the noise reduction unit 52 can be realized by any frequency filtering suitable for noise reduction.

  The short exposure image after the noise reduction processing by the noise reduction unit 52 is referred to as a second intermediate generation image (third image). FIG. 6 shows a second intermediate generated image 312 obtained by performing noise reduction processing on the short exposure image 310 of FIG. As can be seen from the comparison between FIG. 3A and FIG. 6, in the second intermediate generation image 312, the noise included in the short exposure image 310 is reduced, but the edge compared to the short exposure image 310. Is slightly blurred.

  The difference value calculation unit 53 calculates a difference value at each pixel position between the first intermediate generation image and the second intermediate generation image. The difference value at the pixel position (x, y) is represented by DIF (x, y). The difference value DIF (x, y) is the luminance and / or between the pixel at the pixel position (x, y) of the first intermediate generation image and the pixel at the pixel position (x, y) of the second intermediate generation image. It is a value representing a color difference.

The difference value calculation unit 53 calculates the difference value DIF (x, y) based on, for example, the following formula (1). Here, P1 _Y (x, y) is the luminance value of the pixel at the pixel position (x, y) of the first intermediate generation image, and P2 _Y (x, y) is the pixel position of the second intermediate generation image. This is the luminance value of the pixel at (x, y).

It is also possible to calculate the difference value DIF (x, y) based on the following formula (2) or formula (3) by using RGB format signal values instead of formula (1). Here, P1 _R (x, y), P1 _G (x, y), and P1 _B (x, y) are respectively R, G, and R of the pixel at the pixel position (x, y) of the first intermediate generation image. The value of the B signal, P2 _R (x, y), P2 _G (x, y), and P2 _B (x, y) are the pixel values at the pixel position (x, y) of the second intermediate generation image, respectively. R, G and B signal values. The R, G, and B signals of a certain pixel are color signals that represent the red, green, and blue intensities of that pixel.

  The calculation method of the difference value DIF (x, y) based on the above equation (1), equation (2), or equation (3) is merely an example, and the difference value DIF (x, y) is obtained by other methods. It doesn't matter if you do. For example, the difference value DIF (x, y) may be calculated by using the YUV signal value and using the same method as that when the RGB signal value is used. In this case, R, G, and B in formulas (2) and (3) may be replaced with Y, U, and V, respectively. The YUV format signal is composed of a luminance signal represented by Y and a color difference signal represented by U and V.

  FIG. 7 shows an example of a difference image having a difference value DIF (x, y) at each pixel position as a pixel signal value. The difference image 313 in FIG. 7 is a difference image based on the short exposure image 310 and the proper exposure image 311 in FIGS. In the difference image 313, a portion having a relatively large difference value DIF (x, y) is represented in white, and a portion having a relatively small difference value DIF (x, y) is represented in black. Due to the movement of the person SUB during the photographing of the short exposure image 310 and the appropriate exposure image 311, the difference value DIF (x, y) in the movement area of the person SUB in the difference image 313 is relatively large. Further, the difference value DIF (x, y) in the vicinity of the edge (the contour portion of the person or the mountain) is also increased due to the shake on the appropriate exposure image 311 derived from the camera shake.

  The first synthesis calculation unit 54 synthesizes the first intermediate generation image and the second intermediate generation image, and outputs the obtained composite image as a third intermediate generation image (fourth image). This synthesis is realized by weighted addition of pixel signals of corresponding pixels of the first and second intermediate generation images. The pixel signals of the corresponding pixels are mixed by weighted addition, and the mixing ratio (in other words, the combination ratio) can be determined based on the difference value DIF (x, y). The mixing ratio for the pixel position (x, y), which is determined by the first combining calculation unit 54, is represented by α (x, y).

An example of the relationship between the difference value DIF (x, y) and the mixing rate α (x, y) is shown in FIG. When adopting the relationship example of FIG.
When the inequality “DIF (x, y) <TH1 _L ” is satisfied, “α (x, y) = 1” is set.
When the inequality “TH1 _L ≦ DIF (x, y) <TH1 _H ” is satisfied, “α (x, y) = 1− (DIF (x, y) −TH1 _L ) / (TH1 _H −TH1 _L )” And
When the inequality “TH1 _H ≦ DIF (x, y)” is satisfied, “α (x, y) = 0” is set. Here, TH1 _L and TH1 _H are predetermined thresholds satisfying the inequality “0 <TH1 _L <TH1 _H ”. When the relationship example of FIG. 8 is adopted, the corresponding mixing ratio α (x, y) is linear from 1 to 0 as the difference value DIF (x, y) increases from the threshold value TH1 _L toward the threshold value TH1 _H. However, the mixing rate α (x, y) may be decreased nonlinearly.

  The first composition calculator 54 determines the mixing ratio α (x, y) at each pixel position from the difference value DIF (x, y) at each pixel position, and then first and second intermediates according to the following equation (4). The pixel signal of the third intermediate generation image is generated by mixing the pixel signals of the corresponding pixels of the generation image. Here, P1 (x, y), P2 (x, y), and P3 (x, y) are pixel signals at the pixel position (x, y) of the first, second, and third intermediate generation images, respectively. Represents.

  Pixel signals P1 (x, y), P2 (x, y), and P3 (x, y) are respectively the luminance of the pixel at the pixel position (x, y) of the first, second, and third intermediate generation images. A signal representing a color, for example, expressed in RGB format or YUV format. For example, when the pixel signal P1 (x, y) is composed of R, G, and B signals, the pixel signals P1 (x, y) and P2 (x, y) are mixed individually for each R, G, and B signal. Thus, the pixel signal P3 (x, y) may be obtained. The same applies when the pixel signal P1 (x, y) or the like is composed of Y, U, and V signals.

  FIG. 9 shows an example of a third intermediate generation image obtained by the first composition calculation unit 54. A third intermediate generation image 314 shown in FIG. 9 is a third intermediate generation image based on the short exposure image 310 and the proper exposure image 311 shown in FIGS.

  As described above, since the difference value D (x, y) is relatively large in the region where the person SUB moves, the contribution degree (1) of the second intermediate generation image 312 (see FIG. 6) to the third intermediate generation image 314 -Α (x, y)) becomes relatively large. As a result, subject blurring in the third intermediate generated image 314 is significantly suppressed as compared with that in the proper exposure image 311 (see FIG. 3B). Further, since the difference value D (x, y) becomes large near the edge, the contribution (1-α (x, y)) becomes large. As a result, the sharpness of the edge in the third intermediate generation image 314 is improved as compared with that in the proper exposure image 311. However, since the edge in the second intermediate generation image 312 is slightly unclear compared to that of the short exposure image 310, the edge in the third intermediate generation image 314 is also slightly unclear compared to that of the short exposure image 310.

  On the other hand, the region where the difference value D (x, y) is relatively small is presumed to be a flat region with few edge components. Therefore, for the region where the difference value D (x, y) is relatively small, as described above, the contribution degree α (x, y) of the first intermediate generation image with a small amount of noise is made relatively large. . Thereby, the noise of a 3rd intermediate generation image can be suppressed low. Since the second intermediate generation image is generated through the noise reduction process, the contribution degree (1-α (x, y)) of the second intermediate generation image to the third intermediate generation image is relatively large. Even in the area, the noise is hardly noticeable.

  As described above, the edge in the third intermediate generation image is slightly blurred compared with that of the short-exposure image, but this blur is improved by the edge intensity value calculation unit 55 and the second synthesis calculation unit 56.

  The edge strength value calculation unit 55 performs edge extraction processing on the second intermediate generation image, and calculates an edge strength value at each pixel position. The edge intensity value at the pixel position (x, y) is represented by E (x, y). The edge intensity value E (x, y) is an index representing the change amount of the pixel signal in the small block centered on the pixel position (x, y) of the second intermediate generation image, and the larger the change amount, the more the edge The intensity value E (x, y) increases.

For example, the edge intensity value E (x, y) can be obtained according to the following formula (5). In addition, you may make it obtain | require 1/4 of the value of the right side of Formula (5) as edge strength value E (x, y). As described above, P2 _Y (x, y) represents the luminance value of the pixel at the pixel position (x, y) of the second intermediate generation image. F _A (i, j), F _B (i, j), F _C (i, j) and F _D (i, j) extract edges extending in the horizontal, vertical, right diagonal and left diagonal directions, respectively. Represents the filter coefficient of the edge extraction filter for As the edge extraction filter, any spatial filter suitable for edge extraction can be used. For example, a differential filter, a Prewitt filter, or a Sobel filter can be used.

When using a four-direction differential filter as shown in FIGS.
E (x, y) = | P2 _Y (x, y−1) −P2 _Y (x, y + 1) |
+ | P2 _Y (x-1, y) -P2 _Y (x + 1, y) |
+ | P2 _Y (x-1, y-1) -P2 _Y (x + 1, y + 1) |
+ | P2 _Y (x + 1, y−1) −P2 _Y (x−1, y + 1) |
Note that although the edge extraction filter having a 3 × 3 filter size is used in Expression (5), the filter size of the edge extraction filter may be other than 3 × 3.

  FIG. 11 shows an example of an edge image having an edge intensity value E (x, y) at each pixel position as a pixel signal value. An edge image 315 in FIG. 11 is an edge image based on the short exposure image 310 and the proper exposure image 311 in FIGS. In the edge image 315, a portion where the edge intensity value E (x, y) is relatively large is represented in white, and a portion where the edge intensity value E (x, y) is relatively small is represented in black. The edge intensity value E (x, y) is obtained by extracting an edge of the second intermediate generation image 312 obtained by suppressing noise of the short exposure image 310 having a clear edge. Therefore, the noise and the edge are separated, and the position of the edge is specified after clearly distinguishing the noise from the edge of the subject by the edge intensity value E (x, y).

  The second synthesis calculation unit 56 synthesizes the third intermediate generated image and the short exposure image, and outputs the obtained synthesized image as a target correction image. This synthesis is realized by weighted addition of pixel signals of corresponding pixels of the third intermediate generation image and the short exposure image. The pixel signals of the corresponding pixels are mixed by weighted addition, and the mixing ratio (in other words, the combination ratio) can be determined based on the edge intensity value E (x, y). The mixing ratio for the pixel position (x, y), which is determined by the second combining calculation unit 56, is represented by β (x, y).

An example of the relationship between the edge strength value E (x, y) and the mixing ratio β (x, y) is shown in FIG. When adopting the relationship example of FIG.
When the inequality “E (x, y) <TH2 _L ” is satisfied, “β (x, y) = 0” is set.
When the inequality “TH2 _L ≦ E (x, y) <TH2 _H ” is satisfied, “β (x, y) = (E (x, y) −TH2 _L ) / (TH2 _H −TH2 _L )” is established.
When the inequality “TH2 _H ≦ E (x, y)” is satisfied, “β (x, y) = 1” is set.
Here, TH2 _L and TH2 _H is a predetermined threshold that satisfies the inequality "0 <TH2 _L <TH2 _H". When the example of the relationship of FIG. 12 is adopted, the corresponding mixing ratio β (x, y) increases from 0 to 1 as the edge strength value E (x, y) increases from the threshold TH2 _L toward the threshold TH2 _H. Although it increases linearly, the mixing rate β (x, y) may be increased nonlinearly.

After determining the mixing ratio β (x, y) at each pixel position from the edge intensity value E (x, y) at each pixel position, the second synthesis calculation unit 56 then performs the third intermediate generation image according to the following equation (6). And the pixel signal of the corresponding pixel of the short exposure image are mixed to generate the pixel signal of the target corrected image. Here, P _OUT (x, y), P _{IN — SH} (x, y), and P3 (x, y) are the pixel position (x, y) of the target correction image, the short exposure image, and the third intermediate generation image, respectively. Represents a pixel signal.

Pixel signals P _OUT (x, y), P _{IN — SH} (x, y), and P 3 (x, y) are respectively at the pixel position (x, y) of the target correction image, the short exposure image, and the third intermediate generation image. It is a signal representing the luminance and color of a pixel, and is represented in, for example, an RGB format or a YUV format. For example, when the pixel signal P3 (x, y) or the like is composed of R, G, and B signals, the pixel signals P _{IN — SH} (x, y) and P3 (x, y) are individually set for each of the R, G, and B signals. The pixel signal P _OUT (x, y) may be obtained by mixing. The same applies when the pixel signal P3 (x, y) or the like is composed of Y, U, and V signals.

  FIG. 13 shows an example of the target correction image obtained by the second synthesis calculation unit 56. A target correction image 316 shown in FIG. 13 is a target correction image based on the short exposure image 310 and the proper exposure image 311 shown in FIGS. In the edge portion, the contribution β (x, y) of the short-exposure image 310 to the target correction image 316 is large. Therefore, in the target correction image 316, a slight edge of the third intermediate generation image 314 (see FIG. 9) is present. Unsharpness is improved and edges are clearly depicted. On the other hand, in the portion other than the edge, the contribution degree (1-β (x, y)) of the third intermediate generation image 314 to the target correction image 316 is large, so that the short exposure image 310 is compared with the target correction image 316. It is suppressed that the included noise is reflected. Since noise is particularly visually noticeable in a portion other than the edge (flat portion), the adjustment of the synthesis ratio by the mixing rate β (x, y) as described above is effective.

  As described above, according to the first embodiment, the proper exposure image (more specifically, the proper exposure image after alignment (that is, the first intermediate generation image)) and the short exposure image after noise reduction (that is, the second intermediate generation image). ) With the difference values obtained from them, it is possible to generate a third intermediate generation image in which the blur of the proper exposure image and the noise of the short exposure image are suppressed. Thereafter, the third intermediate generation image and the short exposure image are synthesized using the edge intensity value obtained from the short exposure image after noise reduction (that is, the second intermediate generation image), so that the short exposure image is clearly displayed on the target correction image. While a simple edge can be reflected, the reflection of the noise of the short exposure image on the target correction image is suppressed. As a result, the target correction image is an image with less blur and noise.

In order to clearly separate and detect the edge and noise and to avoid the mixing of the noise of the short exposure image into the target correction image, the short intensity image after the noise reduction (that is, the second intermediate image) as described above is used. It is preferable to derive from the generated image). However, as shown in FIG. 14, the edge intensity value may be derived from the short exposure image before noise reduction (that is, the short exposure image 310 in FIG. 3A). . In this case, after substituting the luminance value of the pixel at the pixel position (x, y) of the short-exposure image before noise reduction into P2 _Y (x, y) in Expression (5), the edge intensity value according to Expression (5) E (x, y) may be calculated.

  Although the above-described target correction image generation method is a very useful method, there is room for improvement. For example, consider a case where a short-exposure image and a proper-exposure image are captured using leaves and flowers of a plant as shown in FIG. FIG. 16 is an enlarged view of a part of the target correction image obtained by the method according to the first embodiment in this case. However, in FIG. 16, for the sake of convenience of explanation, the features of the image are highlighted.

  In FIG. 16, a solid line 401 represents the outline of one leaf. Since the edge intensity value is relatively large in the contour portion of the leaf, the composition ratio of the short exposure image is increased. As a result, the edge of the contour portion of the leaf is clear in the target correction image. However, on the inner side of the contour of the leaf, the edge intensity value is relatively small, so the synthesis ratio of the third intermediate generation image is increased, and as a result, sharpness is lacking on the target correction image (however, the signal-to-noise ratio is good) Is). The occurrence of such a phenomenon is attributed to the fact that the combination ratio in the second combination calculation unit 56 is determined based only on the edge intensity value for each pixel position. A method for suppressing the occurrence of such a phenomenon will be described in a second embodiment described later.

<< Second Embodiment >>
A second embodiment of the present invention will be described. FIG. 17 is an internal block diagram of the second synthesis unit 62b according to the second embodiment. An image correction unit provided with a second synthesis unit 62b in FIG. 17 instead of the second synthesis unit 62 in FIG. 5 with respect to the image correction unit 20a in FIG. 5 can be used as the image correction unit 20 in FIG.

  The second synthesis unit 62b includes an edge strength value calculation unit 55, a second synthesis calculation unit 56b, and a region edge amount calculation unit 71. The function of the edge strength value calculation unit 55 in the second synthesis unit 62b is the same as that described in the first embodiment. That is, the edge intensity value calculation unit 55 in the second synthesis unit 62b performs edge extraction processing on the edge evaluation image, thereby calculating the edge intensity value E (x, y) at each pixel position of the edge evaluation image. . As is clear from the description of the first embodiment, the edge evaluation image is a short-exposure image (first image) or a second intermediate generation image (third image).

  In the second synthesizing unit 62b, the entire image area (hereinafter referred to as the entire image area) in which the pixels forming the two-dimensional image 300 (see FIG. 4) are arranged is calculated into a plurality of areas (hereinafter referred to as partial image areas). To be divided). Now, as shown in FIG. 18, by dividing the entire image area of the two-dimensional image 300 into P equal parts in the horizontal direction and Q equal parts in the vertical direction, a total of (P × Q) partial images in the two-dimensional image 300. Assume that an area is set. Here, each of P and Q is an integer of 2 or more. A plurality of pixels (for example, 100 pixels) that are two-dimensionally arranged belong to each partial image region. Although it is possible to vary the number of pixels belonging to a partial image area between different partial image areas, in the following description, the number of pixels belonging to one partial image area is the same in all partial image areas. Suppose that Also, p and q are introduced as symbols representing the horizontal position and vertical position of the partial image area in the two-dimensional image 300 (p is an integer value satisfying 1 ≦ p ≦ P, and q is 1 ≦ q ≦ Q. Integer value to satisfy). A partial image area in which the horizontal position is p and the vertical position is q is referred to as a partial image area [p, q]. In the two-dimensional image 300 and the two-dimensional coordinate system XY (see FIG. 4), as p increases, the horizontal position of the partial image region [p, q] moves toward the right, and as q increases, the partial image region [p, q The vertical position of q] is assumed to be directed downward. Therefore, the partial image areas adjacent to the right side, the left side, the lower side, and the upper side of the partial image area [p, q] are the partial image areas [p + 1, q], [p-1, q], [p, q + 1, respectively. ] And [p, q-1].

The area edge amount calculation unit 71 integrates the edge intensity values E (x, y) of the pixel positions belonging to the partial image area for each partial image area, and obtains the integrated value as the area edge amount. Accordingly, the region edge amount is obtained for each partial image region. The region edge amount obtained for the partial image region [p, q] is represented by E _RG [p, q].

  The second synthesis calculation unit 56b generates a target correction image by synthesizing the third intermediate generation image and the short exposure image in the same manner as the second synthesis calculation unit 56 according to the first embodiment. However, the second synthesis calculation unit 56b sets the mixing rate β (x, y) for the target pixel position (x, y) not only to the edge intensity value E (x, y) for the target pixel position (x, y). This is determined based on the region edge amount of the partial image region to which the target pixel position (x, y) belongs.

For clarification of the description, the target pixel position (x, y), the partial image region [p, q] to which the target pixel position (x, y) belongs, and the mixing ratio β (x , Y) are represented by (x ₀ , y ₀ ), [p ₀ , q ₀ ] and β (x ₀ , y ₀ ), respectively, and then mixed with the target pixel position (x ₀ , y ₀ ). A method for determining the rate β (x ₀ , y ₀ ) will be described. The region edge amount obtained for the partial image region [p ₀ , q ₀ ] is represented by E _RG [p ₀ , q ₀ ].

The graph of FIG. 19A shows an edge intensity value E (x ₀ , y ₀ ) and a mixing ratio β (x that are employed when the first inequality “E _RG [p ₀ , q ₀ ] <E _TH1 ” is satisfied. ₀ , y ₀ ), and the graph of FIG. 19B shows the edge strength value employed when the second inequality “E _TH1 ≦ E _RG [p ₀ , q ₀ ] <E _TH2 ” is satisfied. The relationship between E (x ₀ , y ₀ ) and the mixing ratio β (x ₀ , y ₀ ) is shown, and the graph of FIG. 19C shows the third inequality “E _TH2 ≦ E _RG [p ₀ , q ₀ ]”. Represents the relationship between the edge intensity value E (x ₀ , y ₀ ) and the mixing ratio β (x ₀ , y ₀ ) employed when “ Here, E _TH1 and E _TH2 are thresholds set in advance so as to satisfy the inequality “0 <E _TH1 <E _TH2 ”.

In the case where any of the first, second and third inequalities is satisfied,
When the inequality “E (x ₀ , y ₀ ) <TH2 _L ” is satisfied, “β (x ₀ , y ₀ ) = β _MIN ” is set,
When the inequality “TH2 _L ≦ E (x ₀ , y ₀ ) <TH 2 _H ” is satisfied, “β (x ₀ , y ₀ ) = β _MIN + (1−β _MIN ) × ((E (x ₀ , y ₀ )) −TH2 _L ) / (TH2 _H −TH2 _L )) ”, and“ β (x ₀ , y ₀ ) = 1 ”when the inequality“ TH2 _H ≦ E (x ₀ , y ₀ ) ”is satisfied.

Here, the lower limit value β _MIN is a variable value that depends on the region edge amount E _RG [p ₀ , q ₀ ]. The lower limit value β _MIN when the third inequality is satisfied is set larger than the lower limit value β _MIN when the second inequality is satisfied, and the lower limit value β _MIN when the second inequality is satisfied is the first value It is made larger than the lower limit β _MIN when the inequality is established. In particular, in the example shown in FIG. 19A, the lower limit β _MIN when the first inequality is satisfied is set to zero.

The thresholds TH2 _L and TH2 _H always satisfy the inequality “0 <TH2 _L <TH2 _H ” as described in the first embodiment. However, in the first embodiment, it is assumed that the threshold values TH2 _L and TH2 _H are constant values, but in the second embodiment, at least one of the threshold values TH2 _L and TH2 _H is a variable value. In the example shown in FIGS. 19A to 19C, the threshold value TH2 _L is a fixed value, while the threshold value TH2 _H is a variable value that depends on the region edge amount E _RG [p ₀ , q ₀ ]. Specifically, the threshold TH2 _H when the first inequality is established is set larger than the threshold TH2 _H when the second inequality is established, and the threshold TH2 _H when the second inequality is established is It is larger than the threshold value TH2 _H during the establishment of the third inequality.

19A to 19C, when the inequality “TH2 _L ≦ E (x ₀ , y ₀ ) <TH2 _H ” is satisfied, the edge strength value E (x ₀ , y ₀₎ is the corresponding mixing ratio as increases from the threshold TH2 _L the threshold TH2 _H beta (x _0, y ₀₎ is increased linearly, mixing ratio beta of (x _0, y ₀₎ in a non-linear It may be increased. Further, although the above-described example of changing in three stages the lower limit value beta _MIN and the threshold TH2 _H in accordance with the area edge amount _{E RG [p 0, q 0} ], those regions edge amount E _RG [p _0, q ₀ ] may be changed stepwise in two steps or four or more steps, or may be changed continuously in accordance with the region edge amount E _RG [p ₀ , q ₀ ]. . In any case, as the region edge amount E _RG [p ₀ , q ₀ ] increases, the lower limit β _MIN in determining the mixing rate β (x ₀ , y ₀ ) is increased and the mixing rate β (x ₀ , Y ₀ ), the threshold value TH2 _H when determining is good. As the region edge amount E _RG [p ₀ , q ₀ ] increases, the threshold value TH2 _L can be increased.

As described above, the setting of the mixing ratio (x ₀ , y ₀ ) based on the edge intensity value E (x ₀ , y ₀ ) and the region edge amount E _RG [p ₀ , q ₀ ] is performed for all pixel positions (x , Y) are sequentially executed after sequentially considering the target pixel position (x ₀ , y ₀ ), and finally, the mixture ratio (x, y) for all the pixel positions (x, y) is determined. . The method of generating the target correction image using the mixing ratio β (x, y) by the second synthesis calculation unit 56b is the same as that of the second synthesis calculation unit 56 described in the first embodiment.

According to the second embodiment, for each partial image region, the region edge amount E _RG [p, q] representing the magnitude of the edge strength in the partial image region is extracted as the image feature amount of the partial image region. Then, the degree of contribution of the short-exposure image to the target correction image (that is, β (x, y)) is adjusted according to the region edge amount E _RG [p, q] for each partial image region. If it is assumed that the edge intensity values E (x, y) for all pixel positions are the same by this adjustment, the target for a partial image region having a relatively large region edge amount E _RG [p, q] The contribution of the short-exposure image to the correction image is relatively large, and the contribution of the short-exposure image to the target correction image is relatively small for a partial image region having a relatively small area edge amount E _RG [p, q] Get smaller.

  Therefore, for example, if the image of the partial image region to which the target pixel position belongs is an image of a large flat portion that is continuous, the corresponding region edge amount is small, and therefore the third intermediate generation for the target correction image in the partial image region The contribution degree of the image is increased, and the noise can be reduced satisfactorily. Since noise is conspicuous in a large continuous flat portion, increasing the contribution of the third intermediate generation image contributes to the improvement of the overall image quality.

  On the other hand, when the edge portion and the flat portion are mixed in the image of the partial image region to which the target pixel position belongs, the contribution of the short-exposure image to the target correction image is increased in the partial image region, and as a result, the edge is rounded. The sharpness reduction due to this is avoided. That is, for example, when a leaf or flower of a plant as shown in FIG. 15 is taken as an object to be photographed (see also FIG. 16), not only the edge intensity value for each pixel position but also the entire partial image area including the outline of the leaf Since the mixing rate β (x, y) for the partial image region is determined based on the edge intensity as a result (as a result, the mixing rate β (x, y) for the partial image region is compared with the first embodiment. ) Is corrected to increase), edges such as leaf veins existing inside the contour of the leaf are appropriately stored, and sharpness is increased as compared with the first embodiment.

  In addition, even if the third intermediate generation image input to the second synthesis calculation unit 56b is replaced with the first intermediate generation image (see FIG. 5), the operations and effects peculiar to the second embodiment can be obtained. That is, as shown in FIG. 20, the first intermediate generation image is input to the second synthesis calculation unit 56b instead of the third intermediate generation image, and the first intermediate generation image and the short exposure are input in the second synthesis calculation unit 56b. The target correction image may be generated by combining the images. In this case, the third intermediate generated image in the explanatory text of the second embodiment and the third intermediate generated image in the explanatory text of the function of the second synthesis calculation unit 56 described in the first embodiment are converted into the first intermediate generated image. (P3 (x, y) in the above equation (6) may be replaced with P1 (x, y)). In addition, when the first intermediate generation image is input to the second synthesis calculation unit 56b instead of the third intermediate generation image, the first synthesis unit 61 can be omitted from the image correction unit 20 or 20a. Yes (see FIGS. 1 and 5).

<< Third Embodiment >>
A third embodiment of the present invention will be described. The third embodiment is an embodiment obtained by modifying a part of the second embodiment, and the description of the second embodiment is also applied to the present embodiment with respect to matters not specifically described in the present embodiment.

In the second embodiment, the same region edge amount E [p ₀ , q ₀ ] is applied to all pixel positions belonging to the partial image region [p ₀ , q ₀ ]. As a result, the partial image region [p ₀ , Q ₀ ], a common lower limit value β _MIN and threshold value TH _2H according to the region edge amount E [p ₀ , q ₀ ] are applied to all pixel positions belonging to. Therefore, in the second embodiment, when the obtained region edge amount differs greatly between adjacent partial image regions, the texture (image sharpness, etc.) of the image changes sharply at the boundary between adjacent partial image regions. There are things to do. The technology according to the third embodiment contributes to suppression of such a steep change.

  The technique according to the third embodiment will be specifically described. Now, paying attention to the four partial image regions [p, q], [p + 1, q], [p, q + 1] and [p + 1, q + 1] shown in FIG. 21, the partial image regions [p, q], [p + 1] , Q], [p, q + 1] and [p + 1, q + 1], the pixel positions arranged at the centers are represented by reference numerals 431, 432, 433 and 434, respectively.

As described in the second embodiment, the region edge amount calculation unit 71 integrates the edge intensity values E (x, y) of pixel positions belonging to the partial image region [p, q], and uses the integrated value as the region edge amount. _{Calculated as} E _EG [p, q]. Through similar integration calculations, region edge amounts E _EG [p + 1, q] and E _EG [p, q + 1] are respectively obtained for partial image regions [p + 1, q], [p, q + 1] and [p + 1, q + 1]. ] And E _EG [p + 1, q + 1].

When the pixel position 431 is the target pixel position (x ₀ , y ₀ ), the second composition calculation unit 56b determines the lower limit value β _MIN and the threshold value TH2 when determining the mixing ratio β (x ₀ , y ₀ ). _H is set according to the region edge amount E _EG [p, q]. Similarly, when the pixel positions 432, 433, and 434 are the target pixel position (x ₀ , y ₀ ), the lower limit value β _MIN and the threshold value TH2 _H when determining the mixing ratio β (x ₀ , y ₀ ). Are set according to the region edge amounts E _EG [p + 1, q], E _EG [p, q + 1] and E _EG [p + 1, q + 1], respectively. Thus, the lower limit β _MIN and threshold TH2 _H for the pixel positions 431, 432, 433, and 434 are the region edge amounts E _EG [p, q], E _EG [p + 1, q], E _EG [p, q + 1] and E _EG [p + 1, q + 1] only.

On the other hand, the pixel positions (excluding the pixel positions 431 to 434) in the rectangular region surrounded by the pixel positions 431 to 434, such as the pixel position 441 or 442 shown in FIG. 21, are the target pixel positions (x ₀ , y _0). ), The lower limit value β _MIN and threshold value TH2 _H when determining the mixing ratio β (x ₀ , y ₀ ) are set to the region edge amounts E _EG [p, q], E _EG [p + 1, q]. , E _EG [p, q + 1] and E _EG [p + 1, q + 1] are set based on two or more region edge amounts. Each of the pixel positions 441 and 442 is a pixel position in a rectangular area surrounded by the pixel positions 431 to 434, the pixel position 441 belongs to the partial image area [p, q], and the pixel position 442 is a partial image area. It belongs to [p + 1, q + 1].

The region edge amount calculation unit 71 or the second composition calculation unit 56b calculates the region edge amounts E _EG [p, q], E _EG [p + 1, q], E _EG [p, q + 1], and E _EG [p + 1, q + 1]. _Assume that the region edge amounts for the pixel positions ₄₃₁ , 432, 433, and 434 are the region edge amounts E _{EG —} 441 for the pixel position 441, the region edge amounts E _EG [p, q], E _EG [p + 1, q], E _EG [p, q + 1] and E _EG [p + 1, q + 1] are obtained by bilinear interpolation based on the distance between the pixel position 441 and the pixel positions ₄₃₁ to _434. Similarly, the region edge amount E _{EG —} 442 with respect to the pixel position 442 is obtained. , area edge amount E _EG [p, q], the distance between the _{E EG [p + 1, q} ], E EG [p, q + 1] and _{E EG [p + 1, q} + 1] as well as the pixel position 442 and the pixel position 431-434 Based calculated by bilinear interpolation. Although different from the situation shown in FIG. 21, if the pixel position 441 is arranged at the center of a line segment connecting the pixel positions 431 and 433, for example, “E _EG — ₄₄₁ = (E _EG [p, q] + E _EG [p, q + 1]) / 2 ”and if the pixel position 442 is arranged at the center of the rectangular area surrounded by the pixel positions ₄₃₁ to _434, “ E _EG — ₄₄₂ = (E _EG [p, q ] + E _EG [p + 1, q] + E _EG [p, q + 1] + E _EG [p + 1, q + 1]) / 4 ”.

When the pixel position 441 is the target pixel position (x ₀ , y ₀ ), the second composition calculation unit 56b determines the lower limit value β _MIN and the threshold value TH2 when determining the mixing ratio β (x ₀ , y ₀ ). _{When H} is set according to the region edge amount E _{EG — 441} and the pixel position 442 is the _target pixel position (x ₀ , y ₀ ), the mixing ratio β (x ₀ , y ₀ ) is determined. The lower limit value β _MIN and threshold value TH2 _H are set according to the region edge amount E _{EG — 442} .

As described above, each of the pixel positions 441 and 442 is a pixel position in a rectangular area surrounded by the pixel positions 431 to 434, and the pixel position 441 belongs to the partial image area [p, q] and the pixel position. 442 belongs to the partial image area [p + 1, q + 1]. Therefore, for example, as shown in FIG. 22, numerical values representing the region edge amounts E _EG [p, q], E _EG [p + 1, q], E _EG [p, q + 1], and E _EG [p + 1, q + 1] In the case of 255, 200, 200 and 200, each of the region edge amounts E _{EG — 441} and E _{EG — 442} is less than 255 and greater than 200, and the region edge amount E _{EG — 441} is _greater than the region edge amount E _{EG — 442} . Also grows.

That is, as shown in FIG. 23A, the lower limit β _MIN for the pixel position 441 is corrected to be decreased and the threshold value TH2 _H for the pixel position 441 is increased and corrected as compared to the case where the second embodiment is adopted. As shown in (b), the lower limit value β _MIN for the pixel position 442 is corrected to increase and the threshold value TH2 _H for the pixel position 442 is corrected to decrease compared to the case where the second embodiment is adopted. As a result, compared with the case where the second embodiment is adopted, the mixing rate β (x, y) with respect to the pixel position 441 is corrected to decrease, while the mixing rate β (x, y) with respect to the pixel position 442 is corrected to increase. The

As described above, in the third embodiment, although the lower limit value β _MIN and the threshold value TH2 _H are provisionally determined for each partial image area based on the area edge amount E _EG [p, q], in corrected lower limit value beta _MIN and threshold TH2 provisional limit value in the direction of variation of _H is suppressed beta _MIN and threshold TH2 _H is the final using the lower limit value beta _MIN and the threshold TH2 _H after correction The mixing rate β (x, y) is determined. In other words, the mixing rate β (x, y) for each partial image region derived based on the region edge amount E _EG [p, q] is equal to the mixing rate β (x, y) between adjacent partial image regions. The correction is performed so that the amount of change is suppressed, and the third intermediate generation image and the short-exposure image are combined using the corrected mixing ratio β (x, y). Thereby, it is possible to suppress a sharp change in the texture of the image (image sharpness or the like) at the boundary between adjacent partial image regions.

<< Fourth Embodiment >>
A fourth embodiment of the present invention will be described. FIG. 24 is an internal block diagram of the second synthesis unit 62d according to the fourth embodiment. An image correcting unit provided with a second combining unit 62d in FIG. 24 instead of the second combining unit 62 in FIG. 5 with respect to the image correcting unit 20a in FIG. 5 can be used as the image correcting unit 20 in FIG.

  The second synthesis unit 62d includes an edge strength value calculation unit 55, a second synthesis calculation unit 56d, and a specific area detection unit 81. The function of the edge strength value calculation unit 55 in the second synthesis unit 62d is the same as that described in the first embodiment. That is, the edge strength value calculation unit 55 in the second synthesis unit 62d performs edge extraction processing on the edge evaluation image, thereby calculating an edge strength value E (x, y) at each pixel position of the edge evaluation image. . As is clear from the description of the first embodiment, the edge evaluation image is a short-exposure image (first image) or a second intermediate generation image (third image).

  The specific area detection unit 81 is provided with image data of the detection target image. The detection target image is any one of a short exposure image, a proper exposure image, and first to third intermediate generation images that are the basis of the target correction image. Normally, the short exposure image or the third intermediate generation image (fourth image) input to the second synthesis unit 62d is set as the detection target image. However, the image other than the short exposure image and the proper exposure image that are the basis of the target correction image and the first to third intermediate generation images based on them may be detection target images. For example, when a proper exposure image is photographed next to a short exposure image that is a source of the target correction image, an image photographed by the imaging unit 11 immediately before the short exposure image is photographed, or just after the proper exposure image is photographed. It is also possible to handle an image captured by the imaging unit 11 as a detection target image.

  Based on the image data of the detection target image, the specific area detection unit 81 selects an image area (hereinafter, referred to as a specific area) in which image data of a specific subject (in other words, a specific object) exists as the entire image area of the detection target image. Specific area information is extracted from the image and includes information indicating the position and size of the specific area on the detection target image and the two-dimensional coordinate system XY. The shape of the specific area is defined by the specific area information. For example, if the specific area is a rectangular area, the coordinate values of two vertices on the diagonal of the rectangular area may be included in the specific area information, or the coordinate value of one vertex of the rectangular area and the rectangle The size of the region in the horizontal and vertical directions may be included in the specific region information. As shown in FIG. 25, in any two-dimensional image 300, an image area other than the specific area is referred to as a non-specific area.

  Based on the edge intensity value E (x, y) and the specific area information, the second combination calculation unit 56d generates a target correction image by combining the short exposure image and the third intermediate generation image. At this time, even if the edge intensity value E (x, y) is the same, the combination ratio (that is, β (x, y)) when combining the short-exposure image and the third intermediate generation image is the same. Differentiate between non-specific areas.

  The specific operation of the second combining unit 62d differs depending on the type of the specific subject. Hereinafter, first to third operation examples will be individually described as specific examples of the operation of the second combining unit 62d.

[First operation example]
A first operation example will be described. In the first operation example, it is assumed that the specific subject (in other words, the specific object) is a human face. As shown in FIG. 26, the specific area in the first operation example is a face area where image data of a human face exists, and the non-specific area in the first operation example is a non-face area that is an image area other than the face area. . Any method including a known method can be used as a method for extracting a face region based on the image data of the detection target image.

Based on the specific area information indicating the position and size of the face area, the second composition calculation unit 56d determines whether the target pixel position (x ₀ , y ₀ ) belongs to the face area or the non-face area. Depending on whether the pixel position (x ₀ , y ₀ ) belongs to a face area or a non-face area, the relationship between the edge intensity value E (x ₀ , y ₀ ) and the mixing rate β (x ₀ , y ₀ ) differs. Make it. The relationship between the edge strength value E (x ₀ , y ₀ ) and the mixing ratio β (x ₀ , y ₀ ) has a plurality of types of relationships, and the plurality of types of relationships include the relationships RL _A and RL _B. It is. The broken lines 470 _A and 470 _{B in} FIGS. 27A and 27B indicate the relations RL _A and RL _B , respectively.

When the target pixel position (x ₀ , y ₀ ) belongs to the face area, the relationship RL _A is applied to the target pixel position (x ₀ , y ₀ ). When the relationship RL _A is applied to the target pixel position (x ₀ , y ₀ ),
When the inequality “E (x ₀ , y ₀ ) <TH _LA ” is satisfied, “β (x ₀ , y ₀ ) = 0” is set,
When the inequality “TH _LA ≦ E (x ₀ , y ₀ ) <TH _HA ” holds, “β (x ₀ , y ₀ ) = (E (x ₀ , y ₀ ) −TH _LA ) / (TH _HA −TH _LA ) "
When the inequality “TH _HA ≦ E (x ₀ , y ₀ )” is satisfied, “β (x ₀ , y ₀ ) = 1” is set.
When the target pixel position (x ₀ , y ₀ ) belongs to the non-face area, the relationship RL _B is applied to the target pixel position (x ₀ , y ₀ ). When the relationship RL _B is applied to the target pixel position (x ₀ , y ₀ ),
When the inequality “E (x ₀ , y ₀ ) <TH _LB ” is satisfied, “β (x ₀ , y ₀ ) = 0” is set,
When the inequality “TH _LB ≦ E (x ₀ , y ₀ ) <TH _HB ” is satisfied, “β (x ₀ , y ₀ ) = (E (x ₀ , y ₀ ) −TH _LB ) / (TH _HB −TH _LB) ) "
When the inequality “TH _HB ≦ E (x ₀ , y ₀ )” is satisfied, “β (x ₀ , y ₀ ) = 1” is set.

TH _LA , TH _HA , TH _LB and TH _HB are predetermined thresholds satisfying the inequalities “0 <TH _LA <TH _HA ” and “0 <TH _LB <TH _HB ”. In relations RL _A and RL _B shown in FIGS. 27A and 27B, the edge strength value E (x ₀ , y ₀ ) increases from the threshold value TH _LA toward the threshold value TH _HA or the threshold value TH. _The mixing rate β (x ₀ , y ₀ ) increases linearly from 0 to 1 as it increases from _LB toward the threshold TH _HB , but the mixing rate β (x ₀ , y ₀ ) is increased nonlinearly. It doesn't matter.

As described above, the setting of the mixing rate (x ₀ , y ₀ ) based on the edge intensity value E (x ₀ , y ₀ ) and the specific area information is performed by sequentially setting all pixel positions (x, y) to the target pixel position ( x ₀ , y ₀ ) and then executed one after another, and finally, the mixture ratio (x, y) for all pixel positions (x, y) is determined (second and third operations described later). The same applies to the example). The method of generating the target correction image using the mixing ratio β (x, y) by the second synthesis calculation unit 56d is the same as that of the second synthesis calculation unit 56 described in the first embodiment.

Here, each threshold value is determined so that at least one of the inequalities “TH _LB <TH _LA ” and “TH _HB <TH _HA ”, preferably both, is satisfied. Accordingly, the contribution of the short-exposure image to the target correction image (that is, β (x, y)) is relatively small for the face area, while the contribution (that is, β is applied to the non-face area. (X, y)) becomes relatively large. As a result, in the face area of the target correction image, details are not conspicuous (for example, skin roughness is inconspicuous), and an effect of producing beautiful skin can be obtained.

Among the face region may be only apply the relationship RL _A relative pixel positions having a skin color. That is, only when the target pixel position (x ₀ , y ₀ ) is located in the face area and the color of the pixel at the target pixel position (x ₀ , y ₀ ) is classified as skin color, The relationship RL _A may be applied to x ₀ , y ₀ ), and in other cases, the relationship RL _B may be applied to the target pixel position (x ₀ , y ₀ ). Whether or not the pixel at the target pixel position (x ₀ , y ₀ ) is classified as a skin color can be determined based on the pixel signal at the target pixel position (x ₀ , y ₀ ) of the detection target image. For example, signal value ranges of R, G, and B signals to be classified into skin colors are set in advance, and R, G, and B signals as pixel signals at the target pixel position (x ₀ , y ₀ ) are the signal values. By determining whether or not it falls within the range, it is possible to detect whether or not the pixel at the target pixel position (x ₀ , y ₀ ) is classified into the skin color.

  Even when the specific subject (in other words, the specific object) is other than a human face, the method described above in the first operation example can be applied. If an arbitrary subject whose details are not conspicuous in the target correction image is set as the specific subject, the first operation example is beneficial.

[Second operation example]
A second operation example will be described. In the second operation example, it is assumed that a pet kept by a person is a specific subject (in other words, a specific object). Therefore, the specific area in the second operation example is a pet area where image data of the pet's face and body exists, and the non-specific area in the second operation example is a non-pet area that is an image area other than the pet area.

  It is possible to extract the pet area by the same method as the face area extraction method. When extracting the face area of a person, it is possible to create face dictionary data in advance by learning the image features of the person's face and extract the face area from the detection target image using the face dictionary data. it can. Similarly, for example, pet face dictionary data is created in advance by learning the image characteristics of the face of a pet (for example, a dog), and the pet face region is detected from the detection target image using the pet face dictionary data. Extract. Then, for example, a closed region surrounded by edges including the pet face region may be extracted as a pet region using a known contour extraction process or the like.

The second composition calculation unit 56d determines whether the target pixel position (x ₀ , y ₀ ) belongs to the pet region or the non-pet region based on the specific region information indicating the position and size of the pet region. Depending on whether the pixel position (x ₀ , y ₀ ) belongs to a pet area or a non-pet area, the relationship between the edge intensity value E (x ₀ , y ₀ ) and the mixing rate β (x ₀ , y ₀ ) differs. Make it.

Specifically, when the target pixel position (x ₀ , y ₀ ) belongs to the pet region, the relationship RL _B is applied to the target pixel position (x ₀ , y ₀ ), and the target pixel position (x ₀ , if y ₀₎ belongs to the non-pet area (applying the relationship RL _a relative x _0, y ₀₎ (FIG. 27 (a) the target pixel position reference and (b)). The characteristics of the relations RL _A and RL _B are as described in the description of the first operation example.

  As a result, the contribution of the short-exposure image to the target correction image (that is, β (x, y)) is relatively large for the pet area, while the contribution (that is, β) (X, y)) becomes relatively small. As a result, the sharpness of the pet region is increased in the target correction image, and the pet's hair and the like are clearly depicted.

  For convenience of explanation, it is assumed that the specific subject (in other words, the specific object) in the second operation example is a pet, but the pet may be read as an animal. Furthermore, the method described above in the second operation example can also be applied when the specific subject is other than a pet (animal). If an arbitrary subject to be clearly depicted in the target correction image is set as the specific subject, the second operation example is beneficial.

[Third operation example]
A third operation example will be described. The imaging unit 11 sequentially performs imaging at a predetermined frame period even when the shutter button 17a is not pressed, and the main control unit 13 in FIG. 1 captures one image from the output data of the AFE 12 for one frame period. A video signal of an image (hereinafter also referred to as a frame image) is generated. A group of frame images arranged in time series is called a frame image sequence.

  When the third operation example is used, the main control unit 13 is provided with a tracking processing unit 82 shown in FIG. The tracking processing unit 82 includes a specific area detection unit 81. The tracking processing unit 82 executes a tracking process for tracking the object of interest on the frame image sequence on the frame image sequence based on the image data of the frame image sequence. The target object to be tracked by the tracking process is hereinafter referred to as a tracking target.

  The user can specify a tracking target. For example, a so-called touch panel function is provided in the display unit 15. Then, when the frame image sequence is displayed as a moving image on the display screen of the display unit 27, the user touches the display area on the display screen where the target object is displayed with the finger. An object is set as a tracking target. Alternatively, for example, the user can designate a tracking target by a predetermined operation on the operation unit 17.

  After the tracking target is set, in the tracking process, the position and size of the tracking target in each frame image are sequentially detected based on the image data of the frame image sequence. Actually, an image area where image data representing the tracking target exists is set as a tracking target area in each frame image, and the position (for example, the center or the center of gravity) and the size of the tracking target area are the position and the size of the tracking target. Is detected.

  The tracking process between the first and second frame images can be executed as follows. Here, the first frame image refers to a frame image in which the position and size of the tracking target have already been detected, and the second frame image refers to a frame from which the position and size of the tracking target are to be detected. Refers to the image. The second frame image is a frame image that is usually taken after the first frame image.

  As a tracking processing method based on image data, any method including a publicly known method can be used. For example, the tracking processing unit 82 can perform the tracking process based on the image characteristics of the tracking target. The image feature includes luminance information and color information. More specifically, for example, a tracking frame that is estimated to have the same size as the size of the tracking target area is set in the second frame image, and the image in the tracking frame in the second frame image is set. The similarity between the image feature and the image feature of the image in the tracking target area in the first frame image was executed while sequentially changing the position of the tracking frame in the search area, and the maximum similarity was obtained. It is determined that the center position of the tracking target area in the second frame image exists at the center position of the tracking frame. The search area for the second frame image is set with reference to the position of the tracking target in the first frame image. As a method for detecting the position and size of the tracking target on the frame image, any other method different from the method described above (for example, the method described in Japanese Patent Application Laid-Open No. 2004-94680, or Japanese Patent Application Laid-Open No. 2009-38777). It is also possible to adopt the method described in the publication.

  The short exposure image and the proper exposure image acquired when the shutter button 17a is pressed can also be considered as a part of the frame image sequence to be tracked. Consider a case where, after the shutter button 17a is pressed, the short exposure image 503 and the proper exposure image 504 are continuously photographed in this order as shown in FIG. Further, it is assumed that the frame image 502 is photographed immediately before the short exposure image 503 is photographed, the frame image 501 is photographed immediately before the frame image 502 is photographed, and the frame image 505 is photographed immediately after the proper exposure image 504. As described in the first embodiment, the proper exposure image 504 may be taken before the short exposure image 503.

  When images 501 to 505 as shown in FIG. 29 are taken, the detection target image for the specific area detection unit 81 (see FIGS. 24 and 28) is a target correction image from the short exposure image 503 and the appropriate exposure image 504. In order to generate in real time, for example, a frame image 502 is used. However, the short exposure image 503 or the proper exposure image 504 may be handled as the detection target image. Further, any one of the first to third intermediate generation images based on the short exposure image 503 and / or the proper exposure image 504 may be handled as a detection target image, or is shot after the short exposure image 503 and the proper exposure image 504 are taken. It is also possible to handle a frame image (for example, frame image 505) to be detected as a detection target image.

  The specific area detection unit 81 regards the tracking target area on the detection target image as a specific area and creates specific area information. Therefore, the specific area in the third operation example is a tracking target area in which image data to be tracked exists, and the non-specific area in the third operation example is a non-tracking target area that is an image area other than the tracking target area.

The second composition calculation unit 56d determines whether the target pixel position (x ₀ , y ₀ ) belongs to the tracking target area or the non-tracking target area based on the specific area information indicating the position and size of the tracking target area. Depending on whether the target pixel position (x ₀ , y ₀ ) belongs to the tracking target area or the non-tracking target area, the edge intensity value E (x ₀ , y ₀ ) and the mixing ratio β (x ₀ , y ₀ ) Different relationship with.

Specifically, when the target pixel position (x _0, y ₀₎ belongs to the tracking target area, by applying the relationship RL _B with respect to the target pixel position (x _0, y _0), the target pixel position (x _{When 0} , y ₀ ) belongs to the non-tracking target region, the relationship RL _A is applied to the target pixel position (x ₀ , y ₀ ) (see FIGS. 27A and 27B). The characteristics of the relations RL _A and RL _B are as described in the description of the first operation example.

  Thereby, while the contribution degree of the short exposure image to the target correction image (that is, β (x, y)) is relatively large for the tracking target area, the contribution degree ( That is, β (x, y)) becomes relatively small. As a result, in the target correction image, the sharpness of the tracking target area is increased, and the target object as the tracking target is clearly depicted.

  In addition, even if the third intermediate generation image input to the second synthesis calculation unit 56d is replaced with the first intermediate generation image (see FIG. 5), it is possible to obtain operations and effects peculiar to the fourth embodiment. That is, as shown in FIG. 30, the first intermediate generation image is input instead of the third intermediate generation image to the second synthesis calculation unit 56d, and the first intermediate generation image and the short exposure are input in the second synthesis calculation unit 56d. The target correction image may be generated by combining the images. In this case, the third intermediate generated image in the explanatory text of the fourth embodiment and the third intermediate generated image in the explanatory text of the function of the second composition calculation unit 56 described in the first embodiment are converted into the first intermediate generated image. (P3 (x, y) in the above equation (6) may be replaced with P1 (x, y)). When the first intermediate generation image is input instead of the third intermediate generation image to the second synthesis calculation unit 56d, the first synthesis unit 61 can be omitted from the image correction unit 20 or 20a. Yes (see FIGS. 1 and 5).

<< Fifth Embodiment >>
A fifth embodiment of the present invention will be described. The fifth embodiment is an embodiment obtained by modifying a part of the fourth embodiment. Regarding matters not specifically described in the present embodiment, the description of the fourth embodiment is also applied to the present embodiment.

  In the fourth embodiment, the relationship between the edge intensity value E (x, y) and the mixing ratio (x, y) changes sharply depending on whether the target pixel position belongs to either the specific area or the non-specific area. Therefore, in the fourth embodiment, the texture of the image (image sharpness or the like) may change sharply at the boundary between adjacent partial image areas. The technique according to the fifth embodiment contributes to suppression of such a steep change.

  For specific description, the technique according to the fifth embodiment will be described by taking as an example the case where the specific area is a face area and the non-specific area is a non-face area.

  The specific area detection unit 81 or the second synthesis calculation unit 56d (see FIGS. 24 and 30) classifies the face area indicated by the specific area information into a face center area and a face peripheral area. The outer shape of the face area may be other than a rectangle, but now consider the case where the outer shape of the face area is a rectangle. Then, the face area is classified into a face center area and a face peripheral area as shown in FIG. In FIG. 31 and FIG. 33 described later, the area filled with dots is the face center area, and the area filled with diagonal lines is the face peripheral area. The center of the face area is called the face center. The face center area is an image area formed by pixels whose distance from the face center is a reference distance or less, and includes image data such as the nose of the face. The reference distance is determined based on the size of the face area so that the face center area becomes a part of the face area. Among the face areas, the image area other than the face center area is the face peripheral area.

If the target pixel position (x _0, y ₀₎ belongs to the face central region, the relationship RL _A to the target pixel position (x _0, y ₀₎ is applied, the target pixel position (x _0, y ₀₎ is the non-face If it belongs to the region, the relationship RL _B is applied to the target pixel position (x ₀ , y ₀ ) (see FIGS. 27A and 27B). This is the same as in the fourth embodiment.

However, when the target pixel position (x _0, y ₀₎ belongs to the face peripheral region, the relationship RL _C to the target pixel position (x _0, y ₀₎ is applied. A broken line 470 _C in FIG. 30A represents the relationship RL _C. In FIG. 30B, a broken line 470 _C is shown together with the broken lines 470 _A and 470 _B also shown in FIGS. 27A and 27B.
When the relationship RL _C is applied to the target pixel position (x _0, y _0),
When the inequality “E (x ₀ , y ₀ ) <TH _LC ” is satisfied, “β (x ₀ , y ₀ ) = 0” is set,
When the inequality “TH _LC ≦ E (x ₀ , y ₀ ) <TH _HC ” is satisfied, “β (x ₀ , y ₀ ) = (E (x ₀ , y ₀ ) −TH _LC ) / (TH _HC −TH _LC ) "
When the inequality “TH _HC ≦ E (x ₀ , y ₀ )” is satisfied, “β (x ₀ , y ₀ ) = 1” is set.

When the target pixel position (x ₀ , y ₀ ) belongs to the face peripheral area, the second synthesis calculation unit 56d determines the distance between the target pixel position (x ₀ , y ₀ ) and the face center area and the target pixel position (x ₀ , Y ₀ ) and the non-face area are adjusted to adjust the thresholds TH _LC and TH _HC . However, this adjustment is performed within a range where the inequalities “0 <TH _LC <TH _HC ”, “TH _LB <TH _LC <TH _LA ”, and “TH _HB <TH _HC <TH _HA ” are satisfied.

Consider the case where the pixel position of interest (x ₀ , y ₀ ) is the pixel position 530 as shown in FIG. In this case, assuming a straight line connecting the pixel position 530 and the face center, the pixel position 531 disposed at the intersection of the straight line and the outer periphery of the face central region, and the intersection of the straight line and the outer periphery of the face peripheral region Reference is made to the arranged pixel position 532. The threshold value TH _LC for the pixel position 530 is linearly or gradually from the threshold value TH _LA corresponding to the face center area toward the threshold value TH _LB corresponding to the non-face area as the pixel position 530 moves from the pixel position 531 to the pixel position 532. It is reduced non-linearly. Similarly, the threshold value TH _HC for the pixel position 530 is linear from the threshold value TH _HA corresponding to the face center area toward the threshold value TH _HB corresponding to the non-face area as the pixel position 530 moves from the pixel position 531 to the pixel position 532. Or non-linearly.

  For this reason, if it is assumed that the edge intensity values E (x, y) for all pixel positions are the same, the contribution of the short-exposure image to the target correction image (that is, β (x, y)) is the center of the face. It is the smallest in the area, the second smallest in the face peripheral area, and the largest in the non-face area. The contribution is expected to change smoothly at the boundary between the face area and the non-face area. As a result, a sharp change in the texture of the image (image sharpness or the like) at the boundary between the face area and the non-face area is suppressed.

  Although the case where the specific area is the face area has been described as an example, the same applies to the case where the specific area is the pet area or the tracking target area. However, when the specific area is the pet area or the tracking target area, the contribution of the short-exposure image to the target correction image (that is, β (x, y)) is the largest in the specific central area and in the specific peripheral area. Second largest and smallest in non-specific area (assuming that edge intensity values E (x, y) for all pixel positions are the same). The specific center area is an image area formed by pixels whose distance from the center of the specific area is equal to or less than the reference distance (the reference distance here is the specific area so that the specific central area becomes a part of the specific area). Determined based on size). Among the specific areas, the image area other than the specific central area is the specific peripheral area.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As modifications or annotations of the above-described embodiment, notes 1 to 3 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
Although the example in which the image processing device including the image correction unit (20 or 20a) is provided in the imaging device 1 has been described above, the image correction unit (20 or 20a) is connected to an image processing device (not configured) outside the imaging device 1. It may be provided in the figure). In this case, image data (including image data of a short exposure image and a proper exposure image) obtained by photographing with the imaging device 1 with respect to an image correction unit (20 or 20a) included in the external image processing device. By supplying, the image correction unit (20 or 20a) included in the external image processing apparatus generates image data of the target correction image.

[Note 2]
The imaging device 1 can be realized by hardware or a combination of hardware and software. In particular, all or part of the processing executed in the image correction unit (20 or 20a) can be realized by hardware, software, or a combination of hardware and software. When the imaging apparatus 1 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part.

[Note 3]
For example, it can be considered as follows. In the second and third embodiments, the region edge amount calculation unit 71 (see FIG. 17 or 20) functions as a feature amount extraction unit that extracts an image feature amount for each partial image region. It can also be considered that the edge strength value calculation unit 55 is also included in the constituent elements of the feature amount extraction unit.

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 12 AFE
13 Main control unit 20, 20a Image correction unit 51 Position adjustment unit 52 Noise reduction unit 53 Difference value calculation unit 54 First synthesis calculation unit 55 Edge intensity value calculation unit 56, 56b, 56d Second synthesis calculation unit 61 First synthesis unit 62, 62b, 62d Second composition unit 71 Region edge amount calculation unit 81 Specific region detection unit 82 Tracking processing unit

Claims (6)

The first image obtained by photographing is set as a first composition target image,
A second image obtained by photographing with an exposure time longer than the exposure time of the first image, or a composite image of the third image obtained by reducing noise in the first image and the second image An image processing apparatus for generating an output image by synthesizing the first and second synthesis target images.
The entire image area of the first synthesis target image or the third image is divided into a plurality of partial image areas, and the image feature amount of the partial image area is determined for each partial image area based on the image data of the partial image area. A feature extraction unit for extracting,
An image processing apparatus, wherein a composition ratio when the first and second synthesis target images are synthesized is adjusted for each partial image area based on an image feature amount of the partial image area. The image processing apparatus according to claim 1, wherein the feature amount extraction unit extracts, as the image feature amount, a magnitude of edge strength in the partial image region for each partial image region. For the partial image region having a relatively large edge strength as the image feature amount, the image feature amount is set so that the contribution degree of the first synthesis target image to the output image is relatively large. For the partial image region having a relatively small edge strength, the composition ratio is adjusted so that the contribution of the first composition target image to the output image is relatively small. The image processing apparatus according to claim 2. The first image obtained by photographing is set as a first composition target image,
A second image obtained by photographing with an exposure time longer than the exposure time of the first image, or a composite image of the third image obtained by reducing noise in the first image and the second image An image processing apparatus for generating an output image by synthesizing the first and second synthesis target images.
An image of a specific object from among the detection target images based on image data of the detection target image that is an image obtained by any one of the first to fourth images or before or after the first image is captured. A specific object detection unit for detecting the position and size of an image area where data exists,
Each whole image area of the first and second synthesis target images is classified into a specific area according to the detected position and size and other non-specific areas, and the first and second synthesis target images are classified. An image processing apparatus, wherein a composition ratio at the time of composition is different between the specific area and the non-specific area. The image area in which the image data of the specific object exists is a face area in which image data of a human face exists,
For the specific area corresponding to the face area, the contribution of the first synthesis target image to the output image is relatively small, and for the non-specific area, the output image The image processing apparatus according to claim 4, wherein the composition ratio is adjusted so that the contribution degree of the first composition target image is relatively large. An imaging unit for acquiring an image by shooting;
An image processing apparatus according to any one of claims 1 to 5,
The imaging apparatus, wherein the first and second images are acquired by the imaging unit.