JP2010081002A - Image pickup apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a correction image having a desired depth of field from a picked-up image after image pickup. <P>SOLUTION: A subject is picked up via an optical system 10 having an on-axis color aberration, thereby forming G, R, B signals (G0, R0, B0). The subject distance at each of positions of an original image are estimated from the magnitude of high-frequency components of the G, R, B signals of the original image. Meanwhile, high-frequency components of color signals corresponding to the maximum magnitude, of the sizes of the high-frequency components of G, R, B signals of the original image, are added to the other two color signals, thereby generating G, R, B signals (G1, R1, B1) of an intermediately generated image having high resolution for a wide subject distance range. Thereafter, the resolutions of the G, R, B signals of the intermediately generated image for a subject distance other than the subject distance of a main subject are reduced on the basis of the estimated subject distance, thereby generating G, R, B signals (G2, R2, B2) of a correction signal having shallow depth of field. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an imaging apparatus such as a digital still camera or a digital video camera, and more particularly to a technique for controlling the depth of field.

  A single lens has axial chromatic aberration. An optical system used for an imaging apparatus is usually provided with a lens group for correcting axial chromatic aberration, and as a whole, the axial chromatic aberration is small enough to be ignored. FIG. 23 shows the relationship of incident light with respect to a conventional general lens 910L, the image sensor 911, and the image sensor 911. The lens 910L is obtained by regarding the lens group included in the optical system as one lens.

  The light 901 directed from the point light source 900 to the lens 910L is separated into blue light, green light, and red light by the lens 910L, but the blue light, green light, and red light have the same imaging point due to the axial chromatic aberration correction function. To form an image. By disposing the image sensor 911 at this imaging point, an image of the focused point light source 900 is formed on the imaging surface of the image sensor 911.

  FIG. 24 shows the relationship between the subject distance and the resolution of the image signal obtained from the image sensor 911 when the distance between the lens 910L and the image sensor 911 is fixed. Here, the resolution does not represent the number of pixels of the image but refers to the maximum spatial frequency that can be represented on the image. In other words, the resolution is a scale that represents how much details can be reproduced on the image, and is also called resolution.

  In FIG. 24, curves 920B, 920G, and 920R represent the subject distance dependency of the resolution in the B, G, and R signals of the captured image, respectively. Each of the resolutions of the B, G, and R signals is maximized when the subject distance is at a specific distance, and decreases as the subject distance decreases or increases from the specific distance. That is, an image in which a subject (for example, a person) whose subject distance matches the specific distance is in focus and a subject (for example, a background) whose subject distance is far from the specific distance is blurred as a captured image.

  Note that FIG. 24 assumes that a single-plate image sensor employing a Bayer array is used as the image sensor 910. In a Bayer array image sensor, the density of light receiving pixels that receive the green component of light is greater than the density of light receiving pixels that receive the blue or red component of light. Therefore, the resolution of the G signal is higher than those of the B and R signals.

  920Y in FIG. 24 represents the subject distance dependency of the resolution of the Y signal (luminance signal) generated from the B, G, and R signals of the captured image. At each subject distance, the resolution of the Y signal is about the same as or higher than the resolution of the G signal.

Y signal (or the B, G and all of the R signal) is referred to as a focusing range of the range of the subject distance resolution is one more than a predetermined standard resolution RS _O with the or depth of field.

  A first subject (for example, a person as a main subject) and a second subject (for example, a background) are included in the subject of the imaging device, and the subject distances of the first and second subjects are different from each other. Assume that there is. When focusing on the first subject, the second subject is blurred in an image with a shallow depth of field with only a slight difference in distance between the first and second subject distances. On the other hand, when the depth of field is deep, an image in which both the first and second subjects are in focus can be obtained even if the distance difference is significantly different.

  If a large single-lens reflex camera is used, an image with a shallow depth of field can be easily acquired. This is because a large single-lens reflex camera can employ a large lens, so that the opening of the lens can be made large, and as a result, the depth of field can be set relatively large. On the other hand, in a small digital camera, it is difficult to capture an image with a shallow depth of field as with a single-lens reflex camera. This is because a lens and an image sensor used in a small digital camera are small, and inevitably, the opening of the lens is smaller and the depth of field is deeper than that of a large single-lens reflex camera.

  An image with deep depth of field is not necessarily inferior to an image with shallow depth of field, but sometimes it is shallow to obtain a highly artistic image that reproduces natural blur according to the subject distance. Depth of field is required. Also, if the depth of field of the image can be arbitrarily adjusted after shooting, the convenience is high.

  Patent Document 1 listed below discloses a technique for increasing the speed of focus control by utilizing axial chromatic aberration of a lens. Also disclosed is a technique for correcting an image blur caused by axial chromatic aberration using an inverse function of a point spread function. However, this technique does not contribute to the adjustment of the depth of field.

JP-A-6-138362

  SUMMARY An advantage of some aspects of the invention is that it provides an imaging apparatus that contributes to acquisition of an image having a desired depth of field.

  An image pickup apparatus according to the present invention extracts, for each color signal, an image pickup element that picks up a subject via an optical system having axial chromatic aberration, and a predetermined high-frequency component included in a plurality of color signals representing a picked-up image. And subject distance estimating means for estimating the subject distance in each partial image area in the photographed image based on the magnitude of the high frequency component for each color signal, and image processing means for generating a correction image from the photographed image. And setting means for setting the depth of field for the corrected image by specifying a depth of the depth of field for the corrected image and specifying a specific subject distance to be positioned within the depth of field The image processing means performs image processing on the captured image according to the estimated subject distance by the subject distance estimation means and the set depth of field by the setting means, And generates the corrected image in accordance with the serial setting depth of field.

  The axial chromatic aberration is positively used, and a corrected image according to the set depth of field is generated after shooting based on the estimated subject distance in each partial image region of the shot image. Thereby, it is possible to generate a corrected image having a desired depth of field after shooting. In addition, since focus control after shooting is possible, shooting failures due to focusing errors can be eliminated.

  Specifically, for example, based on the set depth of field and the estimated subject distance in each partial image region in the captured image, the entire image region of each of the captured image and the corrected image is a corresponding estimated subject. An estimated subject distance corresponding to a first partial image area whose distance falls within the set depth of field is classified into a second partial image area located outside the set depth of field, and the image processing means The corrected image is generated from the captured image so that an image in the second partial image region is blurred in the image compared with an image in the first partial image region.

  More specifically, for example, the color signals of the plurality of colors are composed of first, second and third color signals different from each other, and the image processing means includes the high frequency components of the first to third color signals. The first to third color signals after addition of the high frequency band are generated by adding the high frequency component of the color signal corresponding to the maximum size to the other two color signals. First to first representing the corrected image from the first to third post-high color signal by applying predetermined filtering to the correction means and the first to third post-high color signal. Second correction means for generating a third correction color signal, wherein the second correction means is based on the set depth of field and the estimated subject distance in each partial image area in the captured image. In the corrected image, the image in the second partial image area is compared with the image in the first partial image area. Blurred performing said filtering as.

  For example, the filtering by the second correction unit is filtering by a low-pass filter that can variably set a cutoff frequency, and an image in the second partial image region in the corrected image is set by variably setting the cutoff frequency. It is blurred compared with the image in the first partial image area.

  Alternatively, for example, a point spread function derived from the longitudinal chromatic aberration for each color signal in each partial image region in the captured image is determined from the estimated subject distance in each partial image region in the captured image, and the image processing means Is based on the point spread function, the set depth of field, and the estimated subject distance in each partial image area in the captured image, the image in the second partial image area in the corrected image is the first partial image area The corrected image is generated from the photographed image so as to be blurred compared with the image inside.

  And, for example, the image processing means sets a blur amount in each partial image region in the captured image derived from the axial chromatic aberration estimated from the point spread function in each partial image region in the corrected image. The correction image blurred according to the set blur amount is generated by performing filtering for correcting the blur amount on each color signal of the photographed image, and the second partial image is generated in the correction image. The set blur amount is based on the set depth of field and the estimated subject distance in each partial image area in the captured image so that the image in the area is blurred compared with the image in the first partial image area. Determined.

  In addition, for example, the imaging apparatus further includes display means for displaying the captured image, and after capturing the captured image, the captured image or an image based on the captured image is displayed on a display screen of the display means, The setting means receives an operation for designating a specific subject on the display screen in a state where the display is performed, and designates the specific subject distance from an estimated subject distance corresponding to the specific subject, while performing a predetermined operation. By accepting, the depth of field for the corrected image is designated.

  According to the present invention, it is possible to provide an imaging apparatus that contributes to acquisition of an image having a desired depth of field.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle. The first to third embodiments will be described later. First, matters that are common to each embodiment or items that are referred to in each embodiment will be described.

  With reference to FIG. 1, FIG. 2 (a)-(c), and FIG. 3, the characteristic of the lens 10L utilized with the imaging device which concerns on embodiment of this invention is demonstrated. The lens 10L has a relatively large predetermined axial chromatic aberration. Accordingly, as shown in FIG. 1, the light 301 from the point light source 300 toward the lens 10L is separated into blue light 301B, green light 301G, and red light 301R by the lens 10L, and the blue light 301B, green light 301G, and red light 301R are The images are formed on different image forming points 302B, 302G and 302R. Blue light 301B, green light 301G, and red light 301R are the blue, green, and red components of light 301, respectively.

  In FIG. 2A and the like, reference numeral 11 represents an imaging element used in the imaging apparatus. The imaging device 11 is a solid-state imaging device such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor. The image sensor 11 is a so-called single-plate image sensor, and a red filter that transmits only a red component of light and only a green component of light are provided in front of each light receiving pixel in one image sensor as the image sensor 11. Either a green filter that transmits light or a blue filter that transmits only the blue component of light is disposed. The arrangement of the red filter, the green filter, and the blue filter is a Bayer arrangement.

The distances from the center of the lens 10L to the image forming points 302B, 302G, and 302R are represented by XB, XG, and XR, respectively, as shown in FIG. Then, the inequality “XB <XG <XR” is established due to the longitudinal chromatic aberration of the lens 10L. Further, representing the distance to the imaging element 11 in X _IS from the center of the lens 10L. In FIG. 3, “XB <XG <XR <X _IS ” is satisfied, but the distances XB, XG, and XR are changed by changing the distance 310 (see FIG. 1) between the light source 300 and the center of the lens 10L. The magnitude relationship with the distance _XIS changes.

FIGS. 2A to 2C are diagrams showing how the positions of the imaging points 302B, 302G, and 302R change as the distance 310, which should be called the subject distance, changes. FIG. 2A shows the positional relationship between the imaging points 302B, 302G, and 302R and the image sensor 11 when the distance 310 is a relatively small distance and “XB = X _IS ” is satisfied. FIG. 2B shows the positional relationship between the imaging points 302B, 302G, and 302R and the image sensor 11 when “XG = X _IS ” is satisfied when the distance 310 increases from the state of FIG. Show. FIG. 2C shows the positional relationship between the imaging points 302B, 302G and 302R and the image sensor 11 when “XR = X _IS ” is satisfied when the distance 310 further increases from the state of FIG. Is shown.

Distance X _IS distance XB, the position of the lens 10L at the time that matches XG, and XB is a blue light 301B, the green light 301G, focus position of the lens 10L for red light 301R. Therefore, when “XB = X _IS ”, “XG = X _IS ”, and “XR = X _IS ” are satisfied, the blue light 301B, the green light 301G, and the red light 301R are completely focused, respectively. An image in the state is obtained from the image sensor 11. However, the image of the green light 301G and the red light 301R is blurred in the image in a state where the blue light 301B is completely focused. The same applies to an image in a state where the green light 301G and the red light 301R are completely focused. On the image obtained from the image sensor 11 when none of “XB = X _IS ”, “XG = X _IS ”, and “XR = X _IS ” is established, the blue light 301B, the green light 301G, and All images of red light 301R are blurred. The amount of blur (degree of blur) in an image is called a blur amount.

The radii of the blue light 301B, green light 301G, and red light 301R images formed on the imaging surface of the image sensor 11 are represented by YB, YG, and YR, respectively. The blur amounts of the blue light 301B, green light 301G, and red light 301R images are proportional to the radii YB, YG, and YR, respectively. That is, the larger the radius YB, the larger the blur amount of the blue light 301B image (the same applies to the green light 301G and the red light 301R). The characteristics of the lens 10L including the characteristics of axial chromatic aberration are known in advance at the design stage of the imaging apparatus, and the imaging apparatus can naturally recognize the distance _XIS . Thus, the imaging device, if the distance 310 is known, the radius YB using the characteristics of the lens 10L and the distance X _IS, it is possible to estimate the YG and YR (i.e., on the image obtained from the imaging device 11, for each color component The amount of blur can be estimated). Also, if the distance 310 is known, the point spread function of the blue light 301B, green light 301G, and red light 301R is determined, so that the blur of those images can be reduced using the inverse function of the point spread function. It is also possible to remove it.

Although it is possible to change the distance _XIS , in the following description, for the sake of simplicity, it is assumed that the distance _XIS is fixed to a constant distance unless otherwise specified.

  A distance in real space between a certain subject and the imaging device (strictly speaking, a distance in real space between a certain subject and the center of the lens 10L) is called a subject distance. If the point light source 300 is considered to be a focused subject, the distance 310 is a subject distance for the focused subject. Also,

  FIG. 4 shows the relationship between the subject distance and the resolution of the B, G, and R signals of the original image obtained from the image sensor 11. Here, the original image refers to an image obtained by performing demosaicing processing on the RAW data obtained from the image sensor 11. In each embodiment described later, the original image generated by the demosaicing processing unit 14. (Refer to FIG. 6 etc.). The B, G, and R signals are a signal representing a blue component on the image corresponding to the blue light 301B, a signal representing a green component on the image corresponding to the green light 301G, and an image corresponding to the red light 301R, respectively. Refers to the signal representing the red component.

  The resolution in this specification does not represent the number of pixels of an image, but indicates the maximum spatial frequency that can be represented on the image. In other words, the resolution in the present specification is a scale that represents how fine details can be reproduced on an image, and is also referred to as resolving power.

  In FIG. 4, curves 320B, 320G, and 320R represent the subject distance dependency of the resolution in the B, G, and R signals of the original image, respectively. In the graph showing the relationship between the resolution and the subject distance in FIG. 4 (and FIG. 5 and the like to be described later), the horizontal axis and the vertical axis represent the subject distance and the resolution, respectively, and correspond on the horizontal axis from left to right. The subject distance increases, and the corresponding resolution increases as the vertical axis is moved from the bottom to the top.

The subject distances DD _B , DD _G and DD _R are respectively “XB = X _IS ” corresponding to FIG. 2A, “XG = X _IS ” corresponding to FIG. 2B, and FIG. This is the subject distance when the corresponding “XR = X _IS ” holds. Therefore, “DD _B <DD _G <DD _R ” is established.

As shown in curve 320B, the resolution of the original image of the B signal, becomes maximum when the object distance is the distance DD _B, decreases as the object distance decreases or increases the distance DD _B as a starting point. Similarly, as shown in curve 320G, resolution G signal of the original image becomes maximum when the object distance is the distance DD _G, decreases as the object distance decreases or increases the distance DD _G as a starting point. Similarly, as shown in curve 320R, the resolution R signal of the original image becomes maximum when the object distance is the distance DD _R, decreases as the object distance decreases or increases the distance DD _R as a starting point.

  As can be seen from the definition of resolution described above, the resolution of the B signal of the original image represents the maximum spatial frequency of the B signal on the original image (the same applies to the G and R signals). If the resolution of a certain signal is relatively high, the signal contains a relatively large amount of high frequency components. Of the frequency components included in a signal, a frequency component equal to or higher than a predetermined frequency is called a high frequency component (hereinafter abbreviated as a high frequency component), and a frequency component less than a predetermined frequency is referred to as a low frequency component (hereinafter referred to as a high frequency component) Are abbreviated as low-frequency components).

Thus, for the object distance is relatively close to the object (e.g., object subject distance of DD _B), high-frequency frequency components included in the B signal of the original signal, the object distance is relatively far subject (e.g. , for the object distance DD _R of the subject), high-frequency frequency components included in the R signal of the original signal, for a subject of moderate subject distance (e.g., the subject object distance is DD _G) The high frequency component is included in the G signal of the original signal.

  If these high-frequency components are complemented, an image with a wide focusing range, that is, a deep depth of field can be generated. FIG. 5 is a diagram in which a curve 320Y representing the resolution of the Y signal generated by complementing the high-frequency components of the original B, G, and R signals is added to FIG. After performing such complementation, if the subject (for example, the background) whose subject distance is different from the subject (for example, the person as the main subject) desired to be focused is blurred, the subject desired to be focused is focused. A matched image with a narrow focus range (that is, an image with a shallow depth of field) can be generated.

Incidentally, called as described above, Y signal (or the B, G and R signals of all) and the focusing range the scope of the subject distance resolution is a certain or predetermined standard resolution RS _O with the or depth of field .

  In the following first to third embodiments, a method for generating an image with a shallow depth of field will be described more specifically.

<< First Embodiment >>
First, a first embodiment of the present invention will be described. FIG. 1 is an overall block diagram of an imaging apparatus 1 according to the first embodiment of the present invention. The imaging device 1 (and an imaging device 1a described later) is a digital still camera that can capture and record still images, or a digital video camera that can capture and record still images and moving images. Note that shooting and imaging are synonymous.

  The imaging device 1 is provided with each part referred by the codes | symbols 10-24. The optical system 10 is formed by a lens group including a zoom lens for performing optical zoom and a focus lens for adjusting a focal position, and a diaphragm for adjusting the amount of light incident on the image sensor 11, and has a desired angle of view. An image having a desired brightness is formed on the imaging surface of the image sensor 11. The lens 10L described above is obtained by capturing the optical system 10 as a single lens. Therefore, the optical system 10 has the same axial chromatic aberration as that of the lens 10L.

  The image sensor 11 photoelectrically converts an optical image (subject image) representing a subject incident via the optical system 10 and outputs an electrical signal obtained by the photoelectric conversion to the AFE 12. More specifically, the image pickup device 11 includes a plurality of light receiving pixels arranged two-dimensionally in a matrix, and each light receiving pixel stores a signal charge having a charge amount corresponding to the length of the exposure time in each photographing. . An analog signal from each light receiving pixel having a magnitude proportional to the amount of stored signal charge is sequentially output to the AFE 12 in accordance with a drive pulse generated in the imaging device 1.

  An AFE (Analog Front End) 12 amplifies the analog signal output from the image sensor 11, converts the amplified analog signal into a digital signal, and outputs the digital signal. Further, the amplification degree in the signal amplification of the AFE 12 is adjusted in conjunction with the adjustment of the aperture value of the optical system 10 so that the output signal level of the AFE 12 is optimized. The output signal of the AFE 12 is also called RAW data. RAW data can be temporarily stored in a DRAM (Dynamic Random Access Memory) 13. Further, the DRAM 13 can temporarily store not only RAW data but also various data generated in the imaging apparatus 1.

  As described above, since the image sensor 11 is a single-plate image sensor that employs a Bayer array, red color signals are arranged in a mosaic pattern according to the Bayer array in a two-dimensional image represented by RAW data. (Same for green and blue).

  The demosaicing processing unit 14 generates RGB format image data by executing a known demosaicing process on the RAW data. Hereinafter, the two-dimensional image represented by the image data generated by the demosaicing processing unit 14 is referred to as an original image. All R, G, and B signals are assigned to each pixel that forms the original image. The R, G, and B signals for a certain pixel are color signals that represent the red, green, and blue intensities of that pixel, respectively. The R, G, and B signals of the original image are represented by R0, G0, and B0, respectively.

  The high-frequency component extraction / distance estimation unit 15 (hereinafter abbreviated as extraction / estimation unit 15) extracts the high-frequency components of the color signals R0, G0, and B0, and each of the original images through the extraction. The subject distance at the position is estimated, and subject distance information DIST representing the estimated subject distance is generated. Further, information corresponding to the extraction result of the high frequency component is output to the depth of field expansion processing unit 16.

  The depth of field expansion processing unit 16 (hereinafter abbreviated as the expansion processing unit 16) is based on information from the extraction / estimation unit 15, and the depth of field of the original image represented by the color signals R0, G0, and B0. Is expanded (ie, the depth of field is increased) to generate an intermediate generated image. The R, G, and B signals of the intermediate generated image are represented by R1, G1, and B1, respectively. The enlargement processing unit 16 corrects the color signals R0, G0, and B0 representing the original image to generate color signals R1, G1, and B1 representing the intermediate generation image.

  The depth-of-field control unit 17 adjusts the depth of field of the intermediate generation image based on the subject distance information DIST and the depth of field setting information that specifies the depth of field for the corrected image. A corrected image with a shallow depth is generated. The depth of field setting information determines the depth of field with respect to the corrected image and which subject is to be focused. The depth of field setting information is created by the CPU 23 based on a user instruction or the like. The R, G, and B signals of the corrected image are represented by R2, G2, and B2, respectively. The depth-of-field control unit 17 corrects the color signals R1, G1, and B1 representing the intermediate generated image to generate color signals R2, G2, and B2 representing the corrected image.

  The R, G, and B signals of the corrected image are given to the camera signal processing unit 18. It is also possible to supply the R, G, and B signals of the original image or the intermediate generated image to the camera signal processing unit 18.

  The camera signal processing unit 18 converts the R, G, and B signals of the original image, the intermediate generated image, or the corrected image into the luminance signal Y and the color difference signals U and V, and performs the processing on the original image, intermediate generated image, or corrected image. Necessary image quality adjustment such as edge enhancement processing is performed, and a video signal (that is, YUV signal) of the original image, intermediate generation image or corrected image after image quality adjustment is output. A video signal output from the camera signal processing unit 18 is supplied to an LCD 19 that is a liquid crystal display or an external display device (not shown) provided outside the imaging device 1, thereby allowing the display screen of the LCD 19 or an external display device to be displayed. The original image, the intermediate generation image, or the corrected image (strictly, after the image quality adjustment) is displayed on the display screen.

  In the imaging apparatus 1, so-called touch panel operation is possible. The user can perform an operation (that is, a touch panel operation) on the imaging device 1 by touching the display screen of the LCD 19. The touch panel control unit 20 receives a touch panel operation by detecting a pressure applied on the display screen of the LCD 19.

  The compression / decompression processing unit 21 compresses the video signal output from the camera signal processing unit 18 using a predetermined compression method, thereby compressing the compressed video signal (that is, the compressed image data of the original image, the intermediate generation image, or the corrected image). ) Is generated. Further, by decompressing the compressed video signal, the video signal before compression can be restored. The compressed video signal can be recorded on a recording medium 22 which is a nonvolatile memory such as an SD (Secure Digital) memory card. Also, RAW data can be recorded on the recording medium 22. A CPU (Central Processing Unit) 23 comprehensively controls the operation of each part forming the imaging device 1. The operation unit 24 receives various operations on the imaging device 1. The operation content for the operation unit 24 is transmitted to the CPU 23.

[Principle of corrected image generation: Control principle of depth of field]
With reference to FIGS. 7A to 7D, the principle of a method for generating a corrected image from an original image will be described. In FIG. 7A, curves 400B, 400G, and 400R respectively indicate the object distance dependency of the resolution in the B, G, and R signals of the original image, that is, the object distance dependency of the resolution in the color signals B0, G0, and R0. Represents. Curves 400B, 400G, and 400R are the same as curves 320B, 320G, and 320R in FIG. 4, respectively. The distances DD _B , DD _G and DD _R in FIGS. 7A and 7B are the same as those shown in FIG.

  The subject distances in which the resolution of the color signals B0, G0, and R0 increases due to axial chromatic aberration are different from each other. For a subject with a relatively short subject distance, the color signal B0 includes a high frequency component, and for a subject with a relatively long subject distance, the color signal R0 includes a high frequency component, and the subject distance is medium. For a subject, the color signal G0 includes a high frequency component.

  After obtaining such color signals B0, G0, and R0, a signal having the largest high frequency component is specified among the color signals B0, G0, and R0, and the high frequency component of the specified color signal is determined as the other two By adding to the color signal, the color signals B1, G1, and R1 of the intermediate generation image are generated. Since the magnitude of the high frequency component of each color signal changes as the subject distance changes, this generation processing is executed separately for different first, second, third,... Subject distances. . Although subjects having various subject distances appear in the entire image area of the original image, the subject distance of each subject is estimated by the extraction / estimation unit 15 in FIG.

In FIG. 7B, a curve 410 represents the subject distance dependency of the resolution in the B, G, and R signals of the intermediate generated image, that is, the subject distance dependency of the resolution in the color signals B1, G1, and R1. A curve 410 indicates the maximum resolution of the color signals B0, G0, and R0 at the first subject distance, the maximum resolution of the color signals B0, G0, and R0 at the second subject distance, and the color signal at the third subject distance. A curve is formed by connecting the maximum resolution values of B0, G0, and R0,... The focus range (depth of field) of the intermediate generation image is larger than that of the original image, and the distances DD _B , DD _G and DD _R are included in the focus range (depth of field) of the intermediate generation image. The

  While generating the intermediate generation image, a depth-of-field curve 420 as shown in FIG. 7C is set based on a user instruction or the like. The depth-of-field control unit 17 in FIG. 6 adjusts the intermediate generated image so that the curve representing the subject distance dependency of the resolution in the B, G, and R signals of the corrected image is substantially the same as the depth-of-field curve 420. The B, G, and R signals are corrected. A solid line curve 430 in FIG. 7D shows the subject distance dependency of the resolution in the B, G, and R signals of the corrected image obtained by this correction, that is, the subject distance dependency of the resolution in the color signals B2, G2, and R2. Represents. If the depth-of-field curve 420 is appropriately set, a corrected image with a narrow focus range (a corrected image with a shallow depth of field) can be generated. That is, it is possible to generate a corrected image in which only a subject at a desired subject distance is in focus and a subject at another subject distance is blurred.

The principle of the method for generating the color signals B1, G1 and R1 from the color signals B0, G0 and R0 will be supplementarily described. The color signals B0, G0, R0, B1, G1, and R1 are regarded as functions of the subject distance D, and these are respectively B0 (D), G0 (D), R0 (D), B1 (D), and G1 (D). And R1 (D). The color signal G0 (D) can be separated into a high frequency component Gh (D) and a low frequency component GL (D). Similarly, the color signal B0 (D) can be separated into a high frequency component Bh (D) and a low frequency component BL (D), and the color signal R0 (D) is divided into a high frequency component Rh (D) and a low frequency component. It can be separated into RL (D). That is,
G0 (D) = Gh (D) + GL (D),
B0 (D) = Bh (D) + BL (D),
R0 (D) = Rh (D) + RL (D),
Holds.

If it is assumed that the optical system 10 has no axial chromatic aberration, the following equation (1) and the following equation (2) are usually satisfied from the property of the image that the color change is small locally. This is true for any subject distance. The subject in the real space has various color components. However, when the color component of the subject is viewed locally, the brightness changes in a minute region, but the color hardly changes. For example, when the color component of a green leaf is scanned in a certain direction, the color (hue etc.) hardly changes although the luminance changes depending on the leaf pattern. Therefore, if it is assumed that the optical system 10 has no axial chromatic aberration, equations (1) and (2) often hold.
Gh (D) / Bh (D) = GL (D) / BL (D) (1)
Gh (D) / GL (D) = Bh (D) / BL (D) = Rh (D) / RL (D) (2)

On the other hand, since the axial chromatic aberration actually exists in the optical system 10, the high frequency components of the color signals B0 (D), G0 (D), and R0 (D) are different from each other for an arbitrary subject distance. In other words, the high frequency components of the other two color signals can be supplemented by using one color signal having a large high frequency component for a certain subject distance. For example, now, as shown in FIG. 8, the resolution of a color signal G0 (D) is the object distance to take a maximum value and a D _1, and D ₂ large certain object distance than D _1. Further, as shown in FIG. 9, the partial image area in the original image where the image data of the subject SUB _{1 with} the subject distance D ₁ exists is represented by reference numeral 441, and the image data of the subject SUB _{2 with} the subject distance D ₂ exists. A partial image area in the original image is represented by reference numeral 442. A part of an entire image area of an arbitrary image such as an original image, an intermediate generation image, or a corrected image is referred to as a partial image area. Images of the subjects SUB ₁ and SUB ₂ appear in the partial image areas 441 and 442, respectively.

The G signal in the partial image area 441, that is, G0 (D ₁ ) (= Gh (D ₁ ) + GL (D ₁ )) contains a lot of high frequency components. B and R signals in the image area 441, i.e., contained in _{B0 (D 1) (= Bh} (D 1) + BL (D 1)) and _{R0 (D 1) (= Rh} (D 1) + RL (D 1)) There are few high frequency components. The high frequency components of the B and R signals in the partial image area 441 are generated using the high frequency components of the G signal in the partial image area 441. When representing the high frequency component of the B and R signals in the generated partial image region 441 at each Bh '(D ₁₎ and _{Rh' (D 1), Bh} '(D 1) and Rh' (D ₁₎ Is obtained by the following equations (3) and (4).
Bh ′ (D ₁ ) = BL (D ₁ ) × Gh (D ₁ ) / GL (D ₁ ) (3)
Rh ′ (D ₁ ) = RL (D ₁ ) × Gh (D ₁ ) / GL (D ₁ ) (4)

If there is no axial chromatic aberration in the optical system 10, “Bh (D ₁ ) = BL (D ₁ ) × Gh (D ₁ ) / GL (D ₁ )” and “Rh” from the above formulas (1) and (2). (D ₁ ) = RL (D ₁ ) × Gh (D ₁ ) / GL (D ₁ ) ”is considered to hold, but due to the longitudinal chromatic aberration of the optical system 10 that actually exists, the subject distance D ₁ The high frequency components Bh (D ₁ ) and Rh (D ₁ ) are missing from the B and R signals. This missing portion is generated by the above formulas (3) and (4).

In practice, B0 order (D ₁₎ high frequency component Bh of (D ₁₎ and R0 (D ₁₎ of the high frequency component Rh (D ₁₎ is _{small, B0 (D 1) ≒ BL} (D 1) R0 (D ₁ ) ≈RL (D ₁ ). Thus, with respect to the object distance D _1, the formula (5) and in accordance with _{(6), B0 (D 1} ), R0 (D 1) and _{Gh (D 1) / GL (} D 1) using B1 (D ₁₎ And R1 (D ₁ ) are obtained to generate signals B1 and R1 including high frequency components. G1 (D ₁ ) is G0 (D ₁ ) itself as shown in the equation (7).
B1 (D ₁ ) = BL (D ₁ ) + Bh ′ (D ₁ )
_{≒ B0 (D 1) + B0} (D 1) × Gh (D 1) / GL (D 1) ... (5)
R1 (D ₁ ) = RL (D ₁ ) + Rh ′ (D ₁ )
≒ R0 (D ₁ ) + R0 (D ₁ ) × Gh (D ₁ ) / GL (D ₁ ) (6)
G1 (D ₁ ) = G0 (D ₁ ) (7)

  The method for generating the signals B1, G1, and R1 has been described by focusing on the partial image region 441 in which the G signal includes many high-frequency components. However, the partial image region in which the B or R signal includes many high-frequency components. The same generation process is performed for.

[High-frequency component extraction, distance estimation, depth of field expansion]
A detailed configuration example of a part responsible for processing based on the above principle will be described. FIG. 10 is an internal block diagram of the extraction / estimation unit 15 and the enlargement processing unit 16 shown in FIG. Color signals G0, R0, and B0 of the original image are input to the extraction / estimation unit 15 and the enlargement processing unit 16. The extraction / estimation unit 15 includes HPFs (High Pass Filters) 51G, 51R and 51B, LPFs (Low Pass Filters) 52G, 52R and 52B, a maximum value detection unit 53, a subject distance estimation unit 54, a selection unit 55, and a calculation unit 56. . The enlargement processing unit 16 includes selection units 61G, 61R, and 61B and a calculation unit 62.

  An arbitrary two-dimensional image such as an original image, an intermediate generation image, or a corrected image is formed by arranging a plurality of pixels in a matrix in the horizontal and vertical directions, as shown in FIGS. Thus, the position of the pixel of interest on the two-dimensional image is represented by (x, y). x and y represent the horizontal and vertical coordinate values of the pixel of interest, respectively. The color signals G0, R0, and B0 at the pixel position (x, y) of the original image are represented by G0 (x, y), R0 (x, y), and B0 (x, y), respectively. The color signals G1, R1, and B1 at the pixel position (x, y) are represented by G1 (x, y), R1 (x, y), and B1 (x, y), respectively, and the pixel position (x, y of the corrected image) ) Are represented by G2 (x, y), R2 (x, y) and B2 (x, y), respectively.

  HPFs 51G, 51R, and 51G are two-dimensional spatial filters having the same configuration and the same characteristics. The HPFs 51G, 51R, and 51G extract predetermined high frequency components Gh, Rh, and Bh included in the signals G0, R0, and B0 by filtering the input signals G0, R0, and B0. The high frequency components Gh, Rh and Bh extracted for the pixel position (x, y) are represented by Gh (x, y), Rh (x, y) and Bh (x, y), respectively.

The spatial filter outputs a signal obtained by filtering an input signal to the spatial filter. The filtering by the spatial filter refers to an operation for obtaining the output signal of the spatial filter using the input signal at the target pixel position (x, y) and the input signal at the peripheral position of the target pixel position (x, y). The value of the input signal at the target pixel position (x, y) is represented by I _IN (x, y), and the output signal of the spatial filter for the target pixel position (x, y) is represented by I _O (x, y). In this case, both satisfy the relationship of the following formula (8). h (u, v) represents a filter coefficient at the position (u, v) of the spatial filter. The filter size of the spatial filter according to Equation (8) is (2w + 1) × (2w + 1). w is a natural number.

  The HPF 51G is a spatial filter that extracts and outputs a high frequency component of an input signal, such as a Laplacian filter, and inputs the input signal G0 (x, y) and the target pixel position (x, y) at the target pixel position (x, y). ) Is used to obtain an output signal Gh (x, y) for the target pixel position (x, y) using input signals (including G0 (x + 1, y + 1) and the like). The same applies to HPF 51R and 51B.

  The LPFs 52G, 52R and 52G are two-dimensional spatial filters having the same configuration and the same characteristics. The LPFs 52G, 52R, and 52G extract predetermined low-frequency components GL, RL, and BL included in the signals G0, R0, and B0 by filtering the input signals G0, R0, and B0. The low frequency components GL, RL, and BL extracted for the pixel position (x, y) are represented by GL (x, y), RL (x, y), and BL (x, y), respectively. “GL (x, y) = G0 (x, y) −Gh (x, y), RL (x, y) = R0 (x, y) −Rh (x, y) and BL (x, y) = The low frequency components GL (x, y), RL (x, y), and BL (x, y) may be obtained according to “B0 (x, y) −Bh (x, y)”.

  The calculation unit 56 normalizes the high frequency component obtained as described above with the low frequency component for each color signal and for each pixel position, thereby obtaining the value Gh (x, y) / GL (x, y), Rh (x, y) / RL (x, y) and Bh (x, y) / BL (x, y) are obtained. Further, for each color signal and for each pixel position, their absolute values Ghh (x, y) = | Gh (x, y) / GL (x, y) |, Rhh (x, y) = | Rh (x , Y) / RL (x, y) | and Bhh (x, y) = | Bh (x, y) / BL (x, y) |

  FIG. 12 shows the relationship between the signals Ghh, Rhh, and Bhh obtained by the calculation unit 56 and the subject distance. The absolute values Ghh (x, y), Rhh (x, y) and Bhh (x, y) are values at the positions (x, y) of the signals Ghh, Rhh and Bhh, respectively. Curves 450G, 450R, and 450B are plots of how the signals Ghh, Rhh, and Bhh change with respect to changes in the subject distance, respectively. As can be seen from the comparison between the curve 400G in FIG. 7A and the curve 450G in FIG. 12, the subject distance dependency of the signal Ghh is the same as or similar to the subject distance dependency of the resolution of the signal G0 (signal Rhh). And Bhh). This is because the high-frequency component Gh of the signal G0 and the signal Ghh proportional to the absolute value thereof also increase / decrease in conjunction with the increase / decrease of the resolution of the signal G0.

  Without performing normalization with low frequency components, Ghh (x, y) = | Gh (x, y) |, Rhh (x, y) = | Rh (x, y) |, and Bhh (x, y ) = | Bh (x, y) |, absolute values Ghh (x, y), Rhh (x, y) and Bhh (x, y) can be obtained according to the color of the subject. Depending on the magnitude of the absolute values Ghh (x, y), Rhh (x, y), and Bhh (x, y) may vary. Therefore, Ghh (x, y) = | Gh (x, y) / GL (x, y) |, Rhh (x, y) = | Rh (x, y) / RL (x, y) |, and Obtaining absolute values Ghh (x, y), Rhh (x, y) and Bhh (x, y) according to Bhh (x, y) = | Bh (x, y) / BL (x, y) | desirable.

  The maximum value detection unit 53 specifies the maximum value among the absolute values Ghh (x, y), Rhh (x, y), and Bhh (x, y) for each pixel position, and the absolute value Ghh (x, y). ), Rhh (x, y) and Bhh (x, y), a signal SEL_GRB (x, y) indicating which is the maximum is output. Of the absolute values Ghh (x, y), Rhh (x, y), and Bhh (x, y), the case where Bhh (x, y) is the maximum is referred to as case 1, and Ghh (x, y) is the maximum. Is called Case 2, and the case where Rhh (x, y) is maximum is called Case 3.

The subject distance estimation unit 54 calculates the subject distance DIST (x, y) of the subject at the pixel position (x, y) based on the absolute values Ghh (x, y), Rhh (x, y), and Bhh (x, y). ). This estimation method will be described with reference to FIG. In the subject distance estimation unit 54, first, 0 <satisfy D _A <D _B, 2 two object distances D _A and D _B are defined in advance. The subject distance estimation unit 54 changes the subject distance estimation method according to which of the absolute values Ghh (x, y), Rhh (x, y), and Bhh (x, y) is the maximum.

In case 1, it is determined that the subject distance of the object is relatively small at the pixel position (x, y), 0 < DIST (x, y) < Within the D _A is satisfied, Rhh (x, y) / Ghh (x, y) is used to determine the estimated subject distance DIST (x, y). A line segment 461 in FIG. 13 shows the relationship between Rhh (x, y) / Ghh (x, y) and the estimated subject distance DIST (x, y) in Case 1. In case 1 where Bhh (x, y) is maximum, as shown in FIG. 12, Ghh (x, y) and Rhh (x, y) increase as the subject distance corresponding to the pixel (x, y) increases. Although both increase, it is considered that the degree of increase in Ghh (x, y) and Rhh (x, y) with respect to the increase in the subject distance is higher in Ghh (x, y). Accordingly, in case 1, the estimated subject distance DIST (x, y) is obtained so that DIST (x, y) increases as Rhh (x, y) / Ghh (x, y) decreases.

In Case 2, it is determined that the subject distance of the subject at the pixel position (x, y) is medium, and Bhh (x, y) within a range where D _A ≦ DIST (x, y) <D _B is satisfied. y) Estimated subject distance DIST (x, y) is obtained from / Rhh (x, y). A line segment 462 in FIG. 13 shows the relationship between Bhh (x, y) / Rhh (x, y) and the estimated subject distance DIST (x, y) in Case 2. In case 2 where Ghh (x, y) is maximum, as shown in FIG. 12, Bhh (x, y) decreases while Rhh (x, y) decreases as the subject distance corresponding to pixel (x, y) increases. , Y) increases. Therefore, in case 2, the estimated subject distance DIST (x, y) is obtained so that DIST (x, y) increases as Rhh (x, y) / Ghh (x, y) decreases.

In Case 3, it is determined that the subject distance of the subject at the pixel position (x, y) is relatively large, and Bhh (x, y) / Ghh is within a range where D _B <DIST (x, y) is satisfied. Estimated subject distance DIST (x, y) is obtained from (x, y). A line segment 463 in FIG. 13 shows the relationship between Bhh (x, y) / Ghh (x, y) and the estimated subject distance DIST (x, y) in Case 3. In case 3 where Rhh (x, y) is maximum, as shown in FIG. 12, Ghh (x, y) and Bhh (x, y) increase as the subject distance corresponding to the pixel (x, y) increases. Although both decrease, it is considered that Ghh (x, y) has a higher degree of decrease in Ghh (x, y) and Bhh (x, y) as the subject distance increases. Therefore, in case 3, the estimated subject distance DIST (x, y) is determined so that DIST (x, y) increases as Bhh (x, y) / Ghh (x, y) increases.

  As described above, the subject distance estimation unit 54 determines values corresponding to the magnitudes of the high frequency components Gh, Rh, and Bh included in the color signals G0, R0, and B0, that is, absolute values Ghh (x, y), Rhh ( Based on x, y) and Bhh (x, y), the subject distance of the subject at each position of the original image is estimated. DIST (x, y) is an estimated subject distance obtained for the pixel position (x, y). Information including estimated subject distances for all pixel positions is referred to as subject distance information DIST.

The selection unit 55 calculates values Gh (x, y) / GL (x, y), Rh (x, y) / RL (x, y), and Bh (x, y) / BL calculated by the calculation unit 56. One of (x, y) is selected based on the signal SEL_GRB (x, y), and the selected value is output as H (x, y) / L (x, y). In particular,
In case 1 where Bhh (x, y) is maximum, Bh (x, y) / BL (x, y) is output as H (x, y) / L (x, y),
In case 2 where Ghh (x, y) is maximum, Gh (x, y) / GL (x, y) is output as H (x, y) / L (x, y),
In case 3 where Rhh (x, y) is maximum, Rh (x, y) / RL (x, y) is output as H (x, y) / L (x, y).

  The enlargement processing unit 16 is supplied with color signals G0 (x, y), R0 (x, y) and B0 (x, y) of the original image and a signal H (x, y) / L (x, y). It is done. The selectors 61G, 61R, and 61B select one of the first and second input signals based on the signal SEL_GRB (x, y), and select the selected signals as G1 (x, y), R1 (x, y ) And B1 (x, y). The first input signals of the selection units 61G, 61R, and 61B are G0 (x, y), R0 (x, y), and B0 (x, y), respectively, and the second inputs of the selection units 61G, 61R, and 61B. The signals are “G0 (x, y) + G0 (x, y) × H (x, y) / L (x, y)”, “R0 (x, y) + R0 ( x, y) × H (x, y) / L (x, y) ”and“ B0 (x, y) + B0 (x, y) × H (x, y) / L (x, y) ”. .

In case 1 where Bhh (x, y) is maximum,
G1 (x, y) = G0 (x, y) + G0 (x, y) × H (x, y) / L (x, y),
R1 (x, y) = R0 (x, y) + R0 (x, y) × H (x, y) / L (x, y) and B1 (x, y) = B0 (x, y),
Selection processing of the selection units 61G, 61R, and 61B is performed so that
In case 2 where Ghh (x, y) is maximum,
G1 (x, y) = G0 (x, y),
R1 (x, y) = R0 (x, y) + R0 (x, y) × H (x, y) / L (x, y) and B1 (x, y) = B0 (x, y) + B0 (x , Y) × H (x, y) / L (x, y)
Selection processing of the selection units 61G, 61R, and 61B is performed so that
In case 3 where Rhh (x, y) is maximum,
G1 (x, y) = G0 (x, y) + G0 (x, y) × H (x, y) / L (x, y),
R1 (x, y) = R0 (x, y) and B1 (x, y) = B0 (x, y) + B0 (x, y) × H (x, y) / L (x, y)
Selection processing of the selection units 61G, 61R, and 61B is performed so that

For example, when the focused pixel position (x, y) is a pixel position in the partial pixel region 441 in FIG. 9 (see also FIG. 8), the absolute values Ghh (x, y), Rhh (x, y), and Bhh Among (x, y), Ghh (x, y) is the maximum. Therefore, in this case, since H (x, y) / L (x, y) = Gh (x, y) / GL (x, y), the pixel position (x, y) corresponding to the subject distance D ₁ For
G1 (x, y) = G0 (x, y),
R1 (x, y) = R0 (x, y) + R0 (x, y) × Gh (x, y) / GL (x, y) and B1 (x, y) = B0 (x, y) + B0 ( x, y) × Gh (x, y) / GL (x, y). These three equations are equivalent to those obtained by replacing “(D ₁ )” in the above equations (5) to (7) with “(x, y)”.

  FIG. 14 is an internal block diagram of the depth of field control unit 17 shown in FIG. The depth of field control unit 17 in FIG. 14 includes a variable LPF unit 71 and a cutoff frequency control unit 72. The variable LPF unit 71 includes three variable LPFs (low-pass filters) 71G, 71R, and 71B formed so that the cut-off frequency can be variably set. Based on the information DIST and the depth of field setting information, the cutoff frequencies of the variable LPFs 71G, 71R, and 71B are controlled. By inputting the signals G1, R1, and B1 to the variable LPFs 71G, 71R, and 71B, color signals G2, R2, and B2 representing a corrected image are obtained from the variable LPFs 71G, 71R, and 71B.

  Prior to the generation of the color signals G2, R2, and B2, the depth of field setting information is generated based on a user instruction or the like. The same depth-of-field curve 420 in FIG. 15 is set from the depth-of-field setting information, as shown in FIG. In the configuration example of FIG. 14, the cutoff frequency control unit 72 sets the depth of field curve 420 from the depth of field setting information. However, the depth of field setting is different from the cutoff frequency control unit 72. A depth-of-field curve 420 may be set from the information, and information representing the set depth-of-field curve 420 may be given to the cutoff frequency control unit 72.

As described above, the depth of field with respect to the corrected image is designated by the depth of field setting information. In other words, the width of the focus range in the corrected image is designated by the depth of field setting information. Of the focusing range specified by the depth of field setting information represents the shortest object distance in D _MIN, represents the longest object distance at D _MAX. Naturally, 0 <D _MIN <D _MAX .

The depth-of-field setting information includes focus target information. The focus target information determines which subject is the focus target and specifies a specific subject distance (D _CN ) to be positioned within the depth of field of the corrected image. The focus target information is distance setting information for directly specifying a specific subject distance or subject setting information for specifying a specific subject.

When the focus target information is distance setting information, the specific subject distance specified by the focus target information is a distance at the center of the focus range of the corrected image (a distance at the center of the depth of field) _DCN. . D _CN = (D _MIN + D _MAX ) / 2.

On the other hand, when the focus target information is subject setting information, the estimated subject distance of the specific subject specified by the focus target information is read from the subject distance information DIST, and the read estimated subject distance is set to _DCN . The user can specify a specific subject. For example, in a state where the original image or the intermediate generation image is displayed on the display screen of the LCD 19, the user designates a display portion on which the specific subject is displayed using the touch panel function. D _CN is set from the designation result and subject distance information DIST. For example, if the specific subject for the user is the subject SUB ₁ in FIG. 9 and subject setting information specifying the partial image area 441 is given, the subject distance DIST (x, y estimated for the subject SUB ₁₎ ) Is set to D _CN (when the estimation is ideally made, D _CN = D ₁ ).

The depth of field curve 420 is a curve that defines the relationship between the subject distance and the resolution. The resolution on the depth of field curve 420 takes a maximum value at the subject distance D _CN , and as the subject distance increases from the D _CN. It gradually decreases. The resolution on the depth of field curve 420 at the subject distance D _CN is larger than the reference resolution RS _O , and the resolution on the depth of field curve 420 at the subject distances D _MIN and D _MAX matches the reference resolution RS _O.

  The cutoff frequency control unit 72 assumes a virtual signal whose resolution matches the maximum resolution on the depth-of-field curve 420 for any subject distance. A broken line 421 in FIG. 15 represents the subject distance dependency of the resolution in the virtual signal. The cut-off frequency control unit 72 obtains the cut-off frequency of the low-pass filter necessary for converting the broken line 421 into the depth of field curve 420 by the low-pass filter process. That is, when it is assumed that the virtual signals are input signals to the variable LPFs 71G, 71R, and 71B, the output signals of the variable LPFs 71G, 71R, and 71B are called virtual output signals, the cutoff frequency control unit 72 The cutoff frequencies of the variable LPFs 71G, 71R, and 71B are set so that the curve representing the subject distance dependency of the signal resolution matches the depth-of-field curve 420.

Based on the subject distance information DIST, the cutoff frequency control unit 72 determines what cutoff frequency should be set for which partial image region. For example (see FIG. 9), the pixel position (x ₁ , y ₁ ) exists in the partial image area 441 where the image data of the subject SUB _{1 at} the subject distance D ₁ exists, and the subject SUB _{2 at} the subject distance D ₂ Consider a case where a pixel position (x ₂ , y ₂ ) exists in a partial image area 442 where image data exists. In this case, if the estimation error of the subject distance is ignored, the estimated subject distance DIST (x ₁ , y ₁ ) with respect to the pixel position (x ₁ , y ₁ ) and the estimated subject distance with respect to the surrounding pixel positions become D ₁ , and the pixel position (x _2, y ₂₎ estimation for subject distance DIST (x _2, y ₂₎ and estimates the subject distance relative to its surrounding pixel position is D _2. Further, as shown in FIG. 16, it is assumed that the resolutions on the depth-of-field curve 420 corresponding to the subject distances D ₁ and D ₂ are RS ₁ and RS ₂ , respectively.

In this case, the cut-off frequency control unit 72 determines the cut-off frequency CUT ₁ of the low-pass filter necessary for reducing the resolution of the virtual signal to the resolution RS _1, and sets the cut-off frequency CUT ₁ in the partial image region 441. Apply to signals G1, R1 and B1. Thereby, in the variable LPFs 71G, 71R, and 71B, the low-pass filter process of the cutoff frequency CUT ₁ is performed on the signals G1, R1, and B1 in the partial image region 441. The signal after the low-pass filter processing is output as signals G2, R2, and B2 in the partial image region 441 of the corrected image.

Similarly, the cut-off frequency control unit 72 determines the cut-off frequency CUT ₂ of the low-pass filter necessary for reducing the resolution of the virtual signal to the resolution RS _2, and sets the cut-off frequency CUT ₂ in the partial image region 442. Applies to signals G1, R1 and B1. Thereby, in the variable LPFs 71G, 71R, and 71B, the low-pass filter process with the cutoff frequency CUT ₂ is performed on the signals G1, R1, and B1 in the partial image region 442. The signal after the low-pass filter processing is output as signals G2, R2, and B2 in the partial image region 442 of the corrected image.

Table data or an arithmetic expression that defines the relationship between the resolution obtained after the low-pass filter processing and the cutoff frequency of the low-pass filter is prepared in advance, and is set in the variable LPF unit 71 using the table data or the arithmetic expression. The cut-off frequency to be determined can be determined. The table data or the arithmetic expression defines that the cutoff frequencies corresponding to the resolutions RS ₁ and RS ₂ are CUT ₁ and CUT ₂ , respectively.

As shown in FIG. 16, if the subject distance D ₁ belongs to the depth of field of the corrected image and the subject distance D ₂ is not included in the depth of field of the corrected image, CUT ₁ > CUT ₂ is established. Cut-off frequencies CUT ₁ and CUT ₂ are set, and the variable LPF unit 71 blurs the image in the partial image area 442 as compared with the image in the partial image area 441. As a result, the corrected image includes the image in the partial image area 442. The resolution of the image is lower than that of the partial image area 441. Also, unlike the situation shown in FIG. 16, when D _MAX <D ₁ <D ₂ , the cut-off frequencies CUT ₁ and CUT ₂ are set so that CUT ₁ > CUT ₂ is satisfied. Since the subject distances D ₁ and D ₂ do not belong within the depth of field, the images in the partial image areas 441 and 442 in the intermediate generation image are blurred by the variable LPF unit 71. However, the degree of the blur is larger in the partial image area 442 than in the partial image area 441. As a result, the resolution of the image in the partial image area 442 in the corrected image is lower than that in the partial image area 441.

  By performing such a low-pass filter process on the entire image area of the intermediate generation image, the color signals G2, R2, and B2 at each pixel position of the corrected image are output from the variable LPF unit 71. As described above, the subject distance dependency of the resolution of the color signals G2, R2, and B2 is shown by the curve 430 in FIG. The cut-off frequency determined by the cut-off frequency control unit 72 is for converting the resolution characteristic of the virtual signal into the resolution characteristic of the depth-of-field curve 420, but the actual color signals G1, R1. And the resolution characteristics of B1 are different from those of the virtual signal. Therefore, the curve 430 and the depth of field curve 420 are slightly different.

  In the above description, the depth-of-field curve 420 is once set based on the depth-of-field setting information, and then the cutoff frequency is set from the depth-of-field curve 420 and the subject distance information DIST. However, based on the depth-of-field setting information and the subject distance information DIST, the cutoff frequency of the variable LPF unit 71 may be set all at once without setting the depth-of-field curve 420.

<< Second Embodiment >>
Next, a second embodiment of the present invention will be described. FIG. 17 is an overall block diagram of an imaging apparatus 1a according to the second embodiment of the present invention. The imaging device 1a is provided with each part referred by the codes | symbols 10-14, 15a, 26, 18-24. Each part referred by the codes | symbols 10-14 and 18-24 is the same as those in 1st Embodiment (refer FIG. 6).

  In the imaging apparatus 1a, instead of the extraction / estimation unit 15, the enlargement processing unit 16, and the depth of field control unit 17 included in the imaging apparatus 1 of FIG. The imaging device 1 according to the first embodiment and the imaging device 1a according to the second embodiment, except for this difference, are provided with an extraction / estimation unit 15a) and a depth-of-field control unit 26. The same thing. The following description focuses on the differences from the first embodiment. In the second embodiment, the description of the first embodiment is applied to matters that are not particularly described. However, when the matters described in the first embodiment are applied to the second embodiment, the difference in reference numerals (for example, the difference between reference numerals 1 and 1a) is ignored.

  The R, G, and B signals of the original image generated by the demosaicing processing unit 14, that is, the color signals R0, G0, and B0 are given to the extraction / estimation unit 15a and the depth of field control unit 26. The extraction / estimation unit 15a includes HPFs 51G, 51R, and 51B, LPFs 52G, 52R, and 52B, a calculation unit 56, and a subject distance estimation unit 54 illustrated in FIG. 10, and each of the original images is obtained from the color signals R0, G0, and B0. Subject distance at the position is estimated, and subject distance information DIST representing the estimated subject distance is generated. The method for generating subject distance information DIST (including the subject distance estimation method) by the extraction / estimation unit 15a is the same as that by the extraction / estimation unit 15 of FIG.

  The depth-of-field control unit 26 corrects the color signals R0, G0, and B0 of the original image based on the subject distance information DIST generated by the extraction / estimation unit 15a and the depth-of-field setting information. A corrected image having a depth of field of 2 is generated. The R, G, and B signals of the corrected image generated by the depth of field control unit 26 are represented by R2 ', G2', and B2 ', respectively. The R, G, and B signals of the corrected image generated by the depth of field control unit 26 are sent to the camera signal processing unit 18.

  FIG. 18 shows an example of an internal block diagram of the depth of field control unit 26. The depth-of-field control unit 26 in FIG. 18 includes each part referred to by reference numerals 81 to 84.

Since the characteristics of the optical system 10 including the characteristics of axial chromatic aberration are known in advance at the design stage of the imaging device 1a, if the subject distance of the subject of interest in the target partial image region on the original image is known, the target partial image region The blur characteristics of the image for each color signal in FIG. For example, when the subject distance of the subject of interest is the subject distance DD _R that positions the imaging point 302R of the red light 301R on the imaging surface of the imaging device 11 (see FIGS. 1, 2C, and 4). The blue light 301B and the green light 301G from the target subject are blurred in the target partial image region, and if the target subject is regarded as a point light source, the radii of the blue light 301B and the green light 301G are YB and YB. (See FIG. 3). If the object distance is different from the subject distance DD _R is the same.

  It can be considered that the original image is an image deteriorated due to axial chromatic aberration. The deterioration here is blurring of the image due to axial chromatic aberration. A function or a spatial filter representing this deterioration process is called a point spread function (hereinafter referred to as PSF). Since the PSF for each color signal is obtained when the subject distance is determined, the PSF for each color signal at each position on the original image is obtained based on the estimated subject distance at each position on the original image included in the subject distance information DIST. Determined. If such a convolution operation using the inverse function of PSF is performed on the color signals G0, R0, and B0, the deterioration (blur) of the original image due to the longitudinal chromatic aberration is removed. Image processing that removes deterioration is also called image restoration processing. An image obtained by this removal is an intermediate generated image in the second embodiment, and R, G, and B signals of the intermediate generated image in the second embodiment are represented by R1 ', G1', and B1 ', respectively.

  The image restoration filter 81 in FIG. 18 is a two-dimensional spatial filter for applying the above inverse function to the signals G0, R0, and B0. The image restoration filter 81 corresponds to an inverse filter of PSF that represents the deterioration process of the original image derived from axial chromatic aberration. The filter coefficient calculation unit 83 obtains an inverse function of the PSF for the color signals G0, R0, and B0 at each position on the original image from the subject distance information DIST, and applies the obtained inverse function to the signals G0, R0, and B0. Therefore, the filter coefficient of the image restoration filter 81 is calculated. The image restoration filter 81 individually filters the color signals G0, R0, and B0 using the filter coefficients calculated by the filter coefficient calculation unit 83, thereby obtaining the color signals G1 ′, R1 ′, and B1 ′. Generate.

  A broken line 500 in FIG. 19 represents the subject distance dependency of the resolution in the color signals G1 ', R1', and B1 '. As described above, the curves 400G, 400R, and 400B represent the subject distance dependency of the resolution in the color signals G0, R0, and B0. By the image restoration processing for each color signal, an intermediate generated image with high resolution is obtained for all of the G, R, and B signals.

  The depth of field adjustment filter 82 is also a two-dimensional spatial filter. The depth-of-field adjustment filter 82 filters the color signals G1 ', R1', and B1 'for each color signal, thereby generating color signals G2', R2 ', and B2' representing the corrected image. The filter coefficient of the spatial filter as the depth of field adjustment filter 82 is calculated by the filter coefficient calculation unit 84.

  A depth-of-field curve 420 as shown in FIG. 15 or 16 is set according to the depth-of-field setting information. The significance of the depth-of-field setting information and the depth-of-field curve 420 is the same as that described in the first embodiment. Color signals G1 ', R1' and B1 'corresponding to the broken line 500 in FIG. 19 correspond to the virtual signals described in the first embodiment corresponding to the broken line 421 in FIG. The depth-of-field adjustment filter 82 adjusts the color signals G1 ′, R1 ′, and the depth-of-field curve 420 so that the curve representing the subject distance dependency of the resolution of the color signals G2 ′, R2 ′, and B2 ′ matches the depth-of-field curve 420. Filter for B1 '.

  The filter coefficient of the depth-of-field adjustment filter 82 for realizing such filtering is calculated by the filter coefficient calculation unit 84 based on the depth-of-field setting information and the subject distance information DIST.

  Since the resolution characteristics of the color signals of the intermediate generation image and the corrected image are the same between the first and second embodiments, the depth-of-field adjustment filter 82 and the filter coefficient calculation in the depth-of-field control unit 26 in FIG. 14 is replaced with the variable LPF 71 and cut-off frequency control unit 72 shown in FIG. 14, and the low-pass filter processing is performed on the color signals G1 ′, R1 ′, and B1 ′ using the variable LPF 71 and cut-off frequency control unit 72. Accordingly, the color signals G2 ′, R2 ′, and B2 ′ may be generated. In this case as well, as described in the first embodiment, the cutoff frequency of the variable LPF 71 unit may be determined based on the depth of field setting information and the subject distance information DIST.

  The significance of the processing executed by the depth-of-field control unit 26 in FIG. 18 will be supplementarily described. At least two of the blue, green, and red images in the target partial image region of the original image are blurred due to axial chromatic aberration, and the magnitude (blur amount) of the blur is the axial chromatic aberration of the optical system 10. And the subject distance of the subject of interest in the target partial image region. Accordingly, the depth-of-field control unit 26 determines the amount of blurring of the blue, green, and red images in the target partial image region of the original image, and the attention is included in the characteristics of the on-axis chromatic aberration and the subject distance information DIST that are known in advance. It can be estimated based on the estimated subject distance of the subject.

  If image restoration processing is performed on the original image for each color signal, the deterioration (blur) of the original image due to axial chromatic aberration is removed, and as a result, blue, green and red in the target partial image region of the intermediate generated image The blur amount (blur size) of the image becomes small enough to be ignored. That is, if the image restoration process is ideally performed, the diameter of the blur in the intermediate generation image is equal to or less than the diameter of a predetermined allowable circle of confusion.

  In the depth-of-field control unit 26, after the generation of the intermediate generation image, the blur amount (blur size) of the blue, green, and red images in the target partial image region of the corrected image is defined by the depth-of-field curve 420. Correction is applied to the intermediate generated image so that the amount of blurring is reduced.

  The resolution of a noticed color signal and the amount of image blur due to the noticed color signal are generally inversely proportional to each other. The latter decreases as the former increases, and conversely increases as the former decreases. Therefore, in the process of generating the color signals G2 ′, R2 ′, and B2 ′ from the color signals G0, R0, and B0, the blur amount (resolution) in each partial image region in the original image that is derived from the axial chromatic aberration is corrected in the image. It can be said that this is a process for correcting to the set blur amount (set resolution) in each partial image area of FIG. 1, and this process is executed for each color signal. The former amount of blur is estimated based on the characteristics of axial chromatic aberration and subject distance information DIST. In other words, the former amount of blur is estimated from the PSF for each color signal at each position on the original image, which is determined based on the characteristics of longitudinal chromatic aberration and the subject distance information DIST. The latter set blur amount is set based on the depth-of-field setting information that defines the depth-of-field curve 420 and the subject distance information DIST.

  In the configuration of FIG. 18, the filtering for obtaining the intermediate generation image is performed and then the filtering for obtaining the corrected image is performed. However, the filtering of both may be performed at the same time. That is, the depth of field control unit 26 may be configured like the depth of field control unit 26a of FIG. FIG. 20 is an internal block diagram of the depth of field control unit 26a. The depth of field control unit 26 a can be used as the depth of field control unit 26. The depth of field control unit 26 a includes a depth of field adjustment filter 91 and a filter coefficient calculation unit 92.

  The depth-of-field adjustment filter 91 is a two-dimensional spatial filter that performs filtering that integrates filtering by the image restoration filter 81 and filtering by the depth-of-field adjustment filter 82 in FIG. The color signals G2 ′, R2 ′, and B2 ′ are directly generated by filtering the color signals G0 ′, R0 ′, and B0 ′ of the original image by the depth-of-field adjustment filter 91 for each color signal. .

  The filter coefficient calculation unit 92 is a filter coefficient calculation unit in which the filter coefficient calculation units 83 and 84 of FIG. 18 are integrated, and the depth-of-field adjustment filter 91 for each color signal from the subject distance information DIST and the depth-of-field setting information. The filter coefficient of is calculated.

<< Third Embodiment >>
Next, a third embodiment of the present invention will be described. In the third embodiment, an operation flow for generating a corrected image from an original image in relation to a user operation will be described. The operation modes of the imaging apparatus according to the third embodiment include a shooting mode in which an image can be shot and a playback mode in which an image recorded on the recording medium 22 is played back and displayed on the LCD 19. Transition between the modes is performed according to the operation on the operation unit 24. The imaging device 1 or 1a according to the first or second embodiment is used as the imaging device according to the third embodiment. The matters described in the first or second embodiment are applied to the third embodiment as long as there is no contradiction.

  First, the operation of the imaging device (1 or 1a) when a corrected image is generated in the shooting mode will be described with reference to FIG. FIG. 21 is a flowchart showing the flow of this operation. The LCD 19 is in a state where an image can be displayed.

  In the photographing mode, in principle, photographing is performed sequentially at a predetermined frame period, an original image sequence arranged in time series from the image sensor 11 is acquired, and the original image sequence is displayed on the LCD 19 as a moving image. In step S11, the CPU 23 monitors whether or not a shutter button (not shown) provided on the operation unit 24 has been pressed, and performs steps S12 to S14 when it is confirmed that the shutter button has been pressed. In step S12, the CPU 23 causes the DRAM 13 to hold RAW data for one original image obtained immediately after pressing the shutter button in the DRAM 13, and in step S13, the original image or the intermediate generation image based on the RAW data is displayed on the LCD 19. Display on the screen. In step S14, the RAW data and compressed image data corresponding to the RAW data are recorded on the recording medium 22. At this time, the RAW data and the compressed image data are associated with each other, stored in one image file, and recorded on the recording medium 22. Here, the compressed image is an image obtained by compressing the original image or the intermediate generated image based on the RAW data held in the DRAM 13 by the compression / decompression processing unit 21.

  The function of generating a corrected image by the imaging device 1 or 1a is called a depth of field control function. After the shutter button is pressed and the processes in steps S12 to S14 are performed, it is confirmed in step S15 whether the depth-of-field control function is enabled. If it is enabled, the process starts from step S15. On the other hand, the process proceeds to step S16, but if it is not valid, the process returns from step S15 to step S11. In step S16, the original image based on the RAW data held in the DARM 13 in step S12 is set as the correction target original image.

In step S17 following step S16, a focus target designation operation for designating a focus target on the correction target original image by the user is received in a state where the correction target original image or an intermediate generated image based on the correction target original image is displayed on the LCD 19. The user can perform this focusing target designation operation on the LCD 19 using the touch panel function. That is, as described in the first embodiment, the user designates a part on the display screen of the LCD 19 on which the specific subject is displayed by using the touch panel function, so that the specific subject is designated as the focus target. can do. The distance D _CN described in the first embodiment is set from the designation result and the subject distance information DIST about the correction target original image (see FIG. 15). For example, when the specific subject for the user is the subject SUB ₁ in FIG. 9 and the display part of the image in the partial image area 441 in which the image data of the subject SUB ₁ exists is designated, the subject SUB ₁ estimated subject distance DIST (x, y) is set to D _CN (if the estimated subject distance is ideally made, the D _CN = D _1).

Note that the specific subject can be designated as a focusing target by operating the operation unit 24 without using the touch panel function. The user can also directly specify the distance _DCN by operating the operation unit 24.

In step S17, a depth designation operation for designating the depth of field by the user is further accepted. This depth designation operation is executed using the touch panel function or the operation unit 24. The depth of field of the corrected image to be generated from the original image to be corrected (that is, the distance between the distances D _MIN and D _MAX in FIG. 15) according to the depth of field specified by the depth specifying operation. Difference) is determined.

  In step S18 following step S17, the CPU 23 creates the depth-of-field setting information described in the first or second embodiment based on the contents of the focusing target specifying operation and the depth specifying operation in step S17. Thereafter, in step S19, the CPU 23 turns on the depth of field control process. When the depth-of-field control process is turned on, the part in the imaging device 1 or 1a for generating the R, G, and B signals of the corrected image from the R, G, and B signals of the original image to be corrected operates. To do. For this reason, in step S19, based on the depth-of-field setting information created in step S18, the R, G, and B signals of the corrected image are generated from the R, G, and B signals of the correction target original image.

  The R, G, and B signals of the corrected image generated in step S19 are converted into video signals composed of Y, U, and V signals by the camera signal processing unit 18, and then compressed by the compression / decompression processing unit 21. The obtained compressed video signal, that is, compressed image data of the corrected image (image data of the compressed corrected image) is recorded on the recording medium 22 in step S20. At this time, the RAW data recorded in step S14 and / or the image data of the compressed image may be erased from the recording medium 22. Further, the corrected image generated in step S19 may be displayed on the LCD 19. Then, after the depth of field control process is turned off in step 21, the process returns to step S11.

  Next, the operation of the imaging device (1 or 1a) when a corrected image is generated in the playback mode will be described with reference to FIG. FIG. 22 is a flowchart showing the flow of this operation. The LCD 19 is in a state where an image can be displayed.

  First, RAW data that is a source of a corrected image to be generated in the reproduction mode is acquired in steps S31 to S33. The processes in steps S31 to S33 are processes in the shooting mode. In step S31, the CPU 23 monitors whether or not a shutter button (not shown) provided on the operation unit 24 has been pressed, and performs processing in steps S32 and S33 when it is confirmed that the shutter button has been pressed. In step S32, the CPU 23 temporarily stores the raw data for one original image obtained immediately after pressing the shutter button in the DRAM 13, and temporarily stores the original image or the intermediate generated image based on the raw data on the LCD 19. Is displayed on the display screen. In step S33, the RAW data and compressed image data corresponding to the RAW data are recorded on the recording medium 22. At this time, the RAW data and the compressed image data are associated with each other, stored in one image file, and recorded on the recording medium 22. Here, the compressed image is an image obtained by compressing the original image or the intermediate generated image based on the RAW data temporarily stored in the DRAM 13 by the compression / decompression processing unit 21.

  When the recording process in step S33 is completed, it is confirmed whether or not the operation mode of the imaging device (1 or 1a) has changed from the shooting mode to the playback mode. When the operation mode of the imaging device (1 or 1a) is maintained in the shooting mode, the process returns to step S31 and the processes of steps S31 to S33 are repeatedly executed. Thereafter, each process of S41 to S48 is executed. Each process of steps S41 to S48 is a process in the reproduction mode.

  In step S <b> 41, the user selects one from one or a plurality of compressed images recorded on the recording medium 22. The selected compressed image is displayed on the LCD 19 through the decompression process. After the selection in step 41, it is confirmed in step S42 whether the depth-of-field control function is enabled. If it is enabled, the process proceeds from step S42 to step S43 while it is enabled. If not, the process returns from step S42 to step S41. In step S43, the RAW data associated with the compressed image selected in step S41 is read from the recording medium 22, and the original image based on the read RAW data is set as the correction target original image.

  The processing contents of steps S44 to S48 subsequent to step S43 are the same as those of steps S17 to S21 of FIG. 21, and the contents described above with reference to FIG. 21 are also applied to steps S44 to S48.

  That is, in step S44 subsequent to step S43, a focus target designation operation for designating a focus target on the correction target original image by the user is received in a state where the correction target original image or an intermediate generation image based thereon is displayed on the LCD 19. A depth designation operation for designating the depth of field is also accepted. Thereafter, in step S45, the CPU 23 creates the depth-of-field setting information described in the first or second embodiment based on the contents of the focus target specifying operation and the depth specifying operation in step S44. Subsequently, in step S46, the CPU 23 turns on the depth of field control process. Thereby, in step S46, based on the depth-of-field setting information created in step S45, the R, G, and B signals of the corrected image are generated from the R, G, and B signals of the correction target original image.

  The R, G, and B signals of the corrected image generated in step S46 are converted into video signals composed of Y, U, and V signals by the camera signal processing unit 18, and then compressed by the compression / decompression processing unit 21. The obtained compressed video signal, that is, compressed image data of the corrected image (image data of the compressed corrected image) is recorded on the recording medium 22 in step S47. At this time, the RAW data corresponding to the original image to be corrected and / or the image data of the compressed image recorded in the photographing mode may be deleted from the recording medium 22. Further, the corrected image generated in step S46 may be displayed on the LCD 19. Thereafter, after the depth of field control process is turned off in step 48, the process returns to step S41.

  According to the first to third embodiments, an image having a shallow depth of field (an image in which natural blur is reproduced according to the subject distance) that can be obtained only with a large single-lens reflex camera is obtained. It can be generated in a digital camera having a small and simple configuration. In addition, a corrected image having an arbitrary depth of field in which an arbitrary subject is in focus can be generated after the original image is captured. That is, since focus control after shooting is possible, shooting failures due to focus errors can be eliminated.

  The imaging device (1 or 1a) according to the present invention can be realized by hardware or a combination of hardware and software. In particular, the arithmetic processing executed by each part shown in FIG. 10, FIG. 18, or FIG. 20 can be realized by software, hardware, or a combination of hardware and software. When the imaging apparatus is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part. All or part of the arithmetic processing may be described as a program, and the program may be executed on a program execution device (for example, a computer) to realize all or part of the arithmetic processing.

It is a figure for demonstrating the characteristic of the axial chromatic aberration which a lens concerns on embodiment of this invention. FIG. 4 is a diagram illustrating a positional relationship between a point light source, a lens having axial chromatic aberration, an image formation point of each color light, and an image pickup device according to an embodiment of the present invention, and FIG. (A) shows that when the distance between the point light source and the lens is medium, and (c) shows that when the distance between the point light source and the lens is relatively large. FIG. 4 is a diagram illustrating a positional relationship among a point light source, a lens having axial chromatic aberration and an image sensor, and a spread of an image of each color light on the image sensor according to the embodiment of the present invention. It is a figure which shows the resolution characteristic of the color signal in the picked-up image obtained through the lens which has an axial chromatic aberration according to embodiment of this invention. It is a figure which shows the resolution characteristic of the color signal in the picked-up image obtained through the lens which has an axial chromatic aberration according to embodiment of this invention. 1 is an overall block diagram of an imaging apparatus according to a first embodiment of the present invention. FIG. 6 is a diagram for describing a principle that a corrected image is generated from an original image through generation of an intermediate generated image according to the first embodiment of the present invention. It is a diagram for explaining the two subject distance according to the specific example of the first embodiment of the present invention (D ₁ and D _2). FIG. 3 is a diagram illustrating two subjects at two subject distances (D ₁ and D ₂ ) and images of the two subjects on an original image. FIG. 7 is an internal block diagram of a high frequency component extraction / distance estimation unit and a depth of field expansion processing unit shown in FIG. 6. This is for explaining the significance of the pixel positions on the original image, the intermediate generation image, and the corrected image. It is a figure which shows the characteristic of the value produced | generated in the high region component extraction / distance estimation part of FIG. It is a figure for demonstrating the to-be-photographed object distance estimation method by the high region component extraction / distance estimation part of FIG. FIG. 7 is an internal block diagram of a depth of field control unit shown in FIG. 6. It is a figure for demonstrating the content of the process performed in the depth-of-field control part of FIG. It is a figure for demonstrating the content of the process performed in the depth-of-field control part of FIG. It is a whole block diagram of the imaging device concerning a 2nd embodiment of the present invention. FIG. 18 is an internal block diagram of a depth of field control unit shown in FIG. 17. It is a figure for demonstrating the content of the process performed in the depth-of-field control part of FIG. FIG. 18 is a modified internal block diagram of the depth of field control unit shown in FIG. 17. 10 is an operation flowchart according to the third embodiment of the present invention when a corrected image is generated in a shooting mode. 14 is an operation flowchart according to the third embodiment of the present invention when a corrected image is generated in a reproduction mode. It is a figure which concerns on a prior art and is a figure which shows the relationship between the lens which does not have an axial chromatic aberration, an image pick-up element, and the incident light to an image pick-up element. It is a figure which concerns on the prior art and is a figure which shows the resolution characteristic of the color signal in the picked-up image obtained through the lens which does not have an axial chromatic aberration.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 1a Imaging device 10 Optical system 11 Image sensor 14 Demosaicing processing part 15, 15a High frequency component extraction / distance estimation part 16 Depth of field expansion processing part 17, 26, 26a Depth of field control part

Claims (6)

An image sensor for photographing a subject through an optical system having axial chromatic aberration;
Predetermined high frequency components included in a plurality of color signals representing a captured image are extracted for each color signal, and each partial image region in the captured image is based on the size of the high frequency component for each color signal Subject distance estimating means for estimating the subject distance at
Image processing means for generating a corrected image from the captured image;
Setting means for setting the depth of field for the corrected image by specifying the depth of the depth of field for the corrected image and specifying a specific subject distance to be positioned within the depth of field; With
The image processing means performs the image processing according to the estimated subject distance by the subject distance estimation means and the set depth of field by the setting means on the captured image, so that the correction according to the set depth of field is performed. An imaging device that generates an image. Based on the set depth of field and the estimated subject distance in each partial image area in the captured image, the entire estimated image distance of each of the captured image and the corrected image has a corresponding estimated subject distance. The estimated subject distance corresponding to the first partial image area falling within the depth of field is classified into the second partial image area located outside the set depth of field,
The image processing unit generates the corrected image from the captured image so that an image in the second partial image region is blurred in the corrected image compared with an image in the first partial image region. The imaging device according to claim 1. The color signals of the plurality of colors are composed of different first, second and third color signals,
The image processing means includes
By adding the high frequency component of the color signal corresponding to the maximum size among the high frequency components of the first to third color signals to the other two color signals, First correction means for generating a color signal after addition of 3 high frequency bands,
The first to third corrections representing the corrected image from the first to third post-high-frequency-added color signals by performing predetermined filtering on the first to third high-pass-added color signals. Second correction means for generating a color signal,
The second correction unit is configured to determine whether an image in the second partial image area in the corrected image is within the first partial image area based on the set depth of field and an estimated subject distance in each partial image area in the captured image. The imaging apparatus according to claim 2, wherein the filtering is performed so that the image is blurred compared to the image of the image. From the estimated subject distance in each partial image region in the captured image, a point spread function derived from the axial chromatic aberration for each color signal in each partial image region in the captured image is determined,
The image processing means is configured to determine whether the image in the second partial image area in the corrected image is based on the point spread function, the set depth of field, and the estimated subject distance in each partial image area in the captured image. The imaging apparatus according to claim 2, wherein the correction image is generated from the captured image so as to be blurred compared with an image in one partial image region. The image processing means converts a blur amount in each partial image region in the captured image derived from the axial chromatic aberration, estimated from the point spread function, to a set blur amount in each partial image region in the corrected image. And generating the corrected image blurred according to the set blur amount by performing filtering for correcting each color signal of the captured image,
In the corrected image, the set depth of field and the estimated subject distance in each partial image region in the photographed image so that the image in the second partial image region is blurred compared with the image in the first partial image region. The imaging apparatus according to claim 3, wherein the set blur amount is determined based on the image quality. It further comprises display means for displaying the photographed image,
After shooting the shot image, the shot image or an image based on the shot image is displayed on the display screen of the display means,
The setting means accepts an operation for designating a specific subject on the display screen while the display is being performed, and designates the specific subject distance from an estimated subject distance corresponding to the specific subject, while performing a predetermined operation The imaging apparatus according to claim 1, wherein the depth of field with respect to the corrected image is specified by receiving the correction image.

Images