JP5793210B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus capable of blur correction based on distance information of a subject included in a captured image.

  Conventionally, Patent Document 1 discloses an imaging apparatus capable of discretely distributing focus detection pixels between pixel groups of photoelectric conversion means and calculating a subject distance based on a signal from the focus detection pixels. ing. By adopting the configuration disclosed in Patent Document 1, it is possible to acquire the distance distribution for the subject included in the captured image.

  In addition, as a technique for restoring a blurred image and generating an image with the blurred image restored, for example, there is a technique of applying a Wiener filter, a general inverse filter, a projection filter, or the like. Japanese Patent Application Laid-Open No. 2004-228688 discloses a technique for repairing blur using such a method. By using the technique disclosed in Patent Document 2, a deterioration function is obtained by physical analysis based on imaging conditions or the like, or estimation based on output from a measuring device in the imaging device, and an image generally called deconvolution The blurred image can be restored by the restoration algorithm.

  Normally, the subject distance to be focused is determined by the focus state at the time of shooting. For this reason, it is impossible to change the subject distance in focus after shooting. However, if the subject distance distribution in the photographed image is acquired with the technique disclosed in Patent Document 1 and the blur repair technique disclosed in Patent Document 2 is used to correct the blur, it is determined which subject distance to focus on. Can be changed after shooting.

JP 2000-156823 A JP 2000-20691 A

However, when the techniques disclosed in Patent Document 1 and Patent Document 2 are applied to an imaging apparatus, there is a problem that it is difficult for a photographer to check a captured image.
If the techniques disclosed in Patent Document 1 and Patent Document 2 are used, the subject distance in focus can be changed by switching the blur restoration according to the subject distance after shooting. However, the focus lens movement range of the photographic lens is limited, and if the amount of blur is too large, the blur can be repaired because of the extreme conversion processing that occurs when restoring the blur image and noise is likely to occur. The range of subject distance and the amount of blur are limited. Therefore, the subject distance and the amount of blur that can be focused are also limited to a predetermined range. The photographer looks at the confirmation image display immediately after photographing and estimates the range of the subject distance that can be focused. However, it has been very difficult to grasp which range can be focused from an image obtained by focusing on a certain subject distance.

  In addition, it is assumed that the image is displayed in an out-of-focus state in the captured image confirmation display after the image capture, regardless of the area where the image in focus can be acquired by the blur restoration process after the image capture. In this case, even if there is a failure part that the photographer does not intend in that area, it is displayed in a blurred state, so the photographer is not aware of the confirmation immediately after shooting, and it is not until the blur is repaired after shooting. There was a problem of noticing.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to realize an imaging apparatus that allows a photographer to easily check a captured image in an imaging apparatus having a blur restoration unit.

In order to achieve the above object, an image processing apparatus according to the present invention includes an acquisition unit that acquires distance information corresponding to a subject distance in a captured image captured by an image sensor that receives a light beam from a photographing lens, and an acquisition unit. Image processing means for performing image processing for changing the focus state on the captured image based on the distance information obtained in step (1), and display control means for displaying the image processed by the image processing means on the display means. The image processing means generates an image in which the focus state of the distance range corresponding to the subject distance from the first distance to the second distance is changed, and the image processing includes the light transfer characteristics of the photographing lens and the image sensor. wherein the image processing der Rukoto based on.
Another image processing apparatus of the present invention is obtained by an acquisition unit that acquires distance information corresponding to a subject distance in a captured image captured by an image sensor that receives a light beam from a shooting lens, and the acquisition unit. Image processing means for performing image processing for changing the focus state of the captured image based on the distance information, and display control means for displaying the image processed by the image processing means on the display means. The image processing means generates an image in which the focus state of the distance range corresponding to the subject distance from the first distance to the second distance is changed, and the image processing is a deconvolution process. .

Another image processing apparatus of the present invention includes an acquisition means for acquiring the distance information of the object distance in the captured image captured by the image pickup element for receiving light flux from the photographic lens, distance obtained by the obtaining means based on the information, and an image processing means for performing image processing for changing the focus state with respect to the captured image, have a display control means for displaying on the display means an image that has been image processed by the image processing means, display control means Is characterized by displaying a range in which the focus state can be changed by the image processing means after the photographed image is photographed .

  The imaging apparatus of the present invention can display a photographed image obtained by correcting a subject having a plurality of distances in the photographed image within a range from the first distance to the second distance as a confirmation image immediately after photographing. In other words, by displaying a blur repair image that is in focus at a plurality of distances in the captured image, it is possible to display a confirmation image that makes it easy for the photographer to grasp the range of the subject distance that can be repaired. In addition, according to the blurred image of the present invention, it is possible to more easily find a shooting failure point in an area where a focused image can be acquired. Further, by displaying the detected subject position and the number of subjects in a superimposed manner on the blurred image, it is possible to easily grasp the position and number of the detected subjects. Thus, this invention can implement | achieve the imaging device which a photographer tends to confirm a picked-up image.

Configuration diagram of camera of the present invention Circuit diagram of imaging device of the present invention FIG. 3 is a cross-sectional view of a pixel portion of the image sensor of the present invention. The top view and sectional drawing of the pixel for imaging of the image pick-up element of this invention The top view and sectional drawing of a focus detection pixel of an image sensor of the present invention The top view and sectional drawing of the other pixel for focus detection of the image sensor of the present invention The conceptual diagram explaining the pupil division | segmentation condition of the image sensor of this invention Distance information obtained by the distance information acquisition means of the present invention The relationship between the subject distance of the present invention and the distance at which blur can be repaired The relationship between the subject distance imaging plane position of Example 1 and the imaging plane position of the taking lens 137 Photographed image before blur repair is performed by the blur repair means of the present invention A state in which the blur of the photographed image is repaired by the blur repairing means of the present invention Confirmation image display immediately after shooting according to the present invention Main flow diagram of imaging apparatus of the present invention Flow chart of subject distance map creation subroutine Flow chart of shooting subroutine Flow chart of confirmation image display subroutine after shooting Flow chart of blur repair subroutine Flow diagram of blur function generation subroutine Relationship between the subject distance imaging plane position of Example 2 and the imaging plane position of the taking lens 137

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[Example 1]
FIGS. 1-19 is a figure concerning Example 1 of this invention. Hereinafter, the operation of the first embodiment will be described with reference to the drawings.
FIG. 1 is a configuration diagram of an image pickup apparatus according to the present invention, which includes a camera body 138 having an image sensor and a separate shooting lens 137. Shows the camera.

  First, the configuration of the taking lens 137 will be described. Reference numeral 101 denotes a first lens group disposed at the tip of a photographing optical system (imaging optical system), which is held so as to be movable back and forth (movable back and forth) in the optical axis direction. Reference numeral 102 denotes an aperture, which adjusts the light amount at the time of photographing by adjusting the aperture diameter. Reference numeral 103 denotes a second lens group. The diaphragm 102 and the second lens group 103 are integrally moved forward and backward in the optical axis direction, and perform a zooming function (zoom function) in conjunction with the movement operation of the first lens group 101.

  Reference numeral 105 denotes a third lens group that performs focus adjustment by advancing and retreating in the optical axis direction. A zoom actuator 111 rotates a cam cylinder (not shown) to drive the first lens group 111 or the second lens group 103 back and forth in the optical axis direction to perform a zooming operation. An aperture actuator 112 controls the aperture diameter of the aperture 102 and adjusts the amount of photographing light. Reference numeral 114 denotes a focus actuator, which performs focus adjustment by driving the third lens group 105 back and forth in the optical axis direction.

  Reference numeral 136 denotes a camera communication circuit that passes information about the lens to the camera and receives information about the camera. Information on the lens includes a zoom state, a diaphragm state, a focus state, lens frame information, and the like. The camera communication circuit 136 passes these pieces of information to the lens communication circuit 135 provided on the camera side.

  Next, the camera body 138 will be described. Reference numeral 106 denotes an optical low-pass filter, which is an optical element for reducing false colors and moire in a captured image. Reference numeral 107 denotes an image sensor composed of a C-MOS sensor and its peripheral circuits. As the image pickup device, a two-dimensional single-plate color sensor is used in which a Bayer array primary color mosaic filter is formed on-chip on a light receiving pixel in which m pixels are arranged in the horizontal direction and n pixels are arranged in the vertical direction. . The photoelectric conversion means in the claims corresponds to the image sensor 107.

  Reference numeral 139 denotes a shutter unit that controls the exposure time during still image shooting. Reference numeral 140 denotes a shutter actuator for moving the shutter 139.

  Reference numeral 115 denotes an electronic flash for illuminating a subject at the time of photographing, and a flash illumination device using a xenon tube is suitable, but an illumination device including an LED that emits light continuously may be used. Reference numeral 116 denotes an AF auxiliary light unit that projects an image of a mask having a predetermined aperture pattern onto a subject field via a light projecting lens, and improves focus detection capability for a dark subject or a low contrast subject.

  Reference numeral 121 denotes a CPU, which is an in-camera CPU that controls various controls of the camera body, and includes a calculation unit, ROM, RAM, A / D converter, D / A converter, communication interface circuit, and the like. Based on the program, various circuits of the camera are driven to execute a series of operations such as AF, photographing, image processing, and recording.

  An electronic flash control circuit 122 controls lighting of the illumination unit 115 in synchronization with the photographing operation. Reference numeral 123 denotes an auxiliary light driving circuit that controls the lighting of the AF auxiliary light unit 116 in synchronization with the focus detection operation. An image sensor driving circuit 124 controls the image capturing operation of the image sensor 107 and A / D converts the acquired image signal and transmits the image signal to the CPU 121. An image processing circuit 125 performs processes such as γ conversion, color interpolation, and JPEG compression of the image acquired by the image sensor 107.

  A focus drive circuit 126 controls the focus actuator 114 based on the focus detection result, and adjusts the focus by driving the third lens group 105 back and forth in the optical axis direction. Reference numeral 128 denotes an aperture driving circuit that controls the aperture actuator 112 to drive the aperture of the aperture 102. Reference numeral 129 denotes a zoom drive circuit that drives the zoom actuator 111 in accordance with the zoom operation of the photographer. 135 communicates with a lens communication circuit and a camera communication circuit 136 in the photographing lens 137. A shutter drive circuit 145 drives the shutter actuator 140.

  Reference numeral 131 denotes a display device such as an LCD, which displays information related to the shooting mode of the camera, a preview image before shooting and a confirmation image after shooting, a focus state display image at the time of focus detection, and the like. An operation switch group 132 includes a power switch, a release (shooting trigger) switch, a zoom operation switch, a shooting mode selection switch, and the like. Reference numeral 133 denotes a detachable flash memory that records a photographed image. Reference numeral 144 denotes an in-camera memory in which various data necessary for calculations performed by the CPU 121 are stored.

  FIG. 2 is a schematic circuit diagram of the image pickup device of the present invention. For example, the technique disclosed in Japanese Patent Application Laid-Open No. 09-046596 by the present applicant can be used. The drawing shows a pixel array of only 2 columns × 4 rows of a two-dimensional C-MOS area sensor for convenience. However, in an actual image sensor, a large number of illustrated pixels 30-11 to 30-42 are arranged, and a high-resolution image can be acquired. In the present embodiment, the description will be made on the assumption that the pixel pitch is 2 μm, the number of effective pixels is 3000 horizontal pixels × 2000 vertical rows = 6 million pixels, and the imaging screen size is 6 mm horizontal × 4 mm vertical.

  In FIG. 2, 1 is a photoelectric conversion portion of a photoelectric conversion element comprising a MOS transistor gate and a depletion layer under the gate, 2 is a photogate, 3 is a transfer switch MOS transistor, 4 is a reset MOS transistor, and 5 is a source follower amplifier MOS. Transistor, 6 is a horizontal selection switch MOS transistor, 7 is a source follower load MOS transistor, 8 is a dark output transfer MOS transistor, 9 is a light output transfer MOS transistor, 10 is a dark output storage capacitor CTN, and 11 is a light output storage capacitor CTS , 12 are horizontal transfer MOS transistors, 13 is a horizontal output line reset MOS transistor, 14 is a differential output amplifier, 15 is a horizontal scanning circuit, and 16 is a vertical scanning circuit.

  FIG. 3 shows a cross-sectional view of the pixel. In this figure, 17 is a P-type well, 18 is a gate oxide film, 19 is a first-layer poly-Si, 20 is a second-layer poly-Si, and 21 is an n + floating diffusion portion (FD). The FD 21 is connected to another photoelectric conversion unit via another transfer MOS transistor. In the figure, the drains of the two transfer MOS transistors 3 and the FD portion 21 are made common to improve the sensitivity by miniaturization and the capacity reduction of the FD portion 21, but the FD portion 21 may be connected by Al wiring. .

  4 to 5 are diagrams illustrating the structures of the imaging pixels and the focus detection pixels. In the present embodiment, out of 4 pixels of 2 rows × 2 columns, pixels having G (green) spectral sensitivity are arranged in 2 diagonal pixels, and R (red) and B (blue) are arranged in the other 2 pixels. A Bayer arrangement in which one pixel each having a spectral sensitivity of 1 is arranged is employed. The focus detection pixels are distributed and arranged in a predetermined rule between the Bayer arrays. A technique for discretely disposing focus detection pixels between imaging pixels is disclosed in Patent Document 1 and the like and is a known technique, and thus the description thereof is omitted.

  FIG. 4 shows the arrangement and structure of the imaging pixels. FIG. 2A is a plan view of 2 × 2 imaging pixels. As is well known, in the Bayer array, G pixels are arranged diagonally, and R and B pixels are arranged in the other two pixels. The structure of 2 rows × 2 columns is repeatedly arranged.

  A cross section AA of FIG. 4A is shown in FIG. ML is an on-chip microlens disposed on the forefront of each pixel, CFR is an R (Red) color filter, and CFG is a G (Green) color filter. PD is a schematic diagram of the photoelectric conversion unit of the C-MOS sensor described with reference to FIG. 3, and CL is a wiring layer for forming signal lines for transmitting various signals in the C-MOS sensor. TL schematically shows the photographing optical system.

  Here, the on-chip microlens ML and the photoelectric conversion unit PD of the imaging pixel are configured to capture the light beam that has passed through the photographing optical system ML as effectively as possible. In other words, the exit pupil EP of the photographing optical system TL and the photoelectric conversion unit PD are conjugated with each other by the microlens ML, and the effective area of the photoelectric conversion unit is designed to be large. Further, in FIG. 5B, the incident light beam of the R pixel has been described, but the G pixel and the B (Blue) pixel have the same structure. Accordingly, the exit pupil EP corresponding to each of the RGB pixels for imaging has a large diameter, and the S / N of the image signal is improved by efficiently capturing the light flux from the subject.

  FIG. 5 shows the arrangement and structure of focus detection pixels for performing pupil division in the horizontal direction (lateral direction) of the photographic lens. FIG. 5A is a plan view of 2 × 2 pixels including focus detection pixels. When obtaining an imaging signal, the G pixel is a main component of luminance information. Since human image recognition characteristics are sensitive to luminance information, if a G pixel is lost, it is easily recognized as image quality degradation. On the other hand, the R or B pixel is a pixel that acquires color information. However, since humans are insensitive to color information, pixels that acquire color information are less likely to notice deterioration in image quality even if some loss occurs. Therefore, in the present embodiment, among the pixels of 2 rows × 2 columns, the G pixel is left as an imaging pixel, and the R and B pixels are used as focus detection pixels. This is indicated by SHA and SHB in FIG.

  A cross section AA of FIG. 4A is shown in FIG. The microlens ML and the photoelectric conversion unit PD have the same structure as the imaging pixel shown in FIG. In the present embodiment, since the signal of the focus detection pixel is not used as the signal of the captured image, a transparent film CFW (White) is disposed instead of the color filter for color separation. Further, since pupil division is performed by the image sensor, the opening of the wiring layer CL is biased in one direction with respect to the center line of the microlens ML. Specifically, since the pixel SHA and the opening OPHA of the pixel SHA are biased to the right side, the light beam that has passed through the exit pupil EPHA on the left side of the photographing lens TL is received. Similarly, since the opening OPHB of the pixel SHB is biased to the left side, the light beam that has passed through the right exit pupil EPHB of the photographic lens TL is received. Therefore, the pixels SHA are regularly arranged in the horizontal direction, and the subject image acquired by these pixels is defined as an A image. Also, the pixels SHB are regularly arranged in the horizontal direction, and the subject image acquired by these pixels is a B image. By detecting the relative position of the A image and the B image, the amount of focus shift ( Defocus amount) can be detected.

Here, the microlens ML is a pair of optical images of an A image composed of a light beam transmitted through the left exit pupil EPHA of the photographing lens TL and a B image composed of a light beam transmitted through the right exit pupil EPHAB of the photographing lens TL. Plays the function of a lens element that generates
In the pixels SHA and SHB, focus detection is possible for an object having a luminance distribution in the horizontal direction of the photographing screen, for example, a vertical line, but focus detection is not possible for a horizontal line having a luminance distribution in the vertical direction. Therefore, in the present embodiment, pixels that perform pupil division are also provided in the vertical direction (longitudinal direction) of the shooting screen so that the focus can be detected in the latter case.

  FIG. 6 shows the arrangement and structure of focus detection pixels for performing pupil division in the vertical direction of the imaging screen. FIG. 5A is a plan view of pixels of 2 rows × 2 columns including focus detection pixels. As in FIG. 5A, the G pixel is left as an imaging pixel and the R and B pixels are in focus. Detection pixels are used. This is indicated by SVC and SVD in FIG.

  FIG. 6B shows a cross section A-A in FIG. 6A. The pixel in FIG. 5B has a structure in which the pupil is separated in the horizontal direction, whereas the pixel in FIG. The pupil separation direction is just the vertical direction, and the pixel structure does not change. That is, since the opening OPVC of the pixel SVC is biased downward, the light beam that has passed through the upper exit pupil EPVC of the photographing lens TL is received. Similarly, since the opening OPVD of the pixel SVD is biased upward, the light beam that has passed through the lower exit pupil EPVD of the photographing lens TL is received. Therefore, the pixels SVC are regularly arranged in the vertical direction, and the subject image acquired by these pixels is defined as a C image. Also, the pixels SVD are regularly arranged in the vertical direction, and if the subject image acquired by these pixels is a D image, a subject having a luminance distribution in the vertical direction is detected by detecting the relative position of the C image and the D image. The amount of defocus (defocus amount) of the image can be detected.

  FIG. 7 is a diagram for conceptually explaining the pupil division state of the image sensor of the present invention. TL is a photographic lens, 107 is an image sensor disposed on a planned imaging plane of the photographic lens, OBJ is a subject, and IMG is a subject image.

  As described with reference to FIG. 4, the imaging pixel receives the light beam that has passed through the entire exit pupil EP of the photographing lens. On the other hand, the focus detection pixel has a pupil division function as described with reference to FIGS. Specifically, the focus detection pixel SHA in FIG. 5 receives the light beam that has passed through the left pupil when viewed from the imaging surface, that is, the light beam that has passed through the pupil EPHA in FIG. Similarly, the focus detection pixels SHB, SVC, and SVD receive light beams that have passed through the pupils EPHB, EPVC, and EPVD, respectively. The focus detection pixels are arranged so as to be distributed over the entire area of the image sensor 107, thereby enabling focus detection in the entire image pickup area. The CPU 121 that also serves as the distance information acquisition unit calculates the subject distance based on the focus detection information and lens information such as the focal length.

  FIG. 8 shows the distance information obtained by the distance information acquisition means. In the image sensor 107 of the present invention, the focus detection pixels SHA, SHB, SHC, and SHD as described with reference to FIGS. 5 and 6 are distributed over the entire area. It can be acquired. In the obtained distribution of subject distances, the contours of the subjects included in the shooting screen can be extracted by connecting and grouping regions having a short subject distance. Target1, Target2, and Target3 indicate extracted subject areas, and BackGraound1 indicates a background area. Dist1, Dist2, Dist3, and Dist4 are subject distances. Dist1 is the subject distance in the subject area Target1, Dist2 is the subject distance in the subject area Target2, Dist3 is the subject distance in the subject area Target3, and Dist4 is the subject distance in the background area BackGround1. Dist1 is the closest, Dist2 is the second closest, and Dist3 is the third closest. And Dist4 is farthest away.

  The CPU 121 extracts a subject from the distribution of the subject distance obtained from the focus detection pixels, and acquires the area and distance of each subject.

  In the image pickup apparatus of the present invention, the blurred image is corrected based on the distance information. The blur generation process can be estimated from the imaging device characteristics and the imaging lens characteristics. A blur function that models this blur generation process is defined, and a blur image is restored by an image restoration algorithm such as a Wiener filter, which is generally called deconvolution, to restore the blur (the process of changing the degree of blur) Do). Since the blur repair method is described in Patent Document 2, detailed description thereof is omitted.

  FIG. 9 shows the relationship between the subject distances Dist1, Dist2, Dist3, and Dist4 and the distance that can be corrected for blur. The axis represents the subject distance Dist.

  The subject distance that can be focused is determined by the distance that the taking lens 137 can be extended. Since it is difficult to predict a blur shape at a distance where focus cannot be achieved, the distance at which blur repair is possible is also limited to a range near that distance. Therefore, the CPU 121 calculates a range of distance for performing blur repair according to the distance that the photographing lens 137 can be extended. The closest distance end where blur repair is performed is the first distance Dist11, and the infinite distance distance end is the second distance Dist12. The blur restoration means can blur the subject image in the range of the first distance Dist11 and the second distance Dist12. The first distance is determined based on the closest distance end of the photographing lens 137, and the second distance is determined based on the infinite distance end. Here, the first distance and the second distance take into account the manufacturing error of the photographing lens 137, the focus lens stop accuracy, the blur correction variation, and the like, with respect to the closest distance end and the infinite distance end of the photographing lens 137, Set with some margin.

  Among the subject distances Dist1, Dist2, Dist3, and Dist4 described in FIG. 8, the subject distances Dist1 to Dist3 are located in the range of the first distance Dist11 and the second distance Dist12, and the subject distance Dist4 is located outside the range. Suppose you are.

  FIG. 10 shows the relationship between the position of the imaging plane corresponding to the subject distances Dist1, Dist2, Dist3, and Dist4 and the position of the imaging plane of the photographing lens 137. The axis represents the defocus amount Def from the position of the imaging plane of the photographic lens 137. Here, the plus direction of the defocus amount Def corresponds to the rear pin direction.

  “0” indicates the position of the imaging plane of the photographic lens 137 and the amount of deviation is expressed as “0”. Def1, Def2, Def3, and Def4 are defocus amounts at the imaging plane positions of the subject distances Dist1, Dist2, Dist3, and Dist4. Def11 and Def12 are defocus amounts at the position of the image plane of the first distance Dist11 on the near side and the second distance Dist12 on the infinity side.

  On the other hand, Def21 and Def22 are defocus amounts corresponding to blur amounts that can be corrected for blur. When the amount of blur is too large, extreme conversion processing is performed at the time of restoring a blurred image, and noise is likely to occur. For this reason, when restoring the blur, it is desirable that the blur amount be within the range of Def21 and Def22. In FIG. 10, the defocus amounts Def21 and Def22 corresponding to the defocus amount that can be corrected for blur are the defocus amounts Def11 and Def11 at the positions of the image planes of the first distance Dist11 on the closest side and the second distance Dist12 on the infinite side. It exists outside Def12. For this reason, the first distance Dist11 on the near side and the second distance Dist12 on the infinity side are not updated.

  FIG. 11 shows a photographed image before the blur restoration by the blur restoration means. The captured image in FIG. 11 is the same as the captured image in FIG. As will be described in the later-described operation flow, the focus detection position is given priority to the closest subject, and the position of the subject located closest to the extracted subject is set as the focus detection position. For this reason, the photographed image before the blur restoration by the blur restoration means is in focus on the closest subject area Target1 as shown in FIG. The other subject areas Target2, Target3, and BackGranoud1 are not in focus.

  FIG. 12 shows how the blur of the captured image is repaired by the blur repair means. The captured image in FIG. 12 is the same as the captured image in FIGS. FIG. 12A shows an image obtained by performing blur restoration on the entire captured image region based on the subject distance Dist2 of the subject region Target2. A blur function is defined from imaging device characteristic information and photographing lens information corresponding to the subject distance Dist2 in the subject region Target2. By performing a repair process based on the blur function for the entire captured image area, the subject area Target2 is repaired out of focus and becomes an in-focus image. On the other hand, for an area other than the subject area Target2, blur is generated by performing the above-described image restoration process so that the focal position of the photographing lens 137 is in focus with the subject distance Dist2. By doing so, it is possible to acquire an image in which only the subject region Target2 is in focus as shown in FIG.

  FIG. 12B shows an image obtained by performing blur restoration on the entire captured image region based on the subject distance Dist3 of the subject region Target3. A blur function is defined from imaging device characteristic information and photographing lens information corresponding to the subject distance Dist3 in the subject region Target3. By performing a repair process based on this blur function on the entire captured image area, the subject area Target3 is repaired out of focus and becomes an in-focus image. On the other hand, for an area other than the subject area Target3, the above-described image restoration process reproduces the blur that occurs when the focus position of the photographing lens 137 is in focus with the subject distance Dist3. By doing so, it is possible to acquire an image in which only the subject area Target3 is in focus as shown in FIG.

  As described above with reference to FIG. 11 and FIGS. 12A to 12B, in the imaging apparatus capable of blur correction based on subject distance information, based on distance information including the area and distance of each subject. By correcting the blur, it is possible to select a subject to be focused.

  However, as shown in FIG. 11 and FIGS. 12A to 12B, there is a problem in that it is difficult for the photographer to check the photographed image because the subject to be focused has a degree of freedom.

  Although the above-described blur restoration can change the subject distance in focus after shooting, the range of subject distances that can be corrected for blur is limited. Therefore, the subject distance that can be focused is also limited to a predetermined range. The photographer looks at the confirmation image immediately after photographing and estimates the range of the subject distance that can be focused. However, it has been very difficult to grasp which range can be focused from an image obtained by focusing on a certain subject distance.

  In addition, it is assumed that the confirmation image after photographing is displayed in an out-of-focus state in spite of an area where an image in focus can be acquired by the blur restoration processing after photographing. In this case, even if there is a failure part that the photographer does not intend in that area, it is displayed in a blurred state, so the photographer is not aware of the confirmation immediately after shooting, and it is not until the blur is repaired after shooting. There was a problem of noticing.

  Therefore, in the imaging apparatus of the present invention, when the closest distance that can be corrected by the blur repair unit is the first distance, and the most infinite distance that can be corrected by the blur repair unit is the second distance, In the confirmation image display, a photographed image in which the range from the first distance to the second distance is corrected for blur is displayed. That is, an image that has been corrected for blur is displayed as much as possible.

  FIG. 13 shows a confirmation image display immediately after photographing according to the present invention. The captured images in FIG. 13 are the same as the captured images in FIGS. 8, 11, and 12. FIG. 13A shows a case where images in focus are displayed on the subject areas Target1 to Target3. FIG. 13B shows a case where the subject areas Target1 to 3 are in focus, and the detected subject position and the number of subjects are displayed superimposed.

  In FIG. 13A, the subject distances Dist1 to Dist3 of the subject regions Target1 to Target3 are located within the range of the first distance Dist11 and the second Dist12 where the blur can be repaired, so that the blur repair is possible. Therefore, in the imaging apparatus of the present invention, in the confirmation image display immediately after shooting, blur correction is performed for the subject areas Target1 to Target3.

  By performing the repair process based on the imaging device characteristic information and the photographing lens information corresponding to the subject distance Dist1 in the subject region Target1, the subject region Target1 is repaired out of focus and becomes an in-focus image. Further, by performing a repair process based on the imaging device characteristic information and the photographing lens information corresponding to the subject distance Dist2 in the subject region Target2, the subject region Target2 is repaired out of focus and becomes an in-focus image. Further, by performing the repair process based on the imaging device characteristic information and the photographing lens information corresponding to the subject distance Dist3 in the subject region Target3, the subject region Target3 is restored to be out of focus and becomes an in-focus image. That is, the subject area Target1, the subject area Target2, and the subject area Target3 that are located within a distance range where blur restoration is possible are all displayed in focus. In other words, an image in which a plurality of subject distances Dist1, Dist2, and Dist3 are in focus is displayed as a confirmation image display immediately after shooting. The background BackGround1 that is farther than the range where the blur can be repaired is displayed in a blurred state.

  FIG. 13B shows a state in which the detected subject position and the number of subjects are superimposed and displayed on the subject regions Target 1 to 3 in focus. SI1 is a subject position display indicating the position of the subject region Target1. Displaying a contour that is slightly larger than the subject area Target1 indicates the position of the subject area Target1. Similarly, SI2 is a subject position display indicating the position of the subject area Target2, and SI3 is a subject position display indicating the position of the subject area Target3. SI4 is a subject number display indicating the number of detected subjects. As a result, the photographer can easily grasp the position and number of the detected objects when displaying the confirmation image after photographing.

  By displaying the image as described in FIGS. 13A and 13B as a confirmation image after photographing, an imaging device having a photographed image function that allows the photographer to easily confirm the photographed image can be realized. Thereby, the photographer can easily grasp the range where the blur can be repaired. In addition, it is possible to more easily find a failure point in an area where an image that is in focus can be acquired by blur repair. Furthermore, by displaying the detected subject position and the number of subjects superimposed on the blurred image, the photographer can easily display the position and number of the detected subjects when displaying the confirmation image after shooting. Can grasp.

  14 to 19 are flowcharts for explaining the operation of the imaging apparatus of the present invention.

  FIG. 14 is a main flow of the imaging apparatus of the present invention. The main flow operation is a process performed by the CPU 121.

  When the photographer turns on the power switch of the camera (step S101), the CPU 121 checks the operation of each actuator and image sensor in the camera, initializes the memory contents and the execution program, and executes the shooting preparation operation. (Step S102). In step S103, lens communication is performed with the camera communication circuit in the photographing lens via the lens communication circuit 135. The lens operation is confirmed by lens communication, the memory contents in the lens and the execution program are initialized, and a preparatory operation is executed. Further, lens characteristic data necessary for focus detection and imaging is acquired and stored in the in-camera memory 144. In step S104, the imaging operation of the imaging device is started, and a low-resolution moving image for preview is output. In step S105, the read moving image is displayed on the display means 131 provided on the back side of the camera, and the photographer visually determines the composition at the time of photographing by viewing the preview image.

  In step S106, it is recognized whether or not a face exists in the preview moving image. Then, the number, position, and size of the face are detected from the preview moving image and recorded in the in-camera memory 144. Since the technique for recognizing a face is disclosed in Japanese Patent Application Laid-Open No. 2004-317699, etc., it is publicly known, and thus the description thereof is omitted here. The operation of detecting the subject in the photographed image in the claims corresponds to this step, and the subject detection means corresponds to the CPU 121.

  If it is determined in step S107 that a face exists in the shooting area, the process proceeds to step S108, and the focus adjustment mode is set to the face AF mode. Here, the face AF mode refers to an AF mode in which focusing is performed by taking into consideration both the position of the face in the shooting area and the subject distance map created in step S200.

  On the other hand, if it is recognized in step S107 that no face is present in the shooting area, the process proceeds from step S107 to step S109, and the focus adjustment mode is set to the multipoint AF mode. Here, the multi-point AF mode refers to, for example, the main subject analogized based on the focus detection result in each divided region calculated from the subject distance map and the luminance information of the subject. And a mode for focusing the area.

  If the AF mode is determined in step S108 or step S109, it is determined in step S110 whether or not the photographing preparation switch is turned on. If not, the process proceeds to step S116 and whether or not the main switch is turned off is determined. To do.

  When the photographing preparation switch is turned on in step S111, the process proceeds to step S200, and a subject distance map creation subroutine is executed.

  In step S111, the focus detection position is determined based on the subject distance map calculated in step S200. Here, the detection position determination method is given priority on the near side, and the position of the subject located closest to the subject obtained in step S200 is set as the focus detection position.

  In step S112, the defocus amount at the focus detection position determined in step S111 is calculated from the defocus map obtained in step S200, and it is determined whether the obtained defocus amount is equal to or less than an allowable value. If the defocus amount is greater than or equal to the allowable value, it is determined that the subject is out of focus, the focus lens is driven in step S113, and then the process returns to step S110 to determine whether or not the photographing preparation switch has been pressed. If it is determined in step S112 that the in-focus state has been reached, focus display is performed in step S114, and the process proceeds to step S115.

  In step S115, it is determined whether or not the shooting start switch has been turned on. If the switch has not been turned on, the shooting standby state is maintained in step S115. When the shooting start switch is turned on in step S115, the process proceeds to step S300, and a shooting subroutine is executed.

  When the step 300 photographing subroutine is completed, the process proceeds to step S116 to determine whether or not the main switch is turned off. If the main switch is not turned off, the process returns to step S103. When the main switch is turned off, the series of operations is terminated.

  FIG. 15 is a flowchart of a subject distance map creation subroutine. The CPU 121 also performs a series of operations of the subject distance map creation subroutine. The distance information acquisition means in the claims corresponds to the CPU 121.

  When jumping from step S200 of the main flow to step S200 of this subroutine, the focus detection area is set in step S201. A focus detection area is determined from the all focus detection areas, and the processes in and after step S202 are performed. In step S202, the focus detection pixel signal in the focus detection area set in step S201 is read. In step S203, two images for correlation calculation are generated. By arranging the signals of the focus detection pixels read out in step S202, the signals of the two images A and B for correlation calculation are obtained.

  In step S204, a correlation operation is performed based on the obtained image signal, and the phase difference between the two images A and B is calculated. In step S205, the reliability of the correlation calculation result is determined. Here, the reliability refers to the degree of coincidence between the images A and B. When the degree of coincidence between the images A and B is good, the reliability of the focus detection result is generally high. Therefore, the reliability of the phase difference detection result is determined based on whether or not the degree of coincidence exceeds a certain threshold, or when a plurality of focus detection areas are selected, information with high reliability is given priority. It can be used. In step S206, the amount of defocus is calculated by multiplying the phase difference between the images A and B obtained in step S204 by a conversion coefficient that converts the phase difference into the amount of defocus.

  In step S207, it is determined whether or not calculation of the defocus amount has been completed for the all focus detection region. If it is determined that the focus detection area has not been completed, the process returns to step S201 to select and set a focus detection area from the remaining focus detection areas. If it is determined in step S207 that the all-focus detection area is completed, the process proceeds to step S208.

  In step S208, a defocus amount map is created from the defocus amount in the all focus detection region obtained by repeating steps S201 to S207. Here, the defocus amount map is distribution data in which the position on the photographing screen is associated with the defocus amount.

  In step S209, the defocus map obtained in step S208 is converted from the defocus to the subject distance in consideration of the lens information acquired from the photographing lens 137 by the lens communication in step S103. This makes it possible to obtain distribution data in which the position on the shooting screen is associated with the subject distance.

  In step S210, the subject is extracted based on the subject distance distribution data. In the obtained distribution of subject distances, regions close to the subject distance are connected and grouped to extract the contour of the subject included in the shooting screen. As a result, a subject distance map in which each subject area is associated with the subject distance is obtained. The distance information in the claims corresponds to the subject distance map.

  When step S210 ends, the subject distance map creation subroutine ends, and the process proceeds to step S111 in the main flow.

  FIG. 16 is a flowchart of the photographing subroutine. The CPU 121 also performs a series of operations of the photographing subroutine.

  In step S301, the light amount adjusting diaphragm is driven to perform opening control of the mechanical shutter that defines the exposure time.

  In step S302, image reading for high-resolution still image shooting, that is, reading of all pixels is performed.

  In step S303, defective pixel interpolation of the read image signal is performed. That is, the output of the focus detection pixel does not have RGB color information for imaging, and corresponds to a defective pixel in obtaining an image. Therefore, an image signal is generated by interpolation from information on surrounding imaging pixels. .

  In step S304, image processing such as image gamma correction, color conversion, and edge enhancement is performed. In step S305, the captured image is recorded in the flash memory 133. The captured image recorded in step S305 is an image that is not subjected to blur repair. When replaying and displaying a recorded image, it is displayed with blur repair. Thereby, the calculation load of the imaging apparatus is reduced. The CPU 121 performs step S305 for recording a captured image. That is, the recording means in the claims corresponds to the CPU 121.

  In step 306, the characteristic information of the camera body 138 is recorded in the flash memory 133 and the in-camera memory 144 in correspondence with the captured image recorded in step S 305. Here, the characteristic information of the camera body 138 includes the light receiving sensitivity distribution information of the imaging pixels and focus detection pixels of the image sensor 107, the vignetting information of the photographing light flux in the camera body 138, and the attachment of the camera body 138 and the photographing lens 137. It includes optical characteristics such as distance information from the surface to the image sensor 107 and manufacturing error information. Since the light reception sensitivity distribution information of the imaging pixels and focus detection pixels of the imaging element 107 is determined by the on-chip microlens ML and the photoelectric conversion unit PD, these pieces of information may be recorded.

  In step S307, the characteristic information of the photographing lens 137 is recorded in the flash memory 133 and the in-camera memory 144 in correspondence with the photographed image recorded in step S305. Here, the characteristic information of the photographing lens 137 includes optical characteristics such as exit pupil EP information, frame information, F-number information at the time of photographing, aberration information, and manufacturing error information.

  In the image pickup apparatus of the present invention, as described in steps S305 to S307, the flash memory 133 records a shot image that is not subjected to blur repair, shooting lens characteristic information corresponding to the shot image, and image pickup apparatus characteristic information. Thereby, the blur of the photographed image can be corrected after the photographing based on the photographing lens characteristic information and the photographing characteristic information. In addition, it is possible to reduce the calculation load when recording a captured image. Then, the CPU 121 performs recording of the photographed image, the photographing lens characteristic information, and the imaging device characteristic information performed in steps S305 to S307.

  When step S307 ends, step S400 proceeds to a confirmation image display subroutine after shooting.

  When the confirmation image display subroutine after shooting in step S400 is completed, the shooting subroutine in step S300 is ended, and the process proceeds to step S116 of the main routine.

  FIG. 17 is a flowchart of the post-shooting confirmation image display subroutine. The CPU 121 also performs a series of operations in the post-shooting confirmation image display subroutine. The image display control means in the claims corresponds to the CPU 121.

  In step S401, the subject distance map created in step S200 is acquired.

  In step S402, a subject area and subject distance for which blur correction is to be performed are set. Step S200 As described in the subject distance map creation subroutine, information associating the subject area with the subject distance is obtained from the subject distance map. Further, as described with reference to FIG. 9, the distance at which the blur can be repaired differs depending on the photographing lens 137, and the first distance Dist11 that is the closest distance at which blur can be repaired, and the second distance Dist12 that is the most infinite distance. Will change. Accordingly, the subject located within the distance range (first distance Dist11 to second distance Dist12) where the subject distance is determined by the photographing lens 137 and is set to be blurred is set. Thereby, it is possible to set each subject area and a subject distance used when the area is subjected to blur repair.

  In addition, the first distance Dist11 and the second distance Dist12 that define the distance range in which the blur can be repaired are reset so that the amount of deviation from the imaging plane position of the photographing lens 137 is a predetermined amount or less. As a result, the amount of blur can be kept below a predetermined amount, and good blur restoration can be performed.

  Step S402 When the blur repair area / distance setting is completed, the process proceeds to a step S500 blur repair subroutine.

  In step S403, the image repaired in the blur in step S500 is displayed on the display unit 131 for a predetermined time. As an image to be displayed at this time, only the image that has been corrected for blur as shown in FIG. 13A is displayed, or the position of the subject and the position of the subject in the image that has been corrected for blur as shown in FIG. Either the number is overlaid or displayed. Which one is displayed is switched according to the set value input by the photographer with the operation switch 132.

  The operation for displaying the photographed image in the claims on the display means corresponds to this step. The operation of superimposing the position and number of subjects detected by the subject detection means in the claims on the captured image and displaying them on the display means also corresponds to this step.

  When step S403 ends, the post-shooting confirmation image display subroutine ends, and the process returns to the shooting subroutine.

  FIG. 18 is a flowchart of the blur repair subroutine. The CPU 121 also performs a series of operations of the blur repair subroutine. The blur restoration means in the claims corresponds to the CPU 121.

  In step S <b> 501, the CPU 121 acquires conversion information indicating the content of conversion processing in the image processing circuit 125.

  In step S502, the CPU 121 determines a conversion method for converting the image information supplied from the image processing circuit 125. Specifically, the CPU 121 converts the conversion method based on the conversion information acquired in step S501 (if necessary, the imaging device characteristics and the imaging lens characteristics information acquired in step S306 and step S307 in addition to the conversion information). To decide. In the conversion method determined here, in order to ensure linearity, which is a precondition for the algorithm of the image restoration process described in Patent Document 2, the image information is set in a proportional relationship between the exposure value and the pixel value. How to convert.

  For example, when gamma correction is executed by the image processing circuit 125, in step S502, inverse conversion of conversion by gamma correction is executed. As a result, the image before conversion can be reproduced, and an image having linearity can be acquired. Similarly, when color correction is executed by the image processing circuit 125, in step S502, inverse conversion of conversion by color conversion is executed. Thereby, it is possible to acquire an image having linearity. As described above, in step 502, a conversion method corresponding to the inverse conversion of the conversion processing by the image processing circuit 125 is determined.

  In step S <b> 503, a captured image is acquired from the image processing circuit 125. In step S504, the acquired captured image is converted according to the conversion method determined in step S502. When the conversion process ends in step S504, the process proceeds to step S600, and a blur function is generated.

  In step S505, the blur function generated in step S600 is inversely transformed to perform a blur restoration process on the captured image converted in step S504. Here, the blur restoration process is performed by an image restoration algorithm generally called a deconvolution process. As a result, it is possible to obtain a blur repair image in which the blur of the predetermined subject is repaired. Since a method for restoring blur by performing inverse transformation processing of a blur function is disclosed in Patent Document 2, description thereof is omitted.

  When step S505 ends, the blur restoration subroutine ends, and the process proceeds to step S403 in the post-shooting confirmation image display subroutine.

  FIG. 19 is a flowchart of the blur function generation subroutine. The CPU 121 also performs a series of operations of the blur function generation subroutine.

  In step S601, the characteristic information of the camera main body 138 recorded in the camera memory 144 in step S305 at the time of shooting is acquired.

  In step S602, the characteristic information of the photographing lens 137 recorded in the camera memory 144 in step S306 at the time of photographing is acquired.

  In step S603, a parameter used when defining the blur function is acquired. The blur function is determined by the light transfer characteristic between the photographing lens 137 and the image sensor 107. The light transmission means varies depending on factors such as the characteristic information of the camera body 138, the characteristic information of the photographing lens 137, the position of the subject area in the photographed image, and the subject distance. Therefore, table data in which these factors are associated with parameters used when defining the blur function is stored in the in-camera memory 144. When step S603 is executed, the CPU 121 acquires parameters used when defining the blur function from the in-camera memory 144 based on these factors.

ステップＳ６０４が終わったら、ボケ関数生成サブルーチンを終了し、ボケ修復サブルーチン内のステップＳ５０５に進む。 In step S604, a blur function is defined based on the blur parameter acquired in step S603. As an example of the blur function, there is a Gaussian distribution considered that the blur phenomenon follows the normal distribution law. The blur function h (r) is given as follows, where r is the distance from the center pixel, and σ2 is an arbitrary parameter of the normal distribution law.

When step S604 ends, the blur function generation subroutine ends, and the process proceeds to step S505 in the blur repair subroutine.

  In the image pickup apparatus of the present invention, the camera that can replace the photographing lens has been described. The conventional problem also exists in the lens-attached type camera. Then, as described in the present invention, the same effect can be obtained by displaying the photographed image that has been corrected for blur.

  As described above, according to the imaging apparatus of the present invention, it is possible to realize an imaging apparatus in which a photographer can easily check a captured image in an imaging apparatus having a blur restoration unit.

[Example 2]
FIG. 20 is a diagram related to Example 2 of the present invention. Hereinafter, the operation of the second embodiment will be described with reference to the drawings.

  In the first embodiment, the defocus amounts Def21 and Def22 corresponding to the blur amount that can be corrected for blur are the defocus amounts Def11 at the imaging plane position of the first distance Dist11 on the closest side and the second distance Dist12 on the infinite side. It existed outside Def12. For this reason, the first distance Dist11 on the near side and the second distance Dist12 on the infinity side were not updated.

  On the other hand, in the second embodiment, the defocus amounts Def21 and Def22 corresponding to the blur amount that can be corrected for blur are the defocus amounts at the image plane position of the first distance Dist11 on the near side and the second distance Dist12 on the infinite side. It exists inside Def11 and Def12. Therefore, the first distance Dist11 on the near side and the second distance Dist12 on the infinity side are updated according to the defocus amounts Def12 and Def22, which is different from the first embodiment.

  FIG. 20 shows the relationship between the imaging plane position corresponding to the subject distances Dist1, Dist2, Dist3, and Dist4 and the imaging plane position of the taking lens 137. The axis represents the defocus amount Def from the imaging plane position of the photographing lens 137. Here, the plus direction of the defocus amount Def corresponds to the rear pin direction.

  “0” indicates the image plane position of the photographic lens 137, and the amount of deviation is expressed as “0”. Def1, Def2, Def3, and Def4 are defocus amounts at the imaging plane positions of the subject distances Dist1, Dist2, Dist3, and Dist4. Def11 and Def12 are defocus amounts at the imaging plane positions of the first distance Dist11 on the near side and the second distance Dist12 on the infinity side.

  On the other hand, Def21 and Def22 are defocus amounts corresponding to blur amounts that can be corrected for blur. When the amount of blur is too large, extreme conversion processing is performed at the time of restoring a blurred image, and noise is likely to occur. For this reason, when restoring the blur, it is desirable that the blur amount be within the range of Def21 and Def22. Now, Def11 and Def12 are located outside of Def21 and Def22, so there is a possibility that they cannot be restored well.

  Therefore, the first distance Dist11 on the near side is corrected so that the defocus amount Def11 at the first distance Dist11 on the near side becomes the defocus amount corresponding to the defocus amount Def21. Similarly, the infinite second distance Dist12 is corrected so that the defocus amount Def12 at the infinite second distance Dist12 becomes the defocus amount corresponding to the defocus amount Def22. Thereby, since the amount of blur is too large, it is possible to alleviate problems such as generation of noise due to extreme conversion processing at the time of blur image restoration. In other words, it is possible to perform excellent blur repair.

  The operation of updating the first distance Dist11 on the near side and the second distance Dist12 on the infinity side according to the defocus amounts Def12 and Def22 is performed in step S402 of the post-shooting confirmation image display subroutine (FIG. 17). .

  In step S402 in FIG. 17, a subject area and subject distance for which blur correction is performed are set. Step S200 As described in the subject distance map creation subroutine, information associating the subject area with the subject distance is obtained from the subject distance map. The subject located within the distance range (first distance Dist11 to second distance Dist12) where the subject distance is determined by the photographing lens 137 and can be restored is set so as to perform the blur restoration. Further, the distance is reset so that the first distance Dist11 and the second distance Dist12, which are distance ranges that can be corrected for blurring, are less than a predetermined amount of deviation from the imaging plane position of the photographing lens 137. As a result, the amount of blur can be kept below a predetermined amount, and good blur restoration can be performed.

  Since the configuration of the imaging device, the imaging device, and the pixel structure are the same as those in the first embodiment, description thereof is omitted. Since the relationship between the distance information obtained by the distance information acquisition means, the subject distance, and the distance at which the blur can be repaired is the same as in the first embodiment, the description thereof is omitted. Since the operation flow of the imaging apparatus of the second embodiment is also the same as that of the first embodiment, the description thereof is omitted.

  As described above, according to the second embodiment of the present invention, it is possible to realize an imaging apparatus in which a photographer can easily check a captured image in an imaging apparatus having a blur repair unit.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

101 First lens group 103 Second lens group 107 Image sensor 121 CPU
124 Image sensor drive circuit 131 Display means 137 Shooting lens 138 Camera body OBJ Subject IMG Subject image SHA, SHB, SVC, SVD Focus detection pixel
Dist11 First distance determined based on the closest distance end of the photographic lens 137
Dist12 Second distance determined based on the infinite distance end of the photographic lens 137
Dist1, Dist2, Dist3, Dist4 Subject distance
Target1, Target2, Target3 Subject area
Def1, Def2, Def3, Def4 Defocus amount at image plane position with subject distance Dist1-4
Def11, Def12 Defocus amount at image plane position at first distance Dist11 and second distance Dist12
Def21, Def22 Defocus amount SI1, SI2, SI3 corresponding to the amount of blur that can be corrected. Display indicating the position of the subject areas Target1, Target2, Target3 SI4 Display indicating the number of subjects

Claims (22)

An acquisition means for acquiring distance information corresponding to a subject distance in a captured image captured by an image sensor that receives a light flux from a photographing lens;
Image processing means for performing image processing for changing a focus state on the captured image based on distance information obtained by the acquisition means;
Display control means for displaying on the display means the image processed by the image processing means,
The image processing means generates an image in which the focus state of the distance range corresponding to the subject distance from the first distance to the second distance is changed ,
The image processing, the image processing apparatus according to claim image processing der Rukoto based on the light transmission characteristics of the imaging lens and the imaging element. An acquisition means for acquiring distance information corresponding to a subject distance in a captured image captured by an image sensor that receives a light flux from a photographing lens;
Image processing means for performing image processing for changing a focus state on the captured image based on distance information obtained by the acquisition means;
Display control means for displaying on the display means the image processed by the image processing means,
The image processing means generates an image in which the focus state of the distance range corresponding to the subject distance from the first distance to the second distance is changed,
The image processing apparatus is characterized in that the image processing is deconvolution processing. An acquisition means for acquiring distance information corresponding to a subject distance in a captured image captured by an image sensor that receives a light flux from a photographing lens;
Image processing means for performing image processing for changing a focus state on the captured image based on distance information obtained by the acquisition means;
Display control means for displaying on the display means the image processed by the image processing means,
The image processing means generates an image in which the focus state of the distance range corresponding to the subject distance from the first distance to the second distance is changed,
The image processing means determines distance information corresponding to the first distance based on a closest distance end of the photographing lens, and determines distance information corresponding to the second distance based on an infinite distance end of the photographing lens. images processor wherein the deciding. And the image plane position corresponding to the second distance and the first distance, as the deviation amount between the imaging plane position of the taking lens is equal to or less than a predetermined amount, corresponding to the second distance and the first distance the image processing apparatus according to any one of claims 1 to 3, characterized in that setting the distance range. An acquisition means for acquiring distance information corresponding to a subject distance in a captured image captured by an image sensor that receives a light flux from a photographing lens;
Image processing means for performing image processing for changing a focus state on the captured image based on distance information obtained by the acquisition means;
Display control means for displaying on the display means the image processed by the image processing means,
The image processing apparatus, wherein the display control unit displays a range in which a focus state can be changed by the image processing unit after the captured image is captured. The image processing means generates an image in which the focus state of the range in which the captured image can change the focus state is changed immediately after shooting by the image sensor,
The image processing apparatus according to claim 5 , wherein the display control unit displays the image on the display unit. Any of claims 1 to 6, wherein a first photoelectric conversion unit and the second plurality comprises the imaging device a plurality of pixels in which the photoelectric conversion unit is allocated including a photoelectric conversion unit to one microlens whether the image processing apparatus according to (1). The acquisition unit, an image processing apparatus according to claim 7, characterized in that to obtain the pre Ki距 release information by detecting the phase difference between the first and second signals from the photoelectric conversion unit. The image processing apparatus according to claim 8, wherein the acquisition unit determines the reliability of the distance information based on signals from the first and second photoelectric conversion units. The image processing apparatus according to claim 9, wherein the acquisition unit determines the reliability of the distance information based on a degree of coincidence of image signals from the first and second photoelectric conversion units. And the images picked up by the image pickup device, an image according to any one of claims 1 to 10, characterized in that it comprises a recording means for recording the characteristic information at the time of photographing of the photographic image on a recording medium Processing equipment. The image processing apparatus according to claim 11 , wherein the characteristic information includes F number information. The characteristic information, the image processing apparatus according to claim 11 or 12, characterized in that it comprises an aberration information. Wherein the image processing means, with respect to the recorded photographic image on the recording medium, any one of claims 11 to 1 3, characterized in that performing the image processing based on the recorded characteristic information on the recording medium The image processing apparatus according to item 1. Subject detection means for detecting a subject in the captured image;
Wherein the display control unit, wherein the position of the detected object by the object detection unit, in any one of claims 1 to 1 4, characterized in that displayed on the display means by superimposing the photographed image Image processing apparatus. Setting means for setting distance information corresponding to the subject distance for changing the focus state of the captured image;
Wherein the image processing means, the image processing apparatus according to any one of claims 1 to 1 5, characterized in that to change the focus state of the photographed image based on the setting by the setting means. An imaging device that receives a light beam from a photographic lens and includes a plurality of pixels in which a plurality of photoelectric conversion units including a first photoelectric conversion unit and a second photoelectric conversion unit are assigned to one microlens;
Acquisition means for acquiring distance information corresponding to a subject distance in a captured image captured by the imaging element;
Image processing means for performing image processing for changing a focus state on the captured image based on distance information obtained by the acquisition means;
Display control means for displaying on the display means the image processed by the image processing means,
The image processing means generates an image in which the focus state of the distance range corresponding to the subject distance from the first distance to the second distance is changed ,
The image processing, the image processing apparatus according to claim image processing der Rukoto based on the light transmission characteristics of the imaging lens and the imaging element. An imaging device that receives a light beam from a photographic lens and includes a plurality of pixels in which a plurality of photoelectric conversion units including a first photoelectric conversion unit and a second photoelectric conversion unit are assigned to one microlens;
Acquisition means for acquiring distance information corresponding to a subject distance in a captured image captured by the imaging element;
Image processing means for performing image processing for changing a focus state on the captured image based on distance information obtained by the acquisition means;
Display control means for displaying on the display means the image processed by the image processing means,
The image processing means generates an image in which the focus state of the distance range corresponding to the subject distance from the first distance to the second distance is changed,
The image processing apparatus is characterized in that the image processing is deconvolution processing. An imaging device that receives a light beam from a photographic lens and includes a plurality of pixels in which a plurality of photoelectric conversion units including a first photoelectric conversion unit and a second photoelectric conversion unit are assigned to one microlens;
Acquisition means for acquiring distance information corresponding to a subject distance in a captured image captured by the imaging element;
Image processing means for performing image processing for changing a focus state on the captured image based on distance information obtained by the acquisition means;
Display control means for displaying on the display means the image processed by the image processing means,
The image processing apparatus, wherein the display control unit displays a range in which a focus state can be changed by the image processing unit after the captured image is captured. An acquisition step of acquiring distance information corresponding to a subject distance in a photographed image photographed by an image sensor that receives a light flux from a photographing lens;
Based on the distance information obtained in the acquisition step, an image processing step for performing image processing for changing a focus state on the captured image;
A display control step of displaying on the display means the image processed in the image processing step,
In the image processing step, an image in which the focus state of the distance range corresponding to the subject distance from the first distance to the second distance is changed is generated ,
The image processing method for controlling an image processing apparatus according to claim image processing der Rukoto based on the light transmission characteristics of the imaging lens and the imaging element. An acquisition step of acquiring distance information corresponding to a subject distance in a photographed image photographed by an image sensor that receives a light flux from a photographing lens;
Based on the distance information obtained in the acquisition step, an image processing step for performing image processing for changing a focus state on the captured image;
A display control step of displaying on the display means the image processed in the image processing step,
In the image processing step, an image in which the focus state of the distance range corresponding to the subject distance from the first distance to the second distance is changed is generated,
The method for controlling an image processing apparatus, wherein the image processing is deconvolution processing. An acquisition step of acquiring distance information corresponding to a subject distance in a photographed image photographed by an image sensor that receives a light flux from a photographing lens;
Based on the distance information obtained in the acquisition step, an image processing step for performing image processing for changing a focus state on the captured image;
A display control step of displaying on the display means the image processed in the image processing step,
In the display control step, a range in which the focus state can be changed in the image processing step after the captured image is captured is displayed.

Images