JP2020036288A - Image processing apparatus and electronic apparatus - Google Patents

Abstract

To make it possible to check a focus state of a plurality of captured images in an area according to user's intention.SOLUTION: An image processing apparatus 121 is configured to: set a region of interest in a first image among a plurality of captured images obtained by imaging at mutually different time; and detect a corresponding region having relevance to the region of interest in a plurality of second images among the plurality of captured images. A focus state of each region of interest and corresponding area is obtained and a classification process for classifying the captured images into a plurality of groups according to the focus state is performed.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing device that performs image processing on a captured image.

  2. Description of the Related Art Some electronic devices such as a digital camera and a camera-equipped mobile phone can perform an autofocus (hereinafter, referred to as an imaging surface phase difference AF) of an imaging surface phase difference detection method, as disclosed in Japanese Patent Application Laid-Open No. H11-163873. Patent Document 1 discloses a method of classifying and recording a plurality of captured images obtained by continuous shooting depending on whether or not the images are focused images.

JP 2011-209450 A

  However, the region in which the focus state is to be determined in the captured image (focus determination region) differs depending on the intention of the user. For example, when imaging the face of a person, the user arbitrarily selects which of the right eye and the left eye should be focused, and which of the eyelashes and the eye should be focused. On the other hand, Patent Literature 1 does not disclose a method of selecting a focus determination area according to a user's intention.

  The present invention provides an image processing apparatus capable of confirming a focus state in a region according to a user's intention with respect to a plurality of captured images, and an electronic apparatus including the image processing apparatus.

  An image processing apparatus according to one aspect of the present invention includes: an area setting unit configured to set a region of interest in a first image among a plurality of captured images acquired by capturing images at different times from each other; Area detecting means for detecting a corresponding area having relevance to the area of interest in the second image, focus state obtaining means for obtaining the respective focus states of the area of interest and the corresponding area, and setting the plurality of captured images to the focused state Processing means for performing a classification process for classifying the data into a plurality of groups in response to the request.

  Note that an imaging device having an imaging element for imaging a subject image and the image processing device also constitutes another aspect of the present invention.

  Further, an image processing method according to another aspect of the present invention includes a step of setting a region of interest in a first image among a plurality of captured images acquired by capturing images at different times from each other; Detecting a corresponding region having a relevance to the region of interest in the plurality of second images, obtaining a focus state of each of the region of interest and the corresponding region; And performing a classification process for classifying into a group.

  Note that a computer program that causes a computer to execute a process according to the image processing method also constitutes another aspect of the present invention.

  ADVANTAGE OF THE INVENTION According to this invention, the confirmation of the focus state in the area | region according to the user's intention can be made about several imaging images.

FIG. 1 is a block diagram illustrating a configuration of a camera that is Embodiment 1 of the present invention. FIG. 2 is a diagram illustrating a pixel array in the camera according to the first embodiment. 2A and 2B are a plan view and a cross-sectional view of a pixel in Embodiment 1. FIG. 2 is an explanatory diagram of a pixel structure according to the first embodiment. FIG. 4 is an explanatory diagram of pupil division in the first embodiment. FIG. 4 is a diagram illustrating a relationship between a defocus amount and an image shift amount according to the first embodiment. FIG. 3 is a diagram illustrating a focus detection area according to the first embodiment. 6 is a flowchart illustrating AF / imaging processing of the camera according to the first embodiment. 6 is a flowchart illustrating an imaging subroutine of the camera according to the first embodiment. 9 is a flowchart illustrating a focus detection subroutine during continuous shooting in the camera according to the first embodiment. 5 is a flowchart illustrating image reproduction processing in the camera according to the first embodiment. FIG. 4 is a diagram illustrating a method for setting a region of interest from a display image according to the first embodiment. FIG. 2 is a rear view of the camera according to the first embodiment. 5 is a flowchart illustrating a first grouping subroutine in the camera according to the first embodiment. 5 is a flowchart illustrating a focus state detection subroutine in the camera according to the first embodiment. FIG. 4 is a diagram illustrating a second evaluation area according to the first embodiment. FIG. 7 is a diagram for explaining a method of updating a detection result when the reliability of the result obtained from the first evaluation area is low in the first embodiment. 9 is a flowchart illustrating a second grouping subroutine in the camera according to the first embodiment. FIG. 4 is a diagram illustrating an image group subjected to first grouping in the first embodiment. FIG. 9 is a diagram illustrating a result of the second grouping in the first embodiment. FIG. 6 is a diagram illustrating a reproduced image after the second grouping in the first embodiment. FIG. 4 is a diagram illustrating a focus state of a focused area or a corresponding area of the image group on which the first grouping has been performed in the first embodiment. FIG. 14 is a diagram illustrating a focus state of a focused area or a corresponding area of an image group on which a first grouping has been performed in a modified example. 9 is a flowchart illustrating a second grouping subroutine of the camera that is Embodiment 2 of the present invention. FIG. 8 is a diagram for explaining a method of acquiring distance information of a subject region in the camera according to the second embodiment. FIG. 9 is a diagram illustrating a difference in depth of field depending on a subject distance in the second embodiment. FIG. 13 is a diagram illustrating a focus state of a focused region or a corresponding region of an image group on which first grouping has been performed in the second embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a configuration of a camera 100 as an electronic apparatus or an imaging apparatus equipped with an image processing apparatus according to a first embodiment of the present invention. In FIG. 1, a first lens group 101 is disposed closest to a subject (front side) in an imaging optical system as an imaging optical system, and is held so as to be movable in an optical axis direction. The aperture and shutter 102 adjusts the light amount by adjusting the aperture diameter thereof, and also functions as a shutter for controlling the exposure time at the time of capturing a still image. The second lens group 103 moves in the optical axis direction integrally with the stop / shutter 102 and performs magnification (zoom) together with the first lens group 101 moving in the optical axis direction.

  The third lens group (focus lens) 105 moves in the optical axis direction to perform focus adjustment. The optical low-pass filter 106 is an optical element for reducing false colors and moire of a captured image. The first lens group 101, the aperture / shutter 102, the second lens group 103, the third lens group 105, and the optical low-pass filter 106 constitute an imaging optical system.

  The zoom actuator 111 rotates a cam cylinder (not shown) around the optical axis, and moves the first lens group 101 and the second lens group 103 in the optical axis direction by a cam provided on the cam cylinder to change the position. Make a doubling. The aperture shutter actuator 112 drives a plurality of light-shielding blades (not shown) in the opening and closing directions for the operation of adjusting the amount of light of the aperture and shutter 102 and the shutter operation. The focus actuator 114 moves the third lens group 105 in the optical axis direction to perform focus adjustment.

  The focus drive circuit 126 drives the focus actuator 114 in response to a focus drive command from the camera CPU 121, and moves the third lens group 105 in the optical axis direction. The aperture shutter drive circuit 128 drives the aperture shutter actuator 112 in response to an aperture or shutter drive command from the camera CPU 121. The zoom drive circuit 129 drives the zoom actuator 111 according to a zoom operation by the user.

  In this embodiment, a case will be described in which the imaging optical system, the actuators 111, 112, 114, and the driving circuits 126, 128, 129 are provided integrally with the camera body including the imaging element 107. However, an interchangeable lens having an imaging optical system, actuators 111, 112, 114 and drive circuits 126, 128, 129 may be detachable from the camera body.

  The electronic flash 115 has a light-emitting element such as a xenon tube or an LED, and emits light for illuminating a subject. The AF auxiliary light emitting unit 116 has a light emitting element such as an LED, and projects an image of a mask having a predetermined opening pattern onto the subject through a light projecting lens, thereby achieving focus detection performance for a dark or low-contrast subject. Improve. The electronic flash control circuit 122 controls the electronic flash 115 to be turned on in synchronization with the imaging operation. The auxiliary light driving circuit 123 controls the AF auxiliary light emitting unit 116 to light up in synchronization with the focus detection operation.

  The camera CPU 121 controls various controls in the camera 100. The camera CPU 121 includes an arithmetic unit, a ROM, a RAM, an A / D converter, a D / A converter, a communication interface circuit, and the like. The camera CPU 121 drives various circuits in the camera 100 according to a computer program stored in the ROM, and controls a series of operations such as AF, imaging, image processing, and recording. The camera CPU 121 functions as an image processing device.

  The imaging element 107 is composed of a two-dimensional CMOS photosensor including a plurality of pixels and its peripheral circuits, and is arranged on an imaging plane of the imaging optical system. The imaging element 107 photoelectrically converts a subject image formed by the imaging optical system. The image sensor driving circuit 124 controls the operation of the image sensor 107, A / D converts an analog signal generated by photoelectric conversion, and transmits a digital signal to the camera CPU 121. The image processing circuit 125 performs image processing such as pixel defect correction, γ conversion, color interpolation, and JPEG compression on the image data as a digital signal from the image sensor driving circuit 124 to perform live view image display and recording. To generate image data used as a captured image for use.

  The display (display unit) 131 includes a display element such as an LCD, and displays information on an imaging mode of the camera 100, a preview image before imaging, a confirmation image after imaging, an index of a focus detection area, a focused image, and the like. I do. The operation switch group 132 includes a main (power) switch, a release (photographing trigger) switch, a zoom operation switch, a photographing mode selection switch, and the like, and is operated by the user. The flash memory 133 records a captured image. The flash memory 133 is detachable from the camera 100.

  Next, the image arrangement of the image sensor 107 will be described with reference to FIG. FIG. 2 illustrates a pixel array in a range of 4 pixel columns × 4 pixel rows in the image sensor 107 when viewed from the optical axis direction (z direction).

  One pixel unit 200 includes four imaging pixels arranged in 2 rows × 2 columns. By arranging a large number of pixel units 200 on the image sensor 107, a two-dimensional subject image can be photoelectrically converted. In one pixel unit 200, an imaging pixel (hereinafter, referred to as an R pixel) 200R having an R (red) spectral sensitivity is disposed at the upper left, and an imaging pixel (having a G (green) spectral sensitivity at the upper right and lower left is provided. Hereafter, 200 G are arranged. Further, an imaging pixel (hereinafter, referred to as a B pixel) 200B having a spectral sensitivity of B (blue) is arranged at the lower right. Each imaging pixel includes a first focus detection pixel 201 and a second focus detection pixel 202 divided in the horizontal direction (x direction).

  In the imaging element 107 of this embodiment, the pixel pitch P of the imaging pixels is 4 μm, and the number N of imaging pixels is 5575 horizontal (x) × 3725 vertical (y) rows = approximately 207.5 million pixels. The pixel pitch PAF of the focus detection pixels is 2 μm, and the number NAF of focus detection pixels is 11150 horizontal rows × 3725 vertical rows = about 41.5 million pixels.

  In the present embodiment, a case where each imaging pixel is divided into two in the horizontal direction will be described, but it may be divided in the vertical direction. In addition, the image sensor 107 of the present embodiment has a plurality of imaging pixels each including the first and second focus detection pixels, but the imaging pixels and the first and second focus detection pixels may be provided as separate pixels. . For example, the first and second focus detection pixels may be discretely arranged among a plurality of imaging pixels.

  FIG. 3A shows one imaging pixel (200R, 200G, 200B) viewed from the light receiving surface side (+ z direction) of the imaging element 107. FIG. 3B shows an aa cross section of the imaging pixel of FIG. 3A viewed from the −y direction. As shown in FIG. 3B, one imaging lens is provided with one microlens 305 for collecting incident light.

  Further, the imaging pixels are provided with photoelectric conversion units 301 and 302 that are divided into N parts (two parts in this embodiment) in the x direction. The photoelectric conversion units 301 and 302 correspond to the first focus detection pixel 201 and the second focus detection pixel 202, respectively. The centers of gravity of the photoelectric conversion units 301 and 302 are eccentric to the -x side and + x side with respect to the optical axis of the microlens 305, respectively.

  An R, G or B color filter 306 is provided between the micro lens 305 and the photoelectric conversion units 301 and 302 in each imaging pixel. Note that the spectral transmittance of the color filter may be changed for each photoelectric conversion unit, or the color filter may be omitted.

  Light that has entered the imaging pixels from the imaging optical system is condensed by the microlens 305, is separated by the color filter 306, is received by the photoelectric conversion units 301 and 302, and is photoelectrically converted here.

  Next, the relationship between the pixel structure shown in FIG. 3 and pupil division will be described with reference to FIG. FIG. 4 shows an a-a section of the imaging pixel shown in FIG. 3A as viewed from the + y side, and also shows an exit pupil of the imaging optical system. In FIG. 4, the x and y directions of the imaging pixels are inverted with respect to FIG. 3B in order to correspond to the coordinate axes of the exit pupil.

  A first pupil region 501 whose center of gravity is decentered on the + X side of the exit pupil is a region that is substantially conjugated with the light receiving surface of the photoelectric conversion unit 301 on the −x side of the imaging pixel by the microlens 305. The light beam that has passed through the first pupil region 501 is received by the photoelectric conversion unit 301, that is, the first focus detection pixel 201. A second pupil region 502 of the exit pupil, the center of gravity of which is decentered on the −X side, is a region that is approximately conjugated with the light receiving surface of the photoelectric conversion unit 302 on the + x side of the imaging pixel by the microlens 305. is there. The light beam that has passed through the second pupil region 502 is received by the photoelectric conversion unit 302, that is, the second focus detection pixel 202. The pupil region 500 indicates a pupil region in which the entire imaging pixel including the photoelectric conversion units 301 and 302 (the first and second focus detection pixels 201 and 202) can receive light.

  FIG. 5 shows pupil division by the image sensor 107. A pair of light beams that have passed through the first pupil region 501 and the second pupil region 502 respectively enter each pixel of the image sensor 107 at different angles, and are divided by the first and second focus detection pixels 201 and 202 into two. Received. In the present embodiment, a first focus detection signal is generated by collecting output signals from the plurality of first focus detection pixels 201 of the image sensor 107, and a second focus detection signal is generated by collecting output signals from the plurality of second focus detection pixels 202. Generate a focus detection signal. Further, an output signal from the first focus detection pixel 201 of the plurality of imaging pixels and an output signal from the second focus detection pixel 202 are added to generate an imaging pixel signal. Then, an image pickup signal for generating an image having a resolution corresponding to the number N of effective pixels is generated by synthesizing image pickup pixel signals from a plurality of image pickup pixels.

  Next, the relationship between the defocus amount of the imaging optical system and the phase difference (image shift amount) between the first focus detection signal and the second focus detection signal acquired from the image sensor 107 will be described with reference to FIG. An imaging element 107 is disposed on an imaging plane 600 in the figure, and as described with reference to FIGS. 4 and 5, the exit pupil of the imaging optical system includes a first pupil region 501 and a second pupil region 502. Is divided into two. The defocus amount d is such that the distance (magnitude) from the image forming position C of the light flux from the object (801, 802) to the imaging surface 600 is | d |, and the image forming position C is closer to the object than the imaging surface 600. The front focus state is defined by a minus sign (d <0), and the rear focus state where the image forming position C is on the opposite side of the imaging surface 600 from the subject is defined by a plus sign (d> 0). In an in-focus state where the imaging position C is on the imaging surface 600, d = 0. The imaging optical system is in focus (d = 0) with respect to the subject 801 and in front focus (d <0) with respect to the subject 802. The front focus state (d <0) and the rear focus state (d> 0) are collectively referred to as a defocus state (| d |> 0).

  In the front focus state (d <0), among the light beams from the subject 802, the light beams that have passed through the first pupil region 501 (the second pupil region 502) are once collected, and then moved to the center of gravity G1 (G2) of the light beams. The center spreads over a width # 1 (# 2) and forms a blurred image on the imaging surface 600. The blurred image is received by each first focus detection pixel 201 (each second focus detection pixel 202) on the image sensor 107, and a first focus detection signal (second focus detection signal) is generated. That is, the first focus detection signal (the second focus detection signal) is a signal representing the subject image in which the subject 802 is blurred by the blur width Γ1 (Γ2) at the center of gravity G1 (G2) of the light beam on the imaging surface 600. Become.

  The blur width Γ1 (Γ2) of the subject image increases substantially in proportion to the increase of the magnitude | d | of the defocus amount d. Similarly, the magnitude | p | of the image shift amount p (= difference G1-G2 of the center of gravity of the light flux) between the first focus detection signal and the second focus detection signal is also the magnitude | d | of the defocus amount d. Increases almost in proportion to the increase in Even in the back focus state (d> 0), the image shift direction between the first focus detection signal and the second focus detection signal is opposite to that in the front focus state, but is the same.

  As described above, the magnitude of the image shift amount between the first and second focus detection signals increases as the magnitude of the defocus amount increases. In the present embodiment, focus detection is performed by an imaging surface phase difference detection method that calculates a defocus amount from an image shift amount between the first and second focus detection signals obtained by using the imaging element 107.

  Next, a focus detection area of the image sensor 107 for obtaining the first and second focus detection signals will be described with reference to FIG. In FIG. 7, A (n, m) is the focus detection area in the x direction among a plurality of focus detection areas (three in the x direction and the y direction, nine in total) set in the effective pixel area 1000 of the image sensor 107. The nth and mth focus detection areas in the y direction are shown. First and second focus detection signals are generated from output signals from the plurality of first and second focus detection pixels 201 and 202 included in the focus detection area A (n, m). I (n, m) indicates an index for displaying the position of the focus detection area A (n, m) on the display 131.

  Note that the nine focus detection areas shown in FIG. 7 are merely examples, and the number, position, and size of the focus detection areas are not limited. For example, an area in a predetermined range centered on the position designated by the user may be set as the focus detection area.

  The flowchart in FIG. 8 illustrates an AF / imaging process (image processing method) that causes the camera 100 of the present embodiment to perform an AF operation and an imaging operation. More specifically, a process for causing the camera 100 to perform an operation from before image capturing for displaying a live view image on the display 131 to image capturing for a still image is shown. The camera CPU 121 as a computer executes this processing according to a computer program. In the following description, S means a step.

  First, in S1, the camera CPU 121 causes the image sensor driving circuit 124 to drive the image sensor 107, and acquires image data from the image sensor 107.

  Next, in S2, the camera CPU 121 extracts the first and second focus detection pixels from the plurality of first and second focus detection pixels included in each of the nine focus detection areas shown in FIG. Obtain a focus detection signal. Further, the camera CPU 121 generates an imaging signal by adding the first and second focus detection signals of all effective pixels of the imaging element 107, and causes the image processing circuit 125 to perform image processing on the imaging signal (imaging data). Get image data. When the imaging pixels and the first and second focus detection pixels are separately provided, the camera CPU 121 performs complement processing on the focus detection pixels to obtain image data.

  Next, in S3, the camera CPU 121 causes the image processing circuit 125 to generate a live view image from the image data obtained in S2, and causes the display 131 to display the live view image. The live view image is a reduced image that matches the resolution of the display 131, and the user can adjust the imaging composition, exposure conditions, and the like while viewing the image.

  Next, in S4, the camera CPU 121 calculates the image shift amount between the first and second focus detection signals obtained in each of the nine focus detection regions acquired in S2, and calculates the image shift amount for each focus detection region from the image shift amount. Is calculated.

  Next, in S5, the camera CPU 121 determines whether or not the switch Sw1 for instructing the start of the imaging preparation operation has been turned on by half-pressing the release switch included in the operation switch group 132. If Sw1 is not turned on, the camera CPU 121 proceeds to S11. On the other hand, when Sw1 is turned on, the camera CPU 121 proceeds to S6, and sets a focus detection area for obtaining a focused state (hereinafter, referred to as a focus target area). Here, the focus detection area selected by the user may be set as the focus target area, or may be determined based on the defocus amounts of the nine focus detection areas calculated in S4 and the distances of the focus detection areas from the center of the imaging range. Thus, the camera CPU 121 may automatically set the focus target area.

  The camera CPU 121 that has proceeded from S6 to S7 drives the focus lens 105 to a position where a focused state can be obtained (a focused position) based on the defocus amount detected in the set focusing target area.

  Next, in S8, the camera CPU 121 again performs the same imaging data acquisition processing as in S1 and the same focus detection processing as in S4.

  Then, the camera CPU 121 proceeds to S9, and determines whether or not the switch Sw2 for instructing the start of the imaging operation has been turned on by the full-press operation of the release switch. If Sw2 is not turned on, the camera CPU 121 returns to S5. On the other hand, if Sw2 is turned on, the process proceeds to S10, and it is determined whether or not to perform image recording. In the present embodiment, the image acquisition process during continuous shooting (continuous imaging) is switched between for image recording and for focus detection. This switching may be performed alternately, or an image acquisition process for image recording may be performed twice in three times, and an image acquisition process for focus detection may be performed once. Thus, highly accurate focus detection can be performed without greatly reducing the number of times of imaging (the number of captured images) per unit time.

  When performing image recording in S10, the camera CPU 121 proceeds to S300 and executes an imaging subroutine. Details of the imaging subroutine will be described later. When the imaging subroutine ends, the camera CPU 121 returns to S9 and determines whether Sw2 is turned on, that is, whether continuous shooting is instructed.

  On the other hand, when it is determined in S10 that image recording is not to be performed, that is, focus detection is to be performed, the camera CPU 121 proceeds to S400 and executes a focus detection subroutine during continuous shooting. The details of the focus detection subroutine during continuous shooting will be described later. When the focus detection subroutine during continuous shooting ends, the camera CPU 121 returns to S9 and determines whether or not continuous shooting has been instructed.

  In S11, the camera CPU 121 determines whether a main switch included in the operation switch group 132 has been turned off. The camera CPU 121 ends this processing when the main switch is turned off, and returns to S2 when the main switch is not turned off.

  Next, an imaging subroutine executed by the camera CPU 121 in S300 of FIG. 8 will be described using a flowchart shown in FIG.

  In step S301, the camera CPU 121 performs an exposure control process, and determines imaging conditions (shutter speed, aperture value, imaging sensitivity, and the like). This exposure control processing can be performed using luminance information acquired from the image data of the live view image. Details such as the timing of obtaining image data used when performing the exposure control processing will be described later.

  Then, the camera CPU 121 transmits the determined aperture value and shutter speed to the aperture / shutter drive circuit 128 to drive the aperture / shutter 102. Further, the camera CPU 121 causes the image sensor 107 to store charges during the exposure period through the image sensor drive circuit 124.

  In step S302, the camera CPU 121 that has performed the exposure control process causes the image sensor driving circuit 124 to read all pixels of an image signal from the image sensor 107 for capturing a high-pixel still image. Further, the camera CPU 121 causes the image sensor drive circuit 124 to read one of the first and second focus detection signals from the focus detection area (focus target area) in the image sensor 107. The first or second focus detection signal read at this time is used for detecting the focus state of the image at the time of image reproduction described later. By subtracting one of the first and second focus detection signals from the imaging signal, the other focus detection signal can be obtained.

  Next, in step S303, the camera CPU 121 causes the image processing circuit 125 to perform a defective pixel correction process on the imaging data read and A / D converted in step S302.

  Further, in step S304, the camera CPU 121 causes the image processing circuit 125 to perform demosaic (color interpolation) processing, white balance processing, γ correction (tone correction) processing, color conversion processing, and the like on the image data after the defective pixel correction processing. Image processing such as edge enhancement processing and encoding processing are performed.

  In S305, the camera CPU 121 converts the high-pixel still image data as image data obtained by performing the image processing and the encoding process in S304, and one focus detection signal read out in S302, It is recorded in the memory 133 as an image data file.

Next, in S306, the camera CPU 121 records the camera characteristic information as the characteristic information of the camera 100 in the memory 133 and the memory in the camera CPU 121 in association with the high pixel still image data recorded in S305. The camera characteristic information includes, for example, the following information.
・ Imaging conditions (aperture value, shutter speed, imaging sensitivity, etc.)
・ Information on image processing performed by the image processing circuit 125 ・ Information on light-receiving sensitivity distribution of imaging pixels and focus detection pixels of the imaging element 107 ・ Information on vignetting of imaging light flux in the camera 100 ・ Mounting of an imaging optical system in the camera 100 Information on the distance from the surface to the image sensor 107 and information on the manufacturing error of the camera 100.
Information on the light-receiving sensitivity distribution of the imaging pixel and the focus detection pixel (hereinafter, simply referred to as light-receiving sensitivity distribution information) is information on the sensitivity of the imaging element 107 according to the distance (position) on the optical axis from the imaging element 107. Since the light receiving sensitivity distribution information depends on the microlens 305 and the photoelectric conversion units 301 and 302, it may be information on these. Further, the light receiving sensitivity distribution information may be information on a change in sensitivity with respect to the incident angle of light.

  Next, in step S307, the camera CPU 121 records lens characteristic information as characteristic information of the imaging optical system in the memory 133 and the memory in the camera CPU 121 in association with the high-pixel still image data recorded in step S305. The lens characteristic information includes, for example, information about an exit pupil, information about a frame such as a lens barrel for emitting a light beam, information about a focal length and an F-number at the time of imaging, information about aberration of an imaging optical system, and information about a manufacturing error of an imaging optical system. And information on the position (subject distance) of the focus lens 105 at the time of imaging.

  Next, in S308, the camera CPU 121 records image-related information as information relating to high-pixel still image data in the memory 133 and the memory in the camera CPU 121. The image-related information includes, for example, information on a focus detection operation before imaging, information on movement of a subject, and information on focus detection accuracy.

  Next, in step S309, the camera CPU 121 performs a preview display of the captured image on the display 131. Thus, the user can easily check the captured image.

  When the processing in S309 ends, the camera CPU 121 ends the main imaging subroutine, and proceeds to S9 in FIG.

  Next, the continuous shooting focus detection subroutine executed by the camera CPU 121 in S400 of FIG. 7 will be described using the flowchart shown in FIG.

  In S401, the camera CPU 121 performs an exposure control process in the same manner as in S301 in FIG. 9, and determines an imaging condition. Here, in order to perform exposure of the imaging element 107 for focus detection, the camera CPU 121 determines an aperture value suitable for focus detection, and determines a shutter speed and an imaging sensitivity according to the aperture value.

  Then, the camera CPU 121 transmits the determined aperture value and shutter speed to the aperture shutter drive circuit 128 to drive the aperture / shutter 102. Further, the camera CPU 121 causes the image sensor 107 to store charges during the exposure period through the image sensor drive circuit 124. At this time, the exposure may be performed using a slit rolling shutter or a global electronic shutter that electrically controls the exposure time without driving the aperture / shutter 102.

  In step S402, the camera CPU 121 that has performed the exposure control process causes the image sensor drive circuit 124 to output all of the image signal and one of the first and second focus detection signals from the focus target area of the image sensor 107. Pixel reading is performed. At the time of focus detection in S4 of FIG. 7, a signal obtained by performing processing such as addition or thinning on a horizontal pixel output to generate a live view image is used. In S402, signals of all pixels are output. Read and use. Accordingly, it is not necessary to switch the driving of the image sensor 107 between the time of imaging and the time of generating a live view image, so that a focus detection signal can be obtained without requiring a switching time. If the subject has moved from the time point selected in S6 of FIG. 8 at this time, the focus target area may be changed using the subject detection process.

  Next, in S403, the camera CPU 121 causes the image processing circuit 125 to perform a defective pixel correction process on the imaging data read and A / D converted in S402.

  Next, in S404, the camera CPU 121 calculates the defocus amount in the focusing target area using the first and second focus detection signals obtained in the focusing target area.

  Next, in S405, the camera CPU 121 drives the focus lens 105 to a focus position according to the defocus amount of the focus target area. Thereafter, the camera CPU 121 ends the continuous shooting focus detection subroutine, and proceeds to S9 in FIG.

  The flowchart of FIG. 11 shows an image reproduction process performed by the camera CPU 121. At the start of this processing, it is assumed that a plurality of captured images recorded in the memory 133 are displayed on the display 131.

  First, in step S1001, the camera CPU 121 selects one of a plurality of thumbnail images displayed on the display unit 131 according to a user operation, and reproduces a captured image corresponding to the selected thumbnail image on the display unit 131. indicate.

  Next, in step S1002, the camera CPU 121 sets a region of interest in the captured image reproduced and displayed in step S1001 (hereinafter, referred to as a reproduced image) according to a user operation.

  A method of setting a region of interest (first evaluation region) from a reproduced image will be described with reference to FIGS. FIG. 12A shows a state where the entire reproduced image is displayed on the display 131, and FIG. 12B shows a state where a part of the reproduced image is enlarged and displayed on the display 131.

  The user designates the position and size of the region of interest A100 for checking the focus state in the reproduced image through the operation of the operation switch group 132 or the touch operation when the display 131 is a touch panel. The camera CPU 121 sets a region of interest in the reproduced image according to the designation. The operation switch group 132, the display 131 as a touch panel, and the camera CPU 121 correspond to an area setting unit. The setting of the region of interest A100 may be performed in the state shown in FIG. 12A in which the entire reproduced image is displayed, or may be performed in the state shown in FIG.

  Next, in step S1003, the camera CPU 121 as an image group setting unit sets an image group including a plurality of mutually related captured images including a reproduced image based on the region of interest set in step S1002. That is, the first grouping is performed. At this time, the camera CPU 121 as an area detecting unit has relevance to the attention area set by the user in a plurality of other captured images (second images) whose imaging times are close to the reproduced image (first image). It is determined (detected) whether or not (in other words, similar) area is included. When there is a captured image including a region similar to the region of interest, the camera CPU 121 recognizes the similar region as a corresponding region corresponding to the region of interest. Then, an image group including a plurality of other captured images having a reproduction image and a corresponding area is set. The image group is a target of a second grouping described later. The method of setting the image group will be described later in detail. Note that, in the present embodiment, the fact that the corresponding region has “relevance” to the region of interest is expressed as “similar”, but a region having a relationship other than similarity may be the corresponding region.

  Next, in step S1004, the camera CPU 121 as a focus state obtaining unit obtains (detects) the focus state of the attention area and the corresponding area in the image group set in step S1003. That is, the camera CPU 121 detects the focus state of each of the attention area and the corresponding area, and determines the “focus state”, the “front focus state”, and the “back focus state”. The focus state includes a defocus amount and a defocus direction.

  Then, the camera CPU 121 as a processing unit proceeds to S1005, in which a plurality of captured images (reproduced images and other captured images) included in the image group set in the first grouping are determined according to the focus state detection result in S1004. To perform a classification process for classifying into a plurality of groups. This classification processing is called second grouping, and will be described later in detail.

  Next, in step S1006, the camera CPU 121 determines whether or not the user has performed an operation of changing the reproduced image in a time series with the operation switch group 132. If the user has not performed the operation, the camera CPU 121 proceeds to S1007. If the user has performed the operation, the camera CPU 121 proceeds to S1008.

  In step S <b> 1007, the camera CPU 121 determines whether the user has performed an operation of changing the playback image in the focusing direction using the operation switch group 132. If the user has not performed the operation, the camera CPU 121 returns to S1006 and waits for a user operation. If the user has performed the operation, the camera CPU 121 proceeds to S1009.

  An example of the operation switch group 132 operated by the user in S1006 and S1007 will be described with reference to FIG. FIG. 13 shows the camera 100 viewed from the back. The display 131 displays a reproduced image. On the back of the camera 100, operation buttons B1 to B4 are arranged in a cross shape.

  In S1006, the user operation for changing the reproduction image in time series is performed on the operation buttons B2 and B4 as first display image changing means. For example, when the operation button B2 is pressed, a captured image acquired after the current playback image in time is newly displayed as a playback image, and when the operation button B4 is pressed, the captured image is temporally shorter than the current playback image. The captured image acquired earlier is displayed as a new reproduced image.

  In S1007, the user operation for changing the playback image in the focus direction is performed on the operation buttons B1 and B3 as the second display image changing unit. For example, when the operation button B1 is pressed, a captured image focused on the infinity side from the current reproduction image is newly displayed as a reproduction image, and when the operation button B3 is pressed, the image is closer to the current reproduction image. The captured image focused on is displayed as a new reproduced image.

  Instead of the operation buttons B <b> 1 to B <b> 4 arranged in a cross shape, a user operation of changing a playback image by rotating a dial may be performed. In this case, a dial for changing the reproduced image in time series and a dial for changing the reproduced image in the focusing direction may be separately provided. Also, the playback image may be changed in time series and in the focus direction by combining the dial and the operation buttons. Furthermore, instead of the operation switch group 132, a touch operation (for example, a swipe operation in the left-right direction and the up-down direction) on the touch panel of the display 131 may change the time series and the focus direction.

  In step S1008, the camera CPU 121 selects the next captured image in the time series that can be changed as a playback image in response to the user operation in step S1006, in the group of captured images in the same focus state classified in step S1005. Is determined. If the next time series image does not exist, the camera CPU 121 proceeds to S1011 and causes the display 131 to display that the next time series image does not exist. Specifically, the fact that the next time-series image does not exist may be notified by a character, or the next image is not changed even if a user operation (operation of the operation buttons B2, B4 or the touch panel) is performed. You may notify that a time-series image does not exist. On the other hand, when the next time-series image exists, the camera CPU 121 proceeds to S1012, and displays the next time-series image on the display 131. The camera CPU 121 that has completed S1011 or S1012 proceeds to S1013.

  In S1009, the camera CPU 121 determines whether or not there is a group of captured images having different focus states in the focus direction corresponding to the user operation in S1007 with respect to the current playback image. For example, when the current playback image is included in the “focused state” group and the user operation corresponds to the front focus direction, it is determined whether or not the captured image exists in the “front focus state” group. . When the current playback image is included in the “front focus state” group and the user operation corresponds to the front focus direction, it is determined whether or not the captured image exists in the “front focus state” group. I do. If there is a group of captured images corresponding to the user operation with different focus states, the process proceeds to S1010; otherwise, the process proceeds to S1011 to display on the display 131 that there is no next captured image in the focus direction.

  In step S1010, the camera CPU 121 sets the next captured image in the focus direction corresponding to the user operation in step S1007, which is closest in time series to the current reproduced image, to the display 131 as a new reproduced image. indicate.

  The camera CPU 121 that has completed S1010 or S1011 proceeds to S1013.

  In step S1013, the camera CPU 121 determines whether to end the main image reproduction process. If the user turns off the power or instructs to shift from the image playback state to the imaging state, the camera CPU 121 proceeds to S1014 to end the image playback processing. In addition, also in the case where the grouping of the captured images described below in S1014 is reset, the camera CPU 121 determines in S1013 to end the image reproduction process, and proceeds to S1014. On the other hand, if the main image reproduction process is to be continued without ending, the camera CPU 121 returns to S1006 and waits for a user operation.

  In step S1014, the camera CPU 121 determines whether to perform the process regarding the grouping of the captured images performed in step S1003 or S1005 again. When adding or reducing captured images before and after in the time series, when changing the number of groups for classifying captured images according to the focus state, changing the threshold value for determining the focus state, and resetting the region of interest Returns to S1001. On the other hand, when the grouping of the captured images is not reset, the camera CPU 121 ends the main image reproduction process.

  In this embodiment, as described in step S1014, the region of the captured image (reproduced image) for checking the focus state can be changed by resetting the region of interest. For this reason, the focus state can be confirmed not only in the region of interest (focusing target region) set at the time of imaging but also in the region of interest desired by the user. It is also possible to check the focus state in a plurality of regions of interest for one captured image. For example, in a captured image of a person riding a vehicle, a region of interest can be set not only on the face of the person but also on the vehicle.

  Next, a first grouping subroutine performed by the camera CPU 121 in S1003 of FIG. 11 will be described with reference to the flowchart of FIG. In this subroutine, the camera CPU 121 sets a group of mutually related images based on the region of interest set in S1002.

  In S3001, the camera CPU 121 sets a group of detected images. There are various methods for setting a group of detected images, and a plurality of captured images having a corresponding area corresponding to the target area set in S1002 are set as a group of detected images. For example, if a plurality of captured images having a target area and a corresponding area set in S1002 are obtained by continuous shooting, a plurality of continuous captured images (continuously shot image group) having a corresponding area are set as a detection image group. Set.

  Further, a plurality of captured images having a time within a predetermined range (for example, within 10 minutes) with respect to the capturing time of the captured image having the region of interest set in S1002 may be set as the detection image group. . Further, a group of detected images may be set according to a tracking state of focus adjustment or an exposure control state during continuous shooting. For example, a plurality of picked-up images obtained by picking up images while performing so-called servo AF for performing servo control of focus adjustment may be set as a detected image group. Information added to a plurality of consecutively captured images to identify that the images have been recorded by a single image recording instruction, such as the above-described focus adjustment tracking state and exposure control state during continuous shooting, corresponds to continuous imaging information.

  Next, in S3002, the camera CPU 121 determines whether or not each captured image included in the detected image group has a corresponding area. Then, in the captured image having the corresponding area, the position and the size of the corresponding area in the captured image are detected. For this determination and detection, template matching is used. A subject model (template) in template matching is set as a region of interest, and a pixel pattern of the region of interest is treated as a feature amount.

The feature amount T (i, j) is represented by Expression (2), where (i, j) is the coordinate in the template, W is the number of horizontal pixels, and H is the number of vertical pixels.
T (i, j) = {T (0, 0), T (1, 0), ..., T (W-1, H-1)} (2)
In a captured image in which a corresponding area is searched, the range in which matching is performed (search range) is the entire area of the captured image. The coordinates in the search range are represented by (x, y). A partial area for acquiring a matching evaluation value is set, and the luminance signal of each partial area is set as a feature amount S (i, j), similarly to the template feature amount. The feature amount S (i, j) is represented by Expression (3), where (i, j) is the coordinate in the partial area, W is the number of horizontal pixels, and H is the number of vertical pixels.
S (i, j) = {S (0, 0), S (1, 0), ..., S (W-1, H-1)} (3)
As a calculation method for evaluating the similarity between the template and the partial area, an absolute difference sum, a so-called SAD (Sum of Absolute Difference) value is used. The SAD value is calculated by equation (4).

The SAD value V (x, y) is calculated while shifting the partial area one pixel at a time from the upper left of the entire area of the captured image that is the search range. The coordinates (x, y) at which the calculated V (x, y) indicates the minimum value indicate the position most similar to the template. In other words, the position indicating the minimum value is the position where there is a high possibility that the target tracking target subject exists in the search range. Here, an example in which one-dimensional information of a luminance signal is used as the feature amount has been described, but three-dimensional information such as lightness, hue, and saturation signals may be handled as the feature amount. Although the case where the SAD value is used as the matching evaluation value has been described here, a value obtained by a different calculation method such as normalized cross-correlation, so-called NCC (Normalized Correlation Coefficient) may be used.

  If the SAD value V (x, y) at the most similar position is larger than a predetermined value, it is determined that the captured image to be detected does not have a corresponding region. Thereby, the detected image group set in S3001 can be reset.

  Next, in step S3003, the camera CPU 121 sets a captured image having a corresponding region in the reproduced image and the detected image group as an image group, and stores information on the position and size (range) of the corresponding region in each captured image of the image group. Is given. The camera CPU 121 that has completed S3003 ends the processing of this subroutine.

  Next, a subroutine in which the camera CPU 121 detects the focused state and the focus state of the corresponding area in S1004 of FIG. 11 will be described using the flowchart of FIG. In this subroutine, the camera CPU 121 detects the focus state of the focused area or the corresponding area in each captured image included in the image group set in S1003.

  In step S4001, the camera CPU 121 detects a phase difference (image shift amount) of a focused or corresponding region as a first evaluation region in each captured image included in the image group. As described in S302 described above, the camera 100 of this embodiment also records one of the first and second focus detection signals when recording a normal captured image. The camera CPU 121 detects an image shift amount between one of the recorded focus detection signals and the other focus detection signal obtained by subtracting the one focus detection signal from the imaging signal.

  The image shift amount detection performed in step S4001 is the same as the image shift amount detection method performed when a live view image is displayed (live view state), but the pixel pitch of the focus detection signal used is different. For generating a live view image, a signal obtained by performing processing such as addition or thinning on a pixel output in the horizontal direction is used. In step S4001, an image obtained by reading all pixels to record a high-pixel still image is used. Since a signal is used, a signal with a finer pixel pitch is obtained. For this reason, in S4001, it is possible to detect the image shift amount with higher accuracy.

  Further, since the focus detection timing and the recording imaging timing in the live view state are different, the focus states are not necessarily equal at these timings. Although it is possible to predict the focus state at the imaging timing from the focus detection result in the live view state, it is difficult to eliminate the error. On the other hand, image shift amount detection using an imaging signal of a captured image for recording including information on the positions of a plurality of viewpoints and line-of-sight directions from the viewpoints (information on the plurality of viewpoints: viewpoint information) is performed without any time lag. Can be detected.

  Further, in S4001, the camera CPU 121 defocuses the image shift amount detected in each captured image using the camera characteristic information (acquired in S306) and the lens characteristic information (acquired in S307) at the time of acquiring each captured image. It is converted into the amount and subject distance. When converting to a defocus amount, the luminous flux range is calculated using luminous flux range information (information on the position of the exit pupil on the optical axis, F number and frame of the imaging optical system) and light receiving sensitivity distribution information of the focus detection pixels. The light receiving sensitivity distribution information considering the information is calculated. The camera CPU 121 calculates a representative value (representative coordinate) of the pair on the coordinates indicating the light receiving sensitivity distribution information from the light receiving sensitivity distribution information of the pair of focus detection pixels. Then, L / W is calculated using the interval W between the representative value of the pair and the distance L from the image sensor 107 to the position on the optical axis indicating the light-receiving sensitivity distribution information, and the detected image shift amount is calculated as L / W. The defocus amount is calculated by multiplying W. Note that the representative value of the pair may be the barycentric coordinates of the light-receiving sensitivity distribution or the center coordinates of the range of the light-receiving sensitivity distribution.

  Further, a conversion coefficient corresponding to the F-number of the imaging optical system and the exit pupil distance may be stored at the time of acquiring the captured image for recording, and the defocus amount may be calculated by multiplying the conversion coefficient by the image shift amount.

Further, when converting the defocus amount into the distance information, the imaging magnification (lateral magnification) B is obtained from the subject distance information D and the focal distance information f associated with the position of the focus lens 105 at the time of imaging. the focus amount to ² times ^B. Thereby, the defocus amount (distance information) on the subject distance can be calculated. The subject distance is obtained by adding or subtracting the defocus amount on the subject distance to the subject distance information D.

  Next, in step S4002, the camera CPU 121 serving as the reliability determination unit has high reliability of the focus of all the captured images included in the image group and the defocus amount obtained from the corresponding area (first evaluation area) in step S4001. It is determined whether or not.

  In the present embodiment, since the plurality of captured images included in the image group are obtained by continuous shooting, the position (image height) of the subject greatly changes or the lens state (focus lens position, Aperture value) may change significantly. In such a case, there may be a case where the amount of signal in the target and corresponding areas is insufficient, and stable detection of an image shift amount cannot be performed. For this reason, the camera CPU 121 determines the magnitude of the minimum value of the correlation amount between the first and second focus detection signals when detecting the image shift amount in each captured image, and the magnitude of the change in the correlation amount near the minimum value. To determine the reliability of the defocus amount. Specifically, the camera CPU 121 determines that the reliability is high when the minimum value of the correlation amount is small or when the change in the correlation amount near the minimum value is large.

  When the image group includes captured images with low reliability of the defocus amount, at least a part of the captured image of the image group and the first evaluation area that is the attention and corresponding area are included in the image group. A different second evaluation area is set. Then, the defocus amount is calculated in the second evaluation area.

  In step S4002, the reliability of the defocus amount in the attention area and the corresponding area is determined based on the magnitude of the correlation amount and the magnitude of the change in the correlation amount. However, another determination method may be used. For example, it may be determined that the reliability is low when the contrast is low or the signal amount is small, using the information on the contrast of the focused and corresponding region or the information on the signal amount at a predetermined spatial frequency.

  The camera CPU 121 that has determined in S4002 that the reliability of the defocus amount in the focused and corresponding area (first evaluation area) is low, proceeds to S4003.

  In step S4003, the camera CPU 121 sets a second evaluation area near the target area (first evaluation area) of the reproduced image. At this time, the camera CPU 121 sets the second evaluation area so that the detection of the defocus amount with higher accuracy than the first evaluation area can be expected. FIG. 16 shows a second evaluation area A101 in which at least a part thereof is set to be different from the attention area A100 shown in FIG.

  In FIG. 16, by setting a second evaluation area A101 including a hairline area with respect to an area of interest A100 centered on the eye area of the subject, a signal having a higher contrast is obtained and a higher signal is obtained. An accurate defocus amount is detected. A plurality of second evaluation areas may be set. In this case, by calculating the average value of the defocus amounts obtained in each of the plurality of second evaluation areas, a more accurate and accurate defocus amount is detected even when the inclination of the subject changes. be able to.

  Next, in S4004, the camera CPU 121 determines whether or not a plurality of captured images other than the reproduced images included in the image group have a corresponding area similar to the second evaluation area, as in S3002, If so, the position and size of the corresponding area are detected.

  Further, in S4005, the camera CPU 121 detects (calculates) the image shift amount, the defocus amount, and the subject distance in the second evaluation region and the corresponding region, as in S4001.

  Next, in step S4006, the camera CPU 121 determines whether the reliability of the defocus amount obtained in the second evaluation area and the corresponding area is high, as in step S4002. If it is determined that the reliability of the defocus amount obtained in the second evaluation area or the corresponding area is low, the camera CPU 121 returns to S4003, and sets a second evaluation area again. Note that a third evaluation area that is at least partially different from the second evaluation area may be set while holding the second evaluation area.

  On the other hand, the camera CPU 121 that has determined in S4006 that the reliability of the defocus amount obtained in the second evaluation area is high proceeds to S4007. In S4007, the camera CPU 121 updates the defocus amount and the subject distance obtained in the first evaluation area in S4001 with the defocus amount and the subject distance obtained in the second evaluation area.

  A method of updating the defocus amount will be described with reference to FIG. FIG. 17 shows the defocus amounts (A100_2 to A100_5) obtained in the first evaluation region A100 and the defocus amounts (A101_2 to A101_5) obtained in the second evaluation region A101 in chronological order. . The horizontal axis indicates the order of the imaging times, and the vertical axis indicates the detected defocus amount. In the figure, black points A100_2 to A100_5 indicate the defocus amounts obtained in the first evaluation area A100, and white circles A101_2 to A101_5 indicate the defocus amounts obtained in the second evaluation area A101.

  Here, as an example, a method of updating the defocus amount when the reliability of the defocus amount A100_3 obtained from the first evaluation area is low will be described. The defocus amounts A100_2 and A100_4 obtained before and after the time of the defocus amount A100_3 are both highly reliable. Further, the defocus amounts A101_2, A101_3, and A101_4 obtained in the second evaluation area also have high reliability. In such a case, the updated defocus amount A100_3C is calculated by the following equation (5) with respect to the defocus amount A100_3.

  In the equation (5), the highly reliable defocus amount A101_3 is offset by the average value of the difference between the defocus amounts obtained in the first and second evaluation regions A100 and A101 before and after the highly reliable defocus amount A101_3. Then, by updating the defocus amount A100_3 having low reliability with the offset defocus amount, the defocus amount A100_3C having high reliability to some extent is obtained.

  FIG. 17 also shows that the reliability of the defocus amount A100_5 is low. As described in the equation (5), by updating the defocus amount using the defocus amount obtained before and after the defocus amount A100_5 in time, a highly reliable defocus amount A100_5C is obtained.

  In the present embodiment, the evaluation result with low reliability is updated by using the difference between the evaluation results (defocus amount) obtained in the two evaluation regions, but the difference is evaluated in the evaluation region obtained in more evaluation regions. May be obtained from the results. However, if the situation (distance or size) of the subject changes greatly during continuous shooting, the difference between the defocus amounts obtained in the two evaluation areas also changes greatly, so that the difference between the defocus amounts that are closer in time is larger. It is preferable to use Also, although the updating of the defocus amount has been described with reference to FIG. 17, the subject distance may be updated. Since the difference between the subject distances as the evaluation results obtained in the two evaluation areas is hardly influenced by a change in the situation of the subject during continuous shooting, the subject distance can be updated more accurately.

  Further, in the present embodiment, the case has been described where the second evaluation region is set regardless of the size of the region of interest (first evaluation region), but the second evaluation region is set according to the size of the region of interest. The setting may be limited. When the region of interest is small, the subject is small, and there is a high probability that an edge having a light-dark difference exists in the evaluation region. On the other hand, when the region of interest is large, since the subject is large, the probability that an edge having a light-dark difference exists in the evaluation region is low. For this reason, by setting the second evaluation region only when the region of interest is large, it is possible to detect the defocus amount with high accuracy while reducing the amount of calculation relating to the second evaluation region.

  After completing S4007, the camera CPU 121 proceeds to S4008, and stores the focus state of each captured image included in the image group, imaging conditions such as the aperture value at the time of capturing each captured image, the subject distance, and depth-related information to be described later, for each captured image. And store it. Then, this subroutine ends.

  Next, a second grouping subroutine performed by the camera CPU 121 in S1005 of FIG. 11 will be described with reference to the flowchart of FIG. In this subroutine, the camera CPU 121 classifies the image group set in S1003 based on the focus state detected in S1004.

  In S5001, the camera CPU 121 acquires depth-related information from a plurality of captured images included in an image group. The depth-related information is information on the permissible circle of confusion and the aperture value when each captured image is acquired, which is used to calculate the depth of focus. Generally, the permissible circle of confusion depends on the image size and the viewing distance, but is a size that allows the viewer to recognize a blur larger than the diameter of the circle. In the present embodiment, ε is set as the permissible circle of confusion according to the image size. Assuming that the aperture value is F, the depth of focus is represented by ± εF. The camera CPU 121 calculates the depth of focus in each captured image.

  Next, in step S5002, the camera CPU 121 sets a threshold for determining the focus state (hereinafter, referred to as a determination threshold). Specifically, the camera CPU 121 sets the determination threshold value (predetermined defocus amount) to the determination threshold value in the front focus direction (negative first threshold value) Th1 = −εF / 2 and the determination threshold value in the rear focus direction (positive defocus amount). (Threshold of 2) Th2 = + εF / 2 is set. The determination threshold may be changed according to a user operation according to the focus state of the image group. A method for changing the determination threshold will be described later.

  Next, in S5003, the camera CPU 121 sets a captured image (hereinafter, referred to as a determination image) for determining a focus state from the image group. At this time, if the determination image has been set in advance, the camera CPU 121 sets the next captured image in the order of imaging time as a new determination image. On the other hand, if the determination image has not been set in advance, the camera CPU 121 sets the image with the oldest imaging time among the plurality of captured images included in the image group as the determination image.

  Next, in S5004, the camera CPU 121 determines whether or not the defocus amount (focus state) of the region of interest or the corresponding region of the determination image is smaller than the determination threshold Th1. If the defocus amount is smaller than the determination threshold Th1, the camera CPU 121 proceeds to S5006, and stores the determination image in the first group (front pin group) Gr. Judge as 1.

  On the other hand, if the defocus amount is equal to or greater than the determination threshold Th1, the camera CPU 121 proceeds to S5005, and determines whether the defocus amount of the region of interest is smaller than the determination threshold Th2. When the defocus amount is smaller than the determination threshold Th2, the camera CPU 121 proceeds to S5007, and transfers the determination image to the second group (focusing near group) Gr. Judge as 2. Further, if the defocus amount of the region of interest is equal to or greater than the determination threshold Th2, the camera CPU 121 proceeds to S5008, and transfers the determination image to the third group (back focus group) Gr. 3 is determined.

  The camera CPU 121 that has completed any of S5006, S5007, and S5008 proceeds to S5009, and determines whether the focus state determination has been completed for all captured images included in the image group. The camera CPU 121 returns to S5003 if the determination of the focus state of all captured images has not been completed, and determines the next captured image. If the determination of the focus state of all captured images has been completed, the camera CPU 121 ends this subroutine.

  In the present embodiment, a corresponding area of another captured image corresponding to a focused area of one captured image is detected in an image group set using continuous shooting information, presence / absence of a focused area, and capturing time, and the focused and corresponding areas are detected. Detect the defocus amount of the area. Then, the focus state of each captured image is determined from the defocus amount (magnitude and direction) and the depth of focus detected in each captured image. This makes it possible not only to change the reproduced image in time series at the time of reproducing the image, but also to change the focus state.

  In the present embodiment, the grouping of the captured images based on the focus state is performed on the assumption that the focus state of the attention area or the corresponding area varies due to the variation in the focus adjustment accuracy at the time of imaging. In an environment where the focus adjustment accuracy varies greatly, such as when the subject moves violently during imaging or when the contrast of the subject is low, the grouping based on the focus state functions effectively. However, in an environment in which high focus adjustment accuracy is obtained, it is assumed that the focus state of the focused area or the corresponding area does not vary so much that the focus state can be changed. Even in such a case, there is a case where there is a difference between the region of interest where the user wants to obtain the in-focus state and the focus detection region set in the camera 100. May be obtained stably.

  In such a situation, if the user wants to select a focus state after imaging, it is sufficient to intentionally acquire a plurality of captured images with different focus states without expecting a variation in focus adjustment accuracy. For example, a plurality of captured images having different focus states can be acquired by continuously capturing images in the front focus state, the focus state, and the rear focus state according to the depth of focus.

  Further, in the present embodiment, after setting the region of interest in S1002 in FIG. 11, the first grouping is performed in S1003. However, after the first grouping, a picked-up image in which the setting of the focused area can be more easily selected and displayed, and the focused area may be set with the selected captured image. In this case, the first grouping cannot be performed based on the presence or absence of the region of interest, but may be performed in consideration of the imaging time, whether the image is a continuously captured image, and the like. Examples of the captured image in which the region of interest is easier to set include an image in which the subject is projected larger, an image in which the distance to the focused subject is shorter, and an image in which the aperture is more opened. It is preferable to select an image having a smaller depth of focus, such as an image. Thus, the region of interest can be set using a captured image in which setting of the region of interest and confirmation of the focus state are easy.

  Next, a method of determining each captured image of the image group and a method of selecting a reproduced image in the processing described above will be described more specifically.

  19 is obtained as a result of the first grouping as a captured image including a corresponding area similar to the target area set as shown in FIG. 12A among a plurality of captured images obtained by continuous shooting. 5 shows a group of images. The images from Frame 1 to Frame 15 show the captured images in order of the image capturing time being old. The captured image shown in FIG. 12A corresponds to Frame 10. The captured image of Frame 10 has a small depth of focus because the subject distance is short, and is an image in which not only the face but also the entire body is reflected as the subject, and thus is an image suitable for setting the region of interest.

  FIG. 20 shows the result of the second grouping performed in S1005 on the image group shown in FIG. The group name indicating the focus state (information indicating the result of the classification process) is shown as a front focus (Gr. 1), a near focus (Gr. 2), and a rear focus (Gr. 3) at the upper left of each captured image. ing.

  FIGS. 21A and 21B show a state in which the captured image of Frame 10 after the second grouping is reproduced and displayed on the display 131. FIG. The camera CPU 121 as a display processing unit performs an image change process for changing an image to be reproduced and displayed on the display 131. In FIG. 21A, the entire captured image of Frame 10 is displayed, and in FIG. 12B, a part of the captured image is enlarged. A group name (Gr. 2) indicating the focus state is displayed at the upper left of the captured image of Frame 10.

  The camera CPU 121 that has performed the process of S1012 while the captured image of Frame 10 is displayed as shown in FIG. 21A has the same focus state as the first image change process (the focus state belongs to the same group). ) The next time series image is displayed on the display 131. For example, when the operation button B2 shown in FIG. 13 is operated, a captured image of Frame 8 is displayed, and when the operation button B4 is operated, a captured image of Frame 12 is displayed.

  In addition, the camera CPU 121 that has performed the process of S1010 in a state where the captured image of Frame 10 is displayed displays, as the second image change process, a captured image of a different focus state with a similar imaging time (belonging to a group with a different focus state). It is displayed on the display 131. For example, when the operation button B1 shown in FIG. 13 is operated, a captured image of Frame 11 is displayed, and when the operation button B3 is operated, a captured image of Frame 9 is displayed.

  As described above, in the present embodiment, when a large number of captured images having similar compositions are obtained by continuous shooting or the like, selection and reproduction of a captured image of a desired composition (imaging time) and a captured image of a desired focus state are performed. Can be done quickly. Further, the plurality of captured images shown in FIGS. 19 and 20 may be displayed collectively on the display 131 as thumbnail images so that the user can confirm the first grouping.

  Next, a method of changing the determination threshold value set in S5002 will be described with reference to FIGS. FIGS. 22A and 22B show the focus state of the target area and the corresponding area of the image group subjected to the first grouping. The horizontal axis as the first scale indicates the order of imaging time, and the vertical axis as the second scale indicates the focus state of the focused area and the corresponding area in units of depth of focus. The fifteen black dots in the figure indicate the focus states of the captured images of Frame 1 to Frame 15 shown in FIG.

  In the upper left part of the figure, a target region set by the user and a subject region including the vicinity thereof are shown cut out from a captured image in which the target region is set. On the right side of the figure, the group names (Gr. 1, Gr. 2, Gr. 3) indicating the focus state are shown, and the number of captured images belonging to each group is shown. For example, the focus vicinity group Gr. 2 includes seven captured images among the 15 captured images. Dashed lines indicate boundaries between groups.

  When the user changes the determination threshold value in S5002, the display 131 may be displayed as shown in FIGS. By performing such display, the user can intuitively confirm the variation of the focus state in the image group. Also, it is possible to easily confirm how many captured images belong to each group.

  When the determination threshold is changed, the display position of the broken line in the drawing is changed, so that the operation can be performed while checking how many captured images belong to each group in real time. The instruction to change the determination threshold value performed by the camera CPU 121 as the evaluation updating unit may be performed by operating a touch panel or an operation member provided exclusively.

  FIGS. 22A and 22B show changes in the display on the display 131 when the determination threshold is changed by a user operation. In FIG. 22B, the position of the broken line (Th1, Th2) moves in the vertical direction, as compared to FIG. Also, the number of captured images belonging to each group has changed.

  Further, in the display as shown in FIGS. 22A and 22B, the first grouping may be configured to be changeable. In that case, a boundary that can be set in the direction of the horizontal axis may be provided so that the user can operate the boundary.

  In the present embodiment, a plurality of captured images included in the image group are captured by plotting captured images having different focus states at different positions on the vertical axis as shown in FIGS. 22A and 22B. The distribution relating to the time (first scale) and the focus state (second scale) can be easily recognized. The index for classifying a plurality of captured images included in the image group is not limited to the focus state. For example, when the subject is a human face, the degree of smile may be quantified and shown on the vertical axis. At this time, the boundary may be displayed by plotting at different positions on the vertical axis depending on whether the degree of smile is from smile to laughter. Thus, the user can easily recognize how many favorite smile images exist and how to divide the groups. The captured image may be classified according to the orientation of the face of the subject (front, right, left, etc.) without being limited to the focus state and the degree of smile.

  Further, in the present embodiment, as shown in FIGS. 22A and 22B, the number of captured images belonging to groups having different focus states is displayed in a manner that is easy for the user to understand. Accordingly, when the user changes the boundary of the grouping, it is possible to confirm how many captured images belong to which group by the grouping.

  When there are three indices for classifying the captured image, such as the imaging time, the focus state, and the degree of smile, the two indices to be used may be switched to display the group, or the color of the plotted points The group may be displayed using three indices by changing the size or size.

  When the user finishes the setting of the determination threshold, thumbnail display of a plurality of captured images as shown in FIG. 20 or single display of one captured image as shown in FIG. . Further, by selecting a group, the captured images in the selected group may be displayed as thumbnails or individually.

As described above, according to the present embodiment, it is possible to easily confirm the focus state in the attention area and the corresponding area according to the user's intention with respect to a plurality of captured images. .
(Modification 1)
A refocus technique using light field information is known. In this embodiment, only light field information corresponding to two divisions in the horizontal direction can be obtained. The focus state can be changed by the focus processing.

  The change of the focus state by the refocus processing will be described with reference to FIG. FIG. 23 illustrates a display example on the display unit 131 in a case where the focus state of a captured image included in the group of images subjected to the first grouping is changed by the refocus processing. The horizontal axis indicates the order of the imaging times, and the vertical axis indicates the focus state of the attention area or the corresponding area. FIG. 23 shows a state in which the focus state of the captured images belonging to the in-focus vicinity group (Gr. 2) is not changed, and the captured images belonging to the front pin group (Gr. 1) and the rear pin group (Gr. 3). 2 shows an example in which the focus state is changed by the refocus processing.

  The 15 black dots indicate the focus states of the captured images from Frame 1 to Frame 15 shown in FIG. White circles indicate the focus state after the refocus processing. For example, a black point F3_1 indicates a focus state of the captured image of Frame 3 before the refocus processing, and a white circle F3_2 indicates a focus state of the captured image of Frame 3 after the refocus processing. The right side of FIG. 23 illustrates a change in the number of captured images belonging to each group before and after the refocus processing.

By displaying the change in the focus state due to the refocus processing on the display 131 in this way, the user can easily check the distribution of the focus state in the image group.
(Modification 2)
In the first embodiment, when the user can extract a captured image in a desired focus state, information on the set target area (and the corresponding area) may be added to the captured image as supplementary information. The information on the region of interest includes ID information of a captured image belonging to the image group, information on coordinates and size of the region of interest and the second evaluation region, and the like.

  Also, the ID information of the captured images belonging to the image group may be similarly added to all the captured images constituting the image group, corresponding to the case where each captured image is deleted.

  Next, a second embodiment of the present invention will be described. In the present embodiment, when the focus state is determined for each of the captured images of the image group to which the first grouping has been performed and the second grouping is performed, it is determined that one captured image belongs to a plurality of groups. Embodiment 2 is different from Embodiment 1 in the point of allowing.

  In the first embodiment, a case has been described in which one captured image is grouped so as to belong to only one group. However, when the user classifies the captured images in a focused state, more ease of use is required. For example, the user may want to extract a captured image in which the right eye is focused instead of the left eye from a plurality of captured images focused on a human face. In this case, if there is a depth of focus difference between the left eye and the right eye, the user may perform an operation to focus on the right eye. At this time, the captured image focused on the left eye belongs to the rear focus or front focus group with respect to the captured image focused on the right eye.

  However, when the subject is far away or when imaging is performed with a narrowed F value, there may be a captured image in which both the left eye and the right eye are in focus in the image group. Such a captured image should be extracted as a captured image focused on the right eye intended by the user or as a captured image focused on the left eye unintended by the user.

  For this reason, in the present embodiment, information on the defocus amount difference (distance difference) in the subject area and information on the defocus amount difference between the subject and the background (surrounding subject) are acquired, and a plurality of focuses are obtained in accordance with the information. The captured images belonging to the state group are allowed. That is, a captured image in which the defocus amount difference in the subject region is smaller (less than a predetermined value) is classified so as to belong to at least two of the plurality of groups. Thus, the captured image intended by the user can be extracted without omission, and the user can efficiently select and confirm the captured image.

  The configuration of the camera, the focus detection method, the focus detection area, the AF / imaging process, and the image reproduction process according to the present embodiment are the same as the configuration of the camera 100 according to the first embodiment (FIG. 1) and the focus detection method (FIG. 3A to FIG. 6), focus detection area (FIG. 7), AF / imaging processing (FIGS. 8 to 10), and image reproduction processing (FIGS. 11 to 17).

  In the present embodiment, the processing of the image group classification (second grouping) performed by the camera CPU 121 in S1005 of FIG. 11 will be described with reference to the flowchart of FIG.

  In S6001, the camera CPU 121 acquires depth-related information from a plurality of images included in the image group. The depth-related information is information on the permissible circle of confusion ε and the aperture value F, as described in S5001 of FIG. The camera CPU 121 calculates the depth of focus (± εF) in each captured image.

  Next, in step S6002, the camera CPU 121 obtains distance information of a subject area in each captured image. A method of acquiring distance information of a subject area will be described with reference to FIGS.

  FIG. 25A shows a focus detection point for detecting a subject area and acquiring distance information of the subject area in addition to a state in which a focused area A100 is set in the captured image of Frame 10 shown in FIG. Are superimposed. In FIG. 25A, cross marks Ph are shown at a total of 96 locations, 12 locations in the horizontal direction and 8 locations in the vertical direction, as focus detection points.

  In this embodiment, the camera CPU 121 detects a phase difference (image shift amount) at a total of 96 focus detection points in the same manner as in S4001 in FIG. 15, and calculates a defocus amount or a subject distance. Then, a subject area is first extracted from the obtained defocus amount or subject distance at each focus detection point. Specifically, the camera CPU 121 extracts a region where the defocus amount or the subject distance has a small difference equal to or less than a predetermined value with respect to the defocus amount or the subject distance of the region of interest as the subject region. If the subject is a person, the subject region may be extracted by detecting the body or organ.

  Further, the camera CPU 121 calculates the subject distance as a representative value using the defocus amount obtained at the focus detection points arranged in the subject area and the position of the focus lens 105 at the time of acquiring each captured image. . FIG. 25B shows the subject distance obtained from the 15 captured images shown in FIG. The vertical axis indicates the subject distance, and the horizontal axis indicates the order of the imaging time. The polygonal line indicating the subject distance D indicates that the subject gradually approaches from a long distance and the distance is not substantially changed in the last few captured images. The subject distance D (1) indicates the subject distance in the first captured image (Frame1), and the subject distance D (15) indicates the subject distance in the fifteenth captured image (Frame15).

  When the F value or the focal length of the imaging optical system is constant, the depth of field changes depending on the subject distance. Specifically, as the subject distance increases, the depth of field becomes wider (deeper). In this embodiment, the focus state is determined in consideration of the change in the depth of field.

  Next, in step S6003, the camera CPU 121 sets a determination threshold that is a threshold for determining a focus state. Specifically, the camera CPU 121 sets a determination threshold Th2 = −εF / 2 for the front focus state and a determination threshold Th3 = + εF / 2 for the rear focus state. Further, the camera CPU 121 sets a determination threshold Th1 for determining the front focus state and a determination threshold Th4 for determining the back focus state. Since the determination thresholds Th1 and Th4 are different depending on the depth of field of each captured image, if they are expressed as Th1 (i) and Th4 (i), they are calculated by the following equations (6) and (7).

  With reference to FIG. 26, a method of setting a determination threshold for the front focus state and the rear focus state will be described. FIG. 26 shows the subject distance and the depth of field in a state where Frame 1 and Frame 15 are imaged. The subject distance when capturing the image of Frame 1 is D (1), and the depth of field corresponding to the depth of focus ± εF is ± H (1). Similarly, the subject distance when the frame 15 is imaged is D (15), and the depth of field corresponding to the depth of focus ± εF is ± H (15).

  When reproducing and displaying the captured image of Frame 15, if the user desires a focus state in which the defocus amount of the focused area is shifted by the unit depth of focus, the object distance D (15) ± H (15) The state where the focus is on corresponds to that. For example, when the user wants to change the focus state to a state where the left eye is in focus by the unit depth of focus with respect to the right eye where the defocus amount is almost 0, the user sets the in-focus subject distance to D (15). The operation of moving the object by H (15) may be performed.

  Even when this user operation is performed on the subject at the subject distance D (1), the distance between the left eye and the right eye does not change on the subject, so that the focused subject distance is shifted from D (1) by H (15). What is necessary is just to perform the operation which makes it. Since the subject distance D (1) is farther than D (15), the depth of field H (1) is deeper (wider) than H (15). Therefore, the change amount of the focus state of H (15) with respect to the subject distance D (1) is a value smaller than the depth of focus εF. Assuming that the decreasing ratio is k, k is calculated by the following equation (8).

In equation (8), the front depth of field and the rear depth of field are not distinguished, and the depth of field as an approximate value is calculated by dividing the square of the subject distance by the square of the focal length. Is calculated. In order to increase the calculation accuracy of k, the depth of field may be distinguished into the front depth of field and the rear depth of field and used for calculation.

  In the above equations (6) and (7), the determination thresholds Th1 and Th4 are calculated as values obtained by multiplying the depth of focus εF by k calculated in the equation (8). As a result, a captured image in a focus state similar to the focus state selected in a state with a small depth of field (for example, a state in which the right eye or the left eye is focused) is leaked from a captured image with a long subject distance and a large depth of field. It can be extracted without. Further, the focus state can be appropriately changed regardless of the distance of the subject.

  The state in which the focus state determination threshold is set in step S6003 will be described with reference to FIG. 27 showing the focus state of the focused area and the corresponding area of the image group on which the first grouping has been performed. In FIG. 27, the threshold value Th2 for the in-focus state on the front focus side and the threshold value Th3 for the in-focus state on the rear focus side are indicated by broken lines similarly to the determination threshold value in FIG. The determination threshold Th1 of the front focus state shown by the solid line is equal to the determination threshold Th2 of the front focus side focus state in the captured image whose imaging time is the latter half, but is the front focus side focus state in the captured image whose imaging time is the first half. Above the determination threshold value Th2 (closer to the in-focus state). This is because, as described above, the boundary of the second grouping is changed in response to the range in which the focus state is maintained due to the change in the depth of field.

  Similarly, the determination threshold Th4 of the rear focus state indicated by the solid line is equal to the determination threshold Th3 of the rear focus side focus state in the captured image of the latter half of the imaging time, but is equal to the rear focus state in the first half of the captured image of the imaging time. It has changed below (closer to the in-focus state) than the side focus state determination threshold value Th3.

  In FIG. 27, the determination thresholds Th2 and Th3 for the captured image whose imaging time is the latter half are displayed slightly shifted from the determination thresholds Th1 and Th4 for easy understanding, but they may be displayed in an overlapping manner.

  In FIG. 27, the focus state F2 of the second captured image is between Th2 and Th3, so that the second captured image is Gr. Belongs to 2. On the other hand, since F2 is larger than Th4, the second captured image is Gr. Also belongs to 3. In this case, the second captured image indicates that the user has Gr. 2 when extracting a captured image belonging to Gr. Even when the captured image belonging to No. 3 is extracted, it is reproduced and displayed. Therefore, it is possible to reproduce and display the focused image desired by the user without omission.

  Also, by performing the display as shown in FIG. 27, it is possible to easily confirm the regions belonging to a plurality of groups on the scatter diagram. As described in the first embodiment, the determination threshold for determining the focus state can be easily changed.

  After completing S6003, the camera CPU 121 proceeds to S6004, and sets a determination image as a captured image for determining the focus state from the image group, similarly to S5003 in FIG.

  Next, in S6005, the camera CPU 121 determines whether the defocus amount (focus state) of the region of interest or the corresponding region is smaller than the determination threshold Th1. When the defocus amount is smaller than the determination threshold Th1, the camera CPU 121 proceeds to S6006, and stores the determination image in the front focus group Gr. Judge as 1. Thereafter, the camera CPU 121 proceeds to S6007 regardless of the defocus amount.

  In S6007, the camera CPU 121 determines whether the defocus amount is equal to or greater than the determination threshold Th2 and equal to or less than the determination threshold Th3. If the defocus amount is equal to or greater than the determination threshold value Th2 and equal to or less than Th3, the camera CPU 121 proceeds to S6008, and transfers the determination image to the in-focus neighborhood group Gr. Judge as 2. Thereafter, the camera CPU 121 proceeds to S6009 irrespective of the determination result in S6007.

  In S6009, the camera CPU 121 determines whether the defocus amount is larger than the determination threshold Th4. If the defocus amount is larger than the determination threshold Th4, the camera CPU 121 proceeds to S6010, and transfers the determination image to the rear focus group Gr. 3 is determined. After that, the camera CPU 121 proceeds to S6011 irrespective of the determination result of S6009.

  In step S6011, the camera CPU 121 determines whether the focus state determination has been completed for all captured images included in the image group. If the determination of the focus state of all the captured images has not been completed, the camera CPU 121 returns to S6004 to perform determination on the next captured image, and terminates this subroutine if the determination of the focus state of all the captured images has been completed.

  In the present embodiment, attention is paid to the fact that the change in the subject distance in the area where the user desires the focus state is not dependent on the depth of field. It is possible to belong to the group in the focused state. As a result, as described above, it is possible to extract a captured image in which a focus state desired by the user is obtained without omission, and it is possible to efficiently select and confirm the captured image by the user.

  In the present embodiment, a case has been described in which a plurality of captured images included in an image group have the same focal length and aperture value, and the depth of field changes only depending on the subject distance. However, even when the focal length and the aperture value at the time of imaging change, the focusing state (second grouping) may be similarly performed in consideration of the depth of field corresponding thereto. As a result, one captured image may belong to three or more groups.

  In the present embodiment, a case has been described in which a change in the depth of field is obtained by extracting a subject region and calculating the subject distance. However, the depth of field may be considered not only in the subject area but also in the background and other subjects. For example, a captured image in which the entire imaging range including the subject and the background falls within the depth of focus may belong to all groups as a result of the second grouping. Also, the determination threshold for determining the group may be changed using information on the depth and distance in the subject area. This makes it possible to appropriately change the focus state according to the depth of the subject.

  Further, the camera CPU 121 may change the determination threshold value in accordance with a change in the defocus amount in a subject region including the region of interest or the corresponding region and its neighboring region. For example, when the change in the defocus amount is small in the subject area, the absolute value of the determination threshold is reduced, and when the change in the defocus amount is large in the subject area, the absolute value of the determination threshold is increased. Good.

  Furthermore, in this embodiment, one captured image may belong to a plurality of groups having different focus states. In this case, when the display described in the first embodiment with reference to FIGS. 20 and 21 is performed, characters or icons indicating a plurality of group names may be displayed.

Further, in each of the above embodiments, the camera having the image processing device (camera CPU 121) is described. However, other embodiments of the present invention also include a mobile device such as a mobile phone having a built-in camera and image processing device, and an image processing device including a personal computer without an imaging function.
(Other Examples)
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

  Each of the embodiments described above is only a typical example, and various modifications and changes can be made to each embodiment when the present invention is implemented.

Reference Signs List 100 Camera 105 Focus lens 107 Image sensor 121 Camera CPU
133 memory

Claims (13)

Region setting means for setting a region of interest in a first image among a plurality of captured images obtained by imaging at different times from each other;
Region detection means for detecting a corresponding region having relevance to the region of interest in a plurality of second images of the plurality of captured images;
Focus state acquisition means for acquiring the respective focus states of the region of interest and the corresponding region,
Processing means for performing classification processing for classifying the plurality of captured images into a plurality of groups according to the focus state.   The classification process classifies each of the plurality of captured images into a first group when a defocus amount indicating the focus state of the imaging device is smaller than a negative first threshold, and the defocus amount is If the defocus amount is larger than the first threshold value and smaller than the positive second threshold value, the data is classified into a second group, and if the defocus amount is larger than the second threshold value, the data is classified into a third group. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 2, wherein the classification processing unit changes at least one of the first and second thresholds according to a user operation.   The method according to claim 1, wherein the classification processing unit changes at least one of the first and second thresholds in accordance with a change in the defocus amount in a subject area including the target area or the corresponding area. 3. The image processing device according to 2.   The classification processing unit is configured to generate a captured image in which a difference in the defocus amount in a subject area including the target area or the corresponding area among the plurality of captured images is smaller than a predetermined value by at least two of the plurality of groups. The image processing apparatus according to claim 2, wherein the image processing apparatus is classified so as to belong to a group. As the plurality of captured images, has an image group setting means to set an image group obtained by continuous shooting,
The image processing apparatus according to claim 1, wherein the classification processing unit performs the classification processing on the image group. An image for setting an image group including, as the plurality of captured images, the first image and the plurality of second images in which the corresponding region corresponding to the region of interest set in the first image is detected. Having group setting means,
The image processing apparatus according to claim 1, wherein the classification processing unit performs the classification processing on the image group. Display processing means for displaying the captured image on a display means,
The image processing apparatus according to claim 1, wherein the display processing unit displays information indicating a result of the classification process together with the captured image. Display processing means for displaying the captured image or an image generated from the captured image on a display means,
The display processing means includes a first image change processing for changing an image displayed on the display means to an image having the same focus state and a different imaging time, and a second image change processing for changing the image to a different focus state. The image processing apparatus according to claim 1, wherein the image processing apparatus performs an image change process. The captured image includes information on a plurality of viewpoints,
The image processing apparatus according to claim 1, wherein the focus state obtaining unit obtains the focus state using information on the plurality of viewpoints. An image sensor that captures a subject image;
An electronic apparatus, comprising: the image processing apparatus according to claim 1. Setting a region of interest in a first image of a plurality of captured images obtained by imaging at different times from each other;
Detecting a corresponding region having relevance to the region of interest in a plurality of second images of the plurality of captured images;
Acquiring the focus state of each of the noted area and the corresponding area;
Performing a classification process of classifying the plurality of captured images into a plurality of groups according to the focus state.   A computer program for causing a computer of an image processing apparatus to execute a process according to the image processing method according to claim 12.