JP2012108263A - Image calculation unit and image calculation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To acquire accurate depth information of a subject image.SOLUTION: An image calculation unit comprises: an image acquisition part for acquiring parallax image data about mutually-parallactic images that are output from a light-receiving element for detecting incident subject light flux after separating it into multiple wavelength ranges and that correspond to at least two of the wavelength ranges; a distance information acquisition part for acquiring distance information to at least one focused object in a state of being focused in the parallax image data and distance information to at least one non-focused object in a state of not being focused therein; and a calculation part for separating multiple objects included in a parallax image in a depth direction to set an outline by executing matching processing between the parallax image data through the use of a local window regulated according to a window size based on the distance information.

Description

  The present invention relates to an image calculation device and an image calculation program.

A technique is known in which scene depth information is obtained from a single piece of image data by photographing a scene with a camera in which a structured aperture composed of RGB color filters is arranged at the aperture position of the lens optical system ( For example, Non-Patent Document 1).
[Prior art documents]
[Non-Patent Document 1] Depth estimation and foreground mat extraction using a color filter for the diaphragm of a camera (Visual Computing / Graphics and CAD Joint Symposium 2008)

  For the local window of the matching process used in the process of calculating the depth information, accurate depth information cannot be obtained unless the size is set appropriately.

  In order to solve the above-described problem, the image calculation apparatus according to the first aspect of the present invention outputs from a light receiving element that detects an incident subject light beam by separating it into a plurality of wavelength bands, and outputs at least two of the plurality of wavelength bands. An image acquisition unit that acquires parallax image data in which images corresponding to each other have parallax, and distance information to at least one focused object that is in focus in the parallax image data and at least one non-focused state By performing a matching process between parallax image data using a distance information acquisition unit that acquires distance information to a focused object and a local window defined by a window size based on the distance information, a plurality of parallax images are included. A computing unit that separates the object in the depth direction and determines an outline.

  In order to solve the above-described problem, an image calculation program according to the second aspect of the present invention is output from a light receiving element that detects an incident subject luminous flux by separating it into a plurality of wavelength bands, and at least two of the plurality of wavelength bands are detected. An image acquisition step of acquiring parallax image data in which images corresponding to each other have parallax, distance information to at least one in-focus object in the in-focus state in the parallax image data, and at least one non-in-focus state A distance information acquisition step for acquiring distance information to the in-focus object and a matching process between the parallax image data using a local window defined by the window size based on the distance information, thereby performing a plurality of processes included in the parallax image And causing the computer to execute a calculation step of determining an outline by separating the object in the depth direction.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

It is principal part sectional drawing of the single-lens reflex camera which concerns on this embodiment. It is an external view of the structured opening unit concerning this embodiment. It is a figure which shows a mode that a structured opening unit is attached to a lens unit. It is explanatory drawing of the color filter arrange | positioned on the pixel of an image pick-up element. It is a figure which shows the relationship between the wavelength band which the pixel of an image pick-up element has a sensitivity, and the wavelength band which each filter part of structured opening permeate | transmits. 1 is a block diagram schematically showing a system configuration of a single-lens reflex camera. It is a figure which shows the acquisition flow of a main imaging | photography image and an auxiliary | assistant imaging | photography image. It is a figure which shows the to-be-photographed image observed from an optical finder, and a ranging area. It is a figure explaining the object distance acquired. It is a conceptual diagram which shows the relationship between an object area | region and a local window. It is a figure explaining the concept of the distance image output. It is a conceptual diagram showing the relationship between an object and the amount of parallax. It is a figure which shows the processing flow from acquisition of an auxiliary | assistant picked-up image to the output of depth information. It is a figure which shows the other processing flow from acquisition of an auxiliary captured image to output of depth information.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  First, the principle of acquiring scene depth information in this embodiment will be described. Here, the depth information of the scene is distance information indicating a relative front-rear relationship between each object as a subject constituting the scene. For example, the image calculation device outputs the distance of each object from the reference with respect to the focal plane in the depth direction of the scene as the reference. In particular, the image calculation device separates the objects in the scene in the depth direction based on the calculated distance and defines the contour.

  In the present embodiment, a structured aperture is inserted into an optical system that guides a subject light beam to a light receiving element constituted by a CCD, a CMOS, etc., and an auxiliary captured image for calculating depth information is acquired. The structured aperture is provided so that each color filter of R (red), G (green), and B (blue) is decentered with respect to the optical axis of the optical system when inserted into the optical system. Configured. For example, a color filter pattern is formed such as a G filter region extending radially from the optical axis center to the left, an R filter region extending radially upward, and a B filter region extending radially to the right. Is done. The light receiving element has one of RGB color filters arranged on each pixel. For example, a Bayer arrangement in which four pixels are one set is employed. The structure of the structured opening and the structure of the light receiving element according to this embodiment will be described in detail later.

  When the subject light flux that has passed through the structured aperture in which the color filter pattern according to the above example is formed is photographed by the above-described light receiving element, R is below, G is on the right, and B is on the left near the in-focus depth. Observed in different directions. The direction of displacement is reversed at points farther than the in-focus depth. The image will not shift at the depth of focus. In other words, at the out-of-focus depth, the same point in the scene is observed shifted for each color.

In other words, I _r , I _g , and I _b that are RGB components of the captured auxiliary captured image are three-viewpoint stereo images. If the virtual central viewpoint is taken as the reference coordinate, and the parallax at the reference coordinate (x, y) is d pixels, I _r (x, y−d), I _g (x + d, y), I _b (x -D, y) corresponds, and d that best matches them can be used as an estimated value of parallax. However, since the luminance level of the corresponding point is not located because the observation wavelength bands are different, a simple luminance difference cannot be used as a matching index.

Therefore, the fact that the color component of the natural image has a high tendency to show a linear relationship locally is used here. A local window w (x, y) is considered around (x, y), and a set of pixel colors included therein is defined as S (x, y; d) = {(I _r (s, t−d), I _{If g} (s + d, t), I _b (s−d, t)) | (s, t) ∈w (x, y)}, the distribution is considered to be closest to a straight line when true d.

ただしλ_０、λ_１、λ_２（λ_０≧λ_１≧λ_２≧０）は色分布Ｓ（ｘ，ｙ；ｄ）の主成分軸に沿った分散（すなわちＳ（ｘ，ｙ；ｄ）の共分散行列Σの固有値）、σ_ｒ ^２、σ_ｇ ^２、σ_ｂ ^２は色分布のＲ、Ｇ、Ｂ軸に沿った分散である。ここでは表記の簡単のため右辺のｘ、ｙ、ｄへの依存については省略した。 Here, the following is used as an index for measuring the linear relationship between RGB.

However, λ ₀ , λ ₁ , λ ₂ (λ ₀ ≧ λ ₁ ≧ λ ₂ ≧ 0) is a dispersion along the principal component axis of the color distribution S (x, y; d) (that is, S (x, y; d) ), Σ _r ² , σ _g ² , and σ _b ² are variances along the R, G, and B axes of the color distribution. Here, the dependency on the right side x, y, d is omitted for the sake of simplicity.

Since it is λ ₀ + λ ₁ + λ ₂ = σ _r ² + σ _g ² + σ _b ² (= trace (Σ)) due to the nature of the matrix, equation (1) becomes smaller as there is a difference in magnitude between eigenvalues. If the distribution is linear (λ ₀ >> λ ₁ , λ ₂ ), L is close to 0, and the maximum value L = 1 when the correlation is completely uncorrelated (the eigenvalue is equal to the variance along the R, G, and B axes). Take.

It is necessary to calculate L (x, y; d) for each point of the image, and for each disparity d in the range that can be taken, but using Summed Area Table, I _r , I _g , I _b and their squares If the product table is calculated, each element of the covariance matrix Σ can be calculated without depending on the size of the local window. Further, since the numerator of the equation (1) is λ ₀ λ ₁ λ ₂ = det (Σ) due to the matrix property, it is not necessary to actually calculate the eigenvalue.

  The depth information of the scene can be estimated by the parallax d of each object calculated as described above. The single-lens reflex camera as the image arithmetic apparatus according to the present embodiment employs such a principle of acquiring depth information of a scene. However, in the above-described method, if the window size of the local window for performing the matching process is not set appropriately, the calculated distance and the accuracy of the contour of the object determined from the calculated distance are lowered. In view of this, in the single-lens reflex camera of this embodiment, the optimization of the window size is attempted using the distance information acquired in advance in the calculation of the depth information. This embodiment will be described below.

  FIG. 1 is a cross-sectional view of a main part of a single-lens reflex camera 200 according to the present embodiment. The single-lens reflex camera 200 functions as an imaging device by combining a lens unit 210 that is a photographing lens and a camera unit 230 that is a camera body.

  The lens unit 210 includes a lens group 211 arranged along the optical axis 201. The lens group 211 includes a focus lens 212 and a zoom lens 213. An aperture 214 and a structured aperture unit 300 are also arranged along the optical axis 201. The structured aperture unit 300 can be inserted into and removed from the subject light flux.

  The lens unit 210 includes a lens system control unit 216 that controls and operates the lens unit 210 such as driving the focus lens 212 and the diaphragm 214. Each element constituting the lens unit 210 is supported by the lens barrel 217. The position sensor 221 detects the position of the focus lens 212 and outputs it to the lens system control unit 216.

  The lens unit 210 includes a lens mount 218 at a connection portion with the camera unit 230, and engages with the camera mount 231 included in the camera unit 230 to be integrated with the camera unit 230. Each of the lens mount 218 and the camera mount 231 includes an electrical connection portion in addition to a mechanical engagement portion, and realizes power supply from the camera unit 230 to the lens unit 210 and mutual communication. In the following description, as shown in the drawing, the z-axis is determined in the direction in which the subject luminous flux is incident along the optical axis 201.

  The camera unit 230 includes a main mirror 232 that reflects a subject image incident from the lens unit 210 and a focus plate 234 on which the subject image reflected by the main mirror 232 is formed. The main mirror 232 rotates around the rotation axis 233 and can take a state of being obliquely provided in the subject light flux centered on the optical axis 201 and a state of being retracted from the subject light flux. When the subject image is guided to the focus plate 234 side, the main mirror 232 is provided obliquely in the subject light flux. Further, the focus plate 234 is disposed at a position conjugate with the light receiving surface of the image sensor 243.

  The subject image formed on the focusing screen 234 is converted into an erect image by the pentaprism 235 and observed by the user via the eyepiece optical system 236. An AE sensor 237 is disposed above the exit surface of the pentaprism 235, and detects the luminance distribution of the subject image.

  The region near the optical axis 201 of the main mirror 232 in the oblique state is formed as a half mirror, and a part of the incident light beam is transmitted. The transmitted light beam is reflected by the sub mirror 238 that operates in conjunction with the main mirror 232, and is guided to the AF optical system 239. The subject luminous flux that has passed through the AF optical system 239 enters the AF sensor 240. The AF sensor 240 detects a phase difference signal for each of a plurality of predetermined regions from the received subject light flux. Specific detection areas and the like will be described in detail later. The sub mirror 238 retracts from the subject light beam in conjunction with the main mirror 232 when the main mirror 232 retracts from the subject light beam.

  A focal plane shutter 241, an optical low-pass filter 242, and an image sensor 243 are arranged along the optical axis 201 behind the oblique main mirror 232. The focal plane shutter 241 takes an open state when the subject light flux is guided to the image sensor 243, and takes a shielding state at other times. The optical low-pass filter 242 plays a role of adjusting the spatial frequency of the subject image with respect to the pixel pitch of the image sensor 243. The imaging element 243 is a light receiving element such as a CMOS sensor, for example, and converts the subject image formed on the light receiving surface into an electrical signal.

  The electrical signal photoelectrically converted by the image sensor 243 is processed into image data by the image processing unit 246 which is a DSP mounted on the main board 244. In addition to the image processing unit 246, a camera system control unit 245, which is an MPU that integrally controls the system of the camera unit 230, is mounted on the main board 244. The camera system control unit 245 manages the camera sequence and performs input / output processing of each component.

  A display unit 247 such as a liquid crystal monitor is disposed on the back surface of the camera unit 230, and a subject image processed by the image processing unit 246 is displayed. The display unit 247 displays not only a still image after shooting, but also an EVF image as a viewfinder, various menu information, shooting information, and the like. The camera unit 230 houses a detachable secondary battery 248 and supplies power to the lens unit 210 as well as the camera unit 230. A flash 249 that can be used and stored is provided near the pentaprism 235, and the subject is irradiated under the control of the camera system controller 245 in the used state.

  FIG. 2 is an external view of the structured opening unit 300. 2A is a front view seen from the subject side, and FIG. 2B is a side view.

  The structured opening unit 300 includes a filter 302, a base portion 307, an outer peripheral portion 308, and a grip portion 309 as main components. The base portion 307 has a cylindrical hollow portion in the subject light flux range, and the filter 302 is stretched and fixed to the hollow portion of the base portion 307.

  An outer peripheral portion 308 and a grip portion 309 are integrally formed at the end of the base portion 307. The user inserts and removes the structured opening unit 300 with respect to the lens unit 210 by grasping the grip 309.

  The filter 302 includes a blocking filter unit 303 that blocks a subject light beam, an R filter unit 304 that transmits a red wavelength band, a G filter unit 305 that transmits a green wavelength body, and a B filter unit 306 that transmits a blue wavelength band. Composed. As shown in the figure, the R filter unit 304, the G filter unit 305, and the B filter unit 306 are offset from the optical axis 201 so as to extend in the radial direction with respect to the optical axis 201 passing through the center of the filter 302. Is provided. More specifically, the R filter unit 304, the G filter unit 305, and the B filter unit 306 are each square and adjacent to each other, and are arranged such that one vertex is gathered near the optical axis 201.

  FIG. 3 is a diagram illustrating a state in which the structured aperture unit 300 is attached to the lens unit 210. FIG. 3A shows a state immediately before the structured aperture unit 300 is inserted into the lens unit 210, and FIG. 3B shows a state where the structured opening unit 300 is attached.

  The user grips the grip portion 309 and inserts the structured opening unit 300 into the mounting slit 219 provided in the lens barrel 217. Then, in the structured opening unit 300, the side surface of the base portion 307 is guided by the insertion / extraction guide 220 provided inside the lens barrel 217, and reaches the position where the center of the filter 302 coincides with the optical axis 201. When the center of the filter 302 reaches a position that coincides with the optical axis 201, the outer peripheral portion 308 is fitted into the mounting slit 219, and the outer surface of the outer peripheral portion 308 matches the outer appearance surface of the lens barrel 217.

  FIG. 4 is an explanatory diagram of color filters arranged on the pixels of the image sensor 243. The arrangement of the color filters is a so-called Bayer arrangement, and an R pixel filter, a G pixel filter, a G pixel filter, and a B pixel filter are provided on each pixel with four pixels as a set. Therefore, the wavelength band in which each pixel has sensitivity is regulated by the pixel filter provided for each pixel. For example, a pixel provided with an R pixel filter has sensitivity to a red wavelength band. As the entire image sensor 243, each of the two-dimensionally arranged pixels is discretely provided with any one of an R pixel filter, a G pixel filter, and a B pixel filter, and therefore the image sensor 243 is incident. It can be said that the object luminous flux is detected separately for each wavelength band. In other words, the image sensor 243 performs photoelectric conversion by separating the subject image formed on the light receiving surface into three wavelength bands of RGB.

  FIG. 5 is a diagram illustrating a relationship between a wavelength band in which a pixel of the image sensor 243 has sensitivity and a wavelength band transmitted through each filter portion of the structured aperture. In the figure, the vertical axis represents transmittance (%), and the horizontal axis represents wavelength (nm). It represents that the light having a higher transmittance reaches the photodiode constituting the pixel.

  The B curve 701 indicates the sensitivity of the pixel provided with the B pixel filter, the G curve 702 indicates the sensitivity of the pixel provided with the G pixel filter, and the R curve 703 indicates the sensitivity of the pixel provided with the R pixel filter. Indicates. The structured aperture B filter 306 transmits the wavelength band of arrow 711. Similarly, the G filter unit 305 transmits the wavelength band indicated by the arrow 712, and the R filter unit 304 transmits the wavelength band indicated by the arrow 713. Therefore, the pixel provided with the B pixel filter has sensitivity to the subject light beam transmitted through the B filter unit 306 of the structured aperture, and the pixel provided with the G pixel filter is the G filter unit 305 of the structured aperture. It can be said that the pixel provided with the R pixel filter has sensitivity to the subject light beam transmitted through the R filter unit 304 of the structured aperture.

Then, a subject image in the red wavelength band with the R filter unit 304 as an opening, a subject image in a green wavelength band with the G filter unit 305 as an opening, and a subject image in a blue wavelength band with the B filter unit 306 as an opening are once Can be obtained by shooting. These subject images correspond to the above-described I _r , I _g , and I _b , respectively, and constitute a three-viewpoint parallax image.

  Note that the subject image of the red wavelength band, the subject image of the green wavelength band, and the subject image of the blue wavelength band captured in this manner are provided with the G pixel filter only for the output of the pixel provided with the R pixel filter. Since only pixel outputs or only pixel outputs provided with a B pixel filter are collected and separated from each other, they are formed by image processing different from image processing in normal photographing. Therefore, here, a captured image acquired by normal imaging is referred to as a main captured image, and a captured image acquired using the structured aperture is referred to as an auxiliary captured image. The actual captured image is an image captured by extracting the structured aperture unit 300 from the lens unit 210, and the auxiliary captured image is an image captured by attaching the structured aperture unit 300 to the lens unit 210.

  Next, the system configuration of the single-lens reflex camera 200 according to the present embodiment will be described. FIG. 6 is a block diagram schematically showing the system configuration of the single-lens reflex camera 200. The system of the single-lens reflex camera 200 includes a lens control system centered on the lens system control unit 216 and a camera control system centered on the camera system control unit 245 corresponding to the lens unit 210 and the camera unit 230, respectively. Is done. The lens control system and the camera control system exchange various data and control signals with each other via a connection portion connected by the lens mount 218 and the camera mount 231.

  An image processing unit 246 included in the camera control system processes an imaging signal photoelectrically converted by the imaging element 243 into image data in accordance with a command from the camera system control unit 245. The image data processed in the actual photographed image is sent to the display unit 247 and displayed for a certain time after photographing, for example. In parallel with this, the processed image data is processed into a predetermined image format and recorded in the external memory via the external connection IF 254. The image processing unit 246 generates a subject image separated for each wavelength band from an imaging signal captured as an auxiliary captured image. The generated subject image is transferred to the depth information calculation unit 251. The depth information calculation unit 251 calculates depth information by the above-described method. The calculated depth information is recorded in the camera memory 252. Specific calculation processing will be described later.

  The camera memory 252 is a non-volatile memory such as a flash memory, for example, and plays a role of recording a program for controlling the single-lens reflex camera 200, various parameters, and the like in addition to a temporary recording place of the generated captured image. . The work memory 253 is a memory that can be accessed at high speed, such as a RAM, and has a role of temporarily storing image data being processed.

  The release switch 255 has a two-step switch position in the push-in direction, and the camera system control unit 245 detects phase difference information from the AF sensor 240 when detecting that the first-stage switch SW1 is turned on. get. Then, the drive information of the focus lens 212 is transmitted to the lens system control unit 216. Also, the exposure value is determined by obtaining the luminance distribution of the subject image from the AE sensor 237. Furthermore, when it is detected that SW2 which is the second-stage switch is turned on, acquisition of the main captured image or acquisition of the auxiliary captured image is executed according to a predetermined processing flow. A processing flow for acquiring the main captured image and the auxiliary captured image will be described later. The flash 249 irradiates the subject under the control of the camera system control unit 245.

  The lens system control unit 216 receives various control signals from the camera system control unit 245 and executes various operations. In addition, the lens system control unit 216 receives the output from the position sensor 221 and detects the position of the focus lens 212. The lens system control unit 216 transmits the detected focus lens position to the camera system control unit 245 together with the lens information including the focal length.

  Next, an example of a shooting sequence by the single-lens reflex camera 200 will be described. FIG. 7 is a diagram illustrating an acquisition flow of the main captured image and the auxiliary captured image. When the shooting sequence is started, in step S301, the camera system control unit 245 determines whether the shooting operation to be performed from now is the shooting operation of the main shooting image or the shooting operation of the auxiliary shooting image. If it is determined that the actual captured image is captured, the process proceeds to step S302.

  In step S302, the camera system control unit 245 determines whether or not the structured aperture unit 300 is extracted from the subject light beam by the user. Whether it is extracted or not can be determined by providing a sensor such as a photo interrupter.

  If the camera system control unit 245 determines that the structured aperture unit 300 has been extracted, the process proceeds to step S303. It waits at step S302 until it is extracted.

  In step S303, the camera system control unit 245 performs photometry of the subject after waiting for the SW1 of the release switch 255 to be turned on. Specifically, the camera system control unit 245 acquires the luminance distribution of the subject image from the AE sensor 237. After acquiring the luminance distribution of the subject image, the camera system control unit 245 proceeds to step S304 and calculates an exposure value. Here, the exposure value is three numerical values of an exposure time for exposing the image sensor 243 to the subject light beam, an aperture value of the diaphragm 214 for limiting the subject light beam, and ISO sensitivity corresponding to the readout gain of the image sensor 243.

  Further, when SW1 is turned on, the camera system control unit 245 executes photographing preparation processing such as focusing operation in addition to the photometry processing. The focusing operation is realized by measuring the subject using the AF sensor 240 and driving the focus lens 212 by the lens system control unit 216 based on the obtained phase difference information.

  When the shooting preparation process is completed, the camera system control unit 245 proceeds to step S305, waits for the SW2 of the release switch 255 to be turned on, and executes an operation for acquiring a captured image. Specifically, the main mirror 232 and the sub mirror 238 are retracted from the subject light beam, and the diaphragm 214 and the focal plane shutter 241 are operated according to the determined exposure value. Further, the electric charge accumulated in each pixel of the image sensor 243 is read while the focal plane shutter 241 is opened, the image processing unit 246 is caused to generate an image file in accordance with a preset format, and the image file is recorded in the recording unit. To record.

  The camera system control unit 245 proceeds to step S306 when a series of main image capturing operations has been completed, and determines whether or not an auxiliary image capturing operation has been instructed. If the auxiliary photographing image photographing operation is not instructed, the series of processes is terminated.

  If the camera system control unit 245 determines that the auxiliary photographing image capturing operation is instructed in step S301 or step S306, the process proceeds to step S307, and whether or not the structured aperture unit 300 is mounted in the subject luminous flux by the user. Determine whether. The camera system control unit 245 waits in step S307 until the structured aperture unit 300 is mounted.

  If the camera system control unit 245 determines that the structured aperture unit 300 is attached, the process proceeds to step S308. Since the photographing operation of the auxiliary photographed image is executed with the structured aperture interposed in the subject light flux, the optical condition of the subject light flux reaching the image sensor 243 is greatly different from that during the photographing operation of the main photographed image. In brief, since the subject luminous flux reaching the image sensor 243 is limited by the structured aperture, a dark image is obtained under the same conditions as the shooting conditions of the main shooting image. Therefore, the camera system control unit 245 makes the shooting condition of the auxiliary shooting image different from the shooting condition of the main shooting image.

  Therefore, the camera system control unit 245 irradiates the subject by operating the flash 249 during the operation of capturing the auxiliary captured image. In step S308, the camera system control unit 245 waits for SW1 to be turned on, performs pre-light emission as a pre-irradiation operation of the flash 249, and performs subject photometry. Specifically, the camera system control unit 245 acquires the luminance distribution from the AE sensor 237 when the subject image is pre-flashed.

  Further, when SW1 is turned on, the camera system control unit 245 executes photographing preparation processing such as focusing operation in addition to the photometry processing. The focusing operation is realized by measuring the subject using the AF sensor 240 and driving the focus lens 212 by the lens system control unit 216 based on the obtained phase difference information. At this time, distance measurement information including the lens information of the lens unit 210, the distance measurement point where the focusing operation is performed, the defocus amount at the other distance measurement points, the focus lens position, and the like are temporarily stored in the work memory 253.

  After acquiring the luminance distribution of the subject image, the camera system control unit 245 proceeds to step S309 and calculates the exposure value. The exposure value at this time is a value based on the premise that the flash 249 emits light. Therefore, the camera system control unit 245 proceeds to step S310 and determines the irradiation amount of the flash 249 during the photographing operation. The irradiation amount may be calculated from the luminance distribution information obtained from the AE sensor 237. However, if the flash 249 is irradiated even during the photographing operation of the actual captured image, the irradiation amount is calculated from the irradiation amount during the photographing operation. Alternatively, the irradiation amount in the shooting operation of the auxiliary captured image may be calculated. Specifically, the amount of light that is cut by interposing the structured aperture in the subject luminous flux is ascertained by a prior experiment or the like, and thus the amount of light corresponding to the number of exposure steps that decreases according to this is obtained. Calculated by adding to the irradiation amount during the shooting operation.

  Note that the calculation of the exposure value in step S309 is executed in combination with the determination of the irradiation amount of the flash 249 in step S310, but the ISO sensitivity of the exposure value is set larger than the ISO sensitivity calculated in step S304. Good. That is, the blurring of the subject in the auxiliary captured image adversely affects the calculation of the depth information and causes a decrease in accuracy. Therefore, it is preferable that the exposure time for the image sensor 243 is short. Therefore, in order to shorten the exposure time, the ISO sensitivity is set large. Since the auxiliary captured image is not a user-viewed image, the ISO sensitivity may be set large as long as it does not affect the calculation of depth information. Therefore, the upper limit of the ISO sensitivity that can be set by the camera system control unit 245 differs depending on the value at the time of the photographing operation of the main photographic image and the value at the time of the photographing operation of the auxiliary photographic image. Is big.

  When the exposure value and the irradiation amount during the photographing operation are determined in this way, the camera system control unit 245 proceeds to step S311 and waits for the SW2 of the release switch 255 to be turned on to acquire an auxiliary captured image. Perform the action. Specifically, the main mirror 232 and the sub mirror 238 are retracted from the subject light beam, and the diaphragm 214 and the focal plane shutter 241 are operated according to the determined exposure value. At this time, the flash 249 is operated according to the determined irradiation amount to irradiate the subject. Furthermore, the electric charge accumulated in each pixel of the image sensor 243 is read, and the image processing unit 246 generates an auxiliary captured image file including three parallax images of a red image, a green image, and a blue image, and the image file is recorded in the recording unit To record. At the same time, distance measurement information is recorded in association with the auxiliary captured image. The distance measurement information is described as, for example, EXIF information that is supplementary information. Thus, a series of processing ends.

  Further, in this flow, the flash 249 is irradiated during the shooting operation of the auxiliary shooting image to compensate for the light amount reduction due to the structured opening. However, depending on the subject environment, the reduction in the amount of light may be compensated by changing other shooting conditions without irradiating the flash 249. For example, as described above, the ISO sensitivity is set to a large value. Alternatively, the exposure time may be set longer with a predetermined exposure time as a lower limit. That is, the camera system control unit 245 may change the shooting condition of the auxiliary captured image in a direction that captures the subject luminous flux and amplifies the captured light as compared to the shooting condition of the actual captured image.

  Next, acquisition of distance information based on the output of the AF sensor 240 will be described. FIG. 8 is a diagram showing a subject image and a distance measurement area observed from the optical viewfinder. Here, as an example of the scene, a case where a girl 101, a boy 102, and a woman 103 exist in this order from the front will be described.

  The AF sensor 240 has a plurality of distance measuring points 260 that are two-dimensionally and discretely arranged with respect to the subject space. In the case of the example in the figure, eleven distance measuring points 260 are discretely arranged in a substantially rhombus shape as a whole. The AF sensor 240 can independently output a defocus amount corresponding to each distance measuring point 260.

  The camera system control unit 245 detects the defocus amount of the focus range-finding point 261 selected by an algorithm such as near-point priority, and determines the amount and direction of movement of the focus lens 212 leading to focus, The data is transmitted to the lens system control unit 216. The lens system control unit 216 moves the focus lens 212 in accordance with these pieces of information. When the movement of the focus lens 212 is completed, the camera system control unit 245 detects the defocus amount of the focusing distance measuring point 261 again by the AF sensor 240, and focuses on the object corresponding to the focusing distance measuring point 261. Make sure.

  When the camera system control unit 245 confirms that the camera is in focus, the camera system control unit 245 blinks the focusing distance measuring point 261 by superimpose display or the like, and notifies the user of the completion of the focusing operation. In addition, the camera system control unit 245 detects the defocus amount of the focus range-finding point 261 again, and simultaneously detects the defocus amount of other range-finding points 260. At this time, grouping is performed for each distance measuring point 260 showing substantially the same defocus amount. In the case of the figure, the first non-focusing distance measuring point 262 indicating the first defocus amount and the second non-focusing distance measuring point 263 indicating the second defocus amount are recognized as separate groups. It should be noted that the distance measuring point 260 indicating the defocus amount within a certain range from the defocus amount of the specific group and the distance measuring point 260 indicating the defocus amount at infinity are excluded from grouping. If there are a plurality of in-focus distance measuring points 261, these are collectively recognized as a group.

  In the subject space corresponding to the grouped distance measuring points 260, there are objects as subjects. According to the example in the figure, the girl 101 corresponds to the in-focus distance measuring point 261, and the boy 102 corresponds to the second out-of-focus distance measuring point 262, and the boy 102 corresponds to the second out-of-focus distance measuring point 263. A woman 103 exists.

  The camera system control unit 245 calculates the distance to each object from the distance measurement information including the grouping information. FIG. 9 is a diagram for explaining the acquired object distance.

The distance to each object is calculated with reference to the light receiving surface of the image sensor 243 provided in the single-lens reflex camera 200. For example, the distance D _B to the boy 102 is a defocus amount in the first out-of-focus detecting point 262, the lens information of the lens unit 210, is calculated from the focus lens position and the like. More specifically, a correspondence table between the focus lens position and the focused object distance is acquired as lens information, and the current focus lens position is input to obtain the distance to the focused object. This distance corresponds to the distance _{D A} to the girl 101.

In autofocus, a movement amount conversion function for how much the focus lens 212 should be moved to the in-focus position with respect to a certain defocus amount is used. Therefore, the movement amount of the focus lens 212 is calculated from the defocus amount at the first non-focusing distance measuring point 262 using this movement amount function. Then, by inputting a value obtained by adding the movement amounts calculated with the current focus lens position in the correspondence table, obtaining a distance D _B.

Distance _{D C} to women 103 can also be calculated in the same manner. In this case, the second non-focusing distance measuring point 263 includes three distance measuring points 260, but the defocus amount at any one of the distance measuring points 260 may be used as a representative. When the object distance is calculated for each grouped object, the calculated object distance and the arrangement of the grouped distance measuring points 260 are recorded as a pair.

  Note that when the distance to the object corresponding to the out-of-focus distance measuring point is calculated using the movement amount conversion function, the function is held as related information of the auxiliary captured image. Even if the movement amount conversion function is not used, if the focal length of the lens unit 210, the aperture value at the time of distance measurement, and the defocus amount at the first non-focusing distance measuring point 262 are obtained as lens information, the defocus amount is obtained. The distance corresponding to the focus can be calculated. In this case, the calculated distance is added to the distance to the focused object to obtain the distance to the non-focused object.

Next, the matching process performed using these distance information will be described. FIG. 10 is a conceptual diagram showing the relationship between the object area and the local window. Figure, in order from the left, _{I r} image extracted from the auxiliary captured image, _{I g} images, representing a _{I b} images. These are mutually parallax images. Since the image area corresponding to the girl 101 is an in-focus area, there is no parallax between them. On the other hand, the image region corresponding to the boy 102 and the image region corresponding to the woman 103 are regions that are not in focus, and thus have parallax. In particular, since the female 103 exists farther from the girl 101 than the boy 102, it is estimated that the amount of parallax in the image area corresponding to the female 103 is greater than the amount of parallax in the image area corresponding to the boy 102. The

  According to the depth information acquisition principle already described, the distance and contour of the object are calculated by calculating the parallax amount of the same object between the images. The calculation of the amount of parallax is performed by setting a local window for each parallax image and searching for image regions with good matching with each other while relatively shifting. Then, the amount of parallax is determined from the coordinate values that are determined to match each other. At this time, if the window size of the local window to be set is too large with respect to the amount of parallax, the resolution of the boundary portion will be reduced, resulting in the definition of an ambiguous contour. Mismatches that determine that matching is good increase. Therefore, a high-precision contour cannot be defined unless an appropriate window size is defined.

  Therefore, in the present embodiment, the window size considered to be optimal for each object is determined using the above-described distance information calculated from the output of the AF sensor 240. Since the distance information includes the distance of the object corresponding to the arrangement of the grouping distance measuring points 260, first, an area surrounding the image position corresponding to the arrangement of the distance measuring points 260 is defined as an object area. In the example of the figure, a focused object area 271 is set as an object area corresponding to the focus range-finding point 261, a first non-focused object area 272 is set as an object area corresponding to the first non-focus range-finding point 262, and a second A second non-focused object region 273 is defined as an object region corresponding to the non-focusing distance measuring point 263.

Then, at a focus object area 271, it performs moved matching the local window 281 with a side length of a square of W _A vertically and horizontally in each parallax image. Similarly, in the first out-of-focus object region 272, to move the local window 282 with a side length of a square of W _B in the vertical and horizontal directions in each of the parallax images in the second non-focused object area 273 moves the local window 283 with a side length of a square of W _C in the vertical and horizontal directions in each of the parallax images, it performs a matching process.

Because the focus object region 271 is expected that there is no disparity between the parallax images, the window size W _A in the local window 281 having a predetermined size window size for focusing the object area is applied. On the other hand, between the parallax images in the first out-of-focus object region 272, the distance difference _| D B -D _{A |} so that the amount of parallax based on the presence is envisaged, the window size _{W B} in the local window 282, The size calculated from the distance difference | D _B −D _A | is applied. Similarly, in the second non-focused object region 273, it is assumed that there is a parallax amount based on the distance difference | D _C −D _A | between the parallax images, so the window size W _C in the local window 283 is , The size calculated from the distance difference | D _C −D _A | is applied. A specific calculation procedure will be described later.

  When the matching process is performed in this manner, the contour for each object is determined, and a so-called distance image can be generated. FIG. 11 is a diagram for explaining the concept of the output distance image.

As shown in the figure, a contour is defined for each object that exists as a subject, indicating that the regions included in each contour are present at substantially the same distance. _A distance DA is associated with an area A as an image area corresponding to the girl 101. Similarly, a distance D _B is associated with a region B as an image region corresponding to the boy 102, and a distance D _C is associated with a region C as an image region corresponding to the woman 103. Further, the distance of the area D, which is another area, is associated as infinity.

  Note that for the distance image shown in the figure, the contour may be defined based on one of the three parallax images, or the contour image based on a virtual image having no parallax defined by the center of gravity of the three parallax images. May be determined. Further, the contour may be determined in correspondence with the corresponding main captured image.

  The depth information calculation unit 251 can output the depth information calculated in this way as a distance image defined by the object outline and the distance corresponding to the object as shown in FIG. It can also be output as an object table listing enumerated vector data and corresponding distances.

  Next, a calculation procedure for determining the window size based on the distance difference will be described. FIG. 12 is a conceptual diagram showing the relationship between an object and the amount of parallax. Here, it is assumed that the chief ray passes through the center of the pupil of the lens group 211.

と表される。この式により、合焦オブジェクトまでの距離ｓと非合焦オブジェクトまでの距離ｓ_１が既知であれば、その他のパラメータは補助撮影画像の撮影時において確定されているので、視差量εを算出することができる。なお、図による距離ｓおよび距離ｓ_１は、レンズユニット２１０の瞳位置からの距離であるが、撮像素子２４３の結像面から瞳位置までの距離も、レンズ情報により既知であるので、ＡＦセンサ２４０の出力から得られる距離情報を利用することができる。 When the focus lens 212 focuses on a specific object, s is a distance to the focused object existing at the focus position. Further, the distance to the out-of-focus object existing at the defocus position is s ₁ . If the focal length of the lens unit 210 is f and the aperture value is F, the parallax amount ε appearing on the image sensor 243 of the out-of-focus object in s ₁ is

It is expressed. From this equation, if the distance s to the focused object and the distance s ₁ to the non-focused object are known, the parallax amount ε is calculated because the other parameters are determined when the auxiliary captured image is captured. be able to. Note that the distance s and the distance s ₁ shown in the figure are distances from the pupil position of the lens unit 210, but the distance from the imaging surface of the image sensor 243 to the pupil position is also known from the lens information, and therefore, the AF sensor. The distance information obtained from the 240 outputs can be used.

  It can be seen from the equation (2) that the parallax amount ε can be calculated for each out-of-focus object. Therefore, since ε can be calculated for each non-focused object region, a local window that defines a pixel size corresponding to the parallax amount ε as a window size can be applied to each non-focused object region.

  FIG. 13 is a diagram illustrating a processing flow from acquisition of an auxiliary captured image to output of depth information. A series of processes in this flow is started when the camera system control unit 245 receives an instruction from the user via the operation member or when a distance calculation process is incorporated in a predetermined control program. .

  In step S101, the image processing unit 246 acquires an auxiliary captured image that is a processing target. The image processing unit 246 may acquire the image signal output from the image sensor 243 as the auxiliary captured image to be processed as it is, or the auxiliary captured image recorded in the recording unit without calculating the depth information after shooting. May be read and acquired. Therefore, in the present embodiment, the depth information is calculated using the camera unit 230 as the image arithmetic device, but the depth information can also be calculated by an independent image arithmetic device separate from the camera.

  For example, taking a PC as an example, a recording medium can be interposed between the camera and an auxiliary captured image recorded on the recording medium can be read and processed, or a state where the camera is connected to the camera by wire or wirelessly can be established. If so, processing can be performed in conjunction with the shooting of the auxiliary captured image. In this case, distance information based on the output of the AF sensor 240 associated with or recorded integrally with the auxiliary captured image is also taken into the image arithmetic apparatus together with the auxiliary captured image. Further, according to the calculation method, the lens information of the lens unit 210 at the time of shooting the auxiliary captured image, the movement amount conversion function, and the like are also taken into the image calculation device.

  As for the distance information, any acquisition method may be used as long as the distance information includes at least one distance to the in-focus object and at least one distance to the non-focus object, even if the distance information is not based on the output of the AF sensor 240. It may be acquired by. Various acquisition methods are known for the distance to the object. For example, an infrared projection type active distance measurement method can be used. In the following flow, as described above, the distance information in which the object distance and the arrangement of the grouped distance measuring points 260 are recorded as a pair will be described as an example.

The image processing unit 246, extracted in step S102, the auxiliary captured image, R filter portion 304 red image formed through the, i.e., the I _r image formed from the output of a pixel R pixel filter is provided To do. Similarly, to extract the I _b image is I _g images and B blue image formed by passing through the filter unit 306 is a green image formed by passing through the G filter portion 305. When the image processing unit 246 extracts the wavelength band-specific images in this way, the image processing unit 246 delivers these images to the depth information calculation unit 251.

  In step S103, the depth information calculation unit 251 extracts distance information. If the information acquired as the distance information is in a state before being converted into distance such as distance measurement information, the distance to each object is calculated here. Further, the depth information calculation unit 251 counts how many pairs are formed by the arrangement of the distance measuring points 260 grouped with the object distance, and substitutes them into the variable n.

  The depth information calculation unit 251 proceeds to step S104 and sets the increment variable i to 1. In step S105, the object region and window size are determined for the object Oi corresponding to the i-th pair by the above-described procedure.

  The depth information calculation unit 251 proceeds to step S105 and determines a boundary for the object Oi using the local window defined by the determined window size in the vicinity of the determined object region. At this stage, the boundary between the adjacent objects has not been determined, so the adjustment between the boundaries has not yet been made, and the outline of the object has not been determined in that sense.

  The depth information calculation unit 251 increments the increment variable i in step S107 and proceeds to step S108. In step S108, it is determined whether or not the increment variable i is larger than the variable n. That is, it is determined whether or not processing has been completed for all pairs acquired as distance information. If the condition is not satisfied, the process returns to step S105, and if satisfied, the process proceeds to step S109.

  In step S109, the depth information calculation unit 251 confirms whether there is any inconsistency such as whether the boundaries established for each object intersect each other, and adjusts the inconsistent portion. The adjustment of the inconsistent portion is executed according to a predetermined algorithm, for example, giving priority to an object on the near point side. When the boundary adjustment is completed, it is confirmed as contour information and output as depth information in step S110.

  For the depth information, a separate file may be generated as the distance image data, or the auxiliary information may be recorded with the auxiliary captured image. When a separate file is generated separately, link information is recorded for the auxiliary captured image. Of course, as described above, it is also possible to output as an object table in which vector data for defining an outline for each object and corresponding distances are listed. A series of processing flow is complete | finished by the above.

  Next, a modified example of the processing flow will be described. In the above processing, the distance difference between objects is calculated from the acquired distance information, and the window size is calculated from the amount of parallax caused by the distance difference to define the local window. In this modification, a local window is defined by a plurality of window sizes set in advance, and an operation for extracting the distance and boundary of an object is performed on the entire region of the parallax image by each local window. Of the plurality of distances with respect to a certain object extracted in this way, the boundary by the local window that obtains the operation result that most closely matches the acquired distance information is determined as the boundary of the object. The processing flow will be described below.

FIG. 14 is a diagram illustrating another processing flow from acquisition of an auxiliary captured image to output of depth information. In step S201, the image processing unit 246 acquires an auxiliary captured image to be processed. In step S < _b > 102, the image processing unit 246 extracts I _r , I _g , and I _b parallax images from the auxiliary captured image, and delivers them to the depth information calculation unit 251. In step S203, the depth information calculation unit 251 extracts distance information, counts the number of pairs based on the arrangement of the distance measuring points 260 grouped with the object distance, and substitutes it into a variable n.

  In step S204, the image processing unit 246 sets the increment variable j to 1. In step S205, the entire area of each parallax image is searched for in the local window defined by the jth window size Wj among the m window sizes set in advance, and the distance and boundary of each object are determined. Calculate. In addition, since the window size at this stage includes a size that is not appropriate for the amount of parallax, the calculation result may include an error or the like as described above.

  The depth information calculation unit 251 increments the increment variable j in step S206 and proceeds to step S207. In step S207, it is determined whether or not the increment variable j is larger than the variable m. That is, it is determined whether or not the calculation has been completed for all the m window sizes set in advance. If the condition is not satisfied, the process returns to step S205, and if satisfied, the process proceeds to step S208.

  The depth information calculation unit 251 sets the increment variable i to 1 in step S208. In step S209, for the object Oi corresponding to the i-th pair, the m distances calculated in step S205 are compared with the object distance based on the distance information.

Depth information calculating section 251 proceeds to step S210, the distance select the one closest to the object distance Di by the information, the object Oi vicinity computed by the window size Wj ₀ calculated this out of the computed m-number of distance Is determined as the boundary in the vicinity of the object Oi.

  The depth information calculation unit 251 increments the increment variable i in step S211 and proceeds to step S212. In step S212, it is determined whether or not the increment variable i is larger than the variable n. That is, it is determined whether or not processing has been completed for all pairs acquired as distance information. If the condition is not satisfied, the process returns to step S208, and if satisfied, the process proceeds to step S213.

  In step S213, the depth information calculation unit 251 checks whether there is any inconsistency, such as whether the boundaries determined for each object intersect each other, and adjusts the inconsistency location. When the boundary adjustment is completed, it is confirmed as contour information, and is output as depth information in step S214. A series of processing flow is complete | finished by the above.

  In the above embodiment, the in-focus object and the out-of-focus object are determined according to the distance measuring point of the AF sensor 240, but at least one of the plurality of objects is designated by the user via the operation member. You may comprise. In this case, the depth information calculation unit 251 determines the window size for the object specified by the user, and executes the calculation of depth information.

  In the above embodiments, a single-lens reflex camera has been described as an example. However, imaging such as an interchangeable lens camera without an optical finder, a compact camera with an integrated lens unit, and a video camera that can also shoot moving images. It can also be applied to a device.

  In the above embodiment, a single-plate image sensor having a Bayer array color filter as an image sensor has been described as an example. However, other color filter arrays may be used, and not limited to RGB. A filter that transmits the wavelength band may be arranged. In this case, the transmission wavelength band is selected for the filter portion of the structured aperture according to the wavelength band of the color filter of the image sensor.

  Further, the present invention can be applied to an image pickup element that does not have a color filter and has different sensitive wavelength bands in the depth direction of the photodiode. In this case, the transmission wavelength band is selected for the filter portion of the structured aperture according to the wavelength band separated in the depth direction of the photodiode. Furthermore, the present invention is not limited to a camera having a single-plate image sensor, but separates the incident light flux into a plurality of wavelength bands, such as a three-plate camera divided into RGB, for example, to make each wavelength band independent. The present invention can also be applied to a camera including a plurality of image sensors that receive light. Moreover, if the filter which transmits two wavelength bands among RGB is arranged, the above-mentioned distance information acquisition can be performed, so one of the RGB filters of the structured aperture according to the above embodiment is It may be removed.

  Further, the acquired distance information may not be the absolute distance from the camera to each object. In this case, if the relative distance from the focused object to the non-focused object is acquired, the window size based on the parallax amount can be determined. Further, the distance information between the out-of-focus objects may be calculated using the acquired distance information, and may be used for an evaluation calculation for evaluating the certainty of the calculation result of the depth information.

  In the above embodiment, the window size is determined based on the parallax amount calculated corresponding to the object distance difference. However, the window size applied in the object area has some tolerance. Specifically, when the local window is moved within the window size, if the boundary of the object is not included in the local window at the position, the window size may be enlarged as long as the boundary is not included. . Therefore, the range in which the window size can be enlarged is small for an object having a complicated boundary.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

101 Girl, 102 Boy, 103 Woman, 200 Single-lens reflex camera, 201 Optical axis, 210 Lens unit, 211 Lens group, 212 Focus lens, 213 Zoom lens, 214 Aperture, 216 Lens system control unit, 217 Lens barrel, 218 lens Mount, 219 Mounting slit, 220 Insertion / extraction guide, 221 Position sensor, 230 Camera unit, 231 Camera mount, 232 Main mirror, 233 Rotating shaft, 234 Focus plate, 235 Pentaprism, 236 Eyepiece optical system, 237 AE sensor, 238 Sub mirror, 239 AF optical system, 240 AF sensor, 241 focal plane shutter, 242 optical low-pass filter, 243 image sensor, 244 main board, 245 camera system control unit, 246 Image processing unit, 247 display unit, 248 secondary battery, 249 flash, 251 depth information calculation unit, 252 camera memory, 253 work memory, 254 external connection IF, 255 release switch, 260 distance measuring point, 261 focusing distance measuring point 262, first out-of-focus distance measuring point, 263 second out-of-focus distance measuring point, 271 in-focus object area, 272 first out-of-focus object area, 273 second out-of-focus object area, 281, 282, 283 Local window, 300 structured aperture unit, 302 filter, 303 blocking filter section, 304 R filter section, 305 G filter section, 306 B filter section, 307 base section, 308 outer peripheral section, 309 gripping section, 701 B curve, 702 G Curve, 703 R curve, 711, 712, 713 Arrow

Claims (8)

An image acquisition unit that outputs parallax image data that is output from a light receiving element that detects and separates incident subject light flux into a plurality of wavelength bands, and images corresponding to at least two of the plurality of wavelength bands have a parallax;
A distance information acquisition unit that acquires distance information to at least one in-focus object in the in-focus state and distance information to at least one in-focus object in the in-focus state in the parallax image data;
By performing a matching process between the parallax image data using a local window defined by a window size based on the distance information, a plurality of objects included in the parallax image are separated in the depth direction to define an outline. An image calculation device comprising a calculation unit.   2. The calculation unit determines a plurality of window sizes according to the plurality of objects, and performs the matching process using a plurality of local windows respectively defined by the plurality of determined window sizes. The image arithmetic device described in 1.   The image calculation device according to claim 1, wherein the calculation unit determines the window size based on a distance difference between the focused object and the non-focused object.   The calculation unit executes the matching process for each of a plurality of predetermined window sizes, and determines the contour using the window size when the calculated distance most closely matches the distance information. Alternatively, the image arithmetic apparatus according to 2. A designation unit that allows a user to designate at least one of the plurality of objects;
The image calculation device according to claim 1, wherein the calculation unit determines the window size for at least one of the plurality of objects specified by the user.   The image calculation device according to claim 1, wherein the distance information is information based on an output of an AF sensor when the parallax image data is captured. The distance information includes group information in which a plurality of focal points of the AF sensor are divided into a plurality of groups based on the output,
The image calculation device according to claim 6, wherein the calculation unit determines a plurality of window sizes based on the group information. An image acquisition step of acquiring parallax image data output from a light receiving element that detects and splits an incident subject luminous flux into a plurality of wavelength bands, and images corresponding to at least two of the plurality of wavelength bands have parallax with each other;
A distance information acquisition step of acquiring distance information to at least one in-focus object in the in-focus state and distance information to at least one in-focus object in the in-focus state in the parallax image data;
By performing a matching process between the parallax image data using a local window defined by a window size based on the distance information, a plurality of objects included in the parallax image are separated in the depth direction to define an outline. An image calculation program for causing a computer to execute calculation steps.

Images