JP2017009942A - Imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device that can acquire a photographing image having an entire area of a specific subject (a main subject) tilting toward a depth direction of an imaging screen focused, and having a background other than the main subject naturally blurred.SOLUTION: An imaging device comprises: first determination means that determines a main subject by pupil division means on an image pick-up element; second determination means that determines other than the main subject as a background; computation means that computes a tilt of an image formation plane where an entire area of the main subject falls within a depth of field from a defocus map; image generation position setting means that sets an image generation position; angle-of-incidence determination means that determines an angle of incidence upon each pixel; image generation means that determines an amount of shift of an image signal corresponding to the image generation position for each pupil area to shaft the image signal, in accordance with the image generation position set by the image generation position setting means and information on the angle-of-incident determination means, and adds the image signals; and background generation position determination means that determines the image generation position of the background determined by the second determination means as a position with continuity to the image generation position of the main subject determined by the first determination means.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging apparatus typified by a digital camera, and more particularly to image generation of a main subject and a background in a shooting screen.

  Conventionally, in a shooting scene such as a panning shot, when the main subject is in focus and the background is to be blurred, shooting is performed with the aperture set to a value close to or fully open. In such a scene, if the main subject is tilted in the depth direction of the screen and the near side is in focus, the back side of the main subject is out of depth of field and is not in focus. It may end up.

  Therefore, as described in Patent Document 1, by combining the received light signals obtained by the image sensor based on the defocus amount data at a plurality of positions, a composite image focused on a plurality of subjects in the shooting screen is obtained. By applying the obtained technology, it is conceivable that a composite image in which the near side and the far side are in focus can be obtained even with the main subject tilted in the depth direction of the screen as described above.

JP 2010-213038 A

  However, in the case of obtaining a composite image in which the entire area of the main subject is in focus by applying the technique disclosed in Patent Document 1, the background is also in focus other than the main subject having the same defocus amount as the main subject. In some cases, it becomes impossible to obtain a photographed image with a blurred background intended by the photographer.

  SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an imaging apparatus capable of acquiring a shot image in which the entire area of a specific subject (main subject) tilted in the depth direction of the shooting screen is in focus and the background other than the main subject is naturally blurred. Is to provide.

An imaging apparatus according to the present invention
In an imaging apparatus including an imaging optical system including an imaging lens, and an imaging element that photoelectrically converts an optical image of a subject incident through the imaging lens and outputs an image signal,
Pupil dividing means for restricting a light beam of the optical image of the subject incident on each pixel on the image sensor to only a light beam from a specific pupil region of the photographing lens;
Correlation calculation means for calculating the correlation of images passing through different pupil regions;
First determination means for determining a main subject from the result of the correlation calculation means;
Second determination means for determining an optical image of a subject other than the main subject as a background from the first determination means;
A defocus map of the background determined by the second determination unit is created, and a defocus map of the main subject determined by the first determination unit is generated, and the entire area of the main subject is within the depth. Computing means for computing the tilt of the imaging plane;
Image generation position setting means for setting an image generation position for generating a subject image from the image signal based on a result of the calculation means;
Incident angle determining means for determining an incident angle to each pixel on the image sensor;
Based on the image generation position set by the image generation position setting means and the information of the incident angle determination means, the shift amount of the image signal corresponding to the image generation position is determined for each pupil region and the image Image generating means for shifting the signals and adding the electrical signals;
Background generation position determination means for determining the background image generation position determined by the second determination means to be a position that is continuous with the image generation position of the main subject determined by the first determination means. It is characterized by that.

  According to the present invention, there is provided an imaging apparatus capable of acquiring a captured image in which the entire area of a specific subject (main subject) tilted in the depth direction of the shooting screen is in focus and the background other than the main subject is naturally blurred. I can do it.

Block diagram of the camera system Rear view of camera 100 Schematic diagram of optical system of imaging device in embodiment The figure which shows the example of a display of the display part 258 at the time of imaging | photography. Operation flow diagram during composite shooting in the embodiment Schematic diagram showing image generation operation in the embodiment Explanatory diagram of relationship between image generation operation and subject image Operation sub-flow diagram regarding background image generation position at the time of composite shooting in the embodiment Explanatory drawing of the relationship between the background image generation position and subject image at the time of composite shooting in the embodiment Operation flow diagram of correlation calculation in the embodiment Operation flow diagram of image generating means in embodiment

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.
[Example]
The imaging apparatus of the present invention will be described with reference to FIGS. FIG. 1 is a block diagram showing an electrical configuration of a digital camera and a lens which are imaging devices. A camera system including the camera 100 and the lens 102 includes an imaging system, an image processing system, a recording / reproducing system, and a control system. The imaging system includes a photographing optical system 101 and an imaging element 252, and the image processing system includes an image processing unit 150. The recording / reproducing system includes a memory 197 and a display unit 258, and the control system includes a body CPU 109, a lens CPU 103, a zoom lens driving unit 111 that is a lens driving unit, and an AF lens driving unit 121.

  FIG. 2 is a rear view of the camera system. In the camera system according to the present embodiment, a photographing optical system 101 is provided in a lens 102 that can be attached to and detached from the camera 100. The imaging system is an optical processing system that forms an image of light from an object on the imaging surface of the imaging element 252 via the imaging optical system 101.

  The photographing optical system 101 includes a zoom lens 110, and can be driven in the optical axis direction by a zoom lens driving unit 111 using an ultrasonic motor or a stepping motor as a driving source. The zoom lens voltage driver 113 generates a voltage for driving and controlling the zoom lens driving unit 111.

  The zoom position detection unit 112 is a zoom encoder that detects the position of the zoom lens 110 along the optical axis I. The zoom position detection unit 112 outputs a pulse signal corresponding to the focal length value to the zoom lens control unit 105.

  The zoom lens control unit 105 controls the zoom lens driving unit 111 so that, for example, the zoom lens 110 is driven to a position on the optical axis I that becomes a focal length set by the photographer with the camera 100 or the lens 102 using a switch (not shown). This is the part that performs tracking control. Further, the zoom lens control unit 105 takes in the pulse signal output from the zoom position detection unit 112. The zoom lens control unit 105 calculates a drive signal based on the position (target position) on the optical axis I that becomes the focal length set by the photographer, the current position information of the zoom lens 110, and the like, and zooms the digital drive signal. Output to the lens voltage driver 113.

  The zoom lens voltage driver 113 is a driver unit that supplies power to the zoom lens driving unit 111 in accordance with an input driving signal (driving voltage). The zoom lens voltage driver 113 switches the drive signal, applies a voltage to the zoom lens drive unit 111, and drives the zoom lens drive unit 111.

  The photographing optical system 101 includes an AF lens 120 and can be driven in the optical axis direction by an AF lens driving unit 121 using an ultrasonic motor or a stepping motor as a driving source. The AF lens voltage driver 123 generates a voltage for driving and controlling the AF lens driving unit 121.

  The focus position detection unit 122 is a focus encoder that detects the position of the AF lens 120 along the optical axis I. The focus position detection unit 122 outputs a pulse signal corresponding to the subject distance value to the AF lens control unit 104.

  The AF lens control unit 104 controls the AF lens driving unit 121 so that the AF lens 120 is driven to a position on the optical axis I at which the subject distance is detected by operating the release switch 191 of the camera 100 of the photographer, for example. This is the part that performs the follow-up control. The AF lens control unit 104 captures a pulse signal output from the focus position detection unit 122. The AF lens control unit 104 calculates a drive signal based on the position (target position) on the optical axis I that is the subject distance, the current position information of the AF lens 120, and the like, and this digital drive signal is sent to the AF lens voltage driver 123. Output.

  The AF lens voltage driver 123 is a driver unit that supplies power to the AF lens driving unit 121 in accordance with an input driving signal (driving voltage). The AF lens voltage driver 123 performs switching on the driving signal, applies a voltage to the AF lens driving unit 121, and drives the AF lens driving unit 121.

  The photographing optical system 101 has a diaphragm 140, and the size of the aperture of the diaphragm 140 is changed by a diaphragm driving unit 141 using a stepping motor or the like as a driving source. The aperture control unit 106 controls the aperture drive unit 141, for example, calculates an aperture value that is an appropriate exposure amount according to the brightness of the subject to be photographed, calculates a drive signal that is the aperture value, and performs aperture drive. Output to the unit 141.

  Alternatively, as described later, a drive signal is calculated according to the aperture value set by the photographer with the aperture setting switch 194 and output to the aperture drive unit 141. The lens CPU 103 is a central processing unit that performs various controls on the lens 102 side. In the lens CPU 103, a zoom lens control unit 105, an AF lens control unit 104, and an aperture control unit 106 are provided.

  The lens CPU 103 can communicate with the body CPU 109 via a lens contact 190 provided between the lens 102 and the camera 100. The EEPROM 131 is a non-volatile storage unit that stores lens data that is various kinds of unique information regarding the lens 102.

  The body CPU 109 is a central processing unit that performs various controls of the entire camera system. Information on the release switch 191 is input to the body CPU 109, and it can be detected that the release switch 191 is half-pressed or fully pressed. Thereby, driving of the image sensor 252, operation of the image processing unit 150, compression processing of the memory 197, and the like are controlled. Further, the state of each segment of the information display device that displays information on a liquid crystal monitor or the like is controlled by the display unit 258.

  As will be described later, the body CPU 109 has a function of determining a subject that is a main subject at the time of shooting by a photographer and a function of determining a subject other than the main subject as a background. Therefore, the body CPU 109 corresponds to the first determination unit and the second determination unit of the present invention.

  The image processing unit 150 includes an A / D converter, a white balance circuit, a gamma correction circuit, an interpolation calculation circuit, and the like, and can generate a recording image. In addition, image generation means, correlation calculation means, and the like, which are the main parts of the present invention, can be included (in this embodiment, these elements are described assuming that they are arranged in the body CPU 109).

  Specifically, the image sensor 252 is a CMOS process compatible sensor (hereinafter abbreviated as a CMOS sensor) which is one of amplification type solid-state image sensors. One of the features of the CMOS sensor is that the MOS transistors in the area sensor and the peripheral circuits such as the imaging device drive circuit, AD converter circuit, and image processing circuit can be formed in the same process. It can be greatly reduced. Further, it has a feature that random access to an arbitrary pixel is possible, reading that is thinned for display is easy, and real-time display can be performed at a high display rate on the display unit 258.

  The image sensor 252 utilizes the above-described features, and performs a display image output operation (reading in a region where a part of the light receiving region of the image sensor 252 is thinned) and a high-definition image output operation (reading in the entire light receiving region). Do.

  Microlenses are arranged in a lattice pattern on the surface of the image sensor 252 to form a so-called microlens array (hereinafter referred to as MLA20). In this embodiment, the MLA 20 constitutes pupil dividing means. Details of the functions and arrangement of the MLA 20 will be described later with reference to FIG. As will be described later, since the focus evaluation amount / appropriate exposure amount can be obtained from the image sensor 252, the imaging optical system 101 is appropriately adjusted based on this signal, so that an appropriate amount of object light is captured by the image sensor 252. And an object image is formed near the image sensor 252.

  As will be described later, the body CPU 109 operates the image processing unit 150 to obtain the defocus amount of the captured image from the subject image formed on the image sensor 252 so that the defocus map between the background 320 and the main subject 310 is obtained. And the inclination of the image plane where the entire area of the main subject 310 is within the depth is calculated, and thus corresponds to the calculation means of the present invention.

  Further, as will be described later, the body CPU 109 operates the image processing unit 150 to set the image generation position of the subject image formed on the image sensor 252 based on the calculation result, so that the image generation position setting of the present invention is performed. It also corresponds to means.

  Furthermore, as will be described later, the body CPU 109 determines the image generation position of the background 320 to be a position that is continuous with the image generation position of the main subject 310, and therefore corresponds to the background generation position determination means of the present invention.

  FIG. 3 is a diagram for explaining a main part of the photographing optical system in the present embodiment. In order to apply the present invention, it is necessary to acquire angle information in addition to the position of light rays, so-called light space information. In this embodiment, the MLA 20 is disposed in the vicinity of the imaging plane of the photographing optical system 101 for obtaining angle information, and a plurality of pixels are associated with one lens constituting the MLA 20.

  FIG. 3A is a diagram schematically showing the relationship between the image sensor 252 and the MLA 20. FIG. 3B is a schematic diagram showing the correspondence between the pixels of the image sensor 252 and the MLA 20. FIG. 3C is a diagram illustrating that pixels provided below the MLA 20 are associated with a specific pupil region by the MLA 20.

  As shown in FIG. 3A, the MLA 20 is provided on the image sensor 252, and the front principal point of the MLA 20 is arranged in the vicinity of the imaging plane of the photographing optical system 101. FIG. 3A shows a state where the MLA 20 is viewed from the side and from the front of the imaging device, and the lens of the MLA 20 is disposed so as to cover the pixels on the imaging element 252 when viewed from the front of the imaging device. In FIG. 3A, the microlenses constituting the MLA 20 are illustrated in large size so that they can be easily seen. However, in actuality, each microlens is only about several times as large as a pixel (for the actual size). This will be described with reference to FIG.

  FIG. 3B is a partially enlarged view from the front of the apparatus of FIG. A grid-like frame shown in FIG. 3B indicates each pixel of the image sensor 252. On the other hand, each microlens constituting the MLA 20 is indicated by thick circles 20a, 20b, 20c, and 20d. As apparent from FIG. 3B, a plurality of pixels are assigned to one microlens. In the example of FIG. 3B, 5 rows × 5 columns = 25 pixels are one microlens. (That is, the size of each microlens is 5 × 5 times the size of the pixel).

  FIG. 3C is a diagram in which the image sensor 252 is cut so that the micro lens optical axis is included and the longitudinal direction of the image sensor 252 is in the horizontal direction of the drawing. 3, 21, 22, 23, 24, and 25 indicate pixels (one photoelectric conversion unit) of the image sensor 252.

  On the other hand, the figure shown above FIG. 3C shows the exit pupil plane of the photographing optical system 101. Actually, when the direction is aligned with the image sensor 252 shown in the lower part of FIG. 3C, the exit pupil plane is perpendicular to the plane of FIG. 3C, but the projection direction is changed for the sake of explanation. I am letting. In FIG. 3C, one-dimensional projection / signal processing will be described to simplify the description. In an actual apparatus, this can be easily extended to two dimensions.

  Pixels 21, 22, 23, 24, and 25 in FIG. 3C are in a positional relationship corresponding to 21a, 22a, 23a, 24a, and 25a in FIG. As shown in FIG. 3C, each pixel is designed to be conjugate with a specific region on the exit pupil plane of the photographing optical system 101 by the microlens. In the example of FIG. 3C, the pixel 21 and the region 31 correspond to the pixel 22 and the region 32, the pixel 23 and the region 33, the pixel 24 and the region 34, and the pixel 25 and the region 35, respectively. That is, only the light beam that has passed through the region 31 on the exit pupil plane of the photographing optical system 101 enters the pixel 21. The same applies to the other pixels. As a result, angle information can be acquired from the passing area on the pupil plane and the positional relationship on the image sensor 252.

Here, symbols are introduced to simplify the explanation later. As shown in FIG. 3C, the pixel pitch of the image sensor 252 is ΔX, and the angular resolution is Δθ. Further, the number of angle divisions is N _θ (N _θ = 5 in the example of FIG. 3). The pixel pitch is determined by the shape of the image sensor 252, and Δθ is determined by the range for acquiring the angle of the light beam and the number of angle divisions N _θ . That is, these parameters are determined only by the physical structure (the structures of the image sensor 252 and the MLA 20).

  Therefore, the body CPU 109 can determine the angle information on the image sensor 252 by receiving the information of the exit pupil position of the photographing optical system 101 from the lens 102, that is, the incident angle to each pixel. It also corresponds to a determination means.

  As described above, the release switch 191 detects a half-press operation of the release button (not shown) (ON of SW1), starts a series of shooting preparation operations (photometry operation, focus adjustment operation, etc.), and fully presses the release button. This switch detects an operation (ON of SW2) and starts a photographing operation (recording of image data read from the image sensor 252 on a recording medium).

  Reference numeral 192 denotes a selection switch that can be operated in an arbitrary direction, and is a switch that is used when a well-known shooting menu item (not shown) is selected or when a photographer selects a main subject as will be described later. Reference numeral 193 denotes a determination switch which is used when a known shooting menu (not shown) is determined or when a photographer determines a main subject as will be described later. An aperture setting switch 194 is a switch that allows the photographer to set an aperture value by operating the operation dial (not shown).

  Reference numeral 195 denotes a main switch for starting up the camera 100. As shown in FIG. 2, the main switch 195 is a switch capable of selecting two positions (195-1, 195-2). When the main switch 195 is set to the position 195-1, it is “OFF”, and the camera 100 enters a sleep state as will be described later. When the main switch 195 is set to the position 195-2, it is “ON”, and as will be described later, the camera 100 changes from the sleep state to the drive state, and accepts various switch operations and the like for taking a still image or a moving image. It will be in the state which performs operation | movement of.

  Reference numeral 196 denotes a mode switch for setting a shooting mode of the camera 100, which can set a “composite shooting mode” to be described later. A memory 197 includes a processing circuit necessary for recording in addition to an actual storage unit. The memory means outputs to the recording unit and generates and stores an image to be output to the display unit 258. In addition, the memory 197 compresses images, moving images, sounds, and the like using a predetermined method.

  The display unit 258 is attached to the back of the camera 100 so that the photographer can directly observe the display on the display unit 258. If the display unit 258 is formed of an organic EL spatial modulation element, a liquid crystal spatial modulation element, a spatial modulation element using microparticle electrophoresis, power consumption can be reduced, and the display unit 258 can be thinned. . Thereby, the power saving and size reduction of the camera 100 can be achieved.

  FIG. 4 is a diagram illustrating a state of a subject image displayed on the display unit 258 during so-called live view shooting. In FIG. 4, the display unit 258 displays a subject image 258a and shooting information (258b) such as shooting conditions (shutter speed, aperture value, ISO sensitivity, etc.), exposure level display, and battery level display.

  In the subject image 258a display section, a plurality of distance measuring points (ranging areas) 301 are provided, and the distance measuring points 302 selected based on the focus detection result from the image sensor 252 are displayed in another display form ( By displaying in different colors, etc., it is possible to inform the photographer of the main subject portion in focus. In addition, the photographer can select an arbitrary position to be focused from among a plurality of distance measuring points 301 by using the selection switch 192 and the determination switch 193 of the camera 100 regardless of the detection result of the distance measuring unit 253. It is. In that case, the photographer can prevent the image from being out of focus by calculating the distance to the main subject from the focus detection result from the image sensor 252 based on the distance measuring point 302 selected by the photographer.

  FIG. 5 is a flowchart for explaining the operation when the camera system of the present invention performs the depth composite photographing, and the operation of the camera 100 and the lens 102 during the composite photographing will be described below using these. Note that the composite photography referred to here is a feature of the present invention, in which the background 320 of the main subject 310 is blurred, but the main subject 310 is an image that can obtain an image in which the entire area (≈full depth) is in focus. It is a thing.

  In step (hereinafter referred to as S) 1010, the body CPU 109 determines whether or not the main switch 195 is set to the position 195-2 and turned ON. If the main switch 195 is set to the position 195-1 (off), the camera 100 remains in the sleep state, and if the main switch 195 is set to the position 195-2 and turned on, the process proceeds to S1020. move on.

  In S1020, the body CPU 109 determines whether or not the camera 100 is set to the composite shooting mode by the mode switch 196. If it is Yes, it will progress to S1030. On the other hand, if No, the process proceeds to S1250, and a normal still image shooting operation is performed. The operation of S1250 is a known operation and is different from the contents of the present invention, and thus detailed description thereof is omitted.

  In S1030, the body CPU 109 determines whether or not the release switch 191 has been half-pressed. If YES, the process proceeds to S1040. In S1040, the body CPU 109 reads the aperture value set by the photographer using the aperture setting switch 194, and transmits it to the lens CPU 103 via the lens contact 190. Then, the aperture controller 106 drives and controls the aperture driver 141 according to the aperture value so that the aperture 140 has an aperture diameter corresponding to the aperture value set by the photographer.

  In S1050, the image sensor 252 is exposed for an appropriate time and read (A / D converted) to acquire data.

  Although an appropriate exposure amount can be calculated from the exposure time and exposure amount at this time, the description is omitted because it is not a main part of the present invention. In S1060, the correlation calculation means is operated to obtain a result. Details of the operation of the correlation calculating means will be described later with reference to FIG. 10, but at this step, the focus position by so-called phase difference AF can be obtained.

  In S1070, the lens CPU 103 receives the calculation result of S1060 from the body CPU 109 via the lens contact 190, and then the AF lens 120 is driven by the AF lens driving unit 121, and the main subject 310 is focused at the distance measuring point 302 in S1060. The AF lens 120 is moved to a position to be in a state, and image data for each pupil region that has passed at the position of the AF lens 120 is acquired as in S1050.

  In S1080, the defocus amount at each distance measuring point 301 is calculated. It should be noted that the subject in the shooting range (in the area displayed as the subject image 258a in FIG. 4) and outside the distance measuring area (where the distance measuring point 301 is not arranged) is also processed by S1060. The amount of defocus is determined.

  In S1090, the body CPU 109 determines the subject that the photographer is the main subject from the defocus amount at each distance measuring point 301 obtained in S1080. Specifically, for example, the number of distance measurement points 301 having a specific value and a defocus amount in the vicinity thereof is large from the distribution state of the defocus amount at each distance measurement point 301 and the difference between the absolute values. The area is specified, and the subject in the area is determined to be the main subject 310, and the smallest defocus amount in the area is displayed as the focus point 302 at the focus position (see FIG. 4).

  Alternatively, as described above, when the photographer designates the distance measuring point 302 at an arbitrary position to be focused from among the plurality of distance measuring points 301 using the selection switch 192 and the determination switch 193 of the camera 100. The subject at the distance measuring point 302 is determined as the main subject 310. Alternatively, the body CPU 109 selects the subject in the area where the distance measuring points 301 that are equal to or near the defocus amount at the distance measuring point 302 designated by the photographer are distributed. You may determine with 310.

  In this case, when the photographer designates the main subject 310 using the selection switch 192 and the decision switch 193 of the camera 100, for example, “select / determine the main subject” in the display unit 258, etc. If the display is performed to assist the photographer, convenience for the photographer is improved. Regardless of which method is used as the main subject 310, the distance measuring point 302 is a reference position at which the body CPU 109 recognizes the main subject 310 and drives the AF lens 120 to adjust the focus position of the photographing optical system 101. It becomes.

  In S1100, a range in which the main subject 310 determined by the body CPU 109 in S1090 is captured within the shooting range (hereinafter referred to as main subject area 311—see FIG. 7) is calculated. Specifically, a range of pixels that are equal to or near the defocus amount at the distance measuring point 302 on the main subject 310 is calculated as the main subject area 311.

  This is because the defocus amount changes substantially continuously in the portion (or the whole) in which the main subject 310 is within the shooting range, so the defocus amount within the shooting range obtained in S1060 and S1080. By comparing the defocus amount of the main subject 310 with the defocus amount of the background 320 or the like (see FIG. 4) based on the calculation result, the boundary of adjacent pixels where the defocus amount has changed greatly is used as the edge of the main subject 310. The main subject area 311 can be calculated. As described above, since the defocus amount in the main subject area 311 can be calculated, the body CPU 109 creates a defocus map in the main subject area 311.

  The defocus map here refers to a distribution state indicating whether each part of the main subject 310 is on the front pin side or the rear pin side in the main subject area 311 when the distance measuring point 302 is used as a reference. It is a thing. In S1100, the main subject area 311 may be calculated by, for example, detecting an edge from a known subject contrast change or color change (the main subject area 311 is calculated using the defocus amount). Is not limited).

  In S1110, when performing composite photography (image acquisition), which will be described later, the entire area of the main subject area 311 falls within the depth, that is, the defocus map and zoom in the main subject area 311 that are the calculation results in S1100. From the focal length information of the photographing optical system 101 that is the detection result of the position detection unit 112, the body CPU 109 performs the calculation of the tilt of the imaging plane that focuses on the entire main subject 310.

  In S1120, the background 320 determined by the body CPU 109 in S1090 is calculated. Specifically, the area excluding the main subject area 311 calculated in S1100 from the shooting range is calculated as the background 320. As described above, since the defocus amount in the background 320 can be calculated in S1080, the body CPU 109 creates a defocus map in the background 320. At this time, as in S1100, the distribution state of whether each part of the subject in the background 320 is on the front pin side or the rear pin side when the distance measuring point 302 is used as a reference is known.

  In S1130, as will be described in a later-described subflow (FIG. 8), the body CPU 109 calculates the image generation position of the background 320 according to the defocus map of the main subject area 311 that is performed when performing composite shooting (image acquisition). Do. In step S1140, the image generation unit included in the image processing unit 150 is operated to obtain a result. The details of the operation of the image generation means will be described later with reference to FIG. 11, whereby an image of the main subject 310 imaged on the image sensor 252 (image focused at the distance measuring point 302) is generated in the memory 197. Is done.

  In S1140, an image obtained by combining the main subject 310 and the background 320 based on the calculation results in S1110 and S1130 may be generated in the memory 197. In S1150, the image focused on the main subject 310 generated in S1140 is displayed on the display unit 258 as a preview image before photographing.

  The processing in S1140 and S1150 may be performed prior to the processing in S1080 to S1130, and the processing in S1080 to S1130 may be performed while the preview image is displayed in S1150. In this case, since the release switch 191 in S1030 can be confirmed by the preview image from the half-press operation to the preview operation during the calculation from S1080 to S1130, the photographer can A composition or the like to be photographed from now on can be confirmed on the display unit 258.

  In S1160, the body CPU 109 determines whether the release switch 191 has been fully depressed (whether SW2 is turned on). If No, the process returns to S1050 and a series of operations are repeated. If Yes, the process proceeds to S1170. In S1170, a known shooting operation is performed. In S1180, the image generation unit is operated using the image data of the pixel corresponding to the main subject area 311 which is the calculation result in S1100, and the result is obtained. Although this operation will be described later, in S1180, a subject image using the light flux on the tilted image plane calculated in S1110 is obtained.

  As a result, for example, as shown in FIG. 7, the main subject 310 surrounded by the main subject area 311 is closer to the optical axis direction than the distance measuring point 302 from the vicinity of the distance measuring point 302 as a result of the processing in S1180. The image is in focus up to the rear wheel 312 in the back of the drawing and to the vicinity of the front wheel 313 and the distance measuring point 303 near the front of the drawing in the optical axis direction from the distance measuring point.

  In S1190, the image generation unit is operated using the image data of the pixel corresponding to the background 320, which is the calculation result in S1120, and the image generation position, which is the calculation result in S1130, to obtain the result of the background 320.

  As a result, the background 320 becomes a “blurred” image as compared with the main subject 310 as a result of the processing in S1190, and an image of the background 320 is generated at a position continuous with the image generation position of the main subject 310. Therefore, it is possible to prevent the main subject 310 from floating unnaturally.

  In S1200, the pixels that have reached the entire area of the main subject 310 processed in S1180, that is, the light beam in the main subject area 311 and the subjects other than the main subject 310 processed in S1190, that is, pixels that have reached the light beam outside the main subject area 311. And synthesize. As a result, as shown in FIG. 7B, the background 320 of the main subject 310 is blurred even though the main subject 310 is an image in which the entire area (≈full depth) is in focus. I can do it.

  In S1210, the image generated in S1200 is recorded on the memory 197 or a recording medium (not shown) and displayed on the display unit 258. In S1220, the body CPU 109 determines whether or not the state where the release switch 191 is fully depressed (the state where SW2 is turned on) continues. If Yes, the process returns to S1050 to repeat a series of operations. If No, the process proceeds to S1230.

  In S1230, the body CPU 109 determines whether or not the release switch 191 is pressed (SW1 is turned on). If Yes, the process returns to S1040 to repeat a series of operations. If No, the process proceeds to S1240. In S1240, the body CPU 109 determines whether or not the main switch 195 is turned off. In the case of No, the process returns to S1030 to perform a series of shooting operations. If Yes in S1240, the shooting operation is terminated.

Here, the processing in S1180 will be described with reference to FIGS. As described with reference to FIG. 3, the imaging optical system of the present embodiment limits the luminous flux incident on each pixel to a specific pupil region, and thus the incident angle is known. Along this angle, the position of each pixel on the plane constituting the image is constructed. Specifically, it is assumed that a pupil region with a subscript of 1 such as X _{1, i} is incident at an angle as indicated by 41 in FIG. Hereinafter, it is assumed that the subscripts 2, 3, 4, and 5 of the pupil region correspond to 42, 43, 44, and 45, respectively.

  FIG. 6B shows a surface where the image sensor 252 is present at the position corresponding to the distance measuring point 302, that is, the position where the distance measuring point 302 is focused, and the image is acquired.

In FIG. 6B, X _{1, i} , X _{2, i} , X _{3, i} , X _{4, i} , X _{5, i} pass through the pupil regions 1, _{2, 3, 4, 5} respectively and are microscopic. Data obtained by entering the lens X _i is shown. That is, of the subscripts, the first half indicates the passing pupil region, and the second half indicates the pixel number. In FIG. 6B, the data is described as having only one-dimensional expansion for the sake of clarity. In relation to the physical position, X _{1, i} is attached with data obtained from the 21 area in FIG. 3C, and X _{2, i} is attached with data obtained from the 22 area in FIG. The letters 3, 4, and 5 indicate that they correspond to the areas 23, 24, and 25, respectively.

In order to generate an image on the acquisition surface, data incident on the microlens X _i may be added as shown in FIG. Specifically, the integrated value in the angular direction of the light incident on X _i can be obtained with S _i = X _{1, i} + X _{2, i} + X _{3, i} + X _{4, i} + X _{5, i} . As a result, an image of the subject focused on the distance measuring point 302 is generated as in a normal camera.

  Next, due to the inclination of the main subject 310, the portion of the main subject 310 that becomes the front pin from the distance measuring point 302 (for example, the portion of the main subject 310 at a position corresponding to the distance measuring point 303 shown in FIG. Image generation in the case of a lamp will be described with reference to FIG. When the main subject 310 is tilted, a portion of the main subject 310 that is on the front pin side of the distance measuring point 302 (for example, a portion corresponding to the distance measuring point 303 in FIG. 7) is blurred. Therefore, it is only necessary to generate a subject image as shown in FIG. 6A so that the portion of the main subject 310 on the front pin side of the distance measuring point 302 is not blurred.

6A, as in FIG. 6B, X _{1, i} , X _{2, i} , X _{3, i} , X _{4, i} , X _{5, i} are respectively pupil regions 1, 2, 3, 4 5 shows data obtained by passing through 5 and entering the microlens X _i .

As shown in FIG. 6A, the light beam incident on the micro lens X _i is dispersed from X _{i −2} to X _{i + 2} in the portion of the main subject 310 on the front pin side of the distance measuring point 302. Is incident.

More specifically, it is distributed in X1 _{, i-2} , X2 _{, i-1} , X3 _{, i} , X4 _{, i + 1} , and X5 _{, i + 2} . In order to restore the image without blurring the portion of the main subject 310 on the front pin side, not limited to X _i , it is understood that the image may be shifted and added according to the incident angle. In order to generate an image of the main subject 310 on the front pin side, one with a pupil region subscript of 1 shifts to the right by two pixels, and one with a pupil region subscript of 2 shifts to the right with one pixel. Shift, pupil area subscript 3 has no shift, pupil area subscript 4 has 1 pixel shift to the left, pupil area subscript 5 has a shift of 2 pixels to the left Shift according to the angle can be given.

Thereafter, the data of the portion of the main subject 310 on the front pin side can be obtained by adding in the diagonally lower right direction of FIG. 6A according to the inclination of the main subject 310. Specifically, _Si = X1 _{, i-2} + X2 _{, i-1} + X3 _{, i} + X4 _{, i + 1} + X5 _{, i + 2} of main subject 310 on the front pin side In the portion, the integral value in the angular direction of the light incident on X _i can be obtained.

  The above-described series of processing is performed at each part of the main subject 310 on the front pin side by changing the image shift amount in accordance with the inclination of the main subject 310 as the calculation result in S1110. An image with no blur of the subject 310 is obtained.

  Next, due to the inclination of the main subject 310, the portion of the main subject 310 that becomes a pin behind the distance measuring point 302 (for example, the portion of the main subject 310 at a position corresponding to the distance measuring point 304 shown in FIG. 7: the rider's boot) Image generation in the lower motorcycle body will be described with reference to FIG.

  When the main subject 310 is inclined, a portion of the main subject 310 that is on the rear pin side of the distance measuring point 302 (for example, a portion corresponding to the distance measuring point 304 in FIG. 7) is blurred. In view of this, the subject image may be generated as shown in FIG. 6C so that the portion of the main subject 310 on the rear pin side of the distance measuring point 302 is not blurred.

  As can be seen from FIG. 6 (c), an image is also obtained at the portion of the main subject 310 on the rear pin side of the distance measuring point 302 in the same manner as the portion of the main subject 310 on the front pin side of the distance measuring point 302. Can be generated. If the position in the optical axis direction is different from the acquisition plane (meaning whether it is the front side or the rear side with respect to the acquisition plane), the direction in which the pixel is shifted is reversed and the direction to be added according to the inclination of the main subject 310 It is only necessary to reverse.

That is, the data of the portion of the main subject 310 on the rear pin side can be obtained by adding in the diagonally lower left direction of FIG. 6C according to the inclination of the main subject 310. Specifically, S _i = X _{1, i + 2} + X _{2, i + 1} + X _{3, i} + X _{4, i-1} + X _{5, i-2} of the main subject 310 on the rear pin side In the portion, the integral value in the angular direction of the light incident on X _i can be obtained.

  The above-described series of processing is performed at each part of the main subject 310 on the rear pin side by changing the image shift amount according to the inclination of the main subject 310 as the calculation result in S1110. An image with no blur of the subject 310 is obtained.

  Next, the processing in S1130 will be described using the subflow of FIG. In S1310, the defocus amounts of the main subject area 311 and the background 320 are confirmed from the defocus amounts within the photographing range calculated in S1080 (in the area displayed as the subject image 258a in FIG. 4). In S1320, the defocus amount of the background 320 confirmed in S1310 is compared with the defocus amount of the main subject area 311.

  In S1330, the body CPU 109 determines whether or not the defocus amount between the main subject area 311 and the background 320 has continuity from the comparison result in S1320. If the body CPU 109 determines that there is continuity, the process proceeds to S1340. If it is determined that there is no continuity, the process proceeds to S1370.

  In S1340, the body CPU 109 determines whether the continuity determined by the body CPU 109 in S1330 is two or more at the boundary between the subject area 311 and the background 320. If there are two or more boundaries with continuity, the process proceeds to S1350. If there is only one boundary with continuity, the process proceeds to S1360.

  In S 1350, as a result of the determination in S 1330, the boundary closest to the photographer (closest to the photographer) is selected from the boundaries where the defocus amount between the main subject area 311 having two or more points and the background 320 is continuous. It is selected as a reference for the image generation position. For example, in the scene of FIG. 7, the defocus amount between the main subject area 311 and the background 320 becomes a continuous boundary around the front wheel 313 and the rear wheel 312 of the tire. In this case, in S1350, the front wheel close to the photographer The area around 313 is selected as a reference for the image generation position of the background 320.

  After the processing in S1350 is completed, the process returns to the caller (S1130). In S 1360, since there is only one continuous boundary in the defocus amount between the main subject area 311 and the background 320, the boundary is selected as a reference for the image generation position of the background 320. After the processing in S1360 is completed, the process returns to the caller (S1130).

  In S1370, since it is determined in S1330 that there is no continuous boundary in the defocus amount between the main subject area 311 and the background 320 (for example, in the case of a scene as shown in FIG. 9), Is selected as a reference for the image generation position of the background 320. After the processing in S1370 is completed, the process returns to the caller (S1130).

  Hereinafter, the sub-flow of the correlation calculation performed in S1060 will be described using FIG. S1410 indicates the start of the operation of the correlation calculation means included in the image processing unit 150.

  S1420 to S1520 form a loop. In S1420, calculation is repeatedly performed in accordance with the number of evaluation positions (so-called distance measurement visual field number). When the number of distance measuring fields increases, the entire screen can be covered, but there is a problem that it takes time to evaluate. Set appropriately according to the photographer's settings.

  In S1430, the number of evaluation points to be evaluated and the size of the evaluation frame are set. The number of evaluation points here is the number of points for obtaining the correlation value performed in S1440, and is appropriately set according to the shooting conditions, the type of the lens 102, and the like. The number of evaluation points corresponds to the shift amount in the case of obtaining the correlation while shifting the image, and this corresponds to the focus depth at which the focus search is performed in the imaging apparatus.

  On the other hand, if the size of the evaluation frame is increased, it is possible to focus even on textures that do not have a pattern locally, but if it is too large, images of subjects with different distances can be evaluated simultaneously. So-called perspective conflict will occur. The size of the evaluation frame is set appropriately so that these problems can be solved.

S1440 to S1490 form a loop. In S1440, calculation is repeatedly performed so as to obtain an evaluation value corresponding to the evaluation number determined in S1430. S1450 to S1470 form a loop. In S1450, correlation calculation is performed in the range of the number of pixels corresponding to the size of the evaluation frame determined in S1430. Correlation calculation may be performed by, for example, Σ | A _i −B _i | as in S1460.

Here, A _i indicates the luminance of the i-th pixel that has passed through a specific pupil region. B _i indicates the luminance of the i-th pixel that has passed through a pupil region different from A _i . For example, in FIG. 3, it is only necessary to set A _{i for} the pixels corresponding to the pixels 22 and B _i for only the pixels corresponding to the pixels 24. Which pixel of the pupil region to select may be determined according to the length of the base line length, the vignetting situation of the pupil surface, and the like.

  By setting as described above, the correlation between images passing through different pupil regions can be calculated, and an evaluation amount based on so-called phase difference AF can be obtained. In S1480, the obtained correlation value is stored as an evaluation amount.

In the evaluation formula Σ | A _i −B _i | shown above, the location where the correlation value is small corresponds to the location with the best focus state. Here, the correlation calculation is performed by the method of adding the difference absolute value, but the correlation calculation is performed by other calculation methods such as the method of adding the maximum value, the method of adding the minimum value, the method of adding the square difference value, and the like. In S1500, the correlation evaluation value is the best (the point where the correlation value is small in the formula of S1460, as described above, but the best may be determined using other indicators as well). Store as the best focus position based on the evaluation. This calculation is performed for each field of view, and finally the best focus position based on the correlation amount evaluation is obtained, the process proceeds to S1520, and the process returns to the caller (S1080).

Next, details of the operation of the image generating means included in the image processing unit 150 will be described with reference to FIG. S1610 indicates the start of the operation of the image generating means. In S1620, the area data for addition in S1650 is initialized (filled with 0). The size of the data area at this time may be the quantity of MLA 20, and the gradation of data is convenient if it can store the product of the gradation of the original data and the number of pupil divisions. For example, when the original data is 8 bits and divided into 25, if 13 bits (> 8 bits + log ₂ 25), it is not necessary to consider overflow of data.

  S1630 to S1670 form a loop. In S1630, loop calculation is executed in accordance with the number of microlenses constituting the MLA 20. For example, in the example illustrated in FIG. 3, the number of microlens is the number of pixels of the original image sensor ÷ 25 (number of pupil divisions). S1640 to S1660 form a loop. In S1640, the loop calculation is executed by the number corresponding to the number of pupil divisions. For example, in the example shown in FIG. 3, since it is divided into 5 × 5 = 25, the luminous fluxes from the 25 pupil positions are added in S1650. If the shift amount is not an integral multiple of the pixel, in S1650, addition is performed while being appropriately divided (addition may be appropriately performed according to the overlapping area).

  In S1680, the process returns to the calling source S1140 (or S1180, 1190).

  As described above, even when there is a specific subject tilted in the depth direction of the shooting screen, when the main subject and the photographer select it, the entire area of the main subject is in focus and other than the main subject Therefore, it is possible to provide an imaging apparatus capable of acquiring a naturally blurred photographed image.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  For example, in the present invention, the focus position is obtained by the correlation calculation unit included in the image processing unit 150. However, the present invention is not limited to this, and the camera 100 includes a known submirror and a distance measuring unit, and the imaging optical system from the submirror is used. The same effect as that of the present invention can be obtained by a method of receiving a light beam and obtaining a focus position by a phase difference detection method.

  In the present invention, the camera 100 and the lens 102 are detachable. However, the present invention is not limited to this, and the same effect as described above can be obtained even with a camera in which the camera 100 and the lens 102 are integrated. Needless to say.

20 micro lens array (MLA), 100 cameras, 102 lenses,
109 body CPU, 120 AF lens, 192 selection switch,
193 determination switch, 196 mode switch, 252 image sensor, 258 display unit,
301, 302, 303 Distance measuring points, 310 Main subject, 311 Main subject area,
320 Background

Claims (4)

A taking optical system including a taking lens;
In an imaging device including an imaging device that photoelectrically converts an optical image of a subject incident through the photographing lens and outputs an image signal,
Pupil dividing means for restricting a light beam of the optical image of the subject incident on each pixel on the image sensor to only a light beam from a specific pupil region of the photographing lens;
Correlation calculation means for calculating the correlation of images passing through different pupil regions;
First determination means for determining a main subject from the result of the correlation calculation means;
Second determination means for determining an optical image of a subject other than the main subject as a background from the first determination means;
A defocus map of the background determined by the second determination unit is created, and a defocus map of the main subject determined by the first determination unit is generated, and the entire area of the main subject is within the depth. Computing means for computing the tilt of the imaging plane;
Image generation position setting means for setting an image generation position for generating a subject image from the image signal based on a result of the calculation means;
Incident angle determining means for determining an incident angle to each pixel on the image sensor;
Based on the image generation position set by the image generation position setting means and the information of the incident angle determination means, the shift amount of the image signal corresponding to the image generation position is determined for each pupil region and the image Image generating means for shifting the signals and adding the electrical signals;
Background generation position determination means for determining the background image generation position determined by the second determination means to be a position that is continuous with the image generation position of the main subject determined by the first determination means. An imaging apparatus characterized by that. The imaging apparatus according to claim 1, wherein the first determination unit determines, as a main subject, a place where an area in which a defocus amount within a predetermined range is distributed in a shooting screen is the largest. The background generation position determination unit has a plurality of positions where the background defocus amount determined by the second determination unit and the defocus amount of the main subject determined by the first determination unit are continuous. The image pickup apparatus according to claim 1, wherein a position close to the image pickup apparatus in the optical axis direction is determined as a background image generation position. The background generation position determination unit determines whether the background defocus amount determined by the second determination unit and the defocus amount of the main subject determined by the first determination unit are not continuous. The imaging apparatus according to claim 1, wherein the focus position of the main subject calculated by the calculation unit is determined as a background image generation position.