JP2010109921A - Image synthesizing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To synthesize an image focused at a desired position to be focused from one captured image. <P>SOLUTION: An image synthesizing apparatus includes: a photographing optical system 3; a micro lens array 4a which includes a plurality of micro lenses arrayed in a two-dimensional manner and is disposed at the back of the photographing optical system 3; a photoelectric conversion device 4b which is disposed at the back of the micro lens array 4a and in which a plurality of photoelectric conversion area is assigned to each of the plurality of micro lenses; subject specifying means 5a, 5b, 5c for specifying an object to be focused; and an image creating means 5d for synthesizing an image focusing the object specified by the subject specifying means 5a, 5b, 5c on the basis of output of the photoelectric conversion device. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image composition apparatus.

A microlens array in which a plurality of microlenses are arranged two-dimensionally and a photoelectric conversion element array in which photoelectric conversion elements are arranged two-dimensionally are sequentially arranged behind the imaging optical system, and an arbitrary depth position (imaging optical system) There is known an image synthesis camera that synthesizes an image focused in the direction of the optical axis (see, for example, Non-Patent Document 1).
JP 2007-004471 A

   However, the above-described conventional image synthesizing camera synthesizes an image focused on an arbitrary depth position of a captured image, and does not touch on what kind of image is to be synthesized.

(1) The invention of claim 1 has a photographing optical system, a plurality of microlenses arranged two-dimensionally, a microlens array disposed behind the photographing optical system, and disposed behind the microlens array. A photoelectric conversion device in which a plurality of photoelectric conversion regions are assigned to each of a plurality of microlenses, a subject specifying unit for specifying a subject to be focused, and an image focused on the subject specified by the subject specifying unit. And image creating means for synthesizing based on the output of the conversion device.
(2) The invention of claim 2 is the image synthesizing apparatus according to claim 1, wherein the subject specifying means has a focus detecting means, and the image creating means is a photoelectric conversion apparatus previously acquired prior to focus detection by the focus detecting means. Based on the output, an image focused on the subject whose focus is detected by the focus detection means is created.
(3) The invention of claim 3 is the image composition apparatus according to claim 2, wherein the photoelectric conversion area of the photoelectric conversion device is conjugate with the exit pupil of the photographing optical system, and the focus detection means is assigned to each microlens. The amount of defocus to the subject is detected based on the outputs of the pair of photoelectric conversion regions among the plurality of photoelectric conversion regions.
(4) According to a fourth aspect of the present invention, in the image synthesizing apparatus according to the first aspect, the subject specifying means identifies a predesignated feature of the subject and performs focus detection on the feature portion by the focus detection means, Instructing means for instructing the image creating means to synthesize an image focused on the subject that is the focus detection target.
(5) The invention of claim 5 is the image composition apparatus according to claim 1, wherein the image composition means includes display means for creating and displaying an image with a deep focal depth based on the output of the image sensor, The means has setting means for setting a subject to be focused on based on the display on the display means, and obtains an image focused on the subject set by the setting means.
(6) The invention according to claim 6 is the image composition device according to any one of claims 1 to 5, wherein the subject specifying means specifies subjects at different positions in the optical axis direction of the photographing optical system, and The creating means creates one image focused on the plurality of identified subjects.

  According to the present invention, a focused image can be synthesized from a single captured image at a position where focus is desired.

  An embodiment in which the image composition apparatus of the present invention is applied to a digital camera will be described. FIG. 1 is a diagram showing the configuration of a digital camera according to an embodiment. The configuration of the embodiment will be described with reference to FIG. FIG. 1 shows only camera devices and devices that are directly related to the present invention.

  In a digital camera 1 according to an embodiment, a photographing lens 3 is attached to a camera body 2. The camera body 2 includes an image sensor 4, a drive control device 5, a memory 6, and the like. The imaging device 4 includes a microlens array 4a in which minute microlenses are densely arranged on a plane, and a photoelectric conversion element array 4b in which minute photoelectric conversion elements are densely arranged on a plane.

  The light receiving surface of the photoelectric conversion element array 4b is arranged at the position of the focal length of the microlens. The microlens array 4 a and the photoelectric conversion element array 4 b are arranged at a position where the light receiving surface of the photoelectric conversion element array 4 b is conjugated with the exit pupil plane of the photographing lens 3.

  The subject image by the photographic lens 3 is determined by the positional relationship of the subject in the depth direction (the position of the photographic lens 3 in the optical axis direction), and the subject focused on the front side of the microlens apex surface (the photographic lens 3 side) and the rear side. There is a subject to be focused or a subject to be focused at the apex surface of the microlens.

  FIG. 2 is a diagram of the microlens array 4a according to the embodiment as viewed from the photographing lens 3 side. In this embodiment, an example in which minute microlenses are densely arranged in a zigzag manner is shown, and odd columns (or odd rows) and even columns (or even rows) are alternately arranged. Note that the arrangement of the microlenses is not limited to the staggered arrangement, and may be, for example, a square arrangement or a honeycomb arrangement (hexagonal close arrangement).

  FIG. 3 is a diagram in which the outer diameter of one microlens of the microlens array 4a is projected onto the photoelectric conversion element array 4b (the photoelectric conversion element of the photoelectric conversion element array 4b has an outer diameter of the microlens necessary for explanation). Only included). In the photoelectric conversion element array 4b, minute photoelectric conversion elements are arranged in a square lattice pattern on the light receiving surface at the focal length of the microlens. As is clear from FIG. 3, each microlens of the microlens array 4a covers the plurality of photoelectric conversion elements of the photoelectric conversion element array 4b.

  In FIG. 1, the drive control device 5 includes a microcomputer (not shown) and peripheral components such as a memory, an A / D converter, and a drive circuit, and executes various arithmetic controls and sequence controls of the camera 1. The drive control device 5 includes a pupil data extraction unit 5a, a data string formation unit 5b, a focus calculation unit 5c, an image composition unit 5d, and the like according to the software form of the microcomputer. Details of these parts 5a to 5d will be described later.

  The memory 6 stores various data and images. The image memory 6 a stores an image captured by the image sensor 4. The optical axis data memory 6b is a memory for storing position data on the photoelectric conversion element array 4b of the optical axis of each microlens constituting the microlens array 4a. This optical lens position data of the microlens is obtained by developing the coordinates of the photoelectric conversion element position of the photoelectric conversion element array 4b to a decimal level. That is, the optical axis position is accurately expressed as a value smaller than a unit in which one photoelectric conversion element is 1, for example, 0.5 photoelectric conversion element is 0.5.

  The pupil data extraction unit 5a of the drive control device 5 outputs each photoelectric conversion data stored in the optical axis data memory 6b from the photoelectric conversion data for each photoelectric conversion element output from the photoelectric conversion element 4b and stored in the image memory 6a. Photoelectric conversion data of photoelectric conversion elements (usually a plurality) at the left and right target positions with the optical axis position of the microlens as the center is extracted. In this embodiment, as shown in FIG. 3, the photoelectric conversion data of each of the three photoelectric conversion elements at the left and right target positions centered on the microlens optical axis position are respectively integrated, and the right pupil data and the left pupil Data. The pupil data extraction unit 5a extracts a pair of left and right pupil data for each microlens and temporarily stores it in the image data memory 6a.

  In FIG. 3, each of the three photoelectric conversion elements at the left and right target positions centered on the optical axis position of the microlens is projected onto the exit pupil of the photographing lens 3 by the corresponding microlens, and a pair of projections on the exit pupil. Form a region. A pair of light fluxes passing through the pair of projection areas on the exit pupil of the photographing lens 3 is converted into three photoelectric conversions at right and left target positions centered on the optical axis position of the microlens via the photographing lens 3 and the microlens. A pair of images is formed on each of the three photoelectric conversion elements. Since the phase difference of the pair of images, that is, the amount of image shift, changes in accordance with the focus adjustment state of the photographing lens 3, each of the pair of images is based on a pair of image data corresponding to the pair of images output from three photoelectric conversion elements. If the image shift amount is calculated, the focus adjustment state of the photographing lens 3, that is, the defocus amount from the top surface of the microlens can be detected by a so-called pupil division phase difference detection method. The defocus amount of the right pupil and the left pupil due to the photoelectric conversion element behind each microlens indicates the defocus amount of the subject image corresponding to the apex position of each microlens (Japanese Patent Laid-Open No. 2007-11314). Etc.)

  Note that the position and number of photoelectric conversion elements that extract the pair of left and right pupil data are not limited to the position and number of the embodiment shown in FIG.

  The data string forming unit 5b of the drive control device 5 divides a pair of left and right pupil data corresponding to a predetermined number of microlenses arranged in the left-right direction into right pupil data and left pupil data and arranges them in the order of microlenses. A right pupil data string and a left pupil data string are formed.

  FIG. 4 shows an example in which eleven microlenses are selected from the microlens array 4a in order to extract left and right pupil data strings. Since the microlenses are arranged in a staggered pattern in the microlens array 4a, as shown by connecting the black circles in the figure, the even-numbered microlenses and the odd-numbered microlenses are alternately selected to be 13 pieces. A microlens array for detecting the focus of the lens. In this example, the seventh microlens from the left is the center of the focus detection microlens array, and this position is determined as the focus detection position.

  The focus calculation unit 5c of the drive control device 5 reads a pair of left and right pupil data strings from the image memory 6a, performs focus detection calculation (correlation calculation) by the above-described pupil division phase difference detection method, and adjusts the focus adjustment state of the photographing lens 3 Is detected.

The focus detection calculation based on a pair of data strings is well known, but the outline thereof will be described here. The above-described pair of left and right pupil data strings is A {ai} and B {bi}, and the number of data is N (i = 1, 2,..., N), respectively. When the shift amount between the pair of data strings A and B is k, the difference D between the data strings A and B is expressed by the following equation.
Dk = Σ | ai + k−bi | (1)
The shift amount k that minimizes the difference D is obtained from the equation (1). Since the pair of data strings A and B are discrete numbers, it is not possible to obtain a true shift amount k at which the difference D becomes the minimum value with a resolution equal to or higher than the microlens interval.

Here, since the pair of data strings A and B can be regarded as a combination of sine waves, when attention is paid to a sine wave signal having a certain spatial frequency ω, the expression (1) is regarded as the following expression (2). be able to.
D (θ) = K∫ | sin (ωx + θ) −sinωx | dx (2)
Furthermore, the following equation is obtained by modifying equation (2).
D (θ) = K ′ | sin (θ / 2) · sin (ωx + φ) | (3)
Equation (3) can also be applied to other spatial frequencies, all of which are independently multiplied by a term relating to θ, and it can be seen that the difference equation changes with the absolute value of the sine wave. Further, since the sine wave is linear when θ is near 0, it can be considered that the bottom of the valley between the straight lines having the same slope as shown in FIG. 5 is the minimum value.

  A true shift amount k that minimizes the difference D is obtained by the following three-point interpolation calculation. First, the minimum value Di of the difference D and the shift amount k at that time are obtained by the equation (1). Next, of the shift amounts (k-1) and (k + 1) adjacent to the shift amount k that gives the minimum value Di, the one Di-1 having the larger difference D is selected, and two points (k, Di) and (k- Draw a straight line connecting 1, Di-1). If the slope of this straight line is a, a straight line with a slope (-a) passing through the point (k + 1, Di + 1) of the shift amount k + 1 that gives the second smallest difference Di + 1 is drawn. Let the intersection be the true minimum value Dmin of the difference D, and the shift amount k that gives the true minimum difference Dmin be the true shift amount kmin.

The true shift amount kmin is multiplied by a predetermined conversion coefficient m to determine the defocus amount DEF from the microlens apex surface of the subject image plane by the photographing lens 3, that is, the defocus amount DEF.
DEF = m · kmin (4)

  With the above method, the focus adjustment state of the photographing lens 3 can be detected at the center microlens position of the focus detection microlens array shown in FIG.

  The pupil data extraction unit 5a, the data string formation unit 5b, and the focus calculation unit 5c of the drive control device 5 perform the above-described focus detection processing on all the microlenses in the imaging screen of the image sensor 4. Note that the pupil data extraction unit 5a, the data string formation unit 5b, and the focus calculation unit 5c of the drive control device 5 surrounded by a broken line in FIG. 1 constitute the subject specifying means described above. If all the microlenses in the shooting screen are represented with a subscript p indicating the horizontal position and a subscript q indicating the vertical position, the defocus amount at each microlens position in the shooting screen is represented by DEFpq. Is done.

  However, the focus detection range within the shooting screen is narrower by the length of the focus detection microlens array than the vertical and horizontal lengths of the shooting screen, but the areas that cannot be detected can be detected. The focus detection is performed in the entire range of the photographing screen or a range close to it by shortening the length of the focus detection microlens array.

  It should be noted that focus detection is impossible or the reliability of the detection result is low at a portion where the contrast change in the shooting screen is small. In such a case, it is obtained by interpolation using the detection values in the range where the focus detection result with high reliability in the vicinity is obtained. The interpolation method may be simple linear interpolation or spline curve interpolation. Thereby, it is possible to obtain the focus information or the focal curved surface in the entire range of the photographing screen.

  In this embodiment, the focus detection in the left-right direction of the shooting screen will be described as an example, but focus detection is performed in the same way as the focus detection method in the left-right direction in the vertical direction and the diagonal direction of the shooting screen. It can be performed.

  The image composition unit 5d of the drive control device 5 uses the focus information of all the microlens positions in the shooting screen detected by the focus calculation unit 5c, from the photoelectric conversion data output from the image sensor 4 or the arbitrary position or Combine images that are in focus at all positions. An image composition method by the image composition unit 5d will be described with reference to FIGS.

  FIG. 6 illustrates a method for synthesizing an image at the position of the micro lens m0 when the focus detection result of the micro lens (focus detection position) m0 of interest in the shooting screen is focused on the micro lens apex surface S0. It is a figure to do. The luminous flux passing through the focal plane S0 corresponds to the microlens m0 because the aperture diameter of the photographic lens 3 and the aperture diameter of the microlens are set to be the same or the aperture diameter of the photographic lens 3 is smaller. To the photoelectric conversion element. Therefore, if the outputs of the photoelectric conversion elements (photoelectric conversion elements blacked out in the figure) covered by the microlens m0 are integrated, the portion corresponding to the microlens m0 having a focus on the microlens apex surface (planned focal plane) Pixel output (image) is obtained.

FIG. 7 shows a case where the focus detection result of the target microlens (focus detection position) m1 in the shooting screen is focused on the surface S1 at a distance y from the microlens apex surface. It is a figure explaining the image composition method of the micro lens m1 position. 6 and 7, the distance between the microlenses of the microlens array 4a is d, and the distance from the apex surface of the microlens to the light receiving surface of the photoelectric conversion element array 4b, that is, the focal length of the microlens is f. Further, a position where the light beam reaches the light receiving surface of the photoelectric conversion element array 4b through the micro lens m2 adjacent to the micro lens m1 through the intersection P1 between the optical axis of the micro lens m1 of interest and the focal plane S1, and the micro Assuming that the distance from the optical axis of the lens m2 is x, the following relationship is established as is apparent from FIG.
y / d = f / x (5)

  The light beam passing through the point P1 on the focal plane S1 reaches the photoelectric conversion element e1 on the optical axis of the target microlens m1, and is away from the optical axis of the microlens m2 adjacent to the target microlens m1 by a distance x. To the photoelectric conversion element e2. Similarly, the microlens m6 adjacent to the opposite side of the target microlens m1 reaches the photoelectric conversion element e6 at a position away from the optical axis of the microlens m6 by the distance x.

Furthermore, the relationship of the formula (5) is also established for peripheral microlenses m3, m4,... Located at a distance n · d (n = 2, 3,...) From the microlens m1 of interest. Subscript “ij” representing the two-dimensional position of the microlens is attached to n and x, and the expression (5) is expanded to the microlenses m3, m4,.
y / (nij · d) = f / xij (6)
When the distance y to the focal plane S1 with respect to the microlens m1 of interest is fixed and the distance x is moved within the range of the photoelectric conversion elements covered by the surrounding microlenses m3, m4,. It is possible to know which photoelectric conversion element of the microlenses m3, m4,... Around the microlens m1 to which the focused light beam passes the point P1.

  In the example shown in FIG. 7, the light beam passing through the point P1 on the focal plane S1 reaches the photoelectric conversion elements e3 and e7 of the microlenses m3 and m7 located at the distance 2d from the target microlens m1. If the photoelectric conversion elements are specified by the equation (6) for all the microlenses m2, m3,... Around the target microlens m1, and the outputs of these photoelectric conversion elements are integrated, the distance from the apex surface of the microlens A pixel output (image) corresponding to the target microlens m1 having the focal plane S1 at the position y is obtained.

  In the example shown in FIG. 6, the pixel output (image) of the microlens m0 portion having a focal point on the planned focal plane is obtained. In the example shown in FIG. 7, the microlens m1 having a focal point at a distance y from the planned focal plane. A partial pixel output (image) is obtained. These partial pixel outputs, that is, images, are images in focus.

  The pixel output corresponding to each microlens is obtained by integrating the outputs of a plurality of photoelectric conversion elements, and is normalized by the number of photoelectric conversion elements obtained by integrating the integrated values. In the microlens at the end of the shooting screen, the integrated value becomes small because there is no peripheral microlens over the entire 360 ° circumference of the microlens, but the image of the microlens at the end of the screen is obtained by the above normalization process. Can also be as bright as the microlens image in the center of the screen. Further, when the aperture of the photographing lens 3 is reduced, the number of photoelectric conversion elements to be integrated is not uniform within the photographing screen, and a similar problem occurs. However, it can be solved by normalizing the integrated value with the integrated number.

  As described above, since the defocus amount DEFpq is obtained at all the microlens positions in the shooting screen by the focus calculation unit 5c, the above-described method by the image combining unit 5d is used to defocus all the microlenses in the shooting screen. If the image composition processing is performed according to the amount DEFpq, the same number of pixel outputs as the microlenses, that is, images are obtained. If these are developed according to the microlens positions on the two-dimensional plane, they correspond to the respective microlenses. The focused image is developed. In this way, once the signal of the photoelectric conversion element array 4b is captured, an image in focus at any position in the shooting screen can be synthesized.

  Note that it is possible to intentionally defocus some microlens positions, or to synthesize an in-focus image only within a specified range. For example, if a photographed image is processed to detect a person's face from the image, and the above-described image composition processing is performed on the microlens corresponding to the person's face, an image in which all the faces are in focus is obtained. It can also be synthesized.

  According to the above-described embodiment and its modifications, the following operational effects can be achieved. First, a microlens array 4a in which a plurality of microlenses are arranged on a plane is arranged in the vicinity of a planned focal plane of the photographing lens 3, and a photoelectric conversion element array 4 in which a plurality of photoelectric conversion elements are arranged on a plane is arranged as a microlens array. An image sensor 4 that is arranged at the focal position of the microlens behind 4a and images the subject image formed on the planned focal plane by the photographing lens 3, and a pair of photoelectric conversion elements covered by the microlens. Based on the output of the photoelectric conversion element, the amount of deviation of the pair of images due to the pair of light beams that have passed through the exit pupil of the photographing lens 3 is detected, and the photographing lens 3 at a position corresponding to the microlens in the subject image is defocused. The pupil data extraction unit 5a, the data string formation unit 5b, and the focus calculation unit 5c for detecting the amount, and the defocus amount at the position corresponding to the microlens in the subject image Accordingly, the image synthesizing unit that integrates the outputs of the photoelectric conversion elements respectively coated on the microlens and the surrounding microlenses and synthesizes an image focused on a position corresponding to the microlens in the subject image. 5d. As a result, it is possible to synthesize a focused image at a position where the focus is desired from one captured image.

  Next, the subject image is processed to detect a person's face, and the person's face is set to a position to be focused, and the focus is set to the set position in the subject image according to the defocus amount of the set position. Since the existing images are synthesized, it is possible to synthesize an image focused on a person's face from one captured image. In this case, an appropriately blurred image can be used as the background.

  In addition, by specifying all positions of the subject image by the subject specifying means described above, it is possible to synthesize an image that is in focus at all positions of the subject image. Further, according to the focus detection method described above, the focus adjustment state at any position in the screen can be detected, so that an image focused on the entire screen or an arbitrary position can be synthesized.

The figure which shows the structure of the digital camera of one embodiment View of the microlens array from the photographic lens side Figure of the outside diameter of one microlens projected onto the photoelectric conversion element array The figure which shows an example of the micro lens arrangement | sequence for focus detection Diagram for explaining the three-point interpolation method An image at the position of the micro lens m0 when the focus detection result of the micro lens (focus detection position) m0 of interest in the photographing screen is focused on the micro lens apex surface (scheduled focal plane of the photographing lens) S0. Diagram explaining the synthesis method When the focus detection result of the micro lens (focus detection position) m1 of interest in the shooting screen is determined to be focused on the surface S1 at a distance y from the micro lens apex surface (scheduled focal plane of the shooting lens), The figure explaining the image composition method of the micro lens m1 position in an imaging | photography screen

Explanation of symbols

DESCRIPTION OF SYMBOLS 1; Camera, 2; Camera body, 3; Shooting lens, 4; Image pick-up element, 4a; Micro lens array, 4b; Photoelectric conversion element array, 5: Drive control apparatus, 5a; Pupil data extraction part, 5b; Part, 5c; focus calculation part, 5d; image composition part

Claims (6)

Photographic optics,
A plurality of microlenses arranged two-dimensionally, and a microlens array disposed behind the imaging optical system;
A photoelectric conversion device disposed behind the microlens array and assigned a plurality of photoelectric conversion regions for each of the plurality of microlenses;
Subject identification means for identifying the subject to be focused;
An image synthesizing apparatus comprising: an image creating unit configured to synthesize an image focused on the subject specified by the subject specifying unit based on an output of the photoelectric conversion device. The image composition apparatus according to claim 1,
The subject specifying means has focus detection means,
The image creating means creates an image focused on the subject focus-detected by the focus detecting means based on the output of the photoelectric conversion device acquired in advance prior to focus detection by the focus detecting means. Synthesizer. The image composition device according to claim 2,
The photoelectric conversion region of the photoelectric conversion device is conjugate with an exit pupil of the photographing optical system, and the focus detection unit outputs outputs of a pair of photoelectric conversion regions among a plurality of photoelectric conversion regions assigned to the microlenses. An image synthesizing apparatus for detecting a defocus amount to a subject based on the above. The image composition apparatus according to claim 1,
The subject specifying means identifies a pre-designated feature of the subject, performs focus detection on the feature portion by the focus detection means, and synthesizes an image focused on the subject that is a focus detection target. An image synthesizing apparatus comprising instruction means for instructing a creation means. The image composition apparatus according to claim 1,
The image synthesizing means has display means for creating and displaying an image having a deep focal depth based on the output of the image sensor,
The subject specifying means has setting means for setting a subject to be focused based on the display of the display means,
An image synthesizing apparatus for obtaining an image focused on a subject set by the setting means. In the image composition device according to any one of claims 1 to 5,
The subject specifying means specifies subjects at different positions in the optical axis direction of the photographing optical system, and the image creating means creates one image focused on the plurality of identified subjects. Image composition device.

Images