JP2007235862A - Image-taking device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image-taking device by which a clear image can be taken in an entire image region irrespective of a distance from a photographic subject. <P>SOLUTION: The image-taking device is equipped with a plurality of lenses 113, a plurality of image taking regions 123 each equipped with an acceptance surface nearly perpendicular to an optical-axis direction of the lenses 113, an image signal input section 133 to which a plurality of image signals each input from the image taking regions 123 are input, a block dividing section 143 which divides at least one of the plurality of the image signals into a plurality of blocks, a parallax component calculator 144 which performs, by using finer resolution than that of pixels, calculation operations of parallax components which are each formed by the lenses 113 and appear between images, and a parallax component correcting section 145 which corrects the plurality of the image signals and produces an image by synthesizing the image signals based on the parallax components. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a small and thin imaging apparatus.

  There exists a camera module of patent document 1 as a conventional camera module. FIG. 35 is a cross-sectional view showing the configuration of the camera module of Patent Document 1. As shown in FIG.

  In FIG. 35, an imaging system 9010 is an optical processing system that forms an image of light from an object on an imaging surface of an imaging element 9120 via a diaphragm 9110 and an imaging lens 9100. The diaphragm 9110 has three circular openings 9110a, 9110b, and 9110c. Object light that has entered the light incident surface 9100e of the imaging lens 9100 from each of the apertures 9110a, 9110b, and 9110c is emitted from the three lens portions 9100a, 9100b, and 9100c of the imaging lens 9100, and 3 on the imaging surface of the imaging element 9120. Two object images are formed. A light-shielding film (not shown) is formed on the flat portion 9100 d of the imaging lens 9100. Three optical filters 9052a, 9052b, and 9052c that transmit light in different wavelength bands are formed on the light incident surface 9100e of the imaging lens 9100. In addition, three optical filters 9053a, 9053b, and 9053c that transmit light in different wavelength bands are also formed on the three imaging regions 9120a, 9120b, and 9120c on the imaging element 9120. The optical filter 9052a and the optical filter 9053a have spectral transmittance characteristics that mainly transmit green (indicated by G), and the optical filter 9052b and the optical filter 9053b mainly transmit spectral light that transmits through red (indicated by R). Further, the optical filter 9052c and the optical filter 9053c have spectral transmittance characteristics that mainly transmit blue (indicated by B). Therefore, the imaging region 9120a has sensitivity to green light (G), the imaging region 9120b has sensitivity to red light (R), and the imaging region 9120c has sensitivity to blue light (B).

  In such a camera module having a plurality of imaging lenses, when the distance from the camera module to the object changes, the mutual interval between the plurality of object images formed by the plurality of imaging lenses on the imaging surface of the imaging element 9120 changes. To do.

  In the camera module of Patent Document 1, the virtual subject distance D [m] is defined as D = 1.4 / (tan (θ / 2)) as a function of the shooting angle of view θ [°] of a plurality of imaging systems. When the object is at the virtual subject distance D [m], the change between the mutual interval of the plurality of object images and the mutual interval of the plurality of object images when the object is at infinity is twice the pixel pitch of the reference image signal. The optical axis intervals of the plurality of imaging systems are set so as to be smaller than the above. That is, even if the same image processing as the image processing optimized for photographing the object at the virtual subject distance D [m] is performed on the photographing of the object at infinity, both object images on the imaging surface are obtained. Since the optical axis interval is set so that the difference between the two is smaller than twice the pixel pitch of the reference signal, the color shift of the object image at infinity can be suppressed to an acceptable level.

  In recent years, mobile devices such as mobile phones equipped with cameras have become widespread. As these portable devices become smaller, thinner, and higher in performance, camera modules that are smaller, thinner, and higher in performance are required. As a high performance, equipped with auto focus control function, not only landscape shooting (shooting objects at almost infinity) and portrait shooting (usually shooting objects at a distance of several meters), but also macro shooting (several centimeters to several tens of centimeters) (Photographing an object at a distance of 2) is required.

  However, the camera module disclosed in Patent Document 1 has a plurality of lens units 9100a, 9100b, and 9100c, thereby realizing a reduction in thickness. However, the conventional camera module does not have an automatic focus control function. Further, the virtual subject distance D [m] is set in consideration of person photographing. Therefore, the conventional camera module can cope with landscape photography and portrait photography, but cannot cope with macro photography.

As a conventional camera module that solves the problem of Patent Document 1, there is a camera module described in Patent Document 2. FIG. 36 is a diagram illustrating an example of a camera module disclosed in Patent Document 2, and illustrates a captured image and divided small areas. Detection areas k1, k2, k3, and k4 are provided in the center k0 and the periphery thereof on the photographing screen. Then, the parallaxes p0, p1, p2, p3, and p4 of the respective detection areas are obtained. Then, the parallax in a predetermined range is extracted from these parallaxes, and when there are a plurality of extracted parallaxes, the parallax in the region near the center is selected. Further, the entire captured image is corrected using the selected parallax.
JP 2001-78213 A JP 2002-330332 A

  As described above, the conventional camera module described in Patent Document 1 has a plurality of lens portions 9100a, 9100b, and 9100c, thereby realizing a reduction in thickness. However, the conventional camera module does not have an automatic focus control function. Further, the virtual subject distance D [m] is set with the person shooting in mind. Therefore, the conventional camera module can cope with landscape photography and portrait photography, but cannot cope with macro photography.

  Further, in the conventional camera module described in Patent Document 2, a part of an image to be taken is set as a plurality of small areas, parallax of these areas is obtained, one of these parallaxes is selected, and the selected one is selected. The whole captured image is corrected based on the parallax. For example, when a person M that is about 2 m away from the camera and a person N that is about 10 m away from the camera coexist in the center of the shooting screen, the person M located in the center of the shooting screen is considered to be the subject, and this area The entire captured image is corrected based on the parallax p0 at k0. At this time, the parallax of the person M is the same as the selected parallax, the influence of the parallax can be corrected, and a clear image is obtained in the area of the person M. However, since the parallax of the person N is different from the selected parallax, the influence of the parallax cannot be corrected. Therefore, color unevenness occurs in the area of the person N, and a beautiful image is not obtained.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a camera module that can be reduced in size and thickness and can obtain a clear image in the entire image region regardless of the subject distance.

  The imaging apparatus according to the present invention includes a plurality of lens units each including at least one lens, and a light reception that corresponds to the plurality of lens units on a one-to-one basis and is substantially perpendicular to the optical axis direction of the corresponding lens unit. A plurality of imaging regions each having a plane; an imaging signal input unit to which a plurality of imaging signals output from the plurality of imaging regions are input; and a plurality of imaging signals at least one of the plurality of imaging signals. A block division unit that divides the image into blocks, and a parallax calculation unit that calculates the parallax between the images formed by the plurality of lens units for each of the blocks at a resolution finer than the resolution of the pixels, using the imaging signal. A parallax correction unit that corrects the plurality of imaging signals based on the parallax and combines the images.

  The present invention has been made in view of the above problems, and can provide an imaging apparatus that can be reduced in size and thickness and that can obtain a clear image in the entire image region regardless of the subject distance.

  The imaging apparatus according to the present invention includes a plurality of lens units each including at least one lens, and a light reception that corresponds to the plurality of lens units on a one-to-one basis and is substantially perpendicular to the optical axis direction of the corresponding lens unit. A plurality of imaging regions each having a plane; an imaging signal input unit to which a plurality of imaging signals output from the plurality of imaging regions are input; and a plurality of imaging signals at least one of the plurality of imaging signals. A block division unit that divides the image into blocks, and a parallax calculation unit that calculates the parallax between the images formed by the plurality of lens units for each of the blocks with a resolution finer than the resolution of the pixels. And a parallax correction unit that corrects the plurality of imaging signals based on the parallax and combines the images.

  The position of the object image changes relatively according to the subject distance. That is, as the subject distance decreases, the parallax increases. Therefore, when a plurality of subjects having different distances are simultaneously imaged, the parallax differs for each subject. According to the imaging apparatus of the present invention, the parallax for each block is calculated, and after the imaging signal is corrected so as to reduce the influence of the parallax based on the parallax for each block, the image is synthesized. Accordingly, even when a plurality of subjects having different distances are simultaneously imaged, the parallax of each subject can be corrected as appropriate, and a clear image in which the influence of the parallax is reduced in the entire image region can be obtained.

  Furthermore, according to the imaging apparatus of the present invention, for each block, the parallax is calculated with a resolution finer than the pixel resolution, and the imaging signal is reduced so that the influence of the parallax is reduced based on the fine resolution parallax for each block. After correcting the image, the image is synthesized. Accordingly, even when a plurality of subjects having different distances are simultaneously imaged, it is possible to appropriately correct the parallax of each subject and obtain a clear image in which the influence of the parallax is further reduced in the entire image region.

  In the imaging apparatus of the present invention, the parallax calculation unit includes an imaging signal indicating an image formed by one lens unit among the plurality of lens units, and an image formed by a lens unit different from the one lens unit. A plurality of parallax evaluation values are created by comparing the imaging signal indicating a plurality of times while changing the shift amount of the block, and an extreme value shift amount that is the shift amount that gives the extreme value of the parallax evaluation value is set. It is preferable to calculate the parallax with a resolution finer than the resolution of the pixel based on the parallax evaluation value of the extreme value shift amount and the parallax evaluation value of the shift amount before and after the extreme value shift amount. .

  According to this preferred configuration, for each block, the parallax is calculated with a resolution finer than the pixel resolution based on the parallax evaluation value that gives an extreme value with respect to the shift amount and the parallax evaluation value before and after the shift amount. After correcting the imaging signal so as to reduce the influence of the parallax based on the fine resolution parallax for each block, the image is synthesized. Accordingly, even when a plurality of subjects having different distances are simultaneously imaged, the parallax of each subject can be corrected as appropriate, and a clear image in which the influence of the parallax is reduced in the entire image region can be obtained.

  In the imaging apparatus of the present invention, the parallax calculation unit includes an imaging signal indicating an image formed by one lens unit among the plurality of lens units, and an image formed by a lens unit different from the one lens unit. A plurality of parallax evaluation values by changing the block shift amount at a resolution finer than the resolution of a pixel, and a plurality of parallax evaluation values, and giving an extreme value of the parallax evaluation value It is preferable to obtain an extreme value shift amount that is a shift amount, and to calculate the parallax with a resolution finer than the resolution of a pixel based on the parallax evaluation value and the extreme value shift amount of the extreme value shift amount.

  According to this preferable configuration, an imaging signal indicating an image formed by one lens unit for each block and an imaging signal indicating an image formed by a different lens unit only a plurality of times with a resolution finer than the pixel resolution. By changing and comparing the shift amount, the parallax is calculated with a resolution finer than the resolution of the pixel, and after correcting the imaging signal so that the influence of the parallax is reduced based on the fine resolution parallax for each block, Composite images. Accordingly, even when a plurality of subjects having different distances are simultaneously imaged, the parallax of each subject can be corrected as appropriate, and a clear image in which the influence of the parallax is reduced in the entire image region can be obtained.

  In the imaging apparatus of the present invention, the parallax calculation unit includes an imaging signal indicating an image formed by one lens unit among the plurality of lens units, and an image formed by a lens unit different from the one lens unit. A plurality of parallax evaluation values are created by comparing the imaging signal indicating a plurality of times while changing the shift amount of the block, and an extreme value shift amount that is the shift amount that gives the extreme value of the parallax evaluation value is set. To obtain a slightly shifted parallax evaluation value by comparing the image pickup signal by shifting the block smaller than the shifting amount before and after the extreme value shifting amount, and the parallax evaluation value and the slightly shifting time evaluation value. It is preferable to calculate the parallax based on the above.

  According to this preferred configuration, the approximate amount of parallax is obtained by changing the shift amount by a specific width, and then the detailed amount of parallax is obtained by changing the amount of shift by a finer width before and after the approximate parallax. As a result, the amount of calculation can be reduced as compared with the case where the shift amount is changed with a constant width, so that even when a plurality of subjects with different distances are simultaneously imaged with a lower cost arithmetic circuit, each subject is appropriately selected. By correcting the parallax, it is possible to obtain a clear image in which the influence of the parallax is reduced in the entire image region.

  In the imaging apparatus of the present invention, the parallax calculation unit includes a signal obtained by thinning out an imaging signal indicating an image formed by one lens unit among the plurality of lens units, and a lens unit different from the one lens unit. A plurality of parallax evaluation values are created by comparing a signal obtained by thinning an imaging signal indicating an image formed by changing the shift amount of the block a plurality of times, and the parallax is calculated based on the parallax evaluation values It is preferable.

  According to this preferable configuration, the parallax is obtained using a signal obtained by thinning the imaging signal. As a result, the amount of calculation can be reduced compared with the case of obtaining parallax using all imaging signals, so that even when a plurality of subjects with different distances are simultaneously imaged with a lower cost arithmetic circuit, By correcting the parallax of the subject, it is possible to obtain a beautiful image in which the influence of the parallax is reduced in the entire image region.

  In the imaging apparatus according to the present invention, the parallax calculation unit includes a signal obtained by thinning out an imaging signal indicating an image formed by one lens unit among the plurality of lens units, and the one lens unit. A plurality of parallax evaluation values are created by comparing a signal obtained by thinning out imaging signals indicating images formed by different lens portions in a zigzag manner by changing the shift amount of the block, and the parallax evaluation values are generated. It is preferable to calculate the parallax based on this.

  According to this preferable configuration, the parallax is obtained using a signal obtained by thinning the imaging signals in a zigzag pattern. As a result, the amount of calculation can be reduced compared with the case of obtaining parallax using all imaging signals, so that even when a plurality of subjects with different distances are simultaneously imaged with a lower cost arithmetic circuit, By correcting the parallax of the subject, it is possible to obtain a beautiful image in which the influence of the parallax is reduced in the entire image region.

  In the imaging apparatus of the present invention described above, the parallax calculation unit includes a signal obtained by thinning out an imaging signal indicating an image formed by one lens unit among the plurality of lens units, and the one lens unit. A plurality of parallax evaluation values are created by comparing a signal obtained by thinning out every other row of imaging signals indicating images formed by different lens portions by changing the shift amount of the block, and a plurality of parallax evaluation values are created. It is preferable to calculate the parallax based on this.

  According to this preferable configuration, the parallax is obtained using a signal obtained by thinning the imaging signal every other line. As a result, the amount of calculation can be reduced compared with the case of obtaining parallax using all imaging signals, so that even when a plurality of subjects with different distances are simultaneously imaged with a lower cost arithmetic circuit, By correcting the parallax of the subject, it is possible to obtain a beautiful image in which the influence of the parallax is reduced in the entire image region.

  In the imaging apparatus of the present invention, the parallax calculation unit compares the imaging signals of two lens units arranged in a diagonal direction of the imaging element among the plurality of lens units, calculates the parallax, The two lens units have different horizontal distances and vertical distances between optical axes, and the parallax calculation unit responds to a ratio of the horizontal distance to the vertical distance. Thus, the imaging signal is expanded in the vertical direction or the horizontal direction so that the number of pixels in the vertical direction and the number of pixels in the horizontal direction of the imaging signal are substantially equal, and one of the two lens units is The shift amount of the block is changed in a 45 ° direction between a signal obtained by expanding an image pickup signal indicating an image to be formed and a signal obtained by expanding an image pickup signal indicating an image formed by the other lens portion of the two lens portions. Let me compare multiple times Rukoto by creating a plurality of parallax evaluation value, it is preferable to calculate the parallax based on the parallax evaluation value.

  According to this preferable configuration, the imaging signal expanded in the vertical direction by the ratio of the distance in the horizontal direction to the distance in the vertical direction is initially generated, so that no interpolation processing is required at the time of shifting. As a result, the amount of calculation can be reduced as compared with the case of using a non-expanded imaging signal, so that when a plurality of subjects with different distances are simultaneously imaged with a lower cost arithmetic circuit, the parallax of each subject is appropriately selected. Is corrected, and a clear image in which the influence of parallax is reduced in the entire image region can be obtained.

  In the above-described imaging device of the present invention, the parallax calculation unit may calculate the parallax by comparing imaging signals of two lens units arranged in a diagonal direction of the imaging element among the plurality of lens units. good. Alternatively, in the above-described imaging device of the present invention, the parallax calculation unit compares the imaging signals of two lens units arranged in the horizontal direction of the imaging element among the plurality of lens units and calculates the parallax. Also good.

  Hereinafter, specific embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
The imaging apparatus according to Embodiment 1 of the present invention divides the image into blocks, calculates the parallax with a resolution finer than the resolution of the pixels for each block, and corrects the parallax based on the parallax, so that the entire image area is clean. Get an image. Also, after obtaining the approximate parallax, obtaining the detailed parallax, obtaining the parallax using the thinned imaging signal, changing the ratio between the horizontal direction and the vertical direction, and the parallax evaluation value of the block before combining By calculating the combined parallax evaluation value using, the calculation time can be reduced.

  An imaging apparatus according to Embodiment 1 of the present invention will be described with reference to the drawings.

  FIG. 1 is a cross-sectional view showing a configuration of an imaging apparatus according to Embodiment 1 of the present invention. In FIG. 1, the imaging device imaging apparatus includes a lens module unit 110 and a circuit unit 120.

  The lens module unit 110 includes a lens barrel 111, an upper cover glass 112, a lens 113, an actuator fixing unit 114, and an actuator movable unit 115. The circuit unit 120 includes a substrate 121, a package 122, an image sensor 123, a package cover glass 124, and a system LSI (hereinafter referred to as SLSI) 125.

  The lens barrel 111 has a cylindrical shape, and an inner wall surface thereof is black which is matted to prevent irregular reflection of light, and is formed by injection molding a resin. The upper cover glass 112 has a disc shape, is formed of a transparent resin, and is fixed to the upper surface of the lens barrel 111 with an adhesive or the like. The surface of the upper cover glass 112 prevents damage due to friction and the like, and prevents reflection of incident light. An antireflection film is provided.

  FIG. 2 is a top view of the lens 113 of the imaging apparatus according to Embodiment 1 of the present invention. The lens 113 has a substantially disc shape and is formed of glass or transparent resin, and the first lens portion 113a, the second lens portion 113b, the third lens portion 113c, and the fourth lens portion 113d are in a grid shape. Placed in. As shown in FIG. 2, the X axis and the Y axis are set along the arrangement direction of the first to fourth lens portions 113a to 113d. In the first lens unit 113a, the second lens unit 113b, the third lens unit 113c, and the fourth lens unit 113d, the light incident from the subject side is emitted to the image sensor 123 side, and is on the image sensor 123. Four images are formed on the screen. As shown in FIG. 2, the optical axis of the first lens unit 113a and the optical axis of the second lens unit 113b are separated by Dx in the horizontal direction (x-axis direction), and in the vertical direction (y-axis direction). Match. The optical axis of the second lens portion 113b and the optical axis of the third lens portion 113c are separated by Dx in the horizontal direction (x-axis direction) and separated by Dy in the vertical direction (y-axis direction). The optical axis of the third lens portion 113c and the optical axis of the fourth lens portion 113d are separated from each other by Dx in the horizontal direction (x-axis direction) and coincide with each other in the vertical direction (y-axis direction). The optical axis of the fourth lens portion 113d and the optical axis of the first lens portion 113a are separated by Dx in the horizontal direction (x-axis direction) and separated by Dy in the vertical direction (y-axis direction). The first lens unit 113a is designed to transmit green, the second lens unit 113b transmits blue, the third lens unit 113c transmits red, and the fourth lens unit 113d transmits green. Is.

  The actuator fixing portion 114 is fixed to the inner wall surface of the lens barrel 111 with an adhesive or the like. The actuator movable portion 115 is fixed to the outer peripheral edge of the lens 113 with an adhesive or the like. Actuator fixing part 114 and actuator movable part 115 constitute a voice coil motor. The actuator fixing portion 114 has a permanent magnet (not shown) and a ferromagnetic yoke (not shown), and the actuator movable portion 115 has a coil (not shown). The actuator movable portion 115 is elastically supported by the elastic body (not shown) with respect to the actuator fixing portion 114. By energizing the coil of the actuator movable portion 115, the actuator movable portion 115 moves relative to the actuator fixing portion 114, and the relative distance along the optical axis between the lens 113 and the image sensor 123 changes.

  The substrate 121 is made of a resin substrate, and the lens barrel 111 is fixed to the upper surface with an adhesive or the like with the bottom surface in contact therewith. In this way, the lens module unit 110 and the circuit unit 120 are fixed, and the imaging apparatus 101 is configured.

  The package 122 is made of a resin having metal terminals, and the metal terminal portion is fixed to the upper surface of the substrate 121 by soldering or the like inside the lens barrel 111. The image sensor 123 includes a first image sensor 123a, a second image sensor 123b, a third image sensor 123c, and a fourth image sensor 123d. The first imaging element 123a, the second imaging element 123b, the third imaging element 123c, and the fourth imaging element 123d are solid-state imaging elements such as a CCD sensor and a CMOS sensor, respectively. The light receiving surface of each image sensor is such that the center is substantially coincident with the optical axis center of each of the first lens portion 113a, the second lens portion 113b, the third lens portion 113c, and the fourth lens portion 113d. Are arranged so as to be substantially perpendicular to the optical axis of the corresponding lens portion.

  Each terminal of the first image sensor 123a, the second image sensor 123b, the third image sensor 123c, and the fourth image sensor 123d is connected to a metal terminal at the bottom inside the package 122 with a gold wire 127 by wire bonding. Connected and electrically connected to the SLSI 125 through the substrate 121. The first lens unit 113a, the second lens unit 113b, and the third lens unit are provided on the light receiving surfaces of the first image sensor 123a, the second image sensor 123b, the third image sensor 123c, and the fourth image sensor 123d. The light emitted from the lens portion 113c and the fourth lens portion 113d forms an image, and the electrical information converted from the light information by the photodiode is output to the SLSI 125.

  FIG. 3 is a top view of the circuit unit 120 of the imaging apparatus according to Embodiment 1 of the present invention. The package cover glass 124 has a flat plate shape, is formed of a transparent resin, and is fixed to the upper surface of the package 122 by adhesion or the like. On the upper surface of the package cover glass 124, a first color filter 124a, a second color filter 124b, a third color filter 124c, a fourth color filter 124d, and a light shielding portion 124e are disposed by vapor deposition or the like. In addition, an infrared cut filter (not shown; hereinafter referred to as an IR filter) is disposed on the lower surface of the package cover glass 124 by vapor deposition or the like.

  4 is a characteristic diagram of the color filter of the imaging device according to Embodiment 1 of the present invention, and FIG. 5 is a characteristic diagram of the IR filter of the imaging device according to Embodiment 1 of the present invention. The first color filter 124a has a spectral transmission characteristic that mainly transmits green as indicated by G in FIG. 4, and the second color filter 124b has a spectral transmission characteristic that mainly transmits blue as indicated by B in FIG. The third color filter 124c has a spectral transmission characteristic that mainly transmits red as indicated by R in FIG. 4, and the fourth color filter is mainly indicated by G in FIG. It has spectral transmission characteristics that transmit green. The IR filter has a spectral transmission characteristic for blocking infrared light indicated by IR in FIG.

  Accordingly, the object light incident from the upper part of the first lens unit 113a is emitted from the lower part of the first lens unit 113a, and mainly green is transmitted through the first color filter 124a and the IR filter, so that the first imaging is performed. In order to form an image on the light receiving portion of the element 123a, the first imaging element 123a receives the green component of the object light. The object light incident from the upper part of the second lens unit 113b is emitted from the lower part of the second lens unit 113b, and mainly blue light is transmitted through the second color filter 124b and the IR filter, so that the second imaging is performed. In order to form an image on the light receiving portion of the element 123b, the second imaging element 123b receives the blue component of the object light. Further, the object light incident from the upper part of the third lens unit 113c is emitted from the lower part of the third lens unit 113c, and mainly the red color is transmitted through the third color filter 124c and the IR filter. In order to form an image on the light receiving portion of the element 123c, the third imaging element 123c receives the red component of the object light. Further, the object light incident from the upper part of the fourth lens unit 113d is emitted from the lower part of the fourth lens unit 113d, and mainly green is transmitted through the fourth color filter 124d and the IR filter, so that the fourth imaging is performed. In order to form an image on the light receiving portion of the element 123d, the fourth imaging element 123d receives the green component of the object light.

  The SLSI 125 controls the energization of the coil of the actuator movable unit 115, drives the image sensor 123, inputs electrical information from the image sensor 123, performs various image processing, and communicates with the host CPU by a method described later. Output the image to the outside.

  Next, the relationship between subject distance and parallax will be described. The imaging apparatus according to Embodiment 1 of the present invention has four lens units (first lens unit 113a, second lens unit 113b, third lens unit 113c, and fourth lens unit 113d). The relative positions of the four object images formed by the four lens units change according to the subject distance.

  FIG. 6 is a diagram for explaining the position of an object image at infinity in the imaging apparatus according to Embodiment 1 of the present invention. In FIG. 6, only the first lens unit 113a, the first image sensor 123a, the second lens unit 113b, and the second image sensor 123b are shown for simplicity. The incident light L1 to the first lens unit 113a of the light from the object 10 at infinity is parallel to the incident light L2 to the second lens unit 113b. For this reason, the distance between the first lens unit 113a and the second lens unit 113b is equal to the distance between the object image 11a on the first image sensor 123a and the object image 11b on the second image sensor 123b.

  Here, the optical axis of the first lens unit 113a, the optical axis of the second lens unit 113b, the optical axis of the third lens unit 113c, and the optical axis of the fourth lens unit 113d are respectively the first imaging. The center of the light receiving surface of the element 123a, the center of the light receiving surface of the second image sensor 123b, the center of the light receiving surface of the third image sensor 123c, and the center of the light receiving surface of the fourth image sensor 123d are substantially matched. Has been placed. Accordingly, the center of each light receiving surface of the first image sensor 123a, the second image sensor 123b, the third image sensor 123c, and the fourth image sensor 123d and the infinite image formed on each light receiving surface, respectively. The relative positional relationship with the object image is the same for all image sensors. That is, there is no parallax.

  FIG. 7 is a diagram for explaining the position of an object image at a finite distance in the imaging apparatus according to Embodiment 1 of the present invention. In FIG. 7, only the first lens unit 113a, the first image sensor 123a, the second lens unit 113b, and the second image sensor 123b are shown for simplicity. The light L1 incident on the first lens portion 113a and the light L2 incident on the second lens portion 113b of light from the object 12 having a finite distance are not parallel. Therefore, the distance between the object image 13a on the first image sensor 123a and the object image 13b on the second image sensor 123b is longer than the distance between the first lens unit 113a and the second lens unit 113b. . Therefore, the center of each light-receiving surface of the first image sensor 123a, the second image sensor 123b, the third image sensor 123c, and the fourth image sensor 123d and a finite distance imaged on each light-receiving surface. The relative positional relationship with the object image differs for each image sensor. That is, there is parallax.

  If the distance to the object image 12 (subject distance) is A, the distance between the first lens portion 113a and the second lens portion 113b is D, and the focal length of the lens portions 113a and 113b is f, the right angle in FIG. Since the right triangle with two sides A and D sandwiching the right triangle and the right triangle with two sides f and Δ sandwiching the right angle are similar, the parallax Δ is expressed by the following equation (1). expressed. The same relationship is established between the other lens portions. In this way, the relative positions of the four object images formed by the four lens portions 113a, 113b, 113c, and 113d change according to the subject distance. For example, as the subject distance A decreases, the parallax Δ increases.

        Δ = f · D / A (1)

  Next, the relationship between contrast and focal length will be described.

  FIG. 8A is a diagram for explaining the relationship between the image at the time of focusing (when the focus is adjusted) and the contrast evaluation value in the imaging apparatus according to Embodiment 1 of the present invention. (B) is a figure explaining the relationship between the image at the time of out-of-focus (when focus adjustment is not suitable) and a contrast evaluation value in the imaging device which concerns on Embodiment 1 of this invention.

  Each of the left diagrams in FIGS. 8A and 8B is an image obtained by photographing a rectangle in which the left half is white and the right half is black. As shown in the left diagram of FIG. 8A, at the time of focusing, the contour of the captured image is clear and the contrast is large. On the other hand, as shown in the left diagram of FIG. 8B, when out of focus, the contour of the photographed image is blurred and the contrast is small.

  Each of the right diagrams in FIGS. 8A and 8B shows the results when a band-pass filter (BPF) is applied to the information signal in the left diagram. The horizontal axis is the position in the X-axis direction, and the vertical axis is the output value after BPF. 8A, the signal amplitude after BPF is large at the time of focusing, and the signal amplitude after BPF is small at the time of out-of-focusing, as shown in the right diagram of FIG. 8B. Here, the signal amplitude after the BPF is defined as a contrast evaluation value indicating how large the contrast is. Then, as shown in the right diagram of FIG. 8A, the contrast evaluation value is large when focused, and as shown in the right diagram of FIG. 8B, the contrast evaluation value is small when out of focus.

  FIG. 9 is a diagram illustrating the relationship between the lens position and the contrast evaluation value of the imaging apparatus according to Embodiment 1 of the present invention. When photographing a certain object, when the distance between the lens 113 and the image sensor 123 is small (when z1), the contrast evaluation value is small because of out-of-focus. When the distance between the lens 113 and the image sensor 123 is gradually increased, the contrast evaluation value is gradually increased, and the contrast evaluation value is maximized at the time of focusing (when z2). Furthermore, when the distance between the lens 113 and the image sensor 123 is gradually increased (when z3), the lens becomes out of focus and the contrast evaluation value decreases. Thus, at the time of focusing, the contrast evaluation value is maximized.

  Next, the operation of the imaging apparatus according to Embodiment 1 of the present invention will be described. FIG. 10 is a block diagram of the imaging apparatus according to Embodiment 1 of the present invention. The SLSI 125 includes a system control unit 131, an image sensor driving unit 132, an image signal input unit 133, an actuator operation amount output unit 134, an image processing unit 135, and an input / output unit 136. Further, the circuit unit 120 includes an amplifier 126 in addition to the above-described configuration.

  The amplifier 126 applies a voltage corresponding to the output from the actuator operation amount output unit 134 to the coil of the actuator movable unit 115.

  The system control unit 131 includes a CPU (Central Processing Unit), a memory, and the like, and controls the entire SLSI 125. The image sensor driving unit 132 includes a logic circuit and the like, generates a signal for driving the image sensor 123, and applies a voltage corresponding to the signal to the image sensor 123.

  The imaging signal input unit 133 includes a first imaging signal input unit 133a, a second imaging signal input unit 133b, a third imaging signal input unit 133c, and a fourth imaging signal input unit 133d. The first imaging signal input unit 133a, the second imaging signal input unit 133b, the third imaging signal input unit 133c, and the fourth imaging signal input unit 133d are each a CDS circuit (correlated double sampling circuit: Correlated Double sampling circuit). Sampling Circuit), AGC (Automatic Gain Controller), ADC (Analog / Digital Converter) are connected in series. The first image sensor 123a and the second image sensor, respectively. 123b, the third image sensor 123c, and the fourth image sensor 123d are connected, and electric signals from these are input, the fixed noise is removed by the CDS circuit, the gain is adjusted by the AGC, and the analog signal is obtained by the ADC. It is converted into a digital value and written in the memory of the system control unit 131.

  The actuator operation amount output unit 134 includes a DAC (Digital Analog Converter) and outputs a voltage signal corresponding to a voltage to be applied to the coil of the actuator movable unit 115.

  The image processing unit 135 includes a logic circuit, a DSP, or both of them, and performs various types of image processing according to predetermined program control using the memory information of the system control unit 131. The image processing unit 135 includes an automatic focus control unit 141, an image expansion unit 142, a block division unit 143, a parallax calculation unit 144, a parallax correction unit 145, and a distance calculation unit 146.

  The input / output unit 136 communicates with an upper CPU (not shown), and outputs an image signal to an upper display, an external memory (not shown), and an external display device (not shown) such as a liquid crystal display.

  FIG. 11 is a flowchart showing the operation of the imaging apparatus according to Embodiment 1 of the present invention. The imaging apparatus 101 is operated according to this flowchart by the system control unit 131 of the SLSI 125.

  In step S1000, the operation is started. For example, the host CPU (not shown) detects that a shutter button or the like has been pressed, and instructs the imaging device to start the operation via the input / output unit 136, whereby the imaging device 101 starts the operation. To do. Next, step S1100 is executed.

  In step S1100, the automatic focus control unit 141 executes automatic focus control. FIG. 12 is a flowchart showing the operation of automatic focus control according to Embodiment 1 of the present invention. The flowchart in FIG. 12 shows details of the operation in step S1100.

  In the automatic focus control in step S1100, first, step S1121 is executed.

  In step S1121, the counter i is initialized to 0. Next, step S1122 is executed.

  In step S1122, an actuator position command is calculated. As shown in the following equation (2), the actuator position command Xact is calculated using the counter i. The position command Xact indicates a position command in which the direction toward the subject is positive with reference to the focus position of the infinity image. Here, kx is a set value. Next, step S1123 is executed.

    Xact = kx · i (2)

  In step S1123, an actuator operation amount (applied voltage to the coil of the actuator movable portion 115) Vact is calculated using the operation amount function expressed by the following equation (3). Here, ka and kb are respectively set values. Next, step S1124 is executed.

    Vact = ka · Xact + kb (3)

  In step S1124, the actuator is operated. The actuator operation amount output unit 134 changes the voltage signal to be output so that the voltage applied to the coil (not shown) of the actuator movable unit 115 via the amplifier 126 becomes Vact. Next, step S1125 is executed.

  In step S1125, shooting is performed. In response to a command from the system control unit 131, the image sensor driving unit 132 outputs a signal for electronic shutter and transfer as needed. The first imaging signal input unit 133a, the second imaging signal input unit 133b, the third imaging signal input unit 133c, and the fourth imaging signal input unit 133d are synchronized with the signal generated by the imaging element driving unit 132. Then, an image pickup signal that is an analog signal of each image output from the first image pickup element 123a, the second image pickup element 123b, the third image pickup element 123c, and the fourth image pickup element 123d is input, and fixed noise is generated by the CDS. , The AGC automatically adjusts the input gain, the ADC converts the analog signal into a digital value, the first imaging signal I1 (x, y), the second imaging signal I2 (x, y), As the third imaging signal I3 (x, y) and the fourth imaging signal I4 (x, y), digital values are written in a memory at a predetermined address of the system control unit 131. FIG. 13 is a diagram for explaining the coordinates of the imaging signal of the imaging apparatus according to Embodiment 1 of the present invention. I1 (x, y) indicates that the first imaging signal is the xth image in the horizontal direction and the yth image signal in the vertical direction. The number of pixels in the vertical direction of the input image is H, the number of pixels in the horizontal direction is L, the total number of pixels is H × L, x varies from 0 to L−1, and y varies from 0 to H−. It changes to 1. The same applies to the second imaging signal I2 (x, y), the third imaging signal I3 (x, y), and the fourth imaging signal I4 (x, y). That is, I2 (x, y), I3 (x, y), and I4 (x, y) are respectively the x-th second imaging signal in the horizontal direction and the y-th second imaging signal in the vertical direction, the third imaging signal, And the fourth imaging signal. Next, step S1126 is executed.

  In step S1126, an automatic focus control block is set. A rectangular area near the center of the image area is defined as an autofocus control block. Note that this block does not necessarily have to be in the vicinity of the center, and the block may be set by reflecting the intention of the user who operates the camera (for example, the viewpoint direction is detected by a sensor). Note that a plurality of blocks may be selected instead of a single block, and an average of contrast evaluation values for automatic focus control described later may be used in the plurality of blocks. Further, a contrast evaluation value for automatic focus control, which will be described later, may be calculated in a plurality of blocks, and at least one block may be selected later as an automatic focus control block. Next, step S1127 is executed.

  In step S1127, a contrast evaluation value for automatic focus control is created using data on the memory of the system control unit 131. This calculation is performed for the pixels of the automatic focus control block of the first imaging signal I1. The absolute value of the Laplacian that is the sum of the second derivative in the x direction and the y direction is calculated as in the following equation (4), and spatially LPF (low pass filter) as in the following equation (5). Then, this is averaged in the autofocus control block as shown in the following formula (6) to obtain the autofocus contrast evaluation value C3. Here, Naf is the number of pixels of the automatic focus control block. Next, step S1128 is executed.

C1 (x, y) = | I1 (x-1, y) + I1 (x + 1, y) + I1 (x, y-1) + I1 (x, y + 1)-4I1 (x, y) |
... (4)

C2 (x, y) = C1 (x-1, y-1) + C1 (x, y-1) + C1 (x + 1, y-1)
+ C1 (x-1, y) + C1 (x, y) + C1 (x + 1, y)
+ C1 (x-1, y + 1) + C1 (x, y + 1) + C1 (x + 1, y + 1) (5)

    C3 = ΣC2 (x, y) / Naf (6)

  In step S1128, the contrast evaluation value C3 is written as C3 (i) in the memory of the system control unit 131 as shown in the following equation (7). Next, step S1129 is executed.

    C3 (i) = C3 (7)

  In step S1129, 1 is added to the counter i as shown in the following equation (8). Next, step S1130 is executed.

  i = i + 1 (8)

  In step S1130, the counter i is compared with the threshold value Saf, and the result is branched. When the counter i is smaller than the threshold value Saf (the comparison result in step S1130 is Y), next, step 1122 is executed. On the other hand, when the counter i is equal to or greater than the threshold value Saf (the comparison result in step S1130 is N), next, step S1140 is executed. In this way, the counter i is initialized to 0 in step S1121, 1 is added to the counter i in step S1129, and branching is performed by the counter i in step S1130, so that the processing from S1122 to S1128 is repeated Saf times.

  In step S1140, the contrast evaluation value C3 is evaluated. As shown in FIG. 9, the contrast evaluation value C3 is maximized at the in-focus position. As shown in the following equation (9), the counter value i that gives the maximum value is set as a counter value iaf that gives the maximum contrast value. Next, step S1151 is executed.

  iaf = i giving the maximum value of C3 (9)

  In step S1151, an actuator position command is calculated. As shown in the following equation (10), the actuator position command Xact is calculated using the counter value iaf that gives the maximum contrast value. The position command Xact indicates a command in which the direction toward the subject is a positive position with reference to the focus position of the infinity image. Next, step S1152 is executed.

    Xact = kx · iaf (10)

  In step S1152, an actuator operation amount (voltage applied to the coil of the actuator movable portion 115) Vact is calculated using an operation amount function. This operation is the same as that in step S1123, and a description thereof will be omitted. Next, step S1153 is executed.

  In step S1153, the actuator is operated. This operation is the same as that in step S1124, and a description thereof will be omitted. Next, step S1160 is executed.

  In step S1160, the automatic focus control is terminated and the process returns to the main routine. Therefore, step S1200 in FIG. 11 is executed next.

  In step S1200, parallax correction is performed. FIG. 14 is a flowchart showing the parallax correction operation according to Embodiment 1 of the present invention. The flowchart in FIG. 14 shows details of the operation in step S1200. In the parallax correction in step S1200, first, step S1210 is executed.

  In step S1210, shooting is performed. In response to a command from the system control unit 131, the image sensor driving unit 132 outputs a signal for electronic shutter and transfer as needed. The first imaging signal input unit 133a, the second imaging signal input unit 133b, the third imaging signal input unit 133c, and the fourth imaging signal input unit 133d are synchronized with the signal generated by the imaging element driving unit 132. Then, an image pickup signal that is an analog signal of each image output from the first image pickup element 123a, the second image pickup element 123b, the third image pickup element 123c, and the fourth image pickup element 123d is input, and fixed noise is generated by the CDS. , The AGC automatically adjusts the input gain, the ADC converts the analog signal into a digital value, the first imaging signal I1 (x, y), the second imaging signal I2 (x, y), As the third imaging signal I3 (x, y) and the fourth imaging signal I4 (x, y), digital values are written in a memory at a predetermined address of the system control unit 131. Next, step S1220 is executed.

  In step S1220, the image expansion unit 142 performs image expansion using data on the memory of the system control unit 131. Then, the result is written in the memory of the system control unit 131. FIG. 15 is a diagram for explaining image expansion in the imaging apparatus according to Embodiment 1 of the present invention. Since only the first imaging signal I1 and the fourth imaging signal I4 are used for the calculation of the parallax evaluation value, which will be described later, this image expansion processing is performed by the first imaging signal I1 and the fourth imaging signal. Only for I4, the first post-expansion imaging signal I1s and the fourth post-expansion imaging signal I4s are created. Specifically, the y coordinate is converted as shown in the following formulas (11) and (12). In general, since Dx / Dy · y is not an integer, interpolation processing is performed as appropriate. For example, linear interpolation processing such as the following formulas (13), (14), (15), and (16) is performed. Here, the interpolation method is not limited to linear interpolation, and higher-order interpolation may be used. By performing this processing, the number of pixels in the vertical direction is multiplied by Dx / Dy as shown in FIG. 15, and the number of pixels in the vertical direction of the first post-expansion imaging signal I1s and the fourth post-expansion imaging signal I4s is H · Dx / Dy. Next, step S1230 is executed.

I1s (x, y) = I1 (x, Dx / Dy · y) (11)
I4s (x, y) = I4s (x, Dx / Dy · y) (12)

yi = integer part (Dx / Dy · y) (13)
yf = Decimal part (Dx / Dy · y) (14)
I1s (x, y) = (1-yf) · I1 (x, yi) + yf · I1 (x, yi + 1) (15)
I4s (x, y) = (1-yf) · I4 (x, yi) + yf · I4 (x, yi + 1) (16)

  In step S1230, the block dividing unit 143 performs block division using data on the memory of the system control unit 131. Then, the result is written in the memory of the system control unit 131. FIG. 16 is a diagram for explaining block division in the imaging apparatus according to Embodiment 1 of the present invention. As shown in FIG. 16, the first post-expansion imaging signal I1s is divided into M blocks in the horizontal direction, N blocks in the vertical direction, and M × N in total, and each block is divided into B ( u, v). Here, u changes from 0 to M-1, and v changes from N-1. Next, step S1240 is executed.

  In step S1240, the parallax calculation unit 144 uses the data on the memory of the system control unit 131 to calculate the parallax value for each block. Then, the data is written in the memory of the system control unit 131. FIG. 17 is a flowchart showing an operation of parallax calculation according to Embodiment 1 of the present invention. The flowchart in FIG. 17 shows details of the operation in step S1240. In the parallax calculation in step S1240, first, step S1241 is executed.

  In step S1241, approximate parallax is calculated. First, for each block (B (0,0), B (1,0),..., B (u, v),..., B (M-1, N-1)), the parallax evaluation value (R ( (0,0) (k), R (1,0) (k), ..., R (u, v) (k), ..., R (M-1, N-1) (k)). Here, k is a shift amount indicating how much the image is shifted, and is changed as k = 1, 3, 5,..., Kmax (kmax is an odd number). In this way, the shift amount k is changed so as to take only odd numbers. That is, k is changed so as to take every other value. FIG. 18 is a diagram for explaining the calculation area of the parallax evaluation value in the imaging apparatus according to Embodiment 1 of the present invention. As shown in FIG. 18B, the region indicated by B (u, v) (also indicated as I1s) is the u-th in the x direction obtained from the first post-expansion imaging signal I1s in step S1230. , The vth block in the y direction.

  The region indicated by I4s is a region moved from B (u, v) by the shift amount k in the x direction and by the shift amount k in the y direction, that is, moved by 45 °. Then, with respect to the image signals I1s (x, y) and I4s (xk, yk) after full expansion of each region, the absolute value difference sum (SAD: Sum of Absolute Differences) shown in the following equation (17) is calculated, The parallax evaluation value R (u, v) (k). However, in equation (17), the sum operation is performed by thinning out pixels in a zigzag pattern. That is, in FIG. 18B, only the blacked pixels are used for the sum calculation. It should be noted that the parallax evaluation value R (u, v) (k) is determined by the first post-expansion image signal I1s and the fourth post-expansion image signal I4s obtained by expanding the first image pickup signal I1 and the fourth image pickup signal I4. Therefore, the parallax direction is 45 ° and the moving direction is 45 °. If the parallax evaluation value Ri is obtained from the first image pickup signal I1 and the fourth image pickup signal I4 before expansion, as shown in FIG. 18A, the shift amount k in the x direction and k · Since it moves only by Dy / Dx, the y coordinate of the fourth imaging signal I4 is not always an integer as shown in the following equation (18). Therefore, an interpolation process is generally required. Therefore, when calculating the parallax evaluation value Ri from the first imaging signal I1 and the fourth imaging signal I4 before expansion, the amount of calculation is large.

R (u, v) (k) = ΣΣ | I1s (x, y) −I4s (xk, yk) | (17)
R (u, v) (k) = ΣΣ | I1 (x, y) −I4 (xk, yk · Dy / Dx) | (18)

  The parallax evaluation value R (u, v) (k) is the first decompressed image signal I1s of the u-th block in the x direction and the v-th block B (u, v) in the y direction, and the x and y directions. It indicates how much the fourth decompressed image signal I4s in the region separated by (k, k) has a correlation, and indicates that the smaller the value, the larger (similar) the correlation. FIG. 19 is a diagram for explaining the relationship between the schematic parallax and the parallax evaluation value of the imaging apparatus according to Embodiment 1 of the present invention. As shown in FIG. 19, the parallax evaluation value R (u, v) (k) varies depending on the value of the shift amount k, and has a minimum value when the shift amount k = Δg (u, v). This means that the u-th block in the x direction and the v-th block B (u, v) in the y direction of the fourth decompressed image signal I4s are (−Δg (u, v), −Δg ( The image signal obtained by moving only u, v)) is most correlated (similar) with the first decompressed image signal I1s. Therefore, the parallaxes in the x and y directions between the first post-expansion imaging signal I1s and the fourth imaging signal I4s for the block B (u, v) are (Δg (u, v), Δg (u, v)). It turns out that it is. Hereinafter, this Δg (u, v) is referred to as an approximate parallax value Δg (u, v) of the block B (u, v). In this way, the approximate parallax value Δg (u, v) of B (u, v) is obtained from u = 0 to M−1 and v = 0 to N−1. Next, step S1242 is executed.

  In step S1242, a detailed parallax is calculated. In step S1241, since the shift amount k is only an odd number, the resolution is 2. In step S1242, the parallax with resolution 1 is calculated. FIG. 20 is a diagram for explaining the relationship between the detailed parallax and the parallax evaluation value of the imaging apparatus according to Embodiment 1 of the present invention. In step 1241, as indicated by a black circle (●) in FIG. 20, k is changed by 2 in the block B (u, v) to give the minimum value of the parallax evaluation value R (u, v) (k). (Approximate parallax Δg (u, v)) was obtained. In step S1242, the parallax evaluation value R (u, v) (Δg (u, v) −1 at k before and after the approximate parallax Δg (u, v), as indicated by the white circle (◯) in FIG. ), R (u, v) (Δg (u, v) +1), and these parallax evaluation values (R (u, v) (Δg (u, v) -1), R (u, v) ) (Δg (u, v)), R (u, v) (Δg (u, v) +1)), the shift amount k that gives the minimum value is the detailed parallax Δs (u, v). That is, when R (u, v) (Δg (u, v) -1) is minimum as shown in FIG. 20A and the following equation (19), 1 is subtracted from the approximate parallax Δg (u, v). This is defined as a detailed parallax Δs (u, v). Further, as shown in FIG. 20B and the following equation (20), when R (u, v) (Δg (u, v)) is minimum, the approximate parallax Δg (u, v) is changed to the detailed parallax Δs. (u, v). Further, as shown in FIG. 20C and the following equation (21), when R (u, v) (Δg (u, v) +1) is minimum, 1 is set to the approximate parallax Δg (u, v). The addition is defined as a detailed parallax Δs (u, v). Although the resolution of the approximate parallax Δg (u, v) is 2, a detailed parallax Δs (u, v) having a resolution of 1 can be obtained by the above operation. Next, step S1243 is executed.

Δs (u, v) = Δg (u, v)-1 (when R (u, v) (Δg (u, v) -1) is minimum)
... (19)
Δs (u, v) = Δg (u, v) (when R (u, v) (Δg (u, v)) is minimum)
... (20)
Δs (u, v) = Δg (u, v) + 1 (when R (u, v) (Δg (u, v) +1) is minimum)
(21)

  In step S1243, the parallax of the subpixel is calculated. Parallax evaluation value R (u, v) (Δs (u, v) -1) with a parallax evaluation value R (u, v) (Δs (u, v)) giving a detailed parallax and a shift amount k before and after that , R (u, v) (Δs (u, v) +1) is used to calculate the parallax Δf (u, v) of the sub-pixel (finer than the pixel resolution) using the area distribution method . Specifically, when R (u, v) (Δg (u, v) -1) ≧ R (u, v) (Δg (u, v) +1), the calculation is performed as in the following formula (22). When R (u, v) (Δg (u, v) −1) <R (u, v) (Δg (u, v) +1), the calculation is performed as in (23). In order to perform the calculation using the shift calculation, the resolution of the sub-pixel parallax Δf (u, v) is preferably a value obtained by dividing 1 by a power of 2, such as 0.5, 0.25, 0.125, and 0.0625. Next, step S1244 is executed.

Δf (u, v) = Δs (u, v)-0.5
+ 0.5 ・ (R (u, v) (Δg (u, v) -1)-R (u, v) (Δg (u, v)))
/ (R (u, v) (Δg (u, v) +1)-R (u, v) (Δg (u, v)))
(When R (u, v) (Δg (u, v) -1) ≧ R (u, v) (Δg (u, v) +1))
(22)
Δf (u, v) = Δs (u, v) + 0.5
-0.5 ・ (R (u, v) (Δg (u, v) +1)-R (u, v) (Δg (u, v)))
/ (R (u, v) (Δg (u, v) -1)-R (u, v) (Δg (u, v)))
(When R (u, v) (Δg (u, v) -1) <R (u, v) (Δg (u, v) +1))
... (23)

  In step S1244, the parallax calculation is terminated and the process returns to the upper routine. Therefore, next, step S1250 of FIG. 14 is executed.

  In step S1250, the parallax correction unit 145 uses the data in the memory of the system control unit 131 to correct the parallax for each block using the parallax value corresponding to the block, and then combines the images. Then, the result is written in the memory of the system control unit 131. Since the first image sensor 123a and the fourth image sensor 123d mainly receive the green component of the object light, the first image signal I1 and the fourth image signal I4 are information on the green component of the object light. Signal. Further, since the second imaging signal 123b mainly receives the blue component of the object light, the second imaging signal I2 is an information signal of the blue component of the object light. Further, since the third imaging signal 123c mainly receives the red component of the object light, the third imaging signal I3 is an information signal of the red component of the object light. Since the parallax between the first image sensor 123a and the fourth image sensor 123d is predicted to be (Δf (u, v), Δf (u, v) · Dy / Dx), the following equation (23) is satisfied. G (x, y) indicating the green intensity at the pixel coordinates (x, y), the first imaging signal I1 (x, y), and the fourth imaging signal I4 (x−Δf (u, v ), y−Δf (u, v) · Dy / Dx). Further, since the parallax between the first image sensor 123a and the second image sensor 123b is predicted to be (Δf (u, v), 0), the pixel coordinates (x, y B (x, y) indicating the intensity of blue in) is defined as a second imaging signal I2 (x−Δf (u, v), y). Furthermore, since the parallax between the first image sensor 123a and the third image sensor 123c is predicted to be (0, Δf (u, v) · Dy / Dx), (x , Y) R (x, y) indicating the intensity of red is a third imaging signal I3 (x, y-Δf (u, v) · Dy / Dx). As the parallax value Δf (u, v), the parallax value Δf (u, v) of the block B (u, v) when the pixel coordinates (x, y) are mapped to the first post-expansion imaging signal I1. Is used. Next, step S1700 is executed.

G (x, y) = [I1 (x, y) I4 (x-Δf (u, v), y-Δf (u, v) · Dy / Dx)] / 2 (23)
B (x, y) = I2 (x-Δf (u, v), y) (24)
R (x, y) = I3 (x, y-Δf (u, v) · Dy / Dx) (25)

  In step S1700, the distance calculation unit 146 uses the data on the memory of the system control unit 131 to perform distance calculation. Then, the result is written in the memory of the system control unit 131. Solving equation (1) for distance A gives the following equation (26). Therefore, the distance of the subject in the block B (u, v) is calculated as in the following equation (27), and the distance of the subject in the pixel (x, y) included in the block B (u, v) is This is expressed as Expression (28) and stored in the memory of the system control unit 131. Note that the unit is appropriately changed at the time of calculation. Next, step S1800 is executed.

A = f · D / Δ (26)
A (u, v) = f · D / Δf (u, v) (27)
A (x, y) = A (u, v) ((x, y) is included in Bi) (28)

  In step S1800, the result is output. The input / output unit 136 receives image data G (x, y), B (x, y), R (x, y), and distance data A (x, y), which are data on the memory of the system control unit 131. And output to a host CPU (not shown) or an external display device (not shown). Instead of G (x, y), B (x, y), R (x, y), for example, luminance, color difference signals, etc. may be output. Further, values after image processing such as white balance and γ correction may be output. Furthermore, data that has been subjected to lossy compression or lossy compression such as JPEG may be output. Next, S1900 is executed.

  In step S1900, the operation ends.

  The configuration and operation as described above have the following effects.

  The relative positions of the four object images respectively formed by the first to fourth lens portions 113a to 113d change according to the subject distance A as shown in Expression (1). That is, as the subject distance A decreases, the parallax Δ increases. Therefore, when a plurality of subjects having different distances are simultaneously photographed, the parallax Δ is different for each subject. In the image device according to Embodiment 1, in step S1230, the entire image region is divided into blocks, in step S1240, the parallax for each block is calculated, and in step S1250, the influence of parallax is reduced based on the parallax for each block. Thus, parallax correction is performed by combining the images. As a result, even when simultaneously shooting a plurality of subjects at different distances, the parallax of each subject is appropriately corrected using the parallax obtained for each block, and a clean image in which the influence of the parallax is reduced in the entire image area Can be obtained.

  Further, in the imaging apparatus according to the first embodiment, the disparity method is used in S1243 to calculate the parallax with a resolution finer than the pixel resolution, and the influence of the parallax is reduced based on the fine resolution parallax for each block. After correcting the imaging signal as described above, the image is synthesized in step S1250. At this time, the parallax is obtained with sub-pixel resolution (with finer resolution than the pixel resolution). As a result, even when a plurality of subjects with different distances are simultaneously imaged, the parallax of each subject is appropriately reduced with sub-pixel resolution (a finer resolution than the pixel resolution) using fine parallax obtained for each block. Can be corrected (with resolution) to obtain a clean image with reduced parallax effects in the entire image area.

  In the imaging apparatus of Embodiment 1, the approximate parallax Δg (u, v) having a resolution of 2 is obtained in step S1241, and the detailed parallax Δs (u, v) having a resolution of 1 is obtained in step S1242. As a result, the calculation evaluation value can be reduced as compared with the case where k is changed one by one. Therefore, even when a plurality of subjects having different distances are simultaneously imaged with a lower cost calculation circuit, the calculation is calculated for each block. It is possible to appropriately correct the parallax of each subject using the parallax, and obtain a clean image in which the influence of the parallax is reduced in the entire image area. For example, when k is changed to 128, in order to obtain detailed parallax Δs (u, v), the imaging apparatus needs to calculate the parallax evaluation value 129 times when k is changed one by one. The imaging apparatus can suppress the calculation of the parallax evaluation value to 66 times.

  In the imaging device of Embodiment 1, the parallax is obtained using a signal obtained by thinning out the expanded imaging signal in a staggered manner in step S1240. As a result, it is possible to reduce the calculation of the parallax evaluation value as compared with the case where the parallax is calculated using all the post-expansion imaging signals, and therefore when a plurality of subjects with different distances are simultaneously imaged with a lower cost arithmetic circuit. In addition, it is possible to appropriately correct the parallax of each subject using the parallax obtained for each block, and to obtain a beautiful image in which the influence of the parallax is reduced in the entire image region.

  In the imaging device of Embodiment 1, in step S1220, the first imaging signal I1 and the fourth imaging signal I4 are expanded in the vertical direction by Dx / Dy, and the first post-expansion imaging signal I1s and By creating the post-expansion image signal I4s of 4, interpolation processing becomes unnecessary when calculating the parallax evaluation value in step S1240. As a result, the amount of calculation can be reduced as compared with the case where the first imaging signal I1 and the fourth imaging signal I4 are used, so that a plurality of subjects with different distances can be simultaneously imaged with a lower cost arithmetic circuit. Even in this case, it is possible to appropriately correct the parallax of each subject using the parallax obtained for each block, and to obtain a beautiful image in which the influence of the parallax is reduced in the entire image region.

  In the above-described first embodiment, as shown in FIG. 15, since the number of pixels H in the vertical direction is smaller than the number of pixels L in the horizontal direction, the imaging signal is expanded in the vertical direction in step S1220. If the number of pixels H in the vertical direction is larger than the number of pixels L in the horizontal direction, the imaging signal may be expanded in the horizontal direction in step S1220.

  In the imaging apparatus of Embodiment 1, the parallax is calculated in step S1200, the distance is calculated based on this parallax in step S1700, and the result is output in step S1800. Thus, not only the image but also the distance of the subject can be output, and processing using distance information (for example, obstacle detection) can be performed at the upper level. Furthermore, more detailed distance information can be obtained by obtaining the parallax of the sub-pixel (the parallax having a finer resolution than the resolution of the pixel).

(Embodiment 2)
The imaging apparatus according to Embodiment 1 of the present invention uses a proration method to calculate the parallax with a resolution finer than the resolution of the pixels. The imaging device according to Embodiment 2 of the present invention is configured to obtain an imaging signal indicating an image formed by one lens unit and an imaging signal indicating an image formed by one lens unit different from the lens unit. By shifting and comparing with a resolution finer than the resolution, the parallax is calculated with a resolution finer than the resolution of the pixel, and the image is synthesized and parallax corrected so as to reduce the influence of the parallax based on this parallax. A clear image in which the influence of parallax is reduced in a region is obtained.

  In addition, the imaging apparatus according to Embodiment 1 of the present invention creates parallax evaluation values by thinning out the imaging elements after stretching in a zigzag pattern. The imaging apparatus according to Embodiment 2 of the present invention can reduce the calculation of the parallax evaluation value by thinning out the image sensor after every other row to create the parallax evaluation value, so that the calculation circuit with a lower cost can be used. Even when a plurality of subjects having different distances are simultaneously imaged, it is possible to appropriately correct the parallax of each subject and obtain a clear image in which the influence of the parallax is reduced in the entire image region.

  An imaging apparatus according to Embodiment 2 of the present invention will be described with reference to the drawings.

  FIG. 21 is a cross-sectional view showing a configuration of a camera module according to Embodiment 2 of the present invention. The components other than the SLSI 225 of the circuit unit 220 of the camera module 201 are the same as those in the first embodiment, and the same members as those in the first embodiment are denoted by the same reference numerals and the description thereof is omitted.

  FIG. 22 is a block diagram of a camera module according to Embodiment 2 of the present invention. The SLSI 225 includes a system control unit 231, an imaging element driving unit 132, an imaging signal input unit 133, an actuator operation amount output unit 134, an image processing unit 235, and an input / output unit 136. The circuit unit 220 includes an amplifier 126 in addition to the above-described configuration.

  The system control unit 231 includes a CPU, a memory, and the like, and controls the entire SLSI 225.

  The image processing unit 235 includes a logic circuit, a DSP, or both, and performs various types of image processing using the memory information of the system control unit 231. The image processing unit 235 includes an automatic focus control unit 141, an image expansion unit 142, a block division unit 143, a parallax calculation unit 244, a parallax correction unit 145, and a distance calculation unit 146.

  The imaging element driving unit 132, the imaging signal input unit 133, the actuator operation amount output unit 134, the input / output unit 136, and the amplifier 126 are the same as those in the first embodiment, and a description thereof is omitted.

  The imaging apparatus according to the second embodiment is different from the imaging apparatus according to the first embodiment only in the operation of the parallax calculation step (step S1240 in FIG. 14). Other operations are the same as those in the first embodiment, and a description thereof will be omitted.

  In step S1240, the parallax calculation unit 244 uses the data on the memory of the system control unit 231 to calculate the parallax value for each block. Then, the data is written in the memory of the system control unit 231. FIG. 23 is a flowchart showing an operation of parallax calculation according to Embodiment 2 of the present invention. The flowchart in FIG. 23 shows details of the operation in step S1240. In the parallax calculation in step S1240, first, step S2241 is executed.

In step S2241, approximate parallax is calculated. FIG. 24 is a diagram illustrating a parallax evaluation value calculation region in the imaging apparatus according to Embodiment 2 of the present invention.
Since the operation in this step is the same as that in step S1241 in the first embodiment, description thereof is omitted. However, in the imaging apparatus according to the first embodiment, the summation calculation is performed by thinning out the pixels in a staggered manner in Expression (17). However, in the imaging apparatus according to the second embodiment, the summation calculation is performed by adding one pixel in Expression (17). Thin out every other line. That is, in FIG. 24B, only the blacked pixels are used for the sum calculation. Next, step S2242 is executed.

  In step S2242, a detailed parallax is calculated. This operation is the same as step S1242 in the embodiment, and a description thereof will be omitted. Next, step S2243 is executed.

  In step S2243, the parallax is calculated with sub-pixels (with a resolution finer than the pixel resolution). FIG. 25 is a diagram for explaining the relationship between the parallax of the sub-pixel and the parallax evaluation value of the imaging apparatus according to Embodiment 2 of the present invention. The black circles (points indicated by ●) in FIG. 25 are the parallax evaluation value with the detailed parallax Δs (u, v) and the parallax evaluation value when the shift amount k is changed by 1 before and after. Change k with a resolution of 0.125 at 1 before and after the detailed parallax Δs (u, v) (points indicated by white circles and circles) to minimize the parallax evaluation value R (u, v) (k) The sub-pixel parallax Δf (u, v). Since k is a decimal, interpolation processing is performed to create a fourth post-expansion imaging signal I4, which is substituted into equation (17). For example, what is necessary is just to obtain | require using linear interpolation like following formula (29). Here, in Expression (29), ki is an integer part of k, and kf is a decimal part of k. Further, since the shift operation is used instead of the multiplication operation, the resolution of the shift amount k is preferably a value obtained by dividing 1 by a power of 2, such as 0.5, 0.25, 0.125, and 0.0625. Next, step S2244 is executed.

  I4s = (1-kf) ・ I4s (x-ki, y-ki) + kf ・ I4s (x-ki-1, y-ki-1) (29)

  In step S2244, the parallax calculation is terminated and the process returns to the upper routine. Therefore, next, step S1250 of FIG. 14 is executed.

  By configuring and operating as described above, the imaging apparatus of the second embodiment has the same effect as that of the first embodiment.

  That is, the imaging apparatus according to Embodiment 2 of the present invention includes an imaging signal indicating an image formed by one lens unit and an imaging signal indicating an image formed by one lens unit different from the lens unit, The parallax is calculated with a finer resolution than the pixel resolution by shifting and comparing with a finer resolution than the pixel resolution. As a result, even when a plurality of subjects with different distances are simultaneously imaged, the parallax of each subject is appropriately reduced with sub-pixel resolution (a finer resolution than the pixel resolution) using fine parallax obtained for each block. Can be corrected (with resolution) to obtain a clean image with reduced parallax effects in the entire image area.

  In addition, the imaging apparatus according to Embodiment 2 of the present invention can reduce the calculation of the parallax evaluation value by thinning out the image sensor after every other row to create the parallax evaluation value, thereby reducing the calculation at a lower cost. Even when a plurality of subjects with different distances are picked up simultaneously by the circuit, the parallax of each subject is corrected appropriately using the parallax obtained for each block, and the effect of the parallax is reduced in the entire image area. An image can be obtained.

(Embodiment 3)
The imaging apparatus according to Embodiment 1 of the present invention includes four lens units, and color filters (first lens units) corresponding to diagonal lens units (first lens unit 113a and fourth lens unit 113d). The first color expansion 124a and the fourth color filter 124d have the same spectral transmittance, and the corresponding image signals (the first image signal I1 and the fourth image signal I4) are expanded in the vertical direction. The post-imaging signal I1s and the fourth post-expansion imaging signal I4s) were compared to determine the parallax. The imaging apparatus according to Embodiment 3 of the present invention has four lens units, and corresponds to a pair of lens units in the horizontal direction (first lens unit 113a and second lens unit 113b). The spectral transmittances of the first color filter 124a and the fourth color filter 124b are made equal, and the corresponding imaging signals (the first imaging signal I1 and the second imaging signal I2) are compared to obtain the parallax. Is.

  An imaging apparatus according to Embodiment 3 of the present invention will be described with reference to the drawings.

  FIG. 26 is a cross-sectional view showing the configuration of the imaging apparatus according to Embodiment 3 of the present invention. In FIG. 26, the imaging device includes a lens module unit 310 and a circuit unit 320.

  The lens module unit 310 includes a lens barrel 111, an upper cover glass 112, a lens 313, an actuator fixing unit 114, and an actuator movable unit 115. The circuit unit 320 includes a substrate 121, a package 122, an image sensor 123, a package cover glass 324, and a system LSI (hereinafter referred to as SLSI) 325.

  The lens barrel 111 and the upper cover glass 112 are the same as those in the first embodiment, and a description thereof is omitted.

  In the lens 113 of the first embodiment, the first lens portion 113a transmits green, the second lens portion 113b transmits blue, the third lens portion 113c transmits red, and the fourth lens portion 113d transmits green. The optimum design was made. In the lens 313 of Embodiment 3, the first lens portion 313a transmits green, the second lens portion 313b transmits green, the third lens portion 313c transmits red, and the fourth lens portion 313d transmits blue. The optimum design for this purpose has been made. The other parts of the lens 313 are the same as those of the lens 113 of the first embodiment, and a description thereof is omitted.

The actuator fixing unit 114 and the actuator movable unit 115 are the same as those in the first embodiment, and the description thereof is omitted. The substrate 121 is made of a resin substrate, and the lens barrel 111 is in contact with the bottom surface of the upper surface and fixed by an adhesive or the like. Is done. In this way, the lens module unit 310 and the circuit unit 320 are fixed to constitute the imaging device 301.

  The package 122 and the image sensor 123 are the same as those in the first embodiment, and a description thereof is omitted.

  FIG. 27 is a top view of the circuit unit 320 of the imaging apparatus according to Embodiment 3 of the present invention. The package cover glass 324 has a flat plate shape, is formed of a transparent resin, and is fixed to the upper surface of the package 122 by adhesion or the like. On the upper surface of the package cover glass 324, a first color filter 324a, a second color filter 324b, a third color filter 324c, a fourth color filter 324d, and a light shielding portion 324e are disposed by vapor deposition or the like. An infrared cut filter (not shown; hereinafter referred to as an IR filter) is disposed on the lower surface of the package cover glass 324 by vapor deposition or the like.

  FIG. 28 is a characteristic diagram of the color filter of the imaging apparatus according to Embodiment 3 of the present invention. The first color filter 324a has a spectral transmission characteristic that mainly transmits green shown by G in FIG. 28, and the second color filter 324b has a spectral transmission characteristic that mainly transmits green shown by G in FIG. The third color filter 324c has a spectral transmission characteristic mainly transmitting red as indicated by R in FIG. 28, and the fourth color filter 324d is a main color indicated by B in FIG. Has a spectral transmission characteristic of transmitting blue light. The IR filter is the same as that of the first embodiment, and the description thereof is omitted.

  Therefore, the object light incident from the upper part of the first lens unit 313a is emitted from the lower part of the first lens unit 313a, and mainly the green color is transmitted through the first color filter 324a and the IR filter. In order to form an image on the light receiving portion of the element 123a, the first imaging element 123a receives the green component of the object light. In addition, the object light incident from the upper part of the second lens unit 313b is emitted from the lower part of the second lens unit 313b, and mainly the green color is transmitted through the second color filter 324b and the IR filter. In order to form an image on the light receiving portion of the element 123b, the second imaging element 123b receives the green component of the object light. Further, the object light incident from the upper part of the third lens unit 313c is emitted from the lower part of the third lens unit 313c, and mainly the red color is transmitted through the third color filter 324c and the IR filter. In order to form an image on the light receiving portion of the element 123c, the third imaging element 123c receives the red component of the object light. Further, the object light incident from the upper part of the fourth lens unit 313d is emitted from the lower part of the fourth lens unit 313d, and mainly blue light is transmitted through the fourth color filter 324d and the IR filter, so that the fourth imaging is performed. In order to form an image on the light receiving portion of the element 123d, the fourth imaging element 123d receives the blue component of the object light.

  The SLSI 325 controls the energization of the coil of the actuator movable unit 115, drives the image sensor 123, inputs electrical information from the image sensor 123, performs various image processing, and communicates with the host CPU by a method described later. Output the image to the outside.

  Next, the operation of the imaging apparatus according to Embodiment 3 of the present invention will be described. FIG. 29 is a block diagram of an imaging apparatus according to Embodiment 3 of the present invention. The SLSI 325 includes a system control unit 331, an imaging element driving unit 132, an imaging signal input unit 133, an actuator operation amount output unit 134, an image processing unit 335, and an input / output unit 136. The circuit unit 320 includes an amplifier 126 in addition to the above configuration.

  The amplifier 126 applies a voltage corresponding to the output from the actuator operation amount output unit 134 to the coil of the actuator movable unit 115.

  The system control unit 331 includes a CPU (Central Processing Unit), a memory, and the like, and controls the entire SLSI 325.

  The imaging element driving unit 132, the imaging signal input unit 133, and the actuator operation amount output unit 134 are the same as those in the first embodiment, and a description thereof is omitted.

  The image processing unit 335 includes a logic circuit, a DSP, or both of them, and uses the memory information of the system control unit 331 to perform various image processing according to predetermined program control. The image processing unit 335 includes an automatic focus control unit 141, a block division unit 343, a parallax calculation unit 344, a parallax correction unit 345, and a distance calculation unit 146.

  The input / output unit 136 is the same as that of the first embodiment, and a description thereof will be omitted.

  FIG. 30 is a flowchart showing the operation of the imaging apparatus according to Embodiment 3 of the present invention. The imaging apparatus 301 is operated according to this flowchart by the system control unit 331 of the SLSI 325.

  In step S3000, the operation is started. For example, the host CPU (not shown) detects that a shutter button or the like has been pressed, and instructs the imaging device to start the operation via the input / output unit 136, whereby the imaging device 301 starts the operation. To do. Next, step S3100 is executed.

  In step S3100, the automatic focus control unit 141 executes automatic focus control. This operation is the same as step S1100 of the first embodiment, and a description thereof is omitted. Next, step S3200 is executed.

  In step S3200, parallax correction is performed. FIG. 31 is a flowchart showing a parallax correction operation according to Embodiment 3 of the present invention. The flowchart in FIG. 31 shows details of the operation in step S3200. In the parallax correction in step S3200, first, step S3210 is executed.

  In step S3210, shooting is performed. This operation is the same as that in step S1210 of the first embodiment, and a description thereof is omitted. Next, step S3230 is executed.

  In step S3230, the block division unit 343 performs block division using data on the memory of the system control unit 331. Then, the result is written in the memory of the system control unit 331. FIG. 32 is a diagram for explaining block division in the imaging apparatus according to Embodiment 3 of the present invention. As shown in FIG. 32, the first imaging signal I1 is divided into M blocks in the horizontal direction, N blocks in the vertical direction, and M × N in total, and each block is divided into B (u, Shown in v). Here, u changes from 0 to M-1, and v changes from N-1. Next, step S3240 is executed.

  In step S3240, the parallax calculation unit 344 uses the data on the memory of the system control unit 331 to calculate the parallax value for each block. Then, the data is written in the memory of the system control unit 331. FIG. 33 is a flowchart showing an operation of parallax calculation according to Embodiment 3 of the present invention. The flowchart in FIG. 33 shows details of the operation in step S3240. In the parallax calculation in step S3240, first, step S3241 is executed.

  In step S3241, approximate parallax is calculated. First, for each block (B (0,0), B (1,0),..., B (u, v),..., B (M-1, N-1)), the parallax evaluation value (R ( 0,0) (k), R (1,0) (k), ..., R (u, v) (k), ..., R (M-1, N-1) (k), k = 1, 3, 5, ..., kmax (where kmmax is an odd number). Thus, k is changed so as to take only odd numbers. FIG. 34 is a diagram for explaining the calculation area of the parallax evaluation value in the imaging apparatus according to Embodiment 3 of the present invention. As shown in FIG. 34, the area indicated by I2 is an area moved from B (u, v) by k in the x direction. Then, for all the imaging signals I1 (x, y) and I2 (xk, y) in each region, the absolute value difference sum (SAD: Sum of Absolute Differences) shown in the following equation (30) is used as the parallax evaluation value R ( Calculate as u, v) (k). However, in equation (30), the sum operation is performed by thinning out pixels in a zigzag pattern. That is, in FIG. 34, only the blacked pixels are used for the sum calculation.

    R (u, v) (k) = ΣΣ | I1 (x, y) -I2 (x-k, y) | (30)

  The parallax evaluation value R (u, v) (k) is the first image signal I1 of the u-th block B (u, v) in the x-direction and the v-th block in the y-direction and the shift amount k in the x-direction. It shows how much the second image signal I2 in the remote area is correlated, and the smaller the value, the larger (similarly) the correlation. As shown in FIG. 19, the parallax evaluation value R (u, v) (k) varies depending on the value of the shift amount k, and has a minimum value when the shift amount k = Δg (u, v). This is because the image signal obtained by moving the u-th block B (u, v) in the x-direction and the v-th block in the y-direction by −Δg (u, v) in the second image signal I2 is It shows that it is most correlated (similar) with the first image signal I1. Therefore, it can be seen that the parallax in the x direction between the first imaging signal I1 and the second image signal I2 for the block B (u, v) is Δg (u, v). Hereinafter, this Δg (u, v) is referred to as an approximate parallax value Δg (u, v) of the block B (u, v). In this way, the approximate parallax value Δg (u, v) of B (u, v) is obtained from u = 0 to M−1 and v = 0 to N−1. Next, step S3242 is executed.

  In step S3242, detailed parallax is calculated. This operation is the same as in the first embodiment, and a description thereof is omitted. However, Expression (30) is used as the parallax evaluation value R (u, v) (k). Next, step S3243 is executed.

  In step S1243, the parallax is calculated with subpixels. This operation is the same as in the first embodiment, and a description thereof is omitted. However, Expression (30) is used as the parallax evaluation value R (u, v) (k). Next, step S3244 is executed.

  In step S3244, the parallax calculation is terminated, and the process returns to the upper routine. Therefore, next, step S3250 of FIG. 33 is executed.

  In step S3250, the parallax correction unit 345 uses the data on the memory of the system control unit 331 to correct the parallax for each block using the parallax value corresponding to the block, and then combines the images. Then, the result is written in the memory of the system control unit 331. Since the first image sensor 123a and the second image sensor 123b mainly receive the green component of the object light, the first image signal I1 and the second image signal I2 are information on the green component of the object light. Signal. Further, since the second imaging signal 123c mainly receives the red component of the object light, the third imaging signal I3 is an information signal of the red component of the object light. Furthermore, since the fourth imaging signal 123d mainly receives the blue component of the object light, the fourth imaging signal I4 is an information signal of the blue component of the object light. Since the parallax between the first image sensor 123a and the fourth image sensor 123d is predicted to be (Δf (u, v), 0), the pixel coordinates (x, y) are expressed by the following equation (31). G (x, y) indicating the green intensity at is the average of the first imaging signal I1 (x, y) and the second imaging signal I2 (x−Δf (u, v), y). Furthermore, since the parallax between the first image sensor 123a and the third image sensor 123c is predicted to be (0, Δf (u, v) · Dy / Dx), (x , Y) R (x, y) indicating the intensity of red is a third imaging signal I3 (x, y-Δf (u, v) · Dy / Dx). Further, since the parallax between the first image sensor 123a and the fourth image sensor 123d is predicted to be (Δf (u, v), Δf (u, v) · Dy / Dx), the following equation (33) is satisfied. In this way, B (x, y) indicating the blue intensity at the pixel coordinates (x, y) is expressed as the fourth imaging signal I4 (x−Δf (u, v), y−Δf (u, v) · Dy. / Dx). As the parallax value Δf (u, v), the parallax value Δf (u, v) of the block B (u, v) including the pixel coordinates (x, y) is used. Next, step S3700 is executed.

G (x, y) = [I1 (x, y) + I2 (x−Δf (u, v), y)] / 2 (31)
R (x, y) = I3 (x, y-Δf (u, v) · Dy / Dx) (32)
B (x, y) = I4 (x-Δf (u, v), y-Δf (u, v) · Dy / Dx) (33)

  In step S3700, the distance calculation unit 146 uses the data on the memory of the system control unit 331 to perform a distance calculation. Then, the result is written in the memory of the system control unit 331. This operation is the same as that in step S1700 of the first embodiment, and a description thereof is omitted. Next, step S3800 is executed.

  In step S3800, the result is output. This operation is the same as step S1800 in the first embodiment, and a description thereof will be omitted. Next, S3900 is executed.

  In step S3900, the operation ends.

  By configuring and operating as described above, the same effects as those of the first embodiment can be obtained.

  As mentioned above, although some embodiment of this invention was described, these are illustration to the last, Comprising: When implementing this invention, various changes are possible as follows.

  For example, in the imaging devices according to the first to third embodiments, the calculated parallax is used as it is, but it may be appropriately limited. Depending on the lens characteristics, when the subject distance A is smaller than a certain value, the image becomes unclear. Therefore, if this value is determined as the minimum value of the subject distance A, the maximum value of parallax can be determined. Parallax greater than this value may be ignored as an error. In such a case, a value having the second smallest parallax evaluation value may be adopted as the parallax.

  Further, when there are two extreme values that are conspicuous in the parallax evaluation value, the larger parallax may be adopted. Such a block includes a subject and a background, and since the subject distance and the background distance are different, two extreme values appear. Since the subject distance is smaller than the background distance, the subject parallax is larger than the background parallax. Here, if the larger parallax is adopted, the influence of the parallax of the background cannot be reduced, but the influence of the parallax of the subject that directly affects the image quality can be reduced.

  Note that in the imaging devices of the first embodiment and the second embodiment, the first imaging signal I1 (mainly showing green) and the fourth imaging signal I4 (mainly showing green) are parallax. However, the present invention is not limited to this. For example, since a purple subject has a small green component and a large amount of blue and red components, the first imaging signal I1 (mainly green) and the fourth imaging signal I4 (mainly green) are used. If the calculation cannot be performed, the parallax may be calculated from the second imaging signal I2 (mainly showing blue) and the third imaging signal I3 (mainly showing red). In addition, the parallax cannot be calculated from the first parallax signal I1 (mainly green) and the fourth parallax signal I4 (mainly green), and the second imaging signal I2 (mainly blue) If the parallax cannot be calculated from the third imaging signal I3 (mainly red), it is considered that there is no parallax effect, and no parallax is required.

  The same applies to the third embodiment. For example, since a purple subject has a small green component and a large amount of blue and red components, the first imaging signal I1 (mainly green) and the second imaging signal I2 (mainly green) are used. If the calculation is impossible, the parallax may be calculated from the third imaging signal I3 (mainly showing red) and the fourth imaging signal I4 (mainly showing blue). Further, the parallax cannot be calculated from the first parallax signal I1 (mainly showing green) and the second parallax signal I2 (mainly showing green), and the third imaging signal I3 (mainly red) If the parallax cannot be calculated from the fourth imaging signal I4 (mainly blue), it is considered that there is no influence of the parallax, and no parallax is required.

  In addition, when the imaging device according to the first embodiment and the second embodiment is mounted on a camera, the second imaging element 123b (mainly forms an image of blue) on the upper side and the third on the lower side. By arranging the first to fourth image sensors 123a to 123d so that the image sensor 123c (mainly imaging red) is arranged, the upper side is sensitive to blue and the lower side is sensitive to red. Therefore, the color reproduction of landscape photos can be made more natural.

  Further, when the image pickup apparatus according to the third embodiment is mounted on a camera, the third image pickup device 123c below the first image pickup device 123a (mainly forms an image of green color and does not perform coordinate conversion during parallax correction). (Mainly forms an image of red. Coordinate conversion is performed in the y direction during parallax correction), and a fourth image sensor 123d (mainly forms an image of blue color on the diagonal of the first image sensor 123a). By arranging the first to fourth imaging elements 123a to 123d so that the coordinate conversion is sometimes performed in both the x direction and the y direction, the movement due to the coordinate conversion of the red color is reduced compared to the blue color. The visual sensitivity is higher in red than in blue, and therefore, red color unevenness is more conspicuous than blue color unevenness. By disposing like the image pickup apparatus of the third embodiment and reducing movement by coordinate conversion of red compared to blue, an image pickup apparatus in which color unevenness is not noticeable even when color unevenness occurs is realized.

  In addition, the imaging apparatuses according to the first to third embodiments perform focus control but may not operate the actuator. Further, the actuator may be omitted and focus control may be omitted.

  In Embodiments 1 to 3, the image sensor 123 includes a first image sensor 123a, a second image sensor 123b, a third image sensor 123c, and a fourth image sensor 123d. The imaging signal input unit 133 includes a first imaging signal input unit 133a, a second imaging signal input unit 133b, a third imaging signal input unit 133c, and a fourth imaging signal input unit 133d. However, the image sensor 123 may be configured by one image sensor, and four images may be formed by the first to fourth lens portions 113a to 113d at different positions on the light receiving surface. Further, the imaging signal input unit 133 may be configured by one imaging signal input unit to which a signal from one imaging element 123 is input. In this case, a region is appropriately selected from data placed on the memory of the system control units 131, 231, and 331, and the first imaging signal I1, the second imaging signal I2, the third imaging signal I3, and the fourth imaging signal are selected. The imaging signal I4 may be used.

  In Embodiments 1 to 3, the sum of absolute differences as in equations (17) and (30) is used as the parallax evaluation value R (u, v) (k), but the present invention is not limited to this. For example, in the first embodiment and the second embodiment, the sum of the squares of the differences, the difference between the first imaging signal I1 and the average in the block, and the fourth imaging signal I4 in the block average are calculated. The sum of the squares of the difference from the difference and the square value of the difference between the difference between the first imaging signal I1 and the average within the block from the fourth imaging signal I4. Or the sum of the squares of the difference between the difference between the first imaging signal I1 and the average within the block from the fourth imaging signal I4 and the difference between the averages within the block from the first imaging signal I1 Alternatively, a difference obtained by dividing the average in the block by the square root of the sum of the square values and dividing the average in the block from the fourth imaging signal I4 by the square root of the sum of the square values may be used.

  In the first to third embodiments, four imaging regions are provided in a grid pattern, but the present invention is not limited to this. For example, the first image sensor, the second image sensor, the third image sensor, and the fourth image sensor are arranged so as to be on one straight line, the first image signal I1 mainly shows blue, Alternatively, the second imaging signal I2 may be mainly green, the third imaging signal I3 may be mainly green, and the fourth imaging signal may be mainly red. In this case, it is necessary to change the parallax evaluation function as shown in the following formula (34) and to correct the parallax as shown in the following formulas (35), (36), and (37).

R (u, v) (k) = ΣΣ | I2 (x, y) -I3 (xk, y) | (34)
G (x, y) = [I2 (x, y) + I3 (x-Δf (u, v), y)] / 2 (35)
B (x, y) = I1 (x + Δf (u, v), y) (36)
R (x, y) = I4 (x-2 * Δf (u, v), y) (37)

  In Embodiments 1 to 3, four imaging regions are provided, but the present invention is not limited to this. For example, it has three imaging areas, the first imaging element, the second imaging element, and the third imaging element are arranged so as to be on one straight line, and the first imaging signal I1 mainly shows blue, You may change so that the 2nd imaging signal I2 may mainly show green and the 3rd imaging signal I3 may mainly show red. In this case, the parallax evaluation function is changed as in the following formulas (38) and (39), the green component G is created as in the following formula (40), and the parallax Δf (u calculated using the formula (38) , v) is used to create a blue component B as shown in the following equation (41), and the parallax Δf (u, v) obtained using the equation (39) is used as a red component as shown in the following equation (42) Changes such as creating R and correcting parallax are required. Further, a normalized correlation function may be used instead of the equations (38) and (39).

R (u, v) (k) = ΣΣ | I2 (x, y) -I1 (x + k, y) | (38)
R (u, v) (k) = ΣΣ | I2 (x, y) -I3 (xk, y) | (39)
G (x, y) = I2 (x, y) (40)
B (x, y) = I1 (x + Δf (u, v), y) (41)
R (x, y) = I3 (x-Δf (u, v), y) (42)

  In the imaging device according to the first to third embodiments, the parallax correction is performed using the parallax Δf (u, v) in the block, but the parallax of each pixel using the parallax Δf (u, v) for each block Δ (x, y) may be determined, and parallax correction may be performed using the parallax Δ (x, y) of each pixel.

  In the first to third embodiments, the blocks are divided into rectangular shapes, but the present invention is not limited to this. For example, an edge may be detected and divided into non-rectangular blocks based on the edge. Further, instead of obtaining the parallax of the area for each block, the edge may be divided into a plurality of line segments, and the parallax of the line segments may be obtained. Moreover, the parallax calculated | required in a certain block may be evaluated and a block may be divided | segmented or combined.

  Since the imaging device of the present invention is an imaging device that can be reduced in size and thickness, it is useful for a mobile phone having a camera function, a digital still camera, an in-vehicle camera, a surveillance camera, and the like.

Sectional drawing which shows the structure of the imaging device which concerns on Embodiment 1 of this invention. 1 is a top view of a lens of an imaging apparatus according to Embodiment 1 of the present invention. FIG. 3 is a top view of a circuit unit of the imaging apparatus according to Embodiment 1 of the present invention. FIG. 6 is a characteristic diagram of the color filter of the imaging apparatus according to Embodiment 1 of the present invention. IR filter characteristic diagram of the imaging apparatus according to Embodiment 1 of the present invention The figure for demonstrating the position of the object image in infinity in the imaging device which concerns on Embodiment 1 of this invention. The figure for demonstrating the position of the object image in the position of a finite distance in the imaging device which concerns on Embodiment 1 of this invention. (A) is a figure explaining the relationship between the image at the time of focusing and the contrast evaluation value in the imaging apparatus according to Embodiment 1 of the present invention, and (b) is the imaging apparatus according to Embodiment 1 of the present invention. The figure explaining the relationship between the image at the time of out-of-focus and the contrast evaluation value The figure explaining the relationship between the lens position of the imaging device which concerns on Embodiment 1 of this invention, and contrast evaluation value 1 is a block diagram of an imaging apparatus according to Embodiment 1 of the present invention. The flowchart which shows operation | movement of the imaging device which concerns on Embodiment 1 of this invention. The flowchart which shows the operation | movement of the automatic focus control which concerns on Embodiment 1 of this invention. The figure explaining the coordinate of the imaging signal of the imaging device which concerns on Embodiment 1 of this invention. The flowchart which shows the operation | movement of the parallax correction which concerns on Embodiment 1 of this invention. The figure explaining image expansion | extension in the imaging device which concerns on Embodiment 1 of this invention. The figure explaining block division in the imaging device concerning Embodiment 1 of the present invention. The flowchart which shows the operation | movement of the parallax calculation which concerns on Embodiment 1 of this invention. The figure explaining the calculation area | region of a parallax evaluation value in the imaging device which concerns on Embodiment 1 of this invention. The figure explaining the relationship between the rough parallax and parallax evaluation value of the imaging device which concerns on Embodiment 1 of this invention. The figure explaining the relationship between the detailed parallax of the imaging device which concerns on Embodiment 1 of this invention, and a parallax evaluation value. Sectional drawing which shows the structure of the imaging device which concerns on Embodiment 2 of this invention. Block diagram of an imaging apparatus according to Embodiment 2 of the present invention The flowchart which shows the operation | movement of the parallax calculation which concerns on Embodiment 2 of this invention. The figure explaining the calculation area | region of a parallax evaluation value in the imaging device which concerns on Embodiment 2 of this invention. The figure explaining the relationship between the parallax of the sub pixel of the imaging device which concerns on Embodiment 2 of this invention, and a parallax evaluation value. Sectional drawing which shows the structure of the imaging device which concerns on Embodiment 3 of this invention. FIG. 6 is a top view of a circuit unit of an imaging apparatus according to Embodiment 3 of the present invention. Characteristic diagram of color filter of imaging device according to embodiment 3 of the present invention Block diagram of an imaging apparatus according to Embodiment 3 of the present invention The flowchart which shows operation | movement of the imaging device which concerns on Embodiment 3 of this invention. The flowchart which shows the operation | movement of the parallax correction which concerns on Embodiment 3 of this invention. The figure explaining block division in the imaging device concerning Embodiment 3 of the present invention. The flowchart which shows the operation | movement of the parallax calculation which concerns on Embodiment 3 of this invention. The figure explaining the calculation area | region of a parallax evaluation value in the imaging device which concerns on Embodiment 3 of this invention. Sectional drawing which shows the structure of the camera module of patent document 1 The figure which shows an example of the camera module of patent document 2

Explanation of symbols

101, 201, 301 Camera module 110, 310 Lens module 111 Lens barrel 112 Upper cover glass 113, 313 Lens 113a, 313a First lens part 113b, 313b Second lens part 113c, 313c Third lens part 113d, 313d Fourth lens unit 114 Actuator fixing unit 115 Actuator movable unit 120, 220, 320 Circuit unit 121 Substrate 122 Package 123 Image sensor 123a First image sensor 123b Second image sensor 123c Third image sensor 123d Fourth image sensor Element 124, 324 Package cover glass 124a, 324a First color filter 124b, 324b Second color filter 124c, 324c Third color filter 124d, 324d Fourth color Filter 124e shielding portion 125, 225, 325 SLSI
126 Amplifier 131 231 331 System control unit 132 Image sensor drive unit 133 Imaging signal input unit 133a First imaging signal input unit 133b Second imaging signal input unit 133c Third imaging signal input unit 133d Fourth imaging signal Input unit 134 Actuator operation amount output unit 135, 235, 335 Image processing unit 136 Input / output unit 141 Automatic focus control unit 142 Image expansion unit 143, 343 Block division unit 144, 244, 344 Parallax calculation unit 145, 345 Parallax correction unit 146 Distance calculator

Claims (10)

A plurality of lens portions each including at least one lens;
A plurality of imaging regions each corresponding to the plurality of lens units, each having a light receiving surface substantially perpendicular to the optical axis direction of the corresponding lens unit;
An imaging signal input unit to which a plurality of imaging signals respectively output from the plurality of imaging regions are input;
A block dividing unit that divides at least one of the plurality of imaging signals into a plurality of blocks;
A parallax calculation unit that calculates the parallax between images formed by the plurality of lens units for each of the blocks with a resolution finer than the resolution of a pixel, using the imaging signal;
An imaging apparatus comprising: a parallax correction unit that corrects the plurality of imaging signals based on the parallax and combines the images.   The parallax calculation unit includes an imaging signal indicating an image formed by one lens unit among the plurality of lens units and an imaging signal indicating an image formed by a lens unit different from the one lens unit. A plurality of parallax evaluation values are created by changing the shift amount for multiple times, and an extreme value shift amount that is the shift amount that gives the extreme value of the parallax evaluation value is obtained. The imaging apparatus according to claim 1, wherein the parallax is calculated with a resolution finer than a resolution of a pixel based on the parallax evaluation value and the parallax evaluation value of the shift amount before and after the extreme value shift amount.   The parallax calculation unit includes an imaging signal indicating an image formed by one lens unit among the plurality of lens units and an imaging signal indicating an image formed by a lens unit different from the one lens unit. A plurality of parallax evaluation values are created by changing the shift amount at a resolution finer than the resolution of the pixel and comparing a plurality of times, and an extreme value shift amount that is the shift amount that gives the extreme value of the parallax evaluation value is determined. The imaging apparatus according to claim 1, wherein the parallax is calculated with a resolution finer than a pixel resolution based on the parallax evaluation value of the extreme value shift amount and the extreme value shift amount.   The parallax calculation unit includes an imaging signal indicating an image formed by one lens unit among the plurality of lens units and an imaging signal indicating an image formed by a lens unit different from the one lens unit. A plurality of parallax evaluation values are created by changing the shift amount for multiple times, and an extreme value shift amount that is the shift amount that gives the extreme value of the parallax evaluation value is obtained. Creating a slightly shifted parallax evaluation value by comparing the imaging signal by shifting the block smaller than the shift amount before and after, and calculating the parallax based on the parallax evaluation value and the slightly shifted evaluation value; The imaging device according to claim 1.   The parallax calculation unit includes a signal obtained by thinning out an imaging signal indicating an image formed by one lens unit among the plurality of lens units, and an imaging signal indicating an image formed by a lens unit different from the one lens unit. The imaging device according to claim 1, wherein a plurality of parallax evaluation values are created by comparing the thinned signal with a plurality of times by changing a shift amount of the block, and the parallax is calculated based on the parallax evaluation values. .   The parallax calculation unit indicates a signal obtained by thinning out an imaging signal indicating an image formed by one lens unit among the plurality of lens units in a zigzag manner, and an image formed by a lens unit different from the one lens unit. The parallax is calculated based on the parallax evaluation value by creating a plurality of parallax evaluation values by comparing a signal obtained by thinning out the imaging signals in a zigzag manner multiple times by changing a shift amount of the block. 5. The imaging device according to 5.   The parallax calculation unit indicates a signal obtained by thinning out an imaging signal indicating an image formed by one lens unit among the plurality of lens units, and an image formed by a lens unit different from the one lens unit. The parallax is calculated based on the parallax evaluation value by creating a plurality of parallax evaluation values by comparing a signal obtained by thinning the imaging signal every other line with a plurality of times by changing a shift amount of the block. 5. The imaging device according to 5. The parallax calculating unit compares the imaging signals of two lens units arranged in a diagonal direction of the imaging element among the plurality of lens units, calculates the parallax,
The two lens portions have different horizontal distances and vertical distances between optical axes,
The parallax calculation unit converts the imaging signal so that the number of pixels in the vertical direction and the number of pixels in the horizontal direction of the imaging signal are substantially equal according to a ratio of the distance in the horizontal direction to the distance in the vertical direction. A signal obtained by expanding an imaging signal indicating an image formed by one of the two lens units that extends in the vertical direction or the horizontal direction, and an image formed by the other lens unit of the two lens units. A plurality of parallax evaluation values are created by comparing a signal obtained by expanding an imaging signal shown with the block shift amount in a 45 ° direction and a plurality of comparisons, and the parallax is calculated based on the parallax evaluation values. The imaging device according to claim 1.   The parallax calculation unit calculates the parallax by comparing imaging signals of two lens units arranged in a diagonal direction of the imaging element among the plurality of lens units. The imaging device described in 1.   The parallax calculation unit calculates the parallax by comparing image signals of two lens units arranged in a horizontal direction of the imaging element among the plurality of lens units. The imaging device described.

Images