JP2011237215A - Depth map output device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a depth map output device capable of calculating an accurate distance to a subject even in a position where the distance to the subject changes rapidly.SOLUTION: A stereo imaging device 1 comprises an imaging lens 11 for forming an image of a subject image on a predetermined imaging surface, a microlens array 12 having microlenses ML arranged in a two-dimensional pattern near the imaging surface, a light receiving element arrangement 30 arranged corresponding to each of the plurality of microlenses ML, and a control circuit 40 for performing synthesis processing by selecting a plurality of lateral lens arrays arranged in the lateral direction, calculating output of light receiving element arrangement 30 corresponding to each of the microlenses ML constituting the lateral lens arrays for each of the lateral lens arrays, calculating a defocus amount of the microlenses ML of the center part of the lateral lens arrays, creating a lateral derived depth map which is data in which the defocus amount is arranged in a two-dimensional pattern and similarly creating a longitudinal derived depth map for the longitudinal direction, and synthesizing the lateral derived depth map and the longitudinal derived depth map to output a final depth map.

Description

  The present invention relates to a depth map output device that outputs a depth map that is distribution information of a distance to a subject.

  There has been known an imaging apparatus configured to individually handle light beams from different regions of an exit pupil of a photographing optical system (for example, Patent Document 1). Such an imaging apparatus employs a configuration in which a pixel array composed of a plurality of pixels is arranged behind each microlens constituting the microlens array. With such a configuration, light beams from different regions of the exit pupil are incident on different pixels, so that these light beams can be handled individually. By handling these luminous fluxes individually, it is possible to synthesize image data focused on a subject located at an arbitrary shooting distance, for example, by only one imaging. As another example, a plurality of images having parallax can be obtained by one imaging.

  On the other hand, Patent Document 2 describes, as Modification 4, an imaging device that creates a depth map that is distribution information of distances to a subject. This imaging apparatus creates a left viewpoint image and a right viewpoint image by a method similar to that of the imaging apparatus described in Patent Document 1 described above. Thereafter, a phase difference between the two images is calculated by a known stereo matching process, and a depth map is created based on the phase difference.

JP 2007-4471 A JP 2009-165115 A

  The imaging device described in Patent Document 2 has a problem in that an accurate distance to the subject cannot be calculated in the vicinity of a location where the perspective of the subject changes (for example, the boundary between the close subject and the background).

  The invention according to claim 1 includes an imaging optical system that forms an image of a subject on a predetermined imaging plane, a plurality of positive lenses arranged in a two-dimensional manner in the vicinity of the imaging plane, and a plurality of positive lenses. A plurality of light receiving elements arranged at the back of the positive lens corresponding to each of the light receiving elements, wherein at least two of the at least three light receiving elements are arranged in the first direction, and at least Among the three light receiving elements, at least two light receiving elements include a plurality of light receiving elements arranged in a second direction different from the first direction and a plurality of first lenses arranged in the first direction from a plurality of positive lenses. A first direction corresponding to each of the positive lenses constituting the first positive lens array for each of the first selection means for selecting the positive lens array and the plurality of first positive lens arrays selected by the first selection means Receive at least two lights in a row The first distance calculation means that calculates the output of the child and calculates a value representing the distance to the subject whose image is formed on the positive lens in the center of the first positive lens array, and the first distance calculation means. A first depth map creating means for creating a first depth map, which is data in which values representing a plurality of distances are two-dimensionally arranged in accordance with the positions of the positive lenses corresponding to the values, and a plurality of positive lenses Second selection means for selecting a plurality of second positive lens arrays arranged in the second direction, and for each of the plurality of second positive lens arrays selected by the second selection means, the second positive lens array The outputs of at least two light receiving elements arranged in the second direction corresponding to each of the positive lenses constituting the lens are calculated, and the image of the object connected to the positive lens at the center of the second positive lens array is obtained. A second distance calculating means for calculating a value representing the distance; A second depth map that creates a second depth map that is data in which values representing a plurality of distances calculated by the second distance calculating means are arranged two-dimensionally according to the positions of the positive lenses corresponding to the values. Creating means, and depth map output means for executing synthesis processing for synthesizing the first depth map and the second depth map and outputting a third depth map, corresponding to each of the plurality of positive lenses. At least two light receiving elements arranged in the first direction and at least two light receiving elements arranged in the second direction are light beams from different regions of the exit pupil of the imaging optical system via the positive lens. Is a depth map output device characterized by receiving light respectively.

  According to the present invention, a value representing an accurate distance to a subject can be calculated even in the vicinity of a portion where the perspective of the subject changes.

It is a figure which shows the structure of the stereo imaging device by the 1st Embodiment of this invention. 3 is a schematic diagram for explaining a horizontal selection function by a control circuit 40. FIG. It is a figure explaining the signal sequence used for calculation of the defocus amount. It is a figure which shows a mode that a phase difference is calculated. It is a figure which shows micro lens ML from which the defocus amount is calculated by the horizontal origin distance calculation function and the vertical origin distance calculation function. It is a figure which shows the range of the depth map finally output. It is a figure explaining the defocus amount in a horizontal origin depth map.

(First embodiment)
FIG. 1 is a diagram showing a configuration of a stereo imaging device according to the first embodiment of the present invention. The stereo imaging device 1 includes a left imaging unit 100 and a right imaging unit 200 that image a subject, and a control circuit 40 that includes a microprocessor or the like that controls the two imaging units.

  The left imaging unit 100 includes a photographic lens 11, a microlens array 12 disposed behind the photographic lens, and an imaging element 13 disposed on the rear side of the microlens array 12. Usually, the interval between the microlens array 12 and the image pickup device 13 is determined so that the light receiving surface of the image pickup device 13 is in the vicinity of the focal position of the microlens ML constituting the microlens array 12, but in FIG. It is drawn wider than.

  The microlens array 12 has a plurality of microlenses ML arranged in a two-dimensional shape. A plurality of light receiving elements are two-dimensionally arranged on the imaging surface of the imaging element 13 facing the microlens array 12. These light receiving elements are grouped by a certain number, and each group (light receiving element array 30) corresponds to each microlens ML. Hereinafter, this group is called a light receiving element array. In the present embodiment, the light receiving element array 30 including 25 light receiving elements arranged in 5 rows and 5 columns forms one group corresponding to each microlens ML.

  The individual light receiving elements constituting the light receiving element array 30 are arranged at a position substantially conjugate with the exit pupil of the photographing lens 11. As a result, each light receiving element constituting the light receiving element array 30 receives light beams from different regions of the exit pupil of the photographing lens 11 via the microlens ML corresponding to the light receiving element array 30. Note that subject light incident on a certain microlens ML may be incident as stray light on another light receiving element array 30 adjacent to the light receiving element array 30 corresponding to the microlens ML. In order to prevent such stray light, a light shielding mask is provided at the boundary between the microlenses ML, or a partition member for partitioning each light receiving element array 30 is provided between the microlens array 12 and the imaging element 13. It is desirable. In FIG. 1, illustration of such a member is omitted.

  An imaging control circuit 14 and a video circuit 15 are connected to the imaging element 13. The imaging control circuit 14 drives the imaging element 13 and controls exposure time, photoelectric conversion, and the like. The video circuit 15 performs amplification and analog-digital conversion of the electrical signal output from the image sensor 13 and outputs a digital signal to the control circuit 40. The control circuit 40 creates image data representing the subject image based on the digital signal. It should be noted that the resolution of the image data created by the control circuit 40 is limited by the number of microlenses ML in the microlens array 12 in an arbitrary focus photographing mode described later. For example, when the microlens array 12 includes K microlenses ML horizontally and M microlenses ML vertically, the number of pixels of the image data created by the control circuit 40 is K × M at the maximum.

  The right imaging unit 200 includes a photographic lens 21 and an image sensor 23 disposed behind the photographic lens 21. The individual light receiving elements constituting the image sensor 23 have the same size as the light receiving elements of the image sensor 13. Unlike the left imaging unit 100, the right imaging unit 200 does not have microlenses ML corresponding to a plurality of light receiving elements. The positional relationship between the photographing lens 11 of the left imaging unit 100 and the front main surface of the microlens array 12 is the same as the positional relationship between the photographing lens 21 of the right imaging unit 200 and the imaging surface (light receiving surface) of the imaging element 23 and optical. Are equivalent.

  An imaging control circuit 24 and a video circuit 25 are connected to the imaging element 23 as in the imaging element 13 of the left imaging unit 100. The imaging control circuit 24 drives the imaging device 23 to control exposure time, photoelectric conversion, and the like. The video circuit 25 performs amplification and analog-digital conversion of the electrical signal output from the image sensor 23 and outputs a digital signal to the control circuit 40. The control circuit 40 can create image data representing the subject image based on the digital signal. In the present embodiment, the number of pixels of the image data created based on the digital signal output from the right imaging unit 200 is larger than the image data synthesized from the digital signal output from the left imaging unit 100. Will increase. This is because the resolution of the former image data is limited by the number of microlenses ML, whereas the latter image data has no such limitation.

  The taking lens 11 and the taking lens 21 have an optically equivalent configuration. On the image pickup surfaces of the image pickup device 13 and the image pickup device 23, an optical low-pass filter (not shown), an infrared cut filter, and the like are provided. In FIG. 1, the photographic lens 11 and the photographic lens 21 are schematically shown as a single lens, but these may be constituted by a plurality of lenses.

  The left imaging unit 100 and the right imaging unit 200 described above are arranged so that the optical axes of the photographing lens 11 and the photographing lens 21 are separated by 60 mm, which is slightly shorter than the average eye width of adults. That is, the base line length S in FIG. 1 is 60 mm.

  The control circuit 40 is connected to the imaging control circuit 14 and the video circuit 15 of the left imaging unit 100 and the imaging control circuit 24 and the video circuit 25 of the right imaging unit 200, and controls these circuits. The control circuit 40 further includes a horizontal selection function, a horizontal origin distance calculation function, a horizontal origin depth map creation function, a vertical selection function, a vertical origin distance calculation function, a vertical origin depth map creation function, and a depth map synthesis function. And each function. Each of these functions will be described in detail later.

(Description of operation mode of stereo imaging device)
The stereo imaging device 1 includes an input device (not shown) such as a keypad. The user can transmit four types of instruction signals to the control circuit 40 using this input device. The four types of instruction signals are a normal shooting instruction signal, an arbitrary focus shooting instruction signal, a stereo shooting instruction signal, and a depth map creation instruction signal. The control circuit 40 that has received any kind of instruction signal controls the left imaging unit 100 and the right imaging unit 200 in accordance with the instruction signal.

  The control circuit 40 that has received the normal shooting instruction signal starts an operation in the normal shooting mode. In this mode, the control circuit 40 controls the right imaging unit 200 or the left imaging unit 100 and receives a digital signal (image signal) output from the imaging unit. Then, one image data is created by applying known image processing to the image signal. The control circuit 40 stores the image data thus created in a storage medium (not shown) such as a memory card or displays it on a display device (not shown) such as a liquid crystal monitor. In this mode, the image data obtained from the left imaging unit 100 is used to synthesize image data corresponding to an arbitrary focal plane by causing a personal computer or the like to execute a known process described in Patent Document 1 later. Is suitable.

  The control circuit 40 that has received the arbitrary focus shooting instruction signal starts an operation in the arbitrary focus shooting mode. In this mode, the control circuit 40 controls the left imaging unit 100 and receives a digital signal output from the imaging unit. For example, image data corresponding to an arbitrary aperture value and an arbitrary focal plane is created by applying a known process described in, for example, Patent Document 1 to the digital signal. The control circuit 40 stores the image data created in this way in a storage medium or displays it on a display device, as in the normal photographing mode.

  The control circuit 40 that has received the stereo shooting instruction signal starts operation in the stereo shooting mode. In this mode, the control circuit 40 controls both the left imaging unit 100 and the right imaging unit 200, receives digital signals output from the two imaging units, and creates two image data. These two image data are a pair of stereo image data having parallax. The control circuit 40 creates image data from the digital signal output from the right imaging unit 200 as in the normal shooting mode. On the other hand, the image data created from the digital signal output from the left imaging unit 100 is formed by extracting the outputs of one light receiving element for each light receiving element array 30 and arranging them. At that time, from each light receiving element array 30, the output of the light receiving element closer to the right imaging unit 200 than the central part of the light receiving element array 30 is extracted. By extracting in this way, stereo image data having a base length wider than the interval S between the left imaging unit 100 and the right imaging unit 200 is obtained. The stereo image data is data that allows the user to perform stereoscopic viewing by displaying the stereoscopic image data on, for example, a scan backlight type stereoscopic display device.

  The image data created based on the output of the left imaging unit 100 in this way has fewer pixels than the image data created based on the output of the right imaging unit 200. The control circuit 40 matches the resolution of these two image data by applying interpolation processing to the former or applying thinning processing to the latter.

  The control circuit 40 that has received the depth map creation instruction signal starts an operation in the depth map creation mode. In this mode, the control circuit 40 controls only the left imaging unit 100 and receives a digital signal output from the imaging unit. Then, after creating the image data as in the case of the arbitrary focus photographing mode, a depth map (described later) is created. The control circuit 40 stores the image data and the depth map in, for example, a storage medium or displays them on a display device.

(Description of depth map)
The control circuit 40 creates a depth map using the digital signal output from the left imaging unit 100. The depth map is distribution information of the distance to the subject. In the present embodiment, for each microlens ML that constitutes the microlens array 12, a defocus amount that is a value representing a distance to a subject on which an image is formed before and after the microlens ML is set to the microlens ML. Data arranged two-dimensionally according to the position is called a depth map. It is optically clear that the distance to the subject is determined by the defocus amount, the focal length of the taking lens 11, and the focus adjustment plane information. Here, the defocus amount is a numerical value representing the displacement between the imaging position of the subject image and a predetermined reference plane (focus detection plane).

  The depth map includes a horizontal selection function, a horizontal-derived distance calculation function, a horizontal-derived depth map creation function, a vertical selection function, a vertical-derived distance calculation function, a vertical-derived depth map creation function, a depth, which the control circuit 40 has. It is created by each function of the map composition function. Hereinafter, each of these functions will be described.

(Explanation of horizontal selection function)
FIG. 2 is a schematic diagram for explaining the horizontal selection function by the control circuit 40. In the following description, it is assumed that the microlens array 12 is composed of K microlenses ML horizontally and M microlenses ML vertically. The horizontal selection function is a function of selecting a plurality of horizontal lens rows arranged in the horizontal direction from the plurality of microlenses ML constituting the microlens array 12.

  In the horizontal selection function, the microlens array 12 is first partitioned into groups 50 for each horizontal row as shown in FIG. Thereby, M groups 50 are created. In FIG. 2, numbers starting from 1 are assigned in order from the left to the microlenses ML constituting the group 50 divided in this way for the sake of explanation.

  Next, as shown in FIG. 2B, the control circuit 40 extracts a predetermined number of microlenses ML from the left end of the group 50 from the right side of the shift amount determined by the minimum value of N described later. In the present embodiment, this predetermined number is five. That is, the control circuit 40 extracts five microlenses ML (from the third microlens ML to the seventh microlens ML) from the third from the left end of the group 50. In the horizontal selection function, the five microlenses ML extracted in this way are selected as one horizontal lens row 50a. The control circuit 40 then extracts five microlenses ML from the fourth microlens ML from the left to the seventh microlens ML from the left, as shown in FIG. Select as 50b. The control circuit 40 repeats this while shifting the target microlens ML to the right one by one, and selects a plurality of horizontal lens rows from the group 50 so that some microlenses ML overlap each other. To do.

  The control circuit 40 selects a plurality of vertical lens rows arranged in the vertical direction in the same manner for the vertical selection function. The only difference between the horizontal selection function and the vertical selection function is whether the arrangement direction of the microlenses ML and the light receiving elements is the horizontal direction or the vertical direction.

(Explanation of lateral distance calculation function)
The lateral distance calculation function calculates, for each of a plurality of horizontal lens rows selected by the horizontal selection function, a value representing the distance to the subject whose image is formed on the microlens ML at the center of the horizontal lens row. It is a function. As described above, since the defocus amount is used as a value representing the distance to the subject in the present embodiment, the defocus amount is calculated by the lateral origin distance calculation function.

  In the present embodiment, the defocus amount calculation procedure is determined so that the defocus amount becomes positive when the actual imaging position is closer to the subject side than the focus detection surface. Therefore, the direction in which the subject moves away from the stereo imaging device 1 is the direction in which the defocus amount increases.

  FIG. 3 is a diagram for explaining a signal sequence used for calculating the defocus amount. In FIG. 3, the numbers assigned to the microlenses ML are the numbers assigned sequentially from the leftmost microlens ML in the horizontal row of the microlens array 12. In the lateral distance calculation function, the control circuit 40 first creates two signal sequences corresponding to each lateral lens array 50. These two signal sequences are created using the outputs of specific light receiving elements among the light receiving elements included in the light receiving element array 30. The control circuit 40 of the present embodiment creates two signal sequences using outputs of the light receiving element a on the center left side and the light receiving element b on the right side of the center in the light receiving element array 30 shown in FIG.

  The control circuit 40 extracts the output of the specific light receiving element from the light receiving element array 30 corresponding to the microlens ML included in the horizontal lens array 50. For example, in FIG. 3, the output of the light receiving element a and the output of the light receiving element b are extracted from each of the five microlenses ML constituting the horizontal lens array 50. The control circuit 40 arranges the outputs of the light receiving elements a thus extracted in the order of the microlenses ML, and creates the first signal sequence 51. Similarly, the control circuit 40 arranges the outputs of the light receiving elements b and creates the second signal sequence 52. In the following description, the output of the light receiving element a corresponding to the nth microlens ML is represented by a (n), and the output of the light receiving element b is represented by b (n). That is, the first signal sequence 51 is composed of signal values a (3) to a (7), and the second signal sequence 52 is composed of signal values b (3) to b (7).

  As described above, the control circuit 40 determines the phase difference between the first signal train 51 created for each lateral lens train and the second signal train 52 created for the lateral lens train positioned around the lateral lens train. Is calculated to calculate the defocus amount. As a specific defocus calculation method, for example, a known method described in JP-A-60-37513 is used.

  FIG. 4 is a diagram illustrating how the phase difference is calculated. When obtaining the defocus amount for the lateral lens array 50a shown in FIG. 4, the control circuit 40 firstly has a first signal array 51 corresponding to the lateral lens array 50a and a second signal array 52 corresponding to the same lateral lens array 50a. The correlation value of is calculated. Next, the correlation value with the second signal sequence 53 corresponding to the right adjacent horizontal lens sequence, the correlation value with the second signal sequence 54 corresponding to the right adjacent horizontal lens sequence, etc. Calculation is performed for each of a plurality of horizontal lens rows positioned in a certain range on the left and right of the row 50a. The signal sequence corresponding to the horizontal lens column on the left side of the horizontal lens column 50a is similarly calculated. The shift in position between the horizontal lens array that is the target for calculating the defocus amount and the horizontal lens array that is the target for calculating the correlation value is referred to as a shift number. When calculating the defocus amount, the control circuit 40 calculates the correlation value C (N) defined by the following equation (1) while changing the shift number N within a predetermined range (for example, −5 to +5). That is, correlation values C (−5), C (−4), C (−3),..., C (3), C (4), and C (5) are calculated. However, FIG. 4 shows an example in which the shift number N is changed in the range of −2 to +2 for simplicity. Here, pL is the number of the microlens ML corresponding to the leftmost signal value of the first signal sequence 51, and qL is the number of the microlens ML corresponding to the rightmost signal value of the first signal sequence 51.

  Of the correlation values C (N) calculated in this way, the minimum value is referred to as C0. Further, N where C (N) = C0, that is, the number of shifts at which the correlation value is minimized is referred to as N0. The control circuit 40 calculates a more precise shift number Na from the shift number N0 by the following equation (2). Here, Cr is C (N0-1), and Cf is C (N0 + 1).

  The control circuit 40 adds a predetermined correction amount const according to the position of the focus detection surface to the shift number Na obtained by Expression (2), and calculates an image shift amount Δn on the focus detection surface. The final defocus amount Df can be calculated by multiplying the image shift amount Δn by a constant Kf that depends on the detected opening angle. That is, the defocus amount Df = Δn × Kf.

  Here, equation (2) does not always calculate a reliable shift number Na. That is, the calculation result according to the equation (2) may be a shift number Na with low reliability. Therefore, after calculating the shift number Na, the control circuit 40 calculates a reliability parameter R representing the reliability of the shift number Na by the following equation (3). The reliability parameter R decreases as the reliability of the shift number Na increases. When the reliability parameter R is larger than the predetermined value, the control circuit 40 determines that it is impossible to calculate the defocus amount from the horizontal lens row.

  The defocus amount calculated above is the defocus amount with respect to the position of the microlens ML located at the center of the horizontal lens row that is the defocus amount calculation target. The control circuit 40 calculates the defocus amount by the procedure described above for all the horizontal lens rows selected by the horizontal selection function. As a result, the defocus amount is calculated for the microlenses ML except for the microlenses ML near the left and right ends of all the microlenses ML included in the microlens array 12.

  FIG. 5 is a diagram illustrating the microlens ML in which the defocus amount is calculated by the horizontal-derived distance calculation function and the vertical-derived distance calculation function. In the range indicated by hatching in FIG. 5, the defocus amount is not calculated. Accordingly, in FIG. 5A, the defocus amount is calculated for the range 56 excluding the left and right ends of the microlens ML.

  Note that the control circuit 40 similarly applies to the longitudinal distance calculation function, and for each of the plurality of vertical lens rows selected by the vertical selection function, the subject in which an image is formed on the microlens ML at the center of the vertical lens row. The distance to is calculated. A range 57 illustrated in FIG. 5B is a range in which the defocus amount is calculated by the longitudinal-derived distance calculation function. The only difference between the horizontal origin distance calculation function and the vertical origin distance calculation function is whether the arrangement direction of the microlens ML and the light receiving elements used is the horizontal direction or the vertical direction.

(Description of horizontal origin depth map creation function)
The horizontal-derived depth map creation function determines a defocus amount, which is a value representing the distance to the subject corresponding to each microlens calculated by the horizontal-derived distance calculation function, according to the position of the microlens ML corresponding to the distance. This is a function to create a horizontal-derived depth map arranged in two dimensions. The control circuit 40 creates a laterally derived depth map by arranging the defocus amounts in the respective microlenses ML in accordance with the positions on the microlens array 12. The control circuit 40 similarly creates a vertical-derived depth map by arranging a plurality of defocus amounts calculated by the vertical-derived distance calculating function in a two-dimensional manner for the vertical-derived depth map creating function.

  If it is determined that the defocus amount cannot be calculated, the control circuit 40 interpolates the defocus amount from the defocus amount of the surrounding (up / down / left / right) microlenses ML. A horizontal-derived depth map and a vertical-derived depth map are created.

(Description of depth map composition function)
The depth map synthesizing function is a function for executing a synthesizing process for synthesizing the horizontal-derived depth map and the vertical-derived depth map and outputting a final depth map. Specifically, for each microlens ML in which the defocus amount is included in both the horizontal-derived depth map and the vertical-derived depth map, the control circuit 40 determines the defocus amount on the horizontal-derived depth map and the vertical-derived depth. The defocus amount on the map is compared, and the larger defocus amount is adopted as the defocus amount in the final depth map.

  FIG. 6 is a diagram showing the range of the depth map that is finally output. As shown in FIG. 6, the control circuit 40 outputs a depth map in which the defocus amounts for the microlenses ML in the range 58 excluding the upper, lower, left and right ends of the microlens array 12 are arranged. The combination as described above is performed because the defocus amount at the boundary between the short-distance subject and the long-distance subject is easily affected by the short-distance subject. This will be described below.

  FIG. 7 is a diagram for explaining the defocus amount in the laterally derived depth map. In the photographed image 60 shown in FIG. 7A, the boundary lines of the respective light receiving element arrays 30 are superimposed. The shaded area in the photographed image 60 represents that a distant subject has been imaged. Here, in the partial row 61 including the light receiving element arrays 30 at the lowermost end, the light receiving element arrays PP (6) and PP (12) are located at the boundary between the near subject and the far subject.

  Here, since PP (5) corresponds to a distant subject, it is desirable that the defocus amount at this point be large. However, as shown in FIG. 7B, the defocus amount is calculated including the outputs of the light receiving elements corresponding to the left and right microlenses ML. That is, in calculating the defocus amount of PP (5), a (3) to a (7) are used in this example in Equation (1). Here, a (n) is an output extracted from PP (n). Accordingly, the signal value a (6) or the like affects the defocus amount. As a result, the actually calculated defocus amount is smaller than the ideal value. A graph of the ideal defocus amount and the actually calculated defocus amount is shown in FIG. The actually calculated defocus amount 63 (solid line graph in FIG. 7C) is an ideal defocus amount 62 (dotted line in FIG. 7C) near the boundary between a distant subject and a near subject. There is a tendency to become smaller than the graph of

  The reason why the control circuit 40 creates the final depth map by combining the horizontal-derived depth map and the vertical-derived depth map is to eliminate such a problem in the vicinity of the boundary between the distant subject and the nearby subject. It is. That is, even if a far subject and a nearby subject are adjacent in the horizontal direction, the problem at the boundary as described above can be solved if the subject is not adjacent in the vertical direction. . The same applies to the case of adjacent in the vertical direction and not adjacent in the horizontal direction.

According to the stereo imaging device according to the first embodiment described above, the following operational effects can be obtained.
(1) The control circuit 40 selects a plurality of horizontal lens rows arranged in the horizontal direction from the plurality of microlenses ML included in the microlens array 12, and selects the horizontal lens row for each of the selected horizontal lens rows. The outputs of the light receiving elements corresponding to the respective microlenses ML are calculated, and values representing the distances to the subject where the image is formed before and after the microlenses ML at the center of the horizontal lens row are calculated. A horizontal-derived depth map is created by arranging the values of in two dimensions. Similarly, after creating a vertical-derived depth map in the vertical direction, the horizontal-derived depth map and the vertical-derived depth map are combined to output a final depth map. Since it did in this way, the value showing the exact distance to a to-be-photographed object can be calculated also in the vicinity of the location where the distance of a to-be-photographed object changes.

(Second Embodiment)
The stereo imaging device according to the second embodiment has the same configuration as the stereo imaging device 1 according to the first embodiment shown in FIG. In addition, the control circuit 40 has a function of creating a depth map corresponding to the captured image created by the right imaging unit 200 from the depth map created by the left imaging unit 100. Hereinafter, this function will be described.

  Upon receiving the depth map creation instruction signal, the control circuit 40 according to the present embodiment creates a captured image and a depth map using the left imaging unit 100 and simultaneously creates a captured image using the right imaging unit 200. At this time, the photographing lens 11 and the photographing lens 21 are kept in focus on the same surface of the object scene. The control circuit 40 detects information on the focus adjustment surface when the image is taken by an encoder (not shown) attached to the taking lens 11 and the taking lens 21. Here, the information on the focus adjustment surface is a subject distance to which the photographing lens 11 and the photographing lens 21 are focused, and is information such as 1 m or 3 m, for example.

  Thereafter, the control circuit 40 creates a captured image and a depth map using the left imaging unit 100, as in the case of the first embodiment. After that, the control circuit 40 calculates the distance to the subject for each point constituting the depth map from the created depth map and the information on the focus adjustment surface. For example, when the focal length of the photographic lens 11 is 50 mm and the subject distance is 1 m, the defocus amount is 0 for the subject located 1 m ahead of the photographic lens 11. For example, when the defocus amount is 0.5 mm, the subject distance is calculated as 1.25 m. Similarly, in the case of other defocus amounts, the distance to the subject can be calculated according to the value.

  In parallel with this, the control circuit 40 also creates a captured image using the right imaging unit 200. Hereinafter, a captured image created using the left imaging unit 100 is referred to as a left captured image, and a captured image created using the right imaging unit 200 is referred to as a right captured image.

  Thereafter, the control circuit 40 calculates how much each point of the left photographic image is moved in the lateral direction in the right photographic image based on the distance to the subject for each point calculated by the above processing. From the positional relationship between the left imaging unit 100 and the right imaging unit, it is obvious that each point of the left captured image moves only to the left and right in the right captured image and does not move up and down. Further, the amount of movement at this time becomes smaller as the subject pixel located farther from the stereo imaging device 1 and becomes larger as the subject pixel closer to the stereo imaging device 1.

  Finally, the control circuit 40 moves each point on the depth map in the horizontal direction according to the movement amount in the horizontal direction obtained by the above processing. That is, the defocus amount corresponding to the specific point in the left-side captured image is set as the defocus amount corresponding to the left point. At this time, when a plurality of defocus amounts correspond to one point, priority is given to a smaller defocus amount. In addition, the defocus amount estimated from the defocus amount at the adjacent point on the right side is made to correspond to the point that no defocus amount corresponds to. This estimated value is estimated, for example, by extrapolating from the defocus amount distribution on the right side of the point to be estimated.

  At this time, “the point at which any defocus amount no longer corresponds” is considered to be a portion that is a shadow of the subject on the front side in the left imaging unit 100. That is, this is a portion where the subject is behind the subject on the front side.

  As described above, the control circuit 40 creates a depth map corresponding to the right imaging unit 200 from the depth map created based on the left imaging unit 100. The new depth map created here is smaller than the resolution of the right-side captured image. Therefore, when the created depth map is actually used, it is necessary to enlarge the depth map according to the right-side captured image or reduce the right-side captured image according to the depth map. A known image enlargement technique may be used to enlarge the depth map.

According to the stereo imaging device according to the second embodiment described above, the following operational effects can be obtained.
(1) The control circuit 40 performs left-side imaging based on the amount of deviation caused by parallax between the left-side captured image created using the left-side imaging unit 100 and the right-side captured image created using the right-side imaging unit 200. A depth map corresponding to the unit 100 is corrected, and a depth map corresponding to the right-side captured image is created. Since it did in this way, it becomes possible to use the more exact depth map corresponding to the right side picked-up image whose pixel number is higher than the left side picked-up image, and the convenience improves.

  The following modifications are also within the scope of the present invention, and one or a plurality of modifications can be combined with the above-described embodiment.

(Modification 1)
As shown in FIG. 1, in the imaging device 13 according to the first embodiment, each light receiving element array 30 exists independently. That is, each light receiving element array 30 is clearly divided. Instead of doing so, the light receiving elements may be continuously present on the image pickup surface of the image pickup element 13 like the image pickup element 23. That is, the image sensor 13 may have the same configuration as the image sensor 23.

(Modification 2)
In the present invention, if at least two light receiving elements are arranged along each of the two directions for each microlens ML, a depth map creation process can be performed. Therefore, the present invention can be applied to an apparatus having a configuration in which the light receiving element array 30 includes three light receiving elements. In the first embodiment, each light receiving element array 30 is composed of five columns and five rows of light receiving elements. However, the number of light receiving elements constituting the light receiving element array 30 is three or more. There may be.

(Modification 3)
The first signal sequence and the second signal sequence used for calculating the defocus amount may be created from outputs of light receiving elements other than the light receiving elements a and b shown in FIG. Alternatively, a signal sequence may be created from values obtained by calculating outputs of a plurality of light receiving elements for each light receiving element array. For example, an average value of outputs of a plurality of light receiving elements may be calculated, or a weighted average may be calculated with a predetermined weight.

(Modification 4)
The correlation value C (N) between the first signal sequence and the second signal sequence may be calculated by the following equation (4) instead of the equation (1). Here, the function Floor (x) is a so-called floor function that returns a maximum integer equal to or less than x.

(Modification 5)
In the first embodiment, the horizontal lens array and the vertical lens array are always composed of a certain number of microlenses ML. Alternatively, a part of the lens rows may be composed of a smaller number of microlenses ML. Specifically, the control circuit 40 first extracts the outputs of the light receiving elements one by one from the corresponding light receiving element array 30 for each microlens ML. Then, intermediate image data in which the extracted outputs are arranged two-dimensionally is created. By applying a horizontal direction differential filter to the intermediate image data, the change rate of the light amount of the subject image in the horizontal direction is calculated. Similarly, when the vertical direction differential filter is applied, the change rate of the light amount of the subject image in the vertical direction is calculated. In the horizontal selection function and the vertical selection function, the control circuit 40 refers to the rate of change of the light amount of the subject image in the corresponding direction, and is selected at a location where the rate of change is greater than a predetermined threshold and surrounding locations. The lens array is selected so that the lens array to be formed is composed of fewer microlenses ML than usual. In this way, even when the boundary between a distant subject and a nearby subject exists in an oblique direction, the adverse effects of nearby nearby subjects as described in FIG. It becomes possible to calculate the defocus amount.

(Modification 6)
In the depth map composition function, the control circuit 40 may refer to elements other than the magnitude of the defocus amount. For example, a synthesis process may be executed in consideration of the strength of correlation between the horizontal-derived depth map and the vertical-derived depth map.

(Modification 7)
Two imaging units may be arranged in the vertical direction instead of the horizontal direction. Further, the distance between the two imaging units is not limited to the numerical values exemplified in the above embodiment.

(Modification 8)
The defocus amount reliability parameter R may be calculated by an expression other than the above expression (3). For example, the denominator on the right side of the above equation (3) may be used as the reliability parameter R.

  As long as the characteristics of the present invention are not impaired, the present invention is not limited to the above-described embodiments, and other forms conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention. .

DESCRIPTION OF SYMBOLS 1 Stereo imaging device 11, 21 Shooting lens 12 Micro lens array 13, 23 Imaging device 14, 24 Imaging control circuit 15, 25 Video circuit 30 Light receiving element arrangement 40 Control circuit 100 Left imaging unit 200 Right imaging unit ML Micro lens

Claims (7)

An imaging optical system that forms an image of a subject on a predetermined imaging surface;
A plurality of positive lenses arranged two-dimensionally in the vicinity of the imaging plane;
A plurality of light receiving elements disposed at the back of the positive lens corresponding to each of the plurality of positive lenses, wherein at least two of the at least three light receiving elements are in a first direction. A plurality of light receiving elements arranged in a second direction different from the first direction, and at least two of the at least three light receiving elements,
First selecting means for selecting a plurality of first positive lens rows arranged in the first direction from the plurality of positive lenses;
For each of the plurality of first positive lens rows selected by the first selection means, at least two light receptions arranged in the first direction corresponding to each of the positive lenses constituting the first positive lens row. First distance calculation means for calculating an output of the element and calculating a value representing a distance to the subject whose image is formed on the positive lens at the center of the first positive lens row;
A first depth that creates a first depth map that is data in which values representing a plurality of distances calculated by the first distance calculating means are arranged in a two-dimensional manner in accordance with the positions of the positive lenses corresponding to the values. Map creation means;
Second selecting means for selecting a plurality of second positive lens rows arranged in the second direction from the plurality of positive lenses;
For each of the plurality of second positive lens rows selected by the second selection means, at least two light receptions arranged in the second direction corresponding to each of the positive lenses constituting the second positive lens row. Second distance calculation means for calculating an output of the element and calculating a value representing a distance to the subject whose image is formed on the positive lens in the center of the second positive lens row;
A second depth that creates a second depth map that is data in which values representing a plurality of distances calculated by the second distance calculating means are arranged two-dimensionally in accordance with the positions of the positive lenses corresponding to the values. Map creation means;
Depth map output means for combining the first depth map and the second depth map and executing a combining process for outputting a third depth map;
At least two light receiving elements arranged in the first direction and at least two light receiving elements arranged in the second direction corresponding to each of the plurality of positive lenses, via the positive lens, A depth map output device that receives light beams from different regions of an exit pupil of the imaging optical system. The depth map output device according to claim 1,
The composition process executed by the depth map output means is any when the distance represented by the value on the first depth map and the distance represented by the value on the second depth map are the same for the same subject. A depth map output device characterized in that it is a process of outputting one distance and outputting a value representing the larger distance if they are not equal. In the depth map output device according to claim 1 or 2,
The first distance calculating means includes, for each of the plurality of first positive lens rows, at least two light receiving elements arranged in the first direction corresponding to each of the positive lenses constituting the first positive lens row. Two signal sequences are generated from the output of the first, and a value representing a distance to the subject is calculated by calculating a phase difference between the two signal sequences of the first positive lens columns,
The second distance calculating means includes, for each of the plurality of second positive lens rows, at least two light receiving elements arranged in the second direction corresponding to each of the positive lenses constituting the second positive lens row. A depth map for calculating a value representing a distance to the subject by generating two signal trains from the output of the first lens and calculating a phase difference between the two signal trains of the second positive lens trains. Output device. In the depth map output device according to any one of claims 1 to 3,
The first selection unit includes a plurality of the first positive lens rows that overlap with the first positive lens rows from the plurality of positive lenses arranged in a row along the first direction. Choose to meet each other,
The second selection unit includes a plurality of the positive lenses arranged in a line along the second direction, and the plurality of second positive lens rows are overlapped with each other by the second positive lens rows. A depth map output device, wherein selection is performed so as to match each other. In the depth map output device according to any one of claims 1 to 4,
First light amount distribution calculating means for calculating a light amount distribution of the subject image in the first direction based on an output of at least one light receiving element among the light receiving elements corresponding to each of the plurality of positive lenses;
Second light quantity distribution calculating means for calculating the light quantity distribution of the subject image in the second direction based on the output of at least one light receiving element among the light receiving elements corresponding to each of the plurality of positive lenses. ,
The first selection unit selects the plurality of first positive lens rows based on a light amount distribution in the first direction,
The depth map output device, wherein the second selection unit selects the plurality of second positive lens rows based on a light amount distribution in the second direction. In the depth map output device according to claim 5,
The first selection unit includes a smaller number of positive lenses than the first positive lens array selected at a place where the change rate of the light quantity is larger than a predetermined threshold in the light quantity distribution in the first direction. Select the first positive lens array to be
The second selection unit includes a smaller number of positive lenses than the second positive lens array selected at a place where the change rate of the light quantity is greater than a predetermined threshold in the light quantity distribution in the second direction. A depth map output device, wherein the second positive lens array is selected. In the depth map output device according to any one of claims 1 to 6,
An imaging unit that captures a subject image and outputs a first captured image;
A captured image output means for outputting a second captured image having a parallax with respect to the first captured image based on outputs of the plurality of light receiving elements;
The third depth map is corrected based on a shift amount between the first captured image and the second captured image due to the parallax, and a fourth depth map corresponding to the first captured image is obtained. A depth map output device further comprising a depth map correction means for creating.