JP5392198B2 - Ranging device and imaging device - Google Patents

Description

  The present invention relates to a distance measuring device and an imaging device.

  In recent years, it has become practical to use a distance measuring device such as a stereo camera in an embedded environment such as a digital camera or an in-vehicle camera that requires real-time processing. For example, there is a technique of using a stereo camera for high-speed autofocus in a digital camera.

  In conventional autofocus of digital cameras, the focus is changed little by little, and images are acquired each time. The contrast of these images is compared, and the focus position with the highest contrast is determined as the optimum focus position. It was.

  However, the above-described method has a problem that it takes a long processing time because many images must be taken by changing the focus position. On the other hand, there is a method in which a stereo camera is used as a distance measuring device for autofocusing, and a distance to a subject is acquired by one imaging to calculate a focus position according to the distance. When this method is used, the optimum focus position can be calculated by a single imaging process, so that a high-speed autofocus process is possible.

  For example, in Patent Document 1 (Japanese Patent Laid-Open No. 7-225127), vertical edges that are easy to obtain ranging accuracy as a stereo camera (hereinafter simply referred to as vertical edges) are detected, and the parallax data of the vertical edge portion is detected. A method of calculating is disclosed.

  Further, in Patent Document 2 (Japanese Patent Application Laid-Open No. 2009-14444), using the edge information detected by the edge detection unit, the processing region setting unit extracts a candidate region where an object may exist, and the extracted candidate A method is disclosed in which a corresponding area search unit calculates a parallax with respect to an area by a POC method or the like.

  There is also a technique for dividing an image when calculating parallax. For example, in Patent Document 3 (Japanese Patent Laid-Open No. 11-14346), a window of a predetermined size is set in a predetermined area excluding an area such as the sky for one image (reference image) in a stereo camera. An in-vehicle stereo camera that calculates parallax in units is disclosed.

  Patent Document 4 (Japanese Patent Laid-Open No. 2006-32295) first identifies an area corresponding to a predetermined object and the type of the object for one image (reference image) in a stereo camera, A stereo camera that sets a calculation range for performing a parallax calculation with reference to an identification result is disclosed.

  However, when a distance measuring device using a stereo camera as in the prior art is applied to a digital camera, the main image sensor side is controlled by the result of distance measurement. Therefore, it becomes a big problem to match the angle of view of the picked-up image for distance measurement processing with the angle of view of the main image pickup device as a digital camera. In general, a digital camera has a large field angle fluctuation range of 6 ° to 60 °. Such a variation in the angle of view (zoom) is realized by a mechanical mechanism that moves the focal length of the lens by operating the housing (optical zoom) or by electronic processing such as image processing ( Electronic zoom).

  However, if an optical zoom mechanism is also provided on the autofocus distance measuring device side, this leads to a large cost increase and a space in the housing is also required. In the case of the electronic zoom mechanism, there are problems that processing time is required and that an increase in cost is realized if it is realized at high speed.

  In view of the above, it is desirable that the autofocus range finder can cover the entire field angle fluctuation range on the main image sensor side with a single field angle and focal length. However, in such a configuration, as shown in FIG. 1, the corresponding image areas in the corresponding autofocus captured image are greatly different with respect to various angles of view of the main captured image.

  FIG. 1 is a diagram illustrating an example in which the corresponding image areas in the autofocus captured image are different. FIG. 1A shows an example of each image when autofocus imaging angle of view <main imaging angle of view. As shown in FIG. 1A, when the main imaging angle of view is large, that is, at the time of close-up shooting, the main captured image is larger than the corresponding image that is the target for parallax calculation of the autofocus captured image.

  FIG. 1B shows an example of each image when autofocus imaging angle of view = main imaging angle of view. As shown in FIG. 1B, the autofocus captured image and the main captured image have the same image size.

  FIG. 1C shows an example of each image when autofocus imaging angle of view> main imaging angle of view. As shown in FIG. 1C, when the main imaging angle of view is small, that is, at the time of telephoto shooting, the corresponding image area for the parallax calculation target of the autofocus captured image is small.

  In consideration of using a stereo camera for autofocus, it is desirable to be able to calculate parallax for the entire image area of any size. This is because in the case of a digital camera, various subjects are assumed, and the size of the subject in the image varies. However, an invalid value for parallax is unnecessary. For this reason, it is required that an effective parallax map of the entire area of the image area can be detected in the image area corresponding to the main imaging system.

  On the other hand, since autofocus requires real-time performance, there is also a demand for speeding up parallax calculation. Patent Documents 3 and 4 described above are not techniques used for autofocusing, but are techniques for increasing the speed by dividing an image on which parallax calculation is performed according to the presence or absence of an object.

  However, in Patent Documents 3 and 4, since identification and recognition processing are required before the parallax calculation is performed, the corresponding overhead is required, and the accuracy of the identification processing is a problem. Of course, if it is attempted to increase the accuracy of the identification process or to deal with various scenes, it takes a long processing time. Furthermore, it is necessary to assume an object to be identified in performing the identification process. Further, Patent Document 3 discloses a method for processing an imaging process, an identification process, a range setting process, a distance calculation process, and the like in a pipeline, but a process related to parallax calculation (part of the distance calculation process) itself. It is not a technology to speed up.

  Further, for example, as described above, when a stereo camera is applied to autofocus of a digital camera, various objects and scenes viewed by the camera are assumed. In such a situation, rather, information on how the parallax is distributed in a certain region in the image is often required.

  As shown in FIG. 1, if it is known how the parallax is distributed within the image size of the stereo camera corresponding to the main captured image, autofocus is possible. Recognition and identification are often permitted even if processing time is required. Therefore, it is more efficient to separately perform recognition using the calculated disparity distribution and luminance information.

  As described above, in the case of applying a stereo camera to autofocus, there is a problem that it takes time for recognition processing and edge extraction processing for performing parallax calculation at high speed.

  Therefore, the present invention has been made in view of the above-described problems, and an object thereof is to provide a distance measuring device and an imaging device that can increase the processing time required for parallax calculation.

  A distance measuring apparatus according to an aspect of the present invention is a distance measuring apparatus that measures a distance to an object, and includes two imaging units and a plurality of small images that are captured by the two imaging units. A dividing unit for dividing, a texture detection process for selecting a parallax calculation pixel for a small image captured by one of the two imaging units, a selected parallax calculation pixel, and another A parallax calculation unit that performs at least a parallax calculation process that calculates parallax data using a pixel corresponding to the selected parallax calculation pixel among images captured by the imaging unit, and the texture detection process by the parallax calculation unit And control means for controlling the parallax calculation processing to be processed in parallel for each of the small images.

  An imaging device according to another aspect of the present invention includes two imaging units, a dividing unit that divides each image captured by the two imaging units into a plurality of small images, and the two imaging units. A texture detection process for selecting a parallax calculation pixel, a selected parallax calculation pixel, and an image captured by another imaging unit are selected from the small image of the image captured by one imaging unit. A parallax calculation unit that performs at least a parallax calculation process that calculates parallax data using pixels corresponding to the parallax calculation pixel, the texture detection process by the parallax calculation unit, and the parallax calculation process in parallel for each small image. Parallel control means for controlling to process, distance calculation means for calculating a distance based on the parallax data calculated by the parallax calculation means, and the distance calculation means Ri includes a focus control means for controlling the autofocus based on the calculated distance, a.

  According to the present invention, the processing time required for parallax calculation can be increased.

The figure which shows the example from which the corresponding image area | region in an autofocus captured image differs. The principle figure of a distance measuring device. The block diagram which shows an example of a structure of the ranging device in this invention. The figure which shows the relationship between a target image and a small image. The block diagram which shows an example of the function of a parallax calculation part. The block diagram which shows an example of the function of a texture detection part. The figure for demonstrating a texture characteristic value (the 1). The figure for demonstrating a texture characteristic value (the 2). The figure for demonstrating the reliability of correlation. The figure which shows an example of the processing timing flow of pipeline parallel processing. The figure for demonstrating a different buffer size for every process. 5 is a flowchart illustrating an example of parallax data calculation processing according to the first embodiment. 1 is a block diagram illustrating an example of a configuration of an imaging device.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. First, a method for detecting a distance to an object included in an image in the distance measuring device will be described. The distance measuring device is, for example, a stereo camera. The distance measuring device is a device that detects a distance to an object included in a captured image based on two captured images obtained by two imaging units.

  FIG. 2 is a principle diagram of the distance measuring device. As shown in FIG. 2, the distance measuring apparatus includes an imaging unit 1 and an imaging unit 2. The imaging units 1 and 2 are, for example, cameras. In FIG. 2, a pinhole camera will be described as an example. The baseline length (distance between the imaging unit 1 and the imaging unit 2) is B, the focal length of the two imaging units is f, and the distance from the imaging units 1 and 2 to the measurement object 3 is Z.

  The optical axes of the two imaging units are parallel to each other and perpendicular to the base line. The measurement object 3 appears on the imaging surfaces of the two imaging units at a position shifted by p. This p is the distance between corresponding points in the left and right cameras and is called parallax. The magnitude of parallax is often expressed in units of pixels.

Using these values of Z, B, f, and p, the distance Z from the imaging units 1 and 2 to the measurement object 3 can be obtained from the similarity relationship of the triangles using Equation (1).
Z = B × f / p (1)
At this time, the parallax p is calculated by calculating the correlation value by shifting one pixel at a time for each pixel for which the parallax is calculated, and comparing it. For this reason, it is known that the processing takes an enormous amount of time. For example, if the length (search width) for searching by shifting the pixel for searching the parallax p for each pixel is 64 pixels, the processing time for processing a certain image is to search for the parallax for one pixel. Even if it takes one clock cycle, it takes 64 times as much processing time to process one image.

In general, it is impossible to realize processing in one clock cycle in order to calculate a correlation value for one pixel, and this increases exponentially with the block size to be correlated. For this reason, reducing the parallax calculation time has become a major issue. An approximate expression for calculating the parallax calculation time is expressed by Expression (2).
Parallax calculation time for one image = number of parallax calculation pixels × (correlation calculation time for one pixel × search width) Expression (2)
For the problem of reducing parallax calculation time,
It is conceivable to reduce the number of parallax calculation pixels, reduce the correlation calculation time of one pixel, and reduce the search width.

  The search width is determined by the target ranging range. In order to shorten the correlation calculation time for one pixel, a large amount of hardware resources are required, which increases the cost. When the algorithm for the correlation calculation time is shortened, the accuracy tends to deteriorate.

  Therefore, an object of the present invention is to speed up the processing time for parallax calculation while efficiently reducing the number of parallax calculation pixels (selecting parallax calculation pixels efficiently). In the present invention, the processing included in the parallax calculation is pipelined to increase the processing time required for the parallax calculation. Hereinafter, the process for selecting the parallax calculation pixel is referred to as a texture detection process, and the algorithm for the process is referred to as a texture detection algorithm.

[Example 1]
<Configuration>
FIG. 3 is a block diagram showing an example of the configuration of the distance measuring apparatus according to the present invention. A distance measuring device 13 illustrated in FIG. 3 includes an imaging unit 1, an imaging unit 2, an image capturing unit 4, a memory 5, a stereo processing unit 6, and a distance calculation unit 12.

  The imaging units 1 and 2 are arranged in advance such that the base line interval is spaced by B and the optical axes are parallel to each other. The imaging units 1 and 2 are, for example, cameras.

  The image capturing unit 4 acquires the images captured by the imaging units 1 and 2 and outputs them to the memory 5. The memory 5 stores the image input from the image capturing unit 4. The memory 5 is, for example, a RAM (Random Access Memory).

  The stereo processing unit 6 calculates the parallax data based on the reference image captured by one of the two imaging units and the comparison image captured by the other imaging unit. The stereo processing unit 6 includes a dividing unit 7, a parallax calculating unit 10, and a parallel control unit 11.

  The dividing unit 7 acquires a target image (hereinafter, also simply referred to as a target image) that is a target of parallax calculation among the reference images by dividing it into small images of a predetermined size and stores them in the buffer 8. In addition, the dividing unit 7 acquires an image corresponding to the target image among the comparative images by dividing the image into a plurality of small images of a predetermined size, and stores them in the buffer 9.

  FIG. 4 is a diagram illustrating the relationship between the target image and the small image. As illustrated in FIG. 4, the target image 15 is an image corresponding to the main captured image among the autofocus captured images. The small image 16 is an image obtained by dividing the target image 15 into a plurality of predetermined sizes. The predetermined size is, for example, 64 × 42 or 64 × 64.

  The dividing unit 7 reads out the small images 16 of the target image 15 in the reference image one by one from the memory 5 and stores them in the buffer 8. In addition, the dividing unit 7 reads out small images of images corresponding to the target image 15 in the comparison image one by one from the memory 5 and stores them in the buffer 9.

  Returning to FIG. 3, the parallax calculation unit 10 calculates parallax data based on the reference image and the comparison image. Details of the parallax calculation unit 10 will be described with reference to FIG.

  FIG. 5 is a block diagram illustrating an example of the function of the parallax calculation unit 10. The parallax calculation unit 10 illustrated in FIG. 5 includes geometric correction units 21 and 22, buffers 23 and 24, texture detection unit 25, buffer 26, parallax data calculation unit 27, buffer 28, and reliability calculation unit 29.

  The geometric correction unit 21 corrects image distortion and the like for the small image of the reference image acquired from the buffer 8. Even when the entire image is nonlinear, such as barrel distortion, in a small image, a sufficient correction result is often obtained by linear correction. Therefore, the geometric correction unit 21 can save memory resources such as for a lookup table. The geometric correction unit 21 outputs the corrected small image to the buffer 23.

  The geometric correction unit 22 corrects image distortion or the like for the small image of the comparison image acquired from the buffer 9. The basic function of the geometric correction unit 22 is the same as that of the geometric correction unit 21. The geometric correction unit 22 outputs the corrected small image to the buffer 24.

  The buffer 23 stores the corrected small image until it is read by the texture detection unit 25.

  The buffer 24 stores the corrected small image until it is read out by the parallax data calculation unit 27.

  The texture detection unit 25 acquires a small image stored in the buffer 23. The texture detection unit 25 efficiently selects a parallax calculation pixel based on a target image used for parallax calculation of a reference image. Details of the texture detection unit 25 will be described with reference to FIG.

  FIG. 6 is a block diagram illustrating an example of the function of the texture detection unit 25. The texture detection unit 25 illustrated in FIG. 6 includes a determination unit 31, a switching unit 32, a first detection processing unit 33, a second detection processing unit 34, and a selection unit 35.

  The determination unit 31 determines a process (texture detection process) for selecting a pixel for calculating the parallax based on the target image for calculating the parallax. Hereinafter, the pixel for calculating the parallax is referred to as a parallax calculation pixel. The determination unit 31 may directly acquire the size of the target image from the memory 5. Further, when the size of the target image is included in the first small image, the determination unit 31 may acquire the size of the target image from the small image.

  The determination unit 31 determines the first detection processing unit 33 if the target image for calculating the parallax is larger than a predetermined size (size threshold), and the second detection processing unit if the target image is equal to or smaller than the predetermined size. 34. The size threshold is 200 × 100, for example. Further, the determination unit 31 may store a plurality of size threshold values and select one detection processing unit from among three or more detection processing units.

  The determination unit 31 may determine a texture detection algorithm based on the number of parallax calculation pixels. For example, the determination unit 31 determines the number of parallax calculation pixels based on the currently requested processing time (for example, several milliseconds) based on the past processing time taken for parallax calculation, and the number of parallax calculation pixels is a predetermined number (pixel threshold). If it is larger than that, the second detection processing unit 34 may be determined, and if it is equal to or less than a predetermined number, the first detection processing unit 33 may be determined. The required processing time depends on the processing capability of the application and device. For example, according to high-speed autofocus, a processing time of 0.1 to 0.7 seconds is required, and a predetermined number of times is required as a processing time for parallax calculation.

  This is because if the number of parallax calculation pixels is equal to or less than the predetermined number, more pixels are thinned out, so the first detection processing unit 33 is applied. This is based on the idea that the second detection processing unit 34 is applied because there is no need for thinning.

  If the determination unit 31 determines that it is necessary to thin out many pixels from the target image based on the size of the target image and the determined number of parallax calculation pixels, the determination unit 31 determines the first detection processing unit 33. If it is determined that the number of pixels to be thinned out from the target image is small, the second detection processing unit 34 may determine.

  The switching unit 32 is connected to one of the detection processing units according to the determination content of the determination unit 31. The switching unit 32 is described as hardware for convenience of explanation, but may be implemented by software that performs switching by conditional branching of a program.

  The first detection processing unit 33 is a texture characteristic detection process applied when the target image for calculating the parallax is large. The first detection processing unit 33 calculates a texture characteristic value by applying an advanced texture detection algorithm to the small image acquired from the buffer 8.

  FIG. 7 is a diagram for explaining a texture characteristic value (No. 1). In the example illustrated in FIG. 7, when the block size for performing the parallax calculation is 9 × 9, the block size for detecting the texture is also 9 × 9. Note that the block size for detecting the texture shown in FIG. 7 is 9 × 9 for convenience of explanation, but in practice it may be reduced to match the size of the small image.

  According to the example illustrated in FIG. 7, the first detection processing unit 33 calculates the texture characteristic value using Expression (3).

ただし、（ｉ，ｊ）：画素位置
ｘ（ｉ，ｊ）：（ｉ，ｊ）の輝度値
ａ：水平５ＴＡＰ係数、ａ（０）が中心の係数とする
ｂ：５×３ブロックフィルタ係数、ｂ（０，０）が中心の係数とする
ω１：水平５ＴＡＰ計算用重み係数
ω２：５×３ブロックフィルタ用重み係数
図７に示すテクスチャ特性値のテクスチャ検出アルゴリズムは、視差計算を行うブロック（９×９の画素ブロック）内で、水平エッジがどれくらい含まれているかという情報と、視差計算を行う中心の画素のまわりに縦ラインがどれくらい続くか（水平エッジがどれくらい縦に続いているか）という情報によって、テクスチャレベルを評価するアルゴリズムである。

However, (i, j): Pixel position x (i, j): Luminance value at (i, j) a: Horizontal 5TAP coefficient, b: 5 × 3 block filter coefficient with a (0) as a center coefficient, ω1: Horizontal 5TAP calculation weighting coefficient ω2: 5 × 3 block filter weighting coefficient b (0, 0) as a center coefficient The texture characteristic value texture detection algorithm shown in FIG. 7 is a block (9 Information on how many horizontal edges are included in a pixel block of × 9) and how long a vertical line continues around the central pixel for which the parallax calculation is performed (how long the horizontal edge continues) Is an algorithm for evaluating the texture level.

  In this algorithm, for example, by changing the pattern of a 5 × 3 block filter, it is possible to deal with various patterns such as diagonal lines as well as vertical edges. When the size of the target image is large, it is necessary to thin out a large number of pixels. Therefore, a highly accurate texture characteristic value is calculated using such an algorithm. Thereby, it is possible to select an ideal pixel based on a texture characteristic value with high accuracy for performing parallax calculation. The first detection processing unit 33 outputs the calculated texture characteristic value to the selection unit 35 together with the small image.

  The second detection processing unit 34 is a texture characteristic detection process applied when the size of the target image for calculating the parallax is small. The second detection processing unit 34 applies a simple texture detection algorithm giving priority to the processing speed to the small image acquired from the buffer 9 and calculates a texture characteristic value.

  FIG. 8 is a diagram for explaining the texture characteristic value (No. 2). In the example shown in FIG. 8, the texture characteristic value is calculated with a simple texture detection algorithm for the pixels in the target image.

  According to the example shown in FIG. 8, the second detection processing unit 34 calculates the texture characteristic value using the equation (4).

ただし、（ｉ，ｊ）：画素位置
ｘ（ｉ，ｊ）：（ｉ，ｊ）の輝度値
図８に示す例では、隣接する画素の輝度値の差分の絶対値を算出してテクスチャ特性値とする。対象画像のサイズが小さい場合には、それほど画素を間引く必要がないので、処理速度を優先して簡易的にテクスチャ特性値が算出される。第２検出処理部３４は、算出したテクスチャ特性値を小画像と共に選択部３５に出力する。

However, (i, j): luminance value of pixel position x (i, j): (i, j) In the example shown in FIG. 8, the absolute value of the difference between the luminance values of adjacent pixels is calculated to determine the texture characteristic value. And When the size of the target image is small, it is not necessary to thin out pixels so much, and the texture characteristic value is simply calculated giving priority to the processing speed. The second detection processing unit 34 outputs the calculated texture characteristic value to the selection unit 35 together with the small image.

  The first detection processing unit 33 and the second detection processing unit 34 may be configured as one detection processing unit 36. The detection processing unit 36 may include three or more detection processes.

  The selection unit 35 selects a parallax calculation pixel from the small image based on the texture characteristic value acquired by the first detection processing unit 33 or the second detection processing unit 34. For example, when the selection unit 35 acquires a texture characteristic value from the first detection processing unit 33, if the acquired texture characteristic value is equal to or greater than the first threshold (texture threshold), the pixel corresponding to the texture characteristic value is parallaxed. Select as a calculation pixel. Pixels that are not selected are thinned out.

  For example, if the texture characteristic value is acquired from the second detection processing unit 34 and the acquired texture characteristic value is equal to or greater than the second threshold value (texture characteristic value), the selection unit 35 selects a pixel corresponding to the texture characteristic value. Select as a parallax calculation pixel. Pixels that are not selected are thinned out.

  The selection unit 35 may change the first threshold value and the second threshold value so as to be suitable for each. Further, the first threshold value and the second threshold value may be appropriately changed according to the luminance histogram of each small image or the immediately preceding target image. If the small image is a bright image, the threshold is set high, and if the small image is a dark small image, the threshold is set low. This threshold is set to an optimum value by experiment. The selection unit 35 outputs the selected parallax calculation pixel (or its luminance value) to the buffer 26.

  Returning to FIG. 5, the buffer 26 stores the selected parallax calculation pixels until read by the parallax data calculation unit 27.

  The parallax data calculation unit 27 acquires the parallax calculation pixel selected by the texture detection unit 25 from the buffer 26. The parallax data calculation unit 27 acquires the pixels of the comparison image, which are pixels corresponding to the acquired parallax calculation pixels, from the buffer 24.

  The parallax data calculation unit 27 calculates parallax data based on the parallax calculation pixels of the reference image and the pixels corresponding to the parallax calculation pixels of the comparison image. The parallax data calculation unit 27 calculates the parallax data according to the equation (5). In Expression (5), in accordance with the example illustrated in FIG. 7, the parallax data is calculated by setting the block size for performing the parallax calculation to 9 × 9.

ただし、Ｉ（ｉ，ｊ）：視差算出画素の輝度値
Ｃ（ｉ，ｊ）：視差算出画素に対応する比較画像の画素の輝度値
なお、視差データ算出部２７は、視差データ算出に用いるアルゴリズムは式（５）に示すようにＳＡＤ（Sum of Absolute Difference）に限らず、２つの画素ブロックの相関を比較するものであれば適用できる。視差データ算出部２７は、算出した視差データ及び小画像をバッファ２８に出力する。

However, I (i, j): luminance value of the parallax calculation pixel C (i, j): luminance value of the pixel of the comparison image corresponding to the parallax calculation pixel Note that the parallax data calculation unit 27 is an algorithm used for calculating the parallax data. Is not limited to SAD (Sum of Absolute Difference), as shown in Equation (5), and can be applied as long as the correlation between two pixel blocks is compared. The parallax data calculation unit 27 outputs the calculated parallax data and the small image to the buffer 28.

  The buffer 28 stores the parallax data and the small image until read by the reliability calculation unit 29.

  The reliability calculation unit 29 acquires the parallax data and the small image from the buffer 28 and determines whether or not the position of the correlation pixel used for the calculation of the parallax data is reliable. Here, the case where the reliability is not possible means, for example, that the correlation value with the comparison image is calculated by the search width for a certain texture effective pixel position in the reference image by the above equation (5). If the values are similar, etc.

  FIG. 9 is a diagram for explaining the reliability of correlation. The horizontal axis shown in FIG. 9 indicates the search width, and the vertical axis indicates the correlation calculation value (for example, SAD value). As a result of calculating the SAD value for the search width of the small image by the reliability calculation unit 29, the portion of the value with the lowest correlation value calculated by the equation (5) becomes the parallax equivalent position.

  On the other hand, the difference between the correlation calculation value (Min) at the parallax equivalent position and the correlation calculation maximum value (Max) (hereinafter also referred to as correlation difference) does not reach the threshold in the example shown in FIG. This threshold value is a correlation value difference threshold value (correlation value difference threshold value). That is, in the search width, it means that similar blocks are aligned with the pixel block on the reference image side, and it cannot be said that the parallax is very reliable.

  Therefore, as described above, when the correlation difference exceeds the correlation value difference threshold, the reliability calculation unit 29 outputs the parallax data calculated using the parallax equivalent position to the distance calculation unit 12.

  Returning to FIG. 3, the parallel control unit 11 controls each process of the dividing unit 7 and the parallax calculating unit 10 to perform parallel pipeline processing.

  The distance calculation unit 12 uses the parallax data acquired from the reliability calculation unit 29 as the parallax p, and obtains the distance Z by Expression (1). This distance Z indicates the distance to the object.

<Pipeline processing>
In the present invention, each process in the stereo processing unit 6 as described above is performed as a pipeline process in parallel at high speed, thereby speeding up the time required for the parallax calculation process.

  FIG. 10 is a diagram illustrating an example of a processing timing flow of pipeline parallel processing. Each area number shown in FIG. 10 corresponds to each small image 16 divided in FIG. As shown in FIG. 10, it can be seen that even when various processes such as texture detection and parallax calculation are performed, the parallax calculation result is continuously output for each area.

  Further, the example shown in FIG. 10 shows a state in which the processing result is stored in the buffer for each processing. As shown in FIG. 10, a pipeline process is prepared for each pipeline stage. However, the buffer for each process is overwritten for each area. In addition, two systems are required for the geometric image correction for the reference image and the comparison image, but only one system buffer is sufficient for the other processes.

  The size of each buffer may be different for each process as shown in FIG. FIG. 11 is a diagram for explaining different buffer sizes for each process. As shown in FIG. 11, an image 41 used for geometric correction needs a margin for performing geometric correction.

  As shown in FIG. 11, the image 42 used for texture detection and parallax calculation needs a margin for a block for texture detection and parallax calculation for the end pixels in the sub-area.

As shown in FIG. 11, parallax data can be calculated for an image 43 that is a target of parallax calculation at the maximum for all the pixels in this region. As shown in FIG.
The relationship of allowance for geometric correction> allowance for texture detection and parallax calculation is established.

  From the above, when reading a small image from the memory 5 for the first time, it is necessary to read the maximum size in order to have a margin.

  In addition, since these pipeline stages are executed in parallel, it is necessary to control the processing timing to be the same. Therefore, the parallel control unit 11 performs weight control on the processing time of other processes according to the parallax calculation processing cycle that is considered to take the longest. For example, in the parallel control unit 11, several processing time tables are provided with the block size and search width of parallax calculation being input. The parallel control unit 11 controls the weights of other processing stage units according to the processing time table.

<Operation>
Next, the operation of the distance measuring device 13 in the first embodiment will be described. FIG. 12 is a flowchart illustrating an example of parallax data calculation processing according to the first embodiment. The example shown in FIG. 12 is an example of processing one small image.

  In step S101 shown in FIG. 12, the dividing unit 7 divides the target image of the reference image into small images and the comparison image into small images.

  In step S102, the geometric correction units 21 and 22 perform geometric correction on each small image.

  In step S103, the determination unit 31 acquires the number of parallax calculation pixels and / or the size of the target image, and determines a texture detection algorithm from the acquired size of the target image and / or the number of parallax calculation pixels. The method of determination is as described above.

  In step S104, the detection processing unit 36 calculates a texture characteristic value using the determined texture detection algorithm. For example, the texture characteristic value is calculated by Expression (3) or Expression (4).

  In step S105, the selection unit 35 selects a parallax calculation pixel based on the calculated texture characteristic value. For example, if the size of the target image is large, the selection unit 35 selects the parallax calculation pixel so as to thin out many pixels. The texture characteristic value used at this time is a value obtained by a highly accurate algorithm. In addition, if the size of the target image is small, the selection unit 35 selects the parallax calculation pixel with fewer pixels to be thinned out. The texture characteristic value used at this time is a value obtained by a simple algorithm giving priority to the processing time.

  In step S106, the parallax data calculation unit 27 calculates parallax data using the luminance value of the selected parallax calculation pixel and the luminance of the pixel corresponding to the parallax calculation pixel in the comparison image.

  In step S107, the reliability calculation unit 29 determines whether or not the pixel for which the parallax data is obtained is a reliable pixel when performing correlation calculation. The reliability calculation is as described above.

  It is also conceivable to use the texture detection process as the reliability calculation process. In texture detection processing, when many pixels are thinned out for a large target image, it takes time as texture detection processing, but a high-accuracy technique is applied, and few pixels are thinned out for small target images. Texture detection is performed by a simple method.

  Here, when parallax calculation is to be performed on a small target image with almost no pixels thinned out, a highly accurate texture detection method is applied to the pixel during parallax calculation. The result may be calculated as one of the reliability levels.

  As described above, in this processing, the parallax calculation processing is considered to be the most dominant in processing time, so by implementing the pipeline processing as in the present invention, highly accurate texture detection processing is performed on the pixel. It is also possible to do. In other words, if high-precision texture detection processing is performed on the entire image area, the processing becomes heavy, but parallax calculation is performed for pixels that have been determined to perform parallax calculation by a simple method. In the meantime, it is possible to calculate a highly accurate texture detection process for the pixel.

  The reliability calculation unit 29 may determine that the parallax data is reliable if the texture characteristic value calculated by performing high-precision texture detection processing is equal to or greater than a predetermined threshold.

  As described above, according to the first embodiment, a target image of an arbitrary size that is a target of parallax calculation is divided into small images, and geometric correction, texture detection, parallax calculation, and reliability calculation are performed on the small image. The processing executed in performing the parallax calculation such as is executed by pipeline processing.

  Thereby, the calculation of the effective parallax in the target image can be speeded up. Further, since processing can be performed in units of small images, it is not necessary to have a large line memory inside the parallax calculation hardware, and cost can be reduced.

  Note that the geometric correction process and the reliability calculation process are not necessarily required processes, and the parallax data can be calculated if there is at least a texture detection process and a parallax calculation process. Therefore, the present invention can achieve the object of the present invention by pipeline processing the texture detection process and the parallax calculation process.

  Furthermore, when performing parallax calculation for an arbitrary image size, it can be realized by multiple processing of small images as a unit, so there is no need to change control of parallax calculation for various image sizes. It becomes possible to implement with simple processing.

  Further, according to the first embodiment, it is possible to realize a high-speed parallax calculation stably without depending on a scene or an object.

  In the first embodiment, a known texture detection process may be used for the texture detection process. If the texture detection process described above is performed, the texture detection algorithm to be applied can be switched according to the target image size to be used (corresponding field angle), so parallax corresponding to a wide field angle range without optical zoom or electronic zoom Calculation can be executed efficiently.

  Furthermore, if the texture detection process described above is performed, the texture detection algorithm to be applied is defined in advance for each size of the target image, so if the image size is large, a large number of pixels are thinned out, and texture detection is performed. An algorithm with high detection accuracy can be used as the algorithm. On the other hand, when the size of the target image is small, there is almost no need to thin out pixels, so that it is possible to immediately shift to an effective process of parallax calculation with a simple texture detection algorithm.

  In the first embodiment, parallax calculation corresponding to a wide range of angle of view can be used without optical zoom or electronic zoom, and a stereo camera is used in consideration of the processing speed of the parallax calculation, cost reduction, and distance detection accuracy. Auto focus control is possible.

  The dividing unit 7, the parallax calculating unit 10, and the distance calculating unit 12 are realized by, for example, an arithmetic circuit or a processor and a temporary storage memory in the distance measuring device 13, and the parallel control unit 11 can be realized by, for example, a processor.

[Example 2]
Next, an image pickup apparatus in Embodiment 2 will be described. The imaging device according to the second embodiment is an imaging device including the distance measuring device 13 according to the first embodiment.

<Configuration>
FIG. 13 is a block diagram illustrating an example of the configuration of the imaging device 50. An imaging device 50 illustrated in FIG. 13 is, for example, a digital camera. The imaging device 50 includes a power source 51, an imaging optical system 53 that images the subject 52, an imaging unit 54, a processor 55, a distance measuring device 13, a control unit 57, an autofocus unit 58, and a focus driving unit 59.

  The image pickup means 54 converts the subject image picked up by the photographing optical system 53 into an image formation signal and outputs it to the processor 55 and the autofocus means 58.

  The processor 55 performs predetermined processing on the image forming signal from the imaging unit 54 and outputs the processed signal to the monitor. As described in the first embodiment, the distance measuring device 13 measures the distance to the subject 52 based on the principle of triangulation.

  The control means 57 performs control to be described later based on the distance measurement result of the distance measuring device 13. The autofocus unit 58 performs control to be described later based on the image forming signal from the imaging unit 54.

  The focus driving unit 59 moves a part of the photographing optical system 53 in the optical axis direction to change the focus state of the subject image formed on the imaging unit 54.

  The autofocus unit 58 controls the focus driving unit 59 to sequentially change the focus state, sequentially evaluates the image forming signal obtained for each focus state, and obtains a predetermined focus state based on the evaluation value.

  The control means 57 controls the autofocus means 58 so that the evaluation is performed in the focus range near the focus state corresponding to the distance obtained by the distance measuring device 13.

  As described in the first embodiment, the distance measuring device 13 determines a texture detection algorithm according to the size of a target image that is a target of parallax calculation. The distance measuring device 13 thins out pixels based on the determined texture detection algorithm, calculates parallax data based on the parallax calculation pixels, and calculates a distance based on the parallax data. The distance measuring device 13 executes the algorithm determination, texture detection processing including thinning-out processing, and parallax calculation processing by pipeline processing. In addition, the distance measuring device 13 may execute these processes by pipeline processing in addition to the small image reading process, the geometric correction process, and the reliability calculation process in addition to the texture detection process and the parallax calculation process.

  The autofocus means 58 calculates the contrast of the subject image at each lens position of each focus lens group while moving the entire zoom range as the scanning range by moving the focus lens group of the photographing optical system 53, and the maximum after the entire area scan. The position where the contrast is obtained is determined as the optimum focus position, that is, the optimum focus position (focus position).

  This method is called a contrast AF method or a hill-climbing AF method, and focusing is performed while actually looking at the subject image, so that highly accurate and accurate focus detection can be performed.

  However, on the other hand, since the optimum focus position (focus position) is obtained after scanning the entire area, there is a drawback that it takes time to focus. In particular, in the imaging device 50 having a high-magnification zoom function, it takes time to focus.

  Therefore, the control means 57 calculates the lens position A of the focus lens group corresponding to the distance to the subject 52 obtained by the distance measuring device 13, and ±± B scanning range (A−ΔB˜) with the lens position as the center. A + ΔB) is set to the near focus range, and the autofocus means 58 is controlled so that the near focus range is the scan range. That is, the control unit 57 controls the autofocus unit 58 so that the vicinity including the lens position corresponding to the distance is the focus detection scanning range.

  This shortens the time until focusing is achieved, and also realizes accurate focusing. Of course, the focus lens group may be moved only by the distance measurement information (distance) of the distance measuring device 13 and focus adjustment may be performed without performing contrast AF.

  In addition, in the imaging apparatus using the distance measuring apparatus according to the second embodiment, it is possible to reduce the size and cost, and to perform focusing at high speed with high accuracy, thereby missing a photo opportunity. Images can be captured without any problems.

  The distance measuring device 13 of the present invention can be applied to uses such as a vehicle-mounted distance measuring device, a video camera distance measuring device, a portable device mounted camera, a three-dimensional digital camera, and a surveillance camera.

  As mentioned above, although the Example of this invention was explained in full detail, this invention is not limited to the specific Example which concerns, In the range of the summary of this invention described in the claim, in addition to an Example Various modifications and changes are possible.

DESCRIPTION OF SYMBOLS 1, 2 Image pick-up part 4 Image acquisition part 5 Memory 6 Stereo processing part 7 Dividing part 8, 9 Buffer 10 Parallax calculation part 11 Distance calculation part 13 Distance measuring device 21, 22 Geometric correction part 23, 24, 26, 28 Buffer 25 Texture detection unit 27 Parallax data calculation unit 29 Reliability calculation unit 31 Determination unit 32 Switching unit 33 First detection processing unit 34 Second detection processing unit 35 Selection unit 36 Detection processing unit 50 Imaging device 55 Processor 57 Control unit 58 Autofocus unit 59 Focus drive means

JP 7-225127 A JP 2009-14444 A Japanese Patent Laid-Open No. 11-14346 JP 2006-322795 A

Claims (7)

A distance measuring device for measuring a distance to an object,
Two imaging means;
A dividing unit that divides each image captured by the two imaging units into a plurality of small images;
A texture detection process for selecting a parallax calculation pixel, a selected parallax calculation pixel, and a small image of an image captured by one of the two imaging units, and the other imaging unit. Disparity calculating means for performing at least a disparity calculating process for calculating disparity data using a pixel corresponding to the selected disparity calculating pixel in the image;
Control means for controlling the texture detection processing by the parallax calculation means and the parallax calculation processing to be processed in parallel for each of the small images;
Ranging device comprising. The parallax calculation means includes
A geometric correction process for performing geometric correction on the small images of the respective images, and a reliability calculation process for calculating the reliability of the parallax data, and the texture detection process is performed on the geometrically corrected small images. Done,
The control means includes
The distance measuring apparatus according to claim 1, wherein the geometric correction process, the texture detection process, the parallax calculation process, and the reliability calculation process are controlled in parallel for each of the small images. The texture detection process
A first detection process for detecting a texture characteristic with high accuracy or a second detection for detecting a simple texture characteristic in order to select a parallax calculation pixel in accordance with the size of a target image for parallax calculation among the images. The distance measuring apparatus according to claim 1, wherein any one of the processes is determined, and a parallax calculation pixel is selected from the target image based on a detection result of the determined process. The reliability calculation process includes:
When the second detection process is determined in the texture detection process, the first detection process is performed on the selected parallax calculation pixel, and the reliability is determined based on a result of the first detection process. The distance measuring device according to claim 3 to calculate.   5. The distance measuring device according to claim 1, wherein each buffer that holds processed data provided for each process included in the parallax calculating unit has a different memory size. The control means includes
The distance measuring device according to any one of claims 1 to 5, wherein control is performed so that processing start timings of the respective processes included in the parallax calculation unit are the same. Two imaging means;
A dividing unit that divides each image captured by the two imaging units into a plurality of small images;
A texture detection process for selecting a parallax calculation pixel, a selected parallax calculation pixel, and a small image of an image captured by one of the two imaging units, and the other imaging unit. Disparity calculating means for performing at least a disparity calculating process for calculating disparity data using a pixel corresponding to the selected disparity calculating pixel in the image;
Parallel control means for controlling the texture detection processing by the parallax calculation means and the parallax calculation processing to be processed in parallel for each of the small images;
Distance calculating means for calculating a distance based on the parallax data calculated by the parallax calculating means;
Focus control means for controlling autofocus based on the distance calculated by the distance calculation means;
An imaging apparatus comprising:

Images