JP2004069583A - Image processing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To avoid mismatch problem in a parallax image for quickly finding a distance to a target object. <P>SOLUTION: In this image processing device, sequential parallax images of the target object are found by means of a photographing device 13 via an optical shutter 12 having an aperture part moving at a high speed, and on the basis of the image information about the found sequential parallax images, a distance to the target object is computed by a distance computing part 152 in a computing device 15. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image processing apparatus that measures a distance to a target to be tracked based on parallax images captured continuously.
[0002]
[Prior art]
Image processing for obtaining information on the depth direction of an observation target using multiple images is an artificial visual function that restores the three-dimensional structure of the target in the field of computer vision and machine vision and helps to recognize the outside world. It is one of the important methods to realize. A representative method of extracting and restoring depth information of a target using different parallax images is stereo image processing. Here, the parallax image is an image obtained by photographing the same object or the same scene by two or more cameras installed in different places. The stereo image processing is a process of extracting distance and depth information to an observation target using a plurality of parallax images.
[0003]
In conventional stereo image processing using a plurality of cameras, an image formation position of the same object is specified from images taken by each camera, and a distance to the object is measured based on the principle of triangulation. Here, for the sake of simplicity, a two-lens stereoscopic view using two parallax images captured by two cameras will be described.
[0004]
The object of interest OBJ whose distance is to be measured is imaged by two cameras 1 and 2 from different viewpoints. At this time, camera parameters such as the relative positional relationship between camera 1 and camera 2 are known. A region W2 corresponding to the region W1 including the target OBJ in the image I1 captured by the camera 1 is obtained from the image I2 of the camera 2 capturing the target OBJ from a viewpoint different from that of the camera 1. Thus, the distance to the target OBJ is calculated based on the principle of triangulation from the position of the region W1 and the position of the region W2 in the images I1 and I2, and the camera parameters.
[0005]
A matching process such as block matching is used to specify an area W2 corresponding to the area W1 including the target OBJ in the image I1 on the image I2. The matching process is a process of determining a correspondence between pixels or regions between a plurality of parallax images. In this matching process, the correlation value or the SAD (Sum of
Absolute Difference (sum of absolute values of differences) is used as an index indicating the degree of matching.
[0006]
On the other hand, as another method of acquiring a parallax image, there is a method of acquiring a parallax image by moving a monocular camera. This method of acquiring a parallax image by moving the monocular camera is called motion stereo.
[0007]
[Problems to be solved by the invention]
In the above-described compound-eye stereo image processing, when a plurality of parallax images have relatively small parallax and little change in the order of appearance of feature points such as edges and patterns, an area W2 corresponding to the area W1 in the image I1. Is relatively easy to obtain from the image I2. Further, since the search range when searching for the region W2 in the image I2 can be reduced, it is possible to suppress the amount of calculation when calculating the distance.
[0008]
However, when the parallax between the image I1 and the image I2 is large, there is a problem of incompatibility typified by occlusion that feature points are present in one image but not in the other image. Furthermore, since the parallax between the images is large, the search range of the corresponding area is large, and it has been difficult to associate feature points. In particular, when the region of interest is a part of the repetition pattern, the repetition pattern may be included in the search region. For this reason, there is a possibility that an erroneous correspondence such as a case where the target of interest cannot be uniquely determined in the search area or an erroneous correspondence may occur.
[0009]
In such a case, an erroneous response as described in JP-A-2001-184497, JP-A-2001-153633, JP-A-2000-121319, and JP-A-10-289316 is prevented. Or additional processing for detecting an erroneous response has been required. Alternatively, it is necessary to add a complicated elastic matching algorithm for performing flexible association. Performing such additional processing has caused a problem that the above-described series of image processing for calculating the distance to the target becomes complicated and the load increases.
[0010]
On the other hand, in a monocular motion stereo, by sequentially acquiring parallax images while moving the camera, a continuous parallax image with little change between images can be obtained. However, on the other hand, a mechanism for moving the camera is required, and there has been a problem that the device becomes large.
[0011]
Furthermore, when acquiring distance and depth information to a moving target of interest, the viewpoint must be moved in an order faster than the moving speed of the target of interest. For this reason, accurate distance and depth information to the moving target of interest could not be obtained.
[0012]
Further, when using the distance and depth information for, for example, driving control of a vehicle, the distance information must be acquired within a control cycle, so that a continuous parallax image is acquired at a high speed, and the distance and depth information are acquired within the acquisition cycle of the parallax image. This caused a problem that the distance had to be calculated as quickly as possible.
[0013]
Therefore, the present invention has been made in view of the above, and an object of the present invention is to provide an image processing apparatus that avoids the problem of disparity of a parallax image and quickly obtains a distance to a target. is there.
[0014]
[Means for Solving the Problems]
In order to achieve the above object, means for solving the problems of the present invention include: an imaging unit for imaging an image; an imaging direction control unit for controlling a direction of imaging by the imaging unit; A timing control unit for controlling a timing and a change amount of a direction in which the imaging unit captures an image; a first extraction unit for extracting a target to be tracked from the first image captured by the imaging unit; Area determining means for determining an area for extracting the target of interest to be tracked from the second image captured by the image capturing means based on the amount of change in the direction in which the image capturing means captures the second image captured by the image capturing means A second extraction unit for extracting the target of interest to be tracked from an area of the image determined by the area determination unit; and a second extraction unit for extracting the target of interest from the first image by the first extraction unit. Distance calculating means for calculating a distance between the image pickup means and the target of interest based on image information of the target of interest and the image information of the target of interest extracted from the second image by the second extracting means And characterized in that:
[0015]
【The invention's effect】
As described above, according to the present invention, it is possible to provide an image processing apparatus that avoids a problem of disparity in a parallax image and quickly obtains a distance to a target.
[0016]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0017]
FIG. 1 is a diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus according to the embodiment shown in FIG. 1 includes a lens 11, an optical shutter 12 such as a liquid crystal shutter or a PLZT optical shutter that can electrically control the position of an opening arbitrarily, and a photoelectric converter such as a CMOS or a CCD. The imaging device 13 that performs the conversion, the frame memory 14 that stores the image captured by the imaging device 13, the arithmetic device 15, the position of the opening of the optical shutter 12, the image reading position of the imaging device 13, and the arithmetic operation of the distance arithmetic unit 152 The driving device 16 for controlling the timing is provided.
[0018]
The calculation unit 15 tracks the target of interest for which the distance is to be calculated, and performs a process required to obtain a continuous parallax image with sufficient parallax, and the tracking processing unit 151 And a distance calculating unit 152 that calculates a distance as three-dimensional position information to a target of interest based on the amount of change in the image position between the obtained continuous parallax images, the focal length, and the eye distance.
[0019]
The imaging device 13 functions as an imaging unit that captures an image. The optical shutter 12 functions as an imaging direction control unit that controls the imaging direction of the imaging device 13. The driving device 16 functions as timing control means for controlling the timing at which the imaging device 13 captures an image and the amount of change in the direction in which the imaging device 13 captures an image. The tracking processing unit 151 of the arithmetic device 15 includes a first extraction unit that extracts a target of interest to be tracked from the first image captured by the imaging device 13, based on a change amount in a direction in which the imaging device 13 captures an image. An area determining unit that determines an area for extracting a target of interest to be tracked from the second image captured by the imaging device 13, and an area that is determined by the area determining unit in the second image captured by the imaging device 13. , Functions as a second extraction unit for extracting a target to be tracked. The distance calculation unit 152 of the calculation device 16 converts the attention target image information extracted from the first image by the tracking processing unit 151 and the attention target image information extracted from the second image by the tracking processing unit 151. Based on this, it functions as a distance calculation unit that calculates the distance between the imaging device 13 and the target of interest.
[0020]
Next, FIG. 2 for explaining the principle of motion stereo, and FIG. 3 for explaining the corresponding point search processing in stereo image processing, show an example of measuring the distance to the target using the principle of motion stereo. This will be described with reference to FIG.
[0021]
As shown in FIG. 2, after obtaining the image I1 at the camera position C1, the camera is moved from the position C1 to obtain the image I2 at the position C2. The process of obtaining the correspondence of the target (region) of interest from the different parallax images I1 and I2 and calculating the distance to the target will be described as an example of calculating the distance to the region W1 in the image I1 in FIG. .
[0022]
In FIG. 3, in order to obtain an area W2 corresponding to the area W1 in the image I1 in the image I2, for example, the area W1 is used as a template, and an area W corresponding to the area W1 is assumed in the search area S in the image I2. . Then, while scanning the area W in the search area S from left to right in a very small amount (for example, one pixel at a time), the correlation value between the area W and the area W1 is calculated, and the degree of coincidence (or matching error) of the correlation values is calculated. It is quantified to obtain a correlation diagram shown in FIG.
[0023]
In FIG. 3C, it is assumed that the region W having the highest matching degree is a region W2 in the image I2 corresponding to the region of interest W1 in the image I1. Assuming that the pixel position of the region W1 on the image I1 is (x1, y1) and the position of the region W2 on the image I2 is (x2, y2), the camera is moved by a distance H in a direction perpendicular to the optical axis. At this time, the coordinates (x, y, z) of the attention area W1 in the three-dimensional space are:
(Equation 1)
x = x1 · H / (x1-x2)
y = y1 · H / (x1-x2)
= Y2 · H / (x1-x2)
z = fH / (x1-x2)
Is calculated as Here, f is the focal length.
[0024]
As a method of obtaining a parallax image, in addition to the method of obtaining a parallax image by moving the camera position as described above, there is a method of realizing a different shutter opening position and an image reading position as shown in FIG. In FIG. 4, the image of the imaging region C1 on the imaging device 13 when the opening B1 of the optical shutter 12 is opened and the image of the imaging region C2 on the imaging device 13 when the opening B2 is opened are shown. A different parallax image is obtained from each of the read images.
[0025]
On the other hand, when a plurality of openings of the optical shutter 12 are opened and images formed at different positions on the imaging device 13 are read out, a parallax image acquisition method similar to that of the compound-eye stereo image processing is used. When a different parallax image is obtained by scanning the opening of the optical shutter 12, the method is the same as the method of obtaining a parallax image by motion stereo.
[0026]
In the stereo image processing, in order to obtain relative three-dimensional position information such as a distance from an observation point to a target of interest, a process of obtaining a correspondence of the target of interest between parallax images is essential. In the process of obtaining the correspondence between the different parallax images, the reliability of the correspondence can be improved and the processing load can be reduced by using the continuous parallax images obtained continuously by changing the parallax little by little. Can be done.
[0027]
As shown in FIG. 3, when a region W2 corresponding to the reference region W1 is obtained in the image I2, and when the optical axis directions of the cameras are always parallel, the feature of the target to be noted that the parallax is large ( The appearance of the region) and the position on the image differ greatly between the images. Therefore, it is necessary to set a large search area S for searching the corresponding area W2. When the search region S becomes large, a case may occur in which both the feature of the target object and the similar feature are included in the search region S. For this reason, there is a possibility of an erroneous response, and the number of repetitions of the matching process increases, thereby increasing the amount of calculation.
[0028]
On the other hand, when the parallax is relatively small and the order of appearance or deformation of the feature of interest is small, the search region S for searching the corresponding region W2 is set relatively small, so that the feature of the target of interest in the search region S is set. It is possible to exclude the possibility that features similar to are included. Further, since the number of times of the matching process can be reduced, the association can be performed relatively easily. For these reasons, it is understood that the use of continuous parallax images is useful in stereo image processing.
[0029]
In order to acquire a continuous parallax image by the compound-eye stereo method, as shown in FIG. 5, it is necessary to arrange a plurality of cameras B1, B2, and B3 at different positions so as to reduce the parallax, and to acquire an image. There is. On the other hand, in order to acquire a continuous parallax image by the motion stereo method, as shown in FIG. 6, an image is acquired while continuously moving the camera position from B1 to B2 to B3 using one camera unit. I do. Alternatively, as shown in FIG. 7, under the control of the driving device 16, the opening of the optical shutter 12 is continuously moved little by little from the opening B1 to the opening B2, and the position of the opening is determined. Synchronously, a continuous parallax image is obtained by scanning the image reading position of the imaging device 13 from the imaging region C1 to the imaging region C2 under the control of the driving device 16.
[0030]
In such a continuous parallax image acquisition method, when the target of interest does not move, the acquired continuous parallax images are subjected to corresponding point detection processing, and after the feature points are associated with each other, 3 The dimensional coordinates may be calculated. For this reason, the time lag for acquiring each parallax image is not a problem.
[0031]
On the other hand, when the target moves, it is required to acquire a continuous parallax image faster than the moving speed of the target. Therefore, in the process of acquiring a continuous parallax image, if the movement of the target cannot be ignored, an error occurs in calculating the distance to the target. Alternatively, a case occurs where the distance is not obtained.
[0032]
For example, as shown in FIG. 8A, when the target of interest moves from A1 to A2 and A3 while the imaging position moves from B1 to B2 and B3, the relationship between the continuous parallax image and the imaging position Is calculated at a position farther than the actual position. Conversely, if the moving speed of the image of the target A on the image is faster than the speed at which the continuous parallax image is obtained, a negative distance is calculated. Further, as shown in FIG. 8B, when the target of interest changes from A1 to A2 and A3 with the change of the imaging position, that is, when the imaging position of the target of interest A between the continuous parallax images does not change. Cannot calculate the three-dimensional position of the target A.
[0033]
Such a state can be avoided by tracking the target while obtaining a continuous parallax image as fast as the target can be approximated as not moving. As shown in FIG. 9, the position of the target of interest calculated by moving the imaging position from C1 to C2 and C3 faster than the speed at which the target of interest A moves from the position A1 to A2 and A3 is: It is almost the same as the actual position of the target A. Thus, the faster the parallax moves, the smaller the error in the calculated position.
[0034]
It is not realistic to scan the camera position at high speed in order to speed up the movement of parallax, even from the mechanism thereof. However, in the case of the optical shutter 12 used in this embodiment, since the opening position can be arbitrarily controlled electrically, the opening position can be scanned at high speed.
[0035]
In order to acquire a continuous parallax image while scanning the opening of the optical shutter 12 at high speed, it is necessary to read the image at a high frame rate synchronized with the scanning speed of the opening of the optical shutter 12. Imaging at a frame rate at which the movement of the target of interest can be ignored (it can be approximated to a stationary state) has a great effect on the tracking process of the target of interest. By increasing the frame rate of the imaging device 13, the amount of movement on the image of the target of interest decreases. This means that a search area for searching for a target of interest can be limited to an extremely narrow range between temporally consecutive parallax images.
[0036]
FIG. 10 shows the relationship between the amount of movement of the opening of the optical shutter 12 and the image forming position on the imaging surface of the imaging device 13. When the aperture of the optical shutter 12 is fixed and the frame rate is sufficiently higher than the moving speed of the target, the change in the imaging position due to the movement of the target on the imaging surface is very small, and I can ignore it.
[0037]
Next, as shown in FIG. 10A, when the opening of the optical shutter 12 is moved from C (t) to C (t + Δt) in a continuous time, an image is formed on the imaging surface of the target object. The position P moves from P (t) to P (t + Δt). At this time, since the frame rate is sufficiently high, Δt is minute, and the change in the imaging position P is minute. Thus, the search range S (t + Δt) for searching the imaging position P (t + Δt) can be limited to the vicinity of P (t). As the search range S (t + Δt) can be reduced, the number of matching processes for searching for a corresponding point between the parallax images decreases. Further, since extra information does not enter the search area S (t + Δt), it is possible to avoid occurrence of an erroneous response.
[0038]
The above description also holds true for a motion stereo that obtains a continuous parallax image by scanning the opening of the optical shutter 12 and the reading position of the imaging device 13 at high speed. Therefore, the uniqueness of the correspondence between the continuous parallax images obtained by tracking the target of interest is guaranteed.
[0039]
Next, the process of calculating the distance of the target of interest will be described with reference to the flowchart shown in FIG.
[0040]
In FIG. 11, in step S1, the target of interest is imaged by the imaging device 13. Subsequently, in step S2, a region of interest is defined from the captured image. The region of interest may be individual regions (plural regions) obtained by dividing the entire image into rectangular regions of equal size. Next, in step S3, the individual regions defined in step S2 are registered as patterns as reference images used in the matching process in step S7 described later. The reference image may be a grayscale image, an image on which edge extraction processing has been performed, or a binarized image. Next, in step S4, a search area for searching for an area corresponding to each reference area in the subsequently acquired parallax image is defined. When acquiring a continuous parallax image at a high speed, since the movement of the attention area between consecutive parallax images is small, for example, each attention area defined in step S2 is set by enlarging it to an area obtained by enlarging a few pixels in the vicinity. Alternatively, an area enlarged by several pixels only in the scanning direction may be set as the search area.
[0041]
Next, in step S5, the opening of the optical shutter 12 is moved by a very small amount (for example, several pixels when the optical shutter 12 is a liquid crystal shutter), and then an image is captured in step S6. In step S 6, the image is read out by linking the imaging range with the opening position of the optical shutter 12 and stored in the frame memory 14. Next, in step S7, for each of the target patterns registered in step S3, the position where the degree of coincidence is highest is detected from the search regions corresponding to each of the target patterns defined in step S4, and each target region Corresponds to the position in the image acquired in step S6. When the correspondences are obtained for all the corresponding regions, in step S8, a search region for searching for a region corresponding to each reference region in a subsequently acquired parallax image is newly defined, similarly to step S4. .
[0042]
Next, in step S9, it is determined whether sufficient parallax has been obtained for each region of interest. For an area for which it has been determined that sufficient parallax has been obtained, the distance to the target of interest is calculated as three-dimensional position information of the target area based on the amount of change in image position between continuous parallax images, the focal length, and the eye distance. . For the region where it is determined that sufficient parallax has not been obtained, the process returns to step S5, and the opening position of the optical shutter 12 is further moved by a very small amount. Thereafter, the processing up to step S9 is repeatedly performed. In step S9, when the scanning range of the opening of the optical shutter 12 has reached the limit, the distance to the target of interest is calculated as three-dimensional position information of each region using the information obtained so far.
[0043]
(The first embodiment is an embodiment corresponding to the invention described in any one of claims 1, 2, and 3.)
In the first embodiment, since a continuous parallax image with sufficient parallax is obtained, the distance to the target of interest can be accurately and quickly obtained. In addition, by moving the opening of the optical shutter 12 at high speed and moving the image reading position of the imaging device 13 at high speed in synchronization with the movement of the opening of the optical shutter 12, continuous parallax is achieved in tracking the target of interest. The search range of the corresponding region between the images is reduced, and extra information included in the search range is reduced. Thereby, it is possible to avoid an erroneous correspondence between the continuous parallax images.
[0044]
The above effects correspond to the effects achieved by the technical contents described in any one of claims 1, 2 and 3.
[0045]
Next, a second embodiment of the present invention will be described.
[0046]
The feature of the second embodiment is that, in the continuous disparity image acquired as described above, when edge information is used as a feature on the image, it corresponds to the direction (direction) of the edge. The direction in which the opening of the optical shutter 12 is scanned is set. If the edge of interest has a distribution only in the one-dimensional direction, it is difficult to detect a moving component parallel to the edge.
[0047]
As shown in FIG. 12, for example, when attention is paid to the horizontal edge of the roof portion of the vehicle as a feature point, if the horizontal edge has a distribution only in the y direction, the opening of the optical shutter 12 is parallel to the horizontal edge. Is scanned, the parallax images shown in FIGS. 12A and 12B are obtained. When the regions of interest W1 and W2 are associated with each other between the obtained parallax images, since the edge of interest has a distribution only in a single direction, the position is limited in a direction parallel to the edge. It has no features and it is difficult to uniquely determine the correspondence. As described above, when the feature of interest has only a one-dimensional distribution and only a part of the feature is captured, a “window problem” occurs in which a moving component in the distribution direction cannot be extracted.
[0048]
In order to avoid this window problem, it is conceivable to use only features having a two-dimensional distribution. However, at a location where the feature has only a one-dimensional distribution, there are portions that can be tracked and portions that cannot be tracked. The scanning direction of the opening of the optical shutter 12 corresponds to the direction in which parallax is generated between the parallax images with respect to the feature (edge) of the target to be tracked. Therefore, in order to avoid the window problem, it is necessary to scan the opening of the optical shutter 12 or move the imaging device 13 in a direction orthogonal to the one-dimensional distribution of the feature. Then, a parallax image corresponding to the scanning direction or the moving direction is obtained, and the amount of movement in the direction orthogonal to the feature (edge) of interest between the parallax images may be obtained. That is, the process of associating a feature between parallax images may be performed only in a direction in which parallax occurs in the feature.
[0049]
By setting the scanning direction of the opening of the optical shutter 12 and the image reading position to a direction orthogonal to the one-dimensional distribution of the feature, the distance to the feature (edge) of interest can be obtained. Therefore, as shown in FIG. 13, when the feature of interest is a vertical edge, scanning is performed in the horizontal direction, and when the feature of interest is a horizontal edge, scanning is performed in the vertical direction. By obtaining a parallax image in this manner, it is possible to easily calculate a distance to a feature (edge) of interest.
[0050]
(The second embodiment is an embodiment corresponding to the invention described in claim 4.)
In the second embodiment, when a vertical edge is strongly observed as a feature of interest, the opening of the optical shutter 12 and the image reading position of the imaging device 13 are moved in the horizontal direction, and the horizontal edge becomes strong. When it is observed, the opening of the optical shutter 12 and the image reading position of the imaging device 13 are moved in the vertical direction. In other words, the opening of the optical shutter 12 and the image reading position of the image pickup device 13 are scanned in a direction orthogonal to the strong feature distribution (principal axis) direction. As a result, the performance of tracking the change in the scanning direction of the feature of interest is improved, and the position of the feature of interest can be more accurately observed. Therefore, the distance to the target can be accurately calculated.
[0051]
The above effects correspond to the effects achieved by the technical contents of the fourth aspect.
[0052]
Next, a third embodiment of the present invention will be described.
[0053]
In motion stereo that scans a tracked target at a fixed step width, when tracking multiple targets at different positions, the amount of movement of the target on the imaging surface differs depending on the distance to the target. . The amount of movement on the image of the target of interest located relatively close to the observation point is large, and the amount of movement on the image of the object located far away is small. FIG. 5 shows this state.
[0054]
In FIG. 5, the camera scans from B1 to B3 with a constant moving width H. At this time, the positions of the attention targets A1 and A2 on the image move in the order of C11 → C12 → C13 and C21 → C22 → C23, respectively. When it is desired to measure the distances to the attention targets A1 and A2, if the scanning width of the opening of the optical shutter 12 is set so as to reduce the movement amount of the attention target A2, the movement amount of the attention target A1 on the image is minute. It becomes. Therefore, scanning must be repeated until a parallax that can accurately detect the calculated distance to the target A1 is obtained.
[0055]
After a parallax that can accurately calculate the distance to the target A2 is obtained, the scanning width is set to a maximum value within a range where the movement amount of the target A1 on the image falls within the search area S. As a result, the distance to the target A1 can be quickly calculated. That is, when there is only one target to be observed, the scanning width is set so that the movement amount of the target on the image falls within the search area S. On the other hand, when there are a plurality of target objects to be observed, the movement amount on the image becomes larger as the object is closer to the observation point, so that the movement amount on the image of the object closer to the observation point is initially included in the search area S. And calculate the distance to the close target. Subsequently, the scanning width is set so that the moving amount of the next closest target falls within the search area S, and the distance to the target of interest is calculated. As described above, by optimizing the scanning width while sequentially changing the target of interest from the target of interest located at a close position to the target of interest located at a position far from the target, it is possible to quickly calculate the distance to the target of interest. it can.
[0056]
(The third embodiment is an embodiment corresponding to the invention described in claim 5.)
In the third embodiment, the movement amount on the image of the target of interest is observed according to the opening of the optical shutter 12 and the scanning step width of the image reading position. When the scanning step width is small and the moving amount of the target of interest is extremely small, the scanning step width is changed according to the moving amount of the target on the image so as to improve the scanning speed by increasing the scanning step width. As a result, the tracking process of the target of interest can be speeded up, and the distance to the target of interest can be calculated quickly.
[0057]
The above effects correspond to the effects achieved by the technical contents described in claim 5.
[0058]
Next, a fourth embodiment of the present invention will be described.
[0059]
The feature of the fourth embodiment is that the position of the target of interest is accurately detected when the target of interest is concealed due to a change in parallax and when a new target of interest appears. It is in. With reference to FIG. 14, a description will be given of a process in which the target of interest is concealed with a change in parallax and a process of detecting the target of interest in a process in which a new target of interest appears.
[0060]
In FIG. 14, the image acquired at the imaging position B1 is I1, the image acquired at the imaging position B2 is I2, and the image acquired at the imaging position B3 is I3. When the image of the attention area W is captured while the imaging position is moved from B3 to B2 to B1, the attention area W is hidden by the vehicle A1 when the imaging position changes from B2 to B1. By tracking the attention area W by the matching process, it can be determined that the attention area W has disappeared due to the concealment of the attention area W when the degree of coincidence becomes lower than the threshold value.
[0061]
Conversely, when the imaging position changes from B1 to B2, the attention area W newly appears. If the entire image is equally divided into blocks without particularly limiting the area of interest, and tracking processing is performed between continuous parallax images for each block, areas that newly appear during the acquisition of parallax images are newly monitored. Set the area. By performing tracking processing between successive continuous parallax images for the newly set monitoring area, it is possible to calculate the distance from the correspondence between the parallax images. That is, in both the case where the target of interest is concealed and the case where the target of interest newly appears, the distance to the target region W can be obtained by using the parallax images I2 and I3 in which the target is visible.
[0062]
(The fourth embodiment is an embodiment corresponding to the invention described in claim 6 or 7).
In the fourth embodiment, the target of interest is continuously tracked, and the distance to the target of interest is calculated using the image information immediately before the target of interest is hidden. This makes it possible to avoid the incompatibility problem that occurs in compound-eye stereo vision and in which the feature observed in one image does not exist in the other image.
[0063]
In addition, since the target of interest newly appearing in the scanning process is tracked within the maximum scanning range, the feature observed in one image, which occurs in compound-eye stereo vision, does not exist in the other image. This makes it possible to avoid such unresponsive problems.
[0064]
The above effects correspond to the effects achieved by the technical contents described in claim 6 or 7.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention.
FIG. 2 is a diagram for explaining the principle of motion stereo.
FIG. 3 is a diagram for explaining a corresponding point search process in stereo image processing.
FIG. 4 is a diagram for explaining the principle of motion stereo by scanning an opening.
FIG. 5 is a diagram illustrating a continuous parallax image obtained by motion stereo.
FIG. 6 is a diagram for explaining a change in the step width of the motion stereo.
FIG. 7 is a diagram for explaining a principle of acquiring a continuous parallax image.
FIG. 8 is a diagram illustrating a case where an error occurs in a measurement position due to movement of a target of interest.
FIG. 9 is a diagram for explaining a case where a measurement position error is reduced by increasing a scanning speed.
FIG. 10 is a diagram illustrating a tracking process of a target of interest.
FIG. 11 is a flowchart illustrating a process of calculating a distance to a target of interest.
FIG. 12 is a diagram for explaining a window problem.
FIG. 13 is a diagram illustrating a relationship between a feature direction and a scanning direction.
FIG. 14 is a diagram illustrating a case where a target is hidden and appears.
[Explanation of symbols]
11 lenses
12 Optical shutter
13 Imaging device
14 Frame memory
15 Arithmetic unit
16 Drive
151 tracking processing unit
152 Distance calculator

Claims (7)

Imaging means for capturing an image;
Imaging direction control means for controlling a direction in which the imaging means performs imaging,
Timing of imaging by the imaging means, timing control means for controlling the amount of change in the direction of imaging by the imaging means,
First extraction means for extracting a target of interest to be tracked from the first image captured by the imaging means;
An area determining unit that determines an area for extracting the target of interest to be tracked from the second image captured by the imaging unit, based on a change amount of a direction in which the imaging unit captures the image;
A second extraction unit that extracts the target of interest to be tracked from among the regions determined by the region determination unit in the second image captured by the imaging unit;
Based on the image information of the target of interest extracted from the first image by the first extraction unit, and the image information of the target of interest extracted from the second image by the second extraction unit An image processing apparatus comprising: a distance calculation unit that calculates a distance between the imaging unit and the target of interest. The imaging direction control means,
The image processing apparatus according to claim 1, wherein the image processing device is an optical shutter that changes a direction in which the imaging unit captures an image by arbitrarily moving the position of the opening. The imaging speed of the imaging device and the speed at which the imaging direction control unit changes the imaging direction are speeds that can approximate a state where the target of interest is stationary with respect to the imaging device. 3. The image processing device according to 1 or 2. When the vertical edge information is strong as the target of interest, the imaging direction of the imaging direction control unit is scanned in the horizontal direction, and the reading position of the imaging device is synchronized with the scanning position and timing of the imaging direction control unit in the imaging direction. To calculate the distance to the target of interest,
When the horizontal edge information is strong as the target of interest, the imaging direction of the imaging direction control unit is scanned in the vertical direction, and the reading position of the imaging device is synchronized with the scanning position and timing of the imaging direction control unit in the imaging direction. The image processing apparatus according to claim 1, wherein the distance to the target of interest is calculated by changing the distance. The target of interest is sequentially changed from the target of interest closest to the imaging device to the target of interest farthest, and the amount of movement of the image capturing direction in the image capturing direction control unit is optimized according to the distance to each of the target of interest. The image processing apparatus according to claim 1, wherein the image processing is performed. When the target of interest is concealed due to a change in the line of sight caused by the scanning of the imaging direction control unit, the distance to the target of interest is calculated based on the parallax image obtained until the target of interest is concealed. The image processing apparatus according to claim 1, wherein: When the target of interest appears along with a change in the line of sight due to the scanning of the imaging direction control unit, the maximum scanning position of the imaging direction control unit is determined from the imaging direction of the imaging direction control unit at the position where the target of interest appears. The distance to the target of interest is calculated based on the parallax image obtained up to or the parallax image obtained until the target of interest is concealed. The image processing device according to any one of claims 5 and 6.

Images