JP3988574B2 - Image processing device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus that measures a distance to a target object to be tracked based on parallax images that are continuously captured.
[0002]
[Prior art]
Image processing that uses multiple images to obtain information about the depth direction of an observation target is an artificial visual function that helps to recognize the outside world by restoring the three-dimensional structure of the target in the field of computer vision and machine vision. It is one of the important methods to realize. Stereo image processing is a typical technique for extracting and restoring target depth information using different parallax images. Here, the parallax image is an image obtained by photographing the same object or the same scene with two or more cameras installed in different places. Stereo image processing is processing for extracting distance and depth information to an observation target using a plurality of parallax images.
[0003]
In conventional stereo image processing using a plurality of cameras, the imaging position of the same object is specified from the images taken by the respective cameras, and the distance to the object is measured based on the principle of triangulation. Here, for brevity, binocular stereo vision using two parallax images taken by two cameras will be described.
[0004]
The target object OBJ whose distance is to be measured is picked up by two cameras 1 and 2 from different viewpoints. At this time, camera parameters such as the relative positional relationship between the camera 1 and the camera 2 are known. A region W2 corresponding to the region W1 including the target OBJ in the image I1 captured by the camera 1 is obtained from the image I2 of the camera 2 that captures the target OBJ from a different viewpoint from the camera 1. Accordingly, the distance to the target object OBJ is calculated based on the principle of triangulation from the position of the area W1 and the position of the area W2 in each of the images I1 and I2 and the camera parameters.
[0005]
A matching process such as block matching is used to specify a region W2 corresponding to the region W1 including the target OBJ in the image I1 on the image I2. The matching process is a process for determining the correspondence between pixels or regions between a plurality of parallax images. In this matching process, a correlation value or SAD (Sum of Absolute Difference, sum of absolute values of differences) is used as an index indicating the degree of matching.
[0006]
On the other hand, as another method for acquiring a parallax image, there is a method of acquiring a parallax image by moving a monocular camera. This technique of acquiring a parallax image by moving a monocular camera is called motion stereo.
[0007]
[Problems to be solved by the invention]
In the compound-eye stereo image processing, when a plurality of parallax images have relatively small parallax and there is little change in the appearance order of feature points such as edges and patterns, a region W2 corresponding to the region W1 in the image I1 Is relatively easy to obtain from the image I2. In addition, since the search range when searching for the region W2 in the image I2 can be reduced, it is possible to suppress the amount of calculation when calculating the distance.
[0008]
However, when the parallax between the image I1 and the image I2 is large, there is a non-correspondence problem represented by occlusion that a feature point is in one image but not in the other image. Furthermore, since the parallax between the images is large, the search range of the corresponding area becomes large, and it is difficult to associate the feature points. In particular, when the region of interest becomes a part of the repetitive pattern, the repetitive pattern may be included in the search region. For this reason, there is a possibility that an erroneous correspondence such as a case where a target of interest cannot be uniquely determined in the search area or an incorrect association may occur.
[0009]
In such a case, it is possible to prevent erroneous correspondence as described in JP-A-2001-184497, JP-A-2001-153633, JP-A-2000-121319, JP-A-10-289316, etc. Or, an additional process for detecting a false response is required. Alternatively, it is necessary to add a complicated elastic matching algorithm that performs flexible association. Performing such additional processing has caused a problem that the series of image processing for calculating the distance to the target of interest becomes complicated and the load increases.
[0010]
On the other hand, in monocular motion stereo, a continuous parallax image with little change between images can be obtained by sequentially acquiring parallax images while moving the camera. However, on the other hand, a mechanism for moving the camera is required, which causes a problem that the apparatus becomes large.
[0011]
Furthermore, when acquiring the distance and depth information to the moving target object, the viewpoint must be moved in an order faster than the moving speed of the target object. For this reason, accurate distance and depth information to the moving target object cannot be obtained.
[0012]
In addition, when using distance and depth information for vehicle driving control, for example, distance information must be acquired within a control cycle, so a continuous parallax image is acquired at a high speed and within a parallax image acquisition cycle. The problem was that the distance had to be calculated quickly so that it could be settled.
[0013]
Accordingly, the present invention has been made in view of the above, and an object of the present invention is to provide an image processing apparatus that avoids the problem of incompatibility of parallax images and quickly obtains the distance to a target of interest. is there.
[0014]
[Means for Solving the Problems]
In order to achieve the above object, means for solving the problems of the present invention include: In front of the vehicle An imaging unit that captures an image; an imaging direction control unit that controls a direction in which the imaging unit captures an image; An imaging speed for adjusting the imaging speed at which the imaging unit captures an image and the speed at which the imaging direction control unit changes the imaging direction to a speed that can approximate the state in which the target to be tracked with respect to the imaging unit is stationary. Based on the adjusting means and the speed adjusted by the imaging speed adjusting means Tracking from the first image captured by the imaging means Said First extraction means for extracting a target of interest; Speed adjusted by the imaging speed adjusting means The region determination unit for determining the region for extracting the target of interest to be tracked from the second image captured by the imaging unit, and the region determination unit for the second image captured by the imaging unit A second extraction unit that extracts the target object to be tracked from the region that has been tracked, the image information of the target object that is extracted from the first image by the first extraction unit, and the second Based on the image information of the target of interest extracted from the second image by the extraction means, The image reading position of the imaging means is changed in synchronization with the scanning position and scanning timing in the imaging direction controlled by the imaging direction control means, It has a distance calculation means for calculating a distance between the imaging means and the target object.
[0015]
【The invention's effect】
As described above, according to the present invention, it is possible to provide an image processing apparatus that avoids the problem of incompatibility of parallax images and quickly obtains the distance to the target object.
[0016]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0017]
FIG. 1 is a diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus according to the embodiment shown in FIG. 1 has a lens 11, a liquid crystal shutter, a PLZT optical shutter, an optical shutter 12 that can electrically control the position of the aperture, a CMOS, a CCD, and the like. An imaging device 13 that performs conversion, a frame memory 14 that stores an image captured by the imaging device 13, an arithmetic device 15, an opening position of the optical shutter 12, an image readout position of the imaging device 13, and an arithmetic operation of the distance arithmetic unit 152 A drive device 16 for controlling the timing is provided.
[0018]
The calculation unit 15 is obtained by a tracking processing unit 151 that performs processing necessary for acquiring a continuous parallax image in which sufficient parallax is obtained by tracking a target of interest whose distance is to be calculated, and a tracking processing unit 151. A distance calculation unit 152 that calculates the distance as three-dimensional position information to the target of interest based on the amount of change in image position between the continuous parallax images, the focal length, and the eye distance is configured.
[0019]
The imaging device 13 functions as an imaging unit that captures an image. The optical shutter 12 functions as an imaging direction control unit that controls the direction in which the imaging device 13 captures an image. The driving device 16 functions as a timing control unit that controls the timing at which the imaging device 13 captures images and the amount of change in the direction in which the imaging device 13 captures images. The tracking processing unit 151 of the arithmetic device 15 is based on a first extraction unit that extracts a target of interest to be tracked from the first image captured by the imaging device 13, and the amount of change in the direction captured by the imaging device 13. A region determination unit that determines a region for extracting a target of interest to be tracked from the second image captured by the imaging device 13 and a region determined by the region determination unit in the second image captured by the imaging device 13. , And functions as a second extraction unit for extracting the target object to be tracked. The distance calculation unit 152 of the calculation device 16 converts the target image information extracted from the first image by the tracking processing unit 151 and the target image information extracted from the second image by the tracking processing unit 151. Based on this, it functions as a distance calculating means for calculating the distance between the imaging device 13 and the target of interest.
[0020]
Next, an example of measuring the distance to the target object using the principle of motion stereo, FIG. 2 for explaining the principle of motion stereo, and FIG. 3 for explaining corresponding point search processing in stereo image processing Will be described with reference to FIG.
[0021]
As shown in FIG. 2, after obtaining the image I1 at the camera position C1, the camera is moved from the position C1, and the image I2 is obtained at the position C2. The process of obtaining the correspondence between the target (region) of interest from the different parallax images I1 and I2 and calculating the distance to the target will be described by taking as an example the case of calculating the distance to the region W1 in the image I1 in FIG. .
[0022]
In FIG. 3, in order to obtain the area W2 corresponding to the area W1 in the image I1 in the image I2, for example, the area W1 is used as a template, and the area W corresponding to the area W1 is assumed in the search area S in the image I2. . The correlation value between the region W and the region W1 is calculated while scanning the region W from the left to the right within the search region S from the left to the right (for example, one pixel at a time), and the degree of coincidence (or matching error) between the correlation values is calculated. Quantification is performed to obtain the correlation diagram shown in FIG.
[0023]
In FIG. 3C, it is assumed that the region W having the highest degree of coincidence is the region W2 in the image I2 corresponding to the attention region W1 in the image I1. When the pixel position of the area W1 on the image I1 is (x1, y1) and the position of the area W2 on the image I2 is (x2, y2), the camera is moved by a distance H in a direction perpendicular to the optical axis. In this case, the coordinates (x, y, z) of the attention area W1 in the three-dimensional space are
[Expression 1]
x = x1 · H / (x1-x2)
y = y1 · H / (x1-x2)
= Y2 · H / (x1-x2)
z = f · H / (x1-x2)
Is calculated as Here, f is a focal length.
[0024]
As a method for obtaining a parallax image, there is a method for realizing a parallax image by moving a camera position as described above, as well as a method for realizing a parallax image by using different shutter opening positions and image readout positions as shown in FIG. In FIG. 4, the image of the imaging region C1 on the imaging device 13 when the opening B1 of the optical shutter 12 is opened and the image of the imaging region C2 on the imaging device 13 when the opening B2 is opened are shown. Different parallax images are obtained from the read images.
[0025]
On the other hand, when a plurality of openings of the optical shutter 12 are opened and images formed at different positions on the imaging device 13 are read out, the parallax image acquisition method is the same as in the compound-eye stereo image processing. When scanning the opening of the optical shutter 12 to obtain different parallax images, the method is the same as the method for acquiring parallax images by motion stereo.
[0026]
In stereo image processing, in order to acquire relative three-dimensional position information such as a distance from an observation point to a target object, a process for obtaining a correspondence relationship between target objects between parallax images is essential. The processing for obtaining the correspondence between different parallax images can improve the reliability of the correspondence and reduce the processing load by using continuous parallax images obtained by changing the parallax little by little. Can be made.
[0027]
As shown in FIG. 3, when a region W2 corresponding to the reference region W1 is obtained in an image I2, and the optical axis direction of the camera is always parallel, the feature of the target to be noted that the parallax is large ( The appearance of the (region) and the position on the image are greatly different between images. For this reason, it is necessary to set a large search area S for searching the corresponding area W2. When the search area S becomes large, there may be a case where both the target feature of interest and similar features are included in the search area S. For this reason, there is a possibility of erroneous correspondence, and the number of times the matching process is repeated increases, resulting in an increase in the amount of calculation.
[0028]
On the other hand, when the parallax is relatively small and the appearance order and deformation of the feature of interest are small, the feature of the target of interest in the search region S is set by setting the search region S for searching the corresponding region W2 relatively small. It is possible to exclude the possibility that similar features are included. In addition, since the number of matching processes can be reduced, the association can be performed relatively easily. For these reasons, it can be seen that the use of continuous parallax images is useful in stereo image processing.
[0029]
In order to acquire a continuous parallax image by a compound-eye stereo method, as shown in FIG. 5, it is necessary to shift the position so that the parallax is reduced and to arrange a plurality of cameras B1, B2, and B3 to acquire an image. There is. On the other hand, in order to obtain a continuous parallax image by the motion stereo method, as shown in FIG. 6, an image is obtained while continuously moving the camera position from B1 to B2 to B3 using one camera unit. To do. Alternatively, as shown in FIG. 7, under the control of the driving device 16, the opening of the optical shutter 12 is continuously moved little by little from the opening B1 to the opening B2, and the position of the opening is determined. In synchronization, a continuous parallax image is acquired by scanning the image reading position of the imaging device 13 from the imaging region C1 to the imaging region C2 under the control of the driving device 16.
[0030]
In such a continuous parallax image acquisition method, when the target of interest does not move, the corresponding point detection process is performed on the acquired continuous parallax image, and the feature points are associated. What is necessary is just to calculate a dimensional coordinate. For this reason, the time shift which acquires each parallax image does not become a problem.
[0031]
On the other hand, when the target moves, it is required to obtain a continuous parallax image at a higher speed than the moving speed of the target. Therefore, in the process of acquiring the continuous parallax image, an error occurs in the calculation of the distance to the target when the movement of the target cannot be ignored. Or the case where a distance cannot be calculated | required generate | occur | produces.
[0032]
For example, as shown in FIG. 8A, when the target of attention moves from A1 to A2 and A3 while the imaging position moves from B1 to B2 and B3, the relationship between the continuous parallax image and the imaging position. The position of the target of interest A calculated from the above is calculated at a position farther from the actual position. Conversely, when the moving speed of the imaging of the target of interest A on the image is faster than the speed at which a continuous parallax image is obtained, a negative distance is calculated. Further, as shown in FIG. 8B, when the target of interest changes from A1 to A2 and A3 as the imaging position changes, that is, when the imaging position of the target of interest A between the continuous parallax images does not change. Cannot calculate the three-dimensional position of the target of interest A.
[0033]
Such a state can be avoided by tracking the target object while obtaining a continuous parallax image fast enough to approximate that the target object is not moving. As shown in FIG. 9, the position of the target of interest calculated by moving the imaging position from C1 to C2, C3 is faster than the speed of movement of the target of interest A from position A1 to A2, A3. The actual position of the target of interest A is almost the same. Thus, the faster the movement of parallax, the smaller the error in the calculated position.
[0034]
In order to speed up the movement of parallax, scanning the camera position at high speed is not realistic from the mechanism. However, in the case of the optical shutter 12 used in this embodiment, the opening position can be arbitrarily controlled electrically, so that the opening position can be scanned at high speed.
[0035]
In order to acquire a continuous parallax image while scanning the opening of the optical shutter 12 at high speed, it is necessary to read an image at a high frame rate synchronized with the scanning speed of the opening of the optical shutter 12. Imaging at a frame rate at which the movement of the target of interest can be ignored (can be approximated to a stationary state) has a great effect on the tracking processing of the target of interest. Increasing the frame rate of the imaging device 13 reduces the amount of movement on the target image. This means that the search area for searching for a target of interest can be limited to a very narrow range between temporally continuous parallax images.
[0036]
FIG. 10 shows the relationship between the movement amount of the opening of the optical shutter 12 and the imaging position on the imaging surface of the imaging device 13. When the aperture of the optical shutter 12 is fixed and the frame rate is sufficiently high compared to the moving speed of the target object, the change in the imaging position accompanying the movement of the target object on the imaging surface is very small. Can be ignored.
[0037]
Next, as shown in FIG. 10A, when the opening of the optical shutter 12 is moved from C (t) to C (t + Δt) in a continuous time, the image is formed on the imaging surface of interest. The position P moves from P (t) to P (t + Δt). At this time, since the frame rate is sufficiently high, Δt is very small, and the change in the imaging position P is very small. Thereby, the search range S (t + Δt) for searching for the imaging position P (t + Δt) can be limited to the vicinity of P (t). The smaller the search range S (t + Δt), the smaller the number of matching processes for searching for corresponding points between parallax images. Furthermore, since excess information does not enter the search area S (t + Δt), it is possible to avoid the occurrence of an erroneous response.
[0038]
What has been described above holds true for motion stereo that scans the opening of the optical shutter 12 and the readout position of the imaging device 13 at high speed to obtain a continuous parallax image. For this reason, the uniqueness is guaranteed about the correspondence between the continuous parallax images obtained by tracking the target of interest.
[0039]
Next, the process of calculating the distance of the target object will be described with reference to the flowchart shown in FIG.
[0040]
In FIG. 11, in step S <b> 1, the target object is imaged by the imaging device 13. Subsequently, in step S2, a region of interest is defined from the captured image. The region of interest may be individual regions (plurality) obtained by dividing the entire image into rectangular regions of equal size. Next, in step S3, each region defined in step S2 is registered as a pattern as a reference image used in the matching process in step S7 described later. The reference image may be a grayscale image, an image subjected to edge extraction processing, or a binarized image. Next, in step S4, a search area for searching for an area corresponding to each reference area in the subsequently obtained parallax image is defined. When acquiring continuous parallax images at high speed, there is little movement of the region of interest between consecutive parallax images, so for example, each region of interest defined in step S2 is enlarged and set to a region expanded by several pixels. Alternatively, an area enlarged several pixels only in the scanning direction may be used as the search area.
[0041]
Next, in step S5, the opening of the optical shutter 12 is moved by a very small amount (several pixels when the optical shutter 12 is a liquid crystal shutter, for example), and then imaging is performed in step S6. In step S <b> 6, the image is read out in association with the opening position of the optical shutter 12 and stored in the frame memory 14. Next, in step S7, for each target pattern registered in step S3, a position having the highest degree of matching is detected from the search areas corresponding to each target pattern defined in step S4, and each target area is detected. Corresponds to which position in the image acquired in step S6. When correspondences are obtained for all the corresponding regions, in step S8, similarly to step S4, a search region for searching for a region corresponding to each reference region in the subsequently acquired parallax image is newly defined. .
[0042]
Next, in step S9, it is determined whether sufficient parallax is obtained for each region of interest. For a region determined to have sufficient parallax, the distance to the target of interest is calculated as the three-dimensional position information of the target region based on the amount of change in image position between consecutive parallax images, the focal length, and the eye distance. . For an area determined that sufficient parallax is not obtained, the process returns to step S5, and the opening position of the optical shutter 12 is further moved by a minute amount. Thereafter, the processing up to step S9 is repeated. In step S9, when the scanning range of the opening of the optical shutter 12 has reached the limit, the distance to the target object is calculated as the three-dimensional position information of each region using the information obtained so far.
[0043]
In the first embodiment, since a continuous parallax image with sufficient parallax is acquired, the distance to the target of interest can be obtained accurately and quickly. Further, by moving the opening of the optical shutter 12 at high speed and moving the image reading position of the imaging device 13 at high speed in synchronization with the movement of the opening of the optical shutter 12, continuous parallax can be achieved in tracking the target of interest. The search range of the corresponding area between the images is reduced, and excess information included in the search range is reduced. Thereby, it is possible to avoid erroneous correspondence between continuous parallax images.
[0045]
Next explained is the second embodiment of the invention.
[0046]
The feature of the second embodiment is that in the continuous parallax image acquired as described above, when edge information is used as a feature on the image, it corresponds to the direction (direction) of the edge. That is, the scanning direction of the opening of the optical shutter 12 is set. If the edge of interest has a distribution only in the one-dimensional direction, it is difficult to detect a moving component parallel to the edge.
[0047]
As shown in FIG. 12, for example, when the horizontal edge of the roof portion of the vehicle is focused on as a feature point, if the horizontal edge has a distribution only in the y direction, the opening of the optical shutter 12 is parallel to the horizontal edge. Is scanned, parallax images shown in FIG. 12A and FIG. 12B are obtained. Associating the attention areas W1 and W2 between the obtained parallax images limits the position with respect to a direction parallel to the edge because the attention edge has a distribution only in a single direction. There is no feature and it is difficult to uniquely determine the correspondence. As described above, when the feature of interest has only a one-dimensional distribution and only a part of the feature is captured, a “window problem” occurs in which a moving component in the distribution direction cannot be extracted.
[0048]
In order to avoid this window problem, it is conceivable to use only features having a two-dimensional distribution. However, at locations where the features have only a one-dimensional distribution, there are portions that can be tracked and portions that cannot be tracked. The scanning direction of the opening of the optical shutter 12 corresponds to the direction in which the parallax between the parallax images is generated with respect to the feature (edge) of the target object to be tracked. Therefore, in order to avoid the window problem, it is necessary to scan the opening of the optical shutter 12 in the direction orthogonal to the one-dimensional distribution of the feature or to move the imaging device 13. Then, a parallax image corresponding to the scanning direction or the movement direction is obtained, and the movement amount in the direction orthogonal to the feature (edge) of interest between the parallax images may be obtained. That is, the process of associating features between parallax images may be performed only in the direction in which parallax occurs for the features.
[0049]
Thus, the distance to the feature (edge) of interest can be obtained by setting the scanning direction of the opening of the optical shutter 12 and the image reading position in a direction orthogonal to the one-dimensional distribution of the feature. Therefore, as shown in FIG. 13, when the feature of interest is a vertical edge, scanning is performed in the horizontal direction, and when the feature of interest is a horizontal edge, scanning is performed in the vertical direction. By obtaining the parallax image in this way, it becomes possible to easily calculate the distance to the feature (edge) of interest.
[0050]
In the second embodiment, as a feature of interest, when the vertical edge is observed strongly, the opening of the optical shutter 12 and the image reading position of the imaging device 13 are moved in the horizontal direction, and the horizontal edge is strong. When observed, the opening of the optical shutter 12 and the image reading position of the imaging device 13 are moved in the vertical direction. That is, the aperture of the optical shutter 12 and the image reading position of the imaging device 13 are scanned in a direction perpendicular to the distribution (principal axis) direction of strong features. As a result, the performance of tracking the change in the scanning direction of the feature of interest is improved, and the position of the feature of interest can be observed more accurately. Therefore, the distance to the target of interest can be accurately calculated.
[0052]
Next explained is the third embodiment of the invention.
[0053]
When tracking multiple target objects at different positions in a motion stereo that scans the target object to be tracked with a certain step width, the amount of movement of the target object on the imaging surface varies depending on the distance to the target object. . The amount of movement on the target image existing relatively near the observation point is large, and the amount of movement on the target image existing far away is small. This is shown in FIG.
[0054]
In FIG. 5, the camera is scanned from B1 to B3 with a constant movement width H. At this time, the positions of the attention objects A1 and A2 on the image move from C11 to C12 to C13 and from C21 to C22 to C23, respectively. When it is desired to measure the distance to the target of interest A1 and A2, if the scanning width of the opening of the optical shutter 12 is set so that the amount of movement of the target of interest A2 is small, the amount of movement of the target of interest A1 on the image is very small. It becomes. For this reason, scanning must be repeated until a parallax capable of accurately detecting the calculated distance to the target of interest A1 is obtained.
[0055]
After the parallax that can accurately calculate the distance to the target of interest A2 is obtained, the scanning width is set to the maximum within the range in which the movement amount on the image of the target of interest A1 falls within the search region S. Thereby, the distance to attention object A1 can be calculated rapidly. That is, when there is only one target object to be observed, the scanning width is set so that the amount of movement on the target image is the maximum within the range where the search area S can be accommodated. On the other hand, when there are a plurality of objects of interest to be observed, the amount of movement on the image increases as the object is closer to the observation point, so that the amount of movement on the image of the object closer to the observation point is initially stored in the search region S. The distance to the near target is calculated by setting as described above. Subsequently, the scanning width is set so that the movement amount of the next closest object is within the search region S, and the distance to the target object is calculated. In this way, it is possible to quickly calculate the distance to the target object by optimizing the scan width while sequentially changing the target object to the target object existing at a far position from the target object present at a close position. it can.
[0056]
In the third embodiment, the amount of movement on the target image is observed according to the opening of the optical shutter 12 and the scanning step width of the image readout position. When the scanning step width is small and the moving amount of the target object is extremely small, the scanning step width is changed in accordance with the moving amount of the object on the image so as to improve the scanning speed by increasing the scanning step width. As a result, the tracking process of the target object can be speeded up, and the distance to the target object can be calculated quickly.
[0058]
Next explained is the fourth embodiment of the invention.
[0059]
The feature of this fourth embodiment is that the position of the target of interest is accurately detected when the target of interest is concealed and a new target of interest appears due to a change in parallax. There is. With reference to FIG. 14, a description will be given of detection of a target object in the process of concealing the target object and the process of newly appearing the target object as the parallax changes.
[0060]
In FIG. 14, the image acquired at the imaging position B1 is I1, the image acquired at the imaging position B2 is I2, and the image acquired at the imaging position B3 is I3. When capturing an image of the attention area W while moving the imaging position from B3 → B2 → B1, the attention area W is concealed by the vehicle A1 when the imaging position changes from B2 to B1. By tracking the attention area W by the matching process, it can be determined that the attention area W has disappeared due to the concealment of the attention area W when the degree of coincidence becomes lower than the threshold.
[0061]
Conversely, the attention area W newly appears when the imaging position changes from B1 to B2. In particular, if the region of interest is not limited and the entire image is equally divided into blocks and tracking processing is performed between consecutive parallax images for each block, newly appearing regions are newly monitored during the parallax image acquisition. Set the area. It is possible to calculate the distance from the correspondence between the parallax images by performing the tracking process between the continuous parallax images that continue for the newly set monitoring area. That is, the distance to the attention area W can be obtained by using the parallax images I2 and I3 in which the object is visible both when the attention object is concealed and when it newly appears.
[0062]
In the fourth embodiment, the target object is continuously tracked, and the distance to the target object is calculated using image information until immediately before the target object is concealed. As a result, it is possible to avoid a non-correspondence problem that occurs in compound eye stereo vision and that a feature observed in one image does not exist in the other image.
[0063]
In addition, because the target object that newly appears in the scanning process is tracked within the maximum scanning range, the features observed in one image that occur in compound eye stereo vision do not exist in the other image. It becomes possible to avoid such a non-correspondence problem.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention.
FIG. 2 is a diagram for explaining the principle of motion stereo.
FIG. 3 is a diagram for explaining corresponding point search processing in stereo image processing;
FIG. 4 is a diagram for explaining the principle of motion stereo by scanning an opening.
FIG. 5 is a diagram for explaining continuous parallax images obtained by motion stereo.
FIG. 6 is a diagram for explaining a change in the step width of motion stereo.
FIG. 7 is a diagram for explaining the principle of obtaining a continuous parallax image.
FIG. 8 is a diagram for explaining a case where an error occurs in a measurement position due to movement of a target of interest.
FIG. 9 is a diagram for explaining a case where the measurement position error is reduced by increasing the scanning speed.
FIG. 10 is a diagram for explaining attention target tracking processing;
FIG. 11 is a flowchart illustrating processing for calculating a distance to a target of interest.
FIG. 12 is a diagram for explaining a window problem.
FIG. 13 is a diagram illustrating a relationship between a feature direction and a scanning direction.
FIG. 14 is a diagram for explaining a case where a target is concealed and appears;
[Explanation of symbols]
11 Lens
12 Light shutter
13 Imaging device
14 frame memory
15 Arithmetic unit
16 Drive device
151 Tracking processor
152 Distance calculator

Claims (6)

Imaging means for capturing an image in front of the vehicle ;
Imaging direction control means for controlling the direction in which the imaging means captures;
An imaging speed for adjusting the imaging speed at which the imaging unit captures an image and the speed at which the imaging direction control unit changes the imaging direction to a speed that can approximate the state in which the target to be tracked with respect to the imaging unit is stationary. Adjusting means;
First extracting means for extracting the object of interest to the imaging means to track from the first image captured based on the speed which has been adjusted by the imaging speed adjusting means,
Area determining means for determining an area for extracting the target of interest to be tracked from the second image captured by the imaging means based on the speed adjusted by the imaging speed adjusting means ;
A second extraction unit that extracts the target of interest to be tracked from the regions determined by the region determination unit in the second image captured by the imaging unit;
Based on the image information of the target of interest extracted from the first image by the first extraction unit and the image information of the target of interest extracted from the second image by the second extraction unit. A distance calculation means for calculating a distance between the imaging means and the target of interest by changing an image reading position of the imaging means in synchronization with a scanning position and a scanning timing in the imaging direction controlled by the imaging direction control means. An image processing apparatus comprising: The imaging direction control means is
The image processing apparatus according to claim 1, wherein the image processing apparatus is an optical shutter that changes a direction in which the imaging unit captures an image by electrically moving the position of the opening. When the vertical edge information is strong as the target of interest, the imaging direction of the imaging direction control means is scanned in the horizontal direction,
The image processing apparatus according to claim 1, wherein when the horizontal edge information is strong as the target of interest, the imaging direction of the imaging direction control unit is scanned in the vertical direction. The target object is sequentially changed to the target object farthest from the target object closest to the imaging device, and the amount of movement in the imaging direction in the imaging direction control unit is optimized according to the distance to each target object. The image processing apparatus according to any one of claims 1, 2, and 3 . When the target of interest is concealed as the line of sight changes due to scanning by the imaging direction control means, the distance to the target of interest is calculated based on the parallax image obtained until the target of interest is concealed. The image processing apparatus according to any one of claims 1, 2, 3, and 4 . When the target of interest appears as the line of sight changes due to the scanning of the imaging direction controller, the maximum scanning position of the imaging direction controller from the imaging direction of the imaging direction controller at the position where the target of interest appears. claims 1,2,3,4 and and calculates a distance to the target object based on the obtained parallax image or the parallax images obtained by the object of interest is concealed until the image processing apparatus according to any one of 5.

Images