JP2940366B2 - Object tracking recognition device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for capturing an image of the periphery of a vehicle with an on-vehicle camera mounted on the vehicle, and following and recognizing an object around a road based on the captured image.

[0002]

2. Description of the Related Art Various techniques have been proposed for capturing images of the front and rear of a vehicle using a vehicle-mounted camera and recognizing an object in the captured image. For example, if an in-vehicle camera captures an image of the front of the vehicle, and if the captured image can be image-processed to recognize another vehicle in front, the steering and brakes are automatically operated when another vehicle approaches abnormally. Can automatically avoid an accident. If the road sign can be recognized, the information corresponding to the recognized road sign can be used for so-called route guidance in the navigation device.

When recognizing an image of an object captured by a vehicle-mounted camera, the position of the object on the screen changes every moment. When an image of an object moving on the screen is recognized as described above, the behavior of the object on the screen is estimated, and an image portion including the object is cut out from the captured screen using the estimation result. It is necessary to perform an image recognition process on the clipped image.

In order to estimate the behavior of an object on a screen as described above, it is necessary to know not only the speed of the vehicle, but also the posture of the vehicle-mounted camera with respect to the road (road vehicle coordinate system). The parameters must be determined exactly. In the present study of the attitude parameter of the on-vehicle camera, as will be described later in the embodiment, in calculating the attitude parameter, two road parallel lines such as a center separation line of a road and a boundary line of a road side zone are obtained. ,
If a road vanishing point, which is a point where road parallel lines intersect and disappear in a screen imaged by the on-vehicle camera, can be obtained, the attitude parameter of the on-vehicle camera is determined based on the two road parallel lines and the road vanishing point. It is known that it can be calculated (a patent application has been filed in Japanese Patent Application No. 5-223357).

[0005]

However, various objects such as vehicles, pedestrians, and road signs enter and exit from the field of view of the on-vehicle camera. It must be ensured that an object that enters the field of view of the on-vehicle camera in the cycle is identical to an object that is in the field of view of the on-board camera in the next processing cycle.

[0006] For this purpose, the position of the object or a change in the position,
It is necessary to realize a tracking recognition process capable of following an object in consideration of a position detection error, a distinction whether the object is stationary or moving, a pattern of the object, and the like.
In addition, the following recognition process is a process that is performed while the vehicle is running while capturing an image with the on-board camera, and therefore requires quickness.

Therefore, an object of the present invention is to solve the above-mentioned technical problems, to capture an image of the periphery of a vehicle with an on-vehicle camera mounted on the vehicle, and to efficiently detect objects around the road based on the captured image. An object of the present invention is to provide a symmetric object follow-up recognition device capable of following-up recognition.

[0008]

Means and operation for solving the problems] (1) follow-up recognition apparatus of claim 1 object according to an end of, by imaging the front or rear of the vehicle by the onboard camera, processing cycle Image processing means for converting a digital image signal into a digital image signal, and extracting an outline of one or a plurality of recognition objects in the screen based on the converted image obtained from the image processing means. Registering means for determining coordinates on the screen of one or more specific points that determine the position of the object to be recognized when the position is determined, and registering the coordinates in a memory together with features of an image around the specific points; The coordinates of the specific point, the coordinates of which are determined and registered by the registration means, are converted by the spatial coordinate conversion means for converting the coordinates of the specific point into the spatial position of the specific point using the posture parameters of the on-vehicle camera. Specific point From the spatial position and the traveling distance data and the azimuth data of the vehicle included in the output data of the position detection processing device that detects the position of the vehicle based on signals from various sensors, the space of a specific point in a later processing cycle Estimating means for determining a typical position and estimating coordinates on the screen of a specific point corresponding to this position, and estimating means when a converted image in a later processing cycle is obtained from the image processing means. Determining means for determining whether or not the same specific point exists within an error range around coordinates on the screen; and when the determining means determines that the same specific point exists, the determination of the specific point is performed. The recognition target is tracked by obtaining the spatial position, and when the same specific point is determined not to exist for the recognition target by the determination unit, the registration corresponding to the recognition target is deleted. <br/> in what Apply predicates Oh.

According to the object tracking recognition apparatus, the coordinates of one or more specific points corresponding to each recognition object in the converted image are registered in the memory, and the registered coordinates are stored in the memory.
The coordinates of the specific point are converted into the spatial position of the specific point using the posture parameters of the on-vehicle camera in the processing cycle. Then, from the spatial position of the specific point, the traveling distance data and the direction data of the vehicle included in the output data of the position detection processing device, the spatial position of the specific point in a later processing cycle is predicted. The coordinates of the specific point corresponding to the position on the screen are estimated, and when a converted image is obtained in a later processing cycle, the same specific point is within the error range around the estimated coordinates on the screen. Determine if it exists. Limited space within the error range around the coordinates on the screen
Since it is only necessary to search for a specific point at a place,
I'm done. When it is determined that the same specific point exists, the recognition target is followed by obtaining the spatial position of the specific point. When it is determined that the specific point does not exist, the recognition target is determined.
By deleting the registration corresponding to the elephant, the tracking of the recognition target can be performed efficiently.

(2) When registering information of one or more specific points in the memory corresponding to each recognition target, if pattern information for distinguishing each recognition target is also registered, It is possible to follow more reliably (claim 4 ). (3) Also, when predicting the spatial position of a specific point in a later processing cycle, the spatial position of the specific point and the spatial The spatial position of the specific point is determined from the spatial position of the specific point determined in the processing cycle of the above, and based on this change, whether the recognition target object relating to the specific point is stationary or moving at a constant speed May be determined (claim 5 ). (4) If the change is not obtained (for example, the first processing), the processing may be performed assuming that the object to be recognized is stationary (Claim 7 ). If it is determined that there is no, the determination unit determines whether the same specific point exists within a wider range beyond the error range, based on the characteristics of the image around the registered specific point. The determination may be made based on claim 8 .

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an image recognition system according to one embodiment of the present invention will be described in detail. (1) Overview of Image Recognition Processing System The image recognition processing system described below captures the surroundings of a vehicle with an in-vehicle color camera mounted on the vehicle, and performs tracking recognition of objects around the road based on the captured images. It is a device for performing. Recognized objects are, for example, pedestrians, traffic lights, road signs, road markings drawn on the surface of roads, road facilities such as elevated roads that cross over roads that are running, and other vehicles.

The image recognition processing system has the following -4
By executing the two processes, the object is recognized from the image captured by the in-vehicle color camera. It should be noted that the in-vehicle color camera need not always be used for the and processes. Line candidate point extraction processing Road vanishing point calculation processing In-vehicle camera attitude parameter calculation processing Object tracking recognition processing Straight line candidate point extraction processing refers to a straight line portion along the traveling direction of a vehicle in an image captured by an in-vehicle color camera. This is a process of extracting points that make up a parallel road. The straight part includes both sides of the road, a white line and a yellow line on the road, a median strip, a road side zone, an outline of a vehicle ahead, an outline of a road sign, and the like.

The road vanishing point calculation process is a process of calculating a point at which the left and right edges of the road on which the vehicle is traveling intersect and vanish on the screen. Specifically, it is calculated based on the line candidate points extracted by the line candidate point extraction process. The on-board camera attitude parameter calculation processing is processing for obtaining the attitude of the on-board camera with respect to the road.
In this process, the road vanishing point and the road parallel line obtained by the road vanishing point calculation process are used.

The object tracking recognition process is a process for recognizing and following an object in an image captured by a vehicle-mounted camera. In the object tracking recognition processing, one or more specific points, at which the position of the object is substantially determined once the position is determined, are determined and registered for each object, and when the object is tracked, the change of the specific point is determined. A process of estimating the position to be performed is performed based on the posture parameter of the on-vehicle camera, and searching for a specific point that is the same as the registered specific point based on the image data. In this case, the object may be moving or stationary, but for each of them, the position of the specific point is estimated based on the posture parameters of the on-vehicle camera.

According to such an image recognition processing system, objects around the own vehicle can be recognized. Therefore, by utilizing the recognition result for the automatic control of the steering and the brake of the vehicle, the way to the automatic driving of the vehicle is opened, which can contribute to the safe operation of the vehicle. Recognition of a road sign or a traffic light can be useful when guiding a route to a destination.

First, the configuration of the image recognition processing system will be described below. The negative line candidate point extraction processing, road vanishing point calculation processing, posture parameter calculation processing of the on-vehicle camera, and object tracking recognition processing will be described in order. (2) Configuration of Image Recognition Processing System FIG. 1 is a block diagram showing an electrical configuration of the image recognition processing system. This image recognition processing system includes an image recognition processing device 1 for recognizing an image of an object around a vehicle. The image recognition processing device 1 is connected to a position detection processing device 2 for detecting the current position of the vehicle and the traveling direction of the vehicle and displaying the current position and the traveling direction of the vehicle on a display device together with a road map.

The image recognition processing device 1 includes an in-vehicle color camera 11 mounted in, for example, a front portion of a vehicle or a vehicle interior. The in-vehicle color camera 11 captures an image of the front of the vehicle. In addition to the in-vehicle color camera 11,
Alternatively, instead of the in-vehicle color camera 11, another in-vehicle camera capable of capturing an image of the rear of the vehicle or the side of the vehicle may be provided.

The on-vehicle color camera 11 outputs an analog electric signal representing each point of the captured image in color. The analog signal is subjected to processing such as analog / digital conversion in the image processing circuit 13 and is converted into image data. This image data is input to an image recognition processing unit 15 including a microcomputer and the like. A storage unit 17 having a RAM (random access memory) and the like is connected to the image recognition processing unit 15. The image recognition processing unit 15 is provided with support information from the position detection processing device 2.

The position detection processor 2 includes a distance sensor (for example, a wheel speed sensor) 21 for detecting the traveling distance of the vehicle.
And a direction sensor (for example, a gyro) 22 for detecting the traveling direction of the vehicle. Outputs of these sensors 21 and 22 are processed by a sensor processing circuit 23 to be converted into travel distance data and current direction data. These data are input to the position detection processing unit 25 including a microcomputer and the like, and the position detection processing unit 25 calculates the current position data of the vehicle based on the data input from the sensor processing circuit 23. .

The position detection processing unit 25 includes a road map memory 27 storing a road map and a storage unit 2 having a RAM and the like.
8 and a display 29 composed of a CRT (cathode ray tube) or a liquid crystal display panel are connected. The road map memory 27
For example, it is composed of a CD-ROM. Position detector 2
5 searches the road map memory 27 based on the calculated current position data, and reads a road map around the current position. This road map is displayed on the display device 29, and a mark indicating the current position of the vehicle is also displayed on the road map.

The position detection processing unit 25 provides road map data, current azimuth data, current position data and travel distance data to the image recognition processing unit 15 of the image recognition processing device 1 as support information. In addition to these data, azimuth change data representing the amount of change in the traveling azimuth per unit time or per unit traveling distance may be given to the image recognition processing unit 15. (3) Straight Line Candidate Point Extraction Processing Next, straight line candidate point extraction processing will be described.

FIG. 2 is a diagram showing an example of an image picked up by the in-vehicle color camera 11. However, the driver's seat of the vehicle in the foreground is written later for easy imagination. A roadside white line 43 is provided near the roadside of the road in an image captured by the in-vehicle color camera 11 directed forward of the vehicle. A roadside belt 44 is provided on the side of the road. In the screen, the point where the running road disappears is the road vanishing point NP. 46,4
7, 48 are other vehicles traveling forward or sideways.

In the straight line candidate point extraction processing, the image captured by the in-vehicle color camera 11 is scanned from the upper end to the lower end of the screen along the horizontal scanning direction DH. This scanning is executed by sequentially reading out the image data stored in the storage unit 17 from the in-vehicle color camera 11 through the image recognition processing unit 15. Then, when each pixel constituting the screen is scanned along the horizontal scanning direction DH, a stable state in which chromaticity or luminance or both are stable, and an unstable state in which chromaticity or luminance or both greatly change. A line candidate point is detected based on the transition to the state. The stable state and the unstable state are respectively detected as follows.

For example, when the on-vehicle color camera is red (R),
It is assumed that three primary color signals corresponding to green (G) and blue (B) are output. These three primary color signals are signals representing a color tone. The color tone is a quantity expressed by combining chromaticity and luminance. In this case, the storage unit 17 stores the RGB primary color image data. Each of the R, G, and B image data at an arbitrary point on the scanning line along the horizontal scanning direction DH is represented by r (t), g (t), and b (t). t is
It represents a processing cycle and corresponds to one point on a scanning line along the horizontal scanning direction DH.

In this case, the unstable state is, for example,
For the judgment value P (t) defined by the equation (1), the following equation (2) is used.
Is detected based on the following. m1 is a constant, j
₁, J _TwoAnd j_ThreeIs a constant for weighting. example
For example, since the R data has a large variation with respect to a change in brightness,
Constant j₁Is a relatively large value, and the B data
Constant j_ThreeIs relatively small
Value.

P (t) = j ₁ | r (t) −r (t−1) | + j ₂ | g (t) −g (t−1) | + j ₃ | b (t) −b (t− 1) | (1) P (t)> m1 (2) When the linear sum of the absolute values of the color tone changes of two processing target points adjacent in the horizontal scanning direction DH is larger than a predetermined constant m1 , It is detected that the color tone shows a large change. Neighboring processing target points are not necessarily adjacent two points.
The processing target point is not limited to one pixel and may be set at a certain predetermined number of pixel intervals.

On the other hand, the stable state of the color tone is determined by the following equation (3) with respect to the judgment value P (t) of the above equation (1) by a certain number (for example, 1
0) or more consecutive processing target points. n1 is a constant (however, n1
1 <m1). P (t) <n1 (3) That is, a state in which the linear sum of the absolute values of the color tone changes of the two processing target points adjacent in the horizontal scanning direction DH is smaller than a predetermined constant n1 is a certain number of processing targets. If so, it is detected that the color tone is stable.

In this manner, the image picked up by the in-vehicle color camera 11 is scanned in the horizontal scanning direction, and the degree of change in color tone on the scanning line is checked. And
The processing target point where the stable state in which the color tone is stable and the unstable state in which the color tone is unstable is detected as a straight line candidate point. FIG. 3 is a diagram illustrating a straight line portion obtained by connecting the straight line candidate points P ₁₁ , P ₁₂ ,..., P ₂₁ , P ₂₂ ,. . That is, both sides of the road, the roadside white line 43, and the other vehicles 46, 47, 4
8 are shown as straight line portions L1, L2,....

This figure is based on the theory of road vanishing points.
This is a virtual diagram for clarity, and actually shows the straight line part
It does not mean that the image is obtained. (4) Road vanishing point calculation process
The road candidate shown in FIG.
This is a process for obtaining a goal NP. As is clear from FIG.
In addition, the process of obtaining the road vanishing point is performed by selecting the straight line candidate point P₁₁,
P₁₂, ... ・ ； P_{twenty one}, P _{twenty two}The intersection of the straight lines connecting the
This is nothing but the process you want.

In this road vanishing point calculation process, the initial value of the road vanishing point is obtained by repeating the Hough transform process twice on the coordinate sequence of the straight line candidate point. After that, the road vanishing point is periodically updated by a simple method without performing the time-consuming Hough conversion process.
First, the Hough transform will be outlined. FIG. 4 (a) and FIG.
(b) is a diagram for explaining the Hough transform. Fig. 4 (a)
As shown in FIG. 5, a plurality of points (x _i , y _i ) (where i =
..) Exist on the straight line x = ay + b, x _i = ay _i + b holds for any i. When this equation is considered on the ab coordinate plane by regarding (a, b) as variables, the equation of a straight line on this coordinate plane is b = −y _i
a a + x _i. A graph on the ab plane for all i is shown in FIG. That is, a plurality of straight line groups corresponding to a plurality of i are located at a certain point (a ₀ , b
₀ ). This is a natural consequence of the fact that the plurality of points (x _i , y _i ) all exist on one straight line.

Therefore, the ab coordinate plane is set to a sufficiently fine grid.
Partition into the square, (x_i, Y_iCase with straight line corresponding to)
When passing through a child cell, the total of that cell
Increase the number by one. This operation is performed for all (x_i, Y_i) Seki
The operation to be performed is Hough transform. In the above case,
(A₀, B₀The count value of the grid cell at point) is the largest
It is. Therefore, the grid with the largest count value on the ab coordinate plane
If you ask for a square, (a₀, B₀) Is obtained. Accordingly
And a plurality of points (x_i, Y_i) Is x
= A ₀y + b₀Can be defined as like this
In addition, the Hough transform is used in the field of image processing in several points.
(X_i, Y_i) Is used to determine a straight line passing through

FIG. 5 is a diagram for explaining a process for obtaining the road vanishing point by repeating the Hough transform twice. FIG. 5A shows an xy coordinate plane which is a coordinate plane corresponding to a screen imaged by the in-vehicle color camera 11, and FIG. 5B shows the transformed coordinates (first coordinates) in the first Hough transform. FIG. 5C shows an mn coordinate plane which is a transformed coordinate (second transformed coordinate) plane in the second Hough transform.

As shown in FIG. 5A, the straight line candidate points P ₁₁ ,
_{P 12, ····; P 21,} P 22,; P 31, P 32, the straight line L1, L2, L3 which ... belongs respectively, the road vanishing point (x _0,
y ₀ ). The equation of a straight line passing through the coordinates (x ₀ , y ₀ ) is as the following equation (4). In addition,
C is a constant. x = C (y−y ₀ ) + x ₀ = Cy + (x ₀ −Cy ₀ ) (4) Then, if a = C and b = x ₀ −Cy ₀ , the conversion equation x
= Ay + b, and the relationship between a and b is represented by the following equation (5).

B = −ay ₀ + x ₀ (5) Line candidate points P ₁₁ , P ₁₂ ,... P ₂₁ , P ₂₂ , P ₃₁ , P
_{When the} Hough transform is performed on the coordinates of ₃₂ ,.
On the coordinate plane, a plurality of grid cells whose count values take the maximum value should be obtained corresponding to the plurality of straight lines L1, L2, and L3. However, since the straight lines L1, L2, and L3 intersect at one point (x ₀ , y ₀ ), the grid cells D ₁ , D ₂ , and D _{3 that} have the maximum value must be on the straight line of the above equation (5). No (see Fig. 5 (b)).

Therefore, the grid cells D ₁ ,
The _second Hough transform is performed on the mn coordinate plane for the coordinates of D ₂ and D ₃ using the conversion formula of the following expression (6). b = ma + n (6) Since the grid cells D ₁ , D ₂ , and D ₃ at which the count value is maximal on the ab coordinate plane are on the straight line of the equation (5), m = −y ₀ , n on the mn coordinate plane = X ₀ , the count value of the grid cell becomes the maximum. Thereby, the coordinates (x ₀ , y ₀ ) of the road vanishing point nP on the xy coordinate plane can be obtained.

The road vanishing point and the road parallel line obtained in this way are obtained by performing the Hough transform twice, but the Hough transform takes a long processing time. It is not always possible to perform the Hough conversion processing every time during traveling. Therefore, after the second time, a method of easily and quickly finding a road vanishing point and a road parallel line even if the accuracy is slightly reduced is adopted.

FIGS. 6A and 6B are principle diagrams for explaining this simple road vanishing point calculation method.
Line candidate points are extracted based on the image captured by the above, and the road parallel lines obtained in the previous cycle are virtually applied to this screen (see FIG. 6 (a)), and the neighborhood of those road parallel lines is determined. , Several straight line candidate points are specified. Then, these identified points a ₁ , b ₁ , c ₁ ; a
New road parallel lines L ₁ , L ₂ passing through ₁ , b ₁ , c ₁ are calculated (see FIG. 6 (b)), and the road parallel lines in the current cycle are calculated. Intersections are determined and used as road vanishing points in this cycle. Hereinafter, the same procedure is repeated to update the road parallel line and the road vanishing point.

As described above, the road vanishing point can be obtained by performing a simple calculation without referring to the processing result at the previous time, instead of repeating the Hough conversion processing as described above. The road vanishing point can be obtained and updated quickly and reliably. (5) Attitude Parameter Calculation Process of In-Vehicle Camera In this process, an attitude parameter representing the attitude of the in-vehicle camera 11 with respect to the road is obtained. The attitude parameters include a yaw angle that is a rotation angle around a vertical axis, a roll angle that is a rotation angle around a traveling direction of a vehicle, and a pitch angle that is a rotation angle around a direction along a horizontal plane and orthogonal to the traveling direction. , And the lateral displacement distance of the onboard camera from a predetermined reference line parallel to the road (lateral position relative to the road).

Although the on-vehicle camera 11 is accurately mounted on the vehicle in a predetermined posture, occurrence of mounting errors cannot be avoided.
Therefore, in the posture parameter calculation process of the vehicle-mounted camera, the mounting posture of the vehicle-mounted camera 11 with respect to the vehicle is also calculated. First, the coordinate system will be described. A road vehicle coordinate system XYZ and an in-vehicle camera coordinate system X′Y′Z ′ are defined. It is assumed that the vehicle is at the origin of the road vehicle coordinate system XYZ, the Y axis is taken in the direction along the traveling direction of the vehicle (the traveling direction of the vehicle is + Y), and X is
Take the axis. In addition, the Z axis is set in a direction perpendicular to the road.
The road vehicle coordinate system XYZ can be said to be fixed to the vehicle at a point where the direction along the traveling direction of the vehicle is the Y axis.
Since the axis is set vertically regardless of the pitch and roll of the vehicle, it can be said that the axis is fixed to the road. In that sense, it is an eclectic coordinate system. The onboard camera coordinate system and the road vehicle coordinate system share the origin. The imaging surface of the vehicle-mounted camera 11 is assumed to be substantially parallel to the XZ plane and at a distance F (F is the focal length of the lens of the vehicle-mounted camera 11) from the origin.

The rotation angles around the X, Y, and Z axes are a pitch angle θ, a roll angle φ, and a yaw angle そ れ ぞ れ, respectively, and the direction of the right-hand thread is the positive direction. At this time, the conversion formula of the coordinate system of the on-board camera accompanying the mounting error of the on-board camera or the turning of the vehicle is given by the following equation (7). However, in-vehicle camera 1
It is assumed that the Y 'axis is taken in the main axis direction of one lens, and the X' axis and the Z 'axis are taken parallel to the imaging surface.

[0041]

(Equation 1)

If each rotation angle is small, the above equation (7)
It can be transformed into the following approximate expression (8).

[0043]

(Equation 2)

A point P (X, Y, Z) is a point p 'on the imaging surface.
When projected on (x ', y'), the following equation holds. Here, the coordinates (x ′, y ′) are two-dimensional coordinates on the imaging surface. The x 'axis is set in the X'-axis direction of the on-board camera coordinate system, and the y' axis is set in the Z'-axis direction of the on-board camera coordinate system. x '= F.X' / Y '(9) y' = F.Z '/ Y' (10) Therefore, from the above equation (7) and the above equations (9) and (10), the following (11) The equation and the equation (12) are obtained.

[0045] _{x '= F (R 11 X} + R 12 Y + R 13 Z) / (R 21 X + R 22 Y + R 23 Z) (11) y' = F (R 31 X + R 32 Y + R 33 Z) / (R 21 X + R 22 Y + R 23 Z) (12) In particular, when the approximation of the above equation (8) holds, the following equations (13) and (14) are obtained.

X ′ = F (X + ΔY−φZ) / (− ΔX + Y + θZ) (13) y ′ = F (φX−θY + Z) / (− ΔX + Y + θZ) (14) On the other hand, the attitude parameters of the on-vehicle camera are pitch angle θ, The roll angle φ and the yaw angle ψ are divided as follows. Pitch angle θ: (Pitch angle θ _{0 of} vehicle with respect to road) +
(Mounting pitch angle θ ₁ of vehicle-mounted camera 11 with respect to vehicle) Roll angle φ: (Roll angle φ _{0 of} vehicle with respect to road) +
(Mounting roll angle φ _{1 of} vehicle-mounted camera 11 with respect to vehicle) Yaw angle ψ: Mounting yaw angle 車載₁ of vehicle-mounted camera 11 with respect to vehicle Note that yaw angle ψ does not include yaw angle ψ _{0 of} vehicle with respect to road. This is because the Y-axis of the road vehicle coordinate system is set in the traveling direction of the vehicle as described above.

[0047] Now, the vehicle is assumed to be traveling in a direction forming an angle of [psi ₀ relative to the road, the road is assumed to be linear far enough. It is also assumed that the inclination of the road bank can be ignored. Then, it is assumed that the coordinates of the mapping point of the road vanishing point on the imaging surface are (x ₀ , y ₀ ). The coordinates of the mapping point are nothing but the road vanishing point obtained by the road vanishing point calculation process.

The coordinates x of this mapping point₀, Y₀And a car turtle
Pitch angle θ, roll angle φ
To determine the relationship with the yaw angle ψ, the above equations (11) and (12) are used.
And Y ₀= ∞ may be substituted. x₀= R₁₂F / R_{twenty two} y₀= R₃₂F / R_{twenty two} Here, the vehicle₀Running at an angle of
Ψ → ψ₀+ Ψ₁far. Against the road
Roll angle φ of vehicle₀, Pitch angle θ₀Of the on-board camera 11
Mounting angle φ₁, Θ₁Is small, R₁₂≒ sin （ψ₀+ Ψ₁) R_{twenty two}≒ 1 R₃₂≒ (φ₀+ Φ₁) Sin (ψ₀+ Ψ₁) − (Θ₀+ Θ₁) Cos (ψ₀+ Ψ₁) Holds. Therefore, the following equation (15) and (1
6) is obtained.

X ₀ ≒ sin (ψ ₀ + ψ ₁ ) F (15) y ₀ φ (φ ₀ + φ ₁ ) sin (ψ ₀ + ψ ₁ ) − (θ ₀ + θ ₁ ) cos (ψ ₀ + ψ ₁ )｝ F (16) Further, if ψ ₀ and ψ ₁ are minute, the following (17)
Equation (18) is obtained. x ₀ = (ψ ₀ + ψ ₁ ) F = ψF (17) y ₀ = − (θ ₀ + θ ₁ ) F = −θF (18) These equations (17) and (18) are obtained on the image plane of the road vanishing point. (X ₀ , y ₀ ), the pitch angle θ (= θ ₀ + θ ₁ ) and the yaw angle ψ (= ψ ₀ + 姿勢₁ ), which are the attitude parameters of the on-vehicle camera with respect to the road, can be obtained. ing.

On the other hand, the posture parameters include the pitch angle θ,
In addition to the yaw angle ψ, since there is a roll angle φ and a lateral displacement distance A of the vehicle-mounted camera from a predetermined reference line parallel to the road, the roll angle φ and the lateral displacement distance A must be obtained. For this purpose, the position and shape of the road parallel line on the imaging plane are used. The height Z of the parallel road is determined by the on-board camera 1
If the height to 1 is h, then -h. Further, when the displacement in the normal direction of the road-vehicle camera 11 with respect to a reference line parallel to the road and A, vehicle if traveling on the road and [psi ₀ offset direction, the following equation (19) holds. Where Y
As described above, the axis is set in a direction along the traveling direction of the vehicle, and the X axis is set in a direction perpendicular to the Y axis direction.

[0051] _{X = A / cosψ 0 + Y} tanψ 0 ≒ A + Yψ 0 (19) Therefore, the equation (11) and (12), the following equation (20) and
(21), respectively.

[0052]

(Equation 3)

However, ψ in the above equations (11) and (12) is
₁ Thus, when Y is deleted, the following equation (22), which is an equation representing a road parallel line on the imaging surface, is obtained.

[0054]

(Equation 4)

If two road parallel lines are obtained and their distance B is known, the following equations (23), (24) and (25) similar to the above equation (22) are obtained. Where coefficients a and b
And the suffixes “1” and “2” added to A indicate that the coefficients assigned to them correspond to each of the two road parallel lines.

[0056]

(Equation 5)

In the formulas (23) and (24), a ₁ and a ₂
Equation (26), Equation (27) and
Equation (28) is obtained, and A ₁ , A ₂ and φ are obtained, respectively.

[0058]

(Equation 6)

In the above equations (23) and (24), θ, φ,
Since ψ ₀ and ψ ₁ are minute, if these secondary components (θ ² , θ φ, etc.) are ignored, a ₁ = (A ₁ + φh) / (φA ₁ -h) a ₂ = ( A ₂ + φh) / (φA ₂ -h) is obtained. By solving these equations, A ₁ and A ₂ can be obtained.

A ₁ = − (a ₁ + φ) h / (1−φa ₁ ) A ₂ = − (a ₂ + φ) h / (1−φa ₂ ) These equations are obtained if the roll angle φ is known. And the lateral displacement distances A ₁ and A ₂ can be approximately obtained. In particular, when the vehicle ignoring roll angle phi ₀ have straight traveling, the (29) (30) below is obtained.

A ₁ = − (a ₁ + φ ₁ ) h / (1−φ ₁ a ₁ ) (29) A ₂ = − (a ₂ + φ ₁ ) h / (1−φ ₁ a ₂ ) (30) Here, a vehicle is driven on an actual straight road, and data of a road vanishing point (x ₀ , y ₀ ) is obtained. By substituting these data into the equations (17) and (18), the yaw angle ψ and the pitch angle θ of the vehicle-mounted camera with respect to the vehicle can be obtained.

Here, the roll angle φ _{0 of the} vehicle with respect to the road is
And the pitch angle θ ₀ cannot be detected by themselves, but can be grasped as noise that varies with the average value “0”.
That is, if the average values of the roll angle φ and the pitch angle θ over a sufficiently long time are taken, it may be considered that the roll angle φ ₀ and the pitch angle θ ₀ of the vehicle with respect to the road are 0 in the average values. Further, the yaw angle of the vehicle with respect to the road ψ ₀
If the vehicle travels in parallel with the road, this may also be noise that varies with the average value “0”, but if it is desired to obtain it accurately, the road map data and vehicle May be obtained from the current azimuth data. That is, the direction of the road on which the vehicle is traveling is known from the road map data, and the actual traveling direction of the vehicle is known from the current direction data. Therefore, the difference between the actual traveling direction and the azimuth of the road may be set to the yaw angle ψ _{0 of the} vehicle with respect to the road.

Therefore, when a large number of data of road vanishing points (x ₀ , y ₀ ) are taken and averaged, it is considered that θ ₀ is zero as described above and ψ ₀ is zero or a constant value as described above. Therefore, the mounting yaw angle of the vehicle-mounted camera 11 with respect to the vehicle ψ
₁ and the mounting pitch angle θ ₁ can be obtained. Further, a standard deviation which is a deviation from the average value of the yaw angle ψ ₀ and the pitch angle θ ₀ can also be obtained.

Further, the coefficient a or b of the road parallel line on the imaging plane during traveling is determined. If you know the distance B of the road parallel lines, it is a quadratic equation for A ₁ above (26)
A ₁ can be obtained by solving the equation. Also, A ₂ can be obtained from equation (27), and the roll angle φ can be obtained from equation (28). As the interval B, for example, the width of a road or the interval between white lines of a road can be used. These intervals can be obtained from the road map data by filling in the road map data.

As described above, since a large number of data are taken and averaged, the average value of φ ₀ is considered to be zero as described above. Therefore, it is possible to obtain the mounting roll angle phi ₁ of the vehicle-mounted camera 11 to the vehicle running. Further, a standard deviation which is a deviation from the average value of the roll angle φ ₀ can be obtained. As described above, the installation yaw angle ψ ₁ , the pitch angle θ ₁ and the roll angle φ _1, and the standard deviation which is an index indicating the reliability of the yaw angle ψ ₀ , the pitch angle θ ₀ and the roll angle φ ₀ are obtained. Once obtained, the value of the mounting angle is
It can also be used in subsequent runs. Thus, mounting the yaw angle [psi ₁ running a vehicle, pitch angle theta ₁ and the roll angle phi ₁ and the yaw angle [psi _0, the process for obtaining the standard deviation of the pitch angle theta ₀ and the roll angle phi ₀ as "initial process" .

FIGS. 7 to 11 are flowcharts for explaining the processing executed in the image recognition processing section 15 for the posture parameter calculation processing of the vehicle-mounted camera performed after the completion of the initial processing. This process is performed each time the vehicle has completed the initial process, for example, every time the vehicle travels by a certain distance. A parameter representing the number of times the process is executed is hereinafter referred to as a process cycle t (t = 1, 2, 3, 3). …).

FIG. 7 is a flowchart for explaining the process of calculating the yaw angle of the vehicle-mounted camera. After starting the system, in step S 1, the image data of one image picked up by the vehicle-mounted camera 11 is stored in the storage unit 17. In step S2, support data from the position detection processing device 2 is fetched. Then, in step S3, it is determined whether the vehicle is traveling on a curve at an intersection or the like. The determination of the curve traveling is performed when the rate of change of the traveling direction of the vehicle taken in from the position detection processing device 2 is a constant value (for example, 0.1 ° / processing cycle).
Is determined by whether or not it has been exceeded. When the vehicle is traveling on a curve, the roll angle φ _{0 of the} vehicle with respect to the road may become abnormally large due to the centrifugal force.
Proceed to.

When the vehicle speed is within a certain range (for example,
(Within 100 km / h) and whether the acceleration / deceleration of the vehicle is less than a certain value (for example, 0.1 g) may be used as conditions for calculating the posture parameter. If the speed of the vehicle is not within a certain range, the vehicle body may vibrate, and the posture parameter of the vehicle with respect to the road may greatly fluctuate.If the deceleration of the vehicle is equal to or more than a certain value, This is because the pitch angle θ ₀ of the vehicle with respect to the road may be abnormally large.

If it is determined that the vehicle is not traveling on a curve, the simple calculation method of the vanishing point of the road described above is applied.
It is determined whether two road parallel lines on the left and right are determined (steps S4-S6).
From the intersection of these straight lines, obtaining a road vanishing point (x _0, y ₀₎ (step S9). Then, the yaw angle に 対 す る of the on-vehicle camera with respect to the road is obtained by using the equations (17) and (18) (step S10). Yaw angle [psi ₁ relative to the vehicle body at this time vehicle camera, since a need in the initial process as described above, the yaw angle with respect to the road of the vehicle by obtaining the yaw angle [psi [psi
You can know ₀ . This yaw angle ψ ₀ is equal to the processing cycle t
In the following, it is referred to as an observed value ψ _0t ′ in the sense that it is the yaw angle observed from the imaging screen.

In step S11, it is determined whether or not the adopted value ψ _0t-1 of the yaw angle in the immediately preceding processing cycle has been obtained. Here, to explain the meaning of "the adopted value", the yaw angle ψ ₀ to adopt as attitude parameters of the vehicle-mounted camera, there is of course the requirement that it is as much as possible better error is small. However, if we take the observations as they are,
Observed values are basically information obtained from the screen, so if there is a vehicle in front of traffic congestion or if the parallel road is a dashed line, neither the parallel road nor the vanishing point can be obtained There is. Therefore, a value in consideration of the past history of the yaw angle ψ ₀ is obtained by a recurrence formula (for example, the following formula (31)), and this is set as an “estimated value”. Or if both are reasonably reliable, a weighted average of the two is employed. This adopted value is the “adopted value” here.

In step S11, if the adopted value ψ _0t-1 of the yaw angle in the immediately preceding processing cycle is not found, the observed value ψ _0t 'is set to the adopted value ψ _0t (step S12). . Long as it employs value [psi _0t-1 the yaw angle is calculated in step S11 in the preceding processing period, obtaining the adoption values [psi _0t of the yaw angle of the current processing cycle (step S13). The details of this method are as follows.

First, when the road on which the vehicle is traveling is straight and the road direction does not change (this determination can be made based on, for example, road map data), the road for the road between the processing periods t-1 and t is not processed. The amount of change in the yaw angle of the vehicle is
Processing cycles t-1 to t obtained from the position detection processing device 2
Up to _Δt . That is, if the adopted value ψ _0t−1 of the _immediately preceding processing cycle is used, an estimated value ψ _0t ″ of the current processing cycle can be obtained by the following recurrence formula.

Ψ _0t ″ = ψ _0t−1 + Δψ _0t (31) If the adopted value ψ _0t−1 one processing cycle ago has not been determined, the processing cycle in which the adopted value is determined (referred to as ta). Using the adopted value ψ _0t-a and the change amount Δ _{方位 0t} of the azimuth of the vehicle from the processing cycle ta to t, the formula ψ _0t ″ = ψ _0t-a + Δψ _0t (a = 2, 3, 4,...) (32) to obtain the estimated value { _0t} of the current processing cycle.

If the direction of the road changes, the above formula does not hold. Instead, one of the following recurrence formulas corrected using the difference between the adopted value obtained in the preprocessing cycle and the observed value is used. Shall be used. ψ _0t ″ = ψ _0t−1 + Δψ _0t − (ψ _0t-1 ′ −ψ _0t-1 ) (33) ψ _0t ″ = ψ _0t-a + Δψ _0t − (ψ _0t-a ′ -ψ _0t-a ) ( 34) The meaning of this equation is to correct the estimated value that deviated from the previous time this time by taking into account the difference between the past adopted value and the observed value. The reason that the estimation was incorrect last time is probably because the direction of the road has changed.

The above equation is not used and the road map memory 27 is used.
Ψ _0t ″ = ψ _0t−1 + Δψ _0t −Δψ _Mt (35) ψ _0t ″ = ψ _0t-a + Δψ _0t − in consideration of the change Δψ _Mt in the road direction obtained from the road map data stored in Δψ _Mt (36) can also be used. Here, M of Δψ _Mt means road map data.

In step S14, the search is performed as described above.
Estimated value ψ_0t″ And observations ψ_0t′. So
Then, in step S15, is this difference within a certain range?
Check if. How to choose this "fixed range"
For example, 0.3 ° per processing cycle. Within a certain range
If present, estimate ψ_0t″, Observations ψ_0t′ Is neither
The weighted average of the two.
And the adopted value in the processing cycle t _0t(Step S
16).

[0077] The _{αψ 0t '+ (1-α} ) ψ 0t "→ ψ 0t (37) wherein the weight alpha, for example, empirically-determined constants. If within a certain range, at an earlier point in time In step S15, if the determination that the value is not within the predetermined range has continued for a certain number of times P, that is, a predetermined distance or more, the process goes from step S18 to S19, and the observation value ψ _0t ′ is determined in the processing cycle t. The adopted value is set to _ψ0t , because if the determination that the adopted value is not within the range is continued for a certain number of times or more, the adopted value greatly deviates from the true value due to the accumulated error, and it can be determined that the reliability becomes unreliable.

It is determined that the value is not within the predetermined range for P times
If not, the recruitment value obtained previously ψ_0tReliable
There is an estimate ψ_0t”In the processing cycle t
Value ψ _0t(Step S20). To summarize the above,
After starting the system, in the first processing cycle t = 1,
Observed valueψ_0t′ Is the yaw angle of this processing cycle ψ_0tAnd Since
In the processing cycle of _0t'And the recurrence formula
Estimate ψ_0tCheck whether the difference from ″ is within a certain range.
And if it is within a certain range, the observed value 観 測_0t′ And estimated value
ψ_0tTake the weighted average with ″_0tWhen
If it is not within a certain range, it is not within a certain range
If is continuous for more than a certain number of processing cycles, the observed value
ψ_0t′ To the yaw angle ψ_0tState that is not within a certain range
If the number of processing cycles is not continuous, the estimated value
ψ_0t″ Is the yaw angle_0tAnd

Using the yaw angle ψ _0t obtained as described above, the x coordinate of the road vanishing point is corrected by the following equation (38) (step S21). x ₀ = (ψ _0t + ψ ₁ ) F (38) In the above-described simple vanishing point calculation processing, if the left and right two road parallel lines are not found, the observed value of the yaw angle ψ _0t ′
Can not ask. Therefore, if the two left and right road parallel lines have not been determined, it is determined whether or not the adopted value ψ _0t-1 of the yaw angle in the preprocessing cycle has been determined (step S).
7) If obtained, the adopted value 採用_0t of the current processing cycle is obtained by the recurrence formula of the following formula (39) (step S8).

Ψ _0t = ψ _0t-1 + Δψ _0t (39) If the direction of the road changes, the above equation does not hold. Instead, the difference between the adopted value obtained in the preprocessing cycle and the observed value is calculated. Any of the following recurrence formulas (40) (41) modified using
Is used as described above. ψ _0t = ψ _0t-1 + Δψ _0t- (ψ _0t-1 '-ψ _0t-1 ) (40) ψ _0t = ψ _0t-a + Δψ _0t- (ψ _0t-a ' -ψ _0t-a ) (41) If the adopted value ス_{テ ッ プ 0t-1} of the yaw angle in the preprocessing cycle has not been determined in step S7, the process proceeds to step S22. In step S22, the process ends without calculating the yaw angle ψ _0t .

Next, the process of calculating the pitch angle of the vehicle-mounted camera will be described with reference to a flowchart (FIG. 9). In this process, when it is determined that the vehicle is not traveling on a curve after the system is started up, a simple calculation process of a vanishing point is performed based on the image data of the image captured by the on-vehicle camera 11, and the left and right are calculated. The candidate points of the road parallel line are searched out, and their inclinations are checked to determine whether two left and right road parallel lines are obtained (step S32). When the two road parallel lines on the left and right sides are obtained, the vanishing point of the road (x ₀ , y ₀ ) is obtained from the state of the straight lines, and the following equation (42) y ₀ = − (θ ₀ + θ ₁ ) F = − The pitch angle θ of the vehicle-mounted camera with respect to the road is determined using θF (42) (step S33). Pitch angle theta ₁ with respect to the vehicle body at this time vehicle camera, since a need in the initial processing, as described above, it is possible to know the pitch angle theta ₀ with respect to the road of the vehicle by obtaining the pitch angle theta. This pitch angle θ ₀
Is a pitch angle observed from the imaging screen in the processing cycle t, and is hereinafter written as θ _0t .

If only one of the left and right road parallel lines is obtained (step S34), the road vanishing point (x ₀ , y ₀ ) cannot be obtained, so the processing shown in FIGS. The x-coordinate of the road vanishing point is calculated in reverse using the obtained adopted value 採用_0t of the yaw angle of the vehicle (step S3).
5). Then, the y coordinate of the road vanishing point is estimated from the x coordinate value x ₀ and one road parallel line, and the pitch angle θ _0t is obtained (step S36).

The calculation method will be described in detail with reference to FIGS.
X _0t is obtained from the following equation x _0t = (ψ _0t + ψ ₁ ) F (43) using the adopted value of the yaw angle ψ _0t of the vehicle obtained in the above processing, and the y-axis having the x _0t as the x coordinate is obtained. The intersection of the parallel straight line and the one road parallel line is defined as a road vanishing point (x _0t , y _0t ). Then, the pitch angle θ _0t is determined by applying the y coordinate value y _0t to the expression y _0t = − (θ _0t + θ ₁ ) F (44).

Then, step S33 or step S3
It is checked whether or not the pitch angle θ _0t obtained in 6 is within the standard deviation from the average value (= 0) of the pitch angle θ ₀ obtained in the initial processing (step S37). If the pitch angle is within the range, the obtained pitch angle θ _0t is set as the adopted value of the pitch angle (step S38). If not, it is determined that the obtained pitch angle θ _0t is not reliable, and the adopted value of the pitch angle is set to 0 (step S39). Explaining the reason why it is set to 0, as described above, when the vehicle pitches, it is time to accelerate or decelerate. However, even during acceleration or deceleration, it is not considered that only acceleration only decelerates, and the average value of the pitch angle is always 0. Because it can be seen that there is.

If neither the left nor the right road parallel line is obtained, that is, if "NO" is determined in the step S34, the adopted value of the pitch angle is set to 0. Next, a description will be given of a process of calculating the lateral displacement distance of the vehicle-mounted camera. FIG.
It is a flowchart for demonstrating the calculation process of the lateral shift distance of a vehicle-mounted camera. In this process, if it is determined that the vehicle is not traveling on a curve after the system is started, a simple road vanishing point calculation process is performed based on the image data of the image captured by the on-vehicle camera 11, and Then, the candidate points of the road parallel lines are searched for, and their inclinations are checked to determine whether two road parallel lines on the left and right sides are obtained (step S42). If two road parallel lines are obtained, the expression A _1t corresponding to the above-mentioned expressions (29) and (30) is obtained using the inclinations a _1t and a _{2t of} the road parallel lines and the mounting roll angle φ ₁ of the vehicle-mounted camera. = − (A _1t + φ ₁ ) h / (1−φ ₁ a _1t ) (45) A _2t = − (a _2t + φ ₁ ) h / (1−φ ₁ a _2t ) (46) The lateral displacement distance A of the vehicle-mounted camera from a predetermined reference line can be obtained (step S4).
3). However, the vehicle has ignored the roll angle φ ₀ is because it is straight running.

Further, it is determined whether or not the lateral deviation distance At _-1 between the two parallel roads has been determined in the previous processing cycle (step S44). If it has been obtained, before lateral distance A following recurrence formula using _t-1 A _1t = A ₁ processing _{cycle, t-1 + Lψ 0t (} 47) A 2t = A 2, t-1 + Lψ 0t ( Request lateral distance a _t now processing cycle by 48) (step S45). Here, L is the distance traveled by the vehicle in one processing cycle, and is determined based on the support data from the position detection processing device 2. ψ _0t is the adopted value of the yaw angle of the vehicle with respect to the road.

Then, it is compared with the lateral displacement distances A _1t and A _2t obtained from the road parallel lines in step S43 (step S4).
6) It is determined whether it is within a certain range (step S47). If it is within a certain range, the lateral shift distances A _1t and A _2t obtained from the road parallel line in step S43 are adopted (step S49).
The lateral displacement distances A _1t and A _2t obtained from the recurrence formula are adopted (step S48).

If only one road is parallel to the road in step S50
If it is possible to calculate the line,
And the inclination a of the road parallel line_1tAnd rolls for mounting on-vehicle cameras
Angle φ ₁Using, A_1t=-(A_1t+ Φ₁) H / (1-φ₁a_1tThe lateral displacement distance A is obtained by (49) (step S51). So
And the lateral displacement distance A of one road parallel line in the previous processing cycle
_t-1Is determined (step S5).
2). If found, the lateral displacement distance A of the previous processing cycle
_t-1And the following recurrence formula A_1t= A_{1, t-1}+ Lψ_0t According to (50), the lateral shift distance A of the processing cycle is now_t(Step
S53).

Then, it is compared with the lateral displacement distances A _1t and A _2t obtained from the road parallel lines in step S51 (step S5).
4) It is determined whether it is within a certain range (step S55). If it is within a certain range, the lateral shift distance A _1t obtained from the road parallel line is used in step S43 (step S57), and if it is not within the predetermined range, the lateral shift distance A _1t obtained from the recurrence formula is used in step S53. (Step S56).

Although not shown in the flowchart,
For the other road parallel line, the lateral shift distance A is directly
Since it cannot be obtained, the lateral shift distance in the previous processing cycle
Judge whether A is found, and if it is found,
Side shift distance A of previous processing cycle _t-1And the following recurrence formula A_t= A_t-1+ Lψ_0t According to (51), the lateral shift distance A of the processing cycle is_tAsk for. Previous processing
If the lateral displacement distance A is not found in the cycle,
For one road parallel line, determine the lateral displacement distance A
Give up and end the process.

If no parallel road is obtained in step S50, the flow advances to step S61 in FIG.
In the previous processing cycle, the lateral displacement distance A _{1, t-1 of one} road parallel line
Is determined, the lateral displacement distance A is calculated using the recurrence formula if the road parallel line has been determined, and if not, the lateral displacement distance A is not calculated (steps S61-S63).

[0092] Also, regarding another one road parallel line,
It is determined whether or not the lateral displacement distance A2 _{, t-1} has been determined. If the road parallel line has been determined, the lateral displacement distance A is calculated using the recurrence formula. If not, the lateral displacement distance A is not calculated. (Steps S64 to S66). By performing the processing described above, the posture parameters θ, φ, and ψ of the vehicle-mounted camera 11 and the lateral displacement distance A are obtained. These posture parameters are used in an object tracking recognition process described below. (6) Object Tracking Recognition Processing The object tracking recognition processing is performed by a certain type of object such as a road sign, a display on a road, a pedestrian, or another vehicle in an image captured by the vehicle-mounted camera 11 (hereinafter, “recognition”). Object"
This is a process for following recognition. In this image recognition, the posture parameters of the vehicle-mounted camera and the support information given from the position detection processing device 2 are used.

In the following, first, the following recognition of a still object to be recognized such as a road sign or a display on a road will be described, and then the following recognition of a moving object such as another vehicle will be described. Note that when following the recognition target, attention is paid to one or more points specified for the recognition target (hereinafter, referred to as “specific points”). (6-1) Tracking recognition of a still recognition target object The above equations (13) and (14) used in the description of the camera posture parameter calculation process are modified to obtain points (x ′,
y '), the camera attitude parameters ψ, φ, θ and the variable Z are known and the following equation (61) is obtained by solving for X and Y.

[0094]

(Equation 7)

Therefore, in the processing cycle t, the coordinates (x _t ′, y _t ′) of the specific point are obtained, and the camera attitude parameters and the position Z (constant value. If it is known that the height of the ground point is -h, which is lower by the height of the camera), the position ( _Xt , _Yt ) of the specific point viewed from the road is calculated by the above equation (61). You can ask.

Incidentally, each variable or parameter is a function of time, and changes with time. The traveling distance data of the vehicle between the processing cycle t and the processing cycle t + 1 is represented by L _{t + 1} ,
Assuming that the azimuth change data (when the azimuth change data is provided from the position detection processing device 2, it can be used as it is) Δψ _{t + 1} , the position (X _{t + 1)} of the specific point in the processing cycle _{t + 1} , Y _{t + 1} ) is given by the following equation.

[0097]

(Equation 8)

The terms L _{t + 1} sin Δψ _{t + 1} , −L _{t + 1} cos Δψ _{t + 1} of the vector on the right side of the equation (62) indicate that the distance between the object to be recognized and the camera is Shows how close you are, 2x on the right
The two matrices represent the rotation of the camera's field of view based on changes in the camera's yaw angle.

The (X _{t + 1} , Y _{t + 1} ) of the equation (62) and the camera attitude parameters ψ _{t + 1} , φ _{t + 1} , θ _{t + 1} are obtained by the above-mentioned _equation (13).
By substituting into the equation (14), the position (x _{t + 1} ′, y _{t + 1} ′) of the specific point on the screen in the processing cycle t + 1 can be estimated. x _{t + 1} ′ = F (X _{t + 1} + ψ _{t + 1} Y _{t + 1} −φ _{t + 1} Z) / (− ψ _{t + 1} X _{t + 1} + Y _{t + 1} + θ _{t + 1} Z) (63 ) y _{t + 1} ′ = F (φ _{t + 1} X _{t + 1} −θ _{t + 1} Y _{t + 1} + Z) / (− ψ _{t + 1} X _{t + 1} + Y _{t + 1} + θ _{t + 1} Z) ( 64) That is, if the position of the specific point in the processing cycle t is known, it is possible to estimate to which position on the screen the specific point moves in the processing cycle t + 1 based on the camera posture parameter and the like. Therefore, when following a certain recognition target from a screen that is imaged every moment, an image of an area around the estimated position is cut out, and a specific point is searched for therein. Since it is sufficient to perform the recognition processing of the recognition target according to the above, the recognition processing time can be saved.

Image cut-out range around estimated position
The estimated value, which is the estimated position on the screen, is
Considering the error with the actual value that is the position of the identified specific point
Determined. Specifically, first, in the processing cycle t,
Estimate (^Ex_t′,^Ey _t′) And the actual value is (
^Mx_t′,^My_t'). In addition, the average operation is av
e, variance operation is represented by var, and standard deviation is std.
When the square root operation is expressed as sqrt, the following equation is obtained.
Can be

[0101] var (x) = ave (x -ave (x)) 2 std (x) = sqrt (var (x)) x = E x t '- M x t' var (y) = ave (y- ^{ave (y)) 2 std (} y) = sqrt (var (y)) y = E y t '- M y t' Therefore, the error of the x-axis and y-axis position on the screen, for example, k Where k · std (x) and k · st
d (y). Therefore, if this range (hereinafter referred to as “error range σ”) is set as an image cutout range, an image including a specific point can be cut out.

If the constant k is variably set based on the running speed of the vehicle, the change in azimuth, the reliability of the posture parameter, and the like, the error range σ can be set more appropriately. Specifically, the traveling speed of the vehicle is a constant value (for example, 10
0 km / h) or less, the constant k may be set to be constant, and if it is more than 0 km / h, the constant k may be set so as to be substantially proportional to the running speed of the vehicle. Further, when the amount of change in the heading of the vehicle per unit time or per unit traveling distance is equal to or more than a fixed value (for example, 0.1 °), the value of the constant k is set so as to be substantially proportional to the amount of change in heading. May be. Further, the error range σ may be changed for each type of the recognition target to be recognized, or a common error range σ may be used regardless of the type of the recognition target.

It is assumed that Equations (61) , (62), (63), and (64) include errors, and a Kalman filter or the like is used to estimate the mileage, the azimuth change error, and the attitude parameter estimation error. From the difference between the estimated value of the position (x ', y') on the screen and the actual value, etc., the position (X, Y, Z) of the specific point can be estimated every moment by filtering. FIG. 12 illustrates an error range σ ₁ near the specific point P _t (x _t , y _t ) in the processing cycle t of the road sign, which is the object to be recognized for stillness, and the estimated position in the next processing cycle t + 1. When the next processing cycle t + 1 actually occurs, the specific point P _{t + 1} (x _{t + 1} ′, y _{t + 1} ′) becomes
As shown in FIG. 7, it can be found within the error range σ ₁ .

In this way, for example, when a still object to be recognized, such as a road sign or a traffic light, is to be tracked and recognized every moment, the search range is limited by the above-described method.
The tracking process of the recognition target can be speeded up. (6-2) Tracking recognition of a movement recognition target When tracking and recognizing a movement recognition target, the (62)
Expression conversion formula cannot be used. Therefore, instead of the above equation (62), the following equation (65) taking into account the temporal change of the position of the movement recognition target object is used.

[0105]

(Equation 9)

[0106] (65) If the comparison with the (62) formula formula, each vector of the right side _{(X t -X t-1)} -L t sinΔψ t, (66) (Y t -Y t-1) + L _{The term t} cosΔψ _t (67) is added. (X _t -X _t-1) and (Y _t
−Y _t−1 ) is the apparent moving distance of the recognition target object viewed from the vehicle from the previous processing cycle to the current processing cycle, and L _t sin Δψ _t , −L _t cos Δψ _t is the current distance from the previous processing cycle. Thus, the difference between these two equations (66) and (67) is the net movement distance of the recognition target object from the previous processing cycle to the current processing cycle.

By the way, in order to obtain the position of the object to be recognized in the next processing cycle, the equation (65) indicates that the moving distance of the object to be recognized from the previous processing cycle to the current processing cycle is determined by the current processing cycle. It is considered that the distance is equal to the movement distance of the recognition target object from the first processing cycle to the next processing cycle, and the formula takes into account this movement distance. Therefore, if the temporal change of the position of the recognition target object from the previous processing cycle to the current processing cycle is known, (65)
The recognition target object can be tracked and recognized using the equation. It is considered that the movement distance of the recognition object from the previous processing cycle to the current processing cycle is equal to the movement distance of the recognition object from the current processing cycle to the next processing cycle. Although it is assumed that the speed of the object does not suddenly change, this assumption is sufficiently satisfied if the movement recognition target is a vehicle.

FIG. 12 is a view showing a vehicle which is a moving recognition object ahead.
Specific point O in processing cycle t_t(X _t, Y_t) And its
Error range σ in next processing cycle t + 1_TwoIs also shown
Therefore, when the next processing cycle t + 1 actually occurs, the specific point O
_{t + 1}Is the error range σ as shown in FIG._TwoFind in
Indicates that it can be done. (6-3) Processing of the recognition processing unit Next, the recognition processing unit 15 performs tracking recognition of the recognition target object.
The processing to be executed will be described.

The recognition processing unit 15 stores a recognition target object registration table (see Table 1) and a specific point registration table (see Table 2) for registering a recognition target object in the storage unit 17 for follow-up recognition processing. have.

[0110]

[Table 1]

The recognition object registration table in Table 1 captures an image of the front or rear of the vehicle by the on-board camera 11, converts the image into an image signal in each processing cycle, and is recognized based on the converted image. A table for registering one or a plurality of recognition targets, which includes items of a recognition target number, presence / absence of registration, pattern information, movement information, the number of specific points, and a pointer to a specific point. ing.

The "recognition object number" is given a sequential number 1, 2, 3,..., And each number corresponds to each recognition object. “Pattern information” includes the type of the recognition target (passenger car, bus, road sign, pedestrian, etc.), size (width, total length, height, etc.), color tone inside the recognition target (for example, included in the recognition target). RGB primary color image data r, g, b of all the pixels or representative pixels determined from the geometric relationship such as the center of gravity of the contour image, or their average value. May be recorded), and indicates the characteristics of the recognition target object. This feature
It is obtained by pattern recognition of the recognition target. For pattern recognition, for example, a number of standard templates are prepared, the scale is adjusted to the size of the actual object to be recognized based on the position information of the specific point of the object to be recognized, etc., and virtually set at the two-dimensional position on the screen. By superimposing the template on the outline of the object to be recognized, quantitatively measuring the degree of coincidence between the two, and selecting the one that matches the most (see; Makoto Nagao, “Pattern Information Processing”, section 4.3, Corona Company March 10, 1983
First edition published on each day). When color data is used, not only density but also chromaticity data can be used, so that the recognition probability can be increased.

[0113] "Motion information" is three kinds of information for distinguishing whether a recognition target is stationary, moving, or unknown. The “number of specific points” refers to the number of specific points set for the recognition target. Here, a method of selecting a specific point will be described. For example, when a forward image as shown in FIG. 14 is obtained, this image is processed to extract a contour according to the size of the recognition target as shown in FIG. Then, any point on the contour, for example, a corner point of the contour is set as a specific point. As the contour, a boundary between the recognition target and the outside world, a boundary inside the recognition target (for example, a boundary between glass and body), a contour of a shadow of the recognition target, and the like can be selected. In particular,
The image in FIG. 14 is scanned to find points where the color tone (for example, RGB average value) changes for each scanning line, and an outline is extracted by connecting those points. It is good to set.

Note that the specific point is not limited to the corner of the contour as described above, but may be a center within the contour of the object to be recognized (for example, the center of a lamp or a license plate for a vehicle, a sign for a road sign). At the center). However, when the object to be recognized is another vehicle ahead, setting the two lower corner points of the tire contour has the advantage that the height becomes a known value (-h).

"Pointer to specific point" refers to a pointer to a specific point registration table.

[0116]

[Table 2]

The specific point registration table of Table 2 is a table that stores detailed information on specific points set for the recognition target object.
Each item includes a spatial position (X, Y, Z) of a specific point, a spatial position change, and features of a peripheral image of the specific point. The “spatial position (X, Y, Z) of the specific point” refers to the spatial position coordinates (X, Y, Z) of the specific point in the road vehicle coordinate system XYZ obtained by the above equation (61).

"Spatial position change" refers to a change in the spatial position of a specific point viewed from the road as compared with the screen in the previous processing cycle, and a change (ΔX, Δ
Y, ΔZ) and is obtained by subtracting the travel distance of the vehicle. When the change in the spatial position is unknown, for example, all bits are registered as 1 to indicate the unknown. The “characteristics of the peripheral image of the specific point” is information for confirming the identity between the specific point on the screen in the previous processing cycle and the specific point on the screen in the main processing cycle. FIG. 16A shows a specific point Q at the end of the left rear wheel of another vehicle.
₁ and a certain range D _1, and a diagram showing a specific point Q ₂ and a certain range D ₂ in the left backlight, Fig (b)
Is a view cut out a specific point to Q ₁ in a certain range D ₁ and this, FIG. (C) are views cut out a specific point Q ₂ in a certain range D _2, and this. Comparing FIGS. 7B and 7C, it can be seen that the specific points Q ₁ and Q ₂ are not the same because the features of the peripheral images at the specific points are clearly different in terms of color tone and the like.

Thus, as information for confirming the identity of a specific point, a representative point determined from geometrical relationships such as all pixels included within a certain range D from the specific point or the center of gravity of the contour image is used. The three primary color image data r, g, b of RGB of the pixel are adopted. However, it is not limited thereto, three primary color image data r of these pixels, g, b of the surface integral value _{∫ D rdS, ∫ D gdS,} ∫ D bdS (∫dS
Represents an area). Also, their linear sum _{P P = j 1 | ∫ D} rdS | + j 2 | ∫ D gdS | + j 3 | may be represented by | ∫ _D BDS. When the specific point is on the contour, the three primary color image data r, g, b of the pixels constituting the contour
, May be represented by a line integral value of the three primary color image data r, g, b of those pixels, or may be represented by a linear sum thereof. When a point inside the contour is selected as a specific point, the three primary color image data r, which are included within a certain range from the specific point and are inside (not outside) the contour.
It may be represented by g, b, may be represented by an area value of the three primary color image data r, g, b of those pixels, or may be represented by a linear sum thereof. If these pieces of information match or differ from each other and are within the allowable range, it can be determined that the specific point on the screen in the previous processing cycle and the specific point on the screen in this processing cycle are the same specific point. .

Next, referring to the flowcharts of FIGS. 17 to 21, the procedure of the tracking recognition processing of the recognition target will be described. This process is performed every time the vehicle travels for a certain distance or for a certain time during the running of the vehicle after the completion of the initial processing. First, image data captured by the in-vehicle camera 11 is acquired (FIG. 17, step S71), and the posture parameters of the in-vehicle camera are calculated as already described in “(5) Attitude parameter calculation processing of in-vehicle camera” (step S71). S7
2).

Then, the recognition target registration table in Table 1 and the specific point registration table in Table 2 are referred to. First, it is checked whether at least one recognition target is registered in the recognition target registration table (step S73). If the recognition target is registered, the process proceeds to step S81 and subsequent steps. If it is the first acquisition screen, nothing is registered. In that case, the process proceeds to step S75, where the contours of all the images in the screen are extracted based on the image data, and the image patterns are respectively recognized. In this case, the object to be recognized may be limited depending on the purpose of guiding the vehicle. For example, when the purpose is to operate the vehicle safely, only the movement recognition target object may be set as the recognition target. Alternatively, the route guidance information is displayed on the display unit 29 of the position detection processing device 2 so that the route to the destination is displayed. When performing guidance, only stationary recognition targets such as road signs and traffic lights may be set as recognition targets.

If a car, a road sign, a pedestrian, and the like can be respectively recognized, a sequence number is assigned to each recognition target, and a sign of “registered” is entered in the column of “registration” in the recognition target registration table. Write. Furthermore, the pattern information is written (step S76), and the movement information is registered as "unknown" (step S77). Further, specific points are set based on the outline of the extracted image, and the number is registered (step S78).

Next, the spatial position coordinates (X, Y,
Z) and the "characteristics of the peripheral image of the specific point" are registered (step S79), and the spatial position change of the specific point compared with the previous processing cycle is registered as unknown (step S80), and the process returns to the start. The reason why the spatial position change is unknown is that the spatial position change cannot be obtained because no image data has been obtained in the previous processing cycle.

In the next processing cycle, if the object to be recognized is registered, steps S74 to S81 are performed.
To check the movement information in the recognition object registration table. First, reference is made to a recognition target whose movement information is “still”. FIG. 18 is a flowchart for processing a recognition target whose moving / moving information is “still”. According to this processing, a recognition target whose moving / moving information is “still” is referred to (step S91). The registration flag is turned off (step S93). This registration flag is a flag indicating whether or not a specific point in the recognition target object can be followed. That is, if at least one specific point in the recognition target can be followed between the previous processing screen and the current processing screen, the registration flag is turned on for the recognition target.

Then, for the referenced recognition target, a specific point is referenced using a specific point registration table (step S94). If there is a specific point, the position (X, Y,
Z) is estimated, and based on this, the position (x ′, y ′) of the specific point on the screen in the main cycle is calculated using the above-described formulas (63) and (64).
It is estimated according to the equation (step S96). Then, the recognition target is recognized from the new image data, the pattern information is checked, the specific point is checked, and the feature of the peripheral image is checked (step S97). In this case, the processing time can be shortened by using only the outline information of the recognized symmetric object whose registration has been confirmed and the image data in which the inside thereof has been masked to check the specific point. Then, referring to the reliability of the traveling speed, the change in the traveling direction, and the attitude parameter of the vehicle, the magnitude of the error range σ of the estimated position is obtained as described above, and whether the specific point exists in the error range σ A search is made as to whether or not the feature of the peripheral image has been changed (step S98). In this search, the search range is set to the error range σ
, There is an advantage that the search time for a specific point can be shortened. Here, a check criterion of the feature of the peripheral image is described. When a specific point is set on the contour of the image of the recognition target object, the directions of the contours are almost the same, and the images on both sides of the contour are The features are almost the same. When the specific point is set on one side of the outline of the image of the recognition target, the features of the image on one side of the outline are almost the same.

If there is a specific point having the same characteristic of the peripheral image or a specific point having the characteristic of the peripheral image within the allowable range (step S99), the registration flag is turned on (step S100), and the specific point is set. Spatial position coordinates (X,
(Y, Z), spatial position change, and features of the peripheral image are updated to new values on the specific point registration table (step S10).
1). Then, it is checked whether or not the change in the spatial position of the specific point falls within a certain range (step S10).
2). This fixed range is for determining whether the recognition target is moving or stationary, and for example, X, Y, Z
It can be a sphere or spheroid centered on the origin of space. The most appropriate value for the radius of the sphere or spheroid may be obtained by actually running the vehicle and performing this processing for various recognition targets. If not, the motion information of the recognition target object is changed to “movement” (step S103).

If the specific point does not exist in the error range σ in step S99, the spatial position coordinates and the like of the specific point are not updated on the specific point registration table, and the process proceeds to step S9.
4, but instead, as long as the position (x ', y') of the searched specific point exists in the screen, the position (x ', y') is used to determine spatial position coordinates and the like. , The content of the specific point registration table may be updated.

If the specific point does not exist in the error range σ in step S99, the spatial position coordinates and the like of the specific point are not updated on the specific point registration table, and the process proceeds to step S9.
4, but instead of this, the equations (63) and (64)
As long as the position (x ', y') of the specific point estimated according to the equation exists in the screen, the position (x ', y') is regarded as the position of the specific point, and the position is used as a spatial position coordinate. , The content of the specific point registration table may be updated.

In step S98, when searching for a specific point in the error range σ, if the error range σ itself is masked behind another recognition target, the error range σ Since no specific point is found even when the search is performed, it is not necessary to search within the error range σ.
It is desirable that the position (x ', y') of the specific point estimated according to the expression (4) is regarded as the position of the specific point, and the tracking is continued. The fact of whether or not the mask has been made can be determined using the spatial position coordinates (X, Y, Z) of the specific point registration table.

When the above processing has been performed for all the specific points of the recognition object, the flow advances to step S104 to check whether or not the registration flag is turned on (step S1).
04) If it is turned on, the pattern information of the recognition target is checked to determine whether the type or color tone of the recognition target matches the pattern information in the recognition target registration table (step S105). If they match or fall within the permissible range, it is determined that the tracking of the target object has succeeded, and the process returns to step S91 to repeat the same process for another stationary target object.

If the registration flag is not turned on,
In the new image, it is determined that the specific point of the object to be recognized did not exist at all in the error range σ of the estimated position. Therefore, it is determined that the object to be recognized has been lost, and the registration is deleted (step S106). . Also, when the difference in the pattern information of the recognition target object does not fall within the allowable range, it is determined that the recognition target object has been lost, and the registration thereof is deleted (step S106).

When the processing has been completed for all still recognition objects, the flow proceeds to the flow of FIG. 19 for processing the recognition objects whose movement information is “movement”. According to this processing, each recognition target whose movement information is “movement” is referred to (Step S).
111), if there is a movement recognition object, the registration flag and the movement flag are turned off (step S113). This movement flag is a flag indicating whether the recognition target is moving or stationary.

With respect to the object to be recognized, a specific point is extracted with reference to the specific point registration table (step S114). If there is a specific point, according to the above equation (65), two
The dimension position is estimated (step S116), the recognition target is recognized from the new image data, and the pattern information, the specific point,
The feature of the peripheral image is checked (step S117), and it is searched whether the specific point is within the error range σ around the estimated point (step S118). In this search, since the search range is limited to the error range σ, there is an advantage that the search time can be shortened. If there is a specific point in the error range σ where the features of the peripheral images are the same but different but within the allowable range (step S
119), the registration flag is turned on (step S12)
0), the spatial position coordinates (X, Y, Z), spatial position change, and characteristics of the peripheral image of the specific point are updated to new data on the specific point registration table (step S121). Then, it is checked whether or not the change in the spatial position of the specific point is within a certain range, for example, within a sphere centered at the origin of the X, Y, and Z spaces (step S122). Turn on (step S12
3).

When the above processing is performed for all the specific points of the object to be recognized, the flow advances to step S124 to check whether or not the registration flag is turned on (step S124). The pattern information of the object is checked and compared with the pattern information of the recognition object registration table (step S125). If they are the same or similar, it is checked whether the movement flag is on (step S126). If the movement flag is not turned on, it can be determined that the recognition target object has not moved, so the movement information is changed to “still”, but if the movement flag is on, the recognition target object is still Since it can be determined that the object is moving, the movement information is not changed, and in any case, the tracking of the object to be recognized is determined to be successful, the process returns to step S111, and the same processing is repeated for the other object to be moved. .

If the registration flag is not turned on in step S124, it means that the specific point of the object to be recognized did not exist at all in the error range σ of the estimated position in the new image. It is determined that the object to be recognized has been lost, and its registration is deleted (step S128).
Also, even if the pattern information of the recognition target object is different,
It is determined that the recognition target object has been lost, and the registration is deleted (step S128).

When the processing has been completed for all the moving recognition objects, the flow moves to the flowchart of FIG. 20 for processing the recognition objects whose movement information is “unknown”. According to this process, each recognition target whose movement information is "unknown" is referred to (step S131). If there is any unknown recognition target, the movement flag is turned off (step S13).
3). Then, the temporary storage data (data for temporarily storing the spatial position and the like of the specific point) is cleared (step S13).
4) The presence / absence of a specific point is checked for the recognition target object with reference to the specific point registration table (step S135). If there is a specific point, it is assumed that the specific point is stationary, and the two-dimensional position of the specific point to be stationary is estimated by the equation (61) (step S137). Since it is assumed that the specific point is stationary, since this processing is performed on a recognition target whose movement information is `` unknown '', it is estimated by the above equation (61) that the specific point is stationary for the time being. This is because the position is determined and an attempt is made to follow the stationary recognition target object. Then, the recognition target is recognized from the new image data, the pattern information, the specific point, and the features of the peripheral image of the specific point are checked (step S138), and it is searched whether the checked specific point is within the error range σ (step S138). Step S139).

If there is a specific point in the error range σ where the features of the peripheral images are the same or very similar (step S
140), which means that the specific point at which the still point has been stopped has been successfully specified, and the spatial position coordinates (X, Y,
Z), changes in spatial position, and features of the peripheral image are temporarily stored in a predetermined buffer area in the storage unit 17 (step S).
141). Then, it is determined whether or not the change in the spatial position of the specific point falls within a certain range (step S14).
2) If not, the moving flag is turned on (step S143).

After performing the above processing for all the specific points of the recognition target, the process proceeds to step S144, where the pattern information of the recognition target is checked and compared with the pattern information of the recognition target registration table (step S14).
5). If they are the same or similar, the spatial position coordinates (X, Y, Z) of the temporarily stored specific point, the spatial position change, and the feature of the peripheral image are registered in the specific point registration table (step S145). Then, it is checked whether or not the movement flag is turned on (step S146), and if not, the movement information of the recognition target object is “still”.
(Step S147), and if it is turned on, the movement information of the recognition target object is set to “move” (Step S14).
8).

Further, when there is no specific point of the new image in the error range σ in step S140, or when the pattern information does not match in step S144, the registration of the object to be recognized is not deleted. The information of the recognition target is stored in the work area (step S149),
Proceed to the flowchart in FIG. In the processing of FIG. 21, the recognition processing (a kind of loser resurrection processing) is performed again on the recognition target stored in the work area.

The meaning of performing this processing will be described as follows. The processing target of this processing is step S in FIG.
At 77, the motion information is “unknown” and the recognition target object that has failed to follow the stationary recognition object at step S137 in FIG. That is, the recognition target object to be processed is a moving object, and the motion characteristics of the recognition target object are unknown. Therefore, it is not possible to use the above equation (65) in which the temporal change of the position of the movement recognition target is considered. Therefore, in the process of FIG. 21, it is determined that the error range σ is estimated and a search for a specific point is abandoned, and the search range is extended to the entire screen. As a result, even if the recognition target is at an unexpected position, it can be expected that the recognition target can be captured with considerable accuracy.

First, the data of the recognition target stored in the work area is referred to (step S151). Next, a recognition target in the screen is recognized, and a specific point is searched from the screen (step S153). Then, while checking the features of the peripheral image, it is determined whether or not all the specific points of the recognition target stored in the work area exist in the screen (step S154). If all the specific points of the recognition target exist (step S154), it is determined whether the pattern information of the recognition target matches or not (step S155). If the pattern information matches, it is determined that the tracking of the recognition target has succeeded, and the spatial position coordinates (X, Y, Z) of the specific point, the spatial position change, and the characteristics of the peripheral image are registered in the specific point registration table ( (Step S157), the movement information of the recognition target object is set to “move” (step S15).
8). The reason for moving is that if you ’re stationary,
This is because the tracking of the recognition target object must have succeeded in the processing of FIGS. 18 and 20, and only the case of movement can be considered. In addition, when even one specific point of the recognition target object is missing,
Alternatively, when the pattern information of the recognition target does not match, the registration of the recognition target is deleted in step S156. If even one specific point is missing, the registration is deleted because there is a high possibility that a recognition error will occur.

If the above-described processing is completed for all the recognition targets stored in the work area, it is searched whether there is any recognition target that has not yet been recognized in the image data (step S159). If there is a new object to be recognized (step S160), the flow advances to step S76 in FIG. 17 to write the pattern information (step S7).
6) Register the movement information as "unknown" (step S7)
7) Set a specific point and register the number (step S7)
8) The spatial position coordinates (X, Y, Z) of the specific point and the "characteristics of the peripheral image of the specific point" are registered in the specific point registration table (step S79), and the specific point is compared with the previous processing cycle. Register the spatial position change as unknown.

If there is no new object to be recognized (step S
160), and proceed to “return” in FIG.

[0144]

According to the invention of claim 1 Symbol placement as the foregoing, introduces the concept of a specific point corresponding to each recognition object existing around the vehicle, the spatial position of the specific point Is calculated in the memory, and a change in the spatial position of the specific point based on the traveling of the vehicle or the like is predicted. It was decided whether or not.

Therefore, the object to be recognized can be followed. Further, since it is sufficient to search for a specific point in a limited place within the error range on the screen, the processing time can be reduced. According to the fifth aspect of the invention, when predicting the spatial position of a specific point in a later processing cycle, the behavior of the recognition target is estimated, and then the change in the spatial position of the specific point is predicted. Therefore, even when the recognition target is moving, the recognition target can be efficiently followed.

According to the eighth aspect of the invention, when a specific point cannot be searched, it is determined whether or not the same specific point exists within a wider range than the error range. An increase can be prevented.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an image recognition processing system to which an embodiment of the present invention has been applied.

FIG. 2 is a diagram illustrating an example of an image captured by a vehicle-mounted camera.

FIG. 3 is a diagram extracting a straight line portion obtained by connecting straight line candidate points from the image of FIG. 2;

FIG. 4 is a diagram for explaining Hough transform.

FIG. 5 is a diagram illustrating a process for obtaining a road vanishing point by repeating a Hough transform process twice.

FIG. 6 is a diagram for explaining a simple calculation method of a road vanishing point.

FIG. 7 is a flowchart illustrating a yaw angle calculation process of the vehicle-mounted camera.

8 is a flowchart (continued from FIG. 7) for explaining the yaw angle calculation processing of the vehicle-mounted camera.

FIG. 9 is a flowchart illustrating a pitch angle calculation process of the vehicle-mounted camera.

FIG. 10 is a flowchart illustrating a lateral displacement distance calculation process of the vehicle-mounted camera.

11 is a flowchart (continued from FIG. 10) for explaining the lateral displacement distance calculation processing of the vehicle-mounted camera.

FIG. 12 is a diagram showing a specific point defined on a recognition object ahead and an error range σ near an estimated position thereof.

FIG. 13 is a diagram showing that a specific point is actually found in the error range σ in the next processing cycle.

FIG. 14 is a diagram illustrating an example of a front image captured by a vehicle-mounted camera.

FIG. 15 is a diagram illustrating a state in which the image of FIG. 14 is processed to extract a contour of the recognition target object, and a specific point is set at a corner or inside the contour.

[16] FIG. 16 (a) shows, the specific point Q ₁ and a certain range D ₁ of the end of the left rear wheel of the other vehicle, as well as the specific point Q ₂ and a certain range D ₂ in the left backlight FIG.
(b) is a view cut out a specific point to Q ₁ in a certain range D ₁ and this, FIG. (c) is a view obtained by cutting out specific point Q ₂ in a certain range D ₂ and this is there.

FIG. 17 is a flowchart illustrating a procedure of a tracking recognition process of a recognition target object.

FIG. 18 is a flowchart (continuation of FIG. 17) for explaining the procedure of the tracking recognition processing of the recognition target object.

FIG. 19 is a flowchart (continuation of FIG. 18) for explaining the procedure of the tracking recognition processing of the recognition target object.

20 is a flowchart (continued from FIG. 19) illustrating the procedure of the tracking recognition processing of the recognition target object.

21 is a flowchart (continuation of FIG. 20) for explaining the procedure of the tracking recognition processing of the recognition target object.

[Explanation of symbols]

 Reference Signs List 1 image recognition processing device 2 position detection processing device 11 in-vehicle color camera 13 image processing circuit 15 recognition processing unit 17 storage unit 21 distance sensor 22 direction sensor 25 position detection processing unit 27 road map memory 28 storage unit 29 display unit

Continuation of front page (58) Fields investigated (Int. Cl. ⁶ , DB name) H04N 7/18 B60R 21/00 B60K 31/00 G06T 7/20

Claims (9)

(57) [Claims] An apparatus for capturing an image of the periphery of a vehicle with a vehicle-mounted camera mounted on the vehicle and following and recognizing an object around a road on which the vehicle travels based on the captured image. Image processing means for capturing an image of the front or rear of the object and converting the digital image signal into digital image signals at each processing cycle. Based on the converted image obtained from the image processing means, an outline of one or a plurality of recognition targets in the screen is displayed. Extracting, corresponding to each recognition target, determining the coordinates of one or more specific points on the screen that the position of the recognition target is determined if the position is determined, and determining the coordinates of the image around the specific point. Registration means for registering in the memory together with the feature; spatial coordinates for determining coordinates on the screen and converting the coordinates of the specific point registered by the registration means into the spatial position of the specific point using the attitude parameter of the vehicle-mounted camera. conversion And a step, the spatial position of the specific point converted by the spatial coordinate conversion means, and the traveling distance data of the vehicle included in the output data of the position detection processing device that detects the position of the vehicle based on signals from various sensors. Estimating means for predicting the spatial position of a specific point in a later processing cycle from the azimuth data and estimating coordinates on the screen of the specific point corresponding to this position; and And determining means for determining whether or not the same specific point exists within an error range around coordinates on the screen estimated by the estimation means when the converted image is obtained. When it is determined that the specific point exists, the recognition target is followed by obtaining the spatial position of the specific point, and the same specific point does not exist for the recognition target by the determination unit. If it is constant,
A tracking and recognizing device for an object, wherein registration corresponding to the object to be recognized is deleted. 2. When a plurality of specific points are present in a recognition target object, and when the determination means determines that at least one specific point exists within an error range, the recognition target object is followed and the error is tracked. If it is determined that there is no any particular point in the range, follow-up recognition apparatus of claim 1 object, wherein the unregistering corresponding to the recognition object. 3. When a plurality of specific points are present in a recognition target, the recognition target is followed only when the determination means determines that all the specific points exist within the error range. either if the specific point is determined that there is no follow-up recognition apparatus of claim 1 object, wherein the unregistering corresponding to the recognition object within. 4. Registering pattern information for discriminating each object to be recognized in the registering means, and when the determining means determines that the same specific point exists, the pattern information registered in the registering means is further registered. It determines also consistent with the information, if it is determined that they do not match, follow-up recognition device of the object of claim 1, wherein the unregistering corresponding to the recognition object. 5. The method according to claim 1, wherein the estimating unit calculates a spatial position of the specific point based on a spatial position of the specific point converted by the spatial coordinate converting unit and a spatial position of the specific point determined in a preceding processing cycle. The position change is obtained, and if the change is within a certain range, the recognition target object relating to the specific point is determined to be stationary.If the change is outside the certain range, the recognition target object relating to the specific point moves. When it is determined that the recognition target object related to the specific point is moving, when determining the spatial position of the specific point in a later processing cycle, the position change of the specific point is determined. follow-up recognition device of the spatial position and requests the claim 1 object of the description of a particular point in consideration. 6. When a plurality of specific points are present in the object to be recognized, if the position change of at least one of the specific points is out of a predetermined range, the estimating means moves the object to be recognized at the specific point. The tracking and recognizing device for an object according to claim 5 , wherein the object is determined to be stationary, and if the position change is within a certain range for all the specific points, the object to be recognized is determined to be stationary. 7. When the spatial position of a specific point has not been determined in the previous processing cycle, the estimating means determines that the recognition target is stationary and determines the spatial position of the specific point. The apparatus for recognizing and following an object according to claim 1, wherein the apparatus is for obtaining the object. 8. When the determining means determines that the same specific point does not exist within the error range, the determining means determines the same specific point within a wider range exceeding the error range. the but whether present, follow-up recognition apparatus of claim 1 object according is to determine based on the feature of the image around the specific points registered. 9. The method according to claim 1, wherein when the determination unit determines that the same specific point does not exist for the recognition target within the wide range, the registration corresponding to the recognition target is deleted. Item 10. A tracking and recognizing device for an object according to Item 8 .