JP2005025497A - Sign recognizing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To execute processing to specify a solid object in a picked up image, and to recognize the contents of a sign to the specified solid object. <P>SOLUTION: A laser beam is transmitted from a radar 3, and reflected on a solid object, and the position of the solid object is detected based on the reflected light, and the position of the solid image on an image picked up by a camera 2 is specified based on the detected position of the solid object. Then, pattern matching processing to recognize the contents of a sign is carried out to the part of the position of the solid object in the specified image. Thus, it it not unnecessary to carry out the pattern matching to a paint display part on a road surface, and it is possible to reduce an arithmetic load. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a sign recognition device that recognizes the contents by detecting signs such as road signs on a road.
[0002]
[Prior art]
An apparatus that recognizes a road sign based on an image captured by a camera is known (see Patent Document 1). In this conventional sign recognition device, a sign is recognized by extracting a specific color of the sign from the captured image to obtain an extracted image, and performing pattern matching processing on the shape of the extracted image.
[0003]
[Patent Document 1]
Japanese Patent Laid-Open No. 9-185703
[Problems to be solved by the invention]
However, since there are cases where paint on the road is shown in the captured image, the conventional recognition apparatus extracts the paint portion on the road when the specific color of the sign is extracted from the captured image. In some cases. Therefore, pattern matching processing is also performed on a flat object such as paint on the road surface, and there is a problem that an expensive calculation device with a high calculation speed or a high calculation speed is required. .
[0005]
The present invention provides a sign recognition device that identifies a three-dimensional object in a captured image and performs processing for recognizing the content of the sign for the specified three-dimensional object.
[0006]
[Means for Solving the Problems]
The sign recognition device according to the present invention detects a position of a three-dimensional object based on a reflected signal that is sent back from a signal and reflected by an object, and detection on a captured image is performed based on the detected position of the three-dimensional object. The position of the identified three-dimensional object is specified, and processing for recognizing the content of the sign is performed on the position of the specified three-dimensional object in the image.
[0007]
【The invention's effect】
According to the sign recognition device according to the present invention, the three-dimensional object in the captured image is specified, and the processing for recognizing the contents of the sign is performed on the specified three-dimensional object. Therefore, the calculation load for recognizing the sign is reduced. Can be reduced.
[0008]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 is a diagram showing an embodiment in which a sign recognition device according to the present invention is mounted on a vehicle 1. The sign recognition device in one embodiment includes a camera 2, a radar 3, a reflection surface angle estimation device 4, and a processing device 5. The camera 2 is attached to the front center portion of the indoor ceiling 1a of the vehicle 1 and photographs the front of the vehicle through the front window 1b. The camera 2 is preferably a camera equipped with a CCD or CMOS sensor capable of high-sensitivity imaging. In addition, it is preferable that the optical axis of the camera 2 coincides with the straight traveling direction of the vehicle.
[0009]
The radar 3 is attached in front of the vehicle, emits laser light in front of the vehicle, and detects the distance to the target, the direction in which the reflected light returns, the reflection angle, and the reflection intensity. The distance to the target is calculated based on the time from when the laser light is emitted until the reflected light returns. FIG. 2 is a diagram showing a detailed configuration of the radar 3. The radar 3 includes a light projecting window 41, a light receiving window 42, a mirror 43, a motor 44, a light receiving element (PD) 45, a laser light emitting element (LD) 46, an amplifier circuit 47, and a laser light emitting element 46. An LD driving circuit 48 for driving, a motor driving circuit 49 for driving the motor 44, and a CPU 50 are provided.
[0010]
The laser light emitting element 46 transmits laser light in the near-infrared wavelength region based on a command from the LD drive circuit 48. The laser light transmitted from the laser light emitting element 46 is reflected by the reflection mirror 43 and emitted from the projection window 41. At this time, the motor drive circuit 49 drives the motor 44 to rotate the reflecting mirror 43 that rotates in the vertical and horizontal directions with a predetermined angular resolution, so that n beams in the horizontal direction and the vertical direction are within a predetermined angular range. M beams are sent out. The laser light emitted from the light projecting window 41 is reflected by a target such as a preceding vehicle and received by the light receiving element 45 through the light receiving window 42. The light reception signal received by the light receiving element 45 is amplified by the amplification circuit 47 and sent to the CPU 50.
[0011]
The CPU 50 controls the amplifier circuit 47, the LD drive circuit 48, and the motor drive circuit 49 so that the laser light is emitted from the laser light emitting element 45, and the intensity of the received light signal received by the light receiving element 45 is time-resolved. Record and calculate the distance, detection angle, and relative speed with the forward target. The CPU 50 is calculated based on the vehicle speed signal based on these calculation results, a vehicle speed signal output from a vehicle speed sensor (not shown), and a vehicle pitch angle calculated by the vehicle behavior estimation processing unit 20 described later. The height of the target is calculated by correcting the speed of the host vehicle and the target detection angle in the vertical direction. The CPU 50 also determines whether the detected target is a stationary object based on the speed of the host vehicle and the relative speed with the target.
[0012]
The calculation of the height of the target will be supplemented with reference to FIGS. FIG. 3 is a diagram illustrating a state in which three laser beams are transmitted in the vertical direction from the radar 3 when the vertical pitch angle of the radar 3 is zero. From this state, as shown in FIG. 4, when the radar 3 is tilted upward, the area detected by the three beams also changes. Accordingly, it is necessary to correct the height of the target detected by each beam based on the vertical inclination (vehicle pitch angle) of the radar 3, and the CPU 50 calculates the corrected height.
[0013]
The configuration of the radar 3 will be supplemented. A light receiving lens (not shown) is incorporated in the light receiving window 42, and light condensed by the light receiving lens is input to the light receiving element 45. A light projection lens (not shown) is provided in front of the laser light emitting element 45, and the laser light transmitted from the laser light emitting element 45 is spread at a predetermined angle via the light projection lens. Since a light projection lens (not shown) is also incorporated in the light projection window 41, the laser light transmitted from the laser light emitting element 45 is transmitted after being narrowed to a predetermined angle by the light projection lens of the light projection window 41. . The predetermined angle widened by the light projecting lens is determined based on the area size for scanning, the required resolution, the motor performance, the required size of the apparatus, and the like. With this structure of the radar device 3, the laser light projected from the projection window 41 can be scanned with a thick laser beam having a certain area, and a laser beam resistant to dirt and rain can be obtained. .
[0014]
FIG. 5 shows the intensity distribution of the laser beam. FIG. 5A is a diagram showing a distribution in which the above-mentioned laser beams having a certain area have the same intensity, and FIG. 5B is a cross section taken along the center line of FIG. It is a figure which shows the intensity | strength in. As shown in FIG. 5B, the central portion of the laser beam has the strongest intensity.
[0015]
The reflection surface angle estimation device 4 estimates the angle (reflection surface angle) at which the target (marker) detected by the radar 3 is facing the host vehicle. A method of estimating the reflection surface angle will be described. FIG. 6 is a diagram illustrating a state in which laser light is irradiated from the radar 3 to the target. As for the laser light irradiated to the target, scattered light is observed unless the radar 3 and the target are facing each other. Assuming that the target is a complete scattering surface, the intensity I of the observed laser light (reflected light) is expressed by the following equation (1) from Lambert's cosine law.
I = Kd × Ii × cos α (1)
Where Kd is the scattering reflectance, Ii is the intensity of the target incident light, and α is the angle formed between the normal of the target and the incident laser light, as shown in FIG. The perfect scattering surface means a surface having the same brightness when viewed from any direction when the reflected light is viewed.
[0016]
Here, consider a case where a target is detected on the right side of the radar 3 as shown in FIG. In this case, the laser light emitted from the left side of the radar 3 hits the left side of the target and is reflected back, whereas the laser light emitted from the right side is reflected and returned from the right side of the target. come. Since the target is located on the right side of the radar 3, the light reflected on the left side of the target returns faster than the light reflected on the right side of the target. In addition, since the area irradiated on the right side of the target is wider than the area irradiated on the left side, the time when the intensity of the reflected light decreases decreases the light that is reflected back on the right side of the target. Is longer.
[0017]
FIG. 7 is a diagram illustrating a temporal change in the intensity of light that is reflected back from the target. Assuming that the time required for the intensity of the reflected light to reach a maximum value from a predetermined threshold value (for example, 10% of the maximum value) is t1, and the time required for the reflected light intensity to decrease from the maximum value to the predetermined threshold value is t2. When the target is located on the right side of the radar 3, the relationship of t1 <t2 is established for the reason described above. The ratio between t1 and t2 is determined by the angle α indicating the position of the target with respect to the radar 3. Therefore, by measuring t1 and t2 and calculating the ratio between them, the reflection surface angle of the target can be calculated. The relationship between the ratio between t1 and t2 and the reflecting surface angle of the target is obtained in advance through experiments or the like.
[0018]
FIG. 8 is a diagram showing a state in which laser light is emitted from the radar 3 and reflected light is received when there are three targets. The target 1 is present at the position closest to the radar 3, and the target 3 is present at the position farthest from the radar 3. FIG. 9 is a diagram illustrating a temporal change in the intensity of the light reflected and returned by the targets 1 to 3 shown in FIG. As shown in FIG. 9, for the target 1, the above-described times t1 and t2 are represented as t1_1 and t2_1, for the target 2, t1_2 and t2_2, and for the target 3 as t1_3 and t2_3, respectively.
[0019]
As shown in FIG. 9, as the target position is further away from the radar 3, the ratio between the times t1 and t2 becomes closer to 1: 1. The sum (t1 + t2) of the times t1 and t2 depends on the spread angle of the laser light and the distance between the radar 3 and the target. Therefore, although the incident angle α of the laser beam with respect to the target 3 located far away is small, the distance from the radar 3 is long, so the size of t1_3 + t2_3 is not necessarily shorter than t1_1 + t2_1.
[0020]
The relationship between the magnitudes of the times t1 and t2 and the ratio between t1 and t2 is the same when the target is on the left side of the radar 3. Which side of the radar 3 is present is determined based on the direction in which the reflected light returns.
[0021]
FIG. 10 is a diagram illustrating a detailed configuration of the processing device 5. The processing device 5 includes an image memory 10, a vehicle behavior estimation processing unit 20, and a sign detection unit 30. The image memory 10 stores images captured by the camera 2 every predetermined time (for example, 16 msec).
[0022]
The vehicle behavior estimation processing unit 20 includes a white line detection unit 21, a vehicle behavior estimation unit 22, and a vehicle traveling direction estimation unit 23 in terms of processing functions performed internally. The white line detection unit 21 detects a white line on the road based on the image stored in the image memory 10. The detection of the white line is, for example, a method of recognizing the position of the feature pattern in the image based on the shading edge of the image, or by comparing the feature pattern in the image with the feature pattern of the white line stored in advance. A known method such as a method can be employed. As shown in FIG. 11, when the detected white line is two lane marks L1 and L2 which are road boundary lines, the white line detection unit 21 coordinates the two lines L1 and L2 on the image. Get the value.
[0023]
The vehicle behavior estimation unit 22 calculates a pitch angle with respect to the road surface of the camera 2 based on the white line detected by the white line detection unit 21. A known method can be used to calculate the pitch angle. For example, assuming that the two lane marks imaged in a state where the vehicle is stationary are those shown in FIG. 11, when the front part of the vehicle is lifted (the optical axis of the camera 2 is upward), it appears to open. The opening degree of the lane marks L1, L2 is further increased. On the contrary, when the rear part of the vehicle is lifted (the optical axis of the camera 2 is downward), the degree of opening of the two lane marks L1 and L2 shown in FIG. 11 is small (approaching parallel). Based on this principle, the pitch angle of the vehicle, that is, the pitch angle with respect to the road surface of the camera 2 can be calculated (see Japanese Patent Application Laid-Open No. 2002-163641).
[0024]
The vehicle traveling direction estimation unit 23 calculates the yaw angle of the host vehicle 1 with respect to the lane (white line) and the curvature of the road based on the white line detected by the white line detection unit 21. For example, a road model including parameters such as the yaw angle of the host vehicle 1 and the road curvature can be applied, and estimation can be performed by a known method such as using a Kalman filter or a least square method (see Japanese Patent Application Laid-Open No. 2002-236912). ). Note that the pitch angle calculated by the vehicle behavior estimation unit 22 may be calculated by the vehicle traveling direction estimation unit 23.
[0025]
The sign detection unit 30 includes a target specifying unit 31, a target coordinate estimation unit 32, and a pattern matching unit 33 in terms of processing functions performed internally. The target specifying unit 31 includes the yaw angle and road curvature estimated by the vehicle traveling direction estimation unit 23, the position and direction of the target detected by the radar 3, and the object detected by the reflection surface angle estimation device 4. Based on the reflection surface angle of the mark, a sign for the own lane is detected.
[0026]
A specific method for detecting a sign for the own lane will be described with reference to FIG. FIG. 12 shows a state in which the host vehicle 1 enters the curve, and three targets, target A, target B, and target C, are shown in the figure. Usually, the sign for the own lane is installed almost at right angles to the road, and if the road has a curvature, it is almost perpendicular to the road at that point, that is, the face of the sign is the center of the sign and the curve. It is installed so as to be almost parallel to the line connecting the two. Accordingly, when the sign surface is substantially parallel to the line segment connecting the sign and the center of the curve, it can be determined that the sign is for the own lane. Note that if the angle between the line segment connecting the sign and the center of the curve and the line segment indicating the face of the sign is equal to or less than a predetermined angle, it may be determined that the sign is for the own lane.
[0027]
Since the dividing line (lane) on the road is drawn parallel to the road, if the shape of the dividing line is known, the shape of the road can be estimated. Based on the yaw angle of the vehicle and the curvature of the road estimated by the vehicle traveling direction estimation unit 23, the shape of the road at the points of the targets A, B, and C is estimated, and the distance, direction, and height to the target are estimated. Based on this, the position of the target (sign) is specified, and the angle of the line connecting the center of curvature of the road and the target with respect to the host vehicle is calculated. If this angle coincides with the reflection surface angle calculated by the reflection surface angle estimation device 4, it can be determined that the target is a standard for the own lane. By such a method, it is determined that the target B and the target C among the targets A, B, and C shown in FIG. 12 are signs for the own lane.
[0028]
The target coordinate estimation unit 32 is a sign for the own lane based on the pitch angle of the vehicle 1 estimated by the vehicle behavior estimation unit 22 and the height and distance of the target detected by the radar 3. The coordinates of the determined target on the screen are obtained. Specifically, the target coordinates determined by the target specifying unit 21 as a sign for the own lane are projected onto the image memory 10 and the pattern matching processing unit 33 performs pattern matching. Ask for. Here, the projecting process is a process of calculating how the three-dimensional object on the real space coordinates detected by the radar 3 is captured by the camera 2.
[0029]
FIG. 13 is a view of the camera 2 and the target B as seen from the lateral direction. The distance between the lens 2a of the camera 2 and the imaging device 2b, that is, the focal length of the lens 2a is f, the height of the camera 2 is h, the height of the target B is H, and the lens 2a of the camera 2 to the target B Is the distance L, the pitch angle of the camera 2 is θ ₁ , the angle when the target B is viewed from the position of the height h of the camera 2 is θ ₂ , and the coordinate (y coordinate) projected onto the image sensor 2b is Assuming y, the following relationships (2) to (4) are established. The unit of coordinates is the number of pixels, and the focal length f is a value obtained by dividing the distance in real space by the number of pixels, that is, a value obtained by dividing the actual size by the size per pixel.
tan (θ ₁ + θ ₂ ) = y / f (2)
tan (θ ₁ + θ ₂ ) = (tan θ ₁ + tan θ ₂ ) / (1−tan θ ₁ × tan θ ₂ ) (3)
tan (θ ₂ ) = (H−h) / L (4)
[0030]
When the values of θ ₁ and θ ₂ are close to 0, it can be approximated as 1−tan θ ₁ × tan θ ₂ ≈1, so when substituting Equations (2) and (4) into Equation (3), the following equation ( 5) is obtained.
tan ⁻¹ (y / f) = tan ⁻¹ ((H−h) / L) + θ ₁ (5)
[0031]
FIG. 14 is a view of the camera 2 and the target B as viewed from above. In this case, the coordinate (x coordinate) x projected onto the image sensor 2b is expressed by the following equation (6).
x = f · tan (θb) (6)
Here, θb is an angle between the optical axis of the camera 2 (lens 2a) and the target B.
[0032]
The pattern matching processing unit 33 includes a portion corresponding to the coordinates calculated by the target coordinate estimation unit 32, that is, a portion determined to be a sign for the own lane in the image captured by the camera 2, and a predetermined amount. A pattern matching process is performed on the template to recognize the contents of the sign. As a pattern matching method, a method based on having a specific color combination used for a sign, or a recognition method using pattern matching focusing on the shape of the sign is used. That is, matching processing with an image is performed using the combination and shape of the sign color as a template, and the kind and character of the sign are determined. In addition, since the shape of the sign changes depending on the viewing angle, a template corrected based on the reflection surface angle calculated by the reflection surface angle estimation device 4 is used.
[0033]
The sign recognition result obtained by the pattern matching process is output to an external device such as a navigation device. For example, the driver can be alerted by displaying a sign on the navigation device. The sign display can also be used for updating navigation information and automatically controlling the vehicle.
[0034]
According to the sign recognition apparatus in the embodiment, the position of the three-dimensional object is detected based on the reflected light that is sent back from the radar 3 by being reflected from the radar 3 and returned to the target. Based on the position, the position of the three-dimensional object on the image captured by the camera 2 is specified, and a process (pattern matching process) for recognizing the contents of the sign for the part of the three-dimensional object specified on the image Do. As a result, it is not necessary to perform processing for recognizing the content of the sign on a flat object such as a paint display on the road surface, so that the calculation load can be reduced.
[0035]
Further, according to the sign recognition device in the embodiment, the road shape is estimated based on the captured image, and the angle of the reflecting surface of the three-dimensional object is calculated based on the reflected light of the laser beam transmitted from the radar 3. Then, based on the estimated road shape and the calculated reflection surface angle, a sign in the own lane direction is detected. Since the process for recognizing the contents of the sign (pattern matching process) is performed on the sign for the own lane, the process for recognizing the contents of the sign is performed on the sign for other lanes. Therefore, the calculation load can be reduced.
[0036]
The present invention is not limited to the embodiment described above. For example, in detecting a three-dimensional object, a radar that transmits laser light is used, but a radar that transmits other signals such as radio waves and ultrasonic waves may be used.
[0037]
Further, the calculation method of the pitch angle of the vehicle 1, the yaw angle of the vehicle 1 with respect to the lane (white line), and the curvature of the road is not limited to the method described above, and other methods can be used.
[0038]
The correspondence between the constituent elements of the claims and the constituent elements of the embodiment is as follows. That is, the camera 10 is an imaging unit, the radar 3 and the target specifying unit 31 are position detecting units, the target coordinate estimating unit 32 is a position specifying unit, the pattern matching processing unit 33 is a recognition unit, and the radar 3 and the reflection angle. The estimation device 4 constitutes a reflecting surface angle calculating means, and the target specifying unit 31 constitutes a own lane sign determining means. In addition, as long as the characteristic function of this invention is not impaired, each component is not limited to the said structure.
[Brief description of the drawings]
FIG. 1 is a diagram showing an embodiment in which a sign recognition device according to the present invention is mounted on a vehicle. FIG. 2 is a diagram showing a detailed configuration of a radar. FIG. FIG. 4 is a diagram showing a state in which three laser beams are transmitted from the radar in the vertical direction in some cases. FIG. 4 is a diagram illustrating a state in which the laser beam is transmitted when the radar is tilted in the vertical direction. (A) is a figure which shows the distribution which tied the part with the same intensity | strength of the laser beam which has the above-mentioned fixed area, FIG.5 (b) is the cross section cut | disconnected by the centerline of Fig.5 (a). FIG. 6 is a diagram showing the intensity of the laser beam irradiated from the radar to the target. FIG. 7 is a diagram showing the time variation of the intensity of the light reflected back from the target. Diagram showing how the laser beam is emitted from the radar and the reflected light is received when there are three targets. 9 is a diagram showing a temporal change in the intensity of light reflected by the targets 1 to 3 shown in FIG. 13; FIG. 10 is a diagram showing a detailed configuration of the processing device 5; FIG. 12 is a diagram showing how the host vehicle enters the curve. FIG. 13 is a diagram of the camera and the target B viewed from the side. FIG. View of target B from above [Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Vehicle, 1a ... Ceiling, 1b ... Front window, 2 ... Camera, 3 ... Radar, 4 ... Reflection surface angle estimation apparatus, 5 ... Processing apparatus, 10 ... Image memory, 20 ... Vehicle behavior estimation processing part, 21 ... White line Detection unit, 22 ... Vehicle behavior estimation unit, 23 ... Vehicle traveling direction estimation unit, 30 ... Sign detection unit, 31 ... Target identification unit, 32 ... Target coordinate estimation unit, 33 ... Pattern matching processing unit, 41 ... Projection Window, 42 ... Light receiving window, 43 ... Reflecting mirror, 44 ... Motor, 45 ... Light receiving element, 46 ... Laser light emitting element, 47 ... Amplifier circuit, 48 ... LD drive circuit, 49 ... Motor drive circuit, 50 ... CPU

Claims (4)

An imaging means mounted on the vehicle for imaging the surroundings of the vehicle;
A position detection means for transmitting a signal and detecting the position of the three-dimensional object based on the reflected signal returned by being reflected by the object;
Position specifying means for specifying the position of the detected three-dimensional object on the image captured by the imaging means based on the position of the three-dimensional object detected by the position detecting means;
A sign recognition apparatus comprising: a recognition processing means for performing a process for recognizing the contents of the sign for a portion corresponding to the position of the three-dimensional object specified by the position specifying means on the image. In the sign recognition device according to claim 1,
Road shape estimation means for estimating a road shape based on an image captured by the imaging means;
Reflecting surface angle calculating means for calculating the angle of the reflecting surface of the three-dimensional object based on the reflected signal returned from the object reflected from the position detecting means;
Based on the road shape estimated by the road shape estimating means and the angle of the reflecting surface calculated by the reflecting surface angle calculating means, the three-dimensional object detected by the position detecting means is for the traveling lane of the host vehicle. The vehicle further comprises own lane sign judging means for judging that it is a sign,
The position specifying means specifies the position of the sign on the image captured by the imaging means based on the position of the sign determined by the own lane sign determining means,
The sign recognition apparatus characterized in that the recognition processing means performs a process for recognizing the contents of the sign with respect to the position of the sign specified by the position specifying means. The sign recognition apparatus according to claim 2,
The reflecting surface angle calculating means includes a time from when the intensity of the reflected signal reaches a maximum value to a maximum value, and a predetermined threshold value after the intensity of the reflected signal reaches a maximum value. A sign recognition apparatus that calculates an angle of a reflecting surface of a three-dimensional object based on a ratio to a time until the following. In the sign recognition device according to claim 2 or 3,
The road shape estimation means detects the center of curvature of the lane in which the host vehicle is traveling,
The own lane sign determination unit is obtained based on a line segment connecting the center of curvature of the road detected by the road shape estimation unit and the three-dimensional object and the angle of the reflection surface calculated by the reflection surface angle calculation unit. A sign recognition apparatus, characterized in that, when an angle formed by a line segment indicating a reflecting surface is equal to or smaller than a predetermined angle, the detected three-dimensional object is determined as a sign for a traveling lane of the host vehicle.

Images