JP2005347945A - Apparatus and method of monitoring vehicle periphery - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an apparatus and a method of monitoring a vehicle periphery capable of highly accurately recognizing an object of interest with appropriate timing. <P>SOLUTION: The apparatus for monitoring the vehicle periphery applies image processing on images around a vehicle recorded by an imaging means mounted on the predetermined position of the vehicle for determining a three-dimensional positional relation between the vehicle and objects around the vehicle. The positional relation in the vehicle traveling direction is determined based on own vehicle position information obtained by GPS positioning and object position information obtained from given map data. The positional relation in the direction orthogonal to the vehicle traveling direction is determined based on the own vehicle position information obtained by GPS positioning and object position information obtained by the image processing. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a vehicle periphery monitoring apparatus that monitors the periphery of a vehicle using an image processing result of a captured image.

Conventionally, instead of recognizing the existence of an intersection by image processing, and then setting a recognition frame for recognizing a traffic light with a predetermined positional relationship with respect to the intersection, image processing is not performed on the entire area of the captured image. A technique is known in which image processing is performed only within a set recognition frame range to improve the recognition accuracy of a traffic light (see, for example, Patent Document 1).
JP-A-1-265400

  By the way, in the above-described prior art, it is necessary to perform image processing on the entire region of the captured image in order to recognize the intersection, and there is a problem that the processing time cannot be reduced and the recognition rate cannot be improved. In addition, due to the recognition accuracy of the intersection, the recognition accuracy of the traffic light may deteriorate, or the recognition time may be delayed.

  In contrast, when an object to be recognized such as a traffic light is recognized by image processing, the position information of the object is stored, and the next time the next time the same object is recognized, the stored position information is used. It is also conceivable to improve the efficiency of recognition processing from the next time. However, the positional information obtained by the image processing may not be accurate in the line-of-sight direction (depth direction) of the imaging means, and similarly, the recognition accuracy of the object to be recognized may be deteriorated or delayed. sell.

  Therefore, an object of the present invention is to provide a vehicle periphery monitoring device and a vehicle periphery monitoring method that can recognize an object to be recognized at an accurate timing and with high accuracy.

In order to solve the above-described problem, according to one aspect of the present invention, image processing is performed on an image around a vehicle captured by an imaging unit mounted at a predetermined position of the vehicle, and an object between the vehicle and the surrounding object is captured. A vehicle periphery monitoring device for determining the three-dimensional positional relationship of
The positional relationship related to the line-of-sight direction of the imaging means is determined based on the vehicle position information based on GPS positioning and the object position information based on given map data, and relates to a direction perpendicular to the line-of-sight direction of the imaging means. A vehicle periphery monitoring device is provided in which the positional relationship is determined based on own vehicle position information based on GPS positioning and object position information based on image processing.

  In this aspect, the object for which the positional relationship is determined may be an object (for example, a lane display or a temporary stop line) existing in the vicinity of an intersection, and may be a traffic light. Further, the given map data used for the determination of the positional relationship relating to the line-of-sight direction of the imaging means includes coordinate data of an intersection, and the three-dimensional positional relationship between the object and the vehicle is the above-described relationship with respect to the intersection. The object position information determined based on the three-dimensional positional relationship between the object and the vehicle and based on the given map data may be derived based on the position data of the intersection. For example, the object position information based on the given map data may be derived as a relative position with respect to an intersection node. The object position information based on the given map data may be derived based on the intersection position data and the width data of the intersection road. In this case, the object position information as to whether it is on the near side or the far side with respect to the intersection road may be determined based on image processing.

  Further, in the vehicle periphery monitoring device that sets a recognition frame for recognizing an object existing near the intersection and determines the presence / absence of the object in the recognition frame by image recognition processing, the determined line of sight of the imaging unit The size and position of the recognition frame in the image depth direction may be changed based on the positional relationship related to the direction. Further, a start timing of image processing for recognizing the object may be determined based on the positional relationship relating to the determined line-of-sight direction of the imaging unit.

  Further, it effectively includes auxiliary information storage means for storing, as auxiliary information, the positional relationship relating to the direction perpendicular to the line-of-sight direction of the determined imaging means, and the size of the recognition frame at the start of the image processing The height and position are determined using auxiliary information in the auxiliary information storage means. The imaging unit may be a monocular camera that is advantageous in terms of cost with respect to a stereo camera.

According to another aspect of the present invention, the vehicle periphery monitoring for monitoring an object around the vehicle by performing image processing on an image around the vehicle captured by an imaging unit mounted at a predetermined position of the vehicle. In the device
When determining the three-dimensional positional relationship between an object around the vehicle and the vehicle, the positional relationship related to the line-of-sight direction of the imaging means is based on the vehicle position information based on GPS positioning and given map data A vehicle periphery monitoring device is provided that performs correction based on object position information.

According to another aspect of the present invention, the step of acquiring an image around the vehicle captured by the imaging means mounted at a predetermined position of the vehicle;
Performing image processing on the acquired image and determining a three-dimensional positional relationship between an object around the vehicle and the vehicle,
In the determination step, the positional relationship related to the line-of-sight direction of the imaging unit is determined based on the vehicle position information based on GPS positioning and the object position information based on given map data, The vehicle periphery monitoring method is characterized in that the positional relationship in the perpendicular direction is determined based on own vehicle position information based on GPS positioning and object position information based on image processing.

  ADVANTAGE OF THE INVENTION According to this invention, the vehicle periphery monitoring apparatus and vehicle periphery monitoring method which can recognize the object of recognition object with an exact timing and high precision can be obtained.

  The best mode for carrying out the present invention will be described below with reference to the drawings.

  FIG. 1 is a system configuration diagram showing an embodiment of a vehicle periphery monitoring device according to the present invention. The vehicle periphery monitoring apparatus according to the present embodiment is configured around an image recognition / model creation processor 10 (hereinafter simply referred to as “processor 10”). The processor 10 includes a CPU, a ROM, and the like, and executes each process described later. The processor 10 may further include an FPGA (Field Programmable Gate Array) constituting the image processing circuit.

  The processor 10 is connected to position detecting means 12 capable of measuring the position of the vehicle (including the direction of the vehicle). The position detection means 12 includes a GPS (Global Positioning System) receiver. The GPS receiver measures the position and direction of the vehicle based on satellite signals from satellites and information from other reference stations. In addition, as a positioning method of the position of the vehicle, it may be based on independent positioning or interference positioning (kinematic method (RTK-GPS positioning algorithm)).

The position detection means 12 is an INS (Inertial
Navigation Sensor) may be included. The INS may be composed of a gyro sensor that detects rotation parameters around three axes of a body coordinate system defined around the vehicle, and an acceleration sensor that detects acceleration in each direction of the three axes. The INS detects a motion state quantity representing a dynamic state quantity of the vehicle during traveling based on detection values of these sensors. In this case, the position detector 12 may determine the position of the vehicle in consideration of the INS detection result together with the GPS receiver. For example, the position detection unit 12 uses the detection result by the INS until the reception state of the GPS receiver recovers when the reception state of the GPS receiver is not good (typically, when the radio wave is interrupted). The position may be determined.

  A map database 22 is connected to the processor 10. The map database 22 stores given map information on a recording medium such as a DVD or a CD-ROM. The map information includes the coordinate information of each node corresponding to the junction / branch point of the intersection / highway, link information connecting adjacent nodes, and road width information corresponding to each link. In addition, road types such as national roads, prefectural roads, and highways corresponding to each link, traffic regulation information of each link, traffic regulation information between each link, and the like may be included.

  A spatial model database 30 is connected to the processor 10. The space model database 30 is configured by a writable recording medium such as a hard disk. The spatial model database 30 stores useful information (auxiliary information described in detail later) that does not exist in the map information in the map database 22 as needed.

  A radar sensor 19 may be further connected to the processor 10. The radar sensor 19 may be a millimeter wave radar sensor, and is disposed, for example, near the front grill of the vehicle. When the radar sensor 19 is a millimeter wave radar sensor, the relative position of the object ahead of the vehicle (relative position with respect to the host vehicle) may be measured from the phase information of the two frequencies by, for example, a two-frequency CW (Continuous Wave) method. The radar sensor 19 may be configured to scan the radiation beam one-dimensionally or two-dimensionally within a predetermined area in front of the vehicle.

  An imaging means 18 such as a CCD camera is connected to the processor 10. The imaging means 18 is mounted at an appropriate position of the vehicle so as to capture the scenery around the vehicle, and is fixed, for example, near a room mirror in the interior of the vehicle. Image data of a captured image captured by the imaging unit 18 is supplied to the processor 10. The captured image captured by the imaging unit 18 is supplied as a video signal to the display device 24 and displayed on the display device 24 such as a liquid crystal display.

  As will be described in detail below, the processor 10 performs image recognition processing on the captured image from the imaging unit 18 and sets the feature points of a target object to be recognized (hereinafter referred to as “target”) within a predetermined region of the captured image. Search and recognize the presence and state of the target. The processor 10 may execute predetermined control according to the recognition result. For example, the processor 10 recognizes a traffic light as a target, and when the traffic light is in a “red” lighting state, the processor 10 may output an alarm according to the relative positional relationship between the traffic light and the vehicle, the vehicle speed, or the like. Good.

  Such a target recognition function of the vehicle periphery monitoring device is very useful in that it can effectively assist a passenger's eyes as a new “eye” of the imaging means 18. In order to improve the usefulness of the target recognition function by such a vehicle periphery monitoring device, it is necessary to start recognizing the target at an accurate timing and to recognize it with high accuracy.

  FIG. 2 is a flowchart of the target recognition process executed by the processor 10 of the vehicle periphery monitoring device of the present embodiment. In the following processing, it is assumed that the latest vehicle position information is input to the processor 10 from the position detection means 12 at a predetermined cycle.

  Based on the vehicle position information obtained from the position detection means 12, the processor 10 reads map information around the vehicle position from the map database 22 based on the current vehicle position (step 100).

  The processor 10 identifies a road (link) corresponding to the current vehicle position based on the read map information and vehicle position information (step 110), and determines whether or not an intersection exists ahead in the traveling direction. Judgment is made based on the intersection information of the map information (step 120). If it is determined that there is no intersection ahead of the traveling direction, the process returns to step 100.

  When the processor 10 determines that there is an intersection ahead of the traveling direction, the processor 10 checks whether or not auxiliary information associated with the intersection is stored in the spatial model database 30 (step 130). If it is determined that the auxiliary information does not exist in the space model database 30, the process proceeds to step 200 and subsequent steps. On the other hand, if it is determined that the auxiliary information exists in the space model database 30, the process proceeds to step 300 and subsequent steps.

  FIG. 3 is a flowchart showing a process executed when it is determined that the auxiliary information does not exist in the space model database 30. In this case, the processor 10 determines the relative positional relationship between the target around the intersection and the vehicle based on the latest vehicle position information from the position detection unit 12 and the position information of the intersection in the map database 22. (Step 200). At this time, the processor 10 may simply derive the position information of the target around the intersection based on the position information of the intersection. For example, the position information of a traffic light as a target may be position information offset by a height of 5 m from the position of an intersection. Further, the position information of the stop line or the lane display as the target may be position information offset by the width of the road (or width + α) in front of the vehicle traveling direction from the position of the intersection.

  In subsequent step 210, the processor 10 selects the target at a predetermined timing (for example, when the distance in the vehicle traveling direction from the target becomes a predetermined value) based on the relative positional relationship obtained in step 200. Image processing for recognition is started, and thereafter, image processing is continued until an appropriate time point (for example, an intersection passing time point) with respect to the latest captured image input every predetermined cycle (step 220). At this time, the processor 10 determines the relative positional relationship between the target and the vehicle at the time of the imaging (the latest vehicle position from the position detection unit 12) with respect to the latest captured image input every predetermined cycle. Based on the information, the position / size of the image processing range (recognition frame) for recognizing the target is determined. For example, as the vehicle approaches the target (or intersection) (that is, as the relative distance in the vehicle traveling direction between the target and the vehicle becomes smaller), the recognition frame moves in the screen depth direction while being moved to the upper side of the screen. And the size in the screen width direction may be increased.

  When the image processing on the series of captured images is completed in this way, the processor 10 generates position information of the target based on the image processing result (Step 230). The position information of the target is position information relative to a fixed point such as the position of the intersection (coordinate value of the intersection node) or position information based on an absolute coordinate system (for example, latitude, longitude, altitude). . For example, as an example of the former, the position information of the target (traffic light) may be managed as relative coordinate values (Δx, Δy, Δz) with respect to the coordinate values of the intersection node as shown in FIG. Hereinafter, for convenience of explanation, it is assumed that the position information of the target is managed by relative coordinate values (Δx, Δy, Δz), and the Y axis of this relative coordinate system is the traveling direction of the vehicle (that is, the extension of the traveling road). It corresponds to the current direction). In this example, the description will be continued assuming that the line-of-sight direction of the imaging means 18 substantially matches the vehicle traveling direction.

In subsequent step 240, the processor 10 determines the reliability of the position information of the target generated in step 230. At this time, the processor 10 particularly determines the reliability of the position information (that is, Δy) related to the vehicle traveling direction among the position information of the target. This is because, when a monocular camera is used as the imaging means 18, motion parallax (motion parallax) is used for measuring the three-dimensional position of the target.
stereo) is used. In this case, it is difficult to ensure parallax (in terms of space and time) and the measurement error in the depth direction (the direction of the line of sight of the imaging unit 18) is large. Even when a stereo camera is used as the image pickup means 18, it is difficult to secure a sufficient parallax because of being mounted on a vehicle, and the measurement error in the depth direction (the line-of-sight vector direction of the image pickup means 18) is still large. Based on that.

  In step 240, the processor 10 may determine the reliability of position information (that is, Δx, Δz) in a direction perpendicular to the vehicle traveling direction. However, since the measurement error in the direction perpendicular to the line-of-sight vector direction of the image pickup means 18 is usually small, it may be employed as long as it is not an abnormal value.

  When the processor 10 determines that the position information related to the vehicle traveling direction includes a large error (| Δy |> TH, TH is a predetermined threshold), the processor 10 corrects the position information related to the vehicle traveling direction ( Step 250), the corrected target position information (Δx, Δy, Δz) is stored in the space model database 30 as auxiliary information (step 260). On the other hand, when it is determined that the position information related to the vehicle traveling direction is appropriate (| Δy | ≦ TH), the position information (Δx, Δy, Δz) of the target is used as auxiliary information without correction, and the spatial model database 30 is used. (Step 260). In step 260, the auxiliary information is preferably stored in the spatial model database 30 in association with the corresponding intersection along with the type of the target.

  Here, with regard to the correction processing in step 250, the position information related to the vehicle traveling direction may be set to a value offset from the position of the intersection by the width of the intersecting road (or width + α) before the vehicle traveling direction. At this time, as the position information (Δy) related to the vehicle traveling direction obtained by the image recognition, only the sign of the sign may be used (that is, only the position information of the target regarding whether it is before or behind the intersection is employed). Good). For example, Δy = −40 [m] may be corrected as Δy = −D (or Δy = −D−α, where D is the width of the crossing road [m]). Alternatively, in the step 250, the position information related to the vehicle traveling direction of the target is the size and position of the other target obtained by image processing in the vehicle traveling direction and the vehicle of the other target by other detection means. The correction may be performed by using a comparison result with a size or position (or a known size or position) in the traveling direction, or a detection result of the same target by the radar sensor 19.

  Next, a usage mode of the auxiliary information created as described above will be described with reference to FIG. FIG. 5 is a flowchart executed when it is determined in step 130 of FIG. 2 that the auxiliary information does not exist in the space model database 30. In this case, the processor 10 determines the relative positional relationship between the target around the intersection and the vehicle based on the latest vehicle position information from the position detection unit 12 and the auxiliary information in the space model database 30. (Step 300).

  Similarly, in the following step 310, the processor 10 is based on the relative positional relationship obtained in step 300 at a predetermined timing (for example, when the distance in the vehicle traveling direction from the target becomes a predetermined value). Image processing for recognizing a target is started, and thereafter, image processing is continued until an appropriate time point (for example, an intersection passing time point) for the latest captured image input every predetermined period (step 320). ).

  At this time, the processor 10 determines the relative positional relationship between the target and the vehicle at the time of the imaging (the latest vehicle position from the position detection unit 12) with respect to the latest captured image input every predetermined cycle. Based on the information, the position / size of the image processing range (recognition frame) for recognizing the target is determined. At this time, the size and position of the recognition frame in the image depth direction are mainly determined based on position information related to the vehicle traveling direction of the target.

  As described above, in the present embodiment, once the auxiliary information is created and stored, in the subsequent recognition processing, the relative positional relationship between the target and the vehicle (that is, the position information of the target) is the spatial model database 30. Derived using auxiliary information in Thereby, it is possible to accurately determine the three-dimensional relative positional relationship between the vehicle and the target as compared with the determination in FIG. 3 (step 210). In the present embodiment, the auxiliary information is corrected for the component in the line-of-sight vector direction of the image pickup means 18 in consideration of the large measurement error in the line-of-sight vector direction of the image pickup means 18 as described above. It is possible to accurately determine the three-dimensional relative positional relationship between the object and the target.

  Therefore, according to the present embodiment, it is possible to start recognizing a target at an appropriate timing, and it is also possible to appropriately set an image processing range (recognition frame) for the recognition (in particular, It is possible to appropriately set the size and position of the recognition frame in the image depth direction), and as a result, the target can be recognized with high accuracy and with accurate timing.

  Next, with reference to FIG. 6, the Example performed by this inventor is demonstrated. Image processing was performed on an image captured from a vehicle traveling at a speed of 60 km / h. First, assuming that there is no auxiliary information in the spatial model database 30 (see FIG. 3), as shown in FIG. 6A, a range of 130 pixels long × 400 pixels wide is set as an image processing range. Image processing was executed. In this case, the calculation time required to recognize the traffic light located 70 m ahead of the vehicle was 20 ms. When the traffic signal position was calculated using the result of the traffic signal recognition, relative coordinate values (Δx, Δy, Δz) = (7.3, 67.4, 5.2) with respect to the intersection node were obtained. In this case, since Δy = 67.4 is abnormally large, the value of Δy was corrected to the width of the crossing road 5.5 [m].

  Next, auxiliary information (Δx, Δy, Δz) = (7.3, 5.5, 5.2) is stored in the space model database 30 and the same processing is performed (see FIG. 5). As shown in FIG. 6 (B), the range of 70 pixels long × 130 pixels wide can be limited as the image processing range, and the calculation time required to recognize the same traffic light can be reduced to 7 ms.

  On the other hand, when auxiliary information (Δx, Δy, Δz) = (7.3, 67.4, 5.2) is stored in the space model database 30 and the same processing is performed, the start timing of image processing is extremely high. As a result, it became impossible to recognize a traffic light located 70 m ahead of the vehicle.

  From the above, it can be seen that the image processing range and calculation time required for recognizing the traffic light can be greatly reduced by increasing the accuracy of the position information of the traffic light in the vehicle traveling direction.

  The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

  For example, in the above-described embodiment, when it is determined that there is a large error in the position information related to the vehicle traveling direction of the target obtained from the image processing result, the position information related to the vehicle traveling direction is simply a predetermined value ( For example, zero) may be adopted.

  Further, in the above-described embodiment, the corrected position information of the target is used for the same target after the next time (during the second and subsequent runs on the same road in the same direction). However, using the same auxiliary information (position information related to the vehicle traveling direction is corrected in the same way) obtained from the image processing results for several frames of images, the image processing method for subsequent frames of images is modified. (For example, the size and position of the image processing range (recognition frame) may be corrected).

  In the above description, the target position information (auxiliary information) is mainly used for determining the start timing of image processing and the size and position of the image processing range (recognition frame). The position information (auxiliary information) may be used for any control executed based on the relative positional relationship between the target and the vehicle. In particular, the position information related to the vehicle traveling direction of the target can be an important parameter in control such as warning and intervention braking.

  Further, the present invention is applicable not only to a vehicle traffic signal as shown in the figure but also to a pedestrian traffic signal.

1 is a system configuration diagram showing an embodiment of a vehicle periphery monitoring device according to the present invention. It is a flowchart of the target recognition process performed by the processor 10 of the vehicle periphery monitoring apparatus of a present Example. It is a flowchart of the target recognition process performed by the processor 10 of the vehicle periphery monitoring apparatus of a present Example (the 2). It is explanatory drawing which shows an example of the positional information on a target. It is a flowchart of the target recognition process performed by the processor 10 of the vehicle periphery monitoring apparatus of a present Example (the 3). It is a figure which shows the captured image to which one Example of the image recognition process was applied.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image recognition and model creation processor 12 Position detection means 18 Imaging means 19 Radar sensor 22 Map database 24 Display apparatus 30 Spatial model database

Claims (12)

A vehicle periphery monitoring device that performs image processing on an image around a vehicle imaged by an imaging unit mounted at a predetermined position of the vehicle and determines a three-dimensional positional relationship between an object around the vehicle and the vehicle. And
The positional relationship related to the line-of-sight direction of the imaging means is determined based on the vehicle position information based on GPS positioning and the object position information based on given map data, and relates to a direction perpendicular to the line-of-sight direction of the imaging means. The vehicle periphery monitoring device, wherein the positional relationship is determined based on own vehicle position information based on GPS positioning and object position information based on image processing.   The vehicle periphery monitoring apparatus according to claim 1, wherein the object for which the positional relationship is determined is an object existing near an intersection.   The vehicle periphery monitoring device according to claim 2, wherein the object for which the positional relationship is determined is a traffic light. The given map data includes intersection coordinate data;
The three-dimensional positional relationship between the object and the vehicle is determined via the respective three-dimensional positional relationship between the object and the vehicle with respect to an intersection,
The vehicle periphery monitoring device according to claim 2, wherein the object position information based on the given map data is derived based on intersection position data.   5. The vehicle according to claim 4, wherein the object position information based on the given map data is derived based on intersection position data and width data of a road that intersects the line-of-sight direction of the imaging unit at the intersection. Perimeter monitoring device. The vehicle periphery monitoring apparatus according to claim 2, wherein a recognition frame for recognizing an object existing near the intersection is set, and the presence or absence of the object is determined by image recognition processing in the recognition frame.
A vehicle periphery monitoring device that changes the size and position of the recognition frame in the image depth direction based on the determined positional relationship related to the line-of-sight direction of the imaging means.   The vehicle periphery monitoring apparatus according to claim 6, wherein a start timing of image processing for recognizing the object is determined based on the determined positional relationship related to the line-of-sight direction of the imaging unit. Auxiliary information storage means for storing the positional relationship in the direction perpendicular to the line-of-sight direction of the determined imaging means as auxiliary information,
The vehicle periphery monitoring device according to claim 7, wherein the size and position of the recognition frame at the start of the image processing are determined using auxiliary information in auxiliary information storage means.   The vehicle periphery monitoring apparatus according to claim 1, wherein the imaging unit is a monocular camera. In a vehicle periphery monitoring device that monitors an object around a vehicle by performing image processing on an image around the vehicle imaged by an imaging unit mounted at a predetermined position of the vehicle,
When determining the three-dimensional positional relationship between an object around the vehicle and the vehicle, the positional relationship related to the line-of-sight direction of the imaging means is based on the vehicle position information based on GPS positioning and given map data A vehicle periphery monitoring device, wherein correction is performed based on object position information. Acquiring an image around the vehicle imaged by an imaging means mounted at a predetermined position of the vehicle;
Performing image processing on the acquired image and determining a three-dimensional positional relationship between an object around the vehicle and the vehicle,
In the determination step, the positional relationship related to the line-of-sight direction of the imaging unit is determined based on the vehicle position information based on GPS positioning and the object position information based on given map data, The vehicle periphery monitoring method according to claim 1, wherein the positional relationship related to the perpendicular direction is determined based on own vehicle position information based on GPS positioning and object position information based on image processing.   A computer-readable program for realizing the vehicle periphery monitoring method according to claim 11.

Images