JP4857840B2 - Object detection method and object detection apparatus - Google Patents

Description

  The present invention relates to an object detection method and an object detection device for detecting an object based on information obtained from an object sensor such as a radar or a camera, and more particularly to a method and an apparatus for reflecting information related to the external environment in object detection.

  Conventionally, driving support devices such as a collision prevention device, an inter-vehicle distance control device, and a follow-up traveling device have been proposed for vehicles. In these driving support devices, an object (obstacle) such as a vehicle traveling in front of the host vehicle is used. An object detection device for detection is used.

  In such an object detection device, a vehicle detection device that inputs visibility information or weather information from the outside and corrects object detection based on this information is known (see Patent Document 1).

  The vehicle detection device outputs outputs from an environment acquisition device that acquires visibility information or weather information from the outside, a first vehicle observation device that uses a radar, and a second vehicle observation device that uses an infrared camera. A selection device to be selected, a vehicle tracking device that detects a vehicle traveling in front of the host vehicle based on an output from the environment acquisition device and an output selected by the selection device, and an abnormal running or stop of the vehicle by an output from the vehicle tracking device And an event determination device that determines an event such as a vehicle.

  In other words, the first vehicle observation apparatus provided with the millimeter wave radar is less affected by weather such as snow and fog. However, the azimuth angle resolution is low, and the size and identification of the vehicle are difficult. On the other hand, the second vehicle observation apparatus provided with the infrared camera can detect the size and speed position of the vehicle with high accuracy and has excellent detection performance at night. However, the influence of weather such as snow and fog is great.

Therefore, when the second vehicle observation device such as snow or fog cannot be detected with high accuracy, the detection information of the first vehicle observation device is selected. On the other hand, if the second vehicle observation device can detect the weather with good accuracy and the detection information of the second vehicle observation device is selected. Thereby, highly accurate vehicle detection can always be performed.
JP 2000-111644 A

  In the above-described prior art, the detection information of the first vehicle observation device and the detection information of the second vehicle observation device are selected according to the external environment. (Vehicle Information and Communication System) and the like are input from a transmission device outside the apparatus.

  Information obtained from a transmitting device such as VICS is information on the weather at a fixed point determined in advance.

  For this reason, there is a possibility that weather information corresponding to the object detection position can be obtained when the object device is used in a fixed manner. However, when the object detection device is used in a vehicle, a transmission facility such as VICS is used. When traveling on a road that does not have, there is a problem that weather information cannot be obtained and optimum detection information according to the weather is not selected.

  The present invention performs object detection that reflects the actual external environment such as the weather based on the output of the object detection sensor, so that the object detection accuracy can be improved without using a special device for acquiring external environment information. An object of the present invention is to provide an object detection method and an object detection device that can be improved.

  The present invention has been made in view of the above-described circumstances, and executes an object detection step for detecting an object based on information related to an object in the object detection target area detected by an object sensor, so that the object exists in the object detection target area. In the object detection step, the object detection step includes an external environment detection step for detecting an external environment based on the information about the object acquired in the acquisition step, information obtained from the object sensor, and this information. An object detection method characterized by including an external environment reflection step for reflecting an external environment with respect to detection information used for object detection that is at least one of the converted conversion information .

  In the present invention, the external environment information is reflected in the object detection to improve the object detection accuracy, so that the external environment is determined based on the information related to the object detected by the object sensor of the object detection device. Even in a place where information from an external facility cannot be obtained, information on the external environment can be acquired, and external environment information at a position where object detection is performed can be acquired.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The object detection apparatus according to this embodiment detects an object based on information obtained from the object sensor (1, 2) and information obtained from the object sensor (1, 2). An object detection device comprising an object detection means (CU), wherein the object sensor (1, 2) is an image information acquisition means (1) for acquiring image information relating to an object in the object detection target area; Detection information acquisition means (2) for acquiring detection information relating to an object based on reflection of the detection wave acquired by scanning the detection target region with the detection wave, and the object detection means (CU) includes an image External environment detection means (13) for detecting an external environment based on information and detection information is included, and the object detection means (CU) reflects the determination result of the external environment detection means in object detection. And

  Based on FIGS. 1-18, the object detection apparatus of Example 1 of the best mode for carrying out the present invention will be described.

  As shown in FIG. 1, the object detection apparatus according to the first embodiment is mounted on a vehicle MB, and includes a camera (image information acquisition unit) 1 and a radar (detection information acquisition unit) 2 as object sensors. Note that the information on the object detected by the object detection device is output to the driving support control processing unit 20 shown in FIG. 3 which is a part that executes control of the driving support device.

  For example, the camera 1 is mounted in the vicinity of a room mirror (not shown) in the passenger compartment. The camera 1 includes at least one of a visible light camera that captures a luminance image using an image sensor such as a CCD (Charge Coupled Devices) and a CMOS (Complementary Metal Oxide Semiconductor), and an infrared camera that captures an infrared image. In the first embodiment, a visible light camera is used.

  The radar 2 is attached to the front part of the vehicle MB, scans the front of the vehicle (in the direction of the arrow FR) in the horizontal direction, and detects the distance and reflection intensity of an object (detection point) existing in front of the vehicle. The detection point is a position where an object is detected, and is detected as a position on the XZ-axis coordinate shown in FIG.

  As the radar 2, a millimeter wave radar, a laser radar, or an ultrasonic radar can be used. In the first embodiment, a millimeter wave radar is used. In the case of millimeter wave radar, distance, reflection intensity, and relative speed can be obtained as information on the detected object. In the case of laser radar, distance and light reflection intensity can be obtained as detected object information.

  Information obtained by the camera 1 and the radar 2 is input to a control unit CU as object detection means. The control unit CU performs object detection control for detecting an object and discriminating its type by inputting a signal from an in-vehicle sensor including the camera 1 and the radar 2 as an object sensor, and is a RAM (Random Access Memory). , ROM (Read Only Memory), CPU (Central Processing Unit) and the like.

  The flow of processing in object detection control in the control unit CU will be briefly described with reference to FIG. 2. First, an acquisition process is performed in which information is input from sensors including the camera 1 and the radar 2 and stored in a memory not shown. Execute (Step S1).

  Next, an information conversion process for converting the stored input information into information used in the subsequent processes is executed (step S2).

  Next, for the detected input information and information that has been subjected to information conversion processing (conversion information), for detection of objects to be detected for each object type according to the correlation with the type of object to be detected An extraction process for extracting only information is executed (step S3).

  Next, external environment detection processing (step S4) for detecting the external environment is performed on the detection information extracted in step S3.

  Next, an effectiveness weighting process (step S5) is performed on the detection information extracted in step S3 to perform weighting based on the effectiveness corresponding to the weather environment obtained in the external environment detection process in step S4.

  Next, a correlation weighting process that weights the detected detection information according to the effectiveness of the detection information with respect to the object detection based on the correlation between the detection information and the detected object for each type of object. (Step S6) is performed.

  Next, an object existing in the detection area is detected using the detection information after weighting in the correlation weighting process, and an object detection process for determining the type of the object is executed (step S7). In the first embodiment, as the types of objects, the vehicle AB (hereinafter, in order to distinguish the vehicle AB and the vehicle MB, the former is referred to as the preceding vehicle AB, and the latter is referred to as the host vehicle MB), the two-wheeled vehicle MS, the person PE. The road structure (wall WO or the like) is discriminated.

  Next, a reliability determination process (step S8) for determining the reliability of the object detection result based on the external environment obtained by the external environment detection process in step S4 is performed.

  Next, based on the determination result of the reliability determination process of step S8, a control determination process (step S9) for determining the control content of the driving support control processing unit 20 (see FIG. 3) in the driving support control device (not shown). carry out.

  FIG. 3 is a block diagram conceptually showing a portion for performing each of the processes performed in the control unit CU.

  The conversion processing unit 11 receives the image information from the camera 1 and the detection information from the radar 2 and executes the information conversion process in step S2.

  The extraction processing unit 12 executes the extraction process in step S3 for extracting detection information from the image information from the camera 1, the detection information from the radar 2, and the conversion information converted by the conversion processing unit 11.

  The external environment detection processing unit (external environment detection means) 13 inputs detection information and executes the external environment detection process in step S4.

  The effectiveness weighting processing unit (effectiveness weighting processing unit) 14 weights the detection information extracted by the extraction processing unit 12 according to the weather environment detected by the external environment detection processing unit 13 in step S5. The effectiveness weighting process is executed.

  The correlation weighting processing unit 15 further responds to the detection information weighted by the effectiveness weighting processing unit 14 according to the correlation between the object to be detected and the detection information, and according to the effectiveness of the detection information for the object detection. The correlation weighting process of step S6 for performing the weighting is executed.

  The object detection processing unit 16 performs the object detection process in step S7 for detecting an object based on the detection information weighted by the correlation weighting processing unit 15.

  The reliability determination processing unit (reliability determination means) 17 executes the reliability determination processing in step S8 for determining the reliability of the object detected by the object detection processing unit 16, and the detection result of the object and the reliability The determination result is output to the driving support control processing unit 20.

  The driving support control processing unit (changing unit) 20 executes the driving support control, and executes the control determination process of step S9 that determines control based on the determination result of the reliability determination processing unit 17.

Next, each process of said step S1-S9 is demonstrated in detail.
First, in the acquisition process of step S1, as shown in FIG. 4, the image information (luminance image information) captured by the camera 1 and the reflection intensity and distance, which are detection information detected by the radar 2, are stored in the control unit CU. In the memory (not shown). In the first embodiment, the luminance value of a pixel is stored as image information. Further, as the information of the detection points of the radar 2, the distance to the object and the reflection intensity at each angle are stored for each horizontal scanning resolution of the radar 2.

Here, an example of the image information by the camera 1 is shown in FIGS.
5 and 6 show an example of an image in the case where a preceding vehicle AB, a person (pedestrian) PE, and a wall WO as a road structure exist in front of the host vehicle (vehicle MB). The image is projected on the imaging surface 1a of the camera 1 as shown in FIG. 5A shows a state seen from the side, and FIG. 6A shows a state seen from above. In both figures, (c) shows an infrared image when an infrared camera is used as the camera 1.

  5 and 6, the distance from the vertical projection point PA to the point PF on the road surface of the camera 1 is z, and the distance in the x-axis direction between the point PA and the point PF is xs.

Therefore, when the reference coordinate is the origin of the center of the lens 1b of the camera 1, the position of the point PF in the reference coordinate system is represented as (xs, -H, z), and the point PF is located on the image on the imaging surface 1a. The coordinates (xc, yc) to be expressed are expressed by the relationship of the following expressions (1) and (2) using the focal length f.
xc = xs · f / z (1)
yc = −H · f / z (2)

Next, an example of detection information by the radar 2 is shown in FIG.
In FIG. 9, an object detection example similar to the image information shown in FIGS. 5 and 6 shown in FIG. 5A is shown in FIG. As shown in this figure, the presence of an object can be detected by reflecting radio waves at a preceding vehicle AB, a person (pedestrian) PE, and a wall WO existing in front of the host vehicle (vehicle MB). In the figure, qP, qA, and qW indicated by circles indicate detection points of the respective objects.

Next, the information conversion process of step S2 executed by the conversion processing unit 11 will be described.
In this information conversion process, a process for converting image information and a process for converting detection information are performed. As processing for converting this image information, as shown in FIG. 4, edge detection processing for extracting vertical edges, horizontal edges, and edge strengths for luminance image information obtained from the camera 1, and direction vectors are extracted. A direction vector calculation process and an optical flow process for calculating an optical flow are performed. Further, as processing for converting detection information, processing for obtaining a relative velocity based on measurement points obtained from detection information from the radar 2 is performed.

  Edge detection in the above-described edge detection processing can be calculated by a convolution (convolution (multiplication)) operation with a Sobel filter or the like.

  FIG. 7 shows an example of a simple Sobel filter, in which (a) and (b) are filters for vertical edges, and (c) and (d) are filters for horizontal edges. By multiplying such a filter and image information, a vertical edge and a horizontal edge can be obtained. Note that these edge strengths can be obtained as absolute values of these values, for example.

Further, the direction vector can be obtained by the following equation (3) when the vertical edge strength is Dx and the horizontal edge strength is Dy.
Direction vector = Dx / Dy (3)

  The relationship between the direction vector angle and the edge strength is shown in FIG.

Next, the optical flow will be described.
The optical flow is an arrow connecting a video displayed at a certain point (xc, yc) on the image at a certain time and a point on the image where the video is located after Δt seconds, In general, the movement of a certain point on a certain object captured on the image is shown. Such an optical flow can be obtained by applying any conventionally proposed method such as block matching or gradient method.

This optical flow will be specifically described with reference to FIG. 9 and FIG.
9 and 10 show a case where the person PE is stopped and the preceding vehicle AB is moving forward in the same direction as the host vehicle MB. FIG. 10 shows Δt seconds after the time shown in FIG. Indicates the state. In both figures, (a) is an image of the camera 1 similar to that shown in FIGS. 5 and 6, and (b) shows a state in which the detection range by the radar 2 is viewed from above.

  Here, the values xc1, yc1, and hc1 indicating the person PE in FIG. 9 become smaller as the value of z as the denominator becomes smaller in the above-described equations (1) and (2) after Δt seconds shown in FIG. It becomes larger as the vehicle MB moves forward. Therefore, the arrow of the optical flow is the vanishing point VP (the point at which the forward infinity point on the image is imaged, and the optical axis LZ of the camera 1 is parallel to the road surface RS as in the settings of FIGS. In this case, the image center becomes longer in the direction away from the vanishing point VP.

  Similarly, since the points on the wall WO are also stopped, the optical flow becomes long. Also, these optical flows become arrows with the vanishing point VP as the center and from there toward the outside of the image.

  Therefore, the optical flow of the person PE shown in FIG. 10A is directed downward to the right at the feet, and directed to the right near the head near the center of the image.

  On the other hand, the preceding vehicle AB is moving at a constant speed with the host vehicle MB, the distance relationship is substantially constant, and the value of z does not change in the above formulas (1) and (2). Since there is almost no change, the length of the optical flow is shortened.

Next, returning to FIG. 4, the detection information conversion processing from the radar 2 will be described.
As described above, the conversion process of the detection information of the radar 2 is a process for obtaining the relative velocity based on the distance data of the measurement point. This relative velocity is obtained by the distance change / the length of the observation time for the same measurement point for a certain fixed time when distance information from the radar 2 is acquired in time series (for example, every 0.1 second). Can do.

  Alternatively, a least square error line between the same measurement points may be obtained, and the inclination thereof may be obtained as a relative speed. Because there is variation in distance accuracy, for example, by taking the least square error straight line obtained for distance results every 0.1 seconds, the error of about one measurement point or non-measurement point is complemented. Thus, the relative speed can be obtained stably.

  Furthermore, in the information conversion process in step S2, in addition to the conversion process described above, conversion for obtaining a value necessary for the subsequent process is also performed. For example, in the first embodiment, in the external environment detection process, the average value of the edge strength, the maximum value of the edge strength, and the average value or variance of the luminance values are necessary, and such values are also calculated by the conversion processing unit 11. .

Next, the extraction process in step S3 executed by the extraction processing unit 12 will be described.
In this extraction process, only the detection information necessary for the subsequent process is selected from the image information from the camera 1, the detection information from the radar 2, and the conversion information converted by the conversion processing unit 11.

  That is, in the object detection process in step S7, the type of information used for each type of object to be detected is set in advance. For this reason, in this extraction process, only the information used according to the kind of object is extracted.

  Specifically, based on the extraction characteristics shown in FIG. 11, only the detection information that is information indicated by “1” is extracted according to the type of the object to be detected. Are processed in parallel. The setting of the extraction characteristics shown in FIG. 11 is stored in advance in the database.

  The extraction characteristics in FIG. 11 will be described together with the object detection process in step S7.

  In the object detection process, the preceding vehicle AB, the two-wheeled vehicle MS, the person (pedestrian) PE, and the road structure (such as the wall WO) are detected and determined as object detection and determination. Therefore, the correlation between the types of the respective objects and information input from the camera 1 and the radar 2 will be described below.

  Generally, the preceding vehicle AB and the two-wheeled vehicle MS include a reflector (reflecting plate), and the radar 2 has a high reflection intensity at the detection point. Further, since the reflection intensity is high, the distance can be detected with high accuracy, and the relative speed can be calculated with high accuracy.

  On the other hand, as a difference between the preceding vehicle AB and the two-wheeled vehicle MS, generally, in the case of the preceding vehicle AB, the horizontal edge is strong and long on the image, whereas in the two-wheeled vehicle MS, the shape is similar to a pedestrian. Thus, there are no characteristic straight edges, and the edge direction vector has a large variance (the edges are directed in various directions).

  Therefore, as shown in FIG. 11, when the preceding vehicle AB and the two-wheeled vehicle MS are to be detected, the vertical edge, the horizontal edge, the edge intensity, the direction vector dispersion, the reflection intensity, and the relative speed are set to “1”. Other oblique edges are set to “0”. That is, for the detection of the preceding vehicle AB and the two-wheeled vehicle MS, only the information set to “1” is extracted, and the information set to “0” is not used for detection.

  On the other hand, although the person (pedestrian) PE may be detected by the radar 2, the reflection intensity is low. In addition, the person PE has a vertically long shape, is characterized by a vertical edge, and is characterized by a movement of a foot unique to a pedestrian (that is, an optical flow distribution). The person PE has a low moving speed, and the relative speed calculated from the optical flow on the image and the change in the distance of the object obtained from the radar 2 is observed as the speed approaching the host vehicle MB.

  Therefore, as shown in FIG. 11, the person PE sets the vertical edge, the direction vector dispersion, and the relative speed to “1”, extracts only this information, sets the other information to “0”, and extracts it. It is set not to.

  Road structures (walls WO, etc.) are generally difficult to define, but they are placed alongside the road and are artifacts, so they are linear components (edge strength and linearity). There is a feature that is strong. In addition, road structures (walls WO and the like) are stationary objects, and are objects that do not move in time-series observation. Such a stationary object is observed as a speed at which the relative speed calculated from the optical flow on the image and the distance change of the object obtained from the radar 2 approaches the host vehicle MB.

  Therefore, when a road structure (such as a wall WO) is to be detected, as shown in FIG. 11, other than the reflection intensity, that is, the vertical edge, the horizontal edge, the edge intensity, the direction vector dispersion, and the relative velocity are set to “1”. Is set.

  Although extraction of detection information necessary for the object detection process has been described here, information necessary for other processes is also extracted by this extraction process. That is, in the first embodiment, since the edge intensity average value, the reflection intensity average value, and the information related to the luminance used in the external environment detection process are used, these pieces of information are also extracted.

Next, the external environment detection process in step S4 executed by the external environment detection processing unit 13 will be described.
This external environment detection process is a process for determining the weather environment based on information from the camera 1 and the radar 2. That is, in the first embodiment, the determination is made based on the average value of the edge intensity of the image obtained by the camera 1 for a predetermined time and the average value of the reflection intensity obtained by the radar 2 for a predetermined time. These average values are calculated by information conversion processing by the conversion processing unit 11, extracted by extraction processing by the extraction processing unit 12, and input to the external environment detection processing unit 13.

  In general, in the case of fine weather, both the camera 1 and the radar 2 can image and measure a measurement point with high accuracy. Therefore, when both the edge intensity average value and the reflection intensity average value are high, it is determined that the weather is fine such as clear weather or light cloudy weather.

  On the other hand, in an environment where visibility is slightly reduced such as “haze”, “rain”, and “small snow”, the image of the camera 1 becomes unclear compared to when the weather is good, while the measurement of the radar 2 hardly affects the environment. High-precision measurement is possible without receiving it. Therefore, when the average value of the reflection intensity obtained by the radar 2 is kept at a high value while the average value of the edge intensity obtained by the camera 1 is slightly lowered, “haze”, “rain”, “small snow” Judged as minor bad weather such as.

  FIG. 12 shows an example of weather determination. In this example, a person PE, a two-wheeled vehicle MS, a preceding vehicle AB, and a wall WO as a road structure are present in front of the host vehicle MB, and the measurement points of the radar 2 are indicated by circles in FIG. Yes. Further, in this figure (c), the size of the circle represents the reflection intensity. Thus, the radar 2 accurately detects the objects PE, MS, AB, and WO.

  On the other hand, (b) in the figure shows an input image in the case of small bad weather in the same situation as in (c) in the same figure. Here, the image is divided into the respective regions Im1 to Im6 so as to correspond to the regions LL1 to LL6 divided in the direction of the radar 2, and the edge strength decreases in all the regions Im1 and Im6. ing.

  Furthermore, in the case of “dense fog”, “heavy rain”, or “heavy snow”, the image of the camera 1 becomes more blurred than the state shown in FIG. 12B, and the measurement intensity of the radar 2 also decreases. Therefore, when the edge intensity average value obtained by the camera 1 is greatly reduced and the reflection intensity average value obtained by the radar 2 is also reduced, it is determined that the weather is bad weather such as “condensed fog”, “heavy rain” or “heavy snow”.

  Even when the weather is sunny, when driving toward the sun with the sun tilted, such as in the morning or sunset, visibility may be reduced due to so-called backlighting. Therefore, it becomes difficult to obtain information on the pixel in the portion where strong light enters.

  FIG. 12A shows an input image when the camera 1 receives backlight in the same situation as FIG. In this case, among the regions Im1 to Im6, the edge intensity can be observed strongly in Im1 and Im6, whereas the edges cannot be observed in the regions Im3 and Im4.

  Therefore, when the reflection intensity of the radar 2 is high in all the areas LL1 to LL6, while the edge intensity of the camera 1 is observed only in a part of the areas Im1 to Im6, it is determined as “backlight”. Further, in the case of such “backlight”, an area where no edge is detected (for example, Im3 and Im4) generally has a high average value of luminance. Therefore, in the first embodiment, when an edge is not observed in a part of the region, the luminance of the region is further confirmed to improve the determination reliability. As the luminance, an absolute value of luminance may be used, or an average value of luminance per time may be used.

  In the external environment detection process of step S4, as described above, the weather environment is determined based on the average value of the edge intensity of the image for a predetermined time and the average value of the reflection intensity for a predetermined time. As a judgment type of this weather environment, “small bad weather”, “large bad weather”, “backlight”, and “good weather” are discriminated corresponding to the weighting of the effectiveness in step S5. Is set based on the measured values of the edge intensity and the reflection intensity.

Next, the effectiveness weighting process of step S5 executed by the effectiveness weighting processing unit 14 will be described.
This effectiveness weighting process is set based on the effectiveness characteristic set as shown in FIG. That is, in this effectiveness weighting process, each information is weighted according to small bad weather, bad bad weather, backlight, and good weather determined in the external environment detection process of step S3.

  As the degree of weighting, four types of “high”, “medium”, “low”, and “impossible” are set in descending order of weighting, and for example, a coefficient corresponding to these weights is multiplied. Note that “impossible” indicates that the object is not used for detection, and for example, “0” is multiplied as a coefficient.

Next, the correlation weighting process of step S6 executed by the correlation weighting processing unit 15 will be described.
This correlation weighting process is set based on the correlation characteristics set as shown in FIG. 14. As described above, this weighting is information for detection for object detection for each type of object to be detected. It is set according to the effectiveness of.

  That is, in the extraction process of step S3 described above, for each type of object to be detected, detection information used to discriminate the object is selectively extracted, but the type of the object to be discriminated and the extracted detection The effectiveness of information for object detection differs based on the correlation with the business information. For this reason, in step S6, weighting is performed according to the level of effectiveness according to the correlation between the object and information.

  Therefore, as shown in the figure, for example, the information extracted to determine “person PE” is set such that the weight of the vertical edge and the relative velocity is low and the weight of the direction vector dispersion is high. In this embodiment, when the weight is “low”, the measurement value is multiplied by a coefficient of “0.2”, while when the weight is “high”, the measurement value is “1.0”. Multiply by a coefficient.

  In addition, the other (preceding) vehicle AB, the two-wheeled vehicle MS, and the road structure are also weighted as illustrated.

Next, the object detection process in step S7 executed by the object detection processing unit 16 will be described.
In this object detection process, a voting process for voting the extracted and weighted detection information to the voting table TS, and determining whether or not each type of object exists based on the voting result, Detection processing for detecting an object is executed.

  As the voting table TS used for the voting process, a voting table corresponding to each of the person PE, the preceding vehicle AB, the two-wheeled vehicle MS, and the road structure (wall WO or the like) is prepared. Alternatively, in the voting table TS, each divided area is set in parallel with a hierarchy corresponding to each of the person PE, the preceding vehicle AB, the two-wheeled vehicle MS, and the road structure (such as a wall WO). Then, the weighted information corresponding to each of the person PE, the preceding vehicle AB, the two-wheeled vehicle MS, and the road structure (such as the wall WO), which are the types of objects to be identified, is used as a voting table corresponding to each type of object. Or vote in parallel to the hierarchy. This voting may be performed simultaneously in parallel, or may be performed at different voting times.

  FIG. 15 shows an example of the voting table TS. In FIG. 15, in addition to the voting table TS, a luminance image KP that is information from the camera 1, its edge component distance EK, reflection intensity RK that is information from the radar 2, a distance LK, and examples described later. The temperature image SP which is the information from the shown infrared camera is shown. FIG. 15 shows a case where a preceding vehicle AB, a person PE, and a tree TR as a road structure are detected as detection examples.

  The voting table TS corresponds to the XZ plane in the reference coordinate system described above, and this XZ is divided into small regions of Δx and Δz. These Δx and Δz have a resolution of about 1 m or 50 cm, for example. The size of the voting table TS, that is, the dimension in the z-axis direction and the dimension in the x-axis direction are arbitrarily set according to the required distance for object detection, the object detection accuracy, and the like.

  In FIG. 15, only one table is shown as the voting table TS. However, as described above, in the first embodiment, the voting table TS is for the person PE, the one for the preceding vehicle AB, and the two-wheeled vehicle MS. And a road structure (wall WO, tree TR, etc.) are set (not shown). Alternatively, voting is performed in parallel for each type of object in each area of the voting table TS.

  Next, the relationship between the image information and the voting table TS will be described. First, an image table PS with x and y coordinates shown in FIGS. 16A and 17A is also set on the image. The resolutions Δx and Δy of the image table PS indicate a small angle θ on the real coordinate system, as shown in FIG. In the first embodiment, a table is provided on the image for ease of voting of the image processing result, the angle resolution is θ in both the x direction and the y direction, and the edge obtained in the range is the voting table TS on the XZ plane. Vote for Here, θ is set to an angle between about 1 degree and 5 degrees, for example. This resolution angle may be set as appropriate in accordance with the accuracy of the object discrimination process, the required distance and the position accuracy of the object detection.

  Note that the voting table can be set on the XY plane in the reference coordinate system in the same manner as the voting table TS on the XZ plane.

  Next, an example of voting on the voting table TS when there is a preceding vehicle AB, a two-wheeled vehicle MS, a person (pedestrian) PE, and a wall WO as shown in FIGS. 16 and 17 will be described. As an explanation of the voting example, an voting example of the edge obtained by converting the image information by the camera 1 and the distance information by the radar 2 will be described.

  First, as an example of distance information voting by the radar 2, voting to the point Q when the person PE is observed will be described.

  As shown in FIG. 16B, when an object is measured at the position Qxz corresponding to the point Q on the voting table TS by the radar 2, the voting value is applied to the small area Sn including the position of Q of the voting table TS. to add. The voting value is a value weighted based on the above-described weather environment and high necessity.

Next, an edge voting example obtained from the image information of the camera 1 will be described.
First, as shown in FIG. 16A, an image table PS in which the XY axes are divided by Δx and Δy is set and voted. FIG. 17A is an image table PSe as a voting example of edge-processed information. When such an edge exists, the small area of the voting table K on the XZ plane corresponding to the small area where the edge exists. Is added to the vote value indicating this edge. This voting value is also a value that is weighted based on the above-described weather environment and high necessity.

  At this time, the correspondence between the XY plane and the small area in the XZ plane is obtained as follows. For example, the sizes of Δxe and Δye of the image table PSe are set as sizes corresponding to a small angle θ such as Δxe = f × tan θ and Δye = f × tan θ as shown in FIG. Here, f is a focal length.

  The angle formed by the small area of the image table PSe with respect to the image origin (x = 0, y = 0) and the small area of the voting table TS are the origin of the XZ table (x = 0, z = 0). What is necessary is just to convert into the angle made with respect to (point). Specifically, a case will be described in which a vertical edge Be existing at the position of x = xce is voted on the voting table TS in FIG.

  This vertical edge Be exists at a position corresponding to the fifth Δxe from the origin of the image table PSe. Here, since Δxe corresponds to the angle θ of the voting table TS, x = xce is located to the left by α = 5 × θ from the origin of the voting table TS. Therefore, in the voting table TS, voting is performed on a small area corresponding to the width of the angle θ at a position rotated by α = 5 × θ to the left from the origin (voting is performed on the fan-shaped area indicated by B in the figure). ).

In the same way, voting for the position of the object corresponding to the preceding vehicle AB is considered. In this case, the angle calculation is the same. However, when the distance to the preceding vehicle AB is known based on the vote of information from the radar 2, the vote is performed on a small area only at a position corresponding to the distance (here, near z = z0) ( FIG. 17B shows a portion ABR.
The voting process as described above is performed for each piece of detection information extracted in step S3 and weighted in steps S5 and S6. An example of this result is shown in FIG. 15, in which a voting part corresponding to the preceding vehicle AB is indicated by a symbol tAB, a voting part corresponding to the person PE is indicated by a symbol tPE, and a voting part corresponding to the tree TR is indicated by a symbol tTR.

Next, the detection process after voting on this voting table TS is completed will be described.
In general, when some object exists, there are many pieces of information such as distance and edge. That is, it is determined that an object is present at a position where the value of the voting result is high, such as the preceding vehicle tAB, the person (pedestrian) tPE, and the tree tTR shown in FIG.

  That is, the detected object position indicates the position of the small area in the voting result itself. For example, if the position of the preceding vehicle AB in FIG. 17 (ABR), the direction from the origin is α , It is determined that an object is detected at the position of the distance z0.

  Next, the type of the detected object is determined based on the content of information added to this table. That is, it is obtained by collating with the necessity characteristic shown in FIG.

  For example, if the reflection intensity is very strong and the lateral edge is strong among the vote values of the voted area, it is determined that the vehicle is the preceding vehicle AB. Also, if the vertical edge is weak but the edge direction vector has a high variance and the relative speed is high, it is determined as a person PE. Further, when the vertical edge and the edge strength are weak and the direction vector dispersion is strong as in the case of a person, but the reflection strength is strong or the relative speed is low, it is determined as a traveling motorcycle MS. Further, when the relative speed is high and the vertical edge or the edge strength is strong, it is determined as a road structure (wall WO or the like).

  In the first embodiment, these determinations are performed in the voting table for each object type or each layer of the area, and the type discrimination result is reflected in one voting table TS, and the characteristics differ depending on the object type. Therefore, multiple types of determination results do not appear in the same region. That is, if it is determined that the person is a person PE in the voting table or hierarchy for a person, it is not determined as a preceding vehicle AB or a motorcycle MS in the voting table or hierarchy for other types in the same area. .

Next, the reliability determination process in step S8 executed by the reliability determination processing unit 17 will be described.
In the reliability determination process, as described above, the reliability of the object detection result in step S7 is determined based on the weather environment obtained in the external environment detection process in step S4.

  In the first embodiment, the reliability determination process is performed based on the correlation characteristic for each type of detected object illustrated in FIG. 14 and the effectiveness of each information according to the weather environment.

  That is, in the first embodiment, the above-described coefficient indicating the necessity of multiplying each detection information used for the object determination shown in FIG. 14 and the effectiveness according to the weather environment of each detection information are multiplied. The sum of the values is defined as a confidence value AA. Here, the effectiveness according to the weather environment of the information for detection can be determined based on the effectiveness characteristic for each weather in FIG. 13 or is actually measured with respect to the value obtained in the ideal state. It can also be determined by the ratio of absolute values. In the first embodiment, the ratio of the absolute value actually measured to the value obtained in the latter ideal state, that is, the value obtained by dividing the actual measured value by the ideal value is assumed.

  An example of this reliability determination will be specifically described by taking a case where “person PE” is detected as an example. For example, for detection of “person PE”, as shown in FIG. 11, vertical edges, direction vector dispersion, and relative speed are extracted. Based on the correlation characteristics shown in FIG. 14, the vertical edge weighting is low (coefficient 0.2), the direction vector variance weighting is high (coefficient 1.0), and the relative speed weighting is low (coefficient 0.2). ) Is set.

Therefore, the confidence value AA, which is the sum of values obtained by multiplying each detection information by the coefficient multiplied by the effectiveness of each information according to the weather environment, is expressed by the following equation (4) when the person PE is detected. Is done.
AA = 0.2 × vertical edge effectiveness + 1.0 × direction vector dispersion effectiveness + 0.2 × relative velocity effectiveness (4)

In this equation (4), assuming that the effectiveness of all information in the ideal weather environment is “1”, the reliability value AA is 1.4 from the following equation (5).
AA = 0.2 × 1 + 1.0 × 1 + 0.2 × 1 = 1.4 (5)

  In the first embodiment, the reliability when “person PE” is detected is obtained based on 1.4 (= reference value).

  For example, when the weather environment is “good weather”, the vertical edge, the direction vector dispersion, and the relative speed are all the same as in the ideal state, so AA = 1.4 by the above equation (5), and the reference value The comparative value is obtained as 100%. In this case, it is determined that the reliability is high.

  On the other hand, for example, in the case of “backlight”, the effectiveness of information related to image processing decreases. For example, assuming that the effectiveness (measured value / ideal value) of the vertical edge and the direction vector is about 0.3, substituting this into the above equation (4) results in 0.2 × 0.3 + 1.0 × 0. .3 + 0.2 × 1 = 0.56, and the reliability value AA = 0.56.

  The value of the reliability value AA = 0.56 is 40% of the reference value (1.4), and it is determined that the reliability in this case is quite low.

  On the other hand, let us consider a case where only the effectiveness of the relative speed obtained from the position information of the radar 2 is lowered for some reason. Here, assuming that the effectiveness of the relative speed is 0, the reliability value AA = 0.2 × 1 + 1.0 × 1 + 0.2 × 0 = 1.2 is obtained from the above equation (4).

  In this case, the value of the reliability value AA = 1.2 is 85% as compared with the reference value (1.4), and it is determined that the reliability is quite high.

  As described above, the process of setting the sum of values obtained by multiplying the coefficient of necessity for multiplying each piece of detection information and the effectiveness according to the weather environment of each piece of detection information to the reliability value AA is performed for each detection object. Run for each. Then, the reliability is determined by comparing the reliability value AA for each detected object with the reference value in the ideal state.

Next, the control determination process in step S9 will be described.
This control determination process is executed in the driving support control processing unit 20. The driving support control processing unit 20 is a part that executes control of a driving support device (not shown). For example, the driving support control processing unit 20 performs control so as to automatically travel within the white line of the road, or travels following the preceding vehicle AB. Drive support control.

  Therefore, in the control determination process, the support control content of the driving support control processing unit 20 is determined based on the weather environment obtained in step S4 and the reliability obtained in step S8.

  That is, when the reliability is higher than a predetermined value (for example, 60%), the driving support control is executed, but when the reliability is lower than the predetermined value, the driving support is stopped.

  Even if the reliability is higher than a predetermined value, the driving support is also stopped when it is determined in step S4 that the weather is not suitable for driving support such as “heavy rain” or “heavy snow”.

  Here, as the reliability, an average value of reliability of all detected objects may be used, or in the case of follow-up control of an object important for control, for example, preceding vehicle AB, depending on the content of driving support control. The reliability of the preceding vehicle AB may be used.

  Furthermore, in the control determination process, when it is determined that the backlight is used, a notification process for reporting the presence of the detected object is executed using a display device and a speaker of a navigation system (not shown). Specifically, for example, the type of an object existing ahead is notified by voice, or the position of the object is displayed on a display device of the navigation system.

  As described above, in the control determination process, it is determined whether to perform the driving support control and whether to perform the notification process according to the reliability and the weather environment.

  As described above, in the object detection device of the first embodiment, the weather environment as the external environment can be determined based on the input information from the camera 1 and the radar 2 as the object detection sensor. For this reason, even in places where weather information cannot be obtained from external communication equipment such as VICS, it is possible to acquire actual weather information at a position where object detection is performed. In addition, the vehicle MB equipped with the object detection device of the first embodiment moves every moment by traveling, but it is possible to acquire actual weather information of the traveling road.

  Therefore, there is no problem that the weather information cannot be obtained depending on the road as in the past, and the actual weather at the position where the vehicle is traveling is different from the input weather information. It is possible to perform object detection according to the actual weather environment of the traveling position, and it is possible to improve the object detection accuracy as compared with the conventional case.

  In the first embodiment, the conversion information and the input information are voted on the voting table TS, and the type of the object is determined based on the voting result.

  For this reason, the condition that any sensor must detect the object to be detected (for example, the condition that both the camera 1 and the radar 2 must detect) is eliminated. Therefore, even in an environment where only one of the camera 1 and the radar 2 can be detected, an object can be detected and its type can be determined, and an effect that a robust detection is possible can be obtained. In addition, at the same time as this effect, it is possible to maintain the advantage that a highly reliable measurement effect can be obtained at the same time by mounting a plurality of object sensors such as the camera 1 and the radar 2.

  Moreover, in the first embodiment, since the detection information is weighted based on the effectiveness according to the weather environment, the influence of each information change due to the weather is suppressed, and the detection accuracy is improved. Can do.

  That is, in the first embodiment, the weight of the effectiveness of information from the camera 1 is set lower than that in good weather during bad weather when the image of the camera 1 is unclear. Furthermore, even in the information from the radar 2 that is not easily affected by bad weather, the weight of the reflection intensity that tends to be affected by bad weather is set lower in bad weather than in good weather, The effectiveness weight of relative speed that is not easily affected by bad weather is set high.

  Thus, by setting the weighting during bad weather, it is possible to improve the accuracy of detecting an object during bad weather. If the explanation is added, the influence of the noise component becomes large in the information that is affected by the bad weather, for example, the average value decreases. By reducing the weighting of such information, noise components can be removed. Conversely, information that is not easily affected by bad weather and that can maintain a high average value, for example, is also less influenced by noise components, and detection accuracy can be improved by increasing the weighting of such information.

  Furthermore, in the first embodiment, the weather environment can be determined not only by weather such as good weather and bad weather, but also by backlight that is affected by image information even in good weather, so that the control accuracy according to the external environment can be improved. Further improvement can be achieved.

  That is, in the case of backlight, as shown in FIG. 12, the weighting of the effectiveness of the reflection intensity and the relative speed, which is information from the radar 2 that is not affected by the backlight, is increased, while the information by the camera 1 that is affected by the backlight is increased. Has stopped using. In this way, by setting the weighting according to the external environment such as backlight, it is possible to detect an object with high accuracy even in backlight.

  In addition, according to the first embodiment, in accordance with the type of the object to be determined, the information for detection corresponding to the type is extracted and voted. Weighting according to the height and weighting according to the effectiveness of the detection information are performed according to the correlation with the object to be detected. This makes it possible to detect objects and determine object types using only highly effective information according to the weather environment and correlation with objects, improving object detection reliability and object type determination reliability. Can be made.

  In addition, since only necessary information used for object detection is extracted, it is effective in reducing the memory capacity for storing information and reducing the amount of calculation, and reducing the number of information used in detection processing. It is also possible to simplify the detection process.

  Further, in the first embodiment, in the information conversion process for converting the input information, the edge is extracted from the image information, the optical flow is calculated, and the conversion information is used in the subsequent detection process. It is possible to improve the reliability in the type discrimination. That is, in general, an artificial object such as a road structure (wall WO or the like) existing on the road such as a preceding vehicle AB or a guard rail often has a strong edge strength, whereas a person PE or a two-wheeled vehicle MS in a riding state has an edge strength. weak. Further, the direction vector dispersion by the optical flow is low in the preceding vehicle AB and the preceding two-wheeled vehicle MS having a low relative speed, whereas the person PE and road structure (wall WO or the like) having a high relative speed are high. Thus, the correlation with the type of object is high. Since the object detection processing is performed by converting the information into such information highly correlated with the type of object, high detection reliability can be obtained. In addition, as described above, since such highly reliable information is added by voting to detect the object and determine the type, the reliability can be improved.

  In the first embodiment, the reliability value AA of the detected object is calculated, and the driving support control is stopped when the reliability is low or in bad weather according to the reliability obtained from the reliability value AA. . For this reason, driving assistance based on information with low reliability and driving assistance in a situation not suitable for driving assistance are not performed, and the reliability of driving assistance control can be improved.

  Further, in the first embodiment, differentiation processing, integration processing, and filter processing for obtaining edges, optical flows, relative speeds, and the like used for object detection are performed immediately after input in the conversion processing unit 11. For this reason, values that are important for object detection can be made less susceptible to noise, and such important values can be used for both external environment detection, object detection, and reliability determination. Thus, a highly reliable determination can be made with simple processing. In addition, by performing such differential processing, integration processing, and filter processing immediately after input, for example, pipeline processing that performs parallel processing while inputting an image becomes possible, thereby enabling high-speed computation.

  Next, an object detection apparatus according to Example 2 of the embodiment of the present invention will be described with reference to FIG. In the description of the second embodiment, the same or equivalent parts as those in the first embodiment will be denoted by the same reference numerals and different parts will be mainly described.

  The object detection apparatus of the second embodiment is an example in which only a part of the first embodiment is changed. That is, in the second embodiment, when voting on the voting table TS in the valid information extraction process, a threshold value is set for the value to be voted, and only information exceeding this threshold value is voted. .

  FIG. 19 shows the voting result. As can be seen from a comparison between FIG. 19 and FIG. 15 of the first embodiment, the small value voted in FIG. 15 is deleted.

  That is, data with a low vote value (height) shown in FIG. 15 is highly likely to be noise. Therefore, as in the second embodiment, by setting a threshold value for the value to be voted, noise can be removed, erroneous detection can be prevented, and detection accuracy can be improved.

Furthermore, this makes it possible to determine the type of an object using only a relatively small number of types of information, and the effect of reducing the memory capacity for storing information and the amount of calculation is further enhanced.
About another structure and an effect, it is the same as that of Example 1, and abbreviate | omits description.

  Next, an object detection apparatus according to Example 3 of the embodiment of the present invention will be described. In the description of the third embodiment, the same or equivalent parts as those in the first embodiment will be denoted by the same reference numerals and different parts will be mainly described.

In the object detection apparatus of the third embodiment, the object detection process and the effectiveness weighting process are different from those of the first embodiment.
In the third embodiment, detection / non-detection of an object is determined based on the edge strength and the direction vector.

Specifically, an equation for obtaining a determination value TT of the following equation (6) is used by using the variance of the edge strength (this is A) and the variance of the direction vector (this is B). Then, when the obtained determination value TT is larger than a preset threshold value T, it is determined that an object exists (object detection). When the determination value TT is less than the threshold value T, no object exists (object Non-detection).
TT = (1-β) A + βB × 2 (6)

  Here, β is a coefficient for weighting effectiveness and is a value less than 1.0. This coefficient β is changed according to the weather, and a larger value is used in bad weather than in good weather.

That is, during bad weather such as “rain” and “fog”, the reliability of the edge vector is low, while the reliability of the direction vector dispersion is high. Therefore, the reliability of the object detection accuracy can be improved by changing the value of the coefficient β according to this reliability.
Other functions and effects are the same as those of the first embodiment, and thus description thereof is omitted.

  Next, an object detection apparatus according to Example 4 of the embodiment of the present invention will be described. Parts that are the same as or equivalent to those of the first embodiment are given the same reference numerals, and different parts will be mainly described.

  In the object detection apparatus of the fourth embodiment, the extraction processing unit 12 is omitted, and the weighting of the effectiveness and the weighting of the necessity are applied to all the information from the camera 1 and the radar 2 and the information converted by the conversion processing unit 11. After that, all these pieces of information are voted to the corresponding area of the voting table TS.

Then, based on the number of pieces of information voted for each area of the voting table TS, it is determined whether or not an object exists in that area, that is, the object detection is determined. Further, the type of object is determined from the type of information voted. The object type is determined based on, for example, the extraction characteristics of information used for determining the object shown in FIG. That is, based on whether the combination of information showing a high value among the information voted for a certain area matches with the combination of information in which “1” of the type of object shown in FIG. Determine the type of object.
About another structure and an effect, it is the same as that of Example 1, and abbreviate | omits description.

  Next, an object detection apparatus according to Example 5 of the embodiment of the present invention will be described. Parts that are the same as or equivalent to those of the first embodiment are given the same reference numerals, and different parts will be mainly described.

  The object detection apparatus of Example 5 is an example in which an infrared camera (not shown) is used in parallel with the camera 1 as an object sensor. In addition, the example of an image of these cameras 1 and an infrared camera is shown to FIG.5 (b) (c) and (b) (c) of FIG. 15 and 19 also show the temperature image SP.

  An infrared camera is a camera that can convert a value corresponding to temperature into a pixel value. Here, it is generally difficult to distinguish a person riding on the two-wheeled vehicle MS and a person (pedestrian) PE from the image processing of the luminance camera 1 alone.

  In addition, the reflection intensity and the relative speed, which are information from the radar 2, differ from each other. In particular, when the speed of the two-wheeled vehicle MS is low, the difference in the relative speed becomes small, and the person PE and the two-wheeled vehicle MS are different. Discrimination becomes difficult.

  Therefore, in the fifth embodiment, utilizing the fact that the temperature of the muffler of the two-wheeled vehicle MS is much higher than the temperature of the person PE, based on the temperature information obtained from the infrared camera, When information with high temperature is included, it is determined as a motorcycle MS, and when information with high temperature is not included, it is determined as a person (pedestrian) PE.

  In addition, the presence / absence of a region having a high temperature is determined by determining the pixel value (s) at the position where the person PE or the two-wheeled vehicle MS is detected on the infrared image, and the pixel has a threshold value or more. When there are a predetermined number (for example, three pixels) or more of pixels having the pixel value of, the motorcycle is determined to be a motorcycle MS. Here, the number of pixels having a pixel value equal to or greater than the threshold value is determined based on, for example, that at least three pixels are continuous, not single (one pixel) so as not to cause noise. . In addition, the threshold value of the temperature (pixel value) is set to, for example, about 45 ° C. or more as a temperature that cannot be observed by the human body.

  As described above, in the object detection device according to the fifth embodiment, it is possible to improve the accuracy of discrimination between the person PE and the two-wheeled vehicle MS which are usually difficult to discriminate because of their similar shapes.

In addition, in terms of artifacts, the distinction between the preceding vehicle AB and road structures (walls WO, etc.) that have common points is also made clear by adding the temperature element to the discrimination. Accuracy can be improved.
About another structure and an effect, it is the same as that of Example 1, and abbreviate | omits description.

  As described above, the embodiment of the present invention and Examples 1 to 5 have been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment and Examples 1 to 5. Design changes that do not depart from the gist of the present invention are included in the present invention.

  For example, in the first to fifth embodiments, the object detection method and the object detection apparatus of the present invention are implemented by being mounted on a vehicle. However, the present invention is not limited to this, and the present invention is applicable to other than industrial vehicles such as industrial robots. Can do.

  In the first to fifth embodiments, in the extraction process, only information necessary for object detection is extracted for each type of object. At the time of extraction, the validity of the extracted information is determined. Also good. If it is determined that the effectiveness is low, erroneous detection can be prevented by not performing the determination using the information.

  Moreover, in determining the weather as the external environment, the first embodiment shows an example in which the determination is made based on the edge intensity and the reflection intensity, but the present invention is not limited to this. For example, as the information of the camera 1, the number of edges can be used. In this case, for example, the weather is more favorable as the number of edges is larger, and the weather is determined to be worse as the number of edges is smaller. Further, the number of measurement points and the intensity dispersion value can be used as information of the radar 2.

  Further, in the determination of “backlight”, in Example 1, the reflection intensity by the radar 2 is observed in the entire areas LL1 to LL6, whereas the edge intensity by the camera 1 is extremely low in some areas. Furthermore, when the luminance of a part of the area is high, it is determined as “backlight”, but the present invention is not limited to this. For example, the above luminance need not be added to the determination condition. Alternatively, in the case of backlighting, when the host vehicle MB changes its direction, the edge strength of all the regions Im1 to Im6 increases, or the edge strength of some regions suddenly decreases. Changes are observed. Therefore, it may be determined that the backlight is backlit when a part of the edge intensity changes in time series.

  In the embodiment, the weather environment is discriminated by the camera 1 and the radar 2 of the object detection device. However, in addition to this, signals from an in-vehicle raindrop sensor, wiper device, or outside temperature sensor may be used. Good. By using these signals, the weather environment can be determined with higher accuracy.

  Further, in the first embodiment, in the reliability determination, the example in which the effectiveness of the detection information is calculated based on the measurement value / ideal value is shown, but the present invention is not limited to this, and the detected weather environment Accordingly, the effectiveness may be determined based on the effectiveness characteristic of FIG.

  Further, in the determination of the reliability, in the first embodiment, the example in which the determination is made by the sum of the effectiveness of the information used for detecting the object and the weighting coefficient corresponding to the correlation characteristic is shown. It is not limited. For example, in the case of detecting an object using five types of information, if four types are not detected, it is determined that the reliability is low, and if four types or more are detected, the reliability is high. It can also be judged.

  Further, in the determination of the reliability, when an effective signal cannot be observed for a long time from the camera 1 and the radar 2 as the object sensor, it is determined that the object sensor itself has failed and it is determined that the reliability is low. It may be. When it is determined that there is a failure, the driving support control processing unit 20 executes fail-safe control set in advance. As an example of this fail-safe control, it is possible to detect an object using only information of an effective object sensor and notify a driver or the like that such fail-safe control is being executed.

  Moreover, although Example 1-5 showed what determines a weather environment as an external environment determination process, as an external environment, it is not limited to a weather environment. For example, the traveling environment may be determined. Specific examples of the driving environment include road surface conditions such as gravel roads and rough roads, and noise superposition due to lightning and deterioration over time of the device.

  That is, when traveling on a gravel road or a rough road, the camera 1 and the radar 2 vibrate up and down with the vehicle body. For this reason, the whole image obtained by the camera 1 becomes unclear, and the entire edge obtained by this image information becomes unclear. Similarly, the position information obtained by the radar 2 is also discrete because the measurement range of the radar 2 changes as the vehicle body moves up and down. Therefore, from such a phenomenon, it can be judged as a rough road or a gravel road. Further, in such a road surface state, the reliability of object detection is reduced and the road surface state is not suitable for driving support. Therefore, the driving support control processing unit 20 may stop driving support.

  On the other hand, when noise is superimposed due to the occurrence of lightning or device deterioration, the image may become unclear, the edge may become unclear, or the position information may be blurred, as in the case of the rough road. May occur. Therefore, in such a case, it can be determined that the environment is where noise is superimposed. Even in such a case, since the reliability is lowered, it is possible to stop the driving support or to inform the driver of the situation by a screen or sound.

  Moreover, in Example 1, when performing weighting according to a weather environment, as shown in FIG. 13, it performed based on the characteristic beforehand memorize | stored in the database. However, the weighting is not limited to the one performed based on the preset value as described above, and correlates with the input state of the image information such as the edge strength or the average value or the maximum value, for example. Weighting may be performed based on the value. Specifically, when the edge strength, the average value or the maximum value thereof is relatively low, the weighting of the information about the image is reduced or the information obtained from the radar 2 is reduced as compared with the case where the edge strength is relatively high. The weight may be increased.

It is the schematic which shows vehicle MB carrying the object detection apparatus of Example 1 of embodiment of this invention, (a) is the figure seen from the side, (b) is the figure seen from upper direction. It is a flowchart which shows the flow of the object detection control in the object detection apparatus of the said Example 1. FIG. It is a block diagram which shows notionally the part which performs each process performed by control unit CU in the object detection apparatus of the said Example 1. FIG. It is a conceptual diagram explaining the outline of the input process in the object detection control of the object detection apparatus of the said Example 1, and an information conversion process. It is a figure explaining the image information of the camera 1 in the object detection apparatus of the said Example 1, Comprising: (a) shows the state which looked at the object detection apparatus of Example 1 from the side, (b) is the imaging of the camera 1 The brightness | luminance image projected on the surface 1a is shown, (c) has shown the infrared image at the time of using an infrared camera as the camera 1. FIG. It is a figure explaining the image information of the camera 1 in the object detection apparatus of the said Example 1, Comprising: (a) shows the state which looked at the object detection apparatus of Example 1 from upper direction, (b) is the imaging of the camera 1 The brightness | luminance image projected on the surface 1a is shown, (c) has shown the infrared image at the time of using an infrared camera as the camera 1. FIG. It is the schematic which shows the Sobel filter used by the information conversion process of the image information of the camera 1 in the object detection apparatus of the said Example 1. FIG. In the information conversion process of the image information of the camera 1 in the object detection apparatus of the first embodiment, it is an explanatory diagram of how to obtain an edge direction vector, (a) shows a filter for calculating a vertical edge component, (b) Represents a filter for calculating a horizontal edge component, and (c) represents a relationship between the edge strength and the edge direction vector. It is explanatory drawing which shows the state of the distance detection of the image information and radar of the camera 1 in the object detection apparatus of the said Example 1. FIG. It is explanatory drawing which shows the state of the optical flow and the distance detection of a radar in the information conversion process of the image information of the camera 1 in the object detection apparatus of the said Example 1. FIG. It is a characteristic view which shows the extraction characteristic used by the extraction process in the object detection apparatus of the said Example 1. FIG. It is explanatory drawing of an example of the external environment process performed in the external environment detection process part 13 in the object detection apparatus of the said Example 1, (a) shows the example of an image at the time of receiving backlight, (b) shows fog, An example of an image in bad weather such as haze is shown, and (c) shows an example of measurement points by the radar 2. It is a characteristic view which shows the effectiveness weighting characteristic according to the weather used by the effectiveness weighting process performed in the effectiveness weighting process part 14 in the object detection apparatus of the said Example 1. FIG. It is a characteristic view which shows the correlation weighting characteristic used by the correlation weighting process performed in the correlation weighting process part 15 in the object detection apparatus of the said Example 1. FIG. It is explanatory drawing which shows the voting example to the voting table TS used by the object detection process in the object detection apparatus of the said Example 1. FIG. FIG. 4 is an explanatory diagram for explaining a relationship between image information obtained by the camera 1 and measurement points obtained by the radar 2 in the object detection apparatus according to the first embodiment, where (a) represents image information and (b) represents measurement points. . It is explanatory drawing explaining the relationship between the image information by the camera 1, the measurement point by the radar 2, and the vote in the object detection apparatus of the said Example 1, (a) shows the image information which performed the edge process, (b) Indicates the relationship between the measurement points and the voting area. It is explanatory drawing explaining the positional relationship which votes the image information by the camera 1 in the object detection apparatus of the said Example 1 to the ballot table TS, (a) shows the state seen from the side, (b) is from upper direction. Shows the state of viewing. It is explanatory drawing which shows the example of a vote to the ballot table TS used by the object detection process in the object detection apparatus of Example 2 of an embodiment of the invention.

Explanation of symbols

1 Camera (object sensor)
2 Radar (object sensor)
13 External environment detection processing unit 14 Effectiveness weighting processing unit (effectiveness weighting means)
17 Reliability determination processing unit (reliability determination means)
AB Vehicle (object)
CU control unit (object detection means)
MB Vehicle MS Motorcycle (object)
PE person (object)
TR Tree (object)
WO wall (object)

Claims (11)

Image information acquisition means acquires image information related to the object in the object detection target area, and detection information related to the object based on reflection of the detection wave acquired by scanning the object detection target area with the detection wave by the detection information acquisition means. An acquisition step to acquire;
An object detection step of detecting an object based on the acquired image information and detection information;
An object detection method for detecting an object existing in the object detection target region by executing
In the object detection step,
An extraction step of extracting detection information used for object detection including any of the image information, the detection information, and conversion information obtained by converting the information;
An external environment detection step for detecting an external environment including a weather environment that affects the effectiveness of the detection information for the detection of the object based on the image information and the detection information;
In accordance with the external environment detected by the external environment detection step, an effectiveness weighting step for weighting the detection information based on an effectiveness characteristic set in advance corresponding to the external environment,
Voting the weighted information for detection on a voting table corresponding to the object detection target area, and performing a process of determining whether an object exists in the object detection target area based on the voting result Detecting the object. A method for detecting the object. An object sensor for acquiring information about an object in the object detection target area;
Object detection means for detecting an object based on information obtained from the object sensor;
An object detection device comprising:
As the object sensor, and the image information acquisition means for acquiring image information relating to an object of the object detection target region, the detection information about the object based on the reflection of the object detection target area detection wave acquired by scanning in the detection wave Detection information acquisition means for acquiring,
The object detection means includes an extraction processing unit that extracts detection information used for object detection including any of the image information, the detection information, and conversion information obtained by converting the information, the image information, and the detection Based on the information, according to the external environment detection means for detecting the external environment including the weather environment that affects the effectiveness of the detection information for the detection of the object, and the external environment detected by the external environment detection means, Effectiveness weighting means for weighting the detection information based on effectiveness characteristics set in advance corresponding to the external environment, and the weighted detection information as the object detection target vote in the vote table corresponding to the area, the object detection processing unit for detecting the object that voting results by performing a process for determining whether the object in the object detection target area exists Object detection apparatus characterized by comprising a. The said external environment detection means detects a weather environment based on the edge intensity | strength of the object image calculated | required from the said image information, and the reflection intensity of the object calculated | required from the said detection information. Object detection device. As the types of the weather environment, “sunny weather” in which both the edge intensity average value that is an average value of the edge intensity for a predetermined time and the reflection intensity average value that is an average value of the reflection intensity for a predetermined time are relatively high “Good weather” such as “lightly cloudy” and “edge”, “rain” and “light snow” such as “haze”, “rain” and “light snow” in which the edge intensity average value and the reflection intensity average value are relatively lower than those in good weather. “Slightly bad weather” and “Big bad weather” such as “Dense fog”, “Torrential rain” and “Heavy rain” in which the edge intensity average value is significantly lower than that in good weather and the reflection intensity average value is also reduced. The object according to claim 3, wherein the edge intensity is observed only in a partial area of the image information and the reflection intensity is set to be high “backlight” throughout the detection information. Detection device.   5. The object detection device according to claim 2, wherein the object detection unit detects an object that is mounted on a vehicle and exists at least in a traveling direction of the vehicle. The said object detection means is provided with the reliability determination means which determines the reliability of the detection result of the said object detection means according to the external environment which the said external environment detection means detects. The object detection device according to any one of 5. The reliability determination means determines the reliability of the detection result based on a sum of values obtained by multiplying a coefficient corresponding to the effectiveness for each detection information effective for the detection for each type of object. The object detection apparatus according to claim 6 . The object detection apparatus according to claim 6, further comprising a changing unit that changes a process to be performed based on object detection based on a determination result of the reliability determination unit . The detection information includes any one of an edge obtained from the image information, an edge intensity, a direction vector dispersion, a reflection intensity obtained from the detection information, and a relative velocity. Item 9. The object detection device according to any one of Items 2 to 8. The object detection apparatus according to claim 2, wherein the extraction processing unit extracts only information effective for detection from the detection information for each type of object to be detected. . The object detection unit includes a correlation weighting unit that performs weighting according to the effectiveness of the detection information with respect to the object detection based on the correlation between the type of the object to be detected and the detection information. The object detection apparatus according to any one of claims 2 to 10, wherein the object detection apparatus is any one of claims 1 to 10.

Images