CN112084810B

CN112084810B - Obstacle detection method and device, electronic equipment and storage medium

Info

Publication number: CN112084810B
Application number: CN201910505369.0A
Authority: CN
Inventors: 安建平
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2019-06-12
Filing date: 2019-06-12
Publication date: 2024-03-08
Anticipated expiration: 2039-06-12
Also published as: CN112084810A

Abstract

The embodiment of the invention provides an obstacle detection method, an obstacle detection device, electronic equipment and a storage medium. The method is applied to an on-board advanced driving assistance system and comprises the following steps: when the recognition result of the image to be recognized, which is acquired by the vehicle-mounted image acquisition equipment, represents that a target obstacle exists in the image to be recognized, determining a reference position coordinate of a position point corresponding to the target measured value under a preset reference coordinate system, and determining a reference area coordinate of the target obstacle under the reference coordinate system; when the reference position coordinates are located in the range corresponding to the reference region coordinates, the target measured value is determined to be the distance between the target obstacle and the vehicle, the object size information determined based on the reference region coordinates is determined to be the size information of the target obstacle, and the detection result of the target obstacle is obtained. Compared with the prior art, the scheme provided by the embodiment of the invention can reduce the difficulty of decision making of the decision making module and obtain a better final control strategy.

Description

Obstacle detection method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of driving assistance technologies, and in particular, to a method and apparatus for detecting an obstacle, an electronic device, and a storage medium.

Background

In recent years, with the rapid development of automatic driving technology, environmental perception problems during vehicle running are particularly important, wherein obstacle detection problems are the most important part of the environmental perception problems.

In the related art in the field of ADAS (Advanced Driver Assistance Systems, advanced driving assistance system), obstacle detection is performed using multiple sensors.

Specifically, the plurality of sensors are used for respectively acquiring data, and after the data acquired by each sensor are acquired, the data acquired by each sensor are respectively subjected to obstacle analysis, so that the obstacle detection result corresponding to each sensor is determined. Furthermore, the obtained multiple obstacle detection results can be simultaneously input into the decision module of the ADAS, so that the decision module can determine the vehicle control strategy corresponding to each obstacle detection result, and the determined multiple vehicle control strategies are fused to obtain a final control strategy, so that the vehicle operation can be controlled by using the final control strategy.

Obviously, since the obstacle detection result corresponding to each sensor may have a defect in some aspect, the difficulty of determining the final control strategy by the decision module may be increased, so that it is difficult to obtain a better final control strategy.

Disclosure of Invention

The embodiment of the invention aims to provide an obstacle detection method, an obstacle detection device, electronic equipment and a storage medium, so as to reduce the decision difficulty of a decision module and obtain a better final control strategy. The specific technical scheme is as follows:

in a first aspect, an embodiment of the present invention provides a method for detecting an obstacle, which is applied to a vehicle-mounted advanced driving assistance system, and the method includes:

acquiring an image to be identified acquired by vehicle-mounted image acquisition equipment, and identifying an obstacle of the image to be identified to obtain an identification result;

acquiring a target measured value acquired by a vehicle-mounted ultrasonic sensor;

when the identification result represents that a target obstacle exists in the image to be identified, determining a reference position coordinate of a position point corresponding to the target measurement value under a preset reference coordinate system, and determining a reference area coordinate of the target obstacle under the reference coordinate system;

and when the reference position coordinates are positioned in the range corresponding to the reference region coordinates, determining the target measured value as the distance between the target obstacle and the vehicle, and determining the object size information determined based on the reference region coordinates as the size information of the target obstacle to obtain the detection result of the target obstacle.

As an example, the step of acquiring the image to be identified acquired by the vehicle-mounted image acquisition device includes:

and selecting an image with the smallest difference value between the acquisition time of one frame and the current time from the images acquired by the vehicle-mounted image acquisition equipment as an image to be identified.

As an example, the step of acquiring the target measurement value acquired by the in-vehicle ultrasonic sensor includes:

and selecting the ultrasonic measured value with the smallest difference between the acquisition time and the current time from the ultrasonic measured values acquired by the vehicle-mounted ultrasonic sensor as a target measured value.

As an example, when the recognition result characterizes that there is a target obstacle in the image to be recognized, the steps of determining a reference position coordinate of a position point corresponding to the target measurement value under a preset reference coordinate system, and determining a reference area coordinate of the target obstacle under the reference coordinate system include:

when the identification result represents that a target obstacle exists in the image to be identified and the target measured value is not larger than the preset measurement threshold, determining a reference position coordinate of a position point corresponding to the target measured value under a preset reference coordinate system and determining a reference area coordinate of the target obstacle under the reference coordinate system; wherein the preset measurement threshold is not greater than the detection range of the vehicle-mounted ultrasonic sensor.

As an example, the step of determining the reference position coordinates of the position point corresponding to the target measurement value in the preset reference coordinate system, and determining the reference area coordinates of the target obstacle in the reference coordinate system includes:

determining position point coordinates of a position point corresponding to the target measured value in a preset vehicle coordinate system based on the installation position of the vehicle-mounted ultrasonic sensor;

calculating a reference position coordinate corresponding to the position point coordinate under a preset reference coordinate system by using a preset coordinate conversion formula, wherein the reference coordinate system is an image coordinate system of the image to be identified, and the coordinate conversion formula is a coordinate conversion formula from the vehicle coordinate system to the reference coordinate system;

and determining an image area of the target obstacle in the image to be identified, and determining image area coordinates of the image area in the image coordinate system as reference area coordinates of the target obstacle in the reference coordinate system.

As an example, the step of identifying the obstacle to the image to be identified to obtain an identification result includes:

performing obstacle recognition on the image to be recognized to obtain an initial result, and judging whether the initial result represents that an obstacle exists in the image to be recognized;

If yes, taking the initial result as a recognition result;

if not, judging whether obstacles exist in a plurality of continuous images to be recognized before the images to be recognized;

when the fact that a plurality of continuous images to be identified exist in the images to be identified before the images to be identified is judged, determining a tracking result of the images to be identified as an identification result by utilizing an image tracking algorithm based on the initial result and the identification result of the images to be identified;

otherwise, the initial result is taken as a recognition result.

As an example, after the step of acquiring the target measurement value acquired by the in-vehicle ultrasonic sensor, the method further includes:

correcting the target measured value to obtain a corrected target measured value;

the step of determining the target measurement as a distance of the target obstacle from the vehicle includes:

and determining the corrected target measured value as the distance between the target obstacle and the vehicle.

As an example, the method further comprises:

when the reference position information is not located in the reference area coordinates included in the identification result, determining the reference position coordinates as position information of a first obstacle detected by the vehicle-mounted ultrasonic sensor, determining the target measured value as the distance between the first obstacle and a vehicle, and obtaining a detection result of the first obstacle; determining object size information determined based on the reference area coordinates as size information of the target obstacle, determining a monocular ranging distance as a distance between the target obstacle and a vehicle, and obtaining a detection result of the target obstacle; the monocular distance measurement method comprises the step of detecting the monocular distance measurement distance based on the image to be identified by utilizing a monocular distance measurement algorithm.

In a second aspect, an embodiment of the present invention provides an obstacle detection device applied to an advanced driving assistance system on a vehicle, the device including:

the image data acquisition module is used for acquiring an image to be identified acquired by the vehicle-mounted image acquisition equipment, and identifying an obstacle of the image to be identified to obtain an identification result;

the ultrasonic data acquisition module is used for acquiring a target measured value acquired by the vehicle-mounted ultrasonic sensor;

the coordinate information determining module is used for determining a reference position coordinate of a position point corresponding to the target measured value under a preset reference coordinate system when the identification result represents that the target obstacle exists in the image to be identified, and determining a reference area coordinate of the target obstacle under the reference coordinate system;

and the detection result determining module is used for determining the target measured value as the distance between the target obstacle and the vehicle when the reference position coordinate is positioned in the range corresponding to the reference region coordinate, determining the object size information determined based on the reference region coordinate as the size information of the target obstacle, and obtaining the detection result of the target obstacle.

As an example, the image data acquisition module is specifically configured to:

As an example, the ultrasonic data acquisition module is specifically configured to:

As an example, the coordinate information determination module is specifically configured to:

determining position point coordinates of a position point corresponding to the target measured value in a preset vehicle coordinate system based on the installation position of the vehicle-mounted ultrasonic sensor; calculating a reference position coordinate corresponding to the position point coordinate under a preset reference coordinate system by using a preset coordinate conversion formula, wherein the reference coordinate system is an image coordinate system of the image to be identified, and the coordinate conversion formula is a coordinate conversion formula from the vehicle coordinate system to the reference coordinate system; and determining an image area of the target obstacle in the image to be identified, determining an image area coordinate of the image area in the image coordinate system, and determining a reference area coordinate of the target obstacle in the reference coordinate system.

As an example, the image data acquisition module is specifically configured to:

performing obstacle recognition on the image to be recognized to obtain an initial result, and judging whether the initial result represents that an obstacle exists in the image to be recognized; if yes, taking the initial result as a recognition result; if not, judging whether obstacles exist in a plurality of continuous images to be recognized before the images to be recognized; when the fact that a plurality of continuous images to be identified exist in the images to be identified before the images to be identified is judged, determining a tracking result of the images to be identified as an identification result by utilizing an image tracking algorithm based on the initial result and the identification result of the images to be identified; otherwise, the initial result is taken as a recognition result.

As an example, the apparatus further comprises:

an obstacle information determining module, configured to determine, when the reference position information is not located in the reference area coordinates included in the identification result, the reference position coordinates as position information of a first obstacle detected by the vehicle-mounted ultrasonic sensor, determine the target measurement value as a distance between the first obstacle and a vehicle, and obtain a detection result of the first obstacle; determining object size information determined based on the reference area coordinates as size information of the target obstacle, determining a monocular ranging distance as a distance between the target obstacle and a vehicle, and obtaining a detection result of the target obstacle; the monocular distance measurement method comprises the step of detecting the monocular distance measurement distance based on the image to be identified by utilizing a monocular distance measurement algorithm.

In a third aspect, an embodiment of the present invention provides a vehicle-mounted electronic device, on which an advanced driving assistance system is mounted, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;

a memory for storing a computer program;

and a processor, configured to implement any one of the method steps of the method for detecting an obstacle provided in the first aspect when executing the program stored in the memory.

In a fourth aspect, an embodiment of the present invention provides a computer readable storage medium having stored therein a computer program which, when executed by a processor, implements the method steps of any one of the methods for detecting an obstacle provided in the first aspect.

In the above, after the image to be identified and the target measurement value are obtained, the vehicle-mounted advanced driving assistance system can determine whether the vehicle-mounted image acquisition device detects the obstacle through analysis of the image to be identified. Furthermore, when the vehicle-mounted image acquisition device detects an obstacle, the vehicle-mounted advanced driving assistance system can fuse the detection results of the vehicle-mounted image acquisition device and the vehicle-mounted ultrasonic sensor based on the image to be identified and the target measurement value to obtain the detection result of the target obstacle.

In this way, the obtained detection result of the target obstacle is input into the decision module of the vehicle-mounted advanced driving assistance system, and the decision module can directly determine the final control strategy based on the fusion result of the detection results of the vehicle-mounted image acquisition device and the vehicle-mounted ultrasonic sensor. The detection results of the vehicle-mounted image acquisition equipment and the vehicle-mounted ultrasonic sensor can mutually compensate the defects of the detection results of the other side, so that the decision difficulty of the decision module in determining the final control strategy can be reduced, and the better final control strategy is obtained.

Drawings

In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic flow chart of an obstacle detection method according to an embodiment of the present invention;

FIG. 2 is a flow chart of a specific implementation manner of identifying an obstacle in an image to be identified to obtain an identification result;

FIG. 3 is a schematic illustration of ultrasonic measurements taken over a period of time by an on-board ultrasonic sensor;

FIG. 4 is a schematic illustration of the corrected ultrasonic measurement of FIG. 3 corrected;

FIG. 5 is a schematic diagram of a network architecture of a layer LSTM network;

FIG. 6 is a schematic diagram of a three-layer cascaded LSTM network;

FIG. 7 is a flowchart of a specific implementation of determining a reference position coordinate of a position point corresponding to a measured target value in a preset reference coordinate system and determining a reference area coordinate of a target obstacle in the reference coordinate system;

fig. 8 is a flow chart of another method for detecting an obstacle according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of an obstacle detecting device according to an embodiment of the present invention;

fig. 10 is a schematic structural diagram of a vehicle-mounted electronic device according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

In the related art, a vehicle-mounted advanced driving assistance system adopts a plurality of sensors to detect obstacles, specifically, a plurality of sensors are used for respectively acquiring data, and after acquiring the data acquired by each sensor, the data acquired by each sensor are respectively subjected to obstacle analysis, so that the obstacle detection result corresponding to each sensor is determined. Furthermore, the obtained multiple obstacle detection results can be simultaneously input into a decision module of the vehicle-mounted advanced driving assistance system, so that the decision module can determine a vehicle control strategy corresponding to each obstacle detection result, and the determined multiple vehicle control strategies are fused to obtain a final control strategy, so that the vehicle operation can be controlled by utilizing the final control strategy. However, since the obstacle detection result corresponding to each sensor may have some defects, the difficulty of determining the final control strategy by the decision module may be increased, resulting in difficulty in obtaining a better final control strategy

In order to solve the technical problems, an embodiment of the present invention provides an obstacle detection method. The method is applied to an advanced driving assistance system on board.

The advanced driving assistance system (Advanced Driver Assistance System, ADAS) utilizes various sensors installed on a vehicle to sense the environment around the vehicle at any time in the running process of the vehicle, collects relevant data of the vehicle and the environment where the vehicle is located, utilizes the collected data to identify, detect and track static and dynamic objects, and combines navigator map data to calculate and analyze, thereby enabling a driver to perceive possible danger in advance and effectively increasing the comfort and safety of the driving of the vehicle.

In addition, at least one image acquisition device and at least one ultrasonic sensor are mounted on the vehicle, so that image acquisition and obstacle ranging of the environment around the vehicle can be achieved. The method can determine whether the obstacles such as motor vehicles, non-motor vehicles, pedestrians and the like exist around the vehicle by utilizing the image acquired by the image acquisition equipment, and determine the distances between various general obstacles existing around the vehicle and the vehicle by utilizing the ultrasonic ranging principle of the ultrasonic sensor. Wherein, general obstacle means: obstacles within a certain distance from the vehicle body are called general obstacles, such as walls, posts, roadblocks, road edges, etc.

The vehicle-mounted image acquisition equipment acquires images according to preset frequency, namely, acquires an image every interval preset time length, and the vehicle-mounted ultrasonic sensor also acquires ultrasonic measurement values every interval preset time length according to preset frequency. The collection frequency of the vehicle-mounted image collection device and the collection frequency of the vehicle-mounted ultrasonic sensor can be the same or different.

In addition, the types and the number of the image pickup devices and the ultrasonic sensors mounted on the vehicle may be determined according to the performance of the advanced driving assistance system mounted on the vehicle in actual use, the road condition of the road on which the vehicle frequently runs, the requirement for the running safety of the vehicle, and the like. The embodiment of the present invention is not particularly limited in this regard.

For example, the installed image pickup apparatuses may be fish-eye cameras, the number of which is 4, respectively installed in four directions of the front, rear, left and right of the vehicle body. In this way, the photographing range of the image pickup device can cover all areas around the vehicle. The installed ultrasonic sensors can be long-distance ultrasonic sensors and short-distance ultrasonic sensors, wherein the number of the long-distance ultrasonic sensors is 4, 2 ultrasonic sensors are installed on each side of the side edge of the vehicle body, the effective detection distance is 5m, namely the detection range of the long-distance ultrasonic sensors is 5m; the number of the short-distance ultrasonic sensors is 8, 4 short-distance ultrasonic sensors are respectively arranged at the head and the tail of the vehicle, the effective detection distance is 2.5m, and the detection range of the short-distance ultrasonic sensors is 2.5m.

Fig. 1 is a schematic flow chart of an obstacle detection method according to an embodiment of the present invention. As shown in fig. 1, the obstacle detection method may include the steps of:

s101: acquiring an image to be identified acquired by vehicle-mounted image acquisition equipment, and identifying an obstacle of the image to be identified to obtain an identification result;

s102: acquiring a target measured value acquired by a vehicle-mounted ultrasonic sensor;

s103: when the identification result represents that the target obstacle exists in the image to be identified, determining a reference position coordinate of a position point corresponding to the target measurement value under a preset reference coordinate system, and determining a reference area coordinate of the target obstacle under the reference coordinate system;

s104: when the reference position coordinates are located in the range corresponding to the reference region coordinates, the target measured value is determined to be the distance between the target obstacle and the vehicle, the object size information determined based on the reference region coordinates is determined to be the size information of the target obstacle, and the detection result of the target obstacle is obtained.

It will be appreciated that, in accordance with the above description of the image capturing device and the ultrasonic sensor mounted on the vehicle, the vehicle-mounted image capturing device and the vehicle-mounted ultrasonic sensor each perform data capturing at respective corresponding frequencies.

For the vehicle-mounted image acquisition device, when one image related to the surrounding environment of the vehicle is acquired each time, the vehicle image acquisition device can acquire and record the time of acquiring the image, namely the acquisition time of acquiring and recording the image. Correspondingly, for the vehicle-mounted ultrasonic sensor, when one ultrasonic measured value is acquired each time, the vehicle-mounted ultrasonic sensor can acquire and record the time for acquiring the ultrasonic measured value, namely the acquisition time for acquiring and recording the ultrasonic measured value.

Further, based on this, in the above steps S101 and S102, the advanced driving assistance system on the vehicle may also acquire the image to be recognized and the target measurement value at a preset frequency.

It can be understood that the frequency corresponding to the vehicle-mounted advanced driving assistance system, the acquisition frequency of the vehicle-mounted image acquisition device and the acquisition frequency of the vehicle-mounted ultrasonic sensor can be the same or different. Obviously, when the three are the same, the image to be identified acquired by the vehicle-mounted advanced driving assistance system is the image acquired by the vehicle-mounted image acquisition equipment at the current moment, and the target measured value acquired by the vehicle-mounted advanced driving assistance system is the ultrasonic measured value acquired by the vehicle-mounted ultrasonic sensor at the current moment.

The present time is the time at which the in-vehicle advanced driving support system performs the present obstacle detection, and the in-vehicle advanced driving support system acquires the image to be recognized in step S101 and the target measurement value in step S102.

As an example, in order to ensure that the detected result of the detected target obstacle may be relatively matched with an obstacle actually existing in the environment where the vehicle is currently located, the method for acquiring the image to be identified acquired by the vehicle-mounted image acquisition device by the vehicle-mounted advanced driving assistance system to perform the above step S101 may include:

That is, in this example, the image to be recognized acquired by the vehicle-mounted advanced driving assistance system is an image whose acquisition time is closest to the current time among the multiple frame images acquired by the vehicle-mounted image acquisition device.

In addition, it can be understood that, when the image closest to the current moment in the collection moment in the multi-frame images collected by the vehicle-mounted image collection device cannot be used as the image to be identified due to image blurring and the like, the vehicle-mounted advanced driving assistance system can also obtain the image with the inferior difference value between the collection moment and the current moment in the collected multi-frame images as the image to be identified.

In this way, after the image to be identified is obtained, the vehicle-mounted advanced driving assistance system can identify the obstacle of the image to be identified, and an identification result is obtained.

The vehicle-mounted advanced driving assistance system can recognize obstacles on the image to be recognized in various modes to obtain a recognition result. The embodiment of the present invention is not particularly limited in this regard.

As an example, the advanced driving assistance system on board the vehicle may perform image detection on an image to be recognized using various obstacle detection algorithms, resulting in an image detection result as a recognition result.

For example, the obstacle detection algorithm may be: machine learning methods based on HOG (Histogram of Oriented Gradient, directional gradient histogram) features and SVM (support vector machine ) networks. Of course, other obstacle detection algorithms may be used to perform image detection on the image to be identified to obtain an image detection result, and the embodiment of the present invention is not limited in detail.

As an example, the vehicle-mounted advanced driving assistance system may input an image to be identified into a preset obstacle detection model, obtain a detection result output by the obstacle detection model, and use the detection result as the identification result.

For example, the preset obstacle detection model may be: deep learning model based on CNN (Convolutional Neural Network ). Of course, other preset obstacle detection models may be adopted to obtain the recognition result, and the embodiment of the present invention is not limited in detail.

The obstacle detection model is obtained by training a preset initial model based on a plurality of sample images and labels of each sample image. Each sample image marks the image area where the obstacle exists in the sample image, and the sample label of the sample image is the type of the obstacle existing in the sample image. And inputting the plurality of sample images and the labels of each sample image into a preset initial model for training until convergence conditions are met, and obtaining a trained obstacle detection model.

Obviously, when the detection result of the obstacle detection model represents that the target obstacle exists in the image to be identified, the detection result can comprise the image area of the target obstacle in the image to be identified and the type of the target obstacle.

The obstacle detection model may be obtained by local training of the vehicle-mounted advanced driving assistance system, or may be obtained by the vehicle-mounted advanced driving assistance system from other electronic devices in communication connection.

In some cases, an obstacle existing around the vehicle may be blocked by other things, or an image of the obstacle is not acquired in an image to be recognized acquired in an advanced driving assistance system of the vehicle despite the obstacle existing around the vehicle due to the limitation of the angle and the photographing range of the image acquisition apparatus, thereby causing the recognition result obtained by performing obstacle detection on the image to be recognized to indicate that no obstacle exists in the image to be recognized. Obviously, in this case, the decision module of the advanced driving assistance system on board may be caused to make an inappropriate control strategy, increasing the risk of running the vehicle.

Thus, for this case, as an example, as shown in fig. 2, the manner in which the vehicle-mounted advanced driving assistance system performs the above-described step S101 to recognize the obstacle to the image to be recognized and obtain the recognition result may include the following steps:

S201: performing obstacle recognition on the image to be recognized to obtain an initial result, and judging whether the initial result represents that an obstacle exists in the image to be recognized; if yes, go to step S202, otherwise, go to step S203;

s202: taking the initial result as a recognition result;

s203: judging whether barriers exist in a plurality of continuous images to be identified before the images to be identified; if yes, go to step S204, otherwise, go to step S205;

s204: determining a tracking result of the image to be identified based on the initial result and the identification result of the identified image to be identified by using an image tracking algorithm, and taking the tracking result as the identification result;

s205: and taking the initial result as a recognition result.

In this example, the vehicle-mounted advanced driving assistance system first performs obstacle recognition on the image to be recognized by using an obstacle detection algorithm or a preset obstacle detection model, and the like, to obtain an initial result. In this way, the vehicle-mounted advanced driving assistance system can judge whether the initial result represents that an obstacle exists in the image to be identified.

Obviously, when the above-mentioned judgment result is yes, it is indicated that an image of an obstacle around the vehicle is acquired in the image to be recognized, that is, the obstacle around the vehicle is detected by the advanced driving assistance system on the vehicle. Thus, the advanced driving assistance system on board the vehicle can take the initial result as the recognition result.

Correspondingly, when the judgment result is negative, the image of the obstacle around the vehicle is not acquired in the image to be identified, that is, the obstacle around the vehicle is not detected by the vehicle-mounted advanced driving assistance system. Then, there are two possibilities, the first is that there is no obstacle around the vehicle, the second is that there is an obstacle around the vehicle, and the image of the obstacle is not acquired by the image acquisition apparatus because the obstacle is blocked.

Therefore, in order to ensure that in the second situation, the vehicle-mounted advanced driving assistance system can still detect the obstacle around the vehicle, so as to avoid that the decision module of the vehicle-mounted advanced driving assistance system makes an unsuitable control strategy, and increase the running risk of the vehicle. The vehicle-mounted advanced driving assistance system can acquire a plurality of continuous recognized images to be recognized before the images to be recognized, and further judge whether obstacles exist in the continuous plurality of recognized images to be recognized.

Obviously, if there is an obstacle in the above-described continuous plurality of recognized images to be recognized, the current image to be recognized may be in conformity with the above-described second case, i.e., an obstacle exists around the vehicle, and the obstacle is blocked. Therefore, the vehicle-mounted advanced driving assistance system can continuously acquire the recognition result of each of the plurality of continuous recognized images to be recognized. Furthermore, the vehicle-mounted advanced driving assistance system can track the obstacles in the continuous multiple identified images to be identified by utilizing an image tracking algorithm based on the initial result and the acquired identification result of each identified image to be identified, determine the tracking result of the images to be identified, and take the determined tracking result as the identification result of the current images to be identified.

Correspondingly, if no obstacle exists in the plurality of continuous images to be identified, the current image to be identified may be in accordance with the first condition, i.e. no obstacle exists around the vehicle. In this way, the advanced driving assistance system on the vehicle can use the initial result as the recognition result.

Further, if, of the above-described continuous plurality of recognized images to be recognized, no obstacle exists in at least one of the recognized images that is prior to and continuous with the current image to be recognized, and an obstacle exists in the other at least one recognized image to be recognized, the condition that an obstacle exists around the vehicle may be that an obstacle exists before and that the obstacle has disappeared. Therefore, the advanced driving assistance system on board the vehicle can take the above initial result as the recognition result.

In addition, if there is an obstacle in at least one of the plurality of consecutive recognized images that is before the current one and that is consecutive to the one, and there is no obstacle in the other at least one recognized image, then the situation that there is an obstacle around the vehicle may be that there is no obstacle before and that an obstacle appears later. Therefore, the vehicle-mounted advanced driving assistance system can track the obstacles in the continuous multiple identified images to be identified by utilizing an image tracking algorithm based on the initial result and the acquired identification result of each identified image to be identified, determine the tracking result of the images to be identified, and take the determined tracking result as the identification result of the current images to be identified.

Specifically, based on the obtained recognition result of each recognized image to be recognized, an image tracking algorithm is utilized to predict an image area where a target obstacle existing in the image to be recognized is located by combining optical flow information, so as to obtain a tracking result.

After the step S101 is performed, after the image to be identified is obtained and the identification result is obtained, the vehicle-mounted advanced driving assistance system may continue to perform the step S102 to obtain the target measurement value acquired by the vehicle-mounted ultrasonic sensor.

In order to further ensure that the detection result of the detected target obstacle can be matched with the obstacle actually existing in the current environment of the vehicle, as an example, the method for acquiring the target measurement value acquired by the vehicle-mounted ultrasonic sensor by the vehicle-mounted advanced driving assistance system executing the step S102 may include:

That is, in this example, the target measurement value acquired by the in-vehicle advanced driving support system is one of the plurality of ultrasonic measurement values acquired by the in-vehicle ultrasonic sensor, the acquisition time being the closest ultrasonic measurement value to the current time.

In addition, it can be understood that, when, among the plurality of ultrasonic measurement values acquired by the vehicle-mounted ultrasonic sensor, the ultrasonic measurement value whose acquisition time is closest to the current time cannot be taken as the target measurement value due to excessive noise or the like, the vehicle-mounted advanced driving assistance system may also acquire, among the plurality of ultrasonic measurement values acquired, the ultrasonic measurement value whose acquisition time is the next smallest in difference from the current time as the target measurement value.

Further, in practical applications, noise data may exist in the ultrasonic measurement values collected by the vehicle-mounted ultrasonic sensor, as shown in fig. 3, which is a schematic diagram of the ultrasonic measurement values collected by the vehicle-mounted ultrasonic sensor in a period of time, and it can be seen that some maximum or minimum noise data randomly occur in the ultrasonic measurement values.

It can be appreciated that noise data occurring in the ultrasonic measurement values can affect the accuracy of the ultrasonic measurement values acquired by the vehicle-mounted ultrasonic sensor, and further affect the detection accuracy of the vehicle-mounted advanced driving assistance system on the obstacle.

Based on the above, in order to mitigate the influence of noise data appearing in the ultrasonic measurement values on the accuracy of the ultrasonic measurement values acquired by the vehicle-mounted ultrasonic sensor, the detection accuracy of the obstacle by the vehicle-mounted advanced driving assistance system is ensured. As an example, after the step S102 described above is performed, the advanced driving assistance system in the vehicle may perform the following steps:

thus, in the present example, in the step S104 described above, the step of determining the target measurement value as the distance between the target obstacle and the vehicle is: the corrected target measurement value is determined as the distance of the target obstacle from the vehicle.

The vehicle-mounted advanced driving assistance system may correct the target measurement value in various manners, and the embodiment of the invention is not particularly limited to this.

As shown in fig. 3, from an overall point of view, the variation of the ultrasonic measurement values may have a certain continuity and regularity in the ultrasonic measurement values acquired by the in-vehicle ultrasonic sensor for a period of time. Thus, an LSTM (Long Short-Term Memory) network can be used to construct a calibration model to calibrate the ultrasound measurements.

Specifically, in one embodiment, the method for performing the above correction on the target measurement value by the vehicle-mounted advanced driving assistance system to obtain the corrected target measurement value may include the following steps:

inputting the target measured value and n-1 utilized target measured values in the preset correction model in succession before the target measured value, and determining an nth correction result in n correction results output by the correction model as a corrected target measured value;

wherein, the correction model is: taking the 1 st to n th preset sample ultrasonic measurement values as input, taking the 2 nd to n+1st preset sample ultrasonic measurement values as the 1 st to n th preset sample ultrasonic measurement values respectively, and training a preset long-short-term memory LSTM network to obtain n >1.

In this example, after the target measurement value is acquired, the advanced driving assistance system of the vehicle may input the target measurement value and n-1 consecutive n-1 utilized target measurement values before the target measurement value into a preset correction model, that is, n target measurement values are input into the correction model, where the current target measurement value is the nth target measurement value. Thus, n correction results output by the correction model can be obtained. Wherein, since the first n-1 correction results among the obtained correction results are correction results corresponding to n-1 utilized target measurement values in succession before the input target measurement value, respectively, the nth correction result among the n correction results can be determined as the corrected target measurement value.

In this example, the correction model may be obtained by training the advanced driving assistance system locally, or may be obtained by the advanced driving assistance system in the vehicle from other electronic devices connected in communication. This is reasonable.

Next, a specific training process of the above correction model will be described.

Specifically, the correction model is obtained by taking the 1 st to n th preset sample ultrasonic measurement values as input, taking the 2 nd to n+1th preset sample ultrasonic measurement values as the output of the 1 st to n th preset sample ultrasonic measurement values respectively, and training a preset long-short-term memory LSTM network.

The preset training samples may be n+1 continuous target measurement values used by the vehicle-mounted advanced driving assistance system before the current target measurement value, or n+1 sample ultrasonic measurement values obtained by using other electronic devices or ultrasonic sensors. The embodiment of the present invention is not particularly limited in this regard.

In addition, the preset LSTM network may have only one LSTM network structure, as shown in fig. 5, which is a network structure schematic diagram of one LSTM network; the network structure of the multilayer LSTM network can be a cascading LSTM network, as shown in fig. 6, and is a network structure schematic diagram of a three-layer cascading LSTM network.

In embodiment 1, when the preset LSTM network has only one layer of LSTM network structure, taking fig. 5 as an example, training is performed on the preset LSTM network to obtain the correction model in the following manner:

x _i (i∈[1,n]) And y _i (i∈[1,n]) Input and output data of the preset LSTM network respectively, specifically, x _i For the ith sample ultrasonic measurement, y _i Then it is the predicted value of the ith sample ultrasonic measurement and y _i The corresponding target true value is x _i+1 I.e., the i+1th sample ultrasonic measurement. In addition, h _i And (5) setting the network hidden layer state of the preset LSTM network.

In the practical training process, a scheme of predicting the ultrasonic measurement value of the next sample by using the ultrasonic measurement value of the previous sample is adopted, so that the preset LSTM network is a self-supervision network, and no additional labeling is needed for the ultrasonic measurement value of the sample.

In embodiment 2, when the preset LSTM network has a multi-layer LSTM network structure, taking fig. 6 as an example, training the preset LSTM network to obtain the correction model is specifically as follows:

for a first layer LSTM network structure, presetting a sample ultrasonic measurement value x ₁ -x _n Inputting into a first layer LSTM network structure to obtain x ₁ -x _n Respectively corresponding predicted value y ₁ -y _n And determining the predicted value y ₁ -y _n Corresponding true values. Wherein the predicted value y ₁ -y _n The corresponding true value is the ultrasonic measurement value x of the preset sample ₂ -x _n+1 . Further, a loss value between each predicted value and the corresponding real value is calculated. Thus, training of the first layer LSTM network structure of the preset LSTM network can be completed.

For the LSTM network structure of the second layer, the predicted value y of the first layer is calculated ₁ -y _n-1 As input data, input into a second layer LSTM network structure to obtain y ₁ -y _n-1 Respectively corresponding predicted value z ₁ -z _n-1 Wherein the predicted value z ₁ -z _n-1 The corresponding true value is y in the output predicted value of the first layer LSTM network structure ₂ -y _n . And further calculates a loss value between each pre-stored value and the corresponding real value. Thus, training of the second LSTM network structure of the preset LSTM network can be completed.

For the LSTM network structure of the third layer, the predicted value z of the second layer is calculated ₁ -z _n-2 As input data, input into a third layer LSTM network structure to obtain z ₁ -z _n-2 Respectively corresponding predicted value t ₁ -t _n-2 Wherein the predicted value t ₁ -t _n-2 The corresponding true value is z in the output predicted value of the second-layer LSTM network structure ₂ -z _n-1 . And further calculates a loss value between each pre-stored value and the corresponding real value. Thus, training of the third LSTM network structure of the preset LSTM network can be completed.

In the above embodiment 2, since the noise data in the target measurement value can be gradually enhanced in a cascade manner, the step-by-step correction of the target measurement value is realized, so that the corrected target measurement value has a better correction effect, and the detection accuracy of the vehicle-mounted advanced driving assistance system on the obstacle is further improved.

For example, as shown in fig. 4, the corrected ultrasonic measurement value shown in fig. 3 is corrected by using the correction model trained in the above-described example 2. It can be seen that the correction model trained in the above-described example 2 performs good suppression of noise data in the target measurement value.

The execution sequence of step S101 and step S102 may be that step S101 is executed first and then step S102 is executed; step S102 may be executed first, and step S101 may be executed later; it is also possible to perform steps S101 and S102 simultaneously, which is reasonable.

Thus, after the steps S101 and S102 are performed to obtain the recognition result and the target measurement value, the vehicle-mounted advanced driving assistance system may determine whether the recognition result indicates that the target obstacle exists in the image to be recognized. Furthermore, when it is determined that the recognition result indicates that the target obstacle exists in the image to be recognized, the vehicle-mounted advanced driving assistance system may continue to execute the above step S103, determine the reference position coordinates of the position point corresponding to the target measurement value in the preset reference coordinate system, and determine the reference region coordinates of the target obstacle in the reference coordinate system.

The vehicle-mounted advanced driving assistance system may perform the above step S103 in various manners, which is not specifically limited in the embodiment of the present invention.

As an example, as shown in fig. 7, the manner in which the vehicle-mounted advanced driving assistance system performs the determination of the reference position coordinates of the target measurement value corresponding position point in the preset reference coordinate system and the determination of the reference area coordinates of the target obstacle in the reference coordinate system may include the steps of:

s701: determining position point coordinates of a position point corresponding to the target measured value in a preset vehicle coordinate system based on the installation position of the vehicle-mounted ultrasonic sensor;

it can be understood that, according to the installation position of the vehicle-mounted ultrasonic sensor on the vehicle and according to the coordinate operation in the three-dimensional coordinate system, the vehicle-mounted advanced driving assistance system can calculate the coordinate of the vehicle-mounted ultrasonic sensor in the preset vehicle coordinate system. The vehicle coordinate system may be a three-dimensional coordinate system established by taking any point in the vehicle as an origin of coordinates.

Furthermore, according to the measurement principle of the vehicle-mounted ultrasonic sensor and the transmission mode of ultrasonic waves in the air, the vehicle-mounted advanced driving assistance system can determine the position point coordinates of the position point corresponding to the target measurement value in the preset vehicle coordinate system by utilizing the obtained target measurement value according to the coordinate operation in the three-dimensional coordinate system.

The vehicle-mounted advanced driving assistance system may calculate and store the coordinates of the vehicle-mounted ultrasonic sensor in the preset vehicle coordinate system in advance, so that the coordinates may be directly acquired and used when the above step S701 is performed. The coordinates of the in-vehicle ultrasonic sensor in the preset vehicle coordinate system may also be recalculated when the above step S701 is performed.

S702: calculating the corresponding reference position coordinates of the position point coordinates under a preset reference coordinate system by using a preset coordinate conversion formula,

the reference coordinate system is an image coordinate system of an image to be identified, and the coordinate conversion formula is a coordinate conversion formula from a vehicle coordinate system to the reference coordinate system;

in this example, the image coordinate system of the image to be recognized may be determined as the reference coordinate system, and further, a coordinate conversion formula from the vehicle coordinate system to the reference coordinate system may be determined. The coordinate conversion formula may be determined locally by the vehicle-mounted advanced auxiliary driving system, or may be obtained by the vehicle-mounted advanced auxiliary driving system from other electronic devices in communication connection, which is reasonable.

In this way, after the position point coordinates of the position point corresponding to the target measured value in the preset vehicle coordinate system are obtained, the vehicle-mounted advanced auxiliary driving system can calculate the reference position coordinates corresponding to the position point coordinates in the preset reference coordinate system by utilizing the coordinate conversion formula through coordinate operation in the three-dimensional coordinate system.

Preferably, the expression of the above coordinate conversion formula may be as follows:

wherein (X, Y, Z) is the position point coordinate of the position point corresponding to the target measured value in the preset vehicle coordinate system, (f) _x ,f _y ,c _x ,c _y ) And (u, v) is the reference position coordinate corresponding to the position point coordinate under the preset reference coordinate system.

S703: and determining an image area of the target obstacle in the image to be identified, and determining the image area coordinates of the image area in an image coordinate system as the reference area coordinates of the target obstacle in a reference coordinate system.

In this example, since the reference coordinate system is the image coordinate system of the image to be recognized, the advanced driving assistance system on the vehicle can determine the image area coordinates of the image area of the target obstacle in the image to be recognized in the image coordinate system as the reference area coordinates of the target obstacle in the reference coordinate system.

In the above specific implementation manner, the reference coordinate system is an image coordinate system of the image to be identified, and it can be understood that the reference coordinate system may be other coordinate systems.

Based on this, in another specific implementation, the reference coordinate system may be a preset vehicle coordinate system. Thus, in the present example, the manner in which the in-vehicle advanced driving assistance system performs the determination of the reference position coordinates of the target measurement value corresponding position point in the preset reference coordinate system, and the determination of the reference region coordinates of the target obstacle in the reference coordinate system may include the steps of:

Step 1: determining the position point coordinates of the position points corresponding to the target measured values in a preset vehicle coordinate system based on the installation positions of the vehicle-mounted ultrasonic sensors, and determining the position point coordinates as the reference position coordinates of the position points corresponding to the target measured values in the preset reference coordinate system;

as in step S701 in the previous specific implementation, the advanced driving assistance system on board may determine the position point coordinates of the position point corresponding to the target measurement value in the preset vehicle coordinate system based on the installation position of the ultrasonic sensor on board. Further, since the reference coordinate system is the vehicle coordinate system in this example, the in-vehicle advanced driving support system can determine the above-determined position point coordinates as the reference position coordinates of the position point corresponding to the target measurement value in the reference coordinate system.

Step 2: determining an image area of a target obstacle in an image to be identified, and acquiring image area coordinates of the image area in an image coordinate system of the image to be identified; and calculating the reference area coordinates of the image area coordinates under the reference coordinate system by using a preset coordinate conversion formula from the image coordinate system to the vehicle coordinate system.

The vehicle-mounted advanced driving assistance system may first determine an image area of a target obstacle in an image to be identified in the image to be identified, and further obtain an image area coordinate of the image area in an image coordinate system of the image to be identified. Furthermore, based on the imaging principle of the image acquisition device, the vehicle-mounted advanced driving assistance system can calculate the reference area coordinates of the image area coordinates corresponding to the image area of the target obstacle in the image to be identified under the vehicle coordinate system through coordinate operation in the three-dimensional coordinate system by utilizing the coordinate conversion formula from the image coordinate system to the vehicle coordinate system. Thus, since the reference coordinate system is the vehicle coordinate system described above in this example, the advanced driving assistance system on the vehicle can determine the above-described determined reference region coordinate as the reference region coordinate of the image region coordinate corresponding to the image region of the target obstacle in the image to be recognized in the reference coordinate system.

In addition, it can be appreciated that in practical application, in order to ensure the ranging accuracy of the vehicle-mounted ultrasonic sensor, the vehicle-mounted ultrasonic sensor has an effective detection distance. The effective detection distance of the vehicle-mounted ultrasonic sensor is the detection range of the vehicle-mounted ultrasonic sensor. When the ultrasonic measurement value acquired by the in-vehicle ultrasonic sensor exceeds the detection range, the ultrasonic measurement value is invalid.

Therefore, in order to ensure that the target measurement value acquired in the above step S102 is valid for the in-vehicle advanced driving assistance system, and to improve the ranging accuracy of the in-vehicle ultrasonic sensor, further, the detection accuracy of the obstacle by the in-vehicle advanced driving assistance system is improved. As one example, after the target measurement value is obtained, the onboard advanced driving assistance system may first determine the magnitude relation of the target measurement value to a preset measurement threshold. Wherein the preset measurement threshold is not greater than the detection range of the vehicle-mounted ultrasonic sensor.

For example, when the detection range of the vehicle-mounted ultrasonic sensor is 5m, the preset measurement threshold may be 5m, or may be 4m, or may be 3m, as long as the preset measurement threshold is ensured not to be greater than 5 m. The smaller the preset measurement threshold value is, the higher the ranging accuracy of the vehicle-mounted ultrasonic sensor is, and further, the higher the detection accuracy of the vehicle-mounted advanced driving assistance system on the obstacle is, namely, the more reliable the obstacle detection result of the vehicle-mounted advanced driving assistance system is.

When the preset measurement threshold value is smaller than the detection range of the vehicle-mounted ultrasonic sensor, the accuracy of the target measurement value can be further ensured, and then the detection accuracy of the vehicle-mounted advanced driving assistance system to the obstacle is more effectively improved.

Accordingly, in this example, when the vehicle-mounted advanced driving assistance system performs the above step S103 and the recognition result indicates that the target obstacle exists in the image to be recognized, the method for determining the reference position coordinate of the position point corresponding to the target measurement value under the preset reference coordinate system and the reference area coordinate of the target obstacle under the reference coordinate system may include the following steps:

when the identification result indicates that a target obstacle exists in the image to be identified and the target measured value is not greater than a preset measurement threshold, determining a reference position coordinate of a position point corresponding to the target measured value under a preset reference coordinate system and determining a reference area coordinate of the target obstacle under the reference coordinate system; wherein, the preset measurement threshold value is not larger than the detection range of the vehicle-mounted ultrasonic sensor.

In this example, the manner of determining the reference position coordinates of the position point corresponding to the target measurement value in the preset reference coordinate system and the reference area coordinates of the target obstacle in the reference coordinate system is the same as the manner of determining the reference position coordinates of the position point corresponding to the target measurement value in the preset reference coordinate system and the reference area coordinates of the target obstacle in the reference coordinate system in step S103. And will not be described in detail herein.

When the vehicle-mounted advanced driving assistance system corrects the target measurement value, in the above example, the vehicle-mounted advanced driving assistance system determines the magnitude relation between the target measurement value and the preset measurement threshold, that is, determines the magnitude relation between the corrected target measurement value and the preset measurement threshold.

Thus, after the step S103 is performed, the reference position coordinates of the position point corresponding to the target measurement value in the preset reference coordinate system and the reference area coordinates of the target obstacle in the reference coordinate system are obtained, and the vehicle-mounted advanced driving assistance system can determine whether the reference position coordinates are located in the area corresponding to the reference area coordinates.

Obviously, when the reference position coordinates are located in the area corresponding to the reference area coordinates, it may be stated that the target obstacle existing in the image to be identified and the obstacle corresponding to the target measurement value may be the same obstacle. That is, at the present moment, the obstacle detected by the vehicle-mounted image acquisition device and the vehicle-mounted ultrasonic sensor is the same obstacle, and the obstacle is the target obstacle existing in the image to be identified.

In this way, the vehicle-mounted advanced driving assistance system can determine the target measured value as the distance between the target obstacle and the vehicle, and determine the object size information determined based on the reference area coordinates as the size information of the target obstacle, so as to obtain the detection result of the target obstacle.

It will be understood that, with respect to the reference position coordinates of the position point corresponding to the target measurement value in the preset reference coordinate system in the above step S104, the position point may be located in the area corresponding to the reference area coordinates of the target obstacle in the reference coordinate system, or the reference position point may not be located in the area corresponding to the reference area coordinates.

Obviously, when the reference position is not located in the area corresponding to the reference area coordinate, the vehicle-mounted advanced driving assistance system cannot fuse the detection results of the vehicle-mounted image acquisition device and the vehicle-mounted ultrasonic sensor, so that the fusion result cannot be used as the detection result of the target obstacle. In this case, therefore, the advanced driving assistance system on the vehicle needs to obtain respective obstacle detection results for the detection results of the vehicle-mounted image pickup device and the vehicle-mounted ultrasonic sensor, respectively.

Based on this, as shown in fig. 8, another method for detecting an obstacle according to the embodiment of the present invention may include the following steps:

s801: acquiring an image to be identified acquired by vehicle-mounted image acquisition equipment, and identifying an obstacle of the image to be identified to obtain an identification result;

S802: acquiring a target measured value acquired by a vehicle-mounted ultrasonic sensor;

s803: when the identification result represents that the target obstacle exists in the image to be identified, determining a reference position coordinate of a position point corresponding to the target measurement value under a preset reference coordinate system, and determining a reference area coordinate of the target obstacle under the reference coordinate system;

s804: when the reference position coordinates are located in the area range corresponding to the reference area coordinates, determining the target measured value as the distance between the target obstacle and the vehicle, determining the object size information determined based on the reference area coordinates as the size information of the target obstacle, and obtaining a detection result of the target obstacle;

s805: when the reference position information is not located in the reference area coordinates included in the identification result, determining the reference position coordinates as position information of the first obstacle detected by the vehicle-mounted ultrasonic sensor, and determining the target measured value as the distance between the first obstacle and the vehicle to obtain a detection result of the first obstacle; determining object size information determined based on the reference area coordinates as size information of a target obstacle, and determining a monocular ranging distance as a distance between the target obstacle and a vehicle to obtain a detection result of the target obstacle;

The monocular distance measurement method comprises the step of detecting the monocular distance measurement distance based on the image to be identified by utilizing a monocular distance measurement algorithm.

When the determined reference position coordinates of the position point corresponding to the target measured value under the preset reference coordinate system are not located in the range corresponding to the reference region coordinates of the target obstacle under the reference coordinate system, the vehicle-mounted advanced driving assistance system can obtain respective obstacle detection results according to the detection results of the vehicle-mounted image acquisition equipment and the vehicle-mounted ultrasonic sensor.

Specifically, for the detection result of the in-vehicle ultrasonic sensor, the in-vehicle advanced driving assistance system may determine the reference position coordinate of the position point corresponding to the determined target measurement value under the preset reference coordinate system as the position information of the first obstacle detected by the in-vehicle ultrasonic sensor, and determine the target measurement value as the distance between the first obstacle and the vehicle. In this way, the advanced driving assistance system on the vehicle can obtain the position information of the first obstacle detected by the ultrasonic sensor on the vehicle and the distance from the vehicle, so as to obtain the detection result of the first obstacle.

For the detection result of the vehicle-mounted image acquisition device, the vehicle-mounted advanced driving assistance system can firstly determine object size information corresponding to the reference area coordinates of the determined target obstacle under the reference coordinate system, and then determine the object size information as the size information of the target obstacle detected by the vehicle-mounted image acquisition device; furthermore, the vehicle-mounted advanced driving assistance system can also detect the image to be identified by using a monocular ranging algorithm, so as to obtain a monocular ranging distance, and determine the distance as the distance between the target obstacle and the vehicle. In this way, the vehicle-mounted advanced driving assistance system can obtain the size information of the target obstacle detected by the vehicle-mounted image acquisition device and the distance between the vehicle and the target obstacle, so that the detection result of the target obstacle can be obtained.

In this way, when the recognition result indicates that the target obstacle exists in the image to be recognized, the target measured value is not greater than the preset measurement threshold, and the determined reference position coordinate is not located in the area corresponding to the determined reference area coordinate, the vehicle-mounted advanced driving assistance system can still obtain respective obstacle detection results according to the detection results of the vehicle-mounted image acquisition device and the vehicle-mounted ultrasonic sensor.

The specific process of detecting the image to be identified by using the monocular ranging algorithm to obtain the monocular ranging distance can be as follows:

since the relative height between the vehicle-mounted image acquisition device and the ground is fixed in practical application, the distance of each point on the ground can be estimated by using the inverse perspective transformation principle assuming that the ground is a plane. Accordingly, for the target obstacle existing in the image to be recognized, since the target obstacle can be considered to fall on the ground, it can be considered that the midpoint of the bottom edge of the image area where the target obstacle existing in the image to be recognized is also located on the ground. Furthermore, the reverse perspective transformation can be performed on the midpoint of the bottom edge of the image area where the target obstacle exists in the image to be identified, so that the monocular distance measurement distance can be obtained.

Furthermore, the determined detection results of the first obstacle and the determined detection results of the target obstacle, which correspond to the vehicle-mounted image acquisition device and the vehicle-mounted ultrasonic sensor respectively, can be input into a decision module of the vehicle-mounted advanced driving assistance system, so that the decision module of the vehicle-mounted advanced driving assistance system can still obtain a final control strategy to control the vehicle operation.

When the vehicle-mounted advanced driving assistance system determines that the obtained recognition result can represent that the target obstacle exists in the image to be recognized, the image to be recognized can be continuously detected by using a monocular ranging algorithm, so that the monocular ranging distance is obtained, and after the reference area coordinate of the target obstacle under the reference coordinate system is determined, the vehicle-mounted advanced driving assistance system can also determine object size information corresponding to the reference area coordinate based on the determined reference area coordinate of the target obstacle under the reference coordinate system. In this case, the onboard advanced driving assistance system may store the obtained monocular distance measurement and the object size information, so that, when it is determined that the reference position is not located in the area corresponding to the reference area coordinate, the onboard advanced driving assistance system may directly determine the stored monocular distance measurement as the distance between the target obstacle and the vehicle, and determine the stored object size information as the size information of the target obstacle, thereby obtaining the detection result of the target obstacle.

That is, for the detection result of the vehicle-mounted image capturing apparatus, the monocular distance measurement and the object size information may be determined based on the image to be recognized after the step S801 is performed, or may be determined based on the image to be recognized when it is determined that the reference position is not located within the area corresponding to the reference area coordinates. This is reasonable.

The implementation manner of steps S801 to S804 in the embodiment shown in fig. 8 is the same as that of steps S101 to S104 in the embodiment shown in fig. 1, and will not be described herein.

Further, it can be appreciated that, since the performances of the vehicle-mounted image capturing device and the vehicle-mounted ultrasonic sensor may be different, in other cases, a situation may occur in which the recognition result characterizes that a target obstacle exists in the image to be recognized, and the target measurement value is greater than the preset measurement threshold; alternatively, a situation may occur in which the recognition result characterizes that no target obstacle is present in the image to be recognized, and the target measurement value is not greater than a preset measurement threshold. That is, a case may occur in which only one device of the in-vehicle image pickup device and the in-vehicle ultrasonic sensor detects an obstacle.

In order to ensure that the decision-making module of the vehicle-mounted advanced driving assistance system can also obtain a decision-making result to realize the control of the vehicle, the vehicle-mounted advanced driving assistance system can determine a final obstacle detection result aiming at the detection result of the equipment detecting the obstacle, and input the determined final obstacle detection result to the decision-making module of the vehicle-mounted advanced driving assistance system so as to ensure that the decision-making module of the vehicle-mounted advanced driving assistance system can still obtain a final control strategy to control the operation of the vehicle.

Optionally, when the identification result indicates that the target obstacle exists in the image to be identified and the target measurement value is greater than the preset measurement threshold, the vehicle-mounted advanced driving assistance system may determine object size information determined based on the reference area coordinates as size information of the target obstacle, determine the monocular ranging distance as a distance between the target obstacle and the vehicle, and obtain a detection result of the target obstacle. Further, the detection result of the target obstacle is input to the decision module.

Optionally, when the identification result indicates that the target obstacle does not exist in the image to be identified and the target measurement value is not greater than the preset measurement threshold, the vehicle-mounted advanced driving assistance system may determine the reference position coordinate as position information of the first obstacle detected by the vehicle-mounted ultrasonic sensor, determine the target measurement value as a distance between the first obstacle and the vehicle, and obtain a detection result of the first obstacle. Further, the detection result of the first obstacle is input to the decision module.

Corresponding to the method for detecting the obstacle provided by the embodiment of the invention, the embodiment of the invention also provides a device for detecting the obstacle. The device is applied to an advanced driving assistance system on the vehicle.

Fig. 9 is a schematic structural diagram of an obstacle detecting device according to an embodiment of the present invention. As shown in fig. 9, the obstacle detecting apparatus may include the following modules:

the image data acquisition module 910 is configured to acquire an image to be identified acquired by the vehicle-mounted image acquisition device, and identify an obstacle of the image to be identified, so as to obtain an identification result;

the ultrasonic data acquisition module 920 is configured to acquire a target measurement value acquired by the vehicle-mounted ultrasonic sensor;

the coordinate information determining module 930 is configured to determine, when the recognition result indicates that the target obstacle exists in the image to be recognized, a reference position coordinate of a position point corresponding to the target measurement value under a preset reference coordinate system, and a reference region coordinate of the target obstacle under the reference coordinate system;

the detection result determining module 940 is configured to determine the target measurement value as a distance between the target obstacle and the vehicle when the reference position coordinate is located within the range corresponding to the reference region coordinate, determine the object size information determined based on the reference region coordinate as the size information of the target obstacle, and obtain a detection result of the target obstacle.

As an example, the image data acquisition module 910 may be specifically configured to:

As an example, the ultrasonic data acquisition module 920 may be specifically configured to:

As an example, the above-mentioned coordinate information determining module 930 may be specifically configured to determine, when the recognition result indicates that the target obstacle exists in the image to be recognized and the target measurement value is not greater than the preset measurement threshold, a reference position coordinate of the position point corresponding to the target measurement value in the preset reference coordinate system, and determine a reference area coordinate of the target obstacle in the reference coordinate system; wherein, the preset measurement threshold value is not larger than the detection range of the vehicle-mounted ultrasonic sensor.

As an example, the coordinate information determination module 930 may be specifically configured to:

determining position point coordinates of a position point corresponding to the target measured value in a preset vehicle coordinate system based on the installation position of the vehicle-mounted ultrasonic sensor; calculating a reference position coordinate corresponding to the position point coordinate under a preset reference coordinate system by using a preset coordinate conversion formula, wherein the reference coordinate system is an image coordinate system of an image to be identified, and the coordinate conversion formula is a coordinate conversion formula from a vehicle coordinate system to the reference coordinate system; and determining an image area of the target obstacle in the image to be identified, determining the image area coordinates of the image area in an image coordinate system, and determining the image area coordinates as reference area coordinates of the target obstacle in a reference coordinate system.

performing obstacle recognition on the image to be recognized to obtain an initial result, and judging whether the initial result represents that an obstacle exists in the image to be recognized; if yes, taking the initial result as a recognition result; if not, judging whether obstacles exist in a plurality of continuous images to be recognized before the images to be recognized; when the situation that a plurality of continuous images to be identified exist in the images to be identified before the images to be identified is judged, determining a tracking result of the images to be identified based on an initial result and an identification result of the images to be identified by utilizing an image tracking algorithm, and taking the tracking result as an identification result; otherwise, the initial result is taken as the identification result.

As an example, the obstacle detecting apparatus may further include:

the obstacle information determining module is used for determining the reference position coordinates as the position information of the first obstacle detected by the vehicle-mounted ultrasonic sensor when the reference position information is not located in the reference region coordinates included in the identification result, determining the target measured value as the distance between the first obstacle and the vehicle, and obtaining the detection result of the first obstacle; determining object size information determined based on the reference area coordinates as size information of a target obstacle, and determining a monocular ranging distance as a distance between the target obstacle and a vehicle to obtain a detection result of the target obstacle; the monocular distance measurement method comprises the step of detecting the monocular distance measurement distance based on the image to be identified by utilizing a monocular distance measurement algorithm.

Corresponding to the obstacle detection method provided by the embodiment of the invention, the embodiment of the invention also provides vehicle-mounted electronic equipment, and the electronic equipment is provided with an advanced driving assistance system. As shown in fig. 10, includes a processor 1001, a communication interface 1002, a memory 1003, and a communication bus 1004, wherein the processor 1001, the communication interface 1002, the memory 1003 complete communication with each other through the communication bus 1004,

a memory 1003 for storing a computer program;

the processor 1001 is configured to implement the method for detecting an obstacle according to the embodiment of the present invention when executing the program stored in the memory 1003.

Specifically, the obstacle detection method is applied to an advanced driving assistance system installed on the electronic equipment, and comprises the following steps:

when the identification result represents that the target obstacle exists in the image to be identified, determining a reference position coordinate of a position point corresponding to the target measurement value under a preset reference coordinate system, and determining a reference area coordinate of the target obstacle under the reference coordinate system;

When the reference position coordinates are located in the range corresponding to the reference region coordinates, the target measured value is determined to be the distance between the target obstacle and the vehicle, the object size information determined based on the reference region coordinates is determined to be the size information of the target obstacle, and the detection result of the target obstacle is obtained.

It should be noted that, other implementation manners of an obstacle detection method implemented by the processor 1001 executing the program stored in the memory 1003 are the same as those of the embodiment of the method provided in the foregoing method embodiment, and are not repeated here.

In this way, the obtained detection result of the target obstacle is input into the decision module of the vehicle-mounted advanced driving assistance system, and the decision module can directly determine the final control strategy based on the fusion result of the detection results of the vehicle-mounted image acquisition device and the vehicle-mounted ultrasonic sensor. The detection results of the vehicle-mounted image acquisition equipment and the vehicle-mounted ultrasonic sensor can mutually compensate the defects of the detection results of the other side, so that the decision difficulty of the decision module in determining the final control strategy can be reduced, and the better final control strategy is obtained. The communication bus mentioned for the above-mentioned in-vehicle electronic device may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, or the like. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.

The communication interface is used for communication between the vehicle-mounted electronic equipment and other equipment.

The Memory may include random access Memory (Random Access Memory, RAM) or may include Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.

The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processing, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

Corresponding to the obstacle detection method provided by the embodiment of the invention, the embodiment of the invention also provides a computer readable storage medium, and the computer program is executed by the processor to realize the obstacle detection method provided by the embodiment of the invention.

Specifically, the obstacle detection method is applied to a vehicle-mounted advanced driving assistance system, and comprises the following steps:

It should be noted that, other implementation manners of an obstacle detection method implemented when the computer program is executed by the processor are the same as those provided in the foregoing method embodiment, and are not repeated here.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the apparatus embodiment, the in-vehicle electronic device embodiment, and the computer-readable storage medium embodiment, since they are substantially similar to the method embodiment, the description is relatively simple, and the relevant points are referred to in the partial description of the method embodiment.

The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. An obstacle detection method, for application to an advanced driving assistance system on board a vehicle, the method comprising:

when the reference position coordinates are located in the range of the area corresponding to the reference area coordinates, determining the target measured value as the distance between the target obstacle and the vehicle, determining the object size information determined based on the reference area coordinates as the size information of the target obstacle, and obtaining a detection result of the target obstacle;

The step of identifying the obstacle to the image to be identified to obtain an identification result comprises the following steps:

if yes, taking the initial result as a recognition result;

otherwise, when judging that no obstacle exists in at least one of the continuous multiple identified images to be identified, which is before the current image to be identified and is continuous with the image to be identified, and no obstacle exists in the other at least one identified image to be identified, taking the initial result as an identification result; and when judging that the at least one of the continuous multiple identified images which is before the current image to be identified and is continuous with the image to be identified has an obstacle, and the other at least one of the continuous multiple identified images to be identified has no obstacle, tracking the obstacle in the continuous multiple identified images by using an image tracking algorithm, determining a tracking result of the image to be identified, and taking the determined tracking result as the identification result of the current image to be identified.

2. The method according to claim 1, wherein the step of acquiring the image to be recognized acquired by the in-vehicle image acquisition device includes:

3. The method of claim 1, wherein the step of obtaining the target measurement value acquired by the in-vehicle ultrasonic sensor comprises:

4. A method according to any one of claims 1 to 3, wherein the step of determining the reference position coordinates of the position point corresponding to the target measurement value in a preset reference coordinate system when the recognition result indicates that a target obstacle exists in the image to be recognized, and determining the reference region coordinates of the target obstacle in the reference coordinate system comprises:

when the identification result represents that a target obstacle exists in the image to be identified and the target measured value is not larger than a preset measurement threshold, determining a reference position coordinate of a position point corresponding to the target measured value under a preset reference coordinate system, and determining a reference area coordinate of the target obstacle under the reference coordinate system; wherein the preset measurement threshold is not greater than the detection range of the vehicle-mounted ultrasonic sensor.

5. The method of claim 1, wherein the step of determining the reference position coordinates of the position point corresponding to the target measurement value in a preset reference coordinate system, and determining the reference area coordinates of the target obstacle in the reference coordinate system, comprises:

6. The method of claim 1, wherein after the step of obtaining the target measurement value acquired by the in-vehicle ultrasonic sensor, the method further comprises:

7. The method according to claim 1, wherein the method further comprises:

8. An obstacle detection device for use in an advanced driving assistance system on board a vehicle, the device comprising:

the detection result determining module is used for determining the target measured value as the distance between the target obstacle and the vehicle when the reference position coordinate is located in the range corresponding to the reference region coordinate, determining the object size information determined based on the reference region coordinate as the size information of the target obstacle, and obtaining the detection result of the target obstacle;

the image data acquisition module is specifically configured to:

performing obstacle recognition on the image to be recognized to obtain an initial result, and judging whether the initial result represents that an obstacle exists in the image to be recognized; if yes, taking the initial result as a recognition result; if not, judging whether obstacles exist in a plurality of continuous images to be recognized before the images to be recognized; when the fact that a plurality of continuous images to be identified exist in the images to be identified before the images to be identified is judged, determining a tracking result of the images to be identified as an identification result by utilizing an image tracking algorithm based on the initial result and the identification result of the images to be identified; otherwise, when judging that no obstacle exists in at least one of the continuous multiple identified images to be identified, which is before the current image to be identified and is continuous with the image to be identified, and no obstacle exists in the other at least one identified image to be identified, taking the initial result as an identification result; and when judging that the at least one of the continuous multiple identified images which is before the current image to be identified and is continuous with the image to be identified has an obstacle, and the other at least one of the continuous multiple identified images to be identified has no obstacle, tracking the obstacle in the continuous multiple identified images by using an image tracking algorithm, determining a tracking result of the image to be identified, and taking the determined tracking result as the identification result of the current image to be identified.

9. The apparatus of claim 8, wherein the image data acquisition module is specifically configured to:

10. The apparatus of claim 8, wherein the ultrasound data acquisition module is specifically configured to:

11. The apparatus according to any one of claims 8-10, wherein the coordinate information determination module is specifically configured to:

12. The apparatus of claim 8, wherein the coordinate information determination module is specifically configured to:

13. The vehicle-mounted electronic equipment is characterized in that an advanced driving assistance system is arranged on the electronic equipment and comprises a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface and the memory are communicated with each other through the communication bus;

a memory for storing a computer program;

a processor for carrying out the method steps of any one of claims 1-7 when executing a program stored on a memory.

14. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored therein a computer program which, when executed by a processor, implements the method steps of any of claims 1-7.