CN112084810A

CN112084810A - Obstacle detection method and device, electronic equipment and storage medium

Info

Publication number: CN112084810A
Application number: CN201910505369.0A
Authority: CN
Inventors: 安建平
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2019-06-12
Filing date: 2019-06-12
Publication date: 2020-12-15
Anticipated expiration: 2039-06-12
Also published as: CN112084810B

Abstract

The embodiment of the invention provides an obstacle detection method and device, electronic equipment and a storage medium. The method is applied to a vehicle-mounted advanced driving assistance system and comprises the following steps: when the recognition result of the image to be recognized, which is acquired by the vehicle-mounted image acquisition equipment, represents that a target obstacle exists in the image to be recognized, determining the reference position coordinate of a position point corresponding to a target measurement value under a preset reference coordinate system, and determining the reference area coordinate of the target obstacle under the reference coordinate system; and when the reference position coordinate is located in the area range corresponding to the reference area coordinate, determining the target measurement value as the distance between the target obstacle and the vehicle, and determining the object size information determined based on the reference area coordinate as the size information of the target obstacle to obtain the detection result of the target obstacle. Compared with the prior art, the scheme provided by the embodiment of the invention can reduce the decision difficulty of the decision module and obtain a better final control strategy.

Description

Obstacle detection method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of driving assistance technologies, and in particular, to a method and an apparatus for detecting an obstacle, an electronic device, and a storage medium.

Background

In recent years, with the rapid development of automatic driving technology, the environmental perception problem in the vehicle running process becomes more important, wherein the obstacle detection problem is the most important part of the environmental perception problem.

In the related art in the field of ADAS (Advanced Driver Assistance Systems), obstacle detection is performed using multiple sensors.

Specifically, a plurality of sensors are used for respectively acquiring data, and after the data acquired by each sensor is acquired, obstacle analysis is respectively performed on the data acquired by each sensor, so that an obstacle detection result corresponding to each sensor is determined. Furthermore, the obtained multiple obstacle detection results can be simultaneously input into the decision module of the ADAS, so that the decision module can determine the vehicle control strategy corresponding to each obstacle detection result, and fuse the determined multiple vehicle control strategies to obtain a final control strategy, so that the vehicle can be controlled to operate by using the final control strategy.

Obviously, since the obstacle detection result corresponding to each sensor may have some defects, the decision difficulty of the decision module in determining the final control strategy may be increased, which may make it difficult to obtain a better final control strategy.

Disclosure of Invention

The embodiment of the invention aims to provide an obstacle detection method, an obstacle detection device, electronic equipment and a storage medium, so as to reduce the decision difficulty of a decision module and obtain a better final control strategy. The specific technical scheme is as follows:

in a first aspect, an embodiment of the present invention provides an obstacle detection method applied to a vehicle-mounted advanced driving assistance system, where the method includes:

acquiring an image to be identified acquired by vehicle-mounted image acquisition equipment, and identifying an obstacle on the image to be identified to obtain an identification result;

acquiring a target measured value acquired by a vehicle-mounted ultrasonic sensor;

when the identification result represents that a target obstacle exists in the image to be identified, determining a reference position coordinate of a position point corresponding to the target measurement value under a preset reference coordinate system, and determining a reference area coordinate of the target obstacle under the reference coordinate system;

when the reference position coordinate is located in the area range corresponding to the reference area coordinate, the target measurement value is determined as the distance between the target obstacle and the vehicle, the size information of the object determined based on the reference area coordinate is determined as the size information of the target obstacle, and the detection result of the target obstacle is obtained.

As an example, the step of acquiring the image to be identified collected by the vehicle-mounted image collection device includes:

and selecting an image with the minimum difference between the acquisition time and the current time from the images acquired by the vehicle-mounted image acquisition equipment as an image to be identified.

As an example, the step of acquiring the target measurement value acquired by the vehicle-mounted ultrasonic sensor includes:

and selecting the ultrasonic measured value with the minimum difference between the acquisition time and the current time from the ultrasonic measured values acquired by the vehicle-mounted ultrasonic sensor as a target measured value.

As an example, when the recognition result indicates that a target obstacle exists in the image to be recognized, the step of determining the reference position coordinates of the position point corresponding to the target measurement value in a preset reference coordinate system, and determining the reference area coordinates of the target obstacle in the reference coordinate system includes:

when the identification result represents that a target obstacle exists in the image to be identified and the target measurement value is not larger than the preset measurement threshold value, determining a reference position coordinate of a position point corresponding to the target measurement value under a preset reference coordinate system and determining a reference area coordinate of the target obstacle under the reference coordinate system; and the preset measurement threshold is not larger than the detection range of the vehicle-mounted ultrasonic sensor.

As an example, the step of determining the reference position coordinates of the position point corresponding to the target measurement value in a preset reference coordinate system, and determining the reference area coordinates of the target obstacle in the reference coordinate system, includes:

determining the position point coordinates of the position point corresponding to the target measurement value in a preset vehicle coordinate system based on the installation position of the vehicle-mounted ultrasonic sensor;

calculating a reference position coordinate corresponding to the position point coordinate in a preset reference coordinate system by using a preset coordinate conversion formula, wherein the reference coordinate system is an image coordinate system of the image to be recognized, and the coordinate conversion formula is a coordinate conversion formula from the vehicle coordinate system to the reference coordinate system;

and determining an image area of the target obstacle in the image to be recognized, and determining an image area coordinate of the image area in the image coordinate system as a reference area coordinate of the target obstacle in the reference coordinate system.

As an example, the step of performing obstacle recognition on the image to be recognized to obtain a recognition result includes:

carrying out obstacle identification on the image to be identified to obtain an initial result, and judging whether the initial result represents that an obstacle exists in the image to be identified;

if so, taking the initial result as a recognition result;

if not, judging whether obstacles exist in a plurality of continuous recognized images to be recognized before the image to be recognized;

when obstacles exist in a plurality of continuous recognized images to be recognized before the image to be recognized, determining a tracking result of the image to be recognized as a recognition result by using an image tracking algorithm based on the initial result and the recognition result of the recognized image to be recognized;

otherwise, the initial result is used as a recognition result.

As an example, after the step of acquiring the target measurement value acquired by the vehicle-mounted ultrasonic sensor, the method further comprises:

correcting the target measured value to obtain a corrected target measured value;

the step of determining the target measurement as the distance of the target obstacle from the vehicle comprises:

determining the corrected target measurement value as a distance of the target obstacle from a vehicle.

As an example, the method further comprises:

when the reference position information is not located in the reference area coordinates included in the identification result, determining the reference position coordinates as position information of a first obstacle detected by the vehicle-mounted ultrasonic sensor, and determining the target measurement value as the distance between the first obstacle and a vehicle to obtain a detection result of the first obstacle; determining object size information determined based on the reference area coordinates as size information of the target obstacle, and determining a monocular distance as the distance between the target obstacle and the vehicle to obtain a detection result of the target obstacle; and detecting the monocular distance measuring distance based on the image to be identified by utilizing a monocular distance measuring algorithm.

In a second aspect, an embodiment of the present invention provides an obstacle detection apparatus applied to a vehicle-mounted advanced driving assistance system, where the apparatus includes:

the image data acquisition module is used for acquiring an image to be identified, which is acquired by the vehicle-mounted image acquisition equipment, and identifying an obstacle of the image to be identified to obtain an identification result;

the ultrasonic data acquisition module is used for acquiring a target measured value acquired by the vehicle-mounted ultrasonic sensor;

the coordinate information determining module is used for determining a reference position coordinate of a position point corresponding to the target measurement value under a preset reference coordinate system and determining a reference area coordinate of the target obstacle under the reference coordinate system when the identification result represents that the target obstacle exists in the image to be identified;

and the detection result determining module is used for determining the target measurement value as the distance between the target obstacle and the vehicle when the reference position coordinate is located in the area range corresponding to the reference area coordinate, and determining the object size information determined based on the reference area coordinate as the size information of the target obstacle to obtain the detection result of the target obstacle.

As an example, the image data obtaining module is specifically configured to:

As an example, the ultrasound data acquisition module is specifically configured to:

As an example, the coordinate information determination module is specifically configured to:

determining the position point coordinates of the position point corresponding to the target measurement value in a preset vehicle coordinate system based on the installation position of the vehicle-mounted ultrasonic sensor; calculating a reference position coordinate corresponding to the position point coordinate in a preset reference coordinate system by using a preset coordinate conversion formula, wherein the reference coordinate system is an image coordinate system of the image to be recognized, and the coordinate conversion formula is a coordinate conversion formula from the vehicle coordinate system to the reference coordinate system; and determining an image area of the target obstacle in the image to be recognized, determining an image area coordinate of the image area in the image coordinate system, and determining the image area coordinate as a reference area coordinate of the target obstacle in the reference coordinate system.

As an example, the image data obtaining module is specifically configured to:

carrying out obstacle identification on the image to be identified to obtain an initial result, and judging whether the initial result represents that an obstacle exists in the image to be identified; if so, taking the initial result as a recognition result; if not, judging whether obstacles exist in a plurality of continuous recognized images to be recognized before the image to be recognized; when obstacles exist in a plurality of continuous recognized images to be recognized before the image to be recognized, determining a tracking result of the image to be recognized as a recognition result by using an image tracking algorithm based on the initial result and the recognition result of the recognized image to be recognized; otherwise, the initial result is used as a recognition result.

As an example, the apparatus further comprises:

the obstacle information determining module is used for determining the reference position coordinate as the position information of a first obstacle detected by the vehicle-mounted ultrasonic sensor when the reference position information is not located in the reference area coordinate included in the identification result, and determining the target measurement value as the distance between the first obstacle and the vehicle to obtain the detection result of the first obstacle; determining object size information determined based on the reference area coordinates as size information of the target obstacle, and determining a monocular distance as the distance between the target obstacle and the vehicle to obtain a detection result of the target obstacle; and detecting the monocular distance measuring distance based on the image to be identified by utilizing a monocular distance measuring algorithm.

In a third aspect, an embodiment of the present invention provides a vehicle-mounted electronic device, where the electronic device is equipped with an advanced driving assistance system, and the advanced driving assistance system includes a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete mutual communication through the communication bus;

a memory for storing a computer program;

a processor, configured to implement the method steps of any one of the obstacle detection methods provided by the first aspect when executing a program stored in the memory.

In a fourth aspect, the present invention provides a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements the method steps in any one of the obstacle detection methods provided in the first aspect.

Therefore, by applying the scheme provided by the embodiment of the invention, after the image to be recognized and the target measurement value are obtained, the vehicle-mounted advanced driving assistance system can determine whether the vehicle-mounted image acquisition equipment detects the obstacle or not by analyzing the image to be recognized. Furthermore, when the vehicle-mounted image acquisition device detects an obstacle, the vehicle-mounted advanced driving assistance system can fuse the detection results of the vehicle-mounted image acquisition device and the vehicle-mounted ultrasonic sensor based on the image to be recognized and the target measurement value to obtain the detection result of the target obstacle.

Therefore, the obtained detection result of the target obstacle is input into a decision module of the vehicle-mounted advanced driving assistance system, and the decision module can directly determine a final control strategy based on the fusion result of the detection results of the vehicle-mounted image acquisition equipment and the vehicle-mounted ultrasonic sensor. The detection results of the vehicle-mounted image acquisition equipment and the vehicle-mounted ultrasonic sensor can mutually make up for the defects in the detection results of the other side, so that the decision difficulty of the decision module for determining the final control strategy can be reduced, and the optimal final control strategy can be obtained.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic flow chart of a method for detecting an obstacle according to an embodiment of the present invention;

FIG. 2 is a schematic flow chart of a specific implementation of recognizing an obstacle in an image to be recognized to obtain a recognition result;

FIG. 3 is a schematic illustration of ultrasonic measurements taken by an on-board ultrasonic sensor over a period of time;

FIG. 4 is a schematic illustration of a corrected ultrasonic measurement obtained by correcting the ultrasonic measurement of FIG. 3;

FIG. 5 is a schematic diagram of a network structure of a layer of LSTM network;

FIG. 6 is a schematic diagram of a network structure of a three-layer cascaded LSTM network;

FIG. 7 is a schematic flow chart illustrating an embodiment of determining a reference position coordinate of a position point corresponding to a target measurement value in a preset reference coordinate system and determining a reference area coordinate of a target obstacle in the reference coordinate system;

fig. 8 is a schematic flow chart of another obstacle detection method according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of an obstacle detection device according to an embodiment of the present invention;

fig. 10 is a schematic structural diagram of a vehicle-mounted electronic device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In the related art, a vehicle-mounted advanced driving assistance system performs obstacle detection by using multiple sensors, specifically, data acquisition is performed by using multiple sensors, and after data acquired by each sensor is obtained, obstacle analysis is performed on the data acquired by each sensor, so that an obstacle detection result corresponding to each sensor is determined. Furthermore, the obtained multiple obstacle detection results can be simultaneously input into a decision module of the vehicle-mounted advanced driving assistance system, so that the decision module can determine a vehicle control strategy corresponding to each obstacle detection result, and fuse the determined multiple vehicle control strategies to obtain a final control strategy, so that the vehicle can be controlled to operate by using the final control strategy. However, since the obstacle detection result corresponding to each sensor may have some defects, the decision difficulty of the decision module for determining the final control strategy may be increased, and it may be difficult to obtain a better final control strategy

In order to solve the above technical problem, an embodiment of the present invention provides an obstacle detection method. The method is applied to a vehicle-mounted advanced driving assistance system.

The Advanced Driver Assistance System (ADAS) utilizes various sensors mounted on a vehicle to sense the environment around the vehicle at any time in the driving process of the vehicle, collects relevant data of the vehicle and the environment where the vehicle is located, identifies, detects and tracks static and dynamic objects by utilizing the collected data, and performs operation and analysis by combining with navigator map data, thereby leading a Driver to perceive possible dangers in advance and effectively increasing the comfort and safety of vehicle driving.

In addition, at least one image acquisition device and at least one ultrasonic sensor are installed on the vehicle, so that image acquisition and obstacle ranging of the environment around the vehicle can be realized. The distance between various general obstacles and the vehicle can be determined by utilizing the image acquired by the image acquisition equipment to determine whether the obstacles such as motor vehicles, non-motor vehicles, pedestrians and the like exist around the vehicle and utilizing the ultrasonic distance measurement principle of the ultrasonic sensor. Wherein, the general obstacles are: obstacles within a certain distance from the vehicle body are all called general obstacles, such as walls, columns, roadblocks, curbs and the like.

The vehicle-mounted image acquisition equipment acquires images according to preset frequency, namely, one image is acquired at preset time intervals, and the vehicle-mounted ultrasonic sensor also performs distance measurement according to the preset frequency, namely, an ultrasonic measurement value is acquired at preset time intervals. The acquisition frequency of the vehicle-mounted image acquisition equipment and the acquisition frequency of the vehicle-mounted ultrasonic sensor can be the same or different.

In addition, the type and the number of the image acquisition devices and the ultrasonic sensors installed on the vehicle can be determined according to the performance of an advanced driving assistance system installed on the vehicle in practical application, the road condition of a road on which the vehicle frequently runs, the requirement on the running safety of the vehicle and the like. The embodiment of the present invention is not particularly limited.

For example, the image capturing devices may be mounted as fisheye cameras in 4 numbers, respectively mounted in four directions of the front, rear, left, and right of the vehicle body. Thus, the shooting range of the image acquisition device can cover all areas around the vehicle. The ultrasonic sensors can be long-distance ultrasonic sensors and short-distance ultrasonic sensors, wherein the number of the long-distance ultrasonic sensors is 4, 2 ultrasonic sensors are arranged on each side of the side edge of the vehicle body, the effective detection distance is 5m, and the detection range of the long-distance ultrasonic sensors is 5 m; the number of the short-distance ultrasonic sensors is 8, 4 short-distance ultrasonic sensors are respectively arranged at the vehicle head and the vehicle tail, the effective detection distance is 2.5m, namely the detection range of the short-distance ultrasonic sensors is 2.5 m.

Fig. 1 is a schematic flow chart of an obstacle detection method according to an embodiment of the present invention. As shown in fig. 1, the obstacle detection method may include the steps of:

s101: acquiring an image to be identified acquired by vehicle-mounted image acquisition equipment, and identifying an obstacle on the image to be identified to obtain an identification result;

s102: acquiring a target measured value acquired by a vehicle-mounted ultrasonic sensor;

s103: when the recognition result represents that the target obstacle exists in the image to be recognized, determining a reference position coordinate of a position point corresponding to the target measurement value under a preset reference coordinate system, and determining a reference area coordinate of the target obstacle under the reference coordinate system;

s104: and when the reference position coordinate is located in the area range corresponding to the reference area coordinate, determining the target measurement value as the distance between the target obstacle and the vehicle, and determining the object size information determined based on the reference area coordinate as the size information of the target obstacle to obtain the detection result of the target obstacle.

It is understood that, according to the above description of the image capturing apparatus and the ultrasonic sensor mounted on the vehicle, the vehicle-mounted image capturing apparatus and the vehicle-mounted ultrasonic sensor each perform data capturing according to their respective frequencies.

For the vehicle-mounted image acquisition device, each time an image about the surrounding environment of the vehicle is acquired, the vehicle image acquisition device can also acquire and record the time for acquiring the image, namely the acquisition time of the image. Correspondingly, for the vehicle-mounted ultrasonic sensor, when an ultrasonic measurement value is acquired each time, the vehicle-mounted ultrasonic sensor can also acquire and record the time for acquiring the ultrasonic measurement value, namely the acquisition time for acquiring and recording the ultrasonic measurement value.

Further, based on this, in the above steps S101 and S102, the advanced driving assistance system mounted on the vehicle may also acquire the image to be recognized and the target measurement value at a preset frequency.

It can be understood that the frequency corresponding to the vehicle-mounted advanced driving assistance system, the acquisition frequency of the vehicle-mounted image acquisition device and the acquisition frequency of the vehicle-mounted ultrasonic sensor may be the same or different. Obviously, when the three are the same, the image to be recognized acquired by the vehicle-mounted advanced driving assistance system is the image acquired by the vehicle-mounted image acquisition device at the current moment, and the target measurement value acquired by the vehicle-mounted advanced driving assistance system is the ultrasonic measurement value acquired by the vehicle-mounted ultrasonic sensor at the current moment.

Here, the current time is the time when the vehicle-mounted advanced driving assistance system performs the current obstacle detection, and at the current time, the vehicle-mounted advanced driving assistance system acquires the image to be recognized in step S101 and the target measurement value in step S102.

As an example, in order to ensure that the detected result of the target obstacle can be matched with an obstacle that actually exists in the current environment of the vehicle, the vehicle-mounted advanced driving assistance system performs step S101 to obtain the image to be recognized, which is acquired by the vehicle-mounted image acquisition device, and the method may include:

That is, in this example, the image to be identified acquired by the vehicle-mounted advanced driving assistance system is an image whose acquisition time is closest to the current time among the multiple frames of images acquired by the vehicle-mounted image acquisition device.

In addition, it can be understood that, when an image with a capture time closest to the current time in the multi-frame images captured by the vehicle-mounted image capture device cannot be used as an image to be recognized due to image blurring and the like, the vehicle-mounted advanced driving assistance system may further obtain an image with a second smallest difference between the capture time and the current time in the captured multi-frame images as the image to be recognized.

Therefore, after the image to be recognized is obtained, the vehicle-mounted advanced driving assistance system can recognize the obstacle of the image to be recognized, and a recognition result is obtained.

The vehicle-mounted advanced driving assistance system can identify obstacles of the image to be identified in various modes to obtain an identification result. The embodiment of the present invention is not particularly limited.

As an example, the advanced driving assistance system mounted on a vehicle may perform image detection on an image to be recognized using various obstacle detection algorithms, and obtain an image detection result as a recognition result.

For example, the obstacle detection algorithm may be: a machine learning method based on HOG (Histogram of Oriented Gradient) features and SVM (support vector machine) network. Of course, other obstacle detection algorithms may also be adopted to perform image detection on the image to be recognized to obtain an image detection result, and thus, the embodiment of the present invention is not particularly limited.

As an example, the advanced driving assistance system mounted on a vehicle may input an image to be recognized into a preset obstacle detection model, obtain a detection result output by the obstacle detection model, and use the detection result as a recognition result.

For example, the preset obstacle detection model may be: deep learning models based on CNN (Convolutional Neural Network). Of course, other preset obstacle detection models may also be adopted to obtain the recognition result, and thus, the embodiment of the present invention is not particularly limited.

The obstacle detection model is obtained by training a preset initial model based on a plurality of sample images and the label of each sample image. Each sample image is used for marking an image area where the obstacle exists in the sample image, and the sample label of the sample image is the type of the obstacle existing in the sample image. And then, inputting the plurality of sample images and the label of each sample image into a preset initial model for training until a convergence condition is met, and obtaining a trained obstacle detection model.

Obviously, when the detection result of the obstacle detection model indicates that the target obstacle exists in the image to be recognized, the detection result may include an image area where the target obstacle exists in the image to be recognized and the type of the target obstacle.

The obstacle detection model may be obtained by local training in the vehicle-mounted advanced driving assistance system, or may be obtained by the vehicle-mounted advanced driving assistance system from other electronic devices connected in communication.

In some cases, an obstacle existing around the vehicle may be blocked by other things, or, due to the limitation of the angle and the shooting range of the image capturing device, although an obstacle exists around the vehicle, an image of the obstacle is not captured in the image to be recognized acquired in the advanced driving assistance system of the vehicle, so that the recognition result obtained by detecting the obstacle in the image to be recognized represents that no obstacle exists in the image to be recognized. Obviously, in such a case, the decision module of the vehicle-mounted advanced driving assistance system may be caused to make an inappropriate control strategy, increasing the risk of vehicle operation.

Therefore, as an example of this case, as shown in fig. 2, the way in which the vehicle-mounted advanced driving assistance system performs obstacle recognition on the image to be recognized in step S101 to obtain a recognition result may include the following steps:

s201: carrying out obstacle identification on the image to be identified to obtain an initial result, and judging whether the initial result represents that an obstacle exists in the image to be identified; if yes, executing step S202, otherwise, executing step S203;

s202: taking the initial result as a recognition result;

s203: judging whether obstacles exist in a plurality of continuous recognized images to be recognized before the images to be recognized; if yes, executing step S204, otherwise, executing step S205;

s204: determining a tracking result of the image to be recognized as a recognition result based on the initial result and the recognition result of the recognized image to be recognized by utilizing an image tracking algorithm;

s205: and taking the initial result as a recognition result.

In this example, the vehicle-mounted advanced driving assistance system first performs obstacle recognition on an image to be recognized by using an obstacle detection algorithm or a preset obstacle detection model, and the like, to obtain an initial result. Thus, the vehicle-mounted advanced driving assistance system can judge whether the initial result represents that the obstacle exists in the image to be recognized.

Obviously, when the above determination result is yes, it is described that the image of the obstacle around the vehicle is captured in the image to be recognized, that is, the vehicle-mounted advanced driving assistance system detects the obstacle around the vehicle. Therefore, the advanced driving assistance system mounted on the vehicle can use the initial result as the recognition result.

Correspondingly, when the judgment result is negative, the image of the obstacle around the vehicle is not acquired in the image to be recognized, that is, the vehicle-mounted advanced driving assistance system does not detect the obstacle around the vehicle. Then, there are two possibilities, the first is that there is really no obstacle around the vehicle, the second is that there is an obstacle around the vehicle, and the image capturing apparatus does not capture an image of the obstacle because the obstacle is blocked.

Therefore, in order to ensure that in the second case, the vehicle-mounted advanced driving assistance system can still detect the obstacles existing around the vehicle, the decision module of the vehicle-mounted advanced driving assistance system is prevented from making an inappropriate control strategy, and the running risk of the vehicle is increased. The vehicle-mounted advanced driving assistance system can acquire a plurality of continuous recognized images to be recognized before the images to be recognized, and further judge whether obstacles exist in the plurality of continuous recognized images to be recognized.

Obviously, if there is an obstacle in the above-mentioned consecutive plurality of recognized images to be recognized, the current image to be recognized may be the one that conforms to the above-mentioned second case, that is, there is an obstacle around the vehicle, and the obstacle is blocked. Therefore, the vehicle-mounted advanced driving assistance system can continuously acquire the identification result of each identified image to be identified in the plurality of the identified images to be identified. Furthermore, the vehicle-mounted advanced driving assistance system may track obstacles in the consecutive plurality of images to be recognized by using an image tracking algorithm based on the initial result and the recognition result of each of the acquired images to be recognized, determine a tracking result of the images to be recognized, and use the determined tracking result as the recognition result of the current image to be recognized.

Correspondingly, if no obstacle exists in the plurality of consecutive recognized images to be recognized, the current image to be recognized may be the one that meets the first condition, that is, no obstacle exists around the vehicle. Thus, the vehicle-mounted advanced driving assistance system can use the initial result as the recognition result.

Further, if, among the plurality of consecutive images to be recognized, at least one recognized image that is before the current image to be recognized and that is consecutive to the image to be recognized does not have an obstacle, and at least one other recognized image has an obstacle, the obstacle may be present in the vehicle and may have disappeared. Therefore, the advanced driving assistance system mounted on the vehicle can take the above initial result as the recognition result.

In addition, if an obstacle exists in at least one of the plurality of consecutive images to be recognized that are before the current image to be recognized and are consecutive to the image to be recognized, and an obstacle does not exist in at least one other image to be recognized, the obstacle may be present in the vehicle in the case where no obstacle exists before and an obstacle appears later. Therefore, the vehicle-mounted advanced driving assistance system can track the obstacles in the continuous multiple recognized images to be recognized by using an image tracking algorithm based on the initial result and the recognition result of each recognized image to be recognized, determine the tracking result of the images to be recognized, and use the determined tracking result as the recognition result of the current images to be recognized.

Specifically, based on the obtained recognition result of each recognized image to be recognized, an image tracking algorithm may be used to predict an image area where a target obstacle exists in the image to be recognized in combination with optical flow information, so as to obtain a tracking result.

After the step S101 is completed, after the image to be recognized is obtained and the recognition result is obtained, the vehicle-mounted advanced driving assistance system may continue to perform the step S102 to obtain the target measurement value acquired by the vehicle-mounted ultrasonic sensor.

In order to further ensure that the detected result of the detected target obstacle can be matched with an obstacle actually existing in the current environment of the vehicle, as an example, the method for the vehicle-mounted advanced driving assistance system to execute the step S102 to obtain the target measurement value acquired by the vehicle-mounted ultrasonic sensor may include:

That is, in the present example, the target measurement value acquired by the advanced driving assistance system mounted on the vehicle is the ultrasonic measurement value whose acquisition time is closest to the current time among the plurality of ultrasonic measurement values acquired by the ultrasonic sensor mounted on the vehicle.

In addition, it can be understood that, when, among a plurality of ultrasonic measurement values acquired by the vehicle-mounted ultrasonic sensor, an ultrasonic measurement value whose acquisition time is closest to the current time cannot be used as a target measurement value due to excessive noise or the like, the vehicle-mounted advanced driving assistance system may further acquire, among the plurality of acquired ultrasonic measurement values, an ultrasonic measurement value whose acquisition time is the next smallest from the current time as the target measurement value.

Further, in practical applications, noise data may exist in the ultrasonic measurement values acquired by the vehicle-mounted ultrasonic sensor, as shown in fig. 3, which is a schematic diagram of the ultrasonic measurement values acquired by the vehicle-mounted ultrasonic sensor over a period of time, and it can be seen that some maximum or minimum noise data randomly appears in the ultrasonic measurement values.

It can be understood that the noise data occurring in the ultrasonic measurement values may affect the accuracy of the ultrasonic measurement values acquired by the vehicle-mounted ultrasonic sensor, and further, the detection accuracy of the vehicle-mounted advanced driving assistance system on the obstacle.

Based on the above, in order to reduce the influence of noise data occurring in the ultrasonic measurement value on the accuracy of the ultrasonic measurement value acquired by the vehicle-mounted ultrasonic sensor, the detection accuracy of the vehicle-mounted advanced driving assistance system for the obstacle is ensured. As an example, after the step S102 is executed, the advanced driving assistance system on board may execute the following steps:

in this way, in the present example, in step S104, the step of determining the target measurement value as the distance between the target obstacle and the vehicle is: and determining the corrected target measurement value as the distance between the target obstacle and the vehicle.

The vehicle-mounted advanced driving assistance system may correct the target measurement value in various ways, which is not limited in the embodiments of the present invention.

As shown in fig. 3, from an overall perspective, the change of the ultrasonic measurement value in the ultrasonic measurement value acquired by the vehicle-mounted ultrasonic sensor in a period of time may have certain continuity and regularity. Therefore, the ultrasonic measurement values can be corrected by constructing a correction model using an LSTM (Long Short-Term Memory network) network.

Specifically, in an embodiment, the manner in which the vehicle-mounted advanced driving assistance system performs the above-mentioned correction on the target measurement value to obtain the corrected target measurement value may include the following steps:

inputting the target measured value and n-1 utilized target measured values before the target measured value into a preset correction model, and determining the nth correction result of n correction results output by the correction model as a corrected target measured value;

wherein, the correction model is as follows: and taking the ultrasonic measurement values of the 1 st to nth preset samples as input, taking the ultrasonic measurement values of the 2 nd to nth +1 th preset samples as the ultrasonic measurement values of the 1 st to nth preset samples respectively, and training a preset long-short term memory (LSTM) network to obtain the ultrasonic measurement value n > 1.

In this example, after the target measured value is obtained, the advanced driving assistance system of the vehicle may input the target measured value and n-1 consecutive target measured values that are used before the target measured value into a preset correction model, that is, n target measured values are input into the correction model, where the current target measured value is the nth target measured value. Thus, n correction results output by the correction model can be obtained. Wherein, the first n-1 correction results in the obtained correction result are respectively the correction results corresponding to the consecutive n-1 utilized target measurement values before the input target measurement value, so the nth correction result in the n correction results can be determined as the corrected target measurement value.

In this example, the correction model may be obtained by local training in the vehicle-mounted advanced driving assistance system, or may be obtained by the vehicle-mounted advanced driving assistance system from another electronic device connected in communication. This is all reasonable.

Next, a specific training process of the above-described correction model will be described.

Specifically, the calibration model is obtained by training a preset long-short term memory LSTM network, with the 1 st to nth preset sample ultrasonic measurement values as input, and the 2 nd to n +1 th preset sample ultrasonic measurement values as output of the 1 st to nth preset sample ultrasonic measurement values, respectively.

The preset training sample may be n +1 consecutive target measurement values used by the vehicle-mounted advanced driving assistance system before the current target measurement value, or n +1 sample ultrasonic measurement values obtained by using other electronic devices or ultrasonic sensors. The embodiment of the present invention is not particularly limited.

In addition, the preset LSTM network may have only one layer of LSTM network structure, as shown in fig. 5, which is a schematic network structure diagram of one layer of LSTM network; or, the LSTM network structure may be a multi-layer LSTM network structure, that is, a cascaded LSTM network, as shown in fig. 6, which is a schematic network structure diagram of a three-layer cascaded LSTM network.

In embodiment 1, when the preset LSTM network has only one layer of LSTM network structure, taking fig. 5 as an example, the method for training the preset LSTM network to obtain the calibration model is specifically as follows:

x_i(i∈[1,n]) And y_i(i∈[1,n]) Input and output data, in particular, x, of the predetermined LSTM network, respectively_iFor the ultrasonic measurement of the ith sample, y_iThen the predicted value of the ultrasonic measurement value of the ith sample is used as y_iCorresponding target truth value is x_i+1I.e., the i +1 th sample ultrasonic measurement. Furthermore, h_iIs the network hidden state of the preset LSTM network.

In the actual training process, a scheme of predicting the ultrasonic measurement value of the next sample by using the ultrasonic measurement value of the previous sample is adopted, so that the preset LSTM network is a self-supervision network, and no additional marking is required on the ultrasonic measurement value of the sample.

In embodiment 2, when the preset LSTM network has a multi-layer LSTM network structure, taking fig. 6 as an example, a method for training the preset LSTM network to obtain a calibration model is specifically as follows:

presetting sample ultrasonic measurement value x for first layer LSTM network structure₁-x_nInputting into the first layer LSTM network structure to obtain x₁-x_nRespectively corresponding predicted values y₁-y_nAnd determining a predicted value y₁-y_nThe corresponding true value. Wherein y is predicted value₁-y_nThe corresponding true value is the preset sample ultrasonic measurement value x₂-x_n+1. Further, a loss value between each predicted value and the corresponding true value is calculated. Thus, the training of the first layer LSTM network structure of the preset LSTM network can be completed.

Aiming at the LSTM network structure of the second layer, the predicted value y of the first layer is determined₁-y_n-1Inputting the data into the second layer LSTM network structure to obtain y₁-y_n-1Respectively corresponding predicted values z₁-z_n-1Wherein z is predicted₁-z_n-1The corresponding true value is y in the output predicted value of the first layer LSTM network structure₂-y_n. And calculating the loss value between each pre-stored value and the corresponding real value. In this way, a second training of the LSTM network structure of the provisioned LSTM network may be completed.

Aiming at the LSTM network structure of the third layer, the predicted value z of the second layer is determined₁-z_n-2Inputting the data into a third-layer LSTM network structure to obtain z₁-z_n-2Respectively corresponding predicted values t₁-t_n-2Wherein t is predicted₁-t_n-2The corresponding true value is z in the output predicted value of the second layer LSTM network structure₂-z_n-1. And calculating the loss value between each pre-stored value and the corresponding real value. Thus, the third time of training the LSTM network structure of the preset LSTM network can be completed.

In embodiment 2, since the noise data in the target measurement value can be enhanced step by step in a cascade manner, the target measurement value can be corrected step by step, so that the corrected target measurement value can be corrected better, and the detection accuracy of the vehicle-mounted advanced driving assistance system for the obstacle can be further improved.

For example, as shown in fig. 4, the ultrasonic measurement values shown in fig. 3 are corrected by the correction model trained in the above-described embodiment 2, and the corrected ultrasonic measurement values are obtained. It can be seen that the correction model trained in the above embodiment 2 well suppresses the noise data in the target measurement values.

In addition, the execution sequence of the step S101 and the step S102 may be to execute the step S101 first and then execute the step S102; step S102 may be executed first, and then step S101 may be executed; it is also possible to perform steps S101 and S102 simultaneously, which is reasonable.

Thus, after the steps S101 and S102 are executed, and the recognition result and the target measurement value are obtained, the vehicle-mounted advanced driving assistance system can determine whether the recognition result represents that the target obstacle exists in the image to be recognized. Furthermore, when it is determined that the recognition result indicates that the target obstacle exists in the image to be recognized, the vehicle-mounted advanced driving assistance system may continue to perform step S103, where the reference position coordinates of the position point corresponding to the target measurement value in the preset reference coordinate system and the reference area coordinates of the target obstacle in the reference coordinate system are determined.

The advanced driving assistance system mounted on the vehicle may perform the step S103 in various ways, and the embodiment of the present invention is not limited in particular.

As an example, as shown in fig. 7, the manner in which the vehicle-mounted advanced driving assistance system performs the determination of the reference position coordinates of the position point corresponding to the target measured value in the preset reference coordinate system and the determination of the reference area coordinates of the target obstacle in the reference coordinate system may include the steps of:

s701: determining the position point coordinates of the position point corresponding to the target measurement value in a preset vehicle coordinate system based on the installation position of the vehicle-mounted ultrasonic sensor;

it can be understood that, according to the installation position of the vehicle-mounted ultrasonic sensor on the vehicle and according to the coordinate calculation in the three-dimensional coordinate system, the vehicle-mounted advanced driving assistance system can calculate the coordinates of the vehicle-mounted ultrasonic sensor in the preset vehicle coordinate system. The vehicle coordinate system may be a three-dimensional coordinate system established with any point in the vehicle as a coordinate origin.

Furthermore, according to the measurement principle of the vehicle-mounted ultrasonic sensor and the mode of transmitting ultrasonic waves in the air, similarly, according to the coordinate calculation in the three-dimensional coordinate system, the vehicle-mounted advanced driving assistance system can determine the position point coordinates of the position point corresponding to the target measured value in the preset vehicle coordinate system by using the acquired target measured value.

In this case, the advanced driving assistance system mounted on the vehicle may calculate and store coordinates of the ultrasonic sensor mounted on the vehicle in the preset vehicle coordinate system in advance, so that the coordinates may be directly acquired and used when the above step S701 is performed. The coordinates of the on-vehicle ultrasonic sensor in the preset vehicle coordinate system may be calculated again when the above step S701 is executed.

S702: calculating the corresponding reference position coordinate of the position point coordinate in the preset reference coordinate system by using a preset coordinate conversion formula,

the system comprises a vehicle coordinate system, a reference coordinate system, a coordinate conversion formula and a coordinate conversion formula, wherein the reference coordinate system is an image coordinate system of an image to be identified, and the coordinate conversion formula is a coordinate conversion formula from the vehicle coordinate system to the reference coordinate system;

in this example, the image coordinate system of the image to be recognized may be determined as the reference coordinate system, and further, a coordinate conversion formula from the vehicle coordinate system to the reference coordinate system may be determined. It is reasonable that the coordinate transformation formula is determined locally by the vehicle-mounted advanced assistant driving system, or the vehicle-mounted advanced assistant driving system obtains from other electronic devices connected in communication.

Therefore, after the position point coordinates of the position point corresponding to the target measured value in the preset vehicle coordinate system are obtained, the vehicle-mounted advanced assistant driving system can calculate the corresponding reference position coordinates of the position point coordinates in the preset reference coordinate system through coordinate operation in the three-dimensional coordinate system by using the coordinate conversion formula.

Preferably, the expression of the coordinate conversion formula may be as follows:

wherein, (X, Y, Z) is the position point coordinate of the position point corresponding to the target measured value in the preset vehicle coordinate system, (f)_x,f_y,c_x,c_y) The coordinates are internal parameters of the image acquisition equipment, and the (u, v) coordinates are corresponding reference position coordinates of the position point coordinates in a preset reference coordinate system.

S703: and determining an image area of the target obstacle in the image to be recognized, and determining image area coordinates of the image area in an image coordinate system as reference area coordinates of the target obstacle in a reference coordinate system.

In this example, since the reference coordinate system is the image coordinate system of the image to be recognized, the vehicle-mounted advanced driving assistance system may determine the image area coordinates of the image area of the target obstacle in the image to be recognized in the image coordinate system as the reference area coordinates of the target obstacle in the reference coordinate system.

In the above specific implementation, the reference coordinate system is an image coordinate system of the image to be recognized, and it can be understood that the reference coordinate system may be other coordinate systems.

Based on this, optionally, in another specific implementation, the reference coordinate system may be a preset vehicle coordinate system. Therefore, in this example, the manner in which the advanced driving assistance system mounted on the vehicle performs the determination of the reference position coordinates of the position point corresponding to the target measurement value in the preset reference coordinate system, and the determination of the reference area coordinates of the target obstacle in the reference coordinate system may include the steps of:

step 1: determining the position point coordinates of the position point corresponding to the target measured value in a preset vehicle coordinate system based on the installation position of the vehicle-mounted ultrasonic sensor, and determining the position point coordinates as the reference position coordinates of the position point corresponding to the target measured value in a preset reference coordinate system;

as in step S701 in the above embodiment, the advanced driving assistance system mounted on the vehicle may determine the position point coordinates of the position point corresponding to the target measurement value in the preset vehicle coordinate system based on the mounting position of the ultrasonic sensor mounted on the vehicle. Further, since the reference coordinate system is the vehicle coordinate system in this example, the vehicle-mounted advanced driving assistance system can determine the determined coordinates of the position point as the reference coordinates of the position point corresponding to the target measurement value in the reference coordinate system.

Step 2: determining an image area of the target obstacle in the image to be recognized, and acquiring an image area coordinate of the image area in an image coordinate system of the image to be recognized; and calculating the reference area coordinate of the image area coordinate in the reference coordinate system by using a preset coordinate conversion formula from the image coordinate system to the vehicle coordinate system.

The vehicle-mounted advanced driving assistance system can firstly determine an image area of a target obstacle in an image to be recognized in the image to be recognized, and then acquire an image area coordinate of the image area in an image coordinate system of the image to be recognized. Furthermore, based on the imaging principle of the image capturing device, the vehicle-mounted advanced driving assistance system can calculate the reference area coordinates of the image area coordinates corresponding to the image area of the target obstacle in the image to be recognized in the vehicle coordinate system through coordinate operation in the three-dimensional coordinate system by using the coordinate conversion formula from the image coordinate system to the vehicle coordinate system. In this way, since the reference coordinate system is the vehicle coordinate system in this example, the vehicle-mounted advanced driving assistance system can determine the determined reference area coordinates as the reference area coordinates of the image area coordinates of the target obstacle in the image to be recognized in the reference coordinate system, which correspond to the image area.

In addition, it can be understood that, in practical applications, in order to ensure the ranging accuracy of the vehicle-mounted ultrasonic sensor, the vehicle-mounted ultrasonic sensor has an effective detection distance. The effective detection distance of the vehicle-mounted ultrasonic sensor is the detection range of the vehicle-mounted ultrasonic sensor. When the ultrasonic measurement value acquired by the vehicle-mounted ultrasonic sensor exceeds the detection range, the ultrasonic measurement value is invalid.

Therefore, in order to ensure that the target measurement value acquired by the vehicle-mounted advanced driving assistance system in step S102 is valid, the distance measurement accuracy of the vehicle-mounted ultrasonic sensor is improved, and the detection accuracy of the vehicle-mounted advanced driving assistance system on the obstacle is improved. As an example, after acquiring the target measurement value, the advanced driving assistance system on the vehicle may first determine the magnitude relationship between the target measurement value and a preset measurement threshold value. Wherein the preset measurement threshold is not larger than the detection range of the vehicle-mounted ultrasonic sensor.

For example, when the detection range of the vehicle-mounted ultrasonic sensor is 5m, the preset measurement threshold may be 5m, 4m, or 3m, as long as the preset measurement threshold is not greater than 5 m. The smaller the preset measurement threshold is, the higher the distance measurement accuracy of the vehicle-mounted ultrasonic sensor is, and further, the higher the detection accuracy of the vehicle-mounted advanced driving assistance system on the obstacle is, that is, the more reliable the obstacle detection result of the vehicle-mounted advanced driving assistance system is.

When the preset measurement threshold value is smaller than the detection range of the vehicle-mounted ultrasonic sensor, the accuracy of a target measurement value can be further ensured, and then the detection accuracy of the vehicle-mounted advanced driving assistance system on the obstacle is effectively improved.

Accordingly, in this example, the way in which the vehicle-mounted advanced driving assistance system performs the above step S103, determines the reference position coordinates of the position point corresponding to the target measurement value in the preset reference coordinate system and determines the reference area coordinates of the target obstacle in the reference coordinate system when the recognition result indicates that the target obstacle exists in the image to be recognized may include the following steps:

when the recognition result represents that a target obstacle exists in the image to be recognized and the target measurement value is not larger than a preset measurement threshold value, determining a reference position coordinate of a position point corresponding to the target measurement value under a preset reference coordinate system and determining a reference area coordinate of the target obstacle under the reference coordinate system; and the preset measurement threshold value is not larger than the detection range of the vehicle-mounted ultrasonic sensor.

In this example, the reference position coordinates of the position point corresponding to the target measurement value in the preset reference coordinate system and the reference area coordinates of the target obstacle in the reference coordinate system are determined in the same manner as the reference position coordinates of the position point corresponding to the target measurement value in the preset reference coordinate system and the reference area coordinates of the target obstacle in the reference coordinate system in step S103. And will not be described in detail herein.

When the vehicle-mounted advanced driving assistance system corrects the target measurement value, in the above example, the vehicle-mounted advanced driving assistance system determines the magnitude relationship between the target measurement value and the preset measurement threshold, that is, determines the magnitude relationship between the corrected target measurement value and the preset measurement threshold.

Thus, after the step S103 is executed, the reference position coordinates of the position point corresponding to the target measurement value in the preset reference coordinate system and the reference area coordinates of the target obstacle in the reference coordinate system are obtained, and then the vehicle-mounted advanced driving assistance system may determine whether the reference position coordinates are located in the area range corresponding to the reference area coordinates.

Obviously, when the reference position coordinates are located in the area range corresponding to the reference area coordinates, it can be stated that the target obstacle existing in the image to be recognized and the obstacle corresponding to the target measurement value may be the same obstacle. That is to say, at the current moment, the obstacles detected by the vehicle-mounted image acquisition device and the vehicle-mounted ultrasonic sensor are the same obstacle, and the obstacle is the target obstacle existing in the image to be recognized.

Thus, the vehicle-mounted advanced driving assistance system can determine the target measurement value as the distance between the target obstacle and the vehicle, determine the size information of the object determined based on the reference area coordinates as the size information of the target obstacle, and obtain the detection result of the target obstacle.

It can be understood that, with respect to the case that the reference position coordinate of the position point corresponding to the target measurement value in the preset reference coordinate system is located in the area range corresponding to the reference area coordinate of the target obstacle in the reference coordinate system in the step S104, there may be a case that the reference position is not located in the area range corresponding to the reference area coordinate.

Obviously, when the reference position is not located within the range of the area corresponding to the reference area coordinates, the vehicle-mounted advanced driving assistance system cannot fuse the detection results of the vehicle-mounted image capturing device and the vehicle-mounted ultrasonic sensor, and thus cannot use the fused result as the detection result of the target obstacle. Therefore, in this case, the advanced driving assistance system mounted on the vehicle needs to obtain respective obstacle detection results for the detection results of the on-vehicle image capture device and the on-vehicle ultrasonic sensor, respectively.

Based on this, optionally, as shown in fig. 8, another obstacle detection method provided in the embodiment of the present invention may include the following steps:

s801: acquiring an image to be identified acquired by vehicle-mounted image acquisition equipment, and identifying an obstacle on the image to be identified to obtain an identification result;

s802: acquiring a target measured value acquired by a vehicle-mounted ultrasonic sensor;

s803: when the recognition result represents that the target obstacle exists in the image to be recognized, determining a reference position coordinate of a position point corresponding to the target measurement value under a preset reference coordinate system, and determining a reference area coordinate of the target obstacle under the reference coordinate system;

s804: when the reference position coordinate is located in the area range corresponding to the reference area coordinate, determining a target measurement value as the distance between a target obstacle and a vehicle, and determining object size information determined based on the reference area coordinate as the size information of the target obstacle to obtain a detection result of the target obstacle;

s805: when the reference position information is not located in the reference area coordinates included in the identification result, determining the reference position coordinates as the position information of a first obstacle detected by the vehicle-mounted ultrasonic sensor, and determining the target measurement value as the distance between the first obstacle and the vehicle to obtain the detection result of the first obstacle; determining the size information of the object determined based on the coordinates of the reference area as the size information of the target obstacle, and determining the monocular distance as the distance between the target obstacle and the vehicle to obtain the detection result of the target obstacle;

the monocular distance measuring distance is detected based on the image to be identified by utilizing a monocular distance measuring algorithm.

When the reference position coordinate of the position point corresponding to the determined target measurement value in the preset reference coordinate system is not located in the area range corresponding to the reference area coordinate of the target obstacle in the reference coordinate system, the vehicle-mounted advanced driving assistance system can respectively obtain respective obstacle detection results according to the detection results of the vehicle-mounted image acquisition device and the vehicle-mounted ultrasonic sensor.

Specifically, for the detection result of the vehicle-mounted ultrasonic sensor, the vehicle-mounted advanced driving assistance system may determine the reference position coordinate of the position point corresponding to the determined target measurement value in the preset reference coordinate system as the position information of the first obstacle detected by the vehicle-mounted ultrasonic sensor, and determine the target measurement value as the distance between the first obstacle and the vehicle. Thus, the vehicle-mounted advanced driving assistance system can obtain the position information of the first obstacle and the distance between the vehicle-mounted ultrasonic sensor and the vehicle, and accordingly obtain the detection result of the first obstacle.

For the detection result of the vehicle-mounted image acquisition device, the vehicle-mounted advanced driving assistance system may first determine the object size information corresponding to the reference area coordinate of the determined target obstacle in the reference coordinate system, and further determine the object size information as the size information of the target obstacle detected by the vehicle-mounted image acquisition device; furthermore, the vehicle-mounted advanced driving assistance system can also detect the image to be recognized by utilizing a monocular distance measurement algorithm so as to obtain a monocular distance measurement distance, and the distance is determined as the distance between the target obstacle and the vehicle. Therefore, the vehicle-mounted advanced driving assistance system can obtain the size information of the target obstacle and the distance between the vehicle-mounted advanced driving assistance system and the vehicle, which are detected by the vehicle-mounted image acquisition equipment, so that the detection result of the target obstacle is obtained.

In this way, when the recognition result represents that the target obstacle exists in the image to be recognized, the target measurement value is not larger than the preset measurement threshold value, and the determined reference position coordinate is not located in the area range corresponding to the determined reference area coordinate, the vehicle-mounted advanced driving assistance system can still obtain respective obstacle detection results respectively aiming at the detection results of the vehicle-mounted image acquisition device and the vehicle-mounted ultrasonic sensor.

The monocular distance measurement algorithm is utilized to detect the image to be recognized, and the specific process of obtaining the monocular distance measurement distance can be as follows:

since the relative height between the vehicle-mounted image acquisition device and the ground is fixed in practical application, the distance of each point on the ground can be estimated by using the principle of inverse perspective transformation on the assumption that the ground is a plane. Accordingly, for a target obstacle existing in the image to be recognized, since the target obstacle can be considered to be on the ground, the midpoint of the bottom side of the image area where the target obstacle existing in the image to be recognized is located can be considered to be also on the ground. Furthermore, the middle point of the bottom edge of the image area where the target obstacle exists in the image to be recognized can be subjected to inverse perspective transformation, and therefore the monocular distance measurement distance can be obtained.

Furthermore, the detection result of the first obstacle and the detection result of the target obstacle, which correspond to the vehicle-mounted image acquisition device and the vehicle-mounted ultrasonic sensor respectively, can be input into the decision module of the vehicle-mounted advanced driving assistance system, so that the decision module of the vehicle-mounted advanced driving assistance system can still obtain a final control strategy to control the vehicle operation.

When the recognition result determined by the vehicle-mounted advanced driving assistance system can represent that a target obstacle exists in the image to be recognized, the image to be recognized can be detected by continuously utilizing a monocular distance measurement algorithm, so that a monocular distance measurement distance is obtained, and after the reference area coordinate of the target obstacle in the reference coordinate system is determined, the vehicle-mounted advanced driving assistance system can also determine the object size information corresponding to the reference area coordinate based on the determined reference area coordinate of the target obstacle in the reference coordinate system. In this case, the vehicle-mounted advanced driving support system may store the obtained monocular distance measurement distance and the object size information, and when it is determined that the reference position is not located within the area range corresponding to the reference area coordinate, the vehicle-mounted advanced driving support system may directly determine the stored monocular distance measurement distance as the distance between the target obstacle and the vehicle, and may determine the stored object size information as the size information of the target obstacle, to obtain the detection result of the target obstacle.

That is, with respect to the detection result of the vehicle-mounted image capturing apparatus, the monocular distance measuring distance and the object size information may be determined based on the image to be recognized after the step S801 is performed, or may be determined based on the image to be recognized when the reference position is determined not to be located within the area corresponding to the reference area coordinate. This is all reasonable.

The implementation manners of steps S801 to S804 in the embodiment shown in fig. 8 are the same as the implementation manners of steps S101 to S104 in the embodiment shown in fig. 1, and are not described herein again.

Further, it can be understood that, because the performances of the vehicle-mounted image acquisition device and the vehicle-mounted ultrasonic sensor may be different, in some other cases, a case may occur where the recognition result represents that a target obstacle exists in the image to be recognized, and the target measurement value is greater than the preset measurement threshold value; alternatively, the recognition result may represent that no target obstacle exists in the image to be recognized, and the target measurement value is not greater than the preset measurement threshold value. That is, a case may occur in which only one of the on-vehicle image pickup device and the on-vehicle ultrasonic sensor detects an obstacle.

In order to ensure that in the above situation, the decision module of the vehicle-mounted advanced driving assistance system can also obtain a decision result to control the vehicle, and the vehicle-mounted advanced driving assistance system can determine a final obstacle detection result according to a detection result of the device which detects the obstacle, and input the determined final obstacle detection result to the decision module of the vehicle-mounted advanced driving assistance system to ensure that the decision module of the vehicle-mounted advanced driving assistance system can still obtain a final control strategy to control the vehicle operation.

Optionally, when the recognition result represents that the target obstacle exists in the image to be recognized, and the target measurement value is greater than the preset measurement threshold, the vehicle-mounted advanced driving assistance system may determine, as the size information of the target obstacle, the object size information determined based on the reference area coordinates, and determine the single visual distance as the distance between the target obstacle and the vehicle, so as to obtain the detection result of the target obstacle. Further, the detection result of the target obstacle is input to the decision module.

Optionally, when the recognition result indicates that the target obstacle does not exist in the image to be recognized, and the target measurement value is not greater than the preset measurement threshold, the vehicle-mounted advanced driving assistance system may determine the reference position coordinate as the position information of the first obstacle detected by the vehicle-mounted ultrasonic sensor, and determine the target measurement value as the distance between the first obstacle and the vehicle, so as to obtain the detection result of the first obstacle. Further, the detection result of the first obstacle is input to the decision module.

Corresponding to the obstacle detection method provided by the embodiment of the invention, the embodiment of the invention also provides an obstacle detection device. The device is applied to a vehicle-mounted advanced driving assistance system.

Fig. 9 is a schematic structural diagram of an obstacle detection device according to an embodiment of the present invention. As shown in fig. 9, the obstacle detecting apparatus may include the following modules:

the image data acquisition module 910 is configured to acquire an image to be identified, which is acquired by the vehicle-mounted image acquisition device, and perform obstacle identification on the image to be identified to obtain an identification result;

an ultrasonic data acquisition module 920, configured to acquire a target measurement value acquired by a vehicle-mounted ultrasonic sensor;

a coordinate information determining module 930, configured to determine, when the recognition result indicates that the target obstacle exists in the image to be recognized, a reference position coordinate of a position point corresponding to the target measurement value in a preset reference coordinate system, and a reference area coordinate of the target obstacle in the reference coordinate system;

and a detection result determining module 940, configured to determine the target measurement value as a distance between the target obstacle and the vehicle when the reference position coordinate is located within the area range corresponding to the reference area coordinate, and determine the object size information determined based on the reference area coordinate as size information of the target obstacle, so as to obtain a detection result of the target obstacle.

As an example, the image data acquiring module 910 may be specifically configured to:

As an example, the ultrasound data acquisition module 920 may be specifically configured to:

As an example, the coordinate information determining module 930 may be specifically configured to determine, when the recognition result indicates that a target obstacle exists in the image to be recognized, and the target measurement value is not greater than a preset measurement threshold, a reference position coordinate of a position point corresponding to the target measurement value in a preset reference coordinate system, and a reference area coordinate of the target obstacle in the reference coordinate system; and the preset measurement threshold value is not larger than the detection range of the vehicle-mounted ultrasonic sensor.

As an example, the coordinate information determining module 930 may be specifically configured to:

determining the position point coordinates of the position point corresponding to the target measurement value in a preset vehicle coordinate system based on the installation position of the vehicle-mounted ultrasonic sensor; calculating a reference position coordinate corresponding to the position point coordinate in a preset reference coordinate system by using a preset coordinate conversion formula, wherein the reference coordinate system is an image coordinate system of the image to be recognized, and the coordinate conversion formula is a coordinate conversion formula from a vehicle coordinate system to the reference coordinate system; determining an image area of the target obstacle in the image to be recognized, determining image area coordinates of the image area in an image coordinate system, and determining the image area coordinates as reference area coordinates of the target obstacle in a reference coordinate system.

carrying out obstacle identification on the image to be identified to obtain an initial result, and judging whether the initial result represents that an obstacle exists in the image to be identified; if so, taking the initial result as a recognition result; if not, judging whether obstacles exist in a plurality of continuous recognized images to be recognized before the image to be recognized; when obstacles exist in a plurality of continuous recognized images to be recognized before the images to be recognized are judged, determining a tracking result of the images to be recognized as a recognition result by using an image tracking algorithm based on an initial result and a recognition result of the recognized images to be recognized; otherwise, the initial result is used as the recognition result.

As an example, the obstacle detection device may further include:

the obstacle information determining module is used for determining the reference position coordinate as the position information of a first obstacle detected by the vehicle-mounted ultrasonic sensor and determining the target measurement value as the distance between the first obstacle and the vehicle to obtain the detection result of the first obstacle when the reference position information is not located in the reference area coordinate included in the identification result; determining the size information of the object determined based on the coordinates of the reference area as the size information of the target obstacle, and determining the monocular distance as the distance between the target obstacle and the vehicle to obtain the detection result of the target obstacle; the monocular distance measuring distance is detected based on the image to be identified by utilizing a monocular distance measuring algorithm.

Corresponding to the obstacle detection method provided by the embodiment of the invention, the embodiment of the invention also provides vehicle-mounted electronic equipment, and the electronic equipment is provided with an advanced driving assistance system. As shown in fig. 10, the system comprises a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002 and the memory 1003 are communicated with each other through the communication bus 1004,

a memory 1003 for storing a computer program;

the processor 1001 is configured to implement the obstacle detection method according to the embodiment of the present invention when executing the program stored in the memory 1003.

Specifically, the obstacle detection method is applied to an advanced driving assistance system installed on the electronic device, and includes:

when the recognition result represents that the target obstacle exists in the image to be recognized, determining a reference position coordinate of a position point corresponding to the target measurement value under a preset reference coordinate system, and determining a reference area coordinate of the target obstacle under the reference coordinate system;

and when the reference position coordinate is located in the area range corresponding to the reference area coordinate, determining the target measurement value as the distance between the target obstacle and the vehicle, and determining the object size information determined based on the reference area coordinate as the size information of the target obstacle to obtain the detection result of the target obstacle.

It should be noted that other implementation manners of the obstacle detection method implemented by the processor 1001 executing the program stored in the memory 1003 are the same as the obstacle detection method embodiment provided in the foregoing method embodiment section, and are not described herein again.

Therefore, the obtained detection result of the target obstacle is input into a decision module of the vehicle-mounted advanced driving assistance system, and the decision module can directly determine a final control strategy based on the fusion result of the detection results of the vehicle-mounted image acquisition equipment and the vehicle-mounted ultrasonic sensor. The detection results of the vehicle-mounted image acquisition equipment and the vehicle-mounted ultrasonic sensor can mutually make up for the defects in the detection results of the other side, so that the decision difficulty of the decision module for determining the final control strategy can be reduced, and the optimal final control strategy can be obtained. The communication bus mentioned in the above vehicle-mounted electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the vehicle-mounted electronic equipment and other equipment.

The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.

The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.

In response to the obstacle detection method provided in the embodiment of the present invention, an embodiment of the present invention further provides a computer-readable storage medium, and when being executed by a processor, the computer program implements the obstacle detection method provided in the embodiment of the present invention.

Specifically, the obstacle detection method is applied to a vehicle-mounted advanced driving assistance system, and includes:

It should be noted that other implementation manners of the obstacle detection method implemented when the computer program is executed by the processor are the same as the obstacle detection method embodiment provided in the foregoing method embodiment section, and are not described again here.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the apparatus embodiment, the vehicle-mounted electronic device embodiment and the computer-readable storage medium embodiment, since they are substantially similar to the method embodiment, the description is relatively simple, and the relevant points can be referred to the partial description of the method embodiment.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims

1. An obstacle detection method applied to an advanced driving assistance system for vehicle, the method comprising:

2. The method according to claim 1, wherein the step of acquiring the image to be identified acquired by the vehicle-mounted image acquisition device comprises:

3. The method of claim 1, wherein the step of obtaining target measurements acquired by the on-board ultrasonic sensor comprises:

4. The method according to any one of claims 1 to 3, wherein the step of determining the reference position coordinates of the position point corresponding to the target measurement value in a preset reference coordinate system and determining the reference area coordinates of the target obstacle in the reference coordinate system when the recognition result indicates that the target obstacle exists in the image to be recognized comprises:

5. The method of claim 1, wherein the steps of determining reference location coordinates of the location point corresponding to the target measurement in a preset reference coordinate system and determining reference area coordinates of the target obstacle in the reference coordinate system comprise:

6. The method according to claim 1, wherein the step of performing obstacle recognition on the image to be recognized to obtain a recognition result comprises:

if so, taking the initial result as a recognition result;

otherwise, the initial result is used as a recognition result.

7. The method of claim 1, wherein after the step of obtaining target measurements acquired by the on-board ultrasonic sensor, the method further comprises:

8. The method of claim 1, further comprising:

9. An obstacle detection device, for use in an advanced driving assistance system for vehicle, the device comprising:

10. The apparatus of claim 9, wherein the image data acquisition module is specifically configured to:

11. The apparatus of claim 9, wherein the ultrasound data acquisition module is specifically configured to:

12. The apparatus according to any one of claims 9-11, wherein the coordinate information determination module is specifically configured to:

13. The apparatus of claim 9, wherein the coordinate information determination module is specifically configured to:

14. The apparatus of claim 9, wherein the image data acquisition module is specifically configured to:

15. The vehicle-mounted electronic equipment is characterized in that an advanced driving assistance system is mounted on the electronic equipment and comprises a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for completing mutual communication through the communication bus;

a memory for storing a computer program;

a processor for implementing the method steps of any of claims 1 to 8 when executing a program stored in the memory.

16. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of the claims 1-8.