CN111814667B

CN111814667B - Intelligent road condition identification method

Info

Publication number: CN111814667B
Application number: CN202010649661.2A
Authority: CN
Inventors: 方亚东; 杨勤; 王洪添; 徐宏伟; 宋设; 姚民伟
Original assignee: Shandong Inspur Cloud Service Information Technology Co Ltd
Current assignee: Shandong Inspur Cloud Service Information Technology Co Ltd
Priority date: 2020-07-08
Filing date: 2020-07-08
Publication date: 2022-10-14
Anticipated expiration: 2040-07-08
Also published as: CN111814667A

Abstract

The invention discloses an intelligent road condition identification method, which belongs to the technical field of intelligent driving, and comprises the steps of collecting the instantaneous speed of a vehicle, collecting image information and depth distance information of road conditions, constructing a convolutional neural network training and verifying set by using the collected information, training a convolutional neural network model by using the training and verifying set, continuously optimizing parameters, applying the continuously optimized neural network parameter model to acceleration and deceleration of automatic driving, and adaptively controlling the acceleration and deceleration of the vehicle in real-time depth image identification. The convolutional neural network technology based on the depth image can provide an optimized solution for automatic driving from the aspects of road condition identification and acceleration and deceleration, and effectively improve the speed control precision and safety of automatic driving.

Description

Intelligent road condition identification method

Technical Field

The invention relates to the technical field of intelligent driving, in particular to an intelligent road condition identification method.

Background

Under the background of the explosive development of artificial intelligence and machine learning, the automatic driving technology is also produced. The SAE american automobile engineers classify the automatic driving technology into L0 to L5, and the current mainstream automatic driving in the market is L1 and L2 automatic driving, which is limited by the development speed of the intelligent identification technology and the complexity of the road scene, and the L3 automatic driving technology is not common in mass production of automobiles. The L3 automatic driving technology requires that vehicles can realize most vehicle-mounted machine function operations, can deal with vehicles in most situations, and can actively finish a series of actions such as acceleration and deceleration, lane changing, overtaking, left and right turning, traffic light identification and the like, so that the requirements on road condition identification and speed control are high.

Disclosure of Invention

The technical task of the invention is to provide an intelligent road condition identification method aiming at the defects, and the method can effectively improve the speed control precision and the safety of the L3 level automatic driving.

The technical scheme adopted by the invention for solving the technical problems is as follows:

an intelligent road condition identification method comprises the steps of collecting the instantaneous speed of a vehicle, collecting image information and depth distance information of road conditions, constructing a convolutional neural network training and verifying set by utilizing the collected information, training a convolutional neural network model by using the training and verifying set, continuously optimizing parameters, applying the continuously optimized neural network parameter model to acceleration and deceleration of automatic driving, and adaptively controlling the acceleration and deceleration of the vehicle in real-time depth image identification.

The method combines the characteristics of large data volume, complex road conditions and different driving habits of different drivers, acquires depth images aiming at data of various scenes, integrates an object recognition model which is more in line with human mind at a training position by combining the judgment of the driver on the road conditions and the increase and decrease conditions of the vehicle speed, and provides a certain auxiliary basis for the road condition recognition and the acceleration and deceleration of automatic driving.

Preferably, the Kinect depth camera is used for shooting and acquiring image information and depth distance information of road conditions.

Preferably, the vehicle instantaneous speed is collected using the vehicle's OBD module. The OBD interface of the automobile is connected with the computer through the gateway device, so that automobile data can be checked in real time, automobile faults can be detected, and the like.

Specifically, the method utilizes the comparison of the change of the same image depth distance in a certain time period and the starting point vehicle speed to identify the image, and then trains the convolutional neural network by using the identification image set to obtain a training model capable of identifying road condition information of different scenes, so that the speed control precision and the safety of the L3 level automatic driving can be improved to a certain extent.

Further, the specific implementation manner of the method is as follows:

1) Acquiring the instantaneous speed v1 of the vehicle, and acquiring a depth image a of the road condition in front of the vehicle and a depth distance da corresponding to a certain point x;

2) After a short time period t, acquiring a depth image b of the road condition in front of the vehicle and a depth distance db corresponding to a certain point x;

3) Using the formula

Obtaining the average speed vt of the vehicle in a time period t;

4) If vt is larger than or equal to v1, marking the depth image a as barrier-free and accelerating; if vt is less than v1, marking the depth image a as an obstacle, and decelerating;

5) Forming a training data set by using the information obtained in the step 1) and the step 4), performing convolutional neural network training, storing a training result and continuously optimizing a neural network model;

6) And applying the continuously optimized neural network parameter model to an acceleration and deceleration program of automatic driving, and adaptively controlling the acceleration and deceleration of the vehicle within a certain range in real-time depth image recognition so as to achieve better safety.

Preferably, the time period t is 500ms in time length.

Preferably, the convolutional neural network training is performed by using a convolutional neural network and taking 1/3 and 1/4 of the total pixels of the acquired picture as convolutional kernels respectively.

Preferably, the neural network parameter model is applied to an acceleration and deceleration program of automatic driving of the vehicle through a driving computer interface.

The invention also claims an intelligent road condition recognition device, comprising: at least one memory and at least one processor;

the at least one memory to store a machine readable program;

the at least one processor is used for calling the machine readable program and executing the method.

The invention also claims a computer readable medium having stored thereon computer instructions which, when executed by a processor, cause the processor to perform the above-described method.

Compared with the prior art, the intelligent road condition identification method has the following beneficial effects:

the method combines the characteristics of large data volume, complex road conditions and different driving habits of different drivers, acquires depth images aiming at data of various scenes, integrates an object recognition model which is more in line with the thinking of a human brain at a training position by combining the judgment of the driver on the road conditions and the increase and decrease conditions of the vehicle speed, and provides an optimized solution for L3 automatic driving from the aspects of road condition recognition and acceleration and deceleration based on the convolutional neural network technology of the depth images.

Drawings

Fig. 1 is a flowchart of an intelligent road condition identification method according to an embodiment of the present invention;

fig. 2 is a diagram of an application example provided by an embodiment of the present invention.

Detailed Description

The invention is further described with reference to the following figures and specific examples.

In practical application, some neural network structures adopt an improved data enhancement algorithm to solve the problem of insufficient scale of training data, so that the effect of increasing the training data is achieved. However, as the driving scenes of automobiles are more and the road conditions are more and more complex, it is always necessary to continuously collect and integrate different training sets, if the number of training data is increased from the algorithm perspective, the error is often larger in practical application, meanwhile, different drivers react differently to different driving road conditions, and the recognition of the road conditions is optimized by combining the driving habits of the drivers and other factors.

The invention aims to solve the existing problems, provides application of a convolutional neural network technology based on a depth image in intelligent road condition identification, and provides an optimized solution for L3 automatic driving from the aspects of road condition identification and acceleration and deceleration.

An intelligent road condition identification method combines the characteristics of large data volume, complex road condition and different driving habits of different drivers, acquires depth images aiming at data of various scenes, integrates the judgment of the drivers on the road condition and the increase and decrease of the vehicle speed, and integrates an object identification model which is more in line with human brain thinking at the training position:

acquiring the instantaneous speed of the vehicle by using an OBD (on-board diagnostics) module of the vehicle, and shooting and acquiring image information and depth distance information of the front road condition by using a Kinect depth camera;

constructing a convolutional neural network training and verifying set by using the acquired information;

training the convolutional neural network model by using the training and verification set, continuously optimizing parameters, applying the continuously optimized neural network parameter model to acceleration and deceleration of automatic driving, and adaptively controlling the acceleration and deceleration of a vehicle in real-time depth image recognition.

Kinectfor Xbox 360, abbreviated as Kinect, is developed by microsoft and applied to peripheral devices of the Xbox 360 host. The Kinect has three lenses, the middle lens is an RGB color camera, which is used to collect color images. The left and right lenses are 3D structured light depth sensors formed by an infrared emitter and an infrared CMOS camera, respectively, and are used to collect depth data (the distance from an object to the camera in the scene of this embodiment). The color camera can maximally support 1280 × 960 resolution imaging, and the infrared camera can maximally support 640 × 480 imaging. Kinect has still mated the technique of following focus, and the base motor can follow with focusing the removal of object and rotate.

The OBD interface is also called a vehicle-mounted automatic diagnosis system and can detect the working conditions of an engine electric control system and other functional modules of a vehicle in the running process of the vehicle. The OBD interface of the automobile is connected with the computer, so that automobile data can be checked in real time, automobile faults can be detected, and convenience is provided for automobile repair, automobile related function development and automobile equipment modification. In the embodiment, the OBD interface is connected with a computer through a gateway device, so that the real-time checking of automobile data, the detection of automobile faults and the like are realized.

Convolutional Neural Networks (CNN) are an efficient identification method that has been developed in recent years and has attracted considerable attention. CNN has become one of the research hotspots in many scientific fields, especially in the field of pattern classification, and since the network avoids the complex preprocessing of the image and can directly input the original image, it has been more widely applied.

In general, the basic structure of CNN includes two layers, one of which is a feature extraction layer, and the input of each neuron is connected to a local acceptance domain of the previous layer and extracts the feature of the local acceptance domain. Once the local feature is extracted, the position relation between the local feature and other features is determined; the other is a feature mapping layer, each calculation layer of the network is composed of a plurality of feature mappings, each feature mapping is a plane, and the weights of all neurons on the plane are equal. The feature mapping structure adopts a sigmoid function with small influence function kernel as an activation function of the convolution network, so that the feature mapping has displacement invariance. In addition, since the neurons on one mapping surface share the weight, the number of free parameters of the network is reduced. Each convolutional layer in the convolutional neural network is followed by a computation layer for local averaging and quadratic extraction, which reduces the feature resolution.

CNN is mainly used for identifying two-dimensional graphs of displacement, scaling and other forms of distortion invariance, and the part of functions are mainly realized by a pooling layer. Since the feature detection layer of CNN learns from the training data, explicit feature extraction is avoided when CNN is used, while learning from the training data is implicit; moreover, because the weights of the neurons on the same feature mapping surface are the same, the network can learn in parallel, and the advantage of the convolutional network relative to the network in which the neurons are connected with one another is also realized. The convolution neural network has unique superiority in the aspects of voice recognition and image processing by virtue of a special structure with shared local weight, the layout of the convolution neural network is closer to that of an actual biological neural network, the complexity of the network is reduced by virtue of weight sharing, and particularly, the complexity of data reconstruction in the processes of feature extraction and classification is avoided by virtue of the characteristic that an image of a multi-dimensional input vector can be directly input into the network.

If a classical neural network model is used, the whole image needs to be read as an input of the neural network model (i.e. a fully connected mode), and when the size of the image is larger, the connected parameters of the image become more, so that the calculation amount is very large.

The human cognition to the outside world generally is from local to global, namely, the human cognition is that the local part is perceived firstly and then the whole body is perceived gradually, which is the human cognition mode. The spatial relationship in the image is similar, the relationship between pixels in the local range is relatively close, and the correlation between pixels at a relatively long distance is relatively weak. Therefore, each neuron does not need to sense a global image, only needs to sense a local part, and then integrates local information at a higher layer to obtain global information. This model is an important artifact in convolutional neural networks to reduce the number of parameters: local receptive field.

And then, after the images are subjected to feature extraction, convolution, pooling and full connection, a convolution neural network model with accurate parameters can be obtained through training.

In summary, the convolutional neural network is mainly composed of two parts, one part is feature extraction (convolution, activation function, pooling), and the other part is classification identification (full-link layer). In essence, it is an input-to-output mapping that is able to learn a large number of input-to-output mappings without any precise mathematical expression between the inputs and outputs, and the network has the ability to map between inputs and outputs as long as the convolutional network is trained with known patterns.

One feature of CNN is that it is lightweight (the closer to the input weights, the smaller the closer to the output weights), and exhibits a form of a transverse inverted triangle, which well avoids the situation where the gradient loss is too fast when back-propagating in the BP neural network.

The embodiment of the invention also provides an intelligent road condition identification method, which has the following specific implementation mode:

1) Acquiring the instantaneous speed v1 of the vehicle by using an OBD module of the vehicle; acquiring a depth image a of a road condition in front of a vehicle and a depth distance da corresponding to a certain point x by using a Kinect camera;

2) Acquiring a depth image b of the road condition in front of the vehicle and a depth distance db corresponding to a certain point x after a short time period t (t is 500 ms);

3) Using the formula

Obtaining the average speed vt of the vehicle in a time period t;

4) If vt is larger than or equal to v1, marking the depth image a as free, and UP accelerating (marked by 'U'); if vt < v1, then mark depth image a as obstructed, DOWN decelerates (marked with "D");

5) Forming a training data set by using the information obtained in the step 1) and the step 4), respectively taking 1/3 and 1/4 of the total pixels of the collected pictures as convolution kernels by using a convolution neural network, carrying out convolution neural network training, storing a training result and continuously optimizing a neural network model;

6) And applying the continuously optimized neural network parameter model to an automatic driving acceleration and deceleration program through a driving computer interface, and adaptively controlling the acceleration and deceleration of the vehicle within a certain range in real-time depth image recognition so as to achieve better safety.

The method realizes intelligent road condition identification based on a convolutional neural network of a depth image, and the architecture diagram of the method is shown in figure 2.

The embodiment of the present invention further provides an intelligent road condition identification device, including: at least one memory and at least one processor;

the at least one memory to store a machine readable program;

the at least one processor is configured to invoke the machine-readable program to execute the intelligent road condition identification method according to any of the embodiments of the present invention.

An embodiment of the present invention further provides a computer readable medium, where a computer instruction is stored, and when the computer instruction is executed by a processor, the processor is enabled to execute the method for identifying an intelligent road condition in any of the embodiments of the present invention. Specifically, a system or an apparatus equipped with a storage medium on which software program codes that realize the functions of any of the above-described embodiments are stored may be provided, and a computer (or a CPU or MPU) of the system or the apparatus is caused to read out and execute the program codes stored in the storage medium.

In this case, the program code itself read from the storage medium can realize the functions of any of the above-described embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

Examples of the storage medium for supplying the program code include a flexible disk, hard disk, magneto-optical disk, optical disk (e.g., CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD + RW), magnetic tape, nonvolatile memory card, and ROM. Alternatively, the program code may be downloaded from a server computer via a communications network.

Further, it should be clear that the functions of any one of the above-described embodiments may be implemented not only by executing the program code read out by the computer, but also by causing an operating system or the like operating on the computer to perform a part or all of the actual operations based on instructions of the program code.

Further, it is to be understood that the program code read out from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion unit connected to the computer, and then a CPU or the like mounted on the expansion board or the expansion unit is caused to perform part or all of the actual operations based on instructions of the program code, thereby realizing the functions of any of the embodiments described above.

While the invention has been shown and described in detail in the drawings and in the preferred embodiments, it is not intended to limit the invention to the embodiments disclosed, and it will be apparent to those skilled in the art that various combinations of the code auditing means in the various embodiments described above may be used to obtain further embodiments of the invention, which are also within the scope of the invention.

Claims

1. An intelligent road condition identification method is characterized in that the instantaneous speed of a vehicle is collected, image information and depth distance information of the road condition are collected, a convolutional neural network training and verification set is constructed by utilizing the collected information, a convolutional neural network model is trained by using the training and verification set, parameters are continuously optimized, the continuously optimized neural network parameter model is applied to acceleration and deceleration of automatic driving, and the acceleration and the deceleration of the vehicle are adaptively controlled in real-time depth image identification;

the method is realized in the following specific way:

2) Acquiring a depth image b of the road condition in front of the vehicle and a depth distance db corresponding to a certain point x after a time period t;

3) Using the formula vt =

Obtaining the average speed vt of the vehicle in the time period t;

6) And applying the continuously optimized neural network parameter model to an acceleration and deceleration program of automatic driving, and adaptively controlling the acceleration and the deceleration of the vehicle in real-time depth image recognition.

2. The intelligent road condition identification method as claimed in claim 1, wherein a Kinect depth camera is used to capture the image information and depth distance information of the collected road condition.

3. An intelligent road condition identification method as claimed in claim 1, wherein the vehicle instantaneous speed is collected by an OBD module of the vehicle.

4. The intelligent road condition identification method according to claim 1, 2 or 3, characterized in that the images are identified by comparing the change of the same image depth distance in a certain time period with the starting point vehicle speed, and then the convolutional neural network is trained by the identification image set to obtain a training model capable of identifying road condition information of different scenes.

5. The method as claimed in claim 1, wherein the time period t is 500ms.

6. The intelligent road condition identification method according to claim 1, wherein the convolutional neural network training is performed by taking 1/3 and 1/4 of the total pixels of the acquired images as convolutional kernels respectively by using the convolutional neural network.

7. The intelligent road condition identification method as claimed in claim 1, wherein the neural network parameter model is applied to an acceleration and deceleration program of vehicle automatic driving through a driving computer interface.

8. An intelligence road conditions recognition device which characterized in that includes: at least one memory and at least one processor;

the at least one memory to store a machine readable program;

the at least one processor configured to invoke the machine readable program to perform the method of any of claims 1 to 7.

9. A computer readable medium having stored thereon computer instructions which, when executed by a processor, cause the processor to perform the method of any of claims 1 to 7.