CN113537003A - A method and device for visual detection of external environment of a vehicle for assisted driving - Google Patents

Abstract

The present invention provides a method and device for visual detection of the outside environment of the vehicle for assisted driving. An image describing the state of the vehicle is collected by an image acquisition device, and by normalizing the discrete image, after calculating the energy gradient of the normalized discrete image The input neural network model can overcome the information weakening caused by the gray-scale weighted average, and can better retain the characteristic information of the local channel of the color image. Design the convolution window of multiple hidden layers, so that the omnidirectional convolution window of the neural network model can be combined, so as to detect the characteristics of various forms in all directions, and design a nonlinear function as the excitation function of the neural network model to improve the detection of the neural network model. ability, output control variables, and realize intelligent auxiliary control of multiple controlled equipment. The invention can reduce the number of external sensors required by the auxiliary driving system, reduce the load of the auxiliary driving system, and reduce the complexity of the system.

Description

A method and device for visual detection of external environment of a vehicle for assisted driving

technical field

The invention belongs to the technical field of the field of automatic driving, and in particular relates to a method and a device for visual detection of an external environment of a vehicle for assisted driving.

Background technique

The autonomous driving system is a hot field of research in the industry in recent years. It relies on the cooperation of information acquisition and analysis systems such as machine learning, image processing, radar and positioning systems, so that the system can automatically and safely operate in an environment without human active operation. Control motor vehicles.

In the process of autonomous driving, a variety of sensors are required to sense the surrounding environment, and then the control system of the driving vehicle controls the activation or deactivation of the device to achieve control. And often the more sensors, the more complicated the control system of the driving vehicle. After the control system obtains the sensing parameters of the sensor, it often takes more time to find the corresponding controlled device. In addition, the variety of sensors makes the process of the controller processing sensor parameters more complicated, which often fails to achieve the purpose of real-time assistance for automatic driving.

In the prior art, there are also assisted driving strategies that are detected through a neural network model and output to control the start or shutdown of the controlled device. Due to the diversity of sensing parameters input by the sensor, the internal structure of the neural network model is relatively complex, and the recognition effect of each layer structure in the neural network model is not good when recognizing the characteristics of the sensing parameters.

SUMMARY OF THE INVENTION

The present invention provides a method and device for visual detection of the external environment of the vehicle for assisting driving, so as to improve the accuracy and efficiency of the visual detection method of the external environment of the vehicle. The specific technical solution is as follows.

A first aspect. The present invention provides a method for visual detection of an external environment for assisted driving, comprising:

Obtain a collection of raw data describing the state of the vehicle;

Wherein, the set includes a plurality of elements, each element is a discrete image, the discrete images form a sequence according to time, each discrete image is composed of three channels, and each channel is represented as a two-dimensional matrix;

Normalizing each discrete image in the set, so that the value of each discrete image is mapped within a fixed range, to obtain a normalized discrete image;

Calculate the energy gradient of each normalized discrete image according to the gray energy distribution of the image;

The energy gradient of each discrete image is used as the input of the trained neural network model, so that the neural network model performs feature extraction and detection on the energy gradient of each discrete image, and obtains multiple or Multiple sets of control variables;

Wherein, the convolution window structure and convolution direction of each hidden layer of the neural network model are different, the excitation function of the convolution layer of the neural network model is a linear function, and the control variable corresponds to the controlled device;

According to the magnitude relationship between the value of the control variable and the threshold, the controlled device corresponding to the control variable is controlled to perform the corresponding operation.

Optionally, normalizing each discrete image in the set, so that the value of each discrete image is mapped to a fixed range including:

Normalize the two-dimensional matrix of each channel in each discrete image using the first normalization formula, so that the values of each discrete image are mapped to within a fixed range;

Among them, the first normalization formula is:

表示规范化后的每个通道的二维矩阵，

mean表示矩阵I_t(·,·,c)的均值，std为标准差，c＝{red,green,blue}表示离散图像中的三原色通道，i、j表示离散图像的二维空间坐标。where I _t (·,·,c) represents a two-dimensional matrix for each channel,

a two-dimensional matrix representing each channel after normalization,

mean represents the mean of the matrix It (·,·,c), std is the standard deviation, c={red, green, blue} represents the three primary color channels in the discrete image, i, _j represent the two-dimensional spatial coordinates of the discrete image.

Optionally, calculating the energy gradient of each normalized discrete image according to the grayscale energy distribution of the image includes:

Set the rectangular window centered on the two-dimensional spatial coordinates of the two-dimensional matrix in the normalized discrete image, sort the pixels in the area where the rectangular window is located according to the pixel value, and determine the median pixel value;

For each channel of each normalized discrete image, calculate the deviation between each pixel value of the channel and the median pixel value, and determine the pixel value with the largest difference;

The pixel value with the largest difference is determined as the energy gradient at the spatial position of the discrete image.

Wherein, the energy gradient is expressed as:

表示能量梯度，I(i,j,c)表示规范后的离散图像，c＝{red,green,blue}表示图像中的三原色通道，i、j表示图像的二维空间坐标，median(w_ij(u,v))表示中位像素值，w_ij(u,v)表示二维矩阵中以(i,j)为中心、(u,v)为高和宽的一个矩形窗口。in,

Represents the energy gradient, I(i, j, c) represents the normalized discrete image, c={red, green, blue} represents the three primary color channels in the image, i, j represent the two-dimensional spatial coordinates of the image, median(w _ij (u, v)) represents the median pixel value, and w _ij (u, v) represents a rectangular window with (i, j) as the center and (u, v) as the height and width in a two-dimensional matrix.

Optionally, the trained neural network model is obtained through the following steps:

Set the convolution window size of each hidden layer so that the convolution window structure and convolution direction of each hidden layer are different;

The neurons in the input layer, the convolution layer and the output layer are sequentially constructed to construct the initial neural network model;

Get the training dataset;

Wherein, the training data set includes a plurality of samples, and one sample includes the discrete image energy gradient and the manually marked label of the controlled device corresponding to the gradient energy;

Input each sample into the initial neural network, take the control variable corresponding to the label of the controlled device corresponding to the gradient energy as the learning target, and iteratively train the initial neural network model until the learning target is reached or the number of iterations is reached;

The initial neural network that has reached the learning goal or reached the number of iterations is used as the trained neural network model.

Wherein, the excitation function is:

Among them, x represents the input, α represents the function to generate a discontinuous breakpoint at the point x=0, and R represents the set of real numbers.

Wherein, the fixed range is [0, 1], and the value of the control variable is between [0, 1].

Optionally, the controlling the controlled device corresponding to the control variable to perform corresponding operations according to the value of the control variable includes:

When the value of the control variable is greater than the threshold, the controlled device corresponding to the control variable is controlled to be turned on;

When the value of the control variable is not greater than the threshold, the controlled device corresponding to the control variable is controlled to be turned off.

Optionally, the controlled equipment includes: lighting equipment, air conditioning equipment, and braking equipment.

In a second aspect, the present invention provides a driving environment evaluation device based on a dual-mode neural network model, comprising:

The acquisition module is used to acquire the raw data composition set describing the state of the vehicle;

Wherein, the set includes a plurality of elements, each element is a discrete image, the discrete images form a sequence according to time, each discrete image is composed of three channels, and each channel is represented as a two-dimensional matrix;

a normalization module, configured to normalize each discrete image in the set, so that the value of each discrete image is mapped to a fixed range to obtain a normalized discrete image;

The calculation module is used to calculate the energy gradient of each normalized discrete image according to the gray energy distribution of the image;

The detection module is used to use the energy gradient of each discrete image as the input of the neural network model after training, so that the neural network model performs feature extraction and detection on the energy gradient of each discrete image to obtain the neural network model. Multiple or multiple sets of control variables to be output;

Wherein, the convolution window structure and convolution direction of each hidden layer of the neural network model are different, the excitation function of the convolution layer of the neural network model is a linear function, and the control variable corresponds to the controlled device;

The control module is configured to control the controlled device corresponding to the control variable to perform the corresponding operation according to the magnitude relationship between the value of the control variable and the threshold.

The innovative points of the embodiments of the present invention include:

1. The present invention provides a visual detection method for the external environment of the vehicle for assisted driving, which can effectively control the lighting, air conditioning, braking and other on-board systems only by collecting images describing the state of the vehicle through an image acquisition device. , which greatly reduces the number of external sensors required by the assisted driving system, reduces the load of the assisted driving system, and reduces the system complexity.

2. The present invention provides a visual detection method for the external environment of the vehicle for assisted driving, which can improve the reliability of the neural network model by normalizing discrete images.

3. The present invention provides a visual detection method for the external environment of the vehicle for assisted driving. By calculating the energy gradient of the discrete image as the input of the neural network model, the information weakening caused by the gray-scale weighted average can be overcome, and the information can be better. The feature information of the local channel of the color image is preserved, so that the neural network model can achieve better detection results.

4. The present invention provides a visual detection method for the outside environment of the vehicle for assisted driving. Through the convolution windows of multiple hidden layers, the omnidirectional convolution windows of the neural network model are combined, so as to detect the features of various shapes in all directions. , and design a nonlinear function as the excitation function of the neural network model, improve the detection ability of the neural network model, output control variables, and realize intelligent auxiliary control of multiple controlled equipment.

Description of drawings

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that are required in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 is a schematic flowchart of a visual detection method for an external environment of a vehicle for assisted driving according to an embodiment of the present invention;

Fig. 2 is the structural representation of the tracking model constructed;

Figure 3a is a map of convolution layer convolution;

Figure 3b is a schematic diagram of the connection mode of the convolution layer;

FIG. 4 is a schematic structural diagram of an external environment visual detection device for assisting driving according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "comprising" and "having" and any modifications thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the steps or units listed, but optionally also includes steps or units not listed, or optionally also includes For other steps or units inherent to these processes, methods, products or devices.

FIG. 1 is a schematic flowchart of a visual detection method for an external environment of a vehicle for assisted driving according to an embodiment of the present invention. The method is applied to self-driving vehicles. The method specifically includes the following steps.

S1, obtain a set of raw data describing the state of the vehicle;

Among them, the set includes multiple elements, each element is a discrete image, the discrete images form a sequence according to time, each discrete image is composed of three channels, and each channel is represented as a two-dimensional matrix;

The present invention can collect raw data from an image acquisition device installed with the vehicle, and the raw data is a discrete time-sampled image sequence.

Assuming that the original image data sequence collected by the image sensor forms a set V, each element of which is an image I, and the image consists of 3 channels (RGB), that is, c={red, green, blue}, defining the image I =I(i,j,c), where i, j are two-dimensional spatial coordinates, the set of image sequences V={I ₁ , I ₂ ,...,I _t ,...}, t represents the time sequence of image acquisition. Each channel I( _· ,·,red), I(·,·,green), I(·,·,blue) of the image It is a two-dimensional matrix.

S2, normalize each discrete image in the set, so that the value of each discrete image is mapped to a fixed range, and a normalized discrete image is obtained;

S3, according to the gray energy distribution of the image, calculate the energy gradient of each normalized discrete image;

S4, take the energy gradient of each discrete image as the input of the neural network model after training, so that the neural network model can perform feature extraction and detection on the energy gradient of each discrete image, and obtain multiple or multiple groups of outputs of the neural network model control variable;

Among them, the convolution window structure and convolution direction of each hidden layer of the neural network model are different, the excitation function of the convolution layer of the neural network model is a linear function, and the control variable corresponds to the controlled device; the fixed range is [0, 1], the control The value of the variable is between [0,1].

The nonlinear unit of the neural network model is called the excitation function σ, which is used to give the network the ability to classify nonlinear data sets.

Among them, the excitation function is:

Among them, x represents the input, α represents the function to generate a discontinuous breakpoint at the point x=0, and R represents the set of real numbers.

S5, according to the magnitude relationship between the value of the control variable and the threshold, the controlled device corresponding to the control variable is controlled to perform a corresponding operation.

Among them, the controlled equipment includes: lighting equipment, air conditioning equipment and braking equipment.

The present invention provides a method and device for visual detection of the outside environment of the vehicle for assisted driving. An image describing the state of the vehicle is collected by an image acquisition device, and by normalizing the discrete image, after calculating the energy gradient of the normalized discrete image The input neural network model can overcome the information weakening caused by the gray-scale weighted average, and can better retain the characteristic information of the local channel of the color image. Design the convolution window of multiple hidden layers, so that the omnidirectional convolution window of the neural network model can be combined, so as to detect the characteristics of various forms in all directions, and design a nonlinear function as the excitation function of the neural network model to improve the detection of the neural network model. ability, output control variables, and realize intelligent auxiliary control of multiple controlled equipment. The invention can reduce the number of external sensors required by the auxiliary driving system, reduce the load of the auxiliary driving system, and reduce the complexity of the system.

As an optional implementation manner of the present invention, each discrete image in the set is normalized, so that the value of each discrete image is mapped to a fixed range including:

Normalize the two-dimensional matrix of each channel in each discrete image using the first normalization formula, so that the values of each discrete image are mapped to within a fixed range;

Among them, the first normalization formula is:

表示规范化后的每个通道的二维矩阵，

mean表示矩阵I_t(·,·,c)的均值，std为标准差，c＝{red,green,blue}表示离散图像中的三原色通道，i、j表示离散图像的二维空间坐标。where I _t (·,·,c) represents a two-dimensional matrix for each channel,

a two-dimensional matrix representing each channel after normalization,

mean represents the mean of the matrix It (·,·,c), std is the standard deviation, c={red, green, blue} represents the three primary color channels in the discrete image, i, _j represent the two-dimensional spatial coordinates of the discrete image.

进行下一步处理。For each channel I_t(·,·,c) of the image acquired at time t, each element is I_t(i,j,c), the width is w, the height is h, and the first normalization is used separately for each channel The formula is processed, the normalized matrix

Proceed to the next step.

As an optional embodiment of the present invention, calculating the energy gradient of each normalized discrete image according to the grayscale energy distribution of the image includes:

Step 1: Set the rectangular window centered on the two-dimensional spatial coordinates of the two-dimensional matrix in the normalized discrete image, sort the pixels in the area where the rectangular window is located according to the pixel value, and determine the median pixel value;

Step 2: For each channel of each normalized discrete image, calculate the deviation between each pixel value of the channel and the median pixel value, and determine the pixel value with the largest difference;

Step 3: Determine the pixel value with the largest difference as the energy gradient at the spatial position of the discrete image.

In the present invention, the energy gradient of an image is defined as a measure of the energy change of the image. Specifically, the definition is as follows:

Definition w _ij (u, v) represents a rectangular window in a two-dimensional matrix (image) with (i, j) as the center and (u, v) as the height and width. median(w _ij (u,v)) represents the median value of the window, that is, the value in the middle after sorting all the pixels in the window by value size. Then the energy gradient can be expressed as:

表示能量梯度，I(i,j,c)表示规范后的离散图像，c＝{red,green,blue}表示图像中的三原色通道，i、j表示图像的二维空间坐标，median(w_ij(u,v))表示中位像素值对于一幅三通道RGB图像，该图像的能量梯度是一个二维矩阵，首先对每个通道计算像素与其周围像素中位值的偏差，再取各通道偏差最大的那个值作为图像该空间位置上的能量梯度。in,

Represents the energy gradient, I(i, j, c) represents the normalized discrete image, c={red, green, blue} represents the three primary color channels in the image, i, j represent the two-dimensional spatial coordinates of the image, median(w _ij (u, v)) represents the median pixel value. For a three-channel RGB image, the energy gradient of the image is a two-dimensional matrix. First, calculate the deviation of the pixel from the median value of the surrounding pixels for each channel, and then take each channel. The value with the largest deviation is used as the energy gradient at the spatial position of the image.

The invention uses the energy gradient of the image to replace the original image pixels to express the environmental characteristics, on the one hand, the dimension of the data is reduced, on the other hand, it can overcome the information weakening caused by the gray-scale weighted average, and can better retain the characteristics of the local channel of the color image. information for better environmental recognition.

As an optional embodiment of the present invention, the trained neural network model is obtained through the following steps:

Step 1: Set the convolution window size of each hidden layer so that the convolution window structure and convolution direction of each hidden layer are different;

Step 2: Arrange the neurons in the input layer, the convolution layer and the output layer in order to construct the initial neural network model;

Referring to Figure 2, the basic model of the neural network model consists of an input layer, an output layer and a hidden layer. Each layer contains several nodes, called neurons, and the connections between neurons and neurons form a neural network. Determined by the excitation function, weights, and how neurons are connected.

In Figure 2, the leftmost three nodes X ₁ , X ₂ , 1 are the input layer nodes, the right node y is the output layer node, h ₁ , h ₂ , h ₃ are the hidden layer nodes, σ represents the excitation function, the role of It is to make the neural network have nonlinear classification ability. The relationship between the output and input of a neural network is defined by the following formula:

a ₁ =w _1-11 x ₁ +w _1-21 x ₂ +b _1-1

a ₂ =w _1-12 x ₁ +w _1-22 x ₂ +b _1-2

a ₃ =w _1-13 x ₁ +w _1-23 x ₂ +b _1-3

y=σ(w _2-1 σ(a ₁ )+w _2-2 σ(a ₂ )+w _2-3 σ(a ₃ ))

Among them, a ₁ represents the relationship between the input and output of the h ₁ node, a ₂ represents the input and output relationship of the h ₂ node, a ₃ represents the input and output relationship of the h ₃ node, and w with a subscript represents the channel weight between the nodes. b with subscripts indicates the parameters between node 1 and the excitation function. σ in parentheses indicates the excitation function of the channel indicated in the parentheses.

Figure 2 shows a fully connected neural network (the most complete form), that is, each node in the hidden layer is connected to any node in the previous layer (regardless of the excitation function). In practical applications, the hidden layer There can be more than one, and the connection relationship between the number of nodes in each layer and the previous layer can be freely defined under the premise that the implementation allows, that is, the connection is merged or deleted on the basis of full connection.

的神经网络N，并且定义隐藏层如下：Build an input layer for the image energy gradient

The neural network N, and the hidden layer is defined as follows:

(11) The first hidden layer H1 of the neural network model N is defined as follows.

通过卷积窗

后的结果。H1 is based on the input layer data

through the convolution window

the result after.

)对应位置的3x3个节点有连接；将这3x3个连接的权重按行-列的顺序分别定义为

并且H1的每一个节点v与输入连接的3x3个点，其在对应位置的权重均相同。b带下角标表示第几个隐藏层的参数，从下角标为0的第一个隐藏层开始，pq表示卷积核的尺寸。Figure 3a and Figure 3b show the connection when the window size is 3x3, that is, 1≤p, q≤3, and each node is only connected to its upper layer (ie, the input layer data

) 3x3 nodes at the corresponding positions are connected; the weights of these 3x3 connections are defined in row-column order as

And each node v of H1 is connected to the 3x3 points of the input, and its weights at the corresponding positions are the same. b with subscripts indicates the parameters of the first hidden layer, starting from the first hidden layer marked with 0 in the lower corner, and pq indicates the size of the convolution kernel.

In particular, the present invention defines the window size of the H1 layer as 5×5, that is, 1≤p, q≤5.

(12) The second hidden layer H2 of the neural network model N is defined as follows.

后的结果。H2 is based on the output of the H1 layer, through the convolution window

the result after.

Similar to the structure of the H1 layer, the weight window defining the H2 layer is also a rectangular window with a size of 3x7, that is, 1≤p≤3, 1≤q≤7.

(13) The third hidden layer H3 of the neural network model is defined as follows.

后的结果。H3 is based on the output of the H2 layer, through the convolution window

the result after.

Similar to the structure of the H1 and H2 layers, the weight window defining the H3 layer is also a rectangular window with a size of 7x3, that is, 1≤p≤7, 1≤q≤3.

(14) The fourth hidden layer H4 of the network N is defined as follows.

后的结果。The H4 layer is based on the output of the H3 layer, through the convolution window

the result after.

是一个对称矩阵，并且1≤p,q≤7。Convolutional window of H4 layer

is a symmetric matrix, and 1≤p, q≤7.

(15) The fifth hidden layer H5 of the network N is defined as follows.

and when p>q,

后的结果。The H5 layer is based on the output of the H4 layer, through the convolution window

the result after.

是一个上(下)三角矩阵，并且1≤p,q≤7。Convolutional window of H5 layer

is an upper (lower) triangular matrix, and 1≤p, q≤7.

(16) The sixth hidden layer H6 of the network N is defined as follows.

and when p<q,

后的结果。The H6 layer is based on the output of the H5 layer, through the convolution window

the result after.

是一个与卷积窗

相对的下(上)三角矩阵。并且1≤p,q≤7。Convolutional window of H6 layer

is a convolutional window with

Relative lower (upper) triangular matrix. And 1≤p, q≤7.

The H2-H6 layers combine convolution windows of different directions and structures to detect features, so as to identify features in various directions and shapes. This combination of multi-layer omnidirectional detection windows is another important feature of the present invention.

(17) The seventh hidden layer H7 of network N is defined as follows.

H7 is the maximum value of the output of the hidden layer H6 within the window pxq, and 1≤p, q≤4.

(18) The eighth hidden layer H8 of the network N is defined as follows.

后的结果。H8 is based on the output of the H7 layer, through the convolution window

the result after.

The window size of the H8 layer is 5x5, that is, 1≤p, q≤5.

(19) The ninth hidden layer H9 of the network N is defined as follows.

后的结果。The H9 layer is based on the output of the H8 layer, through the convolution window

the result after.

All elements of the convolutional window _w9 of the H9 layer are equal, and 1≤p, q≤5.

(110) The tenth hidden layer H10 of the network N is in the form of a fully connected layer. There is a connection between each node of H10 and each node of H9, and the connection weights are independent.

(111) After the tenth hidden layer H10 of the network N, the output layer Y is connected in a fully connected form.

The output layer Y represents several or several groups of control variables, and the value is between [0, 1], which means that a certain (or a certain group) switch is turned on or off; when a certain component of Y is >0.66, it can be The switch is considered to be on, otherwise the switch is considered to be off. The number of control variables is related to the actual equipment to be controlled. According to the different uses of the equipment in the driving system, it is divided into "lighting", "air conditioning", "braking", etc., and each category is based on the actual number of equipment. The difference corresponds to one or more output components.

The dimensions of the output Y are independent and limited, but the dimensions of the output Y are not limited by the actual device category or number. At the same time, the models described in the present invention are not limited to the type or number of actual devices.

Step 3: Obtain the training data set;

Wherein, the training data set includes a plurality of samples, and one sample includes the discrete image energy gradient and the manually marked label of the controlled device corresponding to the gradient energy;

Step 4: Input each sample into the initial neural network, take the control variable corresponding to the label of the controlled device corresponding to the gradient energy as the learning target, and iteratively train the initial neural network model until the learning target is reached or the number of iterations is reached;

Step 5: The initial neural network that has reached the learning target or reached the number of iterations is used as the neural network model after training.

并人工标注与该梯度图对应的各控制开关所处的位置(开/关：1/0)对初始神经网络模型进行训练。It can be understood that when training the neural network, several sets of video data are selected, and the image energy gradient is established by establishing the image energy gradient.

And manually mark the position of each control switch (on/off: 1/0) corresponding to the gradient map to train the initial neural network model.

The neural network model in the present invention performs special processing on the input image data, and designs a unique image energy gradient feature, a neural network model with a multi-layer omnidirectional detection window, and a nonlinear processing unit of the model. Compared with the classical algorithm, the neural network model of the present invention effectively improves the correct rate of switch state judgment.

As an optional embodiment of the present invention, according to the value of the control variable, controlling the controlled device corresponding to the control variable to perform the corresponding operation includes:

Step 1: when the value of the control variable is greater than the threshold, the controlled device corresponding to the control variable is controlled to be turned on;

Step 2: When the value of the control variable is not greater than the threshold, the controlled device corresponding to the control variable is controlled to be turned off.

The effect of the present invention is verified below with actual data, see Table 1.

Table 1 Effect comparison table

As can be seen from Table 1, the first line is the judgment accuracy obtained by using the Alex network, ReLU nonlinear unit and weighted grayscale image features (the values of the three channels of R, G, and B are averaged); the second line is the use of this Invented the multi-layer omnidirectional detection window neural network model, the non-linear unit provided by the present invention and the judgment accuracy obtained by the weighted gray image features; the third row is the multi-layer omnidirectional detection window neural network model of the present invention, provided by the present invention. The accuracy obtained by the novel nonlinear unit and image energy gradient feature of . It can be seen that the driving environment evaluation method based on the dual-mode neural network model proposed by the present invention effectively improves the correct rate of switch state judgment.

As shown in FIG. 4 , a driving environment evaluation device based on a dual-mode neural network model provided by the present invention includes:

an acquisition module 41, configured to acquire a set of raw data describing the state of the vehicle;

Among them, the set includes multiple elements, each element is a discrete image, the discrete image forms a sequence according to time, each discrete image is composed of three channels, and each channel is represented as a two-dimensional matrix;

The normalization module 42 is used to normalize each discrete image in the set, so that the value of each discrete image is mapped to a fixed range to obtain a normalized discrete image;

The calculation module 43 is used to calculate the energy gradient of each normalized discrete image according to the grayscale energy distribution of the image;

The detection module 44 is used to use the energy gradient of each discrete image as the input of the neural network model after training, so that the neural network model performs feature extraction and detection on the energy gradient of each discrete image, and obtains the output of the neural network model. one or more sets of control variables;

Among them, the convolution window structure and convolution direction of each hidden layer of the neural network model are different, the excitation function of the convolution layer of the neural network model is a linear function, and the control variable corresponds to the controlled device;

The control module 45 is configured to control the controlled device corresponding to the control variable to perform corresponding operations according to the magnitude relationship between the value of the control variable and the threshold.

The above apparatus embodiments correspond to the method embodiments, and have the same technical effects as the method embodiments. For specific descriptions, refer to the method embodiments. The apparatus embodiment is obtained based on the method embodiment, and the specific description can refer to the method embodiment section, which will not be repeated here.

Those of ordinary skill in the art can understand that the accompanying drawing is only a schematic diagram of an embodiment, and the modules or processes in the accompanying drawing are not necessarily necessary to implement the present invention.

Those skilled in the art may understand that: the modules in the apparatus in the embodiment may be distributed in the apparatus in the embodiment according to the description of the embodiment, and may also be located in one or more apparatuses different from this embodiment with corresponding changes. The modules in the foregoing embodiments may be combined into one module, or may be further split into multiple sub-modules.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A method for visual detection of an environment external to a vehicle for driver assistance, the method comprising:

acquiring a raw data composition set describing a vehicle state;

wherein the set comprises a plurality of elements, each element being a discrete image, the discrete images forming a sequence over time, each discrete image consisting of three channels, each channel being represented as a two-dimensional matrix;

normalizing each discrete image in the set so as to map the value of each discrete image in a fixed range to obtain a normalized discrete image;

calculating the energy gradient of each normalized discrete image according to the gray level energy distribution of the image;

taking the energy gradient of each discrete image as the input of a trained neural network model, so that the neural network model performs feature extraction and detection on the energy gradient of each discrete image to obtain a plurality of or a plurality of groups of control variables output by the neural network model;

the convolution window structures and convolution directions of all hidden layers of the neural network model are different, the excitation function of the convolution layer of the neural network model is a linear function, and the control variable corresponds to controlled equipment;

and controlling the controlled equipment corresponding to the control variable to execute corresponding operation according to the magnitude relation between the numerical value of the control variable and the threshold value.

2. A method for visual inspection of an off-board environment according to claim 1, wherein said normalizing each discrete image in said set such that the value of each discrete image maps to within a fixed range comprises:

normalizing the two-dimensional matrix of each channel in each discrete image using a first normalization formula such that the values of each discrete image are mapped to within a fixed range;

wherein the first normalized formula is:

wherein, I_t(-, c) represents a two-dimensional matrix for each channel,

a two-dimensional matrix representing each channel after normalization,

mean represents the matrix I_tThe mean value of ·, c), std is the standard deviation, c ═ { red, green, blue } represents the three primary color channels in the discrete image, and i, j represent the two-dimensional spatial coordinates of the discrete image.

3. The vehicle exterior environment visual detection method according to claim 1, wherein said calculating an energy gradient of each normalized discrete image from a gray scale energy distribution of the image comprises:

setting a rectangular window taking the two-dimensional space coordinate of the two-dimensional matrix as the center in the normalized discrete image, sequencing pixel points of the area where the rectangular window is located according to pixel values, and determining a median pixel value;

calculating the deviation of each pixel value of each channel and the median pixel value aiming at each channel of each normalized discrete image, and determining the pixel value with the maximum difference;

and determining the pixel value with the largest difference as the energy gradient at the spatial position of the discrete image.

4. A method for visual inspection of an off-board environment according to claim 3, characterized in that said energy gradient is expressed as:

wherein,

representing the energy gradient, I (I, j, c) representing the normalized discrete image, c ═ { red, green, blue } representing the three primary color channels in the image, I, j representing the two-dimensional spatial coordinates of the image, mean (w)_ij(u, v)) represents the median pixel value, W_ij(u, v) represents a rectangular window centered at (i, j) and high and wide in the two-dimensional matrix.

5. The vehicle exterior environment visual inspection method according to claim 1, wherein the trained neural network model is obtained by:

setting the convolution window size of each hidden layer to ensure that the convolution window structure and the convolution direction of each hidden layer are different;

sequentially arranging the neurons in the input layer, the convolutional layer and the output layer so as to construct an initial neural network model;

acquiring a training data set;

the training data set comprises a plurality of samples, and one sample comprises a discrete image energy gradient and a manually marked label of controlled equipment corresponding to the gradient energy;

inputting each sample into the initial neural network, taking a control variable corresponding to a label of controlled equipment corresponding to gradient energy as a learning target, and iteratively training the initial neural network model until the learning target is reached or the iteration times are reached;

and taking the initial neural network reaching the learning target or reaching the iteration times as a trained neural network model.

6. A method for visual inspection of an extra-vehicular environment according to claim 1, characterized in that said excitation function is:

where x denotes the input, α denotes the function is made to produce a discontinuous break at the point where x is 0, and R denotes the real number set.

7. The visual detection method for the environment outside the vehicle according to claim 1, wherein the fixed range is [0,1], and the value of the control variable is located between [0,1 ].

8. The visual detection method for the exterior environment of the vehicle according to claim 7, wherein the controlling the controlled device corresponding to the controlled variable to perform corresponding operations according to the numerical value of the controlled variable comprises:

when the value of the control variable is larger than the threshold value, controlling the controlled equipment corresponding to the control variable to be started;

and when the value of the control variable is not greater than the threshold value, controlling the controlled equipment corresponding to the control variable to be closed.

9. The vehicle exterior environment visual inspection method according to claim 1, wherein the controlled preparation includes: lighting equipment, air conditioning equipment, and braking equipment.

10. A driving environment evaluation apparatus based on a dual-mode neural network model, the apparatus comprising:

the acquisition module is used for acquiring a raw data composition set describing the vehicle state;

wherein the set comprises a plurality of elements, each element being a discrete image, the discrete images forming a sequence over time, each discrete image consisting of three channels, each channel being represented as a two-dimensional matrix;

the normalization module is used for normalizing each discrete image in the set so as to map the value of each discrete image in a fixed range and obtain the normalized discrete image;

the computing module is used for computing the energy gradient of each normalized discrete image according to the gray level energy distribution of the image;

the detection module is used for taking the energy gradient of each discrete image as the input of the trained neural network model so that the neural network model performs feature extraction and detection on the energy gradient of each discrete image to obtain a plurality of or a plurality of groups of control variables output by the neural network model;

the convolution window structures and convolution directions of all hidden layers of the neural network model are different, the excitation function of the convolution layer of the neural network model is a linear function, and the control variable corresponds to controlled equipment;

and the control module is used for controlling the controlled equipment corresponding to the control variable to execute corresponding operation according to the magnitude relation between the numerical value of the control variable and the threshold value.

Images