CN113537003A - Method and device for visually detecting external environment of vehicle for assisting driving - Google Patents
Method and device for visually detecting external environment of vehicle for assisting driving Download PDFInfo
- Publication number
- CN113537003A CN113537003A CN202110747205.6A CN202110747205A CN113537003A CN 113537003 A CN113537003 A CN 113537003A CN 202110747205 A CN202110747205 A CN 202110747205A CN 113537003 A CN113537003 A CN 113537003A
- Authority
- CN
- China
- Prior art keywords
- neural network
- discrete image
- image
- network model
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000003062 neural network model Methods 0.000 claims abstract description 74
- 238000001514 detection method Methods 0.000 claims abstract description 29
- 230000005284 excitation Effects 0.000 claims abstract description 18
- 238000010606 normalization Methods 0.000 claims abstract description 11
- 239000011159 matrix material Substances 0.000 claims description 30
- 230000006870 function Effects 0.000 claims description 23
- 238000013528 artificial neural network Methods 0.000 claims description 11
- 238000012549 training Methods 0.000 claims description 10
- 210000002569 neuron Anatomy 0.000 claims description 7
- 238000000605 extraction Methods 0.000 claims description 6
- 238000012886 linear function Methods 0.000 claims description 6
- 230000000007 visual effect Effects 0.000 claims description 6
- 238000004378 air conditioning Methods 0.000 claims description 3
- 238000002360 preparation method Methods 0.000 claims description 3
- 238000012163 sequencing technique Methods 0.000 claims description 3
- 238000011156 evaluation Methods 0.000 claims description 2
- 238000011179 visual inspection Methods 0.000 claims 5
- 238000012545 processing Methods 0.000 abstract description 9
- 230000003313 weakening effect Effects 0.000 abstract description 4
- 230000000694 effects Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000007613 environmental effect Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 241000282414 Homo sapiens Species 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
According to the method and the device for detecting the external environment vision for the auxiliary driving, provided by the invention, the image for describing the vehicle state is acquired through the image acquisition equipment, the discrete image is subjected to normalization processing, and the normalized discrete image is input into the neural network model after the energy gradient is calculated, so that the information weakening caused by gray-scale weighted average can be overcome, and the characteristic information of a local channel of a color image can be better reserved. Designing convolution windows with multiple hidden layers to combine omnidirectional convolution windows of the neural network model, thereby detecting characteristics of various forms in all directions, designing a nonlinear function as an excitation function of the neural network model, improving the detection capability of the neural network model, outputting a control variable and realizing intelligent auxiliary control on multiple controlled devices. The invention can reduce the number of external sensors required by the assistant driving system, reduce the load of the assistant driving system and reduce the complexity of the system.
Description
Technical Field
The invention belongs to the technical field of automatic driving, and particularly relates to a method and a device for detecting external environment vision for driving assistance.
Background
An automatic driving system is a popular field of recent industrial research, and depends on the cooperative cooperation of information acquisition and analysis systems such as machine learning, image processing, radar and positioning systems, so that the system can automatically and safely control motor vehicles in an environment without active operation of human beings.
Various sensors are required to sense the surrounding environment during automatic driving, and then the control is realized by starting or stopping a control system control device of a driving vehicle. And the more sensors that tend to be, the more complex the control system for driving the vehicle. After the control system obtains the sensing parameters of the sensor, more time is often needed to search for the corresponding controlled device. And the variety of sensors causes the process of processing the sensing parameters by the controller to be more complex, and the aim of automatic driving real-time assistance is often not achieved.
In the prior art, an auxiliary driving strategy for controlling the controlled device to start or stop is also output through detection of a neural network model. Due to the diversity of sensing parameters input by the sensor, the internal structure of the neural network model is complex, and the recognition effect of each layer structure in the neural network model is poor when the sensing parameter characteristics are recognized.
Disclosure of Invention
The invention provides a method and a device for vision detection of an external environment for driving assistance, which are used for improving the accuracy and the efficiency of the method for vision detection of the external environment. The specific technical scheme is as follows.
The invention provides a visual detection method for an environment outside a vehicle for assisting driving, which comprises the following steps:
acquiring a raw data composition set describing a vehicle state;
wherein the set comprises a plurality of elements, each element being a discrete image, the discrete images forming a sequence over time, each discrete image consisting of three channels, each channel being represented as a two-dimensional matrix;
normalizing each discrete image in the set so as to map the value of each discrete image in a fixed range to obtain a normalized discrete image;
calculating the energy gradient of each normalized discrete image according to the gray level energy distribution of the image;
taking the energy gradient of each discrete image as the input of a trained neural network model, so that the neural network model performs feature extraction and detection on the energy gradient of each discrete image to obtain a plurality of or a plurality of groups of control variables output by the neural network model;
the convolution window structures and convolution directions of all hidden layers of the neural network model are different, the excitation function of the convolution layer of the neural network model is a linear function, and the control variable corresponds to controlled equipment;
and controlling the controlled equipment corresponding to the control variable to execute corresponding operation according to the magnitude relation between the numerical value of the control variable and the threshold value.
Optionally, the normalizing each discrete image in the set so that the value mapping of each discrete image in a fixed range includes:
normalizing the two-dimensional matrix of each channel in each discrete image using a first normalization formula such that the values of each discrete image are mapped to within a fixed range;
wherein the first normalized formula is:
wherein, It(-, c) represents a two-dimensional matrix for each channel,
a two-dimensional matrix representing each channel after normalization,
mean represents the matrix ItThe mean value of ·, c), std is the standard deviation, c ═ { red, green, blue } represents the three primary color channels in the discrete image, and i, j represent the two-dimensional spatial coordinates of the discrete image.
Optionally, the calculating an energy gradient of each normalized discrete image according to the gray-scale energy distribution of the image includes:
setting a rectangular window taking the two-dimensional space coordinate of the two-dimensional matrix as the center in the normalized discrete image, sequencing pixel points of the area where the rectangular window is located according to pixel values, and determining a median pixel value;
calculating the deviation of each pixel value of each channel and the median pixel value aiming at each channel of each normalized discrete image, and determining the pixel value with the maximum difference;
and determining the pixel value with the largest difference as the energy gradient at the spatial position of the discrete image.
Wherein the energy gradient is represented as:
wherein,
representing the energy gradient, I (I, j, c) representing the normalized discrete image, c ═ { red, green, blue } representing the three primary color channels in the image, I, j representing the two-dimensional spatial coordinates of the image, mean (w)ij(u, v)) represents the median pixel value, wij(u, v) represents a rectangular window centered at (i, j) and high and wide in the two-dimensional matrix.
Optionally, the trained neural network model is obtained through the following steps:
setting the convolution window size of each hidden layer to ensure that the convolution window structure and the convolution direction of each hidden layer are different;
sequentially arranging the neurons in the input layer, the convolutional layer and the output layer so as to construct an initial neural network model;
acquiring a training data set;
the training data set comprises a plurality of samples, and one sample comprises a discrete image energy gradient and a manually marked label of controlled equipment corresponding to the gradient energy;
inputting each sample into the initial neural network, taking a control variable corresponding to a label of controlled equipment corresponding to gradient energy as a learning target, and iteratively training the initial neural network model until the learning target is reached or the iteration times are reached;
and taking the initial neural network reaching the learning target or reaching the iteration times as a trained neural network model.
Wherein the excitation function is:
where x denotes the input, α denotes the function is made to produce a discontinuous break at the point where x is 0, and R denotes the real number set.
Wherein the fixed range is [0,1], and the value of the control variable is positioned between [0,1 ].
Optionally, the controlling, according to the numerical value of the control variable, the controlled device corresponding to the control variable to perform a corresponding operation includes:
when the value of the control variable is larger than the threshold value, controlling the controlled equipment corresponding to the control variable to be started;
and when the value of the control variable is not greater than the threshold value, controlling the controlled equipment corresponding to the control variable to be closed.
Optionally, the controlled preparation includes: lighting equipment, air conditioning equipment, and braking equipment.
In a second aspect, the present invention provides a driving environment assessment apparatus based on a dual-mode neural network model, including:
the acquisition module is used for acquiring a raw data composition set describing the vehicle state;
wherein the set comprises a plurality of elements, each element being a discrete image, the discrete images forming a sequence over time, each discrete image consisting of three channels, each channel being represented as a two-dimensional matrix;
the normalization module is used for normalizing each discrete image in the set so as to map the value of each discrete image in a fixed range and obtain the normalized discrete image;
the computing module is used for computing the energy gradient of each normalized discrete image according to the gray level energy distribution of the image;
the detection module is used for taking the energy gradient of each discrete image as the input of the trained neural network model so that the neural network model performs feature extraction and detection on the energy gradient of each discrete image to obtain a plurality of or a plurality of groups of control variables output by the neural network model;
the convolution window structures and convolution directions of all hidden layers of the neural network model are different, the excitation function of the convolution layer of the neural network model is a linear function, and the control variable corresponds to controlled equipment;
and the control module is used for controlling the controlled equipment corresponding to the control variable to execute corresponding operation according to the magnitude relation between the numerical value of the control variable and the threshold value.
The innovation points of the embodiment of the invention comprise:
1. according to the vision detection method for the external environment of the auxiliary driving system, disclosed by the invention, effective control on vehicle-mounted systems such as light, air conditioners and brakes can be completed only by acquiring the images describing the vehicle states through the image acquisition equipment, so that the number of external sensors required by the auxiliary driving system is greatly reduced, the load of the auxiliary driving system is reduced, and the complexity of the system is reduced.
2. According to the method for detecting the external environment vision for the auxiliary driving, provided by the invention, the reliability of the neural network model can be improved by carrying out standardized processing on the discrete image.
3. According to the method for detecting the external environment vision for the auxiliary driving, provided by the invention, the energy gradient of the discrete image is calculated to be used as the input of the neural network model, so that the information weakening caused by gray-scale weighted average can be overcome, the characteristic information of the local channel of the color image can be better reserved, and the neural network model can obtain a better detection effect.
4. The invention provides a vehicle exterior environment visual detection method for auxiliary driving, which combines omnidirectional convolution windows of a neural network model through convolution windows of multiple hidden layers, thereby detecting characteristics of various forms in all directions, designing a nonlinear function as an excitation function of the neural network model, improving the detection capability of the neural network model, outputting a control variable and realizing intelligent auxiliary control on multiple controlled devices.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is to be understood that the drawings in the following description are merely exemplary of some embodiments of the invention. For a person skilled in the art, without inventive effort, further figures can be obtained from these figures.
FIG. 1 is a schematic flow chart of a method for vision detection of an environment outside a vehicle for driving assistance according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a constructed tracking model;
FIG. 3a is a map of convolutional layer convolution;
FIG. 3b is a schematic diagram of a convolutional layer connection;
fig. 4 is a schematic structural diagram of a vehicle exterior environment vision detection device for driving assistance according to an embodiment of the present invention.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. It is to be understood that the described embodiments are merely a few embodiments of the invention, and not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without inventive effort based on the embodiments of the present invention, are within the scope of the present invention.
It is to be noted that the terms "comprises" and "comprising" and any variations thereof in the embodiments and drawings of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
Fig. 1 is a schematic flow chart of a method for visually detecting an environment outside a vehicle for driving assistance according to an embodiment of the present invention. The method is applied to an autonomous vehicle. The method specifically comprises the following steps.
S1, acquiring a raw data composition set describing the vehicle state;
the set comprises a plurality of elements, each element is a discrete image, the discrete images form a sequence according to time, each discrete image is composed of three channels, and each channel is represented as a two-dimensional matrix;
the invention can collect raw data from an image acquisition device mounted with the vehicle, the raw data being a discrete time-sampled image sequence.
Assuming that the sequence of raw image data acquired by the image sensor forms a set V, each element of which is an image I consisting of 3 channels (RGB), i.e. c ═ { red, green, blue }, an image I ═ I (I, j, c) is defined, where I, j are two-dimensional spatial coordinates, and the set V ═ { I, j, c of the image sequence is defined1,I2,…,It…, and t represents the time order of image acquisition. Image ItIs a two-dimensional matrix.
S2, normalizing each discrete image in the set so that the value of each discrete image is mapped in a fixed range to obtain a normalized discrete image;
s3, calculating the energy gradient of each normalized discrete image according to the gray level energy distribution of the image;
s4, taking the energy gradient of each discrete image as the input of the trained neural network model, so that the neural network model performs feature extraction and detection on the energy gradient of each discrete image to obtain a plurality of or a plurality of groups of control variables output by the neural network model;
the convolution window structures and convolution directions of all hidden layers of the neural network model are different, the excitation function of the convolution layers of the neural network model is a linear function, and the control variable corresponds to the controlled equipment; the fixed range is [0,1], and the value of the control variable is positioned between [0,1 ].
The nonlinear elements of the neural network model are called the excitation function σ, and are used to make the network have the ability to classify nonlinear datasets.
Wherein the excitation function is:
where x denotes the input, α denotes the function is made to produce a discontinuous break at the point where x is 0, and R denotes the real number set.
And S5, controlling the controlled equipment corresponding to the control variable to execute corresponding operation according to the magnitude relation between the numerical value of the control variable and the threshold value.
Wherein the controlled preparation comprises: lighting equipment, air conditioning equipment, and braking equipment.
According to the method and the device for detecting the external environment vision for the auxiliary driving, provided by the invention, the image for describing the vehicle state is acquired through the image acquisition equipment, the discrete image is subjected to normalization processing, and the normalized discrete image is input into the neural network model after the energy gradient is calculated, so that the information weakening caused by gray-scale weighted average can be overcome, and the characteristic information of a local channel of a color image can be better reserved. Designing convolution windows with multiple hidden layers to combine omnidirectional convolution windows of the neural network model, thereby detecting characteristics of various forms in all directions, designing a nonlinear function as an excitation function of the neural network model, improving the detection capability of the neural network model, outputting a control variable and realizing intelligent auxiliary control on multiple controlled devices. The invention can reduce the number of external sensors required by the assistant driving system, reduce the load of the assistant driving system and reduce the complexity of the system.
As an alternative embodiment of the present invention, normalizing each discrete image in the set such that the value mapping of each discrete image is within a fixed range comprises:
normalizing the two-dimensional matrix of each channel in each discrete image using a first normalization formula such that the values of each discrete image are mapped to within a fixed range;
wherein the first normalized formula is:
wherein, It(-, c) represents a two-dimensional matrix for each channel,
a two-dimensional matrix representing each channel after normalization,
mean represents the matrix ItThe mean value of ·, c), std is the standard deviation, c ═ { red, green, blue } represents the three primary color channels in the discrete image, and i, j represent the two-dimensional spatial coordinates of the discrete image.
Acquiring each channel I _ t (·, c) of an image for time t, wherein each element is I _ t (I, j, c), the width is w, the height is h, processing each channel by using a first normalized formula independently, and processing a normalized matrix
And carrying out next treatment.
As an alternative embodiment of the present invention, calculating the energy gradient of each normalized discrete image according to the gray scale energy distribution of the image comprises:
the method comprises the following steps: setting a rectangular window taking the two-dimensional space coordinate of the two-dimensional matrix as the center in the normalized discrete image, sequencing pixel points of the area where the rectangular window is located according to pixel values, and determining a median pixel value;
step two: calculating the deviation of each pixel value of each channel and a median pixel value aiming at each channel of each normalized discrete image, and determining the pixel value with the maximum difference;
step three: and determining the pixel value with the largest difference as the energy gradient at the spatial position of the discrete image.
In the present invention, the energy gradient of an image is defined as a measure of the energy variation of the image, and specifically defined as follows:
definition of wij(u, v) represents a rectangular window in the two-dimensional matrix (image) centered at (i, j) and high and wide (u, v). mean (w)ij(u, v)) represents the median value of the window, i.e., the value in the middle of all pixels in the window sorted by value size. The energy gradient can be expressed as:
wherein,
representing the energy gradient, I (I, j, c) representing the normalized discrete image, c ═ { red, green, blue } representing the three primary color channels in the image, I, j representing the two-dimensional spatial coordinates of the image, mean (w)ij(u, v)) shows that the median pixel value is for a three-channel RGB image, the energy gradient of the image is a two-dimensional matrix, firstly, the deviation of the median value of the pixel and the surrounding pixels is calculated for each channel, and then the value with the maximum deviation of each channel is taken as the energy gradient of the image at the spatial position.
According to the invention, the energy gradient of the image is used for replacing the original image pixel to express the environmental characteristics, so that on one hand, the dimensionality of data is reduced, on the other hand, the information weakening caused by gray-scale weighted average can be overcome, the characteristic information of the local channel of the color image can be better reserved, and a better environmental identification effect can be obtained.
As an optional implementation manner of the present invention, the trained neural network model is obtained by the following steps:
the method comprises the following steps: setting the convolution window size of each hidden layer to ensure that the convolution window structure and the convolution direction of each hidden layer are different;
step two: sequentially arranging the neurons in the input layer, the convolutional layer and the output layer so as to construct an initial neural network model;
referring to fig. 2, the basic model of the neural network model is composed of an input layer, an output layer and a hidden layer, each layer includes a number of nodes called neurons, and the neurons, connections between the neurons constitute a neural network determined by excitation functions, weights and connection patterns between the neurons.
In FIG. 2, three nodes X are on the leftmost side1，X 21 is the input layer node, the right node y is the output layer node, h1,h2,h3For hiding layer nodes, sigma represents an excitation function and has the function of enabling the neural network to have nonlinear classification capability. The relationship between the output and the input of the neural network is defined by the following equation:
a1＝w1-11x1+w1-21x2+b1-1
a2＝w1-12x1+w1-22x2+b1-2
a3＝w1-13x1+w1-23x2+b1-3
y＝σ(w2-1σ(a1)+w2-2σ(a2)+w2-3σ(a3))
wherein, a1Represents h1Relation of input to output of a node, a2Represents h2Node input and output relationship, a3Represents h3Node input and output relationships, w-lower subscripts represent channel weights between nodes. The subscript b indicates the parameters between node 1 and the excitation function. σ parenthesis indicates the excitation function of the channel indicated in parenthesis.
Fig. 2 shows a fully-connected neural network (the most complete form), that is, each node of the hidden layer is connected to any node of the previous layer (without considering the excitation function), in practical applications, there may be a plurality of hidden layers, and the connection relationship between the number of nodes of each layer and the previous layer may be freely defined on the premise of implementation permission, that is, the combination or deletion of the connections is performed on the basis of the full connection.
Establishing an input layer as an image energy gradient
And defining a hidden layer as follows:
(11) the first hidden layer H1 of the neural network model N is defined as follows.
H1 is based on input layer data
By convolution windows
The latter result.
FIGS. 3a and 3b show the connection cases when the window size is 3x3, i.e. 1 ≦ p, q ≦ 3, and each node is only connected to the layer above it (i.e. input layer data)
) 3x3 nodes at corresponding positions have connections; the weights of the 3x3 connections are defined as row-column order
And each node v of H1 is connected to the input at 3x3 points, which have the same weight average at the corresponding position. b denotes the parameters of the first hidden layer with subscripts, starting with the first hidden layer with subscript 0, and pq denotes the size of the convolution kernel.
In particular, the invention defines H1 layer window sizes as 5x5, i.e., 1. ltoreq. p, q. ltoreq.5.
(12) The neural network model N second hidden layer H2 is defined as follows.
H2 is the output from the H1 layer, passing through a convolution window
The latter result.
Similar to the H1 layer structure, the weight window defining the H2 layer is also a rectangular window with the size of 3x7, i.e. p is more than or equal to 1 and less than or equal to 3, and q is more than or equal to 1 and less than or equal to 7.
(13) The third hidden layer H3 of the neural network model is defined as follows.
H3 is the output from the H2 layer, passing through a convolution window
The latter result.
Similar to the H1 and H2 layer structures, the weight window defining the H3 layer is also a rectangular window with the size of 7x3, namely, p is more than or equal to 1 and less than or equal to 7, and q is more than or equal to 1 and less than or equal to 3.
(14) The fourth hidden layer H4 of the network N is defined as follows.
The H4 layer is based on the output of the H3 layer, passing through a convolution window
The latter result.
Convolution window of H4 layer
Is a symmetric matrix, and p is more than or equal to 1 and q is less than or equal to 7.
(15) The fifth hidden layer H5 of the network N is defined as follows.
And when p is>When the sum of q is greater than or equal to q,
the H5 layer is based on the output of the H4 layer, passing through a convolution window
The latter result.
Convolution window of H5 layer
Is an upper (lower) triangular matrix, and p is more than or equal to 1 and q is less than or equal to 7.
(16) The sixth hidden layer H6 of the network N is defined as follows.
And when p is<When the sum of q is greater than or equal to q,
the H6 layer is based on the output of the H5 layer, passing through a convolution window
The latter result.
Convolution window of H6 layer
Is a window of convolution with
The opposite lower (upper) triangular matrix. And p is more than or equal to 1 and q is less than or equal to 7.
The H2-H6 layers combine convolution windows with different directions and different structures to detect features so as to identify features of various directions and various forms, and the combination of the multi-layer omnidirectional detection windows is another important feature of the invention.
(17) The seventh hidden layer H7 of network N is defined as follows.
H7 is the maximum value of the output of hidden layer H6 within the window pxq, and 1 ≦ p, q ≦ 4.
(18) The eighth hidden layer H8 of the network N is defined as follows.
H8 is the output from the H7 layer, passing through a convolution window
The latter result.
The window size of the H8 layer is 5x5, namely p is more than or equal to 1, and q is less than or equal to 5.
(19) The ninth hidden layer H9 of the network N is defined as follows.
The H9 layer is based on the output of the H8 layer, passing through a convolution window
The latter result.
Convolution window w of H9 layer9All of the elements are equal, and 1. ltoreq. p, q. ltoreq.5.
(110) The tenth hidden layer H10 of the network N is shaped like a fully connected layer, and there is a connection between each node of H10 and each node of H9, and the connection weights are independent.
(111) After the tenth hidden layer H10 of the network N, the output layer Y is connected in a fully connected form.
The output layer Y represents a plurality of or a plurality of groups of control variables, the value is between [0 and 1], and the meaning is to turn on or off a certain switch (or a certain group of switches); a switch may be considered open when some component of Y >0.66, otherwise the switch is considered closed. The number of control variables is related to the actual equipment to be controlled, and is divided into "light type", "air conditioner type", "brake type", etc. according to the application of the equipment in the driving system, and each type corresponds to one or more output components according to the difference of the actual equipment number.
The dimension of the output Y is independent and limited, but the dimension of the output Y is not limited by the actual device class or number. Meanwhile, the model described in the present invention is not limited to the type or number of actual devices.
Step three: acquiring a training data set;
the training data set comprises a plurality of samples, and one sample comprises a discrete image energy gradient and a label of controlled equipment corresponding to the artificially labeled gradient energy;
step four: inputting each sample into an initial neural network, taking a control variable corresponding to a label of controlled equipment corresponding to gradient energy as a learning target, and iteratively training an initial neural network model until the learning target is reached or the iteration times are reached;
step five: and taking the initial neural network reaching the learning target or reaching the iteration times as a trained neural network model.
It can be understood that when training the neural network, a plurality of sets of video data are selected and image energy gradients are established
And the position (on/off: 1/0) of each control switch corresponding to the gradient map is manually marked to train the initial neural network model.
The neural network model in the invention performs special processing on input image data, designs a special image energy gradient characteristic, a neural network model with a multilayer omnidirectional detection window and a nonlinear processing unit of the model. Compared with the classical algorithm, the neural network model effectively improves the accuracy of the on-off state judgment.
As an optional implementation manner of the present invention, according to the value of the controlled variable, the corresponding operation executed by the controlled device corresponding to the controlled variable includes:
the method comprises the following steps: when the value of the control variable is larger than the threshold value, controlling the controlled equipment corresponding to the control variable to be started;
step two: and when the value of the control variable is not greater than the threshold value, controlling the controlled equipment corresponding to the control variable to be closed.
The effects of the present invention are verified below with actual data, see table 1.
Table 1 comparison of effects
As can be seen from table 1, the first row is the judgment accuracy obtained by using Alex network, ReLU nonlinear unit and weighted gray scale image features (average of the values of the three channels R, G, B); the second line is the judgment accuracy rate obtained by adopting the multilayer omnidirectional detection window neural network model, the nonlinear unit and the weighted gray level image characteristics provided by the invention; the third line is the accuracy obtained by adopting the multilayer omnidirectional detection window neural network model, the novel nonlinear unit and the image energy gradient characteristic provided by the invention. Therefore, the driving environment evaluation method based on the dual-mode neural network model effectively improves the on-off state judgment accuracy.
As shown in fig. 4, the driving environment assessment apparatus based on the dual-mode neural network model provided by the present invention includes:
an acquisition module 41 for acquiring a set of raw data describing a vehicle state;
the set comprises a plurality of elements, each element is a discrete image, the discrete images form a sequence according to time, each discrete image is composed of three channels, and each channel is represented as a two-dimensional matrix;
a normalization module 42, configured to normalize each discrete image in the set, so that a value of each discrete image is mapped in a fixed range, and obtain a normalized discrete image;
a calculating module 43, configured to calculate an energy gradient of each normalized discrete image according to the gray level energy distribution of the image;
the detection module 44 is configured to use the energy gradient of each discrete image as an input of the trained neural network model, so that the neural network model performs feature extraction and detection on the energy gradient of each discrete image to obtain a plurality of or a plurality of groups of control variables output by the neural network model;
the convolution window structures and convolution directions of all hidden layers of the neural network model are different, the excitation function of the convolution layers of the neural network model is a linear function, and the control variable corresponds to the controlled equipment;
and the control module 45 is configured to control the controlled device corresponding to the control variable to execute a corresponding operation according to the magnitude relationship between the numerical value of the control variable and the threshold.
The above device embodiment corresponds to the method embodiment, and has the same technical effect as the method embodiment, and for the specific description, refer to the method embodiment. The device embodiment is obtained based on the method embodiment, and for specific description, reference may be made to the method embodiment section, which is not described herein again.
Those of ordinary skill in the art will understand that: the figures are merely schematic representations of one embodiment, and the blocks or flow diagrams in the figures are not necessarily required to practice the present invention.
Those of ordinary skill in the art will understand that: modules in the devices in the embodiments may be distributed in the devices in the embodiments according to the description of the embodiments, or may be located in one or more devices different from the embodiments with corresponding changes. The modules of the above embodiments may be combined into one module, or further split into multiple sub-modules.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for visual detection of an environment external to a vehicle for driver assistance, the method comprising:
acquiring a raw data composition set describing a vehicle state;
wherein the set comprises a plurality of elements, each element being a discrete image, the discrete images forming a sequence over time, each discrete image consisting of three channels, each channel being represented as a two-dimensional matrix;
normalizing each discrete image in the set so as to map the value of each discrete image in a fixed range to obtain a normalized discrete image;
calculating the energy gradient of each normalized discrete image according to the gray level energy distribution of the image;
taking the energy gradient of each discrete image as the input of a trained neural network model, so that the neural network model performs feature extraction and detection on the energy gradient of each discrete image to obtain a plurality of or a plurality of groups of control variables output by the neural network model;
the convolution window structures and convolution directions of all hidden layers of the neural network model are different, the excitation function of the convolution layer of the neural network model is a linear function, and the control variable corresponds to controlled equipment;
and controlling the controlled equipment corresponding to the control variable to execute corresponding operation according to the magnitude relation between the numerical value of the control variable and the threshold value.
2. A method for visual inspection of an off-board environment according to claim 1, wherein said normalizing each discrete image in said set such that the value of each discrete image maps to within a fixed range comprises:
normalizing the two-dimensional matrix of each channel in each discrete image using a first normalization formula such that the values of each discrete image are mapped to within a fixed range;
wherein the first normalized formula is:
wherein, It(-, c) represents a two-dimensional matrix for each channel,
a two-dimensional matrix representing each channel after normalization,
mean represents the matrix ItThe mean value of ·, c), std is the standard deviation, c ═ { red, green, blue } represents the three primary color channels in the discrete image, and i, j represent the two-dimensional spatial coordinates of the discrete image.
3. The vehicle exterior environment visual detection method according to claim 1, wherein said calculating an energy gradient of each normalized discrete image from a gray scale energy distribution of the image comprises:
setting a rectangular window taking the two-dimensional space coordinate of the two-dimensional matrix as the center in the normalized discrete image, sequencing pixel points of the area where the rectangular window is located according to pixel values, and determining a median pixel value;
calculating the deviation of each pixel value of each channel and the median pixel value aiming at each channel of each normalized discrete image, and determining the pixel value with the maximum difference;
and determining the pixel value with the largest difference as the energy gradient at the spatial position of the discrete image.
4. A method for visual inspection of an off-board environment according to claim 3, characterized in that said energy gradient is expressed as:
wherein,
representing the energy gradient, I (I, j, c) representing the normalized discrete image, c ═ { red, green, blue } representing the three primary color channels in the image, I, j representing the two-dimensional spatial coordinates of the image, mean (w)ij(u, v)) represents the median pixel value, Wij(u, v) represents a rectangular window centered at (i, j) and high and wide in the two-dimensional matrix.
5. The vehicle exterior environment visual inspection method according to claim 1, wherein the trained neural network model is obtained by:
setting the convolution window size of each hidden layer to ensure that the convolution window structure and the convolution direction of each hidden layer are different;
sequentially arranging the neurons in the input layer, the convolutional layer and the output layer so as to construct an initial neural network model;
acquiring a training data set;
the training data set comprises a plurality of samples, and one sample comprises a discrete image energy gradient and a manually marked label of controlled equipment corresponding to the gradient energy;
inputting each sample into the initial neural network, taking a control variable corresponding to a label of controlled equipment corresponding to gradient energy as a learning target, and iteratively training the initial neural network model until the learning target is reached or the iteration times are reached;
and taking the initial neural network reaching the learning target or reaching the iteration times as a trained neural network model.
6. A method for visual inspection of an extra-vehicular environment according to claim 1, characterized in that said excitation function is:
where x denotes the input, α denotes the function is made to produce a discontinuous break at the point where x is 0, and R denotes the real number set.
7. The visual detection method for the environment outside the vehicle according to claim 1, wherein the fixed range is [0,1], and the value of the control variable is located between [0,1 ].
8. The visual detection method for the exterior environment of the vehicle according to claim 7, wherein the controlling the controlled device corresponding to the controlled variable to perform corresponding operations according to the numerical value of the controlled variable comprises:
when the value of the control variable is larger than the threshold value, controlling the controlled equipment corresponding to the control variable to be started;
and when the value of the control variable is not greater than the threshold value, controlling the controlled equipment corresponding to the control variable to be closed.
9. The vehicle exterior environment visual inspection method according to claim 1, wherein the controlled preparation includes: lighting equipment, air conditioning equipment, and braking equipment.
10. A driving environment evaluation apparatus based on a dual-mode neural network model, the apparatus comprising:
the acquisition module is used for acquiring a raw data composition set describing the vehicle state;
wherein the set comprises a plurality of elements, each element being a discrete image, the discrete images forming a sequence over time, each discrete image consisting of three channels, each channel being represented as a two-dimensional matrix;
the normalization module is used for normalizing each discrete image in the set so as to map the value of each discrete image in a fixed range and obtain the normalized discrete image;
the computing module is used for computing the energy gradient of each normalized discrete image according to the gray level energy distribution of the image;
the detection module is used for taking the energy gradient of each discrete image as the input of the trained neural network model so that the neural network model performs feature extraction and detection on the energy gradient of each discrete image to obtain a plurality of or a plurality of groups of control variables output by the neural network model;
the convolution window structures and convolution directions of all hidden layers of the neural network model are different, the excitation function of the convolution layer of the neural network model is a linear function, and the control variable corresponds to controlled equipment;
and the control module is used for controlling the controlled equipment corresponding to the control variable to execute corresponding operation according to the magnitude relation between the numerical value of the control variable and the threshold value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110747205.6A CN113537003B (en) | 2021-07-02 | 2021-07-02 | External environment visual detection method and device for driving assistance |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110747205.6A CN113537003B (en) | 2021-07-02 | 2021-07-02 | External environment visual detection method and device for driving assistance |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113537003A true CN113537003A (en) | 2021-10-22 |
| CN113537003B CN113537003B (en) | 2022-10-21 |
Family
ID=78097549
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110747205.6A Expired - Fee Related CN113537003B (en) | 2021-07-02 | 2021-07-02 | External environment visual detection method and device for driving assistance |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113537003B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109993074A (en) * | 2019-03-14 | 2019-07-09 | 杭州飞步科技有限公司 | Assist processing method, device, equipment and the storage medium driven |
| CN110059582A (en) * | 2019-03-28 | 2019-07-26 | 东南大学 | Driving behavior recognition methods based on multiple dimensioned attention convolutional neural networks |
| CN110163077A (en) * | 2019-03-11 | 2019-08-23 | 重庆邮电大学 | A kind of lane recognition method based on full convolutional neural networks |
| CN110598637A (en) * | 2019-09-12 | 2019-12-20 | 齐鲁工业大学 | Unmanned driving system and method based on vision and deep learning |
| US20200026936A1 (en) * | 2018-03-30 | 2020-01-23 | Beijing Sensetime Technology Development Co., Ltd. | Vehicle lamp detection methods and apparatuses, methods and apparatuses for implementing intelligent driving, media and devices |
-
2021
- 2021-07-02 CN CN202110747205.6A patent/CN113537003B/en not_active Expired - Fee Related
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200026936A1 (en) * | 2018-03-30 | 2020-01-23 | Beijing Sensetime Technology Development Co., Ltd. | Vehicle lamp detection methods and apparatuses, methods and apparatuses for implementing intelligent driving, media and devices |
| CN110163077A (en) * | 2019-03-11 | 2019-08-23 | 重庆邮电大学 | A kind of lane recognition method based on full convolutional neural networks |
| CN109993074A (en) * | 2019-03-14 | 2019-07-09 | 杭州飞步科技有限公司 | Assist processing method, device, equipment and the storage medium driven |
| CN110059582A (en) * | 2019-03-28 | 2019-07-26 | 东南大学 | Driving behavior recognition methods based on multiple dimensioned attention convolutional neural networks |
| CN110598637A (en) * | 2019-09-12 | 2019-12-20 | 齐鲁工业大学 | Unmanned driving system and method based on vision and deep learning |
Non-Patent Citations (2)
| Title |
|---|
| YUANFANG ZHAO ET AL.: "End-to-end autonomous driving based on the convolution neural network model", 《PROCEEDINGS OF APSIPA ANNUAL SUMMIT AND CONFERENCE 2019》 * |
| 闫龙 等: ""基于视觉的驾驶辅助系统研究"", 《科技创新与应用》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113537003B (en) | 2022-10-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106599773B (en) | Deep learning image identification method and system for intelligent driving and terminal equipment | |
| CN106228125B (en) | Method for detecting lane lines based on integrated study cascade classifier | |
| CN104978580B (en) | A kind of insulator recognition methods for unmanned plane inspection transmission line of electricity | |
| CN107729801A (en) | A kind of vehicle color identifying system based on multitask depth convolutional neural networks | |
| Shinzato et al. | Fast visual road recognition and horizon detection using multiple artificial neural networks | |
| CN108062569B (en) | Unmanned vehicle driving decision method based on infrared and radar | |
| CN110414418B (en) | Road detection method for multi-scale fusion of image-laser radar image data | |
| CN111931764B (en) | Target detection method, target detection frame and related equipment | |
| CN104517103A (en) | Traffic sign classification method based on deep neural network | |
| EP4290408A1 (en) | Feature extraction method and apparatus | |
| CN101813475B (en) | Method for adaptively detecting remote obstacle | |
| CN107229929A (en) | A kind of license plate locating method based on R CNN | |
| CN108197569A (en) | Obstacle recognition method, device, computer storage media and electronic equipment | |
| CN106156765A (en) | safety detection method based on computer vision | |
| Shinzato et al. | A road following approach using artificial neural networks combinations | |
| CN110490155B (en) | Method for detecting unmanned aerial vehicle in no-fly airspace | |
| CN116704273A (en) | Self-adaptive infrared and visible light dual-mode fusion detection method | |
| CN108052989A (en) | A kind of image classification method based on Spline convolution neutral net | |
| CN109271934A (en) | System and method for Traffic Sign Recognition | |
| CN108288038A (en) | Night robot motion's decision-making technique based on scene cut | |
| CN115631344A (en) | Target detection method based on feature adaptive aggregation | |
| Kale et al. | A road sign detection and the recognition for driver assistance systems | |
| CN108932471A (en) | A kind of vehicle checking method | |
| CN115359248A (en) | Robot navigation obstacle avoidance method and system based on meta-learning | |
| Shabarinath et al. | Convolutional neural network based traffic-sign classifier optimized for edge inference |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20221021 |