CN115376104A - Automatic driving scene recognition method, device and system - Google Patents

Automatic driving scene recognition method, device and system Download PDF

Info

Publication number
CN115376104A
CN115376104A CN202211032500.4A CN202211032500A CN115376104A CN 115376104 A CN115376104 A CN 115376104A CN 202211032500 A CN202211032500 A CN 202211032500A CN 115376104 A CN115376104 A CN 115376104A
Authority
CN
China
Prior art keywords
neural network
automatic driving
scene recognition
convolution neural
environment picture
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211032500.4A
Other languages
Chinese (zh)
Inventor
叶武剑
陈华润
刘怡俊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong University of Technology
Original Assignee
Guangdong University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong University of Technology filed Critical Guangdong University of Technology
Priority to CN202211032500.4A priority Critical patent/CN115376104A/en
Publication of CN115376104A publication Critical patent/CN115376104A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/061—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using biological neurons, e.g. biological neurons connected to an integrated circuit
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Neurology (AREA)
  • Microelectronics & Electronic Packaging (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

The application discloses an automatic driving scene recognition method, device and system, wherein the method comprises the following steps: acquiring an environment picture shot when a vehicle is driven; correcting the environment picture into a plane picture to obtain a plane environment picture; inputting the plane environment picture into a pulse convolution neural network hardware platform to obtain a scene recognition result corresponding to the environment picture, wherein a pulse convolution neural network carried in the pulse convolution neural network hardware platform is built by preset neurons, and the preset neurons comprise LIF neurons and/or IF neurons; and sending the scene recognition result to the alarm so that the alarm performs corresponding alarm operation according to the scene recognition result. The technical problems of low operation speed and high power consumption of deep learning in the field of automatic driving are solved, and the pulse convolution neural network meets the use requirement of automatic driving.

Description

Automatic driving scene recognition method, device and system
Technical Field
The present application relates to the field of automatic driving technologies, and in particular, to a method, an apparatus, and a system for identifying an automatic driving scenario.
Background
Pulse convolutional Neural networks (SNNs) are third generation Neural networks, inspired by the way biological brains process information-electrical pulses. The pulse convolution neural network has a solid biological basis, small resource occupation, low power consumption and high efficiency, so that the research on the pulse convolution neural network has great significance.
In automatic driving, the requirement on real-time performance is very high, but the existing pulse convolution neural network is low in operation speed and high in power consumption, and cannot meet the use requirement of automatic driving.
Disclosure of Invention
In view of this, the application provides an automatic driving scene recognition method, device and system, which solve the technical problems of low operation speed and high power consumption in the field of automatic driving in deep learning, and enable a pulse convolution neural network to meet the use requirements of automatic driving.
A first aspect of the present application provides an automatic driving scene recognition system, including:
acquiring an environment picture shot when a vehicle is driven;
correcting the environment picture into a plane picture to obtain a plane environment picture;
inputting the plane environment picture into a pulse convolution neural network hardware platform to obtain a scene recognition result corresponding to the environment picture, wherein a pulse convolution neural network carried in the pulse convolution neural network hardware platform is built by preset neurons, and the preset neurons comprise LIF neurons and/or IF neurons;
and sending the scene recognition result to an alarm so that the alarm performs corresponding alarm operation according to the scene recognition result.
Optionally, the preset neuron membrane potential threshold is determined by a genetic algorithm.
Optionally, the pulse convolution neural network is obtained by training through various data sets;
the plurality of data sets includes: GTSRB dataset, belgiumTSC dataset and CTSDB dataset.
Optionally, the input of the pulse convolutional neural network is the encoded data of the planar environment picture; the coding formula of the plane environment picture is as follows:
Figure BDA0003818020690000021
wherein P (K) is the probability of generating pulses for data, K is the number of pulses, λ T is the excitation frequency proportional to the pixel value, and the probability of not generating pulses within T time is e -λT 。
Optionally, when the preset neuron comprises a LIF neuron, the neuron model of the LIF neuron is:
Figure BDA0003818020690000022
wherein, tau m Is the membrane time constant of the LIF neuron, V is the membrane potential of the LIF neuron, R is the resistance of the LIF neuron, and I is the inflow current of the LIF neuron.
A second aspect of the present application provides an automatic driving scene recognition apparatus, including:
the acquisition unit is used for acquiring an environment picture shot when the vehicle is driven;
the correcting unit is used for correcting the environment picture into a plane graph to obtain a plane environment picture;
the identification unit is used for inputting the plane environment picture into a pulse convolution neural network hardware platform to obtain a scene identification result corresponding to the environment picture, wherein a pulse convolution neural network carried in the pulse convolution neural network hardware platform is built by a preset neuron, and the preset neuron comprises an LIF neuron and/or an IF neuron;
and the sending unit is used for sending the scene recognition result to an alarm so that the alarm carries out corresponding alarm operation according to the scene recognition result.
Optionally, the preset neuron membrane potential threshold is determined by a genetic algorithm.
Optionally, the pulse convolution neural network is obtained by training through various data sets;
the plurality of data sets includes: GTSRB dataset, belgiumTSC dataset and CTSDB dataset.
Optionally, the input of the pulse convolutional neural network is the encoded data of the planar environment picture;
the coding formula of the plane environment picture is as follows:
Figure BDA0003818020690000031
wherein P (K) is the probability of generating pulses for data, K is the number of pulses, λ T is the excitation frequency proportional to the pixel value, and the probability of not generating pulses within T time is e -λT 。
A third aspect of the present application provides an automatic driving scene recognition system, including: the system comprises a camera, an alarm and an automatic driving scene recognition device;
the camera is used for an environment picture when the vehicle is driven;
the automatic driving scene recognition device is used for correcting the environment picture into a plane picture to obtain a plane environment picture; the system is also used for inputting the plane environment picture to a pulse convolution neural network hardware platform to obtain a scene recognition result corresponding to the environment picture; the pulse convolution neural network is set up by preset neurons, and the preset neurons comprise LIF neurons and/or IF neurons;
and the alarm is used for carrying out corresponding alarm operation according to the scene recognition result.
According to the technical scheme, the method has the following advantages:
the automatic driving scene recognition method in the application comprises the following steps: acquiring an environment picture shot when a vehicle is driven; correcting the environment picture into a plane picture to obtain a plane environment picture; inputting the plane environment picture into a pulse convolution neural network hardware platform to obtain a scene recognition result corresponding to the environment picture, wherein a pulse convolution neural network carried in the pulse convolution neural network hardware platform is built by preset neurons, and the preset neurons comprise LIF neurons and/or IF neurons; and sending the scene recognition result to the alarm so that the alarm performs corresponding alarm operation according to the scene recognition result.
According to the method, the LIF and IF neurons are adopted to build the pulse convolution neural network for automatic driving scene recognition, and the pulse convolution neural network is deployed in a pulse convolution neural network hardware platform, so that the neurons in all layers are more suitable for the use requirement of automatic driving, the technical problems of low operation speed and high power consumption existing in the field of automatic driving in deep learning are solved, and the pulse convolution neural network meets the use requirement of automatic driving.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a schematic flowchart of an embodiment of an automatic driving scene recognition method in an embodiment of the present application;
FIG. 2 is a schematic structural diagram of a hardware platform of a pulse convolution neural network according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of an embodiment of an automatic driving scene recognition apparatus in an embodiment of the present application;
fig. 4 is a schematic structural diagram of an embodiment of an automatic driving scene recognition system in the embodiment of the present application.
Detailed Description
The embodiment of the application provides an automatic driving scene recognition method, device and system, solves the technical problems of low operation speed and high power consumption of deep learning in the automatic driving field, and enables a pulse convolution neural network to meet the use requirement of automatic driving.
In order to make the technical solutions of the present application better understood, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.
A first aspect of an embodiment of the present application provides an embodiment of an automatic driving scene recognition method.
Referring to fig. 1, a flow chart of an embodiment of an automatic driving scene recognition method in an embodiment of the present application is schematically illustrated.
The automatic driving scene recognition method in the embodiment comprises the following steps:
step 101, obtaining an environment picture shot when a vehicle is driven.
It is understood that the environmental picture in the present embodiment may be obtained by capturing a driving scene of the autonomous vehicle through a fish-eye camera.
And 102, correcting the environment picture into a plane picture to obtain a plane environment picture.
And (4) picture correction, namely correcting the panoramic image acquired by the panoramic fish-eye camera into a plane image so that the image is more suitable for being processed by a pulse convolution neural network.
103, inputting the plane environment picture into a pulse convolution neural network hardware platform to obtain a scene recognition result corresponding to the environment picture, wherein the pulse convolution neural network carried in the pulse convolution neural network hardware platform is built by preset neurons, and the preset neurons comprise LIF neurons and/or IF neurons.
It is understood that the preset membrane potential threshold in neurons is determined by a genetic algorithm.
Specifically, the pulse convolution neural network is obtained by training various data sets;
the plurality of data sets includes: GTSRB dataset, belgiumTSC dataset and CTSDB dataset.
The method and the device adopt various different data sets to train the pulse convolution neural network, can improve the effectiveness and stability of automatic driving scene recognition, are more suitable for the deployment and reasoning of the SNN (pulse convolution neural network), and have higher speed and lower consumption.
It should be noted that, the present application adopts a poisson coding mechanism to encode and input a planar environment picture. The poisson coding encodes the input data into a pulse sequence whose distribution of issuance times conforms to the poisson process. Whether each data generates the pulse is only related to the input, and the probability of generating the pulse is not interfered with each other, namely the input of the pulse convolution neural network is the coded data of the plane environment picture; the coding formula of the plane environment picture is as follows:
Figure BDA0003818020690000051
wherein P (K) is the probability of generating pulses for data, K is the number of pulses, λ T is the excitation frequency proportional to the pixel value, and the probability of not generating pulses within T time is e -λT 。
Specifically, when the preset neuron includes a LIF neuron, the neuron model of the LIF neuron is:
Figure BDA0003818020690000052
wherein, tau m Is the membrane time constant of the LIF neuron, V is the membrane potential of the LIF neuron, R is the resistance of the LIF neuron, and I is the inflow current of the LIF neuron.
It is understood that the LIF model is one of the most popular simplified pulse neuron models, which can be used to effectively construct SNN, and the basic calculation of LIF neuron has a differential calculation, i.e. the leakage of membrane potential can be realized. The LIF model has small calculated amount and is most widely applied.
In the embodiment, the inference of automatic driving identification of the panoramic fisheye image is realized on a hardware platform by building a pulse convolution neural network hardware platform. The pulse convolution neural network hardware platform in this embodiment may be an FPGA.
The membrane potential thresholds of LIF neurons and IF neurons can directly affect the pulse firing rate, i.e. changing the membrane potential threshold has some impact on the final recognition result. In order to find a suitable membrane potential threshold value for enabling SNNs (pulse convolution neural networks) to identify images more accurately, a search optimization algorithm is required, in which Genetic Algorithms (GAEs) with elite retention strategies are used to optimize the membrane potential threshold values of LIF neurons or IF neurons in each layer of SNNs.
The step of configuring the membrane potential threshold of the LIF neuron or the IF neuron comprises:
A. initializing the GA population, and randomly selecting two values of 0 and 1, namely binary gene coding;
B. during each iteration, the genotypes of all the individuals are decoded into decimal values, and the fitness of all the individuals of the population to the target is completed according to the decoded values;
C. after each fitness calculation, the optimal genotype individuals are retained. And then excluding the individual, selecting, crossing and mutating the population of the remaining individuals.
D. Combining the population after mutation with the optimal individuals retained before into a new population.
N gen ,N indiv ,N bit Are respectively set to be 50,12, N t Will be adjusted according to the actual network. The fitness calculation is to obtain an optimized accuracy of the data set at the corresponding membrane potential threshold. Since the goal is to optimize the recognition accuracy, the neurons of each layer are global in that they participate in the process of solving the global optimum. It can be seen that the entire optimization process does not change the weights of the network, but finds several suitable thresholds to obtain the optimal recognition accuracy.
Further, when the pulse convolution neural network is trained, since a data set used for training is labeled, an Adam algorithm which is a supervised learning algorithm with gradient descent is adopted in the method. The Adam algorithm is a random gradient descent optimization method based on momentum thought. The Adam algorithm comprehensively considers the first moment estimation and the second moment estimation of the gradient, and updates the current parameters after calculating a moving average value.
Fig. 2 shows a specific implementation structure of the pulse convolution neural network hardware platform in this embodiment, and each functional scheme is designed as follows:
the control core is as follows: and is responsible for the coordinated operation of the internal modules in the platform. In the control kernel, there is a network mapping memory, which stores the network topology, the range of target neurons mapped for each neuron in the network, the number of included layers, the number of input channels in each layer, the number of times of multiplexing single-channel PEs, the number of times of multiplexing output channel PEs, the size of convolution kernel, the size of output image and the stride. The network mapping memory can be configured through a serial port to realize different network topologies.
The control core is internally provided with four important registers for recording and updating the current state of convolution calculation in real time, and (a) Did _ Layer represents the number of Layer indexes calculated currently; (b) recording a number Input layer index for the Did _ Input _ Channel; (c) Did _ PE _ Mult _ Times is the number of Times the PE array is multiplexed into a single-channel image; (d) Did _ PE _ Output _ Channel is the number of times all Output channels of the PE array are multiplexed. Also, the control core contains a counter that is started at the beginning of the calculation. When the counter reaches a preconfigured count value, the calculation is complete.
An encoder: responsible for encoding the input pixel data with pulses and buffering them into a pulse buffer. In addition, when the pulse buffer is not empty, the control core retrieves the corresponding pulse and activates the neuronal core, the synucleus, and the neuronal module to update the membrane potential. When a pulse is generated and not from the output layer, it is buffered back into the pulse buffer.
Neuronal nucleus: it contains a fully-connected core and a PE array. The fully-connected core is responsible for the computation of the fully-connected layer. Neurons are computed using parallel-coupled pipeline techniques. For parallel recording in neurons, a local pulse buffer register is used in the neuron core, which is responsible for recording the emission of parallel pulses and is again very simple to implement in hardware. For example 1024 pulses for one layer, 16 parallel pulses are used. In the first calculation, 0-15 neurons of pulse release can be obtained, and only the lower 16 bits of LBR and SR are used for logic tracks, so that parallel pulse recording is realized.
PE array: and finishing the construction of the pulse PE array and being responsible for the realization of convolution and pooling calculation. Including the design of individual PE elements and PE array data combinations. For the PE unit, a data selector and a register are included to hold intermediate calculation values when performing convolution operations. When an input pulse is detected, the weights are accumulated in the register. When the single convolution operation is completed, the membrane potential of the neuron according to the input and the register are added to obtain the final membrane potential. When the potential exceeds the threshold, a pulse is issued and the potential is set to 0. When the pooling operation is carried out, the method can realize the pooling calculation of 2 multiplied by 2, the FSM aiming at the pooling operation exists in the PE to count the number of the pulses in the Map, and then the numerical value which needs to be accumulated at this time is obtained according to the obtained number and is sent to the register to obtain the final neuron membrane potential and the internal pulse.
For the PE calculation array, the PE calculation array comprises 16 × 16 PE units, and is mainly responsible for integration and output of output results of the systolic array, including membrane potential and internal output pulses of convolution neurons. For Map of single channel input, one neuron is only related to pixels of one channel, i.e. calculation is performed for a single time, whereas for Map of multi-channel input, one neuron is related to pixels of all channels of input, and in the design, the method is implemented in an OR mode. Specifically, when the input image is 16 channels, the membrane potential and the output pulse are obtained by adopting a basic calculation mode for the first channel, but the membrane potential is still normally calculated when the following 15 channels are calculated, and the output pulse needs to be subjected to or operation with the previous calculation, so that the multi-channel convolution calculation can be realized.
Leakage FSM: is responsible for coordinating and controlling the leaky neurons.
And (3) synaptic nucleus: the synapse weight obtaining unit is responsible for obtaining the weight of the corresponding address when the system updates the membrane potential. The weights are represented by 16-bit fixed point decimals in this application. Neurons memory stores the membrane potential of neurons.
Pulse counting: the pulse number emitted by the neuron is calculated, and a basis is provided for the identification of the final result.
The whole data processing flow of the pulse convolution neural network hardware platform is as follows: firstly, inputting image pixel data of a plane environment picture into an encoder to obtain binary pulse data, and storing the pulse data into a pulse cache. And the control core starts calculation, takes out the pulse data from the cache for data sorting, sends the rearranged data into the neuron core, and respectively retrieves the neuron state and the weight from the synapse core and the weight storage area to be sent into the neuron core for convolution or full-connection calculation. And sending the calculated result to a neuron module to judge whether to emit pulses or not, immediately updating the state of the neuron, and simultaneously enabling a control core to start a leakage state machine to leak the state of the neuron so as to update the state of the neuron. And if the calculation is not finished, namely the obtained internal pulse is obtained, the pulse data is sent back to the pulse cache to wait for the control core to start the next calculation, if the calculation is finished, the pulse data is output, and meanwhile, a pulse counter module is arranged in the pulse cache to record the number of pulses issued when the hardware platform runs, so that a basis is provided for a final scene recognition result.
And 104, sending the scene recognition result to an alarm so that the alarm performs corresponding alarm operation according to the scene recognition result.
It can be understood that the alarm operation varies with the scene recognition result, for example, if the scene recognition result is that an obstacle is recognized, the alarm operation is to prompt avoidance; if the scene recognition result is that the speed limit is recognized, alarming operation is to prompt deceleration; and if the scene recognition result is turning, alarming to prompt corresponding left turning and right turning.
In the embodiment, the LIF and IF neurons are adopted to build the pulse convolution neural network for scene recognition of automatic driving, and the pulse convolution neural network is deployed in a pulse convolution neural network hardware platform, so that the neurons in all layers are more suitable for the use requirement of automatic driving, the technical problems of low operation speed and high power consumption in the field of automatic driving of deep learning are solved, and the pulse convolution neural network meets the use requirement of automatic driving.
The above is an embodiment of an automatic driving scene recognition method provided in the embodiments of the present application, and the following is an embodiment of an automatic driving scene recognition apparatus provided in the embodiments of the present application.
Referring to fig. 3, a schematic structural diagram of an embodiment of an automatic driving scene recognition apparatus in an embodiment of the present application is shown.
The automatic driving scene recognition device in this embodiment includes:
the system comprises an acquisition unit, a processing unit and a display unit, wherein the acquisition unit is used for acquiring an environment picture shot when a vehicle is driven;
the correcting unit is used for correcting the environment picture into a plane picture to obtain a plane environment picture;
the recognition unit is used for inputting the plane environment picture into a pulse convolution neural network hardware platform to obtain a scene recognition result corresponding to the environment picture, the pulse convolution neural network carried in the pulse convolution neural network hardware platform is built by preset neurons, and the preset neurons comprise LIF neurons and/or IF neurons;
and the sending unit is used for sending the scene recognition result to the alarm so that the alarm carries out corresponding alarm operation according to the scene recognition result.
It is understood that the preset membrane potential threshold in neurons is determined by a genetic algorithm.
In an alternative embodiment, the pulse convolution neural network is trained from a plurality of data sets;
the plurality of data sets includes: GTSRB dataset, belgiumTSC dataset and CTSDB dataset.
Specifically, the input of the pulse convolution neural network is encoded data of a planar environment picture, and an encoding formula of the planar environment picture is as follows:
Figure BDA0003818020690000091
wherein P (K) is the probability of generating pulses for data, K is the number of pulses, λ T is the excitation frequency proportional to the pixel value, and the probability of not generating pulses within T time is e -λT 。
In the embodiment, the LIF and IF neurons are adopted to build the pulse convolution neural network for scene recognition of automatic driving, and the pulse convolution neural network is deployed in a pulse convolution neural network hardware platform, so that the neurons in all layers are more suitable for the use requirement of automatic driving, the technical problems of low operation speed and high power consumption in the field of automatic driving of deep learning are solved, and the pulse convolution neural network meets the use requirement of automatic driving.
The above is an embodiment of an automatic driving scene recognition apparatus provided in the embodiments of the present application, and the following is an embodiment of an automatic driving scene recognition system provided in the embodiments of the present application.
Referring to fig. 4, a schematic structural diagram of an embodiment of an automatic driving scene recognition system in an embodiment of the present application is shown.
The automatic driving scene recognition system in the embodiment includes: the system comprises a camera, an alarm and an automatic driving scene recognition device;
the system comprises a camera, an alarm and an automatic driving scene recognition device;
the camera is used for an environment picture when the vehicle is driven;
the automatic driving scene recognition device is used for correcting the environment picture into a plane picture to obtain a plane environment picture; the system is also used for inputting the plane environment picture into a pulse convolution neural network hardware platform to obtain a scene recognition result corresponding to the environment picture; the pulse convolution neural network carried in the pulse convolution neural network hardware platform is constructed by preset neurons, and the preset neurons comprise LIF neurons and/or IF neurons;
and the alarm is used for carrying out corresponding alarm operation according to the scene recognition result.
The camera in this embodiment: and capturing a real environment image during automatic driving by adopting a panoramic fisheye camera.
The panoramic fish-eye camera acquires the panoramic image, so that the panoramic image is corrected into a plane image for being suitable for pulse convolution neural network processing, namely, the environment image is corrected into the plane image, and the plane environment image is obtained.
The overall process of the automatic driving scene recognition system in the embodiment is as follows: firstly, a pulse convolution neural network built on the basis of LIF/IF neurons is realized by adopting a plurality of data sets on a software level, an Adam gradient descent algorithm is adopted in a training algorithm, then the trained SCNN is deployed on an FPGA after being quantized, a hardware platform is built, and accelerated reasoning of panoramic fisheye image automatic driving recognition is realized.
The automatic driving scene recognition process comprises the following steps: the method comprises the steps of firstly capturing a real environment picture during automatic driving by using a panoramic fisheye camera, and then sending the obtained environment picture into an image corrector. The image corrector processes the image and corrects the panoramic image into a plane image, so that the panoramic image is more suitable for being processed by a pulse convolution neural network hardware platform. And the pulse convolution neural network hardware platform carries out coding, calculation and identification after obtaining the plane graph, and then sends the identification result to the alarm. The alarm performs corresponding operation according to the identification result, for example, if an obstacle is identified, avoidance is prompted; if the speed limit is identified, speed reduction is prompted; and if the turning is recognized, prompting the corresponding left turning and right turning.
In the embodiment, the LIF and IF neurons are adopted to build the pulse convolution neural network for scene recognition of automatic driving, and the pulse convolution neural network is deployed in a pulse convolution neural network hardware platform, so that the neurons in all layers are more suitable for the use requirement of automatic driving, the technical problems of low operation speed and high power consumption in the field of automatic driving of deep learning are solved, and the pulse convolution neural network meets the use requirement of automatic driving.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of a unit is only one logical functional division, and there may be other divisions when the actual implementation is performed, for example, a plurality of units or components may be combined or may be integrated into another grid network to be installed, or some features may be omitted or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in the form of hardware, or may also be implemented in the form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims (10)

1. An automatic driving scene recognition method, comprising:
obtaining an environmental picture shot when a vehicle is driven;
correcting the environment picture into a plane picture to obtain a plane environment picture;
inputting the plane environment picture into a pulse convolution neural network hardware platform to obtain a scene recognition result corresponding to the environment picture, wherein a pulse convolution neural network carried in the pulse convolution neural network hardware platform is set up by a preset neuron, and the preset neuron comprises an LIF neuron and/or an IF neuron;
and sending the scene recognition result to an alarm so that the alarm performs corresponding alarm operation according to the scene recognition result.
2. The automatic driving scenario recognition method of claim 1, wherein the preset neuron membrane potential threshold is determined by a genetic algorithm.
3. The automatic driving scenario recognition method of claim 1, wherein the pulse convolution neural network is trained from multiple data sets;
the plurality of data sets includes: GTSRB dataset, belgiumTSC dataset and CTSDB dataset.
4. The automatic driving scene recognition method according to claim 1, wherein the input of the pulse convolution neural network is encoded data of the plane environment picture;
the coding formula of the plane environment picture is as follows:
Figure FDA0003818020680000011
wherein P (K) is the probability of generating pulses for data, K is the number of pulses, λ T is the excitation frequency proportional to the pixel value, and the probability of not generating pulses within T time is e -λT 。
5. The automatic driving scene recognition method according to claim 1, wherein when the preset neuron includes a LIF neuron, a neuron model of the LIF neuron is:
Figure FDA0003818020680000012
wherein, tau m Is the membrane time constant of the LIF neuron, V is the membrane potential of the LIF neuron, R is the resistance of the LIF neuron, and I is the inflow current of the LIF neuron.
6. An automatic driving scene recognition apparatus, characterized by comprising:
the acquisition unit is used for acquiring an environment picture shot when the vehicle is driven;
the correcting unit is used for correcting the environment picture into a plane picture to obtain a plane environment picture;
the recognition unit is used for inputting the plane environment picture into a pulse convolution neural network hardware platform to obtain a scene recognition result corresponding to the environment picture, wherein a pulse convolution neural network carried in the pulse convolution neural network hardware platform is built by preset neurons, and the preset neurons comprise LIF neurons and/or IF neurons;
and the sending unit is used for sending the scene recognition result to an alarm so that the alarm performs corresponding alarm operation according to the scene recognition result.
7. The automatic driving scenario recognition apparatus of claim 6, wherein the preset neuron membrane potential threshold is determined by a genetic algorithm.
8. The automatic driving scenario recognition device of claim 6, wherein the pulse convolution neural network is trained from multiple data sets;
the plurality of data sets includes: GTSRB dataset, belgiumTSC dataset and CTSDB dataset.
9. The automatic driving scene recognition device of claim 6, wherein the input of the pulse convolution neural network is the encoded data of the planar environment picture;
the coding formula of the plane environment picture is as follows:
Figure FDA0003818020680000021
wherein P (K) is the probability of generating pulses for data, K is the number of pulses, λ T is the excitation frequency proportional to the pixel value, and the probability of not generating pulses within T time is e -λT 。
10. An automatic driving scenario recognition system, comprising: the system comprises a camera, an alarm and an automatic driving scene recognition device;
the camera is used for an environment picture when the vehicle is driven;
the automatic driving scene recognition device is used for correcting the environment picture into a plane picture to obtain a plane environment picture; the system is also used for inputting the plane environment picture to a pulse convolution neural network hardware platform to obtain a scene recognition result corresponding to the environment picture; the pulse convolution neural network is carried in the pulse convolution neural network hardware platform and is built by preset neurons, and the preset neurons comprise LIF neurons and/or IF neurons;
and the alarm is used for carrying out corresponding alarm operation according to the scene recognition result.
CN202211032500.4A 2022-08-26 2022-08-26 Automatic driving scene recognition method, device and system Pending CN115376104A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211032500.4A CN115376104A (en) 2022-08-26 2022-08-26 Automatic driving scene recognition method, device and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211032500.4A CN115376104A (en) 2022-08-26 2022-08-26 Automatic driving scene recognition method, device and system

Publications (1)

Publication Number Publication Date
CN115376104A true CN115376104A (en) 2022-11-22

Family

ID=84067167

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211032500.4A Pending CN115376104A (en) 2022-08-26 2022-08-26 Automatic driving scene recognition method, device and system

Country Status (1)

Country Link
CN (1) CN115376104A (en)

Similar Documents

Publication Publication Date Title
CN109754066B (en) Method and apparatus for generating a fixed-point neural network
CN110210563B (en) Image pulse data space-time information learning and identification method based on Spike cube SNN
US11521043B2 (en) Information processing apparatus for embedding watermark information, method, and computer readable storage medium
US11710044B2 (en) System and method for efficient evolution of deep convolutional neural networks using filter-wise recombination and propagated mutations
KR102545128B1 (en) Client device with neural network and system including the same
CN113449864B (en) Feedback type impulse neural network model training method for image data classification
CN110689109A (en) Neural network method and apparatus
Chowdhury et al. One timestep is all you need: Training spiking neural networks with ultra low latency
US11507829B2 (en) Greedy approach for obtaining an artificial intelligence model in a parallel configuration
Li et al. Efficient and accurate conversion of spiking neural network with burst spikes
Xia et al. Fully dynamic inference with deep neural networks
US11657285B2 (en) Methods, systems, and media for random semi-structured row-wise pruning in neural networks
Chowdhury et al. Towards ultra low latency spiking neural networks for vision and sequential tasks using temporal pruning
CN113396427A (en) Method and system for bit quantization for artificial neural networks
CN111105017A (en) Neural network quantization method and device and electronic equipment
US20200151551A1 (en) Systems and methods for determining an artificial intelligence model in a communication system
US20230376777A1 (en) System and method for efficient evolution of deep convolutional neural networks using filter-wise recombination and propagated mutations
CN115908909A (en) Evolutionary neural architecture searching method and system based on Bayes convolutional neural network
CN116844041A (en) Cultivated land extraction method based on bidirectional convolution time self-attention mechanism
CN113935475A (en) Simulation and training method of pulse neural network with pulse time offset
CN114819114A (en) Pulse neural network hardware accelerator and optimization method thereof in convolution operation
JPWO2021038793A1 (en) Learning systems, learning methods, and programs
US20200151584A1 (en) Systems and methods for determining an artificial intelligence model in a communication system
CN114766024A (en) Method and apparatus for pruning neural networks
CN110443347A (en) Neural network method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination