CN111160481B - Adas target detection method and system based on deep learning - Google Patents
Adas target detection method and system based on deep learning Download PDFInfo
- Publication number
- CN111160481B CN111160481B CN201911412209.8A CN201911412209A CN111160481B CN 111160481 B CN111160481 B CN 111160481B CN 201911412209 A CN201911412209 A CN 201911412209A CN 111160481 B CN111160481 B CN 111160481B
- Authority
- CN
- China
- Prior art keywords
- data
- model
- training
- adas
- data set
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 77
- 238000013135 deep learning Methods 0.000 title claims abstract description 29
- 238000012549 training Methods 0.000 claims abstract description 127
- 238000000034 method Methods 0.000 claims abstract description 55
- 238000013434 data augmentation Methods 0.000 claims abstract description 30
- 238000012545 processing Methods 0.000 claims abstract description 30
- 230000006835 compression Effects 0.000 claims abstract description 11
- 238000007906 compression Methods 0.000 claims abstract description 11
- 238000012795 verification Methods 0.000 claims description 24
- 238000004422 calculation algorithm Methods 0.000 claims description 19
- 230000004927 fusion Effects 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 15
- 238000012360 testing method Methods 0.000 claims description 14
- 230000003416 augmentation Effects 0.000 claims description 13
- 238000010586 diagram Methods 0.000 claims description 12
- 238000013139 quantization Methods 0.000 claims description 11
- 238000012216 screening Methods 0.000 claims description 5
- 230000009466 transformation Effects 0.000 claims description 5
- 230000003044 adaptive effect Effects 0.000 claims description 4
- 238000011002 quantification Methods 0.000 claims description 4
- 238000011426 transformation method Methods 0.000 claims description 4
- 239000000284 extract Substances 0.000 claims description 3
- 238000005282 brightening Methods 0.000 claims 2
- 230000000694 effects Effects 0.000 description 6
- 238000007689 inspection Methods 0.000 description 5
- 238000005457 optimization Methods 0.000 description 4
- 238000013138 pruning Methods 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000004821 distillation Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 238000000844 transformation Methods 0.000 description 2
- 102100034112 Alkyldihydroxyacetonephosphate synthase, peroxisomal Human genes 0.000 description 1
- 101000799143 Homo sapiens Alkyldihydroxyacetonephosphate synthase, peroxisomal Proteins 0.000 description 1
- 229940060587 alpha e Drugs 0.000 description 1
- 238000000848 angular dependent Auger electron spectroscopy Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000011423 initialization method Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000013526 transfer learning Methods 0.000 description 1
- 230000007306 turnover Effects 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
- 230000002087 whitening effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a adas target detection method and system based on deep learning, wherein the method comprises the following steps: acquiring road image data to form an initial dataset; establishing a data augmentation strategy, and performing expansion processing on the initial data set according to the data augmentation strategy to form an expanded data set; providing adas a dataset, combining the adas dataset, the extended dataset, and the initial dataset to form a training dataset; model training is carried out by utilizing the training data set so as to obtain a target detection model; and compression quantizing the object detection model to form an implantable model adapted to adas. The invention starts from adas data sets, combines the disclosed data sets with the collected data sets, so that adas data covers various domestic road conditions, and the problem of strong data pertinence is avoided.
Description
Technical Field
The invention relates to the technical field of target detection, in particular to a adas target detection method and system based on deep learning.
Background
The deep learning technology is increasingly applied to vehicle perception algorithms, and in the field of visual perception, technologies based on target detection and road surface segmentation are numerous, and a training process of the deep learning generally comprises: data enhancement and preprocessing, a brief description of which is provided below.
Because of the specificity of ADAS application scenes, the data acquisition and labeling needs to consider factors such as road conditions, weather, time, illumination and the like. The currently popular adas data processing methods basically collect specific data, then implement the expansion dataset by performing technologies such as color change, brightness adjustment, size scaling, clipping, data deflection or noise addition on the data, and then preprocess, such as normalization, whitening, etc., on the expanded dataset. One approach that is currently popular is wordTree, which can be fused in a variety of datasets to achieve the effect of covering different scenarios. However, the wordTree method has stronger pertinence of the data sets, unbalanced class distribution of samples, some data sets are aimed at logistics parks, more data sets are aimed at truck detection, some data sets are aimed at roads, more data sets are aimed at cars, forced combination can easily lead to model training and fitting, or the occurrence of objects can not be detected well.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, provides a adas target detection method and a adas target detection system based on deep learning, and solves the problem that model training is easy to be fitted due to stronger pertinence of a data set in the existing data enhancement processing.
The technical scheme for achieving the purpose is as follows:
The invention provides a adas target detection method based on deep learning, which comprises the following steps:
Acquiring road image data to form an initial dataset;
Establishing a data augmentation strategy, and performing expansion processing on the initial data set according to the data augmentation strategy to form an expanded data set;
Providing adas a dataset, combining the adas dataset, the extended dataset, and the initial dataset to form a training dataset;
model training is carried out by utilizing the training data set so as to obtain a target detection model; and
Compression quantification of the object detection model results in an implantable model that is adapted to adas.
The invention starts from adas data sets, combines the disclosed data sets with the collected data sets, so that adas data covers various domestic road conditions, and the problem of strong data pertinence is avoided. The invention also adopts data expansion processing, utilizes a data expansion strategy to expand data, realizes collection of a small amount of domestic road image data, and can well adapt the disclosed adas dataset to domestic roads.
The invention further improves the adas target detection method based on deep learning, which comprises the following steps:
establishing a plurality of data augmentation strategies;
Expanding a selected number of data sets in the initial data set by utilizing the established multiple data augmentation strategies to form a test set;
Performing model training by using the test set to obtain a corresponding test model;
Providing a verification set, verifying the verification model by using the verification set to obtain a verification result, and sorting from small to large according to the obtained verification result;
And selecting a data augmentation strategy with a front ordering, and performing expansion processing on the initial data set to form an expanded data set.
The invention further improves the adas target detection method based on deep learning, which is characterized by further comprising the following steps before the initial dataset is subjected to expansion processing:
carrying out sample category distribution statistics on the image data in each initial data set, and drawing a sample category distribution curve graph;
And selecting similar data sets or data set parts in the sample category distribution curve graph to fuse, or fusing the data set parts of the category to which the curve trend is consistent.
The invention further improves the adas target detection method based on deep learning, which is characterized by comprising the steps of utilizing the training data set to carry out model training to obtain a target detection model, and comprising the following steps of:
Training a model by adopting a training algorithm based on feature points, and extracting each image sample in the training data set;
Converting the image sample into a hotspot graph;
And taking the peak point of each hot spot in the hot spot diagram as input data of model training, taking the width information, the height information and the category information of each hot spot as output data, and performing model training to obtain a corresponding target detection model.
The invention further improves the adas target detection method based on deep learning, which comprises the following steps:
Establishing a lightweight network forward propagation frame;
Inputting the compression quantized implantable model which is formed and is adaptive to adas into the lightweight network forward propagation frame, and obtaining the implantable model of the lightweight frame by using the lightweight network forward propagation frame.
The invention also provides a adas target detection system based on deep learning, which comprises:
The acquisition module is used for acquiring road image data to form an initial data set;
the data expansion module is connected with the acquisition module and used for carrying out expansion processing on the initial data set according to the established data expansion strategy to form an expanded data set;
an acquisition module for acquiring adas datasets;
The model training module is connected with the acquisition module, the data expansion module and the acquisition module and is used for utilizing the adas data set, the expansion data set and the initial data set as training data sets and performing model training to obtain a target detection model; and
And the model quantization module is connected with the model training module and is used for compressing and quantizing the target detection model to form an implantable model which is suitable for adas.
The adas target detection system based on deep learning is further improved by further comprising a strategy screening module connected with the data expansion module and the acquisition module;
the strategy screening module is used for carrying out expansion processing on a selected number of data sets in the initial data set by utilizing the plurality of data expansion strategies to form a check set, carrying out model training by utilizing the check set to obtain a corresponding check model, carrying out verification on the check model by utilizing the check set to obtain a verification result, sorting from small to large according to the obtained verification result, and selecting the data expansion strategy with the front sorting to send to the data expansion module.
The adas target detection system based on deep learning is further improved by further comprising a data fusion module connected with the acquisition module;
The data fusion module is used for carrying out sample category distribution statistics on the image data in each initial data set and drawing a sample category distribution curve graph; and selecting similar data sets or data set parts in the sample category distribution curve graph to fuse, or fusing the data set parts of the category to which the curve trend is consistent.
The invention further improves the adas target detection system based on deep learning, wherein the model training module adopts a training algorithm based on characteristic points to perform model training, extracts each image sample in the training data set, converts the image sample into a hot spot diagram, takes the peak point of each hot spot in the hot spot diagram as input data of model training, takes the width information, the height information and the category information of each hot spot as output data, and performs model training to obtain a corresponding target detection model.
A further improvement of the depth learning-based adas target detection system of the present invention is that it further includes a lightweight network forward propagation framework coupled to the model quantification module for converting the compressed quantified implantable model adapted to adas into a lightweight framework implantable model.
Drawings
FIG. 1 is a flow chart of a method for detecting adas targets based on deep learning according to the present invention.
FIG. 2 is a system diagram of a depth learning based adas target detection system of the present invention.
Detailed Description
The invention will be further described with reference to the drawings and the specific examples.
Referring to fig. 1, the invention provides a adas target detection method and a system based on deep learning, which are used for solving the problems that a data set existing in wordTree in the existing data enhancement processing method is strong in pertinence, class distribution of templates is unbalanced, forced combination is easy to cause model training to be fitted, or an object cannot be monitored well. The adas target detection method and system designs a new data strategy, and solves the problem of unbalanced template distribution after fusion by calculating the type distribution of each dataset in advance. By adopting the skills of data expansion and the like, the disclosed adas data set is well applicable to the national roads by collecting a small amount of domestic data sets. The invention is also used for solving the problems that the existing data acquisition standards are different, the labeling levels are different, and the forced combination can cause a plurality of noises to be used as characteristics for learning, so that the difficulty of model training is further increased. And the problems that when the image is scaled or cut, a lot of image semantic information is lost or unnecessary noise is generated, and the expansibility of the image data is greatly reduced are solved. The invention designs 10 strategies for data augmentation, combines the data augmentation strategies with model training, reduces the time for specially carrying out data augmentation, and simultaneously adopts a multi-scale training mode to enable the model framework to be better suitable for data sets with different scales. The invention adopts the training algorithm based on the characteristic points, does not adopt the training algorithm based on the candidate frame, takes the position point of the target pixel as the key point of the target, directly sends the sample to the convolution network region to operate the pixel, thereby greatly reducing the operation amount of the algorithm and simultaneously reducing the resource consumption and the memory consumption. The method and the system for detecting adas targets based on deep learning are described below with reference to the accompanying drawings.
Referring to FIG. 2, a system diagram of the depth learning based adas target detection system of the present invention is shown. The following describes the object detection system adas based on deep learning of the present invention with reference to fig. 2.
As shown in fig. 2, the object detection system based on deep learning adas of the present invention includes an acquisition module 21, a data expansion module 22, an acquisition module 23, a model training module 24, and a model quantization module 25, where the acquisition module 21 is configured to acquire road image data to form an initial dataset; the data expansion module 22 is connected with the acquisition module 21, and is used for receiving the initial data set acquired by the acquisition module 21, and carrying out expansion processing on the initial data set according to the established data expansion strategy to form an expanded data set; the obtaining module 23 is configured to obtain adas a adas data set, which is a public existing data set, but many road conditions of which cannot meet the actual situation of the domestic road. The model training module 24 is connected with the acquisition module 21, the data expansion module 22 and the acquisition module 23, and is used for acquiring adas data sets, expansion data sets and initial data sets, taking adas books, expansion data sets and initial data sets as training data sets, and performing model training to obtain a target detection model; the model quantization module 25 is connected to the model training module 24 for performing a compression quantization process on the object detection model to form an implantable model adapted to adas.
In a specific embodiment of the present invention, the acquisition module 21 is a vehicle-mounted camera, and is configured to capture a road video of a vehicle running to form road image data, preferably the road image data is an actual scene of a domestic road, and the acquired road image data is fused with the adas dataset, so that the model training data can cover various domestic road conditions. The expansion of the existing adas data is realized.
In one embodiment of the present invention, the adas target detection system of the present invention further includes a data fusion module connected to the acquisition module 21; the data fusion module is used for carrying out sample category distribution statistics on the image data in each initial data set and drawing a sample category distribution curve graph; and selecting similar data sets or data set parts in the sample category distribution curve graph to fuse, or fusing the data set parts of the category to which the curve trend is consistent. The fusion of the data sets of the same class and the fusion of the data set parts are realized by utilizing the data fusion module, so that the problem of serious unbalance of the collected samples in the initial data set is solved. Specifically, each data set has the emphasis point, some data sets only comprise vehicles and pedestrians, no identification plates are marked, other data sets just mark identification plates, if forced fusion is carried out without processing, the identification degree of the identification plates is not strong, and serious unbalance of samples is caused. Because the subsequent model training adopts a training algorithm based on characteristic points (namely, training is carried out by adopting a pixel point method), the marking error can be not considered too much. And fusing the data sets of the category to which the curve trend is consistent for the data sets with dissimilar distribution curves, so that the data can be expanded in a targeted manner.
When judging whether the sample category distribution graphs are similar, dividing the sample distribution graphs according to the set interval to obtain a plurality of divided line segments, comparing whether the corresponding divided line segments are identical, calculating the duty ratio of the identical divided line segments, and judging that the two sample category distribution graphs are similar if the duty ratio is more than 85%, otherwise, the two sample category distribution graphs are dissimilar. Similarly, when judging whether or not the curve trends are identical, the two curves are overlapped correspondingly, and if the two curves are overlapped or have a part overlapped, the overlapped part is judged as the curve trend is identical, and the part is fused.
In one embodiment of the present invention, the adas target detection system of the present invention further includes a policy filtering module connected to the data expansion module 22 and the acquisition module 21; the strategy screening module is used for expanding a selected number of data sets in the initial data set by utilizing the plurality of data augmentation strategies to form a checking set, performing model training by utilizing the checking set to obtain a corresponding checking model, verifying the checking model by the checking set to obtain a verification result, sorting from small to large according to the obtained verification result, and selecting the data augmentation strategy with the front sorting to send to the data expansion module.
Preferably, 10 data augmentation strategies are established, 10 data augmentation strategies are used for carrying out expansion processing on a selected number of data sets in an initial data set to form an inspection set, the inspection set is used for carrying out model training to obtain corresponding inspection models, the inspection models are used for verifying to obtain loss of each inspection model, namely verification results, the obtained loss is sequenced from small to large, then the corresponding data augmentation strategies with smaller loss are selected for the first 5, the 5 selected data augmentation strategies are used for carrying out expansion processing on a comfortable data set to form an expanded data set, and during expansion processing, one of the 5 data augmentation strategies is randomly selected for each initial data set to carry out expansion processing so as to complete expansion of all the initial data sets.
Specifically, the 10 data augmentation strategies include: color augmentation method, transformation augmentation method, cut (deletion part) augmentation method, bbox internal transformation method, duck-filling method, SAMPLEPAIRING (sample pairing), mixMatch (super-strong semi-supervised learning), mixup, strategy based on search space, generated image and real image fusion; specifically, the color augmentation method adopts the modes of brightness adjustment, color balance adjustment, contrast adjustment and image blurring to process the image data to obtain an expanded new image, and the method is to artificially add some noise in the original image data. The change augmentation method is to adopt the methods of mirror image, rotation, turnover, scaling and clipping to carry out physical operation on the image data so as to enable the image data to have different transformations and obtain a new image of expansion processing. Cutout is to add one or more black dots to the sample image (i.e., image data). The Bbox internal transformation method is to perform 1 or 2 operations on the target in Bbox boxes. The duck-filling method is to scratch out some targets and then put the targets on a graph without targets, so that the robustness of the image is improved, the targets are transplanted, so that the method has certain randomness, the transplanted targets cannot be randomly placed at any position of the image, and a series of operations of the targets cannot be performed. SAMPLEPAIRING is to average the direct pixel addition of two pictures, and the label is unchanged, but in the training process, firstly, a common data augmentation mode of bdd k data sets is used to finish a plurality of epochs, and then, SAMPLEPAIRING is intermittently forbidden, and after a training loss function and precision are stable, SAMPLEPAIRING is forbidden to carry out fine adjustment. Noise is randomly introduced into the training samples, and guided training samples are artificially introduced into the training samples.
MixMatch is mainly used for preventing overfitting, so that the overfitting can be well represented on a larger data set, the method is mainly used for implicitly performing data augmentation in training, 1) one Batch is randomly selected from the data set to be marked as A, and the conventional augmentation is performed on the data of the Batch, but label is not changed; 2) Taking out a Batch of the same size as A, denoted as B, without considering the Batch label, then performing k times of random augmentation, preferably k is 2, and sending the augmented data to a simple classifier trained in advance, so as to calculate an average classification probability, and then processing the data by using a temperature Sharpen algorithm, wherein a guessed label of a Batch B sample can be obtained, and Sharpen algorithm is as follows:
Where T is a super-parameter (temperature) and p is the probability that the sample belongs to a label, and Sharpen algorithm helps to correct the model to give a low entropy judgment. 3) At this time, after the data of the Batch A is amplified, a certain label is provided, and after the second step of processing, k batches can be obtained, and the label of the Batch is predicted, so that the data can be seen to be amplified by the step, then the A and the k batches are randomly rearranged, one Batch is selected as C from the randomly rearranged data, the C and the original A are processed Mixup, and a Mixup method can be seen from the following description. 4) Then, the rearranged data is selected to be non-A and then subjected to Mixup, so that a new Batch D can be obtained, and the results of D and A and C mixup are denoted as u 'and x' in the following formula. Respectively calculating loss for the data sets:
Mixup field distribution, assuming that the model behaves linearly when processing the sample and the region between samples, this linear modeling reduces the inadaptability when data outside of the predictive training, the formula for Mixup is as follows:
wherein lambda Beta (alpha ), alpha E (0, +#)
where xi,xj are raw input vectors
where yi,yj are one-hot label ecodings
Mixup the super-parameter alpha controls the intensity of interpolation between feature-object vectors, and returns to ERM principle when alpha is 0.
Based on the strategy of the search space, the image enhancement strategy is defined as an unordered set of K sub-strategies, and the strategy is randomly selected from the unordered set to carry out data enhancement on the image during training. Each of these strategies contains N prior image transformations which in turn will act on the enhanced data, the purpose of the study being to search for the most efficient strategy. These variables together define a search space for a discrete optimization problem, setting k= 5,N =2 for data enhancement for the target detection task. Wherein the search space contains five sub-strategies in total, each sub-strategy contains 2 image operations, and each operation contains two parameters of probability p corresponding to the operation and specific value m of the operation. The probability-defined enhancement strategy here is random over the data samples, while m defines the magnitude of the enhancement.
The generated image is fused with the real image, and the generated image is added with shielding, environment and the like. Some of the above strategies may also be included.
Further, in the process of performing expansion processing, multi-scale training is added to dynamically adjust the size of the image, that is, the size of the image sample is not fixed, and after the expansion processing, a new picture size is randomly selected to perform training, so as to obtain each target feature (that is, the target in the sample) with different respective rates. Preferably, considering that the semantic information of the features at different levels is different, namely the semantic information of the shallow features is weaker, but the small target features are more obvious, the method is suitable for detecting the small target at the shallow layer, but is not suitable for detecting the large target at the shallow layer because the semantic features are weaker; the semantic information of the deep features is not strong, and for small targets, the features are basically disappeared, so that the method is suitable for large target detection in deep layers, the range of effective candidate frames is set, in shallow features, the candidate frames of the small targets are considered to be effective candidate frames, only the small targets are subjected to loss calculation, and targets with other scales are ignored; in deep features, the candidate frames of the large targets are considered as effective candidate frames, the loss calculation is only carried out on the large targets, and the sizes of the small targets are ignored, so that the best features can be extracted from the corresponding scale feature layers, and the detection accuracy is increased.
In a specific embodiment of the present invention, the model training module 24 performs model training by using a training algorithm based on feature points, extracts each image sample in the training data set, converts the image sample into a hotspot graph, takes a peak point of each hotspot in the hotspot graph as input data of the model training, takes width information, height information and category information of each hotspot as output data, and performs model training to obtain a corresponding target detection model.
The model training module 24 of the invention does not adopt algorithms such as yolov or ssd based on candidate frames, but adopts a training algorithm based on characteristic points, the training samples are directly sent into a convolution network to obtain heatmap of the training samples, then the peak point of heatmap is taken as a central point, and the position of each peak point of heatmap predicts the width and height information of the target, so thatBbox is target K (its category is C k), whose center point position is/>In addition, each target K regresses to obtain the size/>, of the targetTo reduce computational load, a single size is used for each target class to predict/>In addition, L1loss is added at the center point position. Here, the image size is not normalized, but the original pixel coordinates are directly used, so the loss function needs to be adjusted as follows: l det＝Lk+λsizeLsize+λoffLoff, where λ size＝0.1,λoff =1, the entire object detection model will output the width information and height information of the class and object at each location.
Preferably, when the model training module 24 performs model training, after each epoch (iteration number) is trained for each training data set, the training model is stored, and then an effect diagram test and a thermodynamic diagram test are performed for the stored model, so that the training situation can be known, and the training strategy can be adjusted at any time. The adas target detection system provided by the invention realizes real-time observation of the effect diagram selected by each epoch and timely adjustment.
The model training process is also a process of fusing the initial dataset, the extended dataset, and the adas dataset. After data fusion is completed, selecting a small amount of data in equal proportion, and performing trial training on the small amount of data, wherein the aim is to verify whether the initialization parameter selection of the model is proper or not; the model converges to adjust the learning rate strategy; selecting a proper optimization algorithm according to the convergence condition of the model; and verifying whether the training process is correct. The test training can be regarded as a debug mode, and the code is debugged through debug to judge whether the network structure is good or bad. In order to effectively evaluate the effect of the training model in time, after each epoch is trained, the training model is stored, and then an effect graph test and a thermodynamic diagram test are carried out on the stored model. Thus, the training situation can be known, and the training strategy can be adjusted at any time.
In a specific embodiment of the present invention, the adas target detection system of the present invention further includes a lightweight network forward propagation framework coupled to the model quantification module for converting the compressed quantified implantable model adapted to adas into an implantable model of the lightweight framework. Specifically, because of the different model training frames, there are some difficulties in transplanting the implantable model, and there are many frames that are unnecessary in the embedded computation in the training frames, for this case, only the implementation of summarizing the trained model in the forward propagation process is considered by using the lightweight network forward propagation frame, and since the backward propagation training process is removed, a large number of dependent files can be removed, and for the forward propagation implementation, including pre-selecting the model files to be loaded, inputting pre-processing, forward propagation of the network layer specific implementation, and finally outputting implementation. The pre-training model files are needed to be obtained by training on different frames, and then a developed conversion tool is obtained to convert models of different deep learning frames into models supported by a unified lightweight frame.
The model quantization module 25 carries out compression quantization on the target detection model, the model quantization module 25 adopts a mode of combining channel pruning, quantization and distillation algorithm to realize the slimming of the model, firstly, in the stage of channel pruning, two different methods are adopted to combine and judge the importance of a channel, the first is LASSO regression, namely, an L1 norm is added to restrict the weight, so that the weight is thinned; the second is to determine the importance of the filters based on the entropy value, convert the output of each layer into a vector with length c (number of filters) through a Global average Pooling, obtain a matrix with n x c for n images, divide each filter into m bins, count the probability of each bin, calculate its entropy value to determine the importance of the filter by using the entropy value, then crop the non-important filters, after calculating the entropy value of each channel, set a threshold for pruning, or set a constant compression rate, arrange the entropy values from large to small, and keep only the first k. Where n images are a subset of the training set or validation set. The two methods are independent of each other. The same channel (intersection or union) obtained by the two methods is taken for clipping. (if the same channel obtained by intersection is too few to reach the preset cutting rate), the quality of the cutting channel is ensured.
And then quantifying the obtained pruning model. The ownership weight of each layer is divided into a plurality of clusters, the center point of each cluster is found, then the number of the center point is represented by smaller bit information, and then the position and the value of each center point can be represented. Assuming n connections, each represented by b bits, and assuming k clusters, only log2 (k) bits are needed to represent the index, the compression ratio can be calculated:
Where nb is the total bits needed before no clustering, nlog 2 (k) +kb is the bits of the cluster index plus the bits needed for post-cluster connection.
The clustering method adopts a K-means method to determine the sharing weight of each layer, and the weight sharing in each cluster is carried out, and notice that the cross-layer sharing cannot be carried out and the weight sharing can only be carried out in the layers. The initialization method of the shared weight (the center of the initialization cluster) selects linear initialization, i.e., linear division between the minimum value and the maximum value of the original data is used as the initial cluster center. Then in the last training round, the gradient of each cluster center point is accumulated, multiplied by the learning rate and then subtracted from each cluster center point, and the cluster center is finely adjusted and updated.
Finally, a network distillation method is adopted in the aspect of recovering performance, and the method is based on a teacher-student network method and belongs to one of transfer learning. The teacher network is often a more complex network with very good performance and generalization capability, and can be used as a soft target to guide another simpler student network to learn, so that a simpler student model with less parameter operation can also have performance similar to the teacher network. Thus, the original fine-tune process is replaced, the model compressed in the first step can be further compressed, and the performance of the model is ensured to be basically unchanged.
The adas target detection system of the invention has the following beneficial effects:
The method is matched with a small amount of domestic data sets to collect the data expansion, the disclosed adas data sets are well applied to domestic roads, the consumption of resources and memory consumption can be greatly reduced by adopting a model training framework for key point detection, the detection rate can reach 23ms per frame before uncompressed quantization, and the detection precision of large, medium and small targets can reach map=0.577. After model compression, the detection accuracy is hardly lost, and the detection speed per frame can be up to within 20 ms.
The forward reasoning framework is not limited by the training framework, so that model conversion of all framework training is supported, and deployment and implementation are convenient; because the framework mainly enables forward propagation of the network. Less memory consumption is used, and special optimization is performed for hardware devices, and performance optimization can be performed specifically according to actual deployment devices.
The invention starts from adas data sets, fuses the disclosed data sets with the collected data sets, so that adas data covers various road conditions from home, adopts various data expansion strategies in the training process, and combines a pixel key point detection algorithm to design a new detection method. In the training process, parameter adjusting skills and training skills are used, and an effect diagram of each epoch training can be observed in real time, so that adjustment can be made in time. Finally, according to the trained detection characteristics, a compression quantization method combining various compression strategies is designed, and the method can effectively remove unnecessary weights, so that the model speed is improved, model parameters are reduced, and meanwhile, the precision is not lost too much. Finally, the compressed model is converted into a forward propagation framework suitable for embedded migration.
The invention also provides a adas target detection method based on deep learning, and the adas target detection method is described below.
The adas target detection method based on deep learning comprises the following steps:
As shown in fig. 1, step S11 is performed to acquire road image data to form an initial data set; step S12 is then performed;
Executing step S12, establishing a data augmentation strategy, and performing expansion processing on the initial data set according to the data augmentation strategy to form an expanded data set; step S13 is then executed;
performing step S13, providing adas datasets, combining adas datasets, extended datasets, and initial datasets to form a training dataset; step S14 is then performed;
executing step S14, performing model training by using the training data set to obtain a target detection model; step S15 is then executed;
step S15 is performed to compress and quantify the object detection model to form an implantable model adapted to adas.
The invention starts from adas data sets, combines the disclosed data sets with the collected data sets, so that adas data covers various domestic road conditions, and the problem of strong data pertinence is avoided. The invention also adopts data expansion processing, utilizes a data expansion strategy to expand data, realizes collection of a small amount of domestic road image data, and can well adapt the disclosed adas dataset to domestic roads.
In one embodiment of the present invention, the adas target detection method further includes:
establishing a plurality of data augmentation strategies;
Expanding a selected number of data sets in the initial data set by utilizing the established multiple data augmentation strategies to form a test set;
Model training is carried out by utilizing the test set so as to obtain a corresponding test model;
Providing a verification set, verifying the verification model by using the verification set to obtain a verification result, and sorting from small to large according to the obtained verification result;
And selecting a data augmentation strategy with a front ordering to perform expansion processing on the initial data set to form an expanded data set.
In a specific embodiment of the present invention, before the expanding processing is performed on the initial data set, the method further includes:
carrying out sample category distribution statistics on the image data in each initial data set, and drawing a sample category distribution curve graph;
And selecting similar data sets or data set parts in the sample category distribution curve graph to fuse, or fusing the data set parts of the category to which the curve trend is consistent.
In one embodiment of the present invention, the step of model training using the training dataset to obtain a target detection model comprises:
Training a model by adopting a training algorithm based on feature points, and extracting each image sample in a training data set;
Converting the image sample into a hotspot graph;
And taking the peak point of each hot spot in the hot spot diagram as input data of model training, taking the width information, the height information and the category information of each hot spot as output data, and performing model training to obtain a corresponding target detection model.
In one embodiment of the present invention, the adas target detection method further includes:
Establishing a lightweight network forward propagation frame;
The compression-quantized implantable model which is adaptive to adas is input into a lightweight network forward propagation frame, and the lightweight implantable model is obtained by using the lightweight network forward propagation frame.
The present invention has been described in detail with reference to the embodiments of the drawings, and those skilled in the art can make various modifications to the invention based on the above description. Accordingly, certain details of the illustrated embodiments are not to be taken as limiting the invention, which is defined by the appended claims.
Claims (6)
1. The adas target detection method based on deep learning is characterized by comprising the following steps of:
Acquiring road image data to form an initial dataset;
Establishing a data augmentation strategy, and performing expansion processing on the initial data set according to the data augmentation strategy to form an expanded data set;
Providing adas a dataset, combining the adas dataset, the extended dataset, and the initial dataset to form a training dataset;
model training is carried out by utilizing the training data set so as to obtain a target detection model; and
Compressing and quantifying the target detection model to form an implantable model adapted to adas;
Further comprises:
Establishing a plurality of data augmentation strategies; the data brightening strategies are 10, including a color augmentation method, a transformation augmentation method, a cutout augmentation method, a bbox internal transformation method, a duckling method and SAMPLEPAIRING, MIXMATCH, MIXUP, a strategy based on a search space, and fusion of a generated image and a real image;
Expanding a selected number of data sets in the initial data set by utilizing the established multiple data augmentation strategies to form a test set;
Performing model training by using the test set to obtain a corresponding test model;
Providing a verification set, verifying the verification model by using the verification set to obtain a verification result, and sorting from small to large according to the obtained verification result;
Selecting a data augmentation strategy with a front ordering to perform expansion processing on the initial data set to form an expanded data set; in the process of expansion processing, adding multi-scale training to dynamically adjust the size of an image, and after the expansion processing, randomly selecting a new picture size to train to obtain various target features with different resolutions;
before the initial data set is subjected to the expansion process, the method further comprises the following steps:
carrying out sample category distribution statistics on the image data in each initial data set, and drawing a sample category distribution curve graph;
And selecting similar data sets or data set parts in the sample category distribution curve graph to fuse, or fusing the data set parts of the category to which the curve trend is consistent.
2. The deep learning-based adas target detection method of claim 1, wherein the step of model training using the training dataset to obtain a target detection model comprises:
Training a model by adopting a training algorithm based on feature points, and extracting each image sample in the training data set;
Converting the image sample into a hotspot graph;
And taking the peak point of each hot spot in the hot spot diagram as input data of model training, taking the width information, the height information and the category information of each hot spot as output data, and performing model training to obtain a corresponding target detection model.
3. The deep learning-based adas target detection method of claim 1, further comprising:
Establishing a lightweight network forward propagation frame;
Inputting the compression quantized implantable model which is formed and is adaptive to adas into the lightweight network forward propagation frame, and obtaining the implantable model of the lightweight frame by using the lightweight network forward propagation frame.
4. A depth learning-based adas target detection system, comprising:
The acquisition module is used for acquiring road image data to form an initial data set;
the data expansion module is connected with the acquisition module and used for carrying out expansion processing on the initial data set according to the established data expansion strategy to form an expanded data set;
an acquisition module for acquiring adas datasets;
The model training module is connected with the acquisition module, the data expansion module and the acquisition module and is used for utilizing the adas data set, the expansion data set and the initial data set as training data sets and performing model training to obtain a target detection model; and
The model quantization module is connected with the model training module and is used for compressing and quantizing the target detection model to form an implantable model which is adaptive to adas;
The system also comprises a strategy screening module connected with the data expansion module and the acquisition module;
The strategy screening module is used for carrying out expansion processing on a selected number of data sets in the initial data set by utilizing the plurality of data expansion strategies to form a check set, carrying out model training by utilizing the check set to obtain a corresponding check model, carrying out verification on the check model by utilizing the check set to obtain a verification result, sorting from small to large according to the obtained verification result, and selecting the data expansion strategy with the front sorting to send to the data expansion module; the data brightening strategies are 10, including a color augmentation method, a transformation augmentation method, a cutout augmentation method, a bbox internal transformation method, a duckling method and SAMPLEPAIRING, MIXMATCH, MIXUP, a strategy based on a search space, and fusion of a generated image and a real image;
the system also comprises a data fusion module connected with the acquisition module;
The data fusion module is used for carrying out sample category distribution statistics on the image data in each initial data set and drawing a sample category distribution curve graph; and selecting similar data sets or data set parts in the sample category distribution curve graph to fuse, or fusing the data set parts of the category to which the curve trend is consistent.
5. The deep learning-based adas target detection system as claimed in claim 4, wherein the model training module performs model training by using a feature point-based training algorithm, extracts each image sample in the training data set, converts the image sample into a hotspot graph, takes a peak point of each hotspot in the hotspot graph as input data of the model training, takes width information, height information and category information of each hotspot as output data, and performs model training to obtain a corresponding target detection model.
6. The deep learning based adas target detection system of claim 4, further comprising a lightweight network forward propagation framework coupled to the model quantification module for converting the compressed quantified implantable model adapted to adas into a lightweight framework implantable model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911412209.8A CN111160481B (en) | 2019-12-31 | 2019-12-31 | Adas target detection method and system based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911412209.8A CN111160481B (en) | 2019-12-31 | 2019-12-31 | Adas target detection method and system based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111160481A CN111160481A (en) | 2020-05-15 |
| CN111160481B true CN111160481B (en) | 2024-05-10 |
Family
ID=70560287
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911412209.8A Active CN111160481B (en) | 2019-12-31 | 2019-12-31 | Adas target detection method and system based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111160481B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112163450A (en) * | 2020-08-24 | 2021-01-01 | 中国海洋大学 | Based on S3High-frequency ground wave radar ship target detection method based on D learning algorithm |
| CN112163238B (en) * | 2020-09-09 | 2022-08-16 | 中国科学院信息工程研究所 | Network model training method for multi-party participation data unshared |
| CN112598020A (en) * | 2020-11-24 | 2021-04-02 | 深兰人工智能(深圳)有限公司 | Target identification method and system |
| CN113111587A (en) * | 2021-04-20 | 2021-07-13 | 北京理工雷科电子信息技术有限公司 | Reusable and extensible machine learning method based on plug-in model |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105761306A (en) * | 2016-01-29 | 2016-07-13 | 珠海汇迪科技有限公司 | Road surface model based on field depth image or point cloud and establishment method thereof |
| CN206383949U (en) * | 2017-01-04 | 2017-08-08 | 江西沃可视发展有限公司 | Driving safety system based on the pure image procossings of ADAS |
| CN107316004A (en) * | 2017-06-06 | 2017-11-03 | 西北工业大学 | Space Target Recognition based on deep learning |
| WO2019196130A1 (en) * | 2018-04-12 | 2019-10-17 | 广州飒特红外股份有限公司 | Classifier training method and device for vehicle-mounted thermal imaging pedestrian detection |
| CN110414480A (en) * | 2019-08-09 | 2019-11-05 | 威盛电子股份有限公司 | Training image production method and electronic device |
-
2019
- 2019-12-31 CN CN201911412209.8A patent/CN111160481B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105761306A (en) * | 2016-01-29 | 2016-07-13 | 珠海汇迪科技有限公司 | Road surface model based on field depth image or point cloud and establishment method thereof |
| CN206383949U (en) * | 2017-01-04 | 2017-08-08 | 江西沃可视发展有限公司 | Driving safety system based on the pure image procossings of ADAS |
| CN107316004A (en) * | 2017-06-06 | 2017-11-03 | 西北工业大学 | Space Target Recognition based on deep learning |
| WO2019196130A1 (en) * | 2018-04-12 | 2019-10-17 | 广州飒特红外股份有限公司 | Classifier training method and device for vehicle-mounted thermal imaging pedestrian detection |
| CN110414480A (en) * | 2019-08-09 | 2019-11-05 | 威盛电子股份有限公司 | Training image production method and electronic device |
Non-Patent Citations (1)
| Title |
|---|
| 多任务深度学习框架在ADAS中的应用|分享总结;AI研习社;https://zhuanlan.zhihu.com/p/29816608;第1-8页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111160481A (en) | 2020-05-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111160481B (en) | Adas target detection method and system based on deep learning | |
| CN110458844B (en) | Semantic segmentation method for low-illumination scene | |
| CN110807385B (en) | Target detection method, target detection device, electronic equipment and storage medium | |
| CN108509978B (en) | Multi-class target detection method and model based on CNN (CNN) multi-level feature fusion | |
| CN111008562B (en) | Human-vehicle target detection method with feature map depth fusion | |
| CN108197326B (en) | Vehicle retrieval method and device, electronic equipment and storage medium | |
| CN111461083A (en) | Rapid vehicle detection method based on deep learning | |
| CN110807757B (en) | Image quality evaluation method and device based on artificial intelligence and computer equipment | |
| CN111008639B (en) | License plate character recognition method based on attention mechanism | |
| JP4098021B2 (en) | Scene identification method, apparatus, and program | |
| CN111861925A (en) | Image rain removing method based on attention mechanism and gate control circulation unit | |
| CN112528961B (en) | Video analysis method based on Jetson Nano | |
| CN116342894B (en) | GIS infrared feature recognition system and method based on improved YOLOv5 | |
| CN114255403A (en) | Optical remote sensing image data processing method and system based on deep learning | |
| CN114037640A (en) | Image generation method and device | |
| CN115035298A (en) | City streetscape semantic segmentation enhancement method based on multi-dimensional attention mechanism | |
| CN117011563A (en) | Road damage inspection cross-domain detection method and system based on semi-supervised federal learning | |
| CN113591545B (en) | Deep learning-based multi-level feature extraction network pedestrian re-identification method | |
| CN112132207A (en) | Target detection neural network construction method based on multi-branch feature mapping | |
| CN110490053B (en) | Human face attribute identification method based on trinocular camera depth estimation | |
| CN114567798B (en) | Tracing method for short video variety of Internet | |
| CN107341456B (en) | Weather sunny and cloudy classification method based on single outdoor color image | |
| CN115393802A (en) | Railway scene unusual invasion target identification method based on small sample learning | |
| CN115375672A (en) | Coal gangue detection method based on improved YOLOv4 algorithm | |
| CN115115847A (en) | Three-dimensional sparse reconstruction method and device and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |