CN111860334A - Cascade vehicle type classification method and device based on confusion matrix - Google Patents
Cascade vehicle type classification method and device based on confusion matrix Download PDFInfo
- Publication number
- CN111860334A CN111860334A CN202010708053.4A CN202010708053A CN111860334A CN 111860334 A CN111860334 A CN 111860334A CN 202010708053 A CN202010708053 A CN 202010708053A CN 111860334 A CN111860334 A CN 111860334A
- Authority
- CN
- China
- Prior art keywords
- vehicle type
- sample image
- initial
- network
- vehicle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
- G06V20/54—Surveillance or monitoring of activities, e.g. for recognising suspicious objects of traffic, e.g. cars on the road, trains or boats
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/08—Detecting or categorising vehicles
Abstract
The embodiment of the invention discloses a method and a device for classifying cascade vehicle types based on a confusion matrix. By applying the scheme provided by the embodiment of the invention, the vehicle type classification problem can be divided into two sub-problems by adopting the cascaded classification model, one is the classification problem of the vehicle type with larger characteristic difference, the other is the classification problem of the similar vehicle type, and aiming at the classification problem of the vehicle type with larger characteristic difference, the classification can be carried out by adopting the light-weight classifier with less parameters, so that the classification accuracy is ensured, and the classification efficiency can be improved; the classifier obtained through similar vehicle type training can be further adopted to accurately classify vehicle types contained in the similar set, namely the classification problem of vehicle types with higher similarity, so that the accuracy of vehicle type classification is ensured.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a method and a device for classifying a cascade vehicle type based on a confusion matrix.
Background
The problem of vehicle type classification is one of important automation technologies in automatic driving, intelligent transportation and security. Vehicle type classification is a problem of multi-level classification, and includes classification of vehicle brands and vehicle type classification.
The existing method generally adopts a deep learning-based method to classify vehicle types. The deep learning-based method is characterized in that a large-scale image classification network is trained to classify directly by collecting a large number of samples. However, in practical applications, the difference between some vehicle models is very small, and when the vehicle models are classified, a large amount of calculation amount and memory space are wasted by using a very large network, and the vehicle model classification efficiency is low. Therefore, in order to improve the efficiency of vehicle type classification, a vehicle type classification method is urgently needed.
Disclosure of Invention
The invention provides a method and a device for classifying cascade vehicle types based on a confusion matrix, which are used for improving the efficiency of vehicle type classification. The specific technical scheme is as follows.
In a first aspect, an embodiment of the present invention provides a method for classifying a cascade vehicle type based on a confusion matrix, where the method includes:
acquiring an image to be identified containing a target vehicle;
inputting the image to be recognized into a pre-trained primary classifier to obtain a candidate vehicle type category of the target vehicle; the primary classifier is obtained by training in advance according to each initial sample image and the vehicle type category corresponding to each initial sample image; the parameter number of the primary classifier is smaller than a preset value;
Acquiring a similarity set, and judging whether the candidate vehicle type category is contained in the similarity set; the similarity set is obtained according to a confusion matrix; the confusion matrix is obtained according to the initial vehicle type category of each initial sample image identified by the primary classifier; when i is not equal to j, the ith row and jth column elements in the confusion matrix represent the false recognition rate of the primary classifier for recognizing the initial sample images of the ith vehicle type as the jth vehicle type, wherein the false recognition rate is the number of the primary classifier for recognizing the initial sample images of the ith vehicle type as the jth vehicle type divided by the total number of the initial sample images of the ith vehicle type; the similar set stores a target sample image pair with the error identification rate larger than a preset identification threshold value in the initial sample images and a vehicle type category corresponding to each initial sample image in each target sample image pair;
when the candidate vehicle type category is not contained in the similar set, taking the candidate vehicle type category as a final vehicle type category of the target vehicle;
when the candidate vehicle type category is contained in the similar set, inputting the image to be recognized into a pre-trained secondary classifier to obtain a final vehicle type category of the target vehicle; the secondary classifier is obtained by sampling an initial sample image of an ith vehicle type in the target sample image pair in advance to obtain a first sample image, sampling an initial sample image of a jth vehicle type in the target sample image pair to obtain a second sample image, and training according to the first sample images, vehicle type categories corresponding to the first sample images, and vehicle type categories corresponding to the second sample images and the second sample images.
Optionally, the training process of the first-stage classifier includes:
constructing an initial convolutional neural network; the parameter quantity of the initial convolutional neural network is less than a preset value;
acquiring each initial sample image, and determining the vehicle type category corresponding to each initial sample image;
and inputting the initial sample images and the vehicle type categories corresponding to the initial sample images into the initial convolutional neural network, and taking the current initial convolutional neural network as the primary classifier when the loss function change value of the initial convolutional neural network is smaller than a first threshold value.
Optionally, the initial convolutional neural network includes mobileNet or shuffleNet; the loss function is:
n is the total number of vehicle type categories, and when the obtained identification result is correct, y isnIs 1, otherwise is 0; snA preferred a posteriori probability output for said initial convolutional neural network:
akestimating the estimated vehicle type class k distance, a, of the initial convolutional neural networknAnd estimating the distance with the vehicle type n for the initial convolutional neural network.
Optionally, the method further includes:
obtaining a recognition result of the vehicle type recognition of the initial sample image by the primary classifier;
Constructing a confusion matrix according to the identification result; the element of the ith row and the jth column in the confusion matrix is the number of the initial sample images of the ith type of vehicle recognized by the primary classifier as the jth type of vehicle divided by the total number of the initial sample images of the ith type of vehicle; the dimension of the confusion matrix is N x N, and N is the total number of vehicle type categories;
and taking the initial sample image of the ith vehicle type and the initial sample image of the jth vehicle type corresponding to the element larger than the preset identification threshold value in the confusion matrix as a target sample image pair, and storing the vehicle type category corresponding to each target sample image pair, the initial sample image of the ith vehicle type in each target sample image pair and the vehicle type category corresponding to the initial sample image of the jth vehicle type in a similar set.
Optionally, the method further includes:
acquiring two primary classifiers respectively serving as a first network and a second network;
sampling initial sample images of ith vehicle type in the target sample image pair to obtain first sample images, sampling initial sample images of jth vehicle type in the target sample image pair to obtain second sample images, and acquiring vehicle type categories corresponding to the first sample images and the second sample images from the similarity set;
Inputting each first sample image and the corresponding vehicle type category into the first network, inputting each second sample image and the corresponding vehicle type category into the second network, calculating a value of an objective function according to the recognition results of the first network and the second network, and taking the current first network or the second network as the secondary classifier when the variation value of the objective function is smaller than a second threshold value.
Optionally, the objective function is:
wherein y is the correct result when the first network or the second network obtains the correct resultnIs 1, otherwise is 0; b isnA preferred a posteriori probability output for said first network or said second network:
aka distance, a, of vehicle type class k estimated for the first network or the second networknEstimating a distance with a vehicle type n for the first network or the second network;
vi is the eigenvector output by the topmost layer of the first network, Vi is (x1, x2, …, xn), Vj is the eigenvector output by the topmost layer of the second network, Vj is (y1, y2, …, yn), and n is the dimension of the eigenvector; λ is a constant;
in a second aspect, an embodiment of the present invention provides a cascade vehicle type classification apparatus based on a confusion matrix, where the apparatus includes:
The image acquisition module is used for acquiring an image to be identified containing a target vehicle;
the primary classification module is used for inputting the image to be recognized into a primary classifier obtained by pre-training to obtain the candidate vehicle type category of the target vehicle; the primary classifier is obtained by training in advance according to each initial sample image and the vehicle type category corresponding to each initial sample image; the parameter number of the primary classifier is smaller than a preset value;
the category judgment module is used for acquiring a similarity set and judging whether the category of the candidate vehicle type is contained in the similarity set; the similarity set is obtained according to a confusion matrix; the confusion matrix is obtained according to the initial vehicle type category of each initial sample image identified by the primary classifier; when i is not equal to j, the ith row and jth column elements in the confusion matrix represent the false recognition rate of the primary classifier for recognizing the initial sample images of the ith vehicle type as the jth vehicle type, wherein the false recognition rate is the number of the primary classifier for recognizing the initial sample images of the ith vehicle type as the jth vehicle type divided by the total number of the initial sample images of the ith vehicle type; the similar set stores a target sample image pair with the error identification rate larger than a preset identification threshold value in the initial sample images and a vehicle type category corresponding to each initial sample image in each target sample image pair;
A vehicle type determining module, configured to, when the category determining module determines that the category of the candidate vehicle type is not included in the similarity set, take the category of the candidate vehicle type as a final vehicle type category of the target vehicle;
the secondary classification module is used for inputting the image to be recognized into a secondary classifier obtained by pre-training when the category judgment module determines that the category of the candidate vehicle type is contained in the similarity set, so as to obtain the final vehicle type category of the target vehicle; the secondary classifier is obtained by sampling an initial sample image of an ith vehicle type in the target sample image pair in advance to obtain a first sample image, sampling an initial sample image of a jth vehicle type in the target sample image pair to obtain a second sample image, and training according to the first sample images, vehicle type categories corresponding to the first sample images, and vehicle type categories corresponding to the second sample images and the second sample images.
Optionally, the apparatus further comprises:
the initial network construction module is used for constructing an initial convolutional neural network; the parameter quantity of the initial convolutional neural network is less than a preset value;
the system comprises a sample image acquisition module, a model classification determination module and a model classification determination module, wherein the sample image acquisition module is used for acquiring each initial sample image and determining the model classification corresponding to each initial sample image;
And the primary classifier training module is used for inputting the initial sample images and the vehicle type categories corresponding to the initial sample images into the initial convolutional neural network, and when the loss function change value of the initial convolutional neural network is smaller than a first threshold value, taking the current initial convolutional neural network as the primary classifier.
Optionally, the initial convolutional neural network includes mobileNet or shuffleNet; the loss function is:
n is the total number of vehicle type categories, and when the obtained identification result is correct, y isnIs 1, otherwise is 0; snA preferred a posteriori probability output for said initial convolutional neural network:
akestimating the estimated vehicle type class k distance, a, of the initial convolutional neural networknAnd estimating the distance with the vehicle type n for the initial convolutional neural network.
Optionally, the apparatus further comprises:
the result acquisition module is used for acquiring the recognition result of the primary classifier for vehicle type recognition of the initial sample image;
the matrix construction module is used for constructing a confusion matrix according to the identification result; the element of the ith row and the jth column in the confusion matrix is the number of the initial sample images of the ith type of vehicle recognized by the primary classifier as the jth type of vehicle divided by the total number of the initial sample images of the ith type of vehicle; the dimension of the confusion matrix is N x N, and N is the total number of vehicle type categories;
And the similarity set construction module is used for taking the initial sample image of the ith vehicle type and the initial sample image of the jth vehicle type corresponding to the element, which is greater than the preset identification threshold value, in the confusion matrix as a target sample image pair, and storing the vehicle type category corresponding to each target sample image pair, the initial sample image of the ith vehicle type in each target sample image pair and the vehicle type category corresponding to the initial sample image of the jth vehicle type into the similarity set.
Optionally, the apparatus further comprises:
the network acquisition module is used for acquiring the two primary classifiers respectively as a first network and a second network;
the sample sampling module is used for sampling an initial sample image of an ith vehicle type in the target sample image pair to obtain a first sample image, sampling an initial sample image of a jth vehicle type in the target sample image pair to obtain a second sample image, and acquiring vehicle type categories corresponding to the first sample images and the second sample images from the similarity set;
and the secondary classifier training module is used for inputting each first sample image and the corresponding vehicle type category into the first network, inputting each second sample image and the corresponding vehicle type category into the second network, calculating the value of an objective function according to the recognition results of the first network and the second network, and taking the current first network or the second network as the secondary classifier when the change value of the objective function is smaller than a second threshold value.
Optionally, the objective function is:
wherein y is the correct result when the first network or the second network obtains the correct resultnIs 1, otherwise is 0; b isnA preferred a posteriori probability output for said first network or said second network:
aka distance, a, of vehicle type class k estimated for the first network or the second networknEstimating a distance with a vehicle type n for the first network or the second network;
vi is the eigenvector output by the topmost layer of the first network, Vi is (x1, x2, …, xn), Vj is the eigenvector output by the topmost layer of the second network, Vj is (y1, y2, …, yn), and n is the dimension of the eigenvector; λ is a constant;
as can be seen from the above, the method and the device for classifying the cascading vehicle type based on the confusion matrix provided by the embodiment of the invention can adopt the cascading classification model to divide the vehicle type classification problem into two sub-problems, one is the classification problem of the vehicle type with larger characteristic difference, the other is the classification problem of the similar vehicle type, and aiming at the classification problem of the vehicle type with larger characteristic difference, the classification can be carried out by adopting the lightweight classifier with less parameters, so that the classification accuracy is ensured, and the classification efficiency can be improved; the classifier obtained through similar vehicle type training can be further adopted to accurately classify vehicle types contained in the similar set, namely the classification problem of vehicle types with higher similarity, so that the accuracy of vehicle type classification is ensured. Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
The innovation points of the embodiment of the invention comprise:
1. the classification problem of the vehicle type is divided into two sub-problems by adopting a cascading classification model, one is the classification problem of the vehicle type with larger characteristic difference, the other is the classification problem of the similar vehicle type, aiming at the classification problem of the vehicle type with larger characteristic difference, a lightweight classifier with less parameters can be adopted for classification, the classification accuracy is ensured, and the classification efficiency can be improved; the classifier obtained through similar vehicle type training can be further adopted to accurately classify vehicle types contained in the similar set, namely the classification problem of vehicle types with higher similarity, so that the accuracy of vehicle type classification is ensured.
2. Through the lightweight network with less training parameters, the classification efficiency can be improved while the classification accuracy is ensured when classifying the vehicle types with larger differences.
3. The method comprises the steps of establishing a confusion matrix according to the recognition result of an initial sample image by a primary classifier, determining a vehicle type with higher similarity according to the size of each parameter value in the confusion matrix, establishing a sample image containing the vehicle type with higher similarity and a similarity set of vehicle type categories, training the sample image based on the similarity set to obtain a secondary classifier, classifying the vehicle type with higher similarity by the secondary classifier when classifying the vehicle type, and improving the accuracy of vehicle type classification.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is to be understood that the drawings in the following description are merely exemplary of some embodiments of the invention. For a person skilled in the art, without inventive effort, further figures can be obtained from these figures.
Fig. 1 is a schematic flow chart of a cascade vehicle type classification method based on a confusion matrix according to an embodiment of the present invention.
Fig. 2 is another schematic flow chart of the cascade vehicle type classification method based on the confusion matrix according to the embodiment of the present invention.
Fig. 3 is another schematic flow chart of the cascade vehicle type classification method based on the confusion matrix according to the embodiment of the present invention.
Fig. 4 is a schematic structural diagram of a cascade vehicle type classification apparatus based on a confusion matrix according to an embodiment of the present invention.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. It is to be understood that the described embodiments are merely a few embodiments of the invention, and not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without inventive effort based on the embodiments of the present invention, are within the scope of the present invention.
It is to be noted that the terms "comprises" and "comprising" and any variations thereof in the embodiments and drawings of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
The embodiment of the invention discloses a method and a device for classifying cascade vehicle types based on a confusion matrix, which can improve the vehicle type classification efficiency. The following provides a detailed description of embodiments of the invention.
Fig. 1 is a schematic flow chart of a cascade vehicle type classification method based on a confusion matrix according to an embodiment of the present invention. The method is applied to the electronic equipment. The method specifically comprises the following steps.
S110: and acquiring an image to be identified containing the target vehicle.
The image to be identified may be an image collected by a monitoring device. After the monitoring equipment acquires the image, the image can be sent to the electronic equipment; or, the electronic device may send an image acquisition request to the monitoring device, and after receiving the image acquisition request, the monitoring device may send the acquired image to the electronic device.
S120: inputting the image to be recognized into a primary classifier obtained by pre-training to obtain the candidate vehicle type category of the target vehicle; the primary classifier is obtained by training in advance according to each initial sample image and the vehicle type category corresponding to each initial sample image; the number of parameters of the first-stage classifier is less than a preset value.
In the embodiment of the invention, in order to improve the license plate recognition efficiency on the basis of ensuring the license plate recognition accuracy, a primary classifier and a secondary classifier can be obtained by pre-training. The first-stage classifier can be a lightweight network with few parameters and is mainly used for identifying vehicle type categories with large characteristic differences; the secondary classifier can be a large-scale network and is used for identifying the vehicle type categories with small characteristic difference and ensuring the accuracy of vehicle type classification.
After a primary classifier is obtained through pre-training, when vehicle type classification is carried out, after an image to be recognized is obtained, the image to be recognized can be input into the primary classifier, and candidate vehicle type classes of target vehicles are obtained.
S130: acquiring a similarity set, and judging whether the type of the candidate vehicle type is contained in the similarity set; obtaining a similarity set according to the confusion matrix; the confusion matrix is obtained according to the initial vehicle type category of each initial sample image identified by the primary classifier; when i is not equal to j, the ith row and jth column elements in the confusion matrix represent the false recognition rate of the primary classifier for recognizing the initial sample images of the ith vehicle type as the jth vehicle type, and the false recognition rate is the number of the primary classifier for recognizing the initial sample images of the ith vehicle type as the jth vehicle type divided by the total number of the initial sample images of the ith vehicle type; and the similar set stores a target sample image pair with the error identification rate larger than a preset identification threshold value in the initial sample images and a vehicle type category corresponding to each initial sample image in each target sample image pair.
In the embodiment of the invention, after the first-stage classifier is obtained through training, the confusion matrix can be obtained based on the initial vehicle type class of each initial sample image identified by the first-stage classifier. The similarity between the vehicle types in the current classification task can be determined through the confusion matrix, the probability of sample resampling in the network training process can be adjusted through the similarity, the strength of sample feature learning of similar vehicle types can be improved, meanwhile, the existing pure classification loss is improved through the identification loss, and the classification accuracy can be effectively improved.
When i is not equal to j, the ith row and jth column elements in the confusion matrix represent the false recognition rate of the primary classifier for recognizing the initial sample image of the ith vehicle type as the jth vehicle type, and when i is equal to j, the ith row and jth column elements in the confusion matrix represent the accuracy of the primary classifier for recognizing the initial sample image of the ith vehicle type.
And constructing a similarity set through the confusion matrix, wherein the stored vehicle type category is a vehicle type which is easy to be identified by mistake.
S140: and when the candidate vehicle type category is not contained in the similar set, taking the candidate vehicle type category as the final vehicle type category of the target vehicle.
When the candidate vehicle type category is not contained in the similar set, the vehicle type category of the target vehicle in the image to be recognized is indicated to be greatly different from other vehicle type categories. In this case, the candidate vehicle type obtained by the primary classifier is the accurate vehicle type, and in order to improve the vehicle type classification efficiency, the candidate vehicle type obtained by the primary classifier can be directly used as the final vehicle type of the target vehicle.
S150: when the candidate vehicle type category is contained in the similar set, inputting the image to be recognized into a secondary classifier obtained by pre-training to obtain the final vehicle type category of the target vehicle; the secondary classifier is obtained by sampling an initial sample image of an ith vehicle type in a target sample image pair in advance to obtain a first sample image, sampling an initial sample image of a jth vehicle type in the target sample image pair to obtain a second sample image, and training according to the first sample images, vehicle type categories corresponding to the first sample images and vehicle type categories corresponding to the second sample images and the second sample images.
When the candidate vehicle type category is included in the similarity set, it indicates that there are other vehicle type categories that are less different from the vehicle type category of the target vehicle in the image to be recognized. In this case, in order to improve the accuracy of vehicle type classification, the vehicle type category of the target vehicle in the image to be recognized may be further recognized.
In the embodiment of the invention, the secondary classifier can be obtained according to the similar set training when the network training is carried out. The vehicle type category stored in the similar set is the vehicle type which is easy to be identified by mistake, so the secondary classifier obtained by training by using the image in the similar set as a sample can accurately identify the vehicle type with smaller difference.
When vehicle type classification is performed, when the candidate vehicle type is included in the similar set, the image to be recognized can be input into the secondary classifier, and the final vehicle type of the target vehicle is obtained.
As can be seen from the above, in the embodiment, the vehicle type classification problem can be split into two sub-problems by using a cascaded classification model, one is the classification problem of the vehicle type with a large characteristic difference, the other is the classification problem of the similar vehicle type, and for the classification problem of the vehicle type with a large characteristic difference, a lightweight classifier with less parameters can be used for classification, so that the classification accuracy is ensured, and the classification efficiency can be improved; the classifier obtained through similar vehicle type training can be further adopted to accurately classify vehicle types contained in the similar set, namely the classification problem of vehicle types with higher similarity, so that the accuracy of vehicle type classification is ensured.
In one implementation, as shown in fig. 2, the training process of the above-mentioned primary classifier may include the following steps.
S210: constructing an initial convolutional neural network; the number of parameters of the initial convolutional neural network is less than a preset value.
The initial convolutional neural network may employ a common convolutional neural network structure, such as AlexNet, GoogleNet, or ResNet. In an implementation manner, in consideration of requirements on model operation speed and memory in practical application, the initial convolutional neural network may adopt a lightweight network, such as a mobileNet, a shuffenet, and the like, which is sufficient for solving a classification problem with large feature difference, and has a fast operation speed and a small number of model parameters.
S220: and acquiring each initial sample image, and determining the vehicle type category corresponding to each initial sample image.
The initial sample image may be an image acquired by a monitoring device. In order to ensure the accuracy of network training, the number of initial sample images may be large enough, such as 10 thousands, 15 thousands, 20 thousands, etc. When the vehicle type category corresponding to each initial sample image is determined, the vehicle type category can be determined manually.
S230: and inputting each initial sample image and the vehicle type category corresponding to each initial sample image into an initial convolutional neural network, and taking the current initial convolutional neural network as a primary classifier when the loss function change value of the initial convolutional neural network is smaller than a first threshold value.
After each initial sample image and the vehicle type category corresponding to each initial sample image are input into the initial convolutional neural network, the initial convolutional neural network can classify the vehicle type of each initial sample image, compare the vehicle type with the vehicle type category corresponding to each input initial sample image, determine the accuracy of classification, and calculate to obtain the value of the corresponding loss function. It can be understood that, as the network is continuously optimized, the calculated loss function value becomes smaller and smaller, and no obvious change occurs. At this time, it can be determined that the network training is completed, and the current initial convolutional neural network can be used as a first-stage classifier.
In one implementation, the loss function may be:
n is the total number of vehicle type categories, and when the obtained identification result is correct, y isnIs 1, otherwise is 0; snPreferred posterior probabilities for the initial convolutional neural network output:
akthe estimated vehicle type class is k distance, a, for the initial convolutional neural networknVehicle type classification estimated for initial convolutional neural networkIs a distance of n.
Through the lightweight network with less training parameters, the classification efficiency can be improved while the classification accuracy is ensured when classifying the vehicle types with larger differences.
After the training of the first-level classifier is finished, all training samples can be classified, the recognition result and the real result of each initial sample image are recorded, and a confusion matrix can be constructed. Assuming that the vehicle types involved in the model system are of the N type, the confusion matrix is a matrix of N x N, the elements h of whichijRepresenting the probability that the sample of the ith type of vehicle is mistakenly identified as the jth type of vehicle; and when i is the same as j, indicating the accuracy of the vehicle type identification. h isijAnd the number of samples which are misidentified as the jth vehicle type for the ith vehicle type is divided by the number of samples of the ith vehicle type. h isijThe smaller the vehicle type is, the larger the difference between the ith vehicle type and the jth vehicle type is, so that the classification errors are not easy to occur; otherwise, h ijThe larger the vehicle model is, the greater the similarity between the two types of vehicle models is.
Different classification models result in different confusion matrices. In the embodiment of the invention, the confusion matrix can be calculated by adopting the classification result of the first-stage classifier, so that the first-stage classifier and the second-stage classifier can be effectively connected and more effectively complemented.
After the confusion matrix is obtained, the initial sample image of the ith vehicle type and the initial sample image of the jth vehicle type corresponding to the element in the confusion matrix which is larger than the preset identification threshold value can be used as a target sample image pair, and the vehicle type category corresponding to the initial sample image of the ith vehicle type and the vehicle type category corresponding to the initial sample image of the jth vehicle type in each target sample image pair and each target sample image pair are stored into a similar set S, wherein the vehicle type elements in the S are different. That is, the sample images and vehicle type categories corresponding to vehicle types with high similarity may be stored in the similarity set.
When the secondary network is trained, the training samples of the two categories can be sampled from the target sample images of the similarity set at the same time, and a twin network method can be adopted for training.
In one implementation, as shown in FIG. 3, the training process of the secondary classifier may include the following steps.
S310: and acquiring two primary classifiers as a first network and a second network respectively.
The primary classifier is used as an initial network, so that the training time of the secondary classifier can be shortened, and the training efficiency is improved.
S320: sampling initial sample images of ith vehicle type in the target sample image pair to obtain first sample images, sampling initial sample images of jth vehicle type in the target sample image pair to obtain second sample images, and acquiring vehicle type categories corresponding to the first sample images and the second sample images from the similarity set.
The number of the first sample images and the number of the second sample images may be the same or different, and the embodiment of the present invention does not limit this.
S330: and inputting each first sample image and the corresponding vehicle type category into a first network, inputting each second sample image and the corresponding vehicle type category into a second network, calculating the value of an objective function according to the recognition results of the first network and the second network, and taking the current first network or the second network as a secondary classifier when the variation value of the objective function is smaller than a second threshold value.
After each first sample image and the corresponding vehicle type category are input into a first network, the first network can classify the vehicle type of each first sample image and compare the vehicle type with the vehicle type category corresponding to each input first sample image; after each second sample image and the corresponding vehicle type category are input into a second network, the second network can classify the vehicle type of each second sample image and compare the vehicle type with the vehicle type category corresponding to each input second sample image; and calculating to obtain the value of the corresponding objective function according to the recognition results of the first network and the second network. It can be understood that, as the network is continuously optimized, the calculated value of the objective function becomes smaller and smaller, and no obvious change occurs. At this time, it can be determined that the network training is completed, and the current first network or second network can be used as a secondary classifier because the first network and the second network are the same.
The objective function may be:
wherein, when the first network or the second network obtains the correct identification result, ynIs 1, otherwise is 0; b isnPreferred a posteriori probabilities output for the first network or the second network:
akestimated vehicle type class k distance, a, for the first network or the second networknAnd estimating the distance with the vehicle type class n for the first network or the second network.
Vi is the eigenvector output by the topmost layer of the first network, Vi is (x1, x2, …, xn), Vj is the eigenvector output by the topmost layer of the second network, Vj is (y1, y2, …, yn), and n is the dimension of the eigenvector; λ is a constant;
the method comprises the steps of establishing a confusion matrix according to the recognition result of an initial sample image by a primary classifier, determining a vehicle type with higher similarity according to the size of each parameter value in the confusion matrix, establishing a sample image containing the vehicle type with higher similarity and a similarity set of vehicle type categories, training the sample image based on the similarity set to obtain a secondary classifier, classifying the vehicle type with higher similarity by the secondary classifier when classifying the vehicle type, and improving the accuracy of vehicle type classification.
As shown in fig. 4, a schematic structural diagram of a cascade vehicle type classification apparatus based on a confusion matrix according to an embodiment of the present invention is shown, where the apparatus includes:
An image acquisition module 410, configured to acquire an image to be identified including a target vehicle;
the primary classification module 420 is configured to input the image to be recognized into a pre-trained primary classifier, so as to obtain a candidate vehicle type category of the target vehicle; the primary classifier is obtained by training in advance according to each initial sample image and the vehicle type category corresponding to each initial sample image; the parameter number of the primary classifier is smaller than a preset value;
a category judgment module 430, configured to obtain a similarity set, and judge whether the category of the candidate vehicle type is included in the similarity set; the similarity set is obtained according to a confusion matrix; the confusion matrix is obtained according to the initial vehicle type category of each initial sample image identified by the primary classifier; when i is not equal to j, the ith row and jth column elements in the confusion matrix represent the false recognition rate of the primary classifier for recognizing the initial sample images of the ith vehicle type as the jth vehicle type, wherein the false recognition rate is the number of the primary classifier for recognizing the initial sample images of the ith vehicle type as the jth vehicle type divided by the total number of the initial sample images of the ith vehicle type; the similar set stores a target sample image pair with the error identification rate larger than a preset identification threshold value in the initial sample images and a vehicle type category corresponding to each initial sample image in each target sample image pair;
A vehicle type determining module 440, configured to, when the category determining module 430 determines that the candidate vehicle type category is not included in the similarity set, take the candidate vehicle type category as a final vehicle type category of the target vehicle;
a secondary classification module 450, configured to, when the category determination module 430 determines that the category of the candidate vehicle is included in the similarity set, input the image to be recognized into a secondary classifier obtained through pre-training, so as to obtain a final vehicle type category of the target vehicle; the secondary classifier is obtained by sampling an initial sample image of an ith vehicle type in the target sample image pair in advance to obtain a first sample image, sampling an initial sample image of a jth vehicle type in the target sample image pair to obtain a second sample image, and training according to the first sample images, vehicle type categories corresponding to the first sample images, and vehicle type categories corresponding to the second sample images and the second sample images.
Optionally, the apparatus further comprises:
the initial network construction module is used for constructing an initial convolutional neural network; the parameter quantity of the initial convolutional neural network is less than a preset value;
The system comprises a sample image acquisition module, a model classification determination module and a model classification determination module, wherein the sample image acquisition module is used for acquiring each initial sample image and determining the model classification corresponding to each initial sample image;
and the primary classifier training module is used for inputting the initial sample images and the vehicle type categories corresponding to the initial sample images into the initial convolutional neural network, and when the loss function change value of the initial convolutional neural network is smaller than a first threshold value, taking the current initial convolutional neural network as the primary classifier.
Optionally, the initial convolutional neural network includes mobileNet or shuffleNet; the loss function is:
n is the total number of vehicle type categories, and when the obtained identification result is correct, y isnIs 1, otherwise is 0; snA preferred a posteriori probability output for said initial convolutional neural network:
akestimating the estimated vehicle type class k distance, a, of the initial convolutional neural networknAnd estimating the distance with the vehicle type n for the initial convolutional neural network.
Optionally, the apparatus further comprises:
the result acquisition module is used for acquiring the recognition result of the primary classifier for vehicle type recognition of the initial sample image;
the matrix construction module is used for constructing a confusion matrix according to the identification result; the element of the ith row and the jth column in the confusion matrix is the number of the initial sample images of the ith type of vehicle recognized by the primary classifier as the jth type of vehicle divided by the total number of the initial sample images of the ith type of vehicle; the dimension of the confusion matrix is N x N, and N is the total number of vehicle type categories;
And the similarity set construction module is used for taking the initial sample image of the ith vehicle type and the initial sample image of the jth vehicle type corresponding to the element, which is greater than the preset identification threshold value, in the confusion matrix as a target sample image pair, and storing the vehicle type category corresponding to each target sample image pair, the initial sample image of the ith vehicle type in each target sample image pair and the vehicle type category corresponding to the initial sample image of the jth vehicle type into the similarity set.
Optionally, the apparatus further comprises:
the network acquisition module is used for acquiring the two primary classifiers respectively as a first network and a second network;
the sample sampling module is used for sampling an initial sample image of an ith vehicle type in the target sample image pair to obtain a first sample image, sampling an initial sample image of a jth vehicle type in the target sample image pair to obtain a second sample image, and acquiring vehicle type categories corresponding to the first sample images and the second sample images from the similarity set;
and the secondary classifier training module is used for inputting each first sample image and the corresponding vehicle type category into the first network, inputting each second sample image and the corresponding vehicle type category into the second network, calculating the value of an objective function according to the recognition results of the first network and the second network, and taking the current first network or the second network as the secondary classifier when the change value of the objective function is smaller than a second threshold value.
Optionally, the objective function is:
wherein when the first network or the second network gets the identification nodeWhen the fruit is correct, ynIs 1, otherwise is 0; b isnA preferred a posteriori probability output for said first network or said second network:
aka distance, a, of vehicle type class k estimated for the first network or the second networknEstimating a distance with a vehicle type n for the first network or the second network;
vi is the eigenvector output by the topmost layer of the first network, Vi is (x1, x2, …, xn), Vj is the eigenvector output by the topmost layer of the second network, Vj is (y1, y2, …, yn), and n is the dimension of the eigenvector; λ is a constant;
as can be seen from the above, in the embodiment, the vehicle type classification problem can be split into two sub-problems by using a cascaded classification model, one is the classification problem of the vehicle type with a large characteristic difference, the other is the classification problem of the similar vehicle type, and for the classification problem of the vehicle type with a large characteristic difference, a lightweight classifier with less parameters can be used for classification, so that the classification accuracy is ensured, and the classification efficiency can be improved; the classifier obtained through similar vehicle type training can be further adopted to accurately classify vehicle types contained in the similar set, namely the classification problem of vehicle types with higher similarity, so that the accuracy of vehicle type classification is ensured.
The above device embodiment corresponds to the method embodiment, and has the same technical effect as the method embodiment, and for the specific description, refer to the method embodiment. The device embodiment is obtained based on the method embodiment, and for specific description, reference may be made to the method embodiment section, which is not described herein again.
Those of ordinary skill in the art will understand that: the figures are merely schematic representations of one embodiment, and the blocks or flow diagrams in the figures are not necessarily required to practice the present invention.
Those of ordinary skill in the art will understand that: modules in the devices in the embodiments may be distributed in the devices in the embodiments according to the description of the embodiments, or may be located in one or more devices different from the embodiments with corresponding changes. The modules of the above embodiments may be combined into one module, or further split into multiple sub-modules.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A cascade vehicle type classification method based on a confusion matrix is characterized by comprising the following steps:
acquiring an image to be identified containing a target vehicle;
inputting the image to be recognized into a pre-trained primary classifier to obtain a candidate vehicle type category of the target vehicle; the primary classifier is obtained by training in advance according to each initial sample image and the vehicle type category corresponding to each initial sample image; the parameter number of the primary classifier is smaller than a preset value;
acquiring a similarity set, and judging whether the candidate vehicle type category is contained in the similarity set; the similarity set is obtained according to a confusion matrix; the confusion matrix is obtained according to the initial vehicle type category of each initial sample image identified by the primary classifier; when i is not equal to j, the ith row and jth column elements in the confusion matrix represent the false recognition rate of the primary classifier for recognizing the initial sample images of the ith vehicle type as the jth vehicle type, wherein the false recognition rate is the number of the primary classifier for recognizing the initial sample images of the ith vehicle type as the jth vehicle type divided by the total number of the initial sample images of the ith vehicle type; the similar set stores a target sample image pair with the error identification rate larger than a preset identification threshold value in the initial sample images and a vehicle type category corresponding to each initial sample image in each target sample image pair;
When the candidate vehicle type category is not contained in the similar set, taking the candidate vehicle type category as a final vehicle type category of the target vehicle;
when the candidate vehicle type category is contained in the similar set, inputting the image to be recognized into a pre-trained secondary classifier to obtain a final vehicle type category of the target vehicle; the secondary classifier is obtained by sampling an initial sample image of an ith vehicle type in the target sample image pair in advance to obtain a first sample image, sampling an initial sample image of a jth vehicle type in the target sample image pair to obtain a second sample image, and training according to the first sample images, vehicle type categories corresponding to the first sample images, and vehicle type categories corresponding to the second sample images and the second sample images.
2. The method of claim 1, wherein the training process of the primary classifier comprises:
constructing an initial convolutional neural network; the parameter quantity of the initial convolutional neural network is less than a preset value;
acquiring each initial sample image, and determining the vehicle type category corresponding to each initial sample image;
And inputting the initial sample images and the vehicle type categories corresponding to the initial sample images into the initial convolutional neural network, and taking the current initial convolutional neural network as the primary classifier when the loss function change value of the initial convolutional neural network is smaller than a first threshold value.
3. The method of claim 2, wherein the initial convolutional neural network comprises a mobileNet or a shuffenet; the loss function is:
n is the total number of vehicle type categories, and when the obtained identification result is correct, y isnIs 1, otherwise is 0; snA preferred a posteriori probability output for said initial convolutional neural network:
akestimating the estimated vehicle type class k distance, a, of the initial convolutional neural networknAnd estimating the distance with the vehicle type n for the initial convolutional neural network.
4. The method of claim 2, further comprising:
obtaining a recognition result of the vehicle type recognition of the initial sample image by the primary classifier;
constructing a confusion matrix according to the identification result; the element of the ith row and the jth column in the confusion matrix is the number of the initial sample images of the ith type of vehicle recognized by the primary classifier as the jth type of vehicle divided by the total number of the initial sample images of the ith type of vehicle; the dimension of the confusion matrix is N x N, and N is the total number of vehicle type categories;
And taking the initial sample image of the ith vehicle type and the initial sample image of the jth vehicle type corresponding to the element larger than the preset identification threshold value in the confusion matrix as a target sample image pair, and storing the vehicle type category corresponding to each target sample image pair, the initial sample image of the ith vehicle type in each target sample image pair and the vehicle type category corresponding to the initial sample image of the jth vehicle type in a similar set.
5. The method of claim 4, further comprising:
acquiring two primary classifiers respectively serving as a first network and a second network;
sampling initial sample images of ith vehicle type in the target sample image pair to obtain first sample images, sampling initial sample images of jth vehicle type in the target sample image pair to obtain second sample images, and acquiring vehicle type categories corresponding to the first sample images and the second sample images from the similarity set;
inputting each first sample image and the corresponding vehicle type category into the first network, inputting each second sample image and the corresponding vehicle type category into the second network, calculating a value of an objective function according to the recognition results of the first network and the second network, and taking the current first network or the second network as the secondary classifier when the variation value of the objective function is smaller than a second threshold value.
6. The method of claim 5, wherein the objective function is:
wherein y is the correct result when the first network or the second network obtains the correct resultnIs 1, otherwise is 0; b isnA preferred a posteriori probability output for said first network or said second network:
aka distance, a, of vehicle type class k estimated for the first network or the second networknEstimating a distance with a vehicle type n for the first network or the second network;
vi is the eigenvector output by the topmost layer of the first network, Vi is (x1, x2, …, xn), Vj is the eigenvector output by the topmost layer of the second network, Vj is (y1, y2, …, yn), and n is the dimension of the eigenvector; λ is a constant;
7. a cascade vehicle type classification device based on a confusion matrix is characterized in that the device comprises:
the image acquisition module is used for acquiring an image to be identified containing a target vehicle;
the primary classification module is used for inputting the image to be recognized into a primary classifier obtained by pre-training to obtain the candidate vehicle type category of the target vehicle; the primary classifier is obtained by training in advance according to each initial sample image and the vehicle type category corresponding to each initial sample image; the parameter number of the primary classifier is smaller than a preset value;
The category judgment module is used for acquiring a similarity set and judging whether the category of the candidate vehicle type is contained in the similarity set; the similarity set is obtained according to a confusion matrix; the confusion matrix is obtained according to the initial vehicle type category of each initial sample image identified by the primary classifier; when i is not equal to j, the ith row and jth column elements in the confusion matrix represent the false recognition rate of the primary classifier for recognizing the initial sample images of the ith vehicle type as the jth vehicle type, wherein the false recognition rate is the number of the primary classifier for recognizing the initial sample images of the ith vehicle type as the jth vehicle type divided by the total number of the initial sample images of the ith vehicle type; the similar set stores a target sample image pair with the error identification rate larger than a preset identification threshold value in the initial sample images and a vehicle type category corresponding to each initial sample image in each target sample image pair;
a vehicle type determining module, configured to, when the category determining module determines that the category of the candidate vehicle type is not included in the similarity set, take the category of the candidate vehicle type as a final vehicle type category of the target vehicle;
the secondary classification module is used for inputting the image to be recognized into a secondary classifier obtained by pre-training when the category judgment module determines that the category of the candidate vehicle type is contained in the similarity set, so as to obtain the final vehicle type category of the target vehicle; the secondary classifier is obtained by sampling an initial sample image of an ith vehicle type in the target sample image pair in advance to obtain a first sample image, sampling an initial sample image of a jth vehicle type in the target sample image pair to obtain a second sample image, and training according to the first sample images, vehicle type categories corresponding to the first sample images, and vehicle type categories corresponding to the second sample images and the second sample images.
8. The apparatus of claim 7, further comprising:
the initial network construction module is used for constructing an initial convolutional neural network; the parameter quantity of the initial convolutional neural network is less than a preset value;
the system comprises a sample image acquisition module, a model classification determination module and a model classification determination module, wherein the sample image acquisition module is used for acquiring each initial sample image and determining the model classification corresponding to each initial sample image;
and the primary classifier training module is used for inputting the initial sample images and the vehicle type categories corresponding to the initial sample images into the initial convolutional neural network, and when the loss function change value of the initial convolutional neural network is smaller than a first threshold value, taking the current initial convolutional neural network as the primary classifier.
9. The apparatus of claim 8, further comprising:
the result acquisition module is used for acquiring the recognition result of the primary classifier for vehicle type recognition of the initial sample image;
the matrix construction module is used for constructing a confusion matrix according to the identification result; the element of the ith row and the jth column in the confusion matrix is the number of the initial sample images of the ith type of vehicle recognized by the primary classifier as the jth type of vehicle divided by the total number of the initial sample images of the ith type of vehicle; the dimension of the confusion matrix is N x N, and N is the total number of vehicle type categories;
And the similarity set construction module is used for taking the initial sample image of the ith vehicle type and the initial sample image of the jth vehicle type corresponding to the element, which is greater than the preset identification threshold value, in the confusion matrix as a target sample image pair, and storing the vehicle type category corresponding to each target sample image pair, the initial sample image of the ith vehicle type in each target sample image pair and the vehicle type category corresponding to the initial sample image of the jth vehicle type into the similarity set.
10. The apparatus of claim 9, further comprising:
the network acquisition module is used for acquiring the two primary classifiers respectively as a first network and a second network;
the sample sampling module is used for sampling an initial sample image of an ith vehicle type in the target sample image pair to obtain a first sample image, sampling an initial sample image of a jth vehicle type in the target sample image pair to obtain a second sample image, and acquiring vehicle type categories corresponding to the first sample images and the second sample images from the similarity set;
and the secondary classifier training module is used for inputting each first sample image and the corresponding vehicle type category into the first network, inputting each second sample image and the corresponding vehicle type category into the second network, calculating the value of an objective function according to the recognition results of the first network and the second network, and taking the current first network or the second network as the secondary classifier when the change value of the objective function is smaller than a second threshold value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010708053.4A CN111860334A (en) | 2020-07-22 | 2020-07-22 | Cascade vehicle type classification method and device based on confusion matrix |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010708053.4A CN111860334A (en) | 2020-07-22 | 2020-07-22 | Cascade vehicle type classification method and device based on confusion matrix |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111860334A true CN111860334A (en) | 2020-10-30 |
Family
ID=73001493
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010708053.4A Withdrawn CN111860334A (en) | 2020-07-22 | 2020-07-22 | Cascade vehicle type classification method and device based on confusion matrix |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111860334A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113177527A (en) * | 2021-05-27 | 2021-07-27 | 安阳工学院 | Vehicle type recognition method and device |
-
2020
- 2020-07-22 CN CN202010708053.4A patent/CN111860334A/en not_active Withdrawn
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113177527A (en) * | 2021-05-27 | 2021-07-27 | 安阳工学院 | Vehicle type recognition method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110321923B (en) | Target detection method, system and medium for fusion of different-scale receptive field characteristic layers | |
| US11194331B2 (en) | Unsupervised classification of encountering scenarios using connected vehicle datasets | |
| CN112232241B (en) | Pedestrian re-identification method and device, electronic equipment and readable storage medium | |
| CN111310850B (en) | License plate detection model construction method and system, license plate detection method and system | |
| CN109739904B (en) | Time sequence marking method, device, equipment and storage medium | |
| CN101221623B (en) | Object type on-line training and recognizing method and system thereof | |
| CN113688851B (en) | Data labeling method and device and fine granularity identification method and device | |
| CN110147707B (en) | High-precision vehicle identification method and system | |
| CN108460427A (en) | A kind of disaggregated model training method, device and sorting technique and device | |
| CN112598000A (en) | Question identification method and device, electronic equipment and computer storage medium | |
| CN112183166A (en) | Method and device for determining training sample and electronic equipment | |
| WO2021012383A1 (en) | Age prediction method and device for infrared image | |
| CN112613617A (en) | Uncertainty estimation method and device based on regression model | |
| KR102186974B1 (en) | Smart cctv system for analysis of parking | |
| CN112784494B (en) | Training method of false positive recognition model, target recognition method and device | |
| CN111860334A (en) | Cascade vehicle type classification method and device based on confusion matrix | |
| CN114139624A (en) | Method for mining time series data similarity information based on integrated model | |
| CN116872961B (en) | Control system for intelligent driving vehicle | |
| CN109740750B (en) | Data collection method and device | |
| CN111340041A (en) | License plate recognition method and device based on deep learning | |
| CN115393743A (en) | Vehicle detection method based on double-branch encoding and decoding network, unmanned aerial vehicle and medium | |
| Malmström et al. | Detection of outliers in classification by using quantified uncertainty in neural networks | |
| CN114943290A (en) | Biological invasion identification method based on multi-source data fusion analysis | |
| CN111209860A (en) | Video attendance system and method based on deep learning and reinforcement learning | |
| Maligalig et al. | Machine Vision System of Emergency Vehicle Detection System Using Deep Transfer Learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20201030 |
|
| WW01 | Invention patent application withdrawn after publication |