CN115311690A - End-to-end pedestrian structural information and dependency relationship detection method thereof - Google Patents
End-to-end pedestrian structural information and dependency relationship detection method thereof Download PDFInfo
- Publication number
- CN115311690A CN115311690A CN202211229170.8A CN202211229170A CN115311690A CN 115311690 A CN115311690 A CN 115311690A CN 202211229170 A CN202211229170 A CN 202211229170A CN 115311690 A CN115311690 A CN 115311690A
- Authority
- CN
- China
- Prior art keywords
- pedestrian
- information
- rectangular frame
- frame
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/766—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using regression, e.g. by projecting features on hyperplanes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Multimedia (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Human Computer Interaction (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a method for detecting end-to-end pedestrian structural information and the subordination relationship thereof, which comprises the following steps: the method comprises the steps of firstly developing a universal structural labeling tool and carrying out data labeling, synchronously designing a network structure, needing to enhance the dependency relationship in the training process, distributing labels for a prediction frame, then carrying out reasoning by a model, regressing the coordinates of a pedestrian structural information rectangular frame by utilizing model output information, and finally calculating a loss function loss for updating the model. The invention is suitable for edge equipment; under the condition that the computing resources of the edge device are limited, the effect maximization under effective resources is realized end to end, the dependency relationship can be directly output while the pedestrian rectangular frame and the pedestrian structural information rectangular frame are detected, the subsequent logic judgment is avoided, and the accuracy rate is higher; meanwhile, a universal attribute labeling tool is developed, the detection frames are labeled, and the dependency relationship between the frames can be labeled, so that the labeling efficiency is effectively improved.
Description
Technical Field
The invention relates to the field of computer vision, in particular to a method for detecting end-to-end pedestrian structural information and the dependency relationship thereof.
Background
Pedestrian detection is always a research hotspot in the field of intelligent video monitoring. Pedestrian detection may obtain a pedestrian rectangular frame in the image and video frames. In the application of the industrial park, the position of the pedestrian needs to be acquired, and effective information of the pedestrian needs to be extracted, wherein the effective information generally comprises structural information such as whether a worker wears a worker cap, whether a worker wears a worker clothes and whether a mask is worn.
Currently, there are two general ways to acquire these structured information: one is through the method of detection, the detection method can support the detection function of the particular article attribute, give the rectangular frame of the article, namely utilize the target detection algorithm, while detecting the rectangular frame of the pedestrian, also detect the structural information, give the rectangular frame position of the structural information; the other is a classification method, which can support the identification function of non-entity attributes such as age, gender and the like.
For the first mode, the pedestrian rectangular frame and the pedestrian structured information rectangular frame are both independently detected, and the subordination relation between the pedestrian structured information rectangular frame and the pedestrian rectangular frame needs to be determined after independent detection; at present, the dependency relationship between two frames is determined by calculating the intersection ratio IOU between the two frames; however, if multiple persons overlap, confusion may occur in this way, resulting in inaccurate membership determination.
Specifically, the prior art of the first mode is adopted, for example:
1. the patent names are: a pedestrian detection tracking method and device based on multi-attribute analysis, the disclosure number is: CN114092558A.
The patent redefines a pedestrian detection network, inputs a preprocessed pedestrian image into a preset network structure, enables the network structure to perform feature extraction on the pedestrian image to obtain a feature image, performs detection tracking on each target according to the feature image, and outputs a detection tracking result. The detection tracking result comprises a detected target, a tracked target, target front face information and position information, whether the target rides a bicycle or not and the like. The advantages are that: the method has the advantages that the detection of the target, the tracking of the target, the judgment of whether the target contains the required information, the judgment of the position of the information and the like are completed through a single preset network structure, the multi-attribute analysis is realized, and one model has multiple purposes; meanwhile, whether the pedestrians ride the bicycle or not and whether the pedestrians have the front faces or not in the image can be known, and the pedestrians do not need to be sent to another network for analysis after being captured.
The technical scheme has the following disadvantages:
(1) By adopting a multi-task model, the detection efficiency can be improved, but the model is difficult to train, and the training data labeling cost is high.
(2) The regression box of the attribute is predicted relative to the center of the person, and the regression box of the attribute only predicts the offset relative to the center of the person, namely predicts the offset of the upper left corner and the lower right corner of the attribute target relative to the center of the person to obtain the position information of the box. The center of the person is obtained by model prediction, the center of the person has an error a, and the prediction of the attribute regression frame also has an error b, so that the attribute regression frame obtained indirectly finally has an accumulated error a + b.
(3) The enhancement mode that adopts during training is too single, does not tailor relevant trigk such as waiting, probably influences the model recall rate effect.
(4) The designed model network is customized and improved aiming at the face with small resolution and whether to ride the bicycle or not, and is not necessarily applicable to other attribute scenes.
2. The patent names are: a pedestrian detection method, a system and a terminal device are disclosed as follows: CN 110245799A.
The method comprises the steps that a pedestrian in a target image is identified through a multi-object convolution depth network model, and a first identification frame is added to the pedestrian in the target image; the training task of the multi-object convolution depth network model comprises a semantic task and pedestrian detection; identifying a specific object in the target image through the convolutional neural network VGG19, and adding a second identification frame to the specific object in the target image; and judging whether the first identification frame and the second identification frame are overlapped, if so, judging that the pedestrian carries a specific object, and triggering a preset monitoring event.
The technical scheme has the following disadvantages:
(1) The two models are adopted to detect pedestrians and specific articles respectively, so that detection time consumption and operation memory are increased.
(2) And judging whether the first identification frame and the second identification frame are overlapped or not by means of intersection and comparison to judge whether the pedestrian carries the specific object or not. This approach may be confusing if multiple people overlap.
The disadvantages of the two patents are summarized as follows:
(1) The scheme is inefficient.
(2) The membership between the pedestrian rectangular frame and the pedestrian structural information rectangular frame needs to be calculated by an intersection and parallel equation, and the accuracy is influenced under the condition of overlapping of multiple persons.
(3) The universality is not strong, and the attribute categories cannot be expanded.
(4) The regression mode of the model attribute box is indirect regression, and the accuracy can be influenced.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provide an end-to-end pedestrian structural information and dependency relationship detection method, which is suitable for edge equipment; in case the computational resources of the edge device are limited (the TPU of the edge device has limited computational power with respect to the GPU of the server), the present invention maximizes the effect under the effective resources by end-to-end.
The purpose of the invention is realized by the following technical scheme:
an end-to-end pedestrian structural information and dependency relationship detection method comprises the following steps:
s1, improving a Yolox model: increasing the obj _ output branch and the reg _ output branch of a decoupling Head Decoupled Head of the Yolox model by S and 4 × S channels respectively; wherein S is the category number of the pedestrian structured information;
s2, before training the improved Yolox model, carrying out data annotation on the image of the training sample through an annotation tool, wherein the annotated information comprises a pedestrian rectangular frame, a pedestrian structural information rectangular frame and dependency relationship information between the pedestrian rectangular frame and the pedestrian structural information rectangular frame;
s3, training the improved Yolox model:
enhancing the image data of the training sample, enhancing the dependency relationship between the pedestrian rectangular frame and the pedestrian structural information rectangular frame, and distributing labels for the prediction frames;
inputting the enhanced image, performing inference by using an improved Yolox model, and returning the coordinates of the pedestrian structural information rectangular frame and the coordinates of the pedestrian rectangular frame by using the output information of the improved Yolox model, and simultaneously directly obtaining the membership between the pedestrian rectangular frame and the pedestrian structural information rectangular frame;
calculating loss function loss, updating the improved Yolox model, and finishing training;
and S4, inputting the image to be detected into the improved and trained Yolox model, and directly outputting the pedestrian structural information rectangular frame and the coordinates of the pedestrian rectangular frame, and the subordination relation between the pedestrian rectangular frame and the pedestrian structural information rectangular frame by the model end to end.
The improvement of the Yolox model specifically includes:
firstly, increasing the obj _ output branch of a decoupling Head Decoupled Head of a Yolox model by S channels for output, wherein the output size is H W (1 + S), increasing the reg _ output branch of the decoupling Head Decoupled Head of the Yolox model by 4W (4 + S), and outputting the output size H W (4 + S); the output size of the cls _ output branch of a decoupling Head Decoupled Head of the yolk model is H W1; wherein H is the height of the output characteristic diagram, and W is the width of the output characteristic diagram;
the method for performing inference by using the improved Yolox model specifically includes: merging an obj _ output branch, a reg _ output branch and a cls _ output branch of a decoupling Head Decoupled Head of the Yolox model to obtain final characteristic information, wherein the size of the final characteristic information is pred _ num _ dim _ s; wherein pred _ num = W × H, which is used for representing the number of prediction frames; dim _ s =1+, S +, 4+ S, used for characterizing feature vector dimensions of each prediction frame; each predictor box now contains a feature vector of dimension dim _ s:
[x y w h obj cls attr_1 ... attr_n x_1 y_1 w_1 h_1 ... x_n y_n w_n h_n]
wherein x is the x coordinate information of the center point of the target frame, y is the y coordinate information of the center point of the target frame, and w is the width information of the target frame; h is height information of the target frame; obj is the score information of the target frame; cls is the score information of the target box category; attr _ n is score information of the structured information n, and [ x _ n _ y _ n _ w _ n _ h _ n ] is frame coordinate information of the structured information n;
according to the feature vector, the judgment process of the pedestrian structured information is as follows: when the size of obj × cls meets a score threshold of a rectangular frame of the pedestrian, the current prediction frame is considered to contain pedestrian information, and [ x y w h ] is coordinate information of the rectangular frame of the pedestrian, at the moment, if the score attr _ n of the structured information n meets a structured information probability threshold, the pedestrian is considered to contain the structured information n, and [ x _ n y _ n w _ n h _ n ] is the coordinate information of the rectangular frame of the structured information n of the pedestrian; the whole end-to-end pedestrian structural information detection is completed.
Marking the dependency relationship information between the pedestrian rectangular frame and the pedestrian structural information rectangular frame, wherein the dependency relationship information is established by directly connecting a left upper corner connecting line between the pedestrian rectangular frame and the pedestrian structural information rectangular frame through a marking tool; and acquiring the id pairing information of the pedestrian rectangular frame and the pedestrian structural information rectangular frame through the connection, wherein the pairing information can be stored in the dependency relationship label file.
After the image of the training sample is subjected to data annotation, the annotated data format comprises three parts: image data images, rectangular frame label information labels and dependency relationship label information images relevate; each part is divided into a training part and a testing part.
The enhancing of the image data of the training sample specifically includes: storing the membership label information of the pedestrian rectangular frame and the pedestrian structural information rectangular frame into a queue, then performing mosaic and mixup data enhancement on the image data, the label information of the pedestrian rectangular frame and the pedestrian structural information rectangular frame together, finally combing the pedestrian rectangular frame and the pedestrian structural information rectangular frame which still exist after enhancement again, and judging whether the membership exists or increases according to whether the pedestrian rectangular frame and the structural frame exist or increase newly, thereby deleting or increasing the membership in the original queue, and the updated queue data is the membership label information after enhancement.
The specific implementation manner of allocating the label to the prediction frame is as follows: when the labels of the prediction frames of the pedestrian rectangular frame are distributed, the label information of the real rectangular frame is adopted to carry out positive and negative sample area division on the characteristic image on the decoupling head, namely all the prediction frames in the real rectangular frame are used as positive sample candidate frames, and the rest are negative samples; when the labels are distributed to the prediction frame of the pedestrian structured information rectangular frame, the real frame of the pedestrian structured information rectangular frame is not adopted to carry out region division on the characteristic diagram on the solution head, but the real frame of the pedestrian rectangular frame is still adopted to replace the prediction frame, so that the consistency of label distribution between the pedestrian rectangular frame and the pedestrian structured information rectangular frame is kept, the performance convergence of a training model can be accelerated, and the condition that the pedestrian rectangular frame is detected but the structured information is not detected in the trained model is avoided.
The structured information rectangular frame regression method specifically comprises the following steps:
when the resolution of the input graph is 640 × 640, the three decorupled heads respectively have feature graphs with different down-sampling scales for output, the feature graphs W × H are respectively 20 × 20 (down-sampling 5 times, the down-sampling multiple is 32, and the other same principles are the same), 40 × 40, and 80 × 80;
for each cell of one feature map, a corresponding anchor frame anchor (the anchor frame is a link established between the feature map and an actual pixel coordinate rectangular frame and has the function of accelerating model convergence during training); when the feature maps W × H are 20 × 20, respectively, the anchor frame size is 32 × 32, and is consistent with the down-sampling magnification; according to the improved Yolox model, providing rectangular frame coordinate information [ x _ n _ y _ n _ w _ n _ h _ n ] of the structural information n under a certain cell (U _ w, U _ h), and calculating the rectangular frame coordinate under the actual resolution by combining the anchor frame information of the characteristic diagram where the rectangular frame coordinate information is located; wherein x _ n is the x offset of the central point of the rectangular frame relative to the current cell, and y _ n is the y offset of the central point of the rectangular frame relative to the current cell;
assuming that the width of the anchor frame is anchor _ w and the height of the anchor frame is anchor _ h, the actual pixel coordinates of the rectangular frame of the structured information are as follows:
wherein, (x _ pixel, y _ pixel) is the middle point of the rectangular frame, w _ pixel is the width of the rectangular frame, and h _ pixel is the height of the rectangular frame;
in the above calculation formula, only [ x _ n _ y _ n _ w _ n _ h _ n ] is obtained by network prediction, and the others are preset information, so that: the pedestrian structural information rectangular frame is directly obtained by the improved Yolox model, and the accumulation of intermediate errors is avoided without the intermediate result of the pedestrian rectangular frame information.
For the loss function loss during model training, on the basis of Yolox, structured information foreground probability loss attr _ obj _ loss and pedestrian structured information rectangular box regression loss attr _ reg _ loss are added, wherein the structured information foreground probability loss attr _ obj _ loss adopts a cross entropy loss function BCEWithLogitsLoss, and the pedestrian structured information rectangular box regression loss attr _ reg _ loss adopts a cross-over-loss function IOULoss; and adding newly added structured information foreground probability loss attr _ obj _ loss and pedestrian structured information rectangular frame regression loss attr _ reg _ loss to the original loss function to obtain a final loss function.
And setting an activation function of the Yolox model as relu, and setting a channel coefficient of a decoupling head as 0.5.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. the method can directly output the dependency relationship while detecting the pedestrian rectangular frame and the pedestrian structural information rectangular frame, thereby avoiding subsequent logic judgment and having higher accuracy. Meanwhile, a universal attribute labeling tool is developed, the detection frames are labeled, and the dependency relationship between the frames can be labeled, so that the labeling efficiency is effectively improved.
2. The target detection network Yolox model is obtained by modifying, the dependency relationship between the pedestrian rectangular frame and the pedestrian structural information rectangular frame is output end to end, the judgment of whether the subsequent frames are overlapped is avoided, and the accuracy and the efficiency of structural information detection are effectively improved through a data enhancement and label distribution mode. Meanwhile, the detection of various structured information can be easily supported through the expansion of the model output channel, more time consumption can not be caused, and the universality of the model is ensured. In addition, a new attribute labeling tool is developed, the detection frame is labeled, the dependency relationship between the frames can be labeled, and the labeling efficiency is effectively improved.
Drawings
FIG. 1 is a flow chart of a method for detecting end-to-end pedestrian structural information and its dependency relationship;
FIG. 2 is a schematic diagram of a data enhancement scheme for dependencies;
FIG. 3 is a diagram illustrating the labeled data format.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but the present invention is not limited thereto.
Referring to fig. 1 to 3, a method for detecting end-to-end pedestrian structural information and its dependency relationship includes the following steps: the method comprises the steps of firstly developing a universal structural labeling tool and carrying out data labeling, synchronously designing a network structure, needing to enhance the dependency relationship in the training process, distributing labels for a prediction frame, then carrying out reasoning by a model, regressing the coordinates of a pedestrian structural information rectangular frame by utilizing model output information, and finally calculating a loss function loss for updating the model.
The implementation process is as follows:
1. network architecture design
Yolox adopts yolov3_ spp as a reference model, and adopts the design of such trigks as Decoupled Head and Anchor-free to carry out improvement, so as to finally obtain a Yolox-Darknet53 network structure. The Yolox paper indicates that the expression capability of the current Yolov 3-v 5 series detection heads may be deficient, the expression capability of the detection heads is better without a decorled Head, and the adopted decorled Head not only improves the precision, but also accelerates the convergence speed of the network. There is a further level of importance after this decoupling: the network architecture of the Yolox can be integrated with a plurality of algorithm tasks. Such as: YOLOX + Yolact/CondInst/SOLO, which enables an end-side example segmentation. And outputting a YOLOX + 34 layer to realize the detection of 17 key points of the human body at the end side. Here is also the reason why the invention adopts Yolox, and the integrated detection of the pedestrian structured information can be realized by adding an output channel.
Decorupped Head has three total branches: cls _ output — category and score of the main predicted target box, output size W × H × C; obj _ output-mainly judging whether the target frame is a foreground or a background, and outputting the size of W × H × 1; reg _ output-mainly predicts the coordinate information (x, y, W, H) of the target frame, and the output size is W × H × 4. Where W is the width of the output feature map, H is the height of the output feature map, and C is the number of detection categories, for example, when a detection category is only a pedestrian, C is 1. And finally, combining the last three outputs to obtain the final feature information, wherein the size of the final feature information is pred _ num + dim, pred _ num = W + H, the number of the prediction frames is represented, and dim = C +1+4, and the feature vector dimension of each prediction frame is represented. Each predictor box now contains a feature vector of dimension dim:
[cls_1 cls_2 ... cls_C obj x y w h]
wherein, x is the coordinate information of the center point x of the target frame; y is the coordinate information of the center point y of the target frame; w is the width information of the target frame; h, height information of the target frame; obj is target box score information; cls _1: target box category 1 score; cls _ C-target Box category C score.
In order to increase the detection of the pedestrian structural information, the following changes are made to the decorumled Head: the obj _ output branch is increased by S channels to output, and the output is W × H (1 +S), and the reg _ output branch is increased by 4 × S channels to output W × H (4 +4S). The number of S structural categories of the pedestrians can be changed according to actual project requirements, so that the algorithm has certain universality, for example, whether a mask is worn or not is detected, whether a worker hat is worn or not is detected, and S is 2 at the moment. And finally, combining the three output to obtain final feature information, wherein the size of the final feature information is pred _ num + dim _ s, pred _ num = W + H represents the number of prediction frames, dim _ s = C +1+ S +4+ S represents the feature vector dimension of each prediction frame. Each predictor block then contains a feature vector with dimension dim _ s:
[x y w h obj cls attr_1 ... attr_n x_1 y_1 w_1 h_1 ... x_n y_n w_n h_n]
wherein, attr _ n is score information of the structured information n, and [ x _ n y _ n w _ n h _ n ] is frame coordinate information of the structured information n.
According to the feature vector, the judgment process of the pedestrian structured information is as follows: when the size of obj × cls meets the score threshold of the rectangular frame of the pedestrian, the current prediction frame is considered to contain the pedestrian information, and [ x y w h ] is coordinate information of the rectangular frame of the pedestrian, at this time, if the score attr _ n of the structured information n meets the probability threshold of the structured information, the pedestrian is considered to contain the structured information n, and [ x _ n w _ n _ h _ n ] is the coordinate information of the rectangular frame of the structured information n of the pedestrian. The whole end-to-end pedestrian structured information detection is completed.
2. Network improvements for edge end migration
In order to reduce the inference time after the model edge end transplantation, the following changes are further made to the network: the activation function is modified from silu to relu, and the decoupling head channel coefficient is modified from 1.0 to 0.5.
3. Design of pedestrian structured information rectangular frame regression method
For the improved network, when the input pattern resolution is 640 × 640, the three decorumded heads have feature outputs of different down-sampling scales, and the feature W × H is 20 × 20 (down-sampling 5 times, down-sampling multiple is 32, and the same applies to the rest), 40 × 40, and 80 × 80. For each cell of one of the feature maps, there is a corresponding anchor frame anchor (the anchor frame is a tie established between the feature map and the actual pixel coordinate rectangular frame, and has the effect of accelerating model convergence during training). When the feature patterns W × H are 20 × 20, respectively, the anchor frame size is 32 × 32, and corresponds to the down-sampling magnification. According to the network structure design, rectangular frame coordinate information [ x _ n _ y _ n _ w _ n _ h _ n ] of the pedestrian structural information n in a certain cell (U _ w, U _ h) is given, and rectangular frame coordinates under actual resolution can be calculated by combining anchor frame information of a characteristic diagram where the rectangular frame coordinate information is located. Wherein x _ n is the x offset of the center point of the rectangular frame relative to the current cell, and y _ n is the y offset of the center point of the rectangular frame relative to the current cell. Assuming that the width of the anchor frame is anchor _ w and the height of the anchor frame is anchor _ h, the actual pixel coordinates of the rectangular frame of the structured information are as follows:
where (x _ pixel, y _ pixel) is the rectangular frame midpoint, w _ pixel is the rectangular frame width, and h _ pixel is the rectangular frame height.
In the above calculation formula, only [ x _ n _ y _ n _ w _ n _ h _ n ] is obtained by network prediction, and the others are preset information, which is equivalent to that the pedestrian structural information rectangular frame is directly obtained by the model, and intermediate results such as pedestrian rectangular frame information and the like are not needed, so that the accumulation of intermediate errors is avoided.
4. Data enhancement scheme for dependencies
For the convolutional neural network, in order to avoid overfitting caused by the fact that training data are too single in the model training process, a data-enhanced regularization mode is adopted, the diversity of a data set is increased, and therefore the generalization capability of the model is improved. The data sample of the present invention is divided into three parts, which are image data, rectangular frame tag information (including a pedestrian rectangular frame and a pedestrian structured information rectangular frame), and dependency relationship tag information between the pedestrian rectangular frame and the pedestrian structured information rectangular frame. The original Yolox data enhancement adopts mosaic and mixup, and the training graph is spliced in a random scaling, random cutting and random arrangement mode, so that two parts of image data and rectangular frame label information can be amplified.
The method increases the enhancement of the dependency relationship between the pedestrian rectangular frame and the pedestrian structural information rectangular frame on the basis of the original mosaic and mixup data enhancement, namely, splicing, deleting and cutting each image data in the data enhancement process, wherein the accompanying phenomenon is the new addition, cutting, distortion or disappearance of the rectangular frame label information, and the corresponding dependency relationship is also increased or deleted synchronously.
The basic principle of the implementation is that before data enhancement, the membership label information of a rectangular frame is stored in a queue, then mosaic and mixup data enhancement is carried out on image data and the rectangular frame label information, finally, a pedestrian rectangular frame and a pedestrian structural information rectangular frame which still exist after enhancement are combed again, whether membership exists or is increased is judged according to whether the pedestrian rectangular frame and the pedestrian structural frame exist or are increased, so that membership in the original queue is deleted or increased, and updated queue data is the membership label information after enhancement.
5. Label allocation scheme for prediction boxes
In the design of the network structure, we mention that a feature map on a decoupling head has pred _ num prediction blocks output. The label distribution is that in the model training process, real rectangular frame label information is distributed to each prediction frame, the corresponding relation between the prediction frames and the labeled real rectangular frame label information is established, namely, the prediction frames are determined to be used as foreground positive samples, the prediction frames are determined to be used as background negative samples, then the loss between the prediction frames and the real rectangular frame label information is calculated by using a loss function, and finally the model weight is updated through back propagation.
In the original Yolox, in order to determine the distribution mode of the prediction frame, the feature map on the decoupling head is subjected to positive and negative sample area division by using the label information of the real rectangular frame, that is, all the prediction frames in the real rectangular frame are used as positive sample candidate frames, and the rest are negative samples. The prediction frame of the invention is processed in two cases, namely a prediction frame of a pedestrian rectangular frame (hereinafter referred to as P _ Pred _ Rect) and a prediction frame of a pedestrian structured information rectangular frame (hereinafter referred to as S _ Pre _ Rect). When label allocation is carried out on the P _ Pred _ Rect, the original Yolox label allocation scheme is directly adopted. Let the real frame of the rectangular frame of the pedestrian be P _ True _ Rect, and the real frame of the rectangular frame of the pedestrian structured information be S _ True _ Rect.
When the label distribution is carried out on the S _ Pred _ Rect, the following changes are made, the S _ True _ Rect is not adopted to carry out region division on the characteristic diagram on the decoupling head, and the P _ True _ Rect is still adopted to carry out substitution, so that the consistency of the label distribution between the pedestrian rectangular frame and the pedestrian structural information rectangular frame is kept, the performance convergence of the trained model can be accelerated, and the condition that the pedestrian rectangular frame is detected but the structural information is not detected in the trained model is avoided. The method is mainly characterized in that a detection result, namely whether a pedestrian rectangular frame and a pedestrian structural information rectangular frame are in an independent relation or not, only on the basis of detecting the pedestrian rectangular frame, whether the pedestrian structural information rectangular frame exists or not can be judged, so that when the S _ True _ Rect region is adopted for label distribution, some prediction frames which are located in the P _ True _ Rect region but not located in the S _ True _ Rect region can be distributed as negative samples, and a model trained by the distribution mode can cause that whether the pedestrian structural information rectangular frame exists or not can not be accurately judged after a part of the prediction frames detect the pedestrian rectangular frame. For example, for a cell with coordinates (U _ w, U _ h) on the feature map, if the S _ True _ Rect region is used as S _ Pred _ Rect for label assignment, the S _ Pred _ Rect of the cell is assigned as a negative sample, and the cell is located in the P _ True _ Rect region, so that the P _ Pred _ Rect of the cell is assigned as a positive sample, and thus the predicted frame feature vector of the cell is [ x y w h 1 cls 0 x U1 y \_1 w _1h _1], and after the pedestrian rectangular frame is detected, the pedestrian structured information rectangular frame is predicted not to exist; if the P _ True _ Rect region is used for label allocation for S _ Pred _ Rect, the S _ Pred _ Rect of the cell is allocated as a positive sample, so that the consistency of label allocation between the pedestrian rectangular frame and the pedestrian structural information rectangular frame is maintained.
6. Training loss function loss scheme design
For loss function loss during model training, structured information foreground probability loss attr _ obj _ loss and pedestrian structured information rectangular box regression loss attr _ reg _ loss are added on the basis of Yolox, wherein cross entropy loss function BCEWithLogitsLoss is adopted for attr _ obj _ loss, and cross-over ratio loss function IOULoss is adopted for attr _ reg _ loss. And adding two new losses and the original loss function to obtain a final loss function.
7. Universal structured marking tool development and data reading mode
In order to improve the labeling of the structured information data, a new structured labeling tool LabelImg-Attr is also developed, and the tool can be directly connected with a left upper corner connecting line between a pedestrian rectangular frame and a pedestrian structured information rectangular frame to establish an attribute relationship. The labeled data format is composed of three parts including image data images, rectangular frame label information labels and dependency relationship label information image relevate. Each part is divided into a training part and a testing part.
The txt information under labels is shown in the following figure, the first column is category, the second to fifth columns are frame information, and the last column is added id information.
2 0.374023 0.503125 0.041797 0.122917 0
1 0.363281 0.400694 0.028906 0.063889 1
0 0.073828 0.877431 0.110937 0.245139 2
0 0.369727 0.536111 0.091797 0.340278 3
2 0.084375 0.907292 0.078125 0.177083 4
The corresponding label information under relevate is as follows, wherein the first number of each line is host id, and the second number is attribute id.
3,0
3,1
2,4
The key points of the technical scheme of the embodiment are as follows:
(1) And improving the obtained end-to-end pedestrian structural detection model on the basis of the Yolox model. The method for increasing the number of the channels at the decoupling output head is provided, the structural detection of the pedestrians is realized, the structural detection can be expanded along with the specific structural category number, and the universality of the algorithm is ensured. The current patent has no related general structured information detection method. The improved model can output the dependency relationship between the pedestrian rectangular frame and the structural information rectangular frame end to end. The method avoids the situation that whether frames are overlapped or not is judged by adopting a cross-over ratio scheme in the prior art, and further judges whether the dependency relationship exists or not.
(2) Structured information rectangular frame regression method. The model directly predicts the deviation of the central point of the regression pedestrian structural information rectangular frame relative to the x direction and the y direction of the current cell of the model output characteristic diagram, and index information of the width and the height of the pedestrian structural information rectangular frame relative to the width and the height of the anchor frame of the current cell of the model output characteristic diagram. And combining other preset information to obtain a rectangular frame under the actual pixel coordinate. Equivalently, the pedestrian structural information rectangular frame is directly obtained by the model, compared with the prior art, intermediate results such as pedestrian rectangular frame information and the like are not needed, and accumulation of intermediate errors is avoided.
(3) A data enhancement scheme of membership between a pedestrian rectangular frame and a pedestrian structured information rectangular frame. The diversity of the dependency relationship data in the training process is effectively increased, and no patent is provided for a related data enhancement scheme at present.
(4) And (3) a label distribution scheme of the pedestrian structured information rectangular frame. The method and the device have the advantages that the label distribution consistency between the pedestrian rectangular frame and the pedestrian structural information rectangular frame is kept, the performance convergence of the training model can be accelerated, and meanwhile, the condition that the pedestrian rectangular frame is detected but the structural information is not detected in the trained model is avoided. No patent is currently available that proposes a related label assignment scheme.
(5) A generic structured labeling tool. The invention develops a universal structural labeling tool LabelImg-Attr which can be directly connected with a left upper corner connecting line between a pedestrian rectangular frame and a pedestrian structural information rectangular frame to establish an attribute relationship. No patent is provided for a related labeling tool at present.
Compared with the prior art, the technical scheme of the embodiment has the following advantages:
(1) The model has universality. Based on Yolox, the expandable network output head channel is utilized to adapt to different types of structured information.
(2) And (4) outputting from end to end. Based on the expandable network output head channel, the dependency relationship between the pedestrian rectangular frame and the structural information rectangular frame is output end to end by the model, and subsequent overlapping logic judgment is avoided.
(3) Application of newer target detection techniques. The mosaics data enhancement mode and the SimQTA label distribution are improved for adapting to the training of the structured information data.
(4) The accuracy of the pedestrian structured information rectangular frame is more accurate. And (3) directly predicting and regressing the model to obtain the offset of the central point of the rectangular frame of the pedestrian structural information in the x direction and the y direction relative to the current cell of the model output characteristic diagram, and eliminating intermediate errors.
(5) A universal attribute labeling tool is designed. The marking efficiency of the structured information data is effectively improved.
It will be understood by those skilled in the art that all or part of the steps in the embodiments may be implemented by hardware instructions associated with a computer program, and the program may be stored in a computer readable medium, which may include various media capable of storing program code, such as a flash memory, a removable hard disk, a read-only memory, a random access memory, a magnetic or optical disk, and the like. In one embodiment, the disclosure proposes a computer readable medium, in which a computer program is stored, the computer program being loaded and executed by a processing module to implement a method for detecting end-to-end pedestrian structural information and its dependencies.
The various embodiments or features mentioned herein may be combined with each other as additional alternative embodiments without conflict, within the knowledge and ability level of those skilled in the art, and a limited number of alternative embodiments formed by a limited number of combinations of features not listed above are still within the skill of the disclosed technology, as will be understood or inferred by those skilled in the art from the figures and above.
Moreover, the descriptions of the various embodiments are expanded upon with varying emphasis, and where not already described, may be had by reference to the prior art or other related descriptions herein.
It is emphasized that the above-mentioned embodiments, which are typical and preferred embodiments of the disclosure, are only used for explaining and explaining the technical solutions of the disclosure in detail for the reader to understand, and do not limit the scope of protection or application of the disclosure. Any modifications, equivalents, improvements and the like which come within the spirit and principle of the disclosure are intended to be covered by the scope of the disclosure.
Claims (11)
1. An end-to-end pedestrian structural information and dependency relationship detection method is characterized by comprising the following steps:
s1, improving a Yolox model: increasing S and 4 × S channels respectively for obj _ output branch and reg _ output branch of a decoupling Head Decoupled Head of a Yolox model; wherein S is the category number of the pedestrian structured information;
s2, before training the improved Yolox model, carrying out data annotation on the image of the training sample through an annotation tool, wherein the annotated information comprises a pedestrian rectangular frame, a pedestrian structural information rectangular frame and dependency relationship information between the pedestrian rectangular frame and the pedestrian structural information rectangular frame;
s3, training the improved Yolox model:
enhancing the image data of the training sample, enhancing the subordination relation between the pedestrian rectangular frame and the pedestrian structural information rectangular frame, and distributing a label for the prediction frame;
inputting the enhanced image, performing inference by using an improved Yolox model, and returning the coordinates of the pedestrian structural information rectangular frame and the coordinates of the pedestrian rectangular frame by using the output information of the improved Yolox model, and simultaneously directly obtaining the membership between the pedestrian rectangular frame and the pedestrian structural information rectangular frame;
calculating a loss function loss, updating the improved Yolox model, and finishing training;
and S4, inputting the image to be detected into the improved and trained Yolox model, and directly outputting the pedestrian structural information rectangular frame and the coordinates of the pedestrian rectangular frame, and the subordination relation between the pedestrian rectangular frame and the pedestrian structural information rectangular frame by the model end to end.
2. The method for detecting end-to-end pedestrian structural information and the dependency relationship thereof according to claim 1, wherein the Yolox model is improved, specifically:
firstly, increasing the obj _ output branch of a decoupling Head Decoupled Head of a Yolox model by S channel outputs, wherein the output size is H × W (1 + S), increasing the reg _ output branch of the decoupling Head Decoupled Head of the Yolox model by 4 × S channels, and wherein the output size is H × W (4 + S); the output size of the cls _ output branch of a decoupling Head Decoupled Head of the Yolox model is H W1; wherein, H is the height of the output characteristic diagram, and W is the width of the output characteristic diagram.
3. The method for detecting end-to-end pedestrian structural information and the dependency relationship thereof according to claim 2, wherein the inference is performed by using an improved Yolox model, specifically: merging an obj _ output branch, a reg _ output branch and a cls _ output branch of a decoupling Head Decoupled Head of the Yolox model to obtain final characteristic information, wherein the size of the final characteristic information is pred _ num _ dim _ s; wherein pred _ num = W × H, for representing the number of prediction frames; dim _ s =1+, S +, 4+ S, used for characterizing feature vector dimensions of each prediction frame; each predictor block then contains a feature vector with dimension dim _ s:
[x y w h obj cls attr_1 ... attr_n x_1 y_1 w_1 h_1 ... x_n y_n w_n h_n]
wherein x is the x coordinate information of the center point of the target frame, y is the y coordinate information of the center point of the target frame, and w is the width information of the target frame; h is height information of the target frame; obj is the score information of the target frame; cls is the score information of the target box category; attr _ n is score information of the structured information n, and [ x _ n _ y _ n _ w _ n _ h _ n ] is frame coordinate information of the structured information n;
according to the feature vector, the judgment process of the pedestrian structured information is as follows: when the size of obj × cls meets a score threshold of a rectangular frame of the pedestrian, the current prediction frame is considered to contain pedestrian information, and [ x y w h ] is coordinate information of the rectangular frame of the pedestrian, at the moment, if the score attr _ n of the structured information n meets a structured information probability threshold, the pedestrian is considered to contain the structured information n, and [ x _ n y _ n w _ n h _ n ] is the coordinate information of the rectangular frame of the structured information n of the pedestrian; the whole end-to-end pedestrian structural information detection is completed.
4. The end-to-end pedestrian structural information and dependency relationship detection method according to claim 1, wherein the dependency relationship information between the pedestrian rectangular frame and the pedestrian structural information rectangular frame is established by directly connecting an upper left connecting line between the pedestrian rectangular frame and the pedestrian structural information rectangular frame through a marking tool; and acquiring the id pairing information of the pedestrian rectangular frame and the pedestrian structural information rectangular frame through the connection line, and storing the pairing information into the dependency relationship label file.
5. The end-to-end pedestrian structural information and dependency relationship detection method according to claim 1, wherein after the data annotation of the image of the training sample is completed, the annotated data format comprises three parts: image data images, rectangular frame label information labels and dependency relationship label information images relevate; each part is divided into a training part and a testing part.
6. The method for detecting end-to-end pedestrian structural information and the dependency relationship thereof according to claim 1, wherein the enhancing is performed on the image data of the training sample, specifically: storing the subordinate relation label information of the pedestrian rectangular frame and the pedestrian structural information rectangular frame into a queue, then performing mosaic and mixup data enhancement on the image data, the label information of the pedestrian rectangular frame and the pedestrian structural information rectangular frame together, finally re-carding the pedestrian rectangular frame and the pedestrian structural information rectangular frame which still exist after enhancement, and judging whether the subordinate relation exists or increases according to whether the pedestrian rectangular frame and the structural frame exist or are newly added, so that the subordinate relation in the original queue is deleted or increased, and the updated queue data is the subordinate relation label information after enhancement.
7. The end-to-end pedestrian structured information and dependency relationship detection method according to claim 1, wherein labels are allocated to the prediction frames, and a specific implementation manner is as follows: when the labels are distributed to the prediction frames of the pedestrian rectangular frames, positive and negative sample area division is carried out on the feature map on the decoupling heads by adopting label information of the real rectangular frames, namely all the prediction frames in the real rectangular frames are used as positive sample candidate frames, and the rest are negative samples; when the labels are distributed to the prediction frame of the pedestrian structured information rectangular frame, the real frame of the pedestrian structured information rectangular frame is not adopted to carry out region division on the characteristic diagram on the solution head, but the real frame of the pedestrian rectangular frame is still adopted to replace the prediction frame, so that the consistency of label distribution between the pedestrian rectangular frame and the pedestrian structured information rectangular frame is kept, the performance convergence of a training model can be accelerated, and the condition that the pedestrian rectangular frame is detected but the structured information is not detected in the trained model is avoided.
8. The method for detecting pedestrian structural information and the dependency relationship thereof according to claim 1, wherein the structural information rectangular frame regression method specifically includes:
when the resolution of the input graph is 640 × 640, the three Decoupled heads respectively have feature graph outputs with different down-sampling scales, and the feature graphs W × H are respectively 20 × 20, 40 × 40 and 80 × 80;
for each cell of one of the feature maps, a corresponding anchor frame anchor is provided; when the characteristic graphs W are respectively 20X 20, the size of the anchor frame is 32X 32, and the size is consistent with the down-sampling multiplying power; according to the improved Yolox model, providing rectangular frame coordinate information [ x _ n _ y _ n _ w _ n _ h _ n ] of the structural information n under a certain cell (U _ w, U _ h), and calculating the rectangular frame coordinate under the actual resolution by combining the anchor frame information of the characteristic diagram where the rectangular frame coordinate information is located; wherein x _ n is the x offset of the central point of the rectangular frame relative to the current cell, and y _ n is the y offset of the central point of the rectangular frame relative to the current cell;
assuming that the width of the anchor frame is anchor _ w and the height of the anchor frame is anchor _ h, the actual pixel coordinates of the rectangular frame of the structured information are as follows:
wherein, (x _ pixel, y _ pixel) is the middle point of the rectangular frame, w _ pixel is the width of the rectangular frame, and h _ pixel is the height of the rectangular frame;
in the above calculation formula, only [ x _ n _ y _ n _ w _ n _ h _ n ] is obtained by network prediction, and the others are preset information, so that: the pedestrian structural information rectangular frame is directly obtained by the improved Yolox model, and the accumulation of intermediate errors is avoided without the intermediate result of the pedestrian rectangular frame information.
9. The end-to-end detection method for the pedestrian structured information and the subordinate relationship thereof according to claim 1, characterized in that for the loss function loss during model training, on the basis of Yolox, a structured information foreground probability loss attr _ obj _ loss and a pedestrian structured information rectangular box regression loss attr _ reg _ loss are added, wherein the structured information foreground probability loss attr _ obj _ loss adopts a cross entropy loss function bceath logitss loss, and the pedestrian structured information rectangular box regression loss attr _ reg _ loss adopts a cross-over ratio loss function ioulos; and adding newly added structured information foreground probability loss attr _ obj _ loss and pedestrian structured information rectangular frame regression loss attr _ reg _ loss to the original loss function to obtain a final loss function.
10. The end-to-end pedestrian structural information and dependency detection method according to claim 1, wherein an activation function of the Yolox model is set to relu, and a decoupling head channel coefficient is set to 0.5.
11. A computer-readable medium characterized by: the computer readable medium stores a computer program, which is loaded and executed by a processing module to implement the method for detecting end-to-end pedestrian structural information and its dependency relationship according to any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211229170.8A CN115311690B (en) | 2022-10-08 | 2022-10-08 | End-to-end pedestrian structural information and dependency relationship detection method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211229170.8A CN115311690B (en) | 2022-10-08 | 2022-10-08 | End-to-end pedestrian structural information and dependency relationship detection method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115311690A true CN115311690A (en) | 2022-11-08 |
| CN115311690B CN115311690B (en) | 2022-12-23 |
Family
ID=83866505
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211229170.8A Active CN115311690B (en) | 2022-10-08 | 2022-10-08 | End-to-end pedestrian structural information and dependency relationship detection method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115311690B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114299011A (en) * | 2021-12-27 | 2022-04-08 | 杭州电子科技大学 | Remote sensing target quadrilateral frame rapid detection method based on deep learning |
| CN114332919A (en) * | 2021-12-11 | 2022-04-12 | 南京行者易智能交通科技有限公司 | Pedestrian detection method and device based on multi-spatial relationship perception and terminal equipment |
| US20220180534A1 (en) * | 2020-12-04 | 2022-06-09 | Boe Technology Group Co., Ltd. | Pedestrian tracking method, computing device, pedestrian tracking system and storage medium |
| CN114863366A (en) * | 2022-05-30 | 2022-08-05 | 上海数川数据科技有限公司 | Cross-camera multi-target tracking method based on prior information |
-
2022
- 2022-10-08 CN CN202211229170.8A patent/CN115311690B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220180534A1 (en) * | 2020-12-04 | 2022-06-09 | Boe Technology Group Co., Ltd. | Pedestrian tracking method, computing device, pedestrian tracking system and storage medium |
| CN114332919A (en) * | 2021-12-11 | 2022-04-12 | 南京行者易智能交通科技有限公司 | Pedestrian detection method and device based on multi-spatial relationship perception and terminal equipment |
| CN114299011A (en) * | 2021-12-27 | 2022-04-08 | 杭州电子科技大学 | Remote sensing target quadrilateral frame rapid detection method based on deep learning |
| CN114863366A (en) * | 2022-05-30 | 2022-08-05 | 上海数川数据科技有限公司 | Cross-camera multi-target tracking method based on prior information |
Non-Patent Citations (1)
| Title |
|---|
| YIGUAN LIAO ET AL.: ""Animal Attribute Recognition via Multi-task Learning Based on YOLOX", 《ICVIP 2021》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115311690B (en) | 2022-12-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102635987B1 (en) | Method, apparatus, device and storage medium for training an image semantic segmentation network | |
| CN110569701B (en) | Computer-implemented vehicle damage assessment method and device | |
| CN108256506B (en) | Method and device for detecting object in video and computer storage medium | |
| CN112750150B (en) | Vehicle flow statistical method based on vehicle detection and multi-target tracking | |
| CN103455542B (en) | Multiclass evaluator and multiclass recognition methods | |
| US20210326638A1 (en) | Video panoptic segmentation | |
| CN111274926B (en) | Image data screening method, device, computer equipment and storage medium | |
| JP2022025008A (en) | License plate recognition method based on text line recognition | |
| CN106650670A (en) | Method and device for detection of living body face video | |
| CN112801236B (en) | Image recognition model migration method, device, equipment and storage medium | |
| CN111783844A (en) | Target detection model training method and device based on deep learning and storage medium | |
| CN112613387A (en) | Traffic sign detection method based on YOLOv3 | |
| CN112749653A (en) | Pedestrian detection method, device, electronic equipment and storage medium | |
| CN113378675A (en) | Face recognition method for simultaneous detection and feature extraction | |
| CN112712051A (en) | Object tracking method and device, computer equipment and storage medium | |
| CN116964588A (en) | Target detection method, target detection model training method and device | |
| CN109934096B (en) | Automatic driving visual perception optimization method based on characteristic time sequence correlation | |
| CN114764870A (en) | Object positioning model processing method, object positioning device and computer equipment | |
| CN114550148A (en) | Deep learning-based identification, detection and counting method and system for severely shielded commodities | |
| Rong et al. | Big data intelligent tourism management platform design based on abnormal behavior identification | |
| CN111008622B (en) | Image object detection method and device and computer readable storage medium | |
| CN113111708A (en) | Vehicle matching sample generation method and device, computer equipment and storage medium | |
| CN115311690B (en) | End-to-end pedestrian structural information and dependency relationship detection method thereof | |
| CN111914809A (en) | Target object positioning method, image processing method, device and computer equipment | |
| CN111598000A (en) | Face recognition method, device, server and readable storage medium based on multiple tasks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |