CN112380913A - License plate detection and identification method based on combination of dynamic adjustment and local feature vector - Google Patents
License plate detection and identification method based on combination of dynamic adjustment and local feature vector Download PDFInfo
- Publication number
- CN112380913A CN112380913A CN202011130554.5A CN202011130554A CN112380913A CN 112380913 A CN112380913 A CN 112380913A CN 202011130554 A CN202011130554 A CN 202011130554A CN 112380913 A CN112380913 A CN 112380913A
- Authority
- CN
- China
- Prior art keywords
- license plate
- network
- recognition
- feature
- identification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
- G06V20/54—Surveillance or monitoring of activities, e.g. for recognising suspicious objects of traffic, e.g. cars on the road, trains or boats
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
- G06V20/625—License plates
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a combined license plate detection and identification method based on dynamic adjustment and local feature vectors, which comprises the following steps: carrying out feature extraction on the picture containing the license plate image through a convolution network; the extracted feature images are respectively sent to a license plate positioning network and a license plate recognition network for processing, the license plate positioning network acquires the license plate position in the feature images, and the license plate recognition network recognizes the content of the license plate; cutting the recognition characteristic graph output by the license plate recognition network according to the position of the license plate in the characteristic graph, so that the region of the characteristic graph obtained by cutting corresponds to the position of the license plate in the original picture; and identifying the characteristic image area to obtain a license plate identification result. The method has the advantages that the features extracted by the convolutional network can be reused in the identification stage, the calculated amount is greatly reduced, the reused features can fully learn the surrounding attributes of the license plate, the license plate identification under different environmental conditions is more robust, and the method can be operated on embedded equipment in real time to meet the commercialization requirements.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a license plate detection and identification method based on dynamic adjustment and local feature vector combination.
Background
After the second industrial revolution, automobiles are created, the life style of people is greatly changed, the range of activities of people is expanded, the automobile ownership is explosively increased since the 21 st century, the vehicle management work is increased, and the urban traffic pressure is also heavier. The checking of behaviors such as monitoring and positioning, tracking, random parking and random release, violation of cross regulations and the like, tracking and escaping and the like which depend on license plate recognition require a great deal of manpower to check thousands of cameras, which causes waste of a great deal of labor cost, and human factors also have great influence. With the development of the third industrial revolution, the recognition of license plates becomes possible.
The license plate recognition technology was first proposed in the 80's of the 20 th century and has been extensively studied. The initial license plate recognition technology uses a traditional algorithm for recognition, and influences accuracy on environment, license plate position angle and pollution defect of the license plate. At present, a typical license plate recognition system firstly cuts a picture containing a license plate and then recognizes the cut picture, and the mode usually needs strict cutting operation to reduce the influence on the accuracy.
Disclosure of Invention
In order to overcome the defects, the invention provides a license plate detection and identification method based on dynamic adjustment and local feature vector combination, and an optimized convolutional neural network model is adopted, so that the features extracted by network detection can be reused in an identification stage, and the speed and the accuracy of license plate detection and identification are improved.
The invention provides a license plate detection and identification method based on dynamic adjustment and local feature vector combination, which comprises the following steps:
carrying out feature extraction on the picture containing the license plate image through a convolution network; the extracted feature images are respectively sent to a license plate positioning network and a license plate recognition network for processing, the license plate positioning network acquires the license plate position in the feature images, and the license plate recognition network recognizes the content of the license plate; cutting the recognition characteristic graph output by the license plate recognition network according to the position of the license plate in the characteristic graph, and dynamically adjusting to enable the region of the feature graph obtained by cutting to correspond to the position of the license plate in the original picture; and identifying the characteristic image area to obtain a license plate identification result.
Further, the specific process of acquiring the license plate position in the feature map by the license plate positioning network is to classify the feature points in the feature map and judge whether the category of the feature points is the license plate; and performing frame regression on the license plate region by taking the position of the pixel point judged as the license plate category as a midpoint to obtain the accurate position of the license plate.
Further, the cutting of the recognition feature map output by the license plate recognition network according to the position of the license plate in the feature map specifically comprises: obtaining a license plate position list according to the position of the license plate; and cutting the recognition feature map according to the license plate position list to obtain a feature list corresponding to the number of license plates.
Further, identifying the feature map region specifically includes: adjusting the characteristic diagram area into characteristic attributes with fixed width; performing secondary extraction on the feature map region by using a convolution network to obtain a two-dimensional feature vector with the height of 1; and classifying the two-dimensional characteristic vectors, judging the category of the single pixel point, and encoding the category into a license plate identification result.
Further, the method also comprises the following steps: and performing license plate recognition training on the cut feature map region by using a loss function.
Furthermore, the license plate positioning network and the license plate identification network are multiplexed by the same convolution network.
Further, the convolutional network comprises a convolutional layer, a pooling layer, an activation layer and a normalization layer, wherein the convolutional layer is used for extracting the spatial feature attributes of the pictures, the pooling layer is used for reducing the plane dimension of the feature maps, the activation layer is a nonlinear function and enables the convolutional network to have nonlinear expression capability, and the normalization layer is used for rearranging the distribution of data to robustly express the feature maps.
In a second aspect of the present invention, a storage medium is provided, which includes a program stored in the storage medium, and when the program runs, the apparatus where the storage medium is located is controlled to execute the license plate detection and identification method according to any one of the above technical solutions.
The third aspect of the present invention provides a license plate detection and recognition device, which includes a processor, where the processor is configured to run a program, and the program executes the license plate detection and recognition method according to any one of the above technical solutions when running.
The invention has the beneficial effects that:
1) and designing and identifying networks by utilizing the convolution layer, the pooling layer and the normalization layer of the convolution network to extract the features of the picture. The convolution structure extracts the characteristic attributes, the pooling structure reduces the plane dimension, and the normalization structure enhances the relation between data, so that the identification method has robustness under various environmental conditions, and the influence of factors such as license plate inclination angle, license plate defect, image distortion and the like on the accuracy is reduced.
2) Through the action and the relation among all layers of the convolutional network, the number of network layers of the model is optimized, the structure of the network is simplified, and the calculation amount of the model is reduced by continuously reducing the plane dimension of the feature map by utilizing the pooling layer. Under the condition of not influencing the identification accuracy, the optimized feature extraction model can run and detect on the mobile equipment in real time.
3) The loss function ctc loss used by the invention can greatly reduce the workload of manual marking, does not need to finely cut characters on the license plate picture, and can reduce the influence of cutting on the accuracy.
4) The recognition characteristic graph is multiplexed, and the license plate rotation has strong robustness.
5) End-to-end license plate detection and identification, namely, the license plate detection and identification are realized in the same convolutional network, the detection and identification results can be directly obtained from the picture, the original image does not need to be cut and then identified, and the calculation amount is greatly reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flowchart illustrating a license plate detection and identification method based on dynamic adjustment and local feature vector combination according to an embodiment of the present invention;
FIG. 2 is a diagram of the structure of the convolutional network in the embodiment of FIG. 1;
FIG. 3 is a flow chart of the processing of the feature map in the embodiment of FIG. 1.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In order to improve the speed and accuracy of license plate detection and identification, and to improve the accuracy of license plate identification for different illumination intensities (darkness, strong light, local distortion), various weather environments, larger angles of license plates, and the like, the present embodiment provides a license plate detection and identification method based on dynamic adjustment and local feature vector combination, as shown in fig. 1, the method includes the following steps,
and S1, extracting the characteristics of the picture containing the license plate image through a convolution network.
The convolutional network in the embodiment comprises a convolutional layer, a pooling layer, an activation layer and a normalization layer, wherein the convolutional layer extracts the spatial characteristic attribute of a picture, the pooling layer can reduce the plane dimension of a characteristic diagram, and the pooling layer and the activation layer have a common effect to enable the convolutional network to have nonlinear expression capacity, and the activation layer is a nonlinear function; the normalization layer can rearrange the distribution of data to robustly express the feature map.
As shown in fig. 2, the layer structure of the convolutional network is a convolutional network structure including no active layer, and each color represents a convolutional layer, a pooling layer, and a normalization layer. The following illustrates a process of extracting features when a picture enters a convolutional network, and assuming that the dimension of an original picture is 64x 128, after the picture is input into the convolutional network, the picture is firstly convolved by a convolutional layer and then pooled by a pooling layer, where the planar dimension of the feature map is 32x 64. The convolution and pooling are then repeated so that the final feature map dimensions become 1x16x512, representing feature map height, width, and number of channels, respectively. And then, continuing convolution on the extracted feature map with the height of 1, only changing the channel dimension, putting the feature map into sotmax for classification, and training the classification result. Wherein the normalization layer is used when the feature attribute is 8x16x256 to enhance the expressiveness of the feature map.
In general, the structure of the convolutional network may be arranged to include an active layer, followed by each convolutional layer convolution. The activation layer is a nonlinear function, which can enhance the nonlinear expression capability of the convolutional network. In a general implementation, the non-linear problem is faced, and the over-fitting can be prevented by the active layer, so that the robustness of the network is enhanced. In this model, it can be considered that the feature attribute map extracted by the low-level structure can express low-level semantic features of the picture, such as a point-line plane composed of the picture, and the feature map extracted by the deep-level convolutional network is high-level semantic features, such as association types of various parts of the picture. In the embodiment, the recognition is finally carried out through the extracted high-level semantic features.
The convolutional network structure adopted by the embodiment is a simplified light-weight feature extraction model, only the classical convolutional layer, the pooling layer, the normalization layer and other structures are used, the cyclic neural network structure is not used, the calculation amount is reduced, the method can be operated on various mobile platforms, meanwhile, the license plate recognition method of the embodiment has robustness under various environmental conditions, and the influence of factors such as license plate inclination angle, license plate defect and image distortion on the accuracy is reduced.
And S2, respectively sending the extracted feature maps to a license plate positioning network and a license plate recognition network for processing, wherein the license plate positioning network acquires the license plate position in the feature maps, and the license plate recognition network recognizes the content of the license plate.
In this embodiment, the license plate location network and the license plate recognition network jointly multiplex a convolutional network, which is a combined license plate detection and recognition algorithm based on the dynamically adjusted local feature vector, and not only can reduce the workload of manual labeling, but also can correctly recognize the license plate even under the condition of inaccurate location.
Specifically, the specific process of the license plate positioning network for acquiring the license plate position in the feature map is as follows:
s21, classifying the feature points in the feature map, and judging whether the category of the feature points is a license plate;
and S22, performing frame regression on the license plate area by taking the position of the pixel point judged as the license plate category as a midpoint, and obtaining the accurate position of the license plate.
And S3, cutting the recognition characteristic graph output by the license plate recognition network according to the position of the license plate in the characteristic graph, and dynamically adjusting to enable the local characteristic vector obtained by cutting, namely the characteristic graph region, to correspond to the position of the license plate in the original picture.
Specifically, the process of cutting the recognition characteristic graph output by the license plate recognition network according to the position of the license plate in the characteristic graph specifically comprises the following steps:
s31, obtaining a license plate position list according to the position of the license plate;
and S32, cutting the recognition feature map according to the license plate position list to obtain a feature list corresponding to the number of license plates.
And S4, recognizing the characteristic image area to obtain a license plate recognition result.
Specifically, the step of identifying the feature map region includes the following steps:
s41, adjusting the feature map area to a feature attribute with fixed width;
s42, performing secondary feature extraction on the feature map area by using a convolution network to obtain a two-dimensional feature vector with the height of 1;
and S43, classifying the two-dimensional characteristic vectors, judging the category of the single pixel point, and encoding the category into a license plate identification result.
When the license plate is identified, the convolutional network can fully learn the information around the license plate, and even if the positioning is not accurate, the identification algorithm can also identify the correct result. In the license plate recognition stage, when the license plate features are extracted, the dynamic adjustment based on positioning is used for more robustness of license plate recognition under different environmental conditions, and the method can be operated on embedded equipment in real time to meet the commercialization requirement.
In some embodiments, the license plate recognition training is further performed on the cut feature map region by using a loss function. The loss function adopted by the embodiment is ctc loss, so that the workload of manual marking is greatly reduced, characters do not need to be finely cut for the license plate picture, and the influence of cutting on the accuracy is reduced.
The traditional solution of license plate recognition is to cut character pictures into pictures, then recognize characters separately, and finally synthesize recognition results. In the embodiment, the direct prediction sequence result is used, operations such as binarization, cutting, recognition, comprehensive recognition result and the like are not required to be performed on the picture, the collected license plate is directly sent into a convolutional network, the extracted result is firstly classified through softmax, then the length of the whole license plate sequence is coded through a coder, the coding is completed, the whole recognized serialization result of the license plate picture is directly returned, the whole license plate sequence label is output, and the post-processing is not required.
The invention also provides a storage medium and a license plate recognition device, wherein the storage medium comprises a program, and when the program runs, the device where the storage medium is located is controlled to execute the license plate recognition method in any technical scheme. The storage medium may be a volatile memory (volatile memory), such as a random-access memory (RAM); a non-volatile memory (non-volatile) such as a flash memory (flash memory), a hard disk (HDD) or a solid-state drive (SSD); combinations of the above categories of memory may also be included.
The license plate recognition device comprises a processor, wherein the processor is used for running a program, and the program executes the license plate recognition method in any technical scheme when running. The processor may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a field-programmable gate array (FPGA), a General Array Logic (GAL), or any combination thereof.
Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus, devices (systems) or computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such as a smartphone, tablet, or the like, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the embodiments of the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to encompass such modifications and variations.
Claims (9)
1. The license plate detection and identification method based on the combination of dynamic adjustment and local feature vectors is characterized by comprising the following steps of:
carrying out feature extraction on the picture containing the license plate image through a convolution network;
the extracted feature images are respectively sent to a license plate positioning network and a license plate recognition network for processing, the license plate positioning network acquires the license plate position in the feature images, and the license plate recognition network recognizes the content of the license plate;
cutting the recognition characteristic graph output by the license plate recognition network according to the position of the license plate in the characteristic graph, and dynamically adjusting to enable the region of the feature graph obtained by cutting to correspond to the position of the license plate in the original picture;
and identifying the characteristic image area to obtain a license plate identification result.
2. The license plate detection and identification method according to claim 1, wherein the license plate location network obtains the license plate location in the feature map by a specific process,
classifying the feature points in the feature map, and judging whether the class of the feature points is a license plate;
and performing frame regression on the license plate region by taking the position of the pixel point judged as the license plate category as a midpoint to obtain the accurate position of the license plate.
3. The license plate detection and identification method according to claim 1, wherein the cutting of the identification feature map output by the license plate identification network according to the position of the license plate in the feature map comprises:
obtaining a license plate position list according to the position of the license plate;
and cutting the recognition feature map according to the license plate position list to obtain a feature list corresponding to the number of license plates.
4. The license plate detection and identification method of claim 1, wherein identifying the feature map region specifically comprises:
adjusting the characteristic diagram area into characteristic attributes with fixed width;
performing secondary extraction on the feature map region by using a convolution network to obtain a two-dimensional feature vector with the height of 1;
and classifying the two-dimensional characteristic vectors, judging the category of the single pixel point, and encoding the category into a license plate identification result.
5. The license plate detection and identification method of claim 1, further comprising the steps of: and performing license plate recognition training on the cut feature map region by using a loss function.
6. The license plate detection and identification method according to any one of claims 1 to 5, wherein the license plate location network and the license plate identification network are multiplexed by the same convolutional network.
7. The license plate detection and identification method of claim 6, wherein the convolutional network comprises a convolutional layer, a pooling layer, an activation layer and a normalization layer, wherein the convolutional layer is used for extracting spatial feature attributes of the picture, the pooling layer is used for reducing feature map plane dimensions, the activation layer is a nonlinear function and enables the convolutional network to have nonlinear expression capability, and the normalization layer is used for rearranging data distribution to robustly express the feature map.
8. A storage medium, characterized by: the license plate detection and recognition system comprises a program stored in the storage medium, and when the program runs, the device where the storage medium is located is controlled to execute the license plate detection and recognition method according to any one of claims 1 to 7.
9. A license plate detection and recognition device, characterized by comprising a processor, wherein the processor is used for running a program, and the program is used for executing the license plate detection and recognition method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011130554.5A CN112380913A (en) | 2020-10-21 | 2020-10-21 | License plate detection and identification method based on combination of dynamic adjustment and local feature vector |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011130554.5A CN112380913A (en) | 2020-10-21 | 2020-10-21 | License plate detection and identification method based on combination of dynamic adjustment and local feature vector |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112380913A true CN112380913A (en) | 2021-02-19 |
Family
ID=74580495
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011130554.5A Pending CN112380913A (en) | 2020-10-21 | 2020-10-21 | License plate detection and identification method based on combination of dynamic adjustment and local feature vector |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112380913A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115376119A (en) * | 2022-10-25 | 2022-11-22 | 珠海亿智电子科技有限公司 | License plate recognition method and device, license plate recognition equipment and storage medium |
-
2020
- 2020-10-21 CN CN202011130554.5A patent/CN112380913A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115376119A (en) * | 2022-10-25 | 2022-11-22 | 珠海亿智电子科技有限公司 | License plate recognition method and device, license plate recognition equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103279756B (en) | Vehicle detection based on integrated classifier analyzes system and determination method thereof | |
| CN108268867B (en) | License plate positioning method and device | |
| CN113033604B (en) | Vehicle detection method, system and storage medium based on SF-YOLOv4 network model | |
| CN109993138A (en) | A kind of car plate detection and recognition methods and device | |
| CN111767927A (en) | Lightweight license plate recognition method and system based on full convolution network | |
| CN111160205B (en) | Method for uniformly detecting multiple embedded types of targets in traffic scene end-to-end | |
| Do et al. | Automatic license plate recognition using mobile device | |
| KR102105954B1 (en) | System and method for accident risk detection | |
| CN110909598B (en) | Non-motor vehicle lane traffic violation driving identification method based on deep learning | |
| CN111274926B (en) | Image data screening method, device, computer equipment and storage medium | |
| CN108846387B (en) | Traffic police gesture recognition method and device | |
| CN111435446A (en) | License plate identification method and device based on L eNet | |
| CN115830399B (en) | Classification model training method, device, equipment, storage medium and program product | |
| CN111008554B (en) | Deep learning-based method for identifying pedestrians without giving away in dynamic traffic zebra stripes | |
| Parvin et al. | Vehicle number plate detection and recognition techniques: a review | |
| Kiew et al. | Vehicle route tracking system based on vehicle registration number recognition using template matching algorithm | |
| CN117197763A (en) | Road crack detection method and system based on cross attention guide feature alignment network | |
| CN116597270A (en) | Road damage target detection method based on attention mechanism integrated learning network | |
| CN111401421A (en) | Image category determination method based on deep learning, electronic device, and medium | |
| CN112785610B (en) | Lane line semantic segmentation method integrating low-level features | |
| CN112380913A (en) | License plate detection and identification method based on combination of dynamic adjustment and local feature vector | |
| CN115631197B (en) | Image processing method, device, medium, equipment and system | |
| CN116071557A (en) | Long tail target detection method, computer readable storage medium and driving device | |
| Pramanik et al. | Detection of Potholes using Convolutional Neural Network Models: A Transfer Learning Approach | |
| Kodwani et al. | Automatic license plate recognition in real time videos using visual surveillance techniques |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |