CN115690496A - Real-time regional intrusion detection method based on YOLOv5 - Google Patents
Real-time regional intrusion detection method based on YOLOv5 Download PDFInfo
- Publication number
- CN115690496A CN115690496A CN202211324933.7A CN202211324933A CN115690496A CN 115690496 A CN115690496 A CN 115690496A CN 202211324933 A CN202211324933 A CN 202211324933A CN 115690496 A CN115690496 A CN 115690496A
- Authority
- CN
- China
- Prior art keywords
- intrusion
- detection
- area
- bbox
- polygonal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Burglar Alarm Systems (AREA)
Abstract
The invention discloses a real-time regional intrusion detection method based on YOLOv5, which comprises the following steps: selecting a YOLOv5 model; dividing the detection function of the YOLOv5 model into picture detection, local video detection and real-time camera detection; if the picture detection is selected in the YOLOv5 model, reading a local picture, selecting an identified object by adopting a bbox frame, and acquiring bbox coordinate prediction information and confidence; if the local video detection or the real-time camera detection is selected, accessing a video stream, and judging whether to execute a region intrusion instruction, if so, executing a step 5, otherwise, executing a step 4; adopting a bbox frame to select an identified object, and acquiring bbox coordinate prediction information and confidence; firstly, a polygonal invasion area is generated, then target detection of the invasion area is executed on the video frame, bbox coordinate information, invasion types and confidence degrees in the invasion area are obtained, and alarm information is generated. The invention judges whether the area invasion exists by adopting the relative position of the bbox central point and the invasion area, and has good real-time performance and high efficiency.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a real-time regional intrusion detection method based on YOLOv 5.
Background
With the development of deep learning technology and the generation of massive video monitoring resources, the requirements are far satisfied by analyzing and processing video resources only through manpower, an intelligent video identification technology is explored and applied, illegal behaviors of key monitoring areas are intelligently analyzed and early-warning is carried out, so that the problem of time and labor consumption can be solved, but the existing area intrusion detection method is poor in real-time performance and low in detection precision, and therefore a method for monitoring the object intrusion detection area through the video, which is high in judgment precision and good in real-time performance, is urgently needed to be invented.
Disclosure of Invention
The invention aims to provide a real-time regional intrusion detection method based on YOLOv5 aiming at the defects of the prior art, the method uses the relative position relation between the central point of the prediction result and the polygonal intrusion region to rapidly judge whether regional intrusion exists, and the method has good real-time performance and high efficiency.
In order to solve the technical problems, the invention adopts the following technical scheme:
a real-time regional intrusion detection method based on YOLOv5 comprises the following steps:
step 1, selecting a data set to train a Yolov5 model to obtain a trained Yolov5 model;
step 2, initializing a trained YOLOv5 model, and dividing the detection function of the YOLOv5 model into picture detection, local video detection and real-time camera detection;
step 3, selecting a corresponding detection function in the YOLOv5 model to detect a detection target; if the picture detection is selected, reading the corresponding local picture, executing target detection, selecting an identified object by adopting a bbox frame, acquiring bbox coordinate prediction information and confidence coefficient, and outputting and displaying a result; if the local video detection is selected, firstly judging whether to execute a regional intrusion instruction, if so, executing a step 5, otherwise, executing a step 4; if the real-time camera is selected for detection, accessing a video stream, judging whether to execute a region intrusion instruction, if so, executing a step 5, otherwise, executing a step 4;
step 4, according to the FPS attached to the video, performing target detection on the video frame, selecting an identified object by adopting a bbox frame, acquiring bbox coordinate prediction information and confidence, and outputting and displaying a result;
step 5, according to the FPS attached to the video, if a JSON file for uploading local area coordinate information is selected, firstly generating a polygonal intrusion area, then executing target detection of the intrusion area on a video frame, acquiring bbox coordinate information, intrusion type and confidence coefficient in the intrusion area, and generating alarm information; and if the real-time drawing of the intrusion area is selected, connecting points to form a line to form a polygonal intrusion area, then performing target detection on the intrusion area on the video frame, acquiring bbox coordinate information, intrusion type and confidence coefficient in the intrusion area, and generating alarm information.
Further, in step 3, the resolutions of the picture and the video are unified before performing the object detection.
Further, the method for drawing the intrusion area in real time in step 5 comprises the following steps:
carrying out a regional invasion mode to ensure that the drawing instruction can be identified;
sending a drawing instruction in the area range of the result image output by combining the picture, video or real-time camera monitoring result in the step 3 to complete the operation of creating a single point of the polygonal boundary, and connecting the polygonal boundary to form an intrusion area after three or more different points are created by repeatedly executing the drawing instruction;
and if the polygon needs to be redrawn, sending a clearing instruction to clear all polygon points created previously.
Further, the method for detecting an intrusion object in the polygonal area performed on the video frame in step 5 is as follows:
acquiring a midpoint coordinate of a bbox frame of the object identified by frame selection;
and comparing the position relation between the midpoint coordinate of the bbox frame and the polygonal area, judging whether the midpoint is in the polygonal boundary, leading out a plurality of rays from the midpoint to the outside, counting the number of intersection points of the plurality of rays and the polygonal boundary, if the number of the intersection points is odd, indicating that the midpoint of the bbox frame is in the polygonal area, judging that the object invades the polygonal area, and if the number of the intersection points is even, indicating that the midpoint of the bbox frame is outside the polygonal area, and judging that the object does not invade the polygonal area.
Further, the method for generating the alarm information visualization in step 5 is as follows:
setting an alarm triggering rule of regional invasion that the midpoint of bbox of the framed object is positioned in a polygonal region of the current frame image; the alarm elimination rule of the area intrusion is that the bbox midpoint coordinate of the framed object is not in the polygonal area of the current frame image;
after the YOLOv5 model identifies the object type and the polygonal area of the object in the image, drawing bbox coordinate information, the object type and confidence; and (3) under a region intrusion mode, loading and displaying the polygonal region drawn in the step (5) on the current frame image, judging whether the current frame image has an object meeting the requirement in the polygonal warning region, if so, updating the bbox midpoint coordinate of the object in the warning information frame, and simultaneously judging whether the object belongs to the person category, if so, counting the number of intrusion people in the current frame image, and simultaneously updating the confidence coefficient for judging that the object is in the person category.
Compared with the prior art, the invention has the following beneficial effects: according to the detection model, the YOLOv5 is adopted, so that the operation requirement of the model is lowered, the precision and the speed are considered, and the image, the video and the camera can be accurately detected; the method is used for processing the video stream in real time based on the YOLOv5, and quickly judging whether the regional invasion exists or not by using the relative position relation between the central point of the prediction result and the polygon, and is good in real-time performance and high in efficiency.
Drawings
Fig. 1 is a schematic flowchart of a real-time regional intrusion detection method based on YOLOv5 according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a region intrusion according to an embodiment of the present invention;
fig. 3 is a diagram of the results of the area intrusion detection according to the embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the following embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the embodiments and features of the embodiments of the present invention may be combined with each other without conflict.
The present invention is further illustrated by the following examples, which are not to be construed as limiting the invention.
The embodiment of the invention provides a real-time regional intrusion detection method based on YOLOv5, which fully utilizes the advantages of small YOLOv5 model and high detection speed in regional intrusion detection, ensures the stability and accuracy of an algorithm, reduces the operation requirement of the model, reduces the detection time and realizes the real-time performance of regional intrusion detection. YOLOv5 has higher accuracy than previous YOLO series algorithms, and has higher detection speed than FCN series two-step detection methods.
Referring to fig. 1 in detail, fig. 1 is a schematic general flow chart of a real-time regional intrusion detection method based on YOLOv5 in the present embodiment. As shown in fig. 1, the method for detecting an intrusion into a region of the present embodiment specifically includes the following steps:
step 1, selecting a data set to train a YOLOv5 model to obtain a trained YOLOv5 model; the YOLOv5 model includes YOLOv5m, YOLOv5s, and YOLOv5x with successively higher detection precision, in this embodiment, the YOLOv5 model is pre-trained through the COCO data set to obtain the relevant weight parameters, and the pre-trained weight parameters are selected as the weight parameters of the YOLOv5 model.
Step 2, initializing a trained YOLOv5 model, and dividing the detection function of the YOLOv5 model into picture detection, local video detection and real-time camera detection;
and initializing the YOLOv5 model according to the selected weight parameters to enable the model to have the target detection capability. According to different object types of target detection, the detection function of the YOLOv5 model is divided into three functions of picture detection, local video detection and real-time camera detection.
Step 3, selecting a corresponding detection function in the YOLOv5 model to detect a detection target; if the picture detection is selected, reading a corresponding local picture, executing target detection, selecting an identified object by adopting a bbox frame, acquiring bbox coordinate prediction information and confidence, and outputting and displaying a result; if the video detection is selected, firstly judging whether to execute the area intrusion function, if so, executing a step 5, otherwise, executing a step 4; if the real-time camera is selected for detection, accessing a video stream, judging whether to execute an area intrusion function, if so, executing a step 5, otherwise, executing a step 4;
and when the picture detection function in the YOLOv5 model is selected, reading the picture file which is selected by the user and stored locally. In order to avoid a large resolution difference between different pictures, the loaded picture resolution is uniformly scaled to a preset resolution, which is 640 × 480 in the embodiment, target detection is performed by using a YOLOv5 model, an object in the picture is identified, the identified object position is selected by using a bbox frame, the category of the object is identified, bbox coordinate information and confidence coefficient are obtained, and then the information is output and displayed.
When the local video detection function in the YOLOv5 model is selected, reading a video file which is selected by a user and stored locally, splitting the video into one frame and one frame of image, and performing target detection on each frame of image, wherein the number of Frames Per Second (FPS) of the split video is consistent with that of the original video; when the real-time camera detection function in the YOLOv5 model is selected, the provided cameras in a plurality of different places can be selected as input video streams, the places and the corresponding RTMP video streams are associated, in this embodiment, the FPS is controlled to be 25, and then each frame of image is displayed and target detection is performed.
The steps of the YOLOv5 model for executing the functions of local video detection and real-time camera detection are as follows:
the image is uniformly scaled in size and adjusted to the resolution size of 640 multiplied by 480;
and performing target detection on the current frame image by using a YOLOv5 model, selecting the identified object position by using a bbox frame, identifying the object type, and acquiring information such as a bbox coordinate, confidence coefficient and the like.
And then judging whether to execute the regional intrusion instruction when executing the local video detection and real-time camera detection functions, if so, executing the step 5, otherwise, executing the step 4.
Step 4, according to the FPS attached to the video, performing target detection on the video frame, selecting an identified object by adopting a bbox frame, acquiring bbox coordinate prediction information and confidence, and outputting and displaying a result;
judging whether to execute the 'area intrusion' instruction, if not, displaying the detection result information of the current frame, namely selecting the identified object position by adopting a bbox frame, identifying the object type, acquiring information such as bbox coordinates, confidence coefficient and the like, and updating the result of each frame in a result output window of the system, wherein the function is similar to the picture detection function.
Step 5, according to the FPS attached to the video, if a JSON file for uploading local area coordinate information is selected, firstly generating a polygonal intrusion area, then executing target detection of the intrusion area on a video frame, acquiring bbox coordinate information, intrusion type and confidence coefficient in the intrusion area, and generating alarm information; if the real-time drawing of the intrusion area is selected, connecting points to form a line to form a polygonal intrusion area, then performing target detection on the intrusion area on the video frame, acquiring bbox coordinate information, intrusion type and confidence coefficient in the intrusion area, and generating alarm information;
if the 'area intrusion' instruction is executed, the specific steps of executing the area intrusion instruction in the local video detection function and the real-time camera detection function of the YOLOv5 model are as follows:
and drawing the polygon boundary of the intrusion area in an image output window, wherein the drawing mode is divided into two modes of mouse drawing and local area JSON file loading. The specific steps of mouse drawing are as follows:
when the mouse moves in the image output window, automatically acquiring the (x, y) coordinate of the current mouse relative to the upper left corner (0,0) of the image output window;
clicking a left mouse button to serve as a trigger event for drawing a polygonal boundary point of an intrusion area, and adding the current (x, y) coordinates of the mouse to an ordered list when clicking the left mouse button;
when three or more different (x, y) points are stored in the sequence table, the points are connected into a polygon according to the adding sequence, and then the polygon boundary is redrawn every time a new point is added;
and if the drawn invasion area is not satisfactory, clicking a right mouse button, and after receiving a clearing instruction of the right button, clearing all points in the polygon ordered list to redraw the invasion area.
The specific steps for loading the JSON file of the local region are as follows:
drawing an interested region in a video through software and deriving a corresponding JSON file, wherein the JSON file comprises a two-dimensional array of region coordinates { [ x ] 1 ,y 1 ],[x 2 ,y 2 ],……,[x n ,y n ]In which x n And y n (x, y) coordinates representing a point, n =1,2,3,4, … …;
reading each point in JSON, storing the point in the ordered list, and drawing the polygon boundary according to the order.
The problem that the intrusion areas drawn by the two methods need to be considered is that before drawing, images with different resolution sizes need to be uniformly adjusted to 640 × 480 resolution in advance so as to be convenient for subsequent result display, and the area intrusion algorithm is performed on the image with the original size, so that the proportional relation between the original size of the image and the current size needs to be acquired, otherwise, a certain offset exists between the drawn and displayed area and the actually detected area. Therefore, two sets of polygon ordered lists need to be maintained, one set is used for drawing and displaying the intrusion area, and the other set is used for detecting whether the object crosses the boundary by using the area intrusion algorithm. The point coordinates of the two sets of lists are converted into each other through the proportional relation between the original size and the current size of the image.
After reading the polygon boundary of the intrusion area, judging whether the object intrudes into the drawn intrusion area, as shown in fig. 2, specifically including the following steps:
performing target detection through a YOLOv5 model to obtain the midpoint of a bbox frame of the framed object;
leading out a plurality of rays from the middle point to the outside;
if the number of the intersection points of the ray and the boundary of the invasion area is even, the object is not in the invasion area, and the object is judged not to invade the invasion area; if the number of the intersection points is odd, the object is positioned in the intrusion area, and the object is judged to intrude into the intrusion area.
When the object is judged to be in the intrusion area, screening is carried out according to the obtained object type information in the intrusion area, in the embodiment, only the person type is concerned, so that the object judged to be person is counted in the information of 'illegal intrusion people number', and the coordinates, the intrusion type and the confidence coefficient of all the objects bbox in the intrusion area of the current frame are used as alarm information and displayed in a result output window of the system. The demonstration effect of the system area intrusion method is shown in fig. 3, the middle area is an image output window of the system and displays one frame of a local video, the polygon is the boundary of an intrusion area, and the right detection information counts the number of intrusion people and specific information of a current frame.
In summary, the present embodiment provides a real-time regional intrusion detection method and system based on YOLOv5, which utilize a pre-trained YOLOv5 model to divide the model into three functional modules for different main targets to be detected: 1) The image detection module is used for reading a local image and carrying out target detection by using a YOLOv5 model; 2) The local video detection module is used for reading a local video, performing target detection by using YOLOv5, drawing an intrusion area for each frame of video picture as required, judging the type, bbox coordinate information and confidence coefficient of an object, and judging whether the object is in the area or not by the number of intersection points of a ray led out from the midpoint of bbox and the boundary of the intrusion area, thereby achieving the effect of real-time area intrusion detection; 3) And the real-time camera detection module is used for accessing HTTP video streams to perform target detection, drawing an intrusion area for each frame of video picture as required, judging the object type, bbox coordinate information and confidence coefficient, and judging whether the object is in the area or not through the number of intersection points of rays led out from the midpoint of the bbox and the boundary of the intrusion area, thereby achieving the effect of real-time area intrusion detection. The system outputs the detection results obtained by each functional module, and counts the number and specific information of the invading objects, thereby achieving the landing application of the real-time regional invasion detection method.
While the invention has been described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.
Claims (5)
1. A real-time regional intrusion detection method based on YOLOv5 is characterized by comprising the following steps:
step 1, selecting a data set to train a Yolov5 model to obtain a trained Yolov5 model;
step 2, initializing a trained YOLOv5 model, and dividing the detection function of the YOLOv5 model into picture detection, local video detection and real-time camera detection;
step 3, selecting a corresponding detection function in the YOLOv5 model to detect a detection target; if the picture detection is selected, reading a corresponding local picture, executing target detection, selecting an identified object by adopting a bbox frame, acquiring bbox coordinate prediction information and confidence, and outputting and displaying a result; if the local video detection is selected, firstly judging whether to execute a regional intrusion instruction, if so, executing a step 5, otherwise, executing a step 4; if the real-time camera is selected for detection, accessing a video stream, judging whether to execute a region intrusion instruction, if so, executing a step 5, otherwise, executing a step 4;
step 4, according to the FPS attached to the video, performing target detection on the video frame, selecting an identified object by adopting a bbox frame, acquiring bbox coordinate prediction information and confidence, and outputting and displaying a result;
step 5, according to the FPS attached to the video, if a JSON file for uploading local area coordinate information is selected, firstly generating a polygonal intrusion area, then executing target detection of the intrusion area on a video frame, acquiring bbox coordinate information, intrusion type and confidence coefficient in the intrusion area, and generating alarm information; and if the real-time drawing of the intrusion area is selected, connecting points to form a line to form a polygonal intrusion area, then performing target detection on the intrusion area on the video frame, acquiring bbox coordinate information, intrusion type and confidence coefficient in the intrusion area, and generating alarm information.
2. The YOLOv 5-based real-time intrusion detection method according to claim 1, wherein in step 3, the resolutions of the pictures and the videos are unified before the target detection is performed.
3. The YOLOv 5-based real-time regional intrusion detection method of claim 1, wherein the method for drawing the intrusion region in real time in step 5 is as follows:
carrying out a regional invasion mode to ensure that the drawing instruction can be identified;
sending a drawing instruction in the area range of the result image output by combining the picture, video or real-time camera monitoring result in the step 3 to complete the operation of creating a single point of the polygonal boundary, and connecting the polygonal boundary to form an intrusion area after three or more different points are created by repeatedly executing the drawing instruction;
and if the polygon needs to be redrawn, sending a clearing instruction to clear all polygon points created previously.
4. The YOLOv 5-based real-time regional intrusion detection method of claim 1, wherein the step 5 of performing polygonal regional intrusion target detection on the video frame comprises:
acquiring a midpoint coordinate of a bbox frame of the object identified by frame selection;
and comparing the position relation between the midpoint coordinate of the bbox frame and the polygonal area, judging whether the midpoint is in the polygonal boundary, leading out a plurality of rays from the midpoint to the outside, counting the number of intersection points of the plurality of rays and the polygonal boundary, if the number of the intersection points is odd, indicating that the midpoint of the bbox frame is in the polygonal area, judging that the object invades the polygonal area, and if the number of the intersection points is even, indicating that the midpoint of the bbox frame is outside the polygonal area, and judging that the object does not invade the polygonal area.
5. The YOLOv 5-based real-time regional intrusion detection method according to claim 1, wherein the method for generating the alarm information visualization in the step 5 comprises:
setting an alarm triggering rule of regional invasion that the midpoint of bbox of the framed object is positioned in a polygonal region of the current frame image; the alarm elimination rule of the area intrusion is that the bbox midpoint coordinate of the framed object is not in the polygonal area of the current frame image;
after the YOLOv5 model identifies the object type and the polygonal area where the object is located in the image, drawing bbox coordinate information, the object type and confidence; and (3) under a region intrusion mode, loading and displaying the polygonal region drawn in the step (5) on the current frame image, judging whether the current frame image has an object meeting the requirement in the polygonal warning region, if so, updating the bbox midpoint coordinate of the object in the warning information frame, and simultaneously judging whether the object belongs to the person category, if so, counting the number of intrusion people in the current frame image, and simultaneously updating the confidence coefficient for judging that the object is in the person category.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211324933.7A CN115690496A (en) | 2022-10-27 | 2022-10-27 | Real-time regional intrusion detection method based on YOLOv5 |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211324933.7A CN115690496A (en) | 2022-10-27 | 2022-10-27 | Real-time regional intrusion detection method based on YOLOv5 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115690496A true CN115690496A (en) | 2023-02-03 |
Family
ID=85099280
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211324933.7A Pending CN115690496A (en) | 2022-10-27 | 2022-10-27 | Real-time regional intrusion detection method based on YOLOv5 |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115690496A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116030423A (en) * | 2023-03-29 | 2023-04-28 | 浪潮通用软件有限公司 | Regional boundary intrusion detection method, equipment and medium |
| CN117333995A (en) * | 2023-09-20 | 2024-01-02 | 华能核能技术研究院有限公司 | Yolo-v 3-based high-temperature gas cooled reactor perimeter intrusion detection alarm method and system |
-
2022
- 2022-10-27 CN CN202211324933.7A patent/CN115690496A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116030423A (en) * | 2023-03-29 | 2023-04-28 | 浪潮通用软件有限公司 | Regional boundary intrusion detection method, equipment and medium |
| CN117333995A (en) * | 2023-09-20 | 2024-01-02 | 华能核能技术研究院有限公司 | Yolo-v 3-based high-temperature gas cooled reactor perimeter intrusion detection alarm method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8854469B2 (en) | Method and apparatus for tracking persons and locations using multiple cameras | |
| US6816184B1 (en) | Method and apparatus for mapping a location from a video image to a map | |
| US10810438B2 (en) | Setting apparatus, output method, and non-transitory computer-readable storage medium | |
| CN115690496A (en) | Real-time regional intrusion detection method based on YOLOv5 | |
| CN116188821B (en) | Copyright detection method, system, electronic device and storage medium | |
| CN112669344A (en) | Method and device for positioning moving object, electronic equipment and storage medium | |
| EP3704864B1 (en) | Methods and systems for generating video synopsis | |
| WO2020179065A1 (en) | Image processing device, image processing method, and recording medium | |
| CN111191576A (en) | Personnel behavior target detection model construction method, intelligent analysis method and system | |
| CN112733690B (en) | High-altitude parabolic detection method and device and electronic equipment | |
| CN109815787B (en) | Target identification method and device, storage medium and electronic equipment | |
| CN107122743B (en) | Security monitoring method and device and electronic equipment | |
| CN114003160B (en) | Data visual display method, device, computer equipment and storage medium | |
| US20200349360A1 (en) | Population density determination from multi-camera sourced imagery | |
| CN112422909B (en) | Video behavior analysis management system based on artificial intelligence | |
| CN112530021A (en) | Method, apparatus, device and storage medium for processing data | |
| CN113688820A (en) | Stroboscopic stripe information identification method and device and electronic equipment | |
| US10885704B1 (en) | 3D mapping by distinguishing between different environmental regions | |
| JP2009123150A (en) | Object detection apparatus and method, object detection system and program | |
| CN112528825A (en) | Station passenger recruitment service method based on image recognition | |
| US20180121729A1 (en) | Segmentation-based display highlighting subject of interest | |
| CN113625864B (en) | Virtual scene display method, system, equipment and storage medium based on Internet of things | |
| CN115116136A (en) | Abnormal behavior detection method, device and medium | |
| US9798932B2 (en) | Video extraction method and device | |
| JP2019066909A (en) | Object distribution estimation apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |