CN116402774A - Isolation baffle abnormality detection method and device, electronic equipment and storage medium - Google Patents
Isolation baffle abnormality detection method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116402774A CN116402774A CN202310311470.9A CN202310311470A CN116402774A CN 116402774 A CN116402774 A CN 116402774A CN 202310311470 A CN202310311470 A CN 202310311470A CN 116402774 A CN116402774 A CN 116402774A
- Authority
- CN
- China
- Prior art keywords
- detection
- isolation baffle
- isolation
- baffle
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000002955 isolation Methods 0.000 title claims abstract description 290
- 238000001514 detection method Methods 0.000 title claims abstract description 246
- 230000005856 abnormality Effects 0.000 title claims abstract description 28
- 230000002159 abnormal effect Effects 0.000 claims abstract description 63
- 238000000034 method Methods 0.000 claims abstract description 34
- 238000003708 edge detection Methods 0.000 claims abstract description 32
- 230000004888 barrier function Effects 0.000 claims description 42
- 238000013136 deep learning model Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 11
- 238000012549 training Methods 0.000 claims description 9
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 238000013135 deep learning Methods 0.000 claims description 5
- 239000011159 matrix material Substances 0.000 claims description 5
- 238000013519 translation Methods 0.000 claims description 5
- 239000000203 mixture Substances 0.000 claims description 4
- 238000004891 communication Methods 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 9
- 238000013461 design Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000001537 neural effect Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000011897 real-time detection Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000004576 sand Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000004575 stone Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000000725 suspension Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Biophysics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the application provides an isolation baffle abnormality detection method and device, electronic equipment and storage medium. The method comprises the following steps: shooting a scene of the isolation baffle to be detected by adopting a high-speed camera; determining a detection target in an isolation baffle scene; obtaining world coordinates of the detection target through a calibration algorithm; identifying a detection target by using a YOLOV7 target detection model and a Cannny edge detection algorithm to obtain detection information; inputting the detection information into a pre-trained EfficientNet network model to judge whether an isolation baffle abnormal event occurs to a detection target. Based on the detection method, the detection device and the detection system for the abnormal event of the isolation baffle can achieve abnormal event detection of different types of isolation baffles in an isolation baffle scene.
Description
Technical Field
The application relates to the technical field of isolation baffle safety management, in particular to an isolation baffle abnormality detection method and device, electronic equipment and a storage medium.
Background
Aiming at the occurrence of abnormal time of the current isolation board, the abnormal condition of the isolation board can not be obtained by the existing detection system, the invention provides an abnormal event detection system of the isolation board based on machine vision. Most of the existing abnormal detection aiming at the isolation board adopts manual detection, the detection effect is single, only a certain specific abnormal event can be detected under a certain specific scene, the human detection range is limited, the detection effect is not real-time, the human cost is high, the inconvenience caused by day and night change needs to be overcome, and the abnormal condition of the isolation board cannot be detected all the day. The existing shielding case and isolation board abnormality detection system can only detect single abnormal events of the shielding case and the isolation board, such as whether the isolation baffle is missing or not, and the existing detection method can only detect single abnormal events of the isolation baffle.
Disclosure of Invention
The embodiment of the application mainly aims to provide an isolation baffle abnormality detection method and device, electronic equipment and storage medium, and abnormal event detection of different types of isolation baffles in an isolation baffle scene can be realized.
In order to achieve the above object, a first aspect of an embodiment of the present application provides an isolation baffle abnormality detection method, including:
shooting a scene of the isolation baffle to be detected by adopting a high-speed camera;
determining a detection target in the isolation baffle scene;
obtaining world coordinates of the detection target through a calibration algorithm;
identifying the detection target by using a YOLOV7 target detection model and a Cannny edge detection algorithm to obtain detection information;
and inputting the detection information into a pre-trained EfficientNet network model to judge whether the detection target has an abnormal event of the isolation baffle.
In some embodiments, the training method of the afflicientnets network model is as follows:
constructing a deep learning model;
collecting undamaged isolation baffles, and marking the undamaged isolation baffles as normal samples to obtain first sample data;
manually marking the damaged isolation baffle, and determining the vulnerability grade of the damaged isolation baffle to obtain second sample data;
And inputting the first sample data and the second sample data into the deep learning model, training the deep learning model through deep learning until the target detection precision is reached, and obtaining the EfficientNet network model.
In some embodiments, before the capturing the to-be-detected isolation barrier scene with the high-speed camera, the method further includes:
defining the isolation baffle scene;
and determining the type of the corresponding isolation baffle abnormal event in the isolation baffle scene.
In some embodiments, the categories of isolation barrier exception events include:
the corners of the isolation baffle are missing;
the shape of the isolation baffle is deformed;
breaking the isolation baffle;
the isolation barrier is vulnerable.
In some embodiments, the obtaining the world coordinates of the detection target through a calibration algorithm includes:
acquiring internal parameters of the high-speed camera;
calculating a translation vector of the high-speed camera model perpendicular to the isolation baffle;
calculating a rotation matrix of the world coordinate system around the image coordinate system;
combining the actual position of the high-speed camera to obtain a coordinate conversion relation between an image coordinate and a world coordinate;
and obtaining world coordinates of the detection target based on the coordinate conversion relation.
In some embodiments, the identifying the detection target by using YOLOV7 target detection model and Cannny edge detection algorithm to obtain detection information includes:
detecting the edge of the isolation baffle through a Cannny edge detection algorithm, and detecting whether the isolation baffle has a complete line segment or not by utilizing the Cannny edge detection algorithm when the isolation baffle is identified through a YOLOV7, so as to obtain detection information whether the isolation baffle has corner missing or not;
detecting the isolation baffle through a YOLOV7 detection model, after the isolation baffle is obtained, obtaining the appearance outline of the isolation baffle through a Cannny edge detection algorithm, judging whether line segment dislocation exists or not through comparison with a preset appearance model, and judging whether the line segment composition angle of the baffle changes or not to obtain detection information of whether the appearance of the isolation baffle is deformed or not;
marking line segments of normal lines of the isolation baffle in advance, detecting the isolation baffle and a support of the isolation baffle through a Cannny edge detection algorithm, collecting the detected line segments, and removing the marked line segments to detect positions of cracks of the isolation baffle so as to obtain detection information of whether the isolation baffle is broken or not;
And determining whether the loopholes appearing on the isolation baffle are in a safety range or not by detecting the size of the loopholes and the density of the number of the loopholes, and obtaining detection information of whether the loopholes appear on the isolation baffle or not.
In some embodiments, after the inputting the detection information into the pre-trained efficiency nets network model to determine whether the detection target has an abnormal event of the isolation barrier, the method further includes:
and carrying out early warning and reporting on the abnormal event of the isolation baffle.
To achieve the above object, a second aspect of the embodiments of the present application provides an isolation baffle abnormality detection device, including:
the shooting module is used for shooting a to-be-detected isolation baffle scene by adopting a high-speed camera;
the determining module is used for determining a detection target in the isolation baffle scene;
the calibration module is used for obtaining world coordinates of the detection target through a calibration algorithm;
the identification module is used for identifying the detection target by adopting a YOLOV7 target detection model and a Cannny edge detection algorithm to obtain detection information;
and the judging module is used for inputting the detection information into a pre-trained EfficientNet network model so as to judge whether the detection target has an abnormal event of the isolation baffle.
To achieve the above object, a third aspect of the embodiments of the present application proposes an electronic device, which includes a memory and a processor, the memory storing a computer program, the processor implementing the method according to the first aspect when executing the computer program.
To achieve the above object, a fourth aspect of the embodiments of the present application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of the first aspect.
According to the isolation baffle abnormality detection method and device, the electronic equipment and the storage medium, a high-speed camera is adopted to shoot a scene of the isolation baffle to be detected; determining a detection target in an isolation baffle scene; obtaining world coordinates of the detection target through a calibration algorithm; identifying a detection target by using a YOLOV7 target detection model and a Cannny edge detection algorithm to obtain detection information; inputting the detection information into a pre-trained EfficientNet network model to judge whether an isolation baffle abnormal event occurs to a detection target. Based on this, compared with the existing manual detection method, the embodiment of the application adopts the machine vision technology and the Efficient Net network model, the abnormal detection of the isolation baffle is realized, the world coordinates of the target are obtained through the calibration algorithm, the detection target is identified by adopting the YOLOV7 target detection model and the Cannny edge detection algorithm, the detection information is obtained, the detection of different types of abnormal events in the scene of the isolation baffle can be realized, including but not limited to the detection baffle missing, the isolation baffle corner missing, the isolation baffle appearance deformation, the isolation baffle breakage and the occurrence of loopholes of the isolation baffle, and finally the isolation baffle is accurately and rapidly detected and judged through the Efficient Net network model, so that the real-time safety of the isolation baffle is ensured. Therefore, the embodiment of the application can realize abnormal event detection of different types of isolation baffles in the scene of the isolation baffles.
Drawings
FIG. 1 is a flowchart of an isolation baffle anomaly detection method provided in an embodiment of the present application;
FIG. 2 is a flowchart of a method for training an isolation baffle anomaly detection model provided in an embodiment of the present application;
FIG. 3 is a sub-flowchart of an isolation barrier anomaly detection method provided in an embodiment of the present application;
FIG. 4 is a sub-flowchart of an isolation baffle anomaly detection method provided in an embodiment of the present application;
fig. 5 is a schematic structural diagram of an isolation baffle abnormality detection device provided in an embodiment of the present application;
fig. 6 is a schematic hardware structure of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
It should be noted that although functional block division is performed in a device diagram and a logic sequence is shown in a flowchart, in some cases, the steps shown or described may be performed in a different order than the block division in the device, or in the flowchart. The terms first, second and the like in the description and in the claims and in the above-described figures, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
Aiming at the technical problems that in the prior art, the abnormal event detection of the isolation baffle mainly depends on manual detection, the efficiency is low, and the existing detection method can only detect single abnormal event of the isolation baffle, the embodiment of the application provides an isolation baffle abnormal detection method and device, electronic equipment and storage medium, and a high-speed camera is adopted to shoot an isolation baffle scene to be detected; determining a detection target in an isolation baffle scene; obtaining world coordinates of the detection target through a calibration algorithm; identifying a detection target by using a YOLOV7 target detection model and a Cannny edge detection algorithm to obtain detection information; inputting the detection information into a pre-trained EfficientNet network model to judge whether an isolation baffle abnormal event occurs to a detection target. Based on this, compared with the existing manual detection method, the embodiment of the application adopts the machine vision technology and the Efficient Net network model, the abnormal detection of the isolation baffle is realized, the world coordinates of the target are obtained through the calibration algorithm, the detection target is identified by adopting the YOLOV7 target detection model and the Cannny edge detection algorithm, the detection information is obtained, the detection of different types of abnormal events in the scene of the isolation baffle can be realized, including but not limited to the detection baffle missing, the isolation baffle corner missing, the isolation baffle appearance deformation, the isolation baffle breakage and the occurrence of loopholes of the isolation baffle, and finally the isolation baffle is accurately and rapidly detected and judged through the Efficient Net network model, so that the real-time safety of the isolation baffle is ensured. Therefore, the embodiment of the application can realize abnormal event detection of different types of isolation baffles in the scene of the isolation baffles.
The embodiment of the application provides an isolation baffle abnormality detection method and device, an electronic device and a storage medium, and specifically, the following embodiment is used for explaining, and first describes the isolation baffle abnormality detection method in the embodiment of the application.
Fig. 1 is an optional flowchart of an isolation baffle abnormality detection method provided in an embodiment of the present application, where the method in fig. 1 may include, but is not limited to, steps S101 to S105.
Step S101, shooting a to-be-detected isolation baffle scene by adopting a high-speed camera;
step S102, determining a detection target in an isolation baffle scene;
step S103, obtaining world coordinates of the detection target through a calibration algorithm;
step S104, identifying a detection target by using a YOLOV7 target detection model and a Cannny edge detection algorithm to obtain detection information;
step S105, inputting the detection information into a pre-trained EfficientNet network model to judge whether an abnormal event of the isolation barrier occurs to the detection target.
In some embodiments, the isolation barrier is anomaly detected using a high speed camera mounted to one side of the barrier that can be photographed by a road running vehicle. Because the scene-oriented method is a expressway and a high-speed railway, a high-speed camera is required to be used for shooting when the baffle plate is detected.
In some embodiments, the detected isolation barrier scenario includes, but is not limited to, urban expressways, highways, maglev railways.
In some embodiments, the abnormal events corresponding to the isolation barrier scene include, but are not limited to, isolation barrier corner missing, isolation barrier outline deformation, isolation barrier fracture, and isolation barrier vulnerability.
In some embodiments, in order to improve the detection accuracy of the abnormal event of the isolation baffle, the embodiment of the application calibrates the camera, and determines the actual position of the detected target through a calibration algorithm. Firstly, acquiring internal parameters of a camera, then calculating a translation vector of a camera model perpendicular to an isolation baffle, then calculating a rotation matrix of a world coordinate system around an image coordinate system, and combining the actual position of the camera to obtain a coordinate conversion relation between the image coordinate and the world coordinate. The position of the world coordinate is the position of the target in real space. Therefore, the application adopts the EfficientNet network model trained based on the deep learning algorithm, so that the abnormal position of the isolation baffle can be positioned while the abnormal detection of the baffle is realized.
In some embodiments, a YOLOV7 target detection model is used when detecting the isolation baffle, the model can simultaneously identify a plurality of different targets, and compared with the existing target detection model, the YOLOV7 target detection model can provide more real-time detection information in terms of video detection, and can transmit information in a monitoring video to the YOLOV7 target detection model in real time.
In some embodiments, the isolation baffle anomaly detection and judgment is performed by using an Efficient Net network model, which is a deep convolutional network that can capture more abundant and complex features and perform very well when expanding to other data sets. The model uses neural architecture search to design a baseline network and scales the model to obtain a series of models. The precision and the efficiency of the method are better than those of all previous convolution networks, and the three dimensions of depth, width and resolution are balanced, and the three dimensions are uniformly scaled by using a set of fixed scaling coefficients, so that higher precision is obtained. The EfficientNet network designs a standardized convolution network expansion method, which not only can realize higher accuracy, but also can fully save computational resources.
In some embodiments, since the isolation baffle abnormality detection model is trained in advance through the deep learning model, detection information is input into the isolation baffle abnormality detection model, and various isolation baffles with abnormal conditions can be accurately and rapidly judged by comparing with preset normal isolation baffle sample data, so that abnormal event detection of different types of isolation baffles in an isolation baffle scene can be realized.
Referring to fig. 2, in some embodiments, the training method of the isolation baffle anomaly detection model may include, but is not limited to, steps S201 to S204:
step S201, constructing a deep learning model;
step S202, collecting undamaged isolation baffles, and marking the undamaged isolation baffles as normal samples to obtain first sample data;
step S203, the damaged isolation baffle is marked manually, and the vulnerability grade of the damaged isolation baffle is determined, so that second sample data are obtained;
and S204, inputting the first sample data and the second sample data into a deep learning model, training the deep learning model through deep learning until the target detection precision is reached, and obtaining the EfficientNet network model.
In some embodiments, firstly, a complete isolation baffle without a leak is collected, then, the sample is marked as normal, damaged baffles are marked manually, the grade of the leak is determined, the data is input into a pre-built deep learning model, the deep learning model is trained through deep learning until the target detection precision is reached, and an EfficientNet network model is obtained. It should be noted that when training the Efficient Net network model, 80% of the data is used for training and 20% is used for testing, and if the required detection accuracy is not achieved, the Efficient Net network model is continuously trained through the data until the required detection accuracy is achieved.
Referring to fig. 3, in some embodiments, steps S101 may be preceded by steps S301 to S302:
step S301, defining an isolation baffle scene;
step S302, determining the kind of the corresponding abnormal event of the isolation barrier in the isolation barrier scene.
In some embodiments, a scene is detected by shooting an isolation barrier to be detected by a camera, and the scene to be detected is defined because the barrier to be detected needs to be positioned after an abnormal event is detected, wherein the definition refers to defining the geographic position of the detected barrier and marking the actual position of each barrier. The abnormal events corresponding to the isolation baffle scene include, but are not limited to, missing isolation baffle corners, deformation of the isolation baffle appearance, breakage of the isolation baffle and loopholes of the isolation baffle.
Referring to fig. 4, in some embodiments, step S104 may include, but is not limited to, steps S401 to S405:
step S401, obtaining internal parameters of a high-speed camera;
step S402, calculating a translation vector of the high-speed camera model perpendicular to the isolation baffle;
step S404, calculating a rotation matrix of the world coordinate system around the image coordinate system;
step S404, combining the actual position of the high-speed camera to obtain a coordinate conversion relation between the image coordinates and the world coordinates;
in step S405, world coordinates of the detection target are obtained based on the coordinate conversion relationship.
In some embodiments, in order to improve the detection accuracy of the abnormal event of the isolation baffle, the embodiment of the application calibrates the high-speed camera, and determines the actual position of the detected target through a calibration algorithm. Firstly, obtaining internal parameters of a high-speed camera, then calculating a translation vector of a high-speed camera model vertical to an isolation baffle, then calculating a rotation matrix of a world coordinate system around an image coordinate system, and combining the actual position of the high-speed camera to obtain a coordinate conversion relation between the image coordinate and the world coordinate. The position of the world coordinate is the position of the target in real space.
In some embodiments, step S104 may include, but is not limited to including, step S501 to step S504:
step S501, detecting the edge of the isolation baffle through a Cannny edge detection algorithm, and detecting whether the isolation baffle has a complete line segment or not by utilizing the Cannny edge detection algorithm when the isolation baffle is identified through a YOLOV7, so as to obtain detection information whether the edge of the isolation baffle is missing or not;
step S502, detecting the isolation baffle through a YOLOV7 detection model, after the isolation baffle is obtained, obtaining the appearance outline of the isolation baffle by utilizing a Cannny edge detection algorithm, judging whether line segment dislocation exists or not by comparing the appearance outline with a preset appearance model, and judging whether the line segment composition angle of the baffle changes or not, so as to obtain detection information whether the appearance of the isolation baffle is deformed or not;
step S503, marking the line segments of the normal lines of the isolation baffle in advance, detecting the isolation baffle and the supporting object of the isolation baffle through a Cannny edge detection algorithm, collecting the detected line segments, and removing the marked line segments to detect the positions of cracks of the isolation baffle and obtain detection information of whether the isolation baffle is broken or not;
Step S504, determining whether the loopholes appearing on the isolation baffle are in a safety range or not by detecting the size of the loopholes and the density of the number of the loopholes, and obtaining detection information of whether the loopholes appear on the isolation baffle or not.
In some embodiments, when detecting the corner missing of the isolation barrier, the edge of the isolation barrier is detected through a Cannny model algorithm, when the isolation barrier is identified through YOLOV7, a canny detection module is utilized to detect whether the isolation barrier has a complete line segment, and if a broken line or an incomplete line segment occurs, the corner missing of the isolation barrier is determined.
In some embodiments, when detecting whether the appearance of the isolation baffle is deformed, the isolation baffle is detected through a YOLOV7 detection model, after the isolation baffle is obtained, the appearance outline of the isolation baffle is obtained through a canny detection module, whether line segments are misplaced or not and whether the line segment composition angle of the baffle is changed or not are checked through comparison with a preset appearance model, and if so, the isolation baffle is deformed.
In some embodiments, when detecting whether the isolation baffle is broken, the baffle is broken in two cases, one is that the isolation baffle itself is cracked, one is that a supporting object supporting the isolation baffle is cracked, and since lines are generated when the isolation baffle is broken, the lines generated by the baffle are detected. The isolation baffle itself may have lines, so normal lines need to be marked in advance, then the isolation baffle and a support of the isolation baffle are detected through a canny model, detected line segments are collected, and the positions of cracks of the isolation baffle can be detected through eliminating marked line segments.
In some embodiments, when detecting whether a leak occurs in the isolation barrier, the isolation barrier is damaged due to sand and stone splash when the vehicle is traveling at high speed. And determining whether the loopholes appearing on the baffle are in a safety range or not by detecting the size of the loopholes and the density of the number of the loopholes.
And finally, inputting the detection information into a pre-trained EfficientNet network model to judge whether an isolation baffle abnormal event occurs to a detection target. The isolation baffle abnormality detection judgment is carried out by adopting an Efficient Net network model, wherein the Efficient Net network model is a deep convolution network, and the deep convolution network can capture more abundant and complex characteristics and has very good performance when expanding to other data sets. The model uses neural architecture search to design a baseline network and scales the model to obtain a series of models. The precision and the efficiency of the method are better than those of all previous convolution networks, and the three dimensions of depth, width and resolution are balanced, and the three dimensions are uniformly scaled by using a set of fixed scaling coefficients, so that higher precision is obtained. The EfficientNet network designs a standardized convolution network expansion method, which not only can realize higher accuracy, but also can fully save computational resources.
Based on this, this embodiment carries out anomaly detection to the isolation baffle through the EfficientNet network model, can realize the different kind abnormal event detection of same scene, has not only detected whether the baffle is missing, has still detected isolation baffle corner missing, isolation baffle appearance deformation, isolation baffle fracture, isolation baffle appearance leak, monitors better and whether the isolation baffle is safe. It should be noted that the method can be applied to road isolation baffle detection, and also can be applied to railway and magnetic suspension road isolation baffle detection.
In some embodiments, step S105 may be followed by, but is not limited to including step S106:
and S106, early warning and reporting the abnormal event of the isolation baffle.
In some embodiments, the abnormal detection of the isolation baffle is realized by adopting a machine vision technology and an Efficient Net network model, the world coordinates of the target are obtained through a calibration algorithm, the detection targets are identified by adopting a YOLOV7 target detection model and a Cannny edge detection algorithm, detection information is obtained, different types of abnormal event detection in the scene of the isolation baffle can be realized, including but not limited to detection baffle missing, isolation baffle corner missing, isolation baffle appearance deformation, isolation baffle breakage and isolation baffle loophole occurrence, and finally, the isolation baffle is accurately and quickly detected and judged by the Efficient Net network model, so that whether the abnormal event occurs in the isolation baffle can be comprehensively judged, and the generated event is transmitted to a related department in real time, thereby playing the roles of real-time detection and real-time early warning.
Based on the method, abnormal event detection of different types of isolation baffles in an isolation baffle scene is achieved through high-speed camera shooting in combination with the algorithm model provided by the application.
Referring to fig. 5, an embodiment of the present application further provides an isolation baffle abnormality detection device, which may implement the above isolation baffle abnormality detection method, where the device includes:
a shooting module 510, configured to shoot a scene of an isolation barrier to be detected with a high-speed camera;
a determining module 520, configured to determine a detection target in the isolation barrier scene;
the calibration module 530 is configured to obtain world coordinates of the detection target through a calibration algorithm;
the identifying module 540 is configured to identify a detection target by using a YOLOV7 target detection model and a Cannny edge detection algorithm, so as to obtain detection information;
the judging module 550 is configured to input the detection information into a pre-trained EfficientNet network model, so as to judge whether the detection target has an abnormal event of the isolation barrier.
Based on this, in the isolation barrier abnormality detection device of the embodiment of the present application, the shooting module 510 shoots the isolation barrier scene to be detected with a high-speed camera; the determination module 520 determines a detection target in the isolation barrier scene; the calibration module 530 obtains world coordinates of the detection target through a calibration algorithm; the recognition module 540 recognizes the detection target by using a YOLOV7 target detection model and a Cannny edge detection algorithm to obtain detection information; the determination module 550 inputs the detection information into a pre-trained EfficientNet network model to determine whether an isolation barrier anomaly has occurred at the detection target. According to the embodiment of the application, the high-speed camera is adopted to shoot the scene of the isolation baffle to be detected; determining a detection target in an isolation baffle scene; obtaining world coordinates of the detection target through a calibration algorithm; identifying a detection target by using a YOLOV7 target detection model and a Cannny edge detection algorithm to obtain detection information; inputting the detection information into a pre-trained EfficientNet network model to judge whether an isolation baffle abnormal event occurs to a detection target. Based on this, compared with the existing manual detection method, the embodiment of the application adopts the machine vision technology and the Efficient Net network model, the abnormal detection of the isolation baffle is realized, the world coordinates of the target are obtained through the calibration algorithm, the detection target is identified by adopting the YOLOV7 target detection model and the Cannny edge detection algorithm, the detection information is obtained, the detection of different types of abnormal events in the scene of the isolation baffle can be realized, including but not limited to the detection baffle missing, the isolation baffle corner missing, the isolation baffle appearance deformation, the isolation baffle breakage and the occurrence of loopholes of the isolation baffle, and finally the isolation baffle is accurately and rapidly detected and judged through the Efficient Net network model, so that the real-time safety of the isolation baffle is ensured. Therefore, the embodiment of the application can realize abnormal event detection of different types of isolation baffles in the scene of the isolation baffles.
The specific implementation manner of the isolation baffle abnormality detection device is basically the same as the specific embodiment of the isolation baffle abnormality detection method, and is not described herein.
The embodiment of the application also provides electronic equipment, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the isolation baffle abnormality detection method when executing the computer program. The electronic equipment can be any intelligent terminal including a tablet personal computer, a vehicle-mounted computer and the like.
Referring to fig. 6, fig. 6 illustrates a hardware structure of an electronic device according to another embodiment, the electronic device includes:
the processor 601 may be implemented by a general-purpose CPU (central processing unit), a microprocessor, an application-specific integrated circuit (ApplicationSpecificIntegratedCircuit, ASIC), or one or more integrated circuits, etc. for executing related programs to implement the technical solutions provided in the embodiments of the present application.
The memory 602 may be implemented in the form of read-only memory (ReadOnlyMemory, ROM), static storage, dynamic storage, or random access memory (RandomAccessMemory, RAM). The memory 602 may store an operating system and other application programs, and when the technical solution provided in the embodiments of the present disclosure is implemented by software or firmware, relevant program codes are stored in the memory 602, and the processor 601 invokes the method for detecting the abnormality of the isolation barrier according to the embodiments of the present disclosure, that is, shooting the scene of the isolation barrier to be detected by using a high-speed camera; determining a detection target in an isolation baffle scene; obtaining world coordinates of the detection target through a calibration algorithm; identifying a detection target by using a YOLOV7 target detection model and a Cannny edge detection algorithm to obtain detection information; inputting the detection information into a pre-trained EfficientNet network model to judge whether an isolation baffle abnormal event occurs to a detection target. Based on this, compared with the existing manual detection method, the embodiment of the application adopts the machine vision technology and the Efficient Net network model, the abnormal detection of the isolation baffle is realized, the world coordinates of the target are obtained through the calibration algorithm, the detection target is identified by adopting the YOLOV7 target detection model and the Cannny edge detection algorithm, the detection information is obtained, the detection of different types of abnormal events in the scene of the isolation baffle can be realized, including but not limited to the detection baffle missing, the isolation baffle corner missing, the isolation baffle appearance deformation, the isolation baffle breakage and the occurrence of loopholes of the isolation baffle, and finally the isolation baffle is accurately and rapidly detected and judged through the Efficient Net network model, so that the real-time safety of the isolation baffle is ensured. Therefore, the embodiment of the application can realize abnormal event detection of different types of isolation baffles in the scene of the isolation baffles.
An input/output interface 603 for implementing information input and output.
The communication interface 604 is configured to implement communication interaction between the device and other devices, and may implement communication through a wired manner (such as USB, network cable, etc.), or may implement communication through a wireless manner (such as mobile network, WIFI, bluetooth, etc.).
A bus that transfers information between the various components of the device, such as the processor 601, memory 602, input/output interfaces 603, and communication interfaces 604.
Wherein the processor 601, the memory 602, the input/output interface 603 and the communication interface 604 are communicatively connected to each other within the device via a bus.
The embodiment of the application also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program realizes the isolation baffle abnormality detection method when being executed by a processor.
The memory, as a non-transitory computer readable storage medium, may be used to store non-transitory software programs as well as non-transitory computer executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory optionally includes memory remotely located relative to the processor, the remote memory being connectable to the processor through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
According to the isolation baffle abnormality detection method, the isolation baffle abnormality detection device, the electronic equipment and the storage medium, a high-speed camera is adopted to shoot a scene of the isolation baffle to be detected; determining a detection target in an isolation baffle scene; obtaining world coordinates of the detection target through a calibration algorithm; identifying a detection target by using a YOLOV7 target detection model and a Cannny edge detection algorithm to obtain detection information; inputting the detection information into a pre-trained EfficientNet network model to judge whether an isolation baffle abnormal event occurs to a detection target. Based on this, compared with the existing manual detection method, the embodiment of the application adopts the machine vision technology and the Efficient Net network model, the abnormal detection of the isolation baffle is realized, the world coordinates of the target are obtained through the calibration algorithm, the detection target is identified by adopting the YOLOV7 target detection model and the Cannny edge detection algorithm, the detection information is obtained, the detection of different types of abnormal events in the scene of the isolation baffle can be realized, including but not limited to the detection baffle missing, the isolation baffle corner missing, the isolation baffle appearance deformation, the isolation baffle breakage and the occurrence of loopholes of the isolation baffle, and finally the isolation baffle is accurately and rapidly detected and judged through the Efficient Net network model, so that the real-time safety of the isolation baffle is ensured. Therefore, the embodiment of the application can realize abnormal event detection of different types of isolation baffles in the scene of the isolation baffles.
Those of ordinary skill in the art will appreciate that all or some of the steps, systems, and methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable programs, data structures, program modules or other data, as known to those skilled in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Furthermore, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable programs, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
The embodiments described in the embodiments of the present application are for more clearly describing the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application, and as those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.
It will be appreciated by those skilled in the art that the technical solutions shown in the figures do not constitute limitations of the embodiments of the present application, and may include more or fewer steps than shown, or may combine certain steps, or different steps.
The above described apparatus embodiments are merely illustrative, wherein the units illustrated as separate components may or may not be physically separate, i.e. may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
Those of ordinary skill in the art will appreciate that all or some of the steps of the methods, systems, functional modules/units in the devices disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof.
The terms "first," "second," "third," "fourth," and the like in the description of the present application and in the above-described figures, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the present application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It should be understood that in this application, "at least one" means one or more, and "a plurality" means two or more. "and/or" for describing the association relationship of the association object, the representation may have three relationships, for example, "a and/or B" may represent: only a, only B and both a and B are present, wherein a, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (one) of a, b or c may represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c may be single or plural.
In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the above-described division of units is merely a logical function division, and there may be another division manner in actual implementation, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including multiple instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of the various embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing a program.
Preferred embodiments of the present application are described above with reference to the accompanying drawings, and thus do not limit the scope of the claims of the embodiments of the present application. Any modifications, equivalent substitutions and improvements made by those skilled in the art without departing from the scope and spirit of the embodiments of the present application shall fall within the scope of the claims of the embodiments of the present application.
Claims (10)
1. An isolation baffle abnormality detection method, characterized in that the method comprises:
shooting a scene of the isolation baffle to be detected by adopting a high-speed camera;
determining a detection target in the isolation baffle scene;
obtaining world coordinates of the detection target through a calibration algorithm;
identifying the detection target by using a YOLOV7 target detection model and a Cannny edge detection algorithm to obtain detection information;
and inputting the detection information into a pre-trained EfficientNet network model to judge whether the detection target has an abnormal event of the isolation baffle.
2. The method of claim 1, wherein the training method of the EfficientNets network model is as follows:
constructing a deep learning model;
collecting undamaged isolation baffles, and marking the undamaged isolation baffles as normal samples to obtain first sample data;
manually marking the damaged isolation baffle, and determining the vulnerability grade of the damaged isolation baffle to obtain second sample data;
and inputting the first sample data and the second sample data into the deep learning model, training the deep learning model through deep learning until the target detection precision is reached, and obtaining the EfficientNet network model.
3. The method of claim 1, further comprising, prior to said capturing the baffle scene to be detected with the high speed camera:
defining the isolation baffle scene;
and determining the type of the corresponding isolation baffle abnormal event in the isolation baffle scene.
4. The method of claim 1, wherein the categories of isolation barrier anomaly events include:
the corners of the isolation baffle are missing;
the shape of the isolation baffle is deformed;
breaking the isolation baffle;
the isolation barrier is vulnerable.
5. The method according to claim 1, wherein the obtaining world coordinates of the detection target by a calibration algorithm includes:
acquiring internal parameters of the high-speed camera;
calculating a translation vector of the high-speed camera model perpendicular to the isolation baffle;
calculating a rotation matrix of the world coordinate system around the image coordinate system;
combining the actual position of the high-speed camera to obtain a coordinate conversion relation between an image coordinate and a world coordinate;
and obtaining world coordinates of the detection target based on the coordinate conversion relation.
6. The method of claim 1, wherein identifying the detection target using YOLOV7 target detection model and Cannny edge detection algorithm to obtain detection information comprises:
Detecting the edge of the isolation baffle through a Cannny edge detection algorithm, and detecting whether the isolation baffle has a complete line segment or not by utilizing a canny detection module when the isolation baffle is identified through a YOLOV7, so as to obtain detection information whether the isolation baffle has corner missing or not;
detecting the isolation baffle through a YOLOV7 detection model, after the isolation baffle is obtained, obtaining the appearance outline of the isolation baffle through a Cannny edge detection algorithm, judging whether line segment dislocation exists or not through comparison with a preset appearance model, and judging whether the line segment composition angle of the baffle changes or not to obtain detection information of whether the appearance of the isolation baffle is deformed or not;
marking line segments of normal lines of the isolation baffle in advance, detecting the isolation baffle and a support of the isolation baffle through a Cannny edge detection algorithm, collecting the detected line segments, and removing the marked line segments to detect positions of cracks of the isolation baffle so as to obtain detection information of whether the isolation baffle is broken or not;
and determining whether the loopholes appearing on the isolation baffle are in a safety range or not by detecting the size of the loopholes and the density of the number of the loopholes, and obtaining detection information of whether the loopholes appear on the isolation baffle or not.
7. The method according to any one of claims 1 to 6, further comprising, after said inputting said detection information into a pre-trained EfficientNets network model to determine whether said detection target has an isolation barrier anomaly event:
and carrying out early warning and reporting on the abnormal event of the isolation baffle.
8. An isolation barrier anomaly detection apparatus, the apparatus comprising:
the shooting module is used for shooting a to-be-detected isolation baffle scene by adopting a high-speed camera;
the determining module is used for determining a detection target in the isolation baffle scene;
the calibration module is used for obtaining world coordinates of the detection target through a calibration algorithm;
the identification module is used for identifying the detection target by adopting a YOLOV7 target detection model and a Cannny edge detection algorithm to obtain detection information;
and the judging module is used for inputting the detection information into a pre-trained EfficientNet network model so as to judge whether the detection target has an abnormal event of the isolation baffle.
9. An electronic device comprising a memory storing a computer program and a processor that when executing the computer program implements the isolation barrier anomaly detection method of any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the isolation barrier abnormality detection method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310311470.9A CN116402774A (en) | 2023-03-27 | 2023-03-27 | Isolation baffle abnormality detection method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310311470.9A CN116402774A (en) | 2023-03-27 | 2023-03-27 | Isolation baffle abnormality detection method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116402774A true CN116402774A (en) | 2023-07-07 |
Family
ID=87011640
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310311470.9A Pending CN116402774A (en) | 2023-03-27 | 2023-03-27 | Isolation baffle abnormality detection method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116402774A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118366140A (en) * | 2024-06-18 | 2024-07-19 | 西安市正泰五防工程有限责任公司 | Method for detecting abnormal state of civil air defense door based on machine vision |
-
2023
- 2023-03-27 CN CN202310311470.9A patent/CN116402774A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118366140A (en) * | 2024-06-18 | 2024-07-19 | 西安市正泰五防工程有限责任公司 | Method for detecting abnormal state of civil air defense door based on machine vision |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109887281B (en) | Method and system for monitoring traffic incident | |
| CN110084165B (en) | Intelligent identification and early warning method for abnormal events in open scene of power field based on edge calculation | |
| CN103726879B (en) | Utilize camera automatic capturing mine ore deposit to shake and cave in and the method for record warning in time | |
| US9846823B2 (en) | Traffic lane boundary line extraction apparatus and method of extracting traffic lane boundary line | |
| CN110298300B (en) | Method for detecting vehicle illegal line pressing | |
| CN112017323B (en) | Inspection alarm method and device, readable storage medium and terminal equipment | |
| CN110458126B (en) | Pantograph state monitoring method and device | |
| CN113470374B (en) | Vehicle overspeed monitoring method and device, computer equipment and storage medium | |
| CN109698906B (en) | Image-based jitter processing method and device and video monitoring system | |
| CN108549862A (en) | Abnormal scene detection method and device | |
| CN116402774A (en) | Isolation baffle abnormality detection method and device, electronic equipment and storage medium | |
| CN112270253A (en) | High-altitude parabolic detection method and device | |
| CN112776856A (en) | Track foreign matter intrusion monitoring method, device and system and monitoring host equipment | |
| CN111144301A (en) | Road pavement defect quick early warning device based on degree of depth learning | |
| CN113743151A (en) | Method and device for detecting road surface sprinkled object and storage medium | |
| CN112435479B (en) | Target object violation detection method and device, computer equipment and system | |
| CN103391441A (en) | Surveillance video object deleting, tampering and detecting algorithm based on capacity variance | |
| CN116542916A (en) | Method and device for detecting abnormality of chassis part of train, electronic equipment and medium | |
| CN111371984A (en) | Method and device for determining abnormality of snapshot machine and storage medium | |
| CN102740107A (en) | Damage monitoring system of image surveillance equipment and method | |
| CN113283286B (en) | Driver abnormal behavior detection method and device | |
| CN115187880A (en) | Communication optical cable defect detection method and system based on image recognition and storage medium | |
| CN112926446B (en) | Parabolic detection method and parabolic detection system | |
| CN107976319B (en) | Intelligent detection system and method for car additionally provided with pedal | |
| CN110930362A (en) | Screw safety detection method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |