CN111860283B - System and method for collecting evidence of illegal behaviors in video identification - Google Patents
System and method for collecting evidence of illegal behaviors in video identification Download PDFInfo
- Publication number
- CN111860283B CN111860283B CN202010681448.XA CN202010681448A CN111860283B CN 111860283 B CN111860283 B CN 111860283B CN 202010681448 A CN202010681448 A CN 202010681448A CN 111860283 B CN111860283 B CN 111860283B
- Authority
- CN
- China
- Prior art keywords
- video
- evidence
- generation task
- illegal behaviors
- illegal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000006399 behavior Effects 0.000 title claims abstract description 84
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000001514 detection method Methods 0.000 claims description 25
- 238000013135 deep learning Methods 0.000 claims description 5
- 230000000391 smoking effect Effects 0.000 description 10
- 238000012544 monitoring process Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 239000000779 smoke Substances 0.000 description 3
- 230000003139 buffering effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 235000019504 cigarettes Nutrition 0.000 description 1
- 239000003245 coal Substances 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 239000003208 petroleum Substances 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
- G06V20/42—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items of sport video content
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/01—Detecting movement of traffic to be counted or controlled
- G08G1/017—Detecting movement of traffic to be counted or controlled identifying vehicles
- G08G1/0175—Detecting movement of traffic to be counted or controlled identifying vehicles by photographing vehicles, e.g. when violating traffic rules
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/44—Event detection
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Closed-Circuit Television Systems (AREA)
- Television Signal Processing For Recording (AREA)
Abstract
The invention discloses a system and a method for collecting evidence of illegal behaviors in video identification, which are characterized in that a video stream is firstly read in, the video stream is divided into video clip files with fixed length to be stored, meanwhile, the illegal behaviors in a video image are tracked and detected, and video frame numbers at the starting and ending moments of the illegal behaviors and the position information of the illegal behaviors in the video frames are recorded; then, according to the video frame numbers of the starting and ending moments of the illegal behaviors, a video clip file list corresponding to the illegal behaviors is obtained, and a video evidence generating task is created; periodically checking whether all video clip files in the video evidence generation task are generated; and when all the video clip files in the video evidence generation task are generated, splicing the video clip files to generate a video evidence file.
Description
Technical Field
The invention belongs to the technical field of computer vision, and relates to generation of evidence of illegal behaviors in video identification, which is used as a basis for subsequent illegal processing.
Background
When industries such as building, petroleum, coal, electric power and the like carry out production operation, in order to avoid safety accidents, operators are required to operate according to safety production operation regulations, if illegal operation occurs, illegal behaviors are required to be recorded, and different penalties are carried out on potential hazard degrees of the illegal behaviors.
In recent years, along with the development of monitoring systems, monitoring systems are installed on a plurality of production sites, site operation videos can be collected and transmitted to a monitoring center through cameras, and the monitoring center personnel watch the videos to monitor site operation behaviors. The monitoring system and the monitoring center watch on duty without the need of a supervisory person to the scene, and one supervisory center can supervise a plurality of scattered operation points, so that the relative scene supervision efficiency and coverage are greatly improved, but the problems that personnel are easy to fatigue, visual blind points and the like still exist in a manual supervision mode of watching videos are solved.
With the development of computer vision, supervision of field safety work is made possible by a computer. The patent application (application number 201810120023.4, application publication number CN 110119656A) discloses an intelligent monitoring system for operation site offenders and a site offender monitoring method. In the method disclosed by the application, whether the illegal behaviors exist in the field operation or not is judged by deep learning, target detection and traditional feature detection methods, the problem that the illegal behaviors are automatically identified by a computer vision mode is solved, and fatigue-free and full-coverage operation behavior supervision is realized. The system and method involved in the invention is not described for video evidence collection of offending behavior.
The formation and existence of the violation generally lasts for a period of time, and even a single picture can prove the violation, but in practical application, in order to ensure the stability of the violation detection system, the rule matching of the violation which occasionally occurs is excluded, and whether the violation exists for a period of time is also detected. Some illegal behaviors are determined by combining association relations of target state changes before and after, action behavior modes and the like, and video of a period of time is needed to be relied on as a basis for occurrence of the illegal behaviors.
Therefore, from the first occurrence of the violation or the state that the occurrence of the violation is likely to start, to the determination of the occurrence of the violation, or to the end of the violation, the violation detection system needs to intercept the video of the time as the accessory information of the violation alarm.
The generation of the video evidence of the offensive behavior has the following difficulties:
A. the illegal behavior can occur in any section of the video, but the whole video cannot be cached due to the limitation of capacity or the requirement of performance, and how to support the generation of any illegal behavior video evidence in the limited capacity;
B. multiple violations may occur simultaneously in the same time period, how to intercept different video evidence according to different starting and ending moments respectively;
C. when a cached video file is being written, generating video evidence to read the cached video file, wherein mutual exclusion exists at the moment, and how to ensure reliable cached file reading;
D. the input video has a plurality of target objects including people, tools and equipment, the illegal behaviors generally only relate to partial people, partial tools and equipment, and the targets related to the illegal behaviors are clearly identified in the video, so that the method is an important basis for subsequent illegal behavior treatment.
Disclosure of Invention
In order to solve the problems, the invention provides a robust video evidence collection system and a robust video evidence collection method, which are used for intercepting videos from the occurrence of the illegal behaviors to the end of the illegal behaviors as evidence and are used for the basis of subsequent processing of the illegal behaviors.
The invention adopts the following technical scheme:
video cache module
Dividing a video image into video clip files with fixed lengths and storing the video clip files;
illegal behavior detection module
Tracking and detecting the illegal behaviors in the video image, recording video frame numbers at the starting and ending moments of the illegal behaviors and the position information of a target object related to the illegal behaviors in frames, and notifying a video evidence generation task management module to create a video evidence generation task;
video evidence generation task management module
According to the video frame numbers of the starting and ending moments of the illegal behaviors, obtaining a video clip file list corresponding to the illegal behaviors, and creating a video evidence generation task; adding a video evidence generation task into a video evidence generation task list, periodically checking the video generation task list, checking whether all video clip files in the video evidence generation task are generated, and notifying a video evidence generation module to generate video evidence when the video clip files are generated;
a video evidence generation module:
and splicing the video clip files in the video evidence generation task to generate a video evidence file.
A method for collecting evidence of illegal behaviors in video identification is characterized by comprising the following steps:
1) Reading in a video stream, dividing the video stream into video clip files with fixed lengths, storing, simultaneously tracking and detecting the illegal behaviors in the video image, and recording video frame numbers at the starting and ending moments of the illegal behaviors and the position information of the illegal behaviors in the video frames;
2) According to the video frame numbers of the starting and ending moments of the illegal behaviors, obtaining a video clip file list corresponding to the illegal behaviors, and creating a video evidence generation task; adding a video evidence generation task into a video evidence generation task list, periodically checking the video generation task list, and checking whether a video clip file in the video evidence generation task is generated or not;
3) And if the video clip file in the video evidence generation task is generated, splicing the video clip file to generate a video evidence file.
The invention has the beneficial effects that:
1. collecting video evidence of the illegal behaviors;
the method can support the generation of the video evidence of the illegal behaviors and serve as the evidence of the later processing of the illegal behaviors.
2. Splicing videos according to the violation time span;
according to the method and the device, the video clip files with different caches can be spliced according to different duration of the illegal behaviors, so that video evidence files with different durations can be generated.
3. Collecting parallel violation evidences;
by adopting a mechanism of a video evidence file generation task, even if a plurality of evidence files need to be generated at the same time and the same video clip files are involved, the video evidence files can be reliably generated without worrying about failure of video evidence file generation caused by read-write conflict of the video clip files.
4. Delayed video stitching
The video clip files are cached according to time length, and the moment when the illegal video evidence generation task is created is likely to be that the video clip files are not generated yet. And adding a video generation task, periodically scanning video clip files related to the video generation task, and after determining that the video clip files are well generated, starting the splicing of the whole video evidence files, so that the failure of video evidence file generation can be avoided.
Drawings
FIG. 1 is a block diagram of a evidence collection system for offending behavior;
FIG. 2 is a flow chart for detecting offending behavior;
FIG. 3 is a video buffering flow diagram;
FIG. 4 is a video evidence generation task creation flow diagram;
fig. 5 is a video evidence generation flow chart.
Detailed Description
As shown in fig. 1, the system for collecting evidence of offensive behavior of the present invention comprises:
A. illegal behavior detection module
As shown in fig. 2, the violation detection module detects a target in the video image based on a deep learning or traditional machine vision method, and determines whether the violation occurs according to rules for determining the violation. If the illegal behaviors are detected, tracking and monitoring are carried out, the video frame numbers at the beginning time of the illegal behaviors and the frame numbers at the end time of the illegal behaviors are recorded, and information such as target objects, positioning frames of the target objects in each frame and the like is related to the illegal behaviors, so that a video evidence generation task management module is informed to create a video generation task.
After confirming that the violation video evidence is generated, the violation detection module initiates an alarm of the violation, wherein the alarm contains the violation video evidence.
B. Video cache module
As shown in fig. 3, the video buffer module reads in the video stream, divides the video stream into video clip files with fixed lengths for storage, and the lengths of the files can be configured according to requirements.
The video stream buffer module needs to clear the earliest added video clip file according to the configured buffer capacity or duration so as to maintain the availability of the video stream buffer pool. And deleting the video clip file stored first when the maximum capacity or the maximum duration of the video cache is exceeded.
C. Video evidence generation task management module
Referring to fig. 4, after receiving the message of the video evidence generation task from the violation detection module, the video evidence generation task management module calculates a corresponding start video clip file and an end video clip file according to the start video frame number and the end video frame number of the video evidence, and creates the video evidence generation task by taking a video clip file list related to the video evidence, a video file name to be generated and the like as parameters.
And adding video evidence generation tasks into a task queue, scanning the task list regularly, checking whether video clip files in each video evidence generation task are ready by the system, and informing a video evidence generation module to generate video evidence if all video clip files are ready.
D. Video evidence generation module
Referring to fig. 5, the video evidence generating module receives a command for generating video evidence, and splices video clip files in the video evidence generating task to generate video evidence files.
E. Violation alarm module
After the violation video evidence is generated, the video evidence generation module sends the video evidence to the violation alarm module to initiate the violation behavior alarm.
Example 1
Taking smoking in a smoking-forbidden zone as an illustration of the acquisition of single-item video evidence of violation.
The smoking detection is generally performed by a deep learning mode, and the method is to collect enough smoking videos of different people, different scenes, different angles, different light rays and the like, mark the part of the mouth containing cigarettes as smoking, make the pictures into a deep learning data set, and train a target detection model by using the data set. The method of target detection can adopt a mode of YOLO/SSD and the like, and the modes can be found in the disclosed materials. The trained target detection model can be used for detecting the illegal behavior of smoking in a smoking forbidden zone.
And if one camera is aligned to the smoking-forbidden zone, the video stream obtained by the camera is accessed to the illegal behavior detection module and the video cache module through the network, so that whether people smoke in the smoking-forbidden zone is monitored. The video buffering module generates a video clip file every 10s, assuming that the frame rate of the video is 25 frames per second, that is, 250 frames of pictures constitute a video clip file. Video clip files are named with a sequence number of vc+four bits, such as vc0000.Mp4, vc0001.Mp4, etc. Wherein vc0000.mp4 includes video pictures of frames 0 through 249, and vc0001.mp4 includes video pictures of frames 250 through 499.
The illegal behavior detection module reads in the video stream acquired by the camera, adopts a target detection model to detect the video frame by frame, and detects whether the video stream contains a type of target of smoking. Assuming that a smoking target is detected in 1896 frames, the violation detection module may issue an alarm for "smoking in no-smoking zone". The violation detection module will then keep track of whether "smoke" is present, assuming that "smoke" is not found to be present in 2009, at which point the violation is deemed to have ended.
The method comprises the steps that a violation behavior detection module initiates a video evidence generation command to a video evidence generation task management module, wherein the video evidence generation command comprises a start frame number 1896 and an end frame number 2009 for the occurrence of the violation behavior, and after the video evidence generation task management module receives the command, the video evidence generation task management module firstly calculates the file name of a video clip to be started, wherein the method is that the start frame number/250 q is rounded, for example, 1896/250=7, namely, the number of the video clip file to be started is 7; similarly, the end video clip file number is calculated, 2009/250=8, and since the 8 th video clip file corresponds to frame numbers 1750 to 1999, 2009 refers to the next video clip file, i.e., number 9.
The video clip file list included in the video evidence generation task is as follows: vc0007.mp4, vc0008.mp4, vc0009.mp4, it is notable that at the moment of the end of the violation, i.e. at frame 2009, the vc0009.mp4 video clip file has not yet been completed, which would lead to an error if vc0009.mp4 were read directly.
The video evidence generation task management module creates a video evidence generation task that includes a video evidence file name to be generated, such as chouyan 20200620091010004.mp4, and video clip file lists vc0007.mp4, vc0008.mp4, vc0009.mp4 to be spliced. The task is typed into a video evidence generation task list.
The video evidence generation module periodically detects a list of video evidence generation tasks, for example, every 0.2s, and checks for each video evidence generation task whether all video clip files contained therein are ready. For the video evidence generation task described above, vc0009.mp4 has completed after the corresponding time of 2250 frames, at which point the video evidence generation file concatenates these video clip files vc0007.mp4, vc0008.mp4, vc0009.mp4 into the video evidence file chouyan 20200620091010004.mp4. The splicing method can be realized by calling the video processing software ffmpeg of an open source, firstly creating a text file. Then execute
ffmpeg-f concat-safe 0-I filelist.txt-vcodec libx264chouyan202006200930110004.mp4
In the embodiment, the rule-breaking alarm module is integrated in the rule-breaking behavior detection module, after the video evidence file is generated, the video evidence generation module informs the rule-breaking behavior detection module that the video evidence is ready, and sends the generated video evidence to the rule-breaking behavior detection module, the rule-breaking behavior detection module sends out rule-breaking alarm, for example, sends the video evidence to an upper computer for display, and alarm information comprises information such as a type of rule-breaking behavior, namely smoking in a smoke-breaking area, a file name of the video evidence corresponding to the rule-breaking behavior, chouyan 20200620091010004.
Claims (8)
1. The system for collecting evidence of illegal behaviors in video recognition is characterized by comprising the following components:
video cache module
Dividing a video image into video clip files with fixed lengths and storing the video clip files;
illegal behavior detection module
Tracking and detecting the illegal behaviors in the video image, recording video frame numbers at the starting and ending moments of the illegal behaviors and the position information of a target object related to the illegal behaviors in frames, and notifying a video evidence generation task management module to create a video evidence generation task;
video evidence generation task management module
According to the video frame numbers of the starting and ending moments of the illegal behaviors, obtaining a video clip file list corresponding to the illegal behaviors, and creating a video evidence generation task; adding a video evidence generation task into a video evidence generation task list, periodically checking the video generation task list, checking whether all video clip files in each video evidence generation task are generated, and notifying a video evidence generation module to generate video evidence when the video clip files are generated;
a video evidence generation module:
and splicing the video clip files in the video evidence generation task to generate a video evidence file.
2. The evidence collection system of offending behavior in video identification of claim 1, wherein the video stream caching module deletes an earliest stored video clip file when the stored video clip file exceeds a maximum capacity or maximum duration of the video cache.
3. The system for collecting evidence of offence in video recognition according to claim 1, further comprising a offence alarm module, wherein the video evidence generation module sends the video evidence to the offence alarm module to initiate offence alarm after the offence video evidence is generated.
4. The system for collecting evidence of offensive behavior in video recognition according to claim 1, wherein the offensive behavior detection module detects a target in the video image based on deep learning or a conventional machine vision method, and judges whether offensive behavior occurs according to offensive behavior judgment rules.
5. The system for collecting evidence of offensive behavior in video recognition according to claim 1, wherein the video evidence generation task management module calculates corresponding start video clip files and end video clip file numbers according to the start frame number and end frame number of the video evidence, and creates the video evidence generation task by using a video clip file list related to the video evidence, a video file name to be generated, and the like as parameters.
6. A method for collecting evidence of illegal behaviors in video identification is characterized by comprising the following steps:
1) Reading in a video stream, dividing the video stream into video clip files with fixed lengths, storing, simultaneously tracking and detecting the illegal behaviors in the video image, and recording video frame numbers at the starting and ending moments of the illegal behaviors and the position information of the illegal behaviors in the video frames;
2) According to the video frame numbers of the starting and ending moments of the illegal behaviors, obtaining a video clip file list corresponding to the illegal behaviors, and creating a video evidence generation task; adding a video evidence generation task into a video evidence generation task list, periodically checking the video generation task list, and checking whether all video clip files in each video evidence generation task are generated or not;
3) And when all the video clip files in the video evidence generation task are generated, splicing the video clip files to generate a video evidence file.
7. The method for collecting evidence of offending behavior in video identification of claim 6, wherein when the stored video clip file exceeds a maximum capacity or a maximum duration of the video cache, the earliest stored video clip file is deleted.
8. The method for collecting evidence of illegal actions in video recognition according to claim 6, wherein after the evidence of illegal actions is generated, an alarm of the illegal actions is initiated, and the alarm contains the evidence of the illegal actions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010681448.XA CN111860283B (en) | 2020-07-15 | 2020-07-15 | System and method for collecting evidence of illegal behaviors in video identification |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010681448.XA CN111860283B (en) | 2020-07-15 | 2020-07-15 | System and method for collecting evidence of illegal behaviors in video identification |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111860283A CN111860283A (en) | 2020-10-30 |
| CN111860283B true CN111860283B (en) | 2023-10-31 |
Family
ID=72984103
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010681448.XA Active CN111860283B (en) | 2020-07-15 | 2020-07-15 | System and method for collecting evidence of illegal behaviors in video identification |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111860283B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11574477B2 (en) * | 2021-03-08 | 2023-02-07 | Adobe Inc. | Highlight video generated with adaptable multimodal customization |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1851777A (en) * | 2006-05-22 | 2006-10-25 | 昆明利普机器视觉工程有限公司 | Vehicle video data digging system and method for obtaining evidence about drive against traffic regulations |
| CA2785252A1 (en) * | 2011-08-16 | 2013-02-16 | Xerox Corporation | Automated processing method for bus crossing enforcement |
| CN103400499A (en) * | 2013-07-22 | 2013-11-20 | 青岛海信网络科技股份有限公司 | Ultra-high definition intelligent video electronic police system and vehicle violation judging method of ultra-high definition intelligent video electronic police system |
| CN110826514A (en) * | 2019-11-13 | 2020-02-21 | 国网青海省电力公司海东供电公司 | Construction site violation intelligent identification method based on deep learning |
-
2020
- 2020-07-15 CN CN202010681448.XA patent/CN111860283B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1851777A (en) * | 2006-05-22 | 2006-10-25 | 昆明利普机器视觉工程有限公司 | Vehicle video data digging system and method for obtaining evidence about drive against traffic regulations |
| CA2785252A1 (en) * | 2011-08-16 | 2013-02-16 | Xerox Corporation | Automated processing method for bus crossing enforcement |
| CN103400499A (en) * | 2013-07-22 | 2013-11-20 | 青岛海信网络科技股份有限公司 | Ultra-high definition intelligent video electronic police system and vehicle violation judging method of ultra-high definition intelligent video electronic police system |
| CN110826514A (en) * | 2019-11-13 | 2020-02-21 | 国网青海省电力公司海东供电公司 | Construction site violation intelligent identification method based on deep learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111860283A (en) | 2020-10-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10930151B2 (en) | Roadside parking management method, device, and system based on multiple cameras | |
| JP6425856B1 (en) | Video recording method, server, system and storage medium | |
| US7447331B2 (en) | System and method for generating a viewable video index for low bandwidth applications | |
| US7428314B2 (en) | Monitoring an environment | |
| US8107680B2 (en) | Monitoring an environment | |
| CN101872524B (en) | Video monitoring method, system and device based on virtual wall | |
| CN105632001A (en) | Security and protection monitoring method and system based on face recognition | |
| CN110738178A (en) | Garden construction safety detection method and device, computer equipment and storage medium | |
| JP6013923B2 (en) | System and method for browsing and searching for video episodes | |
| CN104284162A (en) | Video retrieval method and system | |
| CN112382033A (en) | Regional intrusion alarm system | |
| CN108230669B (en) | Road vehicle violation detection method and system based on big data and cloud analysis | |
| CN113011263B (en) | Mine monitoring method, device, terminal equipment and medium | |
| CN105608464A (en) | Face recognition mobile terminal solving system and solving method | |
| CN111860283B (en) | System and method for collecting evidence of illegal behaviors in video identification | |
| CN204926080U (en) | System for video can be retrieveed with target to discernment people's face or video recording of license plate target | |
| CN113990101B (en) | Method, system and processing device for detecting vehicles in no-parking area | |
| CN104346927A (en) | Things internet-based red light-running vehicle monitoring and shooting system | |
| CN111901562B (en) | Campus-oriented audio and video intelligent monitoring system | |
| TWI589158B (en) | Storage system of original frame of monitor data and storage method thereof | |
| CN108122414A (en) | The detection method and device of car on-board and off-board on highway | |
| WO2010087346A1 (en) | Pantograph photographing apparatus by image processing | |
| CN114187666B (en) | Identification method and system for watching mobile phone while walking | |
| CN111931569B (en) | Security monitoring thermodynamic diagram statistical system | |
| CN111343417A (en) | Monitoring system for kindergarten student supervision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |