CN112584073B - 5G-based law enforcement recorder distributed assistance calculation method - Google Patents
5G-based law enforcement recorder distributed assistance calculation method Download PDFInfo
- Publication number
- CN112584073B CN112584073B CN202011550001.5A CN202011550001A CN112584073B CN 112584073 B CN112584073 B CN 112584073B CN 202011550001 A CN202011550001 A CN 202011550001A CN 112584073 B CN112584073 B CN 112584073B
- Authority
- CN
- China
- Prior art keywords
- terminal
- module
- calculation
- law enforcement
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004364 calculation method Methods 0.000 title claims abstract description 198
- 238000004458 analytical method Methods 0.000 claims abstract description 135
- 239000012634 fragment Substances 0.000 claims abstract description 134
- 238000007781 pre-processing Methods 0.000 claims abstract description 30
- 230000001360 synchronised effect Effects 0.000 claims abstract description 18
- 238000001914 filtration Methods 0.000 claims abstract description 3
- 238000001514 detection method Methods 0.000 claims description 27
- 238000007906 compression Methods 0.000 claims description 21
- 230000006835 compression Effects 0.000 claims description 21
- 238000013528 artificial neural network Methods 0.000 claims description 16
- 230000000977 initiatory effect Effects 0.000 claims description 7
- 230000005540 biological transmission Effects 0.000 claims description 6
- 230000001965 increasing effect Effects 0.000 claims description 5
- 238000000034 method Methods 0.000 claims description 4
- 238000013144 data compression Methods 0.000 claims description 3
- 238000010191 image analysis Methods 0.000 claims description 3
- 238000007726 management method Methods 0.000 claims description 3
- 230000002708 enhancing effect Effects 0.000 abstract description 2
- 241000282472 Canis lupus familiaris Species 0.000 description 1
- 241000283086 Equidae Species 0.000 description 1
- 241000282326 Felis catus Species 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/7837—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using objects detected or recognised in the video content
- G06F16/784—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using objects detected or recognised in the video content the detected or recognised objects being people
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/845—Structuring of content, e.g. decomposing content into time segments
- H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Signal Processing (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Library & Information Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a 5G-based law enforcement recorder distributed assistance calculation method, which comprises the following steps of: s1, a terminal video acquisition module of the 5G law enforcement recorder generates YUV video frame bare data and a timestamp which is an absolute time value, and the timestamp is sent to a terminal video display module, a terminal calculation fragment generation module and a terminal analysis preprocessing module; and a terminal GPS time synchronization module of the S2 and 5G law enforcement recorder synchronously updates the local time of the 5G law enforcement recorder by using a GPS or Beidou synchronous satellite signal, so that the time of the 5G law enforcement recorder for assisting analysis is kept consistent. According to the invention, the idle 5G law enforcement recorder is selected to participate in the assistance analysis and calculation through time slicing, so that the intelligent cooperative work is realized; by filtering the unchanged image, repeated analysis and calculation are reduced, and the overall performance is improved; the use of multiple 5G law enforcement recorders assists in enhancing the recognition analysis computing power of one 5G law enforcement recorder.
Description
Technical Field
The invention relates to the field of 5G communication, in particular to a 5G-based law enforcement recorder distributed assistance calculation method.
Background
The edge communication capability of the 5G equipment terminal is greatly improved (edge equipment is in direct communication with the terminal and is decentralized), the communication bandwidth is greatly improved, the network delay change is small, the computing capability of a single law enforcement recorder is limited, and the computing capability can be improved and the computing delay can be reduced by mutual assistance of adjacent law enforcement recorders in a 5G network.
Disclosure of Invention
In order to achieve the purposes of improving the computing capacity and reducing the computing time delay, the invention provides a 5G-based law enforcement recorder distributed assistance computing method, and the detailed technical scheme is as follows:
a law enforcement recorder distributed assistance calculation method based on 5G comprises the following steps:
s1, a terminal video acquisition module of the 5G law enforcement recorder generates YUV video frame bare data and a timestamp which is an absolute time value, and the timestamp is sent to a terminal video display module, a terminal calculation fragment generation module and a terminal analysis preprocessing module;
s2, a terminal GPS time synchronization module of the 5G law enforcement recorder synchronously updates local time of the 5G law enforcement recorder by using a GPS or Beidou synchronous satellite signal, so that the time of the 5G law enforcement recorder assisting analysis is kept consistent, the time deviation of the 5G law enforcement recorder assisting calculation is millisecond, and the terminal calculation fragment generation module and the terminal calculation analysis execution module realize accurate calculation task time distribution;
s3, a terminal calculation fragment generation module of the 5G law enforcement recorder generates video fragments for distributed assistance calculation;
3.1 the terminal computation fragment generation module collects idle time fragments of terminal computation analysis execution modules close to other 5G law enforcement recorders, and the idle time fragments can slice time every 5000 milliseconds;
3.2 the terminal computation fragment generation module selects idle participation assistance 5G law enforcement recorders according to the number of the computation types of the analysis tasks, informs a corresponding terminal computation analysis execution module to lock computation resources, disassembles a plurality of analysis computation tasks and sends the analysis computation tasks to a plurality of 5G law enforcement recorders, and breaks through the bottleneck of the analysis computation capability of a single 5G law enforcement recorder;
3.3 the terminal calculation fragment generation module merges and selects idle time fragments of the 5G law enforcement recorder, each continuous fragment uses video fragments of single I frame and multiple P frames, and more P frames are used for improving data compression amount and reducing data transmission amount;
3.4 the terminal computing fragment generating module divides the video frame into video fragments, video fragment frame data and time stamps according to the time fragments;
3.5 the terminal computation fragment generation module receives the notification that the video frame of the terminal analysis preprocessing module is not changed, and removes the corresponding frame of the video fragment according to the timestamp to reduce the computation amount of repeated image analysis;
3.6 the terminal computing fragment generating module sends the terminal video coding module to other 5G law enforcement recorders for assisting video analysis and computation;
s4, a terminal analysis preprocessing module of the 5G law enforcement recorder preprocesses the video;
4.1 the terminal analysis preprocessing module compares the difference of the front and rear video frame data to generate a variation bitmap, subtracts the data of the Y component of the front and rear YUV video frame data one by one according to pixels, wherein the subtraction value is zero and 0, and the subtraction value is non-zero and 1, and stores the data one by one according to the bit to generate the variation bitmap;
4.2 the terminal analysis preprocessing module counts the variation of the variation bitmap to judge whether the picture is changed, counts the number of 1 bit in the variation bitmap, and if the number of 1 bit is less than the preset value A, the picture is considered to be unchanged;
4.3 if no image changes, the terminal analysis preprocessing module sends the timestamp value of the video frame without change to the terminal computation fragment generating module, and informs the terminal computation fragment generating module to filter the image without change, so that repeated analysis and computation are reduced, and the overall performance is improved;
4.4 if there is image change, the terminal analysis preprocessing module sends the variation bitmap and the timestamp value to the terminal video coding module;
s5, the terminal video coding module codes the video fragments and encodes, analyzes and calculates key information in the video stream;
5.1 the terminal video coding module receives the video fragment of the terminal calculation fragment generation module to stamp the value and analyze the calculation type in time, and carries out H264/H265 coding;
5.2 the terminal video coding module adopts I frame coding to the first frame of the video fragment, uses P frame coding to other frames, uses the video compression fragment to become independent decryption, reduces the size of the compression fragment, is easier to calculate and analyze task allocation, and reduces data transmission amount;
5.3 the terminal video coding module receives the variation bitmap and the timestamp value of the terminal analysis preprocessing module, inserts an SEI custom information frame into each frame of the video compression fragment according to the timestamp value, and comprises the variation bitmap and the timestamp value of the frame and the number and analysis calculation type of a law enforcement recorder;
s6, a terminal 5G calculation distribution module of the 5G law enforcement recorder transmits the video compression fragments to a terminal 5G calculation receiving module of the 5G law enforcement recorder participating in the assistance analysis calculation by using 5G communication, and when a plurality of video analysis calculations are required to be carried out simultaneously, a plurality of 5G law enforcement recorders assist in accelerating intelligent analysis simultaneously;
s7, a terminal 5G calculation receiving module of the 5G law enforcement recorder calculates the video compression fragment of the distribution module by using a 5G communication receiving terminal 5G; sending the video compression fragments to a terminal video decoding module;
s8, a terminal video decoding module of the 5G law enforcement recorder decodes video frames of the video compression fragments to obtain YUV video fragments, and analyzes SEI custom information frames to obtain variable quantity bitmaps and timestamp values of each frame of the video fragments and analysis calculation types; sending the video fragment data and the variable quantity bitmap and the timestamp value of each frame to a terminal calculation analysis execution module;
s9, a terminal calculation analysis execution module of the 5G law enforcement recorder assists in analysis task management;
9.1 the terminal computation analysis execution module sends idle time slices to the terminal computation slice generation module;
9.2 the terminal computation analysis execution module receives the request of the terminal computation fragment generation module for locking the computation resource, carries the number of the locking law enforcement recorder, the associated time fragment and the number of the law enforcement recorder and is only used by the 5G law enforcement recorder; the 5G law enforcement recorders for assisting calculation enhance one law enforcement recorder, and break through the bottleneck of analysis and calculation capacity of a single 5G law enforcement recorder;
9.3 the terminal calculation analysis execution module receives the video fragment data of the terminal video decoding module, the variable quantity bitmap and the timestamp value of each frame and the law enforcement recorder number;
9.4 the terminal calculation analysis execution module finds the corresponding time slice according to the current time, compares the law enforcement recorder number with the law enforcement recorder number associated with the current time slice, and discards the video slice if the law enforcement recorder number is not associated with the current time slice;
9.5 if the terminal video decoding module determines to forward the calculation task to the terminal face comparison module or the terminal target identification module according to the analysis calculation type, the calculation task carries the video fragment data, the variable quantity bitmap and the timestamp value of each frame, and the law enforcement recorder number and the analysis calculation type;
s10, a terminal face comparison module of the 5G law enforcement recorder executes face comparison of the 5G law enforcement recorder;
10.1 the terminal face comparison module receives video fragment data of the terminal calculation analysis execution module, a variable quantity bitmap and a timestamp value of each frame and a law enforcement recorder number;
10.2 the terminal face comparison module carries out full-picture scanning on the first frame of the video fragment to carry out face detection calculation;
10.3 the terminal face comparison module performs face detection calculation on the corresponding regions with the bit 1 of all the variable quantity bitmaps for other video frames according to the variable quantity bitmaps, reduces the detection calculation of the regions which are not changed and are already detected by using the variable quantity bitmaps, and improves the distribution assistance calculation efficiency;
10.4, the face position and size are obtained by face detection calculation of the terminal face comparison module;
10.5 the terminal face comparison module obtains the face image data of the frame data according to the face position and size, carries out face comparison calculation, and finds the name and the identification attribute of the face in a face library;
10.6 the terminal face comparison module sends the name and identification attribute, position, size, timestamp value and law enforcement recorder number to the terminal 5G calculation object feedback module;
and S11, a terminal target identification module of the 5G law enforcement recorder identifies and processes the target object.
11.1 the terminal target identification module uses the neural network identification library to train different neural network identification libraries aiming at different targets so as to improve the identification accuracy;
11.2 the neural network recognition libraries of different targets are subjected to distributed assisted recognition calculation by different 5G law enforcement recorders, so that the recognition accuracy is improved, the recognition types are increased, and the recognition calculation time is reduced;
11.3 the terminal target identification module receives video fragment data of the terminal calculation analysis execution module, a variable quantity bitmap and a time stamp value of each frame and a law enforcement recorder number and an analysis calculation type; the terminal target identification module selects a neural network identification library according to the analysis and calculation type, and performs full-picture scanning on the first frame of the video fragment to perform target identification detection calculation;
11.4 the terminal target identification module carries out target identification detection calculation on other video frames according to the variable quantity bitmap and corresponding regions with the bit of 1 of all the variable quantity bitmaps, reduces the detection calculation of the detected regions and unchanged regions by using the variable quantity bitmap, and improves the distribution assistance calculation efficiency.
11.5 the target identification detection of the terminal target identification module calculates to obtain the target position, size and attribute, and the terminal target identification module sends the identified attribute, position, size, timestamp value and law enforcement recorder number to the terminal 5G calculation object feedback module;
s12, a terminal 5G calculation object feedback module of the 5G law enforcement recorder receives the identification results of the terminal face comparison module and the terminal target identification module, finds the law enforcement recorder initiating the identification calculation according to the law enforcement recorder number in the identification results, and the terminal 5G calculation object feedback module sends the identification results to a terminal synchronous superposition module of the law enforcement recorder initiating the identification calculation;
s13, a terminal synchronous superposition module of the 5G law enforcement recorder receives the YUV video frame bare data and the timestamp of a terminal video acquisition module, the receiving terminal 5G calculates the identification result of an object feedback module, finds out the corresponding YUV video frame according to the timestamp value in the identification result, superposes an identification frame according to the position and the size of the identification result, superposes character description according to the attribute of the identification result, and the terminal synchronous superposition module sends the superposed YUV video frame and the timestamp value to a terminal video display module;
and S14, the terminal video display module receives the YUV video frame and the timestamp value of the superposition target detection calculation result of the terminal synchronous superposition module, and plays the YUV video frame and the timestamp value according to the timestamp value and fixed time delay, so that a user can directly see the assisted identification calculation result on a display screen of the 5G law enforcement recorder, and the decision of the user is assisted.
The invention has the beneficial effects that:
1. an idle 5G law enforcement recorder is selected to participate in assistance analysis and calculation through time slicing, and intelligent cooperative work is achieved;
2. by filtering the unchanged image, repeated analysis and calculation are reduced, and the overall performance is improved;
3. the edge communication capacity of the 5G equipment terminal is greatly improved, a plurality of 5G law enforcement recorders are used for assisting in enhancing the identification, analysis and calculation capacity of one 5G law enforcement recorder, the bottleneck of the calculation capacity of a single 5G law enforcement recorder is broken through, the calculation capacity is improved, and the calculation time delay is reduced.
Drawings
FIG. 1 is a schematic connection diagram of modules in a 5G-based law enforcement recorder distributed assistance calculation method.
Detailed Description
The following examples are illustrative and are not to be construed as limiting the invention.
As shown in fig. 1, a law enforcement recorder distributed assistance calculation method based on 5G includes the following steps:
s1, a terminal video acquisition module 1 of the 5G law enforcement recorder generates YUV video frame naked data and a timestamp which is an absolute time value, and the YUV video frame naked data and the timestamp are sent to a terminal video display module 14, a terminal calculation fragment generation module 3 and a terminal analysis preprocessing module 4;
s2, a terminal GPS time synchronization module 2 of the 5G law enforcement recorder synchronously updates local time of the 5G law enforcement recorder by using a GPS or Beidou synchronous satellite signal, so that the time of the 5G law enforcement recorder assisting analysis is kept consistent, the time deviation of the 5G law enforcement recorder assisting calculation is millisecond, and the terminal calculation fragment generation module 3 and the terminal calculation analysis execution module 6 realize accurate calculation task time distribution;
s3, a terminal calculation fragment generation module 3 of the 5G law enforcement recorder generates video fragments for distributed assistance calculation;
3.1 the terminal computation fragment generation module 3 collects idle time fragments of the terminal computation analysis execution module 6 close to other 5G law enforcement recorders, and the idle time fragments can slice time (00:00 to 23:59) every 5000 milliseconds;
3.2 the terminal computation fragment generation module 3 selects idle participation assistance 5G law enforcement recorders according to the number of the computation types of the analysis tasks, informs the corresponding terminal computation analysis execution module 6 to lock computation resources, disassembles a plurality of analysis computation tasks and sends the analysis computation tasks to a plurality of 5G law enforcement recorders, and breaks through the bottleneck of the analysis computation capability of a single 5G law enforcement recorder;
3.3 the terminal computation fragment generation module 3 merges and selects idle time fragments of the 5G law enforcement recorder, each continuous fragment uses video fragments of single I frame and multiple P frames, and more P frames are used for increasing data compression amount and reducing data transmission amount;
3.4 the terminal computing fragment generating module 3 divides the video frame into video fragments, video fragment frame data and time stamps according to the time fragments;
3.5 the terminal computation fragment generation module 3 receives the notification that the video frame of the terminal analysis preprocessing module 4 is not changed, and removes the corresponding frame of the video fragment according to the timestamp to reduce the computation amount of repeated image analysis;
3.6 the terminal calculation fragment generation module 3 sends the terminal video coding module 5 to other 5G law enforcement recorders for assisting video analysis and calculation;
s4, a terminal analysis preprocessing module 4 of the 5G law enforcement recorder preprocesses the video;
4.1 the terminal analysis preprocessing module 4 compares the difference of the previous and the next video frame data to generate a variation bitmap, the terminal analysis preprocessing module 4 subtracts the data of the Y component of the previous and the next YUV video frame data one by one according to pixels, the subtraction value is zero and is 0, the subtraction value is non-zero and is 1, and the data are stored one by one according to bits to generate the variation bitmap;
4.2 the terminal analysis preprocessing module 4 counts the variation of the variation bitmap to judge whether the picture is changed, counts the number of 1 bit in the variation bitmap, if 1 bit is smaller than the preset value A (such as: 30), the picture is considered to be unchanged;
4.3 if no image changes, the terminal analysis preprocessing module 4 sends the timestamp value of the video frame without change to the terminal computation fragment generating module 3, and informs the terminal computation fragment generating module 3 to filter the image without change, thereby reducing repeated analysis and computation and improving the overall performance;
4.4 if there is image change, the terminal analysis preprocessing module 4 sends the variation bitmap and the timestamp value to the terminal video coding module 5;
s5, the terminal video coding module 5 codes the video film and encodes the analysis and calculation key information in the video stream;
5.1 the terminal video coding module 5 receives the video fragment of the terminal calculation fragment generation module 3 to perform H264/H265 coding according to the timestamp value and the analysis calculation type in time;
5.2 the terminal video coding module 5 adopts I frame coding for the first frame of the video fragment, uses P frame coding for other frames, and uses the video compression fragment to be independently decrypted, thereby reducing the size of the compression fragment, being easier to calculate and analyze task allocation and reducing data transmission amount;
5.3 the terminal video coding module 5 receives the variation bitmap and the timestamp value of the terminal analysis preprocessing module 4, inserts an SEI custom information frame into each frame of the video compression fragment according to the timestamp value, and comprises the variation bitmap and the timestamp value of the frame and the number and analysis calculation type of a law enforcement recorder;
s6, a terminal 5G calculation distribution module 7 of the 5G law enforcement recorder uses 5G communication to compress and divide the video into pieces to be sent to a terminal 5G calculation receiving module 8 of the 5G law enforcement recorder participating in assistance analysis calculation, and when a plurality of video analysis calculations are required to be carried out simultaneously, a plurality of 5G law enforcement recorders assist in accelerating intelligent analysis simultaneously;
s7, a terminal 5G calculation receiving module 8 of the 5G law enforcement recorder calculates the video compression fragment of the distribution module 7 by using a 5G communication receiving terminal 5G; sending the video compression fragment to a terminal video decoding module 9;
s8, a terminal video decoding module 9 of the 5G law enforcement recorder decodes the video frame of the video compression fragment to obtain YUV video fragment, and analyzes the SEI custom information frame to obtain the variation bitmap and the timestamp value of each frame of the video fragment and the analysis and calculation type; sending the video fragment data and the variation bitmap and the timestamp value of each frame to a terminal calculation analysis execution module 6;
s9, a terminal calculation analysis execution module 6 of the 5G law enforcement recorder assists in analysis task management;
9.1 the terminal computation analysis execution module 6 sends the idle time slice to the terminal computation slice generation module 3;
9.2 the terminal calculation analysis execution module 6 receives the request of the terminal calculation fragment generation module 3 for locking the calculation resource, carries the locking law enforcement recorder number, the associated time fragment and the law enforcement recorder number, and is only used by the 5G law enforcement recorder; the 5G law enforcement recorders for assisting calculation enhance one law enforcement recorder, and break through the bottleneck of analysis and calculation capacity of a single 5G law enforcement recorder;
9.3 the terminal calculation analysis execution module 6 receives the video fragment data of the terminal video decoding module 9, the variable quantity bitmap and the timestamp value of each frame and the law enforcement recorder number;
9.4 the terminal calculation analysis execution module 6 finds the corresponding time slice according to the current time, compares the law enforcement recorder number with the law enforcement recorder number associated with the current time slice, and discards the video slice if the law enforcement recorder number is not associated with the current time slice;
9.5 if the terminal video decoding module 9 determines to forward the calculation task to an intelligent analysis planning module such as the terminal face comparison module 10 or the terminal target identification module 11 according to the analysis calculation type, and the intelligent analysis planning module carries the video fragment data, the variable bitmap and the timestamp value of each frame and the number and the analysis calculation type of the law enforcement recorder;
s10, the terminal face comparison module 10 of the 5G law enforcement recorder executes the face comparison of the 5G law enforcement recorder;
10.1 the terminal face comparison module 10 receives the video fragment data of the terminal calculation analysis execution module 6, the variable quantity bitmap and the timestamp value of each frame and the law enforcement recorder number;
10.2 the terminal face comparison module 10 carries out full-picture scanning on the first frame of the video fragment to carry out face detection calculation;
10.3 the terminal face comparison module 10 performs face detection calculation on the corresponding regions with the bit 1 of all the variable quantity bitmaps according to the variable quantity bitmaps for other video frames, reduces the detection calculation of the regions which have not been detected and has not changed by using the variable quantity bitmaps, and improves the distribution assistance calculation efficiency;
10.4 the face detection of the terminal face comparison module 10 is calculated to obtain the face position and size;
10.5 the terminal face comparison module 10 obtains the face image data of the frame data according to the face position and size, carries out face comparison calculation, and finds the name and the identification attribute of the face in a face library;
10.6 the terminal face comparison module 10 sends the name and identification attributes such as occupation, illegal record, position, size, timestamp value, and law enforcement recorder number to the terminal 5G calculation object feedback module 12;
and S11, the terminal target identification module 11 of the 5G law enforcement recorder identifies and processes the target object.
11.1 the terminal target identification module 11 uses the neural network identification library to train different neural network identification libraries aiming at different targets so as to improve the identification accuracy; if aiming at a vehicle neural network recognition library, recognizing the license plate, the vehicle color and the vehicle model; aiming at a human neural network identification library, identifying the color of clothes of a human, the age and the height of the human; identifying cats, dogs and horses aiming at an animal network identification library;
11.2 the neural network recognition libraries of different targets are subjected to distributed assisted recognition calculation by different 5G law enforcement recorders, so that the recognition accuracy is improved, the recognition types are increased, and the recognition calculation time is reduced;
11.3 the terminal target identification module 11 receives the video fragment data of the terminal calculation analysis execution module 6, the variable quantity bitmap and the timestamp value of each frame, the law enforcement recorder number and the analysis calculation type; the terminal target identification module 11 selects a neural network identification library according to the analysis and calculation type, and performs full-picture scanning on the first frame of the video fragment to perform target identification detection calculation;
11.4 the terminal target identification module 11 performs target identification detection calculation on other video frames according to the variation bitmap and corresponding regions with bits 1 of all the variation bitmaps, reduces the detection calculation of the detected regions and unchanged regions by using the variation bitmap, and improves the distribution assistance calculation efficiency.
11.5 the target identification detection of the terminal target identification module 11 calculates to obtain the target position, size and attribute, the terminal target identification module 11 sends the identified attributes such as license plate, person age, position, size, timestamp value and law enforcement recorder number to the terminal 5G calculation object feedback module 12;
s12, a terminal 5G calculation object feedback module 12 of the 5G law enforcement recorder receives the identification results of the terminal face comparison module 10 and the terminal target identification module 11, finds the law enforcement recorder initiating the identification calculation according to the law enforcement recorder number in the identification results, and the terminal 5G calculation object feedback module 12 sends the identification results to a terminal synchronous superposition module 13 of the law enforcement recorder initiating the identification calculation;
s13, a terminal synchronous superposition module 13 of the 5G law enforcement recorder receives YUV video frame bare data and a timestamp of a terminal video acquisition module 1, the receiving terminal 5G calculates an identification result of an object feedback module 12, finds a corresponding YUV video frame according to a timestamp value in the identification result, superposes an identification frame according to the position and the size of the identification result, and superposes character descriptions such as a license plate, the age of a person, names and the like according to the attributes of the identification result; the terminal synchronous superposition module 13 sends the superposed YUV video frames and the timestamp values to the terminal video display module 14;
s14, the terminal video display module 14 receives the YUV video frame and the timestamp value of the calculation result of the target superposition detection of the terminal synchronous superposition module 13, and plays the YUV video frame and the timestamp value according to the timestamp value and a fixed time delay (e.g. 80ms), so that the user can directly view the assisted identification calculation result on the display screen of the 5G law enforcement recorder, thereby assisting the user in making a decision.
The invention also provides a 5G-based law enforcement recorder distributed assistance computing system which comprises a terminal video acquisition module 1, a terminal GPS time synchronization module 2, a terminal computation fragment generation module 3, a terminal analysis preprocessing module 4, a terminal video coding module 5, a terminal computation analysis execution module 6, a terminal 5G computation distribution module 7, a terminal 5G computation receiving module 8, a terminal video decoding module 9, a terminal face comparison module 10, a terminal target identification module 11, a terminal 5G computation object feedback module 12, a terminal synchronization superposition module 13 and a terminal video display module 14.
The terminal video acquisition module 1 is a video acquisition module of a 5G law enforcement recorder, and the acquisition recorder generates YUV video frame naked data and a timestamp;
the terminal GPS time synchronization module 2 is a time synchronization module of the 5G law enforcement recorder, and the terminal GPS time synchronization module 2 updates the local time of the 5G law enforcement recorder by using a GPS or Beidou synchronous satellite signal;
the terminal calculation fragment generation module 3 is a task disassembling module of the 5G law enforcement recorder;
the terminal analysis preprocessing module 4 is a video preprocessing module of the 5G law enforcement recorder;
the terminal video coding module 5 receives the video fragment of the terminal calculation fragment generation module 3 to stamp the value and analyze the calculation type in time, and performs H264/H265 coding;
the terminal calculation analysis execution module 6 is a calculation analysis execution module of the 5G law enforcement recorder;
the terminal 5G calculation distribution module 7 uses 5G communication to compress and divide the video into pieces to send to a terminal 5G calculation receiving module 8 of the 5G law enforcement recorder participating in the assisted analysis calculation, and when a plurality of video analysis calculations are required to be carried out simultaneously, a plurality of 5G law enforcement recorders assist in accelerating intelligent analysis simultaneously;
the terminal 5G calculation receiving module 8 uses the 5G communication receiving terminal 5G to calculate the video compression fragment data of the distribution module 7;
the terminal video decoding module 9 decodes the video frames of the video compression fragments to obtain YUV video fragments, and analyzes the SEI custom information frames to obtain the variation bitmaps and the timestamp values of each frame of the video fragments and the analysis calculation types;
the terminal face comparison module 10 is a face comparison module of a 5G law enforcement recorder;
the terminal target identification module 11 is a terminal target identification module 11 of the 5G law enforcement recorder, and the terminal target identification module 11 uses a neural network identification library to train different neural network identification libraries aiming at different targets so as to improve the identification accuracy;
the terminal 5G calculation object feedback module 12 receives the identification results of the terminal face comparison module 10 and the terminal target identification module 11, and finds out the law enforcement recorder initiating the identification calculation according to the law enforcement recorder number in the identification results;
the terminal synchronous superposition module 13 receives the YUV video frame bare data and the timestamp of the terminal video acquisition module 1, receives the identification result of the terminal 5G calculation object feedback module 12, finds the corresponding YUV video frame according to the timestamp value in the identification result, superposes the identification frame according to the position and the size of the identification result, and superposes character description according to the attribute of the identification result;
the terminal video display module 14 is a display module of the 5G law enforcement recorder, so that a user can directly see the assisted identification calculation result on a display screen of the 5G law enforcement recorder.
In light of the foregoing description of the preferred embodiment of the present invention, many modifications and variations will be apparent to those skilled in the art without departing from the spirit and scope of the invention. The technical scope of the present invention is not limited to the content of the specification, and must be determined according to the scope of the claims.
Claims (4)
1. A law enforcement recorder distributed assistance calculation method based on 5G is characterized by comprising the following steps:
the method comprises the steps that a terminal video acquisition module (1) of an S1 and 5G law enforcement recorder generates YUV video frame naked data and a timestamp, the timestamp is an absolute time value, and the YUV video frame naked data and the timestamp are sent to a terminal video display module (14), a terminal calculation fragment generation module (3) and a terminal analysis preprocessing module (4);
s2, a terminal GPS time synchronization module (2) of the 5G law enforcement recorder synchronously updates local time of the 5G law enforcement recorder by using a GPS or Beidou synchronous satellite signal, so that the time of the 5G law enforcement recorder assisting in analysis is kept consistent, the time deviation of the 5G law enforcement recorder assisting in calculation is millisecond, and the terminal calculation fragment generation module (3) and the terminal calculation analysis execution module (6) realize accurate calculation task time distribution;
s3, a terminal calculation fragment generation module (3) of the 5G law enforcement recorder generates video fragments for distributed assistance calculation;
s4, a terminal analysis preprocessing module (4) of the 5G law enforcement recorder preprocesses the video;
s5, a terminal video coding module (5) codes the video slices and encodes, analyzes and calculates key information in the video stream;
s6, a terminal 5G calculation distribution module (7) of the 5G law enforcement recorder uses 5G communication to compress videos and send the videos to a terminal 5G calculation receiving module (8) of the 5G law enforcement recorder participating in assistance analysis calculation, and when a plurality of video analysis calculations are required to be carried out simultaneously, a plurality of 5G law enforcement recorders assist in accelerating intelligent analysis;
s7, a terminal 5G calculation receiving module (8) of the 5G law enforcement recorder calculates the video compression fragment of the distribution module (7) by using a 5G communication receiving terminal 5G; sending the video compression fragments to a terminal video decoding module (9);
s8, a terminal video decoding module (9) of the 5G law enforcement recorder decodes video frames of the video compression fragments to obtain YUV video fragments, and analyzes SEI custom information frames to obtain variation bitmaps and timestamp values of each frame of the video fragments and analysis calculation types; sending the video fragment data, the variable quantity bitmap of each frame and the timestamp value to a terminal calculation analysis execution module (6);
s9, a terminal calculation analysis execution module (6) of the 5G law enforcement recorder assists in analysis task management;
s10, a terminal face comparison module (10) of the 5G law enforcement recorder executes face comparison of the 5G law enforcement recorder;
s11, a terminal target identification module (11) of the 5G law enforcement recorder identifies and processes the target object;
s12, a terminal 5G calculation object feedback module (12) of the 5G law enforcement recorder receives the identification results of the terminal face comparison module (10) and the terminal target identification module (11), finds the law enforcement recorder initiating the identification calculation according to the law enforcement recorder number in the identification results, and the terminal 5G calculation object feedback module (12) sends the identification results to a terminal synchronous superposition module (13) of the law enforcement recorder initiating the identification calculation;
s13, a terminal synchronous superposition module (13) of the 5G law enforcement recorder receives YUV video frame bare data and a timestamp of a terminal video acquisition module (1), the receiving terminal 5G calculates an identification result of an object feedback module (12), finds a corresponding YUV video frame according to a timestamp value in the identification result, superposes the identification frame according to the position and the size of the identification result, superposes character description according to the attribute of the identification result, and the terminal synchronous superposition module (13) sends the superposed YUV video frame and the timestamp value to a terminal video display module (14);
s14, the terminal video display module (14) receives the YUV video frame and the timestamp value of the superposition target detection calculation result of the terminal synchronous superposition module (13), and plays the YUV video frame and the timestamp value according to the timestamp value and fixed time delay, so that a user can directly see the assistance identification calculation result on a display screen of the 5G law enforcement recorder, and the decision of the user is assisted;
the S3 includes the steps of:
3.1 the terminal computation fragment generation module (3) collects idle time fragments of the terminal computation analysis execution module (6) close to other 5G law enforcement recorders, and the idle time fragments can slice time every 5000 milliseconds;
3.2 the terminal computation fragment generation module (3) selects idle assisting 5G law enforcement recorders according to the number of the computation types of the analysis tasks, informs the corresponding terminal computation analysis execution module (6) to lock computation resources, disassembles a plurality of analysis computation tasks and sends the multiple analysis computation tasks to a plurality of 5G law enforcement recorders, and breaks through the bottleneck of the analysis computation capability of a single 5G law enforcement recorder;
3.3 the terminal computation fragment generation module (3) merges and selects idle time fragments of the 5G law enforcement recorder, each continuous fragment uses video fragments of single I frame and multiple P frames, and more P frames are used for increasing data compression amount and reducing data transmission amount;
3.4 the terminal computing fragment generating module (3) divides the video frame into video fragments, video fragment frame data and time stamps according to the time fragments;
3.5 the terminal computation fragment generation module (3) receives the notice that the video frame is not changed from the terminal analysis preprocessing module (4), and removes the corresponding frame of the video fragment according to the timestamp to reduce the computation amount of repeated image analysis;
3.6 the terminal calculation fragment generation module (3) sends the terminal video coding module (5) to other 5G law enforcement recorders for assisting video analysis and calculation;
the S4 includes the steps of:
4.1 the terminal analysis preprocessing module (4) compares the difference of the front and back video frame data to generate a variation bitmap, the terminal analysis preprocessing module (4) subtracts the data of the Y component of the front and back YUV video frame data one by one according to pixels, the subtraction value is zero and 0, the subtraction value is non-zero and 1, and the data are stored one by one according to bits to generate the variation bitmap;
4.2 the terminal analysis preprocessing module (4) counts the variation of the variation bitmap to judge whether the picture is changed, counts the number of 1 bit in the variation bitmap, and if the number of 1 bit is less than the preset value A, the picture is considered to be unchanged;
4.3 if no image changes, the terminal analysis preprocessing module (4) sends the timestamp value of the video frame without change to the terminal calculation fragment generating module (3), and informs the terminal calculation fragment generating module (3) of filtering the image without change, thereby reducing repeated analysis and calculation and improving the overall performance;
4.4 if there is image change, the terminal analysis preprocessing module (4) sends the change bitmap and the timestamp value to the terminal video coding module (5);
the S9 includes the steps of:
9.1 the terminal computation analysis execution module (6) sends the idle time slice to the terminal computation slice generation module (3);
9.2 the terminal calculation analysis execution module (6) receives the request of the terminal calculation fragment generation module (3) for locking the calculation resource, carries the number of the locking law enforcement recorder, associates the time fragment with the number of the law enforcement recorder, and is only used by the 5G law enforcement recorder; the 5G law enforcement recorders for assisting calculation enhance one law enforcement recorder, and break through the bottleneck of analysis and calculation capacity of a single 5G law enforcement recorder;
9.3 the terminal calculation analysis execution module (6) receives the video fragment data of the terminal video decoding module (9), the variable quantity bitmap and the timestamp value of each frame and the law enforcement recorder number;
9.4 the terminal calculation analysis execution module (6) finds the corresponding time slice according to the current time, compares the law enforcement recorder number with the law enforcement recorder number associated with the current time slice, and discards the video slice if the law enforcement recorder number is not associated with the current time slice;
9.5 if the terminal video decoding module (9) determines to forward the calculation task to the terminal face comparison module (10) or the terminal target identification module (11) according to the analysis calculation type, the calculation task carries the video fragment data, the variable quantity bitmap and the time stamp value of each frame and the law enforcement recorder number and the analysis calculation type.
2. The 5G-based law enforcement recorder distributed assistance computing method according to claim 1, wherein the S5 comprises the following steps:
5.1 the terminal video coding module (5) receives the video fragment of the terminal calculation fragment generating module (3) to perform H264/H265 coding according to the timestamp value and the analysis calculation type;
5.2 the terminal video coding module (5) adopts I frame coding to the first frame of the video fragment, uses P frame coding to other frames, uses the video compression fragment to become independent decryption, reduces the size of the compression fragment, is easier to calculate and analyze task allocation, and reduces data transmission amount;
5.3 the terminal video coding module (5) receives the variation bitmap and the timestamp value of the terminal analysis preprocessing module (4), inserts an SEI custom information frame into each frame of the video compression fragment according to the timestamp value, and comprises the variation bitmap and the timestamp value of the frame and the law enforcement recorder number and analysis calculation type.
3. The 5G-based law enforcement recorder distributed assistance computing method according to claim 1, wherein the S10 comprises the following steps:
10.1, a terminal face comparison module (10) receives video fragment data of the terminal calculation analysis execution module (6), a variable quantity bitmap and a timestamp value of each frame and a law enforcement recorder number;
10.2 the terminal face comparison module (10) carries out full-picture scanning on the first frame of the video fragment to carry out face detection calculation;
10.3 the terminal face comparison module (10) performs face detection calculation on the corresponding regions with the bit 1 of all the variable quantity bitmaps according to the variable quantity bitmaps for other video frames, reduces the detection calculation of the regions which are not changed and are detected by using the variable quantity bitmaps, and improves the distribution assistance calculation efficiency;
10.4, the face detection of the terminal face comparison module (10) calculates to obtain the face position and size;
10.5 the terminal face comparison module (10) obtains the face image data of the frame data according to the face position and size, carries out face comparison calculation, and finds the name and the identification attribute of the face in a face library;
10.6 the terminal face comparison module (10) sends the name and identification attribute, position, size, timestamp value, law enforcement recorder number to the terminal 5G calculation object feedback module (12).
4. The 5G-based law enforcement recorder distributed assistance computing method according to claim 1, wherein the S11 comprises the following steps:
11.1 the terminal target identification module (11) uses a neural network identification library to train different neural network identification libraries aiming at different targets so as to improve the identification accuracy;
11.2 the neural network recognition libraries of different targets are subjected to distributed assisted recognition calculation by different 5G law enforcement recorders, so that the recognition accuracy is improved, the recognition types are increased, and the recognition calculation time is reduced;
11.3 the terminal target identification module (11) receives the video fragment data of the terminal calculation analysis execution module (6), the variable quantity bitmap and the timestamp value of each frame and the law enforcement recorder number and the analysis calculation type; the terminal target identification module (11) selects a neural network identification library according to the analysis and calculation type, and performs full-picture scanning on the first frame of the video fragment to perform target identification detection calculation;
11.4 the terminal target identification module (11) carries out target identification detection calculation on other video frames according to the variable quantity bitmap and corresponding regions with the bit of 1 of all variable quantity bitmaps, reduces the detection calculation of the detected regions and unchanged regions by using the variable quantity bitmap, and improves the distribution assistance calculation efficiency.
11.5 the target identification detection of the terminal target identification module (11) calculates to obtain the target position, size and attribute, the terminal target identification module (11) sends the identified attribute, position, size, timestamp value and law enforcement recorder number to the terminal 5G calculation object feedback module (12).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011550001.5A CN112584073B (en) | 2020-12-24 | 2020-12-24 | 5G-based law enforcement recorder distributed assistance calculation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011550001.5A CN112584073B (en) | 2020-12-24 | 2020-12-24 | 5G-based law enforcement recorder distributed assistance calculation method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112584073A CN112584073A (en) | 2021-03-30 |
| CN112584073B true CN112584073B (en) | 2022-08-02 |
Family
ID=75140468
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011550001.5A Active CN112584073B (en) | 2020-12-24 | 2020-12-24 | 5G-based law enforcement recorder distributed assistance calculation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112584073B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113542842A (en) * | 2021-07-14 | 2021-10-22 | 国网信息通信产业集团有限公司 | Video synchronization processing method and system suitable for edge calculation |
| CN115985007A (en) * | 2022-12-06 | 2023-04-18 | 杭州未兰石云信息科技有限公司 | 5G recorder video inspection method and system based on low power consumption |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101808126A (en) * | 2010-03-12 | 2010-08-18 | 清华大学 | Distributed computation system and method used for real-time video processing |
| CN106375793A (en) * | 2016-08-29 | 2017-02-01 | 东方网力科技股份有限公司 | Superposition method and superposition system of video structured information, and user terminal |
| CN109117821A (en) * | 2018-08-31 | 2019-01-01 | 广东九联科技股份有限公司 | A kind of face identification system and method for law-enforcing recorder |
| US10719899B1 (en) * | 2017-08-31 | 2020-07-21 | Steve Dabell | Method and apparatus for utilizing estimated patrol properties and historic patrol records |
| CN111866450A (en) * | 2020-06-18 | 2020-10-30 | 昇辉控股有限公司 | Intelligent video monitoring device based on cloud edge cooperation |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3785068B2 (en) * | 2001-08-28 | 2006-06-14 | 日本電信電話株式会社 | Video analysis apparatus, video analysis method, video analysis program, and program recording medium |
| CN110430443B (en) * | 2019-07-11 | 2022-01-25 | 平安科技（深圳）有限公司 | Method and device for cutting video shot, computer equipment and storage medium |
-
2020
- 2020-12-24 CN CN202011550001.5A patent/CN112584073B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101808126A (en) * | 2010-03-12 | 2010-08-18 | 清华大学 | Distributed computation system and method used for real-time video processing |
| CN106375793A (en) * | 2016-08-29 | 2017-02-01 | 东方网力科技股份有限公司 | Superposition method and superposition system of video structured information, and user terminal |
| US10719899B1 (en) * | 2017-08-31 | 2020-07-21 | Steve Dabell | Method and apparatus for utilizing estimated patrol properties and historic patrol records |
| CN109117821A (en) * | 2018-08-31 | 2019-01-01 | 广东九联科技股份有限公司 | A kind of face identification system and method for law-enforcing recorder |
| CN111866450A (en) * | 2020-06-18 | 2020-10-30 | 昇辉控股有限公司 | Intelligent video monitoring device based on cloud edge cooperation |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112584073A (en) | 2021-03-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112584073B (en) | 5G-based law enforcement recorder distributed assistance calculation method | |
| CN107004271B (en) | Display method, display apparatus, electronic device, computer program product, and storage medium | |
| CN110996125B (en) | Video stream generation method and device, electronic equipment and storage medium | |
| CA2824330C (en) | An integrated intelligent server based system and method/systems adapted to facilitate fail-safe integration and/or optimized utilization of various sensory inputs | |
| US20120179742A1 (en) | Integrated intelligent server based system and method/systems adapted to facilitate fail-safe integration and/or optimized utilization of various sensory inputs | |
| CN112767711B (en) | Multi-class multi-scale multi-target snapshot method and system | |
| CN107977922B (en) | Image analysis method, device and system | |
| CN111629264B (en) | Web-based separate front-end image rendering method | |
| CN112950951B (en) | Intelligent information display method, electronic device and storage medium | |
| CN111091106A (en) | Image clustering method and device, storage medium and electronic device | |
| CN107071337A (en) | The transmission method and device of a kind of video monitoring image | |
| CN113329139A (en) | Video stream processing method, device and computer readable storage medium | |
| CN104333732A (en) | Distributed video analysis method and system thereof | |
| CN112257691B (en) | Community security implementation method, system and equipment based on 5G edge calculation | |
| CN116962598B (en) | Monitoring video information fusion method and system | |
| CN103150319A (en) | CS (Client Server) framework-based feature retrieval rear video abstraction retrieval system | |
| CN110855947B (en) | Image snapshot processing method and device | |
| CN111447404B (en) | Video camera | |
| CN215186950U (en) | Pedestrian red light running behavior evidence acquisition device based on face recognition technology | |
| CN112613396B (en) | Task emergency degree processing method and system | |
| CN113362233B (en) | Picture processing method, device, equipment, system and storage medium | |
| CN111382697B (en) | Image data processing method and first electronic equipment | |
| CN111818300B (en) | Data storage method, data query method, data storage device, data query device, computer equipment and storage medium | |
| CN113689707B (en) | Video data processing method, device and computer readable storage medium | |
| CN110855930B (en) | Intelligent identification method and system for network equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |