CN113326714A - Target comparison method and device, electronic equipment and readable storage medium - Google Patents
Target comparison method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN113326714A CN113326714A CN202010128658.6A CN202010128658A CN113326714A CN 113326714 A CN113326714 A CN 113326714A CN 202010128658 A CN202010128658 A CN 202010128658A CN 113326714 A CN113326714 A CN 113326714A
- Authority
- CN
- China
- Prior art keywords
- target
- determining
- compared
- blacklist
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000012545 processing Methods 0.000 claims description 38
- 238000012163 sequencing technique Methods 0.000 claims description 28
- 238000004891 communication Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 2
- 230000004044 response Effects 0.000 abstract description 4
- 238000012544 monitoring process Methods 0.000 description 15
- 238000010586 diagram Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Data Mining & Analysis (AREA)
- Human Computer Interaction (AREA)
- Oral & Maxillofacial Surgery (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a target comparison method, a target comparison device, an electronic device and a readable storage medium, wherein the method comprises the following steps: acquiring a control plan; determining a first number of blacklist sub-libraries to be compared and a second number of target models in the blacklist sub-libraries to be compared based on the control plan; the different blacklist sub-libraries store target models of different classifications in the blacklist library; determining a target comparison strategy based on the third quantity, the first quantity and the second quantity of the control plan; and performing target comparison based on the target comparison strategy. The method can improve the rationality of the target comparison strategy in the scene that the same target model needs to be compared with a plurality of different types of target models, and provides support for high-concurrency and high-response alarm.
Description
Technical Field
The present disclosure relates to video surveillance technologies, and in particular, to a method and an apparatus for comparing objects, an electronic device, and a readable storage medium.
Background
Dynamic face recognition control utilizes a face recognition technology to model a face picture captured by a face capturing machine (such as a monitoring front end with a face capturing function) to obtain a face model, the face model is compared with a face model which is loaded in advance in a blacklist library, the similarity between the current face model and the face model in the blacklist library is analyzed, N face models (which can be called Top N) with the similarity exceeding a preset similarity threshold in the blacklist library and the front of the similarity are obtained according to similarity sequencing, and the N face models are output to an application to give an alarm.
However, when the comparison alarm of multiple different types of blacklist libraries needs to be performed on the same face model, a high concurrent high-response alarm capability is required, and the processing performance of the current blacklist library comparison scheme cannot meet the requirement.
Disclosure of Invention
In view of the above, the present application provides a target comparison method, an apparatus, an electronic device and a readable storage medium.
Specifically, the method is realized through the following technical scheme:
according to a first aspect of the embodiments of the present application, there is provided a target matching method applied to a target matching device, the method including:
acquiring a control plan;
determining a first number of blacklist sub-libraries to be compared and a second number of target models in the blacklist sub-libraries to be compared based on the control plan; the different blacklist sub-libraries store target models of different classifications in the blacklist library;
determining a target comparison strategy based on the third quantity, the first quantity and the second quantity of the control plan;
and performing target comparison based on the target comparison strategy.
According to a second aspect of the embodiments of the present application, there is provided a target matching device, applied to a target matching device, the device including:
the acquisition unit is used for acquiring a deployment and control plan;
the first determining unit is used for determining a first number of the blacklist sub-libraries to be compared and a second number of the target models in the blacklist sub-libraries to be compared based on the deployment and control plan; the different blacklist sub-libraries store target models of different classifications in the blacklist library;
a second determining unit, configured to determine a target comparison strategy based on a third quantity, the first quantity, and the second quantity of the deployment and control plan;
and the comparison unit is used for comparing the targets based on the target comparison strategy.
According to a third aspect of the embodiments of the present application, there is provided an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
and the processor is used for realizing the target comparison method when executing the program stored in the memory.
According to a fourth aspect of the embodiments of the present application, there is provided a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements the above-mentioned target matching method.
According to the target comparison method, the control plan is obtained, the first number of the blacklist sub-libraries to be compared and the second number of the target models in the blacklist sub-libraries to be compared are determined based on the control plan, the target comparison strategy is determined based on the third number, the first number and the second number of the control plan, and the target comparison is carried out based on the target comparison strategy, so that the rationality of the target comparison strategy in a scene that the comparison of the target models of different types needs to be carried out on the same target model is improved, and support is provided for high-concurrency and high-response alarm.
Drawings
Fig. 1 is a schematic flowchart of a target matching method according to an exemplary embodiment of the present application;
fig. 2 is a schematic structural diagram of a target alignment apparatus according to an exemplary embodiment of the present application;
fig. 3 is a schematic diagram of a hardware structure of an electronic device according to an exemplary embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
In order to make the technical solutions provided in the embodiments of the present application better understood and make the above objects, features and advantages of the embodiments of the present application more comprehensible, the technical solutions in the embodiments of the present application are described in further detail below with reference to the accompanying drawings.
Referring to fig. 1, a flow chart of a target matching method provided in the present embodiment is shown, wherein the target matching method can be applied to any device with target matching capability (referred to as a target matching device herein), including but not limited to a monitoring front end or a monitoring back end, as shown in fig. 1, the target matching method can include the following steps:
and S100, acquiring a control plan.
In the embodiment of the present application, the deployment plan includes, but is not limited to, a deployment plan for real-time capturing pictures of a monitoring front end (such as a capturing machine), a deployment plan for video capturing, or a deployment plan for picture set data.
For example, the monitoring front end does not refer to a fixed monitoring front end, but may refer to any monitoring front end that provides a target snapshot picture for a target comparison device that applies the target comparison scheme provided by the embodiment of the present application.
Illustratively, the target includes, but is not limited to, a human face, a human body, a vehicle, or the like; the target comparison comprises face model comparison, human body model comparison or vehicle model comparison and the like.
In the embodiment of the present application, the control plan may include, but is not limited to, types of blacklist sub-libraries that need to be compared, a similarity threshold for performing target comparison, Top N, and the like. Different blacklist sub-libraries store different categories of target models in the blacklist.
For example, the target model may be classified according to region (e.g., provincial information), crime level (e.g., suspect, on-flight), and the like.
For example, the similarity threshold for different blacklist sub-libraries may be the same or different; the Top N for different blacklist sub-libraries may be the same or different.
For example, one or more deployment plans may be configured for the same monitoring terminal.
Step S110, determining a first number of blacklist sub-libraries to be compared and a second number of target models in the blacklist sub-libraries to be compared based on the acquired deployment and control plan.
In this embodiment of the application, when the target comparison device obtains the deployment plan, the number of the blacklist sub-libraries to be compared (referred to as a first number) and the number of the target models in the blacklist library to be compared (referred to as a second number) may be determined based on the obtained deployment plan.
For example, assuming that the deployment plan for the monitoring terminal a includes a deployment plan1 and a deployment plan2, the deployment plan1 needs to compare a target model (which may be referred to as a target model to be compared) corresponding to a target snapshot picture of the monitoring terminal with target models in the blacklist sub-library 1 and the blacklist sub-library 2, and the deployment plan2 needs to compare the target model to be compared with target models in the blacklist sub-library 2 and the blacklist sub-library 3, the number of the blacklist sub-libraries to be compared for the monitoring terminal a is 3 (blacklist sub-libraries 1 to 3), and the number of the target models in the blacklist sub-library to be compared is the number of the concentrated target models in the blacklist sub-libraries 1 to 3.
Step S120, determining a target comparison strategy based on the third quantity, the first quantity and the second quantity of the deployment and control plan.
In the embodiment of the application, when one deployment and control plan is associated with a plurality of blacklist sub-libraries (that is, a target model in the plurality of blacklist sub-libraries is required to be compared in the deployment and control plan), when the Top N of the deployment and control plan is determined, the similarity of the target model in each blacklist sub-library can be sorted respectively, and then the plurality of blacklist sub-libraries associated with the deployment and control plan are sorted; alternatively, the similarity of each target model in the union of the blacklist sub-libraries associated with the control plan can be ranked.
It should be noted that, in the embodiment of the present application, unless otherwise specified, the mentioned similarity of the target model in the blacklist sub-library refers to a similarity between the target model to be compared and the target model in the blacklist sub-library.
In addition, when a plurality of deployment plans are configured and one deployment plan is associated with a plurality of blacklist sub-libraries, there may be overlap between blacklist sub-libraries associated with different deployment plans.
For example, still taking the above example as an example, deployment plan1 and deployment plan2 both need to compare the target models in blacklist sub-base 2.
Therefore, when the target comparison is performed according to the target comparison strategy of firstly sorting the similarity of the target models in each blacklist sub-library and then sorting the plurality of blacklist sub-libraries associated with the deployment and control plan, when the number of the blacklist sub-libraries associated with the deployment and control plan is large, the sorting times for the blacklist sub-libraries are also large; when the target comparison is performed according to the target comparison strategy of sequencing the similarity of each target model in the union set of the blacklist sub-libraries related to the control plan, when the blacklist sub-libraries related to different control plans overlap, the target models in the overlapped blacklist sub-libraries need to be repeatedly sequenced.
Therefore, the workload of performing target comparison under different target comparison strategies is related to the number of the deployment and control plans of the monitoring terminal, the number of blacklist sub-libraries related to the deployment and control plans, and the number of target models in the blacklist sub-libraries.
Accordingly, the target alignment apparatus may determine the target alignment strategy for the row target alignment based on the number of deployment plans (referred to herein as the third number), the first number, and the second number.
And step S130, carrying out target comparison based on the determined target comparison strategy.
In the embodiment of the present application, when the target comparison device determines the target comparison policy, the target comparison may be performed based on the target comparison policy.
It can be seen that, in the method flow shown in fig. 1, a target comparison strategy is determined by determining the number of deployment plans, the number of blacklist sub-libraries to be compared, and the number of target models in the blacklist sub-libraries to be compared, and based on the number of deployment plans, the number of blacklist sub-libraries to be compared, and the number of target models in the blacklist sub-libraries to be compared, and further, based on the target comparison strategy, target comparison is performed, so that the rationality of the target comparison strategy in a scene where comparison of multiple different types of target models needs to be performed on the same target model is improved, and support is provided for highly-concurrent and highly-responsive alarm.
In an alternative embodiment, in step S120, determining the target comparison strategy based on the third number of the deployment plan, the first number, and the second number may include:
and determining a target comparison strategy based on the first quantity, the second quantity, the third quantity, the target model quantity threshold and the sorting frequency threshold.
Illustratively, the workload of performing target comparison under different target comparison strategies is related to the number of deployment plans, the number of blacklist sub-libraries associated with the deployment plans, and the number of target models in the blacklist sub-libraries, and the operational characteristics of different hardware (such as a GPU or a CPU) are different, that is, the performance of different types of data processing is different.
For example, in the case of a large data volume, the sorting Processing performance of a GPU (Graphics Processing Unit) is superior to that of a CPU (central Processing Unit). However, when the GPU is used for sorting, data exchange from a memory to a video memory is required, and when the environmental conditions allow, the CPU can improve the sorting processing performance by using the multi-thread concurrent sorting.
Therefore, a target model number threshold and a sorting frequency threshold may be set based on the hardware processing performance, and a target comparison policy may be determined based on the first number, the second number, the third number, the target model number threshold and the sorting frequency threshold.
In one example, determining the target alignment strategy based on the first number, the second number, the third number, the target model number threshold, and the ranking number threshold may include:
when the second quantity is greater than or equal to the threshold value of the quantity of the target models and the third quantity is greater than the first quantity, determining the target comparison strategy as a first target comparison strategy; and/or the first and/or second light sources,
and when the second quantity is smaller than the target model quantity threshold, the first quantity is smaller than the third quantity, and the first quantity is smaller than or equal to the sorting time threshold, determining that the target comparison strategy is a first target comparison strategy.
Illustratively, the first target alignment strategy comprises:
determining a first similarity between a target model to be compared and each target model in a blacklist library through a GPU;
based on the first similarity, sequencing the target models in the blacklist sub-libraries to be compared through the GPU, and determining the Top N in the blacklist sub-libraries to be compared;
respectively determining the Top N of each deployment and control plan through a CPU (Central processing Unit) based on the Top N in each blacklist sub-library to be compared;
and respectively outputting Top N of each control plan according to the control plans.
For example, when the total number of the target models in the blacklist sub-libraries to be compared (that is, the number of the target models in the union of the blacklist sub-libraries to be compared) is large and the number of the deployment plans is greater than the number of the blacklist sub-libraries to be compared, sorting the sub-libraries (that is, sorting the similarity of the target models in each blacklist sub-library respectively, and then sorting the plurality of blacklist sub-libraries associated with the deployment plans), and the processing efficiency of sorting through the GPU is higher.
In addition, since the GPU performs the comparison of the target models, and the performance of determining the similarity between the target model to be compared and the target model in the blacklist library obviously prioritizes the processing performance of the CPU, the comparison of the target models can be performed by the GPU.
The processing performance of the GPU for determining the union of the blacklist sub-libraries to be compared is low, after the GPU determines the union of the blacklist sub-libraries to be compared, the processing performance of comparing the target model to be compared with each target model in the union of the blacklist sub-libraries is lower than the processing performance of comparing the target model to be compared with all target models in the blacklist sub-libraries by the GPU, and therefore when the GPU compares the target models, the target model to be compared can be compared with all target models in the blacklist sub-libraries.
Correspondingly, when the second number is greater than or equal to the threshold of the number of target models, and the third number is greater than the first number, the similarity (referred to as a first similarity herein) between the target model to be compared and each target model in the blacklist library may be determined by the GPU, and the target models in each blacklist library to be compared are sorted by the GPU based on the first similarity, so as to determine the Top N in each blacklist library to be compared.
It should be noted that, for any blacklist sub-library to be compared, if the number of target models in the blacklist sub-library to be compared, whose similarity to the target model to be compared is higher than the preset similarity threshold, is less than N (if M, M is less than N), the Top N corresponding to the blacklist sub-library is actually Top M.
After the Top N of each blacklist sub-library to be compared is determined by the GPU, the Top N of each control plan can be respectively determined by the CPU, and the Top N of each control plan is respectively output according to the control plan.
For example, assuming that the deployment plan1 associates the blacklist sub-base 1 and the blacklist sub-base 2, when the GPU obtains the Top N (assumed to be Top N1) of the blacklist sub-base 1 and the Top N (assumed to be Top N2) of the blacklist sub-base 2, the CPU may sort the Top N1+ Top N2 (therefore, there may be repeated target models, and therefore, the actual number is less than or equal to Top N1+ Top N2 and is greater than or equal to the greater value of the two) target models, and determine the Top N of the deployment plan 1.
Similarly, when the second quantity is smaller than the threshold of the quantity of the target models, the first quantity is smaller than the third quantity, and the first quantity is smaller than or equal to the threshold of the sorting times, the target comparison strategy is determined to be the first target comparison strategy.
In one example, determining the target alignment strategy based on the first number, the second number, the third number, the target model number threshold, and the ranking number threshold may include:
when the second quantity is greater than or equal to the target model quantity threshold value and the third quantity is less than or equal to the first quantity, determining that the target comparison strategy is a second target comparison strategy; and/or the first and/or second light sources,
and when the second quantity is smaller than the target model quantity threshold value, the third quantity is smaller than or equal to the first quantity, and the third quantity is smaller than or equal to the sorting frequency threshold value, determining that the target comparison strategy is a second target comparison strategy.
Exemplary, second target alignment strategies include:
determining a first similarity between a target model to be compared and each target model in a blacklist library through a GPU;
based on the first similarity, determining a centralized target model of a blacklist sub-library to be compared corresponding to each control plan through a GPU for sequencing to obtain Top N of each control plan;
and respectively outputting Top N of each control plan according to the control plans.
For example, when the total number of the target models in the blacklist sub-base to be compared (that is, the number of the target models in the union of the blacklist sub-bases to be compared) is large and the number of the deployment plans is less than or equal to the number of the blacklist sub-bases to be compared, sorting by the deployment plans (that is, sorting the similarity of each target model in the union of the plurality of blacklist sub-bases associated with the deployment plans) is considered, and the processing efficiency of sorting by the GPU is higher.
In addition, since the GPU performs the comparison of the target models, and the performance of determining the similarity between the target model to be compared and the target model in the blacklist library obviously prioritizes the processing performance of the CPU, the comparison of the target models can be performed by the GPU.
The processing performance of the GPU for determining the union of the blacklist sub-libraries to be compared is low, after the GPU determines the union of the blacklist sub-libraries to be compared, the processing performance of comparing the target model to be compared with each target model in the union of the blacklist sub-libraries is lower than the processing performance of comparing the target model to be compared with all target models in the blacklist sub-libraries by the GPU, and therefore when the GPU compares the target models, the target model to be compared can be compared with all target models in the blacklist sub-libraries.
Correspondingly, when the second quantity is greater than or equal to the threshold value of the quantity of the target models, and the third quantity is less than or equal to the first quantity, the similarity (namely the first similarity) between the target model to be compared and each target model in the blacklist library can be determined through the GPU, and the GPU sorts the collected blacklist sub-libraries associated with each deployment and control plan based on the first similarity, determines the Top N of each deployment and control plan, and outputs the Top N of each deployment and control plan respectively.
Similarly, when the second quantity is smaller than the threshold of the quantity of the target models, the third quantity is smaller than or equal to the first quantity, and the third quantity is smaller than or equal to the threshold of the sorting times, the target comparison strategy is determined to be the second target comparison strategy.
In one example, determining the target alignment strategy based on the first number, the second number, the third number, the target model number threshold, and the ranking number threshold may include:
and when the second quantity is smaller than the threshold value of the quantity of the target models, and the first quantity and the third quantity are both larger than the threshold value of the sorting times, determining that the target comparison strategy is a third target comparison strategy.
Illustratively, the third target alignment strategy comprises:
determining a first similarity between a target model to be compared and each target model in a blacklist library through a GPU;
based on the first similarity, sequencing the target models in each blacklist sub-library to be compared through the CPU, and determining the Top N in each blacklist sub-library to be compared
Respectively determining the Top N of each deployment and control plan through a CPU (Central processing Unit) based on the Top N in each blacklist sub-library to be compared;
and respectively outputting Top N of each control plan according to the control plans.
For example, considering that when the total number of target models in the blacklist sub-library to be compared is smaller, although the processing performance of the CPU sequencing is lower than that of the GPU, the performance of sequencing by the CPU is higher by comprehensively considering the sequencing processing performance and data exchange between the memory and the video memory.
In addition, when the number of the deployment and control plans and the number of the sub-libraries of the blacklist to be compared are both larger than the threshold value of the sequencing times, the number of the target models participating in the sequencing when the sub-libraries are sequenced is smaller than the number of the target models participating in the sequencing when the sub-libraries are sequenced, and the processing performance of the CPU for sequencing according to the sub-libraries is higher.
Correspondingly, when the second quantity is smaller than the threshold value of the quantity of the target models, and the first quantity and the third quantity are both larger than the threshold value of the sorting times, the first similarity between the target models to be compared and each target model in the blacklist library can be determined through the GPU, the target models in each blacklist library to be compared are sorted based on the first similarity by the CPU, the Top N in each blacklist library to be compared is determined, further, the Top N of each control plan is determined respectively through the CPU based on the Top N in each blacklist library to be compared, and the Top N of each control plan is output respectively according to the control plans.
In an optional embodiment, before step S110, the method may further include:
and when the target comparison equipment does not deploy the GPU, determining that the target comparison strategy is a fourth target comparison strategy.
Illustratively, the fourth target alignment strategy comprises:
determining a second similarity of each target model in a union set of the target model to be compared and the blacklist sub-base to be compared through a CPU (Central processing Unit);
based on the second similarity, sequencing the target models in the blacklist sub-libraries to be compared through the CPU, and determining the Top N in the blacklist sub-libraries to be compared;
respectively determining the Top N of each deployment and control plan through a CPU (Central processing Unit) based on the Top N in each blacklist sub-library to be compared;
and respectively outputting Top N of each control plan according to the control plans.
For example, when the GPU is not deployed in the target alignment device, the alignment and the sorting need to be performed by the CPU.
Due to the fact that the CPU has low processing performance under the condition of large data volume, and the CPU determines that the performance of the union set of the blacklist sub-libraries to be compared is high, when the CPU compares the target models, the CPU can compare the target models to be compared with the target models in the union set of the blacklist sub-libraries to be compared.
In addition, the number of the target models participating in the sorting when the sorting is performed according to the sub-library is smaller than the number of the target models participating in the sorting when the sorting is performed according to the deployment and control plan, so that the processing performance of the CPU performing the sorting according to the sub-library is higher.
Correspondingly, when the GPU is not deployed in the target comparison device, the CPU may determine similarity (referred to as second similarity herein) between the target model to be compared and each target model in the union of the blacklist sub-base to be compared, and the CPU ranks the target models in each blacklist sub-base to be compared based on the second similarity, determines Top N in each blacklist sub-base to be compared, further determines Top N of each deployment plan based on Top N in each blacklist sub-base to be compared, and outputs Top N of each deployment plan according to the deployment plan.
It should be appreciated that, in the embodiment of the present application, the target comparison policy is not limited to the first target comparison policy, the second target comparison policy, the third target comparison policy, and the fourth target comparison policy, and based on the target comparison policy determination scheme provided in the embodiment of the present application, the adjustment or modification made to the target comparison policy by a person skilled in the art without paying creative labor shall belong to the protection scope of the present application.
In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below with reference to specific examples.
In this embodiment, taking a target as a human face as an example, it is assumed that the deployment Plan for the monitoring front end a includes Plan1, Plan2, and Plan3, the blacklist sub-libraries associated with Plan1 are list1 and list2, the blacklist sub-libraries associated with Plan2 are list2 and list3, and the blacklist sub-libraries associated with Plan3 are list1 and list 4.
In this embodiment, the face comparison process for the face model a to be compared of the monitoring front end a is as follows:
1. and when the GPU is not deployed in the face comparison equipment, selecting a first face comparison strategy.
2. When a GPU is deployed, selection is made based on the following strategies:
and 2.1, counting the deployment plan number PN (namely the third number) aiming at the monitoring front end A, the number LN (namely the first number) of blacklist sub-libraries to be compared and the number of face models (namely the second number) in the blacklist sub-libraries to be compared.
In this embodiment, PN is 3 and LN is 4.
2.2, setting a face model quantity threshold MNT and a sorting time threshold SNT according to different hardware characteristics.
2.3, determining a face comparison strategy based on PN, LN, MN, MNT and SNT:
1) if MN is larger than or equal to MNT and PN is larger than LM, determining the face comparison strategy as a face comparison strategy one;
2) if MN is larger than or equal to MNT and PN is smaller than or equal to LM, determining the face comparison strategy as a face comparison strategy II;
3) if MN is less than MNT, LM is less than PN and LM is less than or equal to SNT, determining the face comparison strategy as a face comparison strategy one;
4) if MN is less than MNT, PN is less than or equal to LN and PN is less than or equal to SNT, determining the face comparison strategy as a second face comparison strategy;
5) and if MN < MNT and min (PN, LM) > SNT, determining the face comparison strategy to be a face comparison strategy III.
Illustratively, the face comparison strategies are as follows:
a first face comparison strategy: GPU comparison and GPU sorting according to a sub-library:
1. comparing all the face models in the black name list library through the GPU (namely comparing the face model A with all the face models in the black name list library respectively, and the same applies below);
2. acquiring a union set of blacklist sub-libraries related to 3 control plans: list1, list2, list3, and list _ 4;
3. sequencing each blacklist sub-library through the GPU respectively to obtain topN of each blacklist sub-library:
list1_ topN, list2_ topN, list3_ topN, and list _4_ topN
4. According to the blacklist sub-libraries related to each deployment and control plan, respectively picking out the alarm topN of the blacklist sub-libraries in each deployment and control plan, and sequencing for 2 times through a CPU:
plan1_topN＝{list1_topN+list2_topN}_topN；
plan2_topN＝{list2_topN+list3_topN}_topN；
plan3_topN＝{list1_topN+list4_topN}_topN；
5. and respectively outputting topN of the 3 control plans according to the control plans.
A second face comparison strategy: GPU comparison and GPU sorting according to a plan:
1. comparing all the face models in the blacklist library through the GPU;
2. respectively sequencing the blacklist sub-library union sets associated with the control plans through a GPU:
plan1_topN＝{list1+list2}_topN；
plan2_topN＝{list2+list3}_topN；
plan3_topN＝{list1+list4}_topN；
3. and respectively outputting topN of the 3 control plans according to the control plans.
A third face comparison strategy: GPU comparison and CPU sorting according to sub-libraries:
1. comparing all the face models in the blacklist library through the GPU;
2. acquiring a union set of blacklist sub-libraries related to 3 control plans: list1, list2, list3, and list _ 4;
3. sequencing each blacklist sub-library through a CPU respectively to obtain a topN of each blacklist sub-library:
list1_ topN, list2_ topN, list3_ topN, and list _4_ topN
4. According to the blacklist sub-libraries related to each deployment and control plan, respectively picking out the alarm topN of the blacklist sub-libraries in each deployment and control plan, and sequencing for 2 times through a CPU:
plan1_topN＝{list1_topN+list2_topN}_topN；
plan2_topN＝{list2_topN+list3_topN}_topN；
plan3_topN＝{list1_topN+list4_topN}_topN；
5. and respectively outputting topN of the 3 control plans according to the control plans.
Face comparison strategy four: CPU comparison and CPU sorting according to sub-libraries:
1. comparing all the face models in the union set of the blacklist sub-libraries related to the 3 control plans by the CPU (comparing the face model A with all the face models in the union set of the blacklist sub-libraries related to the 3 control plans respectively);
2. acquiring a union set of blacklist sub-libraries related to 3 control plans: list1, list2, list3, and list _ 4;
3. sequencing each blacklist sub-library through a CPU respectively to obtain a topN of each blacklist sub-library:
list1_ topN, list2_ topN, list3_ topN, and list _4_ topN
4. According to the blacklist sub-libraries related to each deployment and control plan, respectively picking out the alarm topN of the blacklist sub-libraries in each deployment and control plan, and sequencing for 2 times through a CPU:
plan1_topN＝{list1_topN+list2_topN}_topN；
plan2_topN＝{list2_topN+list3_topN}_topN；
plan3_topN＝{list1_topN+list4_topN}_topN；
5. and respectively outputting topN of the 3 control plans according to the control plans.
In the embodiment of the application, by acquiring the deployment and control plan, determining the first number of the blacklist sub-libraries to be compared and the second number of the target models in the blacklist sub-libraries to be compared based on the deployment and control plan, further determining the target comparison strategy based on the third number, the first number and the second number of the deployment and control plan, and performing target comparison based on the target comparison strategy, the rationality of the target comparison strategy in a scene that the comparison of multiple different types of target models needs to be performed on the same target model is improved, and support is provided for high-concurrency and high-response alarm.
The methods provided herein are described above. The following describes the apparatus provided in the present application:
referring to fig. 2, a schematic structural diagram of a target comparison apparatus according to an embodiment of the present disclosure is shown in fig. 2, where the target comparison apparatus may include:
an obtaining unit 210, configured to obtain a deployment and control plan;
a first determining unit 220, configured to determine, based on the deployment and control plan, a first number of blacklist sub-libraries to be compared and a second number of target models in the blacklist sub-libraries to be compared; the different blacklist sub-libraries store target models of different classifications in the blacklist library;
a second determining unit 230, configured to determine a target comparison strategy based on a third quantity of the deployment and control plan, the first quantity, and the second quantity;
an alignment unit 240, configured to perform target alignment based on the target alignment policy.
In an optional embodiment, the second determining unit 230 determines the target alignment strategy based on the third number, the first number and the second number of the deployment plan, including:
and determining a target comparison strategy based on the first number, the second number, the third number, a target model number threshold and a sorting frequency threshold.
In an optional embodiment, the second determining unit 230 determines the target alignment strategy based on the first number, the second number, the third number, the target model number threshold and the sorting number threshold, including:
when the second number is greater than or equal to the target model number threshold value and the third number is greater than the first number, determining that the target comparison strategy is a first target comparison strategy; and/or the first and/or second light sources,
and when the second quantity is smaller than the target model quantity threshold value, the first quantity is smaller than the third quantity, and the first quantity is smaller than or equal to the sorting time threshold value, determining that the target comparison strategy is a first target comparison strategy.
In an optional embodiment, the second determining unit 230 determines the target alignment strategy based on the first number, the second number, the third number, the target model number threshold and the sorting number threshold, including:
when the second number is greater than or equal to the target model number threshold value and the third number is less than or equal to the first number, determining that the target comparison strategy is a second target comparison strategy; and/or the first and/or second light sources,
and when the second number is smaller than the target model number threshold, the third number is smaller than or equal to the first number, and the third number is smaller than or equal to the sorting time threshold, determining that the target comparison strategy is a second target comparison strategy.
In an optional embodiment, the second determining unit 230 determines the target alignment strategy based on the first number, the second number, the third number, the target model number threshold and the sorting number threshold, including:
and when the second quantity is smaller than the threshold value of the quantity of the target models, and the first quantity and the third quantity are both larger than the threshold value of the sorting times, determining that the target comparison strategy is a third target comparison strategy.
In an optional embodiment, the first target alignment strategy includes:
determining a first similarity between a target model to be compared and each target model in a blacklist library through an image processing unit (GPU);
based on the first similarity, sequencing the target models in each blacklist sub-library to be compared through the GPU, and determining Top N in each blacklist sub-library to be compared;
respectively determining the Top N of each deployment and control plan through a Central Processing Unit (CPU) based on the Top N in each blacklist sub-library to be compared;
and respectively outputting Top N of each control plan according to the control plans.
In an optional embodiment, the second target alignment strategy includes:
determining a first similarity between a target model to be compared and each target model in a blacklist library through an image processing unit (GPU);
based on the first similarity, determining a centralized target model of a blacklist sub-library to be compared corresponding to each control plan through the GPU for sequencing to obtain Top N of each control calculation;
and respectively outputting Top N of each control plan according to the control plans.
In an optional embodiment, the third target alignment strategy includes:
determining a first similarity between a target model to be compared and each target model in a blacklist library through an image processing unit (GPU);
based on the first similarity, sequencing the target models in each blacklist sub-library to be compared through a Central Processing Unit (CPU), and determining Top N in each blacklist sub-library to be compared
Respectively determining the Top N of each deployment and control plan through the CPU based on the Top N in each blacklist sub-library to be compared;
and respectively outputting Top N of each control plan according to the control plans.
In an optional embodiment, before determining, based on the deployment plan, the first number of the blacklist sub-libraries to be compared and the second number of the target models in the blacklist sub-libraries to be compared, the first determining unit 220 further includes:
and when the target comparison equipment is not deployed with the GPU, determining that the target comparison strategy is a fourth target comparison strategy.
In an optional embodiment, the fourth target alignment strategy includes:
determining a second similarity of each target model in a union set of the target model to be compared and the blacklist sub-base to be compared through a CPU (Central processing Unit);
based on the second similarity, sequencing the target models in each blacklist sub-library to be compared through the CPU, and determining the Top N in each blacklist sub-library to be compared;
respectively determining the Top N of each deployment and control plan through the CPU based on the Top N in each blacklist sub-library to be compared;
and respectively outputting Top N of each control plan according to the control plans.
Fig. 3 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present disclosure. The electronic device may include a processor 301, a communication interface 302, a memory 303, and a communication bus 304. The processor 301, the communication interface 302 and the memory 303 communicate with each other via a communication bus 304. Wherein, the memory 303 stores a computer program; the processor 301 may execute the target matching method described above by executing the program stored in the memory 303.
The memory 303 referred to herein may be any electronic, magnetic, optical, or other physical storage device that may contain or store information such as executable instructions, data, and the like. For example, the memory 302 may be: a RAM (random Access Memory), a volatile Memory, a non-volatile Memory, a flash Memory, a storage drive (e.g., a hard drive), a solid state drive, any type of storage disk (e.g., an optical disk, a dvd, etc.), or similar storage medium, or a combination thereof.
The present embodiment also provides a machine-readable storage medium, such as the memory 303 in fig. 3, storing a computer program, which can be executed by the processor 301 in the electronic device shown in fig. 3 to implement the target matching method described above.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims (13)
1. A target comparison method is applied to target comparison equipment, and is characterized by comprising the following steps:
acquiring a control plan;
determining a first number of blacklist sub-libraries to be compared and a second number of target models in the blacklist sub-libraries to be compared based on the control plan; the different blacklist sub-libraries store target models of different classifications in the blacklist library;
determining a target comparison strategy based on the third quantity, the first quantity and the second quantity of the control plan;
and performing target comparison based on the target comparison strategy.
2. The method of claim 1, wherein determining a target alignment strategy based on the third number, the first number, and the second number of the deployment plan comprises:
and determining a target comparison strategy based on the first number, the second number, the third number, a target model number threshold and a sorting frequency threshold.
3. The method of claim 2, wherein determining a target alignment strategy based on the first number, the second number, the third number, a target model number threshold, and a ranking number threshold comprises:
when the second number is greater than or equal to the target model number threshold value and the third number is greater than the first number, determining that the target comparison strategy is a first target comparison strategy; and/or the first and/or second light sources,
and when the second quantity is smaller than the target model quantity threshold value, the first quantity is smaller than the third quantity, and the first quantity is smaller than or equal to the sorting time threshold value, determining that the target comparison strategy is a first target comparison strategy.
4. The method of claim 2, wherein determining a target alignment strategy based on the first number, the second number, the third number, a target model number threshold, and a ranking number threshold comprises:
when the second number is greater than or equal to the target model number threshold value and the third number is less than or equal to the first number, determining that the target comparison strategy is a second target comparison strategy; and/or the first and/or second light sources,
and when the second number is smaller than the target model number threshold, the third number is smaller than or equal to the first number, and the third number is smaller than or equal to the sorting time threshold, determining that the target comparison strategy is a second target comparison strategy.
5. The method of claim 2, wherein determining a target alignment strategy based on the first number, the second number, the third number, a target model number threshold, and a ranking number threshold comprises:
and when the second quantity is smaller than the threshold value of the quantity of the target models, and the first quantity and the third quantity are both larger than the threshold value of the sorting times, determining that the target comparison strategy is a third target comparison strategy.
6. The method of claim 3, wherein the first target alignment strategy comprises:
determining a first similarity between a target model to be compared and each target model in a blacklist library through an image processing unit (GPU);
based on the first similarity, sequencing the target models in each blacklist sub-library to be compared through the GPU, and determining Top N in each blacklist sub-library to be compared;
respectively determining the Top N of each deployment and control plan through a Central Processing Unit (CPU) based on the Top N in each blacklist sub-library to be compared;
and respectively outputting Top N of each control plan according to the control plans.
7. The method of claim 4, wherein the second target alignment strategy comprises:
determining a first similarity between a target model to be compared and each target model in a blacklist library through an image processing unit (GPU);
based on the first similarity, determining a centralized target model of a blacklist sub-library to be compared corresponding to each control plan through the GPU for sequencing to obtain Top N of each control calculation;
and respectively outputting Top N of each control plan according to the control plans.
8. The method of claim 5, wherein the third target alignment strategy comprises:
determining a first similarity between a target model to be compared and each target model in a blacklist library through an image processing unit (GPU);
based on the first similarity, sequencing the target models in each blacklist sub-library to be compared through a Central Processing Unit (CPU), and determining Top N in each blacklist sub-library to be compared
Respectively determining the Top N of each deployment and control plan through the CPU based on the Top N in each blacklist sub-library to be compared;
and respectively outputting Top N of each control plan according to the control plans.
9. The method of any of claims 1-8, wherein prior to determining the first number of blacklist sub-pools to be compared and the second number of target models in the blacklist sub-pools to be compared based on the deployment plan, further comprising:
and when the target comparison equipment is not deployed with the GPU, determining that the target comparison strategy is a fourth target comparison strategy.
10. The method of claim 9, wherein the fourth target alignment strategy comprises:
determining a second similarity of each target model in a union set of the target model to be compared and the blacklist sub-base to be compared through a CPU (Central processing Unit);
based on the second similarity, sequencing the target models in each blacklist sub-library to be compared through the CPU, and determining the Top N in each blacklist sub-library to be compared;
respectively determining the Top N of each deployment and control plan through the CPU based on the Top N in each blacklist sub-library to be compared;
and respectively outputting Top N of each control plan according to the control plans.
11. A target comparison device is applied to target comparison equipment, and is characterized by comprising:
the acquisition unit is used for acquiring a deployment and control plan;
the first determining unit is used for determining a first number of the blacklist sub-libraries to be compared and a second number of the target models in the blacklist sub-libraries to be compared based on the deployment and control plan; the different blacklist sub-libraries store target models of different classifications in the blacklist library;
a second determining unit, configured to determine a target comparison strategy based on a third quantity, the first quantity, and the second quantity of the deployment and control plan;
and the comparison unit is used for comparing the targets based on the target comparison strategy.
12. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method of any one of claims 1 to 10 when executing a program stored in a memory.
13. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method of any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010128658.6A CN113326714B (en) | 2020-02-28 | 2020-02-28 | Target comparison method, target comparison device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010128658.6A CN113326714B (en) | 2020-02-28 | 2020-02-28 | Target comparison method, target comparison device, electronic equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113326714A true CN113326714A (en) | 2021-08-31 |
| CN113326714B CN113326714B (en) | 2024-03-22 |
Family
ID=77412712
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010128658.6A Active CN113326714B (en) | 2020-02-28 | 2020-02-28 | Target comparison method, target comparison device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113326714B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7624107B1 (en) * | 2007-08-09 | 2009-11-24 | Nvidia Corporation | Radix sort algorithm for graphics processing units |
| CN108108711A (en) * | 2017-12-29 | 2018-06-01 | 深圳云天励飞技术有限公司 | Face supervision method, electronic equipment and storage medium |
| CN108446692A (en) * | 2018-06-08 | 2018-08-24 | 南京擎华信息科技有限公司 | Face comparison method, device and system |
| CN109074164A (en) * | 2016-05-03 | 2018-12-21 | 微软技术许可有限责任公司 | Use the object in Eye Tracking Technique mark scene |
| CN109165554A (en) * | 2018-07-24 | 2019-01-08 | 高新兴科技集团股份有限公司 | A kind of face characteristic comparison method based on cuda technology |
| CN109324909A (en) * | 2018-08-01 | 2019-02-12 | 武汉普利商用机器有限公司 | A kind of face alignment service system and method based on cloud service |
| CN109753848A (en) * | 2017-11-03 | 2019-05-14 | 杭州海康威视数字技术股份有限公司 | Execute the methods, devices and systems of face identifying processing |
| CN109886859A (en) * | 2019-01-30 | 2019-06-14 | 上海赜睿信息科技有限公司 | Data processing method, system, electronic equipment and computer readable storage medium |
| CN109993020A (en) * | 2017-12-29 | 2019-07-09 | 浙江宇视科技有限公司 | Face is deployed to ensure effective monitoring and control of illegal activities alarm method and device |
| CN109992687A (en) * | 2019-04-10 | 2019-07-09 | 北京迈格威科技有限公司 | Searching method, device, electronic equipment and the readable storage medium storing program for executing of human face data |
-
2020
- 2020-02-28 CN CN202010128658.6A patent/CN113326714B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7624107B1 (en) * | 2007-08-09 | 2009-11-24 | Nvidia Corporation | Radix sort algorithm for graphics processing units |
| CN109074164A (en) * | 2016-05-03 | 2018-12-21 | 微软技术许可有限责任公司 | Use the object in Eye Tracking Technique mark scene |
| CN109753848A (en) * | 2017-11-03 | 2019-05-14 | 杭州海康威视数字技术股份有限公司 | Execute the methods, devices and systems of face identifying processing |
| CN108108711A (en) * | 2017-12-29 | 2018-06-01 | 深圳云天励飞技术有限公司 | Face supervision method, electronic equipment and storage medium |
| CN109993020A (en) * | 2017-12-29 | 2019-07-09 | 浙江宇视科技有限公司 | Face is deployed to ensure effective monitoring and control of illegal activities alarm method and device |
| CN108446692A (en) * | 2018-06-08 | 2018-08-24 | 南京擎华信息科技有限公司 | Face comparison method, device and system |
| CN109165554A (en) * | 2018-07-24 | 2019-01-08 | 高新兴科技集团股份有限公司 | A kind of face characteristic comparison method based on cuda technology |
| CN109324909A (en) * | 2018-08-01 | 2019-02-12 | 武汉普利商用机器有限公司 | A kind of face alignment service system and method based on cloud service |
| CN109886859A (en) * | 2019-01-30 | 2019-06-14 | 上海赜睿信息科技有限公司 | Data processing method, system, electronic equipment and computer readable storage medium |
| CN109992687A (en) * | 2019-04-10 | 2019-07-09 | 北京迈格威科技有限公司 | Searching method, device, electronic equipment and the readable storage medium storing program for executing of human face data |
Non-Patent Citations (3)
| Title |
|---|
| YC WANG ET AL.: "《Energy-aware real-time face recognition system on mobile CPU-GPU platform》", 《SPRINGER》, 31 December 2012 (2012-12-31) * |
| 彭江锋: "《基于CPU+GPU异构平台的字符串匹配算法研究与实现》", 《中国优秀硕士学位论文全文数据库信息科技辑》, pages 5 * |
| 陈曦: "《面向CPU-GPU异构平台的生物多序列比对 算法的性能优化》", 《中国优秀硕士学位论文全文数据库信息科技辑》, 30 April 2019 (2019-04-30) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113326714B (en) | 2024-03-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20120155759A1 (en) | Establishing clusters of user preferences for image enhancement | |
| US10353954B2 (en) | Information processing apparatus, method of controlling the same, and storage medium | |
| US8417047B2 (en) | Noise suppression in low light images | |
| US20020183984A1 (en) | Modular intelligent multimedia analysis system | |
| US7187787B2 (en) | Method and apparatus for facial identification enhancement | |
| JP2007513555A (en) | Image processing | |
| JP6889132B2 (en) | How to locate one or more candidate digital images that are likely candidates for depicting an object | |
| CN109800329B (en) | Monitoring method and device | |
| Brust et al. | Active and incremental learning with weak supervision | |
| CN112926461B (en) | Neural network training and driving control method and device | |
| US20190278804A1 (en) | Information processing apparatus, method of controlling the same, and storage medium | |
| US20230060211A1 (en) | System and Method for Tracking Moving Objects by Video Data | |
| CN109146923B (en) | Processing method and system for target tracking broken frame | |
| CN113762519A (en) | Data cleaning method, device and equipment | |
| CN114708304B (en) | Cross-camera multi-target tracking method, device, equipment and medium | |
| CN114612774A (en) | Target detection and model construction method thereof, electronic device and storage medium | |
| CN111444816A (en) | Multi-scale dense pedestrian detection method based on fast RCNN | |
| JP5192437B2 (en) | Object region detection apparatus, object region detection method, and object region detection program | |
| CN112507981B (en) | Model generation method, iris image quality evaluation method and electronic equipment | |
| CN112884866B (en) | Coloring method, device, equipment and storage medium for black-and-white video | |
| CN113326714A (en) | Target comparison method and device, electronic equipment and readable storage medium | |
| CN110543887A (en) | Target analysis method and device, electronic equipment and readable storage medium | |
| CN113177917B (en) | Method, system, equipment and medium for optimizing snap shot image | |
| CN109815369B (en) | Filing method and device | |
| CN114511897A (en) | Identity recognition method, system, storage medium and server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |