CN111523445B - Examination behavior detection method based on improved Openpost model and facial micro-expression - Google Patents
Examination behavior detection method based on improved Openpost model and facial micro-expression Download PDFInfo
- Publication number
- CN111523445B CN111523445B CN202010317535.7A CN202010317535A CN111523445B CN 111523445 B CN111523445 B CN 111523445B CN 202010317535 A CN202010317535 A CN 202010317535A CN 111523445 B CN111523445 B CN 111523445B
- Authority
- CN
- China
- Prior art keywords
- abnormal
- state
- examination
- behavior
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 77
- 230000001815 facial effect Effects 0.000 title claims abstract description 23
- 230000006399 behavior Effects 0.000 claims abstract description 80
- 230000002159 abnormal effect Effects 0.000 claims abstract description 55
- 238000000034 method Methods 0.000 claims abstract description 42
- 238000012360 testing method Methods 0.000 claims abstract description 39
- 206010000117 Abnormal behaviour Diseases 0.000 claims abstract description 27
- 230000014509 gene expression Effects 0.000 claims abstract description 25
- 238000013138 pruning Methods 0.000 claims abstract description 6
- 230000005856 abnormality Effects 0.000 claims description 25
- 238000004458 analytical method Methods 0.000 claims description 21
- 230000008569 process Effects 0.000 claims description 17
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 12
- 238000012545 processing Methods 0.000 claims description 8
- 210000000707 wrist Anatomy 0.000 claims description 7
- 230000000694 effects Effects 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 6
- 238000012549 training Methods 0.000 claims description 6
- 230000009471 action Effects 0.000 claims description 5
- 230000009286 beneficial effect Effects 0.000 claims description 4
- 238000010195 expression analysis Methods 0.000 claims description 4
- 210000003128 head Anatomy 0.000 claims description 4
- 230000009467 reduction Effects 0.000 claims description 4
- 230000003542 behavioural effect Effects 0.000 claims description 3
- 238000010224 classification analysis Methods 0.000 claims description 3
- 238000011217 control strategy Methods 0.000 claims description 3
- 238000003745 diagnosis Methods 0.000 claims description 3
- 210000005069 ears Anatomy 0.000 claims description 3
- 238000009963 fulling Methods 0.000 claims description 3
- 210000004247 hand Anatomy 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 230000037361 pathway Effects 0.000 claims 1
- 210000000988 bone and bone Anatomy 0.000 abstract description 3
- 230000008859 change Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 14
- 238000007405 data analysis Methods 0.000 description 10
- 238000011160 research Methods 0.000 description 8
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000013473 artificial intelligence Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 101000622430 Homo sapiens Vang-like protein 2 Proteins 0.000 description 2
- 208000004350 Strabismus Diseases 0.000 description 2
- 102100023520 Vang-like protein 2 Human genes 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 208000019901 Anxiety disease Diseases 0.000 description 1
- 230000036506 anxiety Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 210000000887 face Anatomy 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000013095 identification testing Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000004898 kneading Methods 0.000 description 1
- 238000004451 qualitative analysis Methods 0.000 description 1
- 238000011897 real-time detection Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/20—Education
- G06Q50/205—Education administration or guidance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Educational Administration (AREA)
- Strategic Management (AREA)
- Life Sciences & Earth Sciences (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Tourism & Hospitality (AREA)
- Economics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Marketing (AREA)
- Evolutionary Biology (AREA)
- Biomedical Technology (AREA)
- Educational Technology (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- General Business, Economics & Management (AREA)
- Software Systems (AREA)
- Development Economics (AREA)
- Computing Systems (AREA)
- Bioinformatics & Computational Biology (AREA)
- Molecular Biology (AREA)
- Entrepreneurship & Innovation (AREA)
- Biophysics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Psychiatry (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Game Theory and Decision Science (AREA)
- Oral & Maxillofacial Surgery (AREA)
Abstract
The invention discloses an examination behavior detection method based on an improved Openpost model and facial micro-expressions. The facial information and the upper body bone information are identified through the artificial intelligent model, whether key points can be identified and the distance between the key points are taken as main judging conditions, and the change of the micro expression is taken as auxiliary judging conditions. If a student does not meet the condition for a period of time, judging that the student has abnormal examination behaviors. In addition, a stage in which abnormal behaviors of students are likely to occur is found out through a video stream of a lesson, and the stage is analyzed, so that innovation and innovation of teaching are realized. The invention reduces interference factors and simplifies equipment by machine vision identification, and further optimizes a network model by methods such as residual error network, weight pruning and the like. Compared with the traditional mode, the self-help examination behavior detection and feedback are realized, the test efficiency is high, the accuracy can reach 95%, and the self-help examination behavior detection method can be applied to general examination detection.
Description
Technical Field
The invention relates to the technical field of deep learning, machine vision and image processing, in particular to an examination behavior detection method based on an improved Openpost model and facial micro-expressions.
Background
In recent years, artificial intelligence techniques represented by deep learning have penetrated aspects of life of people and have entered a wide range of application stages. In 2017, the national institute proposed in the "new generation artificial intelligence development planning" issued: the intelligent technology is utilized to accelerate the promotion of talent culture mode and innovation of teaching methods, and a novel education system comprising intelligent learning and interactive learning is constructed.
For a long time, student exams have been the focus of educational research. In student examination activities, students are taken as the main body of the activities, and the behavior states of the students are the direct manifestation of examination conditions. Therefore, analysis of student examination behaviors is an important link of analysis and an important factor affecting student learning and teacher teaching efficiency. At present, education has been in an intelligent era, and higher requirements are put on analysis of examination behaviors of students, so that examination monitoring is becoming more important. Through the academic history combing of related research on examination behaviors at home and abroad and the dynamic learning of related research on literature of mainstream databases at home and abroad, little study is performed on examination behaviors of students in the field of higher education at home and abroad. The basic examination behavior research mainly adopts the research modes of subjective report, qualitative analysis and the like, and the existing research is not suitable for the study of the examination behaviors of students in the complicated higher education field. Under the requirements of accuracy, applicability and anti-interference performance of analysis on student examination behaviors and timely feedback of student information, no method with good analysis experience exists at present. Under the requirements of higher and higher project testing precision in detection and the requirements of unmanned testing, the existing examination behavior methods (Fan Zijian, xu Jing, liu Wei) do not have good testing experience, and the testing efficiency, accuracy and automation degree still need to be improved. The existing analysis method based on machine vision mainly utilizes a convolutional neural network to perform image recognition, however, the conventional convolutional neural network is not satisfactory in recognition accuracy and speed of student behaviors in the application of examination behavior recognition, and in the process of examination behavior recognition, it is very difficult to obtain a large number of training samples. Furthermore, the evaluation of the examination by the feedback of the personal history information of the students and the concentration of the examination is also an aspect yet to be developed.
The artificial intelligence is combined with education and teaching research, is improved based on an original Openpost model ([ 1] Cao Z., hidalgo G., simon T.et. OpenPose: real time Multi-Person 2D Pose Estimation using Part Affinity Fields[J ]. 2018.), and is integrated with a trained microexpressive recognition network model to objectively and quantitatively analyze examination behaviors of students, so that the student training system is better used in the field of education.
In this context, universities in developed countries have established relatively sophisticated examination monitoring and analysis systems to conduct intensive research on examination activities. However, the examination behaviors of students are not easy to detect, so that the intelligent student examination behaviors are difficult to identify. Based on the analysis, the study is to combine the related study results to discuss the student examination behavior recognition based on deep learning so as to improve the accuracy of intelligent chemical examination behavior recognition, assist examination detection and try to be applied to college entrance examination.
Disclosure of Invention
The invention aims to provide an examination behavior detection method based on an improved Openpost model and facial micro-expressions, so as to solve the problems in the background art.
In order to achieve the above purpose, the present invention provides the following technical solutions: an examination behavior detection method based on an improved Openpost model and facial micro-expressions comprises the following steps:
s1, shooting an image of the upper half body of a tester through an equipment camera arranged at the front end of the tester, and continuously shooting a test image of the upper half body of the tester at a certain initial frame rate;
s2, identifying key points of each frame of picture of the test image through the established improved Openpost model and the facial micro-expression, numbering and connecting the key points; taking 18 key points selected by Openpost as references, in order to make recognition simpler and more convenient, selecting 12 key points of left and right eyes, left and right ears, nose, neck, left and right wrists, left and right elbows and left and right shoulders in the upper body, comparing different frames of photos, and defining abnormal expressions in the 12 points except main body connecting lines: when there is rubbing nose, tucking mouth and touching one or more actions, defining abnormal expression;
s3, detecting abnormal behaviors of the examination room, wherein the abnormal behaviors of the examination room are divided into 2 main states: normal and abnormal states; the abnormal state is specifically divided into: state 1: the method is mainly defined as that the hand information exceeds a specified area; state 2: mainly defined as hands being placed under a table; state 3: mainly defined as left and right tenses; state 4: the method is mainly defined as four sub-states of frequently lifting the head; taking a general situation as an example, initially, the system will compare and detect images every 5 seconds, if the numerical value of each connecting line does not reach the abnormal situation, the system will determine that the system is in a normal examination state, if the relation among key points is wrong, the distance between the connecting lines of the key points reaches a threshold value, or the system will increase the detection frequency to every 1 second when a specified micro expression is identified, and then the system will compare and detect the images;
s4, according to the identified key point connecting line image, when the position relation of the key point in a certain frame image or the key point connecting line distance reaches a preset threshold value, setting the frame as a starting frame, and then increasing the detection frequency; initially, the system will compare and detect the images every 5 seconds; if the designated key points are lost, the related numerical values reach a threshold value or abnormal expressions appear, the detection frequency is increased until the comparison detection is carried out every 1 second; when the position of the key point is lost for a long time or the connecting line distance of the key point exceeds a set threshold value, judging that the key point is in an abnormal state;
s5, when an abnormal state is detected, the system continuously analyzes each second of the last 20 seconds, and judges the abnormal information such as the micro expression as a state 1, a state 2, a state 3 or a state 4 according to the abnormal value of the connecting lines among the key points and the loss of the key points and the abnormal information such as the micro expression and the like;
s6, specific judgment standards of abnormal states are as follows: state 1: judging as a state 1 if the key point exceeds the specified space; state 2: judging as a state 2 when the left hand or right hand wrist key point information is lost; state 3: judging as a state 3 when the information of the key points of the single eye and the ear is missing for a long time; state 4: the state 4 is judged when the key point information of the left eye and the right eye exceeds a preset threshold value after being detected for a plurality of times in a short time;
s7, introducing an expert control strategy: analyzing the discipline condition of the student daily examination room, detecting and analyzing the whole examination process for multiple times, and carrying out regional abnormal condition probability statistics; the classroom is divided into a plurality of areas according to experience, and the following strategies are obtained: as shown in fig. 5, the frequency of each area identification increases proportionally; thereby achieving the following steps: optimizing the detection effect and reducing the operation times; analyzing the personal condition of each student, wherein the analysis comprises the introduction of past examination abnormal behavior statistics of the student; in the examination process, counting the abnormal times of each student, and increasing the detection frequency of the students with high abnormal behavior frequency in the past at the initial detection; in the detection process, the classmates higher than the set times are recorded, and the initial value of the subsequent detection frequency is improved for detection and identification;
s8, introducing a searching and optimizing identification strategy, traversing the states 1 to 4, and if a student frequently has a definite diagnosis of the abnormal behavior state 1, preferentially judging whether the student is in the state 1 in the next abnormal identification, thereby improving the analysis efficiency.
And S9, after the information of each student end is acquired, the system transmits the normal and abnormal condition information to the teacher end and gathers the information, and performs information classification analysis.
Preferably, the examination behavior detection system based on the improved Openpost model and the facial micro-expression comprises a student desk, a seat number, a student end display, an identification camera, a camera rod, a data analysis server, a classroom desk, a teacher end display, a teacher end data analysis server, a voice broadcasting system, an IC card reader and a display screen; the face recognition device comprises a camera shooting rod and a face recognition camera arranged at the top of the camera shooting rod, the behavior and micro expression analysis server comprises a behavior examinee end display screen and an analysis data server, and the teacher end console comprises a teacher end display screen and a teacher end processing platform. The personal identification test taker card, the voice device and the test taker end display screen are integrally arranged, the camera is installed on one side of the table, the camera and the behaviors are connected with the microexpressive analysis data server, the teacher end display screen and the teacher end console are installed on a platform desktop, and the behavior analysis data server is connected with the teacher display screen in a communication mode.
Preferably, the step S1 is preceded by identifying the identity of the tester by adopting the test taker card; if the successful number of the identity identification reaches the number of the examination person, normal test is carried out, the examination examinee list is input into the system, and if the successful number of the identity identification does not reach the number of the examination person, the test is stopped, and a teacher is prompted to carry out attendance.
Preferably, the step S4 further includes: if the behavior abnormality is identified to last 10 seconds, marking the behavior abnormality, and if the behavior abnormality is identified to fail to last 10 seconds, marking the behavior abnormality as potential behavior abnormality and continuing to test; if the behavior abnormality is identified next as failing for 20 seconds, the flag is canceled, and if the behavior abnormality is identified as continuing for 20 seconds, the flag is marked as the behavior abnormality.
Preferably, the behavior and micro-expression analysis data server will analyze the identified data according to the following descriptions:
(1) The number of abnormal behavior examinees in any time range;
(2) A list of behavioural anomaly testees;
(3) Any abnormal number of behavior in the examination room seating area;
(4) The number of occurrences of each abnormal behavior;
and synchronously output analysis reports, which can be checked on a teacher display screen.
Preferably, the model uses a residual network to extract bottom layer characteristics so as to improve the detection precision and training speed of the improved model; the network structure refers to the first 10 layers of Res-18 network to form 5 residual blocks; the formed residual block is a residual block added with a soft threshold value; soft thresholding is the basic step of many signal noise reduction algorithms, which can remove extraneous information from a sample; as shown in equation (1), is a soft thresholding equation, where x is the input value, y is the output value, and τ is the threshold.
Preferably, different thresholds are set for different images using a 1X1 convolutional layer. As can be seen from the formula (1), the soft threshold processing sets the absolute value smaller than the threshold value to zero, and reduces the absolute value larger than the threshold value towards zero, so that the threshold value is required to be determined according to the content of irrelevant information of each sample, and the unified value is not beneficial to removing the irrelevant information; the final output result of the residual block is shown in formula (2):
wherein Y represents the output quantity, X represents the input quantity, and X' represents X, COV after general convolution 1X1 Represents the convolution using a 1X1 convolution kernel, sig represents the normalization process,
representing soft thresholding.
Preferably, the model compresses the model in a weight pruning mode so as to reduce the operation amount, speed up the detection and improve the instantaneity under the condition of keeping the detection precision; according to the method, the contribution degree of a certain connection to a final output result is evaluated, and the connection with smaller contribution degree is deleted, so that network parameters and calculated amount can be reduced under the condition that detection accuracy is not affected as much as possible, the size of a model is reduced, and the detection speed is increased; the method for evaluating the contribution degree of a certain connection is to calculate the convolution kernel corresponding to the connection
The L2 norm of (2) is calculated as shown in the formula (3):
wherein the method comprises the steps of
Represents the contribution of the kth junction of the 1 st passage,/->
The convolution kernels corresponding to the connection are h and W which are the height and width of the convolution kernel parameter matrix respectively, n is the number of the convolution kernel parameters, and alpha ij And (5) the parameters corresponding to the ith row and the j columns of the convolution kernel parameter matrix.
Compared with the prior art, the invention has the beneficial effects that: in the invention, a camera is arranged in front of a desk to detect examination behaviors of students in real time. The facial information and the upper body bone information are identified through the artificial intelligent model, whether key points can be identified and the distance between the key points are taken as main judging conditions, and the change of the micro expression is taken as auxiliary judging conditions. If a student does not meet the condition for a period of time, judging that the student has abnormal examination behaviors. In addition, a stage in which abnormal behaviors of students are likely to occur is found out through a video stream of a lesson, and the stage is analyzed, so that innovation and innovation of teaching are realized. The invention also provides a corresponding data analysis and processing system, which reduces interference factors and simplifies equipment by machine vision identification. The invention further optimizes the network model by residual error network, weight pruning and other methods. Compared with the traditional mode, the self-help examination behavior detection and feedback are realized, the test efficiency is high, the accuracy can reach 95%, and the self-help examination behavior detection method can be applied to general examination detection.
Drawings
FIG. 1 is a flowchart of the OpenPose model process;
FIG. 2 is a block diagram of a residual block with addition of second order term fusion;
FIG. 3 is a schematic diagram of a test identification system according to an embodiment of the present invention;
FIG. 4 is a flow chart of test behavior test identification in accordance with the present invention;
FIG. 5 is a schematic diagram of classroom area division;
FIG. 6 is a schematic diagram of the classification of abnormal behavior of the present invention;
FIG. 7 is a model improvement of the invention based on openpost;
FIG. 8 is a frame diagram of an examination of the sitting posture of the present invention;
FIG. 9 is a frame diagram of an examination east-west look action detected by the present invention;
FIG. 10 is a schematic frame diagram of an examination low head and hand-held under-table action as detected by the present invention;
FIG. 11 is a frame diagram of an examination wrist information loss detected by the present invention;
FIG. 12 is a schematic diagram of a test frequently raised frame detected by the present invention;
FIG. 13 is a frame diagram of an examination strabismus expression detected by the present invention;
FIG. 14 is a schematic diagram of a test kneading nose expression according to the present invention;
fig. 15 is a diagram of real-time data information of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the description of the present invention, it should be noted that, unless explicitly specified and limited otherwise, the terms "mounted," "provided," "connected," and the like are to be construed broadly, and may be fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.
Referring to fig. 1-15, the present invention provides a technical solution: the invention provides the following technical scheme: an examination behavior detection method based on an improved Openpost model and facial micro-expressions comprises the following steps:
s1, shooting an image of the upper half body of a tester through an equipment camera arranged at the front end of the tester, and continuously shooting a test image of the upper half body of the tester at a certain initial frame rate;
s2, identifying key points of each frame of picture of the test image through the established improved Openpost model and the facial micro-expression, numbering and connecting the key points; taking 18 key points selected by Openpost as references, in order to make recognition simpler and more convenient, selecting 12 key points of left and right eyes, left and right ears, nose, neck, left and right wrists, left and right elbows and left and right shoulders in the upper body, comparing different frames of photos, and defining abnormal expressions in the 12 points except main body connecting lines: when there is rubbing nose, tucking mouth and touching one or more actions, defining abnormal expression;
s3, detecting abnormal behaviors of the examination room, wherein the abnormal behaviors of the examination room are divided into 2 main states: normal and abnormal states; the abnormal state is specifically divided into: state 1: the method is mainly defined as that the hand information exceeds a specified area and the state 2 is: mainly defined as hand under desk, state 3: mainly defined as left and right hope and state 4: the method is mainly defined as four sub-states of frequently lifting the head; taking a general situation as an example, initially, the system will compare and detect images every 5 seconds, if the numerical value of each connecting line does not reach the abnormal situation, the system will determine that the system is in a normal examination state, if the relation among key points is wrong, the distance between the connecting lines of the key points reaches a threshold value, or the system will increase the detection frequency to every 1 second when a specified micro expression is identified, and then the system will compare and detect the images;
s4, according to the identified key point connecting line image, when the position relation of the key point in a certain frame image or the key point connecting line distance reaches a preset threshold value, setting the frame as a starting frame, and then increasing the detection frequency; initially, the system will compare and detect the images every 5 seconds; if the designated key points are lost, the related numerical values reach a threshold value or abnormal expressions appear, the detection frequency is increased until the comparison detection is carried out every 1 second; when the position of the key point is lost for a long time or the connecting line distance of the key point exceeds a set threshold value, judging that the key point is in an abnormal state;
s5, when an abnormal state is detected, the system continuously analyzes each second of the last 20 seconds, and judges the abnormal information such as the micro expression as a state 1, a state 2, a state 3 or a state 4 according to the abnormal value of the connecting lines among the key points and the loss of the key points and the abnormal information such as the micro expression and the like;
s6, specific judgment standards of abnormal states are as follows: state 1: judging as a state 1 if the key point exceeds the specified space; state 2: judging as a state 2 when the left hand or right hand wrist key point information is lost; state 3: judging as a state 3 when the information of the key points of the single eye and the ear is missing for a long time; state 4: the state 4 is judged when the key point information of the left eye and the right eye exceeds a preset threshold value after being detected for a plurality of times in a short time;
s7, introducing an expert control strategy: analyzing the discipline condition of the student daily examination room, detecting and analyzing the whole examination process for multiple times, and carrying out regional abnormal condition probability statistics; the classroom is divided into a plurality of areas according to experience, and the following strategies are obtained: as shown in fig. 5, the frequency of each area identification increases proportionally; thereby achieving the following steps: optimizing the detection effect and reducing the operation times; analyzing the personal condition of each student, wherein the analysis comprises the introduction of past examination abnormal behavior statistics of the student; in the examination process, counting the abnormal times of each student, and increasing the detection frequency of the students with high abnormal behavior frequency in the past at the initial detection; in the detection process, the classmates higher than the set times are recorded, and the initial value of the subsequent detection frequency is improved for detection and identification;
s8, introducing a searching and optimizing identification strategy, traversing the states 1 to 4, and if a student frequently has a definite diagnosis of the abnormal behavior state 1, preferentially judging whether the student is in the state 1 in the next abnormal identification, thereby improving the analysis efficiency.
And S9, after the information of each student end is acquired, the system transmits the normal and abnormal condition information to the teacher end and gathers the information, and performs information classification analysis.
The invention discloses an examination behavior detection system based on an improved Openpost model and facial micro-expressions, which comprises a student desk 1, a seat number 2, a student end display 3, a face recognition camera 4, a camera rod 5, a data analysis server 6, a classroom desk 7, a teacher end display 8, a teacher end data analysis server 9, a voice broadcasting system 301, an IC card reader 303 and a display screen 304. The face recognition device comprises a camera rod 5 and a face recognition camera 4 arranged at the top of the camera rod 5, wherein the face recognition camera 4 is used for recognizing the face of a tester to determine the identity of the tester. The face recognition camera 4 is used for recognizing and testing the identity of a student, the face recognition camera 4 is used for shooting the upper body of a tester so as to recognize and judge, the student end display 3 can display whether the identity information of the tester is verified and feedback the behavior detection condition, and the data analysis server 6 processes the image shot by the face recognition camera 4 and feeds back the result to the student end display 3 and the teacher end data analysis server 9.
First, a student inserts a student IC card into the IC card reader 303, recognizes the identity of the student, searches a student record file stored in advance therein, and displays the student number, name, and status of the identity test on the display screen 304 to count the information of the examination student for the current examination.
In this embodiment, preferably, the camera rod 5 is mounted on the upper right side, and the face recognition camera 4 is mounted on the top of the camera rod 5, and preferably recognizes the upper body image of the tester.
After the identity registration, the student's hands are placed on a desk, the face faces against a blackboard, a face recognition camera 4 installed above the desk shoots the face of the student, and the face recognition camera is transmitted to a data analysis server 6 through radio waves, and the identity of the student in the seat is confirmed by using a face recognition system.
After the identity of the student is determined, the student is input into a data system, and the student is tested for a certain time (1 minute). The camera 4 starts to recognize. And transmitting the pictures to a data processing server for storage, and synchronously displaying the analysis results by the student side display 3.
The abnormal condition is transmitted to the teacher data analysis server 9, the teacher performs real-time supervision and regulation, and meanwhile, the data is uploaded to real-time information checking software so as to facilitate the teacher to regulate and control the examination condition more conveniently.
In the invention, the step S1 is preceded by adopting the test taker card to identify the identity of the tester; if the successful number of the identity identification reaches the number of the examination person, normal test is carried out, the examination examinee list is input into the system, and if the successful number of the identity identification does not reach the number of the examination person, the test is stopped, and a teacher is prompted to carry out attendance.
In the present invention, the step S4 further includes: if the behavior abnormality is identified to last 10 seconds, marking the behavior abnormality, and if the behavior abnormality is identified to fail to last 10 seconds, marking the behavior abnormality as potential behavior abnormality and continuing to test; if the behavior abnormality is identified next as failing for 20 seconds, the flag is canceled, and if the behavior abnormality is identified as continuing for 20 seconds, the flag is marked as the behavior abnormality.
In the invention, the behavior and micro expression analysis data server analyzes and processes the identified data according to the following descriptions:
(1) The number of abnormal behavior examinees in any time range;
(2) A list of behavioural anomaly testees;
(3) Any abnormal number of behavior in the examination room seating area;
(4) The number of occurrences of each abnormal behavior;
and synchronously output analysis reports, which can be checked on a teacher display screen.
In the invention, the model uses a residual error network to extract the bottom layer characteristics so as to improve the detection precision and training speed of the improved model; the network structure refers to the first 10 layers of Res-18 network to form 5 residual blocks; the formed residual block is a residual block added with a soft threshold value; soft thresholding is the basic step of many signal noise reduction algorithms, which can remove extraneous information from a sample; as shown in equation (1), is a soft thresholding equation, where x is the input value, y is the output value, and τ is the threshold.
In the present invention, different thresholds are set for different images using a 1X1 convolutional layer. As can be seen from the formula (1), the soft threshold processing sets the absolute value smaller than the threshold value to zero, and reduces the absolute value larger than the threshold value towards zero, so that the threshold value is required to be determined according to the content of irrelevant information of each sample, and the unified value is not beneficial to removing the irrelevant information; the final output result of the residual block is shown in formula (2):
wherein Y represents the output quantity, X represents the input quantity, and X' represents X, COV after general convolution 1X1 Represents the convolution using a 1X1 convolution kernel, sig represents the normalization process,
representing soft thresholding.
In the invention, the model is compressed in a weight pruning mode to reduce the operation amount under the condition of keeping the detection precision, accelerate the detection speed and improve the instantaneity; according to the method, the contribution degree of a certain connection to a final output result is evaluated, and the connection with smaller contribution degree is deleted, so that network parameters and calculated amount can be reduced under the condition that detection accuracy is not affected as much as possible, the size of a model is reduced, and the detection speed is increased; the method for evaluating the contribution degree of a certain connection is to calculate the convolution kernel corresponding to the connection
The L2 norm of (2) is calculated as shown in the formula (3):
wherein the method comprises the steps of
Represents the contribution of the kth junction of the 1 st passage,/->
The convolution kernels corresponding to the connection are h and w which are the height and width of the convolution kernel parameter matrix respectively, n is the number of the convolution kernel parameters, and alpha ij And (5) the parameters corresponding to the ith row and the j columns of the convolution kernel parameter matrix.
As shown in fig. 6, the original openpost model is improved, part of key point information is reserved, the original training model is modified and retrained, and the reduction of the identified key information undoubtedly increases the detection efficiency and accuracy. The method is more suitable for examination detection.
Fig. 7 is a frame diagram of a normal posture of an examination, and fig. 8 to 11 are respectively four state diagrams of abnormal behavior of an examination, and abnormal behavior of an examination is detected by an improved openpost model.
As shown in fig. 12 and 13, examination behavior abnormality is detected by the microexpressive assist judgment.
The convolutional neural network with multiple classifications is trained for expression recognition, and the construction of a model is mainly based on 2018 CVPR papers and Google's Going deep design as follows. Compared with the traditional method, the convolutional neural network has better performance, and the model is used for constructing an identification system and providing a GUI interface and real-time detection of a camera. The trained model has good robustness and high and accurate recognition efficiency. Therefore, a network model capable of accurately identifying strabismus and tension and anxiety expressions is trained. Meanwhile, the model is suitable for class behavior detection in a complex environment, and is used as an auxiliary condition and is jointly applied to recognition detection with an improved Openpost model and a facial micro-expression.
A convolutional network model is built to perform microexpression recognition, and the network model structure is shown in table 1.
In summary, in the invention, the camera is arranged in front of the desk to detect the examination behaviors of the students in real time. The facial information and the upper body bone information are identified through the artificial intelligent model, whether key points can be identified and the distance between the key points are taken as main judging conditions, and the change of the micro expression is taken as auxiliary judging conditions. If a student does not meet the condition for a period of time, judging that the student has abnormal examination behaviors. In addition, a stage in which abnormal behaviors of students are likely to occur is found out through a video stream of a lesson, and the stage is analyzed, so that innovation and innovation of teaching are realized. The invention also provides a corresponding data analysis and processing system, which reduces interference factors and simplifies equipment by machine vision identification. The invention further optimizes the network model by residual error network, weight pruning and other methods. Compared with the traditional mode, the self-help examination behavior detection and feedback are realized, the test efficiency is high, the accuracy can reach 95%, and the self-help examination behavior detection method can be applied to general examination detection.
The present invention is not described in detail in the present application, and is well known to those skilled in the art.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Claims (7)
1. An examination behavior detection method based on an improved openpost model and facial micro-expressions is characterized by comprising the following steps of: the method comprises the following steps:
s1, shooting an image of the upper half body of a tester through an equipment camera arranged at the front end of the tester, and continuously shooting a test image of the upper half body of the tester at a certain initial frame rate;
s2, identifying key points of each frame of picture of the test image through the established improved Openpost model and the facial micro-expression, numbering and connecting the key points; taking 18 key points selected by Openpost as references, in order to make recognition simpler and more convenient, selecting 12 key points of left and right eyes, left and right ears, nose, neck, left and right wrists, left and right elbows and left and right shoulders in the upper body, comparing different frames of photos, and defining abnormal expressions in the 12 points except main body connecting lines: when there is rubbing nose, tucking mouth and touching one or more actions, defining abnormal expression;
s3, detecting abnormal behaviors of the examination room, wherein the abnormal behaviors of the examination room are divided into 2 main states: normal and abnormal states; the abnormal state is specifically divided into: state 1: the method is mainly defined as that the hand information exceeds a specified area; state 2: mainly defined as hands being placed under a table; state 3: mainly defined as left and right tenses; state 4: the method is mainly defined as four sub-states of frequently lifting the head; taking a general situation as an example, initially, the system will compare and detect images every 5 seconds, if the numerical value of each connecting line does not reach the abnormal situation, the system will determine that the system is in a normal examination state, if the relation among key points is wrong, the distance between the connecting lines of the key points reaches a threshold value, or the system will increase the detection frequency to every 1 second when a specified micro expression is identified, and then the system will compare and detect the images;
s4, according to the identified key point connecting line image, when the position relation of the key point in a certain frame image or the key point connecting line distance reaches a preset threshold value, setting the frame as a starting frame, and then increasing the detection frequency; initially, the system will compare and detect the images every 5 seconds; if the designated key points are lost, the related numerical values reach a threshold value or abnormal expressions appear, the detection frequency is increased until the comparison detection is carried out every 1 second; when the position of the key point is lost for a long time or the connecting line distance of the key point exceeds a set threshold value, judging that the key point is in an abnormal state;
s5, when an abnormal state is detected, the system continuously analyzes each second of the last 20 seconds, and judges the abnormal information such as the micro expression as a state 1, a state 2, a state 3 or a state 4 according to the abnormal value of the connecting lines among the key points and the loss of the key points and the abnormal information such as the micro expression and the like;
s6, specific judgment standards of abnormal states are as follows: state 1: judging as a state 1 if the key point exceeds the specified space; state 2: judging as a state 2 when the left hand or right hand wrist key point information is lost; state 3: judging as a state 3 when the information of the key points of the single eye and the ear is missing for a long time; state 4: the state 4 is judged when the key point information of the left eye and the right eye exceeds a preset threshold value after being detected for a plurality of times in a short time;
s7, introducing an expert control strategy: analyzing the discipline condition of the student daily examination room, detecting and analyzing the whole examination process for multiple times, and carrying out regional abnormal condition probability statistics; the classroom is divided into a plurality of areas according to experience, and the following strategies are obtained: the identification frequency of each region is increased proportionally; thereby achieving the following steps: optimizing the detection effect and reducing the operation times; analyzing the personal condition of each student, wherein the analysis comprises the introduction of past examination abnormal behavior statistics of the student; in the examination process, counting the abnormal times of each student, and increasing the detection frequency of the students with high abnormal behavior frequency in the past at the initial detection; in the detection process, the classmates higher than the set times are recorded, and the initial value of the subsequent detection frequency is improved for detection and identification;
s8, introducing a searching and optimizing identification strategy, traversing the states 1 to 4, and if a student frequently has a definite diagnosis of the abnormal behavior state 1, preferentially judging whether the student is in the state 1 in the next abnormal identification, thereby improving the analysis efficiency;
and S9, after the information of each student end is acquired, the system transmits the normal and abnormal condition information to the teacher end and gathers the information, and performs information classification analysis.
2. The improved openpost model and facial microexpressive test behavior detection method according to claim 1, wherein: the step S1 is preceded by adopting the test taker card to identify the identity of the tester; if the successful number of the identity identification reaches the number of the examination person, normal test is carried out, the examination examinee list is input into the system, and if the successful number of the identity identification does not reach the number of the examination person, the test is stopped, and a teacher is prompted to carry out attendance.
3. The improved openpost model and facial microexpressive test behavior detection method according to claim 1, wherein: the step S4 further includes: if the behavior abnormality is identified to last 10 seconds, marking the behavior abnormality, and if the behavior abnormality is identified to fail to last 10 seconds, marking the behavior abnormality as potential behavior abnormality and continuing to test; if the behavior abnormality is identified next as failing for 20 seconds, the flag is canceled, and if the behavior abnormality is identified as continuing for 20 seconds, the flag is marked as the behavior abnormality.
4. The improved openpost model and facial microexpressive test behavior detection method according to claim 1, wherein: the behavior and micro expression analysis data server analyzes and processes the identified data according to the following points:
(1) The number of abnormal behavior examinees in any time range;
(2) A list of behavioural anomaly testees;
(3) Any abnormal number of behavior in the examination room seating area;
(4) The number of occurrences of each abnormal behavior;
and synchronously output analysis reports, which can be checked on a teacher display screen.
5. The improved openpost model and facial microexpressive test behavior detection method according to claim 1, wherein: the model uses a residual error network to extract bottom layer characteristics so as to improve the detection precision and training speed of the improved model; the residual error network refers to the first 10 layers of the Res-18 network, and 5 residual error blocks are formed; the formed residual block is a residual block added with a soft threshold value; soft thresholding is the basic step of many signal noise reduction algorithms, which can remove extraneous information from a sample; as shown in formula (1), is a soft thresholding formula, where x is the input value, y is the output value, τ is the threshold;
6. the improved openpost model and facial microexpressive test performance detection method according to claim 5, wherein: the 1X1 convolution layer is used for setting different thresholds for different images, as can be known from the formula (1), the soft threshold processing sets the value with the absolute value smaller than the threshold value to zero, and reduces the value with the absolute value larger than the threshold value towards the zero direction, so that the threshold value is required to be determined according to the content of irrelevant information of each sample, and the unified value is not beneficial to removing the irrelevant information; the final output result of the residual block is shown in formula (2):
wherein Y represents the output quantity, X represents the input quantity, and X' represents X, COV after general convolution 1X1 Represents the convolution using a 1X1 convolution kernel, sig represents the normalization process,
representing soft thresholding.
7. The improved openpost model and facial microexpressive test behavior detection method according to claim 1, wherein: the model is compressed in a weight pruning mode, so that the operation amount is reduced under the condition of keeping the detection precision, the detection speed is increased, and the instantaneity is improved; according to the method, the contribution degree of a certain connection to a final output result is evaluated, and the connection with smaller contribution degree is deleted, so that network parameters and calculated amount can be reduced under the condition that detection accuracy is not affected as much as possible, the size of a model is reduced, and the detection speed is increased; the method for evaluating the contribution degree of a certain connection is to calculate the convolution kernel corresponding to the connection
The L2 norm of (2) is calculated as shown in the formula (3): />
Wherein the method comprises the steps of
Represents the contribution of the kth junction of the first pathway,/->
The convolution kernels corresponding to the connection are h and w which are the height and width of the convolution kernel parameter matrix respectively, n is the number of the convolution kernel parameters, and alpha ij And (5) the parameters corresponding to the ith row and the j columns of the convolution kernel parameter matrix. />
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010317535.7A CN111523445B (en) | 2020-04-21 | 2020-04-21 | Examination behavior detection method based on improved Openpost model and facial micro-expression |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010317535.7A CN111523445B (en) | 2020-04-21 | 2020-04-21 | Examination behavior detection method based on improved Openpost model and facial micro-expression |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111523445A CN111523445A (en) | 2020-08-11 |
| CN111523445B true CN111523445B (en) | 2023-05-26 |
Family
ID=71903391
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010317535.7A Active CN111523445B (en) | 2020-04-21 | 2020-04-21 | Examination behavior detection method based on improved Openpost model and facial micro-expression |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111523445B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112633083A (en) * | 2020-12-04 | 2021-04-09 | 南通大学 | Method for detecting abnormal behaviors of multiple persons and wearing of mask based on improved Openpos examination |
| CN113052041B (en) * | 2021-03-17 | 2024-04-30 | 广东骏杰科技有限公司 | Intelligent teaching system and monitoring method |
| CN113537005B (en) * | 2021-07-02 | 2023-08-11 | 福州大学 | Online examination student behavior analysis method based on attitude estimation |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106920194A (en) * | 2017-03-07 | 2017-07-04 | 佛山市金蓝领教育科技有限公司 | A kind of anti-cheating remote test method |
| WO2018218286A1 (en) * | 2017-05-29 | 2018-12-06 | Saltor Pty Ltd | Method and system for abnormality detection |
| CN109740446A (en) * | 2018-12-14 | 2019-05-10 | 深圳壹账通智能科技有限公司 | Classroom students ' behavior analysis method and device |
-
2020
- 2020-04-21 CN CN202010317535.7A patent/CN111523445B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111523445A (en) | 2020-08-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111523445B (en) | Examination behavior detection method based on improved Openpost model and facial micro-expression | |
| CN109522815B (en) | Concentration degree evaluation method and device and electronic equipment | |
| CN111523444B (en) | Classroom behavior detection method based on improved Openpost model and facial micro-expression | |
| CN111242049B (en) | Face recognition-based student online class learning state evaluation method and system | |
| Hu et al. | Research on abnormal behavior detection of online examination based on image information | |
| WO2019028592A1 (en) | Teaching assistance method and teaching assistance system using said method | |
| CN110428908B (en) | Eyelid motion function evaluation system based on artificial intelligence | |
| CN111046823A (en) | Student classroom participation degree analysis system based on classroom video | |
| WO2021047185A1 (en) | Monitoring method and apparatus based on facial recognition, and storage medium and computer device | |
| CN109805944B (en) | Children's ability analytic system that shares feelings | |
| CN109685007B (en) | Eye habit early warning method, user equipment, storage medium and device | |
| CN115205764B (en) | Online learning concentration monitoring method, system and medium based on machine vision | |
| CN113837153A (en) | Real-time emotion recognition method and system integrating pupil data and facial expressions | |
| CN111814718A (en) | Attention detection method integrating multiple discrimination technologies | |
| CN110660454A (en) | Cancer pain real-time assessment instrument and assessment method thereof | |
| CN112085883A (en) | Intelligent invigilation anti-cheating system | |
| CN105631410B (en) | A kind of classroom detection method based on intelligent video processing technique | |
| Jiang | Analysis of Students’ Role Perceptions and their Tendencies in Classroom Education Based on Visual Inspection | |
| CN113269079B (en) | Child autism early evaluation device and system based on video layering comparison analysis and hierarchical behavior paradigm | |
| CN115312192A (en) | Campus mental health assessment-feedback method based on convolutional neural network expression recognition | |
| CN111950472A (en) | Teacher grinding evaluation method and system | |
| CN114255509A (en) | Student supervises appurtenance based on OpenPose | |
| CN113542668A (en) | Monitoring system and method based on 3D camera | |
| CN117671774B (en) | Face emotion intelligent recognition analysis equipment | |
| Santoso et al. | Development of an online exam security system using ensemble method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |