CN110766912B - Driving early warning method, device and computer readable storage medium - Google Patents
Driving early warning method, device and computer readable storage medium Download PDFInfo
- Publication number
- CN110766912B CN110766912B CN201810851573.3A CN201810851573A CN110766912B CN 110766912 B CN110766912 B CN 110766912B CN 201810851573 A CN201810851573 A CN 201810851573A CN 110766912 B CN110766912 B CN 110766912B
- Authority
- CN
- China
- Prior art keywords
- driver
- preset
- fatigue
- head
- position information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/02—Alarms for ensuring the safety of persons
- G08B21/06—Alarms for ensuring the safety of persons indicating a condition of sleep, e.g. anti-dozing alarms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
Landscapes
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Oral & Maxillofacial Surgery (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- Emergency Management (AREA)
- Traffic Control Systems (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The invention discloses a driving early warning method, which comprises the following steps: acquiring a facial image of a driver, and analyzing the facial image according to a preset face detection algorithm and a face characteristic point alignment algorithm to obtain position information of facial characteristic points of the driver; determining the head posture of the driver according to the position information, judging whether the head posture of the driver meets a preset abnormal condition or not, and/or identifying the fatigue degree of the driver according to the position information; and when the head posture of the driver meets a preset abnormal condition or the fatigue degree of the driver meets a preset early warning condition, triggering corresponding early warning reminding. The invention also discloses a driving early warning device and a computer readable storage medium. The invention can effectively detect and early warn the physiological abnormal condition of the driver, thereby ensuring the driving safety.
Description
Technical Field
The invention relates to the technical field of intelligent driving, in particular to a driving early warning method, driving early warning equipment and a computer readable storage medium.
Background
Along with the improvement of living standard of people, automobiles enter thousands of households, the driving safety of the automobiles becomes more and more the concern of people, the problems of fatigue driving, inattention and the like of drivers in the driving process of the automobiles become potential safety hazards in the driving process of the automobiles, and how to effectively detect and early warn the physiological abnormal conditions of the drivers of fatigue driving, inattention and the like becomes the problem to be solved urgently at present.
Disclosure of Invention
The invention mainly aims to provide a driving early warning method, driving early warning equipment and a computer readable storage medium, and aims to realize effective detection and early warning on physiological abnormal conditions of a driver so as to guarantee driving safety.
In order to achieve the above object, the present invention provides a driving warning method, including the steps of:
acquiring a facial image of a driver, and analyzing the facial image according to a preset face detection algorithm and a face characteristic point alignment algorithm to obtain position information of facial characteristic points of the driver;
determining the head posture of the driver according to the position information, judging whether the head posture of the driver meets a preset abnormal condition or not, and/or identifying the fatigue degree of the driver according to the position information;
and when the head posture of the driver meets a preset abnormal condition or the fatigue degree of the driver meets a preset early warning condition, triggering corresponding early warning reminding.
Preferably, the step of determining the head posture of the driver according to the position information and determining whether the head posture of the driver meets a preset abnormal condition includes:
acquiring position information of the facial feature points of the driver in a preset judgment period, and converting the position information into head deflection angle data of the driver according to a preset geometric projection algorithm;
identifying a head deviation state of the driver according to the head deviation angle data;
detecting whether the duration of single head deviation of the driver reaches a first preset duration or not according to the head deviation state in the judging period;
and if the duration of the single head deviation of the driver reaches a first preset duration, judging that the head posture of the driver meets a preset abnormal condition.
Preferably, the head deviation angle data includes a horizontal deviation angle and a vertical deviation angle, and the step of identifying the head deviation state of the driver from the head deviation angle data includes:
judging whether the horizontal deflection angle falls within a preset horizontal deflection angle interval or not and whether the vertical deflection angle falls within a preset vertical deflection angle interval or not;
and when the horizontal deflection angle does not fall within a preset horizontal deflection angle interval or the vertical deflection angle does not fall within a preset vertical deflection angle interval, judging that the driver is currently in a head deviation state.
Preferably, after the step of detecting whether there is a duration of a single head deviation of the driver for a first preset time period according to the head deviation state in the determination period, the method further includes:
if the duration of the single head deviation of the driver does not reach a first preset duration, counting the accumulated duration of the head deviation state of the driver in the judgment period;
judging whether the occupation ratio of the accumulated time length in the judging period reaches a preset value or not;
and if the occupation ratio of the accumulated time length in the judgment period reaches a preset value, judging that the head posture of the driver meets a preset abnormal condition.
Preferably, the step of capturing the face image of the driver includes:
in the process of collecting the face image of the driver, detecting whether the face image of the driver is not collected within a second preset time length or not;
and if the facial image of any driver is not acquired within a second preset time, judging that the head posture of the driver meets a preset abnormal condition.
Preferably, the step of identifying the fatigue level of the driver from the position information includes:
acquiring position information of the facial feature points of the driver in a preset identification period, and respectively calculating the aspect ratio of eyes and a mouth of the driver according to the position information;
correspondingly recognizing the opening and closing state and the yawning state of the eyes of the driver according to the aspect ratio of the eyes and the mouth;
calculating preset fatigue characteristic parameters according to the recognized eye opening and closing state and the recognized yawning state;
inputting the fatigue characteristic parameters obtained by calculation into a preset classifier so that the classifier outputs a response value corresponding to a preset fatigue level;
and comparing the response values corresponding to different fatigue grades, and selecting one fatigue grade from the preset fatigue grades according to the comparison result to serve as the identification result of the fatigue degree of the driver.
Preferably, the step of comparing the response values corresponding to different fatigue levels, and selecting one fatigue level from the preset fatigue levels according to the comparison result, as the identification result of the fatigue degree of the driver, includes:
calculating the difference value between the maximum value and the second maximum value in the response values, and judging whether the difference value is greater than or equal to a preset threshold value;
if the difference is larger than or equal to a preset threshold value, taking the fatigue grade corresponding to the maximum value as the recognition result of the fatigue degree of the driver;
if the difference value is smaller than a preset threshold value, counting the frequency of yawning of the driver in the identification period;
if the number of times of yawning of the driver is greater than or equal to the preset number of times, taking a higher fatigue grade in the fatigue grades corresponding to the maximum value and the second maximum value as the identification result of the fatigue degree of the driver;
and if the yawning times of the driver are less than the preset times, taking the lower fatigue grade in the fatigue grades corresponding to the maximum value and the second maximum value as the identification result of the fatigue degree of the driver.
Preferably, the fatigue degree of the driver includes wakefulness, light fatigue and deep fatigue, and when the fatigue degree of the driver satisfies a preset early warning condition, the step of triggering a corresponding early warning reminder includes:
when the fatigue degree of the driver is recognized to be light fatigue or deep fatigue, triggering corresponding early warning reminding, wherein the reminding frequency corresponding to the deep fatigue is higher than the reminding frequency corresponding to the light fatigue.
In addition, to achieve the above object, the present invention also provides a driving warning apparatus, including: the driving early warning system comprises a memory, a processor and a driving early warning program which is stored on the memory and can run on the processor, wherein the driving early warning program realizes the steps of the driving early warning method when being executed by the processor.
In addition, to achieve the above object, the present invention also provides a computer readable storage medium having a driving warning program stored thereon, which when executed by a processor implements the steps of the driving warning method as described above.
According to the driving early warning method provided by the invention, the position information of the facial feature points of the driver is obtained, the abnormal situation of the head posture of the driver is detected according to the position information, the fatigue degree of the driver is detected, and the corresponding abnormal reminding is triggered according to the position information and the fatigue degree, so that the effective detection and early warning of the physiological abnormal situations of the driver, such as inattention, fatigue and the like, are realized, and the driving safety is ensured.
Drawings
FIG. 1 is a schematic diagram of an apparatus architecture of a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a first embodiment of the driving warning method according to the present invention;
FIG. 3 is a detailed step diagram of the step in FIG. 2 of determining the head pose of the driver according to the position information and determining whether the head pose of the driver meets a preset abnormal condition;
FIG. 4 is a schematic diagram of collecting position information of facial feature points of a driver through a sliding data window according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of data collection by data flow in an embodiment of the present invention;
FIG. 6 is a detailed step diagram of the step of identifying the fatigue level of the driver according to the position information in FIG. 2.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The main solution of the embodiment of the invention is as follows: acquiring a facial image of a driver, and analyzing the facial image according to a preset face detection algorithm and a face characteristic point alignment algorithm to obtain position information of facial characteristic points of the driver; determining the head posture of the driver according to the position information, judging whether the head posture of the driver meets a preset abnormal condition or not, and/or identifying the fatigue degree of the driver according to the position information; and when the head posture of the driver meets a preset abnormal condition or the fatigue degree of the driver meets a preset early warning condition, triggering corresponding early warning reminding.
At present, the driving safety of automobiles is more and more a problem concerned by people, the problems of fatigue driving, inattention and the like of drivers in the driving process of automobiles are further potential safety hazards in the driving process of automobiles, and how to effectively detect and early warn the physiological abnormal conditions of the drivers such as fatigue driving, inattention and the like is a problem to be solved urgently at present.
According to the driving early warning method provided by the invention, the position information of the facial feature points of the driver is obtained, the abnormal situation of the head posture of the driver is detected according to the position information, the fatigue degree of the driver is detected, and the corresponding abnormal prompt is triggered according to the position information and the fatigue degree, so that the effective detection and early warning on the physiological abnormal situation of the driver are realized, and the driving safety is ensured.
As shown in fig. 1, fig. 1 is a schematic device structure diagram of a hardware operating environment according to an embodiment of the present invention.
The driving early warning device provided by the embodiment of the invention can be a vehicle-mounted terminal, and comprises a camera for collecting facial images of a driver.
As shown in fig. 1, the driving early warning apparatus may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a storage device separate from the processor 1001.
Those skilled in the art will appreciate that the configuration of the apparatus shown in fig. 1 is not intended to be limiting of the apparatus and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is a kind of computer storage medium, may include therein an operating system, a network communication module, a user interface module, and a driving warning program.
In the terminal shown in fig. 1, the network interface 1004 is mainly used for connecting to a backend server and performing data communication with the backend server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and the processor 1001 may be configured to call the driving warning program stored in the memory 1005, and perform the following operations:
acquiring a facial image of a driver, and analyzing the facial image according to a preset face detection algorithm and a face characteristic point alignment algorithm to obtain position information of facial characteristic points of the driver;
determining the head posture of the driver according to the position information, judging whether the head posture of the driver meets a preset abnormal condition or not, and/or identifying the fatigue degree of the driver according to the position information;
and when the head posture of the driver meets a preset abnormal condition or the fatigue degree of the driver meets a preset early warning condition, triggering corresponding early warning reminding.
Further, the processor 1001 may call the driving warning program stored in the memory 1005, and further perform the following operations:
acquiring position information of the facial feature points of the driver in a preset judgment period, and converting the position information into head deflection angle data of the driver according to a preset geometric projection algorithm;
identifying a head deviation state of the driver according to the head deviation angle data;
detecting whether the duration of single head deviation of the driver reaches a first preset duration or not according to the head deviation state in the judging period;
and if the duration of the single head deviation of the driver reaches a first preset duration, judging that the head posture of the driver meets a preset abnormal condition.
Further, the processor 1001 may call the driving warning program stored in the memory 1005, and further perform the following operations:
judging whether the horizontal deflection angle falls within a preset horizontal deflection angle interval or not and whether the vertical deflection angle falls within a preset vertical deflection angle interval or not;
and when the horizontal deflection angle does not fall within a preset horizontal deflection angle interval or the vertical deflection angle does not fall within a preset vertical deflection angle interval, judging that the driver is currently in a head deviation state.
Further, the processor 1001 may call the driving warning program stored in the memory 1005, and further perform the following operations:
if the duration of the single head deviation of the driver does not reach a first preset duration, counting the accumulated duration of the head deviation state of the driver in the judgment period;
judging whether the occupation ratio of the accumulated time length in the judging period reaches a preset value or not;
and if the occupation ratio of the accumulated time length in the judgment period reaches a preset value, judging that the head posture of the driver meets a preset abnormal condition.
Further, the processor 1001 may call the driving warning program stored in the memory 1005, and further perform the following operations:
in the process of collecting the face image of the driver, detecting whether the face image of the driver is not collected within a second preset time length or not;
and if the facial image of any driver is not acquired within a second preset time, judging that the head posture of the driver meets a preset abnormal condition.
Further, the processor 1001 may call the driving warning program stored in the memory 1005, and further perform the following operations:
acquiring position information of the facial feature points of the driver in a preset identification period, and respectively calculating the aspect ratio of eyes and a mouth of the driver according to the position information;
correspondingly recognizing the opening and closing state and the yawning state of the eyes of the driver according to the aspect ratio of the eyes and the mouth;
calculating preset fatigue characteristic parameters according to the recognized eye opening and closing state and the recognized yawning state;
inputting the fatigue characteristic parameters obtained by calculation into a preset classifier so that the classifier outputs a response value corresponding to a preset fatigue level;
and comparing the response values corresponding to different fatigue grades, and selecting one fatigue grade from the preset fatigue grades according to the comparison result to serve as the identification result of the fatigue degree of the driver.
Further, the processor 1001 may call the driving warning program stored in the memory 1005, and further perform the following operations:
calculating the difference value between the maximum value and the second maximum value in the response values, and judging whether the difference value is greater than or equal to a preset threshold value;
if the difference is larger than or equal to a preset threshold value, taking the fatigue grade corresponding to the maximum value as the recognition result of the fatigue degree of the driver;
if the difference value is smaller than a preset threshold value, counting the frequency of yawning of the driver in the identification period;
if the number of times of yawning of the driver is greater than or equal to the preset number of times, taking a higher fatigue grade in the fatigue grades corresponding to the maximum value and the second maximum value as the identification result of the fatigue degree of the driver;
and if the yawning times of the driver are less than the preset times, taking the lower fatigue grade in the fatigue grades corresponding to the maximum value and the second maximum value as the identification result of the fatigue degree of the driver.
Further, the fatigue degree of the driver includes wakefulness, light fatigue and deep fatigue, and the processor 1001 may call the driving warning program stored in the memory 1005, and further perform the following operations:
when the fatigue degree of the driver is recognized to be light fatigue or deep fatigue, triggering corresponding early warning reminding, wherein the reminding frequency corresponding to the deep fatigue is higher than the reminding frequency corresponding to the light fatigue.
The specific embodiment of the driving warning device of the present invention is substantially the same as the following embodiments of the driving warning method, and is not described herein again.
Based on the hardware structure, the embodiment of the driving early warning method is provided.
Referring to fig. 2, fig. 2 is a schematic flow chart of a first embodiment of the driving warning method of the present invention, and the method includes:
step S10, collecting a facial image of a driver, and analyzing the facial image according to a preset face detection algorithm and a face feature point alignment algorithm to obtain position information of the face feature point of the driver;
in this embodiment, in the vehicle driving process, the driving early warning device can gather driver's facial image in real time through infrared light compensation camera, through adopting infrared light compensation camera, can shoot the driver's face clearly under the darker condition of ambient light to can shoot the driver's eyes clearly under the condition that the driver wears sunglasses.
After the facial image of the driver is collected, the facial image is analyzed according to a preset face detection algorithm and a preset face feature point alignment algorithm to obtain the position information of the facial feature points of the driver, wherein the position information can comprise the numbers and the position coordinates of different face feature points. The face detection algorithm may be a Haar (Haar) cascade classification algorithm, or a Support Vector Machine (SVM) face detection algorithm based on a DLIB library, and the face feature point alignment algorithm may be an SDM (supervisory drop Method) face feature point alignment algorithm, or a CLNF (Constrained Local Neural Field) face feature point alignment algorithm.
The specific way of analyzing the facial image to obtain the position information of the facial feature points of the driver by the above algorithm can refer to the prior art, and is not described herein again. The facial image of the driver is converted into the position information of the facial feature points by adopting a preset algorithm, so that the position information of the facial feature points corresponding to each frame of facial image can be obtained.
Step S20, determining the head posture of the driver according to the position information, judging whether the head posture of the driver meets a preset abnormal condition, and/or identifying the fatigue degree of the driver according to the position information;
after the position information of the facial feature points of the driver is obtained, the head posture and the fatigue degree of the driver are respectively detected according to the position information.
During the fatigue degree detection, the height-to-width ratio of the eyes and the mouth of the driver can be calculated according to the position information, the opening and closing state and the yawning state of the eyes of the driver are further identified, then the fatigue degree of the driver is judged according to the opening and closing state and the yawning state of the eyes of the driver in a period of time, wherein the fatigue degree judging mode can be flexibly set, for example, when the frequency of yawning of the driver in a period of time is detected to exceed the preset frequency, the driver is judged to be fatigue driving.
In the head posture detection, as an embodiment, referring to fig. 3, fig. 3 is a detailed step diagram of the step in fig. 2 of determining the head posture of the driver according to the position information and judging whether the head posture of the driver meets a preset abnormal condition, and the step may include:
step S21, acquiring the position information of the facial feature points of the driver in a preset judgment period, and converting the position information into head deflection angle data of the driver according to a preset geometric projection algorithm;
step S22, recognizing the head deviation state of the driver according to the head deviation angle data;
step S23, in the judging period, detecting whether the duration of the single head deviation of the driver reaches a first preset duration according to the head deviation state;
if yes, step S24 is executed to determine that the head posture of the driver satisfies a preset abnormality condition.
Referring to fig. 4, fig. 4 is a schematic diagram illustrating the acquisition of the position information of the facial feature points of the driver through the sliding data window in the embodiment of the present invention. Specifically, a determination period (e.g. 2s) may be preset, where the determination period is used to indicate an interval duration for each determination of whether the head posture of the driver is abnormal, and the position information data of the facial feature point of the driver may be collected in the form of a sliding data window, for example, if the length of the data window in one determination period is set to 5s and the sliding step length is set to 2s, data within 5 seconds (corresponding to window one) is collected for the first time as an analysis object in a first determination period, and then data with a length of 5s (corresponding to window two) is collected for 2 seconds as an analysis object in a subsequent determination period. The data are collected in the overlapping mode according to the fact that the process of the driver doing the action is continuous, partial data of the previous frame are reserved in the next frame, the continuous process can be shown, and false detection caused by discontinuous image frame data can be avoided, so that the detection accuracy is improved.
When a certain judgment period is judged, firstly, the position information of the facial feature points of the driver at each time point in the judgment period is obtained, and then the position information is converted into head deflection angle data of the driver according to a preset geometric projection algorithm, wherein the head deflection angle data comprises a horizontal deflection angle and a vertical deflection angle, the horizontal deflection angle represents the deflection angle of the current head of the driver relative to the preset standard head position in the horizontal direction, and the vertical deflection angle represents the deflection angle of the current head of the driver relative to the preset standard head position in the vertical direction.
The conversion process of converting the position information into the head slip angle data of the driver may be as follows:
in one embodiment, the position information includes 68 aligned face feature point data, where the 0 th to 16 th points mark the outline of the face, the 17 th to 21 st points mark the right eyebrow region, the 22 nd to 26 th points mark the left eyebrow region, the 27 th to 35 th points mark the nose region, the 36 th to 41 th points mark the right eye region, the 42 th to 47 th points mark the left eye region, and the 48 th to 67 th points mark the mouth region; when head deflection angle data are calculated, selecting points (4) at the left eye corner and the right eye corner of the 68 characteristic points, points (1) at the nose tip position, 2 points at the mouth corner position, and 7 personal face characteristic points in total, and performing matrix multiplication on corresponding points on the corresponding standardized face to obtain a rotation matrix of the current face relative to the standardized face, and calculating deflection angles of the current face in all directions by using a conversion formula between the rotation matrix and an Euler angle.
Then, the head deviation state of the driver is identified according to the head deviation angle data, and the specific identification process may be as follows: judging whether the horizontal deflection angle falls within a preset horizontal deflection angle interval or not and whether the vertical deflection angle falls within a preset vertical deflection angle interval or not; and when the horizontal deflection angle does not fall within a preset horizontal deflection angle interval or the vertical deflection angle does not fall within a preset vertical deflection angle interval, judging that the driver is currently in a head deviation state.
It should be noted that the horizontal deflection angle interval and the vertical deflection angle interval may be obtained by learning through a preset machine learning algorithm, and the specific learning process is as follows: when the driver initially enters a driving state, the head of the driver is kept facing the front of the automobile, and at the moment, the horizontal deflection angle and the vertical deflection angle of the head posture of the driver in a period of time are counted according to the geometric projection algorithmObtaining a horizontal deflection angle set and a vertical deflection angle set; then, respectively calculating the mean value and the standard deviation of the two sets of data, and combining the mean value and the standard deviation to construct an initial deflection angle range of the driver in the horizontal direction and the vertical direction in the initial state, wherein the threshold value of the range in the horizontal direction is as follows:
vertical direction range threshold:
then, the horizontal direction range threshold and the vertical direction range threshold are appropriately adjusted according to a preset empirical value, so as to obtain a final horizontal deflection angle interval and a final vertical deflection angle interval, for example, the horizontal deflection angle interval may be set as [ y ] and the vertical deflection angle interval may be set as [ y ]min-10°，ymax+10°]。
The horizontal deflection angle of the head of the driver does not fall in a preset horizontal deflection angle interval, which shows that the degree of the head of the driver deviating from the front in the horizontal direction is large, and at the moment, the driver can be judged to be in a head deviating state currently; similarly, the vertical deflection angle of the current head of the driver does not fall within the preset vertical deflection angle interval, which indicates that the current head of the driver deviates from the front in the vertical direction to a greater extent, and at this time, it is also determined that the driver is currently in a head deviation state.
The horizontal deflection angle interval and the vertical deflection angle interval are calculated through a machine learning algorithm, so that interval setting is realized according to different driving habits of different drivers, the head deflection angles of the different drivers can be self-adapted, the horizontal deflection angle interval and the vertical deflection angle interval are used as judgment standards of head deflection states of subsequent drivers, and the judgment accuracy can be improved. Of course, the horizontal deflection angle interval and the vertical deflection angle interval can also be directly set manually, and can be flexibly set during specific implementation.
And based on a judgment cycle, after the head deviation state of the driver at each time point is recognized, introducing a preset rule to judge whether the head posture of the driver meets a preset abnormal condition. As a determination manner, whether the duration of the single head deviation of the driver in the determination period reaches a first preset duration may be detected, for example, 2s, and if yes, it may be determined that the head posture of the driver meets a preset abnormal condition.
Further, if the duration of the single head deviation of the driver does not reach a first preset duration, counting the accumulated duration of the head deviation state of the driver in the judgment period; judging whether the occupation ratio of the accumulated time length in the judging period reaches a preset value or not; and if the occupation ratio of the accumulated time length in the judgment period reaches a preset value, judging that the head posture of the driver meets a preset abnormal condition. For example, when the determination period is 2s, if the cumulative duration of the head deviation state of the driver within 2s is 1.4s, which reaches the preset percentage of 70%, it may be determined that the head posture of the driver meets the preset abnormal condition.
Specifically, when counting the accumulated time of the driver in the head deviation state in the determination period, the driver needs to perform the detection of the single head deviation time, the detection of the single head deviation time can be performed in a data flow manner, referring to fig. 5, fig. 5 is a schematic diagram of data acquisition in a data flow manner in the embodiment of the present invention, where data 1, data 2, and data 3 are data flows acquired in a preset time period, for each data flow, timing can be started from when the abnormal behavior of the head posture of the driver is first detected, it is determined in the timing process whether the duration of the abnormal state of the head posture of the driver exceeds the preset time, if yes, the timing time is accumulated in the abnormal duration of the head posture of the driver, and if not, it is described that the driver recovers the normal posture before exceeding the set threshold time limit and meets the normal reaction time of the driver, for example: and 0.15 seconds, and at the moment, the timed time length is abandoned to be accumulated into the abnormal duration of the head posture of the driver.
The accumulation method has an assumption that the behavior of the driver is continuous, so that the behavior of the driver does not fluctuate greatly in a very short time, and abnormal data points with higher or lower values in a piece of data can be excluded; secondly, according to the relevant data, the reaction time of the driver taking action is known to be 0.1-0.2 s, and the deviation of the current driver making action or collecting data can be judged according to the threshold of the reaction time of the driver.
Further, in addition to the above judgment rule, it may also be detected whether there is any facial image of the driver not acquired within a second preset time period in the process of acquiring the facial image of the driver; and if the facial image of any driver is not acquired within a second preset time, judging that the head posture of the driver meets a preset abnormal condition. For example, if no facial image of the driver is captured for 3 seconds, it may be determined that the head posture of the driver satisfies a preset abnormal condition.
And step S30, when the head posture of the driver meets a preset abnormal condition, or when the fatigue degree of the driver meets a preset early warning condition, triggering corresponding early warning prompt.
In the step, when the head posture of the driver is judged to meet the preset abnormal condition or the fatigue degree of the driver meets the preset early warning condition, corresponding early warning reminding is triggered, and specific reminding modes include but are not limited to voice, a warning light and the like. For example, when it is determined that the head posture of the driver meets the preset abnormal condition, which indicates that the attention of the driver is not focused, a reminding voice "please notice the road ahead" may be output; when the fatigue degree of the driver meets the preset early warning condition, the driver is in a fatigue state, and at the moment, a reminding voice can be output to ask the driver to reduce the vehicle speed or stop the vehicle for rest.
According to the driving early warning method provided by the embodiment, the position information of the facial feature points of the driver is acquired, the abnormal situation of the head posture of the driver is detected according to the position information, the fatigue degree of the driver is detected, and corresponding abnormal reminding is triggered according to the position information and the fatigue degree, so that the effective detection and early warning of the physiological abnormal situations of the driver, such as inattention, fatigue and the like, are realized, and the driving safety is ensured.
Further, based on the first embodiment of the driving early warning method, the second embodiment of the driving early warning method is provided.
Referring to fig. 6, fig. 6 is a schematic diagram illustrating a detailed step of the step of fig. 2 for identifying the fatigue level of the driver according to the position information, where the step may include:
step S25, acquiring the position information of the facial feature points of the driver in a preset identification period, and respectively calculating the aspect ratio of the eyes and the mouth of the driver according to the position information;
step S26, correspondingly recognizing the opening and closing state and the yawning state of the eyes of the driver according to the aspect ratio of the eyes and the mouth;
step S27, calculating preset fatigue characteristic parameters according to the recognized eye opening and closing state and the recognized yawning state;
step S28, inputting the fatigue characteristic parameters obtained by calculation into a preset classifier so as to enable the classifier to output a response value corresponding to a preset fatigue level;
and step S29, comparing the response values corresponding to different fatigue grades, and selecting one fatigue grade from the preset fatigue grades according to the comparison result as the identification result of the fatigue degree of the driver.
In the present embodiment, when detecting the fatigue degree, an identification period (e.g., 2s) may be set in advance, and the identification period is used to indicate the interval duration of each time the fatigue degree of the driver is identified. When fatigue degree identification is carried out aiming at a certain identification period, firstly, the position information of the facial feature points of the driver at each time point in the identification period is obtained, and then the aspect ratio of the eyes and the mouth of the driver is respectively calculated according to the position information, wherein the aspect ratio of the eyes reflects the opening and closing state of the eyes of the driver, and the aspect ratio of the mouth reflects the opening and closing state of the mouth of the driver.
Specifically, as an embodiment, the height and the width of the eyes and the mouth of the driver can be respectively calculated according to the coordinate difference mode, and then the height and the width are converted into the ratio of the height and the width, namely the aspect ratio, so that the aspect ratio data of the eyes and the mouth of the driver in a period of time are counted, and the aspect ratio threshold value of blinking and yawning of the driver is calculated. For example, the height-width ratio data of the eyes and the mouth of the driver can be counted within 30 seconds, then abnormal data (the height-width ratio is normal data between 0 and 1) is filtered out, and the filtered normal data is placed into a queue to be sorted (from small to large); then, in the queue containing the eye aspect ratio, possible abnormal data may be first eliminated, for example, the first 5% and the last 5% of the statistical data queue are filtered, and then the maximum value EA and the minimum value EI in the queue are taken, where the calculation formula of the eye threshold may be: t _ eye ═ (EA-EI) × 0.1+ EI; in a queue containing the mouth aspect ratio, the data at 0.8 can be taken from the small end of the queue and then amplified by 1.5 times to obtain the threshold value of the mouth aspect ratio when the driver yawns. If the eye threshold value is smaller than the eye threshold value, the eyes are closed, otherwise, the eyes are open; if the value is less than the mouth threshold value, the mouth is normal, otherwise the mouth is yawned.
Of course, other methods can be used to calculate the glasses threshold and the mouth threshold of the driver, and the calculation can be flexibly set in specific implementation.
Then, a preset fatigue characteristic parameter is calculated according to the recognized eye opening and closing state and yawning state, wherein in one embodiment, the fatigue characteristic parameters include 12, which are PERCLOS (cumulative eye closing time as a percentage of the length of the time window over a period of time), MCD (maximum duration of one eye closing time over a period of time), BF (number of blinks per minute), AOL (average of eye opening degrees (percentages) over a period of time), TWLCLOS (cumulative total time required to reach a specific eye closing time), AOT (average of time required for eye opening action over a period of time), MOT (maximum of time required for eye opening action over a period of time), ACT (average of time required for eye closing action over a period of time), MCT (maximum of time required for eye closing action over a period of time), MRCOT (maximum of ratio of eye closing time to eye opening time over a period of time), the calculation method of the arch (average value of the ratio of the eye-closing time to the eye-opening time in a period of time), YAWN (number of YAWNs in a period of time), and the specific fatigue characteristic parameter may refer to the prior art, and will not be described herein.
After each fatigue characteristic parameter is obtained through calculation, the calculation result is input into a preset classifier (such as a Fisher classifier) so that the classifier outputs a response value corresponding to a preset fatigue level, then the response values corresponding to different fatigue levels are compared, and one fatigue level is selected from the preset fatigue levels according to the comparison result and is used as the identification result of the fatigue degree of the driver.
In one embodiment, the fatigue level corresponding to the maximum response value may be used as the recognition result of the fatigue degree of the driver; in further embodiments, the step of comparing the response values corresponding to different fatigue levels and selecting one fatigue level from the preset fatigue levels according to the comparison result may include: calculating the difference value between the maximum value and the second maximum value in the response values, and judging whether the difference value is greater than or equal to a preset threshold value; if the difference is larger than or equal to a preset threshold value, taking the fatigue grade corresponding to the maximum value as the recognition result of the fatigue degree of the driver; if the difference value is smaller than a preset threshold value, counting the frequency of yawning of the driver in the identification period; if the number of times of yawning of the driver is greater than or equal to the preset number of times, taking a higher fatigue grade in the fatigue grades corresponding to the maximum value and the second maximum value as the identification result of the fatigue degree of the driver; and if the yawning times of the driver are less than the preset times, taking the lower fatigue grade in the fatigue grades corresponding to the maximum value and the second maximum value as the identification result of the fatigue degree of the driver.
Taking preset fatigue grades as Clear, Little and Deep fatigue, taking the corresponding response values as F1, F2 and F3, F1> F2> F3 as examples, firstly calculating the difference between the maximum value F1 and the second maximum value F2, judging whether the difference is larger than or equal to a preset threshold value, if so, indicating that the probability of waking is higher, otherwise, indicating that both the waking and the shallow fatigue have high probability, further counting the number of times of yawning of the driver in the identification period, if the number of times of yawning of the driver is larger than or equal to the preset number, taking the shallow fatigue as the identification result of the fatigue degree of the driver, and otherwise, taking the waking as the identification result of the fatigue degree of the driver.
According to the method, the accuracy of identifying the fatigue degree of the driver is improved by combining the magnitude of the response value and the yawning times of the driver in the identification period.
Further, the fatigue degree of the driver includes wakefulness, light fatigue and deep fatigue, and when the fatigue degree of the driver meets a preset early warning condition, the step of triggering a corresponding early warning reminder may include: when the fatigue degree of the driver is recognized to be light fatigue or deep fatigue, triggering corresponding early warning reminding, wherein the reminding frequency corresponding to the deep fatigue is higher than the reminding frequency corresponding to the light fatigue. Therefore, effective early warning of the fatigue state of the driver is achieved.
The invention also provides a computer readable storage medium.
The computer-readable storage medium of the present invention has stored thereon a driving warning program, which when executed by a processor implements the steps of the driving warning method as described above.
The method implemented when the driving warning program running on the processor is executed may refer to each embodiment of the driving warning method of the present invention, and is not described herein again.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (8)
1. A driving early warning method is characterized by comprising the following steps:
acquiring a facial image of a driver, and analyzing the facial image according to a preset face detection algorithm and a face characteristic point alignment algorithm to obtain position information of facial characteristic points of the driver;
determining the head posture of the driver according to the position information, judging whether the head posture of the driver meets a preset abnormal condition or not, and/or identifying the fatigue degree of the driver according to the position information;
when the head posture of the driver meets a preset abnormal condition or the fatigue degree of the driver meets a preset early warning condition, triggering corresponding early warning reminding;
wherein the step of identifying the fatigue level of the driver from the position information comprises:
acquiring position information of the facial feature points of the driver in a preset identification period, and respectively calculating the aspect ratio of eyes and a mouth of the driver according to the position information;
correspondingly recognizing the opening and closing state and the yawning state of the eyes of the driver according to the aspect ratio of the eyes and the mouth;
calculating preset fatigue characteristic parameters according to the recognized eye opening and closing state and the recognized yawning state;
inputting the fatigue characteristic parameters obtained by calculation into a preset classifier so that the classifier outputs a response value corresponding to a preset fatigue level;
comparing the response values corresponding to different fatigue grades, and selecting one fatigue grade from the preset fatigue grades according to a comparison result to serve as an identification result of the fatigue degree of the driver;
the step of comparing the response values corresponding to different fatigue levels, and selecting one fatigue level from the preset fatigue levels according to the comparison result, wherein the step of taking the fatigue level as the identification result of the fatigue degree of the driver comprises the following steps:
calculating the difference value between the maximum value and the second maximum value in the response values, and judging whether the difference value is greater than or equal to a preset threshold value;
if the difference is larger than or equal to a preset threshold value, taking the fatigue grade corresponding to the maximum value as the recognition result of the fatigue degree of the driver;
if the difference value is smaller than a preset threshold value, counting the frequency of yawning of the driver in the identification period;
if the number of times of yawning of the driver is greater than or equal to the preset number of times, taking a higher fatigue grade in the fatigue grades corresponding to the maximum value and the second maximum value as the identification result of the fatigue degree of the driver;
and if the yawning times of the driver are less than the preset times, taking the lower fatigue grade in the fatigue grades corresponding to the maximum value and the second maximum value as the identification result of the fatigue degree of the driver.
2. The driving warning method of claim 1, wherein the determining of the head posture of the driver according to the position information and the determining of whether the head posture of the driver satisfies a preset abnormal condition comprises:
acquiring position information of the facial feature points of the driver in a preset judgment period, and converting the position information into head deflection angle data of the driver according to a preset geometric projection algorithm;
identifying a head deviation state of the driver according to the head deviation angle data;
detecting whether the duration of single head deviation of the driver reaches a first preset duration or not according to the head deviation state in the judging period;
and if the duration of the single head deviation of the driver reaches a first preset duration, judging that the head posture of the driver meets a preset abnormal condition.
3. The driving warning method of claim 2, wherein the head-lean angle data includes a horizontal lean angle and a vertical lean angle, and the step of identifying the head-lean state of the driver from the head-lean angle data includes:
judging whether the horizontal deflection angle falls within a preset horizontal deflection angle interval or not and whether the vertical deflection angle falls within a preset vertical deflection angle interval or not;
and when the horizontal deflection angle does not fall within a preset horizontal deflection angle interval or the vertical deflection angle does not fall within a preset vertical deflection angle interval, judging that the driver is currently in a head deviation state.
4. The driving warning method according to claim 2, wherein after the step of detecting whether there is a duration of a single head deviation of the driver for a first preset duration according to the head deviation state in the determination period, the method further comprises:
if the duration of the single head deviation of the driver does not reach a first preset duration, counting the accumulated duration of the head deviation state of the driver in the judgment period;
judging whether the occupation ratio of the accumulated time length in the judging period reaches a preset value or not;
and if the occupation ratio of the accumulated time length in the judgment period reaches a preset value, judging that the head posture of the driver meets a preset abnormal condition.
5. The driving warning method as set forth in claim 4, wherein the step of collecting the face image of the driver includes:
in the process of collecting the face image of the driver, detecting whether the face image of the driver is not collected within a second preset time length or not;
and if the facial image of any driver is not acquired within a second preset time, judging that the head posture of the driver meets a preset abnormal condition.
6. The driving warning method of any one of claims 1 to 5, wherein the fatigue degrees of the driver comprise wakefulness, light fatigue and deep fatigue, and when the fatigue degrees of the driver meet preset warning conditions, the step of triggering the corresponding warning reminder comprises:
when the fatigue degree of the driver is recognized to be light fatigue or deep fatigue, triggering corresponding early warning reminding, wherein the reminding frequency corresponding to the deep fatigue is higher than the reminding frequency corresponding to the light fatigue.
7. A driving warning apparatus characterized by comprising: memory, a processor and a driving pre-warning program stored on the memory and executable on the processor, the driving pre-warning program when executed by the processor implementing the steps of the driving pre-warning method as claimed in any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a driving warning program, which when executed by a processor implements the steps of the driving warning method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810851573.3A CN110766912B (en) | 2018-07-27 | 2018-07-27 | Driving early warning method, device and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810851573.3A CN110766912B (en) | 2018-07-27 | 2018-07-27 | Driving early warning method, device and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110766912A CN110766912A (en) | 2020-02-07 |
| CN110766912B true CN110766912B (en) | 2022-03-18 |
Family
ID=69328870
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810851573.3A Active CN110766912B (en) | 2018-07-27 | 2018-07-27 | Driving early warning method, device and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110766912B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11352013B1 (en) | 2020-11-13 | 2022-06-07 | Samsara Inc. | Refining event triggers using machine learning model feedback |
| US11341786B1 (en) | 2020-11-13 | 2022-05-24 | Samsara Inc. | Dynamic delivery of vehicle event data |
| US11643102B1 (en) * | 2020-11-23 | 2023-05-09 | Samsara Inc. | Dash cam with artificial intelligence safety event detection |
| CN113408466A (en) * | 2021-06-30 | 2021-09-17 | 东风越野车有限公司 | Method and device for detecting bad driving behavior of vehicle driver |
| US11352014B1 (en) | 2021-11-12 | 2022-06-07 | Samsara Inc. | Tuning layers of a modular neural network |
| US11386325B1 (en) | 2021-11-12 | 2022-07-12 | Samsara Inc. | Ensemble neural network state machine for detecting distractions |
| CN114454891B (en) * | 2022-02-28 | 2023-09-26 | 奇瑞汽车股份有限公司 | Control method and device for automobile and computer storage medium |
| CN116149226B (en) * | 2023-02-22 | 2023-11-10 | 山东中安电力科技有限公司 | Switch cabinet remote control system based on data analysis |
| CN116311181B (en) * | 2023-03-21 | 2023-09-12 | 重庆利龙中宝智能技术有限公司 | Method and system for rapidly detecting abnormal driving |
| CN117064375B (en) * | 2023-07-18 | 2024-03-22 | 江西瑞声电子有限公司 | Head posture monitoring method, main control equipment and intelligent wearable equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101030316A (en) * | 2007-04-17 | 2007-09-05 | 北京中星微电子有限公司 | Safety driving monitoring system and method for vehicle |
| CN101032405A (en) * | 2007-03-21 | 2007-09-12 | 汤一平 | Safe driving auxiliary device based on omnidirectional computer vision |
| CN104688251A (en) * | 2015-03-02 | 2015-06-10 | 西安邦威电子科技有限公司 | Method for detecting fatigue driving and driving in abnormal posture under multiple postures |
| CN105788176A (en) * | 2016-05-25 | 2016-07-20 | 厦门理工学院 | Fatigue driving monitoring and prompting method and system |
| WO2018058266A1 (en) * | 2016-09-27 | 2018-04-05 | 深圳智乐信息科技有限公司 | Mobile terminal-based notification method and system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040032334A1 (en) * | 2002-08-13 | 2004-02-19 | Ataul Haq | Doze-off early warning system for automotives and other applications |
| CN101254765A (en) * | 2005-05-19 | 2008-09-03 | 曹民 | Multifunctional manual/automatic complementary type headlamp dimmer |
| CN101216164B (en) * | 2007-12-29 | 2010-11-10 | 西安交通大学 | Water-cooled wall on-line safe evaluation method |
| CN101375796B (en) * | 2008-09-18 | 2010-06-02 | 浙江工业大学 | Real-time detection system of fatigue driving |
| KR101327007B1 (en) * | 2011-10-17 | 2013-11-13 | 현대자동차주식회사 | A system and method for detection of concentration grade wherein car driving |
-
2018
- 2018-07-27 CN CN201810851573.3A patent/CN110766912B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101032405A (en) * | 2007-03-21 | 2007-09-12 | 汤一平 | Safe driving auxiliary device based on omnidirectional computer vision |
| CN101030316A (en) * | 2007-04-17 | 2007-09-05 | 北京中星微电子有限公司 | Safety driving monitoring system and method for vehicle |
| CN104688251A (en) * | 2015-03-02 | 2015-06-10 | 西安邦威电子科技有限公司 | Method for detecting fatigue driving and driving in abnormal posture under multiple postures |
| CN105788176A (en) * | 2016-05-25 | 2016-07-20 | 厦门理工学院 | Fatigue driving monitoring and prompting method and system |
| WO2018058266A1 (en) * | 2016-09-27 | 2018-04-05 | 深圳智乐信息科技有限公司 | Mobile terminal-based notification method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110766912A (en) | 2020-02-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110766912B (en) | Driving early warning method, device and computer readable storage medium | |
| CN110532976B (en) | Fatigue driving detection method and system based on machine learning and multi-feature fusion | |
| Hossain et al. | IOT based real-time drowsy driving detection system for the prevention of road accidents | |
| CN110826370B (en) | Method and device for identifying identity of person in vehicle, vehicle and storage medium | |
| CN104183091A (en) | System for adjusting sensitivity of fatigue driving early warning system in self-adaptive mode | |
| CN111434553B (en) | Brake system, method and device, and fatigue driving model training method and device | |
| CN105788176A (en) | Fatigue driving monitoring and prompting method and system | |
| CN111414813A (en) | Dangerous driving behavior identification method, device, equipment and storage medium | |
| CN112744174B (en) | Vehicle collision monitoring method, device, equipment and computer readable storage medium | |
| Lashkov et al. | Driver dangerous state detection based on OpenCV & dlib libraries using mobile video processing | |
| CN112754498B (en) | Driver fatigue detection method, device, equipment and storage medium | |
| Charniya et al. | Drunk driving and drowsiness detection | |
| US20240096116A1 (en) | Devices and methods for detecting drowsiness of drivers of vehicles | |
| Guria et al. | Iot-enabled driver drowsiness detection using machine learning | |
| CN111354183B (en) | Early warning information pushing method and terminal | |
| Rani et al. | Development of an Automated Tool for Driver Drowsiness Detection | |
| TWI528331B (en) | Attention detecting device, method, computer readable medium, and computer program products | |
| Bajaj et al. | A real-time driver drowsiness detection using OpenCV, DLib | |
| Mašanović et al. | Driver monitoring using the in-vehicle camera | |
| CN201838333U (en) | Music player based on state of driver | |
| CN113312958B (en) | Method and device for adjusting dispatch priority based on driver state | |
| CN114998874A (en) | Driver abnormal behavior detection method based on deep learning | |
| CN112307920B (en) | High-risk worker behavior early warning device and method | |
| Xie et al. | Revolutionizing Road Safety: YOLOv8-Powered Driver Fatigue Detection | |
| CN111274881B (en) | Driving safety monitoring method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |