CN110390285A - System for distraction of driver detection method, system and vehicle - Google Patents
System for distraction of driver detection method, system and vehicle Download PDFInfo
- Publication number
- CN110390285A CN110390285A CN201910641923.8A CN201910641923A CN110390285A CN 110390285 A CN110390285 A CN 110390285A CN 201910641923 A CN201910641923 A CN 201910641923A CN 110390285 A CN110390285 A CN 110390285A
- Authority
- CN
- China
- Prior art keywords
- attention
- driver
- value
- threshold value
- diverting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
- G06V40/166—Detection; Localisation; Normalisation using acquisition arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Data Mining & Analysis (AREA)
- Human Computer Interaction (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Life Sciences & Earth Sciences (AREA)
- Ophthalmology & Optometry (AREA)
- Fittings On The Vehicle Exterior For Carrying Loads, And Devices For Holding Or Mounting Articles (AREA)
Abstract
This application discloses a kind of system for distraction of driver detection method, system and vehicles.Wherein, system for distraction of driver detection method, comprising: obtain the video image of vehicle driver's human face region and the driving status of vehicle；Head pose detection and line-of-sight detection are carried out according to video image, obtains head pose information and sight information；Judge whether driver diverts attention in conjunction with driving status using head pose information and sight information.The system for distraction of driver detection method of the application accurately regular inspection can measure whether driver diverts attention, and effectively avoid or reduce the generation of error detection, in turn, be able to ascend the safety and reliability of driving.
Description
Technical field
This application involves automobile technical field, in particular to a kind of system for distraction of driver detection method, system and vehicle.
Background technique
Many traffic accidents are caused due to diverting attention in driver's driving procedure, such as: seeing the mobile phone when driving,
Outer landscape of looking glass etc., causes traffic accident.
In the related technology, system for distraction of driver state is detected by head pose, such as: detect that driver turns
Head, whens bowing etc., it is believed that driver is in driving status of diverting attention, and reminds driver, there are following technical problems:
If rotary head or the movement bowed are smaller, driver may not concentrate one's energy really, but may also be still
It concentrates one's energy, is easy to produce error detection at this time, in turn, influence the usage experience of vehicle.
Summary of the invention
The application be intended to solve at least to a certain extent it is above-mentioned in the related technology the technical issues of one of.
For this purpose, first purpose of the application is to propose a kind of system for distraction of driver detection method.This method can be accurate
Regular inspection measures whether driver diverts attention, and effectively avoids or reduces the generation of error detection, in turn, be able to ascend driving safety and
Reliability.
Second purpose of the application is to propose a kind of system for distraction of driver detection system.
The third purpose of the application is to propose a kind of vehicle.
The 4th purpose of the application is to propose a kind of machine readable media.
To achieve the goals above, the first aspect of the application discloses a kind of system for distraction of driver detection method, including with
Lower step: the video image of vehicle driver's human face region and the driving status of vehicle are obtained；Head is carried out according to video image
Attitude detection and line-of-sight detection obtain head pose information and sight information；Using head pose information and sight information, in conjunction with
Driving status, judges whether driver diverts attention.
The system for distraction of driver detection method of the application, can accurate regular inspection measure whether driver diverts attention, effectively avoid or
The generation for reducing error detection is able to ascend the safety and reliability of driving in turn.
In some instances, when the driving status includes state of rolling backward, excessively curved state, the state that turns around or traffic congestion state,
Judge whether driver diverts attention according to the head pose information.
In some instances, described to judge that driver is in conjunction with driving status using head pose information and sight information
It is no to divert attention, comprising: when determining that vehicle is in turning or speed less than preset vehicle speed threshold value according to transport condition, to utilize head appearance
State information and sight information judge that driver watches whether time of preset normal driving watching area is longer than the predetermined time attentively；Such as
Fruit is then to determine that driver is in driving condition of diverting attention.
In some instances, described to judge that driver is in conjunction with driving status using head pose information and sight information
It is no to divert attention, comprising: when determining that vehicle is in forward travel state and speed is greater than preset vehicle speed threshold value according to current driving status, to obtain
Present frame is taken to divert attention value；Diverted attention the accumulated time meter that value, the video frame rate of the video image and triggering divert attention according to present frame
Calculate the accumulative value of diverting attention in the accumulated time；Judge whether driver diverts attention according to the accumulative value of diverting attention.
In some instances, the head pose information includes head yaw angle and head pitch angle, according to sight information
Determine that preset normal driving watching area includes first area and second area, then present frame is diverted attention the method for determination packet of value
It includes: if the head yaw angle is beyond yaw angle threshold value and/or the head pitch angle beyond pitch angle threshold value, institute
Stating present frame value of diverting attention is first to divert attention and be worth threshold value；If the head yaw angle is without departing from yaw angle threshold value and/or described
Head pitch angle is without departing from pitch angle threshold value, and when driver watches first area attentively, present frame value of diverting attention is preset the
One region, which is diverted attention, is worth accumulative parameter；When driver watches second area attentively, the present frame diverts attention value for preset second area point
Mind is worth accumulative parameter；When driver watches improper driving watching area attentively, present frame value of diverting attention is first to divert attention and be worth threshold
Value；When driver watches normal driving watching area attentively, present frame value of diverting attention is second to divert attention and be worth threshold value；Wherein, first point
Refreshing value threshold value is diverted attention greater than second is worth threshold value, and first area, which is diverted attention, is worth accumulative parameter and second area diverts attention that be worth accumulative parameter all big
Divert attention to be worth threshold value and divert attention less than first in second and is worth threshold value.
In some instances, add up to divert attention value calculating process in, if present frame is diverted attention, value, which reaches first and diverts attention, is worth threshold value,
Then accumulative value of diverting attention is accumulated with linear velocity.
In some instances, add up to divert attention value calculating process in, if present frame is diverted attention, value, which reaches second and diverts attention, is worth threshold value,
Then accumulative value of diverting attention is decayed in a manner of any one including exponential damping, linear attenuation or polynomial decay.
In some instances, described to judge whether driver diverts attention according to the accumulative value of diverting attention, comprising: described accumulative point
Refreshing value reaches accumulative and diverts attention when being worth threshold value, determines driver in driving condition of diverting attention.
The second aspect of the present invention discloses a kind of system for distraction of driver detection system, comprising: module is obtained, for obtaining vehicle
The video image of driver's human face region and the driving status of vehicle；Detection module, for carrying out head according to video image
Attitude detection and line-of-sight detection obtain head pose information and sight information；Judgment module, for using head pose information and
Sight information judges whether driver diverts attention in conjunction with driving status.
The system for distraction of driver detection system of the application, can accurate regular inspection measure whether driver diverts attention, effectively avoid or
The generation for reducing error detection is able to ascend the safety and reliability of driving in turn.
In some instances, when the driving status includes state of rolling backward, excessively curved state, the state that turns around or traffic congestion state,
The judgment module judges whether driver diverts attention according to the head pose information.
In some instances, the judgment module for determining that vehicle is in turning or speed is less than according to transport condition
When preset vehicle speed threshold value, judge that driver watches preset normal driving watching area attentively using head pose information and sight information
Time whether be longer than the predetermined time；If it is, determining that driver is in driving condition of diverting attention.
In some instances, the judgment module be used for according to current driving status determine vehicle be in forward travel state and
When speed is greater than preset vehicle speed threshold value, obtains present frame and divert attention value, and according to present frame divert attention value, the video image video
The accumulated time that frame per second and triggering are diverted attention calculates the accumulative value of diverting attention in the accumulated time, and is sentenced according to the accumulative value of diverting attention
Whether disconnected driver diverts attention.
In some instances, the head pose information includes head yaw angle and head pitch angle, according to sight information
Determine that preset normal driving watching area includes first area and second area, then present frame is diverted attention the method for determination packet of value
It includes:
If the head yaw angle exceeds pitch angle threshold beyond yaw angle threshold value and/or the head pitch angle
Value, then present frame value of diverting attention diverts attention for first and is worth threshold value；
If the head yaw angle is without departing from yaw angle threshold value and/or the head pitch angle without departing from pitch angle
Threshold value, when driver watches first area attentively, present frame value of diverting attention is that preset first area diverts attention and is worth accumulative parameter；In
Driver when watching second area attentively present frame value of diverting attention be that preset second area diverts attention and is worth accumulative parameter；It is infused in driver
Present frame value of diverting attention diverts attention as first and is worth threshold value when regarding improper driving watching area；Normal driving note is watched attentively in driver
Present frame value of diverting attention is second to divert attention and be worth threshold value when viewed area；
Wherein, first diverts attention and is worth threshold value and diverts attention greater than second and be worth threshold value, and first area, which is diverted attention, is worth accumulative parameter and the secondth area
Domain, which diverts attention to be worth accumulative parameter and be both greater than second and divert attention to be worth threshold value and divert attention less than first, is worth threshold value.
In some instances, add up to divert attention value calculating process in, if present frame is diverted attention, value, which reaches first and diverts attention, is worth threshold value,
Then accumulative value of diverting attention is accumulated with linear velocity.
In some instances, add up to divert attention value calculating process in, if present frame is diverted attention, value, which reaches second and diverts attention, is worth threshold value,
Then accumulative value of diverting attention is decayed in a manner of any one including exponential damping, linear attenuation or polynomial decay.
In some instances, the judgment module be used for the accumulative value of diverting attention reach it is accumulative divert attention be worth threshold value when, sentence
Determine driver and is in driving condition of diverting attention.
The third aspect of the application discloses a kind of vehicle, comprising: one or more processors；It is instructed with being stored thereon with
One or more machine readable medias, when by one or more of processors execute when so that the vehicle execute basis
System for distraction of driver detection method described in above-mentioned first aspect.The vehicle accurately regular inspection can measure whether driver diverts attention,
The generation for effectively avoiding or reducing error detection is able to ascend the safety and reliability of driving in turn.
The fourth aspect of the application discloses a kind of machine readable media, instruction is stored thereon with, when by one or more
When processor executes, so that the processor executes the system for distraction of driver detection method according to above-mentioned first aspect.
The additional aspect and advantage of the application will be set forth in part in the description, and will partially become from the following description
It obtains obviously, or recognized by the practice of the application.
Detailed description of the invention
The above-mentioned or additional aspect and advantage combination following accompanying drawings of the application will be to will become bright in the description of embodiment
It shows and is readily appreciated that, in which:
Fig. 1 is the flow chart of the system for distraction of driver detection method of the application one embodiment；
Fig. 2 is head yaw angle, pitch angle and roll angle in the system for distraction of driver detection method of the application one embodiment
Schematic diagram；
Fig. 3 is the schematic diagram of sight watching area in the system for distraction of driver detection method of the application one embodiment；
Fig. 4 is the structural block diagram of the system for distraction of driver detection system of the application one embodiment.
Description of symbols:
System for distraction of driver detection system 400 obtains module 410, detection module 420, judgment module 430.
Specific embodiment
Embodiments herein is described below in detail, examples of the embodiments are shown in the accompanying drawings, wherein from beginning to end
Same or similar label indicates same or similar element or element with the same or similar functions.Below with reference to attached
The embodiment of figure description is exemplary, and is only used for explaining the application, and should not be understood as the limitation to the application.
System for distraction of driver detection method, system and vehicle according to the embodiment of the present application are described below in conjunction with attached drawing.
Fig. 1 is the flow chart according to the system for distraction of driver detection method of the application one embodiment.As shown in Figure 1, according to
The system for distraction of driver detection method of the application one embodiment, includes the following steps:
S101: the video image of vehicle driver's human face region and the driving status of vehicle are obtained.
Wherein, the driving status of vehicle includes normally travel state, state of rolling backward, excessively curved state, the state that turns around and traffic congestion
State etc., wherein the state that normally travel state such as moves forward.
In specific example, the video image of driver's human face region can be collected by camera.
It should be noted that being not usually required to all carry out the acquisition of video image and the driving status of vehicle at any time
Acquisition.Such as: after system for distraction of driver detection triggering, detect driver head's posture and direction of visual lines.
S102: carrying out head pose detection and line-of-sight detection according to video image, obtains head pose information and sight letter
Breath.
Such as: according to video image, driver head's posture and direction of visual lines are detected, can be accomplished by the following way:
From extraction face-image in video image (driver's image)；Driver head's posture and sight side are obtained according to face-image
To obtain head pose information and sight information.
Wherein, driver's image can be acquired by vehicle-mounted camera.Such as: receive driver's figure of camera acquisition
Then picture carries out feature extraction to driver's image using convolutional neural networks, human face region is detected, finally, according to face
Testing result extracts face-image, and carries out head pose detection and line-of-sight detection using the methods of deep learning, from
And determine driver head's posture and direction of visual lines, wherein direction of visual lines refers to the direction of gaze that sight is concentrated, such as: sight collection
In on a mirror, then direction of visual lines is the direction for watching reflective mirror attentively.
S103: judge whether driver diverts attention in conjunction with driving status using head pose information and sight information.
In addition, when driving status is state of rolling backward, excessively curved state, the state that turns around or traffic congestion state, it can be only according to the head
Portion's posture information judges whether driver diverts attention.That is, when driving status is, for example, state of rolling backward, excessively curved state and falls
When first-class state, the sight of driver will not have been focused into front, can see rearview mirror etc. from time to time, therefore, can basis
Head pose information judges whether driver diverts attention, such as: head illustrates the sight of driver in when occurring situations such as bowing
Not in front, also rearview mirror, i.e. judgement system for distraction of driver are not being seen.
In other examples, judge whether driver divides in conjunction with driving status using head pose information and sight information
Mind, comprising: when determining that vehicle is in turning or speed less than preset vehicle speed threshold value according to transport condition, believed using head pose
Breath and sight information judge that driver watches whether time of preset normal driving watching area is longer than the predetermined time attentively；If
It is then to determine that driver is in driving condition of diverting attention.
Preset vehicle speed threshold value is, for example, 40,000 ms/h, it is, of course, understood that 40,000 ms/h are only exemplary
, in other examples, increase and decrease appropriate can also be carried out on the basis of 40,000 ms/h.
Preset normal driving watching area is for example including the region where watching instrument board attentively and watches attentively where rearview mirror
Region etc..
Predetermined time is, for example, 3 seconds.
That is, the sight of driver will not have been focused into front, can see from time to time if vehicle is in turning
Rearview mirror etc., at this point, illustrating that driver may not be after often seeing if the time for persistently watching rearview mirror attentively is longer than 3 seconds
Visor, but may divert attention, therefore, determine that driver is in driving condition of diverting attention.
For another example: if speed is lower, such as less than 40,000 ms/h, if driver watches the times such as instrument, rearview mirror attentively
Too long, also explanation is likely to divert attention, it may be assumed that determines that driver is similarly in driving condition of diverting attention.
In other examples, judge whether driver divides in conjunction with driving status using head pose information and sight information
Mind, comprising: when determining that vehicle is in forward travel state and speed is greater than preset vehicle speed threshold value according to current driving status, acquisition is worked as
Previous frame is diverted attention value；Diverting attention that the accumulated time that value, the video frame rate of the video image and triggering divert attention calculates according to present frame should
Accumulative value of diverting attention in accumulated time；Judge whether driver diverts attention according to the accumulative value of diverting attention.
Such as: speed is greater than 40,000 ms/h and vehicle is in forward travel state, and driver is needed to concentrate one's energy to watch front attentively
Road, at this point, if driver does not concentrate one's energy to watch road ahead attentively and may can be very dangerous, such as: it sees the mobile phone, watch the instruments
Disk or inclined head see roadside landscape etc..Therefore, present frame is obtained to divert attention value；According to present frame divert attention value, video image video
The accumulated time that frame per second and triggering are diverted attention calculates the accumulative value of diverting attention in the accumulated time；Driver is judged according to accumulative value of diverting attention
Whether divert attention.
Wherein, judge whether driver diverts attention according to the accumulative value of diverting attention, comprising: accumulative value of diverting attention reaches accumulative and diverts attention
When being worth threshold value, determine that driver is in driving condition of diverting attention.
In specific example, head pose information includes head yaw angle and head pitch angle, is determined according to sight information
Preset normal driving watching area includes first area and second area, then the divert attention method of determination of value of present frame includes:
If the head yaw angle exceeds pitch angle threshold beyond yaw angle threshold value and/or the head pitch angle
Value, then present frame value of diverting attention diverts attention for first and is worth threshold value；
If the head yaw angle is without departing from yaw angle threshold value and/or the head pitch angle without departing from pitch angle
Threshold value, when driver watches first area attentively, present frame value of diverting attention is that preset first area diverts attention and is worth accumulative parameter；In
Driver when watching second area attentively present frame value of diverting attention be that preset second area diverts attention and is worth accumulative parameter；It is infused in driver
Present frame value of diverting attention diverts attention as first and is worth threshold value when regarding improper driving watching area；Normal driving note is watched attentively in driver
Present frame value of diverting attention is second to divert attention and be worth threshold value when viewed area；
Wherein, first diverts attention and is worth threshold value and diverts attention greater than second and be worth threshold value, and first area, which is diverted attention, is worth accumulative parameter and the secondth area
Domain, which diverts attention to be worth accumulative parameter and be both greater than second and divert attention to be worth threshold value and divert attention less than first, is worth threshold value.First, which diverts attention, is worth threshold value and is, for example,
1, second diverts attention, and to be worth threshold value be, for example, 0.
Further, add up to divert attention value calculating process in, if present frame is diverted attention, value, which reaches first and diverts attention, is worth threshold value, tires out
Refreshing value of scoring is accumulated with linear velocity.
Furthermore, add up to divert attention value calculating process in, if present frame is diverted attention, value, which reaches second and diverts attention, is worth threshold value,
Accumulative value of diverting attention is decayed in a manner of any one including exponential damping, linear attenuation or polynomial decay.
In a particular application, first area is, for example, rearview mirror, and second area is, for example, instrument board.
Then, head yaw angle and head pitch angle can be obtained according to driver head's posture first；Then, according to sight
Direction obtains sight watching area；Finally, being divided accordingly according to head yaw angle, head pitch angle and sight watching area
Mind value, wherein value of diverting attention is for example between [0,1].
As shown in Fig. 2, Yaw is head yaw angle, Pitch is head pitch angle, Roll is head roll angle, in conjunction with Fig. 2
It is shown, typically, when head yaw or head pitching occurs along direction shown in Fig. 2, driver may be will affect and concentrate note
Meaning power watches road ahead attentively, and when head occurs along direction shown in Fig. 2 rolling, driver may not be influenced and focus on infusing
Depending on road ahead, therefore, in the specific example of the application, head yaw angle and head pitch angle are determined.
As shown in figure 3, direction of visual lines concentrates on rearview mirror (first area), then sight watching area is reflective mirror, depending on
Line direction concentrates on instrument board (second area), then sight watching area is instrument board, and direction of visual lines is concentrated in the front road
On, then sight watching area is normal driving watching area, and otherwise, when direction of visual lines concentrates on other positions, sight is watched attentively
Region is non-driving watching area.
It is, for example, rearview mirror with first area, for second area is, for example, instrument board, then according to head yaw angle, head
Pitch angle and sight watching area are diverted attention value accordingly, comprising:
(1) if head yaw angle reaches first threshold and/or head pitch angle reaches second threshold, value of diverting attention is 1；
(2) if the head yaw angle is not up to the first threshold, the head pitch angle is not up to second threshold
And the sight watching area is rearview mirror, then the value of diverting attention is α, wherein the α is between (0,1)；
(3) if the head yaw angle is not up to the first threshold, the head pitch angle is not up to second threshold
And the sight watching area is instrument board, then the value of diverting attention is β, wherein the β is between (0,1)；
(4) if the head yaw angle is not up to the first threshold, the head pitch angle is not up to second threshold
And the sight watching area is non-driving watching area, then the value of diverting attention is 1；
(5) if the head yaw angle is not up to the first threshold, the head pitch angle is not up to second threshold
And the sight watching area is normal driving watching area, then the value of diverting attention is 0.
In the above examples, first threshold and second threshold can be demarcated to obtain in advance, such as: when more than corresponding threshold value
When, it may be assumed that it is larger that head yaws larger or head pitching, it may be said that bright driver does not focus on, value of diverting attention accordingly
Maximum is 1.And when being not above corresponding threshold value, it may be assumed that the head smaller or head pitching of yaw is smaller, and explanation may not have
It focuses on driving, at this point, value of diverting attention accordingly can be determined in conjunction with sight watching area, such as between 0 and 1
Numerical value.
It should be noted that when embodiments herein only lists above-mentioned 5 kinds of situations, according to head yaw angle, head
The corresponding value of diverting attention that pitch angle and sight watching area obtain, however, it is also possible to corresponding value of diverting attention, example when having other situations
Such as: if head yaw angle is not up to the first threshold, head pitch angle is not up to second threshold and the sight field of regard
Domain is middle control screen, then can also determine value of diverting attention accordingly, which can also be between 0 and 1.
It is understood that the value of diverting attention in above-mentioned example is the numerical value between 0 and 1, it may be assumed that in the examples described above,
The upper limit for value of diverting attention is 1, and the lower limit for value of diverting attention is 0, but is merely exemplary, and in other examples, can according to need tune
Whole upper and lower bound, such as: the upper limit is set as 5, and lower limit is set as 0, then value of diverting attention is the numerical value between 0 and 5.Certainly, the upper limit
It also can be set to other values with lower limit.
Whether the judgement accumulative value of diverting attention of the first moment in the triggered time of diverting attention reaches preset threshold.
It can be precisely determined whether driver is in driving status of really diverting attention, example by accumulative value of diverting attention
Such as: when driver watches instrument board or rearview mirror the short time attentively, should not be determined as driving status of diverting attention immediately, but work as driver
Watch instrument board or rearview mirror overlong time attentively, should be determined as driving status of diverting attention.Therefore, by adjusting the adjustable of α and β value
Whole driver diverts attention the sensitivity of judgement when watching different zones attentively.
In specific example, whether the judgement accumulative value of diverting attention of the first moment in the triggered time of diverting attention reaches preset threshold,
It include: the detection frequency for obtaining driver head's posture and sight watching area；According to the detection frequency, the triggering of diverting attention
Time and attenuation coefficient obtain first moment accumulative value of diverting attention.
Such as: the first moment accumulative value of diverting attention, formula can be obtained by following formula are as follows:
Dt=(Dt-1+d_t/ (T*f)) * exp (- γ * (1-d_t)/(T*f)),
Wherein, γ is adjustment speed parameter, γ be located at [0,1) between, exp (- γ/(T*f)) is attenuation coefficient, Dt the
The value of diverting attention that one moment added up, d_t are the value of diverting attention at the first moment.In above-mentioned formula, exponential term is to divert attention to be worth dynamic regulation
, f is detection frequency, and the triggered time of diverting attention is T, it may be assumed that triggers the time threshold (dispersion attention time) diverted attention.
That is: when a certain moment divert attention value be 1 when, Dt is accumulated with linear velocity；When a certain moment divert attention value be 0 when, add up
For value of diverting attention by rapid decay, attenuation coefficient is exp (- γ/(T*f)), realizes the fast quick-recovery response for state of diverting attention.Thus can
Can be precisely determined whether driver is in driving status of really diverting attention by accumulative value of diverting attention, when driver is short
When time watches instrument board or rearview mirror attentively, driving status of diverting attention can't be determined as by accumulative value of diverting attention immediately, thus
It avoids judging or reducing probability of miscarriage of justice by accident.
That is, such as very short time, accumulative value of diverting attention can be rapid when driver distraction's attention in the short time
Increase, but not be determined as driving behavior of diverting attention immediately, and when driver be restored in time focus on driving when, it is accumulative
Value of diverting attention will decay rapidly, such as in a manner of exponential damping.It is thereby achieved that the quick response diverted attention.
It should be noted that in above-mentioned formula, decay value of diverting attention using exponential damping mode, it may be assumed that as driver and
When be restored to when focusing on driving, accumulative value of diverting attention will be decayed in a manner of exponential damping.But show other
In example, it can also be decayed using modes such as linear attenuation, polynomial decays to the value of diverting attention of accumulation.
Within the triggered time of diverting attention, if Dt reaches preset threshold, it is determined that driver is in driving status of diverting attention.Its
In, preset threshold can demarcate to obtain in advance, specific value can according to experiment determine or it is empirically determined.
Further, after determining that driver is in driving status of diverting attention, further includes: carry out alarm to driver and mention
Show.Such as: auditory tone cues or light flash prompt, safety belt vibration etc. are carried out to driver.
The system for distraction of driver detection method of the embodiment of the present application accurately regular inspection can measure whether driver diverts attention, effectively
The generation of error detection is avoided or reduced, in turn, is able to ascend the safety and reliability of driving.
Fig. 4 is the structural block diagram according to the system for distraction of driver detection system of the application one embodiment.As shown in figure 4, root
According to the system for distraction of driver detection system 400 of the application one embodiment, comprising: obtain module 410, detection module 420 and judgement
Module 430.
Wherein, it obtains module 410 and is used to obtain the video image of vehicle driver's human face region and the driving status of vehicle.
Detection module 420 is used to carry out head pose detection and line-of-sight detection according to video image, obtains head pose information and sight
Information.Judgment module 430 is used to judge whether driver divides in conjunction with driving status using head pose information and sight information
Mind.
The system for distraction of driver detection system of the embodiment of the present application accurately regular inspection can measure whether driver diverts attention, effectively
The generation of error detection is avoided or reduced, in turn, is able to ascend the safety and reliability of driving.
In one embodiment of the application, the driving status includes state of rolling backward, excessively curved state, the state that turns around or stifled
When car state, the judgment module 430 judges whether driver diverts attention according to the head pose information.
In one embodiment of the application, the judgment module 430 turns for determining that vehicle is according to transport condition
Curved or speed be less than preset vehicle speed threshold value when, using head pose information and sight information judge driver watch attentively it is preset just
Whether the time for often driving watching area is longer than the predetermined time；If it is, determining that driver is in driving condition of diverting attention.
In one embodiment of the application, the judgment module 430 is used to determine vehicle according to current driving status
It in forward travel state and when speed is greater than preset vehicle speed threshold value, obtains present frame and diverts attention value, and diverted attention value, described according to present frame
The accumulated time that the video frame rate of video image and triggering are diverted attention calculates the accumulative value of diverting attention in the accumulated time, and according to institute
It states accumulative value of diverting attention and judges whether driver diverts attention.
In one embodiment of the application, the head pose information includes head yaw angle and head pitch angle, root
Determine that preset normal driving watching area includes first area and second area according to sight information, then present frame diverts attention value really
If determining mode includes: that the head yaw angle exceeds yaw angle threshold value and/or the head pitch angle beyond pitch angle
Threshold value, then present frame value of diverting attention is first to divert attention and be worth threshold value；If the head yaw angle is without departing from yaw angle threshold value
And/or the head pitch angle is without departing from pitch angle threshold value, when driver watches first area attentively, the present frame is diverted attention value
Divert attention for preset first area and is worth accumulative parameter；When driver watches second area attentively the present frame divert attention value be it is preset
Second area, which is diverted attention, is worth accumulative parameter；When driver watches improper driving watching area attentively the present frame divert attention value be first
Divert attention and is worth threshold value；When driver watches normal driving watching area attentively, present frame value of diverting attention is second to divert attention and be worth threshold value；Its
In, first, which diverts attention, is worth threshold value and diverts attention greater than second and be worth threshold value, first area divert attention be worth accumulative parameter and second area divert attention be worth it is tired
Meter parameter, which is both greater than second and diverts attention to be worth threshold value and divert attention less than first, is worth threshold value.
In one embodiment of the application, in the calculating process of accumulative value of diverting attention, if present frame is diverted attention, value reaches first
Divert attention and be worth threshold value, then accumulative value of diverting attention is accumulated with linear velocity.
In one embodiment of the application, in the calculating process of accumulative value of diverting attention, if present frame is diverted attention, value reaches second
Divert attention and be worth threshold value, then adds up to divert attention value in a manner of any one including exponential damping, linear attenuation or polynomial decay
Decaying.
In one embodiment of the application, the judgment module 430 is used to reach accumulative point in the accumulative value of diverting attention
When mind value threshold value, determine that driver is in driving condition of diverting attention.
It should be noted that the specific implementation of the system for distraction of driver detection system of the embodiment of the present application and the application are real
The specific implementation for applying the system for distraction of driver detection method of example is similar, specifically refers to the description of method part, does not do herein
It repeats.
Further, embodiments herein discloses a kind of vehicle, comprising: one or more processors；It deposits thereon
The one or more machine readable medias for containing instruction, when being executed by one or more of processors, so that the vehicle
Execute the system for distraction of driver detection method according to any one above-mentioned embodiment.The vehicle accurately regular inspection can measure driving
Whether member diverts attention, and effectively avoids or reduces the generation of error detection, in turn, is able to ascend the safety and reliability of driving.
In addition, for those of ordinary skill in the art according to other compositions of the vehicle of the embodiment of the present application and effect
For be all it is known, do not repeat them here.
Further, embodiments herein discloses a kind of machine readable media, is stored thereon with instruction, when by one
Or multiple processors are when executing, so that the processor executes the inspection of the system for distraction of driver according to any one above-mentioned embodiment
Survey method.
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means specific features, structure, material or spy described in conjunction with this embodiment or example
Point is contained at least one embodiment or example of the application.In the present specification, schematic expression of the above terms are not
It is that must be directed to identical embodiment or example.Moreover, particular features, structures, materials, or characteristics described can be
It can be combined in any suitable manner in any one or more embodiment or examples.In addition, without conflicting with each other, this field
Technical staff can carry out the feature of different embodiments or examples described in this specification and different embodiments or examples
Combination and combination.
In the description of the present application, it should be noted that unless otherwise specified and limited, term " installation ", " connected ",
" connection " shall be understood in a broad sense, for example, it may be mechanical connection or electrical connection, the connection being also possible to inside two elements can
, can also indirectly connected through an intermediary, for the ordinary skill in the art to be to be connected directly, it can basis
Concrete condition understands the concrete meaning of above-mentioned term
Although embodiments herein has been shown and described above, it is to be understood that above-described embodiment is example
Property, it should not be understood as the limitation to the application, those skilled in the art within the scope of application can be to above-mentioned
Embodiment is changed, modifies, replacement and variant.
Claims (11)
1. a kind of system for distraction of driver detection method, which comprises the following steps:
Obtain the video image of vehicle driver's human face region and the driving status of vehicle；
Head pose detection and line-of-sight detection are carried out according to video image, obtains head pose information and sight information；
Judge whether driver diverts attention in conjunction with driving status using head pose information and sight information.
2. system for distraction of driver detection method according to claim 1, which is characterized in that the driving status includes reversing shape
When state, excessively curved state, the state that turns around or traffic congestion state, judge whether driver diverts attention according to the head pose information.
3. system for distraction of driver detection method according to claim 1, which is characterized in that it is described using head pose information and
Sight information judges whether driver diverts attention in conjunction with driving status, comprising:
When determining that vehicle is in turning or speed is less than preset vehicle speed threshold value according to transport condition, using head pose information and
Sight information judges that driver watches whether time of preset normal driving watching area is longer than the predetermined time attentively；
If it is, determining that driver is in driving condition of diverting attention.
4. system for distraction of driver detection method according to claim 1, which is characterized in that it is described using head pose information and
Sight information judges whether driver diverts attention in conjunction with driving status, comprising:
When determining that vehicle is in forward travel state and speed is greater than preset vehicle speed threshold value according to current driving status, present frame is obtained
Divert attention value；
The accumulated time is calculated according to the present frame accumulated time that value, the video frame rate of the video image and triggering are diverted attention of diverting attention
Interior accumulative value of diverting attention；
Judge whether driver diverts attention according to the accumulative value of diverting attention.
5. system for distraction of driver detection method according to claim 4, which is characterized in that the head pose information includes head
Portion's yaw angle and head pitch angle determine that preset normal driving watching area includes first area and second according to sight information
Region, then the divert attention method of determination of value of present frame include:
If the head yaw angle exceeds pitch angle threshold value beyond yaw angle threshold value and/or the head pitch angle,
Present frame value of diverting attention is first to divert attention and be worth threshold value；
If the head yaw angle is without departing from yaw angle threshold value and/or the head pitch angle without departing from pitch angle threshold
Value, when driver watches first area attentively, present frame value of diverting attention diverts attention for preset first area and is worth accumulative parameter；It is driving
The person of sailing when watching second area attentively present frame value of diverting attention be that preset second area diverts attention and is worth accumulative parameter；Watch attentively in driver
Present frame value of diverting attention is first to divert attention and be worth threshold value when improper driving watching area；Watch normal driving attentively in driver to watch attentively
Present frame value of diverting attention is second to divert attention and be worth threshold value when region；
Wherein, first diverts attention and is worth threshold value and diverts attention greater than second and be worth threshold value, and first area, which is diverted attention, is worth accumulative parameter and second area point
The accumulative parameter of refreshing value, which is both greater than second, diverts attention to be worth threshold value and divert attention less than first and is worth threshold value.
6. system for distraction of driver detection method according to claim 5, which is characterized in that the calculating process of accumulative value of diverting attention
In, if present frame is diverted attention, value, which reaches first and diverts attention, is worth threshold value, and accumulative value of diverting attention is accumulated with linear velocity.
7. system for distraction of driver detection method according to claim 5, which is characterized in that the calculating process of accumulative value of diverting attention
In, if present frame is diverted attention, value, which reaches second and diverts attention, is worth threshold value, accumulative to divert attention value to include exponential damping, linear attenuation or multinomial
Any one mode including formula decaying decays.
8. system for distraction of driver detection method according to claim 4, which is characterized in that described according to the accumulative value of diverting attention
Judge whether driver diverts attention, comprising: the accumulative value of diverting attention reaches accumulative and diverts attention when being worth threshold value, determines that driver is in and diverts attention
Driving condition.
9. a kind of system for distraction of driver detection system characterized by comprising
Module is obtained, for obtaining the video image of vehicle driver's human face region and the driving status of vehicle；
Detection module obtains head pose information and view for carrying out head pose detection and line-of-sight detection according to video image
Line information；
Judgment module, for judging whether driver diverts attention in conjunction with driving status using head pose information and sight information.
10. a kind of vehicle characterized by comprising
One or more processors；With
One or more machine readable medias of instruction are stored thereon with, when being executed by one or more of processors, are made
It obtains the vehicle and executes system for distraction of driver detection method according to claim 1-8.
11. a kind of machine readable media, which is characterized in that it is stored thereon with instruction, when executed by one or more processors,
So that the processor executes system for distraction of driver detection method according to claim 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910641923.8A CN110390285A (en) | 2019-07-16 | 2019-07-16 | System for distraction of driver detection method, system and vehicle |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910641923.8A CN110390285A (en) | 2019-07-16 | 2019-07-16 | System for distraction of driver detection method, system and vehicle |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110390285A true CN110390285A (en) | 2019-10-29 |
Family
ID=68286618
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910641923.8A Pending CN110390285A (en) | 2019-07-16 | 2019-07-16 | System for distraction of driver detection method, system and vehicle |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110390285A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110909718A (en) * | 2019-12-11 | 2020-03-24 | 深圳市锐明技术股份有限公司 | Driving state identification method and device and vehicle |
| CN111325172A (en) * | 2020-02-28 | 2020-06-23 | 英华达（上海）科技有限公司 | Device and method for achieving driving habit formation by sight line tracking |
| CN111391770A (en) * | 2020-04-03 | 2020-07-10 | 广州小鹏汽车科技有限公司 | Interactive region shielding method, device, vehicle and machine readable medium |
| CN111539333A (en) * | 2020-04-24 | 2020-08-14 | 湖北亿咖通科技有限公司 | Method for identifying gazing area and detecting distraction of driver |
| CN112849149A (en) * | 2021-01-28 | 2021-05-28 | 上海商汤临港智能科技有限公司 | Driver state detection method, driver state detection device, driver state detection apparatus, driver state detection medium, and driver state detection program product |
| CN112883834A (en) * | 2021-01-29 | 2021-06-01 | 重庆长安汽车股份有限公司 | DMS system distraction detection method, DMS system distraction detection system, DMS vehicle, and storage medium |
| CN113298041A (en) * | 2021-06-21 | 2021-08-24 | 黑芝麻智能科技(上海)有限公司 | Method and system for calibrating driver distraction reference direction |
| CN113313019A (en) * | 2021-05-27 | 2021-08-27 | 展讯通信(天津)有限公司 | Distracted driving detection method, system and related equipment |
| CN113569785A (en) * | 2021-08-04 | 2021-10-29 | 上海汽车集团股份有限公司 | Driving state sensing method and device |
| CN113696897A (en) * | 2020-05-07 | 2021-11-26 | 沃尔沃汽车公司 | Driver distraction early warning method and driver distraction early warning system |
| CN114475426A (en) * | 2022-01-29 | 2022-05-13 | 深圳智慧车联科技有限公司 | Muck vehicle right-turning early warning reminding system and reminding method |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101540090A (en) * | 2009-04-14 | 2009-09-23 | 华南理工大学 | Driver fatigue monitoring device based on multivariate information fusion and monitoring method thereof |
| CN102510480A (en) * | 2011-11-04 | 2012-06-20 | 大连海事大学 | Automatic calibrating and tracking system of driver sight line |
| CN102963298A (en) * | 2012-09-29 | 2013-03-13 | 重庆长安汽车股份有限公司 | Abnormal behavior monitoring method based on sight detection |
| CN104599443A (en) * | 2015-01-12 | 2015-05-06 | 江苏省交通规划设计院股份有限公司 | Vehicle-mounted forewarning terminal for driving behaviors based on information fusion and forewarning method thereof |
| US20160180677A1 (en) * | 2014-12-18 | 2016-06-23 | Ford Global Technologies, Llc | Apparatus for reducing driver distraction via short range vehicle communication |
| CN105764735A (en) * | 2013-10-29 | 2016-07-13 | 金在哲 | Two-step sleepy driving prevention apparatus through recognizing operation, front face, eye, and mouth shape |
| CN106446771A (en) * | 2015-08-12 | 2017-02-22 | 福特全球技术公司 | Driver attention evaluation |
| CN107334481A (en) * | 2017-05-15 | 2017-11-10 | 清华大学 | One kind drives divert one's attention detection method and system |
| CN107729870A (en) * | 2017-01-24 | 2018-02-23 | 问众智能信息科技（北京）有限公司 | The method and apparatus of in-car safety monitoring based on computer vision |
| CN108256487A (en) * | 2018-01-19 | 2018-07-06 | 北京工业大学 | A kind of driving state detection device and method based on reversed binocular |
| CN108819900A (en) * | 2018-06-04 | 2018-11-16 | 上海商汤智能科技有限公司 | Control method for vehicle and system, vehicle intelligent system, electronic equipment, medium |
| CN109002757A (en) * | 2018-06-04 | 2018-12-14 | 上海商汤智能科技有限公司 | Drive management method and system, vehicle intelligent system, electronic equipment, medium |
| CN109774598A (en) * | 2017-11-15 | 2019-05-21 | 欧姆龙株式会社 | Divert one's attention decision maker, divert one's attention determination method and recording medium |
-
2019
- 2019-07-16 CN CN201910641923.8A patent/CN110390285A/en active Pending
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101540090A (en) * | 2009-04-14 | 2009-09-23 | 华南理工大学 | Driver fatigue monitoring device based on multivariate information fusion and monitoring method thereof |
| CN102510480A (en) * | 2011-11-04 | 2012-06-20 | 大连海事大学 | Automatic calibrating and tracking system of driver sight line |
| CN102963298A (en) * | 2012-09-29 | 2013-03-13 | 重庆长安汽车股份有限公司 | Abnormal behavior monitoring method based on sight detection |
| CN105764735A (en) * | 2013-10-29 | 2016-07-13 | 金在哲 | Two-step sleepy driving prevention apparatus through recognizing operation, front face, eye, and mouth shape |
| US20160180677A1 (en) * | 2014-12-18 | 2016-06-23 | Ford Global Technologies, Llc | Apparatus for reducing driver distraction via short range vehicle communication |
| CN104599443A (en) * | 2015-01-12 | 2015-05-06 | 江苏省交通规划设计院股份有限公司 | Vehicle-mounted forewarning terminal for driving behaviors based on information fusion and forewarning method thereof |
| CN106446771A (en) * | 2015-08-12 | 2017-02-22 | 福特全球技术公司 | Driver attention evaluation |
| CN107729870A (en) * | 2017-01-24 | 2018-02-23 | 问众智能信息科技（北京）有限公司 | The method and apparatus of in-car safety monitoring based on computer vision |
| CN107334481A (en) * | 2017-05-15 | 2017-11-10 | 清华大学 | One kind drives divert one's attention detection method and system |
| CN109774598A (en) * | 2017-11-15 | 2019-05-21 | 欧姆龙株式会社 | Divert one's attention decision maker, divert one's attention determination method and recording medium |
| CN108256487A (en) * | 2018-01-19 | 2018-07-06 | 北京工业大学 | A kind of driving state detection device and method based on reversed binocular |
| CN108819900A (en) * | 2018-06-04 | 2018-11-16 | 上海商汤智能科技有限公司 | Control method for vehicle and system, vehicle intelligent system, electronic equipment, medium |
| CN109002757A (en) * | 2018-06-04 | 2018-12-14 | 上海商汤智能科技有限公司 | Drive management method and system, vehicle intelligent system, electronic equipment, medium |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110909718A (en) * | 2019-12-11 | 2020-03-24 | 深圳市锐明技术股份有限公司 | Driving state identification method and device and vehicle |
| CN110909718B (en) * | 2019-12-11 | 2024-04-19 | 深圳市锐明技术股份有限公司 | Driving state identification method and device and vehicle |
| CN111325172A (en) * | 2020-02-28 | 2020-06-23 | 英华达（上海）科技有限公司 | Device and method for achieving driving habit formation by sight line tracking |
| CN111391770A (en) * | 2020-04-03 | 2020-07-10 | 广州小鹏汽车科技有限公司 | Interactive region shielding method, device, vehicle and machine readable medium |
| CN111539333B (en) * | 2020-04-24 | 2021-06-29 | 湖北亿咖通科技有限公司 | Method for identifying gazing area and detecting distraction of driver |
| CN111539333A (en) * | 2020-04-24 | 2020-08-14 | 湖北亿咖通科技有限公司 | Method for identifying gazing area and detecting distraction of driver |
| CN113696897A (en) * | 2020-05-07 | 2021-11-26 | 沃尔沃汽车公司 | Driver distraction early warning method and driver distraction early warning system |
| CN113696897B (en) * | 2020-05-07 | 2023-06-23 | 沃尔沃汽车公司 | Driver distraction early warning method and driver distraction early warning system |
| CN112849149A (en) * | 2021-01-28 | 2021-05-28 | 上海商汤临港智能科技有限公司 | Driver state detection method, driver state detection device, driver state detection apparatus, driver state detection medium, and driver state detection program product |
| CN112883834A (en) * | 2021-01-29 | 2021-06-01 | 重庆长安汽车股份有限公司 | DMS system distraction detection method, DMS system distraction detection system, DMS vehicle, and storage medium |
| CN113313019A (en) * | 2021-05-27 | 2021-08-27 | 展讯通信(天津)有限公司 | Distracted driving detection method, system and related equipment |
| CN113298041A (en) * | 2021-06-21 | 2021-08-24 | 黑芝麻智能科技(上海)有限公司 | Method and system for calibrating driver distraction reference direction |
| CN113569785A (en) * | 2021-08-04 | 2021-10-29 | 上海汽车集团股份有限公司 | Driving state sensing method and device |
| CN114475426A (en) * | 2022-01-29 | 2022-05-13 | 深圳智慧车联科技有限公司 | Muck vehicle right-turning early warning reminding system and reminding method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110390285A (en) | System for distraction of driver detection method, system and vehicle | |
| JP5171629B2 (en) | Driving information providing device | |
| US10717447B2 (en) | Visual perception assistance system and visual-perception target object detection system | |
| EP2830029B1 (en) | A method for monitoring a vehicle driving state and a vehicle navigation device for implementing the method | |
| US8630793B2 (en) | Vehicle controller | |
| CN112092813A (en) | Vehicle control method, device, electronic device and storage medium | |
| EP1726513A1 (en) | Driving assistance system for lane keeping support, for lane change assistance, and for driver status monitoring for a vehicle | |
| JP6545940B2 (en) | Factor analysis apparatus and factor analysis method | |
| DE102004057947B4 (en) | Method for functional testing of a driver assistance system integrated on the vehicle side | |
| US10558215B2 (en) | Vehicle drive assistance system | |
| EP1557321A1 (en) | Vehicle driver warning procedure | |
| JP4696792B2 (en) | Car alarm system | |
| US20170178509A1 (en) | Method and apparatus for operating a driver assistance system, driver assistance system | |
| JP2016502165A (en) | Driver attention detection method and apparatus | |
| CN105620359B (en) | A kind of lane departure warning method and system based on mobile terminal | |
| CN108698606A (en) | Method for driver's movement of classifying | |
| CN105931430A (en) | Alarm sensitivity detection method and apparatus for driver state early warning system | |
| CN108482367A (en) | A kind of method, apparatus and system driven based on intelligent back vision mirror auxiliary | |
| DE102015226847A1 (en) | Method and device for assisting a driver | |
| CN114872713A (en) | Device and method for monitoring abnormal driving state of driver | |
| JP4186749B2 (en) | Armpit judging device | |
| CN112428953A (en) | Blind area monitoring alarm method and device | |
| CN115384540A (en) | Vehicle-mounted safe driving detection device | |
| CN115366907A (en) | Driver state abnormity prompting method and device, vehicle and storage medium | |
| CN108045311A (en) | A kind of angle of rear-view mirror of vehicle regulating system and adjusting method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191029 |