CN103942527A - Method for determining eye-off-the-road condition by using road classifier - Google Patents
Method for determining eye-off-the-road condition by using road classifier Download PDFInfo
- Publication number
- CN103942527A CN103942527A CN201410021671.6A CN201410021671A CN103942527A CN 103942527 A CN103942527 A CN 103942527A CN 201410021671 A CN201410021671 A CN 201410021671A CN 103942527 A CN103942527 A CN 103942527A
- Authority
- CN
- China
- Prior art keywords
- driver
- road
- catching
- view data
- eyes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 239000011521 glass Substances 0.000 claims abstract description 31
- 238000012706 support-vector machine Methods 0.000 claims abstract description 13
- 230000000007 visual effect Effects 0.000 claims description 40
- 230000001815 facial effect Effects 0.000 claims description 29
- 230000004438 eyesight Effects 0.000 claims description 28
- 210000003128 head Anatomy 0.000 claims description 23
- 238000012549 training Methods 0.000 claims description 18
- 238000005070 sampling Methods 0.000 claims description 17
- 238000013075 data extraction Methods 0.000 claims description 12
- 238000013139 quantization Methods 0.000 claims description 9
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 238000001514 detection method Methods 0.000 description 19
- 230000008569 process Effects 0.000 description 11
- 238000010586 diagram Methods 0.000 description 9
- 238000012544 monitoring process Methods 0.000 description 6
- 238000000605 extraction Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 241000894007 species Species 0.000 description 3
- 241000287181 Sturnus vulgaris Species 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000012876 topography Methods 0.000 description 2
- 206010057315 Daydreaming Diseases 0.000 description 1
- 208000024754 bloodshot eye Diseases 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 201000005111 ocular hyperemia Diseases 0.000 description 1
- 238000007639 printing Methods 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
- B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/163—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state by tracking eye movement, gaze, or pupil change
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/18—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state for vehicle drivers or machine operators
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
- B60W30/08—Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
- B60W30/09—Taking automatic action to avoid collision, e.g. braking and steering
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
- B60W30/08—Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
- B60W30/095—Predicting travel path or likelihood of collision
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2540/00—Input parameters relating to occupants
- B60W2540/221—Physiology, e.g. weight, heartbeat, health or special needs
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- Mechanical Engineering (AREA)
- Transportation (AREA)
- Automation & Control Theory (AREA)
- Pathology (AREA)
- Molecular Biology (AREA)
- Psychology (AREA)
- Psychiatry (AREA)
- Biophysics (AREA)
- Hospice & Palliative Care (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Social Psychology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Educational Technology (AREA)
- Child & Adolescent Psychology (AREA)
- Developmental Disabilities (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The method involves acquiring image data corresponding to a driver of a monocular camera apparatus (10), and detecting whether the driver is wearing glasses based on the data using an eyeglass classifier i.e. linear multi-class support vector machine (SVM) classifier. A location of the driver is detected from the detected face image data while the driver is wearing glasses. By using an eye-offside-the-road (EOTR) classifier determining whether the EOTR condition exists based on the location of the driver face. A warning system is released to warn the driver while the driver is inattentive.
Description
The cross reference of related application
Present patent application requires the right of priority of the U.S. Provisional Application No. 61/754,515 submitting on January 18th, 2013, and this application is incorporated herein by reference.
Technical field
The disclosure relates to monitor vehicle driver and determines whether driver's sight line leaves the road.
Background technology
Statement in this part only provides the background information relevant with the disclosure.Therefore, such statement is not intended to form admission of prior art.
Can monitor the operator of vehicle and detect the vehicle whether operator do not note road scene and allow to take measures to prevent not note because of operator the vehicle collision causing.For example, warning system can enable to remind driver that he or she is absent minded.In addition when, Braking mode and automatic steering system can be enabled to also fail to focus one's attention on after definite driver is even being warned by vehicle stop.
Known and used driver to monitor camera head, this device is configured to monitor driver and detects eyes (EOTR) state that leaves the road, to point out that according to driver's direction of gaze of estimating driver's eyes leave the road.But in the time of driver's wearing spectacles, due to unreliable to the estimation of driver's direction of gaze, performance can reduce.Equally, in the time that driver wears sunglasses, cannot obtain the estimation to driver's direction of gaze.
Summary of the invention
A kind of exist the leave the road method of (EOTR) state of eyes to comprise from monocular camera head and catch the view data corresponding to driver for determining.To the whether detection of the wearing spectacles view data based on using glasses sorter of driver.In the time driver's wearing spectacles being detected, detect driver's facial positions from the view data of catching, and use EOTR sorter to determine whether to exist EOTR state based on driver's facial positions.
The present invention relates to following technical proposal.
1. for determining whether to exist the eyes method for state that leaves the road, comprising:
Catch the view data corresponding to driver from monocular camera head;
Use glasses sorters to detect whether wearing spectacles of described driver based on described view data; And
In the time described driver's wearing spectacles being detected:
Detect driver's facial positions from described view data of catching; And
Use the eyes sorter that leaves the road to determine whether to exist the described eyes state that leaves the road based on described driver's facial positions.
2. according to the method described in technical scheme 2, whether wearing spectacles comprises wherein to use described glasses sorter to detect described driver based on described view data:
From described image data extraction facial characteristics of catching;
Use the dictionary of the multiple vision words that obtained by cluster routine to carry out the visual signature extracting described in quantization;
The described quantized visual signature of pondization is to generate the spatial histogram of described vision word; And
Use described glasses sorter that described spatial histogram is classified to detect whether wearing spectacles of described driver.
3. according to the method described in technical scheme 2, wherein said visual signature is in the following manner from described image data extraction of catching:
In the dense grid of the view data of catching described in intensive yardstick invariant features conversion descriptor is applied to.
4. according to the method described in technical scheme 2, wherein said cluster routine comprises the average cluster routine of k-.
5. according to the method described in technical scheme 1, wherein said glasses sorter comprises multi-category support vector machines linear classifier.
6. according to the method described in technical scheme 5, also comprise:
Use the image of multiple equally distributed training to train described multi-category support vector machines linear classifier, wherein the image of each training comprises the driver corresponding to the corresponding sampling of one of three classifications.
7. according to the method described in technical scheme 6, wherein said three classifications comprise that the driver that the driver of described sampling does not wear glasses, the driver of described sampling wears common spectacles and described sampling wears sunglasses.
8. according to the method described in technical scheme 1, wherein use the described eyes sorter that leaves the road to determine whether to exist the described eyes state of leaving the road to comprise based on described driver's facial positions:
From described image data extraction visual signature of catching;
The visual signature extracting described in the dictionary quantization of multiple vision words that use is obtained by cluster routine;
The described quantized visual signature of pondization is to generate at least one spatial histogram of described vision word;
Generate the proper vector of described at least one spatial histogram of the described vision word being connected in series with described driver's facial positions; And
Use the described eyes sorter that leaves the road that described proper vector is classified to determine whether to exist described EOR state.
9. according to the method described in technical scheme 8, wherein said visual signature is in the following manner from described image data extraction of catching:
In the dense grid of the view data of catching described in intensive yardstick invariant features conversion descriptor is applied to.
10. according to the method described in technical scheme 8, wherein said cluster routine comprises the average cluster routine of k-.
11. according to the method described in technical scheme 8, and wherein the described quantized view data of pondization comprises with the described spatial histogram that generates described vision word:
The described quantized view data of pondization is to generate multiple spatial histograms of described vision word.
12. according to the method described in technical scheme 8, and described multiple spatial histograms of wherein said vision word generate in the following manner:
Described view data of catching is divided into the subregion of more and more refinement; And
Subregion based on described more and more refinement generates described multiple spatial histograms, and wherein every sub regions comprises corresponding spatial histogram in described multiple spatial histogram.
13. according to the method described in technical scheme 8, wherein described proper vector is classified to determine whether to exist described EOR state to comprise:
Described proper vector is classified to extract to the pose information of the facial positions detecting described in being used for.
14. according to the method described in technical scheme 1, and the wherein said eyes sorter that leaves the road comprises scale-of-two support vector machine linear classifier.
15. according to the method described in technical scheme 14, also comprises:
Use the image of multiple equally distributed training to train described scale-of-two support vector machine linear classifier, wherein the image of each training comprises wearing spectacles and the face-image corresponding to the corresponding sampling of one of two classifications.
16. according to the method described in technical scheme 15, and wherein said two classifications comprise the face-image of the face-image of indicating driver's eyes to stare at the corresponding sampling on road surface and the corresponding sampling of indicating described driver's eyes to leave the road.
17. according to the method described in technical scheme 15, and the Part I of the image of wherein said multiple training is hunted down by day and the residue Part II of the image of described multiple training was hunted down at night.
18. according to the method described in technical scheme 1, also comprises:
Only in the time described driver's wearing spectacles not detected, the direction of gaze of the described driver based on estimating determines whether to exist the described eyes state that leaves the road.
19. 1 kinds for determining whether to exist the eyes equipment of state that leaves the road, and comprising:
Monocular camera head in car, for catching the view data of the visual field of pointing to driver; And
Treating apparatus, it is configured to:
Use glasses sorters to detect whether wearing spectacles of described driver based on described view data; And
In the time described driver's wearing spectacles being detected:
Detect driver's facial positions from described view data of catching; And
Use the eyes sorter that leaves the road to determine whether to exist the described eyes state that leaves the road based on described driver's facial positions.
20. according to the equipment described in technical scheme 19, also comprises:
Infrared illuminator, it for illuminating driver under low illumination condition.
Brief description of the drawings
To describe one or more embodiment with reference to the accompanying drawings with way of example now, in the accompanying drawings:
Fig. 1 illustrates the exemplary non-limiting view according to the parts of the driver monitoring system for vehicle in vehicle of the present disclosure;
Fig. 2 illustrates the non-limiting view data according to the driver of vehicle of being caught by the camera head of Fig. 1 of the present disclosure;
Fig. 3 illustrates according to of the present disclosure for selecting to be used for determining whether to exist the eyes exemplary process diagram of one of two kinds of methods of (EOTR) state that leaves the road;
Fig. 4 illustrates the whether exemplary process diagram of the decision block 304 of Fig. 3 of wearing spectacles according to the driver for detection of Fig. 2 of the present disclosure;
Fig. 5 illustrates according to of the present disclosure for using EOTR detection of classifier whether to have the exemplary process diagram 500 of the execution of the direct EOTR detection method of the frame 308 of Fig. 3 of EOTR state; And
Fig. 6 illustrates according to of the present disclosure for building space pyramid visual word bag to generate the exemplary non-limiting example of multiple spatial histograms.
Embodiment
Referring now to accompanying drawing,, content shown in it is only not used in and limits this exemplary embodiment for some exemplary embodiment is shown, Fig. 1 illustrates the exemplary non-limiting view according to the parts of the driver monitoring system for vehicle in vehicle of the present disclosure.Driver monitoring system for vehicle comprises monocular-camera 10 in car, and it is configured to catch the view data in the visual field (FOV) of pointing to vehicle driver.The view data of catching comprises video flowing, and it comprises multiple picture frames of catching continuously.Camera head 10 can receive light or other radiation, and uses the electric signal that in for example charge-coupled device (CCD) (CCD) sensor or complementary metal oxide semiconductor (CMOS) (CMOS) sensor is pixel format by light energy conversion.Camera head 10 and non-instantaneous treating apparatus 15 signal communication, treating apparatus 15 be configured to receive the view data of catching output to driver whether wearing spectacles detection and to whether there is the leave the road judgement of (EOTR) state of eyes.As used herein, term " EOTR state " refers to the judgement on road/Driving Scene to the current out-focus of driver's eyes.Treating apparatus 15 can be embodied in any suitable compartment of the vehicle that can receive the image input data of being caught by camera head 10.Camera head 10 is arranged in the inside of vehicle.In one embodiment, camera head 10 is arranged on the meter panel of motor vehicle of steering wheel shaft top.Driver monitoring system for vehicle also comprises infrared illuminator 12, and it is configured to project infrared light in the direction towards driver, makes to be obtained by camera head 10 picture rich in detail of driver's face under the low illumination condition such as night.Direct light source is different from using, and infrared light does not affect driver's eyesight.There is not " blood-shot eye illness " phenomenon producing in the view data of catching in addition, in the time using near-infrared light source.In one embodiment, camera head does not comprise the infrared filter that stops predetermined wavelength infrared light in addition.Embodiment herein relates to and uses the view data of being caught by camera head 10 to detect whether to have EOTR state, even in the time of driver's wearing spectacles, and do not use the input of high spatial and temporal resolution, and therefore eliminated the demand to expensive video camera and lens.
Control module, module, control, controller, control module, processor and similarly term represent one or more any or the various combination in following: (multiple) special IC (ASIC), (multiple) electronic circuit, (multiple) central processing unit (preferably (multiple) microprocessor) and the storer being associated and the memory storage of the one or more software of execution or firmware program or routine are (read-only, able to programme read-only, random access, hard disk drive etc.), (multiple) combinational logic circuit, (multiple) input/output circuitry and device, suitable Signal Regulation and buffer circuit, and provide other parts of described function.Software, firmware, program, instruction, routine, code, algorithm and similar terms represent to comprise any instruction set of correction card and look-up table.One group of control routine for providing required function to carry out is provided control module.Routine is for example carried out by central processing unit, and operation is for monitoring the input from sensing apparatus and other networking control module, and carry out control and diagnostics routines with the operation of control actuator.Routine can be carried out at regular intervals, for example, during afoot engine and vehicle operating, carries out once every 3.125,6.25,12.5,25 and 100 milliseconds.Alternatively, can be in response to the generation of event executive routine.
Fig. 2 illustrates the non-limiting view data according to the driver who is caught by the camera head of Fig. 1 of the present disclosure.In the illustrated embodiment, driver's wearing spectacles 50.As used herein, term " glasses " refers to corrective glasses, sunglasses, safety goggles, the safety goggles of any type or comprises the glasses of any other form of the lens of the eyes that cover driver.Region 20 comprises EOTR region, and it comprises at least one view data of the information of the eyes for monitoring driver's head position, face feature point and driver.In one embodiment, EOTR region is monitored with the face from driver and extracts visual signature to allow driver's face to follow the tracks of, and wherein the information of driver's eyes can be extracted from face is followed the tracks of.The information of driver's eyes can be finally for estimating watching position attentively and determining EOTR state from it of driver.But, in the time of driver's wearing spectacles, because the ability of information of the eyes that extract driver is subject to the restriction that face is followed the tracks of, the detection that can lead to errors of the EOTR state of watching position attentively based on estimating.Therefore, the existence of determining EOTR state need to be known whether wearing spectacles of driver, makes to select suitable method to determine the existence of EOTR state.
Fig. 3 illustrates according to of the present disclosure for selecting the exemplary process diagram 300 for determining whether to exist one of two kinds of methods of EOTR state.Exemplary process diagram 300 can be in the interior enforcement of non-instantaneous treating apparatus 15 of Fig. 1 and by its execution.Referring to frame 302, catch by the camera head 10 of Fig. 1 corresponding to driver's view data.Decision block 304 is used glasses sorter to detect whether wearing spectacles of driver based on view data.Do not wear glasses (as represented by " 0 ") during when decision block 304 detects driver, at frame, 306 places carry out the EOTR detection method based on watching attentively.If when decision block 304 detects driver's wearing spectacles (as represented by " 1 "), carry out the direct EOTR detection method of the pose information that uses the extraction from driver of being classified by EOTR sorter at frame 308 places.In the time driver's wearing spectacles being detected, embodiment herein relates to direct EOTR detection method.In the direct EOTR detection method of frame 306, can directly for example, detect EOTR state from the visual signature (driver facial characteristics) extracting, and not rely on the estimation of the direction of gaze to driver.Therefore, decision block 304 is carried out glasses sorter to detect whether wearing spectacles of driver, and frame 308 is carried out EOTR sorter and determined whether to exist EOTR state with the facial positions based on driver.
Fig. 4 illustrates the whether exemplary process diagram 400 of the decision block 304 of Fig. 3 of wearing spectacles according to the driver for detection of Fig. 2 of the present disclosure.Exemplary process diagram 400 can be in the interior enforcement of non-instantaneous treating apparatus 15 of Fig. 1 and by its execution.Table 1 provides as the main points of Fig. 4, and wherein the frame with numeral number and corresponding function are described below.
Table 1
Referring to frame 402, obtain the input picture that comprises the view data corresponding to driver of being caught by the camera head 10 of Fig. 1.In one embodiment, input picture comprises the driver's face detecting.Driver may wear common spectacles, sunglasses, or may not wear any glasses.But, in frame 402, do not know whether driver wears common spectacles, sunglasses or do not wear any glasses.As used herein, term " common spectacles " refers to the glasses of any corrective, protectiveness or other type with printing opacity lens.
The image data extraction visual signature of frame 404 from catching.Visual signature is indicated the face feature point of the driver's face detecting.Comprise that the facial input picture detecting can be normalized.In non-limiting example, the face detecting is normalized, and for example, is for example resized, to 200 × 200 pixel squares (, image block).In certain embodiments, Visual Feature Retrieval Process comprises the face extraction dense feature from detecting by being applied to intensive yardstick invariant features conversion (SIFT) descriptor in dense grid in the view data of catching comprising the driver's face detecting.In non-limiting example, the value of the step-length of the feature of extracting and bin size (bin size) is made as respectively 2 and 4.The use of SIFT descriptor makes it possible to calculate the more big collection of topography's descriptor in dense grid, to provide than the more information of corresponding descriptor of evaluating in the picture point set sparse.
Referring to frame 406, the visual signature that uses the dictionary quantization of the multiple vision words that obtained by cluster routine to extract.Quantization is that the visual signature to extracting carries out cluster the cataloged procedure by its generating code.In one embodiment, the dictionary of multiple vision words comprises the visual dictionary of 500 words that use the average cluster routine of k.
Referring to frame 408, the quantized visual signature of frame 406 by pond to generate the spatial histogram of vision word.
Frame 410 uses glasses sorter that the spatial histogram of the frame of generation 408 is classified to detect whether wearing spectacles of driver.In the illustrated embodiment, glasses sorter comprises multi-category support vector machines (SVM) linear classifier.Many classification SVM linear classifiers can use the image (trained image) of multiple equally distributed training to train.The image of each training comprises the face-image corresponding to the corresponding sampling of one of three classifications, comprises the face of sampling: (1) not wearing spectacles, (2) wear common spectacles, and (3) wear sunglasses.Therefore, the image of training be uniformly distributed three equal parts in the image that is included in multiple training, wherein each part is corresponding in three classifications corresponding one.Some in the image of multiple training can be caught during low light photograph or nighttime driving condition.In addition, the face-image of sampling is selected from have the different multiple individualities that change from different races and head pose.
Frame 412 classifies detect driver whether wearing spectacles to spatial histogram based on frame 410 with glasses sorter.Spatial histogram can be classified as driver and not wear glasses 420, wears common spectacles 430 or wear sunglasses 440.In the time that spatial histogram is classified as driver and does not wear glasses, the frame 306 of Fig. 3 estimates to carry out the EOTR detection method based on watching attentively by the direction of gaze that utilizes driver, because can accurately obtain driver's eye information.In the time that spatial histogram is categorized as respectively driver and wears common spectacles 420 or sunglasses 430, frame 308 will use EOTR sorter to carry out direct EOTR detection method to determine whether sight deviating road of driver based on driver's facial positions.
Fig. 5 illustrates according to of the present disclosure for using EOTR detection of classifier whether to have the exemplary process diagram 500 of the execution of the direct EOTR detection method of the frame 308 of Fig. 3 of EOTR state.Exemplary process diagram 500 can be in the interior enforcement of non-instantaneous treating apparatus 15 of Fig. 1 and by its execution.Table 2 provides as the main points of Fig. 5, and wherein the frame with numeral number and corresponding function are described below.
Table 2
Should be appreciated that, in the time of driver's wearing spectacles, the frame 412 of for example Fig. 4 detects common spectacles 430 or sunglasses 440, the estimation of driver's direction of gaze can not obtain or be unreliable, because eye information is blocked because of the existence of glasses.Therefore, use the EOTR detection method detection EOTR state based on watching attentively to be avoided, and instead determine whether to exist EOTR state with the EOTR sorter of training in advance.As described in more detail below, EOTR sorter uses driver's facial characteristics of the image data extraction of catching from the monocular camera head 10 by Fig. 1 to export binary decision, for example, whether has EOTR state.
Referring to frame 502, comprise that the input picture of driver's view data is caught by the camera head 10 of Fig. 1.Big or small from image data extraction area-of-interest or adjustment at frame 504 places.Area-of-interest comprises the driver's face that uses face detector to detect.In non-limiting example, the output of the facial face detector that instruction detects is normalized, for example, for example adjust size, to 200 × 200 pixel squares (, image block).
The image data extraction visual signature of frame 506 from catching.The region of interesting extraction visual signature of the face feature point of driver's facial information particularly, is described from instruction.In certain embodiments, Visual Feature Retrieval Process comprises the face extraction dense feature from detecting by being applied to intensive yardstick invariant features conversion (SIFT) descriptor in dense grid in the view data of catching comprising the driver's face detecting.In non-limiting example, the step-length of the feature of extracting and the value of bin size are all made as 4 respectively.The use of SIFT descriptor makes it possible to calculate the more big collection of topography's descriptor in dense grid, to provide than the more information of corresponding descriptor of evaluating in the picture point set sparse.
Referring to frame 508, the visual signature that uses the dictionary quantization of the multiple vision words that obtained by cluster routine to extract.Quantization is that the visual signature to extracting carries out cluster the cataloged procedure by its generating code.In one embodiment, the dictionary of multiple vision words comprises the visual dictionary of 250 words that use the average cluster routine of k.
The quantized visual signature of frame 510 pondization is to generate at least one spatial histogram of vision word.At least one spatial histogram comprises the vision word that uses quantization visual signature.The spatial histogram feature of vision word because of the targeted species of for example face different because the discriminative information of targeted species is embedded in these features by the image similarity between measurement target kind and nontarget species class.Here, driver's pose information can be determined from the visual signature of the extraction of driver's face of detecting.In one embodiment, the quantized view data of pondization is used and comprises that the space pyramid visual word bag of multiple layers generates the spatial histogram of multiple vision words.Particularly, by the view data of catching is divided into the subregion of more and more refinement and based on this more and more the subregion of refinement generate multiple spatial histograms and generate multiple spatial histograms.Every sub regions comprises corresponding spatial histogram in multiple spatial histograms.The size of subregion depends on the number of plies using in the pyramid visual word bag of space.The corresponding spatial histogram of each layer is connected in series, and causes comprising the longer descriptor of some geological informations (area-of-interest of driver's face that for example, instruction detects) of the view data of catching.Use this distribution for how much of the view data of catching of vision word to improve classification performance.
Fig. 6 illustrate as the frame 510 above with reference to Fig. 5 describe for building space pyramid visual word bag to generate the exemplary non-limiting example of multiple spatial histograms.Ground floor 602 is depicted as and is divided into a region, and the histogram 603 of vision word is depicted as for ground floor 602.The second layer 604 is depicted as the region of ground floor 602 is increased to four (4) sub regions.The histogram 605 of vision word is depicted as each for four sub regions of the second layer 606.Be depicted as for the 3rd layer 606 four sub regions of the second layer 604 are increased to ten six (16) sub regions.The histogram 607 of vision word is depicted as each for 16 sub regions of the 3rd layer 606.Should be appreciated that three layers 602,604 and 606 only illustrate for illustration purpose, and the disclosure is not limited to any number of plies of space pyramid visual word bag.
Again referring to Fig. 5, the image of catching of the position of driver's face that frame 511 detects from instruction is inputted Data Detection driver facial positions.Should be appreciated that driver's facial positions is that space is measured.
Referring to frame 512, at least one spatial histogram of the vision word of frame 510 is connected in series with generating feature vector with driver's facial positions of frame 511.
At frame 514 places, use EOTR sorter by the proper vector classification of the generation of frame 512.Particularly, EOTR sorter is used for proper vector to classify to extract the pose information for detection of the facial positions arriving.In the illustrated embodiment, EOTR sorter comprises scale-of-two SVM linear classifier.Scale-of-two SVM linear classifier uses the image of multiple equally distributed training.The image of each training comprises wearing spectacles and the face-image corresponding to the corresponding sampling of one of two classifications.These two classifications comprise the face-image of sampling, wherein (1) EOTR state exists, for example, face-image instruction driver does not stare at his/her eye on road/Driving Scene, and (2) EOTR state does not exist, for example, face-image instruction driver stares at his/her eye on road/Driving Scene.The sample of therefore, training is waited probability and is distributed in these two classifications.Some in the image of multiple training can be caught during low light photograph or nighttime driving condition.In addition, the face-image of sampling is selected from have the different multiple individualities that change from different races and head pose.Therefore, EOTR sorter is used to the output based on proper vector and space driver's facial positions of obtaining from the view data of catching and estimate whether driver stares at road.
The proper vector of the classification based on frame 514 in frame 516 determines whether to exist EOTR state.In the time EOTR state being detected, can take alarm or other measure to obtain driver's attention, driver is remained on his/her sight in road scene again.
The disclosure has been described some preferred embodiment and amendment thereof.Reading and understanding after this instructions, technician can expect other amendment and modification.Therefore, disclosure intention is not limited to disclosed (multiple) specific embodiment as realizing best mode that the disclosure is contemplated that, and the disclosure also will comprise all embodiment that fall within the scope of claims.
Claims (10)
1. for determining whether to exist the eyes method for state that leaves the road, comprising:
Catch the view data corresponding to driver from monocular camera head;
Use glasses sorters to detect whether wearing spectacles of described driver based on described view data; And
In the time described driver's wearing spectacles being detected:
Detect driver's facial positions from described view data of catching; And
Use the eyes sorter that leaves the road to determine whether to exist the described eyes state that leaves the road based on described driver's facial positions.
2. method according to claim 2, whether wearing spectacles comprises wherein to use described glasses sorter to detect described driver based on described view data:
From described image data extraction facial characteristics of catching;
Use the dictionary of the multiple vision words that obtained by cluster routine to carry out the visual signature extracting described in quantization;
The described quantized visual signature of pondization is to generate the spatial histogram of described vision word; And
Use described glasses sorter that described spatial histogram is classified to detect whether wearing spectacles of described driver.
3. method according to claim 2, wherein said visual signature is in the following manner from described image data extraction of catching:
In the dense grid of the view data of catching described in intensive yardstick invariant features conversion descriptor is applied to.
4. method according to claim 2, wherein said cluster routine comprises the average cluster routine of k-.
5. method according to claim 1, wherein said glasses sorter comprises multi-category support vector machines linear classifier.
6. method according to claim 5, also comprises:
Use the image of multiple equally distributed training to train described multi-category support vector machines linear classifier, wherein the image of each training comprises the driver corresponding to the corresponding sampling of one of three classifications.
7. method according to claim 6, wherein said three classifications comprise that the driver that the driver of described sampling does not wear glasses, the driver of described sampling wears common spectacles and described sampling wears sunglasses.
8. method according to claim 1, is wherein used the described eyes sorter that leaves the road to determine whether to exist the described eyes state of leaving the road to comprise based on described driver's facial positions:
From described image data extraction visual signature of catching;
The visual signature extracting described in the dictionary quantization of multiple vision words that use is obtained by cluster routine;
The described quantized visual signature of pondization is to generate at least one spatial histogram of described vision word;
Generate the proper vector of described at least one spatial histogram of the described vision word being connected in series with described driver's facial positions; And
Use the described eyes sorter that leaves the road that described proper vector is classified to determine whether to exist described EOR state.
9. method according to claim 8, wherein said visual signature is in the following manner from described image data extraction of catching:
In the dense grid of the view data of catching described in intensive yardstick invariant features conversion descriptor is applied to.
10. for determining whether to exist the eyes equipment for state that leaves the road, comprising:
Monocular camera head in car, for catching the view data of the visual field of pointing to driver; And
Treating apparatus, it is configured to:
Use glasses sorters to detect whether wearing spectacles of described driver based on described view data; And
In the time described driver's wearing spectacles being detected:
Detect driver's facial positions from described view data of catching; And
Use the eyes sorter that leaves the road to determine whether to exist the described eyes state that leaves the road based on described driver's facial positions.
Applications Claiming Priority (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US201361754515P | 2013-01-18 | 2013-01-18 | |
| US61/754515 | 2013-01-18 | ||
| US14/041105 | 2013-09-30 | ||
| US14/041,105 US9230180B2 (en) | 2013-01-18 | 2013-09-30 | Eyes-off-the-road classification with glasses classifier |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103942527A true CN103942527A (en) | 2014-07-23 |
| CN103942527B CN103942527B (en) | 2018-01-05 |
Family
ID=51064566
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410021671.6A Active CN103942527B (en) | 2013-01-18 | 2014-01-17 | Classified using the Eyes off road of glasses grader |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN103942527B (en) |
| DE (1) | DE102014100364B4 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017088804A1 (en) * | 2015-11-25 | 2017-06-01 | 腾讯科技（深圳）有限公司 | Method and apparatus for detecting wearing of spectacles in facial image |
| CN109145864A (en) * | 2018-09-07 | 2019-01-04 | 百度在线网络技术（北京）有限公司 | Determine method, apparatus, storage medium and the terminal device of visibility region |
| CN109624850A (en) * | 2017-10-05 | 2019-04-16 | 斯特拉德视觉公司 | Monitor the method for the blind spot of vehicle and the blind spot monitoring device using this method |
| CN111032446A (en) * | 2017-08-16 | 2020-04-17 | 奥迪股份公司 | Method for operating an occupant protection system of a vehicle and occupant protection system of a vehicle |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108860150B (en) * | 2018-07-03 | 2021-05-04 | 百度在线网络技术（北京）有限公司 | Automobile braking method, device, equipment and computer readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH08175218A (en) * | 1994-12-26 | 1996-07-09 | Toyota Motor Corp | Operation state detector |
| US20080037837A1 (en) * | 2004-05-21 | 2008-02-14 | Yoshihiro Noguchi | Behavior Content Classification Device |
| WO2008139328A2 (en) * | 2007-05-09 | 2008-11-20 | Toyota Jidosha Kabushiki Kaisha | Image processing apparatus and image processing method |
| CN101466305A (en) * | 2006-06-11 | 2009-06-24 | 沃尔沃技术公司 | Method and apparatus for determining and analyzing a location of visual interest |
| CN102289660A (en) * | 2011-07-26 | 2011-12-21 | 华南理工大学 | Method for detecting illegal driving behavior based on hand gesture tracking |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8808195B2 (en) * | 2009-01-15 | 2014-08-19 | Po-He Tseng | Eye-tracking method and system for screening human diseases |
| US8811726B2 (en) * | 2011-06-02 | 2014-08-19 | Kriegman-Belhumeur Vision Technologies, Llc | Method and system for localizing parts of an object in an image for computer vision applications |
-
2014
- 2014-01-14 DE DE102014100364.0A patent/DE102014100364B4/en active Active
- 2014-01-17 CN CN201410021671.6A patent/CN103942527B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH08175218A (en) * | 1994-12-26 | 1996-07-09 | Toyota Motor Corp | Operation state detector |
| US20080037837A1 (en) * | 2004-05-21 | 2008-02-14 | Yoshihiro Noguchi | Behavior Content Classification Device |
| CN101466305A (en) * | 2006-06-11 | 2009-06-24 | 沃尔沃技术公司 | Method and apparatus for determining and analyzing a location of visual interest |
| WO2008139328A2 (en) * | 2007-05-09 | 2008-11-20 | Toyota Jidosha Kabushiki Kaisha | Image processing apparatus and image processing method |
| CN102289660A (en) * | 2011-07-26 | 2011-12-21 | 华南理工大学 | Method for detecting illegal driving behavior based on hand gesture tracking |
Non-Patent Citations (2)
| Title |
|---|
| QIANG LI ET.AL.: ""Real-Time Eye, Gaze, and Face Pose"", 《REAL TIME IMAGING》 * |
| 李新光: ""基于局部不变特征的图像匹配技术研究"", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017088804A1 (en) * | 2015-11-25 | 2017-06-01 | 腾讯科技（深圳）有限公司 | Method and apparatus for detecting wearing of spectacles in facial image |
| US10395094B2 (en) | 2015-11-25 | 2019-08-27 | Tencent Technology (Shenzhen) Company Limited | Method and apparatus for detecting glasses in a face image |
| CN111032446A (en) * | 2017-08-16 | 2020-04-17 | 奥迪股份公司 | Method for operating an occupant protection system of a vehicle and occupant protection system of a vehicle |
| US11364867B2 (en) | 2017-08-16 | 2022-06-21 | Audi Ag | Method for operating an occupant protection system of a vehicle, and occupant protection system for a vehicle |
| CN111032446B (en) * | 2017-08-16 | 2023-01-10 | 奥迪股份公司 | Method for operating an occupant protection system of a vehicle and occupant protection system of a vehicle |
| CN109624850A (en) * | 2017-10-05 | 2019-04-16 | 斯特拉德视觉公司 | Monitor the method for the blind spot of vehicle and the blind spot monitoring device using this method |
| CN109624850B (en) * | 2017-10-05 | 2022-04-08 | 斯特拉德视觉公司 | Method for monitoring blind spot of vehicle and blind spot monitor using the same |
| CN109145864A (en) * | 2018-09-07 | 2019-01-04 | 百度在线网络技术（北京）有限公司 | Determine method, apparatus, storage medium and the terminal device of visibility region |
Also Published As
| Publication number | Publication date |
|---|---|
| DE102014100364B4 (en) | 2020-08-13 |
| CN103942527B (en) | 2018-01-05 |
| DE102014100364A1 (en) | 2014-07-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9230180B2 (en) | Eyes-off-the-road classification with glasses classifier | |
| Mbouna et al. | Visual analysis of eye state and head pose for driver alertness monitoring | |
| CN104200192B (en) | Driver's gaze detection system | |
| Coetzer et al. | Eye detection for a real-time vehicle driver fatigue monitoring system | |
| CN111723596B (en) | Gaze area detection and neural network training method, device and equipment | |
| Junaedi et al. | Driver drowsiness detection based on face feature and PERCLOS | |
| WO2016038784A1 (en) | Driver state determination apparatus | |
| US7650034B2 (en) | Method of locating a human eye in a video image | |
| CN105286802B (en) | Driver Fatigue Detection based on video information | |
| CN103942527A (en) | Method for determining eye-off-the-road condition by using road classifier | |
| CN104143098B (en) | Night pedestrian recognition method based on far-infrared camera head | |
| US10803340B2 (en) | Method and apparatus for license plate recognition using multiple fields of view | |
| US11068701B2 (en) | Apparatus and method for vehicle driver recognition and applications of same | |
| Liu et al. | A practical driver fatigue detection algorithm based on eye state | |
| CN110458093B (en) | Safety belt detection method based on driver monitoring system and corresponding equipment | |
| KR102254384B1 (en) | Device and method for driver's gaze detection by deep learning | |
| CN116012822B (en) | Fatigue driving identification method and device and electronic equipment | |
| US8818093B2 (en) | Method and device for analyzing an image of an image recording device for a vehicle | |
| CN110199318A (en) | Driver status estimating device and driver status estimate method | |
| JP2018537787A (en) | Method and apparatus for classifying at least one eye opening data of a vehicle occupant and method and apparatus for detecting drowsiness and / or microsleep of a vehicle occupant | |
| JP6762794B2 (en) | Eyelid opening / closing detection device and eyelid opening / closing detection method | |
| Ma et al. | A real-time fatigue driving detection system design and implementation | |
| Horak et al. | Eyes detection and tracking for monitoring driver vigilance | |
| Shewale et al. | Real time driver drowsiness detection system | |
| JP6815121B2 (en) | Eyelid opening / closing detection device and eyelid opening / closing detection method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |