CN1291320A

CN1291320A - Method and device for detecting drowsiness and proventing a driver of motor vehicle from falling asleep

Info

Publication number: CN1291320A
Application number: CN99802989A
Authority: CN
Inventors: 帕特里克·皮里姆
Original assignee: CARLUS MAGNUS Ltd
Current assignee: Hodding B.E.V. S.A.
Priority date: 1998-01-15
Filing date: 1999-01-14
Publication date: 2001-04-11
Anticipated expiration: 2019-01-14
Also published as: FR2773521B1; CA2320974A1; AU2058099A; DE69902225T2; CN1299498A; BR9906979A; WO1999036893A1; ES2179612T3; EP1050033B1; EP1050032A1; IL137173A0; WO1999036894A1; FR2773521A1; TW436436B; DE69902225D1; ATE221235T1; ES2179620T3; WO1999036893A9; ATE219593T1; US6304187B1

Abstract

The invention relates to a method and device using an optoelectronic sensor in combination with an electronic unit arranged inside a motor vehicle. The sensor is adjusted in line with the head of the driver sitting in said motor vehicle at the same time as the inner rear-view mirror, which comprises a one-way mirror behind which the sensor is positioned. After detecting the presence of a driver seated in the motor vehicle they frame first the whole face and then the eyes of the driver in the video signal raser emitted by the sensor aided by the electronic unit and then determine the successive durations of blinking episodes, which are compared with a limit value situated between one duration typical of an awake person and another duration typical of a drowsy person. A signal (transmitted by an alram unit) suitable for waking the driver is triggered when the duration of blinking exeeds the limit.

Description

Detect method and apparatus sleepy and that avoid the automobilist to fall asleep

The present invention relates to be used for alertness automobile watch-keeping driver to detect and to prevent the method and apparatus that the driver falls asleep.

We know, the part that accounts for remarkable ratio (if not the words of great ratio) in the road traffic accident is by all or part of (dizziness) sleepy causing of automobile (private car, utility device car, van, truck) driver, and so causes a large amount of casualty accidents.

Trigger an alarm when the driver will fall asleep, particularly a voice guard is to wake up with a start him, and this problem it seems to have first-class meaning from humanity and economic viewpoint.

In an automobile, to detect driver's sleepy degree, the following suggestion of special proposition

-on the one hand, any variation when when the driver feels sleepy, detecting him and controlling bearing circle, and

-on the other hand, any variation when when the driver feels sleepy, detecting his eyelid vertical movement.

The present invention realizes the detection of above-mentioned second type (monitoring the motion of eyelid), the present invention is based on a kind of physiology observes, the variation of the time span of promptly blinking and twice variation in the time interval of blink continuously, when a people from waking state changes when feeling dizzy his/her of the rhythm and pace of moving things over to nictation before he feels sleepy situation is: the time span of its nictation is about 100 to 200ms (milliseconds) when the people is clear-headed, when feeling sleepy, he then is about 500 to 800ms, and twice time interval continuously suddenly is stable under waking state, changes in quite wide scope in that sleepy situation is next.It is main within the scope of the invention that what realize is exactly the variation of this nictation of time span.

Can detect the blink growth of time span of driver according to method and apparatus of the present invention, and described time span surpass a setting, be in 200 trigger a sound when between 500ms, for example equaling the threshold value of 350ms or other alarms especially.Here this threshold value can change according to driver's physiological situation.

In international patent application (P.C.T.) PCT/FR97/01354 in french patent application n ° of on July 22nd, 96.09420 and 1997 on July 26th, 1996, inventor the present inventor just of this two application, when emphasizing the right of priority of described french patent application we described a kind of true-time operation, be used for method and apparatus that the section of doing relative motion a scene is positioned and determines extraneous scope and be used for definite speed and moving direction.

In these designed applications of method and apparatus for this reason, section that the head that is used to observe and monitor a driver constituted has been described in described patented claim to detect and to prevent the method and apparatus that this driver feels sleepy.

This specific application according to the method and apparatus of described these patented claims:

-produced the vision signal of consecutive image of a kind of real-time this driver's of expression eyes;

-this vision signal is handled, with in rapid succession

The vertical movement that the reflection of detection eyelid is blinked in the image of this driver's eyes;

Determine the rhythm and pace of moving things of these vertical movements, and

To less than one corresponding to the driver when clear-headed fully nictation the rhythm and pace of moving things the rhythm and pace of moving things of given threshold value position; And

-when this threshold value is surpassed by the described rhythm and pace of moving things, promptly trigger an alarm downwards to wake up with a start this driver.

The objective of the invention is to above-mentioned patented claim method and apparatus, with regard to them the driver is monitored and to carry out perfect with the application that detects his contingent sleepy situation.

Several driver's detection techniques have been analyzed in the article that is entitled as " Development of drowsiness detection system " that deliver on Augusts 31st, 1994 such as Hiroshi Ueno on " the Institute of electrical andelectronics Engineers " magazine (IEEE) 15-20 page or leaf.Exactly, mention and use a kind of video camera that video image is handled counter that has, other has software, on the one hand can detect driver's face, particularly finish a rectangle that includes eyes, can realize on the other hand to widen the view and the detection of the ratio of the time span of closing one's eyes to determine sleepy standard.

In addition, the German patent application DE-19715519 of announcement and the corresponding french patent application n that announces ° 2747346 described estimate one its pin sideband have a video camera, in addition have a counter be used to handle image that this video camera produces and software be used for by determine beginning nictation and finish between time delay detect the driver's of nictation the apparatus and method of sleepy degree.Particularly this counter parts 10 can be realized:

The storage of-video image and processing thereof are to determine to comprise a zone of driver's eyes;

-detect driver's catacleisis and open fully between the time interval; And

-in storer 11 and calculating unit 22 and parts 10 unite and carry out single treatment, to calculate the ratio that occurs nictation at a slow speed.

At last, the main points of the International Application No. WO of announcing 97/01246 are security systems, and it includes one and is installed in what is taking place for video camera in the automobile internal reflector and one and owing to display screen took place and so on for the record of video camera output signal inside and outside a distance is used to analyze car.In fact this is a video camera of hiding (in rearview mirror) with the attention of avoiding the intruder, it can observe inside and its surrounding environment that wide range comprises automobile, record knows what to take place (the 6th page of 13-19 is capable) in this scope after then allowing, and this not to be its visual angle face of obviously being limited to the driver presumable sleepy and with its sensor of waking up with a start to detect him.

Can see, those files (above-mentioned each piece article and application) are using a kind of real video camera and an outside counter, blink detection in these those France of literary composition neutralization and German patent application requires a kind of processing element of complexity, described international application then can not solve the drowsiness detection problem, if the sleepy display screen of can not seeing of driver can not be seen videograph.Can see, can not be merged together two files of this file and front, because they do not belong to same technical scope.

Compare with those files, the present invention can solve driver's record that time span measures of blinking better; Because

Simple sensors (can be the sensor of a miniature video cameras), particularly mos sensor be used in-its permission in the rearview mirror of automobile, it does not require it must is standard video format;

-it can be realized and rearview mirror moves moving of the transducer optical axis that links by the driver, with the face with described subtend driver;

-its permission obtains a kind of comprehensive vision, and an integrated circuit can detect the motion of eyelid immediately;

-it uses a kind of very simple processor to come the information of sensor is handled;

-it can be integrated in this sensor, its relevant electronic circuit and counter in the electronic chip, is installed in the rearview mirror all;

-it makes and might be in rearview mirror the processor in such chip and the panel board to be bound up, this rearview mirror is compared with a kind of standard inside rear view mirror and is only included a chip and one and be used for the mechanical hook-up that this adds the orientation of chip, from but a kind of intelligent drowsiness detection device of moderate cost.

In these cases, purpose of the present invention at first is a kind ofly to be used for continuously monitoring driver's notice to detect and to prevent described driver's any sleepy method in automobile.

This method is,

-vision signal of the consecutive image of driver's face is represented in generation in real time at least;

-continuously handle this signal, thereby

Detect in this signal and the effective corresponding part of driver's head image,

Determine the value of a parameter relevant,, make obvious variation in any side of threshold value when the driver turns to sleepy state from waking state with action nictation of eyelid, and

Determine in real time when this parameter value surpasses this driver and carry out the transition to sleepy effective threshold value from regaining consciousness; And

-be exceeded for this threshold value and react, trigger the warning horn that can wake up with a start this driver;

And it is characterized in that,

-on the one hand, vision signal produces by use a solid-state light electric transducer that links to each other with a rearview mirror on this automobile; This sensor is determined size and is arranged, the face-image in the time of receiving the driver basically and be seated, and point to the head that the optical axis of its receiving beam when correct points to the driver when rearview mirror;

And

-on the other hand, continuously detecting the driver after on its seat, this described Video signal processing is:

Can detect the motion of this driver's horizontal direction according to analysis, in corresponding each successive frame of this vision signal, be in the center with face that will this described driver to the mobile pixel between the two continuous frames of the same nature of described vision signal,

Can detect the motion that this driver is aligned in face's vertical direction at middle part according to analysis to the mobile pixel between the two continuous frames of the same nature of described vision signal, with his/her eye alignment at the middle part,

According to serving as according to can determining that this driver is aligned in continuous nictation of the time span of the eyes at middle part with the analysis to the mobile pixel between the two continuous frames of the same nature of described vision signal, wherein said time span constitutes described parameter,

With continuous nictation of the time span of so determining and compare from effective threshold value of waking state to the transformation of sleepy state,

When the time span of blinking upwards surpasses described threshold value, trigger the alarm that to wake up with a start this driver.

Be very easily, this described sensor is installed in the rear-view mirror box eyeglass back at sensor, this eyeglass is made of a two-way minute surface, the reception optical axis of described sensor and the orthogonal axes symmetry of the axle that is orientated in the vertical central plane of described automobile with respect to described two-way minute surface.

Preferably can detect the driver on his/her seat, at this moment can determine to be detected in the vision signal of remarkable change the number of corresponding pixel in ejusdem generis each frame continuously, simultaneously by the sum of all pixels order of the single frames of this number and this vision signal is compared, whether surpass from the driver not in the effective threshold value during to the transformation of driver on its seat on its seat with the ratio between the total number of determining number of pixels that representative is once moved and single frames pixel.

In several embodiment preferably, this method can include one or several following feature:

-be aligned in the detection-phase that moves horizontally at middle part and driver's eyes got between the detection of various vertical moving of frame at the face that makes the driver, the stage of a very wide eye alignment at the middle part is provided, but only limit to the part that central face comprises eyes and immediate peripheral part thereof, described part that at this moment will be by being used in the people and the anthropological measuring ratio between the entire face;

-when measuring time span stage nictation, provide the mensuration stage in the time interval of a double blink, and when those time intervals present irregular and surpass a preset threshold, trigger primary reinforcement immediately and report to the police;

-revise continuously at least with following parameter in a kind of relevant data, these parameters be move horizontally, vertical moving, nictation time span, continuously in a short time every, this driver on the scene and that be in waking state is at that time finished approximate to the standard value of these parameters;

Each successive stages of-this method is to use continuous computer program to carry out.These programs can be handled the continuous value of the respective pixel of each frame of same nature from the vision signal that described sensor obtains.

This illustrative purposes be equally one be used for automobile continuous monitoring driver notice, to detect and to prevent described driver's any sleepy device.This device uses said method and it is characterized in that it includes as assembly:

-a) photoelectric sensor, the reception generation of corresponding light beam has the vision signal of the frame or the corresponding successive frame of same nature thereby it and subsidiary electronic circuit combine, thereby simultaneously the rearview mirror of described sensor and automobile is connected and chosen yardstick and arrange face-image in the time of receiving the driver basically and be seated properly, and it receives optical axis sensing driver's of light head when rearview mirror is orientated when suitable;

-b) at least one integrated circuit, it includes:

-detect the driver in automobile on its seat, and produce the parts of a signal on the throne;

-the parts of signal triggering on the throne thus, according to analysis moving horizontally of described driver detected the pixel that moves between the two continuous frames of the same nature of described vision signal, make in each corresponding continuous frame of the same nature of this vision signal his/her face to be aligned in the middle part, and be aligned at face and produce a signal when finish at the middle part;

-be aligned in the device that the end signal at middle part activates by this described face, the analysis of the pixel that moves between the two continuous frames according to the same nature of each successive frame part of same nature in the vision signal that described corresponding face is aligned in the middle part detects the vertical moving of this driver's the face that so is aligned in the middle part, make in the described part of each frame of this signal eye alignment with described driver at the middle part, and the face that produces this driver's eyes is aligned in the end signal at middle part;

-the device that triggered at the end signal at middle part by this described eye alignment is determined length continuous time of this driver nictation according to the analysis of the pixel that moves between the two continuous frames of the same nature of each successive frame part of same nature in corresponding eye alignment this vision signal at the middle part;

-will so determine these continuously nictation, time span and this driver changed the device that effective threshold value of sleepy state compares over to from waking state; And

-the device that can wake up with a start this driver's alarm of triggering when this nictation, time span surpassed described threshold value.

In described device, described sensor is arranged in rear-view mirror box and in the eyeglass back of rear-view mirror box, this eyeglass is a two-way minute surface, described sensor is by by the first end support of first bar that intersects of anchor clamps supporting of portion within it of a rotation axis and rear-view mirror box, second end of this bar is linked to first end of crossing second bar of rear-view mirror box by a rotation axis with a connection piece, and second end of said second bar is fixed on the car body of front windscreen top, so that the said reception optic axis of said sensor is symmetrical with the axle at the vertical central plane interior orientation of said automobile about the axle with said two-way minute surface quadrature.

Said detection driver is on his/her seat and produce device that a device that has a signal preferably includes the number of pixels in ejusdem generis each successive frame in the vision signal of determining to be detected remarkable movement, the total number of the single frames pixel of this same number and this vision signal is compared whether surpass the device that is not converted to the effective threshold value of driver on its seat from the driver at its seating state with the rate value between the sum of all pixels order of determining corresponding mobile number of pixels and single frames.

Preferably among the embodiment, this device can comprise one or several following device again, that is: at some

-by said face be aligned in the middle part end signal activated, with in the said part of each successive frame of the said vision signal that is aligned in the middle part corresponding to face, choose one corresponding to comprise eyes and immediate around driver's eyes utilization anthropometry ratio between the wide entire face that is aligned in middle part and people carry out the device of the said wide or rough reduction part that is aligned in the middle part and produce a wide device that is aligned in the middle part end signal of eyes, the said device of this signal activation is with the vertical moving of detection driver face simultaneously;

-and the parallel operation of said device with determine continuously nictation time span and thereby by said eye alignment in middle part end signal activation with time interval of determining the double blink and when these time intervals demonstrate the scrambling that is higher than setting threshold, trigger the once device of pressure warning immediately;

-be used for relating at least: move horizontally, vertical moving, nictation time span, blink one data of these parameters of the time interval are revised continuously continuously, to finish with on the scene and be in the approximate device of the standard value of these relevant parameters of the driver of waking state at that time.

Advantageously, this sensor/electronic circuit processing unit assembly is as realizing for graphic extension in two patented claims of same inventor with the present invention of mentioning in front.

Purpose of the present invention also be invention as the automobile rearview mirror of industrial products, it is characterized in that its minute surface is made up of a two-way eyeglass, and be below this eyeglass, to include a photoelectric sensor and be used for working together with a circuit unit.This circuit unit is described in international patent application (P.C.T.) PCT/FR97/01354 that proposes french patent application n ° of on the July 22nd, 96.09420 and 1997 that proposed on July 26th, 1996, wherein this element also is to be installed in the rearview mirror, when in a single day this described unit determines to see that outrunner's the vertical movement of eyelid of this described eyeglass is corresponding to surpassing the predefined time span and can trigger a warning device during threshold value in the time interval of nictation between the time span of sleepy person nictation that is included in clear-headed people.

Best, this described rearview mirror has a light emitting diode at infrared region at least, this diode illumination of environment at least around becomes and can be driven when being not enough to illuminate driver facial, and this described photoelectric sensor is to the sensitive for infrared radiation of this described diode emission.

Preferably, in this device of the present invention and this rearview mirror, the electronic circuit of sensor, described band and described integrated circuit constitute by an electronic chip that is installed in this rear-view mirror box inside.

Now we will to according to a kind of device of the invention process this kind method of the present invention with and one of some modification preferably embodiment describe with reference to each accompanying drawing, in the accompanying drawing:

Fig. 1 and Fig. 2 are respectively a driver's of concise and to the point graphic extension head and a side view and vertical view of his/her forward and backward optical axis.

The routine arrangement of the inside rear view mirror sheet in automobile of the concise and to the point graphic extension of Fig. 3 and each different optical axis of this driver, this figure is corresponding to prior art.

Fig. 4 and Fig. 5 represent a rearview mirror and photoelectric sensor and the installation with an electronic circuit of connection within the scope of the present invention thereof respectively and are connected.

The scope of the photoelectric sensor that Fig. 6 explanation provides in rearview mirror on Fig. 4 and Fig. 5.

Fig. 7 and Fig. 8 represent when the driver is seated its face to be aligned in the method at middle part.

Fig. 9 and Figure 10 represent when the driver is seated its eyes to be aligned in the method at middle part.

Figure 11 and Figure 12 relate to the driver blink time span measurement and relate to time interval of double blink.

Figure 13 represents the process flow diagram in each continuous working stage.

Figure 14 represents to have as sensor the various advantages of an embodiment of a MOS type sensor.

Last Figure 15 is a kind of variant of Fig. 9, and preferential observation district also comprises nose except that eyes.

At first referring to Fig. 1 to Fig. 6, we will begin from the explanation of the optical-mechanical means of the electric circuitry packages that has photoelectric sensor (miniature video cameras or have the mos sensor of integral lens) and connection band thereof according to this embodiment preferably of the detailed description of the invention.This electric circuitry packages mainly is made up of one or several chip, and the image transformation that this sensor is received becomes vision signal and handles that this driver who is seated is taken place as this described sensor is viewed any sleepy to detect again.

Really, the present invention mainly utilizes the people's variation of the nictation when carrying out the transition to dizzy or sleepy state by waking state.A clear-headed people his/her eyelid that blinks, therefore his/her eye blinked with the time interval of comparison rule, be about per 100 to 200ms, the nictation time span of this people when sleepy then accelerates to 500 to 800ms, and time interval nictation at this moment increases and changes.

In vision signal from the per second 50 of consistent (same property) of photoelectric sensor or 60 frames; respectively per 20 or 16.66ms carry out one-time detection; this makes and to be easy to distinguish 100 to 200ms or 500 to 800ms time span (otherwise waking state is 5 to 10 frames or is 25 to 40 frames at sleepy state under the situation of ejusdem generis 50 frames of per second), thereby can distinguish a people's waking state and dizzy or sleepy state.

When using such differentiation under the situation the driver, suggestion that is to say driver's facial videoization with the incident of this described sensor or reception optical axis sensing driver's face.Essence at the device that this provides among embodiment preferably of the present invention is to utilize the following fact, and promptly the driver points to his/her face to the rearview mirror of his/her automobile, to see the rear of automobile by the reflection on rear view mirror lens.

Referring to figs. 1 through Fig. 3, let us is looked back the work of the conventional rearview mirror be installed in the middle position in the automobile because they fix, might be in this automobile the orientation of a part of adjusted eyeglass of car body.

Fig. 1 and Fig. 2 represent to observe the driver's of his/her automobile street of living in and road a T respectively from the side with the top, see his/her the place ahead (arrow 1) on the one hand, on the other hand owing to the minute surface 3 of this rearview mirror that so suitably is orientated is seen his/her back (

arrow

2a and 2b).This described

arrow

1,2a, 2b represent the track of light, and 2b is corresponding to the light that reflexes on the minute surface 3.

Consider now sketch 3, the minute surface 3 of its expression rearview mirror, this is that on the part 5 that is fixed on car body in the automobile by armed lever 4 one has the possible minute surface of orientation, as seen the axis of orientation of Fig. 1 and Fig. 2 or

arrow

1,2a, 2b analyse clearly.Can notice axle or arrow 1 is parallel with 2b and point to the street or the direction of road.

Fig. 3 is represented by dotted lines optical axis 6 and axle 7, optical axis 6 is perpendicular to the minute surface 3a of the mirror 3 of an inside rear view mirror that the angle of

direction

2a and 2b is divided equally into two equal angles (angle a and b equate) according to reflection law, and axle 7 is perpendicular to axle 2b, thereby be parallel to supporting part 5, and the angle c between direction 7 and 3a equals angle b and a.

After mentioning these principle of work of inside rear view mirror, we illustrate the mechanical component that the input optical axis of optical sensor can be pointed to effectively the driver's who is seated face now with reference to Fig. 4 and Fig. 5, utilize this fact simultaneously, if promptly be not this situation, the minute surface 3 of rearview mirror is by the driver's orientation that is seated, and this driver's orientation axes 2a just points to this driver's a T like this.In fact, if the input optical axis of this sensor points to this driver's face really, the vision signal that described driver produces will comprise the promising described driver's of determining time span information necessary nictation.

At first in the scope of embodiment preferably of the present invention, different with the rearview mirror of routine, rearview mirror 8 includes a two-way mirror 9 (Fig. 4), the minute surface 3a of its sensing driver's the minute surface 9a and the mirror 3 of a conventional rearview mirror (Fig. 3) plays a part same, but work as two-way mirror 9 (with rearview mirror 8) by suitably directed street or the road of driver to see his back, or so during orientation (as the situation of the conventional mirror 3 of Fig. 3), (had by a miniature electronic video camera or one mos sensor of built-in lens forms) sensor 10 that it makes that (with 9 rotations of two-way mirror) support 11 supported can receive the image of the driver's face that is seated at least.

The Cardan type mechanical connection that is illustrated among Fig. 4 and Fig. 5 (Fig. 5 is the more detailed view of the part of Fig. 4) makes it possible to regulate his/her rearview mirror is maybe regulated the receiving plane 10a of support 11 and sensor 10 and this sensor 10 automatically by the driver when this rearview mirror has been conditioned orientation the driver, make it receive the image of the driver's face that is seated, because the angle between the support 11 of minute surface 9 and this sensor 10, his/her input optical axis 10b points to the driver's who is seated head.

For this reason, support 11 used web members include two bars 12 and 13, and they freely connect mutually through a rotation axis 14a (Fig. 4) or a sleeve 14b (Fig. 5).Bar 12 is fixed to through the one end on the part 5 of this car body and by means of rotation axis 15 (being made by a ball and two hemispheric basically caps) and passes rear-view mirror box 8, its other end enters sleeve 14b or is fixed on the rotation axis 14a then, at this moment an end of bar 13 is supporting the support 11 of sensor 10 securely, and (have a ball by means of a hollow rotation axis 17, pass a passage in the ball, bar 13 penetrates in the passage and is essentially in the hemispheric cap by anchor clamps 16 supportings at two and rotates) pass the anchor clamps 16 of rearview mirror 8, be connected to rotation axis 14a or sleeve 14b by its other end then.

Such web member always keeps a suitable angle between mirror 9 and support 11, make it possible to simultaneously inside rear view mirror is carried out usually the orientation of the support 11 of directed and sensor 10 by the driver, make this rearview mirror during by suitable orientation the positive 10a of this sensor can receive the image of the driver's who is seated face at least.

Optical sensor 10 is sent the vision signal that produces according to the image that receives on the positive 10a of sensor 10 by the conductor 18 in the electronic analysis unit 19 (can be made of easily a chip cartridges or some chips that are placed in the rearview mirror 8).

Light emitting diode 20 can be provided, with the light of environment (light that comprises plate) deficiency around so that sensor 10 and electronic circuit cell 19 correct work thereof and this rearview mirror send an infrared radiation that can illuminate the driver's face that is seated at least to the driver who is seated when suitably being orientated.In this case, sensor must be to sensitive for infrared radiation; These diodes can be little by little to be supplied to energy, and are responded and be controlled through a photoelectric cell (drawing) or to the picture element signal in the undercapacity vision signal of (as by driver's 21 schematic representation) by electronic circuit cell 19 for example.

The alarm that is activated by electronic circuit cell 19 then when this driver will feel sleepy is 22 by schematic representation on Fig. 4.Simple for this figure do not expressed the power supply of sensor 10, electronic circuit 19 and each diode 20 in the drawings.

As a kind of change scheme, unit 19 can be positioned in outside the rear-view mirror box.

Our present explanation will from the vision signal of photoelectric sensor 10 electronic analysis unit 19 (have a miniature electronic video camera or one have in dress lens back with the mos sensor that an electronic circuit cell is arranged) in the method handled, this vision signal includes continuously a string (ejusdem generis) and has at every width of cloth picture corresponding to the frame of the rhythm of per second 50 or 60 such frames under the situation of two staggered frames to be even frame or odd-numbered frame, to be independent frame under the situation that every width of cloth picture is a single frames at that time.The target of this processing at this moment is the driver's that is seated of continuous monitoring a notice, simultaneously in real time and continuously measure his/her eye time span of nictation, trigger under the situation of (change of time span is indicated thus) of obviously feeling sleepy and this driver and once can wake up with a start his/her alerting signal.

According to the present invention be used for to a moving area (promptly this driver, his/her is facial and his/her eye, his/her eyelid particularly) carries out being location and this method and this device that surround and this kind travel direction and possible speed determined implemented, this method and this device according to above-mentioned each patented claim, the explanation of each patented claim is used as with reference to the integrated cost explanation, and it is useful that the method described in these patented claims is summarized.

In these applications, (by a video camera or the generation of other the sensor) vision signal that includes the frame (is made up of the corresponding frame of even number or odd number, perhaps is made up of continuous frame) of a succession of same nature under the situation of each image staggered a kind of video system by two frames under each image is a kind of situation of video system of single frames is carried out processing so that continuously

-between the former frame of a frame and correspondence, derive from the numerical value of each pixel or the variation on the intensity

Binary signal that is labeled as DP of one side, effective numerical value that two is possible, a variation that belongs to pixel value, and another is unconverted, its numerical value can for example be labeled as " 1 " and " 0 " respectively, and

Digital signal that is labeled as CO on the other hand, the decreased number of possible numerical value, the amount of this variation of this signal on this pixel number here is effective;

-lick these two signals DPs of same frame of this matrix and the value of CO by matrix distribution by roll mode; And

-from then on matrix distribution is derived desired moving and each parameter (position, direction and speed).

According to above patented claim, this last a kind of mobile detecting operation can be realized preferably

-along two axles for example OX and OY be formed on the signals DP that is matrix distribution in the previous action and the omnidirectional distribution figure of CO at least,

And

-in DP=" 1 " the big variation range of CO about the position among each omnidirectional distribution figure of DP and CO.

By realizing that the way according to above-mentioned each patented claim is following method and apparatus of summing up, the present invention carries out continuously

-detection on the throne a preproduction phase to a driver who is seated;

-in the phase one, the face to the driver in ejusdem generis each frame of this vision signal or corresponding continuous each frame is aligned in the middle part;

-in subordinate phase, the middle part in the middle part of face is aligned in is aligned in the middle part to driver's eyes;

-in the phase III, determine the time span that the driver blinks and can determine time interval of double blink;

-in the quadravalence section, with the time span of each nictation with a certain threshold value compares and comparison shows that when this this described threshold value produces the alerting signal that can wake up with a start this driver in the above immediately when being surpassed by this described time span, equally also variation and another threshold value in time interval of double blink can be compared, simultaneously when this comparison shows that this back one threshold value by from top above the time produce the alerting signal of a reinforcement immediately.

We are by illustrating each the embodiment in this five stages in more detail according to method and apparatus of the present invention now.

Preproduction phase, it is on the throne and begin first face and be aligned in the middle part stage to detect driver who is seated, and by manual or other a switch triggering, clearly is by realizing the method and apparatus of patented claim above-mentioned.As in case of necessity, it can be by regulating this rearview mirror so that the positive 9a subtend driver of the two-way mirror 9 of the former (Fig. 4) makes him can see the street or the road of his back in this mirror.

Fig. 6 is illustrated in the visual field 23 of sensor 10 between direction 23a and the 23b, and at this moment driver's a T should be positioned at the middle section of this conical field 23 as described in reference to figure 4 and Fig. 5 after regulating inside rear view mirror 8.This visual field can be quite narrow, because driver's the mobile of a T is limited (removing in some rare situations) when driving; Can improve the sensitivity of this device to the restriction of field, because suitably the positive 10a of directed sensor and the driver's that minute surface 9 receives simultaneously face-image include very large part in each frame of this vision signal; Therefore represent that with a pixel count this pixel count is sizable part of the total pixel number of every frame.

The direction of Fig. 6 presentation graphs 3 or

light

1,2a and 2b.

The driver take a kind of drive during posture the position can via implement according to above-mentioned two patented claims, can detect each this method and apparatus that moves of summing up above by his/her head, particularly the mobile of his/her face detects.

In fact, this driver sit down and a large amount of pixels that move by this vision signal of his/her T show.Numerical value for this vision signal binary signal DP is " 1 ", and this is corresponding to the big variation of pixel number between corresponding two continuous frames, and digital signal CO gets very high value.

When the driver sat down, the ratio of the total pixel number of a this number of pixels (representing value defined above with DP and CO) and a frame depended on the size in the both sides sensor field of view of this driver's at bearing circle place a T.If visual field very narrow (angle that reduces between the 23a of Fig. 6 and the 23b) for example can be considered, if the DP and the CO with value above-mentioned that show over half of ' moving ' pixel of a given frame, then this driver will sit down.Between the total pixel number of ' a moving ' number of pixels and a frame, can consider one 50% threshold value at that time, and in this case, when this threshold value is exceeded on top, this preproduction phase along with one send this driver signal on the throne, the processing stage of starting this vision signal next, simultaneously begin the phase one zone bit " 1 " generation and stop.Clearly, when considering the visual field of this sensor 10, adopt the threshold value that triggers zone bit " 1 " can be different from 50%.

As changing example, send this driver signal on the throne, start the zone bit " 1 " of phase one and can be produced by an order in the outside of this electronic circuit cell 19, for example securing band or the operating seat that excites, buckles the driver by ignition key can trigger this order under his/her weight.

When driver on the throne being sent signal and zone bit on the throne " 1 " and produced, this vision signal first the processing stage can begin.As noted, this stage is that the face with the driver is aligned in the middle part in vision signal, in the image that detecting device 10 receives any redundance of the upper and lower and both sides of head is removed simultaneously.

For this reason, by realizing according to this method of the present invention and this device, these are to move horizontally, from right to left and mobile being detected from left to right just, because the trend that driver's head has the tendency that moves horizontally rather than vertical moving promptly to see up and down.

Therefore, the dynamic matrix that uses DP and CO numerical value is also by the axes of coordinates along two, the Ox of for example common coordinate and Oy axle are selected, are passed through to implement distribution plan formation method according to above-mentioned patented claim, and represented data stream can extract one and move horizontally signal from the corresponding successive frame of this vision signal.

At the end of these frames the valid pixel that once moves horizontally is counted that can sensed luminance to change be that pixel value changes the maximum peak value that moves along edge, and for example along the horizontal projection of Ox with along the vertical projection of Oy.

This has explanation on Fig. 7, expressed Ox and Oy axle on it and along the distribution plan 24x of Ox with along the distribution plan 24y of Oy axle, promptly respectively in level and vertical projection direction.

The peak 25c of the peak 25a of

distribution plan

24x and 25b and

distribution plan

24y and 25d are drawn out a frame that is limited by the line segment Ya, Yb, Xc, the Xd that surround the facial V of driver that is centered on by each

surge line

27a, 27b, 27c, 27d by their coordinate 26a, 26b, 26c, 26d separately.Surge line explanation driver is having slight motion between described moving period in pixel intensity variation zone greatly.

Thereby the pairing coordinate 26a of four peak 25a, 25b, 25c and 25d of two

distribution plan

24x and 24y, 26b, 26c and 26d location made it possible to stipulate preferably and make this driver facial V position in Z district and for below this vision signal each processing stage elimination with respect to the part up and down of frame Xc, Xd, Ya, Yb, represented as the hatched area on Fig. 8 around facial V, this makes and can improve precision, also may improve the speed that is centered on and include the analysis on the center Z of facial V by line segment Xc, Xd, Ya, Yb at the shadow-free line.

The essence of this operation is to upgrade framing to entire face by for example per ten frame video signals of certain time interval, and regulation again change slightly but quite stable around facial frame Xc, Xd, Ya, Yb the time determine the mean value of coordinate 26a, 26b, 26c, 26d (in time course).In view of this, the position of this described frame (together with the zone of retouching out for later analysis) is quite firm, that is to say time course very stable.

After facial V finishes framing to the driver, produce the zone bit " 1 " of the face of a new framing.

The generation of this zone bit triggers second stage, and this is further reduction to frame just to the processing of the frame of driver's eyes.

This subordinate phase preferably includes pre-operation one time, this be in electronic circuit cell 19 between people's eye areas and entire face, obviously in vertical direction, use common anthropometry ratio, although eye areas is only occupied the finite part of entire face.

In this pre-operation, this electronic circuit cell 19 is determined the more frame Z ' of restriction that includes driver's eyes U then in proportion.Limited by Ya, Yb, Xc, Xd among the frame Z before this face V, and the frame Z ' is here drawn by frame Ya, Yb, inner line segment Y ' a, Y ' b, X ' c and the X ' d of Xc, Xd (Z zone) as represented on Fig. 9.

Like this, the external female hachure on Fig. 9 (single hacures) at this moment is eliminated and only stays frame Z ', and this makes and in subordinate phase the last framing of eyes is easy to, and improves its precision and determine speed with it.

If pre-operation is arranged, it can produce a rough eyes framing zone bit " 1 ", after finishing this pre-operation, after perhaps directly first the processing stage, just respectively at this rough eyes framing zone bit " 1 " or appearance that should face framing zone bit " 1 ", electronic circuit cell 19 carries out second valid framing stage, is at this moment aiming at (zeroing in) on driver's the eyes more and is detecting DP=1 and CO shows the location of pixels of high numerical value in the matrix of DP and CO.This is for the moving of vertical direction, because eyelid is to blink up and down obviously.

When providing under the situation of pre-operation in (Z district) at frame Y ' a, Y ' b, X ' c, X ' d, or in not having a kind of like this frame Ya, Yb of pre-operation, Xc, Xd (Z zone), when such location of pixels number reaches a certain threshold value, at this moment in the former case this threshold value corresponding to for example in frame Y ' a, Y ' b, X ' c, X ' d relatively 20% of sum of all pixels, and under one situation of back in Ya, Yb, Xc, Xd relatively 10% of sum of all pixels, then produce a meticulous eyes framing zone bit " 1 ".In fact this zone bit shows that driver's eyelid is movable, because it is to cause nictation by the driver.Be that driver's face moves the motion of detected vertical direction similarly in the phase one neutralization levels nictation.

Figure 10 represents to retouch out the vertical moving of possible frame Y ' a, Y ' b, X ' c, X ' d and driver's eyelid of rough framing zone Z ' of this driver's eyes along the distribution plan 28x of axle Ox with along the distribution plan 28y of axle Oy, just by their DP and the CO pixel of expressing this mobile matrix.These distribution plans 28x and 28y, corresponding with the 24x that moves horizontally of this driver's face of representing on Fig. 7 with 24y, by their

peak

29a, 29b, 29c, 29d decision horizontal line section X " c and X " d and vertical line segment Y " a and Y " b, they draw out a Z who surrounds driver's eyes in regional Z ' " district; move at 30a and 30b place for its edge of eyes and to indicate, another eyes are then at 30c and 30d place.

Frame Y " a, Y " b, X " c, X " position of d is by determining the mean value in for example per ten frames of time course, the coordinate of

peak value

29a, 29b, 29c, 29d, and according to precision the generation to the zone bit " 1 " of eyes framing obtains revising.These are processed Z in the phase III that zone bit thus triggers just " pixel that comprises in the frame retouched out of the quilt of district (this Z " district be illustrated on Fig. 9) with white.

In this phase III, nictation, time span was determined, also may determine the time interval of double blink, simultaneously by handling corresponding to this Z with electronic circuit cell 19 " various piece of each successive frame of the vision signal in district can make this Z " analysis to the vertical movement of eyelid in the district is more perfect, this can obtain very high precision.

Figure 11 is illustrated in one along determining in the coordinate system of three orthogonal directionss that this driver carries out the transition to three different parameters of sleepy state from waking state.OQ direction note has CO thereby is the pixel change intensity, corresponding to the vertical movement of eyelid.Direction Ot puts down in writing the time interval of double blink, and Oz record time span nictation.Double nictation, C1 and C2 were indicated on Figure 11.

Figure 12 makes every frame of big vertical movement (its DP=1 and CO has very high numerical value) with the curve C explanation of part on (a) number of pixels changed along time of Ot, corresponding to continuous peak P1, P2, the P3 of the mobile number of pixels of nictation.

Corresponding each successive frame relevant with curve C also partly is illustrated on (b) part of Figure 12 with the perpendicular line as 31 is concise and to the point, its peak value P1, P2, P3 are surrounded by rectangle R1, R2, R3 respectively, and Figure 12 (a) and (b) two parts overlapping arrangement simultaneously.At last, Figure 12 expresses the blink time span (5,6,5) and the time interval (14,17) of continuous blink with its numerical value corresponding to the frame number of this driver's waking state.

In this phase III, this electronic circuit cell 19 calculates continuously nictation time span and continuous time interval of double blink, and in time span and carry out a bidimensional statistical study between time interval nictation nictation continuously.It can be determined, whether this time span of blinking surpasses certain threshold value, 350ms for example, and in this case, whether trigger threshold value zone bit nictation " 1 " that is exceeded, and at time interval of double blink quite stable whether, perhaps conversely in time and great changes have taken place, and under second kind of situation, trigger a zone bit " 1 ", express the time interval of transformable blink.

This first zone bit is used to trigger once and reports to the police, the audible alarm that for example once can wake up with a start the driver, and second zone bit strengthened warning, for example increases intensity of sound.

The process flow diagram of enclosing as a spike plate 6 (Figure 13) adds up to each successive stages.

Carry out with serial mode (CAN-VAN) with the dialogue of outside is best.

Electronic circuit and the processor or the counter of this whole converter, connection band can be integrated into an electronic chip in the inside rear view mirror that is installed in automobile easily.

Drive country for the right hand, the rearview mirror of Fig. 4 and Fig. 5 is fit to be sitting in the driver on seat, the right and the seat, the left side, and also may be a kind of external mirror having indicator light obviously aspect the driver.

It is significant especially using a kind of MOS type sensor, and it determines the value of a pixel in location of pixels ground one by one, needn't take out pixel value as ground, a position, position of the every more line by line row of ccd sensor.

Represented as Figure 14, might realize a kind of selection that can change at that time: replace (a) and on entire image, checks all location of pixels PP (a large amount of processing), can (b) only check that composition represents the specific netted pixel PP of a regular network of the overview image of this driver's face and Near-neighbor Environment (image enlarge-effect) thereof to location of pixels; Can (c) select a specific region ZP at last, by the pixel (processing power is constant) of only handling this district.

As the having on same Figure 15 of the variant of Fig. 9 with reference to coordinate, not only can see eye areas in the final stage that the driver monitors, also can see the n of nose N.The profile or the observation of moving of blackening make and can improve the stability in the observation district that comprises eyes.

Unquestionable, the embodiment preferably that the present invention is not limited in explanation and depicts, also be not limited to its variation above-mentioned, on the contrary, the present invention includes the various modifications, the changes and improvements that belong to the range of definition of the present invention that in preamble and subsidiary claim, provides.

Claims

1. the notice of successively monitoring the driver in automobile is to detect and to prevent any possible sleepy method of driver

Be

-produce a real time video signals of the consecutive image of driver's face at least;

-continuously this signal is handled, with

In this signal, detect effective part corresponding to driver's head image,

Determine and the blink value of a relevant parameter of eyelid, when the driver when waking state carries out the transition to sleepy state, have significant change on the both sides of threshold value, and

When surpassing this driver, this parameter value when waking state is transformed into the threshold value of sleepy state, carries out real-time positioning; And

-at this threshold value that is exceeded, trigger the alarm that can wake up with a start this driver;

And it is characterized in that

-on the one hand, by on automobile, use one that link to each other with rearview mirror, face-image when determining size and being arranged to such an extent that can receive the driver basically and be seated, and when rearview mirror directed when correct its reception light optical axis sensing driver photoelectric sensor can produce vision signal; And

-on the other hand, continuously detect this driver on his/her seat after, described Video signal processing is:

From tangential movement to this driver of analyzing and testing of the mobile pixel between the ejusdem generis two continuous frames of described vision signal, be aligned in the middle part with face in each continuous corresponding frame of this vision signal to this described driver,

From vertical movement to this driver's of analyzing and testing of the mobile pixel between the ejusdem generis two continuous frames of described vision signal the face that so is aligned in the middle part, his/her eye being aligned in the middle part,

From continuous nictation of the time span of the eyes that so are aligned in the middle part of the analysis of the mobile pixel between the ejusdem generis two continuous frames of described vision signal being determined this driver, this described time span constitutes this described parameter thus,

These continuous time spans of blinking of so determining are compared from the important threshold value that waking state is transformed into sleepy state with this driver, and

When this nictation, time span surpassed this described threshold value on top, trigger the warning horn that to wake up with a start this driver.

2. according to the method for claim 1, it is characterized in that, this described sensor is placed in the rear-view mirror box in its minute surface back, this rearview mirror is made up of a two-way minute surface, and directed in the reception optical axis (2a) of described sensor and the vertical central plane of a described automobile axle is with respect to symmetrical with the axle (6) of described two-way minute surface quadrature.

3. according to the method for claim 1 or 2, it is characterized in that detecting the driver on his/her seat, at this moment determine to detect in the vision signal of big motion the number of corresponding pixel in ejusdem generis each continuous frame, and compare by total pixel number this number and the every frame of this vision signal, with the number of pixels determining to move and every frame in representative always as the ratio between the Queensland prime number order whether above not being transformed into the effective threshold value of driver on his/her seat at his/her seat from the driver.

4. according to claim 1,2 or 3 method, it is characterized in that, between each detection-phase that moves horizontally that is aligned in the middle part for face and detection for the vertical moving of driver's eyes being got frame to the driver, provide between this described part and people's entire face utilization anthropological measuring ratio eyes be aligned in the very wide stage at middle part, but only limit to this face that is aligned at the middle part comprise eyes and a contiguous part thereof on every side.

5. any one method in requiring according to aforesaid right, it is characterized in that, with the while in mensuration stage of time span nictation, provide the mensuration stage in the time interval of a double blink, and one when these time intervals present irregular and surpass a preset threshold, promptly trigger primary reinforcement and report to the police.

6. any one method in requiring according to aforesaid right, it is characterized in that, at least about move horizontally, vertical moving, nictation time span, a kind of in suddenly these parameters of the time interval constantly revises data continuously, so that driver on the scene at that time and that be in waking state can finish approximate to the standard value of these parameters.

7. any one method in requiring according to aforesaid right, it is characterized in that the different successive stages of this method is to use the successive value of continuous computer programs process respective pixel of each frame of same nature the vision signal that described sensor from this obtains.

8. in automobile, be used for successively monitoring driver's notice,, it is characterized in that its merges to include to detect and to prevent the device of any possible sleepy enforcement of driver according to any one method in the claim 1 to 7:

A) photoelectric sensor (10), the combine reception of corresponding light beam of it and subsidiary electronic circuit (19) produces a vision signal that has the frame or the corresponding successive frame of same nature, simultaneously the rearview mirror (8) of described sensor and automobile is connected and chosen size and arrange face-image in the time of receiving the driver basically and be seated properly, and is orientated the head (T) of optical axis (10b) the sensing driver of the reception light of sensor when suitable when rearview mirror;

B) at least one integrated circuit, it includes:

-detect the driver in automobile on its seat, and produce the device of a signal on the throne;

-the device of signal activation on the throne thus, according to analysis moving horizontally of described driver detected the pixel that moves between the two continuous frames of the same nature of described vision signal, make in each continuous frame of the same nature of this described vision signal his/her face (V) to be aligned in the middle part, and be aligned at face and produce a signal when finish at the middle part;

-be aligned in the device that the described end signal at middle part activates by this face, according to and the facial two continuous frames that is aligned in the same nature of each successive frame part of same nature in corresponding this described vision signal in middle part between the analysis of mobile pixel the vertical moving of the face in the middle part of so being aligned in of this driver is detected, make in the described part of each frame of this signal the eyes (U) with described driver be aligned in the middle part, and the face that produces this driver's eyes is aligned in the middle part and connect the tail signal;

-the device that activates at the end signal at middle part by this described eye alignment, according to and the two continuous frames of eye alignment each successive frame same nature partly of same nature in corresponding this described vision signal in middle part between duration;

-will so really measure these nictation, time span and this driver changed the device that effective threshold value of sleepy state compares over to from waking state continuously; And

-when this nictation, time span surpassed this described threshold value, trigger the device that can wake up with a start this driver's alarm (22).

9. according to the device of claim 8, it is characterized in that, this described sensor (10) is placed in the box of this rearview mirror (8) after the minute surface of this sensor, this minute surface is made up of a two-way minute surface (9), and described sensor is by first end bearing of first bar (13) that is intersected by the anchor clamps of this rear-view mirror box (8) portion's supporting within it through a rotation axis (17) and.Second end of this bar (13) freely is connected to first end of second bar (12) that intersects through a rotation axis (15) and this rear-view mirror box (8) with an interface (14a, 14b).And second end of described second bar (12) is fixed on the car body of front screen screen top (in 5), make the reception optical axis (2a) of described sensor and in the vertical central plane of described automobile a directed axle (2b) with respect to symmetrical with the axle (6) of described two-way minute surface quadrature.

10. according to the device of claim 8 or 9, it is characterized in that, this driver of this described detection on his/her seat and this described device that produces a signal on the throne by measuring it is detected pixel count destination device in each successive frame of same nature in this described vision signal of very big motion, with this number and in this vision signal the sum of all pixels order of every frame compare whether to surpass and do not convert the device of driver in effective threshold value at his/her seat at his/her seat from this driver with the ratio between the total number-of-pixels that is determined at the expression number of a mobile pixel and every frame.

11. according to claim 8,9 or 10 device, it is characterized in that, it also includes and is aligned in the middle part end signal by this described face and activates, with and this face be aligned in the corresponding described vision signal in middle part this described part of each frame in by select one and comprise eyes and corresponding reduction part in the middle part of wide or thick being aligned in of neighbour's driver's eyes on every side at utilization anthropological measuring ratio between the described wide entire face that is aligned in middle part and people, and produce eyes widely be aligned in the middle part end signal, and this described device of this signal activation is to detect the vertical moving of this driver's face.

12. device according to claim 8 to 11, it is characterized in that, it include and this described mensuration blink continuously time span and thereby by this described eye alignment at the middle part end signal, measuring the time interval of double blink, and trigger primary reinforcement when surpassing a setting threshold and report to the police showing when these time intervals.

13. according to any one device in the claim 8 to 12, it is characterized in that, it include be used for to about move horizontally, vertical moving, nictation time span, suddenly at least a data of these parameters of the time interval are revised continuously continuously, so that driver on the scene at that time and that be in waking state can finish the approximate device to the standard value of these parameters.

14. according to any one device in the claim 8 to 13, it is characterized in that, this photoelectric sensor (10)-electronic circuit cell (19) assembly produces a vision signal that includes the corresponding frame of the series of identical character that has a series of row of being made up of a series of pixels, and this described vision signal is handled with continuously

-between a frame and corresponding former frame, deduct from the numerical value of each pixel or the variation of intensity

On the one hand, a binary signal that is labeled as DP, the variation of this pixel number of an expression of two possible numerical value, and another represents that this pixel value is constant, and

On the other hand, a digital signal that is labeled as CO, the decreased number of its possibility numerical value, the size of this variation of this this pixel value of signal indication;

-with roll mode will be by matrix these two signals DPs of same frame and the numerical value of CO distribute by matrix; And

-from then on matrix distribution draws moving and position and direction parameter of requirement.

15. any one device according in the claim 8 to 14 is characterized in that, the electronic circuit (19) of described sensor (10), described band and described integrated circuit are made of an electronic chip that is installed in the rear-view mirror box (8).

16. rearview mirror that is used for automobile, it is characterized in that, its minute surface is made by a two-way mirror (9), and include a sensor that is equipped with the optoelectronic circuit (10) of working together in this two-way mirror back with an electronic circuit cell (19) that also is arranged in this rearview mirror, be that also this photoelectric sensor (10)-electronic circuit cell (19) assembly produces a vision signal that includes the corresponding frame of the series of identical character that has a series of row of being made up of a series of pixels, and this described vision signal is handled with continuously

-from then on matrix distribution draws moving and position and direction parameter of requirement; And

-one when this described unit determine seeing people's the vertical movement of eyelid in the front (9a) of this described minute surface surpass corresponding to one a pre-set threshold be in a clear-headed people nictation time span and a sleepy people between nictation during time span, promptly trigger a warning device (22).

17. the rearview mirror that is used for automobile according to claim 16 is characterized in that, the electronic circuit (19) of described sensor (10), described band and described integrated circuit are made of an electronic chip that is installed in the rear-view mirror box (8).

18. rearview mirror that is used for automobile according to claim 16 or 17, it is characterized in that, it at least also has a light emitting diode (20) that becomes and can be activated when being not enough to illuminate driver facial in the ambient light illumination at least around of infrared region at least, and is that this described photoelectric sensor (10) is also to the sensitive for infrared radiation of this described diode emission.