JPH06344801A - Monitor of driver's eyes on vehicle - Google Patents

Abstract

PURPOSE:To call a driver's attention to napping, looking away, etc., by photographing the driver on a vehicle by a two-dimensional camera and processing the photographed image by an image information processor and detecting the eyes' position by an inspective means and energizing an alrm means when the eyes' position is getting out of a specified area on the photographed image. CONSTITUTION:A CCD camera 3 transmits its image signals to a microprocessor 6 through an interface 16. When the microprocessor 6 reads one frame of image data, it writes it on a frame memory (RAM) 12. The microprocessor 6 is connected to another microprocessor 8 in order to run operation for various image processings and judging thereof. And components such as ROM 9, RAM 10, bus controller 15 and frame memory 12 are connected therewith. The microprocessor 6 judges attention marks and inspects them every time when respective frames are written in the memory 12. When these exceed an allowable value, a buzzer 5 is turned on.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a surveillance device for photographing a driver (driver) on a vehicle with a two-dimensional image pickup device, and detecting and tracking an eye position by image processing of a photographed screen, and particularly to this. Although not intended, the present invention relates to a drowsiness detection device for a driver on a vehicle.

[0002]

In Japanese Patent Publication No. 4-284, a face of a vehicle driver (driver) is photographed by a camera, and the eyes and mouth are recognized and tracked on the photographed image to determine the degree of opening of the eye (opening). ，
Closed) is calculated, and it is determined whether or not the eyes are closed based on the opening degree. The brightness of the lamp that illuminates the driver's face is optimally adjusted to recognize the eyes and mouth based on the image data. The image data is stored in the frame memory and binarized. The driver's eyes and mouth are detected based on the binary data. If the eyes are closed for too long, the driver is considered dozing and an alarm is issued.

[0003]

By the way, when the driver is facing the front of the vehicle, his or her eyes or the like can be detected relatively accurately, but the driver's head is lowered due to drowsiness, extraneous points, or some other reason ( When the face is facing down), when the subject is sideways, or when the subject turns sideways, the eyes themselves cannot be detected. Alternatively, a non-eye is erroneously detected as an eye.

It is an object of the present invention to warn the driver of this type of condition in which eye recognition is not possible.

[0005]

The device for monitoring the eyes of a vehicle driver according to the present invention captures an image of the vehicle driver.
Dimensional image pickup means (3); Warning means (5); Image information processing means (6, 8) for processing the image photographed by the two-dimensional image pickup means (3) for detecting the eye position of the driver; Information processing means
The position (m ₁ , n ₁ / m ₂ , n ₂ ) determined to be the eye position by (6, 8) is determined whether it is inside or outside a predetermined area (ECR) smaller than the screen area on the photographing screen. Verification means (6, 8) for urging the alarm means (5) when it is disengaged; and the two-dimensional imaging means (3) is repeated when the verification means (6, 8) determines that it is within a predetermined area. Tracking means (6,
8); Symbols in parentheses indicate corresponding elements or corresponding matters in the embodiments shown in the drawings and described later.

[0006]

The two-dimensional image pickup means (3) photographs the driver on the vehicle, and the image information processing means (6, 8) processes the photographed image and the eye position (m ₁ , n ₁ / m ₂ , n ₂ ) The verification means (6, 8) detects that the eye position (m ₁ , n ₁ / m ₂ , n ₂ ) is inside or outside a predetermined area (ECR) smaller than the screen area on the photographing screen. If it is judged to be wrong, the alarm means (5) is energized. This will trigger an alarm. By setting the predetermined area (ECR) as the position of the image (eye image) on the screen of the driver's eye who is clearly visually recognizing the front (in a steady state), a margin is added, thereby causing a drowsiness Or, when the driver's head goes down (face faces down), becomes oblivious, or turns sideways for some other reason, the image that the image information processing means (6, 8) considers to be an eye. Even if it is the correct eye image or if another object is falsely detected as the eye,
An alarm is issued because it is out of the predetermined area (ECR).
This alert calls the driver's attention.

Other objects and features of the present invention will become apparent from the following description of embodiments with reference to the drawings.

[0008]

EXAMPLE FIG. 1 shows the configuration of an example. In this embodiment, the driver's face on the vehicle is photographed to detect the eyes in the captured image, the eyes are tracked in time series to recognize whether the eyes are open or closed, and the eyes are closed for a predetermined time or longer. When it is sleeping (sleeping), it activates the buzzer 5. As shown in FIG. 2, the CCD camera 3 and the illuminating lamp 4 for illuminating at least the face portion of the driver are integrally configured to form an instrument panel 2.
In addition, it is fixed so that the pointing direction can be adjusted vertically and horizontally. A buzzer 5 for generating an alarm and a stop switch SS for stopping the alarm of the buzzer 5 are provided on the side surface of the CCD camera 3.
W is provided. The illuminating lamp 4 is an infrared illuminator composed of a light emitting diode, and the CCD camera 3 has sensitivity in the infrared region. The infrared illumination does not give glare to the driver, unlike the illumination by ordinary light, and thus does not make the driver's front view difficult. Therefore, the illumination lamp 4 may be constantly turned on. In FIG. 2, reference numeral 1 is a steering wheel operated by a driver for controlling the traveling direction of the vehicle.

Referring again to FIG. The buzzer 5 is turned on / off by the buzzer controller 19. As will be described later, the buzzer 5 is urged when the driver's eyes are closed for a predetermined time or longer (assuming that the driver is asleep). The illumination lamp 4 is energized by the infrared lamp controller 18. Buzzer controller 19, infrared lamp controller 18, other outputs (controller that controls radio, air conditioner, wiper, headlight, etc.)
Includes a microprocessor 6 and an interface 13
An on / off instruction signal or the like is given via.

The CCD camera 3 is provided with a two-dimensional CCD, and the microprocessor 6 gives an ON / OFF signal to the CCD camera 3 via an interface 16. The CCD camera 3 repeats the captured video signal (video signal: analog) to repeat the A / D converter 1
7 and outputs the pixel sync pulse to the A / D converter 17 as an A / D conversion sync pulse, and also applies the frame sync pulse, the line sync pulse and the pixel sync pulse to the microprocessor 6 via the interface 16.

The microprocessor 6 uses 1 of the image data.
When reading a frame, the frame memory (RAM) 12 is instructed to write in synchronization with the frame sync pulse, and the write address of the frame memory 12 is advanced in synchronization with the line sync pulse and the pixel sync pulse. In the frame memory 12, in this example, the CCD camera 3 captures one screen with 512 horizontal pixels and 485 vertical pixels.
An image memory capable of storing 256-gradation image data for 2 × 485 pixels is used.

The A / D converter 17 converts the video signal into 8
It is converted into digital data of bits (256 gradations). That is, the image data written in the frame memory 12 is 8 bits (256 gradations) per pixel (hereinafter, dot).
Is data indicating the brightness. Therefore, the microprocessor 6 can know the brightness at any position on the screen by reading the value in the frame memory 12.

The microprocessor 6 is connected with another microprocessor 8 for performing various image processing and arithmetic operations for determination, and also a ROM 9 and a RAM 10 for controlling a normal computer system, a bus controller 15 and the above-mentioned. The frame memory 12 of is connected.

FIG. 3 shows an outline (main routine) of the operation of the microprocessors 6 and 8, and details of the operation (subroutine) are shown in FIGS. The control operation of the microprocessors 6 and 8 based on the set program will be described below with reference to these drawings.

First, the outline of the operation will be described with reference to FIG. When the power is turned on, the microprocessor 6 performs initialization (step 1: hereinafter, the word step is omitted in parentheses). In this, the I / O ports are initialized, the internal registers, flags, etc. are cleared, and the CCD
The camera 3 and the infrared lamp controller 18 are supplied with an energizing signal indicating ON, and the buzzer controller 19 is supplied with a signal indicating OFF. Further, the microprocessor 6 has
As shown in FIG. 6, image data to be written in the frame memory 12
25 screens in the center of the screen,
A total of 425 points of interest (cursors), 17 in the vertical direction,
It is arranged at intervals of 15 dots horizontally. This interval varies depending on the size of the human eye on the screen, but as shown in FIG. 17, the maximum vertical width of one eye on the shooting screen of the CCD camera 3 is 30 dots. ， Horizontal direction is 40-5
Since there are 0 dots, the interval is 15 dots in this embodiment so that at least one target point exists in one eye area. As shown in FIG. 16, when the horizontal direction of the image written in the frame memory 12 is the x-axis and the vertical direction is the y-axis (the upper left corner coordinates (0, 0) of the image, the lower right coordinates (511,
484)) and the upper left corner coordinates of the 425 points of interest are (68,
200), the target point is arranged so that the lower right coordinate is (428, 440), and its coordinate data (X _m , Y _n ) is calculated as shown in FIG.
_Write to address A _nm in the memory table of RAM (memory) 10 (FIG. 1) as shown in FIG.

However, X _m = 68 + 15 (m−1); 1 ≦ m ≦ 25, Y _n = 200 + 15 (n−1); 1 ≦ n ≦ 17.

Then, a predetermined time has passed since the initialization (illumination lamp 4
Of the image data for one frame in synchronization with the frame synchronizing pulse from the CCD camera 3 after waiting for the driver's face to be stationary at the driving position). Writing is started, and image data for one frame (8-bit multi-gradation data per dot) is written in the memory 12 (2).

Next, the microprocessor 6 sets a small area centered on each target point and updates the coordinates of the target point to the darkest point in the area (3). This operation is performed for all 425 points of interest. The contents will be described later with reference to FIG.

Next, the microprocessor 6 erases the isolated points having no other points of interest from the points of interest after the coordinate update, or merges other points of interest in the vicinity of a certain point of interest into one. Etc., and finally, it is determined whether the number N of attention points is the number of eyes of a person, that is, whether there are two attention points, and if they are two, whether or not they are in a predetermined area ECR. (4) is performed. This content is shown in Figs.
Will be described later with reference to. Finally, when the number N of attention points is not 2, the process returns to step 2.

If there are two points of interest and they are within the predetermined area ECR, the coordinates of the points of interest are regarded as indicating the positions of both eyes of a person, and the points of interest are tracked each time each frame is written in the memory 12. Then, the open / closed state of both eyes is detected in a time series to determine whether the driver is dozing or looking aside (5). The contents will be described later with reference to FIGS. 14 and 15. If tracking becomes impossible during the tracking process, the process returns to the initialization of step 1.

Referring to FIG. 4, "dark spot detection in a small area"
The contents of (3) will be described. This is a process of setting a small area within a range of ± 10 dots in the vertical and horizontal directions around the target point, and updating the coordinates of the target point to the coordinates indicating the darkest point in the small area.

First, the microprocessor 6 sets 1 in the registers m and n (31) and reads the coordinate data X _m and Y _n at the address A _nm of the memory shown in FIG. 18 (3
2). Initially, the coordinate data X ₁ and Y ₁ of the address A ₁₁ are read. Next, register x ₁ has X _m −10 and y ₁ has Y _n −1.
Set 0 to set the coordinates of the upper left corner of the small area (3
3), X _m +10 is set in the register x ₂ and Y _n +10 is set in y ₂ to set the coordinates of the lower right corner of the small area (34). Then, the data in the frame memory 12 is read out in the area x ₁ ≤x ≤x ₂ & y ₁ ≤y ≤y ₂ to detect the darkest point of the brightness value, and the coordinates are updated to the memory address A _nm. Yes (35). Next, the register m is incremented by 1 (36), and the processes of steps 32 to 36 are repeated until the value of m becomes larger than 25 in step 37. That is, this allows the coordinates (X ₁ ,
Y ₁ ), (X ₂ , Y ₁ ), ..., (X ₂₅ , Y ₁ ) are updated. When the register m becomes larger than 25 (37), the value of m is set to 1 and the register n is incremented by 1 (3
8), the processes of steps 32 to 38 are repeated until the value of n becomes larger than 17 in step 39. As a result, the coordinates (X ₁ , Y ₂ ) of the point of interest, (X ₂ , Y ₂ ), ...
(X ₂₅ , Y ₂ ), (X ₁ , Y ₃ ), (X ₂ , Y ₃ ), ...
(X ₂₅ , Y ₃ ), ... Are updated, and when the coordinates of a total of 425 points of interest are updated (YES in step 39), the routine returns. As a result, the points of interest arranged at equal intervals during initialization move to dark areas such as the human eyes, eyebrows, and hair.

The control operation of "determination" (4) will be described with reference to FIG. In the judgment, since a plurality of points of interest are gathered on the black eye, the isolated point is deleted.
1), when there are multiple points of interest around the point of interest, they are integrated into one point of interest, "deletion of other points of interest near the point of interest"
(42), comparing the average gray scales in the vertical and horizontal directions centering on the point of interest and deleting the point of interest with darker vertical gray scale, "deletion by cross gray scale difference" (43), detected in advance “Delete by template matching”, which removes the points of interest with a large comparison error by comparing it with the template image
(44), comparing the average gray scales in the upper right diagonal direction and the upper left diagonal direction centering on the target point to delete the target point having a large comparison error, "deletion due to diagonal direction gradation difference" (45), remaining By moving the point of interest to the darkest point in the small area by the same processing as in step 3, "dark point detection in small area" (4
6), integrate points of interest that are close to each other, "integrate neighboring points of interest" (47), and determine whether the number N of finally remaining points of interest is 2 for both eyes, "Determination of the number of points of interest" (48) and "attention point verification" (49) of checking whether both of the two images determined to be eyes are in a predetermined area ECR and are in a positional relationship corresponding to each other's eyes Execute.

Referring to FIG. 6, "deletion of isolated points" (4
The processing content of 1) will be described in more detail. First, the microprocessor 6 sets ₁ in the registers m ₁ and n ₁ (410
1), the values of the registers m ₁ and n ₁ are set in the registers m and n (4102), and the coordinate data X _m and Y _{n of the} memory address A _nm are set.
Is read (4103). m ₁ and n ₁ are registers indicating points of interest. Initially, the coordinate data X ₁ and Y ₁ of the address A ₁₁ are read. Next, register x ₁ has X _m −10 and y ₁ has Y _n −1.
0 is set to set the coordinates of the upper left corner of the area for detecting the presence or absence of another target point (4104), and X _m + is set in the register x _2.
10. Set y ₂ to Y _n +10 to set the coordinates of the lower right corner of the detection area (4105).

Next, 1 is set in the registers m ₂ and n _2.
(4106). The registers m ₂ and n ₂ are registers indicating other points of interest with respect to the points of interest indicated by m ₁ and n ₁ . Then m ₁ = m ₂ &
It is checked whether n ₁ = n _2, that is, whether the target point is the same as another target point (4107). If they are the same, it is not necessary to delete the isolated points, and therefore the process proceeds to step 4112, which will be described later.
M ₂ and n ₂ are set to (4108), and it is checked whether or not data exists at the memory address A _nm (4109). If there is no data, the process proceeds to step 4112. The data X _m and Y _n are read (4110). Initially, 1 is set in the registers m ₁ and n ₁ and m ₂ and n ₂ , so the flow advances to step 4112 to increment m ₂ by 1 and step 4
Since the process returns to 107, the coordinate data X ₁ and Y _{1 at the} address A ₁₁ are read as the target points and the coordinate data X ₂ and Y _{1 at the} address A ₂₁ are read as the other target points.

Next, it is checked whether or not the coordinate data X _m , Y _n of another target point exists in the detection region of the target point (411).
1) If it exists, the point of interest is not an isolated point and is left as it is, but if it does not exist in the detection area, there is one whether all other points of interest also exist in the detection area. Repeat until it is detected. That is, m ₂ is incremented by 1 until m ₂ becomes larger than 25 (4112,411
3), the n ₂ sets m ₂ to 1 when greater than 25 1 increments and (4114), n ₂ repeats the processing at steps 4107 to 4111 until greater than 17 (4115). Then, when there is no other target point in the detection area centered on the target point (YES in step 4115), m ₁ in the registers m and n,
sets the n ₁ (4116), to erase the address A _nm coordinate data in the memory (4117).

When another point of interest exists in the area where the coordinate data is erased or detected, the register m ₁ indicating the point of interest is set to 1
Increment (4118), return to step 4102, and leave the above processing (YES at step 4109) and repeat for all points of interest (4118-4121-4102-4118 ...). When the determination of whether all the points of interest are isolated points is completed (Y in 4121
ES), return. That is, ± 10 around the point of interest
If there is no other point of interest in the dot area, and if there is no other point of interest, the reference point of interest is the memory 10
Erased from. As a result, human eyes, eyebrows, hair, etc.
The attention points moved to the dark part are left, but the other attention points are erased from the memory table. With reference to FIG. 7, the processing content of "deletion of other attention points near the attention point" (42) will be described in more detail. First, the microprocessor 6
Sets 1 to registers m ₁ and n ₁ (4201), and register m 1
The values of ₁ and n ₁ are set in the registers m and n (4202), and it is checked whether data exists at the memory address A _nm (4203). This is because the data may have been erased from the memory 10 by the "deletion of isolated points" (41) process. It should be noted that m ₁ and n ₁ are registers indicating a reference point of interest. If there is no data, the process proceeds to step 4217, but if there is data, the coordinate data X _m and Y _n are read (4204). Next, X _m -5 is set in the register x ₁ and Y _n -5 is set in y ₁ to set the coordinates of the upper left corner of the area for detecting the presence or absence of another target point (4205), and register x ₂ To X _m +5 and y ₂ to Y _n +
Set 5 to set the coordinates of the lower right corner of the detection area (42
06).

Next, the m ₁ +1 in the register m _2, sets n ₁ to n ₂ (4207). As in "Deletion of isolated points" (41), m ₂ , n
_The reason why 1 is not set to ₂ is that the data remaining at the address A _nm of the memory is outside the detection area with respect to the point of interest before that and does not need to be detected in duplicate. Therefore, the registers m ₁ and n ₁ indicating the reference point of interest
On the other hand, the registers m ₂ and n ₂ of the other points of interest, which are the objects of existence in the detection area, may be checked if they are larger than m ₁ and n ₁ .

Next, m ₂ and n ₂ are set in the registers m and n (4208), and it is checked whether or not there is data at the memory address A _nm (4209).
12, the coordinate data X _m and Y _n are read if there is data (4210).

Next, it is checked whether or not the coordinate data X _m , Y _n of another target point exists within the detection area of the target point (421).
1) If it exists, the point is near the target point, and the coordinate data of the address A _nm of the memory is deleted (4216). on the other hand,
If it is outside the detection area, the point is left. These processes are performed for all remaining attention points (4212 to 4215 to 4217 to 4220 to 4202 to 4212, ..., Similar to `` deletion of isolated points '' (41).
・). When the determination as to whether all the points of interest are isolated points is completed (YES in 4220), the process returns. That is, the other points of interest within the ± 5 dot area around the point of interest are the memory 10
And the target point is set to be one in the area. As a result, among the points of interest that have moved to dark areas such as human eyes, eyebrows, and hair, the points of interest in the vicinity are set to one, and the number of remaining points of interest decreases.

With reference to FIG. 8, the control operation of "deletion by cross gradation difference" (43) will be described in more detail. First,
The microprocessor 6 sets 1 in the registers m and n
(4301), it is checked whether or not data exists at the memory address A _nm (4302). If there is no data, step 43
The process proceeds to 10, the coordinate data X _m If there is data, it reads out the Y _n (4303), the X _m -20 to register x ₁ to y ₁ Y _n -
20 is set (4304), and X _m +20 is set to y in the register x _2.
Set Y _n +20 to ₂ (4305).

Next, the area, y = Y _n & x ₁ ≤x≤x ₂ , that is, the brightness value of each point in the lateral direction ± 20 dots centering on the target point is detected, and the sum is set in the register T _H. (4306), the area, x = X _m & y ₁ ≦ y ≦ y ₂ , that is, the brightness value of each point in the vertical direction ± 20 dots centering on the target point is detected, and the sum thereof is set in the register T _V. (430
7). Then, T _H and T _V are compared (4308), and only when the vertical gradation T _H is larger than the horizontal gradation T _V , the target point is erased from the memory address A _nm (4309).

This processing is performed by "deleting other attention points near the attention point".
Perform all remaining attention points at the end of (42) (4310-4
313-4302-4310, ...). That is, as shown in FIG. 17, the size of human eyes is longer in the horizontal direction than in the vertical direction, and the horizontal gradation is darker than the vertical gradation. Memory as not in the eye position of the person
Erase from 10. As a result, the number of remaining attention points that have moved to dark areas other than the human eyes and eyebrows, particularly hair, is reduced.

The control operation of "deletion by template matching" (44) will be described in more detail with reference to FIG. First, the microprocessor 6 sets 1 in the registers m and n.
Is set (4401), and it is checked whether data exists at the memory address A _nm (4402). The process proceeds to step 4408 if there is no data, if there is data the coordinate data X _m, reads Y _n (4403), the X _m -30 sets Y _n -20 to y ₁ to the register x ₁ (4404 ), X _{m in} register x ₂
Set +30 to y ₂ and Y _n +20 (4405). Next, the brightness value of each point in the area, x ₁ ≦ x ≦ x ₂ & y ₁ ≦ y ≦ y ₂ , and the area of the template corresponding to the position of each point,
x ₀ −30 ≦ x ≦ x ₀ +30 & y ₀ −20 ≦ y ≦ y ₀ +2
The sum of the absolute values of the differences in the brightness value of each point within 0, is detected,
The register T _n m is set (4406). That is, the gradation of each point (2400 = 60 × 40) in the range of ± 20 dots in the vertical direction and ± 30 dots in the horizontal direction around the point of interest and the center of the human eye (x ₀ , y ₍₀ ) Similarly, each point in the range of ± 20 dots in the vertical direction and ± 30 dots in the horizontal direction (2400 = 60 × 40)
And the gradient of each of the corresponding points, (X _m −30, Y _n −20)
_{(x 0 -30, y 0 -20} ), (X m -29, Y n -20) and (x ₀
−29, y ₀ −20), (X _m −28, Y _n −20) and (x ₀ −2
8, y ₀ -20), ..., (X _m , Y _n ) and (x ₀ , y ₀ ),
_{··, (X m + 30,} Y n +20) and _{(x 0 + 30, y 0} +2
0), respectively. And the set T
Save _nm with address A _nm (4407).

This processing is performed for all remaining attention points at the end of the "deletion by cross tone difference" (43) (4408 to 4411 to 4).
402-4408, ...). Next, the maximum value and the minimum value in the total sum group are detected, the intermediate value T ₀ between them is calculated (4412), and the coordinate data of the address whose total sum is larger than the intermediate value T ₀ is erased from the memory (4413). This reduces the number of remaining attention points that have moved to dark areas such as eyebrows and hairs other than the human eye.

With reference to FIG. 10, the processing contents of "deletion due to diagonal gradation difference" (45) will be described in more detail. This is a process of performing deletion by gradation difference at a position obtained by rotating the cross in the vertical and horizontal directions centered on the point of interest of the above "deletion by cross gradation difference" (43) by 45 degrees. First, the microprocessor 6 sets 1 in the registers m and n (450
1), it is checked whether or not data exists at the memory address A _nm (4502). Proceeds Without data to step 4512, if there is data the coordinate data X _m, reads _{Y n (4503), Y n} -20 to X _m -20 to y ₁ in the register x ₁
Sets (4504), the X _m +20 sets Y _n +20 to y ₂ into register x ₂ (4505).

Next, the brightness value of each point is detected on the straight line connecting (x ₁ , y ₁ ) and (x ₂ , y ₂ ), that is, in the left diagonally upper direction to the right diagonally lower direction passing through the center of the point of interest. Then, the sum is set in the register T _RD (4506), and (x ₁ , y ₂ ) and (x ₂ ,
y ₁ ) on a straight line, that is, the brightness value of each point in the diagonally right upper direction to the left diagonally lower direction passing through the center of the point of interest is detected,
The sum is set in the register T _RU (4507). Then, T _RD and T _RU are compared (4508), and it is checked whether the difference between the two is within + 20% of the smaller gradation sum (4509, 4510). The point is deleted from the memory address A _nm (4511).

This processing is performed for all remaining points of interest at the end of the “deletion by template matching” (44)
(4512-4515-4502-4512, ...). That is, since the attention point located at the center of the human pupil has almost no difference between T _RD and T _RU , the attention point having a large difference is erased from the memory 10. This reduces the number of remaining attention points that have moved to the edge of the human eye or the edge of the eyebrow.

Thereafter, the remaining points of interest are identified in step 46.
In (Fig. 5), the small area and the point of interest are centered in the vertical and horizontal directions ± 2.
The "detection of dark spots in a small area" (46) of moving the target point to the darkest spot within the range of 0 dots is executed. This process (not shown) is the same as step 3 (FIG. 4), except that the area range is ± 20 dots (± 10 dots in FIG. 4) and that between step 31 and step 32 in FIG. The difference is that a determination process of "data is present at memory address A _nm " is provided, and if it exists, the process proceeds to step 32, and if it does not exist, the process proceeds to step 36.

Referring to FIG. 11, "integration of neighboring points of interest"
The processing content of (47) will be described in more detail. In "Delete other attention points in the vicinity of attention point" (42), other attention points within the range of ± 5 dots in the vertical and horizontal directions centering on the attention point were deleted, but in "Integration of neighborhood attention points" (47) , Horizontal and vertical ± 2
If another point of interest exists within the 0 dot range, the point of interest is moved to an intermediate position between the two points and the other point of interest is deleted. In addition, in the "deletion of other attention points near the attention point" (42), the registers m ₁ and n ₁ indicating the reference attention points are set to the other attention points which are the objects of presence or absence in the detection area. It suffices to check registers m ₂ and n _{2 that} are larger than m ₁ and n ₁ , but in “Integration of neighboring attention points” (47), the reference attention point moves, so Check all remaining attention points.

First, the microprocessor 6 uses the register m.
₁ is set to ₁ and n ₁ (4701), the values of the registers m ₁ and n ₁ are set to the registers m and n (4702), and it is checked whether or not data exists at the memory address A _nm. (4703) If there is no data, the process proceeds to step 4722, but if there is data, the coordinate data X _m1 and Y _n1 at the address A _nm of the memory are read (4704). m ₁ and n ₁ are registers indicating a reference point of interest. Next, register x ₁ has X _m1 -20 and y ₁ has Y _n1 -2
Set 0 to set the coordinates of the upper left corner of the area for detecting the presence / absence of another target point (4705), and set X _{m1 in the} register x _2.
The coordinate of the lower right corner of the detection area is set by setting Y _n1 +20 to y ₂ of +20 (4706).

Next, 1 is set in the registers m ₂ and n _2.
(4707). The registers m ₂ and n ₂ are registers indicating other points of interest with respect to the points of interest indicated by m ₁ and n ₁ . Then m ₁ = m ₂ &
It is checked whether n ₁ = n _2, that is, whether the target point is the same as another target point (4708). If they are the same, it is not necessary to integrate them, and the process proceeds to step 4718 described later. If they are not the same, m ₂ and n ₂ are set in the registers m and n (4709), and the data is stored in the memory address A _nm. It is checked whether or not it exists (4710), and if there is no data, the process proceeds to step 4718, but if there is data, the coordinate data X _m2 , Y _n2 is read (4711).

Next, the coordinate data X _m2 and Y _{n2 of the} other points of interest.
Check whether or not exists in the detection area of the point of interest (4
712) If it does not exist, there is no need to integrate it, and it is left as it is, but if it exists in the detection area, both points of interest are integrated. That is, the intermediate position (X _i , Y _i ) between the reference point of interest and the target point of interest is calculated (X _i = (X _m1 + X _m2 ) / 2,
Y _i = (Y _m1 + Y _m2 ) / 2), the attention point is moved to the intermediate position (X _i , Y _i ) and the coordinate data of the address A _nm of the memory is updated and stored (4713 to 4715) to be the target. The coordinate data of the target point is erased from the memory address A _nm (4716, 4717).

This processing is performed for all remaining points of interest.
(4718-4725-4702-4718, ...). As a result, the point of interest approaches the position indicating the center of the human pupil.

Referring to FIG. 12, "determination of the number of points of interest"
The processing content of (48) will be described in more detail. First, the microprocessor 6 registers 1 in registers m and n and 0 in N.
Are set (4801), and it is checked whether data exists at the memory address A _nm (4802). If there is no data, the process proceeds to step 4805, but if there is data, the register N is incremented by 1 (4803), and it is checked whether N is 2 or less (4804). If N is 2 or less, it is checked whether or not data exists in all areas of the memory (4805 to 4809 to 4802 to 4805, ...). That is, it is checked whether the number of data existing in the memory 10, that is, the number of attention points is the number of human eyes (= 2). If N becomes larger than 2 during this check (step 4804 NO),
Alternatively, if N is not 2 at the end of the check for all areas (step 4810 NO; when N = 0 or 1), the positions of both eyes cannot be specified by the points of interest.
The process returns to step 2 of 1), one frame image is input, and the above-mentioned “dark spot detection in a small area” (3) and “determination” (4) are repeated until N becomes 2.

As a result, if N is 2, the positions of both eyes are specified by the point of interest, so the coordinate data (m ₁ , n ₁ ) and (m ₂ , n ₂ ) in the memory are saved (4811). ). FIG. 18 shows a state in which the last two remaining points of interest specify the positions of both eyes of a person.

Referring to FIG. 13, "verification of attention point" (4
The processing content of 9) will be described in more detail. First, the microprocessor 6 determines that the center position coordinates (m1, n1) and (m2, n2) of the image detected as the eye in the above-mentioned “judgment of the number of points of interest” (48) clearly see the front in the steady driving posture. Then, it is checked whether or not it is within the steady eye region ECR shown in FIG. 19 which is set (in the processing program) in advance by adding a margin to the region on the screen where his eyes are located (4901), and within the region. In addition, the difference (m2-m1) in the x (horizontal direction) coordinate value of the center position coordinate is the standard interval (6c) between the pupils of both eyes.
It is checked whether it is within the range of (the number of pixels corresponding to m) ± allowable value (the number of pixels corresponding to 1 cm) (4902), and when it is within the range, the positional difference of the pupils of both eyes in the y direction (vertical direction) (| N2-n1 |) is less than a value (the number of pixels corresponding to 1 cm) which is slightly larger than the vertical shift caused by the temporary left or right inclination of the driver's face during driving or a slight steady inclination. Check if (4903). If any of the checks result in the area or the range, the center position coordinates (m of the image detected as the eye in the above “determination of the number of points of interest” (48) (m
1, n1) is considered to be correct,
The flag register F is cleared (synonymous with writing "0") (4904). The content “0” of the register F means that the eyes are normally detected. Then, the urging of the buzzer 5 is stopped (4905), and the process proceeds from the "determination" 4 to the "tracking" 5. Note that the buzzer 5 deenergization (4905) was detected by the "judgment" or "chase" before the "judgment" 4 this time, or a tracking error.
If it occurs, it is meaningful to stop the ringing of the buzzer 5 in the "judgment" 4 of this time because the detection was successful. If the buzzer 5 is not energized when the process goes to the "judgment" 4, the stop control of the buzzer that is not energized is performed, but no problem occurs in the buzzer control. By the way, the center position coordinates (m1, n1) and (m2, n2) of the image detected as the eye in the above-mentioned “judgment of the number of points of interest” (48) are the area E.
Is it out of CR or the difference in x coordinate value (m2-m1) is
Either the standard distance between the pupils of both eyes is out of the range of ± the allowable value, or the position difference in the y direction (vertical direction) (| n2-n1 |)
However, if the allowable value is exceeded, activate the buzzer-5.
(4906), the flag F indicating whether the buzzer 5 is energized or not energized is 0.
Check if it is (not working) (4907) and flag F
If is 0, the microprocessor 6 turns on the internal timer (4908), sets 1 in the flag F (4909), and indicates that the buzzer is being activated (4909). Then, it is checked whether the timer is over (4910), and if the timer is not over, the image is read.
Returning to (2 in FIG. 3), the “determination” 4 is executed again after the “small area dark spot detection” 3 is performed. The center coordinates of the image that has been determined to be “eyes” again in the “verification point verification” 49 of this determination are
YES in all of the checks 4901 to 4903 in FIG.
The flag register F is cleared (49
04), the urging of buzzer-5 is stopped (4905). However, once again, the "verification of the point of interest" 49 below results in that the eye detection is in error, and when the timer times out, the microprocessor 6 causes the "verification of the point of interest" 49 to occur.
(Last step 4910 of) to step 521 of FIG.
, And waits for input of stop instruction (SSW ON). The driver uses the switch S to stop the buzzer 5
When the SW is pressed, that is, when there is a stop instruction input,
The microprocessor 6 stops the buzzer-5 and returns to the initialization of step 1 (FIG. 3).

The processing contents of the "tracking" (5) shown in FIG. 3 will be described in more detail with reference to FIGS. In addition,
It should be noted that if the eye is accurately detected in the "judgment" 4 (the verification result with the eye is obtained in the verification 49 of the point of interest), the "tracking" 5 is performed. First, the microprocessor 6
Write the frame image data to the frame memory 12 (50
1). Then, m ₁ and n ₁ are set in the registers m and n. That is, a value indicating the address of the point of interest indicating the position of one of the two eyes of a person is set (502). Then, the coordinate data X _m , Y _n indicating the eye position is read from the memory address (503). Next, a small area (in this case, a range of ± 20 dots in the vertical and horizontal directions centering on the point of interest) is set, that is, X _m −20 is set in the register x ₁ in the same manner as “Dark spot detection in small area” (3). Y _n -20 is set in y ₁ to set the coordinates of the upper left corner of the small area (504), and X _m +20 is set in the register x _2.
Is set to y ₂ and Y _n +20 to set the coordinates of the lower right corner of the detection area (505). Then, the region, x ₁ ≦ x ≦ x ₂ & y ₁
The data in the frame memory 12 is read within ≦ y ≦ y ₂ to detect the darkest point of the brightness value, and the coordinates are updated to the memory address A _nm (506). As a result, the position of one eye that changes in time series is detected.
Next, the open / closed state of the eyes is checked. That is, the maximum value T _(max) and the minimum value T _(min) of the brightness values in the vertical direction ± 20 dot range centered on the target point after moving to the darkest point are detected, and the average value is calculated. , Set in the register T _AV (507). And the vertical (y) indicating the brightness value below T _AV
The width W ₁ in the direction is detected (508). In the schematic diagram shown in FIG. 14, the width in the vertical direction is W ₁ = | y′−y ″ |.

The same process is performed for the other eye to detect the width W ₂ in the vertical (y) direction (509 to 515). As a result, the degree of opening and closing of both eyes that changes in time series is detected.

By the processing up to this point, the eye position on the one frame image read this time (501 in FIG. 14) is determined based on the eye position detected based on the previous one frame image data. The eye position data on the one frame image which has been tracked and read this time is assigned to the address An1m1 by the processing of steps 502 to 506 of FIG. 14, and the step 509 of FIG.
It is stored in the address An2m2 by the processing of 513.

When the pupil distances W1 and W2 are calculated as described above, the microprocessor 6 executes the "verification of target point" ECS2 of FIG. The contents are the same as the contents of the “verification point verification” 49 shown in FIG. However, here, the check block 49 of “verification of the attention point” 49
01 coordinate data (m1, n1) is the address An1m1.
Data, coordinate data (m2, n2) is the address An2.
It becomes the data of m2. By executing "verification of target point" ECS2, it is verified whether the "eye" detected (tracked) on the one frame image read this time (501 in FIG. 14) is the correct eye. .

Referring to FIG. "Verification of points of interest" EC
When it is determined in S2 that the eyes are correct, the microprocessor 6 next checks whether or not the widths W ₁ and W _{2 of} both eyes are equal to or less than a predetermined value W ₀ (516). If it is larger than the predetermined value W ₀ , the driver determines that it is normal (not in the dozing state or looking aside) and returns to step 501 to continue the above-mentioned processing (501 to 516 to 501, ...). When the eye widths W ₁ and W ₂ are equal to or less than the predetermined value W ₀ , it is checked whether the flag F indicating whether the timer is operating is 0 (not operating) (517), and if the flag F is 0. For example, the microprocessor 6 turns on the internal timer (518) and sets the flag F to 1 to indicate that the timer is operating (519). Then, it is checked whether the timer has expired (520), and if the timer has not expired, the process returns to step 501 and repeats the processing.If the timer has expired, it is determined that the driver is in a dozing state or is looking aside. A stop instruction from the driver (stop switch S
Keeps issuing alarm until SW turns on (52)
1-522). If the width of any one of the eyes of the driver becomes larger than the predetermined value W ₀ while the timer is once turned on and the timer is not over, the flag (516) is set to 0 (524). The alarm is not issued when the eyes are temporarily closed (less than the set time of the timer) such as the natural blinking of. When the driver gives a stop instruction while the alarm is on, the process returns to the initialization of step 1 (FIG. 3).

In some cases, eye detection may fail during "tracking". At that time, the above-mentioned "verification of attention point" EC
In S2, the judgment result of the eye detection error, or
As a result of the above-mentioned "drowning state or looking aside while driving", the buzzer 5 is activated. Therefore buzzer-5
When it fails to detect the eye during "tracking", and
Sounds when the driver to be monitored is "drowsy or looking aside."

In the above-described embodiment, the driver's dozing or the like is detected in response to the opening / closing of both eyes of the person, and the alarm is issued if the driver is closed for a predetermined time. However, the present invention is not limited to this. Whether the right eye or the left eye is determined (the value of register m ₁ is compared with the value of m ₂ in “tracking” (5)), and only the right eye is closed for a predetermined time (intentional blinking) For example, the air conditioner control may be performed, and the bias control of the electric device disclosed in Japanese Patent Publication No. 4-284 may be performed. Further, the darkest point is detected in the small area for detecting the human eye, and the point of interest is moved to the darkest point, but it may be set to the brightest point or a predetermined brightness value. By performing template matching of a desired image, deletion processing by gradation difference, etc., the object to be photographed is not limited to the human eye, and other objects or a part thereof, such as a dark pattern or a bright pattern characteristic of the photographed image. It may provide a pattern.

In the above embodiment, the coordinates of the points of interest dispersed in advance at a predetermined pitch are written in the memory table, but this writing is omitted, and instead, for example, the first points of interest (68,
200) and write the coordinate data to the coordinate register, detect the darkest part of the periphery and write the coordinates to the memory table, and then set the x coordinate data of the coordinate register to 1
The coordinate data may be sequentially incremented and the points of interest may be sequentially specified, such that the coordinates of the second points of interest are incremented by 5 dots. In the above embodiment, the memory table is
Although the coordinate data of the target point is held in the blue memory, a frame memory for allocating one frame binary data table to the RAM 10 or a separate memory and writing the address to the multi-gradation data. The point of interest position information is binary-coded by writing "1" indicating the "point of interest" in the binary data table address corresponding to the coordinates of the point of interest in a one-to-one correspondence with the 12 addresses. -When the data is held in the table and the point of interest is confirmed (searched or read), the address is sequentially updated to 2
The data of the value data table is sequentially read and the read data is read.
The read address when the data is "1" (point of interest) may be saved (extracted as the address of the point of interest).

[0056]

[Effects of the Invention] For example, the driver's head goes down (face turns down) due to drowsiness, extra sight, or some other reason.
When, or when it comes to the sideways, or sideways, when the eye position by the image information processing means (6, 8) (m ₁ , n ₁ /
The detection of m ₂ , n ₂ ) becomes error or impossible. In such a case, the verification means (6, 8) activates the alarm means (5). This triggers an alarm, which alerts the driver.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an exemplary embodiment of the present invention.

FIG. 2 is a perspective view showing the external appearance of the arrangement of a CCD camera 3, an illumination lamp 4 and a buzzer 5 shown in FIG.

3 is a flowchart showing an outline of processing contents of a microprocessor 6 shown in FIG.

FIG. 4 is a flowchart showing the processing content of “dark spot detection in a small area” (3) shown in FIG.

FIG. 5 is a flowchart showing the processing contents of “determination” (4) shown in FIG.

FIG. 6 is a flowchart showing the processing content of “deletion of isolated point” (41) shown in FIG.

[FIG. 7] “Delete other attention points near the attention point” shown in FIG.
It is a flowchart which shows the processing content of (42).

FIG. 8 shows “deletion due to cross gradation difference” (4
It is a flowchart which shows the processing content of 3).

9 is a flowchart showing the processing contents of "deletion by template matching" (44) shown in FIG.

[FIG. 10] “Delete due to gradation difference in diagonal direction” shown in FIG.
It is a flowchart which shows the processing content of (45).

[FIG. 11] “Integration of neighboring points of interest” shown in FIG. 5 (47)
5 is a flowchart showing the processing contents of FIG.

FIG. 12 is a flowchart showing the processing contents of “determination of the number of points of interest” (48) shown in FIG.

FIG. 13 is a flowchart showing the processing content of “verification of attention point” (49) shown in FIG.

FIG. 14 is a flowchart showing the processing content of “tracking” (5) shown in FIG.

FIG. 15 is a flowchart showing the processing content of “tracking” (5) shown in FIG.

16 is a state diagram showing a state in which 425 points of interest are arranged in one screen distribution of image data to be written in the frame memory 12. FIG.

FIG. 17 is a block diagram showing vertical and horizontal sizes of a human eye.

18 is a block diagram showing the contents of a memory table of RAM 10 shown in FIG.

FIG. 19 is a state diagram showing how two attention points specify the positions of both eyes of a person.

[Explanation of symbols]

3: CCD camera 4: Illumination lamp 5: Buzzer 6, 8: Microprocessor 7: Address bus & control bus 9: ROM 10: RAM 12: Frame memory 17: A / D converter 18: Infrared lamp controller 19: Buzzer controller SSW : Stop switch

Claims (4)

[Claims] 1. A two-dimensional image pickup means for photographing a driver on a vehicle; a warning means; an image information processing means for processing an image photographed by the two-dimensional image pickup means for detecting the eye position of the driver; The position determined by the means as the eye position is
Verification means for activating the alarm means when it is determined whether it is inside or outside a predetermined area smaller than the screen area on the photographing screen; and the two-dimensional when the verification means determines that the area is within the predetermined area. An apparatus for monitoring the eyes of a driver on a vehicle, comprising: a tracking unit that tracks an eye position on a screen that is repeatedly captured by an image capturing unit. 2. The verification means further includes the alarm means when determining whether the eye position obtained by the tracking by the tracking means is inside or outside a predetermined area smaller than the screen area on the photographing screen. The device for monitoring the eyes of a driver on a vehicle according to claim 1, wherein the device is a moving device. 3. The drowsiness detecting means for detecting whether the eye image at the eye position tracked by the tracking means detects either eye opening or eye closing and activating the alarm means when the eye image is detected.
Alternatively, the device for monitoring the eyes of a driver on a vehicle according to claim 2. 4. The image information processing means converts the image signal obtained by the two-dimensional image pickup means into image data for representing multi-gradation and stores it in the storage means, and designates a point of interest at a predetermined pitch or randomly. Then, for each point of interest, the maximum or minimum position of the image data in a predetermined small area based on it is detected,
The position of the point of interest is changed to the detected maximum or minimum position, the isolated points of interest are erased, the points of interest that are close to each other are thinned, and the image data in a predetermined small area based on the remaining points of interest are used as the reference. , And whether or not there is an eye in the predetermined small area according to a discriminant logic that is predetermined according to the feature that appears on the shooting screen of the eye image. , A device for monitoring the eyes of a driver on a vehicle.