JPH02183811A - Energization controller for electrical equipment - Google Patents

Abstract

PURPOSE:To shorten the tracking time of a target and to attain the scan in real time by integrating the shift data on the target by the prescribed frequency and at each fixed time in a tracking scan process of the target and calculating the mean value and the scattering value of the data equivalent to the prescribed frequency to set a medium area window. CONSTITUTION:The shift data on a target are integrated at each prescribed time to calculate the mean value and the scattering value of the data equivalent to the prescribed frequency in a process where the target is tracked by a target tracking means I applying a narrow band window. When the calculation is through, a decision flag PFLAG of a medium area window setting possibility deciding means M is set at 1. A medium area window setting means N sets a medium area window having a high target catching probability assured statistically based on the mean value and the scattering value which are calculated and stored by a mean value/scattering value calculation means L. Then a target detecting means P detects the target by means of the medium area window. Thus the overall target tracking speed is improved for an electrical equipment.

Description

Detailed Description of the Invention [Field of Industrial Application] The present invention relates to an energization control device that provides an energization control signal such as on/off to an electrical device. Alternatively, the present invention relates to a device that automatically detects the movement of an object and controls an electrical device by applying an energizing signal to a required electrical device in response to mouth movements without contact.

[Prior Art] With the advancement of CCD (charge-coupled device), this type of energization control device for electrical equipment has been used as a monitoring and control device for high-speed moving objects in printing photographs, paper manufacturing, electro-X, machining, textiles, and electric wires. Practical applications are progressing in the pharmaceutical, food, and other industries. However, its scope of use is limited to the detection of specific movements of specific target objects under the selected best lighting conditions, and is limited to the detection of minute objects that undergo complex changes under varying lighting conditions ( No technology has been developed to track the movements of a car driver's eyes or mouth, for example, to control certain electrical devices.

In order to solve this problem, the present inventor has conducted various studies and developed an energization control device for electrical devices that controls various electrical devices in response to movement or changes in a narrow area of an object (e.g., eyes or mouth). I applied for an invention.

(Jikai No. 62-247410) The energization control device for an electric device according to the application includes an image input means for converting and inputting optical information of an image into digital image information, and a target object for detecting the position of a specific target portion of the image. The detection means is used to photograph an object or a person (hereinafter, a driver on a car will be cited as an example) and detect the position of a specific target part such as a portion of the photographed image, such as the driver's eyes and mouth.

Since the brightness inside the vehicle fluctuates, the vehicle is equipped with a lighting device that keeps the brightness of the lighting device constant and prevents image processing errors due to fluctuations in brightness.

It also tracks changes in the position of the face due to vibrations of the vehicle body, unconscious minute movements of the driver, or changes in posture, as well as intentional changes in the position of the mouth or eyes to control electrical devices, accurately adjusting the position of the eyes and eyes. In order to extract the mouth from photographic information, a storage means for storing a detected position, a window setting means for setting an area narrower than the imaging screen of the imaging means based on the stored position, and the specific target portion of the narrow area. and storage means for updating and storing the position of. As a result, after the position of the pupils or mouth is detected, the information scanning for detecting the pupils and mouth on the screen is limited to a narrow area, the time to detect the pupils and mouth is shortened, and the position of the pupils and mouth is detected. Can be tracked quickly and accurately.

The energization control device for an electric device further includes a target object tracking means for detecting time-series state changes of the pupils and mouth, and a control signal discriminating means for supplying a control signal corresponding to the state change of the target object to the target electric device, When an intentional change for controlling the electrical device appears in the eye or mouth being tracked, a control signal corresponding to the change is given to the electrical device.

As described above, in the energization control device for electrical equipment disclosed in Jikai No. 62-247410, the driver remains in the driving position.
Electrical devices can be controlled by moving the eyes or mouth, allowing for comfortable and safe driving.For example, when you give a command by speaking, the electrical device is controlled according to the shape of your mouth. Even if you are not making a sound, if you intentionally make the same mouth shape as when you are making a sound, the electrical device will be controlled according to this mouth shape, and since it does not depend on the sound, it is possible to eliminate detection errors caused by noise inside the car.

However, when the brightness of the object to be imaged (for example, the face of a vehicle driver) is relatively uniform across different parts of the face, typically there is no oncoming vehicle at night and only the interior lights illuminating the face are used. When the illumination is substantially for imaging, the detection accuracy of the driver's head, face, and eyes is high, and the above-mentioned effects can be obtained. However, when a driver's face and head are exposed to the strong headlights of an oncoming or following vehicle at night, or when the sunlight is strong during the day, the driver's face and head reflect stronger external light than the headlights. It may be blocked. At this time, the intensity of reflected light in only one part of the face is strong, while other parts are in shadow and dark, which often results in uneven face illumination.1 For example, when driving under clear skies when the sun is on your right. , the area around the right eye becomes extremely bright and the area around the left eye becomes extremely dark. In the case of such non-uniform illumination, the device described above binarizes the entire captured image using one threshold value, so there is a problem that the accuracy of detecting the driver's pupils and the shape of the driver's mouth also decreases. There was a point.

In order to solve the above-mentioned problems, the present inventor conducted further research and determined that the first
A gradation histogram is calculated, a threshold value is determined for each area based on this, and the image gradation information is binarized, so that the density gradation of the plurality of adjacent areas in the imaged object (for example, a face) is determined. 2
A boundary that changes in value (for example, the boundary between hair and forehead) is detected, a second tone histogram of image tone information of a narrow area (for example, an area including the eyes) based on the detected boundary is calculated, and this is An electric device is equipped with a detection means that determines a threshold value based on the above, binarizes the image gradation information of the narrow area, and detects the position of a specific small part (for example, a pupil) within the narrow area. He invented and applied for a power control device.

(Patent Application No. 169325/1982) Setting a threshold value based on a gradation histogram and binarization processing based on the threshold value are useful when the background in the screen (e.g. It is suitable for cutting out objects in front of the background (for example, the head) from the back sheet), and even if there is non-target illumination and brightness fluctuations of the illumination itself,
It is possible to cut out and separate a portion (for example, the pupil) where the gradation changes rapidly within a required region from the background (for example, around the pupil) with high precision.

Therefore, according to this patent application No. 169325/1982,
Even if an object is asymmetrically illuminated or the illumination itself has large variations in brightness, it is possible to detect a specific part with high precision.

[Problem M to be solved by the invention] However, according to the above-mentioned Jikai No. 62-247410 and patent application No. 62-169325, under various driving situations encountered by a driver, a target object within a narrow area (e.g. Although it has become possible to track and detect the eyes (pupils or mouth) in time series, the narrow area window frame for tracking is fixed to a predetermined size in advance, and when the window frame size is set small, Although the detection time becomes faster, there is a greater possibility that the target object will deviate from the frame during tracking.Also, if the frame size is set large, the possibility of tracking failure becomes smaller, but the detection time becomes longer. If detection fails, the process returns to the initial step, detects the image boundary area, sets a wide window, redetects the target using the wide window, and then re-tracks the target using a narrow window of the same size. However, there is a problem in that it takes a long time to enable re-tracking after tracking has failed.

That is, in the prior art configuration shown in FIG. 9a,
If the target object detection means G using a wide area window and the target object tracking means I using a narrow area window fail to detect or track the target, the process returns to the image boundary detection means E, and the wide area window setting means F then detects the target in the wide area window. Through means G and narrow window setting means H,
In order to re-track the target object with the target object tracking means I, complicated steps are required to discover the target object and make re-tracking possible, and a long time is required.

[Means for Solving the Problem] The present inventor solved the above problem by accumulating movement data of a target object measured at predetermined time intervals a predetermined number of times in the process of tracking and scanning a target object with a target object tracking means. means for calculating the average value and variance of data for a predetermined number of times, and further changing and storing the average and variance values sequentially each time new data is input; When the initial values of the average value and the variance value are determined, a midrange window setting possibility determining means M for instructing that the midrange window can be set; This problem was solved by providing a region window setting means N and a target object detection means P using a mid-range window for detecting a target within the set mid-range window.

[Operation] In the configuration diagram of the present invention shown in FIG. 9b, in the process of tracking a target by the narrow window target tracking means I, movement data of the target is accumulated a predetermined number of times at predetermined time intervals, and The average value and the variance value of the data for a predetermined number of times are calculated, and when the calculation is completed, the determination flag DFLAG of the mid-range window setting possibility determining means M is set to 1.

The medium range window setting means N uses the average value and variance value calculated and stored by the mean value and variance value calculation means to set a medium range window with a high statistically guaranteed probability of capturing the target. A target object is detected by the target object detection means P using the mid-range window.

If tracking of the target fails while tracking the target using the narrow window target tracking means (■), the process returns to the medium range window setting possibility determining means M, and if DFLAG is set to 1, the medium range window is set. is determined to be possible, the target is promptly re-detected through the medium-range window determination means N and the target object detection means P using the medium-range window, and the target object tracking means I using the narrow-range window is used to track the target. resume.

Therefore, in the initial state at the time of startup of this device, while DFLAG is set to 0 by the medium area window setting possibility determining means M, if tracking of the target fails, the image boundary detecting means E, the wide area window setting means Although it is necessary to re-track the target after F. target detection means G using a wide area window and narrow window setting means H, after a predetermined time has elapsed and DFLAG is set to 1, the medium area Window setting means N, target detection means P using a medium range window
By-pass circuit equipped with 100mm enables quick re-tracking of the target.

Embodiment FIG. 1a shows a block diagram of an embodiment of the present invention, which turns on-vehicle electrical devices on and off in response to intentional movements of the eyes and mouth of the driver's face. This is an on-vehicle control device that performs off, up/down, etc. control.

Image input means A includes a television camera 3 and an A/D converter 1
The television camera 3 includes a two-dimensional COD (charge-coupled device) that obtains an image signal (analog video signal) of 256×256 pixels per frame. To this TV camera 3,
Microprocessor 6 via interface 17
gives an on/off signal. The television camera 3 repeats the brightness signal of 256 x 256 pixels per frame and performs A/
It outputs to the D converter 18, provides a pixel synchronization pulse to the A/D converter 18 as an A/D conversion synchronization pulse, and provides a frame synchronization pulse, a line synchronization pulse, and a pixel synchronization pulse to the microprocessor 6 via the interface 15. In this example, the A/D converter 18 converts the video signal into 8-bit (256 gradation) digital data.

The lighting means B is composed of a lighting lamp 4 and a lighting controller 19. The lighting lamp 4 is an incandescent light bulb that uses a filament as a light emitting source, and the lighting controller 19 chops DC voltage with a thyristor chopper and supplies the lighting lamp 4 with a thyristor chopper. Apply. The lighting controller 19 turns off the thyristor chopper when the control signal is off, which indicates off, and when the control signal is on, which indicates on, the lighting controller 19 first controls the thyristor chopper on/off at a standard duty.
When a signal instructing brightness up (brightness increase) arrives, the duty is updated one step higher.When a signal instructing the brightness up and down is reached, the duty is updated lower by one step. Stop down there. lighting controller 19
The microprocessor 6 is connected to the interface 14.
The above-mentioned on/off and up/down control signals are provided through the .

The television camera 3 of the image input means A and the illumination lamp 4 constituting the illumination means B that illuminates at least the driver's face are:
As shown in FIG. 1b, it is integrated into the instrument panel 2, and the pointing direction can be adjusted vertically and horizontally.
Fixed.

Mode switches 5E and 5M are connected to the microprocessor 6 to constitute target selection means C.

The switch 5E is for specifying the pupil response mode, and when the switch 5E is closed (instructing the Ilfi response mode), the microprocessor 6 turns on the electrical device in accordance with the pupil position and blink pattern. Control.

Switch 5M specifies oral response mode,
When this switch 5M is closed (indicating mouth response mode), the microprocessor 6 controls the electrical device in response to the mouth shape pattern.

When the microprocessor 6 of the image control storage means reads one frame of imaging data (image data),
The frame memory (RA) is synchronized with the frame synchronization pulse.
M) 13 to write, and advance the write address of the frame memory 13 in synchronization with the line synchronization pulse and the pixel synchronization pulse.

The microprocessor 6 is connected to another image calculation microprocessor 8 that performs image data calculations, and stores one frame's worth of binarized data (one bit per pixel indicating the presence or absence of an image point). Frame memory (RAM) for storing data) 11. A ROM 9 and a RAM 10 for processor control, a bus controller 16, and the above-mentioned frame memory 13 are connected, and electrical devices 20 to 30 to be controlled and a lighting controller 19 are also sold separately.

The image calculation microprocessor 8 reads the ROM 9 according to the control instructions from the input/output control microprocessor 6.
The program built into the frame memory 11.
M image boundary detection means E wide area window setting means F target object detection means
G Narrow window setting means H
Means for tracking the target using a narrow window ■ Means for calculating the average value and variance of the movement of the target L control signal discriminating means
J mid range window setting means
N Target detection means using medium range window P
The following arithmetic processing is performed by each means.

The medium range window setting possibility determination means M is set to 0 at initialization when the average value and variance value of the movement of the target object are calculated in the average value and variance value calculation means of the movement of the target object. Using the flag DFLAG, when DFLAG=O, it is impossible to set the midrange window, and DFLAG=
When the value is 1, the operation of the midrange window setting means N is enabled.

The image boundary detection means E binarizes the input image information and detects the boundaries of the image.

The wide area window setting means F sets a wide area window for scanning and detecting a target from the detected boundary.

The target detection means G scans the wide area window to detect a target object.

The narrow window setting means H sets a narrow window for tracking the detected target.

The middle range window setting means N sets a middle range window for detecting a target that has failed in tracking and has been lost, using the average value of the movement of the target object and the results calculated by the variance value calculation means. .

Target object detection means P using a mid-range window detects a target object using the window.

Once the size of the narrow window for tracking the target object and the center position for setting the window (the center of the detected target object) are determined, one new frame of digital image information is taken in, and the target tracking means I , tracks the target and returns to means M if tracking fails.

The average value and variance value calculation means for the movement of the target object accumulates the movement data of the target object tracked at predetermined time intervals a predetermined number of times, calculates the average value and analysis value of the data for the predetermined number of times, and further calculates the average value and the analysis value of the data for the predetermined number of times. Average values and analysis values are updated and stored every time new data is input. Moreover, when the initial values of the average value and analysis value are obtained, the DF
Setting LAG to 1, medium range window setting determination means M
Set it so that it determines that it is configurable.

When the motion pattern of the tracked target matches a predetermined pattern, the control signal determining means J detects the electric device controlling means K.
The control signal is then transmitted.

To summarize the above image processing sequence, (1) one frame's worth of digital image information is initially M-E-F-G
-H → I-L → J, and while the target is successfully tracked, J-I-L → J → I is repeated, and when tracking fails, ■ → M-E → F →G-H→■→L→J is processed.

(2) If tracking fails after it becomes possible to calculate the average movement value and analysis value of the target while repeating the tracking work, ■→M-N→P→I→L→ Proceed with J, and repeat J→IL→J→I while tracking is successful.

(3) Since the mid-range window set by means N is a window with a statistically high probability of target acquisition, the probability of failure in acquisition is extremely low.

Furthermore, in the event that tracking fails, the means N sets a middle range window based on the latest average value and analysis value each time, thereby enabling high-speed re-tracking of the target.

In this embodiment, two microprocessors are used to
Although the microprocessor 6 is used for input/output control and the microprocessor 8 is used for calculation, it is of course possible to use one microprocessor for both input/output control and calculation processing.

The electrical device control means includes a lighting controller 19;
It is composed of controllers shown in figures 20 to 29 and 30.

The radio controller 20 electrically controls the power on/off of the radio and the step up/down of the volume. When the activation signal indicates on, the radio controller 20 turns on the radio power and turns off the radio. If the signal indicates "up", the radio power is turned off, and when the activation signal indicating "up" arrives once, the volume is increased by one step. When the limit is reached in up and down, up and down are respectively stopped.

The air conditioner controller 21 only controls on/off of the air conditioner. A controller in the air conditioner itself (not shown) controls cooling/heating selection, power up/down, cleaner on/off, etc. in response to room temperature and air pollution.

When autodrive is instructed, the autodrive (automatic constant speed driving) unit memorizes the vehicle speed at that time and then controls the throttle opening so that the actual vehicle speed becomes the memorized vehicle speed, and autodrive is canceled. (off) to stop this throttle opening control. Alternatively, when autodrive is instructed, the throttle opening is controlled so that the actual vehicle speed matches a preset or stored vehicle speed value, or a vehicle speed value modified by these, and autodrive is canceled. When it is turned off (off), this throttle opening control is stopped. In either case, when an up command is given, the target vehicle speed is updated and set one step higher, and when a down command is given, the target vehicle speed is updated and set one step lower. The auto drive controller 22 controls this on/off and up/
If the energization signal indicates on, it starts auto drive, if it indicates up, the target vehicle speed is updated and set one step higher, and if it indicates down. Set the update to l step lower.

The door lock controller 23 controls the driver seat lock/unlock switch and all door lock/unlock switches attached to the driver seat door, and each door lock/unlock activation attached to other doors. However, if the activation signal (ONloFF) indicates ON, each door is activated to be locked in sequence unless other blocking conditions are satisfied. When the energizing signal indicating OFF arrives, each door is energized to unlock in sequence unless other blocking conditions are satisfied.

In this example, the sunroof controller 24 controls the opening and closing of a sliding sunroof, and controls the opening and closing of the sunroof in response to the operation of a normally provided switch.
/DOWN) indicates on, the sunroof is fully opened, when it is off, it is fully closed, and when it is up, the sunroof is closed by one step (predetermined opening degree). However, when it indicates a down state, it is driven closed by one step.

The window controller 25 controls the opening/closing (up and down) driving of the sliding glass of each door. The controller 25 controls the opening/closing drive of the sliding glass in accordance with the operation of a normally provided switch, and when the energizing signal (ON10F/UP/DOWN) indicates ON, the sliding glass of the driver's door is opened/closed. When it indicates OFF, it controls full open drive, and when it indicates up, it controls close drive by one step.
If it indicates a down state, the closing drive is controlled by one step.

The wiper controller 26 controls the drive of the wiper in response to a switch operation normally provided, and outputs an energizing signal (ON).
When 10FF/UP/DOWN) indicates ON, the wiper drive is energized at the standard speed, when it indicates OFF, the wiper drive is stopped and the wiper is positioned at the standby position, and when it indicates UP, the wiper speed is changed one step higher. When indicating down, set one step lower.

The cigarette lighter controller 27 causes the cigarette lighter retraction/extrusion mechanism to be retracted when the energizing signal (ONloFF) is turned on, and to be pushed out even if the cigarette lighter is not at a predetermined temperature when the energizing signal indicates OFF. to strengthen The rest is the same as the conventional one, and the cigarette lighter is energized by mechanical contact in the retracted state, and at a predetermined temperature, the cigarette lighter is automatically pushed out and the energization is cut off.

The headlight controller 28 responds to the large light/small light/off switch, turns on the small light when the switch is in the preferred position, raises the headlight from the retracted position, and turns on the headlight, and responds to the operation of the angle indicator switch. to set the beam angle, and when the small position is reached, the headlights are turned off and only the small lights are turned on, and when the off position is reached, the small lights are turned off and the headlights are retracted.
When the energizing signal (ON10FF/UP/DOWN) indicates ON, the small light is turned on and the headlight is raised from the retracted position. The illumination angle is raised by one step upwards, and lowered by one step if it is a down command, and if it is off, the headlights are turned off and only the small lights are turned on.

The defroster controller 29 energizes/de-energizes the built-in heater in the rear window according to the on/off switch operation normally set by INN, and also energizes the heater (
When ONloFF) is on, power is supplied to the embedded heater, and when it is off, power is stopped.

The buzzer controller 30 activates a buzzer (not shown) when the activation signal is on, and stops the buzzer when the activation signal is off. As will be described later, this buzzer is activated when the driver's eyes remain closed for a predetermined period of time or longer (assuming that the driver is dozing off). Further, it is activated when image pattern processing becomes impossible while the mode switch 5E or 5M is closed (@response control and/or mouth response control is instructed).

The above is an overview of the configuration of the present invention using the block diagram of the embodiment, and will be explained below using the flowchart. 2a and 2b are main flowcharts of the present invention, and FIGS. 3a to 3u are subflowcharts explaining the structure and operation of each functional means.

Referring to FIGS. 2a and 2b, when the power is turned on, the microprocessor 6 executes initialization (step l, hereinafter the word step will be omitted in parentheses),
Initialize the input/output board, set the electrical devices 19 to 30 and the television camera 3 to the standby state (off), and set the internal registers, counters, flags, and RAMs 10 and 11.1.
Clear 3. And switch 5 of target selection means C
If either E or 5M is closed, the microprocessor 6 provides an energizing signal to the lamp controller 19 of the lighting means E and the television camera 3 to indicate on. 5E,
5M are both open, an energizing signal indicating OFF is given to the illumination light controller 19 and the television camera 3, and an energizing signal indicating OFF is given to the buzzer controller 30 of the electric device control means, Give a warning to the driver.

By closing at least one of the mode selection switches 5E and 5M, an energizing signal indicating ON is given to the lighting controller 19 and the television camera 3 of the image input means A for a predetermined period of time (the brightness of the lighting lamp is stabilized). , the waiting time for the driver's face to stop at the driving position), one frame (256 x 256 pixels) worth of image data (
The gradation data (8 bits per pixel) is written to the frame memory 13 (5).

Next, the microprocessor 6 calculates the gradations of the image data in the memory 13 from the lowest level (gradation O: highest brightness) to the highest level (gradation 255: darkest). Calculate the number of pixels, create a brightness histogram, and store it in RAM 10 (6). Calculate the deviation value (deviation per each gradation) between the histogram in (6) and the standard histogram stored in ROM 9, and square the deviation. is added, and the added value is compared with the reference value (7). If the added value is out of a predetermined range, the lighting lamp controller 1 sends an energizing signal to instruct up or down according to the sum of the deviations.
9 (10), and after a predetermined time (time until the brightness of the illumination lamp 4 stabilizes to a brightness corresponding to the new setting), one frame of image take is read into the memory 13 again (5). In this way, the brightness of the illumination lamp 4 is adjusted so that the histogram of the image data falls within a predetermined range (5.6.7.10).

In this adjustment, the lighting controller 19 sets the brightness to the highest brightness and the lowest brightness.
A signal indicating this is sent to the microprocessor 6, and when the processor 6 receives a signal indicating that the lowest brightness has been set (8), it instructs the lighting controller 19 to turn off (11), and then adjusts the brightness. After the illumination lamp 4 is turned off, one frame of image data is read and written to the memory 13 (12). When a signal indicating that the brightness has been set to the maximum is received, the brightness is Finish adjusting the height (9).

After completing the brightness adjustment, the determining means (12A) constituting the mid-range window setting possibility determining means M tests whether or not the mid-range window can be set. Since (12A) is cleared in step (1) and DFLAG=0, the process advances to the "determination of head detection threshold THh" subroutine (13).

The following subroutine (1) for determining the threshold value THh for head detection constitutes the image boundary detection means E and the wide area window setting means F.
3) “Determination of threshold value THf for forehead detection” subroutine “Determination of threshold value THe for pupil detection” subroutine and “Head detection” subroutine that constitutes target object detection means G (15) “Forehead detection” subroutine (22) "Pupil detection" subroutine (27) is sequentially executed, and the "pupil tracking window We setting" subroutine (34) constituting the narrow window setting means H is executed.
) and rro tracking window Wm setting subroutine, the pupil center positions ECX, ECY, the mouth center position MCX,
Setting of MCY, narrow window We for pupil tracking, and narrow window Wm for mouth tracking is completed, and preparations for tracking the pupil or the mouth are completed.

The steps up to this point will be explained step by step again with reference to FIGS. 2a and 2b.

[Determination of head detection threshold THh] subroutine (
13), the threshold value THh is determined, and then "0: Unsuccessful" is first set in the flag register 5FLAG that stores data indicating success or failure of head detection (14), and the subroutine for "head detection" (15) is set. ), the head tip position ATY (Fig. 4c) is detected using the threshold value THh. When the detection is successful, flag register 5FLA is set in subroutine (15).
Set 1 to G, and if unsuccessful, set the corresponding register 5FLAG
The content of is left as O.

When subroutine (15) ends, register 5FLAG
(16), and if it is O, register I for counting the number of errors (unsuccessful).
If you want to update the content of the PDF to a value 1 higher than 17), it will be 1.
If it is 6 or more, the buzzer controller 30 is instructed to turn on the buzzer (19), and the process returns to image reading (5).

When the value is less than 16, the buzzer is not energized and the image is read (5
).

If the contents of the register 5FLAG exceeds 0 in the check at step 16, then the [amount detection threshold TH
f setting] subroutine (20) is executed. Then, the flag register 5FLAG is cleared (21), and the "forehead detection" subroutine (22) is executed. If forehead detection is successful in this subroutine (22), register 5FLA
Put 1 in G, and if unsuccessful, register 5FLAG
The content of is left as 0.

When the "forehead detection" subroutine (22) is finished, the contents of register 5FLAG are checked (23), and if it is 0, the contents of register IPDF for counting the number of errors (unsuccessful) are increased to a larger value. Update 17
), check if it is greater than or equal to 16 (
18), if it is 16 or higher, the buzzer controller 3
0 to instruct the buzzer to turn on (19), and image reading (5).
). When the number is less than 16, the buzzer is not energized and the process returns to image reading (5).

If the contents of register 5FLAG exceeds O in the check in step 23, 0.05 is added to each of registers 1 and 2 for storing constants for pupil detection.
and 0.4 are written (24), and the subroutine (25) of "determination of threshold value THe for pupil detection" is executed.

When the subroutine (25) ends, flag register 5
FLAG is initialized (26), and the "pupil detection" subroutine (27) is executed. In this subroutine (27), if a detection is performed without satisfying the second condition (fine condition), 2 is written to register 5FLAG, and if a detection is performed which does not satisfy the second condition but satisfies the first condition (coarse condition), register 5FLAG is written.
When it is detected that neither the first nor second conditions are satisfied, the contents of the register 5FLAG are set to O.

When the subroutine (27) ends, register 5FLAG
(28, 29), and if it is 2, proceed to the calculation of the pupil center position in steps 31 and below.If the content of register 5FLAG is 1, the "threshold change" subroutine (30) Execute the pupil detection threshold (determination of 1aT He) subroutine (25) again.
). If the contents of register 5FLAG are O, the process advances to step 17.

Now, when the content of register 5 FLAG is 2, first instruct the buzzer controller 19 to turn off the buzzer.
), then clear the error count register IPDF (32), then execute the subroutine (33) of []Calculating the child's center position ECX, ECY' to obtain the pupil center position data, In the subroutine (34) of ``Setting the window We'', a small area of a predetermined area centered on the center position of the pupil is determined.In the subroutine (35), both ends of the eyebrows BRX and BLX (Fig. 41) are detected, Calculate the eye area in subroutine (36), and subroutine: / (3
B) TO(7) central position ffiMcX, McY (4th
(Fig. j) is detected, and in a subroutine (39), a small region Wm of a predetermined area centered on the center position of the mouth is determined.

Now, when the sizes of the pupil tracking window We and the mouth tracking window Wm and the current center coordinates of the pupils and mouth have been determined in the previous steps, the middle range window setting possibility determining means (12A) separates merges with the bypass,
The following mouth 1 or mouth tracking work is started.

First, a new image is captured (40), and the switch 5 of the target object setting means C is checked (41) (42), and the process proceeds to the "pupil tracking" subroutine (43) that constitutes the target object tracking means (■) to move the pupil. is tracked, and if successful, a subroutine (45) is used to calculate the average value and variance of the movement of the target object.
Proceed to (5) if tracking fails.

In the subroutine (45), the coordinate data of the 11g center position of the images that are sequentially input at a synchronous speed (1/30 second) is input a predetermined time (for example, 1 second) R a predetermined number of times (for example, 300 times).
collect data, calculate the average value and variance value of the data for a predetermined number of times, set the frame setting possible determination flag register DFLAG to 1, and calculate the average value and variance value of the movement of the pupil center each time new data is input. Update and always store the latest average value and variance value for window settings. The details will be explained in the description of the subroutine (45).

Next, the process proceeds to the control signal determining means J, and in the "blinking of the eyes determination" subroutine (46), the time and interval of the blinking of the eyes are calculated, and if it is determined that the driver is drowsy, a warning is issued with a buzzer to determine whether the driver is intentionally driving. If it is determined that it is a signal blink, a predetermined signal is transmitted to the electrical device control means.

Next, the next “judgment based on pupil movement direction” routine (47
), by comparing the pupil movement vector when focusing on a predetermined part of the device to be operated with a predetermined value, a signal is sent to the desired electrical device control means, and the process returns to (40).

The above are the steps in the initial state when the operation of this device is started, and until the data is accumulated a predetermined number of times (for example, 300 times) in the [Calculation of average value and variance value of target movement] routine (45), If tracking of the pupil fails (when the center position of the pupil cannot be determined), the process returns to step 5 each time it fails, and the center position of the pupil is determined again in step 33 via the above-mentioned steps.

However, in step 45, a predetermined number of times (for example, 300
After the data has been accumulated, DFLAG is set to 1, and if tracking fails, the process proceeds to step 50 as determined in step 12A. A range window is set, and the middle range window is used to detect the center of the pupil in step 51, and the process returns to step 40 to capture the image.
Tracking can be continued in the narrow window We (4
3).

Therefore, after DFLAG is set to 1 in step 12A, if there is no tracking failure, repeated tracking is performed at high speed between step 40 and step 47, and even if tracking fails, step 50 is executed. The latest mid-range window is set, allowing quick resumption of tracking. That is, a new detection window is set in the subroutine (50) for setting a mid-range window that constitutes the setting means N, and a subroutine for detecting pupils using a mid-range window that constitutes the high-precision target detection means P is performed. In (51), the pupils are detected,
If the detection is successful, the process advances to step (40) to capture an image to continue tracking the pupil.

The above is the main routine for tracking the eyes, and the routine for tracking the mouth includes a "mouth tracking" and "variance value calculation of mouth movement" subroutines in step 48.
This becomes possible by converting the medium range window setting means N and the target object detection means P using the medium range window into daily use.

The details of the subroutine will be explained below.

Referring to FIG. 3a, "Head detection threshold T I-1h
The contents of the subroutine (13) of ``determination of ``determination'' will be explained. This subroutine (13) is executed by the microprocessor 8 in response to instructions from the microprocessor 6. On the imaging screen, for example, as shown in FIG. 4a, almost the entire face of the driver in the driver's seat is contained within one frame Fin,
Furthermore, the imaging magnification of the camera 3 is determined so that the tip of the head and its background (inner surface of the vehicle roof) are both present. The microprocessor 8 first determines in the image data processing area of the memory 13 a small area Htp that is set (fixed) so that the probability that the tip of the head and the background both exist is highest (55), and A simple gradation histogram of the image data of p is calculated (56).As explained earlier, the image data is 8 bits, and the gradation ranges from O to 255.
There are 256 gradations. In calculating this gradation histogram, the processor 8 reads out all the image data in the area Htp in a predetermined order, and counts the number of images showing the same gradation for each gradation.

Next, processor 8 clears register Bmax (
57), calculate the average density Mt of the tone histogram (
58). Then, clear the gradation register i (include the data indicating the floor? JJO) (59), and then clear the number of pixels wl in the gradation histogram that are less than the gradation i (contents of register i).
and its average density Ml, the number W2 of pixels with gradation level 1 or higher, and its average density M2 are calculated (60). B=
wl(Ml-Mt)2+w2(M2-Mt)'' (61), that is, the total concentration M of the first group.
The sum of the square of the agricultural degree deviation (for each group) and that of the second group, that is, the discrimination degree B of the first group is calculated. Note that the gradation i is different from the area Htp in density 2
When the group is clearly distinguished, the value of B is large, i.e., the value of B is large.

Next, the processor 8 checks whether the calculated B is greater than or equal to the contents of the register Bmax (62), and if yes, updates and writes B to the register Bmax (63), and sets i to the threshold register THh. and updates the contents of register i to its current contents plus 1 (
65).

In step 62, if B is less than the contents of register Bmax, the process directly advances to step 65.

Next, the processor 8 determines that the content i of the register i is 256 or more (the B operation has been completed for all gradations).
(66).

If not, the calculations in steps 6o and 61 are performed again. In this way, when the content i of register i becomes 256 or more, the processor 8 notifies the processor 6 that the processing has ended, and the processor 6 returns to the main routine ( 2a) is resumed (proceeding to step 14).

At this time, B calculation (
60, 61) has finished, and register Bmax contains
The highest value of all the B values obtained by the calculation is entered, and the gradation level i at which the highest value is obtained, that is, the threshold value THh is written in the threshold register THh. For reference, FIG. 4b shows a binarized image obtained by binarizing the image Fim shown in FIG. 4a using the threshold value THh obtained in this way.

In step 14, processor 6 clears register 5FLAG. Then, the processor 8 is instructed to execute the "head detection" subroutine (15).

Referring to FIG. 3b, the "head detection" subroutine (15
). The processor 8 enlarges the area Htp by twice in the sub-scanning direction Y of the screen to create an area 2Htp (a fourth area).
b) is set in the image data readout area of the memory 13 (67), and the area 2 Ht is set in the line register LN.
Minimum address data of p in main scanning direction X (right end position)
68).

Next, the X address is LN (contents of register LN 2 and below,
The image data of each pixel on the vertical line (line extending in the y direction) of the register (the same symbol as the register symbol indicates the contents of the register to which the symbol is attached) is sequentially read out from the memory 13, and the image data [O ;Maximum brightness (white)/
255: The lowest brightness (black)) is binarized with THh,
The number of pixels BNP of black (gradation data of T Hh or more) is counted (69), and it is checked whether BNP exceeds 50 (76). If it does not exceed 50, LN is in the area t[2
Check whether the left end value of Ht p) has reached the max (71), and if it has not, update the contents of the register LN to a value larger by 1 (72), and count the number of black pixels on the vertical line LN. (69) In this way, when the number of black pixels BNP exceeds 50 for one of the vertical lines,
Assuming that this is the right end of the head, LN at that time is written into the right end register ARX (73).The above is the detection of the right end position ARX of the head.

In the next steps 74 to 79, the left end position tlRALX of the head is similarly detected and written to the left end register ALX.
Indicates LX.

Next, the processor 8 calculates the head AW=ALX-ARX, writes it to the register AW (80), and then writes it to the line register LN at the Y address Y of the upper end of the area 2Htp.
Write min (81) and write THh of horizontal line LN.
Count the number of black pixels BNP (82), then B
Check whether NP exceeds AW/2 and
If it does not exceed W/2, the contents of register LN will be in area 2 Ht
Check whether the lower end position Ymax of I) has been reached (
84), and if the value has not been reached, the contents of the register LN are updated by one increment (85), and the number of black pixels on the horizontal line LN is counted (82). In this way, for one horizontal line, When the number of C black pixels BNP exceeds AW/2, it is assumed that this is the tip of the head, and the LN at that time is loaded into the tip register ATY (86), and the flag register 5F is loaded.
Add 1 to LAG (87). Above is the tip of the head 1f
1ATY detection. When you finish this, processor 8
notifies the processor 6 of the completion of processing.

In response to this notification, the processor 6 restarts the main routine and first checks whether the content of the register 5FLAG is O (16), and if it is not O (the head has been successfully detected). ) and instructs the processor 8 to execute the "Determination of forehead detection threshold value THh" subroutine (20).

FIG. 3c shows the content of the subroutine (20) for "determining threshold value THf for forehead detection." In this subroutine (20), the processor 8 first executes the subroutine (88) to X=△LX (see Figure 4c)
The threshold value THfr of the right half region (X = ARX side), which divides the range of Then, in the next subroutine (89), the left half area (X=
The zero-order processor 8 determines the threshold value T Hf t on the ALX side), which determines the threshold value THfr and the threshold value THfL.
(90), and if THfL is larger than THfr, THfL is written to register THfm (91), and TH
When fr is greater than or equal to T Hf L, THfr is written to register THfm (91), and then X=ARX to X=
The range of ALX is divided into three equal regions 1 to 3 as shown in FIG. 4C, and the threshold value THfr of the register THfr is set in region 1.
, threshold value THfm of register THfm in area 2,
Also, in area 3, the threshold value T Hf of the register T Hf L is
L is assigned (93), and here processor 8 notifies processor 6 of the completion of processing. In response, the processor 6 clears the register 5FLAG (21) and instructs the processor 8 to execute the "forehead detection" subroutine (22).

The content of the "forehead detection" subroutine (22) is shown in FIG. 3d. The processor 8 writes the head tip position ATY into the line register LN (94), clears the register MXL, register CK and register RL, and Write the head rightmost position [ARX (95) to XP, then y address is L
The image data of the pixels in the range from ARX to ALX of N horizontal lines are sequentially read out, and it is checked which of the regions 1 to 3 the pixel is in. If the pixel is in the region 1, THfr is changed to the region 2. The image data is binarized using THfm as the threshold when it is in region 3, and THfb as the threshold when it is in region 3. When white is obtained by binarization, the run length (continuous length) RL is counted and the line L
The highest value of the white run length of N is loaded into the register MXL (96, 101-112). When black appears on the way, the previous count value RL is compared with MXL, and the current count value RL is higher than MXL. If it is large, store this in register M
Write an update to XL, clear register l)L, and when the mouth appears, start counting the run length again. ARX to AL of the horizontal line whose y address is LN
When this is completed for the range up to X, the longest value of the continuous length of the number of white pixels in the horizontal line will have been written in the register MXL.

Here, the processor 6 performs steps 96 to 97.
Proceed to and check if the longest run length MXL exceeds AW/2. If not, check if LN has reached the maximum value of 255 (98), 25
If the value is not 5, the contents of register LN are updated to a number larger by 1 (99), and the maximum value of the white run length below step 95 is detected. If it is 255, this is used for forehead detection. It means that it has failed (Register 5FL
Since we return to the main routine without putting 1 in AG,
The main routine proceeds from step 23 to step 17).

Now, if MXL>AW/2 in step 97, LN at this time is the upper part of the forehead (write HXL to i7HW, write MXL to forehead position register HW, and write MXL to forehead position register HW).
Writes LN to TY and writes 1 to register 5 FLAG (100). When this is completed, processor 8 notifies processor 6 of the completion of processing. At this point, register HT Y has the upper 11qATY of the amount, register HRX has the rightmost position 1iZ HRX of the amount, register HLX has the leftmost position HLX of the amount, and register HW has the width HW of the upper end of the amount.
is written. These are shown in Figure 4d.

When the processor 6 receives the completion notification, it checks the contents of the flag register SFLΔG (23), that is, it checks whether or not the forehead detection was successful. And 5th floor
If the content of LAG is 0 (failure), the process proceeds to step 17; if the content is 1 (success), 0.05 is written to register Kt, 0.4 is written to register 2 (24), and processor 8
The subroutine (25) of "determination of threshold value THe for pupil detection" is instructed to be executed.

In Figure 3e, "Determination of pupil detection threshold 'I'He"
This shows the content of the subroutine (25), which means that in the screen area Sd (Figure 4d), the sum of pixels that are the edges of eyebrows, eyes, etc. (edges where the gradation changes extremely: black edges) is A threshold value T He that is a predetermined number of times suitable for distinguishing those pixels from others is detected. Processor 8 first calculates the coordinates (HRX, HTY) and (HRX+HW
/2, HTY+HW) as the diagonal point (113), and calculates the difference tone histogram of the area Sd (114), that is, the gradation indicated by the image data of the pixels existing in the area Sd. Let the gradation be i, and the number in Sd of pixels with gradation i be 01, and the gradation of each of these pixels j: From AI, the gradation of the pixel that is one previous (value y months smaller) than pixel j in the y direction. Assuming that the difference obtained by subtracting the tone is dDj, the square mean of the difference f1M DFMi=[Σ(dDj)"]/ni is calculated for each tone i (0 to 255). Note that the absolute value of the difference is An average value may be used.Then, from the gradation i where the average value DFMi is the highest value DFMmax, the gradation i is set as the threshold value (i t 5).
, binarize the image data of the area Sd, count the number sb of pixels that become black as a result (116),
Checks whether the ratio Sb/S of sb to the total number of pixels S of d is within a predetermined range +<Sb/S<K2 (117
), KI and NI2 are the main routine (second
These are 0, 05, and 0°4 set in step 24 of Fig. a). If it is not within the predetermined range, update the gradation i to the gradation that yields the next largest root mean square value of the difference (118).
, the image data of Sd is binarized using it as a threshold, and the number of black pixels S of the binarized image similarly satisfies K l< S
Check whether b/S<K2 is satisfied (116.117). If it is within the predetermined range, the gradation level i written in the register i at that time is written into the threshold value register THe (119), and the processor 6 is notified of the end of the process.

The processor 6 clears the register 5FLAG (26), and the processor 8 instructs the processor 8 in the "pupil detection" subroutine (
27).

FIG. 3f shows the contents of the "pupil detection" subroutine (27).
Image data T ranging from TY to the lower end HTY+HW
Binarize with He, calculate the dimensions of each black area in the binarized image, determine whether it matches the eye size, and if it matches well, set 2 to register 5FLAG.
If an eye-like object is detected although it does not match well, 5FLAG is set to 1. If nothing is detected, the contents of 5FLAG are left as 0.

That is, first, register CK is cleared, and line N next to front line HTY is stored in line register LN.

(120), the image data in the Sd area of the horizontal line LN is binarized using the threshold value THe, and the number of black pixels BPN is detected (121), and the black of that line is Compare the number of pixels BPN with 4 (122
), this is to detect whether or not that line is the starting end of a certain black area in the Y direction. If it is 4 or less (not the start end), it is checked whether the Y direction scanning position has reached the lower end of Sd (123), and if it has not, the contents of register LN are updated to a number larger by 1 ( 124), the number of black pixels BPN of the line LN is detected 121), and it is compared with 4 (122).

When BPN exceeds 4, it is assumed that the starting edge of a certain black area in the Y direction is detected, and the BPN is stored in the maximum black pixel number register MXL, the vertical width register W is cleared, and LN is written in the black area upper starting edge register TP. The black area lower end register BP is also L.
Enter N by four (127), and check whether the Y direction scanning position has reached the lower end of Sd (128), and if it has not, update the contents of register LN to a number larger by one (129), Count the number of black dots in the same way as step 121 (130), and compare the count value BPN with 4 (131). If it is 4 or more, the black area is still in this LN.
Seeing that it is also on the line, compare the contents of the maximum sunspot number register MXL with BPN (132), and if BPN is larger, update and write BPN to register MXL (133)
, and also checks whether the Y-direction scanning position has reached the lower end of Sd (128), and if it has not, similarly counts the number of black dots on the next line (129, 130) and counts the count value BPN of 4. This is performed by sequentially updating the line position ti2LN to a larger y value, and when BPN becomes less than 4, it is assumed that the lower end of one black area in the y direction is reached, and the lower end position register BP is compared with (131). Write LN to , and write BP-'rp+l to vertical width register W (
134), and the vertical width W of the black area detected in this way
Checks whether it exceeds HW/20 (in the case of eyes, it exceeds it) (135), and if it does, the width MXL is HW.
Check whether it exceeds /15 (in the case of pupils, it exceeds) (136), and if it exceeds, the ratio of width to height M
Checks whether XL/W (in the case of a pupil, it is 1 if the eyelids are not closed) is 3 or less (137). If it is 3 or less, it is assumed that a pupil is detected, and W is written in the pupil height register EYEW. TP is written in the pupil lower end position register ETY, BP is written in the pupil lower end position register EBY, and 2 is written in the flag register 5FLAG, indicating that the pupil has been detected with certainty (138).

Then, from step 140 via step 139, CK
=O, so proceed to step 141 and register 5F.
Clear LAG, write 1 to register CK, and start from the next horizontal line as described above (7) MXL, W, T.
Detect each value of the next black area, such as P and BP (124
-121-...127-137), the first
The first time the pupil is detected with certainty, it is usually the eyebrows, so the next time the pupil is detected with certainty, it is definitely the pupil. Each value of the other black area is detected.

In this second reliable pupil detection flow, each value detected in step 13B is written in the register in the same way as described above, and the process proceeds to step 139, and from there to step 140. This time, CK= 1, so “
The pupil detection subroutine (27) is terminated, and the processor 6 is notified of the end of the process. By the way, when the condition of step 137 is not satisfied, that is, MXL/W is 3.
When it exceeds ^, this is a binary image in which the entire eye and/or its surroundings are integrated with the pupil, or the eyebrows and the eye are continuous, such as (a) in Figure 4g or ((a) in Figure 4h).
As shown in a), there is a high probability that a large black area is formed.

In this case, it is assumed that there is a pupil in this black area (something that looks like a pupil has been detected), and TP is entered in the pupil lower end position register ETY, and BP is entered in the pupil lower end position register EBY, but in register 5FLAG. writes l (142
), proceed to step 139 and from there to step 124
Proceed to.

Now, when the sunspot number count value BPN is less than 4 and the y scan advances to the lower end (HTY+HW) of the area Sd, step 122
-123, and then in step 125, it is checked whether the contents of register CK are 1 or not. If this is 1, the pupil was reliably detected once (steps 137-138-1
39-140-141), but since the same detection was not performed again after that, the flag register 5FLA is set so that the first detection value is the pupil detection value.
2 is rewritten in G (126), and this subroutine (27) is ended. The contents of register 5FLAG are O(
(Detection failure) or l (An undetermined pupil-like object was detected: (a) in Fig. 4g or (a) in Fig. 4h). In this case, the subroutine (27) is ended there.

When the subroutine (27) ends T in this manner, the processor 8 notifies the processor 6 that the processing has ended.

In response, the processor 6 registers the flag register 5FL.
Check the contents of AG28゜29), and if it is 1, in order to obtain a binary image with a clearer pupil shape,
Execute the subroutine (30) of "Change threshold value".

Its contents are shown in FIG. 3g, namely register K l
Update and write the value obtained by multiplying the contents of 2 by 0.7 to the register, update and write the value obtained by multiplying the contents of 2 by 0.7 to the register 2 (143), and then return to the main routine. Returning, the subroutine (25) of ``Determination of pupil detection threshold value THe'' is executed. This content is shown in FIG. 3e, but here, step 117
Since the value of and KN is 0.7 times the previous value,
The threshold THe determined here is a larger value than the previous one, and when the image data of the area Sd is binarized in the next "pupil detection" subroutine (27), the area of the black area is For example, the black area around the eye appears in the shape of the eye as shown in FIG. 4g (b) or FIG. 4h (b). Therefore, register 5F
There is a high possibility that 2 will be written to LAG. Furthermore, if 1 is written to flag register 5FLAG even in the execution of the current "pupil detection" subroutine (27), the "threshold change" subroutine (30) will be written in the same way. ), “pupil detection threshold TI (
The subroutine (25) for "determination of e" and the subroutine (27) for "pupil detection" are executed.

Figure 4e shows the relationship between the pupil shape and the detected f+'ri,
Fig. 4f shows the right eye area determined from the detected values already obtained.

Now, in the "pupil detection" subroutine (27), register 5
When 2 is written to FLAG, processor 6
3 instructs the buzzer controller 30 to turn off the buzzer.
1), clear the error count register IPDF (
32) In subroutine 33, the detected center positions ECX and ECY of the right pupil are calculated. This is the area shown in Figure 4f (coordinates (HRX, ETY) and (HRX+HW/
2. A rectangular area whose diagonal point is EBY) is regarded as the right eye area, and the binarized data of that area by THe is given to the microprocessor 8 to perform gravity center calculation (center calculation of black pixel distribution). Obtain the center of gravity position data and convert it to an address on the image frame. The center position data obtained by this is x=EcX, y=EcY.

Next, the microprocessor 6 outputs center position data ECX,
A window We (FIGS. 5a to 5f) defining the pupil tracking scanning area centered on ECY is set (34). The area of window We is ECX-EW NECX+EW.

ECY-EW NECY+EW, where EW is the width of the pupil. This window We is
■It is a very small area compared to the area of the frame.

Next, both ends BRX and BLX (Fig. 41) of the eyebrows are detected (35). In this embodiment, the energization IJ control of the electric device corresponding to the shape and position of the eyebrows is not performed, but it is necessary to detect the eye area. In detecting both ends of the eyebrows BRX and BLX (35), this is done to find the scanning range of
The X-direction scanning range is HRX-HLX, the y-direction scanning range is ETY-HW/2 to ETY-17EYEN, and the number of black pixels is counted for each scanning line within the range, assuming that the scanning line is the X direction. The X address BRX of the rightmost black pixel of the line with the largest number is memorized as the rightmost position, and the X address BLX of the leftmost black pixel is memorized as the leftmost position. If this detection fails, step 17 in Figure 2a
Proceed to.

Next, a scanning area for mouth detection is calculated and a threshold value for the area is determined (36). In other words, the scanning area for mouth detection is set to BRX~BLX, ETY-EMW~255 (lower edge of the frame). (Figure 4j). BMW is a constant. Then, in the same manner as the threshold value determination by the differential tone histogram calculation in subroutine (25), the threshold value T Hm of the scanning area is set, and the mouth detection is performed in the same manner as in the "pupil detection" subroutine (27). If sufficient detection is not possible, the "Change threshold value" subroutine (
In the same manner as in 30), set a threshold value that allows sufficient detection.

Next Nicono area BRX-BLX%ETY-EMW~255
The image data of is binarized with a threshold value THm, and the binarized image is operated with x=BRX, y=ETY-EMW as the starting point, and the scanning line is horizontal (X), and the number of black pixels for each line is calculated. , and each time the counting of the number of black pixels of the - line is completed, the count value is compared with the maximum f1m held, and if the count value of the negative line is large this time, it is updated and held as the maximum value. In this way, when the count value of the number of black pixels for each line is greater than the count value of all lines, scanning is proceeded to the next line. If the count value of the current line is smaller than the maximum value held, it is assumed that the line just before it was on the horizontal line of the mouth (the tangent line between the upper and lower lips), and the y address MCY of the line just before it is (Fig. 4j) is memorized as the X address of the center of the mouth, and the value MCX between the black start end and the black end of the previous line is memorized as the X address of the center of the mouth (mouth center position MCX, Calculation of MCY: 38) If detection fails, proceed to step 17 in FIG. 2a.

Next, the microprocessor 6 generates center position data MCX.
, MCY, a window Wm (FIG. 8a) is set (39) to capture the tracking scanning area of the mouth, and the area of the window Wm is MCX-0, 6MW-MCX+0 .
6 MW.

MCY-MH-MCY+MH1 where MW is the width of the mouth and MH is the height of the mouth, which are stored in memory. This window Wm is a very small range compared to the area of the image l frame.

Next, the processor 6 newly loads the image data for one frame into the memory 13 (40), binarizes it, and converts it into 2
In this binarization in which value image data is written to the memory 11, the area of the window We is the above-mentioned THe, the area of the window Wm is the above-mentioned 'l'Hm, and the image data is written into the memory 11.
Value.

After this binarization, if the switch 5E is on, ``pupil tracking J (43)'' is executed, and if the switch 5M is on, ``mouth tracking J (48) is executed.

``The contents of pupil tracking J (43) are shown in Figure 3h and 3.
This will be explained with reference to FIG. Note that the window We area has a threshold value THe, the window Wm area has a threshold value THm (determined in subroutine 36), and other areas belonging to area 1 have a threshold value THf.
r, the area belonging to region 2 is the threshold value THfr, and the area belonging to the area 3 is the threshold value T Hf L, which is 2
(The image data for l frames thus binarized is loaded into the memory 11.

Proceeding to pupil tracking (43), a window is set at the center of the previous pupil (501), and subroutine ``Line Scan J'' (502) is entered (see Figure 3h). (See FIG. 31), the center X coordinate data of the window We is vertically scanned from top to bottom to count the number of consecutive blacks, and when the count value reaches a predetermined number, the upper end of the pupil MTY
(Fig. 5a) is detected (144, 145), the y address MTY at that time is memorized, and then the X coordinate is kept as it is, scanning from bottom to top, counting the number of consecutive black pixels, and counting the number of consecutive black pixels. However, when the number reaches a predetermined value, the lower end of the pupil MB
Assuming that Y is detected, store the y address MBY at that time in memory (146), then calculate the center WCY (vertical center position) of the top and bottom ends (147), then set the X coordinate to WCY and move from right to left. Scan horizontally to detect the right end MRX of the pupil (Figure 5b) and store that address MRX in memory.
48) Next, scan from left to right to detect the left end MLX of the pupil (Fig. 5b), store that address MLX149), and calculate the center WCX of the left and right ends in the horizontal direction.
50), the obtained data WCX and WCY are regarded as the center of the pupil, and the process returns.

If the detection of the upper end of the pupil fails in step 144 described above (the pupil is shifted to the left or right from the center of the window), a horizontal line passing through the center of the window We is scanned from right to left, and the MRX (predetermined Black sequence of numbers; Figure 5C)
is detected (152), and once this is detected, the next step is to scan from left to right to detect MLX (Fig. 5c) 154
) Calculate the center WCx of the right end and left end of l1lfi.
55), Next, scan the data of x = wcx from top to bottom to detect the top end MTY (Figure 5d)156), then scan from the bottom to find the bottom end MBY157), the top and bottom ends in the vertical direction. The center WCY of is calculated. (158), the data wcx and wCY thus detected are regarded as the center of the pupil, and the process returns.

MTY could not be detected in step 144, and
If MRY cannot be detected in step 52, step 1
59 (Fig. 31), the binary data is scanned from the upper left corner to the lower right corner of the window we (step 5e).
Figure: This is done by setting the start address for reading binary data from the memory 11 to the upper left corner of the window We, and then sequentially incrementing the x and X addresses by m and each at the same time. 159), then scan the binary data from the lower right corner to the upper left corner of the window We (set the start address to the window W
As the lower right corner of e, x and y are simultaneously decremented (m and n, respectively) to detect point B (lower left corner of the pupil) and its address MRX%MBY is memorized, (161),
Calculate the midpoints wcx and wcy of the detected address, and (
162, 163) Memory. The data WCX and WCY thus detected are regarded as the center of the pupil, and the process returns.

MTY could not be detected in step 144, and
If MLX cannot be detected in step 52 and point A cannot be detected in step 159, the process advances to step 164.
2 from the upper right corner of window Wa toward the lower left corner
Scan the value data (Figure 5f: Read binary data from the memory 11. Set the start address at the upper right corner of the window We, then sequentially decrement the X address by m, and increment the X address by n. ), A
Detect the point (diagonally upper left corner of the pupil) and write its address MRX,
Store MTY in memory (164), then scan the binary data from the lower left corner to the upper right corner of the window We (
Set the start address to the lower left corner of the window We, sequentially increment the X address by m, decrement the , calculate the midpoints WCX and WCY of the detected addresses, and (1
67,168) memorize. The data WCY and WCX thus detected are regarded as the center of the pupil, and the process returns.

Sub-subroutine "Line scan J (502>) determines the result of tracking in (503). If the tracking is successful, the process returns to the main routine; if it fails, the process proceeds to (504), and the position of the pupil from the time before the previous time and the previous pupil position are determined. The position of the pupil that failed to be tracked is predicted from the position of (see Figure 5g), the window is set at the predicted position (505), and the pupil position is tracked again using the sub-subroutine ``Line Scan J'' (502). Then, if the judgment of (507) is successful, return,
If it is a failure, the window is set to the initial position again (50B) and a third "line scan" (502) is performed.

In step 44, it is determined whether or not the tracking was successful. If it is unsuccessful, the process returns to step 12A, and if it is successful, it is determined in step 45.
Proceed to.

``Calculation of average value variance of target object movement'' subroutine (45) constituting means for calculating average value variance value of target object movement
constitutes the most important part of the present invention, and the 3j
The details 332 will be explained in detail in FIG.

In this embodiment, a case is illustrated in which data is collected 300 times every second. Counter TI shown in step (510)
ME is a frame synchronization pulse counter, which counts 0.1°2...129 every time the video is switched.
If TIME=O, jump from step (510) to step (522) and set (TIME+1) mod
30, that is, the remainder of dividing (TIME+1) by 30 is returned as a new TIME.

In other words, TIME is 1.2, 3. ...If it is 29, jump to the step (522), sequentially 2, 3, 4.
．． ...Converted to 29.0. The next pulse entering step (522) at TIME=29 is therefore TIM
Since E=O is counted, the counter TIME is O,
l, 2. ...29°0.1,2. ...29 will be repeated, and when TIME=O, that is, l/30
A signal that enters every second will proceed to step (511) for the 30th time, that is, every second, and by this procedure, data that enters at a synchronous speed of 1/30 second will be captured every second. The Rukoto.

The data captured every second is sent to the DFL in step 511.
If AG is O, the process goes to step 516; if it is l, the process goes to step 512; DFLAG is set to O at initialization in step 1.
is still set to , so naturally step 5 is the first step.
Proceed to step 16 and set the pupil center coordinates as X (0) and Y, respectively.
(0) and (517)', (518) sU
MX, SUMY, SUMXX% SUMYY are calculated, and in step 521, they are processed by DTP-(DTP+l) mod 299, and DTP=1 and the process returns.

Here, DTP is the number of times data is captured every second, and is 0, 1, 2. In the process of progressing, DPT=
299, that is, the 300th time, in step 519, YE
It is determined that the value is S, and the process proceeds to step 520 where DFLAG is set to 1.
is set to (DTP+1)mo at step 521.
In response to the process of d300, DTP is set to O and the process returns.

That is, the number of data t111 captures, that is, the data number starts from 0 and changes to 1, 2, 3, etc. every second. ..., and when it reaches 299, it is restored to 0, and continues to be 0.
1, 2. ...299 will be repeated, and 300
It becomes possible to capture data for 10 times.

When the DTP reaches 299 (corresponding to the 300th time) in step 520, 1 is set in DFLAG and 301
The data corresponding to the 300 times are processed from step 511 to step 512, and the pupil center data A variance value is calculated.

Average value, variance value calculation sub-subroutine (513), (
515) (see figure 3), it should be noted that a procedure is adopted in which the average value and variance value for the first 300 times are updated every time new data is received.

Note that if DFLAG is set to 1 in step 520 (see Figure 3J), step 12A of the main routine
Is it possible to set the frame? In the determination (see FIG. 2a), it is determined that the setting is possible, and the process proceeds to the bypass from step 50 onward. Therefore, the "pupil tracking" subroutine (43
), when tracking fails and returns to step 12A,
It is possible to immediately proceed to the "setting middle range frame" subroutine (50).

In this example, the data capture interval and number of captures are
Although the optimal values are 1 second and 300 times, the capture interval should be changed depending on the synchronization speed, and it is important to note that if the number of captures is 300 times, the captured data will have a normal distribution. This is determined based on prediction, and if a normal distribution can be predicted even with a smaller number of captures, there is no problem in reducing the number of captures.

Next, the subroutines ``Mid-range window ideal setting J (50)'' and ``Pupil detection using a mid-range window J (51)'' constituting the mid-range window setting means N and the target object detection means P using the mid-range window of the present invention are executed. Explain the details.

Referring to FIG. 3n and FIG. 5h, the subroutine "Setting middle range window" (50) includes the subroutine "Pupil tracking J (43)" and "Calculating the variance value of the average value of target object movement J ( 45), the average value of the latest pupil movements at the time when pupil tracking failed XBER%YBE
Using R, variance values XBUNSAN, YBUNSAN and pupil width EW, center on XBER, YBER, and set the coordinates of the diagonal vertices WXI, WYl and Wx2, WY2 as ('t'LWX1=XBER-3nτ11)l-EWW
X2=XBER+3 XBUNSAN+EWwy1=
yBER-3tF'fi Go Ao "Ding AN-EWWY2=Y
Calculate and set the midrange window frame for BER+3 "7" Ding πN+EW (Figure 5h).

In ■, ``1 block Wπ and 1 block JN block X ball have standard deviations σ)c and ay in the X direction and Y direction, respectively, and 3σ
By employing y, it becomes possible to set a highly accurate window (with extremely low probability of detection failure) using the 30 method.

Of course, it is not necessary to limit it to 3a, and it is possible to pursue even higher accuracy.The point is that it is fine as long as it is possible to set a statistically guaranteed high probability window. Assuming that the data over 300 times is normally distributed, the 30 method guarantees a probability of 99.7%.

Subroutine configuring the target object detection means P using the middle range window [Pupil detection J using the middle range window (51
) 11 is completed in Figure 3p, Figure 3q, Figure 3r, Figure 6a.
6b.

This subroutine further includes “pupil region detection” (528).
and “determine whether or not the pupil is a pupil” (529).
Since it is similar to (27), detailed explanation will be omitted. (
6a and 6b).

The above is a description of the control signal determining means J which, when the intentional movement pattern of the eyes or mouth to be tracked matches a predetermined pattern that enables a predetermined electrical device by 1++ J ta, sends a signal to control the electrical device. Now that we have finished explaining each subroutine except for , the control signal discriminating means J will be explained below.

The control signal discrimination means J is composed of a subroutine 46 for "judgment based on the blinking of the pupils," a subroutine 47 for "judgment based on the movement direction of the pupils," and a subroutine 49 for "judgment based on the movement of the mouth," as shown in FIG. 2b. has been done.

The “judgment based on blinking of the eyes” subroutine (46)
This will be explained with reference to FIG. 3β and FIGS. 7a and 7b.

In Figure 7b, the normal blink time is 0.2 to 0.3 seconds for eye opening and 1.0 to 5 seconds for one blink.
．． In the present invention, in order to distinguish between involuntary eye-closing time and intentional blinking for signal transmission, the predetermined time TI, T2,
T3 is defined and the detected eye-closing time T is TI<T<T2.
If so, the eyes are intentionally closed for signal transmission, and the eye opening time T
If T > T3, it is a normal, unconscious eye-opening time, and if the eye-opening time T is T<T3, it is an intentional eye-opening time. A flag CT for the number of times of eye closure is provided for a signal to calculate the number of times of eye closure, and C
It is specified that when T=1, the radio switch is operated, when CT=2, the air conditioner switch is operated, and when CT=3, the headlights are operated.

According to the above rules, as shown in FIG. 7a, in the "judgment based on blinking of pupils" subroutine (46), the pupil aspect ratio EH/EW is calculated in steps 550 and 551, and compared with the judgment value in step 552. Are your eyes open?
Determine whether the eyes are open. When the eyes are closed, the eye-closed time T is detected (553), and the eye-closed time is set to a predetermined value TI,
554) (556), when T<72, a buzzer sounds as a warning that there is a possibility of drowsy driving557), and furthermore, detects the eye opening time TERM before closing the eyes (558), TERM> At T3, it is determined to be normal and the flag CT is set to 0.

When TERM≦T3, this eye closure is considered as intentional eye closure for signal transmission, and the flag CT is set to CT+1.

In step 552, if it is determined that the eyes are open, the eye opening time TERM before opening the eyes is detected (562), and TERM≦
If T1, it is judged as normal operation and returns, then TERM
> Tl, it is determined that the eyes are intentionally closed for signal transmission, examines CT in step 564, returns when C=0, operates radio SW when CT=1, and sets CT=2. If , turn the air conditioner on/off; if CT=3, turn the headlight switch on/off; in either case, clear the flag CT and return.

Next, the “judgment based on the direction of pupil movement” subroutine (47)
This will be explained with reference to FIG. 3m and FIG. 7c. This control signal discriminating means J needs to match the movement vector of the pupil with a predetermined vector of the desired controller when the driver focuses on the controller of the electrical device that he/she desires to control. The arrangement is
set in the appropriate position. In this example, FIG. 7e (
As shown in C), the positions of the controls for radios, air conditioners, and other devices are -Y1, +Y2, Y3 in the vertical direction and horizontally & m from the center of the driver's normal pupils, respectively.
-XI, +X2, -X3G: are set, and in subroutine 47, the deviations X and Y from the initial position of the pupil are detected in steps 570 and 571, and the deviations X and Y from the initial position of the pupil are detected in steps 572 and 5.
After 73,574,575, air conditioner, radio, aluminum
Controls 1 device, 8 devices, and C device.

According to the determination means based on the blinking of the pupils, the control device of the present invention can be used extremely effectively as a drowsy driving prevention device in addition to its original purpose. It will be easily understood that by counting the number of times the number of movements exceeds m, it can be effectively used as a device for preventing dangerous distracted driving.

The "judgment based on mouth movement" subroutine (49) realizes an extremely wide range of control signal discrimination means J in the present invention, and is shown in Fig. 3s, Fig. 3, Fig. 3u, Fig. 8a,
The embodiment will be described in detail with reference to FIG. 8b.

In FIG. 3s, the microprocessor 6 first reads the binary data of the window Wm updated and set in the mouth tracking (48) from the memory 11 and gives it to the microprocessor 8. When the microprocessor 8 receives the binary data of the window Wm, it first sets the line direction to vertical (y) and aligns the first line with the left end of the window Wm. The number of consecutive black pixels (black run count). When the black run length exceeds a predetermined value, it is assumed that that point (X address of the line) is the right edge of the mouth, and the X address NM at that time is determined.
RX is memorized (207), then the line address is set at the right end of the window Wm, the line address is similarly updated to the left, the black run length on each vertical line is counted, and the count value is a predetermined value. If it is above, it is assumed that this is the left end of the mouth, and the X address NMLX at that time is memorized (208), and the mouth (7) width MW = NMLX - NM
RX is calculated and stored in memory (209).

Next, set the X address to NMRX+ (NMLX-NMRX
)/2 (horizontal center of the mouth), read the binary image data of the vertical line at that X address from the top to the bottom of window Wm, and when the data is white, clear the run counter, and At this time, the run counter is incremented by 1, and when the count value, that is, the black run length becomes more than a predetermined number, the upper end of the lips is detected, and the y address NMTY at that time is memorized.210°211)
Next, the binary image data is read from the lower end of the window Wm, the black run length is counted, and when the count value exceeds a predetermined number, the lower end of the lips is detected and the y address NMBY at that time is stored in memory (210, 211). Next, the vertical width M
Calculate H=NMBY-NMTY and store it in memory, MW, MH, NMRX+ (NMLX-NMRX)/2
and NMTY+ (NMBY-NMTY)/2 are transferred to the microprocessor 6. Microprocessor 6 has NMRX+ (NMLX-NMRX)/2 and NMT
Convert Y+ (NMBY-NMTY)/2 (these are the mouth center coordinates in window Wm) to frame position coordinates, update the memory as MCX and MCY, and center window Wm at this MCX%MCY. Update settings to what you want. Then, MW is defined as Wl (predetermined discriminant value), MH is pre-stored discriminant value H1,
Compared to H2, H3, and H4, the mouth shape is “A”.
whether it is the pronunciation shape, the "i" pronunciation shape, the "tsu" pronunciation shape, the "tech" pronunciation shape, or the "i" pronunciation shape.
It is determined whether the pronunciation shape is “o” (213 to 217), and “
If it is “A”, the data that indicates “A” is “A”, and if it is “I”, it is “
Register the data indicating ``a'', the data indicating ``tsu'' when ``tsu'', the data indicating ``tech'' when ``tech'', and the data indicating ``o'' when ``sai''. Set to I (218-222).

As shown in Figure 8b, the vowels ``a'' to ``sai'' have widths and heights that can be distinguished from each other, so the above-mentioned Wl
, H1 to H4, etc., and MW, MH. These vowels can be identified by comparing them with reference values such as MW and MH.

Here, for the purpose of explaining the determination of changes in the speech shape pattern of the mouth below, the names assigned to the electrical devices 20 to 29 shown in FIG. 1a are shown in Table 1 below.

(Taking the example of energization control of the first radio on the 5th page of Jonin, in this embodiment,
``Radio'' If you say ``Move'' and select just that vowel, it will become ``I-o'' and ``Tsu-oni.'' These vowel pronunciation shapes are detected and sequentially stored in registers 1.2.3, and the vowel is selected. Each time a shape is detected (213 in Figure 3S)
~222) First, store the detected data (data indicating the detected vowel) in register I, and compare the contents of register ■ with the contents of register 3.223) If the two match, it means that the same vowel shape continues. Therefore, return without doing anything, and if it is different, the utterance shape has changed, so move the contents of register 2 to register L, the contents of register 3 to register 2, and transfer the contents of register I to register 3. The contents are stored in memory (224).

That is, the detection data of the latest vowel pronunciation shape different from the vowel shape detected immediately before is stored in the register 3.

As a result of this memory shift, when the mouth shape first moves to the shape of "radio," and when the last shape of "sai" is detected, the contents of registers 1 to 3 are the vowels enclosed by the lines in the table above. The data will be stored. Since it is unknown at what point the vowel voicing shape detection data has been stored in registers 1 to 3, each time the vowel detection data is stored in register 3, in steps 225 to 227, the It is determined whether vowel pronunciation shape data corresponding to the device or action command is stored in registers 1 to 3. This is to check whether the contents of registers 1 to 3, that is, the time-series pattern of the utterance shape, are the contents shown in the above-mentioned clothes.

If one of these is selected, for example, as shown in the radio field, register 3 stores "Sai" data, register 2 stores "i" data, and register L stores "aco data," then step 225-2 and enter " in the NAME register.
"Radio" is set (266).

If the contents of registers 1 to 3 are the data shown in I"0NJlr4 in the table above, step 225-234-2
43-253-254-255, set "ON" in the operation register (257), and store operation command data (ON, OFF, UP, DOWN) in the operation register (242, 250, 257, 259), clears registers 1 to 3, updates the data in the output register to one that instructs, for example, the radio controller 20 (this is determined by the contents of the NAME register), and sends the contents of the output register to the interface 14. Output.

The operation of the driver and the operation of the above embodiment are summarized as follows.

(1) When the driver closes either switch 5E or 5M while both switches 5E and 5M are open, the illumination light 4 is energized with the standard duty, and the camera 3
Turn on. Then, after the required delay time, one frame of image data of the driver's face is read into the memory 13, an illuminance histogram is created, and it is compared with the set value to determine whether the illumination is appropriate or not, and the illuminance is adjusted accordingly. Adjust brightness. Each time the brightness of the illumination lamp is changed, one frame of image data is newly read into the memory 13 (2 to 12).

(2) After completing the brightness adjustment, in step 12A to test whether the midrange window frame can be set, DFLAG is
If so, branch to (13); if l, branch to (48).

Immediately after the power is turned on (1), DFLAG is initialized to O, so proceed to (13).

(3) In step 13, a simple gradation histogram of a predetermined area Htp of l frames of image data stored in the memory 13 is calculated to determine a threshold value THh for cutting out hair from the background. 2 image data with threshold
The head right end position ARX, left end position ALX, and tip position ATV are detected by converting into values (15).

The area between the left and right end positions of the head and below the tip position ATY is divided into left and right halves, and thresholds THfr (right half) and T are set in each area based on a simple gradation histogram.
Determine Hf L, divide the area horizontally into three equal parts from the right side, area 1, area 2, and area 3 (Fig. 4c), and divide THfr into area l, and THfr and THf into area 2. L
, T Hf L for region 3, and definitely 20). Binarize the image data of the region (regions 1 to 3) with these to determine the position of the upper end of the forehead (boundary between forehead and hair). HT Y
and left and right ends HLX, HRX22), front width H
Calculate W = HLX - HRX.

The threshold value THe for pupil detection is determined based on the differential tone histogram in the area Sd (Figure 4d) of the right half of the front width HW of the upper edge of the forehead HT Y x the lower direction HW (subroutine 25).
, the gradation data of the area Sd is binarized, and the right eye area E (fourth
Figure f) is detected27), the center position of the pupil is detected (3
3), a small area window We centered on the center position
Establish.

The outer ends of both eyebrows are detected from the area (35), a mouth detection area is determined, and the threshold value THm of the mouth detection area is determined based on the difference tone histogram (36), and the center position of the mouth is detected. (37, 38), a window Wm of a small area centered on the mouth center position is determined.

With the processing up to this point, the current position of the pupil to be tracked, the size of the tracking window We, the center position of the mouth to be tracked, and the size of the mouth tracking window We can be determined, and the initial stage is completed. , merges with the bypass branched off at step 12A. Since this initial stage is complex and time-consuming, the present invention achieves faster image processing by bypassing this initial stage once the moving average value and variance II'1 of the target are calculated in step 45. do.

(4) After the initial stage, each time a new image is read (40), the threshold value THe (We area, THm
(Wm area) to binarize each part of the image and window W
Eye detection or mouth detection is performed in the area of e and W m. If detection fails, the process returns to the initial stage, and if detection is successful,
At (45), the process of collecting data on the movement of the target object is started.

(5) “Calculation of mean and variance” routine (45)
In the process of pupil or mouth tracking scanning, pupil or mouth movement data measured every second is accumulated 300 times, the average value and variance value of the data for these 300 times are calculated, and the average value and the variance value are calculated. Mid-range window setting data used in step 50 of the bypass circuit is prepared by sequentially updating and storing the dispersion value each time new data is input. In addition, when the first 300 data are accumulated and the initial average value and variance value are determined, DFLAG is set to 1, and step 12
At A, it is possible to shift to the bypass circuit.

(6) After completing step 45, when the mode switch 5E is in the leap mode, determination based on the blinking of the pupil (46) and determination based on the moving direction of the pupil (47) are performed, and when the mode switch 5M is closed, It makes judgments based on mouth movements, and then captures new images and continues tracking.

(7) In the process of tracking the pupil in step 43, if tracking fails, the process returns to step 12A, but if DFLΔG=1, proceed to the bypass circuit and set the latest mid-range window with a high probability of successful tracking. 50), scan this window to find the lost eye or mouth, capture image data again, and restart tracking.

The above embodiment is an example of an on-vehicle control device that controls an on-vehicle electrical device in response to intentional movements of the eyes and mouth of the driver's face. The same effect occurs even when the target is not the driver. For example, a critically ill patient at a hospital can use his or her eyes and mouth to activate or stop nearby medical equipment (shallow type or rescue equipment), or power up/down. You can

Furthermore, as an abnormality monitoring and protection device for machinery, the object to be photographed by the imaging device is the machine, the target object is the part or part to be monitored for abnormality, and if the object moves abnormally, the Isochiro is stopped and an alarm device is activated. Even if the target is not a person, the present invention can be implemented in the same manner and similar effects can be obtained.

Furthermore, the present invention can be implemented in the same manner in which the monitoring target is a wide place (natural background) and the specific target is an animal in the place or a moving body such as a car. For example, it can be similarly implemented to control the opening and closing of a gate according to the movement of a table in a safari bark, or to control the gate according to the movement of a wild beast. Translating this to a factory, for example, if a belt conveyor line is to be imaged, and the target object is a part or product on the conveyor, when it moves in a predetermined direction, a safety device is activated or equipment for the next process is activated. , similar effects can be obtained by implementing the above device in the same manner as in the case described above.

[Effects of the Invention] As described in detail in the embodiments, the present invention accumulates movement data of the target a predetermined number of times at predetermined time intervals in the process of tracking and scanning the target object by the target tracking means, and collects movement data of the target object for the predetermined number of times. By calculating the average value and variance value of the data, and then changing and storing the average value and variance value sequentially each time new data is input, if the target object tracking fails, the stored average value and variance value are A high-precision mid-range window with a high probability of target capture is set, the lost target is rediscovered using that window, and then the target is tracked using a narrow window. Therefore, (1) By setting a narrow window for tracking time-series changes in the state of the target, the time required to track the target is shortened. (2) If tracking fails, a bypass path is used to
By setting a high-speed, high-precision mid-range window using the latest average and variance values, it is possible to re-detect and re-track the target, greatly improving the target tracking speed of the entire device and enabling real-time scanning. shall be.

Furthermore, the present invention is capable of controlling the energization of the on-vehicle electric device by determining the intentional movement of the driver's eyes or mouth using the control signal determination means when the target object is the eyes or mouth of the on-vehicle driver. shall be.

Further, the present invention provides an electric device for medical equipment or rescue equipment by determining the intentional movement of the eyes or mouth of a seriously ill patient by using a control signal determining means, when the target object is the eyes or mouth of a critically ill patient. It can be used as a support device for critically ill patients to control their symptoms.

Furthermore, the present invention can be used as a driver's drowsy driving prevention device that detects drowsy driving using a control signal determining means based on the blink interval of the driver's eyes without providing any special accessory equipment, and sends a U-warning. .

Furthermore, since the main part of the present invention is composed only of software, there is a large degree of freedom and flexibility in design, and it is easy to design a device that corresponds to the individual characteristics of the target. It can be suitably used in private vehicles.

[Brief explanation of the drawing]

FIG. 1a is a block diagram showing the configuration of an embodiment of the present invention, and FIG. 1b is a television camera 3 shown in FIG. 1a.
FIG. 4 is a perspective view showing the appearance of the arrangement of illuminating lights 4. FIG. Figures 2a and 2b are flowcharts outlining the present invention. FIGS. 3a to 3u are sub-flow charts (subroutines) constituting the main flow shown in FIGS. 2a and 2b. Figure 3a is a flowchart of the image boundary detection means E (head boundary). Figure 3b is a flowchart of the target object detection means G (head detection). Figure 3c is a flowchart of the image boundary detection means E (forehead boundary). Figure 3d is a flowchart of the image boundary detection means E (forehead boundary). Figure 3e is a flowchart of the target object detection means G (forehead detection). Figure 3e is a flowchart of the image boundary detection means E (border around the pupil). Figure 3f is a flowchart of the target object detection means G (pupil detection). Figure 3g. Figures 3H and 31 are flowcharts for changing the threshold value (face) and Figures 3J and 3J are flowcharts for the target object tracking means (pupil tracking), respectively. FIG. 31 is a flowchart of the determination means based on the blinking of the pupils, which constitutes the control signal determination means J. FIG. 3M is a flowchart of the determination means based on the movement direction of the pupils, which constitutes the control signal determination means J. Figures 3p, 3q and 3r are flowcharts of the middle range window setting means N, and Figures 3s and 3 are flowcharts of the target object detection means P using the middle range window.
1 and 3U are flowcharts of the control signal determining means J (determining means based on mouth movements). Figures 4a, 4b, 4c, 4d, 4e,
4f, 4g, 4h, 41, and 4j are plan views showing the whole or part of the image taken by the camera 3. FIG. Figure 5a, Figure 5b, Figure 5C, Figure 5d, Figure 5e,
FIGS. 5f and 5g are a plan view showing the area of the window set for pupil tracking and the detected pupils, and a linear prediction diagram of the pupil position, and FIG. 5h is a diagram showing how to set the mid-range window. FIGS. 6a and 6b are diagrams showing how to detect the pupil using the mid-range window. Fig. 7a is a diagram showing the procedure for testing pupil blinking, Fig. 7b is a diagram showing the procedure for detecting intentional blinking, and Fig. 7C is a diagram defining the relationship between the direction of movement of the pupil and the electric device. FIG. 8a is a plan view showing the region of the window Wm set for mouth tracking and the detected mouth. FIG. 8b is a plan view showing the shape of the mouth when pronouncing vowels. FIG. 9a shows a configuration diagram of a device according to the prior art, and FIG.
The figure shows a block diagram of a device according to the invention. A00 image input means 3, television camera 18, A/D converter B, illumination means 4, illumination light C1 target selection means 5E, 5M mode instruction switch D0 image control storage means 6, general-purpose microprocessor 7, address bus and control bus 8. Image processor 11.12. Image frame Memory E6 Image boundary detection means F, wide area window setting means G, target object detection means using wide area window H1 Narrow area window setting means ■, using wide area window. The target tracking means J, the control signal discrimination means, and the electric device control means 19.20 to 30. Controller L, mean value variance value calculation means M for movement of a target, means W6 for determining whether a middle-mid range window can be set. Wm1 window patent applicant Aisin Seiki Co., Ltd. and 1 other person 1d Engraving of drawings (no change in content) Engraving of drawings (no change in content) Engraving of drawings IF (no change in content) Engraving of multi-3rr-I drawings (No change in content) Voice 4e ku ÷ 3d ■ Katsu 49 Figure 4h Figure Interval C] Five snoring [East 4b Figure East 5a Figure ``(city'' L (o-) and downward resistance of cod showa 6 minutes Cb) Zuoya same Lu old n rotten fu 1 minute] Kishiba taste=shiF-0-・shii de・廓 [next ship・kaimuzaiq
b Figure C Yu) Consider ヰ0zu 2 2 saliva same (E)〕Sho Puhitsu Ji? 100Tl, T2.T3 k;S
(Kots 7i ri ``GoC3 wa tachi id, irr? - 1n sai needle blood procedure amendment (rei) 24th 7th, amendment. Contents (1) From page 95, line 9 to page 96, line 20 of the specification, “Figures 3a to 3u are charts.” Correct as shown.

Claims (4)

[Claims] (1) Image input means (A) for converting optical information of an image into digital image information and inputting it; illumination means (B) for illuminating the image and controlling and adjusting its brightness; and plurality of images in the image. target object selection means (C) for selecting one of the target objects; image control storage means (D) for storing the digital image information and controlling input/output; image boundary detection means (E) that determines a threshold value from the histogram and scans the binarized information using the threshold value to detect image boundaries; wide-area window setting means (F) for setting a scanning area; wide-area window target detection means (G) for scanning within the wide-area window and detecting the position of the target; and detecting with the target object detection means. Narrow window setting means (H) for setting a narrow window for tracking the movement of a target object
and narrow window target tracking means (I) that scans within the narrow window to track the target; and an intentional movement pattern of the tracked target that can control a predetermined electrical device control signal discriminating means (for transmitting a signal to control the electric device when the electric device matches a predetermined pattern);
J); and an electric device control means (K) for controlling a predetermined electric device according to the signal of the control signal discriminating means, wherein the target tracking means using the narrow window detects the target. In the process of tracking and scanning an object, the movement data of the target object measured at predetermined time intervals is accumulated a predetermined number of times, the average value and variance value of the data for the predetermined number of times are calculated, and the average value and variance value are Once the average value and variance calculation means (L) of the movement of the target object are changed and memorized each time new data is input, and the initial values of the average and variance values are determined, it is possible to set the middle range window frame. a mid-range window setting means (N) for setting a mid-range window using the average value and the variance value; , a target object detection means (P) using a mid-range window for scanning and detecting a target object. (2) In the energization control device for an electric device according to claim (1), the target object is the eyes or mouth of the onboard driver,
An automobile driving control device that controls the energization of an on-vehicle electrical device by determining intentional eye or mouth movements using a control signal determining means (J). (3) In the energization control device for an electric device according to claim (1), the target object is the eyes or mouth of a critically ill patient, and the control signal determining means (J) detects intentional movements of the eyes or mouth. A support device for critically ill patients that controls the energization of electrical equipment in medical equipment or rescue equipment by making a determination. (4) In the vehicle driving control device according to claim (2), the driver's drowsy driving prevention device detects drowsy driving using the control signal determination means (J) based on the blink interval of the driver's eyes and issues an alarm. .