DE112021007503T5 - Condition detection device and condition detection method - Google Patents

Abstract

Comprising an image acquisition section (11) for acquiring an image image on which a region in which a face of a target person should be located is imaged, a face detection section (12, 12a) through which based on the image image acquired by the image acquisition section ( 11), it is determined whether the target person makes a face of a hard-to-detect degree of eye opening by squinting his eyes, in which the degree of eye opening is difficult to detect, an eye degree of eye opening determination section (13) which is determined based on the image acquisition section (11 ) acquired image image determines the eye-opening degree of the target person, and a state determining section (14) which determines whether the current state of the target person is an eyes-open state based on the eye-opening degree determined by the eye-opening degree determining section (13). or an eye-closed state when it is determined by the face detection section (12, 12a) that the target person does not make a face of an eye-opening degree that is difficult to detect, and based on the image image and eyes-on acquired by the image acquisition section (11). /To-detection conditions determine whether the current state of the target person is an eyes-on state or an eyes-closed state when it has been determined by the face detection section (12, 12a) that the target person has a face a degree of eye opening that is difficult to determine.

Description

[Technical area]

The present disclosure relates to a condition detection device and a condition detection method for detecting an eye open/close condition of a person's eyes.

[background technology]

Conventionally, there is known a technique in which, based on an image obtained by imaging a person's face, the degree of eye opening is determined by detecting the outline of the person's eyes, and based on the detected degree of eye opening, an eye open/close state is determined, i.e . H. whether that person's condition is an eyes-open state with eyes open or an eyes-closed state with eyes closed (e.g., Patent Document 1).

[Prior Art Documents]

[patent documents]

[Patent Document 1] Patent Publication No. JP 2009-118989 A

[Overview of the invention]

[Problem to be solved by the invention]

It happens that a person squints his eyes. If a person narrows their eyes, e.g. For example, the edges of the eyelids go to the inside of the eyes and the edges of the eyes can no longer be detected, or the eyelid becomes an unusual shape, making it difficult to determine the degree of eye opening of the person based on the image. Hereinafter, a face in which it is difficult to determine the eye opening degree due to the squinting of a person's eyes is referred to as a “difficult eye opening degree face.”

In the prior art, detection of the eye open/close state is not provided in the case where a person's face has become a face of a difficult to detect degree of eye opening, so that a problem arises in making a sufficiently reliable detection of the eye open state /Close state does not occur when a person's face has become a face of a difficult to detect degree of eye opening.

The present disclosure has been made to solve the above problem and has as its object to provide a state inference device by which the eye open/close state of a person can be determined with sufficient reliability even when that person's face is Face of a degree of eye opening that is difficult to determine.

[Means for solving the task]

The condition detection device according to the present disclosure includes an image acquisition section for acquiring an image image on which is imaged a region in which a face of a target person should be located, with respect to which it is to be determined whether it is an eye-open condition or an eye-closed state, a face detection section by which, based on the image image acquired by the image acquisition section, it is determined whether the target person makes a face of a hard-to-detect degree of eye opening by squinting his eyes, in which the detection of the eye-opening degree is difficult, an eye-opening degree determining section that determines the eye-opening degree of the target person based on the image image acquired by the image acquisition section, and a condition determining section that determines whether the eye-opening degree is determined based on the eye-opening degree determined by the eye-opening degree determining section the current state of the target person is an eyes-on state or an eyes-closed state when it is determined by the face detection section that the target person does not make a face of an eye opening degree that is difficult to detect, and based on the image image acquired by the image acquisition section and eye open/close determination conditions determine whether the current state of the target person is an eyes open state or an eyes closed state when it is determined by the face detection section that the target person has a face makes it difficult to determine the degree of eye opening.

[Effects of the invention]

According to the present disclosure, the eye open/close state of a person can be detected with sufficient reliability even if that person's face has become a face of an eye opening degree that is difficult to detect.

[Brief explanation of the characters]

• [ 1 ] 1 is a view showing a structural example of a condition detection device according to a first embodiment.
• [ 2 ] 2A , 2 B and 2C are views for explaining concrete examples of detecting the eye open/closed state of a driver by the state detecting device in the first embodiment.
• [ 3 ] 3A , 3B and 3C are views for explaining determination results of the driver's eye open/closed state output by the state determining device in the first embodiment.
• [ 4 ] 4 is a flowchart for explaining the operation of the condition detection device according to the first embodiment.
• [ 5 ] 5 is a flowchart for a more detailed explanation of the eye open/close state determination processing by the state determination section of step ST4 in 4 .
• [ 6 ] 6 is a view showing a structural example of the condition detection device when determining whether a driver makes a face of a hard-to-detect degree of eye opening using machine learning.
• [ 7 ] 7A and 7B are views for explaining hardware structure examples of the condition detection device according to the first embodiment.

[Forms for Carrying Out the Invention]

Forms for carrying out the present disclosure are explained in detail below using the figures.

First embodiment

A condition detection device according to the first embodiment determines whether a person's condition is an eyes-on state or an eyes-closed state. Hereinafter, a person whose state determination device determines whether it is an eyes-on state or an eyes-closed state is referred to as a “target person.”

In the first embodiment, it is assumed that the condition detection device e.g. B. is installed in a vehicle 100 and it is determined whether the driver's condition is an eyes-on state or an eyes-closed state. That is, in the first embodiment, the driver is assumed to be the target person.

The determination result determined by the state determination device as to whether the driver is in an eyes-on state or eyes-closed state is z. B. used to detect that the driver is falling asleep.

1 is a view showing a structural example of a condition detection device 1 according to the first embodiment.

The condition determination device 1 is connected to a camera 2. The camera 2 is installed in the vehicle 100 and is mounted in such a way that an area can be imaged in which at least the driver's face should be located.

The camera 2 is mounted, for example, in the middle section of the dashboard or in the instrument panel of the vehicle 100. The camera 2 can z. B. can be used together with a so-called DMS (Driver Monitoring System) that is installed to monitor the passenger compartment. Camera 2 is a visible light camera or an infrared camera.

In 1 Only one camera 2 is mounted in the vehicle 100, which is just an example. A large number of cameras 2 can also be mounted in the vehicle 100.

The camera 2 outputs an imaged image (hereinafter referred to as “image image”) to the condition detection device 1. In the first embodiment, the image may be a still image or a moving image.

Based on an image acquired by the camera 2, the condition determination device 1 determines whether the driver's condition is an eyes-on state or an eyes-closed state (hereinafter referred to as “eyes determination “Open/Close state”).

Data regarding the eye-open/close state detection result (hereinafter referred to as “eye-open/close state detection result”) is output by the state detection device 1, for example. B. to various devices (not shown) that are connected to the condition detection device 1. The various devices are, for example: B. a microsleep detector, through which a microsleep of the driver of the vehicle 100 is detected.

The microsleep detector detects a microsleep of the driver based on the eye sleep detection result output from the condition detection device 1. Open/closed status. For example, the microsleep detector detects a driver's microsleep when the driver continues to be in an eyes-closed state for a predetermined time or longer (hereinafter referred to as a "microsleep detection time"). Then the microsleep detector causes a display warning of a microsleep or a voice output warning of a microsleep by an output device (not shown).

As in 1 As shown, the condition detection device 1 according to the first embodiment includes an image acquisition section 11, a face detection section 12, an eye opening degree detection section 13, a condition detection section 14, and a detection result output section 15.

The image acquisition section 11 acquires an image in which the camera 2 has imaged an area in which the face of a target person should be located.
The image acquisition section 11 outputs the acquired image image to the face detection section 12 and the eye opening degree detection section 13.

The face detection section 12 determines whether the driver makes a face of an eye opening degree that is difficult to detect based on the image image acquired by the image acquisition section 11.
In the first embodiment, a “difficult-to-determine-eye-opening-degree face” is a face in which it is difficult to determine the eye-opening degree due to the squinting of a person's eyes. In the first embodiment, a state in which a person squints his eyes is a state in which a person squeezes his eyes with great force. Squeezing the eyes together with strong force is different from closing the eyes without force, such as when blinking, etc., for example: For example, it is a condition in which the section of the eyelash roots sinks under the upper eyelid, the eyelid has an abnormal shape, or the boundary between the upper eyelid and lower eyelid cannot be identified. If a person squeezes their eyes together with great force, the force exerted on the eyes causes wrinkles to form at the outer corner of the eye or between the eyebrows, which do not appear if the eyes are closed without force, and the shape of the eyebrows is deformed. If the driver makes a face with a degree of eye opening that is difficult to determine, it is conceivable that he is, for example, fighting fatigue, is hit by bright light such as strong sunlight, or that he is yawning.

The face detection section 12 follows predetermined rules for distinguishing a face of a difficult-to-detect eye-opening degree (hereinafter referred to as “difficult face discrimination rules”) and determines whether the driver makes a face of a difficult-to-detect eye-opening degree.

• - Falten der äußeren Augenwinkel haben eine vorgegebene Form (im Folgenden als „Form zur Feststellung der Falten der äußeren Augenwinkel“ bezeichnet)
• - Augenbrauen haben eine vorgegebene Form (im Folgenden als „Form zur Feststellung der Augenbrauen“ bezeichnet)
• - Falten zwischen den Augenbrauen haben eine vorgegebene Form (im Folgenden als „Form zur Feststellung der Falten zwischen den Augenbrauen“ bezeichnet)

Difficult face discrimination rules can be, for example: B. Rules of the following content can be drawn up:

• - Wrinkles of the outer corners of the eyes have a predetermined shape (hereinafter referred to as the “shape for determining the wrinkles of the outer corners of the eyes”)
• - Eyebrows have a predetermined shape (hereinafter referred to as “eyebrow determination shape”)
• - Wrinkles between the eyebrows have a predetermined shape (hereinafter referred to as the “shape for determining the wrinkles between the eyebrows”)

The shape of wrinkles in the outer corners of the eyes that can arise when a person squints is used to determine the shape of wrinkles in the outer corners of the eyes. The shape of the eyebrows that can arise when a person squints is set in advance to determine the shape of the eyebrows. The way to determine the wrinkles between the eyebrows is to first set the shape of wrinkles between the eyebrows that can arise when a person squints their eyes. Data of the shape for detecting the wrinkles of the outer corners of the eyes, the shape for detecting the eyebrows or the shape for detecting the wrinkles between the eyebrows are stored in the face detection section 12.

A variety of patterns can be set as the shape for detecting the wrinkles of the outer corners of the eyes, the shape for detecting the eyebrows or the shape for detecting the wrinkles between the eyebrows.

The face detection section 12 detects the shape of the wrinkles of the outer corners of the eyes, the shape of the eyebrows, or the shape of the wrinkles between the eyebrows of the driver in the image image using a known image recognition technology.

For example, the face detection section 12 performs known edge detection in the image image and extracts feature points of the driver's face that represent parts of the driver's face. Parts of the face are the eyes, the outer corners of the eyes, the inner corners of the eyes, the nose, the mouth, the eyebrows, the chin or wrinkles. For example, feature points of the driver's face are expressed by coordinates on the mapping image.

Since the installation position and the image angle of the camera 2 are known in advance, it can be distinguished by the face detection section 12 that a face that is imaged in the area of the image image is the face of the driver. For example, in an image image, a region in which the driver's face may exist (hereinafter referred to as "driver detection region") is predetermined, and the face detection section 12 extracts the feature points of the driver's face from the driver detection region using a well-known image recognition technology.

Then, the face detection section 12 detects the shape of the wrinkles of the outer corners of the eyes, the shape of the eyebrows, or the shape of the wrinkles between the eyebrows of the driver based on the extracted feature points of the driver's face.

The face detection section 12 determines whether the driver makes a face of a hard-to-detect degree of eye opening based on the detected shape of the wrinkles of the outer corners of the eyes, the shape of the eyebrows, or the shape of the wrinkles between the eyebrows of the driver and the difficult face discrimination rules . For example, the face detection section 12 carries out a pattern comparison of the detected shape of the wrinkles of the outer corners of the driver's eyes and the shape for detecting the wrinkles of the outer corners of the eyes, with a match between the shape of the wrinkles of the outer corners of the eyes and the shape for detecting the wrinkles the outer corner of the eye is determined that the driver makes a face with a degree of eye opening that is difficult to determine.

Further, for example, the face detection section 12 carries out a pattern comparison of the detected shape of the driver's eyebrows and the shape for detecting the eyebrows, whereby if there is a match between the shape of the driver's eyebrows and the shape for detecting the eyebrows, it is determined that the driver makes a face with a degree of eye opening that is difficult to determine. Further, for example, by the face detecting section 12, a pattern comparison is made of the detected shape of the wrinkles between the driver's eyebrows and the shape for detecting the shape of the wrinkles between the eyebrows, whereby a match between the shape of the wrinkles between the driver's eyebrows and the Form for determining the shape of the wrinkles between the eyebrows is determined that the driver makes a face with a degree of eye opening that is difficult to determine.

In the present first embodiment, the “match” is not limited to a complete match, but the “match” also includes an approximate match within a given framework.

Further, in the present first embodiment, it can be determined that the driver makes a face of a difficult-to-detect degree of eye opening by the face detection section 12 also by combining a plurality of difficult face discrimination rules. For example, in the example described above, it can be determined by the face detection section 12 that the driver makes a face of a hard-to-detect degree of eye opening when the shape of the wrinkles of the outer corners of the driver's eyes matches the shape for detecting the wrinkles of the outer corners of the eyes, and the shape of the driver's eyebrows matches the shape for determining the shape of the eyebrows.

The face detection section 12 outputs data on whether it is determined that the driver is making a face of an eye-opening degree that is difficult to detect to the eye-opening degree detecting section 13.

The eye opening degree determining section 13 determines the eye opening degree of the driver based on the image image acquired by the image acquisition section 11.

More specifically, the eye opening degree detecting section 13 detects the driver's upper eyelid and lower eyelid in the imaging image using a known image recognition technology.

For example, the eye opening degree detection section 13 performs known edge detection in the driver detection area of the image image and extracts feature points representing the upper eyelid and feature points representing the lower eyelid of the driver.

Next, the eye opening degree determining section 13 calculates, for example, the vertical distance between the upper eyelid and the lower eyelid of the driver in the image image (hereinafter referred to as “eyelid distance”) from the left in the image image, and calculates the maximum eyelid distance.

Then, the eye opening degree determining section 13 determines the eye opening degree based on the relationship between the calculated maximum eyelid distance and a reference value of the eye opening degree. That is, the unit of eye opening degree is percent.

An average value of the degree of eye opening of an adult of average stature when awake is set as the reference value of the degree of eye opening, which, for. B. was calculated in advance in a test procedure. This is just an example, but it is also possible, e.g. B. to determine an average value of the driver's eye opening degrees at predetermined times from the driver's entry into the vehicle 100, or an average value of all eye opening degrees previously determined by the eye opening degree determining section 13 as a reference value of the eye opening degree.

For example, also by the face detection section 12, when extracting feature points of the driver's face, extracting feature points representing the driver's upper eyelid and feature points representing the driver's lower eyelid and outputting to the feature points of the driver's upper eyelid is also possible Driver and data related to the feature points of the driver's lower eyelid to the eye opening degree detection section 13 possible. In this case, the eye opening degree determining section 13 can determine the eye opening degree of the driver based on the data output from the face detecting section 12 regarding the feature points representing the driver's upper eyelid and the feature points representing the driver's lower eyelid.

The eye opening degree detecting section 13 can also detect the eye opening degree using various known algorithms besides the methods described above.

If the driver makes a face with a degree of eye opening that is difficult to determine, e.g. For example, the edges of the eyelids are on the inside of the eye and the edges of the driver's upper and lower eyelids cannot be detected. That is, it happens that the feature points representing the driver's upper eyelid and the feature points representing the driver's lower eyelid are not extractable. As a result, the eye opening degree detecting section 13 cannot detect the eye opening degree of the driver.

If the driver makes a face with a degree of eye opening that is difficult to determine, it can also happen that the eyelid takes on an unusual shape. Here, even if detection of the driver's eye opening degree is possible by the eye opening degree determining section 13, the detected eye opening degree cannot necessarily be considered as an appropriate value.

Therefore, by the eye opening degree determining section 13, detection processing of the driver's eye opening degree can also be omitted when the driver makes a face of an eye opening degree that is difficult to detect, in other words, when data indicating that the driver is a is output from the face determining section 12 Face has a degree of eye opening that is difficult to determine.

By the eye opening degree detecting section 13, detection processing of the driver's eye opening degree should be performed at least when the driver does not make a face of an eye opening degree that is difficult to detect, in other words, when data indicating that the driver does not have been output from the face detecting section 12 Face has a degree of eye opening that is difficult to determine.

The eye opening degree determination section 13 outputs the determination result of the driver's eye opening degree in relation to the image image acquired by the image acquisition section 11 to the condition determination section 14. For example, if the eye opening degree of the driver was not detected by the eye opening degree determining section 13, or the eye opening degree of the driver could not be determined, the eye opening degree determining section 13 correlates data of this content with the image image.

Further, the eye opening degree detecting section 13 outputs the data output from the face detecting section 12 on whether it has been determined that the driver is making a face of a difficult to detect eye opening degree, together with the image to which the eye opening degree has been related -Determination section 14.

In the following first embodiment, the image image that has been related to the degree of eye opening output from the eye opening degree detecting section 13 to the condition determining section 14 is referred to as an “eye opening degree image image”.

The state determination section 14 carries out eye open/close state determination processing in which a determination of the eye open/close state is made as to whether it is the current one The driver's condition is an eyes-on state or an eyes-closed state.

More specifically, the state determining section 14 determines whether the current state of the driver is an eyes-on state or an eyes-closed state based on the eye-opening degree determined by the eye-opening degree determining section 13 when determined by the face determining section 12 it was found that the driver does not make a face with a degree of eye opening that is difficult to determine. The condition determination section 14 determines the eye opening degree of the driver determined by the eye opening degree determining section 13 based on the eye opening degree image image.

The state determination section 14 determines whether the current state of the driver is an eyes-on state or an eyes-closed state based on the image image acquired by the image acquisition section 11 and the eye-open/closed state. when it is determined by the face detection section 12 that the driver makes a face of an eye opening degree that is difficult to detect. In other words, the image image acquired by the image acquisition section 11 is the eye-opening degree image image output from the eye-opening degree determining section 13.

The condition detection section 14 performs eye open/close condition detection for each image image.

In the first embodiment, the eye-open/closed determination conditions are conditions for determining whether the driver is in an eye-open state or an eye-closed state. The eye open/close detection conditions are preset and stored in the condition detection section 14.

Eye open/close detection conditions include: B. Conditions such as the following <condition (1)> to <condition (3)> are set.

<Condition (1)>

The current state of the target person is considered to be an “eyes-closed state” if retroactively from the time when the driver was found to be making a face of a difficult-to-determine degree of eye opening based on the detection of the eyes-open/closed state of the target person Target person with respect to image images acquired before the driver's detection of a face of a difficult to detect degree of eye opening, at greater than or equal to a predetermined number (hereinafter referred to as the "continuation detection threshold") of image images continued an "eye-close" Condition” of the target person was determined.

<Condition (2)>

The current state of the target person is considered to be an “eyes-open state” if retroactively from the time when the driver is found to be making a face of a difficult-to-determine degree of eye opening based on the detection of the eyes-open/closed state of the target person target person with respect to a predetermined number (an appropriate number of greater than or equal to 2) of image images acquired before the driver is found to be making a face of a degree of eye opening that is difficult to detect, of the image images in question, the proportion of image images in which a " The target person's "eyes-closed state" was determined to be below a predetermined threshold value (hereinafter referred to as the "proportion detection threshold").

<Condition (3)>

The current state of the target person is considered to be an “eyes-on state” if, based on the determination of the eyes-open/closed state of the target person with respect to an image image immediately preceding the determination, the driver makes a face of a degree of eye opening that is difficult to detect , was acquired, an “eyes-open state” of the target person was detected.

Hereinafter, an explanation will be given of the details of the eye-open/close state determination by the state determination section 14 in the eye-open/close state determination processing separately after the “determination that the driver does not make a face of an eye-opening degree that is difficult to detect.” “ and the “determination that the driver makes a face with a degree of eye opening that is difficult to determine.”

First, the details of determining the eye open/close state by the state determining section 14 in “determining that the driver does not make a face of an eye opening degree that is difficult to detect” will be explained.

If it has been determined that the driver does not make a face of an eye-opening degree that is difficult to detect, that is, if it is determined by the face detection section 12 that the driver does not make a face of an eye-opening degree that is difficult to detect, the state detection section 14 determines the eye-opening degree of the driver that is determined by the eye opening degree detection section 13 was determined with a predetermined one Threshold (hereinafter referred to as “eyes-on detection threshold”) compared. The eyes-on detection threshold can be adjusted appropriately. In the first embodiment, the eye-on detection threshold is set to “40 (%)” as an example.

If the driver's eye opening degree is greater than or equal to the eye-open detection threshold, an eye-open condition of the driver is detected by the condition detection section 14. On the other hand, if the eye opening degree of the driver is less than the eyes-on detection threshold, an eye-closed state of the driver is detected by the condition detection section 14.

The status determination section 14 z. For example, the driver's state is determined to be an eye-open state when the driver's eye opening degree is greater than or equal to 40, and the driver's state is determined to be an eye-closed state when the driver's eye opening degree is less than 40.

Next, the details of determining the eye open/close state by the state determining section 14 in “detecting that the driver makes a face of an eye opening degree that is difficult to detect” will be explained using a concrete example.

<Concrete example (1)>

The eye open/close detection condition is e.g. B. the above <condition (1)> is set. Further, in <condition (1)>, the continuation detection threshold is set to “2 (piece)”.

This shows 2A an example of an eye open/closed state of the driver determined based on the image image by the condition determination section 14, in other words, based on the eye opening degree image image output from the eye opening degree determination section 13, before the face Detection section 12 determined that the driver makes a face of an eye opening degree that is difficult to detect.

For example, as in 2A shown with the eye opening degree imaging image acquired immediately before it is determined by the face detecting section 12 that the driver makes a face of a difficult to detect eye opening degree, the eye opening degree is related to = 8 (%), and since this eye opening degree is below the Eyes-on detection threshold is, it is determined by the condition detection section 14 that the driver is in an “eyes-closed state”. In addition, the eye opening degree = 10 (%) is related to the previously acquired eye opening degree image image, and since this eye opening degree is below the eye open detection threshold, it is determined by the condition detection section 14 that the driver is is in an “eyes-closed” state.

This will continue retroactively from the point in time at which it was determined that the driver is making a face with a degree of eye opening that is difficult to detect with respect to two image images with a degree of eye opening that were acquired before the determination that the driver is making a face with a degree of eye opening that is difficult to determine The driver was diagnosed with a “closed eyes” condition. Therefore, in this case, it is determined by the condition determination section 14 that the <condition (1)> is satisfied and that the current condition of the driver in which a face of a difficult-to-detect degree of eye opening has been detected is an “eyes-closed condition " is.

Therefore, for example, it is a plurality of image images acquired by the image acquisition section 11 before it is determined by the face detection section 12 that the driver makes a face of an eye opening degree that is difficult to detect, and is made with respect to a plurality of image images acquired with the eye opening degree of the driver determined by the eye opening degree determination section 13 (image images with eye opening degree), a determination of the driver's eye open/close state per image image, the state determination section 14 makes by comparing the determination result of the eye opening degree. Driver's open/closed state per image image with the eyes open/closed determination conditions determines whether the current state of the driver is an eyes open state or an eyes closed state. For example, if it has been continuously determined that the driver is in an eye-closed state based on the determination of the driver's eye-open/closed state per image image for a number of image images greater than or equal to the continuation determination threshold value, then the State determination section 14 determines that the eye-open/closed state is satisfied (<condition (1)>), which states that the current state of the driver is an eye-closed state.

For example, if the driver's microsleep state persists, even if temporarily, a face that is difficult to detect must appear If the degree of eye opening is present, it can be determined that it is a continued eye-closed state. If an eye-closed state is not detected on a face with a degree of eye opening that is difficult to determine, it may happen that a microsleep state, which is detected, for example, by the continuation time of the eyes-closed state, or the detection of the microsleep is delayed. Since the state determining section 14 determines the current state of the driver according to the <Concrete Example (1)> based on the eye open/close state of the driver until the appearance of a face of a difficult to detect eye opening degree, the state of a driver who for example, temporarily squinting during an ongoing eye-closed condition can be appropriately identified as an eye-closed condition.

<Concrete example (2)>

The eye open/close detection condition is e.g. B. the above <condition (2)> is set. Further, in <condition (2)>, the proportion determination threshold is set to “80 (%)”. Furthermore, in <condition (2)>, a number of “5 (piece)” image images are specified as the relevant image images.

This shows 2 B another example of an eye open/closed state of the driver determined based on the image image by the state determining section 14, in other words, based on the eye opening degree image image output from the eye opening degree determining section 13 before through the face -Detection section 12 determined that the driver makes a face with a degree of eye opening that is difficult to detect. For example, as in 2 B shown with the eye opening degree imaging image acquired immediately before it is determined by the face detecting section 12 that the driver makes a face of a difficult to detect eye opening degree, the eye opening degree is related to = 20 (%), and since this eye opening degree is below the Eyes-on detection threshold is, it is determined by the condition detection section 14 that the driver is in an “eyes-closed state”. In addition, it is assumed that with the eye opening degree image images acquired one picture before, two pictures before, three pictures before and four pictures before, the driver's eye opening degree is 40 (%), 60 (%), 80 (%), respectively. and 70 (%) is related. Hereby, it is assumed that based on the eye opening degree image images acquired two images to five images before the determination by the face detection section 12 that the driver makes a face of a difficult to detect eye opening degree is determined by the condition detection section 14 that the driver is in an “eyes-open” state.

Due to the determination of the driver's eyes open/closed state with respect to the five image images in question, which was retroactively made from the time at which the driver was determined to be making a face of a degree of eye opening that was difficult to detect, before the determination that the driver was making a face a degree of eye opening that is difficult to determine, the proportion of the image images in question in which an “eyes-closed state” of the driver was determined (20 (%)) is below the proportion detection threshold (80 (%)) . Therefore, in this case, it is determined by the condition determination section 14 that the <condition (2)> is satisfied and that the current condition of the driver in which a face of a difficult-to-detect degree of eye opening has been detected is an “eyes-on condition " is.

Therefore, for example, the condition determination section 14 performs a plurality of image images, which are a plurality of image images acquired by the image acquisition section 11 before determining that the driver has a difficult face by the face detection section 12 detectable eye opening degree, and which are further related to the eye opening degree of the driver determined by the eye opening degree determining section 13 (image images with eye opening degree), makes a determination of the driver's eye open/closed state per image image, and makes a comparison the determination result of the driver's eyes open/closed state per image image with the eyes open/closed determination conditions determines whether the current state of the driver is an eyes open state or an eyes closed state. For example, based on the determination of the driver's eye-open/closed state per image image from a large number of image images, the proportion of image images in which it was determined that the driver is in an eye-closed state is below the proportion determination - Threshold value, then the satisfaction of the eyes open/closed determination condition (<condition (2)>) is determined by the state determination section 14, which states that the current state of the driver is an eyes open state.

For example, it may happen that a driver, even if he is awake and eyes are sufficiently open, temporarily squints his eyes when his face is hit by bright light from strong sunlight. The driver makes a face with a degree of eye opening that is difficult to determine. However, since the driver is awake, an eyes-open condition should be detected for the driver. Since the state determination section 14 determines the current state of the driver according to the <Concrete Example (2)> based on the eye open/close state of the driver until the appearance of a face of a difficult to detect eye opening degree, the state of a driver who Temporarily squinting the eyes can be appropriately identified as an eye-open condition.

<Concrete example (3)>

The eye open/close detection condition is e.g. B. the above <condition (3)> is set.

This shows 2C another example of one based on an image image, in other words, based on an eye-opening degree image image output from the eye-opening degree detecting section 13 before it is determined by the face detecting section 12 that the driver makes a face of an eye-opening degree that is difficult to detect, the driver's eye open/closed state determined by the state detection section 14. For example, as in 2C shown with the eye opening degree imaging image acquired immediately before it is determined by the face detecting section 12 that the driver makes a face of a difficult to detect eye opening degree, the eye opening degree is related to = 100 (%), and since this eye opening degree is greater than or is equal to the eyes-on detection threshold, it is determined by the condition detection section 14 that the driver is in an “eyes-on state”.

In this case, it is determined by the condition determination section 14 that the <condition (3)> is satisfied and that the current condition of the driver in which a face of a hard-to-detect degree of eye opening has been detected is an “eyes-open condition.” is.

Therefore, the condition determination section 14 can detect the determination result of the driver's eye open/close state determined based on, for example, image images, which are image images acquired by the image acquisition section 11 immediately before determined by the face detection section 12 that the driver makes a face of an eye-opening degree that is difficult to detect, and which are also related to the driver's eye-opening degree detected by the eye-opening degree detecting section 13 (eye-opening degree image images), are compared with the eye open/closed detection conditions and thereby determine whether the driver's current state is an eyes-on state or an eyes-closed state. For example, based on image images, which are image images acquired by the image acquisition section 11 immediately before it is determined by the face detection section 12 that the driver makes a face of an eye opening degree that is difficult to detect, and which is also consistent with the by The eye-opening degree of the driver detected by the eye-opening degree determination section 13 is related, if it is determined that it is an eye-open state of the driver, the state determining section 14 determines that the eyes-open/closed determination condition (<condition (3)>), which states that the current state of the driver is an eyes-open state.

For example, if it was an eye-open state immediately before the driver makes a face with a degree of eye opening that is difficult to detect, there is a high probability that he is in an awake state. As with the <Concrete Example (2)> described above, it is e.g. For example, it is conceivable that the driver, although awake, temporarily squinted because his face was hit by bright light from strong sunlight. Since the driver is awake at this time, an eyes-open condition should be detected for the driver. Since the state determination section 14 determines the current state of the driver according to <Concrete Example (3)> based on the eye open/close state of the driver immediately before the appearance of a face of a difficult to detect eye opening degree, the state of a driver, who temporarily squints his eyes can be appropriately identified as an eye-open condition.

In the above-described eye open/close detection conditions (<Condition (1)> to <Condition (3)>) and the specific examples (<Specific Example (1)> to <Specific Example 3>) of detecting the eye On/off states by the state determination section 14 are just examples. Conditions other than <Condition (1)> to <Condition (3)> can also be set as the eye open/close detection conditions. The eye open/close detection conditions can be set appropriately. Conditions can be set as eye open/close detection conditions: which allows separation of whether the target's condition is an "eyes-open" or “Eye-closed condition”.

For example, the condition that the current state of the target person is an "eyes open state" can be set as the eye open/close detection condition when it is determined that the target person makes a face of a degree of eye opening that is difficult to detect, and further, the subject face of a hard-to-determine eye-opening degree is due to yawning". In other words, the image image acquired by the image acquisition section 11 is the eye-opening degree image image output from the eye-opening degree determining section 13. Here, the eye-opening degree image image is related to the information set, which states that the degree of eye opening could not be determined.

If it was determined by the face detection section 12 that the driver makes a face of a hard-to-detect eye-opening degree, and it was further inferred that the face of a hard-to-detect eye-opening degree is due to yawning of the driver based on image images acquired by the image acquisition section 11, then by the state determination section 14, the fulfillment of the eye-open/closed determination condition that the current state of the driver is an eye-open state is determined. That is, it is determined by the state determination section 14 that the current state of the driver is an “eyes-on state”.

For example, in an image in which it is determined that the driver makes a face with a degree of eye opening that is difficult to detect, the distance between a point on the upper lip and a point on the lower lip is greater than or equal to a predetermined threshold (hereinafter referred to as “yawn detection "Threshold value"), it can be concluded by the condition determination section 14 that the face of a difficult-to-determine degree of eye opening is due to the driver yawning. For example, the condition detection section 14 may acquire data regarding feature points on the driver's upper lip and feature points on the driver's lower lip from the face detection section 12 via the eye opening degree detection section 13. The eye opening degree detection section 14, in addition to the methods described above, can also infer that the driver is yawning from image images using various known algorithms.

A yawn is a gesture of trying to remain as awake as possible. Therefore it can be said that the driver is awake. Since the driver is awake, an eye-open condition must be determined for a face with a difficult to detect degree of eye opening due to yawning. Since the state determining section 14 determines the current state of the driver as an eye-open state, for example, when it is inferred that the face of a hard-to-determine degree of eye opening is due to yawning as described above, the state of the driver can be appropriately determined.

For example, whether the current state of the driver is an eyes-on state or an eyes-closed state can also be determined by the state determination section 14 through a combination of a plurality of eyes-on/closed state determination conditions .

When the condition detection section 14 performs detection of the driver's eyes open/closed condition, it outputs the detection result of the driver's eyes open/closed condition to the detection result output section 15. The determination result of the eye open/closed state includes e.g. B. the time at which it was determined whether the driver is in an eyes-on state or an eyes-closed state, data indicating whether the driver is in an eyes-on state or an eyes-closed state. closed state and the degree of eye opening of the driver. If the driver's eye opening degree could not be determined, data about this is recorded in the driver's eye opening degree. The eye-open/closed state determination result may also not include the time at which it was determined whether the driver is in an eye-open state or an eye-closed state, or may be in addition to the time in question also include an image time added to the image or the time at which the image was acquired by the image acquisition section 11.

These are merely examples, and the determination result of the eye open/closed state may also include seat data indicating the location where the driver is sitting, driver authentication data, or data by which the driver can be identified. At least the determination result of the eye open/closed state should include whether whether the driver is in an eyes-on state or an eyes-closed state.

Through the detection result output section 15, the eye open/closed detection result output from the condition detection section 14 is output. For example, by the detection result output section 15, the detection result of the eye open/close state is output to a microsleep detector.

Further, for example, by the detection result output section 15, the detection result of the eye open/close state can be stored in a storage section (not shown). The memory section in question can be set up in the status determination device 1, but it can also be set up outside the status detection device 1 at a location to which the status detection device 1 can take reference.

The condition determination section 14 can also be made to have the functions of the determination result output section 15. In this case, the condition determination device 1 need not include the determination result output section 15.

3A , 3B and 3C are views for explaining determination results of the driver's eye open/closed state output by the state determining device 1 in the first embodiment.

In 3A , 3B and 3C The history of the eye open/close state determination results output from the condition determination section 1, including the eye open/close state determination results output in the past, is displayed. As an example, the determination result of the eye-open/closed state includes the time at which it was determined whether the driver is in an eye-open state or an eye-closed state, data indicating whether the driver is in an eyes-on state or an eyes-closed state (see the determination results of 3A , 3B and 3C ), and the degree of eye opening of the driver. In 3A , 3B and 3C however, the depiction of the time at which it was determined whether the driver is in an eyes-on state or an eyes-closed state has been omitted.

3A 1 shows a detection result of the eye-open/closed state outputted when the state determining device 1 detects the driver's eye-open/closed state by the state determining section 14 as in the above-described <Concrete Example ( 1)> has taken place. 3B 1 shows a detection result of the eye-open/closed state outputted when the state determining device 1 detects the driver's eye-open/closed state by the state determining section 14 as in the above-described <Concrete Example ( 2)> has taken place. 3C 1 shows a detection result of the eye-open/closed state outputted when the state determining device 1 detects the driver's eye-open/closed state by the state determining section 14 as in the above-described <Concrete Example ( 3)> has taken place.

For example, a microsleep detector detects a microsleep of the driver when the driver is continuously in an eye-closed state for a predetermined time or longer based on the eye-open/closed state detection result.

As described above, in the prior art, detection of the eye open/close state is not assumed in the case where a person's face has become a face of a difficult to detect degree of eye opening. Accordingly, in the prior art, when, for example, a person's face has become a face of an eye opening degree that is difficult to detect, the eye open/close state could not be detected, and the eye open degree could not be detected.

Assume that by the condition detection device 1, like the prior art, detection of the eye open/close condition is not possible when a person's face has become a face of a difficult to detect eye opening degree and the eye opening degree could not be detected. Then, if through the condition determination section 14 z. B. a determination of the driver's eyes open/closed state has been made as in the above-described <Concrete Example (1)>, from the state determining device 1 the determination results of the eyes open/closed state "eyes closed" Status” → “Eyes-closed status” → “Determination not possible” is output. In this case, it is possible that, for example, a microsleep detector, through which microsleep detection of the driver is carried out based on the detection result of the eyes open/closed state, detects that the driver's eyes closed state has been interrupted. This makes it possible that a microsleep of the driver that needs to be detected cannot be detected.

On the other hand, the condition detection device 1 according to the first embodiment determines whether the driver's condition is an eyes-on state or an eyes-closed state even when the face has a difficult-to-determine eye opening degree, and the detection result of the eyes -Open/closed status output. As a result, the state detection device 1 can prevent, for example, a microsleep detector, through which a microsleep detection of the driver is carried out based on the determination result of the eyes open/closed state, from erroneously determining that the eyes closed state of the driver Driver was interrupted, and a microsleep of the driver to be detected cannot be detected.

If, due to the specification of the microsleep detector, data that the eye open/close state cannot be detected was acquired, it is also conceivable that this could be the case, for example. B. can be viewed as a continuation of the eye open/closed state acquired immediately before. Here, for example, after being hit by bright light and temporarily entering an eyes-closed state, the driver makes a face of an eye-opening degree that is difficult to detect, assuming that a detection result of the eyes-open/closed state is “Eyes -Closed state” → “Determination not possible” is output. This is viewed by the microsleep detector as a continuation of an “eyes-closed state” as in “eyes-closed state” → “eyes-closed state.” This creates the possibility that the microsleep detector will incorrectly detect a driver's microsleep. On the other hand, the condition detection device 1 according to the first embodiment determines whether the driver's condition is an eyes-on state or an eyes-closed state even with a face of a difficult to detect eye opening degree, and the detection result of the eyes -Open/closed status output. If, as in the <Concrete Example (2)>, it is conceivable that the eye-closed state immediately before is a temporary eye-closed state, the driver's state when he leaves after the eye-closed in question -Condition that makes a face with a degree of eye opening that is difficult to detect is determined to be an eye-open condition. Then a determination result of the eye open/close state of this content is output (see e.g. 3B) . In this way, the microsleep detector can prevent a non-continuing eye-closed state from being viewed as continuing and the driver's state from being incorrectly detected.

The operation of the condition detection device 1 according to the first embodiment will be explained.
4 is a flowchart for explaining the operation of the condition detection device 1 according to the first embodiment.

The image acquisition section 11 acquires an image image in which an area in which a target person's face should be located has been imaged by the camera 2 (step ST1).

The image acquisition section 11 outputs the acquired image image to the face detection section 12 and the eye opening degree detection section 13.

The face detection section 12 determines whether the driver makes a face of an eye opening degree that is difficult to detect based on the image image acquired by the image acquisition section 11 in step ST1 (step ST2).

The face detection section 12 outputs data on whether it is determined that the driver is making a face of an eye-opening degree that is difficult to detect to the eye-opening degree detecting section 13.

The eye opening degree determining section 13 determines the eye opening degree of the driver based on the image image acquired by the image acquiring section 11 in step ST1 (step ST3).

The eye opening degree determination section 13 outputs the eye opening degree determination result related to the image image acquired by the image acquisition section 11 to the condition determination section 14. Further, the eye opening degree detecting section 13 outputs the data output from the face detecting section 12 on whether it has been determined that the driver is making a face of a difficult to detect eye opening degree, together with the image to which the eye opening degree has been related -Determination section 14.

The state determination section 14 executes eye-open/close state determination processing in which a determination of the eye-open/close state is made whether the current state of the driver is an eye-open state or an eye-open state Eyes-closed state acts (step ST4).

By the determination result output section 15, the eye open/closed determination result output from the condition determination section 14 in step ST4 is output (step ST5).

5 is a flowchart for a more detailed explanation of the eye open/close state determination processing by the state determining section 14 of step ST4 in 4 .

The condition determination section 14 determines whether in step ST2 of 3 it is determined by the face detection section 12 whether the driver makes a face of an eye opening degree that is difficult to detect (step ST41).

If it is determined by the face detection section 12 that the driver makes a face of a hard-to-detect degree of eye opening (at “YES” in step ST41), the condition detection section 14 determines based on that in step ST1 of 4 by the image acquisition section 11 and the eye-open/closed determination conditions, it is determined whether the current state of the driver is an eyes-on state or an eyes-closed state (step ST42).

If it is determined that an eye open/close determination condition of the content that the current state of the driver is an eyes closed state is satisfied (if “YES” in step ST42), it is determined by the state determining section 14 determined that the current state of the driver is an eyes-closed state (step ST43). Then, the state determination section 14 outputs the determination result of the driver's eye open/close state to the determination result output section 15.

If it is determined that an eye open/close determination condition of the content that the current state of the driver is an eye open state is satisfied (if “NO” in step ST42), it is determined by the state determining section 14 determined that the current state of the driver is an eyes-open state (step ST44). Then, the state determination section 14 outputs the determination result of the driver's eye open/close state to the determination result output section 15.

On the other hand, if it has been determined by the face detection section 12 that the driver does not make a face of an eye opening degree that is difficult to detect (if “NO” in step ST41), it is determined by the condition detection section 14 whether the driver does not make a face of an eye opening degree that is difficult to detect (at “NO” in step ST41). 3 The driver's eye opening degree detected by the eye opening degree detecting section 13 is greater than or equal to the eye open detection threshold value (step ST45).

If the driver's eye opening degree is greater than or equal to the eye-open detection threshold (“YES” in step ST45), an eye-open condition of the driver is detected by the condition detection section 14 (step ST46). Then, the state determination section 14 outputs the determination result of the driver's eye open/close state to the determination result output section 15.

If the eye opening degree of the driver is below the eye-open detection threshold (“NO” in step ST45), an eye-closed state of the driver is determined by the condition detection section 14 (step ST47). Then, the state determination section 14 outputs the determination result of the driver's eye open/close state to the determination result output section 15.

In this way, the condition determination device 1 according to the first embodiment determines whether the driver makes a face of an eye opening degree that is difficult to detect based on the image image depicting the area where the driver's face should be located. When it is determined that the driver does not make a face of a hard-to-detect eye opening degree, the condition detecting device 1 performs eye open/close state detection based on the eye opening degree of the driver, and it is determined that the driver is making a hard-to-detect face Eye opening degree, it performs eye open/close detection based on the image image and the eye open/close detection conditions. As a result, the eye open/closed state of a person can be determined by the state detection device 1 with sufficient reliability even if the face of this person has become a face with an eye opening degree that is difficult to detect.

In the above first embodiment, in the condition detection device 1, whether the driver makes a face of a difficult-to-detect eye opening degree is determined following difficult face discrimination rules specified by the face detection section 12, which is merely an example.

For example, in the above first embodiment, a face detection section can also determine whether a driver makes a face of a hard-to-detect eye opening degree using a learned model in machine learning (hereinafter referred to as a “machine learning model”).

6 is a view showing a structural example of the condition detection device 1a when determining whether a driver makes a face of a hard-to-detect eye-opening degree using machine learning.

In 6 become structures that are the same as those of the state detection device 1 based on 1 has been explained, provided with the same reference numerals and a duplicate explanation is omitted.

In the 6 State detection device 1a shown differs from that based on 1 explained state determination device 1 in that a model storage section 16 is included. Furthermore, the specific operation of a face detection section 12a differs in the case of FIG 6 shown condition detection device 1a from the specific operation of the face detection section 12 in the case of 1 explained condition detection device 1.

In the model storage section 16, a machine learning model is stored, into which a mapping image depicting a person's face is input and which outputs data indicating whether the person makes a face of a difficult-to-detect degree of eye opening.

The machine learning model is generated in advance by a learning device 3 and stored in the model storage section 16. A structural example of the learning device 3 will be described below.

In 6 is the model storage section 16 included in the state detection device 1a, but this is only an example. The model storage section 16 can also be set up outside the state determination device 1a at a location to which the state determination device 1a can take reference.

The face detection section 12a determines whether the driver makes a face of an eye opening degree that is difficult to detect based on the image image acquired by the image acquisition section 11 and the machine learning model. More specifically, the face detection section 12a inputs the image image acquired by the image acquisition section 11 into the machine learning model and acquires data indicating whether the driver makes a face of an eye opening degree that is difficult to detect. The face detecting section 12a determines that the driver makes a face of an eye-opening degree that is difficult to detect when it obtains data indicating that the driver makes a face of an eye-opening degree that is difficult to detect. The face detecting section 12a determines that the driver does not make a face of a hard-to-detect eye-opening degree when it obtains data indicating that the driver does not make a face of a hard-to-detect eye-opening degree.

The face detection section 12a outputs data on whether it is determined that the driver is making a face of an eye-opening degree that is difficult to detect to the eye-opening degree detecting section 13.

The flowchart explaining the operation of the condition detection device 1a is the same as that in FIG 4 flowchart shown.

In the condition detection device 1a, in step ST2, the face detection section 12a determines whether the driver makes a face of an eye opening degree that is difficult to detect, based on the image image acquired by the image acquisition section 11 and the machine learning model.

The learning device 3 includes as in 6 1, a learning data acquisition section 31 and a model creation section 32 are shown. The learning device 3 is connected to the state detection device 1a.

The learning data acquisition section 31 acquires learning data. The learning data is data including a plurality of image images, which include image images depicting the face of a person making a face of an eye opening degree that is difficult to detect, and image images depicting the face of a person, which does not make a face with a degree of eye opening that is difficult to determine, information is added as a monitoring label that indicates whether a face with a degree of eye opening that is difficult to determine is being made. Learning data is generated, for example, by an administrator.

The learning data acquisition section 31 outputs the acquired learning data to the model creation section 32.

The model creation section 32 creates, based on learning data acquired from the learning data acquisition section 31, a machine learning model into which an image image on which a person's face is depicted and data is input by means of a so-called supervised learning outputs that display whether the person in question makes a face with a degree of eye opening that is difficult to determine.

The model creation section 32 stores a learned machine learning model in the model storage section 16.

In this way, even in a structure in which it is determined whether the driver makes a face of an eye opening degree that is difficult to detect using a machine learning model, the state detection device 1a can also detect the eye open/close state of a person with sufficient reliability can be detected when that person's face has become a face with a degree of eye opening that is difficult to determine.

Further, the condition detection device 1, 1a in the above first embodiment is an onboard device for installation in the vehicle 100, wherein the image acquisition section 11, the face detection section 12, 12a, the eye opening degree detection section 13, etc Condition detection section 14 and the detection result output section 15 are included in the condition detection device 1, 1a.

However, there is no limitation to this, but it is also possible that a part of the image acquisition section 11, the face detection section 12, 12a, the eye opening degree detection section 13, the condition detection section 14 and the detection result output section 15 into an onboard device of the vehicle, and the remaining parts are present on a server which is connected to the onboard device via a network, so that a condition detection system is formed by the onboard device and the server.

Further, the image acquisition section 11, the face detection section 12, 12a, the eye opening degree detection section 13, the condition detection section 14 and the detection result output section 15 may also all be on one server.

Further, in the above first embodiment, the target person is the driver of the vehicle 100, and this is just an example. In addition to the driver, the target person may also be a passenger of the vehicle 100.

Further, in the above first embodiment, the condition detection device 1, 1a is installed in the vehicle 100, which is just an example. The condition determination device 1, 1a is not limited to an occupant of the vehicle 100 including the driver, but can determine, as a condition of each person, whether that person is in an eye-open state or an eye-closed state.

7A and 7B show examples of the hardware structure of the state detection device 1, 1a according to the first embodiment.

The functions of the image acquisition section 11, the face detection section 12, 12a, the eye opening degree detection section 13, the condition detection section 14 and the detection result output section 15 in the first embodiment are realized by a processing circuit 701. That is, the condition detection device 1, 1a includes the processing circuit 701 to control the detection of the eye open/close condition of a person based on an image image.

The processing circuit 701 may have special hardware as in 7A be shown, but it can also be as in 7B shown is a processor 704 that executes programs stored in memory.

If the processing circuit 701 is a special hardware, the processing circuit 701 corresponds, for example. B. a simple circuit, a complex circuit, a programmed processor, parallel programmed processors, ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or their combination.

When the processing circuit is the processor 704, the functions of the image acquisition section 11, the face detection section 12, 12a, the eye opening degree detection section 13, the condition detection section 14 and the condition detection section 15 are performed by software, firmware or a Combination of software and firmware realized. The software or firmware is represented as a program and stored in a memory 705. The processor 704 reads out and executes programs stored in the memory 705, thereby executing the functions of the image acquisition section 11, the face detection section 12, 12a, the eye opening degree detection section 13, the condition detection section 14 and the condition detection section 15 become. That is, when executed by the processor 704, the state detection device 1, 1a includes the memory 705 for storing programs that are finally executed in the steps ST1 to ST5 described above 4 be executed. It can also be said that the ones in memory are 705 stored programs are the order or a method for executing the image acquisition section 11, the face detection section 12, 12a, the eye opening degree detection section 13, the condition detection section 14 and the condition detection section 15 on a computer. The memory 705 can be a non-volatile or volatile semiconductor memory such as. B. a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory) or an EEPROM (Electrically Erasable Programmable Read Only Memory), a magnetic disk, a floppy disk, an optical disk , a compact disc, a minidisk or a DVD (Digital Versatile Disc).

The functions of the image acquisition section 11, the face detection section 12, 12a, the eye opening degree detection section 13, the condition detection section 14 and the condition detection section 15 can also be realized partly by special hardware and partly by software or firmware. For example, by the processing circuit 701 as special hardware, the functions of the image acquisition section 11 and the detection result output section 15 can be realized, and the functions of the face detection section 12, 12a, the eye opening degree detection section 13 and the condition detection section 14 can be realized by the processor 704 reads and executes programs stored in memory 705.

The model storage section 16 consists, for example, of a memory.

Further, the condition detection device 1 includes an input interface device 702 and an output interface device 703 for performing wired or wireless communication with devices such as the camera 2 or a microsleep detector.

A hardware structure example of the learning device 3 is the same as that shown in FIG 7A and 7B shown hardware structure examples of the state detection device 1, 1a.

The functions of the learning data acquisition section 31 and the model creation section 32 are e.g. B. realized by the processing circuit 701. That is, the learning device 3 includes the processing circuit 701 to control the generation of a machine learning model. The processing circuit 701 can be special hardware or the processor 704, which executes programs stored in the memory 705.

If the processing circuit is special hardware, the processing circuit corresponds to 701 e.g. B. a simple circuit, a complex circuit, a programmed processor, parallel programmed processors, ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or their combination.

When the processing circuit is the processor 704, the functions of the learning data acquisition section 31 and the model creation section 32 are realized by software, firmware, or a combination of software and firmware.

As stated above, according to the first embodiment, it is a structure in which the condition detection device 1, 1a according to the present disclosure has an image acquisition section 11 for acquiring an image image on which a region in which a face of a target person is located is imaged should be located, with respect to which it is to be determined whether it is an eyes-on state or an eyes-closed state, a face detection section 12, 12a, by which based on the image image acquired by the image acquisition section 11 , determining whether the target person makes a face of a difficult-to-detect eye-opening degree by squinting his eyes, at which the eye-opening degree is difficult to detect, an eye-opening degree determining section 13 that determines the eye-opening degree of the target person based on the image image acquired by the image acquisition section 11 , and a state determining section 14 that determines whether the current state of the target person is an eyes-on state or an eyes-closed state based on the eye-opening degree determined by the eye-opening degree determining section 13 when through the face detection section 12 determines that the target person does not have a face of a difficult to detect eye opening degree, and determines whether the current state of the target person is one based on the image image and eye open/close detection conditions acquired by the image acquisition section 11 Eyes-on state or an eyes-closed state when it is determined by the face detection section 12 that the target person has a face of a difficult to detect degree of eye opening. Therefore, by the condition detection device 1, 1a, the eye open/close condition of a person can be detected with sufficient reliability even if the face of this person has become a face of a difficult to detect degree of eye opening.

It is possible to change any components of the embodiments or omit any components of the embodiments.

[Industrial application area]

By the condition detection device of the present disclosure, the eye open/close condition of a person can be detected with sufficient reliability even when the person's face has become a face of a difficult to detect eye opening degree.

[Explanation of reference numerals]

1, 1a

Condition detection device

11

Image acquisition section

12, 12a

Face detection section

13

Eye opening degree detection section

14

Condition Determination Section

15

Detection result output section

16

Model storage section

2

camera

3

Learning device

31

Learning data acquisition section

32

Model creation section

100

vehicle

701

Processing circuit

702

Input interface device

703

Output interface device

704

processor

705

Storage

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• JP 2009118989 A [0003]

Claims (14)

Condition detection device comprising an image acquisition section for acquiring an image image depicting an area in which a face of a target person should be located, with respect to which it is to be determined whether it is an eyes-on state or an eyes-closed state, a face detection section that determines whether the target person makes a face of a hard-to-detect degree of eye opening by squinting his eyes based on the image image acquired by the image acquisition section, in which the degree of eye opening is difficult to detect, an eye opening degree determining section that determines the eye opening degree of the target person based on the image image acquired by the image acquisition section, and a state determining section that determines whether the current state of the target person is an eye-open state or not based on the eye-opening degree determined by the eye-opening degree determining section is an eye-closed condition when it is determined by the face detection section that the target person does not have a face of a difficult-to-detect eye opening degree, and determines whether it is based on the image image and eye-open/closed detection conditions acquired by the image acquisition section in the current state of the target person it is an eyes-open state or an eye-closed condition when it is determined by the face detection section that the target person makes a face of an eye opening degree that is difficult to detect. Condition detection device Claim 1 , wherein the condition detection section provides a detection result of the target person's eye open/closed condition determined based on image images, which are image images acquired by the image acquisition section before being detected by the face detection section, that the target person makes a face of an eye-opening degree that is difficult to detect, and which are further related to the eye-opening degree of the target person detected by the eye-opening degree detecting section, compares with the eye open/closed detection conditions, and thereby determines whether it is in the current state the target person is in an eyes-open state or an eyes-closed state. Condition detection device Claim 2 , wherein the condition detection section detects the detection result of the target person's eyes open/closed state based on image images, which are image images acquired by the image acquisition section, immediately before being detected by the face detection section that the target person makes a face of an eye-opening degree that is difficult to detect, and which are further related to the eye-opening degree of the target person detected by the eye-opening degree detecting section, compares with the eye open/closed detection conditions, and thereby determines whether it is the current one The target person's state is an eyes-on state or an eyes-closed state. Condition detection device Claim 3 , wherein by the condition detection section, when based on image images which are image images acquired by the image acquisition section immediately before it is determined by the face detection section that the target person makes a face of a hard-to-detect degree of eye opening, and which are further related to the eye-opening degree of the target person detected by the eye-opening degree detecting section, it is determined that the target person is in an eye-open state, satisfying the eyes-open/closed determining condition is determined, which states that the current state of the target person is an eyes-open state. Condition detection device Claim 2 , wherein the condition detection section relates to a plurality of image images, which are a plurality of image images acquired by the image acquisition section before it is determined by the face detection section that the target person makes a face of a hard-to-detect degree of eye opening, and which are further related to the eye opening degree of the target person detected by the eye opening degree detecting section, performs eye open/closed state detection of the target person per image image, and by comparing the eye open/closed state detection results of the target person Target person determines per image with the eyes open/closed determination conditions whether the current state of the target person is an eyes open state or an eyes closed state. Condition detection device Claim 5 , wherein by the state determination section, if based on the determination of the target person's eyes open/closed state from a plurality of image images carried out per image image, the proportion of image images in which it was determined that the target person is in an eye-closed state State is below a share detection threshold, meeting the eyes open/closed detection condition is determined, which states that the current state of the target person is an eyes open state. Condition detection device Claim 5 , wherein by the state determination section, if the target person is continuously determined to be in an eye-close state based on the determination of the eye-open/closed state of the target person carried out per image image for a number of image images greater than or equal to a continuation detection threshold value -State, the fulfillment of the eyes-on/closed detection condition is determined, which states that the current state of the target person is an eyes-closed state. Condition detection device Claim 1 , wherein by the condition detection section, when it was determined by the face detection section that the target person makes a face of a difficult to detect eye opening degree, and further, based on image images acquired by the image acquisition section, it was concluded that the face of a difficult to detect Eye opening degree is due to a yawn, the satisfaction of the eyes open/closed detection condition is determined, which states that the current state of the target person is an eyes open state. Condition detection device Claim 1 , wherein it is determined by the face detection section whether the target person makes a face of a hard-to-detect degree of eye opening based on wrinkles of the outer corners of the target person's eyes detected in image images acquired by the image acquisition section. Condition detection device Claim 1 , wherein it is determined by the face detection section whether the target person makes a face of a hard-to-detect degree of eye opening based on the shape of the target person's eyebrows detected in image images acquired by the image acquisition section. Condition detection device Claim 1 , wherein it is determined by the face detection section whether the target person makes a face of a hard-to-detect degree of eye opening based on the shape of the wrinkles between the eyebrows of the target person detected in image images acquired by the image acquisition section. Condition detection device Claim 1 , wherein the face detection section based on the image images acquired by the image acquisition section and a machine learning model into which the image images on which a person's face is depicted acquired by the image acquisition section are input and data indicating whether the subject is output If the person makes a face with a degree of eye opening that is difficult to determine, it is determined whether the person makes a face with a degree of eye opening that is difficult to determine. Condition detection device Claim 1 , where the target person is an occupant of a vehicle. A condition detection method comprising a step in which an image image is acquired by an image acquisition section on which a region in which a face of a target person should be located, in respect of which it is to be determined whether there is an eyes-on condition, is acquired or an eye-closed state, a step in which it is determined by a face detection section based on the image image acquired by the image acquisition section whether the target person makes a face of a hard-to-detect degree of eye opening by squinting his eyes, in which the detection of the degree of eye opening is difficult, a step in which the degree of eye opening of the target person is determined by an eye opening degree determination section based on the image image acquired by the image acquisition section, and a step in which the degree of eye opening of the target person is determined by a state determination section based on the image acquired by the image acquisition section Whether the current state of the target person is an eyes-on state or an eyes-closed state is determined by the eye-opening degree determined by the eye-opening degree detection section when it is determined by the face detection section that the target person does not have a face that is difficult to detect eye opening degree, and based on the image image and eye open/close detection conditions acquired by the image acquisition section, determines whether the current state of the target person is an eyes open state or an eyes closed state when determined by the Face detection section found that the target person makes a face of a difficult to detect degree of eye opening.

Images