JP2021007717A - Occupant observation device, occupant observation method and program - Google Patents

Abstract

To provide an occupant observation device, an occupant observation method and a program which can accurately determine whether or not an occupant is falling asleep.SOLUTION: An occupant observation device comprises: an imaging unit which images a head of an occupant of a vehicle to generate an image; an index value derivation unit which derives an index value indicating a level of opening/closing of eyes of the occupant on the basis of the image generated by the imaging unit; an inclination estimation unit which estimates a change level of the inclination of the head of the occupant using time when the occupant is awake as the reference; and a determination unit which determines whether or not the occupant is falling asleep on the basis of a determination result according to the index value derived by the index value derivation unit and the change level detected by the inclination estimation unit.SELECTED DRAWING: Figure 1

Description

The present invention relates to an occupant observation device, an occupant observation method, and a program.

Conventionally, research has been conducted on a mechanism for determining that a vehicle occupant is asleep by a device. An important factor for determining that an occupant is asleep is the condition of the eyes. Therefore, a device that captures an image of an occupant with a camera, analyzes the image, and observes the state of the eye is being put into practical use (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 6-266981

Here, the drowsiness of the occupant may appear in places other than the eyes. However, with the conventional technique, it has not been possible to determine whether or not the occupant is dozing based on factors other than the condition of the eyes.

The present invention has been made in consideration of such circumstances, and an object of the present invention is to provide an occupant observation device, an occupant observation method, and a program capable of accurately determining whether or not an occupant is asleep. Let's try.

The occupant observation device, the occupant observation method, and the program according to the present invention have the following configurations.
(1) The occupant observation device according to one aspect of the present invention is an imaging unit that images the head of a vehicle occupant and generates an image, and the occupant's eyes based on the image generated by the imaging unit. The index value derivation unit for deriving the index value indicating the degree of opening / closing, the inclination estimation unit for estimating the degree of change of the inclination of the occupant's head with respect to the awakening time, and the index value derivation unit derived by the index value derivation unit. It is provided with a determination unit for determining whether or not the occupant is asleep based on the result of determination based on both the index value and the degree of change detected by the inclination estimation unit.

The aspect (2) is an eye detection unit in which the occupant observation device according to the above aspect (1) detects at least a part of the contour of the occupant's eye based on the image generated by the imaging unit. Further, the index value deriving unit derives the index value based on the positional relationship of a plurality of feature points in the contour detected by the eye detecting unit.

The aspect (3) is the occupant observation device according to the above aspect (1) or (2), and the determination unit indicates that the index value indicates that the degree of eye opening is less than the first threshold value, or the degree of eye closure. When the state indicating that is equal to or higher than the second threshold value continues for the first predetermined time or longer, it is determined that the occupant is dozing.

In the aspect (4), in the occupant observation device according to any one of the above aspects (1) to (3), the determination unit is in a state where the degree of change is tilted by the third threshold value or more for the second predetermined time or more. If it continues, it is determined that the occupant is dozing.

(5) In another aspect of the occupant observation method of the present invention, a computer images the head of a vehicle occupant to generate an image, and based on the generated image, the degree of opening / closing of the occupant's eyes is determined. The indicated index value is derived, the degree of change in the inclination of the occupant's head with respect to the time of awakening is estimated, and the result of judgment based on both the derived index value and the estimated degree of change. Based on the above, it is determined whether or not the occupant is asleep.

(6) The program of another aspect of the present invention uses a computer to image the head of a vehicle occupant to generate an image, and based on the generated image, an index indicating the degree of opening and closing of the occupant's eyes. A value is derived, the degree of change in the inclination of the occupant's head with respect to the time of awakening is estimated, and based on the result of determination based on both the derived index value and the estimated degree of change. Therefore, it is determined whether or not the occupant is asleep.

According to (1) to (6), it is possible to accurately determine whether or not the occupant is asleep.

According to (3) to (4), it is possible to more accurately determine whether or not the occupant is dozing.

It is a figure which shows an example of the structure and use environment of the occupant observation device 1. It is a figure which illustrated the position where the image pickup unit 10 is installed. It is a figure which shows typically the content of the processing by an eye detection unit 22. It is a figure (the 1) for demonstrating the process of the eye opening rate derivation part 24. It is a figure (the 2) for demonstrating the process of the eye opening rate derivation part 24. It is a figure which shows typically the content of the processing by the inclination estimation unit 26. It is a flowchart which shows an example of the flow of processing executed by image processing apparatus 20.

Hereinafter, the occupant observation device, the occupant observation method, and the embodiment of the program of the present invention will be described with reference to the drawings.

<Embodiment>
FIG. 1 is a diagram showing an example of the configuration and usage environment of the occupant observation device 1. The occupant observation device 1 includes, for example, an image pickup unit 10 and an image processing device 20. The image processing device 20 includes, for example, an eye detection unit 22, an eye opening rate derivation unit 24, a tilt estimation unit 26, and a determination unit 28. The occupant observation device 1 determines, for example, whether the state of the occupant of the vehicle is dozing, and outputs the determination result to various in-vehicle devices 100. The occupants may include at least the driver and may include passenger seat occupants. The various in-vehicle devices 100 are a driving support device, an automatic driving control device, an agent device, and other devices, and the occupant observation device 1 estimates and outputs the state of the occupants according to the type and purpose of the various in-vehicle devices 100. ..

FIG. 2 is a diagram illustrating the position where the imaging unit 10 is installed. The image pickup unit 10 is installed in the central portion of the instrument panel of the vehicle, for example, and images at least the head of the occupant of the vehicle to generate an image. The image pickup unit 10 laterally faces both the driver's seat DS (an example of the seating position) provided with the steering wheel SW and the passenger seat AS (another example of the seating position) from a position facing them. It is installed at a position offset to. Therefore, the image generated by the imaging unit 10 is an image generated by imaging the head of the occupant diagonally in the lateral direction. In other words, at least the head of the occupant does not exist on the optical axis of the imaging unit 10.

Returning to FIG. 1, each part of the image processing apparatus 20 will be described. The components of the image processing device 20 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components are hardware (circuit parts;) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit). It may be realized by (including circuits), or it may be realized by the cooperation of software and hardware. The program may be stored in advance in a storage device (a storage device including a non-transient storage medium) such as an HDD (Hard Disk Drive) or a flash memory, or a removable storage device such as a DVD or a CD-ROM. It is stored in a medium (non-transient storage medium) and may be installed by mounting the storage medium in a drive device.

As an example of the function of the eye detection unit 22, a method of detecting an eye after extracting an edge will be described. For example, the eye detection unit 22 first extracts an edge from an image generated by the imaging unit 10 (hereinafter, referred to as an captured image). An edge is a pixel (or a group of pixels) in which the difference in pixel value from the pixels around itself is larger than the reference, that is, a characteristic pixel. The eye detection unit 22 extracts edges by using, for example, an edge extraction filter such as a SOBEL filter. The SOBEL filter is used only as an example, and the eye detection unit 22 may extract edges based on another filter or an algorithm.

The eye detection unit 22 detects at least a part of the occupant's eye in the captured image, for example, based on the distribution of the edges extracted in the captured image. At this time, the eye detection unit 22 sets the eye of the occupant, which is closer to the imaging unit 10, as the detection target. The eye closer to the image pickup unit 10 is the left eye if the occupant is seated in the driver's seat DS, and the right eye if the occupant is seated in the passenger seat AS. The eye detection unit 22 directly inputs a part of the eye (or a feature point described later) by inputting an image to a trained model generated by a machine learning method such as deep learning instead of extracting an edge. It may be detected as a target.

FIG. 3 is a diagram schematically showing the content of processing by the eye detection unit 22. In the figure, IM represents an image in which the edge EG is superimposed on the captured image. In addition, this figure focuses exclusively on the occupants seated in the driver's seat DS. First, as shown in the upper part of FIG. 3, the eye detection unit 22 extracts the contour CT of the face by fitting a model such as an ellipse or an egg shape to the edge EG. Next, as shown in the middle figure of FIG. 3, the eye detection unit 22 sets the nose detection window NM with reference to the contour CT of the face, and the nose detection window NM is a portion where edges are easily extracted. The position of the nasal bridge BN is detected. Next, as shown in the lower figure of FIG. 3, the eye detection unit 22 sets an eye detection window EW of a predetermined size on the right side of the nasal bridge BN where the left eye of the occupant should be present, with reference to the position of the nasal bridge BN. Then, at least a part of the eye is detected in the eye detection window EW. As a result, the eye detection window EW is set at a position overlapping the left eye of the occupant seated in the driver's seat DS, so that the left eye is detected in the eye detection window EW. Although specific examples of the process of "detecting at least a part of the eye" are defined in various ways, in the following description, it is assumed that "at least a part of the contour of the eye is detected". When detecting the contour, the eye detection unit 22 detects the contour by, for example, fitting a curve model to the distribution of the edge EG.

The eye opening rate deriving unit 24 derives the eye opening rate α of the occupant's eye based on the positional relationship of a plurality of feature points in the contour of the eye detected by the eye detecting unit 22. The plurality of feature points include, for example, a first feature point that is an end portion (corresponding to the outer corner of the eye) on the side of the contour of the eye that is closer to the imaging unit 10 in the lateral direction, and a second feature point that is an upper end portion. Includes a third feature point, which is the lower end. FIG. 4 is a diagram (No. 1) for explaining the processing of the eye opening rate deriving unit 24. In the figure, P1 is the first feature point, P2 is the second feature point, and P3 is the third feature point. For example, the eye opening rate deriving unit 24 virtually moves a vertical line from the right end of the eye detection window EW to the left in the eye detection window EW, and sets the intersection point when it first intersects the eye contour ECT. Let it be a feature point P1. Further, the eye opening rate deriving unit 24, for example, virtually moves the horizon from the upper end of the eye detection window EW downward in the eye detection window EW, and sets the intersection point when the eye contour ECT first intersects. 2 Let it be a feature point P2. Further, the eye opening rate deriving unit 24 virtually moves the horizon from the lower end of the eye detection window EW upward in the eye detection window EW, and sets the intersection point when the eye contour ECT first intersects. 3 Let it be a feature point P3.

Then, the eye opening rate deriving unit 24 sets the angle between the first straight line connecting the first feature point P1 and the second feature point P2 and the second straight line connecting the first feature point P1 and the third feature point P3. Based on this, the eye opening rate α of the occupant is derived. FIG. 5 is a diagram (No. 2) for explaining the processing of the eye opening rate deriving unit 24. In the figure, L1 is the first straight line, L2 is the second straight line, and θ1 is the angle formed by them. For example, the eye opening rate deriving unit 24 sets the reference angle θini, which is obtained by calculating the average of the angles derived based on the captured images for the first few minutes after the occupant boarded the vehicle, as 100 [%] of the eye opening rate α. The angle θ1 derived after the definition is divided by the reference angle θini to derive the eye opening rate α (see equation (1)). Not limited to this, when the person authentication of the occupant is performed, the reference angle corresponding to 100 [%] for each occupant may be stored in the memory, and the reference angle for each occupant may be read from the memory and used for the calculation. Good. Further, the specified value may be set to the reference angle θini, or the specified value may be used at first and gradually adjusted to the average angle of the occupants.

α = MIN {θ1 / θini, 100 [%]}… (1)

Further, in the explanation so far, the eye opening rate α is derived based on the angle θ1 on the image plane. However, for example, a three-dimensional model of the eye is prepared, and the facial contour CT and the nasal bridge BN are prepared. The estimation accuracy of the eye opening rate α can be improved by performing the processing described above after two-dimensionally mapping the eye model rotated according to the face orientation angle estimated from the above relationship. Further, the eye opening rate deriving unit 24 may derive the occupant's eye closing rate as an index value indicating the degree of opening and closing of the eyes. In this case, the eye opening rate deriving unit 24 derives a value obtained by subtracting the eye opening rate α from 100 [%] as the eye closing rate β. The eye opening rate α and the eye closing rate β are examples of “index values indicating the degree of opening and closing of the eyes”, the eye opening rate derivation unit 24 is an example of the “index value derivation unit”, and the eye opening rate α is “eye opening rate α”. The degree of eye closure is an example of the degree of eye closure, and the eye closure rate β is an example of the degree of eye closure.

FIG. 6 is a diagram schematically showing the content of processing by the inclination estimation unit 26. In addition, this figure focuses exclusively on the occupants seated in the driver's seat DS. First, the tilt estimation unit 26 utilizes a part of the processing process of the eye detection unit 22, and by fitting a model such as an ellipse or an egg shape to the edge EG as shown in the upper figure of FIG. The contour CT of the face is extracted. Next, the tilt estimation unit 26 extracted from the midpoint of the virtual first line segment CL1 that connects the left eye and the right eye detected by the eye detection unit 22, as shown in the middle figure of FIG. The position of the second line segment CL2 connecting to the midpoint of the arc of the contour CT of the face (that is, the position of the jaw) is detected. Instead of the above, the tilt estimation unit 26 may directly detect the second line segment CL2 by inputting an captured image into a trained model generated by a machine learning method such as deep learning. ..

Then, as shown in the lower figure of FIG. 6, the inclination estimation unit 26 estimates the degree of change of the second line segment CL2 with respect to the predetermined line segment CL2d. The predetermined line segment CL2d is detected when the occupant is awake. The predetermined line segment CL2d is used as a standard for evaluating how much the second line segment CL2 is tilted as compared with the time of awakening. The predetermined line segment CL2d is detected, for example, at the timing when the occupant gets on the vehicle. The information indicating the detected predetermined line segment CL2d is stored (updated) in, for example, a storage unit (not shown) included in the occupant observation device 1. The inclination estimation unit 26 sets the angle θ2 formed by the second line segment CL2 and the predetermined line segment CL2d when the second line segment CL2 and the lower end of the predetermined line segment CL2d are overlapped with each other, for example. It is estimated as the degree of change of the second line segment CL2 with reference to the minute CL2d.

In the explanation so far, the degree of change is derived based on the angle θ2 on the image plane. However, for example, a three-dimensional model related to the second line segment CL2 is prepared and the contour of the face. After mapping two-dimensionally from the model of the second line segment CL2 rotated according to the face orientation angle estimated from the relationship between CT and the nasal bridge BN, the above-described processing is performed to improve the estimation accuracy of the degree of change. Can be made to.

The determination unit 28 determines whether the occupant is dozing based on, for example, the eye opening rate α derived by the eye opening rate derivation unit 24 and the angle θ2 estimated by the inclination estimation unit 26, and various in-vehicle devices 100. Output to. For example, the determination unit 28 determines that the smaller the eye opening rate α is, the stronger the sleepiness of the occupant is, and when the eye opening rate α is less than the first threshold value Th1, the occupant is strongly drowsy or the occupant is dozing. judge. Further, the determination unit 28 determines that the greater the eye closure rate β, the stronger the sleepiness of the occupant, and when the eye closure rate β is equal to or higher than the second threshold value Th2, the occupant is strongly drowsy or the occupant is dozing. judge. For example, the first threshold Th1 is a value indicating an eye opening rate α that can distinguish between a state in which the occupant is not asleep and a state in which the occupant is asleep, and the second threshold Th2 is a state in which the occupant is not asleep. , It is a value indicating the eye closure rate β that can be distinguished from the dozing state.

Further, the determination unit 28 determines that the greater the angle θ2, the stronger the drowsiness of the occupant, and when the angle θ2 is equal to or greater than the third threshold value Th3, the determination unit 28 determines that the occupant is drowsy or the occupant is dozing. .. The third threshold value Th3 is a value indicating an angle θ2 that can distinguish between a state in which the occupant is not dozing and a state in which the occupant is dozing.

The values of the first threshold value Th1, the second threshold value Th2, and the third threshold value Th3 may be predetermined, may be determined for each occupant in the vehicle, and the image generated by the imaging unit 10 may be determined. It may be derived using a learning model learned by deep learning using.

Further, in the determination unit 28, the state where the eye opening rate α is less than the first threshold value Th1 continues for the first predetermined time or more, or the state where the eye closing rate β is the second threshold value Th2 or more continues for the first predetermined time or more. In some cases, it is determined that the occupant is dozing. Further, the determination unit 28 determines that the occupant is dozing when the state in which the angle θ2 is equal to or greater than the third threshold value Th3 continues for the second predetermined time or longer.

[Operation flow]
FIG. 7 is a flowchart showing an example of the flow of processing executed by the image processing apparatus 20. First, the image processing device 20 acquires the image generated by the imaging unit 10 (step S100). Next, the eye detection unit 22 extracts an edge from the acquired image (step S102).

The eye detection unit 22 detects at least a part of the occupant's eye in the captured image based on the distribution of the edges extracted in the captured image (step S104). The eye opening rate deriving unit 24 derives the eye opening rate α of the occupant's eye based on the positional relationship of a plurality of feature points in the contour of the eye detected by the eye detecting unit 22 (step S106). The eye opening rate deriving unit 24 may derive the eye closing rate β of the occupant's eyes based on the positional relationship of the plurality of feature points.

The determination unit 28 determines whether or not the eye opening rate α derived by the eye opening rate deriving unit 24 is less than the first threshold Th1 or whether the eye closing rate β is equal to or more than the second threshold Th2 (step S108). ). When the determination unit 28 determines that the eye opening rate α is equal to or higher than the first threshold value Th1 or the eye closing rate β is less than the second threshold value Th2, the determination unit 28 terminates the process assuming that the occupant does not feel drowsy. When the determination unit 28 determines that the eye opening rate α is less than the first threshold value Th1 or the eye closing rate β is equal to or higher than the second threshold value Th2, it is assumed that the occupant feels drowsy, and the eye opening rate α is further determined. It is determined whether or not the state in which the first threshold value is less than Th1 or the eye-closure rate β is equal to or higher than the second threshold value Th2 continues for the first predetermined time or more (step S110). When the determination unit 28 determines that the state in which the eye opening rate α is less than the first threshold value Th1 or the state in which the eye closing rate β is the second threshold value Th2 or more continues for the first predetermined time or more, the process proceeds to step S120. , It is determined that the occupant is dozing (step S120).

In the tilt estimation unit 26, the occupant feels drowsy due to the determination unit 28, but the eye opening rate α is less than the first threshold value Th1 or the eye closure rate β is the second threshold value Th2 or more for the first predetermined time. If it is determined that the above is not continued, the position of the second line segment CL2 is detected based on the edge EG extracted by the eye detection unit 22 (step S112). The inclination estimation unit 26 estimates the degree of change (that is, the angle θ2) of the second line segment CL2 with respect to the predetermined line segment CL2d (step S114).

The determination unit 28 determines whether or not the angle θ2 estimated by the inclination estimation unit 26 is equal to or greater than the third threshold value Th3 (step S116). When the determination unit 28 determines that the angle θ2 is less than the third threshold value Th3, the determination unit 28 terminates the process assuming that the occupant does not feel drowsy. When the determination unit 28 determines that the angle θ2 is the third threshold value Th3 or more, it is assumed that the occupant feels drowsy, and the state in which the angle θ2 is the third threshold value Th3 or more continues for the second predetermined time or more. It is determined whether or not this has been done (step S118).

Although the determination unit 28 determines that the occupant feels drowsy, the state in which the eye opening rate α is less than the first threshold value Th1 or the eye closing rate β is in the second threshold value Th2 or more continues for the first predetermined time or more. If the occupant does not do so, or if the state in which the angle θ2 is equal to or greater than the third threshold value Th3 does not continue for the second predetermined time or more, the process is terminated assuming that the occupant has not fallen asleep. When the determination unit 28 determines that the state in which the angle θ2 is the third threshold value Th3 or more continues for the second predetermined time or more, the determination unit 28 determines that the occupant is dozing (step S120).

In the above description, the image processing apparatus 20 first performs determination processing related to the eye opening rate α or eye closing rate β (processing in steps S104 to S110), and then determination processing related to the angle θ2 (processing in steps S112 to S118). ) Has been described, but it is not limited to this. For example, the image processing device 20 may perform the determination process related to the eye opening rate α or the eye closing rate β and the determination process related to the angle θ2 in parallel, and after performing the determination process related to the angle θ2, the eye opening The determination process relating to the rate α or the eye closure rate β may be performed.

As described above, according to the occupant observation device 1 of the present embodiment, the index value indicating the degree of opening / closing of the eye derived by the processing of the eye detection unit 22 and the eye opening rate derivation unit 24, and the inclination estimation unit 26 Based on the result of the judgment based on both the estimated inclination of the occupant's head and the degree of change based on the awakening time, it is judged whether the occupant is dozing more accurately than using either one index. can do.

Although the embodiments for carrying out the present invention have been described above using the embodiments, the present invention is not limited to these embodiments, and various modifications and substitutions are made without departing from the gist of the present invention. Can be added.

1 ... Crew observation device, 10 ... Imaging unit, 20 ... Image processing device, 22 ... Eye detection unit, 24 ... Eye opening rate derivation unit, 26 ... Estimating unit, 28 ... Judgment unit, 100 ... Various in-vehicle devices, BN ... Nose bridge, CL1 ... 1st line segment, CL2 ... 2nd line segment, CL2d ... Predetermined line segment, CT ... Face contour, ECT ... Eye contour, EG ... Edge, EW ... Eye detection window, NM ... Nose detection window, Th1 ... 1st threshold, Th2 ... 2nd threshold, Th3 ... 3rd threshold, α ... eye opening rate, β ... eye closing rate, θ1 ... angle, θ2 ... angle, θini ... reference angle

Claims (6)

An imaging unit that captures the head of the occupant of the vehicle and generates an image,
An index value deriving unit that derives an index value indicating the degree of opening and closing of the eyes of the occupant based on the image generated by the imaging unit, and
A tilt estimation unit that estimates the degree of change in the tilt of the occupant's head with respect to the time of awakening,
Judgment to determine whether or not the occupant is dozing based on the result of determination based on both the index value derived by the index value deriving unit and the change degree detected by the inclination estimation unit. Department and
An occupant observation device equipped with. An eye detection unit that detects at least a part of the contour of the occupant's eye based on the image generated by the imaging unit is further provided.
The index value deriving unit derives the index value based on the positional relationship of a plurality of feature points in the contour detected by the eye detecting unit.
The occupant observation device according to claim 1. The determination unit indicates that the index value indicates that the degree of eye opening is less than the first threshold value, or the state indicating that the degree of eye closure is equal to or more than the second threshold value continues for the first predetermined time or longer. Judges that he is dozing,
The occupant observation device according to claim 1 or 2. The determination unit determines that the occupant is dozing when the state in which the degree of change is tilted by the third threshold value or more continues for the second predetermined time or more.
The occupant observation device according to any one of claims 1 to 3. The computer
The head of the occupant of the vehicle is imaged to generate an image,
Based on the generated image, an index value indicating the degree of opening and closing of the eyes of the occupant is derived.
Estimate the degree of change in the inclination of the occupant's head with respect to the time of awakening.
Based on the result of the determination based on both the derived index value and the estimated degree of change, it is determined whether or not the occupant is asleep.
Crew observation method. On the computer
The head of the occupant of the vehicle is imaged to generate an image,
Based on the generated image, an index value indicating the degree of opening and closing of the eyes of the occupant is derived.
Estimate the degree of change in the inclination of the occupant's head with respect to the time of awakening.
Based on the result of the determination based on both the derived index value and the estimated degree of change, it is determined whether or not the occupant is asleep.
program.

Images