WO2020179656A1

WO2020179656A1 - Driver monitoring device

Info

Publication number: WO2020179656A1
Application number: PCT/JP2020/008262
Authority: WO
Inventors: 一輝三浦
Original assignee: オムロン株式会社
Priority date: 2019-03-06
Filing date: 2020-02-28
Publication date: 2020-09-10
Also published as: JP2020144573A

Abstract

In the present invention, a camera (1) generates: a first reference image in which a driver is imaged while not riding in a vehicle; a second reference image in which the driver is imaged while riding with proper posture; and a driving image in which the driver is imaged while driving. A difference calculation unit (31) generates: a reference difference image which is the difference between the first reference image and the second reference image; and a driving difference image which is the difference between the first reference image and the driving image. A difference total calculation unit (32): calculates, as a first difference total, the total number of pixels of a threshold value or above in the binarized reference difference image; and calculates, as a second difference total, the total number of pixels of the threshold value or above in the binarized driving difference image. A posture determination unit (34) determines whether the posture of the driver is proper, on the basis of a comparison result between the first difference total and the second difference total. An alarm output unit (35) outputs an alarm if the posture of the driver is determined to be improper.

Description

Driver monitoring device

The present invention relates to a device for monitoring the condition of a driver of a vehicle, and particularly to a technique for determining whether or not the posture of the driver is appropriate.

Driver driver monitors are known that capture and monitor drivers with cameras to prevent traffic accidents such as drowsy driving and looking aside. The driver monitor is installed in the driver's seat of the vehicle and outputs an alarm to alert the driver when a driver's abnormality is detected from the image captured by the camera, or to perform predetermined control on the vehicle. I will go. Examples of the abnormality of the driver include a drowsiness, looking aside, and the case where the driving posture is not proper. For example, when the driver is in a reclining posture with a large inclination to the rear, the face is facing upward and the visibility in front of the vehicle deteriorates.Therefore, if the vehicle approaches the front vehicle abnormally or another vehicle suddenly interrupts. When a dangerous situation occurs, such as when possible, it is not possible to immediately shift to driving to avoid danger, which is a safety issue. Further, even when the driver is in a forward leaning posture with a large inclination to the front, the same problem occurs because the face is downward. Therefore, there have been proposed devices that monitor the posture of the driver with a camera or a sensor and take measures such as outputting an alarm when the posture is not appropriate (Patent Documents 1 to 3).

In Patent Document 1, when detecting an incorrect posture (posture collapse) of the driver, the posture collapse amount is calculated based on the image captured by the camera. For example, the center of the headrest or steering is used as a reference point, and the distance between the center of the driver's face and the reference point is calculated as the amount of posture collapse. Then, when the calculated posture collapse amount is larger than the threshold value, it is determined that the driver is in the posture collapse state.

In Patent Document 2, driver's posture information (elbow angle, knee angle, upper body angle, etc.) is acquired from a captured image of a camera, and the driver's confidence in driving is calculated from the degree of dispersion of this posture information. To do. If the calculated confidence level is less than a predetermined value, the driver is encouraged to improve the driving posture by displaying the driving posture of the driver and the appropriate driving posture.

In Patent Document 3, a body pressure distribution sensor for detecting the body pressure distribution of the driver is provided in the seat of the driver seat as a means for detecting the body signal of the driver according to the driving posture, and the body pressure detected by the sensor is provided. The driving posture of the driver is determined based on the distribution. Then, the driver's driving posture and the proper driving posture are displayed in a comparable manner, and the motion of the body part necessary to correct the driving posture of the driver to the proper driving posture is notified. There is.

JP, 2016-38793, A JP, 2005-193274, A JP, 2005-20681, A

In Patent Document 1, since it is necessary to measure the distance between the driver's face center and the reference point in order to detect posture collapse, the driver monitor must be equipped with a distance measurement function. In Patent Document 2, since the captured image is analyzed and the posture information is obtained by quantifying the angles of the elbow, knee, upper body, and the like, the calculation process is complicated. In Patent Document 3, a body signal detecting means such as a body pressure distribution sensor is required separately from the driver monitor, and the configuration becomes complicated.

An object of the present invention is to provide a driver monitoring device that can easily detect an improper posture of a driver without a distance measuring function.

A driver monitoring device according to the present invention includes a camera for capturing an image of a driver, an attitude determination unit for determining the attitude of the driver based on an image captured by the camera, and a case where the attitude of the driver is determined to be inappropriate. An alarm output unit that outputs an alarm to the camera, a difference calculation unit that calculates the difference between two images captured by the camera, and a difference image obtained by the calculation of the difference calculation unit is binarized using a threshold value, and the binary value And a total difference calculation unit that calculates the total number of pixels equal to or larger than a threshold value in the converted difference image as the total difference. The camera has a first reference image captured while the driver is not in the vehicle and a second reference image captured when the driver is in the vehicle and the posture of the driver is proper. A driving image taken while the person is driving is generated. The difference calculation unit calculates a difference between the first reference image and the second reference image to generate a reference difference image which is a difference between the two images, and a difference between the first reference image and the driving image. Is calculated to generate a difference image during driving which is a difference between the two images. The total difference calculation unit calculates the total number of pixels equal to or larger than the threshold value in the binarized reference difference image as the first total difference X, and calculates the total number of pixels equal to or larger than the threshold value in the binarized driving difference image as the second total. It is calculated as the total difference Y. The posture determination unit determines whether or not the posture of the driver is appropriate based on the comparison result of the first total difference X and the second total difference Y.

According to such a driver monitoring device, a reference difference image and a driving difference image are generated from three images, a first reference image, a second reference image, and a driving image. Then, by comparing the total difference obtained by binarizing the reference difference image with the total difference obtained by binarizing the driving difference image, it is determined whether or not the posture of the driver is appropriate. Therefore, even with a driver monitoring device that does not have a distance measuring function, it is possible to easily determine whether or not the posture of the driver is appropriate by a simple arithmetic process.

In the present invention, when the difference between the first total difference X and the second total difference Y is |XY−, and the first reference value is P, the posture determination unit determines that |XY−≦P. May determine that the driver's posture is appropriate, and if |XY−>P, determine that the driver's posture is incorrect.

In the present invention, the barycentric position of the pixel region of the threshold value or more in the binarized reference difference image is calculated as the first barycentric position M, and the barycentric position of the pixel region of the threshold value or more in the binarized driving difference image is calculated. A barycenter calculation unit may be provided to calculate as the second barycentric position N. Then, the posture determination unit determines whether the driver's posture is proper based on the comparison result between the first total difference X and the second total difference Y and the comparison result between the first center-of-gravity position M and the second center-of-gravity position N. It may be determined whether or not.

In this case, the difference between the first difference total X and the second difference total Y is |XY|, the first reference value is P, and the difference between the first center of gravity position M and the second center of gravity position N is |MN. , And the second reference value is Q, the posture determination unit determines that the posture of the driver is proper when |XY−≦P and |M−N|≦Q However, if |XY|>P or |MN|>Q, it may be determined that the driver's posture is inappropriate.

In the present invention, when the posture determination unit determines that the posture of the driver is inappropriate, the posture determination unit measures the time period during which the inappropriate posture continues as the duration time, and the alarm output unit determines that the duration time is predetermined. An alarm may be output when the reference time has elapsed.

In the present invention, when the camera generates the second reference image, the posture determination unit determines whether or not the posture of the driver who has boarded the vehicle is appropriate based on the face information of the driver and the vehicle information of the vehicle. You may judge based on this.

In this case, the posture determination unit may determine whether or not the posture of the driver who has boarded the vehicle is appropriate by taking into consideration personal learning data regarding the posture of the driver in addition to the face information and the vehicle information. Good.

In the present invention, the camera may capture the first reference image when it is confirmed that the driver having the electronic key for opening and closing the door of the vehicle approaches the vehicle.

According to the present invention, it is possible to provide a driver monitoring device that can easily detect an inappropriate posture of a driver without a distance measuring function.

FIG. 1 is a block diagram of a driver monitoring device according to the first embodiment of the present invention. FIG. 2 is a diagram showing a state in which a driver is imaged by a camera. FIG. 3 is a diagram illustrating an installation example of a camera. FIG. 4 is a diagram showing an example of an improper posture of the driver. FIG. 5 is a diagram showing another example of the driver's improper posture. FIG. 6 is a diagram showing a reference image A captured in the absence of a driver. FIG. 7 is a diagram showing a reference image B captured while the driver is on board. FIG. 8 is a diagram showing a reference difference image G which is a difference between the reference images A and B. FIG. 9 is a diagram schematically showing binarization of the reference difference image G. FIG. 10 is a diagram showing a driving image C1 captured while the vehicle is driving. FIG. 11 is a diagram showing a driving difference image H1 which is a difference between the reference image A and the driving image C1. FIG. 12 is a diagram schematically showing the binarization of the difference image H1 during operation. FIG. 13 is a diagram showing another driving image C2 taken while driving the vehicle. FIG. 14 is a diagram showing a driving difference image H2 which is a difference between the reference image A and the driving image C2. FIG. 15 is a diagram schematically showing the binarization of the difference image H2 during operation. FIG. 16 is a flowchart showing in detail the posture determination procedure in the first embodiment. FIG. 17 is a diagram illustrating the driver's movement in the left-right direction. FIG. 18 is a diagram schematically showing the principle of the second embodiment of the present invention. FIG. 19 is a block diagram of the driver monitoring device according to the second embodiment. FIG. 20 is a flowchart showing in detail the posture determination procedure in the second embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, the same parts or corresponding parts are designated by the same reference numerals.

FIG. 1 shows the configuration of the driver monitoring device (hereinafter referred to as “driver monitor”) according to the first embodiment. In FIG. 1, a driver monitor 100 is a device mounted on a vehicle to monitor a driver's state, and includes a camera 1, an image processing unit 2, a control unit 3, a storage unit 4, a communication unit 5, and a power supply circuit 6. Equipped with.

The camera 1 is provided, for example, as shown in FIG. 2, in the vehicle interior 51 of the vehicle 50 so as to face the driver DR, and images the driver DR seated on the seat 54. The broken line indicates the imaging range of the camera 1, and the upper body including the face, neck, and shoulders of the driver DR is imaged by the camera 1. Vehicle 50 is, for example, an automobile. Reference numeral 55 in FIG. 2 indicates a headrest provided on the upper portion of the seat 54.

FIG. 3 is a view of the windshield W and the instrument panel 52 as viewed from the inside of the passenger compartment 51 toward the traveling direction of the vehicle 50. The camera 1 is installed at a location on the instrument panel 52 that faces the driver DR, and images the driver DR through the steering wheel 53. Note that the installation location of the camera 1 is not limited to the example of FIG. 3, and may be installed near the meter 56, the display 57, etc. arranged on the instrument panel 52, or on the windshield W, for example. It may be installed in the vicinity of the room mirror 58 located at.

As shown in FIG. 1, the camera 1 includes an imaging unit 11 that captures the driver DR as a subject, and a light emitting unit 12 that irradiates the driver DR with light. The image pickup section 11 has an image pickup element such as a CMOS image sensor, and the light emitting section 12 has a light emitting element such as a light emitting diode that emits infrared light. In addition, the camera 1 is also equipped with an optical system such as a lens (not shown).

The image processing unit 2 processes the image captured by the image capturing unit 11 of the camera 1 to extract the face of the driver DR and feature points such as eyes, nose, and mouth of the face. Then, these are analyzed to detect the direction of the driver's face, the direction of the line of sight, the open / closed state of the eyelids, and the like.

The control unit 3 includes a CPU and the like, and includes a difference calculation unit 31, a total difference calculation output unit 32, a vehicle state determination unit 33, an attitude determination unit 34, and an alarm output unit 35. The functions of each of these parts are actually realized by software processing, but are shown here as hardware blocks for convenience.

The difference calculation unit 31 calculates the difference between the two images (described later) captured by the camera 1 to generate a difference image. The total difference calculation unit 32 binarizes the difference image into “0” and “1” by a threshold value, and calculates the total number of pixels of “1” which is equal to or more than the threshold value (total difference). The vehicle state determination unit 33 determines the state of the vehicle 50 (stopped state, running state, turning state, etc.) based on vehicle information such as vehicle speed and yaw rate input to the driver monitor 100 from the outside. The posture determination unit 34 determines whether or not the posture of the driver DR is appropriate, based on the total difference calculated by the total difference calculation unit 32. The alarm output unit 35 outputs an alarm when the attitude determination unit 34 determines that the posture of the driver DR is inappropriate. Details of the processing in each of these blocks 31 to 35 will be described later in more detail.

In addition to the blocks 31 to 35 described above, the control unit 3 includes a drowsiness determination unit that determines whether or not the driver DR is drowsy driving, and whether or not the driver DR is looking aside. Although an inattentive judgment unit for judging is provided, these are not shown because they are not directly related to the present invention.

The storage unit 4 is composed of a semiconductor memory or the like, and stores various parameters required for processing by the control unit 3. The parameters related to the present invention include a binarization threshold when the difference image is binarized by the total difference calculation unit 32, the reference value P and the reference time To used for the posture determination unit 34 to determine the posture of the driver DR. , There are personal learning data obtained from learning results such as the posture habit of the driver DR. In addition, the storage unit 4 is also provided with an area in which a program used for software processing in the control unit 3 is stored and an area in which the calculation result, the determination result, and the like in the control unit 3 are temporarily stored ( (Not shown).

The communication unit 5 sends and receives signals and data to and from the ECU (Electronic Control Unit) 200 mounted on the vehicle 50, and also sends and receives signals and data to and from the control unit 3. The communication unit 5 also receives signals from various sensors provided in the vehicle 50 as the vehicle information described above. The communication unit 5 and the ECU 200 are connected via a CAN (Controller Area Network). The ECU 200 is composed of a plurality of ECUs provided for each control target, but in FIG. 1, they are collectively shown as one block. The electronic key 300 is a mobile terminal for remotely controlling the opening and closing of the door of the vehicle 50, and performs wireless communication with the ECU 200.

The power supply circuit 6 supplies power to each part of the driver monitor 100 under the control of the control unit 3. Power is supplied to the power supply circuit 6 from a battery (not shown) mounted on the vehicle 50.

FIG. 4 shows an example of an improper posture of the driver DR boarding the vehicle 50. Here, the driver DR reclines the seat 54 and assumes a posture inclining rearward largely (reclining posture). In this case, the face of the driver DR faces upward and the visibility in front of the vehicle 50 deteriorates. Therefore, in the case of an abnormal approach to the vehicle in front or a sudden interruption of another vehicle, the danger is immediately avoided. There is a risk of accidents due to carelessness ahead as in the case of inattentive driving, and there is a safety problem.

FIG. 5 shows another example of the incorrect posture of the driver DR. Here, the driver DR is in a posture of being greatly tilted forward (a leaning forward posture). In this case, the face of the driver DR faces downward, and the visibility in front of the vehicle 50 also deteriorates. Therefore, it is possible to immediately perform the driving for avoiding the danger against the abnormal approach or sudden interruption. It is not possible to shift, and there is a risk of an accident due to inattention to the front as in the case of looking aside, which is a safety issue.

Therefore, if the driver DR has such an inappropriate posture, it is necessary to ensure safety by detecting this and warning the driver DR. As a method of detecting this improper posture, a method of measuring the distance from the camera 1 to the face of the driver DR based on the captured image of the camera 1 and determining whether the posture is proper or not can be considered. .. According to this, when the distance from the camera 1 to the face of the driver DR is too long, it is determined that the driver DR is in the reclining posture as shown in FIG. 4, and the distance from the camera 1 to the face of the driver DR is determined. If the distance is too short, it is determined that the driver DR is in the forward leaning posture as shown in FIG. However, this method cannot be adopted unless the driver monitor 100 has a distance measuring function.

The present invention makes it possible to determine whether or not the posture of the driver DR is appropriate, even if the driver monitor 100 does not have a distance measuring function. Hereinafter, the principle of the present invention will be described.

In the present invention, first, the camera 1 captures the reference image A (first reference image) shown in FIG. 6 and the reference image B (second reference image) shown in FIG. 7. The reference image A of FIG. 6 is an image captured in a state where the driver DR is not in the vehicle 50, and the reference image B of FIG. 7 is the image of the driver DR in the vehicle 50 and the driver DR of the driver DR. It is an image picked up in a proper posture. The reference image B is captured after the driver DR starts driving the vehicle 50.

Next, the difference calculation unit 31 calculates the difference between the reference image A and the reference image B to generate a reference difference image G as shown in FIG. 8 which is the difference between the two images A and B. In the reference difference image G, the background portion other than the driver DR is darkened by the difference calculation, and the driver DR portion is brightened. The actual reference difference image G is not such a simple light and dark image, but in FIG. 8, only the driver DR portion is drawn brightly for simplification (also in FIGS. 11 and 14). ). The posture of the driver DR in the reference difference image G is the same as the posture of the driver DR in the reference image B of FIG. 7, and is an appropriate posture.

The total difference calculation unit 32 binarizes the reference difference image G as shown in FIG. 9 using the binarization threshold value stored in the storage unit 4 (FIG. 1). FIG. 9 schematically shows pixels forming the reference difference image G. White pixels are "0" pixels whose luminance is less than the threshold value, and black pixels are "1" pixels whose luminance value is the threshold value or more. (The same applies to FIGS. 12, 15, and 18). It should be noted that the white color and the black color of the pixel are colored separately for convenience of distinction between “0” and “1” and have no relation to the actual luminance. The total difference calculation unit 32 calculates the total number of pixels of “1” in the binarized reference difference image G of FIG. 9 as the total difference X (first total difference).

After that, while the vehicle 50 continues to run, the camera 1 constantly captures the driver DR and generates a driving image captured while the driver DR is driving. If the posture of the driver DR is appropriate, the captured driving image is the same as the reference image B shown in FIG. 7. On the other hand, if the posture of the driver DR is inappropriate, the captured driving image will be an image as shown in FIGS. 10 and 13.

FIG. 10 shows a driving image C1 when the driver DR is in the reclining posture shown in FIG. In this driving image C1, since the driver DR is retracted rearward when viewed from the camera 1, the driver DR is captured smaller than the reference image B in FIG. 7.

The difference calculation unit 31 calculates the difference between the reference image A in FIG. 6 and the driving image C1 in FIG. 10 to calculate the difference between the two images A and C1, and the driving difference image H1 as shown in FIG. To generate. The total difference calculation unit 32 binarizes the driving difference image H1 using a binarization threshold as shown in FIG. Then, the total difference calculation unit 32 calculates the total number of pixels of “1” in the binarized driving difference image H1 as the total difference Y (second total difference).

The posture determination unit 34 calculates the total difference X calculated for the reference difference image G in FIG. 9, the total difference Y calculated for the driving difference image H1 in FIG. 12, and the reference value P stored in the storage unit 4. Based on (first reference value), it is determined whether or not the posture of the driver DR is appropriate. Specifically, the posture determination unit 34 calculates the difference between the total difference X and the total difference Y as |XY− (absolute value), and compares this |XY− with the reference value P. Then, if |XY|≦P, it is determined that the posture of the driver DR is proper, and if |XY−>P, it is determined that the posture of the driver DR is inappropriate. ..

When the posture of the driver DR is appropriate, the face of the driver DR is neither too far from the camera 1 nor too close to the camera 1, so the difference |XY| between the total difference X and the total difference Y is | , Which is less than or equal to the reference value P. On the other hand, when the posture of the driver DR is the reclining posture (FIG. 4), the face of the driver DR is too far away from the camera 1, so that the total difference Y also becomes smaller as the image of the driver DR becomes smaller. Therefore, the difference | XY | between the total difference X and the total difference Y is a value exceeding the reference value P.

FIG. 13 shows a driving image C2 when the driver DR is in the forward leaning posture shown in FIG. In the driving image C2, since the driver DR is approaching the front when viewed from the camera 1, the driver DR is imaged larger than the reference image B in FIG.

The difference calculation unit 31 calculates the difference between the reference image A in FIG. 6 and the driving image C2 in FIG. 13 to calculate the difference between the two images A and C2, and the driving difference image H2 as shown in FIG. To generate. The total difference calculation unit 32 binarizes this driving difference image H2 using a binarization threshold as shown in FIG. Then, the total difference calculation unit 32 calculates the total number of pixels of “1” in the binarized driving difference image H2 as the total difference Y (second total difference).

The posture determination unit 34 calculates the total difference X calculated for the reference difference image G of FIG. 9, the total difference Y calculated for the driving difference image H2 of FIG. 15, and the reference value P stored in the storage unit 4. Based on (first reference value), it is determined whether or not the posture of the driver DR is appropriate. Specifically, similar to the above, the difference between the total difference X and the total difference Y is calculated as |XY−(absolute value), and if |XY−≦P, the posture of the driver DR is It is determined that the posture is appropriate, and if | XY |> P, it is determined that the posture of the driver DR is inappropriate.

When the posture of the driver DR is the forward leaning posture (FIG. 5), the face of the driver DR is too close to the camera 1, and the total difference Y increases as the image of the driver DR increases. Therefore, in this case also, the difference |XY− between the total difference X and the total difference Y becomes a value exceeding the reference value P.

When the posture determination unit 34 determines that the posture of the driver DR is inappropriate, the alarm output unit 35 outputs an alarm. This alarm is output from a speaker (not shown) provided in the vehicle 50 as voice guidance such as "the driving posture is not proper. Please return to the correct posture."

As described above, in the first embodiment of the present invention, the reference image A when the driver DR is not present, the reference image B when the driver DR is on board and the posture is proper, and the driver during driving Three images of the driving image C (C1, C2) obtained by capturing the DR are captured, and the reference difference image G that is the difference between the images A and B and the driving difference image H (H1 that is the difference between the images A and C). , H2) and. Whether the difference |XY− between the total difference X obtained by binarizing the reference difference image G and the total difference Y obtained by binarizing the driving difference image H is less than or equal to the reference value P. Whether or not the posture of the driver DR is appropriate is determined based on whether or not the posture is correct. Therefore, even if the driver monitor 100 does not have a distance measuring function, it is possible to easily determine that the posture of the driver DR is a reclining posture or a leaning posture by a simple calculation process.

FIG. 16 is a flowchart showing the posture determination procedure in the driver monitor 100 of the first embodiment in more detail.

In step S1, the driver DR holding the electronic key 300 (FIG. 1) waits for approaching the vehicle 50. The ECU 200 communicates with the electronic key 300, and detects that the electronic key 300 approaches the vehicle 50 based on receiving the response signal transmitted from the electronic key 300. As a result, the door opening / closing portion (not shown) unlocks the door of the vehicle 50.

Further, based on the detection of the approach of the electronic key 300, in step S2, the control unit 3 drives the power supply circuit 6 to turn on the power supply of the driver monitor 100. Following this, in step S3, the camera 1 captures the reference image A (FIG. 6) in the absence of the driver DR.

Next, in step S4, it is determined whether or not the driver DR has boarded the vehicle 50. This determination is made on the output of, for example, a door sensor (not shown) that detects that the door of the driver's seat has been opened, or a seating sensor (not shown) that detects that the driver DR is seated on the seat 54 of the driver's seat. It is done based on.

When the driver DR gets on the vehicle 50 and the driving of the vehicle 50 is started, the process proceeds to step S5 to determine whether or not the posture of the driver DR is appropriate. This determination is based on, for example, face information of the driver DR (face direction, line-of-sight direction, etc.) obtained by image processing in the image processing unit 2 and vehicle information of the vehicle 50 input via the communication unit 5. It is done based on. As described above, the vehicle speed, the yaw rate, and the like are input as the vehicle information, and the vehicle state determination unit 33 determines the state of the vehicle 50 (running state, stopped state, turning state, etc.) based on these.

The posture determination unit 34 determines that the posture of the driver DR is not appropriate when the face of the driver DR faces sideways, and also when the vehicle 50 is turning, for example. It is judged that the posture is not appropriate because there is a risk of shaking. On the other hand, when the face of the driver DR is facing the front and the vehicle 50 is traveling straight, the posture determination unit 34 determines that the posture of the driver DR is appropriate.

In addition to the face information and the vehicle information described above, the posture determination unit 34 also takes into consideration personal learning data regarding the posture of the driver DR stored in the storage unit 4, and the posture of the driver DR is appropriate. Or not.

If it is determined in step S5 that the posture of the driver DR is not proper, the driver waits until the posture becomes proper. Then, when it is determined that the posture of the driver DR is appropriate, the process proceeds to step S6, and the camera 1 captures the reference image B (FIG. 7) including the driver DR.

Next, in step S7, the difference calculation unit 31 calculates the difference between the reference image A and the reference image B to generate the reference difference image G (FIG. 8). Further, the difference calculation unit 31 binarizes the reference difference image G in step S8 (FIG. 9), totals the “1” pixels of the binarized reference difference image G in step S9, and calculates the total difference. Calculate X.

After that, the process proceeds to step S10, and the camera 1 constantly photographs the driver DR who is driving, and photographs the driving image C (C1 in FIG. 10, C2 in FIG. 13).

Next, in step S11, the difference calculation unit 31 calculates the difference between the reference image A and the driving image C to generate a driving difference image H (H1 in FIG. 11, H2 in FIG. 14). Further, the difference calculation unit 31 binarizes the driving difference image H in step S12 (FIGS. 12 and 15), and totals the “1” pixels of the binarized reference difference image H in step S13. Then, the total difference Y is calculated.

Next, in step S14, the posture determination unit 34 compares the difference |X−Y| between the total difference X calculated in step S9 and the total difference Y calculated in step S13 with the reference value P. Then, when |XY−≦P (determination in step S14: NO), it is determined that the posture of the driver DR is proper, and the process returns to the beginning without executing steps S15 to S18. After that, the operations after step S1 are performed. When |XY−>P (determination in step S14: YES), it is determined that the posture of the driver DR is inappropriate, and the process proceeds to step S15.

In step S15, the posture determination unit 34 measures the time during which the incorrect posture of the driver DR continues as the duration Tc. This is because even if the posture of the driver DR temporarily becomes an improper posture, if the posture is returned to the proper posture in a very short time, there is substantially no safety problem, and in such a case, the alarm is triggered irregularly. This is to avoid being done. Vehicle information is also taken into consideration in the measurement of the duration Tc. For example, when the gear is in the R (Reverse) range to move the vehicle 50 backward, the operation of checking the rear frequently occurs, so that the posture of the driver DR collapses, but there is no safety problem. Therefore, in such a case, the time during the backward confirmation of the driver DR is excluded from the duration Tc of the improper posture.

The posture determination unit 34 compares the measured duration time Tc with the reference time To stored in the storage unit 4 in the subsequent step S16. As a result, when the duration time Tc≦reference time To (determination in step S16: NO), it is determined that the improper posture is temporary, and the process returns to the beginning without executing steps S15 to S18. After that, the operations after step S1 are performed. On the other hand, when the duration Tc> the reference time To (determination in step S16: YES), it is determined that the improper posture continues for a long time, and the process proceeds to step S17.

In step S17, the alarm output unit 35 outputs an alarm for notifying the driver DR of an improper posture to prompt correction of the posture as a voice message as described above.

In the next step S18, the control unit 3 determines whether or not the engine of the vehicle 50 has been turned off (stopped). This determination is made based on whether or not the ignition switch (not shown) is turned off. While the engine is not turned off (determination in step S18: NO), the process returns to step S10, the photographing of the driver DR by the camera 1 is continued, and the subsequent steps S11 to S18 are repeatedly executed. Then, when the engine is turned off (determination in step S18: YES), the process returns to the beginning and the operations in step S1 and subsequent steps are performed.

Next, a second embodiment of the present invention will be described. In the above-described first embodiment, the reclining posture or the forward leaning posture of the driver DR, that is, the inappropriate posture in the front-back direction when viewed from the camera 1 is detected. However, the inappropriate posture in the left-right direction, the vertical direction, or the diagonal direction viewed from the camera 1 is detected. It is difficult to detect the posture. For example, as shown in FIG. 17, when the driver DR in an appropriate posture moves in the left-right direction (left in the figure) as viewed from the camera 1 and comes to the position indicated by the broken line, the driver DR′ is seated normally. Since it is out of position, it has an incorrect posture in the left-right direction.

However, the size on the image of the driver DR′ having an improper posture is almost the same as the size on the image of the driver DR having the proper posture. There is no big difference. For this reason, when the suitability of the posture is determined based on only the difference between the total difference X and the total difference Y as in the first embodiment, both the driver DR and the driver DR′ are determined to have proper postures. There is a risk that The second embodiment solves this problem.

FIG. 18 is a diagram schematically showing the principle of the second embodiment. 18A shows a binarized image of the reference difference image corresponding to the driver DR of FIG. 17, and FIG. 18B shows a binarized image of the driving difference image corresponding to the driver DR′ of FIG. The binarized image is shown. Since the total difference (the total number of black pixels) in these binarized images is almost the same, the suitability of the posture cannot be determined only by the total difference as described above.

Therefore, in the second embodiment, the position of the center of gravity of the "1" pixel region (black-painted region in FIG. 18) in the reference difference image and the driving difference image is calculated. In FIG. 18A, the barycentric position M (first barycentric position) of the reference difference image G is indicated by X, and in FIG. 18B, the barycentric position N of the driving difference image H is ×. It is indicated by a mark. These center of gravity positions M and N can be calculated by assigning the X coordinate and the Y coordinate to each pixel arranged in a matrix and using these coordinate values.

Here, the difference between the X coordinate of the center of gravity position M and the X coordinate of the center of gravity position N is |Mx−Nx|, and the difference between the Y coordinate of the center of gravity position M and the Y coordinate of the center of gravity position N is |My−Ny| These are collectively referred to as |MN|. When the center of gravity position N moves with respect to the center of gravity position M, the moving distance of the center of gravity position N can be known by calculating the difference | MN | of the center of gravity positions.

FIG. 18 shows an example in which the center of gravity position N is horizontally moved to the left with respect to the center of gravity position M by a distance δ for simplification. The X coordinate of the center of gravity position N has changed from the X coordinate of the center of gravity position M, but the Y coordinate of the center of gravity position N is the same as the Y coordinate of the center of gravity position M. That is, | Mx—Nx | = δ, | My—Ny | = 0. By comparing the moving distance δ of the center of gravity position N with the reference value Q (second reference value) stored in the storage unit 4, is the driver DR moving in the horizontal direction and taking an inappropriate posture? It can be determined whether or not. Specifically, if the movement distance δ is equal to or less than the reference value Q (δ≦Q), the posture of the driver DR is appropriate, and if the movement distance δ exceeds the reference value Q (δ>Q), The posture of the driver DR is determined to be inappropriate.

When the center-of-gravity position M of the reference difference image G and the center-of-gravity position N of the driving difference image H are compared, if the posture of the driver DR is appropriate, the amount of movement of the center-of-gravity position N is small. N | (here | Mx-Nx |, that is, δ) is a value equal to or less than the reference value Q. On the other hand, when the driver DR moves in the left-right direction and takes an improper posture, the amount of movement of the center of gravity N is large, so |MN| (same as above) exceeds the reference value Q.

Even if the driver DR moves in the vertical direction or in the diagonal direction, by comparing the moving amount |MN| of the center of gravity position with the reference value Q based on the same principle as above, the driver DR The suitability of the DR posture can be determined. In addition, individual reference values Q1, Q2, and Q3 may be set for each of the left-right, up-down, and diagonal directions.

As described above, in the second embodiment, in addition to the total differences X and Y in the first embodiment, the position of the center of gravity M and N of the driver on the image is also taken into consideration to determine the appropriateness of the posture. It is possible to detect not only the unsuitable posture of the driver DR in the front-rear direction but also the unsuitable posture of the driver DR in the left, right, up, and down directions.

FIG. 19 shows the configuration of the driver monitor 100 according to the second embodiment. In FIG. 19, the same parts as those in FIG. 1 are designated by the same reference numerals. In FIG. 19, the difference from FIG. 1 is that the control unit 3 is provided with the center of gravity calculation unit 36 and the storage unit 4 stores the reference value Q. The configurations other than these are the same as those in FIG. 1, and the description thereof will be omitted.

FIG. 20 is a flowchart showing the posture determination procedure in the driver monitor 100 of the second embodiment in more detail. In FIG. 20, steps that perform the same processing as in FIG. 16 are assigned the same reference numerals.

Steps S1 to S9, S10 to S13, and S15 to S18 in FIG. 20 are the same processes as in FIG. 16, so description thereof will be omitted.

In step S9a, the center-of-gravity calculating unit 36 calculates the center-of-gravity position M (FIG. 18A) of the pixel region of "1" in the binarized reference difference image G. In step S13a, the center of gravity calculation unit 36 calculates the center of gravity position N (FIG. 18B) of the pixel region “1” in the binarized operating difference image H.

In step S14a, the posture determination unit 34 compares the difference |XY| in the total difference with the reference value P and compares the difference |MN| in the center of gravity position with the reference value Q. Then, if |XY|≦P and |MN|≦Q (determination in step S14a: NO), the posture determination unit 34 determines that the posture of the driver DR is appropriate. To do. In this case, the process returns to the beginning without executing steps S15 to S18. On the other hand, if |XY|>P or |MN|>Q (determination in step S14a: YES), the posture determination unit 34 determines that the posture of the driver DR is incorrect. judge. In this case, the process proceeds to step S15, and the processes of steps S15 to S18 are executed in the same manner as in the first embodiment.

In the present invention, in addition to the above-described embodiments, the following various embodiments can be adopted.

In the above-described embodiment, the difference |X−Y| between the total difference X and the total difference Y is compared with the reference value P, but the ratio Y/X (or X/Y) between the total difference X and the total difference Y is calculated. You may compare with a predetermined reference value. In short, the suitability of the posture of the driver DR may be determined based on the comparison result of the total difference X and the total difference Y.

In the above-described embodiment, the difference |MN| between the center of gravity position M and the center of gravity position N is compared with the reference value Q, but the ratio M/N (or N/M) of the center of gravity position M and the center of gravity position N is calculated. You may compare with a predetermined reference value. In short, the suitability of the posture of the driver DR may be determined based on the comparison result between the center of gravity position M and the center of gravity position N.

In the above-described embodiment, the camera 1 captures the reference image B after the driver DR starts driving the vehicle 50. However, between the time the driver DR boards the vehicle 50 and the driving starts, The camera 1 may capture the reference image B.

In the above embodiment, an example in which the image processing unit 2 is provided independently of the control unit 3 has been described, but the image processing unit 2 may be incorporated in the control unit 3. Similarly, in the above-described embodiment, an example in which the storage unit 4 is provided independently of the control unit 3 has been described, but the storage unit 4 may be incorporated in the control unit 3.

In the above embodiment, an example in which the driver monitor 100 is mounted on a four-wheeled vehicle has been described, but the present invention can also be applied to a driver monitor mounted on another vehicle such as a truck or a bus.

1 Camera 2 Image processing unit 3 Control unit 4 Storage unit 5 Communication unit 6 Power supply circuit 11 Imaging unit 12 Light emitting unit 31 Difference calculation unit 32 Difference total calculation output unit 33 Vehicle condition judgment unit 34 Attitude judgment unit 35 Alarm output unit 36 Center of gravity calculation unit 50 Vehicle 100 Driver monitor (driver monitoring device)
200 ECU
300 electronic key DR driver

Claims

A camera provided so as to face the driver of the vehicle and capturing an image of the driver on the vehicle;
A posture determination unit that determines the posture of the driver based on the image captured by the camera, and
In a driver monitoring device including an alarm output unit that outputs an alarm when the posture determination unit determines that the driver's posture is inappropriate.
A difference calculation unit that calculates the difference between two images captured by the camera,
A difference total calculation output unit that binarizes the difference image obtained by the calculation of the difference calculation unit using a predetermined threshold value and calculates the total number of pixels equal to or larger than the threshold value in the binarized difference image as the total difference. Further equipped with,
The camera is
A first reference image taken by the driver when he is not in the vehicle, and
A second reference image taken by the driver in the vehicle and in a proper posture of the driver;
A driving image captured by the driver while driving is generated.
The difference calculation unit
The difference between the first reference image and the second reference image is calculated to generate a reference difference image which is the difference between the two images.
The difference between the first reference image and the driving image is calculated to generate a driving difference image which is the difference between the two images.
The difference total calculation output unit is
The total number of pixels above the threshold value in the binarized reference difference image is calculated as the first difference total X.
The total number of pixels above the threshold value in the binarized difference image during operation is calculated as the second total difference total Y.
The driver monitoring device is characterized in that the posture determination unit determines whether or not the posture of the driver is appropriate based on the comparison result between the first difference total X and the second difference total Y. ..
The driver monitoring device according to claim 1,
The posture determination unit,
When the difference between the first total difference X and the second total difference Y is |XY−, and the first reference value is P,
If |XY−≦P, it is determined that the posture of the driver is appropriate,
When | XY |> P, the driver monitoring device is characterized in that it is determined that the posture of the driver is inappropriate.
In the driver monitoring device according to claim 1,
The position of the center of gravity of the pixel region above the threshold value in the binarized reference difference image is calculated as the first center of gravity position M, and the position of the center of gravity of the pixel area above the threshold value in the binarized difference image during operation is calculated. Is further provided with a center of gravity calculation unit that calculates as the second center of gravity position N.
The posture determination unit is based on the comparison result of the first difference total X and the second difference total Y, and the comparison result of the first center of gravity position M and the second center of gravity position N of the driver. A driver monitoring device characterized in that it determines whether or not the posture is appropriate.
The driver monitoring device according to claim 3,
The posture determination unit,
The difference between the first total difference total X and the second difference total Y is | XY |, the difference between the first center of gravity position M and the second center of gravity position N is | MN |, the first reference value. When is P and the second reference value is Q,
When | XY | ≤P and | MN | ≤Q, it is determined that the posture of the driver is appropriate.
A driver monitoring device, characterized in that, when | XY |> P, or | MN |> Q, it is determined that the posture of the driver is inappropriate.
In the driver monitoring device according to any one of claims 1 to 4.
When the posture determination unit determines that the driver's posture is inappropriate, the posture determination unit measures the duration of the improper posture as the duration.
The driver monitoring device, wherein the alarm output unit outputs an alarm when the duration has passed a predetermined reference time.
In the driver monitoring device according to any one of claims 1 to 5.
When the camera generates the second reference image, the posture determination unit determines whether or not the posture of the driver who has boarded the vehicle is appropriate, the face information of the driver and the vehicle information of the vehicle. A driver monitoring device characterized by making a judgment based on.
The driver monitoring apparatus according to claim 6,
The posture determination unit determines whether or not the posture of the driver who has boarded the vehicle is appropriate by taking into consideration the personal learning data regarding the posture of the driver in addition to the face information and the vehicle information. A driver monitoring device characterized in that it does.
In the driver monitoring device according to any one of claims 1 to 7.
The camera captures the first reference image when it is confirmed that the driver, who possesses an electronic key for opening and closing the door of the vehicle, has approached the vehicle. Person monitoring device.