JP2013156707A - Driving support device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a driving support device that precisely determines awakening degree.SOLUTION: A driving support device includes: imaging means 11 for taking an image of a driver's face; and a control device 10 that detects movement of a mouth that is a feature point of the driver's face from a face image taken by the imaging means, detects feature behavior from a change of the movement, determines the awakening degree of the driver from the change of the feature behavior over time, detects an object having a predetermined positional relationship with a face image area, and stops detection of the feature behavior when the object is detected.

Description

  The present invention relates to a driving support device.

  The movement of the mouth is detected from the imaged image of the driver's face, the feature motion is detected from the detected shape change of the mouth, and the driver is mild based on the change over time of the detected feature motion. A device that determines an initial state of a decrease in arousal level that starts to feel drowsiness is known (Patent Document 1).

JP 2010-204984 A

  However, when capturing a driver's face image, if a disturbance occurs in the face image due to the driver's action of covering his mouth with his hand or moving his hand in the vicinity of his mouth, facial features such as his mouth Therefore, there is a possibility that the driver's characteristic motion may be erroneously detected without being accurately captured, and as a result, the determination accuracy of the driver's arousal level is lowered.

  The problem to be solved by the present invention is to improve the determination accuracy of the driver's arousal level even when the driver performs the operation of covering the mouth with a hand or moving the hand in the vicinity of the mouth. .

  In the present invention, when an object other than a face having a predetermined positional relationship with the face image area is detected, the above-described problem is solved by stopping the motion feature point detection process.

  According to the present invention, a scene where a disturbance is likely to occur in the face image is determined, and the motion feature point detection process is stopped. Even when the hand is moved, the driver's arousal level can be accurately determined.

It is a block diagram which shows schematic structure of the driving assistance system to which embodiment of this invention is applied. It is a graph which shows the relation between the eye closure rate of a driver concerning the present invention, and the frequency of secondary operation with time. It is a control block diagram which shows schematic structure of the driving assistance device to which one embodiment of this invention is applied. It is a figure which shows an example of a captured image. FIG. 4 is a control block diagram illustrating a sub-block for motion detection in FIG. 3. FIG. 6 is a control block diagram showing sub-blocks of sigh / deep breath detection and yawn detection in FIG. 5. It is a figure for demonstrating the procedure of the sigh / deep breath detection of FIG. 3, and a yawn detection. It is a figure for demonstrating the procedure of the sigh / deep breath detection of FIG. 3, and a yawn detection. It is a figure for demonstrating the procedure of the sigh / deep breath detection of FIG. 3, and a yawn detection. It is a figure for demonstrating the procedure of the sigh / deep breath detection of FIG. 3, and a yawn detection. It is a figure for demonstrating the procedure of the sigh / deep breath detection of FIG. 3, and a yawn detection. It is a figure for demonstrating the procedure of the sigh / deep breath detection of FIG. 3, and a yawn detection. It is a figure for demonstrating the procedure of the motion detection of the hand of FIG. It is a figure for demonstrating the procedure of the motion detection of the hand of FIG. It is a figure for demonstrating the procedure of the motion detection of the hand of FIG. It is a figure for demonstrating the procedure of the motion detection of the neck of FIG. It is a figure which shows the example of the weighting of the secondary operation | movement set in the struggle detection of FIG. It is a figure which shows the other example of a captured image. It is a flowchart figure which shows the control procedure of the driving assistance device of embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the present embodiment, a case where the driving support apparatus according to the present invention is applied to a driving support system 1000 for supporting driving of a vehicle will be described as an example. In this example, an example of mounting on a vehicle will be described, but it can be mounted on a moving body such as an airplane, a ship, or a heavy machine. The application of the driving support device is not limited to a mobile control device, but can be incorporated into various operation devices provided in factories, workplaces, and management towers that require high reliability in work and operation.

  FIG. 1 is a block configuration diagram of a driving support system 1000 including a driving support device 100 according to the present embodiment. As shown in FIG. 1, the driving support system 1000 of the present embodiment includes a camera 1 installed in a vehicle, a light source 2, a driving support device 100, a vehicle controller 200, a notification device 300, and a storage device 400. The communication device 500 can be provided. Moreover, although it is an example, the driving assistance system 1000 of this embodiment can be provided with the driving assistance device 610, the braking device 620, the output device 630, the air conditioner 640, the seat heater 650, the memory 660, and the navigation device 670. Each of these devices is connected by a CAN (Controller Area Network) or other in-vehicle LAN, and can exchange information with each other. Furthermore, an external device (computer) that can exchange information by communication via the communication device 500 can be provided.

  The camera 1 can be composed of a photoelectric conversion module in which a plurality of photoelectric conversion elements such as a two-dimensional CCD (Charge Coupled Devices) image sensor, a MOS sensor, or a CID are two-dimensionally arranged. The camera 1 is installed on the upper surface of an instrument panel, for example, which can capture the driver's face. The camera 1 captures the occupant's head region a plurality of times within a preset tracking time, and sends the captured image to the driving support apparatus 100 with information on the imaging timing.

  The light source 2 is an infrared irradiation device that irradiates infrared rays toward the driver's face so as not to interfere with driving. The camera 2 captures an image of the driver's face irradiated with infrared rays. This light source 2 is also installed at the center of an instrument panel or a handle, for example, which can irradiate the driver's face with infrared light without unevenness.

  As shown in FIG. 1, the control device 10 of the driving assistance device 100 according to the present embodiment executes a ROM 12 that stores a program for determining the degree of wakefulness of an occupant and a program stored in the ROM 12. A CPU 11 as an operation circuit that functions as the driving support device 100 and a RAM 13 that functions as an accessible storage device are provided.

  The control device 10 of the driving support device 100 according to the present embodiment realizes an image acquisition function, a first detection function, a feature detection function, a determination function, a second detection function, a detection control function, and an output function. Each function can be executed by the cooperation of the above-described software and the hardware described above. In the present embodiment, an example in which the control device 10 transmits each control command will be described as an example. However, the control device 10 according to the present embodiment can also transmit each control command via the vehicle controller 200. .

  The driving assistance apparatus 100 of this embodiment determines the driver's arousal level. Here, the driver's arousal level will be described. FIG. 2 is a graph showing the relationship between the frequency X of the secondary movement and the eye closure rate Y of the driver over time, the horizontal axis is the elapsed time Time, and the vertical axis is the frequency X of the secondary movement or the eye closure rate Y. Respectively. The closed eye rate Y correlates with the driver's arousal level. The higher the closed eye rate Y, the lower the awakening level, and the lower the closed eye rate Y, the higher the awakening level.

  As shown in FIG. 2, when drowsiness strikes during driving, the driver often takes several actions so as not to fall asleep. For example, when you sleep, your eyes will dry, so you can rub your eyes, stroke your face, bend your neck, do stretching exercises, sigh and deep breath to send oxygen to the brain, To do. Hereinafter, these operations are also referred to as secondary operations leading to falling asleep.

  When the present inventors have observed the relationship between the frequency X of such a secondary movement and the eye closure rate Y (or arousal level), the driver's eye closure rate Y shown at the left end of FIG. There is a strong tendency that the frequency X of the secondary movement takes the peak value Xmax between the normal state A and the dozing state D in which the closed eye rate Y shown at the right end of the figure is extremely high (the degree of arousal is very low). It has been found.

  This is because when the wakefulness starts to decrease from the normal state, the driver first tries to wake up as much as possible by struggling with or struggling with the sleeper, such as hand movement, neck movement, sigh or deep breathing, yawning, etc. This is because the frequency of secondary operations increases. On the other hand, if the awakening level is further reduced without winning the sleeper, the frequency is low because the awakening level is not high enough to allow such a secondary action.

  And, as shown in the figure, between the normal state A and the dozing progress state D, the struggle or struggling state B struggling with or struggling with the sleeper, and the initial state of dozing where the arousal level further decreases When divided into states C, the period of time until the secondary action frequency X increases to reach the peak value Xmax is struggling or conflicting state B, and the secondary action frequency X starts to decrease from the peak value Xmax and decreases greatly. Can be identified as the initial state of falling asleep. Regarding the relationship between the frequency X of the secondary motion and the normal state A and the dozing progress state D, the absolute value of the frequency X of the secondary motion is predetermined rather than the temporal change of the frequency X of the secondary motion. Although the absolute value of the frequency X of the secondary operation is small in both the normal state A and the dozing progress state D, the eye closure rate Y or the like is separately used to identify these. It is desirable to use the characteristic value of

  That is, in this example, the initial state C before the dozing progress state D is continuously detected over time with the frequency X of the secondary operation, and the predetermined time after the peak value Xmax of the frequency X shown in FIG. Are detected as the initial state C of dozing. The detection result is used in each in-vehicle device 600. The mode of use will be described later.

  Next, specific embodiments will be described.

  3 is a control block diagram showing the driving support apparatus 1 of FIG. 1 in more detail, FIG. 4 is a diagram showing an example of a captured image, FIG. 5 is a control block diagram showing a sub-block of motion detection of FIG. FIG. 4 is a control block diagram showing sub-blocks of sigh / deep breath detection and yawn detection in FIG. 3.

  The software recorded in the ROM (12) of the control device 10 in FIG. 1 includes an image acquisition function (F1) for acquiring a captured image including the driver's face, and driving included in the face image area of the acquired captured image. A first detection function (F2) for detecting the movement of the feature point of the person's face, a feature detection function (F3) for detecting the feature action of the driver from the detected change in the feature point of the face, and the detected feature A determination function (F4) for determining the driver's arousal level from the temporal change of the operation, a second detection function (F5) for detecting an object other than the face in a predetermined positional relationship with the face image area, and an object detected If so, a detection control function (F6) for canceling the detection of the characteristic motion is executed.

  As shown in FIG. 3, the image signal of the face image of the captured image including the driver's head imaged by the camera 1 detects neck and head movements, sighs and deep breaths, and yawns from changes in the temporal shape of the face. Output to the shape estimation function F12 for detecting the movement of the body (for example, a hand) other than the driver's face from the face background image of the face image.

  FIG. 4 is a diagram illustrating an example of a captured image K that is captured so as to include the driver's face. As shown in FIG. 4, a face image area FC is set at the center of the image including the eyes, mouth, nose, and eyebrows that are characteristic points of the driver's face. In order to detect an object other than a face (an object such as a driver's body such as a hand or an object held by the driver's hand) in a predetermined positional relationship with the face image area FC, the face image area FC is predetermined. The disturbance determination area V is set at the position of. In this embodiment, the case where the possibility of disturbance in the face image is high is a case where the feature point is hidden (image cannot be acquired), the periphery of the feature point is hidden, and another object moves near the feature point. The disturbance determination area V of the present embodiment is set around the face image area FC set at the center of the captured image and along at least a part of the outer edge of the face image area FC. Although not particularly limited, the disturbance determination areas V3 and V4 can be set at least on the right side or the left side of the face image area FC. Further, the disturbance determination areas V1 and V2 can be set at least above or below the face image area FC. In this example, four disturbance determination areas V1 to V4 are set vertically and horizontally without overlapping the face image area FC so as to surround the outer edges of the four sides of the rectangular face image area FC. Furthermore, a body image detection region D for detecting the movement of a body such as a hand other than the driver's face can be set outside the face image region FC. In this example, disturbance determination areas V1 to V4 are set outside the face image area FC, and body image detection areas D1 to D4 are set outside the disturbance determination areas V1 to V4. The disturbance determination areas V1 to V4 and the body image detection areas D1 to D4 in this example are set in a band shape substantially parallel to the edge of the rectangular captured image. The widths and lengths of the disturbance determination areas V1 to V4 and the body image detection areas D1 to D4 can be appropriately set according to the size of the captured image K and the size and position of the face image.

Hereinafter, functions executed by the driving support device 100 of the present embodiment will be described.
First, the first detection function according to this embodiment of the present invention will be described. The first detection function has a function of detecting movement of feature points of the driver's face included in the face image area FC. Further, the first detection function further includes a function of detecting a movement of the body other than the driver's face included in the body image detection region D. Note that the process of the first detection function described below can be applied to the process of the second detection function for detecting an object in the disturbance determination region V described later.

  First, the shape estimation function F12 that detects the movement of the feature point of the driver's face will be described. As shown in FIG. 5, a shape change calculation process F211 based on the shape estimation signal, a noise removal process F212, and various operation detection processes F213 are performed. In the noise removal processing F212, the movement of the driver's face and the steering wheel operation are performed using the image signal from the camera 1, the shape estimation signal from the shape estimation function F12, and the steering signal of the steering control function 2 obtainable via the vehicle controller 200. Predictable movements such as hand movements are detected and removed.

  The shape estimation function F12 receives a digital image signal (hereinafter also referred to as a video stream) captured continuously in time, and uses an AAM (Active Appearance Model) method using a statistically expressed learning model. The position of the mouth, nose, and chin. The mouth, nose, and jaw have shapes and changes in shape.

  In addition, the control device 10 of the present embodiment includes a feature detection function that detects a driver's feature motion based on a change in the facial feature point detected by the first detection function and a body movement described later. The control device 10 of the present example detects movements of the neck and head, sighs and deep breaths, and yawns using the mouth shape, the nose shape, the jaw shape, and the like estimated by the shape estimation function F12. This process will be described with reference to FIG. The neck and head movements are processed by the neck movement detection function F22 shown in FIG. 3, the sigh and deep breathing are processed by the sigh / deep breath detection function F23 shown in FIG. 3, and the yawn is processed by the yawn detection function F24 shown in FIG.

  As shown in FIG. 6, first, in step S11, the shape estimation signal from the shape estimation function F12 described above is input, the previous image is compared with the current image, and a feature point change vector is calculated. Then, it is identified whether or not the feature point change vector is a face motion.

  Feature coordinates are extracted based on the shape estimation signal estimated by the shape estimation function F12. This process shows a case where the shape estimation signal of the mouth shape is extracted from the shape estimation signals for the mouth shape, the nose shape and the jaw shape shown in the upper diagram of FIG.

  Next, in step S12, coordinate normalization processing is executed. Since the feature points extracted in the above steps are coordinates according to the size, yaw angle, pitch angle, and roll angle of the driver's face, it is necessary to normalize them before using them for determination. However, in this example, since only the front side of the driver's face needs to be handled, only the roll angle is considered.

  Then, as shown in FIG. 8, paying attention to both ends of the mouth (× in the left figure) in the mouth shape estimation signal, the roll angle θ is detected. Further, the center of the mouth is detected using the mouth shape estimation signal, and the feature point is rotated so that the previously obtained roll angle θ becomes zero. Next, as shown in the right figure of the figure, the minimum rectangle including all the feature points is specified, and the coordinates of each feature point are coordinate-converted to a value between 0 and 1.

  When the feature point coordinate normalization process is performed as shown in FIG. 8, in step S13, the feature vector is calculated by combining the distances between the feature points. In this process, as shown in FIG. 9, vectors f0, f1,... M0, m1,... That are considered to most reflect the motion in the vertical direction are extracted, and these are extracted as one vector P = {f0, f1,. , M1,. Further, a difference vector X = P−Pave is calculated by setting an average value of the past n vectors P as an average vector Pave. The average vector Pave is used to ignore a slight difference.

  The difference vector X thus calculated is a feature vector. This is, for example, as shown in FIG. 11, “Yawning (Yes (1))” or “No yawning (No (0))”. Or any of the above. The classification performed in step S14 can be executed by a machine learning technique such as ANN (Artificial Neural Networks), SVM (Support Vector Machines), or SOM (Self-Organizing Maps). This is the binarization classification process shown in FIG.

  Subsequently, in step S15, when binarization classification processing is performed, specific operations such as neck or head motion, sigh or deep breath, and yawn are extracted based on the binary data (operation extraction in FIG. 6). processing).

  Yawn detection is performed by the yawn detection function F24 in FIG. When yawning, the mouth changes greatly in the vertical direction. In the feature vector of FIG. 9, the vertical vector strongly reflects the yawn, and therefore the vertical change of the mouth shape change is important for detecting the yawn. Further, when such a vertical feature vector is extracted and the binarization classification process shown in FIG. 11 is performed, the yawn maintains its shape for a relatively long time. It is determined that a long action is yawning, and an action shorter than a predetermined time is not yawning. In the figure, three actions are extracted, but the first and third actions are determined to be yawning because of the long time, but the second action is not yawning because the time is short. Is likely to be a conversation, for example a conversation. Therefore, it is determined that the operation is not yawning. By setting the time threshold in this way, the distinction from other operations is enhanced.

  The detection of sigh or deep breath is executed by the sigh or deep breath detection function F23 of FIG. The sigh here refers to the action of exhaling greatly, and does not mean the gesture of dropping the shoulder when discouraged. When such sighs and deep breaths are taken, the mouth narrows and the lateral change increases. In the feature vector of FIG. 10, since the horizontal vector strongly reflects sighs and deep breaths, the change in the horizontal direction among the mouth shape changes is important in detecting sighs and deep breaths. In addition, when such horizontal feature vectors are extracted and the binarization classification process shown in FIG. 11 is performed, sighs and deep breaths are not as long as yawning but maintain their shapes for a longer time than conversation. If such a predetermined time is set, it is possible to improve discrimination from other actions such as conversation.

  The detection of the neck or head motion is executed by the neck motion detection function F22 shown in FIG. The action of the neck or head here means an action of bending or turning the neck left and right, such as a stretching exercise for waking up sleepiness.

  When the neck or head is bent or rotated to the left or right, as shown in FIG. 13, the inclination θ of the straight line connecting both ends of the mouth with respect to the x axis of the shape estimation signal estimated by the shape estimation function F12 changes with time. The upper middle figure and the upper right figure in the figure show that the neck is inclined to the left and right. If the change in the difference Δθt = θt-θt-1 from the previous inclination angle θt-1 changes according to the prescribed rule as shown by the arrow in the lower figure of the figure, Is determined to be bent or rotated to the left or right, otherwise it is determined not to be a movement of the neck to wake up sleepiness.

  As described above, the method of detecting the movement of the feature point of the driver's face in the first detection function of the present embodiment, specifically, the shape estimation function F12 to the neck motion detection function F22, sigh / deep breath detection shown in FIG. The function F23 and the yawning detection function F24 have been described.

  Next, a technique for detecting the movement of the body (hand) other than the driver's face in the first detection function, specifically, the motion detection function F11 and the hand motion detection function F21 shown in FIG. 3 will be described.

  The motion detection function F11 detects a driver's motion by an optical flow method that expresses a motion of an object as a vector from a digital image signal captured continuously in time by the camera 1. This movement of the hand is used to detect the secondary movement that finally rubs the eyes.

  In this case, image data of an area other than the face area of the image captured by the camera 1 is also used. However, the camera 1 is installed on the upper surface of the instrument panel or the like in order to photograph the driver's face. The movement of the driver's hand holding the steering wheel in the field of view may also be reflected. For this reason, as shown in FIGS. 3 and 5, an actual steering signal by the steering wheel operation is input from the vehicle steering control device 2 to the hand motion detection function F21. This will be described later.

  In this example, in order to determine whether or not the driver's hand touches the eyes or face, as shown in FIG. 4, body image detection is performed on the four sides around the face of the image data captured by the camera 1. Regions D1 to D4 are set. In addition, in the same figure, the symbols of the dots and arrows shown outside the screen are described outside the box in order to show the detection results of these detection regions D1 to D4 in an easily understandable manner.

  When the object intersects the body image detection areas D1 to D4, the direction is detected by the optical flow method. For example, the hand in the body image detection region D3 in FIG. 4 shows an optical flow that moves from the outside of the screen toward the face.

  The hand motion detection function F21 includes a shape change calculation function F211, a noise removal function F212, and a hand motion extraction process F213 shown in FIG. 5, and the following processes are executed.

  First, a shape estimation signal from the shape estimation function F12 described above is input, and the previous image and the current image are compared to calculate a feature point change vector. Then, it is identified whether or not the feature point change vector is a face motion.

  In the noise removal processing F212, it is possible to predict the movement of the driver's face and the hand operating the steering wheel using the image signal from the camera 1, the shape estimation signal from the shape estimation function F12, and the steering signal from the steering control device 2. Detect motion and remove it.

  The signal output from the noise removal processing F212 is subjected to the following processing in the hand motion extraction processing F213. That is, the optical flow in each of the detection areas D1 to D4 shown in FIG. 4 is the same except that the direction of the optical flow is the face movement direction or the handle operation direction as shown in FIG. A relatively large number of optical flows are extracted, and it is determined whether the flow is closer to the face or away from the face. The optical flows for the detection areas D1 to D4 are classified as shown in FIG.

  Then, as shown in the lower part of FIG. 16, for each detection region, when an optical flow in the direction approaching the face is detected, flag 1 is detected, and when an optical flow in the direction away from the face is detected, flag 0 is not detected. When flag 0 is set after flag 1 is set, that is, when the driver's hand once passes through body image detection regions D1 to D4 from the outside to the inside, the flag n is set to the outside. It is determined that the face has been touched when passing toward. As described above, the driver's characteristic motion can be detected from the change in the body motion in the body image detection region D detected by the first detection function.

  As described above, according to the driving support device 1 of the present example, the degree of arousal such as the action of bending the neck and head, sighing and deep breathing, yawning, and touching the face with the hand from the captured image of the driver's face and body is reduced. The secondary operation observed in the initial state C can be detected.

  Next, the determination function of the control apparatus 10 of this embodiment is demonstrated. The control device 10 of the present example determines the driver's arousal level based on the feature motion detected by the feature detection function. Although not particularly limited, the frequency per unit time of the secondary operation detected by the above processing is counted by the struggle detection function F31 shown in FIG. Then, in the initial state determination function F4 for arousal level reduction, for example, the initial state C for the reduction in arousal level is determined from the change in frequency shown in FIG.

  In addition, when counting the frequency of the secondary motion by the struggle detection function F31, the motions of bending the neck and head, sighing and deep breathing, yawning, and touching the face with hands as shown in FIG. 17 all have the same weighting ω0 to ω3. However, the weights ω0 to ω3 can be changed according to the driver's habit.

  For example, if there are many movements that move the neck or head extremely during the learning operation in advance, it is determined that the movement is also a driver's eyelid, and the weight of the detected movement of the neck or head is reduced.

  Further, by combining with a device that detects the closed state of the driver, all of the normal state A, the struggle or conflict state B, the initial state C of dozing and the progress state D of dozing shown in FIG. 2 can be detected.

  As described above, according to the driving support device of the present example, since the frequency of the secondary motion that occurs in a state where the arousal level, particularly the arousal level is reduced, is detected, the struggle or conflict state B or the initial state C of doze is determined. It can be determined with high accuracy.

  Moreover, since the determined state can determine not only the initial state C of dozing but also the struggle or conflict state B having a higher arousal level, it is possible to alert the driver in advance.

  Yawn detection is based on the vertical shape change of the mouth, sighs and deep breaths are based on the horizontal shape change of the mouth, and neck and head movements are based on the angle of inclination of both ends of the mouth. Therefore, yawning, sighing / deep breathing, and neck movement can be accurately detected, and erroneous recognition of these can be suppressed.

  In addition, since the hand movement is determined by a combination of the movement in the direction approaching the face and the movement away from the face by removing the steering operation using the steering control device 2, the movement of the hand related to the secondary movement. Can be detected with high accuracy.

  In addition, by changing the weighting for the secondary action depending on the driver's habit, it is possible to determine the initial state of the arousal level reduction with higher accuracy.

  Then, the 2nd detection function of this embodiment of this invention is demonstrated. The control device 10 according to the present embodiment detects an object other than a face that has a predetermined positional relationship with the face image area FC. Note that the objects other than the face include non-body objects such as a driver's hand, shoulder, and elbow, and a handkerchief, drink, and cigarette that the driver has.

  Incidentally, as shown in the captured image shown in FIG. 18, when the driver moves his / her hand, a part of the driver's face may be hidden. As described above, when detecting feature points of the driver's face included in the face image area FC of the captured image, for example, movements of P1 to P4 and P11 to P14, an object such as a driver's hand is a face image. If a disturbance occurs in the face image by entering the area FC, the position of the feature point may not be detected accurately. In particular, as shown in FIG. 18, when the image of the driver's hand enters the outside of the face image area FC and moves so as to be represented by a broken line, the face image is disturbed and the position of the feature point is accurately detected. It becomes difficult to do. For this reason, the control device 10 detects an object having a predetermined positional relationship with the face image area FC in order to determine the possibility of disturbance in the face image. Although the positional relationship with the face image area FC is not particularly limited, in the present embodiment, the position within the outer edge of the image area FC, the position on the outer edge of the image area FC, the outer position of the outer edge of the image area FC, the position of the image area FC A position at a predetermined distance from the center can be defined as a predetermined positional relationship.

  The control device 10 of the driving assistance device 100 according to the present embodiment of the present invention has a disturbance in a predetermined positional relationship with the face image region FC in order to determine whether the object affects the face image included in the image information. The determination area V is set. As described above, the disturbance determination area V is preferably provided on the right side or the left side outside the face image area FC. In this embodiment, since the disturbance determination area V is set in the left and right areas, it is possible to detect the movement of the hand that tends to be moved along the left and right directions with high accuracy. The disturbance determination area V is preferably provided on the upper side or the lower side outside the face image area FC. In the present embodiment, since the disturbance determination area V is set in the upper and lower areas, it is possible to detect with high accuracy that the hand has been moved along the vertical direction.

  The control device 10 detects an object within the disturbance determination area V set at a predetermined position around the face image area FC as an object having a predetermined positional relationship with the face image area FC, and detects an object outside the disturbance determination area V. It is detected as an object that is not in a predetermined positional relationship with the face image area FC. A method for determining whether or not an object exists within the disturbance determination region V is not particularly limited, but when a part of the object exists within the disturbance determination region V, it is determined that the object exists within the disturbance determination region V. Alternatively, when the area occupied by the object image with respect to the disturbance determination region V is greater than or equal to the threshold value, it may be determined that the object exists within the disturbance determination region V. Thus, it is possible to make an accurate determination by determining whether there is an object that affects the face image according to a predetermined criterion.

  Although not particularly limited, the control device 10 of the driving support device 100 according to the present embodiment sets the threshold of the occupation area ratio when the disturbance determination region V is set on the right side or the left side of the face image region FC. It can be made lower than the threshold of the occupation area ratio when the disturbance determination region V is set on the upper side or the lower side. As described above, the movement of the driver's hand is frequently detected along the left-right direction. For this reason, in the present embodiment, a relatively high threshold is set for the disturbance determination area V along the left and right to detect the movement of the object based on a strict standard, and a relatively low threshold is set for the disturbance determination area V along the top and bottom. To detect the movement of an object with a strict and strict standard. This makes it possible to detect the movement of the hand or other object that causes disturbance in the face image with high accuracy.

  Next, the detection control function will be described. When the object is detected by the second detection function, the driving support apparatus 100 according to the present embodiment of the present invention stops the detection of the feature motion based on the change of the facial feature point by the first detection function. On the other hand, when the object detected by the second detection function is not in a predetermined positional relationship with the face image area FC, the feature action is detected. As a result, the detection of the feature motion can be stopped when the possibility of generating disturbance in the face image is high, and the detection of the feature motion can be executed when the possibility of generating disturbance in the image is low. It is possible to accurately execute the detection of the operation and, in turn, the determination of the driver's arousal level. As a result, reliable driving assistance can be performed.

  In addition, the driving assistance device 100 according to the present embodiment of the present invention can detect a driver's movement from a change in body movement such as a hand other than the driver's face even when an object is detected by the second detection function. Continue the process of detecting feature motion. That is, the detection process of the driver's characteristic motion based on a change in the movement of the body such as the hand is not stopped. Even if there is a high possibility that disturbance will occur in the face image, no disturbance will occur in the hand image other than the driver's face, so detection of feature motion based on body images other than the face will continue. . Since the accuracy of detection of feature motion can be maintained based on an image of a body other than the face that is less likely to cause disturbance in the face image, driving support based on arousal level can be continued. As a result, stable driving assistance can be performed without causing the driver to feel uncomfortable.

  Moreover, the control apparatus 10 of the driving assistance apparatus 100 deletes the detection result of the feature point for the predetermined time used for determination of an arousal level, when an object is detected by the 2nd detection means. The control device 10 according to the present embodiment detects a feature motion from changes in feature points of the face and body (hand), and determines the driver's arousal level based on the change in the feature motion over time. That is, the arousal level is determined based on the detection result of the feature points detected in the predetermined time. For this reason, even when a situation in which there is a high possibility of causing a disturbance in the face image occurs temporarily, the reliability of the feature point detection result at a predetermined time including that point is lowered. In this embodiment, even when a situation that is highly likely to cause a disturbance in the face image occurs temporarily, in order to maintain the accuracy of the determination, the feature for a predetermined time used for the determination of the arousal level Delete the point detection result. Thereby, determination of a driver | operator's arousal level can be performed correctly and reliable driving assistance can be performed.

  FIG. 19 is a flowchart for explaining the control procedure of the driving support apparatus 10 according to the present embodiment of the present invention, in particular, the processing of the detection control function of the present embodiment. In this embodiment, after starting, the determination of the arousal level in steps S110 to S112 is executed, and the processes in steps S101 to S107 are executed in a predetermined cycle in parallel with this process. Hereinafter, the processing of the detection control function will be mainly described on the assumption that the determination of the arousal level is executed.

  First, in step S101, the control device 10 sets a disturbance determination region V in order to detect an object other than a face having a predetermined positional relationship with the face image region FC. The disturbance determination area V is in a predetermined positional relationship with the face image area FC. In step S102, the control device 10 operates the second detection function to detect an object within the disturbance determination region V. As the object detection technique, the technique of the first detection function can be used.

  Subsequently, in step S <b> 103, the control device 10 calculates the occupation ratio occupied by the area of the object image with respect to the area of the disturbance determination region V. In step S104, the control device 10 determines whether or not the occupation ratio is equal to or greater than a predetermined value and the state has continued for a predetermined time. This determination makes it possible to determine whether or not the image information is likely to be disturbed. If the state where the occupancy rate of the object image with respect to the disturbance determination region V is equal to or greater than the predetermined value continues for a predetermined time, the process proceeds to step S105, and otherwise, the process proceeds to step S110.

  In step S104, when it is determined that the state in which the occupation ratio continues to be equal to or higher than the predetermined value and there is a high possibility of disturbance in the image information, in step S105, determination of the arousal level based on the movement of the face. Cancel processing. Taking the above-described process as an example, the determination process of the agony state is stopped. Further, in step S106, the detection result of the feature points of the face for a predetermined time used for the determination of the arousal level is deleted. Thereby, it is possible to prevent an erroneous determination of arousal level. After deleting the detection result, the arousal level determination process and the detection control process are repeated.

  If No in step S104, Yes in step S108, and there is a low possibility that disturbance will occur in the image information, the processing after step S110 is performed. Since the arousal level determination process in step S110 has been described above, a description thereof will be omitted. In subsequent step S111, a state in which the arousal level is reduced is detected. In the present embodiment, an initial state in which the arousal level is reduced, that is, an agony state in which the driver resists sleep is detected. Note that the method for determining the arousal level in the present invention is not particularly limited, and a method for determining the driver's situation such as the arousal level based on the face image can be appropriately applied.

  Even if it is determined in step S104 that the occupation ratio continues to be greater than or equal to the predetermined value and there is a high possibility of disturbance in the image information, the process proceeds from step S107 to step S110. The arousal level determination process based on body movements other than the face is continued at a predetermined cycle. Since disturbances do not occur in the image information of the driver's hand, the determination of the arousal level should be continued based on the body such as the hand other than the driver's face while suppressing the output of erroneous determination results. Can do.

  In step S112, the driving support apparatus 100 outputs a determination result of the arousal level. The detected driver's arousal level is used to support safer driving. The control device 10 of the mood determination device 100 of the present embodiment includes a vehicle controller 200 that can control the driving of the vehicle, a notification device 300 for notifying the driver, a storage device 400, and the driver. And output to any one or more of the communication devices 500 capable of transmitting / receiving information to / from the outside. Then, the determination result of the arousal level is fed back to the control such as the driving operation of the driver, the drive control of the vehicle, the next driving operation, and the driving operation of another driver. Although not particularly limited, the feedback method of the determination result of the arousal level in the present embodiment is exemplified below.

  As a first example, the vehicle controller 200 that has acquired the determination result of the arousal level sends a command for setting a long inter-vehicle distance to be maintained to the inter-vehicle distance maintaining device 611 when the awakening level is low in order to ensure safe driving. When the awakening level is low, a command for setting the vehicle speed to be maintained low is sent to the vehicle speed maintenance device 620. Similarly, a control command for realizing safe traveling such as assisting a steering wheel operation by the steering assist system can be sent to another traveling support device 610. Further, in order to ensure safe traveling, the vehicle controller 200 can send a command for decelerating the vehicle to the braking device 620 and guiding the vehicle to stop when the arousal level is low. Thereby, the driving | running | working assistance according to a driver | operator's alertness can be performed.

  As a second example, the notification device 300 that has acquired the determination result of the arousal level indicates the current driver's arousal level via the display 631 and / or the speaker 632 by using characters, figures, flashing signals, voices, and the like. Can be notified. In addition, the notification device 300 that has acquired the determination result of the arousal level can send a command to blow air or cold air to the air conditioner 640 in order to notify the driver of the awakening level and change the driver's mood. . Furthermore, the notification device 300 that has acquired the determination result of the arousal level can send a cooling command to the seat heater 650 in order to notify the driver of the arousal level and increase the driver's arousal level. Thus, the notification device 300 can notify the driver himself / herself directly or indirectly of the driver's arousal level.

  As a third example, the driver can be alerted that he is in the initial state of falling asleep, and for example, a parking area existing in the traveling direction of the own vehicle can be searched for and guided by the car navigation device.

  As a fourth example, the storage device 400 that has acquired the determination result of the arousal level can be stored in the storage memory 660 in association with the tracking time T. In addition, the storage device 400 uses the obtained determination result of the arousal level, the map information 671 disposed on the vehicle or server used by the navigation device 670, the road information 672, the tracking time T and the travel position at the tracking time T, Can be stored in association with each other. Thereby, the awakening degree of the driving | running | working driver | operator can be written in the map information 671 and the road information 672.

  As a fifth example, the communication device 500 that has acquired the determination result of the arousal level can send the acquired determination result of the arousal level to the external device 700 via a communication line. Thereby, a driver | operator's attention can also be alerted from the outside. For example, when the initial state of dozing is detected, the signal is transmitted to an external road management system, and “Do n’t sleep!” Or the like is displayed on the roadside signboard, as is done for an overspeeding vehicle, for example. . Furthermore, awakening information of a large number of drivers traveling can be collected in the external device 700 and used for road design and road environment maintenance.

  Thus, the determination result of the driver's arousal level during the tracking time T is output to the vehicle controller 200, the notification device 300, the storage device 400, the communication device 500, etc., and each device feeds back the determination result of the arousal level. By this, it can be used for vehicle control performed later, a driver | operator's operation, and another driver | operator's operation.

  When the output process ends, the process proceeds to step S113, and the arousal level determination and detection control process are repeated at a predetermined cycle until an end command is input.

  The driving support device 100 of the present embodiment configured and operating as described above has the following effects.

  (1) According to the driving support apparatus 100 according to the present embodiment of the present invention, when an object other than a face having a predetermined positional relationship with the face image area FC is detected, disturbance may occur in the face image. Because it detects scenes with high characteristics and stops detecting movements of facial feature points, driving even when the driver moves the hand around the mouth or moves the hand near the mouth It is possible to accurately determine a person's arousal level.

  (2) According to the driving assistance apparatus 100 according to the present embodiment of the present invention, when the object detected by the second detection function is not in a predetermined positional relationship with the face image area FC, the feature motion is detected. Since it is resumed, the detection of feature motion is stopped when there is a high possibility of causing disturbance in the face image, while the detection of feature motion is executed when the possibility of occurrence of disturbance in the face image is low. Therefore, it is possible to accurately execute detection of characteristic movements and, in turn, determination of the driver's arousal level. As a result, highly reliable driving assistance can be performed.

  (3) According to the driving support apparatus 100 according to the present embodiment of the present invention, even when a situation that is highly likely to cause a disturbance in the face image occurs temporarily, the accuracy of the determination is maintained. In addition, the detection result of feature points for a predetermined time used for determination of the degree of arousal is deleted. Thereby, determination of a driver | operator's arousal level can be performed correctly. As a result, highly reliable driving assistance can be performed.

  (4) According to the driving support apparatus 100 according to the present embodiment of the present invention, an object within the disturbance determination area V set at a predetermined position around the face image area FC has a predetermined positional relationship with the face image area FC. Since an object outside the disturbance determination area V is detected as an object that is not in a predetermined positional relationship with the face image area FC, it is determined whether or not the captured image is in a state of disturbance based on a clear standard. Judgment can be made. In addition, the load of determination processing can be reduced.

  (5) According to the driving assistance apparatus 100 according to the present embodiment of the present invention, the disturbance determination area V is set in the left and right areas, and thus the movement of the hand that is likely to be moved along the left and right direction is detected with high accuracy. can do.

  (6) According to the driving assistance apparatus 100 according to the present embodiment of the present invention, the disturbance determination area V is set in the upper and lower areas, so that it is detected with high accuracy that the hand is moved along the vertical direction. Can do.

  (7) According to the driving support apparatus 100 according to the present embodiment of the present invention, when the area occupied by the image of the object occupies the disturbance determination region V is greater than or equal to the threshold value, the object is determined as a disturbance. Since it is determined that the object exists within the region V, it can be determined according to a predetermined standard whether or not there is an object that affects the face image, and thus an accurate determination can be made.

  (8) According to the driving assistance apparatus 100 according to the present embodiment of the present invention, a relatively high threshold is set for the disturbance determination region V along the left and right, and the movement of the object is detected based on a strict standard, and the vertical direction is followed. For the disturbance determination region V, a relatively low threshold value is set and the movement of the object is detected based on a strict and strict standard. Therefore, the movement of a hand or other object that causes a disturbance in the face image can be detected with high accuracy. .

  (9) According to the driving support apparatus 100 according to the present embodiment of the present invention, driving from a change in body movement other than the driver's face included in the body image detection region D set outside the face image region FC. The person's characteristic motion is detected. Even if there is a high possibility that disturbance will occur in the face image, no disturbance will occur in the image of the hand other than the driver's face, so feature motion detection is performed based on the body image other than the face. The determination of arousal level can be made.

  (10) Furthermore, the driving support apparatus 100 according to the present embodiment of the present invention can detect a change in the movement of a body such as a hand other than the driver's face even when an object is detected by the second detection function. The process of detecting the driver's characteristic motion is continued. That is, the detection process of the driver's characteristic motion based on a change in the movement of the body such as the hand is not stopped. Thereby, since the accuracy of detection of the feature motion can be maintained based on the image of the body other than the face that is less likely to cause disturbance in the face image, driving support based on the arousal level can be continued. As a result, stable driving assistance can be performed without causing the driver to feel uncomfortable.

  The imaging means according to the present invention corresponds to the camera 1, the first detection means according to the present invention, the second detection means correspond to the motion detection function F11 and the shape estimation function F12, and the feature detection means according to the present invention is It corresponds to a hand motion detection function F21, a neck motion detection function F22, a sigh / deep breath detection function F23, and a yawn detection function F24. The determination means according to the present invention corresponds to the initial state determination function F4 of arousal level reduction. The detection control means according to the invention corresponds to the detection control function F5.

DESCRIPTION OF SYMBOLS 1000 ... Driving assistance system 100 ... Driving assistance apparatus 10 ... Control apparatus 1 ... Camera 2 ... Light source 13 ... Control apparatus F11 ... Motion detection function F12 ... Shape estimation function F21 ... Hand motion detection function F22 ... Neck motion detection function F23 ... Sigh / deep breath detection function F24 ... yawn detection function F31 ... struggle detection function F4 ... initial state determination function F5 ... detection control function 200 ... vehicle controller 300 ... notification device 400 ... storage device 500 ... communication device 600 ... in-vehicle device 610: Driving support device 611: Inter-vehicle maintenance device 612 ... Key plane device 620 ... Braking device 630 ... Output device 631 ... Display 632 ... Speaker 640 ... Air conditioner 650 ... Seat heater 660 ... Memory 670 ... Navigation device 700 ... External device 700

Claims (10)

Image acquisition means for acquiring a captured image including a driver's face;
First detection means for detecting movements of feature points of the driver's face included in the face image area of the captured image acquired by the image acquisition means;
Feature detecting means for detecting a feature action of the driver from a change in the feature point of the face detected by the first detecting means;
Determination means for determining the driver's arousal level based on the characteristic motion detected by the characteristic detection means;
Second detection means for detecting an object other than a face in a predetermined positional relationship with the face image region;
A driving support apparatus, comprising: a detection control unit that stops detection of a feature action based on a change in a feature point of the face when an object is detected by the second detection unit.   2. The detection control unit according to claim 1, wherein when the object detected by the second detection unit is not in a predetermined positional relationship with the face image region, the detection of the characteristic motion is performed. The driving assistance apparatus as described.   2. The detection control unit according to claim 1, wherein when the second detection unit detects an object, the detection control unit deletes the detection result of the feature points for a predetermined time used for the determination of the arousal level. Or the driving assistance apparatus of 2 description.   The second detection means detects an object within a disturbance determination area set at a predetermined position around the face image area as an object having a predetermined positional relationship with the face image area, and an object outside the disturbance determination area The driving support device according to any one of claims 1 to 3, wherein the vehicle is detected as an object that is not in a predetermined positional relationship with the face image area.   The driving assistance device according to claim 4, wherein the disturbance determination area is set on a right side or a left side of the face image area.   The driving assistance device according to claim 4 or 5, wherein the disturbance determination area is set above or below the face image area.   The said 2nd detection means detects as an object in the said disturbance determination area | region, when the image whose occupation area ratio with respect to the said disturbance determination area | region exists more than a threshold value exists. The driving support device according to one item.   The threshold value when the disturbance determination area is set on the right side or the left side of the face image area is lower than the threshold value when the disturbance determination area is set above or below the face image area. The driving support apparatus according to claim 7, wherein The first detection means detects a movement of the body other than the driver's face included in the body image detection area set outside the face image area,
The feature detection means detects the driver's characteristic action from a change in the body movement in the body image detection area detected by the first detection means. The driving support device according to one item.   The detection control means continues the process of detecting the driver's characteristic action from a change in the movement of the body other than the driver's face even when the object is detected by the second detection means. The driving support apparatus according to claim 9, wherein the driving support apparatus is characterized.

Images