JP2017188017A - Device and method for identifying facial orientation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device and method and device for identifying facial orientation, which offer improved accuracy in identifying facial orientation of a driver.SOLUTION: A facial orientation identification device 1 comprises; an image acquisition unit 111 configured to acquire a captured image obtained by photographing the face of a driver and a luminous part provided on a wearable member worn on the head of the driver; a characteristics identification unit 112 configured to identify characteristics of the face of the driver from the captured image; a position identification unit 113 configured to identify the position of the luminous part from the captured image; and a facial orientation identification unit 114 configured to identify orientation of the luminous part based on the position of the luminous part identified by the position identification unit, and to identify orientation of the face of the driver based on the characteristics of the face of the driver identified by the characteristics identification unit and the orientation of the luminous part.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a face direction specifying device and a face direction specifying method for specifying the face direction of a driver who drives a vehicle.

  2. Description of the Related Art Conventionally, a technique for capturing a face of a driver driving a vehicle with a camera and specifying the direction of the driver's face based on the facial features recognized from the captured image is known. Patent Document 1 discloses a driver monitoring system that detects that a driver is looking aside or falling asleep based on recognized facial feature points.

JP-A-2015-194484

  However, when the driver's face is photographed with a camera, the face angle is excessive, the light intensity is insufficient, the light intensity is excessive, the driver is wearing sunglasses, etc. Under the circumstances, there was a problem that the driver's face could not be recognized with high accuracy. Therefore, there is a need for a method for specifying the direction of the driver's face with high accuracy even in a situation where the driver's face cannot be photographed satisfactorily by the camera.

  Therefore, the present invention has been made in view of these points, and an object thereof is to provide a face orientation identifying device that can improve the accuracy of identifying the driver's face orientation.

  In the first aspect of the present invention, image acquisition for acquiring a captured image generated by photographing a driver's face and a light emitting unit provided on a mounting member worn on the head of the driver. A feature specifying unit that specifies a driver's facial feature based on the captured image, a position specifying unit that specifies a position of the light emitting unit based on the captured image, and the position specified by the position specifying unit A face that identifies the direction of the light emitting unit based on the position of the light emitting unit, and that identifies the direction of the driver's face based on the facial features of the driver identified by the feature identifying unit and the orientation of the light emitting unit There is provided a face orientation identification device having an orientation identification section.

  The face orientation specifying device further includes a storage unit that stores relationship information indicating a relationship between the orientation of the light emitting unit and the orientation of the face, and the face specifying unit is configured such that the feature specifying unit is a feature of the face. When the accuracy of the result of specifying is lower than a predetermined level, the face direction may be specified by referring to the face direction stored in the storage unit in association with the direction of the light emitting unit. .

  Whether the face orientation specifying unit records, in a predetermined timing, relation information indicating a relationship between the face orientation and the light emitting unit based on the facial feature specified by the feature specifying unit in the storage unit. It may be determined whether or not.

  The face orientation specifying unit may record the relationship information in the storage unit when the accuracy with which the feature specifying unit specifies the facial feature satisfies a predetermined condition.

  The face orientation specifying unit is configured to store the relationship information after a predetermined time has elapsed since the relationship between the face orientation based on the facial feature specified by the feature specifying unit and the orientation of the light emitting unit has changed. Recording may be resumed.

  The face orientation specifying unit determines the weight of the face feature and the weight of the light emitting unit used for specifying the face orientation based on the accuracy with which the feature specifying unit specifies the feature of the face. Also good.

  In the second aspect of the present invention, a captured image generated by photographing a driver's face and a light emitting unit provided on a mounting member mounted on the head of the driver, which is executed by a computer. Obtaining a feature of the driver's face based on the captured image, identifying a position of the light emitting unit based on the captured image, and based on the position of the light emitting unit Identifying the direction of the light emitting unit, identifying the characteristics of the driver's face, and identifying the direction of the driver's face based on the identified direction of the light emitting unit, Provide a specific method.

  According to the present invention, there is an effect that it is possible to improve the identification accuracy of the direction of the driver's face.

It is a figure for demonstrating the outline | summary of the face direction specific device which concerns on embodiment. It is a figure showing composition of a face direction specific device concerning an embodiment. It is the table | surface which showed the example of the relationship information by which the time at the time of recording, the direction of a face, and the direction of the light emission part were recorded on each address of a memory | storage part. It is a figure for demonstrating the method in which a face direction specific | specification part specifies the direction of a face. It is a figure for demonstrating the method in which a face direction specific | specification part specifies the direction of a face. It is a flowchart which shows the flow of the signal processing which the face direction identification apparatus which concerns on embodiment performs.

<Embodiment>
[Outline of Face Orientation Identification Device 1]
FIG. 1 is a diagram for explaining an outline of the face orientation identifying device 1 according to the present embodiment. The face orientation identifying device 1 identifies the orientation of the driver's face based on the captured image taken by the camera 2 installed in the driver's seat of the vehicle. Mounted on the driver's head is a mounting member A provided with a light emitting part 3 composed of a light emitting element or a light reflecting mirror. The light emitting element is, for example, an LED (Light Emitting Diode) element connected to a power source. The mounting member A is, for example, a hat with a collar or a hat-type sun visor with a collar, but may be a hat without a collar or a headband-shaped member.

  The face orientation identifying device 1 identifies the face orientation by recognizing the characteristics of the driver's face based on the captured image. The face orientation identifying device 1 is based on the orientation of the light emitting unit 3 provided on the mounting member A when the facial features cannot be identified, such as when strong light is incident or when light is insufficient. Identify the driver's face orientation.

  The direction of the light emitting unit 3 is specified by each position of one or more light emitting elements constituting the light emitting unit 3 or a mirror that reflects light. When the position of the one or more light emitting elements or the mirror that reflects light is set as the position of the light emitting unit 3, the position of the light emitting unit 3 and the direction of the light emitting unit 3 correspond one-to-one. The direction of the light emitting unit 3 and the direction of the mounting member A are the same direction.

  The face orientation specifying device 1 outputs alarm information to the outside when the orientation of the driver's face is outside a predetermined range for a predetermined time. The alarm information is notified to, for example, an ECU (Electronic Control Unit), and the ECU controls the alarm sound to sound or stops the vehicle. In FIG. 1, the face orientation specifying device 1 and the camera 2 are connected by wire, but the face orientation specifying device 1 and the camera 2 may be connected wirelessly.

[Configuration of Face Orientation Identification Device 1]
Below, the structure of the face direction specific | identification apparatus 1 which concerns on embodiment is demonstrated concretely. FIG. 2 is a diagram illustrating a configuration of the face orientation identification device 1 according to the embodiment. As shown in FIG. 2, the face orientation identification device 1 includes a control unit 11 and a storage unit 12. The storage unit 12 is a storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), and a hard disk. The storage unit 12 stores a program executed by the control unit 11 and data such as relationship information indicating the relationship between the direction of the light emitting unit 3 and the direction of the driver's face.

  The control unit 11 is, for example, a CPU (Central Processing Unit). The control unit 11 functions as an image acquisition unit 111, a feature specifying unit 112, a position specifying unit 113, and a face orientation specifying unit 114 by executing a program stored in the storage unit 12.

  The image acquisition unit 111 acquires a captured image of the driver generated by the camera 2 and transmits a signal of the captured image to the feature specifying unit 112 and the position specifying unit 113.

  The feature specifying unit 112 specifies the feature of the driver's face based on the received captured image of the driver. The feature specifying unit 112 specifies the positions of the driver's eyes and nose, for example. The number of objects specified by the feature specifying unit 112 can be arbitrarily set. The feature specifying unit 112 transmits feature information including coordinates indicating the specified eye and nose positions in the captured image to the face orientation specifying unit 114.

  The feature specifying unit 112 may not be able to completely specify the features of the driver's face, and when a captured image is generated in a state where strong external light is hitting the face or too dark, and the orientation of the face is not appropriate In some cases, the specific accuracy may be insufficient. Therefore, the feature specifying unit 112 transmits information indicating the specification accuracy of the facial features to the face orientation specifying unit 114 together with the feature information. For example, the feature specifying unit 112 transmits the ratio of the number of specified features to the number of specified features to the face orientation specifying unit 114 as information indicating accuracy.

  The position specifying unit 113 specifies the position of the light emitting unit 3 based on the received captured image, and transmits the specified position of the light emitting unit 3 to the face direction specifying unit 114. One or more light emitting elements are provided in the mounting member A worn by the driver on the head, and the position specifying unit 113 specifies the positions of the one or more light emitting elements. Even when the external light is strong and it is difficult for the feature specifying unit 112 to specify the feature of the driver's face, the position specifying unit 113 can specify the position of the light emitting unit 3.

  In order to make it easier to identify the orientation of the mounting member A based on the number and position of the light emitting elements shown in the captured image even if the position where the driver mounts the mounting member A is inappropriate, the mounting member In A, a plurality of light emitting elements are preferably provided in a wide range. At this time, in order to easily identify the orientation of the mounting member A, it is preferable that the number of light emitting elements provided on the right side of the mounting member A is different from the number of light emitting elements provided on the left side.

  When the accuracy of the facial feature identified by the feature identifying unit 112 is higher than a predetermined value, the face orientation identifying unit 114 is based on the driver's facial feature identified by the feature identifying unit 112 and the position coordinates of the facial feature. To determine the direction of the driver's face.

  In addition, when the accuracy of the facial feature specified by the feature specifying unit 112 is higher than a predetermined value, the face orientation specifying unit 114 determines the orientation of the light emitting unit 3 based on the position of the light emitting unit 3 specified by the position specifying unit 113. Is also specified, and calibration is performed by recording relation information indicating the relation between the direction of the light emitting unit 3 and the direction of the driver's face in the storage unit 12. The relationship information includes the relationship between the face direction specified by the face direction specifying unit 114 from the same captured image and the direction of the light emitting unit 3.

  The direction of the face is represented by an angle with respect to the traveling direction of the vehicle, for example. In the present embodiment, the face orientation when the driver is facing the traveling direction of the vehicle is 0 degree, and the face orientation when the driver is facing the right side is + n degrees (n is a natural number from 1 to 90). ), The orientation of the face when facing left is -n degrees.

  The direction of the light emitting unit 3 is represented by an angle with respect to the traveling direction of the vehicle, for example. The direction of the light emitting unit 3 when the light emitting unit 3 is facing the traveling direction of the vehicle is 0 degree, and the direction of the light emitting unit 3 when it is facing the right side is + n degrees (n is a natural number from 1 to 90) The direction of the light emitting unit 3 when facing the left side is -n degrees.

  Here, when the position of the mounting member A that is mounted changes unexpectedly, the driver may immediately return the direction of the mounting member A to the front. The relationship information recorded by the face orientation identifying unit 114 between the time when the driver greatly changes the position of the mounting member A and before the mounting member A is returned to the correct position is information inappropriate for identifying the face orientation. Is likely.

  Therefore, when the direction of the light emitting unit 3 with respect to the driver's face direction changes from a predetermined value, the face direction specifying unit 114 stops recording the related information, and after the predetermined time has elapsed, The recording of the related information may be resumed (or updated) on condition that the direction of the light emitting unit 3 with respect to the direction is the same. By doing so, the face orientation specifying unit 114 can reduce the probability of erroneous determination because the relationship information of the state in which the position of the mounting member A has temporarily changed is not used.

  When the accuracy of the facial feature specified by the feature specifying unit 112 is lower than a predetermined value, the face orientation specifying unit 114 is based on the direction of the light emitting unit 3 and the relationship information recorded in the storage unit 12. Identify the face orientation.

  FIG. 3 is a table showing an example of the relationship information stored in the storage unit 12. In the table shown in FIG. 3, the address is written in the first column C1, the recording time is written in the second column C2, the face orientation is written in the third column C3, and the fourth column C4 is written. Describes the direction of the light emitting unit 3. The related information is stored in a rewritable RAM in the storage unit 12. The address indicates a RAM address. The recording time is the time when the face orientation identifying unit 114 writes the face orientation and the light emitting unit 3 orientation in the storage unit 12.

  In FIG. 3, it can be seen that the driver changes the face direction over time. In FIG. 3, for example, at address 2, it is recorded that the driver is facing the front at 10:56, and the mounting member A is mounted 10 degrees from the front to the left. At address 3, the angle between the face and the mounting member A is 20 degrees, so the driver changed the position of the mounting member A between 10:56 and 10:57. I understand. In FIG. 3, the face orientation specifying unit 114 records the relationship information every minute, but the 10:55 relationship information is not recorded because the facial feature specification accuracy was insufficient. . In FIG. 3, the face orientation specifying unit 114 records the related information every minute, but the recording interval of the related information may be an arbitrary interval.

[Example of processing to specify face orientation]
Subsequently, referring to FIG. 4 and FIG. 5, when the accuracy of the result of specifying the facial features by the feature specifying unit 112 is lower than a predetermined level, the face orientation specifying unit 114 determines the driver's A method for specifying the orientation of the face will be specifically described. 4 and 5 are diagrams for explaining a method by which the face orientation identifying unit 114 identifies the face orientation. Here, the relationship information referred to by the face orientation identifying unit 114 to identify the face orientation is referred to as reference information. The reference information is related information at a time closest to the time when the face orientation identifying unit 114 identifies the face orientation.

  In FIG. 4, the relationship information when the direction of the light emitting unit 3 and the driver's face is simultaneously 0 degrees is used as the reference information (a), and the face orientation specifying unit 114 specifies the driver's face orientation. It is the figure which showed the result (b), (c), (d). In the reference information (a), since the feature specifying unit 112 can accurately specify the features of the driver's face, the specified face part is drawn on the face. Further, in the identification results (b), (c), and (d), since the feature identification unit 112 cannot accurately identify the facial features of the driver, the facial part is not drawn on the face.

  In the specific result (b), a state in which the light emitting unit 3 configured by the light emitting elements 3a to 3f faces the front is illustrated. In the reference information, since the direction of the light emitting unit 3 matches the direction of the driver's face, the face direction specifying unit 114 has the driver's face direction of 0 degrees based on the direction of the light emitting unit 3. It is specified.

  In the specific result (c), the light emitting unit 3 is turned 90 degrees from the front to the right, and the light emitting elements 3c and 3d are hidden. In the reference information, since the direction of the light emitting unit 3 matches the direction of the driver's face, the face direction specifying unit 114 has the driver's face direction of 90 degrees based on the direction of the light emitting unit 3. It is specified.

  In the specific result (d), the light emitting unit 3 changes its direction from the front to the left by 45 degrees, and the light emitting elements 3e and 3f are hidden. In the reference information, since the direction of the light emitting unit 3 matches the direction of the driver's face, the face direction specifying unit 114 determines that the driver's face direction is −45 degrees based on the direction of the light emitting unit 3. It is specified that there is.

  FIG. 5 shows the relationship information when the direction of the light emitting unit 3 is 45 degrees and the driver's face direction is 0 degree as reference information (a), and the face direction specifying unit 114 is the driver's face direction. It is the figure which showed the result (b), (c), (d) which identified this. In the reference information (a), since the feature specifying unit 112 can accurately specify the features of the driver's face, the specified face part is drawn on the face. Further, in the identification results (b), (c), and (d), since the feature identification unit 112 cannot accurately identify the facial features of the driver, the facial part is not drawn on the face.

  In the specific result (b), a state in which the light emitting unit 3 configured by the light emitting elements 3a to 3f faces the front is illustrated. In the reference information, since the direction of the light emitting unit 3 and the direction of the driver's face are deviated from each other by 45 degrees, the face direction specifying unit 114 determines that the driver's face direction is −45 based on the direction of the light emitting unit 3. The degree is specified.

  In the specific result (c), the light emitting unit 3 is turned 90 degrees from the front to the right, and the light emitting elements 3c and 3d are hidden. In the reference information, since the direction of the light emitting unit 3 and the face direction of the driver coincide with each other, the face direction specifying unit 114 is 45 degrees based on the direction of the light emitting unit 3. It is specified.

  In the specific result (d), the light emitting unit 3 changes its direction from the front to the left by 45 degrees, and the light emitting elements 3e and 3f are hidden. In the reference information, since the direction of the light emitting unit 3 and the face direction of the driver are the same, the face direction specifying unit 114 has the driver's face direction of −90 degrees based on the direction of the light emitting unit 3. It is specified.

[Operation Flowchart of Face Orientation Identification Device 1]
FIG. 6 is a flowchart showing a flow of signal processing performed by the face orientation identification device 1. First, the image acquisition unit 111 acquires a captured image from the camera 2 (S1). Subsequently, the feature identifying unit 112 determines whether or not the face orientation identifying device 1 can accurately identify the feature of the driver's face based on the acquired captured image (S2).

  The feature specifying unit 112 determines that the features of the driver's face can be specified with high accuracy (YES in S2), and the face orientation specifying unit 114 determines that a predetermined time has elapsed since the previous recording of the relationship information. In the case (YES in S3), the face orientation identifying unit 114 records the relationship between the face orientation and the light emitting unit 3 in the storage unit 12 (S4). The feature specifying unit 112 determines that the features of the driver's face can be specified with high accuracy (YES in S2), and the face orientation specifying unit 114 determines that the predetermined time has not elapsed since the previous recording of the relationship information. In the case (NO in S3), the control unit 11 stands by until a predetermined time elapses.

  When the feature specifying unit 112 determines that the driver's facial features cannot be specified with high accuracy (NO in S2), the face orientation specifying unit 114 specifies the position of the light emitting unit 3 (S5), and then the driver. The face orientation is specified (S6). If the driver's face orientation is within a predetermined normal range (YES in S7), the face orientation specifying unit 114 determines whether or not the engine of the vehicle driven by the driver is stopped (S9). If the driver's face orientation is outside the predetermined normal range (NO in S7), the face orientation specifying unit 114 outputs a warning (S8), and then whether or not the engine of the vehicle driven by the driver is stopped. Is determined (S9). If the vehicle engine is stopped (YES in S9), control unit 11 ends the process of specifying the face orientation. If the vehicle engine is not stopped (NO in S9), control unit 11 returns to step S1 and performs the processes from step S1 to step S9.

[Modification 1]
In the face orientation identifying device 1, the face orientation identifying unit 114, when the feature identifying unit 112 is able to identify a facial feature, identifies the face orientation identified from the facial features and the face orientation identified from the position of the light emitting unit 3. And the face direction may be specified. For example, the face orientation specifying unit 114 performs weighting based on the accuracy with which the feature specifying unit 112 specifies facial features.

  For example, the face orientation specifying unit 114 determines the weighting based on the number of items that can be actually detected among the items to be detected by the feature specifying unit 112. By doing so, even if the accuracy when the feature specifying unit 112 specifies facial features is not sufficiently high, the face orientation specifying unit 114 can be used by using the position of the light emitting unit 3 together. The accuracy of specifying the orientation of the can be made to a sufficient level.

[Modification 2]
In the above description, the face orientation specifying unit 114 records the related information in the storage unit 12 when the feature specifying unit 112 can specify the facial features. At this time, the face orientation specifying unit 114 has the orientation of the light emitting unit 3 when the feature specifying unit 112 can specify the facial features within a predetermined range with respect to the face orientation specified by the feature specifying unit 112. On this condition, the related information may be recorded in the storage unit 12. Further, when the direction of the light emitting unit 3 is not within a predetermined range with respect to the face direction, the face direction specifying unit 114 notifies the driver of warning information that prompts the user to mount the mounting member A correctly. May be. By doing in this way, the state where the driver is not wearing wearing member A appropriately, or removing wearing member A can be prevented from continuing for a long time.

[Effect of this embodiment]
According to the face orientation specifying device 1 described above, the face orientation can be specified based on both the facial features of the driver and the position of the light emitting unit 3. For example, when the accuracy of the result of specifying the facial feature by the feature specifying unit 112 is lower than a predetermined level, the face orientation specifying unit 114 is associated with the direction of the light emitting unit 3 specified by the position specifying unit 113 and stored in the storage unit 12. The orientation of the face is specified by referring to the orientation of the face stored in. Therefore, even if the feature specifying unit 112 cannot specify the driver's facial feature, the driver's facial feature can be specified based on the direction of the light emitting unit 3, so the face orientation specifying device 1 It is possible to specify the face direction with higher accuracy than specifying the face direction based on only one of the feature and the position of the light emitting unit 3.

  Further, the face orientation identifying device 1 records the related information in the storage unit 12 when the accuracy with which the feature identifying unit 112 identifies the facial feature satisfies a predetermined condition. Since the direction of the light emitting unit 3 relative to the direction of the face is recorded in the storage unit 12 in this way, the face direction identifying device 1 can identify the direction of the driver's face even when only the direction of the light emitting unit 3 can be identified. Can be identified.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Face direction identification apparatus 2 Camera 3 Light emission part 11 Control part 111 Image acquisition part 112 Feature specification part 113 Position specification part 114 Face direction specification part 12 Storage part A Mounting member

Claims (7)

An image acquisition unit that acquires a captured image generated by photographing a driver's face and a light emitting unit provided on a mounting member that the driver is wearing on the head;
A feature identifying unit that identifies a feature of the driver's face based on the captured image;
A position specifying unit for specifying the position of the light emitting unit based on the captured image;
The direction of the light emitting unit is specified based on the position of the light emitting unit specified by the position specifying unit, and the driver is based on the facial feature of the driver specified by the feature specifying unit and the direction of the light emitting unit. A face orientation identifying unit for identifying the face orientation of
A device for identifying face orientation, comprising: A storage unit that stores relationship information indicating a relationship between the orientation of the light emitting unit and the orientation of the face;
The face orientation specifying unit, when the accuracy of the result of specifying the facial features by the feature specifying unit is lower than a predetermined level, the face orientation stored in the storage unit in association with the direction of the light emitting unit Identify the orientation of the face by referring to
The face orientation identifying device according to claim 1. Whether the face orientation specifying unit records, in a predetermined timing, relation information indicating a relationship between the face orientation and the light emitting unit based on the facial feature specified by the feature specifying unit in the storage unit. Whether or not
The face orientation identifying device according to claim 2. The face orientation specifying unit records the relation information in the storage unit when the accuracy with which the feature specifying unit specifies the facial feature satisfies a predetermined condition.
The face orientation identifying device according to claim 3. The face orientation specifying unit is configured to store the relationship information after a predetermined time has elapsed since the relationship between the face orientation based on the facial feature specified by the feature specifying unit and the orientation of the light emitting unit has changed. Characterized by resuming recording,
The face orientation identification device according to claim 3 or 4. The face orientation specifying unit determines the weight of the face feature and the weight of the light emitting unit used for specifying the face orientation based on the accuracy with which the feature specifying unit specifies the facial feature. Characterized by the
The face orientation identifying device according to any one of claims 1 to 5. The computer runs,
Acquiring a captured image generated by photographing a driver's face and a light emitting unit provided on a mounting member worn by the driver on the head;
Identifying a driver's facial features based on the captured image;
Identifying the position of the light emitting unit based on the captured image;
The direction of the light emitting unit is specified based on the position of the light emitting unit, and the driver's face direction is specified based on the result of specifying the characteristics of the driver's face and the specified direction of the light emitting unit. And steps to
A face orientation specifying method characterized by comprising:

Images