JP7217838B2 - Mouth opening calculator - Google Patents

Description

The present disclosure relates to technology for calculating the degree of mouth opening of a person.

2. Description of the Related Art A technology for determining a driver's condition from changes in the degree of opening of the mouth of a vehicle driver is known. For example, Patent Literature 1 below discloses a technique for determining a driver's absentmindedness and arousal based on a change in the degree of opening of the driver over time.

JP 2010-241371 A

In the technique of Patent Document 1, the degree of opening is calculated by detecting the driver's mouth from the driver's face image captured by a camera. Therefore, when the driver's mouth is covered, for example, when the driver wears a mask, the camera cannot photograph the driver's mouth, so the degree of opening cannot be calculated.

SUMMARY OF THE INVENTION The present disclosure has been made to solve the above-described problems, and provides an opening degree calculation device capable of calculating the degree of opening of a person even when the person's mouth is blocked. With the goal.

The mouth opening degree calculation device according to the present disclosure includes a face part detection unit that detects each part of a person's face from a person's face image, and a specific part above the mouth of the person's face and the chin in the face image. a degree-of-mouth opening calculation unit that calculates the degree of opening, which is the degree of opening of a person's mouth , based on the distance between a current facial image comprising: a history storage unit; If the specific part of the person is not detected from the image, the degree of opening calculation unit calculates the degree of opening based on the estimated current position of the specific part of the person .

The degree of mouth opening calculation device according to the present disclosure can calculate the degree of mouth opening based on the distance between the chin and a specific part of the face above the mouth of the person. It is also possible to calculate the degree of mouth opening.

Objects, features, aspects and advantages of the present disclosure will become more apparent with the following detailed description and accompanying drawings.

1 is a diagram showing a configuration of an aperture calculation device according to Embodiment 1; FIG. 4 is a flowchart showing the operation of the mouth opening degree calculation device according to Embodiment 1; 4 is a flowchart showing opening degree calculation processing according to Embodiment 1. FIG. It is a figure which shows the hardware structural example of an opening degree calculation apparatus. It is a figure which shows the hardware structural example of an opening degree calculation apparatus. FIG. 10 is a diagram showing the configuration of an aperture calculation device according to Embodiment 2; 9 is a flowchart showing the operation of the mouth opening degree calculation device according to Embodiment 2; 10 is a flowchart showing aperture correction processing according to Embodiment 2. FIG. FIG. 10 is a diagram showing the configuration of an aperture calculation device according to Embodiment 3; 14 is a flowchart showing opening degree calculation processing in Embodiment 3. FIG. 10 is a flowchart showing aperture correction processing in Embodiment 3. FIG. It is a figure which shows the example of application to a passenger|crew state determination apparatus of an opening degree calculation apparatus. 4 is a flow chart showing the operation of a yawn determination unit; It is a flowchart which shows operation|movement of a conversation determination part.

<Embodiment 1>
FIG. 1 is a diagram showing the configuration of an aperture calculation device 10 according to Embodiment 1. As shown in FIG. As shown in FIG. 1, the mouth opening degree calculation device 10 includes a face part detection unit 11 and a mouth opening degree calculation unit 12, and is connected to a camera 20 for photographing the face of a person who is a user.

The face part detection unit 11 detects each part of the user's face from an image of the user's face captured by the camera 20 (hereinafter referred to as "face image"). The face part detection unit 11 can detect at least the mouth, the chin, and specific parts above the mouth among the parts of the face. As the specific part, for example, the eyes, nose, eyebrows, between the eyebrows, the center point between the eyes, etc. can be considered, but since the eyebrows and between the eyebrows are easily hidden by hair, the eyes, the nose, the center point between the eyes are preferable. .

The degree-of-openness calculation unit 12 calculates the degree of opening, which is the degree of opening of the user's mouth, based on the results of detection of each part of the face by the face part detection unit 11 . The degree-of-openness calculation unit 12 calculates the degree of opening from the shape of the mouth (for example, the distance between the upper and lower lips in the face image) when the facial part detection unit 11 detects the user's mouth. However, if the facial part detection unit 11 fails to detect the user's mouth, the opening degree calculation unit 12 calculates the distance between the specific part in the face image and the chin (also called "the distance from the specific part to the chin"). ) to calculate the degree of opening. Therefore, even if the mouth of the user is blocked by a mask or the like, the degree of mouth opening calculation unit 12 can calculate the degree of mouth opening of the user.

As shown in FIG. 1 , the mouth opening degree calculation unit 12 includes a reference data storage unit 121 and a reliability calculation unit 122 .

The reference data storage unit 121 is a storage medium storing reference data representing the relationship between the distance from a specific part of a person to the chin and the degree of opening. The degree-of-openness calculation unit 12 compares the distance from the specific part of the user detected by the face part detection unit 11 to the chin with the reference data stored in the reference data storage unit 121, thereby calculating the degree of opening of the user. calculate.

The reference data may be fixed data, but since there are individual differences in the relationship between the distance from the specific site to the jaw and the degree of mouth opening, it is desirable that the reference data be set for each individual user. In the present embodiment, when the facial part detection unit 11 can detect all of the user's specific parts, chin and mouth, the opening degree calculation unit 12 calculates the opening degree based on the shape of the person's mouth. , based on the detected distance from the specific part of the user to the chin and the calculated degree of opening, calculate the relationship between the distance from the specific part of the user to the chin and the degree of opening, and use the calculation result to update the reference data. As a result, the reference data becomes data suitable for the current user.

Further, when the mouth opening degree calculation device 10 has a personal authentication function (or when combined with a device having a personal authentication function), the reference data storage unit 121 may store reference data for each individual user.

The reliability calculation unit 122 calculates the reliability of the degree of mouth opening calculated by the degree of opening calculation unit 12 . An arbitrary method may be used to calculate the reliability. For example, the image quality of the face image captured by the camera 20 (degree of blown-out highlights and blocked-up shadows), the detection accuracy of facial parts by the facial part detection unit 11, and the amount of reference data. The reliability may be calculated based on the accuracy (for example, the number of updates of the reference data). Note that the reliability calculation unit 122 may be omitted when the degree of mouth opening calculation device 10 is applied to an application that does not require information on the degree of reliability of the degree of mouth opening.

Next, the operation of the degree-of-openness calculation device 10 will be described based on the flow charts of FIGS.

Referring to FIG. 2, when the mouth opening degree calculation device 10 starts operating, the face part detection unit 11 acquires the user's face image from the camera 20 (step S101), and searches for each part of the face from the acquired face image. (step S102).

Next, an opening degree calculation process is performed in which the degree of opening calculation unit 12 calculates the degree of opening of the user from the face image (step S103). FIG. 3 is a flowchart showing the opening degree calculation process.

In the degree-of-openness calculation process, first, it is confirmed whether or not the mouth is detected from the face image by the facial part detection unit 11 (step S201). If the mouth is detected (YES in step S201), the degree of opening calculation unit 12 calculates the degree of opening from the shape of the mouth (for example, the distance between the upper lip and the lower lip in the face image) (step S202). ). Further, it is confirmed whether or not the specific part and the chin are detected from the face image by the face part detection unit 11 (step S203). If the specific part and the chin have been detected (YES in step S203), the degree-of-openness calculation unit 12, based on the distance between the detected specific part of the user and the chin and the degree of opening calculated in step S203, The reference data stored in the reliability calculation unit 122 is updated (step S204), and the opening calculation process ends. Note that if the specific site and jaw are not detected (NO in step S203), step S204 is not performed.

On the other hand, if the mouth is not detected from the face image (NO in step S201), it is checked whether or not the specific part and chin have been detected by the facial part detection unit 11 (steps S205 and S206). If the specific site and the chin have been detected (YES in both steps S205 and S206), the degree-of-openness calculator 12 calculates the user's degree of mouth opening based on the distance between the specific site and the chin (step S207).

If the specific part or jaw is not detected (NO in step S205 or S206), the mouth opening degree calculation unit 12 determines that calculation of the mouth opening degree has failed (step S208). After step S208 is performed, the process returns to step S101 in FIG. In step S208, the degree-of-openness calculation unit 12 may output a notification indicating that calculation of the degree of opening has failed, or an invalid value of the degree of opening.

Returning to FIG. 2, when the degree of mouth opening calculation processing (step S103) ends, the reliability calculation unit 122 of the degree of opening calculation unit 12 calculates the reliability of the degree of opening calculated in the degree of opening calculation processing (step S104). . Then, the mouth opening degree calculation unit 12 outputs the calculation results of the mouth opening degree and reliability (step S105), and returns to step S101.

As described above, the degree of mouth opening calculation device 10 according to the first embodiment can calculate the degree of mouth opening based on the distance between the chin and a specific part (eyes, nose, etc.) above the mouth of the face. Therefore, even if the user's mouth is covered with a mask or the like, the degree of mouth opening can be calculated.

4 and 5 are diagrams each showing an example of the hardware configuration of the aperture calculation device 10. FIG. Each function of the constituent elements of the mouth opening degree calculation device 10 shown in FIG. 1 is realized by, for example, a processing circuit 50 shown in FIG. That is, the mouth opening calculation device 10 detects each part of the person's face from the face image of the person, and based on the distance between the chin and a specific part of the person's face above the mouth in the face image , a processing circuit 50 for calculating the degree of mouth opening of a person. The processing circuit 50 may be dedicated hardware, or a processor (central processing unit (CPU: Central Processing Unit), processing device, arithmetic device, microprocessor, microcomputer, etc.) that executes a program stored in a memory. Also called a DSP (Digital Signal Processor)).

When the processing circuit 50 is dedicated hardware, the processing circuit 50 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of these. Each function of the constituent elements of the aperture calculation device 10 may be implemented by individual processing circuits, or these functions may be collectively implemented by one processing circuit.

FIG. 5 shows an example of the hardware configuration of the aperture calculation device 10 when the processing circuit 50 is configured using a processor 51 that executes a program. In this case, the functions of the constituent elements of the aperture calculation device 10 are implemented by software or the like (software, firmware, or a combination of software and firmware). Software or the like is written as a program and stored in the memory 52 . The processor 51 implements the function of each part by reading and executing the program stored in the memory 52 . That is, when executed by the processor 51, the mouth opening degree calculation device 10 performs a process of detecting each part of a person's face from a person's face image, and a process of detecting each part of the person's face above the mouth in the face image. A memory 52 is provided for storing a program that results in the execution of a process of calculating the degree of opening, which is the degree of opening of a person's mouth, based on the distance between the specific part and the chin. In other words, it can be said that this program causes a computer to execute the procedures and methods of the operations of the constituent elements of the degree-of-openness calculation device 10 .

Here, the memory 52 is, for example, a non-volatile or Volatile semiconductor memory, HDD (Hard Disk Drive), magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc) and its drive device, etc., or any storage medium that will be used in the future. may

The configuration in which the functions of the components of the mouth opening degree calculation device 10 are realized by either hardware or software has been described above. However, the configuration is not limited to this, and may be a configuration in which some of the constituent elements of the opening degree calculation device 10 are realized by dedicated hardware, and other constituent elements are realized by software or the like. For example, the functions of some of the components are realized by the processing circuit 50 as dedicated hardware, and the processing circuit 50 as a processor 51 executes the programs stored in the memory 52 for some of the other components. Its function can be realized by reading and executing it.

As described above, the mouth opening degree calculation device 10 can realize each of the functions described above using hardware, software, etc., or a combination thereof.

Although FIG. 1 shows a configuration in which the camera 20 is externally attached to the degree of mouth opening calculation device 10 , the camera 20 may be built in the degree of opening calculation device 10 . Also, part of the functions of the facial part detection unit 11 and the degree of opening calculation unit 12 may be implemented on an external server that can communicate with the degree of opening calculation device 10 .

In addition, when the mouth opening degree calculation device 10 has a personal authentication function, it is also possible to use past stored information when using it together with the system.

<Embodiment 2>
Embodiment 1 is based on the premise that the positional relationship between the user's face and the camera 20 is constant. and the camera 20 may change. When the orientation of the user's face with respect to the camera 20 changes, the distance between the specific part in the face image and the chin changes. Further, when the distance between the user's face and the camera 20 changes, the size of the user's face in the face image changes, so the distance between the specific part in the face image and the chin changes. Therefore, when calculating the degree of opening from the distance between the specific part in the face image and the chin, it is preferable to correct the calculated degree of opening in consideration of the orientation and size of the user's face in the face image. . Accordingly, the second embodiment proposes an opening degree calculation device 10 that calculates the degree of opening in consideration of the orientation and size of the user's face in the face image.

FIG. 6 is a diagram showing the configuration of the degree-of-openness calculation device 10 according to the second embodiment. The configuration of the mouth opening degree calculation device 10 in FIG. 6 is obtained by adding a face orientation calculation unit 13 and a face size calculation unit 14 to the configuration in FIG. is.

The face orientation calculation unit 13 calculates the orientation of the user's face with respect to the camera 20 from the positional relationship of each part of the face in the face image captured by the camera 20 . The face size calculator 14 detects the contour of the user's face from the face image captured by the camera 20, and calculates the size of the user's face.

The openness correction unit 123 of the openness calculation unit 12 corrects the opening degree calculation unit 12 based on the orientation of the user's face calculated by the face orientation calculation unit 13 and the size of the user's face calculated by the face size calculation unit 14. The calculated degree of opening of the user is corrected.

For example, it is assumed that the normal state is a state in which the camera 20 is positioned slightly above the front of the user's face and at a position 50 cm from the face. In this case, when the user looks down, the distance between the specific part in the facial image and the chin becomes smaller than in the normal state, and the degree of opening calculated by the degree-of-openness calculation unit 12 becomes smaller than in the normal state. The degree-of-openness correction unit 123 performs correction to increase the degree of opening calculated by the degree-of-openness calculation unit 12 . Further, when the user brings the face closer to the camera 20 and the distance between the face and the camera 20 becomes smaller than 50 cm, the size of the face in the face image becomes larger than the normal state, and the opening degree calculation unit 12 calculates the size of the face. Since the degree of opening is larger than that in the normal state, the degree of opening correcting section 123 performs correction to reduce the degree of opening calculated by the degree of opening calculating section 12 . The opening degree calculator 12 outputs the corrected opening degree. As a result, the user's mouth opening degree output from the mouth opening degree calculation unit 12 is a value that takes into consideration the orientation and size of the user's face in the face image.

The operation of the reliability calculation unit 122 of the degree of mouth opening calculation unit 12 may be the same as that in the first embodiment. degrees can be calculated.

FIG. 7 is a flow chart showing the operation of the degree-of-openness calculation device 10 according to the second embodiment. The flow of FIG. 7 is obtained by adding an opening degree correction process (step S103a) for correcting the opening degree calculated in step S103 after the opening degree calculation process (step S103) of the flow of FIG. Since the other steps are the same as those in FIG. 2, only the aperture correction process will be described here.

FIG. 8 is a flowchart of the aperture correction process. When the degree of opening correction process is started, it is confirmed whether or not the face orientation calculation unit 13 has calculated the orientation of the user's face, and whether or not the face size calculation unit 14 has calculated the size of the user's face. (steps S301 and S302). If the orientation and size of the user's face have been calculated (YES in both steps S301 and S302), the degree-of-openness correction unit 123 performs the degree-of-openness calculation processing (step The degree of opening calculated in S103) is corrected (step S303), and the degree of opening correcting process ends. If either the orientation or size of the user's face has not been calculated (NO in step S301 or S302), the degree of mouth opening calculation unit 12 determines that the degree of mouth opening has been calculated unsuccessfully (step S304). After step S304 is performed, the process returns to step S101 in FIG. In step S304, the degree-of-openness calculation unit 12 may output a notification indicating that calculation of the degree of opening has failed, or an invalid value of the degree of opening.

As described above, since the degree of mouth opening calculation device 10 according to Embodiment 2 can calculate the degree of opening taking into consideration the orientation and size of the user's face in the face image, the position of the user's face and the camera 20 is calculated. Even if the relationship is not constant, it is possible to calculate the degree of opening with high accuracy.

<Embodiment 3>
In the mouth opening calculation device 10 of Embodiments 1 and 2, when the user's specific part and chin cannot be detected from the face image, or when the orientation and size of the face cannot be calculated from the face image, the mouth opening calculation is performed. was judged to have failed. Embodiment 3 proposes an opening degree calculation device 10 capable of estimating the position of a specific part of the user, the position of the chin, the direction and size of the face, and calculating the degree of opening using the estimation results in such a case. do.

FIG. 9 is a diagram showing the configuration of the degree-of-openness calculation device 10 according to the third embodiment. 9 is different from the configuration shown in FIG. , a history storage unit 141 and an estimation unit 142 are provided in the face size calculation unit 14, respectively.

The history storage unit 111 and the estimation unit 112 of the face part detection unit 11 will be described. The history storage unit 111 stores a history of positions of specific parts of the user detected in past frames of the face image and a history of positions of the user's chin detected in past frames of the face image. When the specific part of the user is not detected from the current face image, the estimating unit 112 obtains the trajectory of the position of the specific part of the user from the history of the positions of the specific part of the user stored in the history storage unit 111, Based on the trajectory, the current position of the specific part of the user is estimated. In this case, the mouth opening degree calculation unit 12 calculates the mouth opening degree based on the estimated position of the current specific part of the user. In addition, when the user's jaw is not detected from the current face image, the estimation unit 112 obtains the trajectory of the user's jaw position from the history of the user's jaw position stored in the history storage unit 111. Estimate the current user jaw position based on the trajectory. In this case, the mouth opening degree calculation unit 12 calculates the mouth opening degree based on the estimated current position of the user's jaw. The history of the position of the specific part and the history of the position of the jaw stored in the history storage unit 111 may include the position of the specific part and the position of the jaw estimated by the estimation unit 112 in the past.

The history storage unit 131 and the estimation unit 132 of the face orientation calculation unit 13 will be described. The history storage unit 131 stores a history of the orientation of the user's face detected in past frames of the face image. When the user's face orientation cannot be calculated from the current face image, the estimation unit 132 obtains the trajectory of the user's face orientation from the history of the user's face orientation stored in the history storage unit 131, and calculates the trajectory of the user's face orientation. Estimate the current user's face orientation based on the trajectory. In this case, the degree-of-openness correction unit 123 of the degree-of-openness calculation unit 12 corrects the degree of opening based on the estimated current orientation of the user's face. Note that the face orientation history stored in the history storage unit 131 may include the face orientation estimated by the estimation unit 132 in the past.

The history storage unit 141 and the estimation unit 142 of the face size calculation unit 14 will be described. The history storage unit 141 stores a history of the size of the user's face detected in past frames of the face image. When the size of the user's face cannot be calculated from the current face image, the estimation unit 142 obtains the trajectory of the size of the user's face from the history of the user's face size stored in the history storage unit 141. Estimate the size of the current user's face based on the trajectory. In this case, the degree of mouth opening correction unit 123 of the degree of opening calculation unit 12 corrects the degree of opening based on the estimated size of the current face of the user. Note that the face size history stored in the history storage unit 141 may include the face size previously estimated by the estimation unit 142 .

The basic operation of the degree-of-openness calculation device 10 according to the third embodiment is the same as the flow of FIG. ) is different from that in the second embodiment. Therefore, the degree-of-opening calculation processing and the degree-of-opening correction processing according to the third embodiment will be described here.

FIG. 10 is a flowchart showing opening degree calculation processing according to the third embodiment. When the degree-of-openness calculation process is started, it is checked whether or not the mouth is detected from the face image by the facial part detection unit 11 (step S201). If the mouth is detected (YES in step S201), the processing of steps 201 to 204 similar to that of FIG. 3 is executed. Since the processes of steps 201 to 204 are the same as those described in the first embodiment, the description is omitted here.

If the mouth is not detected from the face image (NO in step S201), it is checked whether or not the specific part and chin have been detected by the facial part detection unit 11 (steps S205 and S206). If the specific site and the chin have been detected (YES in both steps S205 and S206), the degree-of-openness calculator 12 calculates the user's degree of mouth opening based on the distance between the specific site and the chin (step S207).

However, if the specific part is not detected (NO in step S205), the estimation unit 112 of the facial part detection unit 11 refers to the history of the position of the specific part stored in the history storage unit 111, and from the history, It is determined whether or not the current position of the specific part can be estimated (specifically, whether or not the trajectory of the position of the specific part can be obtained from the history) (step S205a). If the current position of the specific part can be estimated (YES in step S205a), the estimation unit 112 estimates the current position of the specific part (step S205b).

If the chin is not detected (NO in step S206), the estimation unit 112 of the facial part detection unit 11 refers to the history of the chin position stored in the history storage unit 111, It is determined whether or not the position of the jaw can be estimated (specifically, whether or not the trajectory of the position of the jaw can be obtained from the history) (step S206a). If the current jaw position can be estimated (YES in step S206a), the estimation unit 112 estimates the current jaw position (step S206b).

When the position of the specific part is estimated (step S205b) or the jaw position is estimated (step S206a), in step S207, the degree of opening calculation unit 12 uses the estimation result to calculate the degree of opening of the user. calculate.

If it is determined in step S205a or S206a that the current position of the specific part or jaw cannot be estimated, the degree of mouth opening calculation unit 12 determines that calculation of the degree of mouth opening has failed (step S208). After step S208 is performed, the process returns to step S101 in FIG.

FIG. 11 is a flowchart of aperture correction processing according to the third embodiment. When the degree of opening correction process is started, it is confirmed whether or not the face orientation calculation unit 13 has calculated the orientation of the user's face, and whether or not the face size calculation unit 14 has calculated the size of the user's face. (steps S301 and S302). If the direction and size of the user's face have been calculated (YES in both steps S301 and S302), the degree-of-openness correction unit 123 performs the degree-of-openness calculation processing (step The degree of opening calculated in S103) is corrected (step S303), and the degree of opening correcting process ends.

However, if the user's face orientation has not been calculated (NO in step S301), the estimation unit 132 of the face orientation calculation unit 13 refers to the history of the user's face orientation stored in the history storage unit 131. , it is determined whether or not the current face orientation can be estimated from the history (specifically, whether or not the trajectory of the face orientation can be obtained from the history) (step S301a). If the current face orientation can be estimated (YES in step S301a), the estimation unit 132 estimates the current face orientation (step S301b).

If the user's face size has not been calculated (NO in step S302), the estimation unit 142 of the face size calculation unit 14 refers to the history of the user's face size stored in the history storage unit 141. , whether or not the current face size can be estimated from the history (specifically, whether or not the locus of the face size can be obtained from the history) is determined (step S302a). If the current face size can be estimated (YES in step S302a), the estimation unit 142 estimates the current face size (step S302b).

If it is determined in step S301a or S302a that the current face orientation and size cannot be estimated, the degree of mouth opening calculation unit 12 determines that the degree of mouth opening calculation has failed (step S304). After step S304 is performed, the process returns to step S101 in FIG.

As described above, the mouth opening degree calculation device 10 according to the third embodiment can detect the user's specific part and chin from the face image, or even when the face orientation and size cannot be calculated from the face image. By estimating them, the degree of opening can be calculated.

In Embodiment 3, the reliability calculation unit 122 uses the information (the position of the specific part, the position of the chin, the orientation of the face, the size of the face) used by the mouth opening calculation unit 12 to calculate the degree of opening, It may be determined that the more items that are estimated (that is, the more items that could not be detected or calculated), the lower the reliability of the degree of mouth opening.

<Embodiment 4>
Embodiment 4 shows an example of a specific application to which the degree of opening calculation device 10 is applied. FIG. 12 is a diagram showing an example in which the degree of opening calculation device 10 is applied to the occupant state determination device 30. As shown in FIG. The opening degree calculation device 10 shown in FIG. 12 may be according to any one of the first to third embodiments.

The occupant state determination device 30 is an in-vehicle device that determines the state of a vehicle occupant (including a driver). and a conversation determination unit 32 for determining whether or not the user is present. For example, the occupant state determination device 30 determines that a yawning occupant has a low wakefulness level, or that a conversational occupant has a high alertness level. Note that the occupant state determination device 30 may include only one of the yawn determination section 31 and the conversation determination section 32 .

In the fourth embodiment, the user of the degree of opening calculation device 10 is an occupant of a vehicle in which the occupant state determination device 30 is mounted. That is, the camera 20 captures an image of the passenger's face, and the degree of opening calculation device 10 calculates the degree of opening of the passenger and its reliability. In addition, the output of the degree-of-openness calculation device 10 , that is, the degree of opening of the occupant and its reliability is provided to the yawning determination section 31 and the conversation determination section 32 of the occupant state determination device 30 .

FIG. 13 is a flow chart showing the operation of the yawn determination unit 31. As shown in FIG. The operation of the yawn determination unit 31 will be described below with reference to FIG.

The yawn determination unit 31 acquires the occupant's degree of opening calculated by the degree of opening calculation device 10 and its reliability (step S401). Then, the yawn determination unit 31 confirms whether or not the state in which the opening degree of the occupant is equal to or greater than a predetermined threshold has continued for a certain period of time or longer (step S402).

If the degree of opening of the occupant has remained equal to or greater than the threshold for a certain period of time or longer (YES in step S402), the yawning determination unit 31 further determines whether the reliability of the degree of opening is equal to or greater than a predetermined threshold. Confirm (step S403). At this time, if the reliability of the degree of opening is equal to or higher than the threshold (YES in step S403), the yawning determination unit 31 determines that the occupant is yawning (step S404). However, if the reliability of the degree of mouth opening is lower than the threshold (NO in step S403), the yawning determination unit 31 determines that the yawning determination has failed (step S405). In step S405, the yawn determination unit 31 may output a notification indicating that yawn determination has failed, or an invalid value for yawn determination.

On the other hand, if the degree of opening of the occupant has not continued for a certain period of time or longer (NO in step S402), the yawning determination unit 31 further determines whether the reliability of the degree of opening is equal to or greater than the predetermined threshold. Confirm whether or not (step S406). At this time, if the reliability of the degree of opening is greater than or equal to the threshold (YES in step S406), the yawning determination unit 31 determines that the occupant is not yawning (step S407). However, if the reliability of the degree of mouth opening is lower than the threshold (NO in step S406), the yawn determination unit 31 determines that the yawn determination has failed (step S408). Also in step S408, the yawn determination unit 31 may output a notification indicating that yawn determination has failed, or an invalid value for yawn determination.

The yawn determination unit 31 repeats the above operations. Note that the reliability threshold used in step S403 and the reliability threshold used in step S406 may be the same value or different values.

FIG. 14 is a flow chart showing the operation of the conversation determining section 32. As shown in FIG. The operation of the conversation determination unit 32 will be described below with reference to FIG.

The conversation determination unit 32 acquires the occupant's degree of opening calculated by the degree of opening calculation device 10 and its reliability (step S501). Then, the conversation determination unit 32 confirms whether or not the opening degree of the passenger is periodically changing (step S502).

If the occupant's openness is periodically changing (YES in step S502), the conversation determination unit 32 further checks whether the reliability of the openness is equal to or greater than a predetermined threshold (step S503). . At this time, if the reliability of the degree of opening is equal to or higher than the threshold (YES in step S503), the conversation determination unit 32 determines that the passenger is having a conversation (step S504). However, if the reliability of the degree of opening is lower than the threshold (NO in step S503), the conversation determining unit 32 determines that the conversation has failed (step S505). In step S505, the conversation determination unit 32 may output a notification indicating that the conversation determination has failed, or an invalid value for the conversation determination.

On the other hand, even if the occupant's openness does not change periodically (NO in step S502), the conversation determination unit 32 further checks whether the reliability of the openness is equal to or higher than a predetermined threshold value ( step S506). At this time, if the reliability of the degree of opening is equal to or greater than the threshold (YES in step S506), the conversation determining unit 32 determines that the passenger is not speaking (step S507). However, if the reliability of the degree of opening is lower than the threshold (NO in step S506), the conversation determination unit 32 determines that the conversation has failed (step S508). Also in step S508, the conversation determination unit 32 may output a notification indicating that the conversation determination has failed, or an invalid value for the conversation determination.

The conversation determination unit 32 repeats the above operations. Note that the reliability threshold used in step S503 and the reliability threshold used in step S506 may be the same value or different values.

Since the degree of opening calculation device 10 can calculate the degree of opening even if the mouth of the occupant is blocked by a mask or the like, the occupant state determination device 30 can , it is possible to determine the state of the occupant.

In addition, it is possible to combine each embodiment freely, and to modify|transform and abbreviate|omit each embodiment suitably.

It is to be understood that the above description is illustrative in all aspects and that countless variations not illustrated can be envisaged.

10 opening degree calculation device 11 face part detection unit 12 opening degree calculation unit 13 face orientation calculation unit 14 face size calculation unit 20 camera 121 reference data storage unit 122 reliability calculation unit 123 opening degree correction unit , 111 history storage unit, 112 estimation unit, 131 history storage unit, 132 estimation unit, 141 history storage unit, 142 estimation unit, 30 occupant state determination device, 31 yawn determination unit, 32 conversation determination unit, 50 processing circuit, 51 processor , 52 memory.

Claims (11)

a face part detection unit that detects each part of the face of the person from the face image of the person;
an opening degree calculation unit that calculates an opening degree, which is the degree of opening of the mouth of the person, based on the distance between the chin and a specific part of the face above the mouth of the person in the face image;
with
The facial part detection unit includes:
a history storage unit that stores a history of positions of specific parts of the person in the face image;
an estimating unit for estimating the current position of the specific part of the person from a history of positions of the specific part of the person when the specific part of the person is not detected from the current face image;
with
When the specific part of the person is not detected from the current face image, the degree of opening calculation unit calculates the degree of opening based on the estimated position of the current specific part of the person.
Mouth opening calculation device. wherein the specific site is the eye, nose, or midpoint between the eyes;
The mouth opening degree calculation device according to claim 1 . The facial part detection unit includes:
a history storage unit that stores a history of the position of the chin of the person in the face image;
an estimating unit for estimating the current chin position of the person from the history of the chin positions of the person when the chin of the person is not detected from the current face image;
with
If the chin of the person is not detected from the current face image, the degree of opening calculation unit calculates the degree of opening based on the estimated current position of the chin of the person.
The mouth opening degree calculation device according to claim 1 . further comprising a face orientation calculation unit that calculates the orientation of the face of the person from the face image,
The degree of opening calculation unit calculates the degree of opening by taking into account the orientation of the face of the person.
The mouth opening degree calculation device according to claim 1 . The face orientation calculation unit
a history storage unit that stores a history of the orientation of the person's face in the face image;
an estimation unit that estimates the current face orientation of the person from the history of the face orientation of the person when the face orientation of the person cannot be calculated from the current face image;
with
When the face orientation of the person cannot be calculated from the current face image, the degree of opening calculation unit calculates the degree of opening by adding the estimated current face orientation of the person.
The opening degree calculation device according to claim 4 . further comprising a face size calculation unit that calculates the size of the person's face in the face image,
The degree of opening calculation unit calculates the degree of opening by taking into account the size of the person's face in the face image.
The mouth opening degree calculation device according to claim 1 . The face size calculator,
a history storage unit that stores a history of the size of the person's face in the face image;
an estimation unit that estimates the current face size of the person from a history of the face size of the person when the size of the face of the person cannot be calculated from the current face image;
with
When the size of the face of the person cannot be calculated from the current face image, the degree of opening calculation unit calculates the degree of opening by adding the estimated current face size of the person.
The opening degree calculation device according to claim 6 . When the mouth of the person is detected from the face image, the degree of opening calculation unit calculates the degree of opening based on the shape of the mouth of the person.
The mouth opening degree calculation device according to claim 1 . The opening degree calculation unit stores reference data representing a relationship between a distance between a specific part of the person and the chin and the degree of opening, and calculates the distance between the specific part and the chin of the person detected from the face image. calculating the degree of opening by comparing the distance between and the reference data;
The mouth opening degree calculation device according to claim 1 . When a specific part, chin, and mouth of the person are detected from the face image, the opening degree calculation unit calculates the opening degree based on the shape of the mouth of the person, and specifies the detected person. updating the reference data using the distance between the part and the jaw and the calculated openness;
The opening degree calculation device according to claim 9 . The mouth opening degree calculation unit further calculates a reliability of the calculated mouth opening degree,
The mouth opening degree calculation device according to claim 1 .

Images