JP7458216B2 - Image processing device, computer program, and abnormality estimation system - Google Patents

Description

The present invention relates to an image processing device that processes an image in which a driver of a moving object is included. The present invention also relates to a computer program executable by the processing unit of the image processing device. The present invention also relates to an abnormality estimation system that estimates whether the driver's posture is abnormal based on the image.

Patent Document 1 discloses a technique for detecting an abnormality in the posture of a driver of a vehicle, which is an example of a moving object. The abnormality in posture is estimated based on an image in which the driver is captured by an imaging device installed in the vehicle.

JP 2019-105872 Publication

An object of the present invention is to increase the certainty of estimating an abnormal posture of a driver of a moving object based on images acquired by an imaging device.

One aspect of achieving the above object is an image processing device, comprising:
a reception unit that repeatedly receives image information corresponding to images in which a driver of a mobile object is captured;
a processing unit that estimates whether the posture of the driver is abnormal based on the image information;
It is equipped with
Each time the processing unit receives the image information,
determining the possibility that the posture of the driver reflected in the image corresponds to each of a plurality of abnormal postures;
assigning a score corresponding to the possibility to each of the plurality of abnormal postures;
If the representative value of the score given to at least one of the plurality of abnormal postures exceeds a threshold value, it is estimated that the driver is in the at least one abnormal posture.

One aspect of achieving the above object is a computer program executable by a processing unit of an image processing device, comprising:
By being executed, the image processing device
Repeatedly accepts image information corresponding to images in which the driver of a moving object is captured,
Each time the image information is received,
determining the possibility that the posture of the driver reflected in the image corresponds to each of a plurality of abnormal postures;
assigning a score corresponding to the possibility to each of the plurality of abnormal postures;
When the representative value of the score given to at least one of the plurality of abnormal postures exceeds a threshold value, it is estimated that the driver is taking the at least one abnormal posture.

One aspect of achieving the above objective is an abnormality estimation system,
an imaging device that repeatedly outputs image information corresponding to an image in which a driver of a moving object is captured;
an image processing device that estimates whether the driver's posture is abnormal based on the image information;
a control device that controls the operation of a controlled device mounted on the moving body based on a result of estimation by the image processing device that the posture is abnormal;
It is equipped with
Each time the image processing device receives the image information,
determining the possibility that the posture of the driver reflected in the image corresponds to each of a plurality of abnormal postures;
assigning a score corresponding to the possibility to each of the plurality of abnormal postures;
If the representative value of the score given to at least one of the plurality of abnormal postures exceeds a threshold value, it is estimated that the driver is in the at least one abnormal posture.

Generally, the driver's transition from a normal posture to an abnormal posture progresses in a complex manner, including the various abnormal posture elements described above. A specific abnormal posture that was dominant at the time of acquiring certain image information may no longer be dominant at the time of acquiring the next image information, and another abnormal posture may become dominant.

According to the configurations according to each of the above aspects, the possibility that the driver is taking each of the plurality of prescribed abnormal postures is determined each time image information is received, and a score based on the possibility is assigned. Since the abnormal posture is estimated based on whether the score exceeds the threshold, it is possible to grasp the possibility that the driver is taking multiple abnormal postures, and ultimately determine whether the driver is taking at least one abnormal posture. Three abnormal postures can be estimated. Therefore, it is possible to increase the reliability of estimation of the driver's posture abnormality based on the image acquired by the imaging device.

1 illustrates a functional configuration of an abnormality estimation system according to an embodiment. 2 illustrates an example of a vehicle in which the abnormality estimation system of FIG. 1 can be installed. 2 illustrates an example of the flow of processing executed by the image processing apparatus of FIG. 1. FIG. 2 illustrates an example of an image that may be acquired by the imaging device of FIG. 1; 5 illustrates an example of a state in which a skeleton model is applied to the image of FIG. 4. An example of the flow of the abnormality estimation process in FIG. 3 is shown. An example of the flow of the posture estimation process in FIG. 3 is shown. 4 illustrates a state in which a skeletal model is applied to the image in FIG. 4. 1 illustrates a method for determining the head rotation angle in the vehicle yaw direction. Another example of the flow of the posture estimation process in FIG. 3 is shown. It is a figure for explaining a "shrimp warped posture." FIG. 3 is a diagram for explaining a "prone posture." 1 illustrates a method for determining the head rotation angle in the vehicle pitch direction. 4 illustrates another example of the flow of the attitude estimation process in FIG. 3 . FIG. 6 is a diagram for explaining a "head-only tilted posture" to the left. FIG. 13 is a diagram for explaining a "head-only sideways posture" to the right. A method for identifying the rotation angle of the head in the roll direction of the vehicle is illustrated. An example of the flow of the posture estimation process in FIG. 3 is shown. 19 is a diagram for explaining details of the posture estimation process in FIG. 18. FIG. 19 is a diagram for explaining details of the posture estimation process in FIG. 18. FIG. FIG. 13 is a diagram for explaining the "leaning sideways position" to the left. FIG. 3 is a diagram for explaining a "side-leaning posture" toward the right. FIG. 13 is a diagram for explaining a "sideways posture" to the left. FIG. 3 is a diagram for explaining a "sideways falling posture" to the right. Another example of the flow of the posture estimation process in FIG. 3 is shown. It is a diagram for explaining a "prone posture." An example of the flow of the posture estimation process in FIG. 3 is shown. FIG. 28 is a diagram for explaining details of the posture estimation process in FIG. 27 . 28 is a diagram for explaining details of the posture estimation process in FIG. 27. FIG. FIG. 3 is a diagram for explaining a "backward posture".

Examples of embodiments will now be described in detail with reference to the accompanying drawings. FIG. 1 illustrates a functional configuration of an abnormality estimation system 10 according to an embodiment. The abnormality estimation system 10 is a system that estimates whether the posture of the driver 30 is abnormal based on an image in which the driver 30 of the vehicle 20 illustrated in FIG. 2 is reflected. Vehicle 20 is an example of a moving object.

The abnormality in posture is estimated to detect an abnormality in the driver 30. As used in this specification, the term "abnormality in the driver" refers to a sudden change in physical condition that is difficult to predict in advance.

In the attached drawings, arrow F represents the forward direction as seen by the driver 30. Arrow B represents the rearward direction as seen by the driver 30. Arrow L represents the leftward direction as seen by the driver 30. Arrow R represents the rightward direction as seen by the driver 30. Arrow U represents the upward direction as seen by the driver 30. Arrow D represents the downward direction as seen by the driver 30.

As illustrated in FIG. 1, the abnormality estimation system 10 includes an imaging device 11. The imaging device 11 is placed at an appropriate location in the vehicle 20. A driver 30 seated on a seat 22 arranged in a cabin 21 of a vehicle 20 illustrated in FIG. 2 is imaged by an imaging device 11.

As illustrated in FIG. 1, the imaging device 11 is configured to output image information I corresponding to an acquired image. The image information I may be in the form of analog data or digital data.

The abnormality estimation system 10 includes an image processing device 12. The image processing device 12 includes a reception section 121 and a processing section 122. The reception unit 121 is configured to receive image information I from the imaging device 11. When the image information I is in the form of analog data, the reception unit 121 may include an appropriate conversion circuit including an A/D converter. The processing unit 122 processes image information I in the form of digital data.

The processing unit 122 is configured to perform a process of estimating whether the posture of the driver 30 is abnormal based on the image information I. Details of this processing will be described later.

The image processing device 12 includes an output unit 123. The processing unit 122 is configured to output control information C through the output unit 123 when the posture of the driver 30 is estimated to be abnormal. The control information C may be in the form of digital data or analog data. When the control information C is in the form of analog data, the output unit 123 may include an appropriate conversion circuit including a D/A converter.

The abnormality estimation system 10 includes a control device 13. The control device 13 is mounted on the vehicle 20. The control device 13 is configured to control the operation of the controlled device 40 mounted on the vehicle 20 based on the control information C output from the image processing device 12.

Specifically, the control device 13 is configured to activate driving assistance for the vehicle 20 when the driver 30 is estimated to be in an abnormal position. The term "driving assistance" as used in this specification refers to a control process that at least partially performs at least one of the following: driving operations (steering, acceleration, deceleration, etc.), monitoring of the driving environment, and backup of driving operations. In other words, it includes everything from partial driving assistance such as a collision mitigation brake function and a lane keep assist function to fully automated driving operations.

For example, the control device 13 causes the controlled device 40 to perform operations necessary to decelerate the vehicle 20 and stop the vehicle 20 on the shoulder of the road, based on the control information C output from the image processing device 12. Examples of the controlled device 40 include devices that constitute the drive system of the vehicle 20, lamps, and devices that notify occupants of the vehicle 20, other vehicles, and pedestrians that the driving support operation is effective.

Next, the process of estimating the posture of the driver 30 executed by the processing unit 122 of the image processing device 12 will be described in detail with reference to FIGS. 3 to 5. FIG. 3 illustrates the flow of the process.

The processing unit 122 receives image information I from the imaging device 11 through the reception unit 121 (STEP 1). FIG. 4 illustrates an image IM in which the driver 30 is captured, which is acquired by the imaging device 11. Image information I corresponds to image IM.

Subsequently, the processing unit 122 performs a process of applying the skeleton model (STEP 2 in FIG. 3). The term "processing that applies a skeletal model" as used in this specification refers to detecting a plurality of feature points defined in the skeletal model on a driver captured in an image obtained by an imaging device, and This means connecting feature points with each other using a plurality of skeletal lines defined in the skeletal model.

FIG. 5 shows an example in which the skeletal model M is applied to the driver 30 reflected in the image IM acquired by the imaging device 11. In this example, the skeletal model M includes a head feature point H, a neck feature point NK, a left shoulder feature point LS, and a right shoulder feature point RS.

The head feature point H is a point corresponding to the center of the head of the model human body. The neck feature point NK is a point corresponding to the neck of the model human body. The left shoulder feature point LS is a point corresponding to the left shoulder of the model human body. The right shoulder feature point RS is a point corresponding to the right shoulder of the model human body. The head feature point H and neck feature point NK are connected by a skeleton line. The neck feature point NK is connected to each of the left shoulder feature point LS and the right shoulder feature point RS by a skeleton line.

The processing unit 122 detects points corresponding to the head characteristic point H, neck characteristic point NK, left shoulder characteristic point LS, and right shoulder characteristic point RS on the driver 30 captured in the image IM, and connects the detected points with the above-mentioned skeleton lines. The algorithm for carrying out this process is well known, so a detailed description will be omitted.

As illustrated in FIG. 1, the image processing device 12 includes a storage unit 124. The processing unit 122 stores the position of each feature point in the image IM in the storage unit 124 as representing the posture of the driver 30 in a normal state. Specifically, the position of the pixel in the image IM corresponding to each feature point is stored in the storage unit 124.

Subsequently, the processing unit 122 determines whether the posture of the driver 30 is normal (STEP 3). For example, the determination can be made based on whether each feature point detected in the image IM by applying the skeleton model M is located within a defined range.

When it is determined that the posture of the driver 30 is not normal (NO in STEP 3), the processing unit 122 executes posture estimation processing (STEP 4). The posture estimation process is a process of estimating whether the driver 30 is taking each of a plurality of predefined abnormal postures. Details of the posture estimation process will be described later.

Subsequently, the processing unit 122 performs a process of adding a score for at least one of the plurality of abnormal postures estimated to be taken by the driver 30 as a result of the posture estimation process (STEP 5). The score is a numerical value corresponding to the possibility that the posture of the driver 30 corresponds to each of a plurality of abnormal postures.

Subsequently, the processing unit 122 executes abnormality estimation processing (STEP 7). FIG. 6 illustrates the flow of processing executed in the abnormality estimation processing.

As a result of the score addition process, the processing unit 122 determines whether there is any posture whose score exceeds the threshold value among the plurality of abnormal postures (STEP 71).

If it is determined that there is no abnormal posture with a score exceeding the threshold (NO in STEP 71), the processing unit 122 ends the abnormality estimation process. The process returns to STEP 1 in FIG. 3, and the next image information I is accepted. The cycle at which the image information I is repeatedly accepted may correspond to the frame rate of the imaging device 11.

Further, when it is determined that the posture of the driver 30 is normal (YES in STEP 3), the processing unit 122 performs a process of subtracting the score assigned to each abnormal posture (STEP 6). The score may be reset to 0 or may be subtracted by a predetermined value. Thereafter, the process returns to STEP 1 in FIG. 3, and the next image information I is accepted.

In the abnormality estimation process, if it is determined that the score for at least one of the multiple abnormal postures exceeds the threshold value as a result of the score addition process (YES in STEP 71 of FIG. 6), the processing unit 122 estimates that the driver 30 is in at least one of the abnormal postures (STEP 72).

Subsequently, the processing unit 122 outputs the control information C from the output unit 123 (STEP 73). Control information C is transmitted to the control device 13. The control information C may cause the controlled device 40 to perform the same operation, or may cause the controlled device 40 to perform different operations depending on the type of the estimated abnormal posture.

Next, an example of the posture estimation process (STEP 4 in FIG. 3) executed by the processing unit 122 will be described with reference to FIGS. 7 to 9. FIG. 7 shows an example of the flow of processing.

In this example, it is estimated whether the driver 30 is "looking away". The term "looking away" as used herein refers to a state in which the driver 30 continues to not look in the direction in which the vehicle 20 is traveling.

In this example, when the rotation angle of the head 31 of the driver 30 in the yaw direction of the vehicle 20 that is reflected in the image IM acquired by the imaging device 11 exceeds a threshold value, the processing unit 122 It is determined that there is a possibility that the person is ``looking away''.

The term "yaw direction" as used in this specification means the direction of rotation around an axis extending in the vertical direction of the vehicle 20. With regard to the "yaw direction," the counterclockwise direction as viewed from above the head of the driver 30 is defined as the "left yaw direction," and the clockwise direction as viewed from above the head of the driver 30 is defined as the "right yaw direction."

In order to detect the rotation angle of the head 31 in the yaw direction of the vehicle 20, the processing unit 122 assumes the center coordinates of the head 31 of the driver 30 reflected in the image IM acquired by the imaging device 11. Processing is performed (STEP 11). Details of the processing will be described with reference to FIGS. 8 and 9.

Figure 8 shows an enlarged view of the portion of image IM illustrated in Figures 4 and 5 in which the head 31 of the driver 30 is captured. In this example, the process of applying skeletal model M can detect the left eye feature point LY in addition to the head feature point H. The left eye feature point LY is a point that corresponds to the left eye of the model human body. The process of applying skeletal model M also sets a rectangular frame FM that estimates the area in image IM in which the head 31 of the driver 30 is located.

Figure 9 is a diagram for explaining a method for assuming the center coordinate CT of the head 31 using the set frame FM. In this figure, the head 31 is shown as viewed from above. In this example, the width of the head 31 in the left-right direction of the vehicle 20 and the width of the head 31 in the front-rear direction of the vehicle 20 are considered to be the same as the width W of the frame FM set in the image IM shown in Figure 8. Under this premise, the center coordinate CT of the head 31 is assumed to be a point (W/2) away from the head feature point H toward the rear of the vehicle 20. The processing unit 122 stores the number of pixels corresponding to the distance (W/2) in the memory unit 124.

At this time, the angle θY0 that the straight line connecting the center coordinate CT and the left eye feature point LY makes with the straight line connecting the center coordinate CT and the head feature point H is the angle θY0 between the head feature point H and the left eye feature point LY in the left-right direction of the vehicle 20. It can be specified by determining the distance dHL between the two and the arctangent of (W/2) mentioned above. This angle θY0 is stored in the storage unit 124 as an initial angle when the driver 30 is in a normal posture (looking in the direction of travel of the vehicle 20).

When the driver 30 takes the "looking away" position, the head 31 rotates in the yaw direction of the vehicle 20. At this time, the angle θY1 that the straight line connecting the center coordinate CT and the left eye feature point LY makes with the straight line connecting the center coordinate CT and the head feature point H (in the longitudinal direction of the vehicle 20) changes from the initial angle θY0. The amount of change (θY1−θY0) can be regarded as the rotation angle θY of the head 31 in the yaw direction of the vehicle 20. Therefore, when the absolute value of the amount of change (θY1−θY0) exceeds the threshold value, it can be estimated that the driver 30 is in a “looking away posture”. The threshold value can be statistically determined as the rotation angle θY that may occur when the general driver 30 takes a “looking away posture”.

Based on the above principle, the processing unit 122 performs a process of specifying the rotation angle θY of the head 31 in the yaw direction of the vehicle 20 using the center coordinate CT specified in STEP 11 (STEP 12 in FIG. 7). Specifically, as illustrated in FIG. 8, the distance dHL between the head feature point H and the left eye feature point LY of the driver 30 reflected in the acquired image IM in the left-right direction of the vehicle 20 is specified. . The distance dHL is specified as the number of pixels. Subsequently, the processing unit 122 determines the above angle θY1 by determining the arctangent of the distance (W/2) stored in the storage unit 124 and the distance dHL.

Furthermore, the processing unit 122 specifies the rotation angle θY of the head 31 in the yaw direction of the vehicle 20 by taking the difference value between the specified angle θY1 and the initial angle θY0. If the difference value is a positive value, it can be determined that the head 31 is rotating in the left yaw direction. If the difference value takes a negative value, it can be determined that the head 31 is rotating in the right yaw direction.

Subsequently, the processing unit 122 determines whether the absolute value of the rotation angle θY specified in STEP 12 exceeds the threshold value stored in the storage unit 124 (STEP 13).

If it is determined that the absolute value of the rotation angle θY exceeds the threshold value (YES in STEP 13), the processing unit 122 determines that the driver 30 may be taking a "looking away" posture (STEP 14). . In this case, the processing unit 122 adds a score to the "looking away posture" (STEP 5 in FIG. 3).

On the other hand, if it is determined that the absolute value of the rotation angle θY does not exceed the threshold value (NO in STEP 13), the processing unit 122 determines that there is no possibility that the driver 30 is taking a "looking away posture" ( STEP 15). In this case, the processing unit 122 does not add a score to the "looking away posture".

If the rotation angle θY of the head 31 in the yaw direction of the vehicle 20 can be specified, an appropriate feature point included in the head 31 can be used instead of the left eye feature point LY. Examples of such feature points include a right eye feature point, a left ear feature point, a right ear feature point, etc. that can be detected at a position away from the center of the face in the left-right direction of the vehicle 20.

Next, another example of the posture estimation process (STEP 4 in FIG. 3) executed by the processing unit 122 will be described with reference to FIG. 8 and FIGS. 10 to 13. FIG. 10 shows an example of the flow of processing.

In this example, when the rotation angle of the head 31 of the driver 30 reflected in the image IM acquired by the imaging device 11 in the pitch direction of the vehicle 20 exceeds a threshold, the processing unit 122 It is determined that there is a possibility that the patient is taking a ``shrimp posture'' or ``prone posture.''

The term "pitch direction" used in this specification means a direction of rotation about an axis extending in the left-right direction of the vehicle 20. Regarding the "pitch direction," the counterclockwise direction when viewed from the left of the driver 30 is defined as the "lower pitch direction," and the clockwise direction when viewed from the left of the driver 30 is defined as the "upper pitch direction." do.

"Shrimp posture" is one of multiple types of "postural collapse patterns" defined by the Ministry of Land, Infrastructure, Transport and Tourism. The "shrimp posture" is defined as a state in which the driver's upper body is arched and the driver's face is turned upward.

FIG. 11 illustrates the "shrimp warped posture". The "shrimp posture" is defined as a state in which the rotation angle θPU of the driver's face in the upward pitch direction exceeds a threshold value. The threshold value of the rotation angle θPU is, for example, 25°.

"Downward posture" is one of several "posture problems patterns" defined by the Ministry of Land, Infrastructure, Transport and Tourism. "Downward posture" is defined as a state in which the driver continues to maintain a posture with their face facing downward.

FIG. 12 shows an example of a "prone posture". The "prone posture" is defined as a state in which the rotation angle θPD of the driver's face in the downward pitch direction exceeds a threshold value. The threshold value of the rotation angle θPD is, for example, 20°.

In order to detect the rotation angle of the head 31 in the pitch direction of the vehicle 20, the processing unit 122 assumes the center coordinates of the head 31 of the driver 30 reflected in the image IM acquired by the imaging device 11. Processing is performed (STEP 21 in FIG. 10). Details of the processing will be described with reference to FIGS. 8 and 13.

As illustrated in FIG. 8, in this example, in addition to the head feature point H, the nose feature point NS can also be detected by the process of applying the skeleton model M. The nose feature point NS is a point corresponding to the nose of the model human body. Further, by the process of applying the skeleton model M, a rectangular frame FM is set in the image IM to estimate the area where the head 31 of the driver 30 is located.

FIG. 13 is a diagram for explaining a method of assuming the center coordinates CT of the head 31 using the set frame FM. In the figure, the head 31 is schematically shown as viewed from the left. In this example, the width of the head 31 in the vertical direction of the vehicle 20 and the width of the head 31 in the longitudinal direction of the vehicle 20 are the same as the width W of the frame FM set in the image IM shown in FIG. It is assumed that Under this premise, the center coordinate CT of the head 31 is assumed to be a point distant from the head feature point H by (W/2) toward the rear of the vehicle 20. The processing unit 122 stores the number of pixels corresponding to the distance (W/2) in the storage unit 124.

At this time, the angle θP0 that the straight line connecting the center coordinate CT and the nose feature point NS makes with the straight line connecting the center coordinate CT and the head feature point H is the angle θP0 between the head feature point H and the nose feature point NS in the left-right direction of the vehicle 20. This can be determined by determining the distance dHN between the two and the arctangent of (W/2) described above. This angle θP0 is stored in the storage unit 124 as an initial angle when the driver 30 is in a normal posture.

When the driver 30 assumes a "hunchback posture" or a "prone posture", the head 31 rotates in the pitch direction of the vehicle 20. At this time, the angle θP1 that the straight line connecting the center coordinate CT and the nose feature point NS makes with the straight line connecting the center coordinate CT and the head feature point H (the longitudinal direction of the vehicle 20) changes from the initial angle θP0. The amount of change (θP1-θP0) can be regarded as the rotation angle θP of the head 31 in the pitch direction of the vehicle 20. Therefore, when the absolute value of the amount of change (θP1-θP0) exceeds the threshold value, it can be estimated that the driver 30 is in a "hunchback posture" or a "prone posture." The threshold value may be determined as appropriate based on the definition by the Ministry of Land, Infrastructure, Transport and Tourism.

Based on the above principle, the processing unit 122 performs a process of specifying the rotation angle θP of the head 31 in the pitch direction of the vehicle 20 using the center coordinate CT specified in STEP 21 (STEP 22 in FIG. 10). Specifically, as illustrated in FIG. 8, the distance dHN between the head feature point H and the nose feature point NS of the driver 30 reflected in the image IM acquired in the vertical direction of the vehicle 20 is specified. . The distance dHN is specified as the number of pixels. Subsequently, the processing unit 122 determines the above-mentioned angle θP1 by determining the arctangent of the distance (W/2) stored in the storage unit 124 and the distance dHN.

Furthermore, the processing unit 122 specifies the rotation angle θP of the head 31 in the pitch direction of the vehicle 20 by taking the difference value between the specified angle θP1 and the initial angle θP0. If the difference value is a positive value, it can be determined that the head 31 is rotating in the downward pitch direction. If the difference value takes a negative value, it can be determined that the head 31 is rotating in the upward pitch direction.

Next, the processing unit 122 determines whether the absolute value of the rotation angle θP identified in STEP 22 exceeds the threshold value stored in the memory unit 124 (STEP 23).

If it is determined that the absolute value of the rotation angle θP exceeds the threshold value (YES in STEP 23), the processing unit 122 determines that the driver 30 is likely to be in a "shrimp posture" or a "prone posture." It is determined that there is one (STEP 24).

Specifically, when the rotation angle θP takes a positive value, it is determined that the driver 30 may be in a “prone position”, and when the rotation angle θP takes a negative value, the driver 30 It is determined that there is a possibility that the person is taking a ``shrimp posture.'' If it is determined that there is a possibility that the driver 30 is taking the "curved posture", the processing unit 122 adds a score to the "curved posture" (STEP 5 in FIG. 3). If it is determined that there is a possibility that the driver 30 is in a "down-down posture", the processing unit 122 adds a score to the "down-down posture" (STEP 5 in FIG. 3).

On the other hand, if it is determined that the absolute value of the rotation angle θP does not exceed the threshold value (NO in STEP 23), the processing unit 122 determines the possibility that the driver 30 is in a "shrimp posture" or a "prone posture". It is determined that there is no one (STEP 25). In this case, the processing unit 122 does not add scores to the "shrimp posture" and the "prone posture."

As long as the rotation angle θP of the head 31 in the pitch direction of the vehicle 20 can be specified, an appropriate feature point included in the head 31 can be used instead of the nose feature point NS. Examples of such feature points include left eye feature points, right eye feature points, mouth feature points, facial feature points, etc. that can be detected on the surface of the face.

Next, another example of the posture estimation process (STEP 4 in FIG. 3) executed by the processing unit 122 will be described with reference to FIGS. 14 to 17. FIG. 14 shows an example of the flow of processing.

In this example, when the rotation angle of the head 31 of the driver 30 reflected in the image IM acquired by the imaging device 11 in the roll direction of the vehicle 20 exceeds a threshold value, the processing unit 122 It is estimated that there is a possibility that only the head is in a sideways posture.

The term "roll direction" used herein means the direction of rotation of the vehicle 20 about an axis extending in the longitudinal direction. Regarding the "roll direction", a counterclockwise direction as seen from the driver 30 is defined as a "left roll direction", and a clockwise direction as seen from the driver 30 is defined as a "right roll direction".

The ``posture where only the neck falls sideways'' is one of multiple types of ``postural collapse patterns'' defined by the Ministry of Land, Infrastructure, Transport and Tourism. The "head tilted sideways posture" is defined as a state in which the driver's head continues to be tilted to the left or right.

FIG. 15 exemplifies a "posture where only the neck is tilted sideways" to the left. The "head tilted sideways posture" to the left is defined as a state in which the rotation angle θRL of the driver's face in the left roll direction exceeds a threshold value. The threshold value of the rotation angle θRL is, for example, 30°.

Figure 16 illustrates an example of a "head-only sideways posture" to the right. A "head-only sideways posture" to the right is defined as a state in which the rotation angle θRR of the driver's face in the right roll direction exceeds a threshold value. The threshold value of the rotation angle θRR is, for example, 30°.

Figure 17 shows an example of a method for determining the rotation angle θR in the roll direction of the vehicle 20. In this example, the left eye feature point LY and the right eye feature point RY detected by applying the skeleton model M are used. The inclination angle of the straight line connecting the left eye feature point LY and the right eye feature point RY from the direction corresponding to the left-right direction of the vehicle 20 can be considered as the above-mentioned rotation angle θR. Therefore, the rotation angle θR can be determined by obtaining the distance dLR between the left eye feature point LY and the right eye feature point RY in the left-right direction of the vehicle 20 and the distance dUD between the left eye feature point LY and the right eye feature point RY in the up-down direction of the vehicle 20, and calculating the arctangent of the distance dLR and the distance dUD.

When the driver 30 assumes a posture in which only the head is tilted sideways, the absolute value of the rotation angle θR increases. Therefore, when the absolute value of the rotation angle θR exceeds the threshold value, it can be estimated that the driver 30 is in a “head-only tilted posture”. The threshold value may be determined as appropriate based on the definition by the Ministry of Land, Infrastructure, Transport and Tourism.

Based on the above principle, the processing unit 122 performs a process of specifying the rotation angle θR of the head 31 in the roll direction of the vehicle 20 (STEP 31 in FIG. 14). Specifically, the above-mentioned distance dLR and distance dUD are specified for the left eye feature point LY and right eye feature point RY of the driver 30 reflected in the image IM acquired in the vertical direction of the vehicle 20. Subsequently, the processing unit 122 determines the rotation angle θR by determining the arctangent of the distance dLR and the distance dUD.

When the rotation angle θR is a positive value, the head 31 can be determined to be rotating in a left roll direction. When the rotation angle θR is a negative value, the head 31 can be determined to be rotating in a right roll direction.

Subsequently, the processing unit 122 determines whether the absolute value of the rotation angle θP specified in STEP 31 exceeds the threshold value stored in the storage unit 124 (STEP 32).

If it is determined that the absolute value of the rotation angle θR exceeds the threshold value (YES in STEP 32), the processing unit 122 determines that the driver 30 may be in a “sideways posture with only the neck tilted”. (STEP 33).

Specifically, if the rotation angle θR takes a positive value, it is determined that there is a possibility that the driver 30 is taking a “position with only the neck tilted sideways” to the left, and if the rotation angle θR takes a negative value. In this case, it is determined that there is a possibility that the driver 30 is taking a ``position with only the head tilted sideways'' to the right. In this case, the processing unit 122 adds a score to "only the neck is tilted sideways" (STEP 5 in FIG. 3).

On the other hand, if it is determined that the absolute value of the rotation angle θR does not exceed the threshold value (NO in STEP 32), the processing unit 122 determines that there is no possibility that the driver 30 is in a “sideways posture with only the neck tilted”. Make a judgment (STEP 34). In this case, the processing unit 122 does not add a score to "only the neck falls sideways".

If the rotation angle θR of the head 31 in the roll direction of the vehicle 20 can be specified, a combination of appropriate feature points included in the head 31 is used instead of the combination of the left eye feature point LY and the right eye feature point RY. It can be done. Examples of such combinations include left ear feature points and right ear feature points, left eye feature points and right ear feature points, left eye feature points and left ear feature points, left eye feature points and nose feature points, and the like.

Next, another example of the posture estimation process (STEP 4 in FIG. 3) executed by the processing unit 122 will be described with reference to FIG. 18 to FIG. 22. FIG. 18 shows an example of the process flow.

First, the processing unit 122 identifies distances in a specified direction between a plurality of pixels corresponding to the plurality of feature points in the image IM used to identify the plurality of feature points described above (STEP 41). In this example, the specified direction is the left-right direction of the vehicle 20.

FIG. 19 shows an enlarged portion of the image IM illustrated in FIGS. 4 and 5 in which the head 31 of the driver 30 is reflected. In this example, the left ear feature point LE and the right ear feature point RE can also be detected by the process of applying the skeleton model M. The left ear feature point LE is a point corresponding to the left ear of the model human body. The right ear feature point RE is a point corresponding to the right ear of the model human body. In FIG. 19 , the direction in which the straight line connecting the left ear feature point LE and the right ear feature point RE extends corresponds to the left-right direction of the vehicle 20 .

In this example, the processing unit 122 specifies the inter-pixel distance DP, which is the distance between the pixel PL corresponding to the left ear feature point LE and the pixel PR corresponding to the right ear feature point RE in the left-right direction of the vehicle 20. . The inter-pixel distance DP is specified by the number of pixels.

Subsequently, as illustrated in FIG. 18, the processing unit 122 performs a process of specifying the correspondence between the distance between two points in the image IM and the distance between two points in real space (STEP 42).

The distance between two points in the image IM is specified as the inter-pixel distance DP described with reference to STEP41. In this example, this inter-pixel distance DP is regarded as the width of the head 31 of the driver 30 reflected in the image IM in the left-right direction of the vehicle 20.

The average dimensions of each part of the human body are statistically investigated and provided as a database. An example of such a database is the "AIST Human Body Size Database 1997-98" provided by the National Institute of Advanced Industrial Science and Technology. FIG. 20 illustrates the head width WH in the left-right direction of the human body obtained from such a database. It can be said that the head width WH corresponds to the real spatial distance DR between the left ear and the right ear of an average person.

The value of the real spatial distance DR corresponding to the width WH is stored in the storage unit 124. The processing unit 122 reads the value of the real space distance DR from the storage unit 124, and calculates the ratio P (=DR/DP) with the inter-pixel distance DP specified in STEP41. Thereby, if the inter-pixel distance DP in the image IM is specified, the real spatial distance DR in the cabin 21 of the vehicle 20 can be calculated based on the ratio P (DR=DP×P).

In this example, it is estimated whether the driver 30 is in a "leaning position" based on the correspondence between the inter-pixel distance DP and the real spatial distance DR identified in STEP 42.

The "sideways leaning posture" is one of multiple types of "posture collapse patterns" defined by the Ministry of Land, Infrastructure, Transport and Tourism. A "side-leaning posture" is defined as a state in which the driver's upper body continues to lean toward the left or right.

FIG. 21 illustrates a "sideways leaning posture" to the left. The leftward "leaning posture" is defined as a state in which the leftward movement amount ΔDL of the vehicle 20 from the normal state of a specific position on the driver's face exceeds a threshold value. The threshold value is, for example, 200 mm.

FIG. 22 illustrates a "sideways leaning posture" toward the right. The "sideways leaning posture" to the right is defined as a state in which the rightward movement amount ΔDR of the vehicle 20 from the normal state of a specific position on the driver's face exceeds a threshold value. The threshold value is, for example, 200 mm.

In order to estimate whether the driver 30 is in a "leaning posture", the processing unit 122 identifies the position of the pixel corresponding to the head feature point H of the driver 30 reflected in the acquired image IM. , and the position of the pixel corresponding to the head feature point H in a normal state stored in the storage unit 124. Note that feature points such as the nose, eyes, face, etc. may be used for estimation depending on the specifications of the applied skeletal model.

As a result of the comparison, the pixel distance in the left-right direction of the vehicle 20 between the position of the pixel corresponding to the head feature point H of the driver 30 captured in the acquired image IM and the position of the pixel corresponding to the head feature point H under normal conditions can be identified. This pixel distance is converted into a real spatial distance within the vehicle 20 based on the ratio identified in STEP 42. This real spatial distance corresponds to the amount of movement of the head 31 of the driver 30 captured in the acquired image IM in the left-right direction of the vehicle 20 from under normal conditions. That is, the processing unit 122 identifies the amount of movement of the head 31 of the driver 30 in the left-right direction of the vehicle 20 (STEP 43 in FIG. 18).

Subsequently, the processing unit 122 determines whether the converted real space distance exceeds a threshold (STEP 44). That is, it is determined whether the movement amount ΔDL of the head 31 of the driver 30 to the left of the vehicle 20 exceeds a threshold value. Similarly, it is determined whether the movement amount ΔDR of the head 31 of the driver 30 to the right of the vehicle 20 exceeds a threshold value.

If it is determined that the converted real space distance does not exceed the threshold (NO in STEP 44), the processing unit 122 determines that there is no possibility that the driver 30 is in a “side-leaning posture” (STEP 45). ). In this case, the processing unit 122 does not add a score to the "side leaning posture".

If it is determined that the converted real space distance exceeds the threshold (YES in STEP 44 of FIG. 18), the processing unit 122 determines that the driver 30 may be in a "side-leaning posture". (STEP 46).

Specifically, if it is determined that the amount of movement ΔDL of the driver's 30 head 31 to the left of the vehicle 20 exceeds a threshold, it is determined that the driver 30 may be in a "leaning-toward-side position" to the left. If it is determined that the amount of movement ΔDR of the driver's 30 head 31 to the right of the vehicle 20 exceeds a threshold, it is determined that the driver 30 may be in a "leaning-toward-side position" to the right. In this case, the processing unit 122 adds a score to the "leaning-toward-side position" (STEP 5 in FIG. 3).

If there are a plurality of feature points included in the head 31, a different combination from the left ear feature point LE and the right ear feature point RE may be adopted depending on the specifications of the applied skeletal model M. Examples include a combination of left eye feature points and right eye feature points, a combination of nose feature points and left ear feature points, a combination of head feature points and right ear feature points, a combination of left eye feature points and left ear feature points, and the like.

The "side leaning posture" is estimated based on the amount of movement of the head 31 in the left-right direction of the vehicle 20, so if a reference for the distance in the left-right direction of the vehicle 20 can be provided, a combination of multiple feature points can be used. is not limited to what is included in the head 31. Examples include a combination of left shoulder feature points and right shoulder feature points, a combination of left ear feature points and left shoulder feature points, a combination of neck feature points and right shoulder feature points, and the like.

As illustrated in FIG. 18, the processing unit 122 of the image processing device 12 can estimate whether the driver 30 is in a “sideways falling posture”.

The "sideways falling posture" is one of multiple types of "postural collapse patterns" defined by the Ministry of Land, Infrastructure, Transport and Tourism. The "sideways falling posture" is defined as a state in which the driver's upper body is tilted to the left or right, and the driver's face is also tilted in the same direction.

Figure 23 illustrates an example of a "sideways posture" to the left. A "sideways posture" to the left is defined as a state in which the amount of movement ΔDL of a specific position on the driver's face to the left of the vehicle 20 from normal exceeds a threshold, and the roll angle θRL of the driver's face to the left exceeds a threshold. The threshold for ΔDL is, for example, 200 mm. The threshold for θRL is, for example, 15°.

FIG. 24 illustrates a "sideways falling posture" to the right. A "sideways falling posture" to the right is defined as a situation where the rightward movement amount ΔDR of the vehicle 20 from the normal state of a specific position on the driver's face exceeds a threshold value, and the driver's face is tilted to the right. This is defined as a state in which the roll angle θRR exceeds a threshold value. The threshold value of ΔDR is, for example, 200 mm. The threshold value of θRR is, for example, 15°.

Specifically, when the processing unit 122 determines that the amount of movement of the head 31 of the driver 30 reflected in the image IM in the left-right direction of the vehicle 20 exceeds a threshold (YES in STEP 44 of FIG. ), the rotation angle θR of the head 31 in the roll direction of the vehicle 20 is specified (STEP 47). In specifying the rotation angle θR, the method described with reference to FIG. 17 can be used.

Then, the processing unit 122 determines whether the identified rotation angle θR exceeds a threshold value (STEP 48 in FIG. 18). That is, it determines whether the rotation angle θR of the head 31 of the driver 30 in the left roll direction exceeds a threshold value. Similarly, it determines whether the rotation angle θR of the head 31 of the driver 30 in the right roll direction exceeds a threshold value.

If it is determined that the rotation angle θR does not exceed the threshold value (NO in STEP 48), the processing unit 122 determines that the driver 30 may be in a "leaning posture" (STEP 46). In this case, the processing unit 122 adds a score to the "side leaning posture" (STEP 5 in FIG. 3).

If it is determined that the rotation angle θR exceeds the threshold value (YES in STEP 48 of FIG. 18), the processing unit 122 determines that the driver 30 may be in a “sideways falling posture” (STEP 49). ).

Specifically, when it is determined that the rotation angle θRL of the head 31 of the driver 30 in the left roll direction of the vehicle 20 exceeds a threshold value, the driver 30 assumes a “sideways falling posture” to the left. It is determined that there is a possibility that the If it is determined that the rotation angle θRR of the head 31 of the driver 30 in the right roll direction of the vehicle 20 exceeds the threshold, there is a possibility that the driver 30 is in a "sideways falling posture" to the right. It is determined that there is. In this case, the processing unit 122 adds a score to the "sideways falling posture" (STEP 5 in FIG. 3).

FIG. 25 shows another example of the flow of the posture estimation process (STEP 4 in FIG. 3) executed by the processing unit 122. Elements that are common to the process flow illustrated in FIG. 18 are given the same reference numerals, and repetitive explanations will be omitted.

In this example, it is estimated whether the driver 30 is in a "prone position" based on the correspondence between the inter-pixel distance DP and the real spatial distance DR identified in STEP 42.

The "prone posture" is one of multiple types of "posture collapse patterns" defined by the Ministry of Land, Infrastructure, Transport and Tourism. The "prone position" is defined as a state in which the driver continues to fall forward with his or her face near the steering wheel.

Figure 26 illustrates an example of a "prone posture." The "prone posture" is defined as a state in which the amount of forward movement ΔDF of the vehicle 20 from normal at a specific position on the driver's face exceeds a threshold, the amount of downward movement ΔDD of the vehicle 20 exceeds a threshold, and the downward pitch angle θPD of the driver's face exceeds a threshold. The threshold for ΔDF is, for example, 200 mm. The threshold for ΔDD is, for example, 180 mm. The threshold for θPD is, for example, 30°.

In this example, in order to estimate whether the driver 30 is in a "prone posture", the processing unit 122 selects a pixel corresponding to the head feature point H of the driver 30 reflected in the acquired image IM. The position is specified and compared with the position of the pixel corresponding to the head feature point H in a normal state stored in the storage unit 124. Note that feature points such as the nose, eyes, face, etc. may be used for estimation depending on the specifications of the applied skeletal model.

As a result of the comparison, the front and back of the vehicle 20 are between the position of the pixel corresponding to the head feature point H of the driver 30 reflected in the acquired image IM and the position of the pixel corresponding to the head feature point H in normal conditions. The inter-pixel distance in the direction may be determined. This inter-pixel distance is converted into a real spatial distance within the vehicle 20 based on the ratio specified in STEP42. This real spatial distance corresponds to the amount of movement of the head 31 of the driver 30 reflected in the acquired image IM in the forward direction of the vehicle 20 from the normal state. That is, the processing unit 122 specifies the amount of movement of the head 31 of the driver 30 in the forward direction of the vehicle 20 (STEP 53 in FIG. 25).

Subsequently, the processing unit 122 determines whether the converted real space distance exceeds a threshold (STEP 54). That is, it is determined whether the amount of movement ΔDF of the head 31 of the driver 30 toward the front of the vehicle 20 exceeds a threshold value.

If it is determined that the converted real space distance does not exceed the threshold (NO in STEP 54), the processing unit 122 determines that there is no possibility that the driver 30 is in a "prone position" (STEP 55). In this case, the processing unit 122 does not add a score to the "prone posture".

If it is determined that the converted real space distance exceeds the threshold (YES in STEP 54 of FIG. 25), the processing unit 122 determines that the driver 30 may be in a "prone position". (STEP 26). In this case, the processing unit 122 adds a score to the "prone posture" (STEP 5 in FIG. 3).

In other words, in this example, of the movement amount ΔDF, the movement amount ΔDD, and the pitch angle θPD included in the definition of the "prone posture," processing focuses only on the movement amount ΔDF. However, even with this configuration, it is possible to estimate the "prone posture."

Next, another example of the posture estimation process (STEP 4 in FIG. 3) executed by the processing unit 122 will be described with reference to FIGS. 27 to 29. FIG. 27 shows an example of the flow of processing. In this example, it is estimated whether the driver 30 is in a "prone position".

Figure 28 shows an enlarged view of the portion of the image IM illustrated in Figures 4 and 5 in which the upper body of the driver 30 is captured. In this example, a nose feature point NS can also be detected by the process of applying the skeletal model M. The nose feature point NS is a point that corresponds to the nose of the model human body.

First, the processing unit 122 specifies the distance d in the direction corresponding to the vertical direction of the vehicle 20 between the nose feature point NS and the left shoulder feature point LS of the driver 30 reflected in the image IM acquired by the imaging device 11. (STEP 61 in FIG. 27). The nose feature point NS and the left shoulder feature point LS are examples of multiple feature points on the upper body. The distance d is specified by the number of pixels. The value of the specified distance d is stored in the storage unit 124.

Subsequently, the processing unit 122 compares the value of the distance d specified in STEP 11 with the value of the distance d stored in the storage unit 124, and determines whether the amount of decrease exceeds the threshold value dth (STEP 62).

FIG. 29 illustrates a change over time in the distance d between the nose feature point NS and the left shoulder feature point LS when the driver 30 takes a "prone posture". As the driver 30 falls forward, the distance d in the vertical direction of the vehicle 20 between the nose feature point NS and the left shoulder feature point LS rapidly decreases. In the illustrated example, the distance d begins to decrease at time t1, and the amount of decrease exceeds the threshold dth at time t2.

Therefore, if it is determined that the amount of decrease in the distance d between the nose feature point NS and the left shoulder feature point LS in the vertical direction of the vehicle 20 exceeds the threshold value dth (YES in STEP 62), the processing unit 122 determines that the driver 30 may be in a "prone posture" (STEP 63). In this case, the processing unit 122 adds a score to the "prone posture" (STEP 5 in FIG. 3). The threshold value dth can be statistically determined as the amount of decrease in the distance d that may occur when a typical driver 30 is in a "prone posture".

On the other hand, if it is determined that the amount of decrease in the distance d in the vertical direction of the vehicle 20 between the nose feature point NS and the left shoulder feature point LS does not exceed the threshold value dth (NO in STEP 62), the processing unit 122 It is determined that there is no possibility that the person 30 is in a "prone position" (STEP 64). In this case, the processing unit 122 does not add a score to the "prone posture".

Depending on the direction and position of the driver's 30 face, it may be difficult to identify the nose feature point NS. Therefore, as illustrated in FIG. 27, the processing unit 122 can determine whether the nasal feature point NS cannot be specified (STEP 65).

If the nose feature point NS has been identified (NO in STEP 65), the process proceeds to STEP 62, and as described above, the amount of decrease in the distance d in the vertical direction of the vehicle 20 between the nose feature point NS and the left shoulder feature point LS is set to the threshold dth. It is judged whether it exceeds.

If the nose feature point NS cannot be identified (YES in STEP 65), the processing unit 122 calculates the distance d' in the vertical direction of the vehicle 20 between the head feature point H and the left shoulder feature point LS, as illustrated in FIG. Specify (STEP 66). The head feature point H and the left shoulder feature point LS are examples of multiple feature points on the upper body. The distance d' is specified by the number of pixels. The value of the specified distance d' is stored in the storage unit 124.

In this case, the processing unit 122 compares the value of the distance d' specified in STEP 66 with the value of the distance d' stored in the storage unit 124, and determines whether the amount of decrease exceeds the threshold value dth' ( STEP 62). The threshold value dth can be statistically determined as the amount of decrease in the distance d' that may occur when the general driver 30 takes a "prone posture."

If it is determined that the amount of decrease in the distance d' in the vertical direction of the vehicle 20 between the head feature point H and the left shoulder feature point LS exceeds the threshold value dth' (YES in STEP 62), the processing unit 122 It is determined that 30 may be in a "prone position" (STEP 63). In this case, the processing unit 122 adds a score to the "prone posture" (STEP 5 in FIG. 3).

On the other hand, if it is determined that the decrease in the distance d' between the head feature point H and the left shoulder feature point LS in the vertical direction of the vehicle 20 does not exceed the threshold value dth' (NO in STEP 62), the processing unit 122 determines that there is no possibility that the driver 30 is in a "prone posture" (STEP 64). In this case, the processing unit 122 does not add a score to the "prone posture."

According to such a configuration, the influence of the direction and position of the face on the estimation of the "prone posture" can be further suppressed.

In this example, the nose feature point NS or head feature point H and left shoulder feature point LS are used to estimate whether the driver 30 is in a "prone posture." However, as long as the distance between the feature points in the vertical direction of the vehicle 20 decreases as the posture deviates from the normal posture, an appropriate combination can be selected from those included in the upper body of the driver 30. Examples of such combinations include nose feature points and neck feature points, left ear feature points and left shoulder feature points, and the like.

By appropriately selecting multiple characteristic points from the upper body of the driver 30, whose vertical distance to the vehicle 20 decreases as the driver deviates from the normal posture, the processing unit 122 of the image processing device 12 can estimate whether the driver 30 is in the "sideways posture" described with reference to Figures 23 and 24.

In the posture estimation process (STEP 4) illustrated in FIG. 3, it is determined whether the driver 30 is taking two or more abnormal postures among the various abnormal postures explained with reference to FIGS. 7 to 29. can be estimated. Therefore, in the abnormality estimation process illustrated in FIG. 6, it is determined whether the scores of the two or more abnormal postures exceed the threshold.

In a situation where a score has been assigned for at least one abnormal posture and the score does not exceed the threshold (NO in STEP 71), there is a possibility that the driver 30 is in the abnormal posture, but the abnormal posture is not higher than the threshold value. This means that it has not yet been estimated that the

A situation in which a score is assigned to at least one abnormal posture and the score exceeds the threshold value (YES in STEP 71) means that the driver 30 is estimated to be in the abnormal posture. do.

The transition from the normal posture of the driver 30 to the abnormal posture generally progresses in a complex manner, including the various abnormal posture elements described above. A specific abnormal posture that was dominant at the time when a certain image information I was acquired may no longer be dominant at the time when the next image information I is acquired, and another abnormal posture may become dominant.

According to the above configuration, the possibility that the driver 30 is taking each of a plurality of prescribed abnormal postures is determined each time image information I is received, and a score is assigned based on the possibility. Since the abnormal posture is estimated based on whether the score exceeds the threshold, it is possible to grasp the possibility that the driver 30 is taking multiple abnormal postures, and ultimately determine which posture the driver 30 is taking. At least one abnormal posture can be estimated. Therefore, the reliability of the estimation of the abnormal posture of the driver 30 based on the image IM acquired by the imaging device 11 can be increased.

As a result of determining whether the assigned score exceeds the threshold (STEP 71 in FIG. 6), a plurality of abnormal postures may exceed the threshold at the same time. Therefore, the processing unit 122 can determine whether a plurality of abnormal postures exceed the threshold at the same time (YES in STEP 71, STEP 74).

If there is only one abnormal posture for which the assigned score exceeds the threshold (NO in STEP74), the process proceeds to STEP72. That is, it is estimated that the driver 30 is in the abnormal posture.

If there are a plurality of abnormal postures whose assigned scores exceed the threshold (YES in STEP 74), the processing unit 122 identifies the abnormal posture with the maximum score value (STEP 75). After that, the process advances to STEP72. That is, it is estimated that the driver 30 is taking the abnormal posture with the maximum score value.

According to such a configuration, when the movement of the driver 30 includes elements of a plurality of abnormal postures, the most dominant abnormal posture can be estimated. Thereby, for example, it is possible to output control information C that can correspond to the most prevalent abnormal posture.

As a result of the estimation processing for the various abnormal postures described with reference to FIGS. 7 to 29, it may be determined that there is no possibility that the driver 30 is in any of the abnormal postures. However, the posture estimation process (STEP 4 in FIG. 3) is executed under a situation where the posture of the driver 30 is determined not to be normal (NO in STEP 3). Therefore, there is a possibility that the driver 30 is in an undefined abnormal posture.

At the end of the posture estimation process, the processing unit 122 can determine whether the posture of the driver 30 does not correspond to any of the abnormal postures. If it is determined that the posture of the driver 30 does not correspond to any of the abnormal postures, the processing unit 122 can add the score to "undefined abnormal posture" in the score addition process of STEP 5.

According to such a configuration, even if it is determined that the posture of the driver 30 reflected in the image IM does not correspond to any of a plurality of prescribed abnormal postures, there is no abnormality in the posture of the driver 30. It is possible to avoid the conclusion that there is no abnormality, and treat it as having some kind of abnormal posture. Therefore, the flexibility of the abnormality estimation system 10 for undefined abnormal postures can be increased.

The processing unit 122 having each of the functions described above can be realized by a general-purpose microprocessor that operates in cooperation with a general-purpose memory. Examples of general-purpose microprocessors include CPUs, MPUs, and GPUs. Examples of general-purpose memory include ROM and RAM. In this case, the ROM may store a computer program that executes the above-described processing. A ROM is an example of a storage medium that stores a computer program. The processor designates at least a part of the computer program stored on the ROM, loads it on the RAM, and executes the above-described processing in cooperation with the RAM. The computer program described above may be preinstalled in general-purpose memory, or may be downloaded from an external server via a communication network and installed in general-purpose memory. In this case, the external server is an example of a storage medium that stores a computer program.

The processing unit 122 may be realized by a dedicated integrated circuit such as a microcontroller, ASIC, or FPGA that can execute the above-mentioned computer program. In this case, the above-mentioned computer program is preinstalled in the memory element included in the dedicated integrated circuit. The storage element is an example of a storage medium that stores a computer program. The processing unit 122 can also be realized by a combination of a general-purpose microprocessor and a dedicated integrated circuit.

The storage unit 124 can be realized by a semiconductor memory or a hard disk device. The storage unit 124 may be realized by the above-mentioned general-purpose memory or storage element.

The above embodiments are merely illustrative to facilitate understanding of the present invention. The configuration according to the embodiments described above can be modified and improved as appropriate without departing from the spirit of the present invention.

In the above embodiment, if it is determined that the driver 30 may be assuming at least one of the abnormal postures defined as a result of the posture estimation process (STEP 4 in FIG. 3), the score value assigned in the score addition process (STEP 5 in FIG. 3) is constant. However, multiple scores may be assigned according to the possibility that the driver 30 may be assuming each abnormal posture. Specifically, multiple threshold ranges are set in the posture estimation process performed for each abnormal posture, and different scores may be associated with each threshold range.

For example, in a situation in which it is determined that the driver 30 may be in a "downward posture" and a "sideways posture" simultaneously, if it is determined based on the threshold range that the "downward posture" is more likely to be present, the score assigned to the "downward posture" will be higher than the score assigned to the "sideways posture." With this configuration, when the movement of the driver 30 includes elements of multiple abnormal postures, it is possible to increase the likelihood that the more prevalent abnormal posture will ultimately be estimated. In other words, it becomes possible to estimate the abnormal posture more precisely.

In the embodiment described above, the determination as to whether the score given to each of the plurality of defined abnormal postures exceeds the threshold value (STEP 71 in FIG. 6) is made based on the cumulative value of the scores. The cumulative value is an example of a representative value. However, a representative value such as an average value, median value, or mode may be used, and a threshold value corresponding to the representative value may be set.

In the embodiment described above, the abnormal postures that the driver 30 may take are "looking away", "heavy", "head down", "head tilted to the side", "leaning sideways", and "sideways". ”, and “prone posture”. However, it is possible to specify the head 31 of the driver 30 reflected in the image IM based on the feature points obtained by applying the skeleton model M, and the position and moving direction of the feature points and the specified If the relationship between the rotation angles of the head 31 in the opposite directions can be found, an appropriate abnormal posture can be estimated based on the relationship.

FIG. 30 shows a "supine posture" as an example of such an abnormal posture. The "backward posture" is one of multiple types of "postural collapse patterns" defined by the Ministry of Land, Infrastructure, Transport and Tourism. The "upside down posture" is defined as a continued state in which the driver's upper body is tilted toward the rear of the vehicle and the driver's face is facing upward.

Specifically, the "upside down posture" is defined as a situation in which the rearward movement amount ΔDB of the vehicle 20 from the normal state of a specific position on the driver's face exceeds a threshold value, and the driver's face is in an upward pitch direction. is defined as a state in which the rotation angle θPU exceeds a threshold value. The threshold value of ΔDB is, for example, 100 mm. The threshold value of θPU is, for example, 20°.

The image processing device 12 may be mounted on the vehicle 20, or may be provided as an external device capable of communicating with the vehicle 20 via a well-known wireless communication network. In this case, image information I output from the imaging device 11 is transmitted to the image processing device 12 via the wireless communication network. Control information C that may be output as a result of the attitude estimation process by the processing unit 122 is transmitted to the control device 13 via the well-known wireless communication network.

The abnormality estimation system 10 can also be applied to moving objects other than the vehicle 20. Examples of other moving objects include trains, aircraft, ships, and the like.

10: Abnormality estimation system, 11: Imaging device, 12: Image processing device, 121: Reception unit, 122: Processing unit, 13: Control device, 20: Vehicle, 30: Driver, 40: Controlled device, I: Image Information, IM: Image

Claims (6)

a reception unit that repeatedly receives image information corresponding to images in which a driver of a mobile object is captured;
a processing unit that estimates whether the posture of the driver is abnormal based on the image information;
It is equipped with
Each time the processing unit receives the image information,
determining the possibility that the posture of the driver reflected in the image corresponds to each of a plurality of abnormal postures;
assigning a score corresponding to the possibility to each of the plurality of abnormal postures;
If the representative value of the score given to at least one of the plurality of abnormal postures exceeds a threshold, it is estimated that the driver is in the at least one abnormal posture;
If it is determined that the posture of the driver reflected in the image is not normal but does not fall under any of the plurality of abnormal postures, assigning the score to the undefined abnormal posture;
Image processing device. When the representative value exceeds the threshold for a plurality of abnormal postures, the processing unit identifies one abnormal posture to which a score having the maximum representative value is assigned, and causes the driver to identify the one abnormal posture. It is estimated that the
The image processing device according to claim 1. The plurality of abnormal postures include a prone posture, a prone posture, a supine posture, a bowed posture, a posture where only the neck falls sideways, a sideways posture, a sideways leaning posture, and a looking away posture.
The image processing device according to claim 1 or 2 . configured to be mounted on the mobile body,
The image processing device according to any one of claims 1 to 3 . A computer program executable by a processing unit of an image processing device,
When the above-mentioned program is executed, the image processing device
Repeatedly receiving image information corresponding to an image in which a driver of the moving object is captured;
Each time the image information is received,
determining the possibility that the driver's posture captured in the image corresponds to each of a plurality of abnormal postures;
assigning a score corresponding to the likelihood to each of the plurality of abnormal postures;
When a representative value of the scores assigned to at least one of the plurality of abnormal postures exceeds a threshold value, it is estimated that the driver is taking the at least one abnormal posture;
When the posture of the driver captured in the image is estimated to be abnormal but is determined not to correspond to any of the plurality of abnormal postures, the score is assigned to an undefined abnormal posture.
Computer program. an imaging device that repeatedly outputs image information corresponding to an image in which a driver of a moving object is captured;
an image processing device that estimates whether the driver's posture is abnormal based on the image information;
a control device that controls the operation of a controlled device mounted on the moving body based on a result of estimation by the image processing device that the posture is abnormal;
It is equipped with
Each time the image processing device receives the image information,
determining the possibility that the posture of the driver reflected in the image corresponds to each of a plurality of abnormal postures;
assigning a score corresponding to the possibility to each of the plurality of abnormal postures;
If the representative value of the score given to at least one of the plurality of abnormal postures exceeds a threshold, it is estimated that the driver is in the at least one abnormal posture;
If it is determined that the posture of the driver reflected in the image is not normal but does not fall under any of the plurality of abnormal postures, assigning the score to the undefined abnormal posture;
Anomaly estimation system.

Images