JP7314084B2 - Image processing device, computer program, and anomaly estimation system - Google Patents

Description

The present invention relates to an image processing apparatus that processes an image in which a driver of a mobile object is captured. The invention also relates to a computer program executable by a processing unit of the image processing apparatus. The present invention also relates to an abnormality estimation system for estimating whether the posture of the driver is abnormal based on the image.

Patent Literature 1 discloses a technique for detecting an abnormality in the posture of a driver of a vehicle, which is an example of a moving object. The posture abnormality is estimated based on an image of the driver captured by an imaging device mounted on the vehicle.

JP 2019-105872 A

SUMMARY OF THE INVENTION An object of the present invention is to increase the likelihood of estimating an abnormal posture of a driver of a mobile object based on an image acquired by an imaging device.

One aspect for achieving the above object is an image processing device,
a reception unit that receives image information corresponding to an image in which the driver of the moving object is captured;
a processing unit that estimates whether the posture of the driver is abnormal based on the image information;
and
The processing unit, based on the image information,
identifying a plurality of feature points on the driver's body reflected in the image;
identifying a ratio of a distance between pixels corresponding to the feature points in a specified direction to a known distance between the feature points in the specified direction in real space;
identifying an amount of movement in the specified direction of the position of the driver's head reflected in the image;
The posture is estimated to be abnormal when the amount of movement of the position of the driver's head in the real space specified based on the ratio in the specified direction exceeds a threshold.

One aspect for achieving the above object is a computer program executable by a processing unit of an image processing apparatus,
By being executed, the image processing device
receiving image information corresponding to an image in which a driver of a mobile object is captured;
Based on the image information,
specifying a plurality of feature points on the driver's body reflected in the image;
determining a ratio of a distance between pixels corresponding to the feature points in a specified direction to a known distance between the feature points in the specified direction in real space;
specifying a movement amount in the specified direction of the position of the driver's head reflected in the image;
If the amount of movement of the position of the driver's head in the real space specified based on the ratio exceeds a threshold in the specified direction, the driver's posture is estimated to be abnormal.

One aspect for achieving the above object is an anomaly estimation system,
an imaging device that outputs image information corresponding to an image in which a driver of a mobile object is captured;
an image processing device that estimates whether the posture of the driver is abnormal based on the image information;
a control device that controls the operation of a controlled device mounted on the moving body based on the estimation result that the posture is abnormal by the image processing device;
and
The image processing device, based on the image information,
identifying a plurality of feature points on the driver's body reflected in the image;
identifying a ratio of a distance between pixels corresponding to the feature points in a specified direction to a known distance between the feature points in the specified direction in real space;
identifying an amount of movement in the prescribed direction of the position of the driver's head reflected in the image;
The posture is estimated to be abnormal when the amount of movement of the position of the driver's head in the real space specified based on the ratio in the specified direction exceeds a threshold.

The position of the seat in the vehicle and how to sit on the seat differ from driver to driver. Therefore, how the image is reflected in the image acquired by the imaging device also differs from driver to driver. When trying to estimate an abnormal posture based on the amount of movement of the driver's head, the above-described difference in how the image is reflected in the image can cause a decrease in the certainty of estimation.

However, according to the configuration according to each of the above aspects, the ratio between the pixel-to-pixel distance corresponding to the plurality of feature points detected for each driver captured in the image and the known real space distance between the plurality of feature points is specified, so the amount of movement of the head in the image can be specified accurately for each driver. As a result, it is possible to increase the likelihood of estimating the abnormal posture of the driver based on the image acquired by the imaging device.

1 illustrates a functional configuration of an abnormality estimation system according to one embodiment; 1 illustrates a vehicle in which the abnormality estimation system of FIG. 1 can be installed. 2 illustrates the flow of processing executed by the image processing apparatus of FIG. 1; 2 illustrates an image that may be acquired by the imaging device of FIG. 1; The image in FIG. 4 illustrates a state in which a skeleton model is applied. 4 shows an example of the flow of posture estimation processing in FIG. 3 ; FIG. 11 is a diagram for explaining details of posture estimation processing; FIG. 11 is a diagram for explaining details of posture estimation processing; It is a figure for demonstrating the "side leaning posture" to the left. It is a figure for demonstrating the "side leaning posture" to the right. It is a figure for demonstrating the "sideways posture" to the left. It is a figure for demonstrating the "lateral fall posture" to the right. FIG. 10 is a diagram for explaining a method of specifying the rotation angle of the head in the roll direction; FIG. 4 shows another example of the flow of posture estimation processing in FIG. 3 ; FIG. It is a figure for demonstrating a "push-down posture."

Exemplary embodiments are described in detail below with reference to the accompanying drawings. FIG. 1 illustrates the functional configuration of an abnormality estimation system 10 according to one embodiment. The abnormality estimation system 10 is a system that estimates whether the posture of the driver 30 is abnormal based on the image of the vehicle 20 illustrated in FIG. 2 in which the driver 30 is reflected. Vehicle 20 is an example of a mobile object.

Posture anomalies are estimated to detect anomalies of driver 30 . The term "driver's abnormality" used in this specification means a sudden change in physical condition that is difficult to predict in advance.

In the accompanying drawings, arrow F represents the forward direction as seen from driver 30 . Arrow B represents the rearward direction viewed from driver 30 . An arrow L indicates the left direction as seen from the driver 30 . An arrow R indicates the right direction as seen from the driver 30 . An arrow U represents an upward direction as seen from the driver 30 . An arrow D represents the downward direction viewed from the driver 30 .

As illustrated in FIG. 1 , an abnormality estimation system 10 includes an imaging device 11 . The imaging device 11 is arranged at an appropriate location on the vehicle 20 . A driver 30 seated on a seat 22 arranged in a cabin 21 of a vehicle 20 illustrated in FIG. 2 is imaged by the imaging device 11 .

As illustrated in FIG. 1, the imaging device 11 is configured to output image information I corresponding to the captured image. The image information I may be in the form of analog data or in the form of digital data.

The abnormality estimation system 10 includes an image processing device 12 . The image processing device 12 includes a reception section 121 and a processing section 122 . The reception unit 121 is configured to receive the image information I from the imaging device 11 . If the image information I is in the form of analog data, the reception unit 121 can include an appropriate conversion circuit including an A/D converter. The processing unit 122 processes image information I in the form of digital data.

The processing unit 122 is configured to execute processing for estimating whether the posture of the driver 30 is abnormal based on the image information I. Details of the processing will be described later.

The image processing device 12 has an output unit 123 . The processing unit 122 is configured to output the control information C through the output unit 123 when it is estimated that the posture of the driver 30 is abnormal. The control information C may be in the form of digital data or in the form of analog data. If the control information C is in the form of analog data, the output section 123 may include suitable conversion circuitry including a D/A converter.

The abnormality estimation system 10 includes a control device 13 . The control device 13 is mounted on the vehicle 20 . The control device 13 is configured to control the operation of the controlled device 40 mounted on the vehicle 20 based on the control information C output from the image processing device 12 .

Specifically, the control device 13 is configured to enable driving assistance for the vehicle 20 when it is estimated that the posture of the driver 30 is abnormal. The term “driving assistance” as used herein means control processing that at least partially performs at least one of driving operation (steering, acceleration, deceleration, etc.), monitoring of the driving environment, and driving operation backup. In other words, it includes everything from partial driving assistance such as the collision damage mitigation brake function and lane keeping assist function to fully automated driving operations.

For example, based on the control information C output from the image processing device 12, the control device 13 causes the controlled device 40 to decelerate and stop the vehicle 20 on the shoulder of the road. Examples of the controlled device 40 include a device that configures the driving system of the vehicle 20, a lamp, and a device that notifies passengers and pedestrians of the vehicle 20 and other vehicles that the driving support operation is effective.

Next, the processing for estimating the posture of the driver 30 executed by the processing unit 122 of the image processing device 12 will be described in detail with reference to FIGS. 3 to 5. FIG. FIG. 3 exemplifies the flow of the processing.

The processing unit 122 receives the image information I from the imaging device 11 through the receiving unit 121 (STEP 1). FIG. 4 exemplifies an image IM in which the driver 30 is captured by the imaging device 11 . Image information I corresponds to image IM.

Subsequently, the processing unit 122 performs processing for applying the skeleton model (STEP 2 in FIG. 3). As used herein, the term "process of applying a skeleton model" means detecting a plurality of feature points defined in the skeleton model in a driver captured in an image captured by an imaging device, and connecting the plurality of feature points with a plurality of skeleton lines defined in the skeleton model.

FIG. 5 shows an example in which the skeletal model M is applied to the driver 30 appearing in the image IM acquired by the imaging device 11 . In this example, the skeleton model M includes a head feature point H, a neck feature point NK, a left shoulder feature point LS, and a right shoulder feature point RS.

The head feature point H is a point corresponding to the center of the head of the model human body. The neck feature point NK is a point corresponding to the neck of the model human body. The left shoulder feature point LS is a point corresponding to the left shoulder of the model human body. The right shoulder feature point RS is a point corresponding to the right shoulder of the model human body. The head minutiae H and the neck minutiae NK are connected by a skeleton line. The neck feature point NK is connected to each of the left shoulder feature point LS and the right shoulder feature point RS by skeleton lines.

The processing unit 122 detects points corresponding to each of the head feature point H, the neck feature point NK, the left shoulder feature point LS, and the right shoulder feature point RS in the driver 30 captured in the image IM, and connects the plurality of detected points with the skeleton lines described above. Algorithms for performing this process are well known and will not be described in detail.

As illustrated in FIG. 1 , the image processing device 12 includes a storage section 124 . The processing unit 122 stores the position of each feature point in the image IM in the storage unit 124 as representing the posture of the driver 30 in the normal state. Specifically, the position of the pixel in the image IM corresponding to each feature point is saved in the storage unit 124 .

Subsequently, the processing unit 122 executes posture estimation processing for estimating whether the posture of the driver 30 is abnormal (STEP 3 in FIG. 3). The details of the orientation estimation process will be described later.

As a result of the posture estimation processing, when it is estimated that the posture of the driver 30 is not abnormal (NO in STEP4), the processing returns to STEP1, and the next image information I is accepted. The cycle in which the reception of the image information I is repeated can correspond to the frame rate of the imaging device 11 .

When it is determined that the posture of driver 30 is abnormal as a result of posture estimation processing (YES in STEP4), processing unit 122 outputs control information C from output unit 123 (STEP5). Control information C is transmitted to the control device 13 . The control information C may cause the controlled device 40 to perform the same action, or may cause the controlled device 40 to perform different actions depending on the type of the estimated abnormal posture.

Next, details of posture estimation processing (STEP 3 in FIG. 3) executed by the processing unit 122 will be described with reference to FIGS. 6 to 10. FIG. FIG. 6 shows an example of the flow of posture estimation processing.

First, the processing unit 122 identifies the distance in a prescribed direction between the plurality of pixels corresponding to the plurality of feature points in the image IM used to identify the plurality of feature points (STEP 11). In this example, the specified direction is the left-right direction of the vehicle 20 .

FIG. 7 shows an enlarged portion of the image IM illustrated in FIGS. 4 and 5 in which the head 31 of the driver 30 is reflected. In this example, the left ear feature point LE and the right ear feature point RE can also be detected by applying the skeletal model M. FIG. The left ear feature point LE is a point corresponding to the left ear of the model human body. The right ear feature point RE is a point corresponding to the right ear of the model human body. In FIG. 7 , the direction in which the straight line connecting the left ear feature point LE and the right ear feature point RE extends corresponds to the lateral direction of the vehicle 20 .

In this example, processing unit 122 identifies inter-pixel distance DP, which is the distance between pixel PL corresponding to left ear feature point LE and pixel PR corresponding to right ear feature point RE in the left-right direction of vehicle 20 . The inter-pixel distance DP is specified by the number of pixels.

Subsequently, as exemplified in FIG. 6, the processing unit 122 performs a process of identifying the correspondence relationship between the distance between two points in the image IM and the distance between two points in the real space (STEP 12).

The distance between two points in the image IM is specified as the inter-pixel distance DP described with reference to STEP11. In this example, the inter-pixel distance DP is regarded as the width of the head 31 of the driver 30 captured in the image IM in the lateral direction of the vehicle 20 .

The average dimensions of each part of the human body are statistically investigated and provided as a database. An example of such a database is the "AIST Human Body Size Database 1997-98" provided by the National Institute of Advanced Industrial Science and Technology. FIG. 8 illustrates the head width WH in the lateral direction of the human body obtained from such a database. It can be said that the head width WH corresponds to the real spatial distance DR between the left and right ears of an average human.

A value of the actual space distance DR corresponding to the width WH is stored in the storage unit 124 . The processing unit 122 reads the value of the actual space distance DR from the storage unit 124 and calculates a ratio P (=DR/DP) to the inter-pixel distance DP specified in STEP11. Accordingly, if the inter-pixel distance DP in the image IM is specified, the actual space distance DR in the vehicle interior 21 of the vehicle 20 can be calculated based on the ratio P (DR=DP×P).

In this example, based on the correspondence between the inter-pixel distance DP and the real space distance DR identified in STEP 12, it is estimated whether the driver 30 is in a "reclining position".

The “side leaning posture” is one of multiple types of “posture collapse patterns” defined by the Ministry of Land, Infrastructure, Transport and Tourism. A "side leaning posture" is defined as a state in which the upper half of the body of the driver continues to lean leftward or rightward.

FIG. 9 illustrates a "side leaning position" to the left. The leftward “lateral leaning posture” is defined as a state in which the amount of movement ΔDL of a specific position on the driver's face in the leftward direction of the vehicle 20 from the normal state exceeds a threshold. The threshold is, for example, 200 mm.

FIG. 10 exemplifies a "side leaning position" to the right. The "lateral leaning posture" to the right is defined as a state in which the movement amount ΔDR of a specific position on the driver's face in the right direction of the vehicle 20 from the normal state exceeds a threshold. The threshold is, for example, 200 mm.

In order to estimate whether or not the driver 30 is in the “side leaning posture”, the processing unit 122 identifies the position of the pixel corresponding to the head feature point H of the driver 30 reflected in the acquired image IM, and compares it with the position of the pixel corresponding to the normal head feature point H stored in the storage unit 124. Note that feature points such as the nose, eyes, and face may be used for estimation depending on the specifications of the applied skeleton model.

As a result of the comparison, the inter-pixel distance in the left-right direction of the vehicle 20 between the position of the pixel corresponding to the head feature point H of the driver 30 captured in the acquired image IM and the position of the pixel corresponding to the head feature point H in the normal state can be specified. This inter-pixel distance is converted into a real spatial distance within the vehicle 20 based on the ratio specified in STEP12. This real space distance corresponds to the amount of lateral movement of the vehicle 20 from the normal state of the head 31 of the driver 30 captured in the acquired image IM. That is, the processing unit 122 identifies the amount of movement of the head 31 of the driver 30 in the lateral direction of the vehicle 20 (STEP 13 in FIG. 6).

Subsequently, the processing unit 122 determines whether the converted real space distance exceeds the threshold (STEP 14). That is, it is determined whether or not the amount of movement ΔDL of the head 31 of the driver 30 to the left of the vehicle 20 exceeds a threshold value. Similarly, it is determined whether the amount of movement ΔDR of the head 31 of the driver 30 to the right of the vehicle 20 exceeds the threshold.

If it is determined that the converted actual space distance does not exceed the threshold (NO in STEP 14), processing unit 122 estimates that driver 30 is not in the "side leaning posture" (STEP 15). The processing is reflected in the determination of NO in STEP4 of FIG.

If it is determined that the converted real space distance exceeds the threshold (YES in STEP14 of FIG. 6), the processing unit 122 estimates that the driver 30 is in a "reclining posture" (STEP16). Specifically, when it is determined that the amount of movement ΔDL of the head 31 of the driver 30 to the left of the vehicle 20 exceeds a threshold value, it is estimated that the driver 30 is in a leftward “side leaning posture”. When it is determined that the amount of movement ΔDR of the head 31 of the driver 30 to the right of the vehicle 20 exceeds the threshold, it is estimated that the driver 30 is in a "side leaning posture" to the right. The processing is reflected in the YES determination in STEP4 of FIG.

The position of the seat 22 in the vehicle compartment 21 and how to sit on the seat 22 differ for each driver 30 . Therefore, how the driver 30 is reflected in the image IM acquired by the imaging device 11 also differs for each driver 30 . When trying to estimate an abnormal posture based on the amount of movement of the head 31 of the driver 30, such as the "side leaning posture", the difference in how the image is reflected in the image IM as described above can be a cause of lowering the certainty of estimation.

However, according to the configuration as described above, the ratio between the pixel-to-pixel distance corresponding to the plurality of feature points detected for each driver 30 captured in the image IM and the known real space distance between the plurality of feature points is specified, so the amount of movement of the head 31 in the image IM can be specified accurately for each driver 30. As a result, it is possible to increase the likelihood of estimating the posture abnormality of the driver 30 based on the image IM acquired by the imaging device 11 .

Especially in this example, the left ear feature point LE and the right ear feature point RE included in the head 31 of the driver 30 are selected as a plurality of feature points for specifying the ratio between the inter-pixel distance and the real space distance. It is known that there are individual differences in the physique of the driver 30 captured in the image IM acquired by the imaging device 11, but the individual differences in the dimensions of the head 31 are relatively small. Therefore, by using a plurality of feature points included in the head 31 to specify the ratio between the inter-pixel distance and the real spatial distance, it is possible to suppress the influence of individual differences when converting the inter-pixel distance into the real spatial distance.

Therefore, as long as there are a plurality of feature points included in the head 31, a combination different from the left ear feature point LE and the right ear feature point RE can be adopted according to the specifications of the skeletal model M to be applied. Examples include a combination of left eye and right eye feature points, a combination of nose and left ear feature points, a combination of head and right ear feature points, and a combination of left eye and left ear feature points.

Since the "side leaning posture" is estimated based on the amount of movement of the head 31 in the left-right direction of the vehicle 20, the combination of a plurality of feature points is not limited to that included in the head 31 as long as the distance in the left-right direction of the vehicle 20 can be given as a reference. For example, a combination of left shoulder feature points and right shoulder feature points, a combination of left ear feature points and left shoulder feature points, a combination of neck feature points and right shoulder feature points, and the like.

As exemplified in FIG. 6, the processing unit 122 of the image processing device 12 can estimate whether the driver 30 is in a "sideways posture."

The "sideways posture" is one of multiple types of "posture collapse patterns" defined by the Ministry of Land, Infrastructure, Transport and Tourism. A "sideways posture" is defined as a state in which the driver's upper body is tilted leftward or rightward and the driver's face is also tilted in the same direction.

FIG. 11 exemplifies a "sideways posture" to the left. The leftward “sideways posture” is defined as a state in which the leftward movement amount ΔDL of the vehicle 20 from the normal state at a specific position on the driver's face exceeds a threshold and the leftward roll angle θRL of the driver's face exceeds a threshold. A threshold value of ΔDL is, for example, 200 mm. A threshold value of θRL is, for example, 15°.

FIG. 12 exemplifies a "sideways posture" to the right. The rightward "sideways posture" is defined as a state in which the rightward movement amount ΔDR of the vehicle 20 from the normal state at a specific position on the driver's face exceeds a threshold, and the rightward roll angle θRR of the driver's face exceeds a threshold. A threshold value of ΔDR is, for example, 200 mm. A threshold value of θRR is, for example, 15°.

As used herein, the term "roll angle" refers to the angle of rotation about a longitudinally extending axis of the vehicle 20 . As used herein, the term “roll direction” refers to the direction of rotation about an axis extending longitudinally of vehicle 20 . Regarding the "roll direction", the counterclockwise direction viewed from the driver 30 is defined as the "left roll direction", and the clockwise direction viewed from the driver 30 is defined as the "right roll direction".

Specifically, when it is determined that the amount of movement of the head 31 of the driver 30 in the lateral direction of the vehicle 20 in the image IM exceeds the threshold value (YES in STEP 14 in FIG. 6), the processing unit 122 specifies the rotation angle θR of the head 31 in the roll direction of the vehicle 20 (STEP 17).

FIG. 13 shows an example of a technique for specifying the rotation angle θR of the vehicle 20 in the roll direction. In this example, the left-eye feature point LY and the right-eye feature point RY detected by applying the skeleton model M are used. The inclination angle of the straight line connecting the left-eye feature point LY and the right-eye feature point RY from the direction corresponding to the left-right direction of the vehicle 20 is equal to the rotation angle θR. Therefore, the processing unit 122 acquires the distance dLR between the left-eye feature point LY and the right-eye feature point RY in the horizontal direction of the vehicle 20 and the distance dUD between the left-eye feature point LY and the right-eye feature point RY in the vertical direction of the vehicle 20, and obtains the arctangent of the distance dLR and the distance dUD, thereby specifying the rotation angle θR.

Subsequently, the processing unit 122 determines whether the identified rotation angle θR exceeds the threshold (STEP 18 in FIG. 6). That is, it is determined whether the rotation angle θR of the head 31 of the driver 30 in the left roll direction exceeds the threshold value. Similarly, it is determined whether the rotation angle θR of the head 31 of the driver 30 in the right roll direction exceeds the threshold.

If it is determined that the rotation angle θR does not exceed the threshold (NO in STEP 18), the processing unit 122 estimates that the driver 30 is in the "side leaning posture" (STEP 16). The processing is reflected in the YES determination in STEP4 of FIG.

If it is determined that the rotation angle θR exceeds the threshold (YES in STEP18 of FIG. 6), the processing unit 122 estimates that the driver 30 is in a "sideways posture" (STEP19). Specifically, when it is determined that the rotation angle θRL of the head 31 of the driver 30 in the left roll direction of the vehicle 20 exceeds the threshold, it is estimated that the driver 30 is in a leftward “sideways posture”. If it is determined that the rotation angle θRR of the head 31 of the driver 30 in the right roll direction of the vehicle 20 exceeds the threshold, it is estimated that the driver 30 is in a "sideways posture" to the right. The processing is reflected in the YES determination in STEP4 of FIG.

FIG. 14 shows another example of the flow of posture estimation processing (STEP 3 in FIG. 3) executed by the processing unit 122 . Elements common to the flow of processing illustrated in FIG. 3 are assigned the same reference numerals, and repeated descriptions are omitted.

In this example, based on the correspondence between the inter-pixel distance DP and the real space distance DR specified in STEP 12, it is estimated whether the driver 30 is in the "bent down posture".

The “bent-down posture” is one of multiple types of “posture collapse patterns” defined by the Ministry of Land, Infrastructure, Transport and Tourism. A "bent down posture" is defined as a state in which the driver continues to fall forward with his/her face positioned near the steering wheel.

FIG. 15 exemplifies the "prostrate posture". The “bent down posture” is defined as a state in which the forward movement amount ΔDF of the vehicle 20 from the normal state exceeds the threshold, the downward movement amount ΔDD of the vehicle 20 exceeds the threshold, and the downward pitch angle θPD of the driver's face exceeds the threshold. A threshold value of ΔDF is, for example, 200 mm. A threshold value of ΔDD is, for example, 180 mm. A threshold value of θPD is, for example, 30°.

In this example, in order to estimate whether the driver 30 is in a “bent down posture,” the processing unit 122 identifies the position of the pixel corresponding to the head feature point H of the driver 30 appearing in the acquired image IM, and compares it with the position of the pixel corresponding to the normal head feature point H stored in the storage unit 124. Note that feature points such as the nose, eyes, and face may be used for estimation depending on the specifications of the applied skeleton model.

As a result of the comparison, the inter-pixel distance in the longitudinal direction of the vehicle 20 between the position of the pixel corresponding to the head feature point H of the driver 30 captured in the acquired image IM and the position of the pixel corresponding to the head feature point H in the normal state can be specified. This inter-pixel distance is converted into a real spatial distance within the vehicle 20 based on the ratio specified in STEP12. This real space distance corresponds to the amount of forward movement of the vehicle 20 from the normal state of the head 31 of the driver 30 captured in the acquired image IM. That is, the processing unit 122 identifies the amount of movement of the head 31 of the driver 30 in the forward direction of the vehicle 20 (STEP 23 in FIG. 14).

Subsequently, the processing unit 122 determines whether the converted real space distance exceeds the threshold (STEP 24). That is, it is determined whether the amount of movement ΔDF of the head 31 of the driver 30 toward the front of the vehicle 20 exceeds the threshold value.

If it is determined that the converted actual space distance does not exceed the threshold (NO in STEP 24), processing unit 122 estimates that driver 30 is not in the "bent down posture" (STEP 25). The processing is reflected in the determination of NO in STEP4 of FIG.

If it is determined that the converted actual space distance exceeds the threshold (YES in STEP24 of FIG. 14), processing unit 122 estimates that driver 30 is in a "bent down posture" (STEP26). The processing is reflected in the YES determination in STEP4 of FIG.

That is, in this example, among the movement amount ΔDF, the movement amount ΔDD, and the pitch angle θPD included in the definition of the "bent down posture", only the movement amount ΔDF is focused on and processed. However, even with such a configuration, it is possible to estimate the "push-down posture".

The processing unit 122 having each function described so far can be implemented by a general-purpose microprocessor operating in cooperation with a general-purpose memory. A CPU, MPU, and GPU can be exemplified as a general-purpose microprocessor. Examples of general-purpose memory include ROM and RAM. In this case, the ROM may store a computer program for executing the above-described process. ROM is an example of a storage medium that stores computer programs. The processor designates at least part of the computer program stored in the ROM, develops it on the RAM, and cooperates with the RAM to execute the above-described processing. The above computer program may be pre-installed in a general-purpose memory, or may be downloaded from an external server via a communication network and installed in a general-purpose memory. In this case, the external server is an example of a storage medium storing computer programs.

The processing unit 122 may be implemented by a dedicated integrated circuit such as a microcontroller, ASIC, FPGA, etc., capable of executing the above computer programs. In this case, the computer program is pre-installed in a storage element included in the dedicated integrated circuit. The storage element is an example of a storage medium storing a computer program. Processing unit 122 may also be implemented by a combination of a general-purpose microprocessor and a dedicated integrated circuit.

Storage unit 124 can be realized by a semiconductor memory or a hard disk device. The storage unit 124 may be implemented by the general-purpose memory or storage element described above.

The above embodiments are merely examples for facilitating understanding of the present invention. The configurations according to the above embodiments can be modified and improved as appropriate without departing from the scope of the present invention.

In the above-described embodiment, when performing the posture estimation process illustrated in FIG. 6, the ratio between the inter-pixel distances related to the plurality of feature points and the known real space distance is specified, the number of pixels corresponding to the displacement of the feature points related to the head 31 reflected in the image IM is converted into the actual movement amount of the head in the vehicle interior 21 based on the ratio, and it is determined whether the converted actual movement amount exceeds the threshold. However, by specifying the ratio, it is also possible to specify the number of pixels corresponding to the threshold for the amount of movement. In this case, the abnormal posture may be estimated based on whether the number of pixels corresponding to the displacement of the feature points of the head 31 reflected in the image IM exceeds the number of pixels corresponding to the threshold.

In the above-described embodiment, the abnormal postures that the driver 30 can take are exemplified by the “side leaning posture”, the “sideways falling posture”, and the “bent over posture”. However, if the inter-pixel distance can be identified using a plurality of feature points that provide a reference for a specified direction of the vehicle 20, and if the actual space distance between the plurality of feature points is known, an appropriate abnormal posture that can be based on the amount of movement in the specified direction can be estimated based on the ratio specified from the two.

The image processing device 12 may be mounted on the vehicle 20, or may be provided as an external device capable of communicating with the vehicle 20 via a wireless communication network based on appropriate standards. When image processing device 12 is mounted on vehicle 20, image processing by image processing device 12 and control processing by control device 13 may be performed by a common device or element. When the image processing device 12 is provided as an external device capable of communicating with the vehicle 20 via a wireless communication network, the image information I output from the imaging device 11 is transmitted to the image processing device 12 via the wireless communication network. Control information C that can be output as a result of posture estimation processing by the processing unit 122 is transmitted to the control device 13 via the wireless communication network.

The abnormality estimation system 10 can also be applied to mobile bodies other than the vehicle 20 . Examples of other mobile objects include railroads, aircraft, ships, and the like.

10: Abnormality estimation system 11: Imaging device 12: Image processing device 121: Receiving unit 122: Processing unit 13: Control device 20: Vehicle 30: Driver 31: Head 40: Controlled device DP: Distance between pixels DR: Real space distance I: Image information IM: Image LE: Left ear feature point PL, PR: Pixel RE: Right ear feature point WH: Head width ΔDL, ΔDR, ΔDD : Movement amount, θR: Rotation angle in roll direction

Claims (8)

a reception unit that receives image information corresponding to an image in which the driver of the moving object is captured;
a processing unit that estimates whether the posture of the driver is abnormal based on the image information;
and
The processing unit, based on the image information,
identifying a plurality of feature points on the driver's body reflected in the image;
identifying a ratio of a distance between pixels corresponding to the feature points in a specified direction to a known distance between the feature points in the specified direction in real space;
identifying an amount of movement in the prescribed direction of the position of the driver's head reflected in the image;
estimating that the posture is abnormal when the movement amount in the specified direction of the position of the driver's head in the real space specified based on the ratio exceeds a threshold;
Image processing device. the defined direction corresponds to the left-right direction of the moving body,
The processing unit estimates that the driver is in a side-lying posture when the amount of movement in the left-right direction exceeds the threshold.
The image processing apparatus according to claim 1. The processing unit is
identifying the rotation angle of the driver's head reflected in the image in the roll direction of the moving body;
estimating that the driver is lying sideways when the amount of movement in the left-right direction exceeds the threshold and the rotation angle exceeds the threshold;
The image processing apparatus according to claim 2. the defined direction corresponds to the forward direction of the moving object;
The processing unit estimates that the driver is in a prone posture when the amount of movement in the forward direction exceeds the threshold.
The image processing apparatus according to claim 1. each of the plurality of feature points corresponds to a point included in the head;
The image processing apparatus according to any one of claims 1 to 4. configured to be mounted on the mobile body,
The image processing apparatus according to any one of claims 1 to 5. A computer program executable by a processing unit of an image processing device,
By being executed, the image processing device
receiving image information corresponding to an image in which a driver of a mobile object is captured;
Based on the image information,
specifying a plurality of feature points on the driver's body reflected in the image;
determining a ratio of a distance between pixels corresponding to the feature points in a specified direction to a known distance between the feature points in the specified direction in real space;
specifying a movement amount in the specified direction of the position of the driver's head reflected in the image;
estimating that the driver's posture is abnormal when the movement amount in the specified direction of the position of the driver's head in the real space specified based on the ratio exceeds a threshold;
computer program. an imaging device that outputs image information corresponding to an image in which a driver of a mobile object is captured;
an image processing device that estimates whether the posture of the driver is abnormal based on the image information;
a control device that controls the operation of a controlled device mounted on the moving body based on the estimation result that the posture is abnormal by the image processing device;
and
The image processing device, based on the image information,
identifying a plurality of feature points on the driver's body reflected in the image;
identifying a ratio of a distance between pixels corresponding to the feature points in a specified direction to a known distance between the feature points in the specified direction in real space;
identifying an amount of movement in the prescribed direction of the position of the driver's head reflected in the image;
estimating that the posture is abnormal when the movement amount in the specified direction of the position of the driver's head in the real space specified based on the ratio exceeds a threshold;
Anomaly estimation system.

Images