WO2021156914A1 - Attention direction determination device and attention direction determination method - Google Patents

Abstract

A purpose of the present invention is to provide an attention direction determination device and an attention direction determination method that determine the direction in which the attention of an occupant is directed. This attention direction determination device comprises: an image acquisition unit that acquires an image of an occupant in a vehicle, said image having been captured by a camera; a face characteristic point estimation unit that estimates a characteristic point of the face of the occupant, on the basis of the image that was acquired by the image acquisition unit; a skeleton estimation unit that estimates the direction in which a hand of the occupant is extending, on the basis of the image that was acquired by the image acquisition unit; a face orientation detection unit that detects the face orientation of the occupant, on the basis of the face characteristic point that was estimated by the face characteristic point estimation unit; and an attention direction determination unit that determines the direction in which the attention of the occupant attention is directed, on the basis of the direction in which the hand of the occupant is extended that was estimated by the skeleton estimation unit and the face orientation of the occupant that was detected by the face orientation detection unit.

Description

Attention direction determination device and attention direction determination method

The present disclosure relates to a caution direction determination device and a caution direction determination method for determining a direction in which an occupant in a vehicle is paying attention.

Conventionally, the human body feature points of the occupants in the vehicle are detected from the images acquired by the camera mounted in the vehicle, and the behavior of the occupants in the vehicle is predicted based on the position change of the detected human feature points. The technology is disclosed. (See, for example, Patent Document 1)

Japanese Unexamined Patent Publication No. 2011-123733

In the conventional technology, the behavior of the occupant in the vehicle is predicted from the position change of the human feature point, but since the direction in which the occupant is paying attention is not estimated, it is related to the direction in which the occupant is paying attention. There is the problem of predicting no behavior. For example, in the prior art, even when the occupant changes the sitting posture without paying attention to the direction of the in-vehicle device, the occupant's behavior is the behavior of operating the in-vehicle device due to the change in the position of the human body feature point. There is a problem of predicting that.

The present disclosure has been made to solve the above-mentioned problems, and an object of the present disclosure is to provide a caution direction determination device and a caution direction determination method for determining a direction in which an occupant is paying attention.

The caution direction determination device according to the present disclosure is an image acquisition unit that acquires an image of the occupant in the vehicle taken by the camera, and a facial feature that estimates the feature points of the occupant's face based on the image acquired by the image acquisition unit. Based on the point estimation unit, the skeleton estimation unit that estimates the direction in which the occupant's hand is extending based on the image acquired by the image acquisition unit, and the facial feature points estimated by the facial feature point estimation unit, the occupant's The occupant is paying attention based on the face orientation detection unit that detects the face orientation, the occupant's hand extension direction estimated by the skeleton estimation unit, and the occupant's face orientation detected by the face orientation detection unit. A caution direction determining unit for determining a direction is provided.

The attention direction determination method according to the present disclosure includes a step of acquiring an image of the occupant in the vehicle taken by the camera, a step of estimating a feature point of the occupant's face based on the image, and a step of estimating the occupant's face based on the image. An occupant based on a step of estimating the direction in which the hand is extended, a step of detecting the occupant's face orientation based on facial feature points, and an occupant's hand extension direction and the occupant's face orientation. Has a step of determining the direction in which the attention is directed.

According to the attention direction determination device according to the present disclosure, it is possible to determine the direction in which the occupant is paying attention.

According to the attention direction determination method according to the present disclosure, it is possible to determine the direction in which the occupant is paying attention.

It is a block diagram which shows an example of the structure of the attention direction determination apparatus which concerns on Embodiment 1 of this disclosure. It is a figure which shows an example of the image of the occupant sitting in the passenger seat which concerns on Embodiment 1 of this disclosure. It is a figure which shows an example of the feature point of the face of the occupant which concerns on Embodiment 1 of this disclosure. It is a figure which shows an example of a plurality of parts of an occupant in the image which concerns on Embodiment 1 of this disclosure. It is a figure which shows an example of a plurality of parts of an occupant in the image which concerns on Embodiment 1 of this disclosure. It is a figure which shows an example of a plurality of parts of an occupant in the image which concerns on Embodiment 1 of this disclosure. It is a figure which shows an example of a plurality of parts of an occupant in the image which concerns on Embodiment 1 of this disclosure. It is a figure which shows an example for detecting the extending direction of the occupant's hand in the image which concerns on Embodiment 1 of this disclosure. It is a figure which shows an example of the face orientation of the occupant which concerns on Embodiment 1 of this disclosure. It is a figure which shows an example of the face orientation of the occupant in consideration of the line-of-sight direction angle of the occupant which concerns on Embodiment 1 of this disclosure. It is a flowchart which shows an example of the operation of the attention direction determination apparatus which concerns on Embodiment 1 of this disclosure. It is a block diagram which shows an example of the structure of the attention direction determination apparatus which concerns on Embodiment 2 of this disclosure. It is a flowchart which shows an example of the operation of the attention direction determination apparatus which concerns on Embodiment 2 of this disclosure. It is a block diagram which shows an example of the hardware composition of the attention direction determination apparatus which concerns on Embodiments 1 and 2 of this disclosure. It is a block diagram which shows an example of the hardware composition of the attention direction determination apparatus which concerns on Embodiments 1 and 2 of this disclosure.

Hereinafter, in order to explain the present disclosure in more detail, a mode for carrying out the present disclosure will be described with reference to the attached drawings. The embodiments shown below are examples, and the present disclosure is not limited to these embodiments.

Embodiment 1.
FIG. 1 is a block diagram showing an example of the configuration of the attention direction determination device 100 according to the present embodiment. Further, it is assumed that the caution direction determination device 100 is mounted in the vehicle.

As shown in FIG. 1, the attention direction determination device 100 includes an image acquisition unit 1, a face feature point estimation unit 2, a head position estimation unit 3, a site estimation unit 4, a skeleton estimation unit 5, and a face orientation detection. A unit 6, a caution direction determination unit 7, and a determination result output unit 8 are provided. Further, the image acquisition unit 1 is connected to the camera 9. The camera 9 is mounted in the vehicle. The determination result output unit 8 is connected to the control unit 10. The control unit 10 receives the determination result output by the determination result output unit 8 and controls the in-vehicle device or the like based on the determination result. Here, the in-vehicle device is, for example, an in-vehicle lighting device, a navigation device, an audio device, or the like.

The image acquisition unit 1 acquires an image of the occupant in the vehicle taken by the camera 9. The camera 9 photographs the occupants seated in the driver's seat, the passenger seat, and the rear seat, respectively. FIG. 2 is a diagram showing an example of an image of an occupant seated in the passenger seat. The example of FIG. 2 shows a case where the occupant is seated in the passenger seat, but the present invention is not limited to this. The image taken by the camera 9 may include an occupant seated in the driver's seat and an occupant seated in the rear seat.

The facial feature point estimation unit 2 estimates the facial feature points of the occupant based on the image acquired by the image acquisition unit 1. Specifically, the face feature point estimation unit 2 collates the image acquired by the image acquisition unit 1 with a learning dictionary (not shown) prepared in advance, and the face of the occupant included in the image acquired by the image acquisition unit 1. Estimate facial feature points that indicate positions such as, eyes, and nose. For example, the facial feature point estimation unit 2 includes the position and size of the occupant's pupil, the distance between the centers of the pupils of both eyes, the position of both ends of the eyebrows, the position of both ends of the eyes, the position of the tip of the nose, and both ends of the mouth. The position and the like are estimated as facial feature points, but the present invention is not limited to these, and other facial feature points may be estimated. In the learning dictionary, the feature points of the faces of a plurality of people are recorded. When the image includes a plurality of occupants, the face feature point estimation unit 2 estimates the facial feature points of each occupant.

FIG. 3 is a diagram showing an example of the feature points of the occupant's face. The occupant shown in FIG. 3 is seated in the passenger seat. In FIG. 3, the line surrounding the occupant's face with a square indicates the position of the occupant's face. The dashed circle in the center of the occupant's eyes indicates the position and size of the occupant's pupil. The plurality of dots shown on the occupant's face indicate the positions of both ends of the eyebrows, the positions of both ends of the eyes, the position of the tip of the nose, and the positions of both ends of the mouth. The eye width is the distance between the occupant's eyes, specifically the distance between the centers of each pupil. The facial feature point estimation unit 2 may estimate not only the facial feature points shown in FIG. 3 but also other facial feature points.

The head position estimation unit 3 estimates the relative position of the occupant's head with respect to the camera 9 based on the facial feature points estimated by the face feature point estimation unit 2. Specifically, the head position estimation unit 3 estimates the physical relative position of the occupant's head with respect to the camera 9 based on the distance between the occupant's eyes or the size of the pupil estimated by the face feature point estimation unit 2. .. That is, the head position estimation unit 3 estimates the distance between the camera 9 and the occupant's head.

In the present embodiment, the relative position of the occupant's head with respect to the camera 9 is estimated based on the distance between the occupant's eyes or the size of the pupil, which is less affected by the physical disparity. Therefore, regardless of the physique of the occupant, the relative position of the occupant's head with respect to the camera 9 can be estimated accurately. Further, even when the seat position in which the occupant is seated changes, the relative position of the occupant's head with respect to the camera 9 can be accurately estimated according to the change.

The part estimation unit 4 estimates the sizes of a plurality of parts of the occupant in the image based on the image acquired by the image acquisition unit 1. Specifically, the site estimation unit 4 collates the image acquired by the image acquisition unit 1 with a learning dictionary (not shown) prepared in advance, and the neck and shoulders of the occupant included in the image acquired by the image acquisition unit 1. Estimate the size and position of each part in the image, such as, arms, hands, and upper body. In the learning dictionary, the size and position of each part of a plurality of people in the image are recorded. When a plurality of occupants are included in the image, the part estimation unit 4 estimates the sizes of the plurality of parts of each occupant. Further, the unit of the size of each part of the occupant in the image estimated by the part estimation unit 4 is pix (pixel).

4 to 7 are diagrams showing an example of the size and position of a plurality of parts of the occupant in the image acquired by the image acquisition unit 1. 4 to 6 show the size and position of each part of the occupants having different physiques. In FIGS. 4 to 7, the straight line connecting the points corresponds to the occupant's part, and the unit of the size of each part is pix (pixel). For example, the double-headed arrow shown in FIG. 7 indicates the shoulder width (n pixels) of the occupant.

The skeleton estimation unit 5 determines the skeleton of the occupant based on the relative position of the occupant's head with respect to the camera 9 estimated by the head position estimation unit 3 and the size of each part of the occupant in the image estimated by the part estimation unit 4. And estimate the direction in which the occupant's hand is extending. Specifically, first, the skeleton estimation unit 5 converts the size of each part of the occupant in the image estimated by the part estimation unit 4 into the actual size of each part of the occupant. When there are a plurality of occupants, the skeleton estimation unit 5 estimates the actual size of each part of each occupant.

Here, a method in which the skeleton estimation unit 5 converts the occupant's shoulder width in the image estimated by the site estimation unit 4 into an actual shoulder width will be described. The skeleton estimation unit 5 calculates the actual shoulder width d according to the following equation (1).
d = Mn / {2xtan (θ / 2)} ... (1)

In formula (1), n indicates the size of the shoulder width in the image. M indicates the width of the image, which is determined by the angle of view θ of the camera 9. x indicates the distance between the camera 9 and the occupant's head, and corresponds to the relative position of the occupant's head with respect to the camera 9 estimated by the head position estimation unit 3. In this way, the skeleton estimation unit 5 corrects the size of each part in the image estimated by the part estimation unit 4 with the relative position of the occupant's head estimated by the head position estimation unit 3, thereby causing the occupant to Calculate the actual size of each part.

As shown in the above equation (1), the skeleton estimation unit 5 uses the relative position of the occupant's head with respect to the camera 9 estimated by the head position estimation unit 3 to determine the actual size of each part of the occupant. It is calculated. Therefore, even when the seat position in which the occupant is seated changes, the skeleton estimation unit 5 can accurately calculate the actual size of each portion of the occupant. In the above description, the case of calculating the shoulder width of the occupant has been described, but the calculation can be performed in the same manner for other parts of the occupant. For example, the skeleton estimation unit 5 may calculate the sitting height of the occupant, the size of the face, and the length of the arm.

The size of each part of the occupant in the image can be converted to the actual size of each part of the occupant because all the parts of the occupant are present at the relative position of the occupant's head with respect to the camera 9. This is because it is assumed that there is. It can be assumed that the shoulders, spine, and face are located approximately at the same positions as the occupant's head relative to the camera 9, but this is not the case because the arms can be moved significantly around the shoulders. With respect to the arm, the site estimation unit 4 may estimate the maximum arm size as the occupant's arm size when the arm size becomes maximum due to a change over time in the image. In this case, the skeleton estimation unit 5 converts the arm size in the image estimated by the site estimation unit 4 into the actual arm size.

Next, the skeleton estimation unit 5 estimates the skeleton of the occupant based on the actual size of each part of the occupant. Specifically, the skeleton estimation unit 5 inputs the actual size of each part of the occupant into a learned learner (not shown) that has been machine-learned to estimate the skeleton of the occupant, and learns. By executing the arithmetic processing of the instrument, the estimation result of the skeleton of the occupant is acquired from the learner. When there are a plurality of occupants, the skeleton estimation unit 5 estimates the skeleton of each occupant.

The learning device has a learning dictionary (not shown) recorded by associating the actual size of each part of a plurality of persons with the skeleton of each person. For example, when the skeleton estimation unit 5 inputs the occupant's sitting height, shoulder width, face size, and arm length into the learning device, the learning device collates the actual size of each input part with the learning dictionary. Then, it is presumed that the skeleton with the highest likelihood is the skeleton of the occupant having each input part. In this way, there is a correlation between the actual size of each part of the occupant and the skeleton of the occupant. The skeleton estimation unit 5 acquires the estimation result of the skeleton of the occupant from the learner.

Next, the skeleton estimation unit 5 estimates the direction in which the occupant's hand extends from the estimated occupant's skeleton. For example, the skeleton estimation unit 5 detects the direction in which the occupant's hand is extended by detecting the occupant's hand in the preset skeleton detection area. FIG. 8 is a diagram showing an example for detecting the extending direction of the occupant's hand in the image acquired by the image acquisition unit 1.

In the example shown in FIG. 8, the occupant is seated in the passenger seat. In FIG. 8, each area surrounded by a dotted line indicates each preset skeleton detection area. In the example shown in FIG. 8, the areas set as the skeleton detection area are the adjacent seat area for the occupant to be detected, the rear seat area for the occupant to be detected, and the doorknob area of the door for the occupant to be detected to get on or off. Is. Each skeleton detection area set in the example shown in FIG. 8 is an example in the case where the occupant is seated in the passenger seat, and is appropriately set according to the seating position of the occupant to be detected. Further, the size of each detection area can be arbitrarily set, and is appropriately set according to the occupant to be detected.

In the example shown in FIG. 8, when the occupant's hand is detected in the skeleton detection area A, the skeleton estimation unit 5 estimates that the occupant's hand extends toward the adjacent seat. In the example shown in FIG. 8, when the occupant's hand is detected in the skeleton detection area B, the skeleton estimation unit 5 estimates that the occupant's hand extends toward the rear seat. In the example shown in FIG. 8, when the occupant's hand is detected in the skeleton detection area C, the skeleton estimation unit 5 estimates that the occupant's hand extends in the direction of the doorknob. If the occupant's hand is detected outside the preset skeleton detection area, or if the occupant's hand is detected in a plurality of skeleton detection areas, the skeleton estimation unit 5 cannot estimate the direction of the occupant's hand. Judge. Although FIG. 8 shows an example in which the occupant to be detected is seated in the passenger seat, the occupant to be detected is not limited to the occupant seated in the passenger seat.

The face orientation detection unit 6 detects the face orientation based on the positional relationship of the face, eyes, nose, etc., which are the facial feature points of the occupant estimated by the face feature point estimation unit 2. For example, the face orientation detection unit 6 extracts feature points corresponding to the face portion from the face image of the occupant, and calculates the face orientation angle of the occupant based on the positional relationship of the face portion indicated by the feature points. The face orientation angle can be detected using, for example, a known technique, the POSIT (Pose from Orthography and Scaling with Iterations) algorithm.

The face orientation detection unit 6 detects the occupant's face orientation from the detected face orientation angle. For example, the face orientation detection unit 6 faces the occupant in the direction of the adjacent seat with respect to the occupant to be detected, is oriented toward the rear seat with respect to the occupant to be detected, or the occupant to be detected is facing the rear seat. Detects whether the door is facing the door knob for getting on or off. Here, FIG. 9 is a diagram showing an example of the occupant's face orientation, showing a state in which the occupant's head is viewed from directly above. In FIG. 9, the face orientation of the occupant is indicated by a chain line, and when the occupant's face orientation coincides with the traveling direction Y of the vehicle, the face orientation angle is set to 0 °.

Next, an example of detecting the face orientation of the occupant by the face orientation detecting unit 6 will be described with reference to FIG. In the example shown in FIG. 9, the occupant is seated in the passenger seat.
As shown in FIG. 9A, when the occupant's face orientation is rotating to the right from the vehicle's traveling direction Y, the face orientation detection unit is when the face orientation angle is 0 ° to a predetermined angle α or less. 6 determines that the occupant is facing the direction of the next seat.
As shown in FIG. 9B, when the occupant's face orientation is rotating to the left from the vehicle's traveling direction Y, the face orientation detection unit is when the face orientation angle is 0 ° to a predetermined angle β or less. 6 determines that the occupant is facing the door knob direction.
As shown in FIG. 9C, when the occupant's face orientation is rotating to the right from the traveling direction Y of the vehicle by exceeding a predetermined angle α, the face orientation detection unit 6 causes the occupant to face the rear seat direction. Judge that
As shown in FIG. 9D, when the occupant's face orientation is rotating to the left from the traveling direction Y of the vehicle by exceeding a predetermined angle β, the face orientation detection unit 6 causes the occupant to face the rear seat direction. Judge that

Here, as the predetermined angle α and the predetermined angle β, the angles that can be taken in the process of facing the rear seat direction are set. The predetermined angle α and the predetermined angle β are set to, for example, 50 degrees. The predetermined angle α and the predetermined angle β can be arbitrarily set, and different values can be set between the predetermined angle α and the predetermined angle β. Although FIG. 9 shows an example in which the occupant to be detected is seated in the passenger seat, the occupant to be detected is not limited to the occupant seated in the passenger seat. Further, the face orientation of the occupant determined by the face orientation detection unit 6 is determined according to the seating position of the occupant to be detected.

Further, the face orientation detection unit 6 may be configured to detect the visual field or the line of sight of the occupant from the feature points estimated by the face feature point estimation unit 2. That is, the face orientation detection unit 6 calculates the occupant's face orientation angle and the occupant's field of view or line of sight based on the occupant's face feature points estimated by the face feature point estimation unit 2, and calculates the occupant's face orientation. The configuration may be such that the face orientation of the occupant is detected based on the angle and the occupant's field of view or line of sight. For example, the face orientation detection unit 6 includes the position and size of the occupant's pupils estimated by the face feature point estimation unit 2, the distance between the centers of the pupils of both eyes, the distance between both eyes, the position at both ends of the eyebrows, and the position at both ends of the eyes. Detect the visual field or line of sight from. The visual field or line of sight can be detected using a known line of sight detection process.

The face orientation detection unit 6 detects the line-of-sight direction angle as a field of view or a line of sight. The line-of-sight angle is an angle indicating the direction of the occupant's line of sight with respect to the occupant's face-facing angle. When the face orientation detection unit 6 is configured to detect the line-of-sight direction angle of the occupant, the face orientation detection unit 6 calculates the face-direction angle in consideration of the line-of-sight direction angle of the occupant and detects the face orientation. For example, when the rotation direction of the occupant's face orientation angle with respect to the vehicle's traveling direction Y and the rotation direction of the occupant's line-of-sight direction angle with respect to the occupant's face orientation angle, the face orientation detection unit 6 is the occupant's face. The value obtained by adding the occupant's line-of-sight direction angle to the direction angle is calculated as the face direction angle in consideration of the occupant's line-of-sight direction angle. On the other hand, when the rotation direction of the occupant's face orientation angle with respect to the vehicle's traveling direction Y and the rotation direction of the occupant's line-of-sight direction angle with respect to the occupant's face orientation angle are different, the face orientation detection unit 6 faces the occupant's face. The value obtained by subtracting the occupant's line-of-sight direction angle from the angle is calculated as the face orientation angle in consideration of the occupant's line-of-sight direction angle. That is, the face orientation detection unit 6 detects the face orientation based on the face orientation angle in consideration of the occupant's line-of-sight direction angle calculated by further utilizing the occupant's line-of-sight direction angle in addition to the occupant's face-direction angle. ..

Here, FIG. 10 is a diagram showing an example of the occupant's face orientation in consideration of the occupant's line-of-sight direction angle, and shows a state in which the occupant's head is viewed from directly above. In FIG. 10, the face orientation of the occupant is indicated by a chain line, and the line-of-sight direction angle of the occupant is indicated by a broken line. If the occupant's face orientation coincides with the vehicle's traveling direction Y, the face orientation angle is set to 0 °. If the occupant's face orientation and the occupant's line-of-sight direction angle match, the occupant's line-of-sight direction angle is set to 0 °.

Next, an example of occupant's face orientation detection in consideration of the occupant's line-of-sight direction angle by the face orientation detection unit 6 will be described with reference to FIG. In the example shown in FIG. 10, the occupant is seated in the passenger seat. In the example shown in FIG. 10, the face-facing angle of the occupant is A ° to the right from the traveling direction Y of the vehicle. The line-of-sight direction angle of the occupant is a ° to the left of the occupant's face-facing angle. That is, in the example shown in FIG. 10, the face orientation detection unit 6 calculates A ° −a ° as the face orientation angle in consideration of the line-of-sight direction angle of the occupant, and the face orientation angle A in consideration of the line-of-sight direction angle of the occupant. When ° -a ° is from 0 ° to a predetermined angle α or less, it is determined that the occupant is facing the direction of the next seat.

That is, when the face orientation angle considering the occupant's line-of-sight direction angle is 0 ° to a predetermined angle α or less, the face orientation detection unit 6 determines that the occupant is facing the adjacent seat direction and determines the occupant's line-of-sight direction angle. If the considered face-facing angle exceeds the predetermined angle α, it is determined that the occupant is facing the rear seat direction. Further, the face orientation detection unit 6 determines that the occupant is facing the door knob direction when the face orientation angle considering the occupant's line-of-sight direction angle is 0 ° to a predetermined angle β or less, and determines the occupant's line-of-sight direction angle. If the considered face-facing angle exceeds the predetermined angle β, it is determined that the occupant is facing the rear seat direction.

The attention direction determination unit 7 determines the direction in which the occupant is paying attention based on the direction in which the occupant's hand is extended estimated by the skeleton estimation unit 5 and the occupant's face orientation estimated by the face orientation detection unit 6. judge. Further, the attention direction determination unit 7 outputs the determined determination result to the determination result output unit 8. For example, in the attention direction determination unit 7, when the occupant's hand extension direction estimated by the skeleton estimation unit 5 and the occupant's face orientation estimated by the face orientation detection unit 6 are the same direction, the occupant It is determined that the direction of attention is the direction estimated by the skeleton estimation unit 5 and the face orientation detection unit 6. On the other hand, in the attention direction determination unit 7, if the direction in which the occupant's hand is extended estimated by the skeleton estimation unit 5 and the occupant's face orientation estimated by the face orientation detection unit 6 are not the same, the skeleton estimation unit 5 may perform the skeleton estimation unit 5. If the estimated occupant's hand extension direction cannot be estimated, or if the occupant's face orientation estimated by the face orientation detection unit 6 cannot be estimated, it is determined that the attention direction cannot be determined.

The determination result output unit 8 outputs the determination result determined by the attention direction determination unit 7 to the control unit 10. Here, the determination result determined by the attention direction determination unit 7 also includes the determination result that the attention direction determination unit 7 determines that the attention direction determination is impossible.

The control unit 10 receives the determination result output from the determination result output unit 8 and controls the in-vehicle device or the like corresponding to the determination result. For example, when the control unit 10 receives a determination result that the direction in which the occupant is paying attention is the direction of the next seat, the control unit 10 controls the occupant to be detected to turn on the lighting for illuminating the next seat. Further, when the control unit 10 receives the determination result that the direction in which the occupant is paying attention is the rear seat direction, the control unit 10 controls the occupant to be detected to turn on the illumination for illuminating the rear seat. Further, when the control unit 10 receives the determination result that the direction in which the occupant is paying attention is the doorknob direction, the control unit 10 controls to turn on the lighting that illuminates the feet of the occupant to be detected. The control unit 10 does not control the in-vehicle device or the like when the received determination result cannot determine the caution direction.

FIG. 11 is a flowchart showing an example of the operation of the attention direction determination device 100 shown in FIG.

In step S101, the image acquisition unit 1 acquires an image of the occupant in the vehicle taken by the camera 9. In step S102, the facial feature point estimation unit 2 estimates the facial feature points of the occupant based on the image acquired by the image acquisition unit 1.

In step S103, the head position estimation unit 3 estimates the relative position of the occupant's head with respect to the camera 9 based on the facial feature points estimated by the face feature point estimation unit 2. In step S104, the part estimation unit 4 estimates the sizes of a plurality of parts of the occupant in the image based on the image acquired by the image acquisition unit 1.

In step S105, the skeleton estimation unit 5 estimates the skeleton of the occupant based on the relative position of the occupant's head estimated by the head position estimation unit 3 and the size of each part estimated by the site estimation unit 4. .. In step S106, the skeleton estimation unit 5 determines whether the estimated skeleton is detected in the skeleton detection area, and estimates the direction in which the occupant's hand is extended.

In step S107, the face orientation detection unit 6 detects the occupant's face orientation based on the positional relationship of the occupant's face feature points such as the face, eyes, and nose estimated by the face feature point estimation unit 2.

In step S108, the attention direction determination unit 7 directs the occupant's attention based on the direction in which the occupant's hand is extended estimated by the skeleton estimation unit 5 and the occupant's face orientation estimated by the face orientation detection unit 6. Determine the direction in which you are.

In step S109, the determination result output unit 8 outputs the determination result determined by the attention direction determination unit 7. In the example of FIG. 1, the determination result output unit 8 outputs to the control unit 10 the direction in which the occupant determined by the attention direction determination unit 7 is paying attention.

The attention direction determining device according to the first embodiment estimates the feature points of the occupant's face based on the image acquisition unit that acquires the image of the occupant in the vehicle taken by the camera and the image acquired by the image acquisition unit. Based on the facial feature point estimation unit, the skeleton estimation unit that estimates the direction in which the occupant's hand is extending based on the image acquired by the image acquisition unit, and the facial feature points estimated by the facial feature point estimation unit. , The occupant pays attention based on the face orientation detection unit that detects the occupant's face orientation, the occupant's hand extension direction estimated by the skeleton estimation unit, and the occupant's face orientation detected by the face orientation detection unit. It is provided with a caution direction determining unit for determining the direction in which the vehicle is directed.

Further, in the caution direction determination unit of the caution direction determination device according to the first embodiment, the direction in which the occupant's hand is extended estimated by the skeleton estimation unit and the occupant's face orientation detected by the face orientation detection unit are the same. In the case of the direction of, the direction in which the occupant is paying attention is determined.

Further, in the caution direction determination unit of the caution direction determination device according to the first embodiment, the direction in which the occupant's hand is extended is the direction of the adjacent seat estimated by the skeleton estimation unit, and the occupant's face detected by the face orientation detection unit. When the direction is the direction of the next seat, it is determined that the direction in which the occupant is paying attention is the direction of the next seat.

Further, in the caution direction determination unit of the caution direction determination device according to the first embodiment, the direction in which the occupant's hand is extended as estimated by the skeleton estimation unit is the rear seat direction, and the occupant's face orientation detection unit detects it. When the face direction is the rear seat direction, it is determined that the direction in which the occupant is paying attention is the rear seat direction.

Further, in the caution direction determination unit of the caution direction determination device according to the first embodiment, the direction in which the occupant's hand is extended is the door knob direction estimated by the skeleton estimation unit, and the occupant's face detected by the face orientation detection unit. When the direction is the door knob direction, it is determined that the direction in which the occupant is paying attention is the door knob direction.

Further, the attention direction determination device according to the first embodiment includes a head position estimation unit that estimates the relative position of the occupant's head with respect to the camera based on the facial feature points estimated by the face feature point estimation unit. Based on the image acquired by the image acquisition unit, it has a part estimation unit that estimates the size of a plurality of parts of the occupant in the image, and the skeleton estimation unit is the relative of the occupant's head estimated by the head position estimation unit. The occupant's skeleton is estimated based on the target position and the size of each part estimated by the part estimation unit, and the direction in which the occupant's hand is extended is estimated from the estimated occupant's skeleton estimation result.

Further, the attention direction determination device according to the first embodiment is further provided with a determination result output unit that outputs a direction in which the occupant determined by the attention direction determination unit is paying attention.

With the above configuration, the attention direction determination device 100 according to the first embodiment can determine the direction in which the occupant is paying attention. Further, since the face orientation detection unit 6 detects the face orientation based on the positional relationship of the occupant's face feature points such as the face, eyes, and nose estimated by the face feature point estimation unit 2, the attention direction determination is made. The device 100 can determine the direction in which the occupant is paying attention even when the occupant is wearing sunglasses.

The attention direction determination method according to the first embodiment includes a step of acquiring an image of the occupant in the vehicle taken by the camera, a step of estimating a feature point of the occupant's face based on the image, and a step based on the image. Based on the step of estimating the extending direction of the occupant's hand, the step of detecting the occupant's face orientation based on the facial feature points, the extending direction of the occupant's hand, and the occupant's face orientation. It has a step of determining the direction in which the occupant is paying attention.

With the above configuration, the attention direction determination method according to the first embodiment can determine the direction in which the occupant is paying attention.

Embodiment 2.
The configuration of the caution direction determination device 200 according to the second embodiment of the present disclosure will be described. The same or corresponding configurations as those in the first embodiment will be omitted, and only the parts having different configurations will be described.

FIG. 12 is a block diagram showing an example of the configuration of the caution direction determination device 200 according to the second embodiment. As shown in FIG. 12, the attention direction determination device 200 includes a seating information acquisition unit 21 and a vehicle information acquisition unit 22.

The seating information acquisition unit 21 acquires the seating information of the occupant in the vehicle from the image of the occupant in the vehicle taken by the camera 9. Here, the seating information is information indicating which seat in the vehicle the occupant is seated in. The acquisition of the seating information of the occupant in the vehicle by the seating information acquisition unit 21 is not limited to the configuration acquired from the image of the occupant in the vehicle taken by the camera 9. For example, when the vehicle is provided with a seating sensor, the seating information acquisition unit 21 may be configured to acquire the seating information of the occupant in the vehicle from the seating sensor.

The vehicle information acquisition unit 22 acquires vehicle information of the vehicle. Here, the vehicle information is, for example, information regarding the stop of the vehicle. The vehicle information acquisition unit 22 acquires vehicle information from a vehicle sensor (not shown) such as a vehicle speed sensor provided in the vehicle, for example.

The attention direction determination unit 7 according to the second embodiment of the present disclosure acquires the seating information, the direction in which the occupant's hand is extended estimated by the skeleton estimation unit 5, the occupant's face orientation detected by the face orientation detection unit 6, and the seating information. The direction in which the occupant is paying attention is determined based on at least one information of the seating information acquired by the unit 21 or the vehicle information acquired by the vehicle information acquisition unit 22.

Next, the attention direction determination unit 7 determines the direction in which the occupant's hand is extended estimated by the skeleton estimation unit 5, the occupant's face orientation detected by the face orientation detection unit 6, and the seating acquired by the seating information acquisition unit 21. An example of determination in the case of determining the direction in which the occupant is paying attention based on the information will be described.

The attention direction determination unit 7 is further seated when the direction in which the occupant's hand is extended estimated by the skeleton estimation unit 5 is the direction of the next seat and the face direction of the occupant detected by the face orientation detection unit 6 is the direction of the next seat. The seating information for the next seat is acquired from the seating information acquired by the information acquisition unit 21, and the direction in which the occupant is paying attention is determined. Here, the next-seat seating information is information indicating whether or not the occupant is seated in the next-seat with respect to the occupant to be detected. That is, in the attention direction determination unit 7, the direction in which the occupant's hand is extended estimated by the skeleton estimation unit 5 is the direction of the next seat, and the direction of the occupant's face detected by the face orientation detection unit 6 is the direction of the next seat, and the seat is seated. When the seating information acquired by the information acquisition unit 21 indicates that the occupant is not seated in the seat next to the occupant to be detected, the direction in which the occupant is paying attention may be determined to be the direction of the next seat. .. On the other hand, when the seating information acquired by the seating information acquisition unit 21 indicates that the occupant is seated in the seat next to the occupant to be detected, the attention direction determination unit 7 determines that the attention direction determination is not possible.

When the attention direction determination unit 7 indicates that the occupant's hand extension direction estimated by the skeleton estimation unit 5 is the rear seat direction and the occupant's face direction detected by the face orientation detection unit 6 is the rear seat direction. Further, the rear seat seating information is acquired from the seating information acquired by the seating information acquisition unit 21, and the direction in which the occupant is paying attention is determined. Here, the rear seat seating information is information indicating whether or not the occupant is seated in the rear seat with respect to the occupant to be detected. That is, in the attention direction determination unit 7, the direction in which the occupant's hand is extended estimated by the skeleton estimation unit 5 is the rear seat direction, and the occupant's face direction detected by the face orientation detection unit 6 is the rear seat direction. Further, when the seating information acquired by the seating information acquisition unit 21 indicates that the occupant is not seated in the rear seat with respect to the occupant to be detected, it is determined that the direction in which the occupant is paying attention is the rear seat direction. It may be configured. On the other hand, when the seating information acquired by the seating information acquisition unit 21 indicates that the occupant is seated in the rear seat with respect to the occupant to be detected, the attention direction determination unit 7 determines that the attention direction determination is not possible.

Next, the attention direction determination unit 7 determines the direction in which the occupant's hand is extended estimated by the skeleton estimation unit 5, the occupant's face orientation detected by the face orientation detection unit 6, and the vehicle acquired by the vehicle information acquisition unit 22. An example of determination in the case of determining the direction in which the occupant is paying attention based on the information will be described.

When the occupant's hand extension direction estimated by the skeleton estimation unit 5 is the door knob direction and the occupant's face direction detected by the face orientation detection unit 6 is the door knob direction, the attention direction determination unit 7 further determines the vehicle. The vehicle information acquired by the information acquisition unit 22 is used to determine the direction in which the occupant is paying attention. Here, the vehicle information is information regarding the stop of the vehicle. That is, in the attention direction determination unit 7, the occupant's hand extension direction estimated by the skeleton estimation unit 5 is the door knob direction, the occupant's face direction detected by the face orientation detection unit 6 is the door knob direction, and the vehicle. When the vehicle information acquired by the information acquisition unit 22 indicates that the vehicle is stopped, the direction in which the occupant is paying attention may be determined to be the door knob direction. On the other hand, when the vehicle information acquired by the vehicle information acquisition unit 22 indicates that the vehicle is not stopped, the attention direction determination unit 7 determines that the attention direction determination is not possible.

Further, when the occupant to be detected is an occupant seated in the driver's seat, the caution direction determination unit 7 acquires information on the vehicle stop from the vehicle information acquired by the vehicle information acquisition unit 22, and the vehicle stops. The configuration may be such that the direction in which the occupant is paying attention is determined when the occupant is paying attention. That is, the attention direction determination unit 7 may be configured to determine that the attention direction cannot be determined when the occupant to be detected is a occupant seated in the driver's seat and the vehicle is not stopped.

Further, according to the seating information acquired by the seating information acquisition unit 21 or the vehicle information acquired by the vehicle information acquisition unit 22, the face orientation detection unit 6 uses a predetermined angle value or a skeleton used when detecting the face orientation of the occupant. The estimation unit 5 may be configured to change the size of each skeleton detection area used when estimating the extending direction of the occupant's hand. For example, when the vehicle information acquired by the vehicle information acquisition unit 22 indicates that the vehicle gear is parking or the ignition is off, the skeleton estimation unit 5 uses each skeleton detection area when the vehicle is running. It may be configured to be larger than.

By acquiring the seating information, the attention direction determination unit 7 determines the direction in which the occupants are paying attention in consideration of the information regarding the plurality of occupants, so that it is possible to reduce the erroneous determination of the attention direction. Further, since the attention direction determination unit 7 determines the direction in which the occupant is paying attention in consideration of the state of the vehicle by acquiring the vehicle information, it is possible to reduce the erroneous determination of the attention direction.

When the attention direction determination unit 7 does not use the seating information and the vehicle information when determining the direction in which the occupant is paying attention, the control unit 10 outputs the determination result output from the determination result output unit 8. The vehicle-mounted device or the like may be controlled based on at least one piece of seating information acquired by the seating information acquisition unit 21 or vehicle information acquired by the vehicle information acquisition unit 22.

Next, the control unit 10 according to the second embodiment of the present disclosure controls the in-vehicle device or the like based on the determination result output from the determination result output unit 8 and the seating information acquired by the seating information acquisition unit 21. An embodiment in the case of performing the above will be described.

When the control unit 10 receives the determination result that the direction in which the occupant is paying attention is the direction of the next seat, the control unit 10 acquires the next seat seating information from the seating information acquired by the seating information acquisition unit 21. When the seating information acquired by the control unit 10 is information indicating that the occupant is not seated in the seat next to the occupant to be detected, the control unit 10 is for turning on the lighting for illuminating the occupant next to the occupant to be detected. Take control. On the other hand, when the seating information acquired by the control unit 10 is information indicating that the occupant is seated in the seat next to the occupant to be detected, the control unit 10 turns on the light illuminating the occupant next to the occupant to be detected. No control for

When the control unit 10 receives the determination result that the direction in which the occupant is paying attention is the rear seat direction, the control unit 10 acquires the rear seat seating information from the seating information acquired by the seating information acquisition unit 21. When the seating information acquired by the control unit 10 is information indicating that the occupant is not seated in the rear seat of the occupant to be detected, the control unit 10 turns on the light illuminating the rear seat of the occupant to be detected. Control for. On the other hand, when the seating information acquired by the control unit 10 is information indicating that the occupant is seated in the rear seat of the occupant to be detected, the control unit 10 illuminates the rear seat of the occupant to be detected. No control is performed to turn it on.

Next, the control unit 10 according to the second embodiment of the present disclosure controls the in-vehicle device or the like based on the determination result output from the determination result output unit 8 and the vehicle information acquired by the vehicle information acquisition unit 22. An embodiment in the case of performing the above will be described.

When the control unit 10 receives the determination result that the direction in which the occupant is paying attention is the doorknob direction, the control unit 10 acquires the vehicle information acquired by the vehicle information acquisition unit 22. When the vehicle information acquired by the control unit 10 is information indicating that the vehicle is stopped, the control unit 10 controls to turn on the lighting that illuminates the feet of the occupant to be detected. On the other hand, when the vehicle information acquired by the control unit 10 is information indicating that the vehicle is not stopped, the control unit 10 does not perform control for turning on the lighting that illuminates the feet of the occupant to be detected.

Further, the attention direction determination unit 7 does not use the seating information and the vehicle information when determining the direction in which the occupant is paying attention, and the attention direction determination unit 7 determines the direction in which the attention is directed. When the occupant is a occupant seated in the driver's seat, the control unit 10 corresponds to the determination result when the vehicle information acquired by the vehicle information acquisition unit 22 indicates that the vehicle is stopped. It may be configured to control the equipment or the like. That is, the control unit 10 is a occupant whose attention direction is determined by the attention direction determination unit 7 in the driver's seat, and the vehicle information acquired by the vehicle information acquisition unit 22 is the vehicle. If it indicates that the vehicle is not stopped, the vehicle-mounted device or the like may not be controlled.

By using the seating information when the control unit 10 controls the in-vehicle device, the control unit 10 can control the in-vehicle device and the like based on the information about a plurality of occupants, thus reducing erroneous control of the in-vehicle device. can do. Further, since the control unit 10 can control the in-vehicle device and the like in consideration of the state of the vehicle by using the vehicle information when controlling the in-vehicle device, erroneous control of the in-vehicle device can be performed. Can be reduced.

FIG. 13 is a flowchart showing an example of the operation of the caution direction determination device 200 shown in FIG. Note that steps S201 to S207 in FIG. 13 correspond to steps S101 to S107 in FIG. 11, and step S211 in FIG. 13 corresponds to step S109 in FIG. 11, so the description thereof is omitted here. Hereinafter, steps S208 to S210 will be described.

In step S208, the seating information acquisition unit 21 acquires the seating information of the occupant in the vehicle. In step S209, the vehicle information acquisition unit 22 acquires vehicle information.

In step S210, the attention direction determination unit 7 has acquired the direction in which the occupant's hand is extended estimated by the skeleton estimation unit 5, the occupant's face orientation detected by the face orientation detection unit 6, and the seating information acquisition unit 21. The direction in which the occupant is paying attention is determined by using at least one information of the seating information or the vehicle information acquired by the vehicle information acquisition unit 22.

The attention direction determination device according to the second embodiment has a seating information acquisition unit that acquires seating information of the occupant in the vehicle, and the attention direction determination unit extends the occupant's hand estimated by the skeleton estimation unit. The direction in which the occupant is paying attention is determined based on the direction, the occupant's face orientation detected by the face orientation detection unit, and the seating information acquired by the seating information acquisition unit.

Further, the attention direction determination device according to the second embodiment has a vehicle information acquisition unit that acquires vehicle information of the vehicle, and the attention direction determination unit is a direction in which the occupant's hand is extended as estimated by the skeleton estimation unit. The direction in which the occupant is paying attention is determined based on the occupant's face orientation detected by the face orientation detection unit and the vehicle information acquired by the vehicle information acquisition unit.

Further, in the caution direction determination unit of the caution direction determination device according to the second embodiment, the direction in which the occupant's hand is extended is the direction of the adjacent seat estimated by the skeleton estimation unit, and the occupant's face detected by the face orientation detection unit. When the direction is the direction of the next seat and the seating information acquired by the seating information acquisition unit indicates that the occupant is not seated in the next seat, it is determined that the direction in which the occupant is paying attention is the direction of the next seat.

Further, in the caution direction determination unit of the caution direction determination device according to the second embodiment, the direction in which the occupant's hand is extended as estimated by the skeleton estimation unit is the rear seat direction, and the occupant's face orientation detection unit detects it. When the face direction is the rear seat direction and the seating information acquired by the seating information acquisition unit indicates that the occupant is not seated in the rear seat, the direction in which the occupant is paying attention is the rear seat direction. judge.

Further, in the caution direction determination unit of the caution direction determination device according to the second embodiment, the direction in which the occupant's hand is extended is the doorknob direction estimated by the skeleton estimation unit, and the occupant's face detected by the face orientation detection unit. When the direction is the doorknob direction and the vehicle information acquired by the vehicle information acquisition unit indicates that the vehicle is stopped, it is determined that the direction in which the occupant is paying attention is the doorknob direction.

With the above configuration, the attention direction determination device 200 according to the second embodiment can determine the direction in which the occupant is paying attention. Further, the attention direction determination device 200 according to the second embodiment determines the direction in which the occupants are paying attention in consideration of the information regarding a plurality of occupants by acquiring the seating information, so that the attention direction is erroneously determined. Can be reduced. Further, since the attention direction determination device 200 according to the second embodiment determines the direction in which the occupant is paying attention in consideration of the state of the vehicle by acquiring the vehicle information, the erroneous determination of the attention direction is reduced. can do.

Image acquisition unit 1, face feature point estimation unit 2, head position estimation unit 3, site estimation unit 4, skeleton estimation unit 5, face orientation detection unit 6, attention direction determination unit in the attention direction determination devices 100 and 200 described above. 7. Each function of the determination result output unit 8, the seating information acquisition unit 21, and the vehicle information acquisition unit 22 is realized by the processing circuit. That is, the attention direction determination devices 100 and 200 acquire the image of the occupant in the vehicle taken by the camera, estimate the feature points of the occupant's face, estimate the relative position of the occupant's head with respect to the camera, and image. The size of multiple parts of the occupant is estimated, the skeleton of the occupant and the direction in which the hand is extended are estimated, the face orientation of the occupant is detected, the direction in which the occupant is paying attention is determined, and the determination result is obtained. A processing circuit for outputting is provided. The processing circuit may be dedicated hardware, and is a processor (CPU (Central Processing Unit), a central processing unit, a processing unit, an arithmetic unit, a microprocessor, a microcomputer, a DSP) that executes a program stored in a memory. It may also be called Digital Signal Processor).

When the processing circuit is dedicated hardware, as shown in FIG. 14, the processing circuit 31 is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, or an ASIC (Application Specific Integrated Circuit). , FPGA (Field Processor Gate Array), or a combination thereof. Image acquisition unit 1, face feature point estimation unit 2, head position estimation unit 3, site estimation unit 4, skeleton estimation unit 5, face orientation detection unit 6, attention direction determination unit 7, determination result output unit 8, seating information acquisition unit Each function of 21 and the vehicle information acquisition unit 22 may be realized by the processing circuit 31, or each function may be collectively realized by one processing circuit 31.

When the processing circuit 31 is the processor 32 shown in FIG. 15, the image acquisition unit 1, the face feature point estimation unit 2, the head position estimation unit 3, the site estimation unit 4, the skeleton estimation unit 5, the face orientation detection unit 6, and the attention direction. Each function of the determination unit 7, the determination result output unit 8, the seating information acquisition unit 21, and the vehicle information acquisition unit 22 is realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 33. The processor 32 realizes each function by reading and executing the program recorded in the memory 33. That is, the attention direction determination devices 100 and 200 are programs in which each step from step S101 to step S109 shown in FIG. 11 or each step from step S201 to step S211 shown in FIG. 13 is executed as a result. A memory 33 for storing the above is provided. Further, these programs include an image acquisition unit 1, a face feature point estimation unit 2, a head position estimation unit 3, a site estimation unit 4, a skeleton estimation unit 5, a face orientation detection unit 6, a attention direction determination unit 7, and a determination result output. It can also be said that the computer is made to execute the procedure or method of the unit 8, the seating information acquisition unit 21, and the vehicle information acquisition unit 22. Here, the memory is, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), an EPROM (Electrically Memory) It may be a semiconductor memory, a magnetic disk, a flexible disk, an optical disk, a compact disk, a DVD (Digital Versaille Disc), or any other storage medium that will be used in the future.

The image acquisition unit 1, the face feature point estimation unit 2, the head position estimation unit 3, the site estimation unit 4, the skeleton estimation unit 5, the face orientation detection unit 6, the attention direction determination unit 7, the determination result output unit 8, and the seating information. For each function of the acquisition unit 21 and the vehicle information acquisition unit 22, some functions may be realized by dedicated hardware, and other functions may be realized by software or firmware.

In this way, the processing circuit can realize each of the above-mentioned functions by hardware, software, firmware, or a combination thereof.

The caution direction determination device described above is an appropriate combination of not only an in-vehicle navigation device, that is, a car navigation device, but also a PND (Portable Navigation Device) that can be mounted on the vehicle, a server provided outside the vehicle, and the like. It can also be applied to a navigation device constructed as a system or a device other than the navigation device. Even if each function of the attention direction determination device is distributed and arranged in each function for constructing the system, the same effect as that of the above-described embodiment can be obtained.

In the present disclosure, each embodiment can be freely combined, and each embodiment can be appropriately modified or omitted within the scope of the disclosure.

Although the present disclosure has been described in detail, the above description is exemplary in all embodiments and the present disclosure is not limited thereto. It is understood that a myriad of variants not illustrated can be envisioned without departing from the scope of the present disclosure.

100,200 Attention direction determination device 1 Image acquisition unit 2 Face feature point estimation unit 3 Head position estimation unit 4 Part estimation unit 5 Skeleton estimation unit 6 Face orientation detection unit 7 Attention direction determination unit 8 Judgment result output unit 9 Camera 10 Control unit 21 Seating information acquisition unit 22 Vehicle information acquisition unit 31 Processing circuit 32 Processor 33 Memory

Claims (13)

1. An image acquisition unit that acquires images of occupants in the vehicle taken by the camera,
   A facial feature point estimation unit that estimates the facial feature points of the occupant based on the image acquired by the image acquisition unit, and a facial feature point estimation unit.
   Based on the image acquired by the image acquisition unit, the skeleton estimation unit that estimates the extending direction of the occupant's hand, and the skeleton estimation unit.
   A face orientation detection unit that detects the face orientation of the occupant based on the face feature points estimated by the face feature point estimation unit, and a face orientation detection unit.
   A caution direction for determining the direction in which the occupant is paying attention based on the direction in which the occupant's hand is extended and the occupant's face orientation detected by the face orientation detection unit estimated by the skeleton estimation unit. Judgment part and
   A caution direction determination device.
2. In the caution direction determination unit, when the direction in which the occupant's hand is extended estimated by the skeleton estimation unit and the occupant's face orientation detected by the face orientation detection unit are the same direction, the occupant The attention direction determination device according to claim 1, wherein the attention direction is determined.
3. In the caution direction determination unit, when the direction in which the occupant's hand is extended, which is estimated by the skeleton estimation unit, is the direction of the next seat, and the direction of the occupant's face detected by the face orientation detection unit is the direction of the next seat. The attention direction determining device according to claim 2, wherein the direction in which the occupant is paying attention is determined to be the direction of the adjacent seat.
4. In the caution direction determination unit, the direction in which the occupant's hand is extended as estimated by the skeleton estimation unit is the rear seat direction, and the face orientation of the occupant detected by the face orientation detection unit is the rear seat direction. The attention direction determining device according to claim 2, wherein the direction in which the occupant is paying attention is the rear seat direction.
5. In the caution direction determination unit, when the occupant's hand extension direction estimated by the skeleton estimation unit is the door knob direction and the occupant's face direction detected by the face orientation detection unit is the door knob direction. The attention direction determination device according to claim 2, wherein the direction in which the occupant is paying attention is determined to be the door knob direction.
6. It has a seating information acquisition unit that acquires seating information of the occupants in the vehicle.
   The attention direction determination unit includes the direction in which the occupant's hand is extended estimated by the skeleton estimation unit, the occupant's face orientation detected by the face orientation detection unit, and the seating acquired by the seating information acquisition unit. The attention direction determination device according to claim 1, wherein the occupant determines the direction in which the occupant is paying attention based on the information.
7. It has a vehicle information acquisition unit that acquires vehicle information of the vehicle, and has a vehicle information acquisition unit.
   The attention direction determination unit includes the direction in which the occupant's hand is extended estimated by the skeleton estimation unit, the occupant's face orientation detected by the face orientation detection unit, and the vehicle acquired by the vehicle information acquisition unit. The attention direction determination device according to claim 1, wherein the occupant determines the direction in which the occupant is paying attention based on the information.
8. In the caution direction determination unit, the direction in which the occupant's hand is extended as estimated by the skeleton estimation unit is the direction of the next seat, the face direction of the occupant detected by the face orientation detection unit is the direction of the next seat, and the above. The caution direction according to claim 6, wherein when the seating information acquired by the seating information acquisition unit indicates that the occupant is not seated in the next seat, it is determined that the direction in which the occupant is paying attention is the next seat direction. Judgment device.
9. In the caution direction determination unit, the direction in which the occupant's hand is extended, which is estimated by the skeleton estimation unit, is the rear seat direction, and the face direction of the occupant detected by the face orientation detection unit is the rear seat direction. In addition, when the seating information acquired by the seating information acquisition unit indicates that the occupant is not seated in the rear seat, it is determined that the direction in which the occupant is paying attention is the rear seat direction. The caution direction determination device described.
10. In the caution direction determination unit, the direction in which the occupant's hand is extended, which is estimated by the skeleton estimation unit, is the doorknob direction, the occupant's face direction detected by the face orientation detection unit is the doorknob direction, and the above. The caution direction according to claim 7, wherein when the vehicle information acquired by the vehicle information acquisition unit indicates that the vehicle is stopped, it is determined that the direction in which the occupant is paying attention is the door knob direction. Judgment device.
11. A head position estimation unit that estimates the relative position of the occupant's head with respect to the camera based on the facial feature points estimated by the face feature point estimation unit.
    Based on the image acquired by the image acquisition unit, a part estimation unit that estimates the size of a plurality of parts of the occupant in the image, and a part estimation unit.
    Have,
    The skeleton estimation unit estimates the skeleton of the occupant based on the relative position of the occupant's head estimated by the head position estimation unit and the size of each of the parts estimated by the site estimation unit. The caution direction determination device according to any one of claims 1 to 10, wherein the direction in which the occupant's hand is extended is estimated from the estimated skeleton determination result of the occupant.
12. The caution direction determination device according to any one of claims 1 to 11, further comprising a determination result output unit that outputs a direction in which the occupant has determined the attention direction determination unit.
13. The steps to acquire the image of the occupant in the vehicle taken by the camera,
    A step of estimating the facial feature points of the occupant based on the image, and
    Based on the image, the step of estimating the extending direction of the occupant's hand and
    A step of detecting the face orientation of the occupant based on the facial feature points, and
    A step of determining the direction in which the occupant is paying attention based on the direction in which the occupant's hand is extended and the face orientation of the occupant.
    A method of determining the direction of attention.

Images