WO2024034109A1

WO2024034109A1 - Physique determination device and physique determination method

Info

Publication number: WO2024034109A1
Application number: PCT/JP2022/030703
Authority: WO
Inventors: 晃平鎌田; 祥平 ▲高▼原; 直哉馬場; 大暉市川
Original assignee: 三菱電機株式会社
Priority date: 2022-08-12
Filing date: 2022-08-12
Publication date: 2024-02-15

Abstract

The present invention comprises: a skeletal point extraction unit (102) that extracts skeletal points of a vehicle occupant on the basis of a captured image of the vehicle occupant; a normal sitting determination unit (104) that determines, on the basis of situation information relating to the situation of the vehicle or the vehicle occupant, whether the posture of the vehicle occupant is in a normal sitting state; a feature quantity calculation unit (105) that determines whether or not the skeletal points of the vehicle occupant that were extracted by the skeletal point extraction unit (102) are normal sitting skeletal points extracted on the basis of a captured image of the vehicle occupant in a normal sitting state, and, if the skeletal points of the vehicle occupant that were extracted by the skeletal point extraction unit (102) are determined to be the normal sitting skeletal points, calculates a feature quantity for physique determination on the basis of information relating to the normal sitting skeletal points; and a physique determination unit (107) that determines the physique of the vehicle occupant on the basis of the feature quantity for physique determination.

Description

Physique determination device and physique determination method

The present disclosure relates to an occupant physique determination device and a physique determination method.

2. Description of the Related Art Conventionally, a technique is known in which the posture of a vehicle occupant is determined based on a captured image of the vehicle occupant, and the physique of the vehicle occupant is determined from the determined posture.
For example, Patent Document 1 discloses that based on a captured image obtained by capturing an image of the inside of a vehicle, the occupant's body physique estimation index including the coordinates of a plurality of feature points of the occupant in the captured image extracted by deep learning etc. An indoor monitoring device has been disclosed that estimates a posture and estimates the physique of an occupant based on the estimated posture.

JP2020-104680A

When a passenger is seated in a seat with a crooked posture, there are countless possible patterns of crooked posture. Therefore, when an occupant is seated in a seat with a crooked posture, it is difficult to cover the countless possible patterns of crooked posture, and the imaging The posture of the occupant determined based on the image tends to include errors, or the errors become large. As a result, the accuracy of physique determination decreases.
Since the conventional technology disclosed in Patent Document 1 does not take this into consideration, there is a problem that when the posture of the occupant collapses, the accuracy of determining the occupant's physique may decrease. .

The present disclosure has been made in order to solve the above-mentioned problems, and an object of the present disclosure is to provide a physique determination device that prevents a decrease in the accuracy of determining the physique of an occupant even if the posture of the occupant collapses. shall be.

The physique determination device according to the present disclosure includes an image acquisition unit that acquires a captured image of an occupant of a vehicle, and extracts skeletal points of the occupant indicating body parts of the vehicle based on the captured image acquired by the image acquisition unit. a normal seating determination section that determines whether or not the occupant's posture is in a normal seating state based on situation information regarding the vehicle or the occupant's situation; The skeletal points of the occupant extracted by the skeletal point extraction section are based on information about the skeletal points of It is determined whether or not the occupant's skeletal point is a normal seated skeletal point extracted based on an image taken in a seated state, and when it is determined that the skeletal point of the occupant extracted by the skeletal point extraction unit is a normal seated skeletal point. , a feature amount calculation unit that calculates a feature amount for physique determination based on information regarding normal sitting skeletal points, and a physique determination unit that determines the physique of the occupant based on the feature amount for physique determination calculated by the feature amount calculation unit. It is prepared.

According to the present disclosure, the physique determination device can prevent the accuracy of determining the physique of the occupant from decreasing even if the occupant's posture collapses.

1 is a diagram illustrating an example configuration of a system in which a physique determination device according to a first embodiment is used; FIG. 1 is a diagram illustrating a configuration example of a physique determination device according to a first embodiment; FIG. 2 is a flowchart for explaining an example of the operation of the physique determination device according to the first embodiment. FIG. 3 is a diagram illustrating a configuration example of a physique determination device in a case where inclination information is used as situation information in the first embodiment. FIG. 2 is a diagram illustrating a configuration example of a physique determination device in a case where vehicle information is used as situation information in Embodiment 1. FIG. 6A and 6B are diagrams illustrating an example of the hardware configuration of the physique determination device according to the first embodiment.

Embodiments of the present disclosure will be described in detail below with reference to the drawings.
Embodiment 1.
FIG. 1 is a diagram showing a configuration example of a system in which a physique determination device 100 according to the first embodiment is used.
In the system shown in FIG. 1, the physique determination device 100 according to the first embodiment is connected to an imaging device 200, a seatbelt control device 300, an airbag control device 400, an abandonment detection device 500, and an output device 600. Ru.
In the first embodiment, it is assumed that the physique determination device 100 is mounted on a vehicle.
Further, the seatbelt control device 300, the airbag control device 400, the left-behind detection device 500, and the output device 600 are mounted on the vehicle.

The imaging device 200 is, for example, a near-infrared camera or a visible light camera, and images the occupant of the vehicle. For example, the imaging device 200 may be shared with an imaging device included in a so-called "Driver Monitoring System (DMS)" that is installed in a vehicle to monitor the condition of a driver in the vehicle.
The imaging device 200 is installed to be able to image an area within the vehicle that includes at least an area where the upper body of a vehicle occupant should be present. The range in which the upper body of the occupant in the vehicle should exist is, for example, a range corresponding to the space near the backrest of the seat and the front of the headrest.
For example, the imaging device 200 can image the front seats including the driver's seat and the front passenger seat from the center of the instrument panel (hereinafter referred to as "instrument panel") of the vehicle. It is installed so that it can image the occupant in the seat (hereinafter referred to as "passenger seat occupant"). Note that this is just an example, and the imaging device 200 may be installed to be able to image only the driver, or may be installed to be able to image the passenger in the rear seat, for example.

The physique determination device 100 according to the first embodiment determines the physique of a vehicle occupant based on a captured image of the vehicle occupant taken by the imaging device 200.
At this time, the physique determination device 100 determines whether or not the posture of the occupant of the vehicle is in a normally seated state, and determines the physique of the occupant based on the captured image taken with the occupant in the normally seated state. Make a judgment.
In the first embodiment, "regular seating" means that the occupant is seated on the seat in a normal state with no posture collapse. In the first embodiment, a state in which the occupant's face is not facing forward is also considered to be a posture collapse. When the occupant's face is not facing forward, it is estimated that there is a high possibility that the occupant's whole body is not facing forward.
When the physique determination device 100 determines the physique of the vehicle occupant, the physique determination device 100 transmits the result of determining the physique of the vehicle occupant (hereinafter referred to as “physique determination result”) to the seat belt control device 300, the airbag control device 400, and the left-behind detection. Output to device 500. Furthermore, the physique determination device 100 outputs information to the output device 600 for outputting a message or the like urging the occupant to sit properly.
Details of a configuration example of the physique determination device 100 will be described later using FIG. 2.
In addition, in the following Embodiment 1, the passenger of a vehicle is also simply called a "passenger."

The seat belt control device 300 controls the seat belt. For example, the seatbelt control device 300 has a seatbelt reminder function that outputs a warning in consideration of the occupant's physique based on the physique determination result output from the physique determination device 100.

The airbag control device 400 is, for example, an ECU (Engine Control Unit) for an airbag system, and controls the airbag based on the physique determination result output from the physique determination device 100.

The left-behind detection device 500 detects if an infant or the like is left behind in a vehicle. For example, the abandonment detection device 500 determines whether an infant or the like is left behind in the vehicle based on the physique determination result output from the physique determination device 100.

The output device 600 is, for example, a display device or an audio output device provided in a center cluster, a meter panel, or the like. For example, the output device 600 displays a message urging the occupant to sit properly, outputs the message in audio, or outputs a warning sound urging the occupant to sit properly.

A configuration example of the physique determination device 100 according to the first embodiment will be described.
FIG. 2 is a diagram showing a configuration example of the physique determination device 100 according to the first embodiment.
The physique determination device 100 includes an image acquisition section 101, a skeletal point extraction section 102, a situation information acquisition section 103, a normal sitting determination section 104, a feature calculation section 105, a feature leveling section 106, a physique determination section 107, and a posture determination section 104. A warning section 108 is provided. The situation information acquisition unit 103 includes a face orientation detection unit 1031.

The image acquisition unit 101 acquires a captured image of a passenger captured by the imaging device 200.
The image acquisition unit 101 outputs the acquired captured image to the skeleton point extraction unit 102.

The skeletal point extraction unit 102 extracts skeletal points of the occupant, which indicate body parts of the occupant, based on the captured image acquired by the image acquisition unit 101. Specifically, the skeletal point extracting unit 102 extracts skeletal points of the occupant, which indicate joint points determined for each part of the occupant's body, based on the captured image acquired by the image acquiring unit 101. Joint points determined for each part of the occupant's body include, for example, nose joint points, neck joint points, shoulder joint points, elbow joint points, waist joint points, wrist joint points, and knee joints. point, the ankle joint point. The joint points are predefined.
The skeletal point extraction unit 102 may extract the skeletal points of the occupant using a known method using a known technique such as an image recognition technique or a technique using a trained model (hereinafter referred to as a "machine learning model"). .
A skeleton point is, for example, a point in a captured image, and is expressed by coordinates in the captured image. The skeletal point extraction unit 102 detects the coordinates of a skeletal point of the occupant and which part of the body of the occupant the skeletal point indicates.

Note that the skeletal point extraction unit 102 extracts skeletal points for each occupant.
For example, in a captured image, an area corresponding to each seat (hereinafter referred to as a "seat corresponding area") is set in advance for each seat. The seat corresponding area is set in advance according to the installation position of the imaging device 200 and the angle of view.
For example, the skeleton point extraction unit 102 extracts skeleton points for each seat corresponding area. For example, if a certain seat-corresponding area is a seat-corresponding area corresponding to a driver's seat, the skeleton point extraction unit 102 assumes that the skeleton points extracted in the seat-corresponding area are the skeleton points of the driver. Further, for example, if a certain seat corresponding area is a seat corresponding area corresponding to a passenger seat, the skeleton point extraction unit 102 determines that the skeleton points extracted in the seat corresponding area are the skeleton points of the front passenger seat occupant. . In this way, the skeleton point extracting unit 102 extracts skeleton points for each occupant, for example, by extracting skeleton points from the seat corresponding area for each seat corresponding area.

Skeletal point extraction section 102 outputs information regarding the extracted skeleton points (hereinafter referred to as "skeletal point information") to situation information acquisition section 103.
The skeletal point information includes information in which information indicating the occupant, information indicating the coordinates of the skeletal point of the occupant, and information indicating which part of the body of the occupant the skeletal point indicates are associated; Also included are captured images acquired by the image acquisition unit 101.
In the first embodiment, the information indicating the occupant is, for example, information indicating the seat where the occupant is seated.

When the skeletal point extraction unit 102 extracts the skeletal points of the occupant, in other words, when the skeletal point information is output from the skeletal point extraction unit 102, the situation information acquisition unit 103 acquires information regarding the situation of the occupant (hereinafter referred to as “situation information”). ). The situation information is, for example, information regarding the direction of the occupant's face.

The face orientation detection unit 1031 detects the passenger's face orientation based on the captured image acquired by the image acquisition unit 101. The captured image acquired by the image acquisition unit 101 is included in the skeleton point information.
The face orientation detection unit 1031 may detect the passenger's face orientation using a known method using a known technique such as an image recognition technique or a technique using a machine learning model. The direction of the occupant's face is expressed, for example, by an angle.
Note that the face orientation detection unit 1031 detects the face orientation for each passenger.

The situation information acquisition unit 103 acquires information regarding the passenger's face direction detected by the face direction detection unit 1031 (hereinafter referred to as “face direction information”) as situation information. The face orientation information includes information in which information indicating the occupant and information indicating the face orientation are associated with each other. The information indicating the occupant is, for example, information indicating the seat where the occupant is seated. For example, the face direction detection unit 1031 may determine which seat the occupant is seated in based on the seat corresponding area. In addition to these, the face orientation information may also include a captured image acquired by the image acquisition unit 101.
The situation information acquisition unit 103 outputs the acquired situation information, here face orientation information, to the normal seating determination unit 104.

The normal seating determination unit 104 determines whether the occupant's posture is in a normal seating state based on the situation information acquired by the situation information acquisition unit 103.
Specifically, here, the normal seating determination unit 104 determines whether the occupant's posture is in the normal seating state based on the face orientation information.
Note that the normal seating determination unit 104 determines for each passenger whether or not the passenger's posture is in a normal seating state. The normal seating determination unit 104 can identify the facial orientation of each occupant from the facial orientation information.

An example of a method for determining whether or not the passenger's posture is a normal seating state by the normal seating determination unit 104 will be described.
For example, the normal seating determination unit 104 uses conditions (hereinafter referred to as (referred to as "regular seating determination conditions"), it is determined whether the occupant's posture is in a regular seating state.
The conditions for determining normal seating include, for example, conditions under which the occupant's posture is considered to be normal seating. The normal seating determination conditions are generated in advance by an administrator or the like and stored in a location that can be referenced by the physique determination device 100.

For example, a condition such as <Condition 1> below is defined as the condition for determining normal seating.

<Condition 1>
The occupant's face must be facing forward.
If the passenger's face direction is within a predetermined range (referred to as a "face direction determination range") with the front as a reference, it is assumed that the passenger's face direction is facing the front direction.

Note that the face orientation determination range in <Condition 1> is set in advance as an appropriate range in which it is assumed that the face is facing the front direction.

Based on the face orientation information, the normal seating determination unit 104 determines that the posture of the occupant is the normal seating state when the passenger's face orientation satisfies the normal seating determination conditions.
On the other hand, based on the face orientation information, if the occupant's face orientation does not satisfy the conditions for determining normal seating, the normal seating determination unit 104 determines that the occupant's posture is not the normal seating state.

The normal seating determination unit 104 outputs a determination result (hereinafter referred to as a “seating state determination result”) as to whether or not the occupant's posture is determined to be a normal seating state to the feature value calculation unit 105. At this time, normal seating determination section 104 outputs the skeleton point information output from skeleton point extraction section 102 to feature amount calculation section 105 together with the seating state determination result.
Further, the normal seating determination unit 104 outputs the seating state determination result to the posture attention unit 108 .

The feature value calculation unit 105 calculates the position of the occupant based on the skeleton point information output from the skeleton point extraction unit 102 and the seating state determination result of the normal seating determination unit 104 as to whether or not the occupant's posture is in the normal seating state. Whether or not the occupant's skeletal points extracted by the skeletal point extraction unit 102 are skeletal points extracted based on an image taken with the occupant in a normally seated state (hereinafter referred to as "normally seated skeletal points"). Determine.
For example, if the seating state determination result output from the normal seating determination unit 104 indicates that the occupant's posture is a normal seating state, the feature value calculation unit 105 extracts The skeleton point thus obtained is determined to be a normal seated skeleton point. That is, the feature amount calculation unit 105 determines that the skeleton point information output from the skeleton point extraction unit 102 is regular sitting skeleton point information.
Note that in the first embodiment, the feature amount calculation unit 105 acquires the skeleton point information output from the skeleton point extraction unit 102 via the normal seating determination unit 104, but this is only an example. do not have. For example, the feature amount calculation unit 105 may directly acquire skeleton point information from the skeleton point extraction unit 102. In this case, in FIG. 1, the arrow from the skeleton point extraction unit 102 is drawn toward the feature quantity calculation unit 105.

Then, when the feature value calculation unit 105 determines that the occupant's skeletal point extracted by the skeletal point extraction unit 102 is a normal seating skeletal point, the feature calculation unit 105 provides skeletal point information of the normal seating skeletal point (hereinafter referred to as “regular sitting skeletal point Based on the information (hereinafter referred to as "feature amount for physique determination") used for determining the occupant's physique (hereinafter referred to as "feature amount for physique determination").
The feature amount calculation unit 105 calculates a feature amount for physique determination for each occupant. The feature amount calculation unit 105 can identify the skeletal points of each occupant from the normal seating skeletal point information.
Note that if the feature amount calculation unit 105 does not determine that the occupant's skeletal points extracted by the skeletal point extraction unit 102 are normal seating skeletal points, the feature amount calculation unit 105 does not calculate the feature amount for physique determination.

For example, the feature amount calculation unit 105 calculates the feature amount for physique determination from the length in the captured image of a line segment connecting two of the skeletal points in the captured image, based on the normal sitting skeletal point information.
In detail, for example, the feature amount calculation unit 105 calculates the arm length, shoulder width, and neck length of the occupant in the captured image as the feature amount for physique determination based on the normal sitting skeletal point information. .
For example, the feature amount calculation unit 105 may set the length of a line segment connecting a skeletal point indicating a shoulder and a skeletal point indicating an elbow as the length of the arm of the occupant as the feature amount for physique determination. In addition, the feature value calculation unit 105 calculates, for example, the length of a line segment connecting the skeletal points indicating both shoulders, the length of the line segment connecting the skeletal points indicating the neck and the skeletal point indicating the right shoulder, or the length of the line segment connecting the skeletal points indicating the neck and the skeletal point indicating the right shoulder. The length of the line segment connecting the skeleton point shown in FIG. Further, the feature quantity calculation unit 105 may, for example, set the length of the line segment connecting the skeletal point indicating the nose and the skeletal point indicating the neck as the length of the neck to be used as the feature quantity for physique determination.
Note that this is just an example, and the feature amount calculation unit 105 does not need to calculate all the arm length, shoulder width, and neck length of the occupant as feature amounts for physique determination; At least one of the length, shoulder width, and neck length may be calculated as the feature quantity for physique determination. Further, the feature amount calculation unit 105 may use the length of a line segment connecting two skeletal points other than the arm length, shoulder width, and neck length of the occupant as the feature amount for physique determination.

The feature quantity calculation unit 105 outputs information regarding the calculated feature quantity for physique determination (hereinafter referred to as “feature quantity information”) to the feature quantity leveling unit 106.
In the feature amount information, for each occupant, information indicating the occupant is associated with a feature amount for physique determination. The information indicating the occupant is, for example, information indicating the seat where the occupant is seated.

The feature quantity leveling unit 106 calculates a plurality of physique determination feature quantities for a past preset period (hereinafter referred to as “leveling period”), in other words, the feature quantity calculation unit 105 The plurality of physique determination feature quantities calculated by are equalized.
The leveling period can be set to an appropriate length. The leveling period is set by an administrator or the like, and is stored in a location that can be referenced by the feature amount leveling unit 106.

For example, the feature leveling unit 106 associates the feature information output from the feature calculating unit 105 with the acquisition date and time of the feature information, and stores it at a location where the feature leveling unit 106 can refer to it. Memorize them in sequence. The feature quantity leveling unit 106 may acquire a plurality of physique determination feature quantities in the leveling period from the stored feature quantity information. Note that when the feature values for physique determination for the leveling period are not stored, such as immediately after the vehicle starts running, the feature value leveling unit 106 stores only the stored feature values for physique determination, for example. The feature amount for physique determination may be set for the leveling period.
The feature amount information stored by the feature amount calculation unit 105 is deleted, for example, when the power of the vehicle is turned on.

The feature leveling unit 106 levels the plurality of physique determination feature quantities for the leveling period, for example, by averaging, taking a median value, or taking a value corresponding to a set percentile. .
In other words, the feature leveling unit 106 calculates, for example, the average value of the plurality of physique determination feature quantities for the leveling period, the median value of the plurality of physique determination feature quantities for the leveling period, or the average value of the plurality of physique determination feature quantities for the leveling period. The value corresponding to the set percentile of the plurality of features for physique determination for the period of time is set as the feature for physique determination after leveling (hereinafter referred to as "the feature for physique determination after leveling"). calculate.

Note that the feature amount leveling unit 106 calculates a post-leveling physique determination feature amount for each occupant. The feature amount leveling unit 106 can specify the feature amount for determining the physique of each occupant from the feature amount information.
In addition, when the feature quantity calculation unit 105 calculates a plurality of types of feature quantities for physique determination, such as the length of the arms, shoulder width, and neck length of the occupant, the feature quantity leveling unit 106 level the respective feature quantities.

For example, when the feature value leveling unit 106 equalizes the feature values for physique determination, it is assumed that the feature values for physique determination are temporarily deviated from the feature values corresponding to the original physique of the occupant. Also, the physique determination device 100 can reduce the influence on the occupant's physique determination using the physique determination feature (specifically, the leveled physique determination feature). Note that the physique determination section 107 performs the determination of the occupant's physique. Details of the physique determining section 107 will be described later.

For example, when the feature amount leveling unit 106 averages the plurality of physique determination feature amounts for the leveling period to level the plurality of physique determination feature amounts, the feature amount leveling unit 106 averages the plurality of physique determination feature amounts for the leveling period. Even if the number of a plurality of feature quantities for physique determination is small, an accurate value, in other words, a value closer to the feature quantity according to the original physique of the occupant can be used as the feature quantity for physique determination after leveling. Therefore, in this case, for example, the administrator etc. can perform face orientation determination defined in the regular seating determination conditions used by the regular seating determination unit 104 to determine whether or not the passenger's posture is in the regular seating state. The scope of use can be narrowed. By narrowing the face orientation determination range, it is assumed that the frequency at which the normal seating determining unit 104 determines that the passenger's posture is normal seating will decrease; Even if the number of multiple physique determination feature quantities is small when the passenger's posture is determined to be in a normal seating state, accurate physique determination after leveling can be performed from a small number of physique determination feature quantities. Feature quantities can be calculated. As a result, the physique determination device 100 can prevent a decrease in the accuracy of determining the occupant's physique using the leveled physique determination feature amount.

For example, the feature value leveling unit 106 may take the median value of a plurality of feature values for physique determination for the leveling period, or may take a value corresponding to a set percentile to determine the When leveling feature quantities, the feature leveling unit 106 calculates accurate values from a large number of physique determination feature quantities, in other words, even if the number of physique determination feature quantities is large. A value closer to the corresponding feature amount can be set as the feature amount for physique determination after leveling. Therefore, in this case, for example, the administrator etc. can perform face orientation determination defined in the regular seating determination conditions used by the regular seating determination unit 104 to determine whether or not the passenger's posture is in the regular seating state. By widening the range of use, the normal seating determination unit 104 can easily determine that the occupant's posture is in the normal seating state. It is assumed that by making it easier for the normal seating determining unit 104 to determine that the occupant's posture is normal seating, the accuracy of determining the normal seating state of the occupant will decrease; however, after that, the feature leveling unit 106 is capable of calculating an accurate leveled physique determination feature amount from many physique determination feature amounts when the posture state of the occupant is determined to be a normal seating state. As a result, the physique determination device 100 can prevent a decrease in the accuracy of determining the occupant's physique using the leveled physique determination feature amount.

The feature leveling unit 106 outputs information regarding the leveled physique determination feature (hereinafter referred to as “leveled feature information”) to the physique determining unit 107.
In the leveled feature amount information, information indicating the occupant and leveled feature amount information are associated for each occupant. The information indicating the occupant is, for example, information indicating the seat where the occupant is seated.

The physique of the occupant is determined based on the leveled physique determination feature quantity leveled by the physique determination unit 107 and the feature quantity leveling unit 106 .
Note that the physique determination unit 107 determines the physique of each occupant. The physique determination unit 107 can identify the leveled physique determination feature of each occupant from the leveled feature information.

The physique determining unit 107 may determine the physique of the occupant by a known method using a known technique such as an image recognition technique or a technique using a learned model (hereinafter referred to as a "machine learning model").
The physique determining unit 107 determines the physique of the occupant, such as "infant", "small", "normal", or "large", using a known method. Note that this is just an example, and the definition of the physique determined by the physique determination unit 107 can be set as appropriate.

The physique determination unit 107 outputs the physique determination result to the seatbelt control device 300, the airbag control device 400, and the left-behind detection device 500.
In the physique determination result, for each occupant, information indicating the occupant is associated with information indicating the occupant's physique. The information indicating the occupant is, for example, information indicating the seat where the occupant is seated.

The posture warning unit 108 determines whether the occupant's posture, as determined by the normal seating determination unit 104, is not the normal seating state for a preset period of time (hereinafter referred to as “posture abnormality determination time”). If so, information (hereinafter referred to as "posture caution information") urging the occupant to adopt a normal seating posture is output to the output device 600.
Note that, based on the seating state determination result output from the normal seating determination unit 104, the posture warning unit 108 determines that the normal seating determination unit 104 has determined that the occupant's posture is not the normal seating state. can.

For example, the posture caution section 108 associates the sitting state determination result output from the regular sitting determination section 104 with the acquisition date and time of the sitting state determination result, and stores it in a chronological order in a location where the posture caution section 108 can refer to. Remember it. Based on the stored seating state determination result, the posture warning unit 108 can determine whether or not the occupant's posture is not the normal seating state for the posture abnormality determination period. Note that the seating state determination result stored by the posture attention unit 108 is deleted, for example, when the vehicle is powered on.
If the posture warning unit 108 determines that the passenger's posture is not in the normal seating state for the posture abnormality determination period, the posture warning unit 108 outputs posture caution information to the output device 600.

By outputting the posture caution information, the posture caution section 108 displays, for example, a message to the output device 600 urging the occupant to sit properly, outputs the message in audio, or displays a message urging the occupant to sit properly. A warning sound is output to prompt the user to sit down.

The posture warning unit 108 outputs posture caution information and urges the occupant to sit properly, so that the physique determination device 100 can make the occupant maintain a correct posture. As a result, the physique determining device 100 can easily determine that the occupant's posture is in a normal seating state.

The operation of the physique determination device 100 according to the first embodiment will be described.
FIG. 3 is a flowchart for explaining an example of the operation of the physique determination device 100 according to the first embodiment.
For example, the physique determination device 100 periodically repeats the operation shown in the flowchart of FIG. 3 while the vehicle is running.
Furthermore, for example, when the vehicle is powered on, the physique determining device 100 may repeat the operation shown in the flowchart of FIG. 3 until the vehicle is powered off.

The image acquisition unit 101 acquires a captured image of a passenger captured by the imaging device 200 (step ST1).
The image acquisition unit 101 outputs the acquired captured image to the skeleton point extraction unit 102.

The skeletal point extraction unit 102 extracts the skeletal points of the occupant, which indicate the body parts of the occupant, based on the captured image acquired by the image acquisition unit 101 in step ST1 (step ST2).
Skeleton point extraction section 102 outputs skeleton point information to situation information acquisition section 103.

The situation information acquisition unit 103 acquires situation information when the skeleton point extraction unit 102 extracts the skeleton points of the occupant in step ST2 (step ST3).
In step ST3, the face orientation detection unit 1031 detects the passenger's face orientation based on the captured image acquired by the image acquisition unit 101 in step ST1. The situation information acquisition unit 103 acquires face direction information regarding the passenger's face direction detected by the face direction detection unit 1031 as situation information.
The situation information acquisition unit 103 outputs the acquired situation information, here face orientation information, to the normal seating determination unit 104.

The normal seating determination unit 104 determines whether the posture of the occupant is normal seating based on the situation information acquired by the situation information acquisition unit 103 in step ST3 (step ST4). Specifically, here, the normal seating determination unit 104 determines whether the occupant's posture is in the normal seating state based on the face orientation information.
The normal seating determination unit 104 outputs the seating state determination result to the feature amount calculation unit 105. At this time, normal seating determination section 104 outputs the skeleton point information output from skeleton point extraction section 102 to feature amount calculation section 105 together with the seating state determination result.
Further, the normal seating determination unit 104 outputs the seating state determination result to the posture attention unit 108 .

The feature value calculation unit 105 determines whether or not the posture of the occupant is a normal seating state based on the skeleton point information output from the skeleton point extraction unit 102 in step ST2 and the state of the occupant's posture determined by the normal seating determination unit 104 in step ST4. Based on the seating state determination result, it is determined in step ST2 whether the skeletal points of the occupant extracted by the skeletal point extraction unit 102 are regular seating skeletal points.
Then, when it is determined that the occupant's skeleton point extracted by the skeleton point extracting unit 102 in step ST2 is a regular sitting skeleton point, the feature amount calculation unit 105 adds the regular sitting skeleton point information of the regular sitting skeleton point. Based on this, a feature quantity for physique determination is calculated (step ST5).
The feature quantity calculation unit 105 outputs the feature quantity information to the feature quantity leveling unit 106.

If the skeletal points of the occupant extracted by the skeletal point extraction unit 102 in step ST2 are not determined to be regular seating skeletal points, the feature amount calculation unit 105 does not calculate the feature amount for physique determination, and The operation of the determination device 100 ends the processing as shown in the flowchart of FIG.

The feature value leveling unit 106 calculates a plurality of physique determination feature values for the past leveling period based on the normal sitting skeleton point information output from the normal sitting determining unit 104, in other words, The plurality of feature quantities for physique determination calculated by the feature quantity calculation unit 105 are leveled (step ST6).
The feature leveling unit 106 outputs the leveled feature information to the physique determining unit 107.

The physique determination section 107 determines the physique of the occupant based on the leveled physique determination feature amount leveled by the feature leveling section 106 in step ST6 (step ST7).
The physique determination unit 107 outputs the physique determination result to the seatbelt control device 300, the airbag control device 400, and the left-behind detection device 500.

The posture warning unit 108 transmits the posture caution information when the state of the occupant's posture, which is determined by the normal seating determination unit 104 in step ST4, is not the normal seating state and continues for the posture abnormality determination period. , is output to the output device 600 (step ST8).

In this way, the physique determination device 100 extracts the skeletal points of the vehicle occupant based on the captured image of the vehicle occupant. Furthermore, the physique determination device 100 determines whether the occupant's posture is in a normal seating state based on situation information, in this case, face orientation information. The physique determination device 100 determines whether or not the extracted skeletal points of the occupant are normal seated skeletal points based on the skeletal point information and the determination result as to whether the posture state of the occupant is a normal seating state. If the point is determined to be a regular sitting skeleton point, a feature quantity for physique determination is calculated based on the regular sitting skeleton point information. Then, the physique determination device 100 determines the occupant's physique based on the calculated physique determination feature amount.

As described above, when an occupant is seated in a seat with a crooked posture, there are countless possible patterns of crooked posture. Therefore, when an occupant is seated in a seat with a crooked posture, it is difficult to cover the countless possible patterns of crooked posture. The posture of the occupant determined by the above-described method is likely to include an error, or the error becomes large. For example, the skeletal points of the occupant extracted from the captured image tend to include errors, or the errors become large. If the skeletal points of the occupant extracted from the captured image tend to include errors, or if the error becomes large, the feature values calculated based on the skeletal points of the occupant tend to include errors, or The error becomes larger. Then, the posture of the occupant determined based on the feature amount is likely to include an error, or the error becomes large. As a result, the accuracy of physique determination decreases.
Note that when the occupant is seated without changing his or her posture, that is, when the occupant is seated normally, the position of the skeleton point, which is the feature point of the occupant, on the captured image will fall within a certain range. is assumed. Therefore, when the occupant is properly seated, errors are unlikely to occur in the occupant's skeletal points extracted from the captured image and the feature amounts calculated from the skeletal points, and as a result, the determination is made based on the feature amounts. Errors in the occupant's posture are also less likely to occur. Even if an error occurs, the influence of the error on the physique determination is not large.

The occupant's physique determination result based on the captured image is used, for example, in functions such as seatbelt control, airbag control, and abandonment detection. Functions such as seatbelt control, airbag control, and left-behind detection are safety-related functions, and therefore the physique determination results used for these functions are required to have high accuracy. Therefore, the less accurate results of determining the occupant's physique when the occupant is seated in a crooked position may be affected by functions such as seatbelt control, airbag control, or left-behind detection. cannot be used. If the physique determination device determines the physique used for functions such as seatbelt control, airbag control, or left-behind detection, it is determined that the occupant is sitting in a crooked position in the seat. If the occupant's physique determination result is provided, there is a risk of erroneous control or erroneous detection in the above function.

On the other hand, as described above, the physique determination device 100 according to the first embodiment determines whether or not the posture of the occupant is in the normal seating condition, and when it is determined that the occupant is in the normal seating condition. The feature amount based on the skeletal points extracted from the captured image is used as the feature amount for physique determination, and the physique of the occupant is determined based on the feature amount for physique determination. The physique determination device 100 calculates a physique determination feature amount based on a captured image when the posture state of the occupant is in a normal seating state, and determines the occupant's physique based on the physique determination feature amount. , it is possible to prevent a decrease in the accuracy of determining the occupant's physique. That is, the physique determining device 100 can prevent the accuracy of determining the physique of the occupant from decreasing, even if the occupant's posture collapses while the vehicle is running.
The physique determination device 100 provides the occupant's physique determination result that prevents deterioration of accuracy, so the physique determination result can be used for seatbelt control, airbag control, abandonment detection, etc., which require high accuracy. It is also possible to provide a physique determination result that can be used in the function.

Note that in the first embodiment described above, in the physique determination device 100, the situation information acquisition unit 103 includes the face orientation detection unit 1031, and the face orientation detection unit 1031 detects the captured image based on the captured image acquired by the image acquisition unit 101. The direction of the passenger's face was detected. However, this is just one example. For example, the situation information acquisition unit 103 acquires facial orientation information of the occupant from a device provided outside the physique determination device 100 at a location that the physique determination device 100 can refer to, such as an occupant state detection device included in the DMS. You may obtain it. Note that in this case, it is assumed that the imaging device 200 is shared with the DMS.
In this case, in the physique determination device 100, the situation information acquisition section 103 can be configured without the face orientation detection section 1031.

Furthermore, in the first embodiment described above, in the physique determination device 100, the situation information acquired by the situation information acquisition unit 103 is information regarding the passenger's facial orientation, but this is only an example. In the first embodiment described above, the physique determination device 100 may use the situation information acquired by the situation information acquisition unit 103 as skeletal point information, for example.
In this case, the normal seating determination section 104 may have the function of the situation information acquisition section 103. That is, it is not essential for the physique determination device 100 as shown in FIG. 1 to include the situation information acquisition unit 103.
In this case, the skeleton point extraction section 102 outputs skeleton point information to the normal seating determination section 104.
The normal seating determination unit 104 determines whether the occupant's posture is in a normal seating state based on the skeleton point information output from the skeleton point extraction unit 102.

For example, in this case, a condition such as <Condition 2> below is defined as the condition for determining normal seating.

<Condition 2>
On the captured image, the position of the skeleton point indicating the occupant's elbow is not located within a preset range (hereinafter referred to as "elbow position determination range").

Note that the elbow position determination range in <Condition 2> is set in advance, for example, to a predetermined range including the armrest or a predetermined range near the door of the vehicle.

Based on the skeleton point information, the normal seating determination unit 104 determines that the posture of the occupant is the normal seating state when the position of the skeleton point indicating the elbow of the occupant satisfies the conditions for determining normal seating.
On the other hand, based on the inclination information, if the inclination of the occupant's skeleton line does not satisfy the condition for determining normal seating, the normal seating determination unit 104 determines that the posture of the occupant is not the normal seating state.
For example, if the position of the skeleton point indicating the occupant's elbow is located within the elbow position determination range, the normal seating determination unit 104 determines that the occupant's posture is not the normal seating state.

In the physique determination device 100, a flowchart showing the operation of the physique determination device 100 when the situation information is skeletal point information is the same as the flowchart shown in FIG.
However, in step ST3, the normal seating determination unit 104 acquires the skeleton point information output by the skeleton point extraction unit 102 in step ST2 as situation information.
Then, in step ST4, the normal seating determination unit 104 determines whether or not the occupant's posture is in a normal seating state based on the skeleton point information acquired in step ST3.

The other specific operations of the physique determination device 100 in steps ST1 to ST2 and steps ST5 to ST8 are the same as the specific operations that have already been explained, and therefore, redundant explanations will be omitted.

In this way, the situation information is the occupant's skeletal point information, and in the physique determining device 100, the normal seating determining section 104 determines that the occupant's posture is normal based on the occupant's skeletal point information extracted by the skeletal point extracting section 102. It may also be determined whether the user is in a seated state. In this case as well, the physique determination device 100 calculates the physique determination feature amount based on the captured image when the posture state of the occupant is the normal seating state, and calculates the physique determination feature amount of the occupant based on the physique determination feature amount. Therefore, it is possible to prevent a decrease in the accuracy of determining the occupant's physique. That is, the physique determining device 100 can prevent the accuracy of determining the physique of the occupant from decreasing, even if the occupant's posture collapses while the vehicle is running.

Further, in the first embodiment described above, the physique determination device 100 may obtain, for example, information regarding the inclination of a skeletal line connecting the skeletal points of the occupant (hereinafter referred to as "inclination information") as the situation information. In this case, even if a situation occurs in which only the occupant's face is not facing forward, it is not considered to be a posture collapse.
Here, FIG. 4 is a diagram illustrating a configuration example of the physique determination device 100 in the first embodiment when tilt information is used as situation information.
The configuration example of the physique determination device 100 shown in FIG. 4 differs from the configuration example of the physique determination device 100 shown in FIG. are different.
Further, in the physique determining device 100 shown in FIG. 4, the specific operation of the normal sitting determining section 104 is different from the specific operation of the normal seating determining section 104 in the physique determining device 100 shown in FIG. The specific operations of the image acquisition unit 101, skeletal point extraction unit 102, feature value calculation unit 105, feature level leveling unit 106, physique determination unit 107, and posture attention unit 108 are the same as those already explained. As this is true, repeated explanation will be omitted.

The inclination detection unit 1032 detects the inclination of a skeletal line, which is a line segment connecting a plurality of skeletal points of the occupant, based on the skeletal point information output from the skeletal point extraction unit 102.
For example, the inclination detection unit 1032 detects the inclination of a skeletal line of the occupant's shoulders that connects skeletal points indicating both shoulders of the occupant. Note that this is just an example, and the inclination detection unit 1032 can detect the inclination of an appropriate skeletal line, such as the inclination of a skeletal line indicating a torso (for example, a skeletal line indicating sitting height). However, the fact that the occupant's posture is disrupted is likely to be reflected in the inclination of the skeletal line of the occupant's shoulders. Therefore, when the inclination detection unit 1032 detects the inclination of the shoulder skeleton line of the occupant, it is possible to reduce the erroneous determination by the normal seating determination unit 104 as to whether or not the occupant's posture is in the normal seating state. can.

The situation information acquisition unit 103 acquires the tilt information as situation information. The inclination information includes information in which information indicating the occupant is associated with information indicating the inclination of the skeleton line. In addition to these, the tilt information may also include a captured image acquired by the image acquisition unit 101.
The situation information acquisition unit 103 outputs the acquired situation information, here tilt information, to the normal seating determination unit 104.

When the configuration example of the physique determination device 100 is as shown in FIG. It is determined whether the state is a normal seating state.
For example, in this case, a condition such as <Condition 3> below is defined as the condition for determining normal seating.

<Condition 3>
The inclination of the occupant's skeletal line must be within a preset range (hereinafter referred to as the "inclination determination range").

Note that the range for determining inclination in <Condition 3> is set in advance as a predetermined range based on the inclination of the skeletal line of the occupant, which is assumed to be facing forward, for example.

Based on the inclination information, the normal seating determination unit 104 determines that the posture of the occupant is normal seating if the inclination of the skeletal line of the occupant satisfies the conditions for determining normal seating.
On the other hand, based on the inclination information, if the inclination of the occupant's skeleton line does not satisfy the condition for determining normal seating, the normal seating determination unit 104 determines that the posture of the occupant is not the normal seating state.

A flowchart showing the operation of the physique determination device 100 when the configuration example of the physique determination device 100 is as shown in FIG. 4 is the same as the flowchart shown in FIG.
However, if the configuration example of the physique determination device 100 is as shown in FIG. 4, the situation information acquisition unit 103 acquires the inclination information as the situation information in step ST3.
Then, the situation information acquisition unit 103 outputs the acquired situation information, here the tilt information, to the normal seating determination unit 104.
In step ST4, the normal seating determination unit 104 determines whether the occupant's posture is in a normal seating state based on the inclination information.

In this way, the situation information is tilt information, and the physique determination device 100 includes the tilt detection unit 1032 that detects the tilt of the occupant's skeletal line based on the information regarding the occupant's skeletal points extracted by the skeletal point extraction unit 102, and The seating determination unit 104 may determine whether the occupant is in a normal seating position based on the inclination of the occupant's skeletal line detected by the inclination detection unit 1032. In this case as well, the physique determination device 100 calculates the physique determination feature amount based on the captured image when the posture state of the occupant is the normal seating state, and calculates the physique determination feature amount of the occupant based on the physique determination feature amount. Therefore, it is possible to prevent a decrease in the accuracy of determining the occupant's physique. That is, the physique determining device 100 can prevent the accuracy of determining the physique of the occupant from decreasing, even if, for example, the occupant's posture collapses while the vehicle is running.

Further, in the above-described first embodiment, the situation information may be not only information about the situation of the occupant, but also information about the situation of the vehicle, for example. Note that in this case as well, even if a situation occurs in which only the occupant's face is not facing the front direction, this is not considered to be a posture collapse.
For example, in the first embodiment described above, the physique determination device 100 may acquire information regarding the situation of the vehicle (hereinafter referred to as "vehicle information") as the situation information.
Vehicle information includes, for example, information regarding the steering angle of the vehicle, vehicle speed, the opening/closing status of vehicle doors, the ignition status of the vehicle, or the operation status of in-vehicle devices (e.g., car navigation device, interior lamp, etc.). include.

Here, FIG. 5 is a diagram showing a configuration example of the physique determination device 100 in the first embodiment when vehicle information is used as situation information.
The configuration example of the physique determination device 100 shown in FIG. 5 is different from the configuration example of the physique determination device 100 shown in FIG. The difference is in the preparation.
Further, in the physique determining device 100 shown in FIG. 5, the specific operation of the normal sitting determining section 104 is different from the specific operation of the normal seating determining section 104 in the physique determining device 100 as shown in FIG.
The specific operations of the image acquisition unit 101, skeletal point extraction unit 102, feature value calculation unit 105, feature level leveling unit 106, physique determination unit 107, and posture attention unit 108 are the same as those already explained. As this is true, repeated explanation will be omitted.

The vehicle information acquisition unit 1033 acquires vehicle information from various devices installed in the vehicle. For example, the vehicle information acquisition unit 1033 may acquire information regarding the steering angle status from the steering angle sensor. Further, for example, the vehicle information acquisition unit 1033 may acquire the vehicle speed from a vehicle speed sensor. Further, for example, the vehicle information acquisition unit 1033 may acquire information regarding the opening/closing status of the door from a sensor provided on the door. Further, for example, the vehicle information acquisition unit 1033 may acquire information regarding the ignition status from the ignition sensor. Further, for example, the vehicle information acquisition unit 1033 may acquire information regarding the operation status of the in-vehicle device from an in-vehicle device such as a car navigation device.

The situation information acquisition unit 103 acquires the vehicle information acquired by the vehicle information acquisition unit 1033 as situation information.
The situation information acquisition unit 103 outputs the acquired situation information, here vehicle information, to the normal seating determination unit 104.

When the configuration example of the physique determination device 100 is as shown in FIG. 5, the normal seating determination unit 104 determines that the state of the occupant's posture is the normal seating state based on the vehicle information acquired by the vehicle information acquisition unit 1033. Determine whether or not.
For example, in this case, conditions such as the following <Condition 4> to <Condition 8> are defined as the conditions for determining normal seating.

<Condition 4>
The vehicle steering angle must be within a preset range (hereinafter referred to as the "steering angle determination range").

<Condition 5>
The vehicle speed exceeds a preset range (hereinafter referred to as "vehicle speed determination range").

<Condition 6>
vehicle door not open

<Condition 7>
The elapsed time after the ignition was turned on must be within a preset time (hereinafter referred to as "start determination time").

<Condition 8>
The in-vehicle device is not being operated.

Note that the range for steering angle determination in <Condition 4> is set to, for example, a range of steering angles that cannot occur without bending the arm.
Furthermore, the vehicle speed determination range in <Condition 5> is set to, for example, a vehicle speed assuming that the vehicle is traveling on an expressway.
Regarding <Condition 6>, for example, if the door of the vehicle is open, it is assumed that the occupant is getting in or out of the vehicle, and that the occupant is not properly seated.

If the vehicle information satisfies the condition for determining normal seating, the normal seating determination unit 104 determines that the posture of the occupant is normal seating.
On the other hand, if the vehicle information does not satisfy the conditions for determining normal seating, the normal seating determination unit 104 determines that the posture of the occupant is not the normal seating state.

A flowchart showing the operation of the physique determination device 100 when the configuration example of the physique determination device 100 is as shown in FIG. 5 is the same as the flowchart shown in FIG. 3.
However, if the configuration example of the physique determination device 100 is the configuration example shown in FIG. 5, in step ST3, the situation information acquisition unit 103 acquires the vehicle information acquired by the vehicle information acquisition unit 1033 as the situation information. .
Then, the situation information acquisition unit 103 outputs the acquired situation information, here vehicle information, to the normal seating determination unit 104.
In step ST4, the normal seating determination unit 104 determines whether the occupant's posture is in a normal seating state based on the vehicle information acquired by the vehicle information acquisition unit 1033 in step ST3.

In this way, the situation information is vehicle information, the physique determination device 100 includes a vehicle information acquisition unit 1033 that acquires vehicle information, and the normal seating determination unit 104 uses vehicle information acquired by the vehicle information acquisition unit 1033. It may be determined whether the passenger's posture is in a normal seating state. In this case as well, the physique determination device 100 calculates the physique determination feature amount based on the captured image when the posture state of the occupant is the normal seating state, and calculates the physique determination feature amount of the occupant based on the physique determination feature amount. Therefore, it is possible to prevent a decrease in the accuracy of determining the occupant's physique. That is, the physique determining device 100 can prevent the accuracy of determining the physique of the occupant from decreasing, even if the occupant's posture collapses while the vehicle is running.

In this way, the physique determination device 100 may acquire not only face orientation information but also skeletal point information, tilt information, or vehicle information as situation information.
However, devices other than the physique determination device 100 often have the function of detecting the direction of the occupant's face based on the captured image captured by the vehicle. As described above, for example, the occupant state detection device included in the DMS may have a function of detecting the direction of the occupant's face. For example, the physique determination device 100 can acquire the face orientation information from a device external to the physique determination device 100 by using the face orientation information as situation information, and is not required to have a function of detecting the facial orientation. do not have. Furthermore, since the physique determination device 100 does not need to perform processing for detecting the occupant's face orientation, it is possible to reduce the processing load associated with determining the occupant's physique.

Note that the physique determination device 100 does not need to add a function for acquiring situation information even if the skeletal point information is used as situation information. However, in general, detecting the position of the eyes or mouth based on the captured image has higher detection accuracy than detecting the position of the skeletal point based on the captured image. For example, determining whether the occupant is facing forward based on the direction of the occupant's face has higher determination accuracy than determining whether the occupant is facing forward based on the occupant's skeletal points. Note that, generally, the direction of a person's face is detected using the position of the eyes or mouth.
Therefore, the physique determination device 100 determines whether the occupant's posture is in the normal seating state with more accuracy when the face orientation information is used as the situation information than when the skeletal point information is used as the situation information. can.

Furthermore, in the first embodiment described above, the physique determination device 100 may acquire two or more of face orientation information, skeleton point information, tilt information, and vehicle information as situation information.
That is, in the physique determination device 100, the situation information acquisition section 103 may include, for example, two or more of the face direction detection section 1031, the tilt detection section 1032, and the vehicle information acquisition section 1033.

Further, in the first embodiment described above, in the physique determination device 100, the normal seating determination unit 104 determines whether the posture of the occupant is the normal seating state based on the situation information and according to the conditions for determining normal seating. However, this is just one example.
For example, the normal seating determination unit 104 determines whether the occupant's posture is the normal seating state based on the situation information and a machine learning model (hereinafter referred to as the "regular seating determination model"). Good too.
For example, the normal seating determination model is a machine learning model that receives situational information as input and outputs information indicating whether or not the passenger's posture is normal seating.
The normal seating determination model is generated in advance by an administrator or the like and stored in a location that can be referenced by the physique determination device 100.

Further, in the first embodiment described above, the normal seating determination unit 104 uses both the normal seating determination conditions and the normal seating determination model based on the situation information to determine whether the occupant is in the normal seating state. It may be determined whether or not.
For example, the normal seating determination unit 104 determines whether the occupant's posture is in a normal seating state (first determination) based on the situation information and in accordance with the conditions for determining normal seating. Further, the normal seating determination unit 104 determines whether the occupant's posture is in a normal seating state (second determination) based on the situation information and the normal seating determination model. The normal seating determination unit 104 determines that the occupant's posture is normal seating when it is determined in both the first determination and the second determination that the occupant's posture is normal seating; If it is determined in either the first determination or the second determination that the occupant's posture is not the normal seating condition, it is determined that the occupant's posture is not the normal seating condition.
Further, for example, it may be determined whether or not the occupant's posture is a normal seating state by weighting the seating state determination result of the first determination and the seating state determination result of the second determination.

Further, in the first embodiment described above, the physique determination device 100 was provided with the feature leveling unit 106, but this is only an example. The physique determination device 100 may be configured without the feature leveling unit 106.
In this case, the physique determination unit 107 may determine the physique of the occupant based on the physique determination feature calculated by the feature calculation unit 105.
Furthermore, in the operation of the physique determining device 100 described using the flowchart of FIG. 3, the physique determining device 100 can omit the process of step ST6. In step ST7, the physique determination unit 107 may determine the physique of the occupant based on the physique determination feature calculated by the feature calculation unit 105 in step ST5.

Further, in the first embodiment described above, the physique determination device 100 was provided with the posture attention section 108, but this is only an example. The physique determination device 100 may be configured without the posture attention section 108. Further, the physique determination device 100 does not need to be connected to the output device 600.
In this case, in the operation of the physique determining device 100 described using the flowchart of FIG. 3, the physique determining device 100 can omit the process of step ST8.

Further, in the first embodiment described above, the physique determination device 100 is connected to the seatbelt control device 300, the airbag control device 400, and the left-behind detection device 500, but this is only an example.
The physique determination device 100 may be connected only to either the seatbelt control device 300, the airbag control device 400, or the left-behind detection device 500, or may perform various processes using the physique determination results. It may be connected to devices other than the seatbelt control device 300, the airbag control device 400, and the left-behind detection device 500.
For example, the physique determination device 100 may be connected to a display device mounted on a vehicle. The display device performs display according to the physique determination result output from the physique determination device 100, for example. For example, if an infant is among the occupants, the display device displays an icon indicating that a child is in the vehicle.

Further, in the first embodiment described above, it is assumed that there is one imaging device 200, but this is only an example. In the first embodiment, a plurality of imaging devices 200 may be installed in a vehicle. For example, a plurality of imaging devices 200 may be installed so as to be able to image each passenger seated in each seat in the vehicle.
In this case, the physique determination device 100 acquires captured images from the plurality of imaging devices 200 and determines the physique of the occupant.

6A and 6B are diagrams illustrating an example of the hardware configuration of the physique determination device 100 according to the first embodiment.
In the first embodiment, an image acquisition unit 101, a skeleton point extraction unit 102, a situation information acquisition unit 103, a face orientation detection unit 1031, a tilt detection unit 1032, a vehicle information acquisition unit 1033, and a normal seating determination unit The functions of 104 , feature amount calculation section 105 , feature amount leveling section 106 , physique determination section 107 , and posture attention section 108 are realized by processing circuit 1001 . That is, the physique determination device 100 includes a processing circuit 1001 for performing control to determine the physique of the vehicle occupant based on a captured image of the vehicle occupant.
The processing circuit 1001 may be dedicated hardware as shown in FIG. 6A, or may be a processor 1004 that executes a program stored in memory as shown in FIG. 6B.

When the processing circuit 1001 is dedicated hardware, the processing circuit 1001 is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Circuit). Gate Array), or a combination of these.

When the processing circuit is the processor 1004, the image acquisition unit 101, the skeleton point extraction unit 102, the situation information acquisition unit 103, the face orientation detection unit 1031, the tilt detection unit 1032, the vehicle information acquisition unit 1033, and the normal seating The functions of the determination unit 104, feature amount calculation unit 105, feature amount leveling unit 106, physique determination unit 107, and posture attention unit 108 are realized by software, firmware, or a combination of software and firmware. . Software or firmware is written as a program and stored in memory 1005. The processor 1004 reads out and executes the program stored in the memory 1005, thereby controlling the image acquisition unit 101, the skeleton point extraction unit 102, the situation information acquisition unit 103, the face direction detection unit 1031, and the tilt detection unit 1032. The functions of the vehicle information acquisition section 1033, normal seating determination section 104, feature amount calculation section 105, feature amount leveling section 106, physique determination section 107, and posture attention section 108 are executed. That is, the physique determination device 100 includes a memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST8 in FIG. 3 described above. Further, the programs stored in the memory 1005 include an image acquisition section 101, a skeleton point extraction section 102, a situation information acquisition section 103, a face orientation detection section 1031, a tilt detection section 1032, and a vehicle information acquisition section 1033. , normal seating determination section 104, feature amount calculation section 105, feature amount leveling section 106, physique determination section 107, and posture attention section 108 can be executed by a computer. Here, the memory 1005 is, for example, RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically non-volatile or volatile This includes semiconductor memory, magnetic disks, flexible disks, optical disks, compact disks, mini disks, DVDs (Digital Versatile Discs), and the like.

Note that the image acquisition unit 101, the skeleton point extraction unit 102, the situation information acquisition unit 103, the face orientation detection unit 1031, the tilt detection unit 1032, the vehicle information acquisition unit 1033, the normal seating determination unit 104, and the feature Regarding the functions of the quantity calculation unit 105, the feature quantity leveling unit 106, the physique determination unit 107, and the posture attention unit 108, some of them are realized by dedicated hardware, and some of them are realized by software or firmware. It's okay. For example, the function of the image acquisition unit 101 is realized by a processing circuit 1001 as dedicated hardware, and the function is realized by a skeletal point extraction unit 102, a situation information acquisition unit 103, a face orientation detection unit 1031, and a tilt detection unit 1032. The processor 1004 is stored in the memory 1005 for the vehicle information acquisition section 1033, normal seating determination section 104, feature amount calculation section 105, feature amount leveling section 106, physique determination section 107, and posture attention section 108. The function can be realized by reading and executing the program.
The physique determination device 100 also includes an input interface device 1002 that performs wired or wireless communication with devices such as an imaging device 200, a seatbelt control device 300, an airbag control device 400, an abandonment detection device 500, or an output device 600. and an output interface device 1003.

In the above first embodiment, the physique determination device 100 is an in-vehicle device mounted on a vehicle, and includes a skeleton point extraction section 102, a situation information acquisition section 103, a normal seating determination section 104, and a feature amount calculation section. 105, the feature value leveling unit 106, the physique determination unit 107, and the posture attention unit 108 are assumed to be included in the vehicle-mounted device.
However, the present invention is not limited to this, but includes the skeletal point extraction section 102, the situation information acquisition section 103, the normal seating determination section 104, the feature amount calculation section 105, the feature amount leveling section 106, the physique determination section 107, and the posture attention section. 108, part of which is installed in the on-vehicle device of the vehicle, and the other part is provided in a server connected to the in-vehicle device via a network, so that the in-vehicle device and the server constitute a physique determination system. Good too.
In addition, the skeletal point extraction section 102, the situation information acquisition section 103, the normal seating determination section 104, the feature amount calculation section 105, the feature amount leveling section 106, the physique determination section 107, and the posture attention section 108 are all It may be provided in the server.

As described above, according to the first embodiment, the physique determination device 100 includes the image acquisition unit 101 that acquires a captured image of an occupant of a vehicle, and a skeletal point extracting unit 102 that extracts skeletal points of the occupant indicating the body parts of the occupant; The seating determination unit 104 determines whether or not the occupant's posture is a normal seating state based on the information regarding the occupant's skeletal points extracted by the skeletal point extraction unit 102 and the seating state determination result of the normal seating determination unit 104 as to whether or not the occupant's posture is in a normal seating state. , the skeletal point extraction unit 102 determines whether or not the skeletal points of the occupant extracted are normal seating skeletal points extracted based on an image taken with the occupant normally seated. When it is determined that the extracted skeletal point of the occupant is a normal sitting skeletal point, the feature calculating unit 105 calculates a feature for physique determination based on information about the normal sitting skeletal point, and the feature calculating unit 105 calculates The vehicle is configured to include a physique determination unit 107 that determines the physique of the occupant based on the physique determination feature quantity. The physique determination device 100 calculates a physique determination feature amount based on a captured image when the posture state of the occupant is in a normal seating state, and determines the occupant's physique based on the physique determination feature amount. , it is possible to prevent a decrease in the accuracy of determining the occupant's physique. That is, even if the occupant's posture collapses, the physique determination device 100 can prevent the accuracy of determining the occupant's physique from decreasing.

Note that in the present disclosure, any component of the embodiments can be modified or any component of the embodiments can be omitted.

A physique determination device according to the present disclosure calculates a physique determination feature amount based on a captured image when the posture state of an occupant is in a normally seated state, and determines the physique of the occupant based on the physique determination feature amount. Therefore, it is possible to prevent a decrease in the accuracy of determining the occupant's physique.

100 physique determination device, 101 image acquisition unit, 102 skeletal point extraction unit, 103 situation information acquisition unit, 1031 face orientation detection unit, 1032 tilt detection unit, 1033 vehicle information acquisition unit, 104 normal seating determination unit, 105 feature amount calculation unit , 106 feature leveling unit, 107 physique determination unit, 108 posture warning unit, 200 imaging device, 300 seatbelt control device, 400 airbag control device, 500 left behind detection device, 600 output device, 1001 processing circuit, 1002 input interface Device, 1003 Output interface device, 1004 Processor, 1005 Memory.

Claims

an image acquisition unit that acquires a captured image of a vehicle occupant;
a skeletal point extraction unit that extracts skeletal points of the occupant indicating body parts of the occupant based on the captured image acquired by the image acquisition unit;
a normal seating determination unit that determines whether or not the occupant's posture is in a normal seating state based on situation information regarding the vehicle or the occupant's situation;
Based on the information regarding the skeletal points of the occupant extracted by the skeletal point extracting unit and the seating state determination result of whether or not the occupant's posture state is the normal seating state by the normal seating determining unit, It is determined whether the skeletal points of the occupant extracted by the skeletal point extracting unit are normal seated skeletal points extracted based on the captured image of the occupant in the normally seated state, and the skeletal points are extracted from the skeletal points. a feature amount calculation unit that calculates a feature amount for physique determination based on information regarding the normal sitting skeletal point when the extracting unit determines that the skeletal point of the occupant is the normal sitting skeletal point;
A physique determination unit that determines the physique of the occupant based on the physique determination feature calculated by the feature calculation unit.
The situation information is information regarding the direction of the occupant's face detected based on the captured image,
The physique according to claim 1, wherein the normal seating determination unit determines that the passenger's posture is the normal seating state when the face of the passenger is facing forward. Judgment device.
comprising a face orientation detection unit that detects the face orientation of the occupant based on the captured image acquired by the image acquisition unit,
The normal seating determination unit determines that the posture state of the occupant is the normal seating state when the face orientation of the occupant detected by the face orientation detection unit is facing the front direction. The physique determination device according to claim 2.
The situation information is information regarding the inclination of a skeletal line connecting the skeletal points of the occupant,
comprising an inclination detection unit that detects the inclination of the skeletal line of the occupant based on information regarding the skeletal points of the occupant extracted by the skeletal point extraction unit;
The normal seating determination unit determines that the posture state of the occupant is the normal seating state when the inclination of the skeleton line of the occupant detected by the inclination detection unit is within a range for inclination determination. The physique determining device according to any one of claims 1 to 3, characterized in that:
The physique determining device according to claim 4, wherein the skeletal line of the occupant is a line connecting the skeletal points indicating both shoulders of the occupant.
The situation information is vehicle information related to the situation of the vehicle, including a steering angle situation of the vehicle, a vehicle speed, a door opening/closing situation of the vehicle, an ignition situation of the vehicle, or an operation situation of an on-vehicle device,
comprising a vehicle information acquisition unit that acquires the vehicle information,
The regular seating determination unit determines, based on the vehicle information acquired by the vehicle information acquisition unit, that if the steering wheel steering angle of the vehicle is within a steering angle determination range, the vehicle speed exceeds a vehicle speed determination range. In this case, if the door of the vehicle is not open, if the elapsed time after the ignition is turned on is within the start determination time, or if the in-vehicle device is not operated, the posture of the occupant The physique determining device according to any one of claims 1 to 5, wherein the state is determined to be the normal sitting state.
The situation information is information regarding the skeletal points of the occupant extracted by the skeletal point extraction unit,
The normal seating determination unit determines that the occupant is not in the normal seating state when the position of the skeleton point indicating the elbow of the occupant extracted by the skeleton point extraction unit is located within an elbow position determination range. The physique determining device according to any one of claims 1 to 6, characterized in that it determines that.
The normal seating determination unit determines whether the occupant is seated based on the situation information and a machine learning model that receives the situation information and outputs information indicating whether or not the passenger's posture is in the normal seating state. The physique determining device according to claim 1, wherein it is determined whether or not the posture state is the normal sitting state.
comprising a feature leveling unit that levels the plurality of physique determination feature quantities calculated by the feature quantity calculation unit retroactively over a leveling period;
The physique determination section determines the physique of the occupant based on the leveled physique determination feature amount leveled by the feature leveling section. The physique determination device according to any one of the items.
The feature amount leveling unit is characterized in that the average value, median value, or value corresponding to a set percentile of the plurality of the feature amounts for physique determination is set as the feature amount for physique determination after leveling. The physique determining device according to claim 9.
If the state of the posture of the occupant determined by the normal seating determination unit is not the normal seating state for a period of time for determining posture abnormality, the occupant is instructed to assume the normal seating posture. The physique determination device according to any one of claims 1 to 10, further comprising a posture warning section that outputs posture warning information to prompt.
a step in which the image acquisition unit acquires a captured image of an occupant of the vehicle;
a skeletal point extraction unit extracting skeletal points of the occupant indicating body parts of the occupant based on the captured image acquired by the image acquisition unit;
a step in which the normal seating determination unit determines whether or not the posture of the occupant is in a normal seating state based on situation information regarding the vehicle or the occupant;
A feature amount calculation unit determines whether or not the posture of the occupant is the normal seating state based on the information regarding the skeletal points of the occupant extracted by the skeletal point extraction unit and the normal seating judgment unit. Based on the results, it is determined whether the skeletal points of the occupant extracted by the skeletal point extracting unit are normal seated skeletal points extracted based on the captured image of the occupant in the normally seated state. and calculating a feature amount for physique determination based on information regarding the regular sitting skeletal point when the skeletal point of the occupant extracted by the skeletal point extracting unit is determined to be the regular sitting skeletal point. and,
A physique determining method, comprising: a physique determining section determining the physique of the occupant based on the physique determining feature calculated by the feature calculating section.