JP2020194294A - Jointpoint detection apparatus - Google Patents

Abstract

To provide, as an example, a jointpoint detection apparatus capable of classifying a plurality of jointpoints acquired from an image for each human body with a small calculation amount.SOLUTION: A jointpoint detection apparatus according to the embodiment comprises: a detection unit that detects a plurality of jointpoints included in a target area based on at least one of a distance image and a brightness image of the target area; and a classification unit that classifies the plurality of jointpoints for each human body based on the distance image.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to a joint point detector.

Conventionally, a technique of estimating the coordinates of a joint point of a human body from an image captured by an imaging device has been known.

Non-Patent Document 1 discloses a technique for calculating the degree of association (PAF: Part Affinity Fields) between joint points and classifying a plurality of joint points for each human body based on the calculated degree of association. According to such a technique, it is possible to estimate the posture of each human body even when a plurality of human bodies are reflected in the image.

“Realtime Multi-Person 2D Pose Optimization using Part Affinity Fields” Zhe Cao <Internet> https://arxiv.org/pdf/1611.08050.pdf (Searched April 10, 2019)

However, since the degree of relevance between the joint points is large in the amount of calculation, for example, in order to detect the joint points of the human body in real time, an arithmetic unit having a high calculation ability is required.

As an example, the present disclosure provides a joint point detection device capable of classifying a plurality of joint points acquired from an image for each human body with a small amount of calculation.

As an example, the joint point detection device according to the present disclosure is based on a detection unit that detects a plurality of joint points included in the target area based on at least one of a distance image and a brightness image of the target area, and based on the distance image. A classification unit for classifying the plurality of joint points for each human body is provided. By classifying a plurality of joint points included in the target region for each human body based on a distance image, it is possible to reduce the amount of calculation as compared with a conventional classification method using, for example, PAF.

In the above joint point detection device, the classification unit is one of a target joint point among the plurality of joint points and a plurality of connection candidate joint points that are candidates for the joint point connected to the target joint point. The degree of deviation based on the sum of the distances of the points existing on the line connecting the two is calculated for each of the plurality of connection candidate joint points, and the connection having the smallest degree of deviation among the plurality of connection candidate joint points. A candidate joint point is determined as the joint point connected to the target joint point.

On the line segment connecting the joint points of different people, there is a high possibility that there is a place where the distance becomes large, such as a background part. Therefore, the classification unit determines the connection candidate joint point having a low degree of deviation from the plurality of connection candidate joint points as the joint point connected to the target joint point. As a result, a plurality of joint points acquired from the image can be classified for each human body with a small amount of calculation.

In the above-mentioned joint point detection device, the classification unit selects as the target joint point in order from the said joint point having the shortest distance among the plurality of joint points. In this way, the degree of divergence can be appropriately obtained by selecting the target joint points in order from the joint point having the closest distance.

In the joint point detection device, the classification unit calculates the coordinates of each point existing on the line segment using Bresenham's algorithm. Since the Bresenham's algorithm can be implemented only by adding and subtracting integers and bit shifting, the coordinates of each point existing on the line segment can be calculated at high speed with a small amount of calculation by using the Bresenham's algorithm.

In the above joint point detection device, the classification unit is one of a target joint point among the plurality of joint points and a plurality of connection candidate joint points that are candidates for the joint point connected to the target joint point. Calculate the vector between the joint points connecting the ones, and calculate the vector between the plurality of division points connecting the plurality of division points that divide the line segment connecting the target joint point and one of the plurality of connection candidate joint points. Then, the degree of coincidence between the directions of the vector between the joint points and the vector between the plurality of division points is calculated for each of the plurality of connection candidate joint points, and among the plurality of connection candidate joint points, the degree of coincidence is the highest. A connection candidate joint point is determined as the joint point connected to the target joint point.

When the joint points of the same person are set as the target joint point and the connection candidate joint point, the direction of the vector between the joint points and the direction of the vector between the plurality of division points are theoretically the same, so that the degree of coincidence is high. On the other hand, the degree of coincidence when the joint points of different persons are set as the target joint points and the connection candidate joint points is smaller than that when the joint points of the same person are set as the target joint points and the connection candidate joint points. Become. This is because the directions of the vector between the joint points and the directions of the vectors between the plurality of division points do not match when the target joint points and the connection candidate joint points are the joint points of different persons. Therefore, the classification unit determines the connection candidate joint point having a high degree of coincidence among the plurality of connection candidate joint points as the joint point connected to the target joint point. As a result, a plurality of joint points acquired from the image can be classified for each human body with a small amount of calculation.

FIG. 1 is a plan view of the interior of a vehicle equipped with the joint point detection device according to the first embodiment as viewed from above. FIG. 2 is a block diagram showing a configuration of a control system according to the first embodiment. FIG. 3 is a block diagram showing a functional configuration of the ECU according to the first embodiment. FIG. 4 is a diagram showing an example of joint point information. FIG. 5 is an explanatory diagram of a procedure for calculating the degree of deviation. FIG. 6 is a flowchart showing a procedure of processing executed by the ECU according to the first embodiment. FIG. 7 is an explanatory diagram of a procedure for calculating the degree of agreement. FIG. 8 is an explanatory diagram of a procedure for calculating the degree of agreement. FIG. 9 is an explanatory diagram of a procedure for calculating the degree of agreement. FIG. 10 is a flowchart showing a procedure of processing executed by the ECU according to the second embodiment.

Hereinafter, a mode for carrying out the joint point detection device according to the present disclosure (hereinafter, referred to as “the embodiment”) will be described in detail with reference to the drawings. It should be noted that this embodiment does not limit the joint point detection device according to the present disclosure. Further, in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate description is omitted.

(First Embodiment)
[1. Configuration of vehicle 1]
FIG. 1 is a plan view of the interior of the vehicle 1 on which the joint point detection device according to the first embodiment is mounted, as viewed from above. As shown in FIG. 1, a plurality of seats 2 are provided in the vehicle interior of the vehicle 1. For example, a driver's seat 2a and a passenger seat 2b are provided on the front side of the vehicle interior, and a plurality of rear seats 2c are provided on the rear side.

An image pickup device 3 is provided on the front side of the vehicle interior. The image pickup device 3 is, for example, a TOF (Time OF Flight) distance image camera, and captures a luminance image and a distance image. The luminance image is an image in which the luminance value is a pixel value. Further, the distance image is an image in which the distance value from each point included in the target area (imaging area) in the vehicle interior captured by the imaging device 3 to the imaging device 3 (imaging position) is used as a pixel value.

In the first embodiment, the image pickup device 3 determines the orientation, the angle of view, the installation position, and the like so that all the seats 2 in the vehicle interior can be imaged, in other words, all the occupants in the vehicle interior can be imaged. Be done. For example, the image pickup device 3 may be installed on a dashboard, a rearview mirror, a ceiling, or the like. Not limited to this, the imaging device 3 may be arranged at a position where only an occupant seated in a specific seat 2 (for example, the driver's seat 2a) can be imaged.

Here, it is assumed that both the brightness image and the distance image are captured by using one image pickup device 3, but the vehicle 1 includes, for example, an image pickup device that captures the brightness image and an image pickup device that captures the distance image. May be provided separately. The imaging device that captures the distance image is not limited to the TOF distance image camera, and may be, for example, a stereo camera, a 3D scanner by a structured light method, or the like.

[2. Configuration of control system 100]
The vehicle 1 is provided with a control system 100 including a joint point detecting device. The configuration of the control system 100 will be described with reference to FIG. FIG. 2 is a block diagram showing a configuration of the control system 100 according to the first embodiment.

As shown in FIG. 2, the control system 100 includes an image pickup device 3, an in-vehicle device 8, an ECU 10, and an in-vehicle network 20. The ECU 10 is an example of a joint point detecting device.

The image pickup device 3 is connected to the ECU 10 via an output line such as an NTSC (National Television System Committee) cable, and outputs the captured luminance image and distance image to the ECU 10 via the output line.

The in-vehicle device 8 is a device mounted on the vehicle 1 and controlled by the ECU 10. For example, a plurality of airbag devices provided corresponding to a plurality of seats 2. The airbag device protects the occupant seated in the seat 2 from impact by deploying the airbag in the event of a collision of the vehicle 1. It is assumed that the airbag device can switch the deploying force of the airbag according to the control of the ECU 10.

The in-vehicle device 8 is not limited to the airbag device. For example, the in-vehicle device 8 may be a seat adjusting device that adjusts the position of the seat 2. For example, the seat adjusting device is provided in the driver's seat 2a and the passenger seat 2b, and adjusts the front-rear position, the height position, and the like of the driver's seat 2a and the passenger seat 2b.

The ECU 10 can control various in-vehicle devices 8 by sending control signals via the in-vehicle network 20. In addition, the ECU 10 can execute control of the brake system, control of the steering system, and the like.

The ECU 10 includes, for example, a CPU (Central Processing Unit) 11, an SSD (Solid State Drive) 12, a ROM (Read Only Memory) 13, and a RAM (Random Access Memory) 14. The CPU 11 realizes a function as a joint point detection device by executing a program installed and stored in a non-volatile storage device such as a ROM 13. The RAM 14 temporarily stores various data used in the calculation by the CPU 11. The SSD 12 is a rewritable non-volatile storage device, and can store data even when the power of the ECU 10 is turned off. The CPU 11, ROM 13, RAM 14, and the like can be integrated in the same package. The ECU 10 may have a configuration in which another logical operation processor such as a DSP (Digital Signal Processor), a logic circuit, or the like is used instead of the CPU 11. Instead of the SSD 12, a rewritable non-volatile storage device such as an HDD (Hard Disk Drive) or Flash Memory may be provided, or the SSD 12 or the HDD may be provided separately from the ECU 10.

[3. Functional configuration of ECU 10]
Next, the functional configuration of the ECU 10 will be described with reference to FIG. FIG. 3 is a block diagram showing a functional configuration of the ECU 10 according to the first embodiment.

As shown in FIG. 3, the ECU 10 includes a control unit 50 and a storage unit 60. The control unit 50 includes an acquisition unit 51, a joint point detection unit 52, a classification unit 53, a human body information generation unit 54, and a device control unit 55. The storage unit 60 stores the luminance image 61, the distance image 62, the joint point information 63, the classified joint point information 64, and the human body information 65.

The acquisition unit 51, the joint point detection unit 52, the classification unit 53, the human body information generation unit 54, and the device control unit 55 are realized by the CPU 11 executing the program stored in the ROM 13. Note that these configurations may be realized by a hardware circuit. The storage unit 60 is composed of, for example, an SSD 12.

The acquisition unit 51 acquires a luminance image and a distance image captured by the image pickup device 3 from the image pickup device 3. The acquisition unit 51 stores the acquired luminance image and distance image in the storage unit 60 (corresponding to the luminance image 61 and the distance image 62 in FIG. 3).

The joint point detection unit 52 detects a plurality of joint points included in the target area by using the luminance image 61 stored in the storage unit 60. A joint point is a characteristic point indicating the position of each part of the human body, and includes not only joints such as shoulders and elbows but also parts such as the base of the neck, nose, eyes, and ears. As a method for detecting the joint point, for example, PCM (Part Confidence Maps) can be used, but the technique is not limited to PCM, and any known technique may be used. The joint point detection unit 52 stores the detected information on the plurality of joint points in the storage unit 60 as the joint point information 63.

FIG. 4 is a diagram showing an example of joint point information. As shown in FIG. 4, the joint point information 63 includes information on the portion of each joint point in addition to the two-dimensional coordinates (X coordinate, Y coordinate) of each joint point. For example, among the plurality of joint points detected from the target region R, the joint points p1 and q1 include information indicating that the site of the joint points p1 and q1 is the “shoulder”. Similarly, the joint points p2 and q2 include information indicating that the site of the joint points p2 and q2 is the "elbow", and the joint points p3 and q3 include the site of the joint points p3 and q3 as the "neck". Information is included to indicate that.

It is not possible to determine whether the detected joint point is the joint point of the person M1 or the person M2 only from the joint point information 63. It is possible to classify multiple joint points for each human body based on PAF (Part Affinity Fields), but since PAF has a large amount of calculation, for example, it is expensive to detect the joint points of the human body in real time. An arithmetic unit with capability is required.

The classification unit 53 classifies a plurality of joint points for each person based on the distance image 62 stored in the storage unit 60.

The classification unit 53 according to the first embodiment calculates the “degree of deviation” using the distance image 62. The degree of divergence is a value indicating the degree of low connection between a certain joint point (target joint point described later) and another joint point (connection candidate joint point described later).

FIG. 5 is an explanatory diagram of a procedure for calculating the degree of deviation. FIG. 5 shows an example when the joint point p1 is selected as the target joint point.

The classification unit 53 selects a joint point of a portion adjacent to the portion of the joint point p1 which is the target joint point as a connection candidate joint point. As shown in FIG. 5, the site of the joint point p1 is the “shoulder”. Therefore, the classification unit 53 selects the joint points p2 and q2 of the “elbow”, which is a portion adjacent to the shoulder, as the connection candidate joint points.

Subsequently, the classification unit 53 calculates the degree of deviation between the joint point p1 and the joint point p2. Specifically, the classification unit 53 first calculates the coordinates of each point existing on the line segment L1 having the joint point p1 and the joint point p2 as end points. This process is performed, for example, using the Bresenham's line algorithm. Bresenham's algorithm is an algorithm for placing a plurality of consecutive points between a given start point and end point and drawing an approximate straight line. Since the Bresenham's algorithm can be implemented only by adding and subtracting integers and bit shifting, it is possible to identify a plurality of pixels existing on the line segment L1 at high speed with a small amount of calculation by using the Bresenham's algorithm.

Subsequently, the classification unit 53 acquires the distance value of the pixels located at each coordinate on the line segment L1 from the distance image 62. Then, the classification unit 53 calculates the value obtained by dividing the sum of the distance values of the plurality of pixels existing on the line segment L1 by the number of the plurality of pixels existing on the line segment L1 as the degree of deviation. That is, when the number of pixels present on the line segment L1 k, the distance value of each pixel was Z _i, the deviation degree is defined by the following equation (1).

The degree of divergence is normalized by the number of pixels k, because the number of pixels existing on the line segment differs for each line segment. The number k of pixels existing on the line segment L1 may or may not include the joint points p1 and q1 which are the end points.

The classification unit 53 calculates the degree of deviation of the line segment L2 having the joint point p1 and the joint point q2 as end points in the same procedure as described above.

In many cases, the joint points of the same person M1 such as the joint points p1 and p2 are connected by a flat surface. This is because, basically, only the pixels of the person exist on the line segment connecting the joint points of the same person, and the pixels of the background located farther than the person can exist on the line segment. Because there is no such thing.

On the other hand, there is a high possibility that there is a place where the distance becomes large on the line segment L2 connecting the joint points of different persons M1 and M2 such as the joint points p1 and q2. For example, in the case of the example shown in FIG. 5, the pixels of the rear seat 2c located behind the persons M1 and M2 are present on the line segment L2 connecting the joint point p1 and the joint point q2. Since the distance value of the pixels in this portion is larger than the distance value of the pixels in the parts of the persons M1 and M2, the degree of deviation of the combination of the joint point p1 and the joint point q2 is the combination of the joint point p1 and the joint point p2. It will be higher than the degree of divergence.

The classification unit 53 determines the joint point p2 having a low degree of divergence among the joint points p2 and q2 which are the connection candidate joint points as the joint points of the “elbow” connected to the joint point p1 which is the target joint point. Specifically, the classification unit 53 adds information (hereinafter, referred to as “person information”) indicating that the joint points p1 and the joint points p2 are the joint points of the same person M1.

In addition to the elbow described above, the classification unit 53 also classifies the "neck", which is a site adjacent to the shoulder, which is the site of the joint point p1, based on the degree of divergence. That is, the classification unit 53 sets the joint point p1 as the target joint point and the joint points p3 and q3 (see FIG. 4) as connection candidate joint points, and the combination of the joint points p1 and p3 and the combination of the joint points p1 and q3 deviate from each other. The degree is calculated, and the connection candidate joint point having the lower degree of deviation, that is, the joint point p3, is determined as the joint point of the "neck" connected to the joint point p1.

When the classification of all combinations of all the joint points is completed in this way, the classification unit 53 stores the classified joint point information 64 in which the person information is added to the joint point information 63 in the storage unit 60. The classified joint point information 64 is information including the two-dimensional coordinates of each joint point, the site of each joint point, and the person information of each joint point. As a result, the same person information (for example, M1) is added to the joint points of the person M1 including the joint points p1 to p3. Further, the same person information (for example, M2) is added to the joint points of the person M2 including the joint points q1 to q3. By adding the person information to the joint point information 63 in this way, it is possible to classify the plurality of joint points detected from the target area R for each person.

The human body information generation unit 54 generates human body information including the length of each part of the occupant based on the classified joint point information 64 stored in the storage unit 60. For example, the human body information generation unit 54 can obtain information on the shoulder width of the person M1 by calculating the distance between the joint point p1 of the left shoulder and the joint point of the right shoulder for the person M1. The human body information generation unit 54 generates human body information for each person M1 and M2 using the classified joint point information 64 classified for each person M1 and M2. Then, the human body information generation unit 54 stores the generated human body information 65 in the storage unit 60.

The human body information generation unit 54 may estimate the postures of the persons M1 and M2 based on the classified joint point information 64.

The device control unit 55 controls various in-vehicle devices 8 mounted on the vehicle 1 based on the human body information 65 stored in the storage unit 60. As an example, the device control unit 55 can control the airbag device as the in-vehicle device 8 based on the human body information 65. For example, when the shoulder width of the occupant included in the human body information 65 is equal to or less than the threshold value, it is determined that the occupant is a child, and the deployment mode of the airbag device is set to the weak deployment mode in which the airbag device is deployed weaker than the normal mode for adults. You may switch. In this way, by deploying the airbag with strength according to the physique of the occupant, the safety of the airbag can be enhanced.

Further, the device control unit 55 automatically sets the front-rear position, the height position, etc. of the seat 2 according to the height of the occupant, etc. by controlling the seat adjustment device as the in-vehicle device 8 based on the human body information 65. Can be adjusted. As a result, it is possible to save the occupant from having to adjust the position of the seat 2 by himself / herself.

[4. Specific operation of ECU 10]
Next, the specific operation of the ECU 10 will be described with reference to FIG. FIG. 6 is a flowchart showing a procedure of processing executed by the ECU 10 according to the first embodiment. Note that FIG. 6 shows a procedure for determining one combination of one target joint point.

As shown in FIG. 6, the ECU 10 selects the joint point having the smallest distance value among the plurality of joint points detected from the target region R as the target joint point (step S101). Subsequently, the ECU 10 selects a joint point of a portion adjacent to the portion of the target joint point as a connection candidate joint point among the plurality of joint points detected from the target region R (step S102).

Subsequently, the ECU 10 calculates the coordinates of each point on the line segment connecting the target joint point and the connection candidate joint point (step S103), and calculates the degree of deviation using the calculated distance value of each coordinate (step S103). S104).

Subsequently, the ECU 10 determines whether or not the deviation degrees of all the connection candidate joint points have been calculated (step S105). In this process, if there is a connection candidate joint point for which the degree of deviation has not been calculated (steps S105, No), the ECU 10 returns the process to step S103, and steps for the connection candidate joint point for which the degree of deviation has not been calculated. Processes S103 to S105 are performed.

On the other hand, when it is determined in step S105 that the deviation degrees of all the connection candidate joint points have been calculated (steps S105, Yes), the ECU 10 connects the connection candidate joint points having the lowest degree of deviation to the target joint points. It is determined as a joint point (step S106), and the process for one combination of one target joint point is completed.

After determining all combinations for one target joint point (for example, if the target joint point is a shoulder, "shoulder and elbow", "shoulder and neck"), the ECU 10 is the distance next to the target joint point. A joint point having a small value is selected as the target joint point (step S101), and the processes after step S102 are performed. At this time, the ECU 10 excludes the target joint points for which all combinations have been determined from the connection candidate joint points for the next target joint point.

For example, if the joint point of a person seated in the back seat 2c is selected as the target joint point, and the joint point of the person seated in the driver's seat 2a is selected as the connection candidate joint point, the joint points of different persons are selected. The degree of divergence may be lower than the degree of divergence between the joint points of the same person. On the other hand, by selecting the target joint points in order from the joint point closest to the distance and not selecting the target joint point for which all combinations have been determined as the connection candidate joint point for the next target joint point. The degree of divergence can be calculated appropriately.

(Second Embodiment)
Next, a method for classifying a plurality of joint points according to the second embodiment will be described with reference to FIGS. 7 to 9. 7 to 9 are explanatory views of the procedure for calculating the degree of agreement.

The classification unit 53 according to the second embodiment calculates the “matching degree” using the distance image 62. The degree of coincidence is the direction of the vector connecting the target joint point and the connection candidate joint point, and the vector between each division point when the line segment connecting the target joint point and the connection candidate joint point is divided by a plurality of division points. It is the degree of agreement with the direction of.

For example, as shown in FIG. 7, the classification unit 53 calculates the vector va0 between the joint points from the target joint point p1 to the connection candidate joint point p2 based on the distance value of the distance image 62. To do.

Specifically, the three-dimensional coordinates of the articulation points p1, p2 (X coordinate, Y coordinate, and distance value), respectively _{P1 = (X 1, Y 1} , Z 1), P2 = (X 2, Y 2, Z ₂ ), the vector between the joint points va0 is (X _2- X ₁ , Y _2- Y ₁ , Z _2- Z ₁ ).

Further, the classification unit 53 sets a plurality of division points a1, a2, and a3 that evenly divide the line segment on the two-dimensional coordinates connecting the joint point p1 and the joint point p2. Then, the classification unit 53 includes a division point vector va1 from the joint point p1 to the division point a1, a division point vector va2 from the division point a1 to the division point a2, and a division point vector from the division point a2 to the division point a3. The vector between the division points va4 from the division point a3 to the joint point p2 is calculated based on the distance value of the distance image 62.

Specifically, the three-dimensional coordinates of the division points a1 to a3 are a1 = (X _a1 , Y _a1 , Z _a1 ), a2 = (X _a2 , Y _a2 , Z _a2 ), a3 = (X _a3 , Y _a3 ), respectively. , Z _a3 ), the inter-division point vector va1 is (X _a1- X ₁ , Y _a1- Y ₁ , Z _a1- Z ₁ ), and the inter-division point vector va2 is (X _a2- X _a1 , Y _{a2). -Y a1, Z a2 -Z a1)} , division point between vectors va3 _{is, (X a3 -X a2, Y} a3 -Y a2, Z a3 -Z a2), division point between vectors va4 _{is, (X} 2 -X _{a3, Y} 2 _{-Y a3,} a Z 2 _{-Z a3).}

Then, the classification unit 53 calculates the degree of coincidence of the directions of the vectors with respect to the joint points p1 and p2. Here, the degree of coincidence is defined as 1- (sin ² θ ₁ + sin ² θ ₂ + ... + sin ² θ _k ) / k. “K” is the number of divisions of the line segment, and in the case of the example shown in FIG. 7, “k” is 4. Further, "θ ₁ " is the angle formed by the vector between the division points and the vector between the joint points. In the case of the example shown in FIG. 7, "θ ₁ " is the vector between the division points va1 and the vector between the joint points va0. It is the angle between the two.

In addition, 1- (1-cos ² θ ₁ + sin ² θ ₂ + ... + sin ² θ _k ) / k is 1- (1-cos ² θ ₁ + 1-cos ² θ ₂ + ... + 1-cos ² θ. It can be converted to _k ) / k. Therefore, the classification unit 53 further converts this to {((va1 · va0) / (| va1 | · | va0 |)) ² + ((va2 · va0) / (| va2 | · | va0 |)). ² + ... ((Vak · va0) / (| vak | · | va0 |)) ² } / k is used to calculate the degree of coincidence.

Similarly, the classification unit 53 calculates the inter-joint vector vb0 from the target joint point p1 to the connection candidate joint point q2. Further, the classification unit 53 sets a plurality of division points b1, b2, b3 for dividing the line segment connecting the joint point p1 and the joint point q2, and the vector between the division points vb1 from the joint point p1 to the division point b1. The inter-division vector vb2 from the division point b1 to the division point b2, the inter-division vector vb3 from the division point b2 to the division point b3, and the inter-division vector vb4 from the division point b3 to the joint point q2 are calculated. Then, the classification unit 53 calculates the degree of coincidence using the vector between the joint points vb0 and the vectors vb1 to vb4 between the division points.

Here, the degree of agreement when the joint points of the same person M1 as the joint points p1 and p2 are set as the target joint point and the connection candidate joint point is theoretically 1. For example, as shown in FIG. 8, if the target joint point (joint point p1) and the connection candidate joint point (joint point p2) are the joint points of the same person M1, the direction of the vector between the joint points va0 , The directions of the vectors va1 to va4 between the dividing points are theoretically the same, and "θ ₁ " to "θ _k " are all 0, so that "1-cos ² θ ₁ " and "1-cos ² θ" _This is because the values of " ₂ " ... "1-cos ² θ _k " are all 0.

On the other hand, when the joint points of different persons M1 and M2 such as the joint points p1 and q2 are set as the target joint points and the connection candidate joint points, the degree of coincidence is lower than 1. For example, as shown in FIG. 9, when the target joint point (joint point p1) and the connection candidate joint point (joint point q2) are different joint points of the persons M1 and M2, the vector between the joint points vb0 This is because the orientation does not match the orientation of the vectors vb1 to vb4 between the division points.

The classification unit 53 determines the joint point p2 having a high degree of coincidence among the joint points p2 and q2 which are the connection candidate joint points as the joint points of the “elbow” connected to the joint point p1 which is the target joint point. Specifically, the classification unit 53 adds personal information indicating that the joint points p1 and the joint points p2 are the joint points of the same person M1.

Next, the specific operation of the ECU 10 according to the second embodiment will be described with reference to FIG. FIG. 10 is a flowchart showing a procedure of processing executed by the ECU according to the second embodiment. FIG. 10 shows a procedure for determining one combination of one target joint point.

As shown in FIG. 10, the ECU 10 selects the joint point having the smallest distance value among the plurality of joint points detected from the target region R as the target joint point (step S201). Subsequently, the ECU 10 selects a joint point of a portion adjacent to the portion of the target joint point as a connection candidate joint point among the plurality of joint points detected from the target region R (step S202).

Subsequently, the ECU 10 calculates the vector between the joint points (step S203), calculates the vector between the division points (step S204), and calculates the degree of agreement (step S205).

Subsequently, the ECU 10 determines whether or not the degree of coincidence of all the connection candidate joint points has been calculated (step S206). In this process, if there is a connection candidate joint point for which the degree of coincidence has not been calculated (steps S206, No), the ECU 10 returns the process to step S203, and steps for the connection candidate joint point for which the degree of coincidence has not been calculated. Processes S203 to S206 are performed.

On the other hand, when it is determined in step S206 that the matching degrees of all the connection candidate joint points have been calculated (steps S206, Yes), the ECU 10 connects the connection candidate joint points having the highest matching degree to the target joint points. It is determined as a joint point (step S207), and the process for one combination of one target joint point is completed.

(Modification example)
The ECU 10 may calculate both the degree of deviation and the degree of coincidence described above, and determine the joint point connected to the target joint point based on the degree of deviation and the degree of agreement. At this time, the ECU 10 may weight either the degree of deviation or the degree of coincidence according to the situation such as the brightness of the surroundings.

In the above-described embodiment, an example in which the joint point detection unit 52 generates the joint point information 63 using the luminance image 61 has been described. Not limited to this, the joint point detection unit 52 may generate the joint point information 63 using the distance image 62. Further, the joint point detection unit 52 may generate the joint point information 63 by using both the luminance image 61 and the distance image 62.

In the above-described embodiment, an example is shown in which the ECU 10 as the joint point detection device is used to acquire the human body information of the occupant existing in the vehicle interior of the vehicle 1, but the joint point detection device can be used, for example, in a factory. It can also be used for other purposes such as acquiring human body information such as the posture of a worker in order to monitor the load of the worker.

As described above, the joint point detection device (for example, ECU 10) according to the embodiment includes a distance image (as an example, a distance image 62) and a brightness image (for example, an example) of a target region (for example, the target region R). A detection unit (for example, a joint point detection unit 52) that detects a plurality of joint points included in the target region based on at least one of the brightness images 61), and a plurality of joint points for each human body based on a distance image. It is provided with a classification unit (as an example, a classification unit 53) for classification. By classifying a plurality of joint points included in the target region for each human body based on a distance image, it is possible to reduce the amount of calculation as compared with a conventional classification method using, for example, PAF.

In the above-mentioned joint point detection device, the classification unit is a line connecting a target joint point among a plurality of joint points and one of a plurality of connection candidate joint points that are candidates for a joint point connected to the target joint point. The degree of divergence based on the sum of the distances of the points existing on the line segment is calculated for each of a plurality of connection candidate joint points. Then, the classification unit determines the connection candidate joint point having the lowest degree of divergence among the plurality of connection candidate joint points as the joint point connected to the target joint point.

On the line segment connecting the joint points of different people, there is a high possibility that there is a place where the distance becomes large, such as a background part. Therefore, the connection candidate joint point with a low degree of divergence is determined as the joint point connected to the target joint point. As a result, a plurality of joint points acquired from the image can be classified for each human body with a small amount of calculation.

In the above-mentioned joint point detection device, the classification unit selects the target joint points in order from the joint point having the shortest distance among the plurality of joint points. In this way, the degree of divergence can be appropriately obtained by selecting the target joint points in order from the joint point having the closest distance.

In the joint point detection device, the classification unit calculates the coordinates of each point existing on the line segment using Bresenham's algorithm. Since the Bresenham's algorithm can be implemented only by adding and subtracting integers and bit shifting, the coordinates of each point existing on the line segment can be calculated at high speed with a small amount of calculation by using the Bresenham's algorithm.

In the above-mentioned joint point detection device, the classification unit connects a target joint point among a plurality of joint points and one of a plurality of connection candidate joint points that are candidates for a joint point connected to the target joint point. Calculate the vector between points. Further, the classification unit calculates a vector between a plurality of division points connecting the plurality of division points for dividing the line segment connecting the target joint point and one of the plurality of connection candidate joint points. Further, the classification unit calculates the degree of coincidence between the directions of the vector between the joint points and the vector between the plurality of division points for each of the plurality of connection candidate joint points. Then, the classification unit determines the connection candidate joint point having the highest degree of coincidence among the plurality of connection candidate joint points as the joint point connected to the target joint point.

When the joint points of the same person are set as the target joint point and the connection candidate joint point, the direction of the vector between the joint points and the direction of the vector between the plurality of division points theoretically match, so that the degree of coincidence in this case is high. .. On the other hand, the degree of coincidence when the joint points of different persons are set as the target joint points and the connection candidate joint points is smaller than that when the joint points of the same person are set as the target joint points and the connection candidate joint points. Become. This is because the directions of the vector between the joint points and the directions of the vectors between the plurality of division points do not match when the target joint points and the connection candidate joint points are the joint points of different persons. Therefore, the classification unit determines the connection candidate joint point having a high degree of coincidence among the plurality of connection candidate joint points as the joint point connected to the target joint point. As a result, a plurality of joint points acquired from the image can be classified for each human body with a small amount of calculation.

Although the embodiments of the present disclosure have been illustrated above, the above-described embodiments and modifications are merely examples, and are not intended to limit the scope of the invention. The above-described embodiment and modification can be implemented in various other forms, and various omissions, replacements, combinations, and changes can be made without departing from the gist of the invention. Further, the configuration and shape of each embodiment and each modification can be partially exchanged.

1 ... Vehicle, 2 ... Seat, 3 ... Imaging device, 8 ... In-vehicle device, 10 ... ECU, 50 ... Control unit, 51 ... Acquisition unit, 52 ... Joint point detection unit, 53 ... Classification unit, 54 ... Human body information generation unit , 55 ... device control unit, 60 ... storage unit, 61 ... brightness image, 62 ... distance image, 63 ... joint point information, 64 ... classified joint point information, 65 ... human body information.

Claims (5)

A detection unit that detects a plurality of joint points included in the target area based on at least one of a distance image and a brightness image of the target area.
A joint point detection device including a classification unit that classifies the plurality of joint points for each human body based on the distance image. The classification unit
Each point existing on a line connecting a target joint point among the plurality of joint points and one of a plurality of connection candidate joint points that are candidates for the joint point connected to the target joint point. The degree of divergence based on the total distance is calculated for each of the plurality of connection candidate joint points, and the connection candidate joint point having the smallest degree of divergence among the plurality of connection candidate joint points is connected to the target joint point. The joint point detection device according to claim 1, which is determined as the joint point. The classification unit
The joint point detecting device according to claim 2, wherein the target joint point is selected in order from the joint point having the closest distance among the plurality of joint points. The classification unit
The joint point detection device according to claim 3, wherein the coordinates of each point existing on the line segment are calculated by using the Bresenham's algorithm. The classification unit
A vector between the target joint points connecting the target joint point among the plurality of joint points and one of the plurality of connection candidate joint points that are candidates for the joint point connected to the target joint point is calculated. A line segment connecting a target joint point and one of the plurality of connection candidate joint points is divided. A plurality of division point vectors connecting the plurality of division points are calculated, and the joint point vector and the plurality of divisions are calculated. The degree of coincidence of the orientation with the point-to-point vector is calculated for each of the plurality of connection candidate joint points, and the connection candidate joint point having the highest degree of coincidence among the plurality of connection candidate joint points is connected to the target joint point. The joint point detection device according to any one of claims 1 to 4, which is determined as the joint point to be used.

Images