JP7117878B2 - processor and program - Google Patents

Description

The present invention relates to a processing device and program, and more particularly to a processing device and program that perform detection processing based on a captured image.

Conventionally, there is a technique of capturing an image in which an occupant having coordinate information on a two-dimensional plane is captured and processing the captured image to identify characteristic points (for example, joint points) of the occupant (for example, See Patent Document 1).

The processing device of Patent Document 1 includes image acquisition means for repeatedly acquiring images including a plurality of operation targets and an operator existing at a position where the operation targets can be operated, and images repeatedly acquired by the image acquisition means. each time, based on the human body feature point specifying means for specifying a predetermined human body feature point of the operator included in the image, and the human body feature point for each image specified by the human body feature point specifying means, the operator an operation estimating means for estimating the content of an operation to be performed, the operation estimating means estimating a posture trajectory (estimated posture trajectory) that is estimated to be followed by an operator who operates an operation target for each operation content. with the operator's posture trajectory (actual posture trajectory) obtained from each of the human body feature points for each image specified by the human body feature point specifying means; Among the estimated posture trajectories modeled in , for the estimated posture trajectory whose degree of approximation to the actual posture trajectory satisfies a predetermined threshold value, the operator is about to perform the operation content corresponding to the estimated posture trajectory. The processing unit is configured to estimate that

Japanese Patent Application Laid-Open No. 2008-140268

However, in the technique of Patent Document 1, since the feature points of the occupant are specified from the image having the coordinate information on the two-dimensional plane, it is required to improve the accuracy of specifying the feature points.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a processing apparatus and a program for improving the accuracy of specifying feature points in a technique for specifying feature points of a human body.

[1] In order to achieve the above object, an acquisition unit for acquiring a distance image in which an imaging area including an occupant of a vehicle is an imaging target, the distance image obtained by assigning distance information to the imaging target to pixels; According to the correspondence relationship between the coordinate system in the interior space of the vehicle and the coordinate system in the range image, the detection image is converted into a detection image having a specific coordinate system, and at least one feature point of the occupant is detected in the detection image. and a control unit that executes feature point detection processing.
[2] The control unit identifies one defined feature point, and with the one defined feature point as a reference, sequentially collates predefined feature point search models to detect the remaining feature points. The processing device according to the above [1] may be configured to execute, as the feature point detection processing, to
[3] Further, the control unit considers a set of voxels including voxels located around the feature point, and determines whether the sum of voxel values in the set of voxels is less than a threshold, and determines the likelihood of the feature point. The processing device according to the above [1] or [2] may be configured to execute a threshold process for determining as one of the feature point detection processes.
[4] Further, the control unit determines whether the sum of the voxel values is less than a threshold value, and divides the set of voxels from an origin that is one specified feature point to another feature point The processing apparatus according to [3] above may be used, wherein the voxel group exists at a predetermined interval between the target feature point.
[5] In addition, if the sum of voxel values in each of all voxel groups from the starting point to the target feature point is equal to or greater than a threshold value, the control unit determines that the target feature point is probable. 4] may be used.
[6] In addition, in the voxel group from the starting point to the target feature point, even if the sum of the voxel values in the voxel group of the number indicating the specified distance is less than a threshold, the target feature point The processing apparatus according to the above [4] or [5] may be one in which it is probable that
[7] Further, the processing device according to any one of [1] to [6] above, wherein, when the feature point is not detected, the control unit performs complementary processing using detection results up to the previous frame. may be
[8] The processing device according to any one of [1] to [7] above, wherein the defined one feature point is a head.
[9] In order to achieve the above object, an acquisition step of acquiring a distance image in which an imaging area including an occupant of a vehicle is an imaging target, the distance image obtained by assigning distance information to the imaging target to pixels; A feature inspection that converts into a detection image having a specific coordinate system according to the correspondence relationship between the coordinate system in the vehicle interior space and the coordinate system in the range image, and detects at least one feature point of the occupant in the detection image. A program for causing a computer to execute a control step for executing output processing is provided.
[10] The control step identifies one specified feature point, and with the one specified feature point as a reference, sequentially collates predefined feature point search models to detect the remaining feature points. The program according to the above [9] may be executed as the feature point detection process.
[11] The control step considers a set of voxels including voxels located around the feature point, and determines the likelihood of the feature point based on whether the sum of voxel values in the set of voxels is less than a threshold. The program according to the above [9] or [10] may be configured to execute the threshold processing for detecting the feature points as one of the feature point detection processing.
[12] In addition, the control step determines whether the sum of the voxel values is less than a threshold, and divides the set of voxels from an origin that is one specified feature point to another feature point. The program according to the above [11] may be such that a voxel group exists at a predetermined interval from the target feature point to .
[13] In addition, in the control step, if the sum of voxel values in each of all voxel groups from the starting point to the target feature point is equal to or greater than a threshold, the target feature point is likely to be the above [ 12].
[14] Further, in the control step, in the voxel group from the starting point to the target feature point, even if the sum of the voxel values in the voxel group of the number indicating the prescribed distance is less than a threshold value, the target feature point may be the program described in [12] or [13] above, which makes it probable that
[15] Further, in the program according to any one of [9] to [14] above, in the control step, when the feature point has not been detected, interpolation processing is performed using the detection result up to the previous frame. It can be.
[16] Further, the control step may be the program according to any one of [9] to [15] above, wherein the defined single feature point is the head.

According to the processing device and the program of the present invention, it is possible to improve the accuracy of specifying the feature points in the technique of specifying the feature points of the human body.

FIG. 1(a) is a diagram of a skeleton model defining the size of each skeleton of the human body and the range of motion of joints, and FIG. 1(b) is an example of defining the size of each skeleton of the human body and the range of motion of joints. FIG. 1(c) is a diagram showing the forearm as an example of the movable range (angle defined in the skeleton model definition) of each part (shoulder, elbow, wrist, hand) is. FIG. 2 is a coordinate diagram three-dimensionally showing the relationship of the coordinate system when the processing device according to the embodiment of the present invention is mounted on a vehicle. FIG. 3 is a diagram showing an example of a three-dimensional pixel group. FIG. 4(a) is a diagram showing an example of voxel creation considering the size of the vehicle interior space, and FIG. 4(b) is a chart showing an example of voxel creation parameter settings. FIG. 5(a) is a diagram showing a search range of voxels included in the length and movable range defined by the skeleton model, and FIG. 5(b) is a chart showing examples of search range narrowing parameters. FIG. 6 is a diagram showing an example of 3×3×3 voxels including one surrounding voxel. FIG. 7 is a diagram showing an example in which 3×3×3 voxels are arranged at intervals of X [mm] from the starting point P. FIG. FIG. 8 is a chart showing an example of parameter setting in each part detection. FIG. 9 is a flow chart showing the operation of the processing device according to the embodiment of the present invention. FIG. 10 is a diagram showing the detected shoulder, elbow, wrist, and hand positions on an image.

(Embodiment of the present invention)
The processing device 1 according to the embodiment of the present invention acquires a distance image obtained by assigning distance information to the imaging target to pixels, which is a distance image whose imaging target is an imaging region including an occupant 5 of a vehicle 8. According to the correspondence relationship between the TOF camera 10 as a unit and the coordinate system in the interior space of the vehicle 8 and the coordinate system in the range image, the detection image is converted into a detection image having a specific coordinate system. and a control unit 20 that executes a feature point detection process for detecting at least one feature point.

The processing device 1 according to the present embodiment uses the TOF camera 10 to perform feature point detection processing for detecting parts such as the shoulders, elbows, wrists, and hands of the vehicle occupant as feature points. A pixel group (three-dimensional pixel group) is created by transforming each pixel of the range image obtained from the TOF camera 10 into a three-dimensional space, and a skeleton model (definition of the skeleton size and movable range of the human body) is created with the head as the starting point. Based on this, parts such as shoulders, elbows, wrists, and hands are detected as feature points.

Note that the pixel group is a set of points corresponding to the pixels captured by the TOF camera 10, and is a three-dimensional image whose position is displayed by a coordinate system in the vehicle interior space of the vehicle 8, a coordinate system in the TOF camera 10, or the like. A set of points in space.

(skeleton model)
The skeleton model, as shown in FIG. 1(a), defines the size of each skeleton of the human body and the movable range of joints, and an example is shown in FIG. 1(b). This definition can be determined, for example, based on information such as the human body size database published by the National Institute of Advanced Industrial Science and Technology and the human characteristics database published by the National Institute of Technology and Evaluation. The movable range (angle defined in the skeleton model definition) of each part (shoulder, elbow, wrist, hand) is defined as shown in FIG. 1(c), taking the forearm as an example.

(TOF camera 10)
Any acquisition unit capable of three-dimensional recognition of an imaging target can be used. In the present embodiment, a TOF (Time Of Flight) camera 10 is used as the acquisition unit. The TOF camera 10 can detect the time for each pixel until the light from the light source hits the object to be measured and returns, and can capture a three-dimensional image including position information corresponding to the distance in the depth direction. After emitting infrared light or the like, the TOF camera 10 receives the reflected light that is reflected by the object and returns, measures the time from the light emission to the light reception, and calculates the distance to the imaging object for each pixel. to detect

The TOF camera 10 as an acquisition unit acquires a distance image in which an imaging area including the occupant 5 of the vehicle 8 is an imaging target, and in which distance information to the imaging target is assigned to each pixel. This distance image can be obtained as one frame of frames captured at predetermined time intervals. The acquired range image functions as an input (three-dimensional pixel group) in the following processing. The distance image is transformed into the camera coordinate system and vehicle coordinate system described above. Thereby, a three-dimensional pixel group included in the detection area can be extracted.

For example, as shown in FIG. 2, the TOF camera 10 is attached near the rearview mirror, and captures an imaging area including the occupant 5 of the vehicle 8 as an imaging target. An image captured by the TOF camera 10 includes coordinates (u, v) and a pixel value d(u, v) as depth information at the coordinates (u, v). The obtained pixel values d(u, v) (u=0, 1, . . . U−1, v=0, 1, . or in-vehicle equipment). In addition, the code|symbol U in embodiment means the horizontal width [pixel] in a captured image, and the code|symbol V means the vertical width [pixel] in a captured image. That is, a distance image is generated by assigning distance information in a three-dimensional space to each pixel in the image captured by the TOF camera 10 .

なお、（ｃ_ｘ、ｃ_ｙ）は、画像中心座標、ｆは、レンズ焦点距離である。 (Camera coordinate system obtained from range image)
The coordinate values u, v in the range image and the pixel value d (u, v) at the coordinates (u, v) are converted to a point (x, y, z) in the three-dimensional space using the following formula. , a three-dimensional pixel group in the camera coordinate system (x, y, z) shown in FIG. 2 can be created.

Note that (c _x , c _y ) are the image center coordinates, and f is the lens focal length.

(vehicle coordinate system)
The camera coordinate system (x, y, z) can be transformed into a vehicle coordinate system (w, l, h) with the driver to be detected (for example, the position of the hipbone) as the origin. The transformation method is a combination of general coordinate rotation transformation, translation transformation, and scale transformation. The coordinate axes after conversion are the w-axis in the right direction of the driver, the l-axis (L-axis) in front, and the h-axis in the upper direction, and the origin is the hipbone position 203 of the driver. FIG. 3 shows an example of a three-dimensional pixel group.

(Voxel creation)
As shown in FIG. 4(a), the control unit 20 considers the size of the vehicle interior space, and creates voxels (three-dimensional space of a fixed size) with the size of the voxel creation parameter setting example shown in FIG. to divide it into a grid and express the area discretely). Each voxel has a voxel value, and the density information of the three-dimensional pixel group (here, the number of three-dimensional pixel groups included in each voxel×3) is used as the voxel value.

(control unit 20)
The control unit 20 includes, for example, a microcomputer for performing coordinate conversion, parietal region identification processing, head identification processing, and the like. The control unit 20 is connected to the TOF camera 10 as shown in FIG. The control unit 20 includes a CPU (Central Processing Unit) 21 that performs calculations and processing on acquired data according to a stored program, a semiconductor memory such as a RAM (Random Access Memory) 22 and a ROM (Read Only Memory) 23. I have.

(head detection, identification)
The head is one defined feature point, and in this embodiment, the remaining feature points (left and right shoulders, elbows, wrists, hands) are obtained by sequentially collating predefined feature point search models. It is a standard for detecting Any method can be used to detect and specify the head, and the head can be detected and specified by a known method. Also, three-dimensional data (w, l, h) of the head may be input to the control unit 20 instead of detection and identification by the control unit 20 .

(Search range narrowing down)
As shown in FIG. 5A, the control unit 20 searches for and extracts voxels included in the length and movable range defined by the skeleton model. The movable range is defined as a conical shape defined by a generatrix 250 shown in FIG. The search is performed based on search range narrowing parameters as shown in FIG. 5(b).

(Part candidate extraction as feature points)
(1) Threshold Processing For each voxel whose search range has been narrowed down, the control unit 20 considers a set of voxels including surrounding voxels. Here, as shown in FIG. 6, 3×3×3 voxels including one surrounding voxel are used. Here, if the total voxel value of the voxel is less than the threshold, it is excluded from the candidates. That is, the threshold processing considers a set of voxels including voxels located around the feature point, and determines the likelihood of the feature point based on whether the sum of the voxel values in this set of voxels is less than the threshold. .

(2) Continuity check The control unit 20, for each voxel extracted by the threshold processing in (1), as shown in FIG. Considering (3×3×3 voxels), if the sum of voxel values at all positions is equal to or greater than the threshold, the voxel is determined as a part candidate. However, considering wearable objects such as wristwatches that may not be able to acquire information with a TOF camera, the number of allowable gaps is set so that even if there is a point that does not satisfy the voxel value threshold between the starting point P, it is acceptable. Define. For example, when the number of allowable gaps is 1, it is determined that there is continuity if only one point does not satisfy the threshold.

The control unit 20 determines whether the sum of the voxel values is less than a threshold, and divides the set of voxels from the starting point, which is one specified feature point, to the target feature point, which is another feature point. A group of voxels may be present at predetermined intervals between them. With this, when the probability is high, the target feature point can be regarded as highly probable.

Furthermore, the control unit 20 can determine that the target feature point is probable if the sum of the voxel values in each of all the voxel groups from the starting point to the target feature point is equal to or greater than the threshold. As a result, even if a part of the feature point is missing due to noise or the like, the target feature point can be made probable.

FIG. 8 shows an example of parameter settings for detection of each part, and the detection of each part shown below is executed based on these parameters. The above-mentioned (1) threshold processing and (2) continuity check are used to detect each part as part detection common logic. The unit cubic size is the cubic size of voxels used for density calculation. Also, the density threshold is a density threshold used for rejecting termination candidates and confirming continuity. The continuity step length is the width between central voxels when the unit cube is shifted when confirming continuity. Also, the number of allowable gaps is the number of steps whose density is equal to or less than the threshold, which is allowed when confirming continuity.

(shoulder detection)
The control unit 20 calculates the neck position from the detected or specified head position. Here, the neck position is a point that is offset directly below the coordinates of the head position. For example, the offset value is 100 mm. With the front part starting point as the head position and the starting point P as the neck position, shoulder candidates are extracted using the part detection common logic. However, considering the possibility that the neck position is located in a voxel with a low voxel value, no continuity check is performed. A shoulder candidate on the left side (−w axis side) of the neck position is taken as a left shoulder candidate, and a shoulder candidate on the right side (+w axis side) is taken as a right shoulder candidate.

(elbow detection)
The control unit 20 extracts an elbow candidate using the part detection common logic, with the front part starting point as the neck position and the starting point as the shoulder position. The control unit 20 performs a continuity check. If the continuity check is NG and the elbow candidate point is 0, it is considered that the shoulder is floating in the air (located at a position with a low voxel value), and the position is corrected and searched again. The correction method is, for example, selecting voxels having voxel values equal to or greater than the nearest threshold. The center of gravity of the extracted elbow candidate is set as the elbow position. The position starting from the left shoulder is the left elbow, and the position starting from the right shoulder is the right elbow.

(wrist detection)
The control unit 20 extracts a wrist candidate using the part detection common logic, with the front part starting point as the shoulder position and the starting point as the elbow position. The control unit 20 performs a continuity check, and if the continuity check is NG, corrects the position and performs a search again. The center of gravity of the extracted wrist candidate is set as the wrist position. A position starting from the left elbow is defined as a left wrist, and a position starting from the right elbow is defined as a right wrist.

(hand detection)
The control unit 20 extracts a hand candidate using the part detection common logic, with the elbow position as the front part starting point and the wrist position as the starting point. The control unit 20 performs a continuity check, and if the continuity check is NG, corrects the position and performs a search again. The center of gravity of the extracted hand candidate is set as the hand position. The position starting from the left wrist is defined as the left hand, and the position starting from the right wrist is defined as the right hand.

(Response when not detected)
When each part is not detected as a feature point (when the shoulder is hidden, when the wrist is cut by a wristwatch, etc.), the control unit 20 performs complementation processing using the detection results up to the previous frame. For example, there is a method that reuses the detection result of one frame before as it is, and a method that utilizes a tracking technique such as the KCF method.

(Operation of processing device 1)
Description will be made based on the flowchart showing the operation of the processing device 1 according to the embodiment of the present invention shown in FIG. The control unit 20 executes the following calculations and processes according to the flow charts.

(Pretreatment, Step 1)
A camera image from the TOF camera 10 is input to the control unit 20 . This camera image is input as one frame of continuous frame images. As preprocessing for the feature point detection processing, the control unit 20 converts the camera image into the vehicle coordinate system and extracts a three-dimensional pixel group as shown in FIG. Also, voxels are created based on voxel creation parameter setting examples. Also, the head is detected and identified as a reference for searching for feature points.

(Shoulder detection, Step 2)
As shown in FIG. 10 , the control unit 20 calculates a neck position 305 as a point offset by a predetermined distance from the head position 300 , which is one specified reference feature point. With the neck position 305 as the starting point, shoulder candidates are extracted using the part detection common logic. As shown in FIG. 10, the shoulder candidate on the left side (−w axis side) of the neck position is the left shoulder candidate, and the shoulder candidate on the right side (+w axis side) is the right shoulder candidate. A right shoulder position 310 is assumed.

(Step 3)
The control unit 20 determines whether or not the shoulder has been successfully detected. The control unit 20 can determine whether or not the shoulder has been successfully detected by (1) threshold processing and (2) continuity check described above. However, as far as shoulder detection is concerned, the possibility that the neck position is located in a voxel with a low voxel value is taken into consideration, and no continuity check is performed. If the shoulder detection is successful, the process proceeds to Step 5 (Step 3: Yes), and if the shoulder detection is not successful, the process proceeds to Step 4 (Step 3: No).

(Complementary processing, Step 4)
When the shoulder region is not detected (when the shoulder is hidden, etc.), the control unit 20 performs complementation processing using the detection results up to the previous frame. For example, there is a method that reuses the detection result of one frame before as it is, and a method that utilizes a tracking technique such as the KCF method.

(Elbow detection, Step 5)
The control unit 20 extracts the elbow candidate using the part detection common logic, with the neck position 305 as the front part starting point and the shoulder positions 310 and 311 as the starting point. If the continuity check is NG and the elbow candidate point is 0, it is considered that the shoulder is floating in the air (located at a position with a low voxel value), and the position is corrected and searched again. The correction method is, for example, selecting voxels having voxel values equal to or greater than the nearest threshold. The center of gravity of the extracted elbow candidate is set as the elbow position. As shown in FIG. 10, a left elbow 321 is a position starting from the left shoulder, and a right elbow 320 is a position starting from the right shoulder.

(Step 6)
The control unit 20 determines whether or not the elbow has been successfully detected. The control unit 20 can determine whether or not the elbow has been successfully detected by (1) threshold processing and (2) continuity check described above. If the elbow detection is successful, the process proceeds to Step 8 (Step 6: Yes), and if the elbow detection is not successful, the process proceeds to Step 7 (Step 6: No).

(Complementary processing, Step 7)
When the elbow region is not detected (when the elbow is hidden, etc.), the control unit 20 performs complementation processing using the detection results up to the previous frame. For example, there is a method that reuses the detection result of one frame before as it is, and a method that utilizes a tracking technique such as the KCF method.

(Wrist detection, Step 8)
The control unit 20 uses shoulder positions 310 and 311 as starting points of the front part and elbow positions 320 and 321 as starting points, and extracts wrist candidates using the common logic for part detection. The center of gravity of the extracted wrist candidate is set as the wrist position. As shown in FIG. 10, the left wrist 331 is the starting point of the left elbow, and the right wrist 330 is the starting point of the right elbow.

(Step 9)
The control unit 20 determines whether or not the wrist has been successfully detected. The control unit 20 can determine whether or not the wrist has been successfully detected by (1) threshold processing and (2) continuity check described above. If the wrist detection is successful, the process proceeds to Step 11 (Step 9: Yes), and if the wrist detection is not successful, the process proceeds to Step 10 (Step 9: No).

(Complementary processing, Step 10)
When the wrist region is not detected (eg, when the wrist is cut by a wrist watch, etc.), the control unit 20 performs complementary processing using the detection results up to the previous frame. For example, there is a method that reuses the detection result of one frame before as it is, and a method that utilizes a tracking technique such as the KCF method.

(Hand detection, Step 11)
The control unit 20 uses elbow positions 320 and 321 as starting points of the front part and wrist positions 330 and 331 as starting points, and extracts hand candidates using the common logic for part detection. The center of gravity of the extracted hand candidate is set as the hand position. As shown in FIG. 10, a left hand 341 is the starting point of the left wrist, and a right hand 340 is the starting point of the right wrist.

(Step 12)
The control unit 20 determines whether or not the hand has been successfully detected. The control unit 20 can determine whether or not the hand has been successfully detected by (1) threshold processing and (2) continuity check described above. If the hand detection is successful, the process proceeds to Step 14 (Step 12: Yes), and if the hand detection is not successful, the process proceeds to Step 13 (Step 12: No).

(Complementary processing, Step 13)
When the part of the hand is not detected (when the hand is hidden, etc.), the control unit 20 performs complementation processing using the detection results up to the previous frame. For example, there is a method that reuses the detection result of one frame before as it is, and a method that utilizes a tracking technique such as the KCF method.

(Step 14)
The control unit 20 can output the positions of left and right shoulders, elbows, wrists, and hands as part detection results, as shown in FIG. Each position is output in the vehicle coordinate system (w, l, h), but it can also be output as a value in another coordinate system such as the camera coordinate system (x, y, z) by coordinate conversion. is.

The above process execution can be repeatedly executed as a series of operations shown above.

(Embodiment as a program)
an acquisition step of acquiring, in a computer, a distance image in which an imaging area including an occupant of a vehicle is taken as an imaging target, and in which distance information to the imaging target is assigned to pixels; According to the correspondence relationship between the coordinate system in the vehicle interior space and the coordinate system in the range image of 8, the detected image is converted into a detected image having a specific coordinate system, and at least one characteristic point of the occupant 5 is detected in the detected image. A program for causing a computer to execute the control step of executing the feature point detection process is also one of the embodiments of the present invention.

Step 1 described in the operation of the processing device 1 is an example of an acquisition step for acquiring a distance image, and Steps 2 to 14 are an example of control steps for executing specific processing. , can be an embodiment of a program to be executed by a computer.

A computer-readable recording medium recording the above program is also one embodiment of the present invention.

(Effect of Embodiment)
According to the embodiment of the present invention, the following effects are obtained.
(1) The processing device according to the embodiment of the present invention acquires a distance image in which an imaging area including the occupant 5 of the vehicle 8 is an imaging target, and in which distance information to the imaging target is assigned to pixels. A TOF camera 10 as an acquisition unit that converts to a detection image having a specific coordinate system according to the correspondence relationship between the coordinate system in the vehicle interior space of the vehicle 8 and the coordinate system in the range image, and in the detection image, and a control unit 20 that executes a feature point detection process for detecting at least one feature point of the occupant 5 . This makes it possible to improve the accuracy of specifying the feature points in the technique of specifying the feature points of the human body.
(2) By utilizing three-dimensional information, it is possible to accurately detect a human skeleton without being affected by changes in gradation due to ambient light or the like.
(3) Since the head position, which is difficult to hide behind other objects, is used as the starting point of detection, detection can be performed regardless of whether the body is hidden.
(4) Processing speed can be increased by narrowing down the region search range using the skeleton model.
(5) By creating voxels in the preprocessing, it is possible to easily obtain density information of the three-dimensional pixel group, and to speed up the processing.
(6) By confirming the continuity with the front part using the three-dimensional information, the position of the part can be detected with high accuracy.
As described in (7) Continuity check, by setting the number of allowable gaps, it is possible to cope with the interruption of the three-dimensional information due to the influence of a wristwatch or the like.
(8) Complementary processing makes it possible to output the detection result even when it is not detected due to the hiding of each part, etc., improving robustness.

Although several embodiments of the present invention have been described above, these embodiments are merely examples and do not limit the invention according to the claims. In addition, these novel embodiments can be implemented in various other forms, and various omissions, replacements, changes, etc. can be made without departing from the scope of the present invention.

For example, the camera installation positions and angles are not limited to the examples shown in the above description. Also, the camera is not limited to a TOF camera. Other distance sensors such as a stereo camera may also be used. Also, the parameters shown in the definition of the skeleton model are not limited to that. Also, the complementary processing is not limited to the example shown here. Moreover, the parts to be detected are not limited to shoulders, elbows, wrists, and hands. For example, shoulders, elbows, and hands may be used. Further, the calculation of the neck position is not limited to the method described above. For example, a calculation method that considers the inclination of the head may be used.

Moreover, not all the combinations of features described in these embodiments are essential to the means for solving the problems of the invention. Furthermore, these embodiments are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

Reference Signs List 1 processing device 5 occupant 8 vehicle 10 TOF camera 20 control unit 203 hipbone position 250 generatrix 300 head position 305 neck position 310, 311 shoulder position 320, 321... Elbow position 330, 331... Wrist position 340, 341... Hand position P... Starting point

Claims (4)

an acquisition unit that obtains a distance image in which an imaging area including an occupant of a vehicle is taken as an imaging target, and distance information to the imaging target is assigned to each pixel corresponding to coordinates of a given two-dimensional coordinate system;
converting the coordinates of the pixels in the two-dimensional coordinate system of the distance image acquired by the acquisition unit and the distance information of the coordinates into points in a three-dimensional space to create a three-dimensional pixel group; The first coordinate system of the pixel group is transformed into a second coordinate system having the occupant as the origin, and the three-dimensional pixel group in the transformed second coordinate system, the size of each skeleton of the human body, and the joints. a control unit that executes feature point detection processing for detecting feature points of shoulders, elbows, wrists, and hands with the occupant's head defined in advance based on a skeleton model defining a movable range as a starting point;
has
The feature point detection processing of the control unit sequentially collates predetermined feature point search models starting from the head of the occupant to identify feature points of the shoulder, elbow, wrist, and hand. When detecting
Considering the set of voxels including voxels located around the feature point before determination, a threshold for determining the likelihood of the feature point before determination based on whether the sum of voxel values in the set of voxels is less than the threshold. perform the processing,
The set of voxels for determining whether the sum of the voxel values is less than a threshold is set at a predetermined interval from the feature point of the occupant's head to the target feature point, which is a feature point different from the feature point. Let the voxel group that exists in
processing equipment. 2. The method according to claim 1, wherein the control unit determines that the target feature point is probable if the sum of the voxel values in each of all voxel groups from the starting point to the target feature point is equal to or greater than a threshold. processing equipment. an acquisition step of obtaining a distance image in which an imaging area including an occupant of a vehicle is taken as an imaging target and distance information to the imaging target is assigned to each pixel corresponding to coordinates of a given two-dimensional coordinate system;
converting the coordinates of the pixels in the two-dimensional coordinate system of the distance image acquired in the acquisition step and the distance information of the coordinates into points in a three-dimensional space to create a three-dimensional pixel group; The first coordinate system of the pixel group is transformed into a second coordinate system having the occupant as the origin, and the three-dimensional pixel group in the transformed second coordinate system, the size of each skeleton of the human body, and the joints. A control step of executing a feature point detection process for detecting feature points of shoulders, elbows, wrists, and hands with the occupant's head defined in advance based on a skeleton model defining a movable range as a starting point;
has
The feature point detection processing of the control step sequentially collates predetermined feature point search models with the occupant's head as a starting point to identify feature points of the shoulder, elbow, wrist, and hand. When detecting
Considering the set of voxels including voxels located around the feature point before determination, a threshold for determining the likelihood of the feature point before determination based on whether the sum of voxel values in the set of voxels is less than the threshold. perform the processing,
The set of voxels for determining whether the sum of the voxel values is less than a threshold is set from the occupant's head as a starting point to a target feature point that is a feature point different from the occupant's head feature point. A program for causing a computer to create a group of voxels existing at predetermined intervals. 4. The method of claim 3 , wherein the control step makes the target feature point probable if the sum of the voxel values in each of all voxel groups from the origin to the target feature point is equal to or greater than a threshold. program.

Images