JP6988406B2 - Hand position detection method, hand position detection device, and hand position detection program - Google Patents

Description

The present invention relates to a hand position detection method, a hand position detection device, and a hand position detection program.

In recent years, at work sites such as factories, it has been studied to detect the movement of the upper limbs of a worker in a predetermined space by using a distance measuring device such as a depth sensor and manage the working condition of the worker. In this type of technique, for example, a computer calculates the three-dimensional coordinates of a worker's palm and joint in the measurement space based on the distance information in the measurement space obtained by the distance measurement device. At this time, the computer detects the upper limb region corresponding to the upper limb of the worker based on the distance information, detects the position of the palm, the position of the joints such as the wrist, elbow, and shoulder from the upper limb region, and then each. Calculate the three-dimensional coordinates of.

However, when measuring the distance with a distance measuring device, a part of the worker's upper limbs is hidden by the worker's own head or body, or a shelf installed in the work area (that is, a state where occlusion has occurred). ) May be measured. The upper limb region detected based on the distance information measured in a state where a part of the upper limb is hidden lacks a region corresponding to a part of the upper limb (for example, the palm). When the upper limb region in which the region corresponding to a part of the upper limb is missing is detected, the position of the palm or the position of each joint may be erroneously detected, and the erroneous three-dimensional coordinates may be calculated.

As one of the techniques to prevent erroneous detection of the position of the palm and the position of each joint by occlusion, the distance to the first joint position in the distance measurement result and the distance to the second joint position adjacent to the first joint position. There is a method of learning the relationship with (see, for example, Patent Document 1). With this type of technique, even if the distance to the 2nd joint position is not included in the measurement result due to occlusion, the 2nd joint is based on the measured distance to the 1st joint position and the learned relationship. It is possible to estimate the distance to the position.

Japanese Unexamined Patent Publication No. 2016-212688

However, when learning the relationship of the distance to each joint position in the distance measurement result, a large amount of measurement results (learning data) are collected in advance while the distance to each joint of the palm or upper limb is measured. There is a need. In addition, in the method of estimating the distance to the joint position based on the relationship of the distance to the joint position, if the distance measurement result does not include the information on the distance to the palm and the distance to each joint, the palm or It becomes difficult to detect the position of each joint.

In one aspect, it is an object of the present invention to correctly detect the position of the palm and each joint from the distance information measured in a state where each joint of the palm and the upper limb is hidden.

In one aspect of the hand position detection method, the computer performs the following five processes. The first process is a process of detecting an upper limb region corresponding to a target person's upper limb portion from an image including information on the distance to an object existing within a predetermined measurement range. The second process detects the positions of the center of the hand and the joints of the upper limbs in the image based on the shape and position of the upper limb region in the image, and within the respective measurement ranges of the detected center of the hand and the joints. This is a process for calculating three-dimensional coordinates. The third process determines whether or not each of the calculated three-dimensional coordinates of the hand center and the three-dimensional coordinates of the joint is consistent with the predetermined positional relationship between the hand center and each joint in the upper limb of the subject. It is a judgment process. The fourth process changes the position of the hand center in the image when the calculated three-dimensional coordinates do not match the predetermined hand center and the positional relationship of each joint, and the hand center three-dimensional. It is a process to correct the coordinates and the three-dimensional coordinates of the joint. The fifth process measures the current 3D coordinates of the hand center and the 3D coordinates of the joint when the calculated 3D coordinates match the predetermined hand center and the positional relationship of each joint. It is a process to determine the three-dimensional coordinates within.

According to the above aspect, it is possible to correctly detect the position of the palm and each joint from the distance information measured in the state where each joint of the palm and the upper limb is hidden.

It is a figure which shows the functional structure of the hand position detection apparatus which concerns on 1st Embodiment. It is a figure which shows the content of the dimensional information. It is a flowchart explaining the content of the coordinate calculation process which concerns on 1st Embodiment. It is a flowchart explaining the content of the coordinate correction processing in 1st Embodiment. It is a flowchart explaining the content of the consistency determination processing in 1st Embodiment. It is a flowchart explaining the content of the coordinate recalculation processing in 1st Embodiment. It is a figure explaining the application example of the hand position detection device. It is a figure which shows the example of the depth image which lacked the information which shows the distance of a hand area. It is a figure explaining the detection method of the upper limb region by the background subtraction method. It is a figure explaining an example of the detection method of the center of a hand and the position of each joint in a depth image. It is a figure explaining the method of discriminating the spatial consistency. It is a figure explaining the method of changing the position of a hand center. It is a graph which explains the estimation method of the depth value used for the calculation of three-dimensional coordinates. It is a flowchart (the 1) explaining the modification of the coordinate correction processing. It is a flowchart (the 2) explaining the modification of the coordinate correction processing. It is a flowchart explaining the content of the adjacent area discrimination process. It is a figure explaining the relationship between the adjacent area and the discrimination result. It is a flowchart (the 1) which explains the modification of the consistency determination process. It is a flowchart (the 2) explaining the modification of the consistency determination process. It is a figure which shows the functional structure of the hand position detection apparatus which concerns on 2nd Embodiment. It is a flowchart explaining the content of the coordinate calculation process which concerns on 2nd Embodiment. It is a flowchart explaining the content of the process of detecting a body region from a depth image. It is a flowchart explaining the content of the process of detecting the upper limb area from a depth image. It is a flowchart explaining the content of the process of calculating the three-dimensional coordinates of a hand center and a joint. It is a flowchart explaining the content of the consistency determination processing in 2nd Embodiment. It is a figure explaining the imaging range of a depth image. It is a figure which shows the example of the depth image. It is a figure (the 1) explaining the detection method of a body region. It is a figure (the 2) explaining the detection method of a body region. It is a figure which shows the example of the detection result of the upper limb region. It is a figure which shows the example of the detection result of the position of a hand center. It is a figure explaining the method of determining the position of the joint of the left upper limb. It is a figure which shows the position of the determined hand center and joint. It is a figure explaining the correction method of the position of a hand center and a joint. It is a figure which shows the hardware configuration of a computer.

[First Embodiment]
FIG. 1 is a diagram showing a functional configuration of the hand position detection device according to the first embodiment.

As shown in FIG. 1, the hand position detection device 1 of the present embodiment includes a depth image acquisition unit 110, a joint position calculation unit 120, a correction unit 130, an output unit 140, and a storage unit 190.

The depth image acquisition unit 110 acquires a depth image from the depth sensor 2 including information indicating the distance (depth) to the surface of the object existing in the detection region for detecting the position of the center of the hand. Here, the position of the center of the hand is the three-dimensional position of the central portion of the hand (the portion beyond the wrist) in the detection area. The depth image acquisition unit 110 acquires, for example, a depth image including depth values corresponding to each of U × V points in a plane whose normal direction is the depth direction from the depth sensor 2. The depth image is an example of information showing the distance to the surface of an object existing in the detection area for detecting the position of the center of the hand. That is, the depth image acquisition unit 110 is an example of an acquisition unit that acquires information indicating the distance to the surface of an object existing in the detection region for detecting the position of the center of the hand. In the following description, the detection area that detects the position of the center of the hand is also referred to as the imaging range of the depth image.

The joint position calculation unit 120 calculates three-dimensional coordinates indicating the positions of the joints of the center of the hand and the upper limbs existing in the detection area in the detection area based on the depth image. The joint position calculation unit 120 in the hand position detection device 1 of the present embodiment includes an upper limb region detection unit 121, an in-image position detection unit 122, and a three-dimensional coordinate calculation unit 123.

The upper limb region detection unit 121 determines a region (upper limb region) corresponding to the human upper limb portion in the depth image based on the depth image for detecting the position of the center of the hand and the background image 191 stored in the storage unit 190. To detect. The background image 191 is a depth image captured by the depth sensor 2 within the detection range in a state where the upper limbs of a person are not present. The upper limb region detection unit 121 may directly acquire the depth image to be detected from the depth image acquisition unit 110, or may read and acquire the depth image 194 stored in the storage unit 190.

The in-image position detection unit 122 detects the position of the center of the hand and the position of the joint of the upper limb in the depth image based on the position and shape of the upper limb region in the depth image. That is, the position detection unit 122 in the image detects the position of the center of the hand and the position of the joint in the depth image based on the detection result of the upper limb region in the upper limb region detection unit 121. The position detection unit 122 in the image detects, for example, the positions of the wrists, elbows, and shoulders as the positions of the joints of the upper limbs. The in-image position detection unit 122, for example, obtains a line segment group indicating the extension direction of the upper limb based on the upper limb region in the depth image, and then detects the position of the center of the hand and the position of each joint based on the line segment group. .. At this time, the position detection unit 122 in the image detects the position of the center of the hand, and then determines the position of each joint based on the position of the center of the hand and the size of the upper limb of the person within the imaging range of the depth image. To detect.

The three-dimensional coordinate calculation unit 123 has three-dimensional coordinates indicating the position of the hand center in the detection area and three dimensions indicating the position of each joint based on the positions of the hand center and each joint in the depth image and the depth value. Calculate the coordinates. The three-dimensional coordinate calculation unit 123 calculates the three-dimensional coordinates indicating the position of the center of the hand and the three-dimensional coordinates indicating the position of each joint according to a known calculation method. For example, the three-dimensional coordinate calculation unit 123 applies a pinhole camera model to calculate three-dimensional coordinates indicating the position of the center of the hand and three-dimensional coordinates indicating the position of each joint.

The correction unit 130 corrects the three-dimensional coordinates of the hand center and each joint calculated from the depth image. The correction unit 130 determines whether or not to correct the three-dimensional coordinates based on the three-dimensional coordinates of the hand center and each joint calculated from the depth image, and the dimensional information 192 and the coordinate history 193 stored in the storage unit 190. do. Then, when it is determined that correction is necessary, the correction unit 130 corrects the three-dimensional coordinates of the hand center and each joint calculated from the depth image. The dimensional information 192 includes a dimension from the center of the hand to the wrist and a dimension from the wrist to the elbow in the upper limb of a person (subject) who moves the upper limb within the imaging range of the depth image. The coordinate history 193 includes the three-dimensional coordinates of the hand center and each joint determined by the processing for the past depth image in the hand position detection device 1. For example, the coordinate history 193 is determined by processing the depth image captured at a time between the time when the depth image currently being processed is captured and the time before a predetermined period (for example, about several seconds). The three-dimensional coordinates are stored. The correction unit 130 in the hand position detection device 1 of the present embodiment includes a consistency determination unit 131 and a recalculation unit 132.

The consistency determination unit 131 determines whether or not the three-dimensional coordinates need to be corrected based on the spatial consistency and the time consistency of the three-dimensional coordinates of the hand center and the joint calculated from the depth image. Spatial consistency is the consistency between the dimensions of the subject's upper limbs based on the three-dimensional coordinates of the center of the hand and the joint calculated from the depth image, and the dimensions of the subject's upper limbs in the dimensional information 192. The time consistency is the consistency with respect to the time change of the three-dimensional coordinates of the hand center and the joint calculated from the depth image, which is calculated based on the coordinate history 193. When the value indicating consistency of either spatial consistency or time consistency is smaller than the threshold value, the consistency determination unit 131 determines that the three-dimensional coordinates need to be corrected. When it is determined that the three-dimensional coordinates need to be corrected, the consistency determination unit 131 causes the recalculation unit 132 to recalculate (correct) the three-dimensional coordinates of the hand center and the joint. On the other hand, when it is determined that it is not necessary to correct the three-dimensional coordinates, the consistency determination unit 131 (correction unit 130) stores the three-dimensional coordinates of the hand center and the joint used for the determination in the coordinate history 193.

The recalculation unit 132 changes the position of the hand center in the depth image, and calculates the three-dimensional coordinates of the hand center and the three-dimensional coordinates of each joint within the imaging range based on the changed position of the hand center (recalculation). calculate). For example, the recalculation unit 132 first moves the position of the center of the hand by a predetermined distance (number of pixels) in the direction of the vector having the elbow or wrist as the starting point and the center of the hand as the ending point in the depth image. In addition, the recalculation unit 132 calculates the positions of joints such as wrists and elbows in the depth image based on the position of the center of the hand after movement. After that, the recalculation unit 132 calculates the three-dimensional coordinates indicating the position of the hand center and the three-dimensional coordinates indicating the position of each joint within the imaging range based on the calculated position of the hand center and the position of each joint. When the changed position of the hand center in the depth image is in the upper limb region, the recalculation unit 132 calculates the three-dimensional position of the hand center based on the depth value of the position of the hand center. On the other hand, when the changed position of the hand center in the depth image is outside the upper limb region, the recalculation unit 132 calculates the three-dimensional position of the hand center by using, for example, the depth value in the upper limb region.

That is, when the correction unit 130 in the hand position detection device 1 of the present embodiment determines that the value indicating consistency is smaller than the threshold value in the consistency determination unit 131, the recalculation unit 132 determines that the center of the hand and each joint The three-dimensional coordinates are calculated again and corrected. When the recalculation unit 132 corrects the three-dimensional coordinates of the center of the hand and each joint, the correction unit 130 uses the corrected three-dimensional coordinates, the dimensional information 192, and the coordinate history 193 in the consistency determination unit 131. , Judge the necessity of re-correction of the three-dimensional coordinates of the hand center. When it is determined that it is not necessary to correct the three-dimensional coordinates, the consistency determination unit 131 (correction unit 130) stores the three-dimensional coordinates of the hand center and each joint used for the determination in the coordinate history 193. The consistency determination unit 131 (correction unit 130) stores, for example, the three-dimensional coordinates of the center of the hand and each joint in the coordinate history 193 in association with the information indicating the imaging time of the depth image to be processed. ..

The output unit 140 outputs information including the three-dimensional coordinates of the center of the hand and each joint included in the coordinate history 193 to the external device 3. The external device 3 is, for example, a server device that manages whether or not the work of the worker is appropriate based on the time change of the three-dimensional coordinates of the center of the hand.

FIG. 2 is a diagram showing the contents of dimensional information.
The dimensional information 192 includes, for example, as shown in FIG. 2, three dimensions L1, L2, and L3 in the upper limb of the subject, which are measured from the subject who moves the upper limb within the imaging range of the depth image by the depth sensor 2. The first dimension L1 is the length from the center of the hand to the wrist. The second dimension L2 is the length from the wrist to the elbow. The third dimension L3 is the length from the elbow to the shoulder. The first dimension L1, the second dimension L2, and the third dimension L3 may be the dimensions measured by either the upper right limb or the upper left limb of the subject, respectively, or may be measured by the upper right limb. It may be the average value of the dimension and the dimension measured by the upper left limb. Further, when there are a plurality of target persons, the first dimension L1, the second dimension L2, and the third dimension L3 of each target person are registered in the dimension information 192 in a mode that can be identified for each target person. I'll keep it. For example, when the worker ID of the subject is "A", the consistency determination unit 131 of the hand position detection device 1 sets the dimensions L1, L2, and L3 of the upper limbs of the subject as the dimensional information of FIG. 2, respectively. The dimensions LA1, LA2, and LA3 are read from 192.

The hand position detection device 1 of the present embodiment sequentially acquires depth images captured by the depth sensor 2 at predetermined time intervals, and detects the center of the hand and the joints of the upper limbs from each of the acquired plurality of depth images. Calculate the three-dimensional coordinates of the center and joint. The hand position detection device 1 of the present embodiment performs coordinate calculation processing according to the flowchart of FIG. 3 as a process of detecting the center of the hand and the joints of the upper limbs from the depth image and calculating the three-dimensional coordinates.

FIG. 3 is a flowchart illustrating the contents of the coordinate calculation process according to the first embodiment.
As shown in FIG. 3, the hand position detection device 1 of the present embodiment first selects a depth image to be processed (step S1), and detects an upper limb region from the selected depth image (step S2). The processing of steps S1 and S2 is performed by the upper limb region detection unit 121 of the joint position calculation unit 120 in the hand position detection device 1. The upper limb region detection unit 121 selects, for example, the depth image having the earliest imaging time among the depth images that have not been processed to detect the upper limb region. After selecting the depth image, the upper limb region detection unit 121 determines the upper limb region in the depth image based on the depth value of each point (each pixel) in the selected depth image and the depth value of each point in the background image 191. To detect.

Next, the hand position detection device 1 determines whether or not the upper limb region has been detected from the depth image (step S3). The determination in step S3 is performed by the upper limb region detection unit 121. When the upper limb region is not detected (step S3; NO), the upper limb region detection unit 121 stores, for example, information indicating that the upper limb region was not detected from the selected depth image in the coordinate history 193 (step). S11), the coordinate calculation process for the depth image is terminated. On the other hand, when the upper limb region is detected from the depth image (step S3; YES), the hand position detection device 1 performs the processes of steps S4 to S7.

When the upper limb region is detected from the depth image (step S3; YES), the hand position detection device 1 next detects the positions of the hand center and the joint in the upper limb region in the depth image (step S4). The process of step S4 is performed by the position detection unit 122 in the image of the joint position calculation unit 120 in the hand position detection device 1. The position detection unit 122 in the image extracts, for example, a line segment group indicating the extension direction of the upper limb in the depth image based on the shape and position of the upper limb region, and then the position of the center of the hand on the line segment of the line segment group and the position of the center of the hand. Detects the position of upper limb joints (wrist, elbow, and shoulder). The position detection unit 122 in the image reduces the dimension in the direction orthogonal to the extension direction of the upper limb in the upper limb region, based on the direction that can be the extension direction of the upper limb (the direction of the line segment connecting the joints) in the depth image, for example. , Extract a line segment group indicating the extension direction of the upper limb. After extracting the line segment group, the position detection unit 122 in the image first detects the position of the center of the hand from each point on the line segment included in the line segment group. The in-image position detection unit 122 detects the position of the center of the hand on the line segment, for example, based on the information indicating the area corresponding to the size of the hand (palm) in the depth image. After detecting the position of the center of the hand, the position detection unit 122 in the image detects, for example, the position of the detected hand center and the information indicating the distance (number of pixels) from the center of the hand to the wrist in the depth image. Detect the position. After that, the position detection unit 122 in the image sequentially detects the positions of the adjacent joints based on the detected joint positions and the information indicating the distance (number of pixels) between the joints in the depth image. When detecting the position of a joint, the position detection unit 122 in the image sets the position where the line segment bends in the line segment group as a candidate for the joint, and includes the angle formed by the two line segments connected at the bent position. Detects the position of.

Next, the hand position detection device 1 calculates the three-dimensional coordinates of the detected hand center and joint (step S5). The process of step S5 is performed by the three-dimensional coordinate calculation unit 123 of the joint position calculation unit 120 in the hand position detection device 1. The three-dimensional coordinate calculation unit 123 applies a pinhole camera model, and is based on the imaging range of the depth image, the number of pixels of the depth image, the positions of the hand center and the joint in the depth image, and the depth value. And calculate the three-dimensional coordinates of the joint. The three-dimensional coordinates calculated in step S5 are the coordinates in the world coordinate system corresponding to the measured value of the distance between arbitrary two points in the three-dimensional space for detecting the position of the center of the hand.

Next, the hand position detection device 1 performs a coordinate correction process (step S6) for correcting the three-dimensional coordinates of the center of the hand and the joint. The coordinate correction process includes a process of determining the necessity of correcting the three-dimensional coordinates and a process of correcting the position of the hand center and calculating (recalculating) the three-dimensional coordinates of the hand center and the joint. The coordinate correction process is performed by the correction unit 130 in the hand position detection device 1. As described above, the correction unit 130 determines whether or not the three-dimensional coordinates need to be corrected based on the spatial consistency and the time consistency. This determination is performed by the consistency determination unit 131 of the correction unit 130. The consistency determination unit 131 calculates a value indicating spatial consistency based on the three-dimensional coordinates of the center of the hand and the joint calculated from the depth image and the dimensional information 192. Further, the consistency determination unit 131 calculates a value indicating time consistency based on the three-dimensional coordinates of the center of the hand and the joint calculated from the depth image and the coordinate history 193. Then, when the consistency determination unit 131 determines that the three-dimensional coordinates do not match spatially or temporally based on the calculated value indicating spatial consistency and the value indicating time consistency, the recalculation unit 132 determines. Recalculate (correct) the 3D coordinates.

The consistency determination unit 131 is a value indicating spatial consistency for each of the center of the hand and the joint, for example, the difference between the distance value calculated based on the three-dimensional coordinates and the depth image and the depth value in the depth image. Is calculated. Further, the consistency determination unit 131 includes, for example, the dimensions of the upper limbs calculated based on three-dimensional coordinates and the upper limbs of the subject in the dimensional information 192 as values indicating spatial consistency for each of the center of the hand and the joint. Calculate the difference from the dimensions. At this time, the consistency determination unit 131 determines that the three-dimensional coordinates are spatially matched when the calculated difference is equal to or less than the threshold value. Further, the consistency determination unit 131 calculates, for example, the movement speed as a value indicating the time consistency for each of the center of the hand and the joint. At this time, the consistency determination unit 131 determines that the three-dimensional coordinates are time-matched when the calculated movement speed is equal to or less than the threshold value.

The recalculation unit 132 changes the position of the hand center in the depth image to calculate the three-dimensional coordinates of the hand center, and calculates the three-dimensional coordinates of the joint based on the recalculated three-dimensional coordinates of the hand center. .. When the recalculation unit 132 corrects the three-dimensional coordinates, the consistency determination unit 131 of the correction unit 130 determines whether or not the correction of the three-dimensional coordinates is necessary based on the corrected three-dimensional coordinates. After that, the correction unit 130 repeats the correction of the three-dimensional coordinates and the determination of the necessity of the correction until the consistency determination unit 131 determines that the correction does not need to be performed again. In the coordinate correction process, an upper limit may be set for the number of corrections of the three-dimensional coordinates. When the 3D coordinates are corrected by the set upper limit number of times, the correction unit 130 uses the current 3D coordinates as the center of the hand and the joint at the imaging time of the depth image to be processed, regardless of whether or not they are consistent. Determined to the three-dimensional coordinates of.

When the coordinate correction process is completed, the hand position detection device 1 stores information including the three-dimensional coordinates of the hand center and the joint determined by the coordinate correction process in the coordinate history 193 (step S7), and is currently selected as the processing target. The calculation process for the existing depth image is terminated. The processing of step S7 is performed by the correction unit 130. The correction unit 130 stores the determined three-dimensional coordinates of the hand center and the joint in the coordinate history 193 in association with the information indicating the imaging time of the depth image currently being processed.

The hand position detection device 1 repeats the processing according to the flowchart of FIG. 3 until the processing after step S2 is performed on all of the plurality of depth images to be processed. The hand position detection device 1 may pipeline the processes of steps S1 to S7 and S11 for a plurality of depth images.

As described above, the hand position detection device 1 of the present embodiment calculates the three-dimensional coordinates of the hand center and the joint based on the positions of the hand center and the joint detected from the depth image, and then performs the coordinate correction process by the correction unit 130 ( Step S6) is performed to determine the three-dimensional coordinates to be stored in the coordinate history 193. As the coordinate correction process, as described above, the correction unit 130 performs a process of correcting the three-dimensional coordinates of the center of the hand and the joint, which are not spatially or temporally matched, so as to be matched. .. The correction unit 130 performs, for example, a process according to the flowchart of FIG. 4 as the coordinate correction process.

FIG. 4 is a flowchart illustrating the contents of the coordinate correction process in the first embodiment.

As shown in FIG. 4, the correction unit 130 first sets the variable r indicating the number of corrections of the three-dimensional coordinates to “0” (step S601). The process of step S601 is performed by, for example, the consistency determination unit 131.

Next, the correction unit 130 selects one of the upper limb regions detected from the depth image, and reads out the three-dimensional coordinates of the hand center and each joint detected from the upper limb region (step S602). The process of step S602 is performed by the consistency determination unit 131 of the correction unit 130.

Next, the correction unit 130 performs a consistency determination process (step S603) for the three-dimensional coordinates read in step S602, that is, the three-dimensional coordinates of the hand center and each joint currently being processed. The consistency determination process is performed by the consistency determination unit 131. The consistency determination unit 131 determines whether or not the three-dimensional coordinates of the center of the hand and each joint are spatially aligned and temporally aligned.

The consistency determination unit 131 calculates, for example, the position and distance in the depth image from the three-dimensional coordinates for each of the center of the hand and each joint, and calculates the difference between the calculated distance and the depth value in the depth image. .. Then, when the absolute value of the calculated difference is equal to or greater than the threshold value, the consistency determination unit 131 determines that the three-dimensional coordinates are not spatially matched. Further, the consistency determination unit 131 calculates, for example, the three-dimensional distance between the center of the hand and each joint with the adjacent joint based on the three-dimensional coordinates, and the calculated three-dimensional distance and the target person in the dimensional information 192. Calculate the difference from the dimensions of the upper limbs. Then, when the absolute value of the calculated difference is equal to or greater than the threshold value, the consistency determination unit determines that the three-dimensional coordinates are not spatially matched.

Further, the consistency determination unit 131 calculates the movement speed based on the three-dimensional coordinates, for example, for the center of the hand and each of the joints. Then, when the calculated movement speed is equal to or higher than the threshold value, the consistency determination unit 131 determines that the three-dimensional coordinates do not match in time.

After completing the consistency determination process, the correction unit 130 then determines whether or not there is a hand center or joint that is temporally or spatially inconsistent in the hand center and each joint currently being processed. Determination (step S604). The determination in step S604 is performed by the consistency determination unit 131. When the center of the hand and all of the joints are temporally and spatially aligned (step S604; NO), the consistency determination unit 131 (correction unit 130) determines the current three-dimensional coordinates of the center of the hand and each joint. , The three-dimensional coordinates to be stored in the history information 193 are determined (step S608).

On the other hand, when either the center of the hand or each joint is not consistent in time or space (step S604; YES), the consistency determination unit 131 then determines whether the variable r is smaller than the threshold value N. (Step S605). The threshold value N is an upper limit of the number of times the three-dimensional coordinates are corrected. When r ≧ N (step S605; NO), the consistency determination unit 131 (correction unit 130) determines the current three-dimensional coordinates of the hand center and each joint as the three-dimensional coordinates stored in the history information 193 (step S605; NO). Step S608). On the other hand, in the case of r <N (step S605; YES), the consistency determination unit 131 (correction unit 130) performs a coordinate recalculation process (step S606) for recalculating the three-dimensional coordinates of the hand center and each joint. Let the recalculation unit 132 perform the calculation. In the coordinate recalculation process, the recalculation unit 132 changes the position of the hand center in the depth image, and then calculates the three-dimensional coordinates of the hand center and each joint based on the changed position of the hand center (recalculation). do). After calculating the three-dimensional coordinates of the center of the hand and each joint, the recalculation unit 132 notifies the consistency determination unit 131 of the calculated three-dimensional coordinates, and ends the coordinate recalculation process.

When the recalculation unit 132 finishes the coordinate recalculation process (step S606), the correction unit 130 updates the variable r to r + 1 (step S607) and performs the consistency determination process (step S603). The process of step S607 is performed by the consistency determination unit 131. When the consistency determination unit 131 receives the three-dimensional coordinates from the recalculation unit 132, the consistency determination unit 131 updates the variable r and performs the consistency determination process S603. After that, the correction unit 130 will perform steps S603 to S607 until the center of the hand and all of the joints are temporally and spatially aligned (step S604; NO) or r ≧ N (step S605; NO). Repeat the process. Then, when the center of the hand and all of the joints are temporally and spatially aligned, or when r ≧ N, the consistency determination unit 131 (correction unit 130) stores the current three-dimensional coordinates in the coordinate history 193. Determine the three-dimensional coordinates to be used (step S608).

After finishing the processing of step S608, the correction unit 130 next determines whether or not there is an unselected upper limb region in the upper limb region detected from the depth image to be processed (step S609). The determination in step S609 is performed by the consistency determination unit 131. When there is an unselected upper limb region (step S609; YES), the consistency determination unit 131 (correction unit 130) performs the processing after step S601 on the unselected upper limb region. Then, when the processing after step S601 is performed on all the upper limb regions (step S609; NO), the consistency determination unit 131 (correction unit 130) ends the coordinate correction processing on the depth image to be processed.

As described above, the coordinate correction process performed by the correction unit 130 includes the consistency determination process (step S603) and the coordinate recalculation process (step S603). Then, when the correction unit 130 determines that the coordinates are spatially and temporally consistent in the consistency determination process, or when the number of times the coordinate recalculation process is performed becomes a predetermined number of times (N times) or more. , The three-dimensional coordinates at that time are determined to be the three-dimensional coordinates stored in the coordinate history 193.

FIG. 5 is a flowchart illustrating the content of the consistency determination process in the first embodiment.

The consistency determination process (step S603) is performed by the consistency determination unit 131 of the correction unit 130. As shown in FIG. 5, the consistency determination unit 131 in the hand position detection device 1 of the present embodiment first sets the variable i for identifying the hand center and each joint to “0” and sets the consistency parameter Ci to “True”. (Step S60301). Here, the variable i is a numerical value (0, 1, 2, ...) Attached to identify each of the hand center and each joint when the position of the hand center and each joint is detected from one upper limb region.・). The consistency parameter Ci is a value indicating whether or not the three-dimensional coordinates of the point Pi indicating the hand center or joint specified by the variable i are spatially and temporally matched. In the present embodiment, the consistency parameter Ci is set to one of two values, "True" and "False". "True" is a value indicating that the three-dimensional coordinates are spatially and temporally aligned. "False" is a value indicating that the three-dimensional coordinates are not spatially or temporally aligned.

Next, the consistency determination unit 131 calculates the coordinates (xi, yi) and the distance zi in the depth image from the three-dimensional coordinates (Xi, Yi, Zi) of the point Pi indicating the center of the hand or the joint (step). S60302). In step S60302, the consistency determination unit 131 performs back calculation based on the calculation formula when calculating the three-dimensional coordinates (Xi, Yi, Zi) corresponding to the position of the point Pi indicating the center of the hand or the joint in the depth image (Xi, Yi, Zi). The coordinates (xi, yi) and the distance zi in the depth image are calculated by the inverse transformation).

Next, the consistency determination unit 131 reads out the depth value zi'of the coordinates (xi, yi) in the depth image (step S60303), and determines whether or not zi'> zi + M (step S60304). Here, M is a margin with respect to the distance zi. The margin M is consistent with the three-dimensional coordinates (Xi, Yi, Zi) of the point Pi, the coordinates (xi, yi) in the depth image, and the three-dimensional coordinates calculated from the depth value z'. Set based on the range considered to be). In other words, the margin M is, for example, the distance to an object whose depth value zi'is different from that of the subject's upper limb when the depth value zi'is zi'> zi + M, based on, for example, the thickness of the upper limb. Set to a value that is recognized as the indicated value. That is, when zi'> zi + M, the consistency determination unit 131 calculates from the three-dimensional coordinates (Xi, Yi, Zi) of the point Pi, the coordinates (xi, yi) in the depth image, and the depth value zi'. It is determined that the three-dimensional coordinates are not spatially consistent. Therefore, when zi'> zi + M (step S60304; YES), the consistency determination unit 131 changes the consistency parameter Ci to "False" (step S60312).

On the other hand, when zi'≤ zi + M (step S60304; NO), the consistency determination unit 131 next has a point Pi indicating the hand center or joint currently being processed and adjacent to the hand center or joint. The three-dimensional distance len with the point Pi + 1 indicating the joint is calculated (step S60305). That is, in the process of step S60305, the consistency determination unit 131 has the three-dimensional coordinates (Xi, Yi, Zi) of the hand center or the joint specified by the current variable i and the three-dimensional coordinates of the joint specified by the variable i + 1. The distance len with the coordinates (Xi + 1, Yi + 1, Zi + 1) is calculated.

Next, the consistency determination unit 131 refers to the dimension information 192 and calculates the difference ΔL between the calculated three-dimensional distance len and the dimension corresponding to the three-dimensional distance len in the upper limb of the subject (step S60306). For example, the consistency determination unit 131 can recognize whether the point Pi is a point indicating the center of the hand, the wrist, the elbow, or the shoulder by the value of the variable i. The consistency determination unit 131, which recognizes whether the point Pi is the center of the hand, the wrist, the elbow, or the shoulder, determines the dimension corresponding to the three-dimensional distance len in the upper limb of the subject based on the recognition result. Recognize which of the three dimensions L1, L2 and L3 in 192 is. For example, if the point Pi is the center of the hand, the consistency determination unit 131 recognizes that the dimension corresponding to the three-dimensional distance len is the first dimension L1 indicating the distance from the center of the hand to the wrist, and the dimension information 192. The first dimension L1 of the subject is read from. The three-dimensional coordinates of the point Pi and the point Pi + 1 are the coordinates calculated in the process of step S5 in the flowchart of FIG. 3, respectively. That is, the three-dimensional coordinates of the point Pi and the point Pi + 1 are the coordinates in the world coordinate system corresponding to the measured values of the distances between arbitrary two points in the three-dimensional space for detecting the position of the center of the hand, respectively. Therefore, the three-dimensional distance len between the point Pi and the point Pi + 1 has a dimension of length. Therefore, after reading the dimension corresponding to the three-dimensional distance len, the consistency determination unit 131 calculates the difference ΔL between the three-dimensional distance len and the read dimension.

Next, the consistency determination unit 131 determines whether or not the calculated absolute value | ΔL | of the difference ΔL is larger than the threshold value THL (step S60307). For example, if the point Pi and the point Pi + 1 correctly correspond to the positions of the subject's wrist and wrist, respectively, the three-dimensional distance len calculated in step S60305 is the dimension from the subject's wrist to the wrist. It almost matches. Further, when the points Pi and the points Pi + 1 correctly correspond to the positions of the wrist and elbow of the subject, respectively, the three-dimensional distance len calculated in step S60305 substantially matches the dimension from the wrist to the elbow of the subject. do. Therefore, when the difference ΔL between the three-dimensional distance len and the dimension of the target person corresponding to the three-dimensional distance len is within a predetermined range, the positional relationship between the two points Pi and Pi + 1 and the upper limb of the target person. It can be said that the positional relationship based on the dimensions of is spatially consistent. That is, when the absolute value | ΔL | of the difference ΔL between the three-dimensional distance len and the dimension of the target person corresponding to the three-dimensional distance len is larger than the threshold THL, the positional relationship between the two points Pi and Pi + 1 It can be said that the positional relationship based on the dimensions of the subject's upper limbs is not spatially consistent. Therefore, when | ΔL |> THL (step S60307; YES), the consistency determination unit 131 changes the consistency parameter Ci to “False” (step S60312).

On the other hand, when | ΔL | ≦ THL (step S60307; NO), the consistency determination unit 131 next calculates the moving speed Si of the point Pi (step S60308). In the process of step S60308, the consistency determination unit 131 first reads the past three-dimensional coordinates of the point Pi with reference to the coordinate history 193, and the three-dimensional distance between the current three-dimensional coordinates and the past three-dimensional coordinates. Calculate Di. The three-dimensional coordinates read from the coordinate history 193 are, for example, three-dimensional coordinates determined based on the depth image captured at the imaging timing immediately before the imaging timing of the depth image currently being processed. When the three-dimensional coordinates determined from the depth image captured at the previous imaging timing do not exist in the coordinate history 193, the consistency determination unit 131 determines, for example, from the depth image captured at the imaging timing two or more times before. Read out the three-dimensional coordinates. After reading out the three-dimensional coordinates, the consistency determination unit 131 calculates the three-dimensional distance Di of the two three-dimensional coordinates. After calculating the three-dimensional distance Di, the consistency determination unit 131 calculates the moving speed Si based on the calculated three-dimensional distance Di and the difference in the imaging time of the depth image.

Next, the consistency determination unit 131 determines whether or not the calculated movement speed Si is larger than the threshold value THS (step S60309). The threshold THS is set based on, for example, the center of the hand when a person moves the upper limbs and the average moving speed of each joint. That is, when the calculated movement speed Si is Si> THS, the consistency determination unit 131 determines that the positional relationship between the two three-dimensional coordinates used for calculating the movement speed Si does not match in time. do. Therefore, when Si> THS (step S60309; YES), the consistency determination unit 131 changes the consistency parameter Ci to “False” (step S60312).

When the processing of step S60312 is performed, or when Si ≦ THS (step S60309; NO), the consistency determination unit 131 is then undetermined among the points indicating the center of the hand and the joint currently being processed. It is determined whether or not there is a point (step S60310). That is, at the time of determining in step S60310, the consistency parameter Ci for the point Pi indicating the center of the hand or the joint currently being processed is determined to be either “True” or “False”.

When there is a point where the consistency is not determined (step S60310; YES), the consistency determination unit 131 updates the variable i to i + 1 and sets the consistency parameter Ci to “True” (step S60311). After that, the consistency determination unit 131 performs the processes after step S60302. Then, when the consistency of all the hand centers and joints is determined (step S60310; NO), the consistency determination unit 131 ends the consistency determination process.

As described above, in the consistency determination process in the present embodiment, the consistency determination unit 131 determines the consistency parameter Ci for each of the center of the hand and each joint as either “True” or “False”. In this case, in the determination of step S604 performed after the consistency determination process (step S603), the consistency determination unit 131 determines the presence or absence of the center of the hand or the joint where Ci = False. When the consistency parameter Ci for all hand centers or joints is "True", the consistency determination unit 131 uses the current three-dimensional coordinates as the three-dimensional coordinates at the time of capturing the depth image, that is, the coordinate history 193. It is determined to be the three-dimensional coordinates to be stored in (step S608).

Further, even when the center of the hand or the joint where Ci = False exists, when the number of executions r of the coordinate recalculation process is N times or more (step S605; NO), the consistency determination unit 131 Determines the current three-dimensional coordinates as the three-dimensional coordinates stored in the coordinate history 193.

When there is a hand center or joint where Ci = False and the number of times the coordinate recalculation process is executed r is less than N times, the consistency determination unit 131 causes the recalculation unit 132 to perform the coordinate recalculation process. (Step S606) is performed. The recalculation unit 132 performs, for example, a process according to the flowchart of FIG. 6 as the coordinate recalculation process.

FIG. 6 is a flowchart illustrating the content of the coordinate recalculation process in the first embodiment.

As shown in FIG. 6, the recalculation unit 132 first selects one of the upper limb regions including the hand center or joint where Ci = False (step S60601), and changes the position of the hand center in the depth image (step S60601). Step S60602). In the process of step S60602, the recalculation unit 132 is, for example, the position of the center of the hand by a predetermined distance (number of pixels) in the direction of the vector whose starting point is the position of the wrist or elbow and the end point is the position of the center of the hand before the change. To move.

Next, the recalculation unit 132 detects the position of each joint in the depth image based on the position of the center of the hand after the change (step S60603). As the process of step S60603, the recalculation unit 132 performs the same process as that performed by the position detection unit 122 in the image, and detects the positions of the wrist, elbow, and shoulder.

Next, the recalculation unit 132 selects one of the center of the hand and the joint (step S60604), and determines whether or not the position of the selected center of the hand or the joint is within the upper limb region (step S60605). .. When the position of the selected hand center or joint is within the upper limb region (step S60605; YES), the recalculation unit 132 uses the depth value of the position in the depth image to obtain the three-dimensional coordinates of the selected hand center or joint. Is calculated (step S60606). On the other hand, when the position of the selected hand center or joint is outside the upper limb region (step S60605; NO), the recalculation unit 132 calculates the depth value of the selected hand center or joint and uses the calculated depth value. Calculate the three-dimensional coordinates (step S60607). In the process of step S60607, the recalculation unit 132 is based on, for example, the amount of change in the depth value on the line segment passing between the position of the unchanged hand center or joint in the upper limb region and the position of the adjacent joint. , Calculate (estimate) the depth value at the position of the center of the hand or the joint after the change.

After calculating the three-dimensional coordinates by step S60606 or S60607, the recalculation unit 132 determines whether or not there is an unselected hand center or joint among the hand centers and joints detected from the upper limb region to be processed. (Step S60608). When there is an unselected hand center or joint (step S60608; YES), the recalculation unit 132 performs the processing after step S60604. Then, when all the three-dimensional coordinates of the hand center and the joint are calculated (step S60608; NO), the recalculation unit 132 is then unselected in the upper limb region including the hand center or the joint where Ci = False. It is determined whether or not there is an upper limb region (step S60609).

When there is an unselected upper limb region (step S60609; YES), the recalculation unit 132 performs the processing after step S60601 on the unselected upper limb region. Then, when the recalculation of the three-dimensional coordinates for all the target upper limb regions is completed (step S60609; NO), the recalculation unit 132 notifies the calculated three-dimensional coordinates to the consistency determination unit 131 (step S60610). , Ends the coordinate recalculation process.

When the recalculation unit 132 finishes the coordinate recalculation process, the consistency determination unit 131 updates the variable r indicating the number of coordinate recalculation processes to r + 1 (step S607), and the recalculated hand center. And the consistency determination process (step S603) for the three-dimensional coordinates of each joint is performed.

As described above, in the hand position detecting device 1 of the present embodiment, the three-dimensional coordinates of the joints of the hand center and the upper limbs calculated based on the depth image are the positions of the joints of the hand center and the upper limbs based on the dimensions of the upper limbs of the subject. If it does not match the relationship, change the position of the center of the hand to correct the three-dimensional coordinates of the center of the hand and the joint. At this time, the hand position detection device 1 changes the position of the center of the hand in a direction away from the position of the wrist or elbow. Therefore, the hand position detection device 1 of the present embodiment captures the position of the center of the hand detected from the depth image even when the upper limb region detected from the depth image does not include the partial region corresponding to the hand. It is possible to approach the position of the center of the hand of the subject within the imaging range of. Therefore, in the hand position detection device 1 of the present embodiment, the joints of the center of the hand and the upper limbs corresponding to the dimensions of the upper limbs of the subject can be seen from the depth image in which the distance information about the part of the subject's hand is missing due to occlusion or the like. It is possible to calculate the appropriate three-dimensional coordinates of.

FIG. 7 is a diagram illustrating an application example of the hand position detection device.
FIG. 7 shows, as an application example of the hand position detection device 1 of the present embodiment, a management system 5 that manages a work situation based on a time history of a position centered on the hand of a worker 7 who works in the work space 6. Shows. The management system 5 includes a hand position detection device 1, a depth sensor 2, and a management device 3A. The hand position detection device 1 and the depth sensor 2 in the management system 5 each include a communication function that can be connected to a communication network 4 such as the Internet. Further, the management device 3A in the management system 5 is an example of the external device 3 in FIG. The x-direction, y-direction, and z-direction shown in FIG. 7 indicate the x-axis direction, the y-axis direction, and the z-axis direction of the coordinate system of the depth sensor 2 in the work space 6, respectively.

In this type of management system 5, for example, the manager of the management system 5, the worker 7, or the like creates dimensional information 192 in which the dimensions of the upper limbs of the worker 7 are input in advance, and the dimensional information 192 is detected by hand. It is stored in the storage unit 190 of the device 1. Further, in this type of management system 5, for example, a depth image is captured by a depth sensor 2 installed above the work space 6 while the worker 7 is not working, and the depth image is used as a background image 191. It is stored in the storage unit 190. The depth sensor 2 is a depth value (distance information) for U × V points within the imaging range 10 including the work area in the first workbench 8A and the second workbench 8B on which the worker 7 works. Take a depth image with.

The hand position detection device 1 in the management system 5 acquires a depth image from the depth sensor 2 while the worker 7 is working, and calculates the three-dimensional positions of the hand center of the worker 7 and the joints of the upper limbs. Perform processing. For example, the depth sensor 2 captures a depth image including information (depth value) of a distance within the imaging range at predetermined time intervals, and sequentially feeds the captured depth image to the hand position detecting device 1 via the communication network 4. Send.

In the hand position detection device 1, the image acquisition unit 110 sequentially receives the depth images captured by the depth sensor 2 and stores them in the storage unit 190. Further, in the hand position detection device 1, the joint position calculation unit 120 and the correction unit 130 calculate the three-dimensional coordinates of the hand center of the worker 7 and the joints of the upper limbs based on the depth image. The joint position calculation unit 120 and the correction unit 130 calculate, for example, the three-dimensional coordinates of the joints of the center of the hand and the joints of the upper limbs according to the flowchart of FIG. Further, in the hand position detection device 1, the output unit 140 outputs (transmits) the calculated information such as the three-dimensional coordinates of the center of the hand and the joints of the upper limbs to the management device 3A.

In the management device 3A, for example, based on the time history of the three-dimensional coordinates of the joints of the center of the hand and the upper limbs calculated by the hand position detection device 1, and the movements of the upper left limb 710 and the upper right limb 720 in the work performed by the worker 7. , It manages whether or not the worker 7 is performing the work correctly.

The imaging range 10 of the depth sensor 2 in the management system 5 of FIG. 7 corresponds to a work area on the surface of the first workbench 8A and the second workbench 8B. Therefore, in the captured image captured by the depth sensor 2, there is a region including information on the distance corresponding to the portion included in the working region in the first workbench 8A and the second workbench 8B. Further, as shown in FIG. 7, when the shelf 9 is installed in the portion of the second workbench 8B included in the work area, the captured image captured by the depth sensor 2 includes the surface of the shelf 9 ( There is a region containing information on the distance to the surface facing the depth sensor 2 side). Further, the depth image captured by the depth sensor 2 while the worker 7 is working includes the distance to each point in the portion of the upper left limb 710 and the upper right limb 720 that exists within the imaging range 10. There is an upper limb area that contains information indicating. The depth image captured by the depth sensor 2 while the worker 7 is working includes, for example, an upper limb region containing information indicating the distance to each point in the hand 711 and wrist 712 portions of the upper left limb 710. exist. Further, the depth image includes, for example, an upper limb region containing information indicating the distance to each point in the hand 721, the wrist 721, and the elbow portion (not shown) of the upper right limb 720. In the following description, the hand 711, wrist 712, elbow 713, and shoulder 714 of the left upper limb 710 are also referred to as the left hand 711, left wrist 712, left elbow 713, and left shoulder 714, respectively. Further, in the following description, the hand 721, wrist 722, elbow (not shown), and shoulder 724 of the right upper limb 720 are also referred to as right hand 721, right wrist 722, right elbow, and right shoulder 724, respectively.

When the worker 7 works in the work space 6 of FIG. 7, for example, the right hand 721 may be inserted between the surface of the second work table 8B and the shelf 9 for the purpose of putting in and taking out tools. be. When the worker 7 inserts the right hand 721 between the surface of the second work table 8B and the shelf 9, the shelf 9 exists between the depth sensor 2 and the right hand 721 of the worker 7. Therefore, the depth image captured by the depth sensor 2 when the worker 7 inserts the right hand 721 between the surface of the second workbench 8B and the shelf 9 is, for example, as shown in FIG. It is a depth image lacking information indicating the distance of the hand area (right hand 721) of the member 7.

FIG. 8 is a diagram showing an example of a depth image in which information indicating the distance of the hand region is missing.
The depth image 1101 of FIG. 8 includes depth values for each of the U × V points in the imaging region 10. The four points Q (0,0), Q (U-1,0), Q (0, V-1), and Q (U-1, V-1) in FIG. 8 are depth images, respectively. The pixels at the four corners (depth value detection positions) in 1101 are shown.

The depth image 1101 includes a region showing the first workbench 8A, a region showing the second workbench 8B, and a region showing the shelf 9. In the depth image 1101 of FIG. 8, the depth value of each region is indicated by the density of the dot pattern in the region, and the region having a higher density of the dot pattern has a smaller depth value (that is, the distance from the depth sensor 2 is shorter).

Further, the depth image 1101 of FIG. 8 includes a first upper limb region corresponding to the upper left limb 710 of the worker 7 and a second upper limb region corresponding to the upper right limb 720 of the worker 7. The depth image of FIG. 8 is a depth image taken when the worker 7 extends the upper right limb 720 and inserts his right hand between the second work table 8B and the shelf 9. Therefore, the second upper limb region is a region including the wrist 722 in the upper right limb 720 to the position between the elbow 723 and the shoulder (not shown), and the hand region (right hand) beyond the wrist 722. Not included. On the other hand, the first upper limb region includes the hand region (left hand 711) beyond the wrist 712 in the left upper limb 710.

When the hand position detecting device 1 of the present embodiment calculates the three-dimensional coordinates of the center of the hand and each joint from the depth image 1101 of FIG. 8, the hand position detecting device 1 performs processing according to the flowchart of FIG. That is, the hand position detection device 1 first detects the first upper limb region and the second upper limb region from the depth image 1101 of FIG. 8 (step S2). The hand position detection device 1 detects a first upper limb region and a second upper limb region by, for example, a background subtraction method using a background image 191.

FIG. 9 is a diagram illustrating a method of detecting an upper limb region by the background subtraction method.
The background image 191 (a) of FIG. 9 is a depth image captured by the depth sensor 2 when the worker 7 is not performing the work, as described above. The background image 191 includes a region showing the first workbench 8A, a region showing the second workbench, and a region showing the shelf 9. In the background image 191 the depth value of each region is shown by the density of the dot pattern in the region, and the higher the density is, the smaller the depth value is (that is, the distance from the depth sensor 2 is short).

The process of detecting the upper limb region from the depth image 1101 by the background subtraction method is performed by the upper limb region detection unit 121 of the hand position detection device 1. The upper limb region detection unit 121 calculates, for example, the difference from the depth value of the same position in the background image 191 for each depth value of each position (each pixel) in the depth image 1101 of FIG. The depth value of each position in the depth image 1101 is binarized based on the magnitude relationship between the image and the threshold value. Here, the threshold value is, for example, a value that is about several mm to several cm when converted into an actual distance in a three-dimensional space. In this way, the difference in depth value becomes larger than the threshold value at each position in the upper limb region in the depth image 1101, and the difference in depth value becomes smaller than the threshold value at each position outside the upper limb region. For example, the depth value of each position in the region corresponding to the upper left limb 710 in the depth image 1101 of FIG. 8 is smaller than the depth value of the surface of the first workbench 8A, and the difference is several cm in the actual distance. It will be a dozen centimeters from. Similarly, the depth value of each position in the region corresponding to the upper right limb 720 in the depth image 1101 of FIG. 8 is smaller than the depth value of the surface of the first workbench 8A, and the difference is a numerical value at the actual distance. It becomes from cm to a dozen cm. On the other hand, at each position in the region showing the depth value of the first workbench 8A and the region showing the depth value of the second workbench 8B in the depth image 1101 of FIG. The difference from the depth value in the background image 191 is about 0 to several mm. Further, at each position in the region showing the depth value of the shelf 9 in the depth image 1101 of FIG. 8, the difference between the depth value in the depth image 1101 and the depth value in the background image 191 is about 0 to several mm.

Therefore, the upper limb region detection unit 121 generates a binary image 1102 as shown in FIG. 9B based on the depth image 1101 of FIG. 8 and the background image 191 of FIG. 9A. In the binary image 1102 of FIG. 9B, the first upper limb region 1102A and the second upper limb region 1102B including pixels having a depth value difference larger than the threshold value are shown on a white background, and the depth value difference is equal to or less than the threshold value. A dot pattern is attached to the area including the pixels.

When the binary image 1102 is generated by the background subtraction method, the upper limb region detection unit 121 is for preventing, for example, the contour of the upper limb region (the boundary between the inside of the upper limb region and the outside of the upper limb region) from becoming discontinuous. Smoothing processing or the like may be performed.

When the upper limb regions 1102A and 1102B are detected from the depth image 1101 by the background subtraction method, the hand position detection device 1 detects the positions of the center of the hand and each joint from the upper limb regions 1102A and 1102B, respectively (step S4). As described above, the process of step S4 is performed by the position detection unit 122 in the image of the hand position detection device 1. The in-image position detection unit 122 generates, for example, a line segment indicating the extending direction of the upper limb in the depth image 1101 (or the binary image 1102) based on the position and shape of the upper limb region, and then the line segment and the upper limb region. The position of the center of the hand and the position of each joint are detected based on the contour of.

FIG. 10 is a diagram illustrating an example of a method of detecting the center of the hand and the position of each joint in a depth image.

The in-image position detection unit 122 detects the position of the center of the hand and each joint in the depth image by using, for example, the binary image 1102 generated when detecting the upper limb region. When the first upper limb region 1102A and the second upper limb region 1102B are detected as in the binary image 1102 of FIG. 9B, the position detection unit 122 in the image first, for example, first of all, first of all, the first upper limb region. The center of the hand and the position of each joint are detected from the region 1102A.

When the position of the center of the hand and each joint is detected from the first upper limb region 1102A, the position detection unit 122 in the image indicates the extension direction of the upper limb in the first upper limb region 1102A, as shown in FIG. To generate. In the binary image 1102 of FIG. 10, the subject is present below the image. Therefore, the extending direction of the upper left limb corresponding to the first upper limb region 1102A in the binary image 1102 is the direction from the lower side to the upper side of the binary image 1102. Therefore, for example, the position detection unit 122 in the image first shortens the distance between the ends in the left-right direction of the first upper limb region 1102A and converts the first upper limb region 1102A into a curve. After that, the position detection unit 122 in the image converts the curve corresponding to the first upper limb region 1102A obtained by the conversion into a plurality of line segments that approximate the curve. As a result, a line segment group FL including a plurality of line segments indicating the extending direction of the upper left limb as shown in the binary image 1102 of FIG. 10 is generated.

After generating the line segment group FL indicating the extension direction of the upper left limb, the position detection unit 122 in the image is positioned at the center of the hand (center of the left hand) based on the line segment group FL and the contour of the first upper limb region 1102A. To decide. As described above, the worker 7 is located below the binary image 1102. Therefore, when the center of the hand and each joint are on the line segment of the line segment group, the distance from the lower side of the binary image 1102 along the line segment to the center of the hand is larger than the distance to the joints such as the wrist and elbow. become longer. Further, since the center of the hand is the center of the palm, the size and shape of the portion corresponding to the center of the hand in the first upper limb region 1102A substantially coincides with the size and shape of the palm. Therefore, the position detection unit 122 in the image starts from the end Q10 farther from the lower side of the binary image 1102 at both ends of the line segment group FL, and moves toward the end Q13 in contact with the lower side of the binary image 1102. Performs processing to detect the position of the center. At this time, in the position detection unit 122 in the image, for example, from among a plurality of points on the line segment group FL, a circle H having a predetermined radius centered on the point on the line segment is inscribed in the first upper limb region 1102A. Detect points. In the first upper limb region 1102A in the binary image 1102 of FIG. 10, the circle H is inscribed in the first upper limb region 1102A at the bending point Q11 in the line segment group FL. Therefore, the position detection unit 122 in the image detects the bending point Q11 as the point P0 indicating the position of the center of the hand.

After detecting the position of the center of the hand, the position detection unit 122 in the image is based on the position of the center of the hand, the position of the bending point in the line segment group, and the average value of the distance from the center of the hand to the wrist in the binary image 1102. And detect the position of the wrist. When detecting the position of the wrist, the position detection unit 122 in the image detects a bending point existing in the direction from the position of the center of the hand to the position of the wrist in the line segment group, and determines whether or not the position of the wrist is appropriate. .. The position detection unit 122 in the image detects a bending point Q12 existing in the direction from the position Q11 at the center of the hand to the position of the wrist in the line segment group FL of the binary image 1102 in FIG. 10, and the bending point Q12 is the position of the wrist. It is judged whether it is appropriate or not. At this time, the position detection unit 122 in the image has the bending point Q12 of the wrist, for example, based on whether or not the distance from the position Q11 at the center of the hand to the bending point Q12 corresponds to the distance from the center of the hand to the wrist. Determine if it is appropriate as a position. Further, the position detection unit 122 in the image is, for example, based on whether or not the angle formed by the two line segments connected at the bending point Q12 is within the bendable angle range of the wrist, the bending point Q12. Determines whether is the appropriate wrist position. Then, when it is determined that the bending point Q12 is appropriate as the position of the wrist, the position detection unit 122 in the image detects the bending point Q12 as the point P2 indicating the position of the wrist.

After that, the position detection unit 122 in the image subsequently detects the position of the elbow. However, the first upper limb region 1102A in the binary image 1102 of FIG. 10 does not include the left elbow portion. Therefore, the position detection unit 122 in the image ends the process of detecting the position of the hand center and each joint from the first upper limb region 1102A, and then the position of the hand center and each joint from the second upper limb region 1102B. Is processed to detect.

When detecting the center of the hand and the position of each joint from the second upper limb region 1102B, the position detection unit 122 in the image first generates a line segment group FR indicating the extension direction of the upper limb corresponding to the second upper limb region 1102B. do. After that, the position detection unit 122 in the image performs a process of detecting the position of the center of the hand toward the other end Q23 in the line segment group FR, starting from the end Q20 farthest from the lower side of the day image 1102 in the generated line segment group FR. .. At this time, the position detection unit 122 in the image detects the bending point Q21 in the line segment group FR as the point P0 indicating the position of the center of the hand.

After detecting the position of the center of the hand in the second upper limb region 1102B, the position detection unit 122 in the image detects the position of the wrist on the line segment group FR. However, in the line group FR in the binary image of FIG. 10, the bending point Q22 that first appears when viewed from the bending point Q21 in the direction of the wrist is located at the elbow portion of the upper right limb corresponding to the second upper limb region 1102B. exist. Therefore, the distance from the position of the center of the hand (bending point Q21) to the bending point Q22 is longer than the distance from the center of the hand to the wrist. That is, when the bending point Q21 in the line segment group FR is set to the position of the center of the hand, the bending point corresponding to the position of the wrist does not exist in the line segment group FR. In this way, when there is no bending point corresponding to the position of the wrist in the line segment group, the position detection unit 122 in the image is, for example, on the line segment connecting the position of the center of the hand (the bending point Q21) and the bending point Q22. Point Q24 is detected as point P1 indicating the position of the wrist. The point Q24 is a point where the distance from the position of the center of the hand is the distance from the center of the hand to the wrist in the depth image among the plurality of points on the line segment connecting the position of the center of the hand and the bending point Q22.

After that, the position detection unit 122 in the image detects the bending point Q22 in the line segment group FR as the point P2 indicating the position of the elbow. Further, after detecting the position of the elbow, the position detection unit 122 in the image performs a process of detecting the position of the shoulder, but the second upper limb region in the binary image 1102 of FIG. 10 includes the right shoulder portion. No. Therefore, the position detection unit 122 in the image ends the process of detecting the position of the center of the hand and the position of each joint from the first upper limb region 1102A.

After detecting the center of the hand and the position of each joint in the upper limb region based on the binary image, the hand position detecting device 1 calculates the three-dimensional coordinates of the center of the hand and each joint detected by the three-dimensional coordinate calculation unit 123. .. The three-dimensional coordinate calculation unit 123 applies a pinhole camera model, and is based on the imaging range of the depth image, the number of pixels of the depth image, the positions of the hand center and the joint in the depth image, and the depth value. And calculate the three-dimensional coordinates of the joint.

The three-dimensional coordinate calculation 123 calculates the three-dimensional coordinates of the detected hand center and each joint for each upper limb region detected from the depth image 1101 (binary image 1102). For example, the three-dimensional coordinate calculation unit 123 first obtains the three-dimensional coordinates of the point P0 indicating the position of the center of the hand detected from the first upper limb region 1102A in FIG. 10 and the three-dimensional coordinates of the point P1 indicating the position of the wrist. calculate. The three-dimensional coordinate calculation unit 123 applies a pinhole camera model to the position Q11 of the center of the hand in the depth image 1101, the depth value of the position Q11, the imaging range of the depth image 1101, and the number of pixels of the depth image 1101. Based on, the three-dimensional coordinates of the center of the hand and the wrist are calculated. After that, the three-dimensional coordinate calculation unit 123 determines the three-dimensional coordinates of the point P0 indicating the position of the center of the hand detected from the second upper limb region 1102B in FIG. 10, the three-dimensional coordinates of the point P1 indicating the position of the wrist, and the elbow. The three-dimensional coordinates of the point P2 indicating the position are calculated.

After calculating the three-dimensional coordinates of the hand center and each joint detected from the depth image 1101, the hand position detection device 1 determines and corrects whether or not the three-dimensional coordinates are corrected by the correction unit 130. The consistency determination unit 131 determines whether or not the three-dimensional coordinates are corrected. The consistency determination unit 131 performs the consistency determination process according to the flowchart of FIG. 5 to determine whether or not the three-dimensional coordinates are corrected. The consistency determination process performed by the consistency determination unit 131 includes determination of spatial consistency and determination of time consistency.

FIG. 11 is a diagram illustrating a method for determining spatial consistency.
The depth image 1101 of FIG. 11 shows the upper limb regions 1101A and 1101B, the center of the hand, and the positions of the joints.

The consistency determination unit 131 determines the spatial consistency and the time consistency of the three-dimensional coordinates (positions) of the center of the hand and each joint for each upper limb region in the depth image 1101. At this time, the consistency determination unit 131 in the hand position detection device 1 of the present embodiment first determines the spatial consistency as shown in the flowchart of FIG.

For example, in the first upper limb region 1101A in the depth image 1101 of FIG. 11, a point P0 indicating the center of the hand and a point P1 indicating the wrist are shown. In the determination of spatial consistency, the consistency determination unit 131 determines the spatial consistency regarding the positional relationship between the center of the hand and the wrist. First, the consistency determination unit 131 sets the consistency parameter C0 for the point P0 indicating the center of the hand to "True" (step S60301), and sets the three-dimensional coordinates (X0, Y0, Z0) of the point P0 in the depth image 1101. The coordinates (x0, y0) and the distance z0 are calculated (step S60302). Here, the consistency determination unit 131 sets the three-dimensional coordinates (X0, Y0, Z0) into the coordinates (x0) in the depth image by the back calculation formula for the formula for calculating the three-dimensional coordinates from the coordinates and the depth value of the depth image 1101. , Y0) and the distance z0.

Next, the consistency determination unit 131 determines whether or not the magnitude relationship between the distance z0 with respect to the point P0 and the depth value z0'of the coordinates (x0, y0) in the depth image 1101 is z0'> z0 + M. (Steps S60303 and S60304). When the depth value of the coordinates (x0, y0) in the depth image 1101 corresponds to the distance to the hand center existing in the imaging range of the depth image, the magnitude relationship between the distance z0 and the depth value z0'is described above. As shown in this case, z0'≦ z0 + M. The point P0 indicating the center of the hand in the depth image 1101 of FIG. 11 is included in the first upper limb region 1101A and is located in the central part of the palm. Therefore, the magnitude relationship between the distance z0 and the depth value z0'with respect to the point P0 indicating the center of the hand is z0'≤ z0 + M. In this case, the consistency determination unit 131 then calculates a three-dimensional distance len between the position of the center of the hand and the position of the adjacent joint (wrist), and the distance len is the dimension from the center of the hand to the wrist of the subject. It is determined whether or not it is consistent with (steps S6035-S60307).

When z0'≤ z0 + M, the consistency determination unit 131 then determines the three-dimensional coordinates (X0, Y0, Z0) of the point P0 indicating the center of the hand and the three-dimensional coordinates (X1) of the point P1 indicating the adjacent wrist. , Y1, Z1) and the three-dimensional distance len is calculated (step S60305). The distance len1'between the points P0 and the point P1 in the depth image 1101 of FIG. 11 indicates the distance (in-plane distance) in the depth image corresponding to the three-dimensional distance len. After calculating the three-dimensional distance len, the consistency determination unit 131 reads out the dimension from the center of the subject's hand to the wrist (first dimension L1) from the dimension history 192 and calculates the difference ΔL = len-L1 (1). Step S60306).

After calculating the difference ΔL between the three-dimensional distance len and the dimension of the subject, the consistency determination unit 131 determines whether the relationship between the absolute value | ΔL | of the calculated difference ΔL and the threshold value THL is | ΔL |> THL. It is determined whether or not (step S60307).

The position of the point P1 in the first upper limb region 1101A in the depth image 1101 of FIG. 11 exists at the wrist portion. Therefore, the coordinates (x1, y1) of the position of the point P1 in the depth image 1101 and the three-dimensional coordinates of the point P1 calculated based on the depth value correspond to the position of the wrist in the imaging range of the depth image. Therefore, the three-dimensional distance len between the three-dimensional coordinates of the point P0 and the three-dimensional coordinates of the point P1 substantially coincides with the dimensions from the center of the hand to the wrist of the subject within the imaging range of the center of the hand and the depth image. That is, since the difference ΔL calculated by the consistency determination unit 131 is a very small value, the magnitude relationship between the absolute value | ΔL | and the threshold value THL is | ΔL | ≦ THL. In this case, the consistency determination unit 131 spatially matches the three-dimensional coordinates (X0, Y0, Z0) of the point P0 indicating the hand center with the position of the subject's hand center within the imaging range of the depth image. judge. When it is determined that the consistency is spatially matched, the consistency determination unit 131 next determines whether or not the three-dimensional coordinates (X0, Y0, Z0) of the point P0 indicating the hand center are temporally matched (time consistency). (Steps S60308 and S60309).

In order to determine the time consistency with respect to the three-dimensional coordinates of the point P0, the consistency determination unit 131 of the present embodiment calculates the moving speed S0 of the point P0 (step S60308). In order to calculate the movement speed S0, the consistency determination unit 131 uses the three-dimensional coordinates (X0', Y0', Z0') of the point P0 indicating the center of the hand detected from the first upper limb region 1101A in the past depth image 1101. Is read from the coordinate history 193. For example, when the depth image 1101 of FIG. 11 is a depth image captured at time t, the consistency determination unit 131 uses, for example, the three-dimensional coordinates of the point P0 detected from the depth image captured at time t−Δt. read out. Here, Δt is, for example, the imaging interval of the depth image in the depth sensor 2.

After reading the three-dimensional coordinates of the past point P0, the consistency determination unit 131 uses the three-dimensional coordinates of the current point P0 (X0, Y0, Z0) and the three-dimensional coordinates of the past point P0 (X0', Y0'). , Z0'), and the moving speed S0 is calculated based on the three-dimensional distance and the imaging interval Δt of the depth image.

When the upper limbs of the worker 7 are moving within the imaging range 10 of the depth image, there is an upper limit to the moving speed of the hand moved by the worker 7. Therefore, the three-dimensional coordinates (X0, Y0, Z0) of the current point P0 and the three-dimensional coordinates (X0', Y0', Z0') of the past point P0 are objects within the imaging range of the depth image, respectively. When corresponding to the center of the left hand of the person, the movement speed S0 is always equal to or less than the threshold value THS. In other words, when the moving speed S0 is larger than the threshold THS, the position in the upper limb of the subject corresponding to the three-dimensional coordinates (X0, Y0, Z0) of the current point P0 is the three-dimensional coordinates of the past point P0. It can be said that the position in the upper limb of the subject corresponding to (X0', Y0', Z0') is different. Therefore, when S0 ≦ THS, the consistency determination unit 131 temporally changes the three-dimensional coordinates (X0, Y0, Z0) of the point P0 indicating the hand center and the three-dimensional coordinates of the past hand center. Judged to be consistent. When it is determined that they are consistent in time, the consistency determination unit 131 keeps the consistency parameter C0 with respect to the point P0 indicating the center of the hand as “True”. On the other hand, when S0> THS, the consistency determination unit 131 temporally determines the three-dimensional coordinates (X0, Y0, Z0) of the point P0 indicating the hand center and the three-dimensional coordinates of the past hand center. Judge as inconsistent. When it is determined that they do not match in time, the consistency determination unit 131 changes the consistency parameter C0 for the point P0 indicating the center of the hand to "False" (step S60312).

When the consistency determination unit 131 finishes determining the consistency of the point P0 indicating the center of the hand detected from the first upper limb region 1101A, the consistency determination unit 131 next, regarding the point P1 indicating the wrist detected from the first upper limb region 1101A. To determine the consistency of. In the depth image 1101 of FIG. 11, the only joint detected from the first upper limb region 1101A is the wrist. Therefore, when the consistency determination for the point P1 is completed, the consistency determination unit 131 ends the consistency determination process for the center of the hand and the joint detected from the first upper limb region 1101A.

Further, the consistency determination unit 131 performs the following processing in the consistency determination processing with the hand center and the joint detected from the second upper limb region 1101B in the depth image 1101 of FIG. 11 as the processing target.

The consistency determination unit 131 first sets the consistency parameter C0 of the point P0 indicating the center of the hand detected from the second upper limb region 1101B to "True", and determines the spatial consistency of the three-dimensional coordinates of the point P0. Make a judgment. The consistency determination unit 131 first calculates the coordinates (x0, y0) in the depth image and the distance z0 from the three-dimensional coordinates of the point P0, and the depth value of the distance z0 and the coordinates (x0, y0) in the depth image. It is determined whether or not the relationship with z0'is z0'> z0 + M. In the depth image 1101 of FIG. 11, the position of the point P0 is within the second upper limb region 1101B. Therefore, the relationship between the distance z0 and the depth value z0'is z0'≤z0 + M. Therefore, the consistency determination unit 131 then calculates a three-dimensional distance len between the point P0 indicating the center of the hand and the point P1 indicating the wrist detected from the second upper limb region 1101B, and the three-dimensional distance len is used. The difference ΔL from the dimension from the center of the subject's hand to the wrist is calculated. After that, the consistency determination unit 131 determines whether or not the relationship between the calculated absolute value | ΔL | of the difference ΔL and the threshold value THL is | ΔL |> THL.

The position of the point P1 indicating the wrist in the second upper limb region 1101B of the depth image 1101 of FIG. 11 is, for example, on a line segment connecting the point P0 indicating the center of the hand and the point P2 indicating the elbow, from the point P0. Is the point where the distance is the dimension from the center of the hand to the wrist. Therefore, the three-dimensional distance len between the point P0 indicating the center of the hand and the point P1 indicating the wrist substantially coincides with the dimension from the center of the hand to the wrist of the subject. Therefore, the magnitude relationship between the absolute value | ΔL | and the threshold value THL is | ΔL | ≦ THL. In this case, the consistency determination unit 131 spatially matches the three-dimensional coordinates (X0, Y0, Z0) of the point P0 indicating the hand center with the position of the subject's hand center within the imaging range of the depth image. judge. When it is determined that they are spatially aligned, the consistency determination unit 131 next calculates the moving speed S0 of the three-dimensional coordinates (X0, Y0, Z0) of the point P0 indicating the center of the hand, and the three-dimensional coordinates of the point P0. Determines whether or not is time-consistent (time-consistent). When matching in time, the consistency determination unit 131 leaves the consistency parameter C0 at the point P0 as “True”. On the other hand, if they do not match in time, the consistency determination unit 131 changes the consistency parameter C0 at the point P0 to "False". After that, the consistency determination unit 131 determines the spatial consistency of the three-dimensional coordinates of the point P1 indicating the wrist detected from the second upper limb region 1101B.

Although detailed description will be omitted, the relationship between the distance z1 calculated by the consistency determination unit 131 from the three-dimensional coordinates of the point P1 and the depth value z1'in the depth image 1101 is z1'<z1 + M. Therefore, the consistency determination unit 131 next determines whether or not the three-dimensional distance len between the point P1 indicating the wrist and the point P2 indicating the elbow matches the dimension from the wrist to the elbow of the subject. As described above, the second upper limb region 1101B in the depth image 1101 of FIG. 11 lacks the region 1101C corresponding to the hand portion in the upper right limb of the subject due to occlusion. When the second upper limb region 1101B includes the region 1101C corresponding to the hand portion, the positions of the hand center and the wrist detected from the second upper limb region 110B are the points PC0 and the points in the depth image 1101 of FIG. 11, respectively. It is the position of PC1. That is, in the depth image 1101 in which the region 1101C corresponding to the hand portion in the upper right limb of the subject is missing due to occlusion, the positions of the point P0 indicating the center of the hand and the point P1 indicating the wrist are respectively from the actual positions. Is also closer to the elbow joint side. Therefore, the three-dimensional distance len between the point P1 indicating the wrist and the point P2 indicating the elbow is shorter than the three-dimensional distance between the point PC1 indicating the correct wrist position and the point P2 indicating the elbow, and the difference in distance is approximately short. The size is hard to say that they are the same. The distance len2'between points P1 and P2 in the second upper limb region 1101B in the depth image 1101 of FIG. 11 is the distance (plane) in the depth image corresponding to the three-dimensional distance len of the two points P1 and P2. Inner distance) is shown. Further, the distance lenc'between the points PC1 and the point P2 in the depth image 1101 of FIG. 11 indicates the distance (in-plane distance) in the depth image corresponding to the three-dimensional distance between the two points PC1 and P2. ..

That is, when the point P1 indicating the wrist detected from the second upper limb region 1101B is the determination target, the difference ΔL between the three-dimensional distance len between the points P1 and the point P2 and the dimension from the wrist to the elbow of the subject. The absolute value | ΔL | becomes | ΔL |> THL. When | ΔL |> THL, the consistency determination unit 131 has a space in which the three-dimensional coordinates of the point P1 indicating the wrist detected from the second upper limb region 1101B and the three-dimensional coordinates of the point P2 indicating the adjacent elbow are spatial. Judged as inconsistent. Therefore, the consistency determination unit 131 changes the consistency parameter C1 of the point P1 indicating the wrist detected from the second upper limb region 1101B to “False” (step S60312). When it is determined that the points are not spatially consistent in this way, the consistency determination unit 131 omits the determination of the time consistency with respect to the point P1, and then determines the consistency of the point P2. In the depth image 1101 of FIG. 11, the only joints detected from the second upper limb region 1101B are the wrist and the elbow. Therefore, when the determination of the consistency of the point P2 is completed, the consistency determination unit 131 ends the consistency determination process for the center of the hand and the joint detected from the second upper limb region 1101B.

When the consistency determination process for the hand center and the joint detected from the depth image 1101 of FIG. 11 is completed, the consistency determination unit 131 has a hand center or a joint in which the consistency parameter Ci is "False" for each upper limb region. It is determined whether or not to do so (step S604). When all the consistency parameters of the hand center and the joint corresponding to the upper limb region to be determined are "True", the consistency determination unit 131 (correction unit 130) is the current of the hand center and each joint. The three-dimensional coordinates are determined to be the three-dimensional coordinates stored in the coordinate history 193 (step S608).

On the other hand, when there is a hand center or joint whose consistency parameter is "False", the consistency determination unit 131 changes the position of the hand center and recalculates the three-dimensional coordinates of the hand center and each joint. The recalculation unit 132 is made to perform the calculation process (step S606). The consistency determination unit 131 in the hand position detection device 1 of the present embodiment is used in the recalculation unit 132 when the number of times r of the coordinate recalculation processing is performed is N times or less as shown in the flowchart of FIG. Have the coordinates recalculated.

When the depth image to be processed is the depth image 1101 of FIG. 11 and the determination target of the consistency determination process is the first upper limb region 1101A, all the consistency parameters are "True". Therefore, the consistency determination unit 131 stores the current three-dimensional coordinates of the point P0 indicating the center of the hand corresponding to the first upper limb region 1101A and the current three-dimensional coordinates of the point P1 indicating the wrist in the coordinate history 193. Determine the three-dimensional coordinates to be used.

On the other hand, when the depth image to be processed is the depth image 1101 of FIG. 11 and the determination target of the consistency determination processing is the second upper limb region 1101B, the consistency of the point P1 indicating the wrist The parameter C1 is "False". Therefore, the consistency determination unit 131 causes the recalculation unit 132 to perform a coordinate recalculation process for recalculating the three-dimensional coordinates of the hand center and each joint corresponding to the second upper limb region 1101B. The recalculation unit 132 performs a process according to the flowchart of FIG. 6 as a coordinate recalculation process. That is, the recalculation unit 132 changes the position of the point P0 indicating the center of the hand in the second upper limb region 1101B in the depth image 1101 of FIG. 11, and the second upper limb is based on the changed position of the point P0. The three-dimensional coordinates of the center of the hand corresponding to the region 1101B and each joint are calculated.

First, the recalculation unit 132 changes the position of the point P0 indicating the center of the hand corresponding to the second upper limb region 110B in the depth image 1101 (step S60601). In the process of step S60601, the recalculation unit 132 determines, for example, in the direction of the vector in the depth image 1101 starting from the point P1 indicating the wrist or the point P2 indicating the elbow and ending at the point P0 indicating the center of the hand. The position of the point P1 is moved by the distance (the number of pixels) of.

FIG. 12 is a diagram illustrating a method of changing the position of the center of the hand. FIG. 12 shows an example of a method of changing the position of the point P0 indicating the center of the hand corresponding to the second upper limb region 1101B in the depth image 1101 of FIG.

The position of the point P0 before the change in the depth image 1101 of FIG. 12 is the position of the bending point Q21 in the line segment group FR of FIG. Further, the position of the point P2 indicating the elbow in the depth image 1101 of FIG. 12 is the position of the bending point Q22 in the line segment group FR of FIG. Further, the position of the point P1 indicating the wrist in the depth image 1101 of FIG. 12 is the position of the point Q24 on the line segment connecting the point Q21 and the point Q22 in the line segment group FR.

When changing the position of the point P0 indicating the center of the hand, the recalculation unit 132 starts at the point P1 indicating the wrist and ends at the point P0 indicating the center of the hand as a predetermined distance in the direction of the vector in the depth image 1101. Move by (number of pixels). The positions of the points P0 and P1 in the depth image 1101 before the position of the point P0 is changed are the positions Q21 and Q24, respectively. When the depth image 1101 of FIG. 12 is the target of processing, the recalculation unit 132 changes the position of the point P0 indicating the center of the hand in the direction of the vector starting from the position Q24 and ending at the position Q21. That is, the recalculation unit 132 is centered on the line segment LS1 passing through the positions Q21 and Q24 in the depth image 1101 by a predetermined number of pixels in the direction in which the distance from the position Q24 is longer than the position Q21. The position P0 of is changed.

After changing the position of the point P0 indicating the center of the hand, the recalculation unit 132 changes the position of the point P1 indicating the wrist and the position of the point P2 indicating the elbow based on the changed position of the point 0 (step S60602). ). When the position of the point P0 in the depth image 1101 of FIG. 12 is changed from the position Q21 to the position Q25, the recalculation unit 132 positions the position of the point P1 indicating the wrist by the amount of movement of the point P, for example, from the position Q24. Move to the Q21 side. Further, the recalculation unit 132 keeps the position of the point P2 indicating the elbow at the position Q22.

After changing the positions of the hand center and each joint in the depth image 1101, the recalculation unit 132 calculates the three-dimensional coordinates of the hand center and each joint (steps S60603 to S60607). When calculating the three-dimensional coordinates of the hand center and each joint, the recalculation unit 132 first determines whether or not the position of the hand center for calculating the three-dimensional coordinates or the joint in the depth image is within the upper limb region. (Step S60604). When the position of the center of the hand or the joint is within the upper limb region, the recalculation unit 132 calculates the three-dimensional coordinates based on the depth value of the position of the center of the hand or the joint (step S60605). On the other hand, when the position of the center of the hand or the joint is outside the upper limb region, the recalculation unit 132 calculates the depth value of the center of the hand or the joint, and calculates the three-dimensional coordinates based on the depth value (step S60606). ..

For example, when calculating the three-dimensional coordinates of the hand center in the depth image 1101 of FIG. 12, the recalculation unit 132 determines whether or not the changed position of the hand center is within the upper limb region 1101B. When the position of the hand center after the change is within the upper limb region 1101B (for example, the position Q25), the recalculation unit 132 uses the coordinates of the position Q25 in the depth image 1101 and the depth value of the position Q25 to determine the hand center. The three-dimensional coordinates of the indicated point P0 are calculated. On the other hand, when the position of the center of the hand after the change is outside the upper limb region 1101B (for example, the position Q26), the recalculation unit 132 calculates the depth value of the center of the hand at the position Q26 and uses the depth value to center the hand. The three-dimensional coordinates of the point P0 indicating the above are calculated.

When the position of the center of the hand after the change is the position Q26 outside the upper limb region 1101B, the depth value of the position Q26 in the depth image 1101 is not the distance to the subject's hand, for example, the shelf in the depth image 1101 of FIG. Corresponds to the distance to 9. Therefore, the coordinates of the position Q26 in the depth image 1101 and the three-dimensional coordinates calculated by using the depth value of the position Q26 are values indicating the position of the shelf 9 instead of the position of the center of the hand of the target person. Therefore, when the position of the hand center after the change is outside the upper limb region 1101B, the recalculation unit 132 calculates the depth value of the hand center at the position of the hand center after the change in order to correctly calculate the three-dimensional coordinates of the hand center. Calculate (estimate).

When the position of the center of the hand is outside the upper limb region, the recalculation unit 132 calculates the depth value of the center of the hand based on the distribution of the depth value on the line segment passing through the center of the hand and the wrist in the upper limb region 1101. ..

FIG. 13 is a graph illustrating a method of estimating a depth value used for calculating three-dimensional coordinates. The horizontal axis in the graph of FIG. 13 is the position on the line segment LS1 in the depth image 1101 of FIG. Further, the vertical axis in the graph of FIG. 13 is the depth (mm) in the depth image 1101 and corresponds to the distance from the depth sensor 2.

The positions Q30, Q22, Q21, Q25, Q31, Q26, and Q32 on the horizontal axis of the graph of FIG. 13 correspond to the positions on the line segment LS1 in the depth image 1101 of FIG. 12, respectively. That is, the thick line BZ1 in the section from the position Q30 to the position Q31 in the graph of FIG. 13 indicates the depth value corresponding to each position on the line segment LS1 in the upper limb of the subject. Further, the thick line BZ2 in the section from the position Q31 to the position Q32 in the graph of FIG. 13 is the depth value of each position in the depth image 1101 and indicates the depth value corresponding to the surface of the shelf 9. On the other hand, the thick dotted line BZ3 in the section from the position Q31 to the position Q32 in the graph of FIG. 13 corresponds to each position on the line segment LS1 in the upper limb of the subject, which is estimated from the change in the depth value of the thick line BZ1. Shows the value.

When the changed position of the point P0 indicating the center of the hand is outside the upper limb region 1101B as in the position Q26, the recalculation unit 132 distributes the thick dotted line BZ3 based on the distribution of the depth value in the upper limb region 1101B. Estimate. The recalculation unit 132, for example, is based on the amount of change in the depth value from the position Q21 of the center of the hand or the position Q24 of the wrist before the change to the position Q31 intersecting the outer circumference of the upper limb region 1101B on the line segment LS1. A thick dotted line BZ3 indicating a change in the depth value of the upper limb from the position Q31 to the position Q32 is calculated. After that, the recalculation unit 132 calculates the depth value Z6 of the point corresponding to the position Q26 on the calculated thick dotted line BZ3, and calculates the three-dimensional coordinates of the hand center using the depth value Z6 as the depth value of the hand center.

In this way, when the changed position of the point P0 indicating the center of the hand is outside the upper limb region, the depth value of the center of the hand is calculated (estimated) and the three-dimensional coordinates are calculated to obtain the depth image by occlusion. It is possible to correctly calculate the three-dimensional coordinates of the center of the hand that are not shown in.

In addition, the recalculation unit 132 calculates the depth value of the wrist based on the distribution of the depth value in the upper limb region even when the position after the change of the point P1 indicating the wrist is outside the upper limb region, for example. (Estimate) Calculate 3D coordinates.

When the calculation of the three-dimensional coordinates of the center of the hand and each joint is completed, the recalculation unit 132 notifies the consistency determination unit 131 of the calculated three-dimensional coordinates (step S60609), and ends the coordinate recalculation process. After that, the consistency determination unit 131 determines the spatial consistency and the time consistency of the three-dimensional coordinates of the hand center and each joint calculated by the recalculation unit 132. When it is determined that the center of the hand and all of the joints are spatially and temporally aligned, the consistency determination unit 131 stores the current three-dimensional coordinates of the center of the hand and each joint in the coordinate history 193. Determine to 3D coordinates. On the other hand, when either the center of the hand or each joint is not spatially or temporally aligned, the consistency determination unit 131 performs a process of correcting the three-dimensional coordinates of the center of the hand and each joint to the recalculation unit 132. Let me do it.

As described above, the hand position detection device 1 of the present embodiment detects the position of the subject's hand center and each joint based on the position and shape of the upper limb region in the depth image, and the position and depth in the depth image. Based on the value, the three-dimensional coordinates of the center of the hand and each joint are calculated. Further, when the three-dimensional coordinates of the hand center and each joint calculated from the depth image and the dimensions of the upper limbs measured from the subject do not spatially match, the hand position detecting device 1 calculates the hand center and the hand center calculated from the depth image. Correct the 3D coordinates of each joint. When correcting the three-dimensional coordinates of the hand center and each joint, the hand position detection device 1 allows the position of the hand center and each joint in the depth image to be a position outside the upper limb region in the depth image. .. Therefore, even if the depth image does not include the depth value to the center of the hand due to occlusion, the hand position detection device 1 of the present embodiment can correctly calculate the three-dimensional coordinates of the center of the hand. ..

Further, in the hand position detection device 1 of the present embodiment, the temporal consistency is determined separately from the spatial consistency determination. That is, when the moving speed Si of the three-dimensional coordinates of the center of the hand or the joint calculated from the depth image is larger than the predetermined speed threshold THS, the hand position detecting device 1 corrects the three-dimensional coordinates of the center of the hand and each joint. By determining whether or not the spatial and temporal alignment is consistent in this way, the hand position detection device 1 of the present embodiment can more accurately calculate the three-dimensional position of the hand center.

Further, in the hand position detection device 1 of the present embodiment, after the recalculation unit 132 calculates (recalculates) the three-dimensional coordinates of the hand center and each joint, the consistency determination unit 131 spatially and temporally matches. Determine whether or not to do so. Then, the recalculation unit 132 calculates (recalculates) the three-dimensional coordinates of the hand center and each joint a plurality of times until the consistency determination unit 131 determines that they are spatially and temporally consistent. Therefore, even when the deviation between the correct position of the hand center in the depth image and the position of the hand center detected based on the upper limb region is large, it is possible to correctly calculate the three-dimensional coordinates of the hand center.

In the hand position detection device 1 of the present embodiment, as described above, the upper limit is set for the number of times the three-dimensional coordinates are calculated by the recalculation unit 132. By setting an upper limit on the number of times the three-dimensional coordinates are calculated, the hand position detection device 1 can prevent an increase in the processing load and a delay in processing due to an increase in the number of times the three-dimensional coordinates are recalculated by the recalculation unit 132. It becomes.

Further, the flowchart of FIG. 4 is only an example of the coordinate correction process performed by the hand position detection device 1 (correction unit 130) of the present embodiment. The coordinate correction process performed by the hand position detection device 1 of the present embodiment can be appropriately changed as long as it does not deviate from the gist of the present embodiment. The hand position detection device 1 of the present embodiment may perform, for example, a process according to the flowcharts of FIGS. 14A and 14B as the coordinate correction process.

FIG. 14A is a flowchart (No. 1) for explaining a modification of the coordinate correction process. FIG. 14B is a flowchart (No. 2) for explaining a modification of the coordinate correction process. Of the processing steps in the flowcharts of FIGS. 14A and 14B, the processing steps having the same processing contents as the processing steps in the flowchart of FIG. 4 have the same reference numerals as the processing steps of the flowchart of FIG. 4 (for example, S601 and S601 and). S602 etc.) is attached.

The coordinate correction process according to the flowcharts of FIGS. 14A and 14B is performed by the correction unit 130 of the hand position detection device 1. First, as shown in FIG. 14A, the correction unit 130 sets the variable r indicating the number of times the three-dimensional coordinates have been recalculated by the recalculation unit 132 to “0” (step S601), and subsequently selects one upper limb region. The three-dimensional coordinates of the center of the hand and the joint are read out (step S602). The processing of steps S601 and S602 is performed, for example, by the consistency determination unit 131 of the correction unit 130.

Next, the correction unit 130 performs a consistency discrimination process (step S603) for discriminating the spatial consistency and the time consistency of the three-dimensional coordinates of the read hand center and the joint, and whether or not there is a hand center or the joint that does not match. (Step S604). The consistency determination process of step S603 and the determination of step S604 are performed by the consistency determination unit 131. The consistency determination unit 131 performs, for example, a process according to the flowchart of FIG. 5 as the consistency determination process (step S603). When all the three-dimensional coordinates of the center of the hand and the joint are spatially and temporally aligned (step S604; NO), the consistency determination unit 131 uses the current three-dimensional coordinates as the coordinate history as shown in FIG. 14B. The three-dimensional coordinates to be stored in 193 are determined (step S608). On the other hand, when either the center of the hand or the three-dimensional coordinates of the joint do not match spatially or temporally (step S604; YES), the consistency determination unit 131 then sets the variable r <. It is determined whether or not it is N (step S605). When r ≧ N (step S605; NO), the consistency determination unit 131 determines the current three-dimensional coordinates as the three-dimensional coordinates to be stored in the coordinate history 193 (step S608). On the other hand, when r <N (step S605; YES), the consistency determination unit 131 then performs the adjacent region determination process (step S611).

In the adjacent area determination process (step S611), the adjacent area in the depth image, which is the moving direction when changing the position of the center of the hand, is in front of the upper limb (depth sensor 2 side). It is a process of determining whether or not the area indicates the depth value of the object existing in. In the following description, the region showing the depth value of the object existing in front of the upper limb (depth sensor 2 side) in the depth image is referred to as the front region. The consistency determination unit 131 determines whether or not the adjacent region including the position Q26 is the front region, for example, based on the depth value of each point on the line segment LS1 in the depth image 1101 of FIG. To determine.

After the adjacent region determination process, the consistency determination unit 131 determines whether or not there is a front region adjacent to the upper limb region (step S612). When there is no front region adjacent to the upper limb region (step S612; NO), the consistency determination unit 131 determines the current three-dimensional coordinates as the three-dimensional coordinates stored in the coordinate history 193 (step). S608). On the other hand, when there is a front region adjacent to the upper limb region (step S612; YES), the consistency determination unit 131 causes the recalculation unit 132 to perform the coordinate recalculation process (step S606). The recalculation unit 132 performs, for example, a process according to the flowchart of FIG. 6 as the coordinate recalculation process.

When the recalculation unit 132 finishes the coordinate recalculation process (step S606), the consistency determination unit 131 of the correction unit 130 performs the consistency determination process (step S603) for the recalculated three-dimensional coordinates. After that, until it is determined as "NO" in any of steps S604, S605, and S612, the correction unit 130 recalculates the three-dimensional coordinates by the recalculation unit 132 and the consistency determination unit 131. Repeat the judgment. Then, when it is determined as "NO" in any of steps S604, S605, and S612, the correction unit 130 (consistency determination unit 131) converts the current three-dimensional coordinates into the coordinate history 193 as shown in FIG. 14B. It is determined to be the three-dimensional coordinates to be stored (step S608).

After finishing the process of step S608, the correction unit 130 (consistency determination unit 131) next determines whether or not there is an unselected upper limb region (step S609). When there is an unselected upper limb region (step S609; YES), the consistency determination unit 131 performs the processing after step S601 on the unselected upper limb region. Then, when the processing after step S601 is performed on all the upper limb regions, the correction unit 130 ends the coordinate correction processing.

As described above, in the modified example of the coordinate correction process, the hand position detecting device 1 performs the coordinate recalculation process before performing the coordinate recalculation process when either the center of the hand or the three-dimensional coordinates of the joint do not match spatially or temporally. The adjacent area determination process (step S611) is performed. The adjacent area determination process is a process for determining the presence or absence of a front area adjacent to the upper limb area as described above. The adjacent area determination process is performed, for example, by the consistency determination unit 131 of the correction unit 130 of the hand position detection device 1. The consistency determination unit 131 performs, for example, a process according to the flowchart of FIG. 15 as an adjacent area determination process.

FIG. 15 is a flowchart illustrating the contents of the adjacent area discrimination process.
In the adjacent region determination process, the consistency determination unit 131 is first on a line segment extending from the position of the center of the hand in the currently selected upper limb region in the direction opposite to the position of the joint, and is the position of the center of the hand. The position Pa at a predetermined distance is calculated from (step S61101).

Next, in the consistency determination unit 131, the position Pa indicates a part of the upper limb based on the depth value at each point on the line segment passing through the position of the center of the hand and the position of the joint in the upper limb region. The depth value za of the case is calculated (step S61102). In step S61102, the consistency determination unit 131 estimates the thick dotted line BZ3 based on the thick line BZ1 in the graph of FIG. 13, and calculates the depth value za when the position Pa is a point indicating a part of the upper limb. do.

Next, the consistency determination unit 131 reads out the depth value za'of the coordinates of the position Pa in the depth image (step S61103), and determines whether or not za-Ma> za'(step S61104). Here, Ma is a margin with respect to the depth value za. The margin Ma is set based on a range in which the depth value za'of the position Pa in the depth image can be regarded as the depth value of the object existing in front of the upper limb (depth sensor 2 side). That is, when the margin Ma is za-Ma> za', it is considered that the depth value za'of the position Pa in the depth image is the depth value of the object existing in front of the upper limb (depth sensor 2 side). It is a possible value. Therefore, when za-Ma> za'(step S61104; YES), the consistency determination unit 131 determines that there is a front region adjacent to the upper limb region (step S61105), and ends the adjacent region determination process. On the other hand, when za-Ma ≤ za'(step S61104; NO), the consistency determination unit 131 determines that there is no front region adjacent to the upper limb region (step S61106), and ends the adjacent region determination process.

FIG. 16 is a diagram illustrating the relationship between the adjacent region and the discrimination result.
The depth image 1101 of FIG. 16 shows the first upper limb region 1101A and the second upper limb region 1101B detected from the depth image 1101. Further, in the depth image 1101 of FIG. 16, the area 1101D showing the depth value of the shelf 9 in the depth image 1101, the area 1101E showing the depth value of the first workbench 8A, and the depth of the second workbench 8B are shown. The area 1101F showing the value is shown. Further, in the depth image 1101 of FIG. 16, the region 1101C indicating the right-hand portion of the subject existing between the shelf 9 and the second workbench 8B is shown by a two-dot chain line.

When the right-hand portion of the subject is between the shelf 9 and the second workbench 8B, the second upper limb region 1101B detected from the depth image 1101 includes the region 1101C corresponding to the right-hand portion of the subject. No. Therefore, when the hand position detecting device 1 detects the position of the point P0 indicating the center of the hand based on the second upper limb region 1101B, the position of the second upper limb region 1101B actually corresponding to the wrist. The position Q21 that is closer to the elbow is detected. Therefore, when the depth image 1101 of FIG. 16 is the target of processing, the consistency determination unit 131 of the hand position detection device 1 has any of the three-dimensional coordinates of the hand center and the joint calculated from the second upper limb region 1101B. Judged as spatially inconsistent. When it is determined that they are not spatially consistent, the consistency determination unit 131 performs an adjacent region determination process (step S611) to determine the presence or absence of a front region adjacent to the second upper limb region 1101B.

When the second upper limb region 1101B is the target of processing, the consistency determination unit 131 is on the line segment LS1 passing through the positions Q21 and Q24 in the depth image, and is in the direction opposite to the position Q24 from the position Q21. The position Pa at which the distance is LL is calculated (step S61101).

Next, the consistency determination unit 131 calculates the depth value z when the position Pa is a part of the upper limb based on the distribution of the depth value on the line segment LS1 in the second upper limb region 1101B (step). S61102). In step S61102, the consistency determination unit 131 calculates the depth value za when the position Pa is a part of the upper limb, for example, based on the graph of FIG. The position Pa in the depth image 1101 of FIG. 16 is outside the second upper limb region 1101B. Therefore, the consistency determination unit 131 calculates the depth value za based on the thick dotted line BZ3 in the graph of FIG. 13.

After that, the consistency determination unit 131 reads out the depth value za'of the position Pa in the depth image 1101 (step S61103), and determines whether or not za-Ma> za'(step S61104). The position Pa in the depth image 1101 is included in the region 1101D adjacent to the second upper limb region 1101B. The area 1101D is an area indicating the depth value of the shelf 9 installed on the second workbench 1B, and exists in front of the area 1101C indicating the right hand portion of the subject (on the side of the depth sensor 2). At this time, it cannot be said that the depth value za calculated in step S61102 and the depth value za'in the depth image 1101D both indicate the depth value of the same part of the same object. (See, for example, FIG. 13). Therefore, the consistency determination unit 131 determines that the front region adjacent to the second upper limb region 1101B exists (step S61105), and ends the adjacent region determination process. In this case, the consistency determination unit 131 causes the recalculation unit 132 to perform the coordinate recalculation process (step S606).

Further, it was determined that the point P0 indicating the center of the hand or the point P1 indicating the wrist in the first upper limb region 1101A in the depth image 1101 of FIG. 16 is not spatially or temporally aligned and r <N. In this case, the consistency determination unit 131 performs the adjacent area determination process (step S611). In this case, the consistency determination unit 131 sets a position Pa on the line segment LS2 passing through the two points P0 and P1 in the first upper limb region 1101A and having a distance LL from the point P0 in the direction opposite to the point P1. Calculate (step S61101).

Next, the consistency determination unit 131 calculates the depth value when the position Pa is a part of the upper limb based on the depth value of each point on the line segment LS2 in the first upper limb region 1101A (step). S61102).

After that, the consistency determination unit 131 reads out the depth value za'of the position Pa in the depth image 1101 (step S61103), and determines whether or not za-Ma> za'(step S61104). The position Pa in the depth image 1101 is included in the region 1101E adjacent to the first upper limb region 1101A. The region 1101E is a region showing the depth value of the first workbench 1A, and is located behind the first upper limb region 1101A including the left hand portion of the subject (far side from the depth sensor 2). At this time, the relationship between the depth value za calculated in step S61102 and the depth value za'in the depth image 1101D is za <za'(that is, za-Ma <za'). Therefore, the consistency determination unit 131 determines that the front region adjacent to the second upper limb region 1101B does not exist (step S61106), and ends the adjacent region determination process. In this case, the consistency determination unit 131 omits the coordinate recalculation process (step S606) and determines the current three-dimensional coordinates as the three-dimensional coordinates stored in the coordinate history 193 (step S608).

When it is determined in the adjacent area discrimination process that there is a front area adjacent to the upper limb area, the subject who is hidden by an object existing in front of the upper limb such as the shelf 9 is correct in the moving direction of the hand center in the depth image. It is thought that there is a hand center. Therefore, in the modified example of the coordinate correction process, when there is a front region adjacent to the upper limb region as shown in FIG. 14A, the position of the hand center is moved and the three-dimensional coordinates of the hand center and the joint are recalculated. The coordinate recalculation process (step S606) is performed. On the other hand, when there is no front region adjacent to the upper limb region, there is no object hiding a part of the upper limb at a position in front of the upper limb in the movement direction of the center of the hand in the depth image. Therefore, if the position of the moved hand center is outside the upper limb region, the three-dimensional coordinates of the hand center become erroneous coordinates. Therefore, in the modified example of the coordinate correction process, when there is no front area adjacent to the upper limb area as in FIGS. 14A and 14B, the coordinate recalculation process is not performed and the current three-dimensional coordinates are converted to the coordinate history 193. Determine the 3D coordinates to be stored. This makes it possible to prevent the recalculation unit 132 from calculating the wrong three-dimensional coordinates of the hand center due to the wrong depth value when the position of the hand center after the movement is outside the upper limb region.

Further, the flowchart of FIG. 5 is only an example of the consistency determination process performed by the hand position detection device 1 (consistency determination unit 131) of the present embodiment. The consistency determination process performed by the hand position detection device 1 of the present embodiment can be appropriately changed as long as it does not deviate from the gist of the present embodiment. The hand position detection device 1 of the present embodiment may perform, for example, a process according to the flowcharts of FIGS. 17A and 17B as the consistency determination process.

FIG. 17A is a flowchart (No. 1) for explaining a modified example of the consistency determination process. FIG. 17B is a flowchart (No. 2) for explaining a modified example of the consistency determination process. Of the processing steps in the flowcharts of FIGS. 17A and 17B, the processing steps having the same processing contents as the processing steps in the flowchart of FIG. 5 have the same reference numerals as the processing steps of the flowchart of FIG. 5 (for example, S60302 and the like). ) Is attached.

The consistency determination process according to the flowcharts of FIGS. 17A and 17B is performed by the consistency determination unit 131 of the correction unit 130 in the hand position detection device 1. First, as shown in FIG. 17A, the consistency determination unit 131 sets the variable i for identifying the hand center and each joint to “0”, and sets the consistency parameter Ci for the hand center or joint specified by the variable i. The value is set to "1.0" (step S60321).

Next, the consistency determination unit 131 calculates the coordinates (xi, yi) and the distance zi in the depth image from the three-dimensional coordinates of the point Pi indicating the center of the hand or the joint (step S60302). After that, the consistency determination unit 131 reads out the depth value zi'of the coordinate coordinates (xi, yi) from the depth image (step S60303), and determines whether or not zi'> zi + M (step S60304). When zi'> zi + M (step S60304; YES), the consistency determination unit 131 subtracts 0.3 from the value of the consistency parameter Ci (step S60322).

On the other hand, when zi'≤ zi + M (step S60304; NO), the consistency determination unit 131 does not change the value of the consistency parameter Ci, and then sets the three-dimensional distance len between the point Pi and the point Pi + 1. Calculate (step S60305). After that, the consistency determination unit 131 calculates the difference ΔL between the three-dimensional distance len and the dimension of the subject (step S60306), and determines whether or not | ΔL |> THL (step S60307). When | ΔL |> THL (step S60307; YES), the consistency determination unit 131 subtracts 0.3 from the value of the consistency parameter Ci (step S60322).

On the other hand, when | ΔL | ≦ THL (step S60307; NO), the consistency determination unit 131 next calculates the moving speed Si of the point Pi (step S60308), and whether or not Si> THS. (Step S60309). Further, even when the processing of step S60322 is performed, the consistency determination unit 131 calculates the moving speed Si of the point Pi (step S60308) after the processing of step S60322, and determines whether or not Si> THS. Determination (step S60309). When Si> THS (step S60309; YES), the consistency determination unit 131 subtracts 0.2 from the value of the consistency parameter Ci (step S60323), and the subtracted consistency parameter Ci is matched with the point Pi. Determine the sex parameter. After that, as shown in FIG. 17B, the consistency determination unit 131 determines the presence / absence of a hand center or a joint for which the value of the consistency parameter Ci has not been determined (step S60310). On the other hand, when Si ≦ THS (step S60309; NO), the consistency determination unit 131 next determines in step S60310 without changing the value of the consistency parameter Ci. That is, when Si ≦ THS, the consistency determination unit 131 determines the consistency parameter Ci at the time of determination in step S60309 as the consistency parameter of the point Pi.

When there is a hand center or a joint for which the consistency parameter Ci has not been determined (step S60310; YES), the consistency determination unit 131 updates the variable i to i + 1 and sets the consistency parameter Ci to “1.0”. (Step S60324). After that, the consistency determination unit 131 performs the processes after step S60302. Then, when all the consistency parameters Ci of the hand center and the joint currently being processed are determined (step S60310; NO), the consistency determination unit 131 ends the consistency determination process.

When the values of the consistency parameter Ci of the hand center and each joint are determined according to the flowcharts of FIGS. 17A and 17B, the consistency determination unit 131 is based on whether or not Ci <1.0 in step S604. To determine the presence or absence of inconsistent hand centers or joints. In the flowcharts of FIGS. 17A and 17B, the value of the consistency parameter Ci is subtracted when they are not spatially matched and when they are not matched temporally. Therefore, in step S604, when all of the hand centers and joints to be determined are Ci = 1.0, the consistency determination unit 131 matches all of the hand centers and joints ( Step S604; NO) is determined.

As described above, the consistency parameter Ci used for determining the spatial and temporal consistency of the three-dimensional coordinates of the center of the hand and the joint is not limited to the binary values of "True" and "False", and may be quantified. .. Further, when the consistency parameter Ci is quantified, the combination of the values to be subtracted in steps S60322 and S60323 can be appropriately changed.

Further, the consistency determination process performed by the consistency determination unit 131 may only determine the consistency of either the spatial consistency or the time consistency of the three-dimensional coordinates of the hand center and the joint. Further, in the consistency determination process, the consistency determination unit 131 uses the three-dimensional coordinates of the center of the hand and the joint stored in the coordinate history 193 instead of the dimensions of the upper limbs of the subject registered in the dimension information 192. The spatial consistency may be determined.

Further, when calculating the three-dimensional coordinates of the center of the hand and each joint in the hand position detection device 1 of the present embodiment, the depth image 1101 and the background image 191 captured by the depth sensor 2 are not limited to other distance measurements. Information on the distance measured by the device may be used.

[Second Embodiment]
FIG. 18 is a diagram showing a functional configuration of the hand position detection device according to the second embodiment.

As shown in FIG. 18, the hand position detection device 1 of the present embodiment includes a depth image acquisition unit 110, a joint position calculation unit 120, a correction unit 130, an output unit 140, and a storage unit 190.

The depth image acquisition unit 110 acquires a depth image from the depth sensor 2 including information indicating the distance (depth) to the surface of the object existing in the detection region for detecting the position of the center of the hand.

The joint position calculation unit 120 calculates three-dimensional coordinates indicating the positions of the joints of the center of the hand and the upper limbs existing in the detection area in the detection area based on the depth image. The joint position calculation unit 120 in the hand position detection device 1 of the present embodiment includes an upper limb region detection unit 121, an in-image position detection unit 122, and a three-dimensional coordinate calculation unit 123. Further, the joint position calculation unit 120 in the hand position detection device 1 of the present embodiment includes the body area detection unit 124.

The body region detection unit 124 is a body region showing the body portion (trunk portion) of the subject in the depth image based on the depth image for detecting the position of the center of the hand and the background image 191 stored in the storage unit 190. Is detected. The upper limb region detection unit 121 determines a region (upper limb region) corresponding to the human upper limb portion in the depth image based on the depth image for detecting the position of the center of the hand and the background image 191 stored in the storage unit 190. To detect.

The in-image position detection unit 122 detects the position of the center of the hand and the position of the joint of the upper limb in the depth image based on the position and shape of the body region in the depth image and the position and shape of the upper limb region. The in-image position detection unit 122 estimates the position of the shoulder in the depth image from, for example, the position and shape of the body region. Further, the position detection unit 122 in the image detects the positions of the wrist, elbow, and shoulder based on the position of the center of the hand detected in the upper limb region and the estimated position of the shoulder.

The three-dimensional coordinate calculation unit 123 has three-dimensional coordinates indicating the position of the hand center in the detection area and three dimensions indicating the position of each joint based on the positions of the hand center and each joint in the depth image and the depth value. Calculate the coordinates.

The correction unit 130 corrects the three-dimensional coordinates of the hand center and each joint calculated from the depth image. The correction unit 130 in the hand position detection device 1 of the present embodiment includes the consistency determination unit 131 and the recalculation unit 132, as in the correction unit 130 of the first embodiment. The consistency determination unit 131 provides spatial consistency of the three-dimensional coordinates and three-dimensional coordinates based on the three-dimensional coordinates of the center of the hand and each joint calculated from the depth image, and the dimensional information 192 and the coordinate history 193 stored in the storage unit 190. Make a judgment about time consistency. The consistency determination unit 131 determines that the three-dimensional coordinates are corrected when either the hand center or the joint is not spatially or temporally aligned, and the recalculation unit 132 tells the recalculation unit 132 that the hand center and the joint are three-dimensional. Have the coordinates recalculated. First, the recalculation unit 132 changes the position of the center of the hand in the depth image, and calculates the position of each joint in the depth image based on the changed position of the center of the hand. After that, the recalculation unit 132 calculates the three-dimensional coordinates of the center of the hand and each joint.

The output unit 140 outputs information including the three-dimensional coordinates of the center of the hand and each joint included in the coordinate history 193 to the external device 3. The external device 3 is, for example, a server device that manages whether or not the work of the worker is appropriate based on the time change of the three-dimensional coordinates of the center of the hand.

The hand position detection device 1 of the present embodiment sequentially acquires depth images captured by the depth sensor 2 at predetermined time intervals, and detects the center of the hand and the joints of the upper limbs from each of the acquired plurality of depth images. Calculate the three-dimensional coordinates of the center and joint. The hand position detection device 1 of the present embodiment performs coordinate calculation processing according to the flowchart of FIG. 19 as a process of detecting the center of the hand and the joints of the upper limbs from the depth image and calculating the three-dimensional coordinates.

FIG. 19 is a flowchart illustrating the contents of the coordinate calculation process according to the second embodiment. Of the processing steps in the flowchart of FIG. 19, the processing step having the same processing content as the processing step in the flowchart of FIG. 3 has the same reference numeral as the processing step of the flowchart of FIG. 3 (for example, S1 and S2, etc.). Is attached.

As shown in FIG. 19, the hand position detection device 1 of the present embodiment first selects a depth image to be processed (step S1), and detects a body region from the selected depth image (step S12). The processing of steps S1 and S12 is performed, for example, by the body region detection unit 124 of the joint position calculation unit 120 in the hand position detection device 1. The body region detection unit 124 selects, for example, the depth image having the earliest imaging time among the depth images that have not been processed to detect the body region. After selecting the depth image, the body region detection unit 124 determines the body region in the depth image based on the depth value of each point (each pixel) in the selected depth image and the depth value of each point in the background image 191. To detect. In the process of step S12 for detecting the body region, the body region detection unit 124 first detects a body region candidate from within the body region detection area by, for example, a background subtraction method for the body region detection area in the depth image. do. After that, the body region detection unit 124 corrects the contour of the body region and extracts the body region based on the distribution of the depth value in the contour portion of the candidate body region.

Next, the hand position detection device 1 detects the upper limb region from the selected depth image (step S2). The process of step S2 is performed by the upper limb region detection unit 121 of the joint position calculation unit 120 in the hand position detection device 1. The upper limb region detection unit 121 detects the upper limb region from within the upper limb region detection area by, for example, a background subtraction method for the upper limb region detection area in the depth image. At this time, the upper limb region detection unit 121, for example, among a plurality of regions in which the difference from the depth value in the background image 191 is equal to or greater than the threshold value, the upper limb region detection unit 121 determines a region in which the depth value corresponds to the distance from the depth sensor 2 to the upper limb. Detect as upper limb area.

Next, the hand position detection device 1 determines whether or not the upper limb region has been detected from the depth image (step S3). The determination in step S3 is performed by the upper limb region detection unit 121. When the upper limb region is not detected (step S3; NO), the upper limb region detection unit 121 stores, for example, information indicating that the upper limb region was not detected from the selected depth image in the coordinate history 193 (step). S11), the coordinate calculation process for the depth image is terminated. On the other hand, when the upper limb region is detected from the depth image (step S3; YES), the hand position detection device 1 performs the processes of steps S14, S15, S6, and S7.

When the upper limb region is detected from the depth image (step S3; YES), the hand position detection device 1 then detects the position of the center of the hand in the upper limb region in the depth image (step S14). The process of step S14 is performed by the position detection unit 122 in the image of the joint position calculation unit 120 in the hand position detection device 1. The in-image position detection unit 122 detects the position of the center of the hand in the upper limb region in the depth image, for example, based on the shape and position of the upper limb region. The position detection unit 122 in the image reduces the dimension in the direction orthogonal to the extension direction of the upper limb in the upper limb region, based on the direction that can be the extension direction of the upper limb (the direction of the line segment connecting the joints) in the depth image, for example. , Extract a line segment indicating the extension direction of the upper limb. After extracting the line segment, the position detection unit 122 in the image detects the position of the center of the hand from each point on the line segment. The in-image position detection unit 122 detects the position of the center of the hand on the line segment, for example, based on the information indicating the area corresponding to the size of the hand (palm) in the depth image.

After detecting the position of the center of the hand, the hand position detection device 1 calculates the three-dimensional coordinates of the center of the hand and the joints of the upper limbs (step S15). The process of step S15 is performed by the three-dimensional coordinate calculation unit 123 of the joint position calculation unit 120 in the hand position detection device 1. The three-dimensional coordinate calculation unit 123 applies a pinhole camera model and calculates the three-dimensional coordinates of the center of the hand and the joint for each upper limb region detected from the depth image. In the process of calculating the three-dimensional coordinates of the hand center and the joint corresponding to one upper limb region, the three-dimensional coordinate calculation unit 123 first calculates the three-dimensional coordinates of the hand center. Next, the three-dimensional coordinate calculation unit 123 estimates the position of the shoulder corresponding to the upper limb region based on the shape and position of the body region detected from the depth image. After that, the three-dimensional coordinate calculation unit 123 sequentially calculates the three-dimensional coordinates of the wrist, the three-dimensional coordinates of the elbow, and the three-dimensional coordinates of the shoulder based on the three-dimensional coordinates of the center of the hand and the estimated shoulder position. .. When calculating the three-dimensional coordinates of the wrist, elbow, and shoulder, the three-dimensional coordinate calculation unit 123 determines and determines the extension direction of the upper limb based on, for example, the position of the center of the hand and the estimated position of the shoulder. The three-dimensional positions of the wrist, elbow, and shoulder are detected on the line indicating the extension direction of the upper limb, and the three-dimensional coordinates of each are calculated.

After calculating the three-dimensional coordinates of the center of the hand and the joint, the hand position detection device 1 then performs the coordinate correction process (step S6). The coordinate correction process is performed by the correction unit 130 of the hand position detection device 1. As described in the first embodiment, the correction unit 130 first determines the spatial consistency and the time consistency of the current three-dimensional coordinates of the center of the hand and the joint for each upper limb region. Then, when all the three-dimensional coordinates of the hand center and the joint corresponding to one upper limb region are spatially and temporally aligned, the correction unit 130 measures the three-dimensional coordinates of the hand center and the joint corresponding to the upper limb region. It is determined that correction is unnecessary. In this case, the correction unit 130 determines the current three-dimensional coordinates of the center of the hand and the joint corresponding to the upper limb region to be determined as the three-dimensional coordinates to be stored in the coordinate history, and ends the coordinate correction process. On the other hand, when the three-dimensional coordinates of either the center of the hand or the joint do not match spatially or temporally, the correction unit 130 determines that the three-dimensional coordinates of the center of the hand and the joint are corrected. In this case, the correction unit 130 first corrects the three-dimensional coordinates of the center of the hand, and then corrects the three-dimensional coordinates of the wrist, elbow, and shoulder based on the three-dimensional coordinates of the center of the hand after correction. Further, when the three-dimensional coordinates are corrected, the correction unit 130 determines the consistency of the corrected three-dimensional coordinates. After that, the correction unit 130 repeats the correction of the three-dimensional coordinates until it is determined that they match or the number of corrections of the three-dimensional coordinates reaches the upper limit value. When it is determined that the corrected 3D coordinates are spatially and temporally consistent, or when the number of corrections of the 3D coordinates reaches the upper limit, the correction unit 130 uses the current 3D coordinates as the coordinate history. The three-dimensional coordinates to be stored in 193 are determined, and the coordinate correction process is completed.

When the coordinate correction process is completed, the hand position detection device 1 stores the three-dimensional coordinates of the hand center and the joint in the coordinate history 193 (step S7), and ends the coordinate calculation process for the depth image currently being processed. do.

As described above, the hand position detecting device 1 of the present embodiment detects the body region indicating the body portion of the subject from the depth image, and utilizes the position of the shoulder of the subject estimated based on the body region. Calculate the position (three-dimensional coordinates) of the joints of the upper limbs of the subject.

FIG. 20 is a flowchart illustrating the content of the process of detecting the body region from the depth image.

The body region detection unit 124 performs the process of detecting the body region from the depth image (step S12). As shown in FIG. 20, the body region detection unit 124 first binarizes the body region detection area in the depth image based on the depth image and the background image (step S1201). In the process of step S1201, the body region detection unit 124 divides the depth image for detecting the center of the hand and the joint into the upper limb region detection area and the body region detection area, and detects the body region by the background subtraction method for the body region detection area. Binarize the area. Here, the upper limb region detection area is a partial region in the imaging range of the depth image in which the upper limb (particularly the hand portion) of the subject moves. On the other hand, the body area detection area is an area in the imaging range of the depth image in which the body portion of the subject exists. The boundary between the upper limb area detection area and the body area detection area in the depth image is set in advance based on, for example, the imaging range of the depth image and the standing position of the subject within the imaging range. The body area detection unit 124 calculates the difference between the depth value in the depth image and the depth value of the corresponding pixel in the background image for each pixel included in the body area detection area of the depth image, and the calculated difference and the threshold value. The depth value is binarized based on the magnitude relationship.

Next, the body region detection unit 124 selects, in the binarized body region detection area, the maximum region among the continuous regions where the difference in depth value is equal to or greater than the threshold value as the body region (step S1202).

After that, the body region detection unit 124 corrects the body region based on the depth value in the contour portion of the selected body region (step S1203), and ends the process of detecting the body region from the depth image. In the process of step S1203, the body region detection unit 124 determines the contour of the body region based on the difference in depth value between the pixels in the adjacent body region and the pixels outside the body region with the contour of the body region as the boundary in the depth image. Correct the position of. For example, when the difference in depth value between the pixel inside the body region and the pixel outside the body region is within a predetermined range, the body region detection unit 124 widens the contour so that the pixel outside the body region is included in the body region. Make corrections. At this time, the body region detection unit 124 allows the contour of the body region to project to the upper limb region detection area.

After the process of detecting the body region, the hand position detection device 1 performs a process of detecting the upper limb region from the depth image (step S2). The process of detecting the upper limb region is performed by the upper limb region detection unit 121 of the joint position calculation unit 120 in the hand position detection device 1. The upper limb area detection unit 121 performs, for example, a process according to the flowchart of FIG. 21 as a process of detecting the upper limb area from the depth image.

FIG. 21 is a flowchart illustrating the content of the process of detecting the upper limb region from the depth image.

In the process of detecting the upper limb region, the upper limb region detection unit 121 first, as shown in FIG. 21, first, based on the depth image and the background image, the depth of each point (each pixel) in the upper limb region detection area in the depth image. The difference between the values is calculated (step S201).

Next, the upper limb region detection unit 121 divides the upper limb region detection area into a plurality of regions based on the difference in the calculated depth values (step S202). In the process of step S202, the upper limb region detection unit 121 divides the difference in depth value into a plurality of sections, and based on which section of the plurality of sections the difference in the depth value of each pixel is included. The upper limb area detection area is divided into multiple areas. At this time, the difference in depth value is, for example, the difference in depth value between the object existing in the background image and the upper limb of the subject, and the difference in depth value between the object existing in the background image and the head of the subject. Divide into different sections.

After that, the upper limb region detection unit 121 extracts a region in which the difference in depth value is within a predetermined range as the upper limb region (step S203), and ends the process of detecting the upper limb region from the depth image. In step S203, the upper limb area detection unit 121 has an object having a difference in depth value in the background image and the upper limb of the subject among a plurality of areas in the upper limb area detection area divided based on the difference in depth value. The area included in the section corresponding to the difference in the depth value of is extracted as the upper limb area.

After detecting the upper limb region from the depth image, the hand position detection device 1 detects the position of the center of the hand for each upper limb region (step S14). The process of step S14 is performed by the position detection unit 122 in the image. The position detection unit 122 in the image detects the position of the center of the hand for each upper limb region by the method as described in the first embodiment. If the upper limb region cannot be converted into a line segment group to detect the position of the center of the hand, the position detection unit 122 in the image detects an arbitrary point in the upper limb region as the position of the center of the hand.

After detecting the position of the center of the hand, the hand position detection device 1 calculates the three-dimensional coordinates of the center of the hand and the joint for each upper limb region (step S15). The process of step S15 is performed by the three-dimensional coordinate calculation unit 122. The three-dimensional coordinate calculation unit 122 in the hand position detection device 1 of the present embodiment performs, for example, a process according to the flowchart of FIG. 21 as a process of calculating the three-dimensional coordinates of the center of the hand and the joint.

FIG. 22 is a flowchart illustrating the contents of the process of calculating the three-dimensional coordinates of the center of the hand and the joint.

As shown in FIG. 22, the three-dimensional coordinate calculation unit 122 according to the present embodiment first calculates the three-dimensional coordinates of the weight center based on the body region detected from the depth image (step S1501), and the shoulder in the depth image. Estimate the position of (step S1502). In the process of step S1501, the three-dimensional coordinate calculation unit 122 first calculates the position of the body weight center in the body region detected from the depth image. After that, the three-dimensional coordinate calculation unit 122 applies the pinhole camera model, and based on the position (coordinates) of the weight center in the depth image and the depth value of the position, the weight center in the three-dimensional space where the depth image is imaged. Calculate the three-dimensional coordinates of. At this time, the three-dimensional coordinate calculation unit 122 calculates the three-dimensional coordinates in the world coordinate system.

Further, in the process of step S1502, the three-dimensional coordinate calculation unit 122 first calculates the approximate height and shoulder width of both shoulders in the three-dimensional space in which the depth image is captured from the predetermined height value. Here, the shoulder width is calculated based on, for example, the Witruwiwis-like human body proportions. After that, the three-dimensional coordinate calculation unit 122 calculates the positions of both shoulders so that the lateral center in the depth image becomes the center of the shoulder width, for example.

Next, the three-dimensional coordinate calculation unit 122 selects one upper limb region and calculates the three-dimensional coordinates of the hand center detected from the upper limb region (step S1503). In the process of step S1503, the three-dimensional coordinate calculation unit 122 selects the upper limb region for which the three-dimensional coordinates of the hand center are not calculated, among the upper limb regions detected from the depth image. Further, in the process of step S1503, the three-dimensional coordinate calculation unit 122 applies the pinhole camera model, and is based on the position (coordinates) of the point P0 indicating the center of the hand in the depth image and the depth value of the position. , Calculate the three-dimensional coordinates of the center of the hand in the three-dimensional space where the depth image is captured.

Next, the three-dimensional coordinate calculation unit 122 detects the position of the wrist in the depth image and calculates the three-dimensional coordinates of the wrist (step S1504). In the process of step S1504, the three-dimensional coordinate calculation unit 122 uses, for example, the depth image based on the position of the center of the hand detected from the line segment group indicating the extension direction of the upper limb extracted from the depth image and the bending position of the line segment. Detects the position of the wrist in. If the position of the wrist cannot be detected using the line segment, such as when the line segment group cannot be extracted from the upper limb region, the three-dimensional coordinate calculation unit 122 may, for example, measure the dimension from the center of the hand to the wrist. The position of the wrist in the depth image is detected based on the average value (for example, 60 mm). In this case, the three-dimensional coordinate calculation unit 122 is, for example, on a line segment connecting the position of the center of the hand and the estimated position of the shoulder in the depth image, and the three-dimensional distance from the center of the hand is the dimension from the center of the hand to the wrist. The position that becomes the average value of is detected as the position of the wrist.

Next, the three-dimensional coordinate calculation unit 122 detects the position of the elbow in the depth image and calculates the three-dimensional coordinates of the elbow (step S1505). In the process of step S1505, the three-dimensional coordinate calculation unit 122 in the depth image is based on, for example, the position of the wrist detected from the line segment group indicating the extension direction of the upper limb extracted from the depth image and the bending position of the line segment. Detect the position of the elbow. If the position of the elbow cannot be detected using the line segment, such as when the line segment group cannot be extracted from the upper limb region, the three-dimensional coordinate calculation unit 122 may, for example, average the dimensions from the wrist to the elbow. The position of the elbow in the depth image is detected based on the value (eg 260 mm). In this case, the three-dimensional coordinate calculation unit 122 is, for example, on a line segment connecting the position of the wrist and the estimated position of the shoulder in the depth image, and the three-dimensional distance from the wrist is the average value of the dimensions from the wrist to the elbow. The position where becomes is detected as the position of the elbow.

Next, the three-dimensional coordinate calculation unit 122 detects the position of the shoulder in the depth image and calculates the three-dimensional coordinates of the shoulder (step S1506). In the process of step S1505, the three-dimensional coordinate calculation unit 122 determines the depth based on, for example, the position of the elbow detected from the line segment group indicating the extension direction of the upper limb extracted from the depth image, and the end point or the bending position of the line segment. Detects the position of the shoulder in the image. If the shoulder position cannot be detected using the line segment, such as when the line segment group cannot be extracted from the upper limb region, the three-dimensional coordinate calculation unit 122 may use, for example, the estimated shoulder position and elbow. The position of the shoulder in the depth image is detected based on the average value of the dimensions from to the shoulder. In this case, the three-dimensional coordinate calculation unit 122 is, for example, on a line segment connecting the elbow position and the estimated shoulder position in the depth image, and the three-dimensional distance from the wrist is the average value of the dimensions from the elbow to the elbow. The position where becomes is detected as the position of the elbow.

After calculating the three-dimensional coordinates of the wrist, elbow, and shoulder in steps S1504 to S1506, the three-dimensional coordinate calculation unit 122 determines whether or not there is an unselected upper limb region (step S1507). In the determination of step S1507, the three-dimensional coordinate calculation unit 122 determines that there is an unselected upper limb region when there is an upper limb region for which the three-dimensional coordinates of the center of the hand and each joint have not been calculated. When there is an unselected upper limb region (step S1507; YES), the three-dimensional coordinate calculation unit 122 repeats the processes after step S1503. Then, when the three-dimensional coordinates of the hand center and each joint are calculated in all the upper limb regions (step S1507; NO), the three-dimensional coordinate calculation unit 122 ends the process of calculating the three-dimensional coordinates of the hand center and the joint. ..

When the process of calculating the three-dimensional coordinates of the center of the hand and the joint (step S15) is completed, the hand position detection device 1 then performs the coordinate correction process (step S6). The coordinate correction process is performed by the correction unit 130 of the hand position detection device 1. The correction unit 130 in the hand position detection device 1 of the present embodiment performs, for example, a process according to the flowchart of FIG. 4 as the coordinate correction process. The correction unit 130 according to the present embodiment performs, for example, a process according to the flowchart of FIG. 23 as the consistency determination process (step S603) in the flowchart of FIG.

FIG. 23 is a flowchart illustrating the content of the consistency determination process in the second embodiment.

The consistency determination process (step S603) of FIG. 23 is performed by the consistency determination unit 131 of the correction unit 130. As shown in FIG. 23, the consistency determination unit 131 in the hand position detection device 1 of the present embodiment first sets the variable i for identifying the hand center and each joint to “0” and sets the consistency parameter Ci to “True”. (Step S60301). Here, the variable i is a numerical value (0, 1, 2, ...) Attached to identify each of the hand center and each joint when the position of the hand center and each joint is detected from one upper limb region.・). The consistency parameter Ci is a value indicating whether or not the three-dimensional coordinates of the point Pi indicating the hand center or joint specified by the variable i are spatially and temporally matched. In the present embodiment, the consistency parameter Ci is set to one of two values, "True" and "False". "True" is a value indicating that the three-dimensional coordinates are spatially and temporally aligned. "False" is a value indicating that the three-dimensional coordinates are not spatially or temporally aligned.

Next, the consistency determination unit 131 calculates the coordinates (xi, yi) and the distance zi in the depth image from the three-dimensional coordinates (Xi, Yi, Zi) of the point Pi indicating the center of the hand or the joint (step). S60302). In step S60302, the consistency determination unit 131 back-calculates based on the calculation formula when calculating the three-dimensional coordinates (Xi, Yi, Zi) corresponding to the position of the point Pi indicating the center of the hand or the joint in the depth image (Xi, Yi, Zi). The coordinates (xi, yi) and the distance zi in the depth image are calculated by the inverse transformation).

Next, the consistency determination unit 131 reads out the depth value zi'of the coordinates (xi, yi) in the depth image (step S60303), and determines whether or not zi'> zi + M (step S60304). When zi'> zi + M (step S60304; YES), the consistency determination unit 131 changes the consistency parameter Ci to "False" (step S60312).

On the other hand, when zi'≤ zi + M (step S60304; NO), the consistency determination unit 131 next calculates the moving speed Si of the point Pi (step S60308). In the process of step S60308, the consistency determination unit 131 first reads the past three-dimensional coordinates of the point Pi with reference to the coordinate history 193, and the three-dimensional distance between the current three-dimensional coordinates and the past three-dimensional coordinates. Calculate Di. After calculating the three-dimensional distance Di, the consistency determination unit 131 calculates the moving speed Si based on the calculated three-dimensional distance Di and the difference in the imaging time of the depth image.

Next, the consistency determination unit 131 determines whether or not the calculated movement speed Si is larger than the threshold value THS (step S60309). The threshold THS is set based on, for example, the center of the hand when a person moves the upper limbs and the average moving speed of each joint. That is, when the calculated movement speed Si is Si> THS, the consistency determination unit 131 determines that the positional relationship between the two three-dimensional coordinates used for calculating the movement speed Si does not match in time. do. Therefore, when Si> THS (step S60309; YES), the consistency determination unit 131 changes the consistency parameter Ci to “False” (step S60312).

When the processing of step S60312 is performed, or when Si ≦ THS (step S60309; NO), the consistency determination unit 131 is then undetermined among the points indicating the center of the hand and the joint currently being processed. It is determined whether or not there is a point (step S60310). That is, at the time of determining in step S60310, the consistency parameter Ci for the point Pi indicating the center of the hand or the joint currently being processed is determined to be either “True” or “False”.

When there is a point where the consistency is not determined (step S60310; YES), the consistency determination unit 131 updates the variable i to i + 1 and sets the consistency parameter Ci to “True” (step S60311). After that, the consistency determination unit 131 performs the processes after step S60302. Then, when the consistency of all the hand centers and joints is determined (step S60310; NO), the consistency determination unit 131 ends the consistency determination process.

As described above, in the consistency determination process of FIG. 23, the processes from step S60305 to step S60307 in the consistency determination process of FIG. 5 are omitted. That is, in the consistency determination process performed by the consistency determination unit 131 of the present embodiment, the distance zi calculated from the three-dimensional coordinates of the point Pi indicating the center of the hand or the joint does not match the depth value zi'in the depth image. , It is determined that the three-dimensional coordinates of the point Pi are spatially inconsistent. Therefore, in the hand position detection device 1 of the present embodiment, the number of processes is reduced as compared with the case where the three-dimensional distance len is calculated to determine the consistency with the dimensions of the target person, and the process of the hand position detection device 1 is performed. It is possible to reduce the load. Of course, as the consistency determination process in the hand position detection device 1 of the present embodiment, the consistency determination unit 131 may perform the process according to the flowchart of FIG. Further, the consistency determination unit 131 may perform a process according to the flowcharts of FIGS. 17A and 17B as the consistency determination process.

FIG. 24 is a diagram illustrating an imaging range of a depth image.
In FIG. 24, as an application example of the hand position detection device 1 of the present embodiment, a management system 5 that manages a work situation based on a time history of a position centered on the hand of a worker 7 who works in the work space 6 is shown. Shows. The management system 5 includes a hand position detection device 1, a depth sensor 2, and a management device 3A. The hand position detection device 1 and the depth sensor 2 in the management system 5 each include a communication function that can be connected to a communication network 4 such as the Internet. Further, the management device 3A in the management system 5 is an example of the external device 3 in FIG. The x-direction, y-direction, and z-direction shown in FIG. 24 indicate the x-axis direction, the y-axis direction, and the z-axis direction of the coordinate system of the depth sensor 2 in the work space 6, respectively.

In this type of management system 5, for example, the manager of the management system 5, the worker 7, or the like creates dimensional information 192 in which the dimensions of the upper limbs of the worker 7 are input in advance, and the dimensional information 192 is detected by hand. It is stored in the storage unit 190 of the device 1. Further, in this type of management system 5, for example, a depth image is captured by a depth sensor 2 installed above the work space 6 while the worker 7 is not working, and the depth image is used as a background image 191. It is stored in the storage unit 190. The depth sensor 2 captures a depth image having depth values (distance information) for U × V points within the imaging range 10 including the work area on the workbench 8A on which the worker 7 works.

Further, in the management system 5 illustrated in the present embodiment, as shown in FIG. 24, the imaging range 10 of the depth sensor 2 projects toward the worker 7 side from the work area. Therefore, the depth image captured by the depth sensor 2 while the worker 7 is working includes information on the depth values of the head 750 and the body 760 of the worker 7.

FIG. 25 is a diagram showing an example of a depth image.
The depth image 1101 of FIG. 25 is a depth image captured by the depth sensor 2 while the worker 7 is working in the work area 6 of FIG. 24. The depth image 1101 contains information on the depth value for each of the U × V points in the imaging range 10.

The imaging range 10 of the depth image 1101 includes a working area 8A and an area in which an adjacent worker 7 stands. Therefore, the depth image 1101 includes a region showing the workbench 8A, a region showing the upper left limb 710 of the worker 7, a region showing the upper right limb 720, and a region showing the depth value of the head 750. Includes a portion indicating the depth value of the body portion 760 of the worker 7. The depth sensor 2 that captures the depth image 1101 is installed above the worker 7. Therefore, the region showing the upper left limb 710 of the worker 7 and the region showing the upper right limb 720 in the depth image 1101 may be partially missing due to the head 750 and the body portion 760 of the worker 7, respectively. be. In the depth image 1101 of FIG. 25, the portion of the hand 711 of the upper left limb 710 of the worker 7 is hidden in the head portion 750, and the portion of the elbow 713 is hidden in the body portion 760. Similarly, in the depth image 1101 of FIG. 25, the portion of the hand 721 of the upper right limb 720 of the worker 7 is hidden in the head portion 750, and the portion of the elbow 713 is hidden in the body portion 760.

As described above, the hand position detection device 1 of the present embodiment first performs a process of detecting a body region from a depth image to be processed (step S12) and a process of detecting an upper limb region (step S2). ..

FIG. 26A is a diagram (No. 1) illustrating a method for detecting a body region. FIG. 26B is a diagram (No. 2) illustrating a method for detecting a body region.

FIG. 26A (a) shows the body region detection area 1105 and the upper limb region detection area 1106 in the depth image 1101. The body area detection area 1105 is an area in which the worker 7 stands, which is adjacent to the work area in the work table 8A. Therefore, in the following description, the position at the lower end of the workbench 8A in the background image is defined as the boundary B, and the partial region below the boundary B in the depth image 1101 is defined as the body region detection area 1105. That is, the body region detection unit 124 first generates a binary image based on the depth value of each position (pixel) included in the body region detection area 1105 below the boundary B in the depth image 1101.

Although not shown, the background image used by the body region detection unit 124 to generate the binary image is only the region showing the depth value of the workbench 8A above the boundary B including the boundary B in the depth image 1101. Exists, and there is only a region indicating the depth value of the floor surface below the boundary B. Therefore, the binary image generated based on the depth value of the body region detection area 1105 is, for example, the binary image 1102 of FIG. 26A (b). In the binary image of FIG. 26A (b), the depth value of each position (pixel) in the upper limb region detection area 1106 is matched with the depth value outside the body region 1102C in the body region detection area 1105.

That is, the body region 1102C in the binary image 1102 generated based on the difference in depth values in the body region detection area 1105 does not include the body portion of the worker 7 existing in the upper limb region detection area 1106. Therefore, it cannot be said that the body region 1102C correctly indicates the body portion of the worker 7. Therefore, the body region detection unit 124 next corrects the body region 1102C based on the depth value in the contour portion of the body region 1102C.

Comparing the depth image 1101 of FIG. 26A and the binary image 1102 of FIG. 26A, the depth value in the body region 1102C and the body region 1102C are located in the portion of the contour of the body region 1102C that overlaps the boundary B. There is a section where the difference from the outside depth value is approximately 0. Therefore, the body region detection unit 124 determines the portion of the contour of the body region 1102C where the difference between the depth value inside the body region 1102C and the depth value outside the body region 1102C is substantially 0, in the upper limb region detection area 1106. Spread to the side. As a result, the body region detection unit 124 detects the body region 1102D corresponding to the region showing the depth value of the body portion 760 in the depth image 1101, as in the binary image 1102 of FIG. 26B, for example.

After detecting the body region 1102D from the depth image 1101, the upper limb region detection unit 121 detects the upper limb region from the depth image 1101 in the hand position detection device 1. The upper limb region detection unit 121 detects the upper limb region from within the upper limb region detection area 1106 in the depth image 1101 by the background subtraction method.

FIG. 27 is a diagram showing an example of the detection result of the upper limb region.
The binary image 1102 of FIG. 27 shows a binary image generated based on the depth image 1101 of FIG. 25 and the background image 191 (not shown). The depth image 1101 of FIG. 25 includes a region showing depth values for each part of the worker 7's head 750, body portion 760, upper left limb 710, and upper right limb 720, which are not present in the background image 191. Therefore, in the binary image 1102 generated based on the depth image 1101 and the background image 191 of FIG. 25, the first upper limb region 1102A, the second upper limb region 1102B, the body region 1102D, and the head region 1102E are included. , Isolated from the background. Therefore, the upper limb region detection unit 121 next extracts the upper limb regions 1102A and 1102B from the four regions 1102A, 1102B, 1102D, and 1102E separated from the background.

Of the four regions 1102A, 1102B, 1102D, and 1102E separated from the background, the upper limb regions 1102A and 1102B and the body regions 1102D and the head region 1102E have the depth calculated when the binary image 1102 is generated. The magnitude of the difference in values is different. The upper limbs 710 and 720 of the worker 7 corresponding to the upper limb areas 1102A and 1102B are located closer to the workbench 8A than the torso portion 760 corresponding to the body area 1102D and the head 750 corresponding to the head area 1102E. Therefore, the difference in depth value between the pixels in the upper limb region 1102A and 1102B is smaller than the difference in depth value between the pixels in the body region 1102D and the head region 1102E. Therefore, the upper limb region detection unit 121 is a small region among the four regions 1102A, 1102B, 1102D, and 1102E separated from the background, in which the difference in depth value is between the first threshold value and the second threshold value. 1102A and 1102B are detected as the upper limb region. In this case, in the binary image 1102, the upper limb region detection unit 121 considers the regions 1102D and 1102E in which the difference between the depth values is larger than the second threshold value as the background, as shown in FIG. 27. Therefore, when the position detection unit 122 in the image detects the position of the center of the hand in the depth image 1101, the white background region (first upper limb region 1102A and second upper limb region) in the binary image 1102 of FIG. 27 The position of the center of the hand is detected from each of 1102B).

FIG. 28 is a diagram showing an example of the detection result of the position of the center of the hand.
The depth image 1101 of FIG. 28 shows the first upper limb region 1101A and the second upper limb region 1101B detected based on the binary image 1102 of FIG. 27, and the hand center detected from the respective upper limb regions 1101A and 1101B. The position of P0 is shown.

As described in the first embodiment, the position detection unit 122 in the image linearly differentiates the upper limb region and converts it into a line segment group indicating the extension direction of the upper limb, and determines the position of the center of the hand on the line segment of the line segment group. To detect. Therefore, the in-image position detection unit 122 detects the position Q10 in the first upper limb region 1101A as the position of the point P0 indicating the hand center in the first upper limb region 1101A, as shown in the depth image 1101 of FIG. The position Q20 in the upper limb region 1102B is detected as the position of the point P0 indicating the center of the hand.

After the position detection unit 122 in the image detects the position of the point P0 indicating the center of the hand, in the hand position detection device 1 of the present embodiment, the three-dimensional coordinate calculation unit 123 calculates the three-dimensional coordinates of the center of the hand and the joints of the upper limbs. Processing (step S15) is performed.

FIG. 29 is a diagram illustrating a method of determining the position of the joint of the left upper limb. FIG. 30 is a diagram showing the determined positions of the center of the hand and the joint.

The three-dimensional coordinate calculation unit 123 determines the position of the joint for each upper limb region detected from the depth image 1101 (binary image 1102). The depth image 1101 of FIG. 29A shows an example of the position Q11 of the point P1 indicating the wrist when the processing target is the first upper limb region 1101A. At the time of imaging the depth image 1101 of FIG. 29 (a), most of the wrist-to-shoulder portion of the upper left limb 710 of the worker 7 is hidden by the body portion 760. Therefore, the first upper limb region 1101A in the depth image 1101 has a short dimension in the direction in which the upper limb can be extended, and the three-dimensional coordinate calculation unit 123 detects the position of the wrist from the line segment group obtained by line-differentiating the upper limb region. I can't. In this case, the three-dimensional coordinate calculation unit 123 is based on the position Q10 of the point P0 indicating the center of the hand detected from the first upper limb region 1101A and the average value (for example, 60 mm) of the dimensions from the center of the hand to the wrist. The position of the point P1 indicating the wrist is determined. At this time, the three-dimensional coordinate calculation unit 123 is, for example, on the line segment LS1 passing through the position Q10 of the hand center in the depth image 1101 and the estimated position QS1 of the shoulder, and the distance from the position Q10 of the hand center is The position Q11 at L1'is the position of the point P1 indicating the wrist. Here, the distance L1'between the position Q10 at the center of the hand and the position Q11 at the wrist is a distance at which the three-dimensional distance substantially coincides with the dimension from the center of the hand to the wrist. That is, when the position of the wrist cannot be detected from the line segment group obtained by line-differentiating the upper limb region in the depth image 1101, the three-dimensional coordinate calculation unit 123 calculates the three-dimensional coordinates of the point P1 indicating the wrist and then the depth image. A process of determining the position of the point P1 in 1101 is performed.

After determining the position of the wrist corresponding to the first upper limb region 1101A, the three-dimensional coordinate calculation unit 123 determines the position of the point P2 indicating the elbow based on the determined position of the wrist. The three-dimensional coordinate calculation unit 123 determines the position of the point P2 indicating the elbow by the same method as when determining the position of the wrist based on the position of the center of the hand. That is, the three-dimensional coordinate calculation unit 123 is on the line segment LS1 passing through the wrist position Q11 as in the depth image 1101 of FIG. 29 (b), and the distance L2'from the position Q11 is from the wrist to the elbow. The position Q12, which is the distance corresponding to the dimension up to, is determined at the position of the point P2 indicating the elbow. Here, the distance L2'between the wrist position Q11 and the elbow position Q12 is a distance whose three-dimensional distance substantially coincides with the dimension from the wrist to the elbow. That is, when the position of the elbow cannot be detected from the line segment group obtained by line-differentiating the upper limb region in the depth image 1101, the three-dimensional coordinate calculation unit 123 calculates the three-dimensional coordinates of the point P2 indicating the elbow and then the depth image. A process of determining the position of the point P2 in 1101 is performed.

After determining the position of the wrist corresponding to the first upper limb region 1101A, the three-dimensional coordinate calculation unit 123 determines the position of the point P3 indicating the left shoulder based on the determined elbow position and the estimated shoulder position. do.

After determining the positions of the wrist, elbow, and shoulder of the upper left limb corresponding to the first upper limb region 1101A and calculating the three-dimensional coordinates of each, the three-dimensional coordinate calculation unit 123 corresponds to the second upper limb region 1102B. Determine the positions of the wrists, elbows, and shoulders of the right upper limb and calculate the three-dimensional coordinates of each. For example, the three-dimensional coordinate calculation unit 123 determines the position Q21 on the line segment LS2 in the depth image 1101 of FIG. 30 at the position of the point P1 indicating the wrist corresponding to the second upper limb region 1101B, and calculates the three-dimensional coordinates. do. Further, the three-dimensional coordinate calculation unit 123 determines, for example, the position Q22 on the line segment LS2 in the depth image 1101 of FIG. 29 to the position of the point P1 indicating the wrist corresponding to the second upper limb region 1101B, and the three-dimensional coordinates. Is calculated.

After calculating the three-dimensional coordinates of the hand center and the joint, the hand position detection device 1 performs the consistency determination process (step S603) in the coordinate correction process (step S6) to correct the three-dimensional coordinates of the hand center and the joint. Judge the necessity. The hand position detection device 1 of the present embodiment performs a process according to the flowchart of FIG. 23 as a consistency determination process. That is, when the relationship between the distance value zi calculated from the three-dimensional coordinates of the point Pi indicating the center of the hand or the joint and the depth value zi'in the depth image 1101 is zi'> zi + M, the hand position detecting device 1 is the point Pi. It is determined that the center of the hand or the joint corresponding to the above is not spatially aligned. In the depth image 1101 of FIG. 30, the point P1 indicating the wrist and the point P2 indicating the elbow corresponding to the first upper limb region 1101A are located outside the first upper limb region 1101A, respectively. However, the position Q11 of the point P1 is inside the body region 1101D, and the position Q12 of the point P2 is outside the body region 1101D. That is, the depth value z1 at the position Q11 is a value indicating the distance from the depth sensor 2 to the body portion, and is smaller than the value indicating the distance from the depth sensor 2 to the upper limbs (wrist). On the other hand, the depth value of the point P2 is a value indicating the distance from the depth sensor 2 to the floor, and is larger than the value indicating the distance from the depth sensor 2 to the upper limb (wrist). Therefore, in the consistency determination process for the center of the hand and the joint corresponding to the first upper limb region 1101A, the consistency determination unit 131 of the hand position detection device 1 spatially has a point P2 indicating the elbow. Determine that they are not consistent. In this case, the consistency determination unit 131 causes the recalculation unit 132 to perform a process of recalculating the three-dimensional coordinates of the hand center and each joint. As described in the first embodiment, the recalculation unit 132 changes the position of the point P0 indicating the hand center in the depth image 1101, and based on the changed position of the point P0, the hand center and each joint. Calculate (recalculate) the three-dimensional coordinates of.

FIG. 31 is a diagram illustrating a method of correcting the positions of the center of the hand and the joint.
The depth image 1101 of FIG. 31 shows an example of correcting the positions of the center of the hand and the elbow corresponding to the first upper limb region 1101A.

When the point P1 indicating the wrist and the point P2 indicating the elbow corresponding to the first upper limb region 1101A are not spatially aligned, the recalculation unit 132 first indicates the point P0 indicating the center of the hand in the depth image 1101. Change the position of. At this time, the recalculation unit 132 is in the direction of the vector starting from the wrist position Q11 and ending at the hand center position Q10 before the change, as shown in FIG. 30, from the hand center position Q10. The position of the point P0 indicating the center of the hand is changed to the position Q10'which is a predetermined distance. In other words, the recalculation unit 132 sets the position of the center of the hand on the line segment LS1 passing through the position Q10 of the center of the hand before the change and the position Q11 of the wrist in the direction away from (away from) the position Q11 of the wrist. change.

After changing the position of the point P0 indicating the center of the hand to the position Q10', the recalculation unit 132 uses the three-dimensional coordinates of the point P0 indicating the center of the hand based on the changed position Q10'(coordinates) in the depth image 1101. Is calculated. At this time, as described in the first embodiment, the recalculation unit 132 is used to calculate the three-dimensional coordinates depending on whether or not the changed position Q10'is within the first upper limb region 1101A. Switch the depth value. When the changed position Q10'is in the first upper limb region 1101A, the recalculation unit 132 calculates the three-dimensional coordinates using the depth value of the position Q10'. On the other hand, when the changed position Q10'is outside the first upper limb region 1101A, the recalculation unit 132 may, for example, based on the distribution of the depth value on the line segment LS1 in the first upper limb region 1101A. Calculate (estimate) the depth value used to calculate the three-dimensional coordinates.

After that, although not shown, the recalculation unit 132 changes the position of the point P1 indicating the wrist based on the changed position Q10'of the hand center, and calculates the three-dimensional coordinates of the changed point P1. .. After that, the recalculation unit 132 further changes the position of the point P2 indicating the elbow from the position Q12 to the position Q12'as shown in FIG. 31, and calculates the three-dimensional coordinates of the changed point P2. Here, assuming that the changed position Q12 of the point P2 indicating the elbow is included in the body region 1101D corresponding to the body portion as shown in FIG. 31, the depth value of the position Q12'in the depth image 1101 is the depth sensor 2. It is smaller than the value indicating the distance from the upper limb (elbow). Therefore, in the consistency determination process performed by the consistency determination unit 131 after the recalculation unit 132 calculates the three-dimensional coordinates, the distance z2 calculated from the three-dimensional coordinates of the point P2 indicating the elbow and the position of the depth image 1101. The relationship of Q12'with the depth value z2'is z2'≤z2 + M. Therefore, the consistency determination unit 131 determines that the three-dimensional coordinates of the point P2 indicating the elbow are spatially matched. Therefore, if the relationship between the moving speed S2 of the point P2 indicating the elbow and the threshold value THS is S2 ≦ THS, the consistency parameter C2 of the point P2 indicating the elbow remains “True”. In this case, the consistency determination unit 131 determines that the point P2 indicating the elbow is spatially and temporally aligned.

Further, in the depth image 1101 of FIG. 31, the position Q21 of the point P1 indicating the wrist corresponding to the second upper limb region 1101B is within the second upper limb region 1101B, and the position Q22 of the point P2 indicating the elbow is the body region 1101D. Inside. In this case, the consistency determination unit 131 spatially matches the three-dimensional coordinates of the points P1 and P2 in the consistency determination process targeting the center of the hand and the joint corresponding to the second upper limb region 1101B. It is determined that it is. Therefore, when the three-dimensional coordinates of the points P1 and the points P2 are consistent in time, the consistency determination unit 131 uses the three-dimensional coordinates of the center of the hand and each joint corresponding to the second upper limb region 1101B. Is determined to be the three-dimensional coordinates stored in the coordinate history 193.

As described above, in the hand position detecting device 1 of the present embodiment, after detecting the position of the point indicating the hand center from within the upper limb region in the depth image, the hand center and the upper limb are located based on the detected position of the hand center. Calculate the three-dimensional coordinates of the joint. At this time, the hand position detecting device 1 allows the position of the joint in the depth image to be outside the upper limb region, so that the positional relationship between the center of the hand and each joint in the three-dimensional space substantially matches the dimension of the upper limb. Determine the position of the 3D coordinates of each joint (3D coordinates). After that, the hand position detecting device 1 determines the spatial consistency of the three-dimensional coordinates based on the distance calculated from the three-dimensional coordinates of the center of the hand and each joint and the depth value in the depth image, and spatially matches them. If not, correct the 3D coordinates of the center of the hand and each joint. In the hand position detection device 1 of the present embodiment, when the depth value in the depth image is larger than the distance calculated from the three-dimensional coordinates of the joint, it is determined that the three-dimensional coordinates of the joint are not spatially matched. Correct the 3D coordinates. Therefore, in the hand position detecting device 1 of the present embodiment, it is possible to correctly calculate the three-dimensional coordinates of the joints of the upper limbs from the depth image in which the joints of the upper limbs are hidden by the body portion and the head.

The flowchart of FIG. 3 is only an example of the coordinate calculation process performed by the hand position detection device 1 of the present embodiment. The coordinate calculation process performed by the hand position detection device 1 of the present embodiment can be appropriately changed as long as it does not deviate from the gist of the present embodiment. For example, in the coordinate calculation process, the order of the process of detecting the body region from the depth image (step S12) and the process of detecting the upper limb region from the depth image (step S2) may be reversed, or the second. The two processes may be performed in parallel.

Further, the flowchart of FIG. 20 is only an example of a process of detecting a body region from a depth image. Similarly, the flowchart of FIG. 21 is only an example of the process of detecting the upper limb region from the depth image. The process of detecting the body region from the depth image and the process of detecting the upper limb region can be appropriately changed within a range that does not deviate from the gist of the present embodiment. Further, for example, the process of detecting the body region from the depth image and the process of detecting the upper limb region may be one process. That is, the hand position detection device 1 has a larger difference in depth value than the upper limb regions 1102A and 1102B, and the body region detection area, like the body region 1102D detected when the binary image 1102 of FIG. 27 is generated. The region included in may be detected as a body region.

Further, the flowchart of FIG. 22 is only an example of the process of calculating the three-dimensional coordinates of the center of the hand and the joint. The process of detecting the position of the center of the hand performed by the hand position detection device 1 of the present embodiment (step S14) and the process of calculating the three-dimensional coordinates of the center of the hand and the joint (step S15) deviate from the gist of the present embodiment. It can be changed as appropriate to the extent that it does not. Further, in the process of calculating the three-dimensional coordinates of the center of the hand and the joint, the dimension of the subject registered in the dimension information 192 is not limited to the average value of the dimensions of the upper limbs (for example, the average value of the dimensions from the center of the wrist to the wrist). May be used to calculate the three-dimensional position.

Further, the flowchart of FIG. 22 is only an example of the consistency determination process. The consistency determination process performed by the hand position detection device 1 of the present embodiment may be, for example, a process according to the flowchart of FIG. 5 or a process according to the flowcharts of FIGS. 17A and 17B. ..

The above-mentioned hand position detection device 1 can be realized by a computer and a program executed by the computer. Hereinafter, the distance measuring device 1 realized by the computer and the program will be described with reference to FIG. 32.

FIG. 32 is a diagram showing a hardware configuration of a computer.
As shown in FIG. 32, the computer 20 includes a processor 2001, a main storage device 2002, an auxiliary storage device 2003, an input device 2004, an output device 2005, an input / output interface 2006, a communication control device 2007, and a medium. The drive device 2008 is provided. These elements 2001-2008 in the computer 20 are connected to each other by a bus 2010, and data can be exchanged between the elements.

The processor 2001 is a Central Processing Unit (CPU), a Micro Processing Unit (MPU), or the like. Processor 2001 controls the overall operation of the computer 20 by executing various programs including the operating system. Further, the processor 2001 obtains the three-dimensional coordinates of the center of the hand and the joints of the upper limbs in the three-dimensional space in which the depth image is captured by executing the hand position detection program including each process in the flowcharts of FIGS. 3 to 6, for example. calculate. The hand position detection program may be, for example, a program including each process in the flowcharts of FIGS. 19 to 23. Further, the coordinate correction process (step S6) in the hand position detection program is not limited to the process according to the flowchart of FIG. 4, and may be, for example, the process according to the flowcharts of FIGS. 14A and 14B. Further, the consistency determination process (step S603) in the hand position detection program may be, for example, a process according to the flowcharts of FIGS. 17A and 17B.

The main storage device 2002 includes a Read Only Memory (ROM) and a Random Access Memory (RAM) (not shown). In the ROM of the main storage device 2002, for example, a predetermined basic control program read by the processor 2001 when the computer 20 is started is recorded in advance. Further, the RAM of the main storage device 2002 is used as a working storage area by the processor 2001 as necessary when executing various programs. The RAM of the main storage device 2002 can be used for storing, for example, the background image 191 and the dimensional information 192, the coordinate history 193, the depth image 194, and the like.

The auxiliary storage device 2003 is a storage device having a larger capacity than the RAM of the main storage device 2002, such as a Hard Disk Drive (HDD) or a non-volatile memory (including a Solid State Drive (SSD)) such as a flash memory. be. The auxiliary storage device 2003 can be used to store various programs, various data, and the like executed by the processor 2001. The auxiliary storage device 2003 can be used, for example, for storing the above-mentioned hand position detection program. Further, the auxiliary storage device 2003 can be used for storing, for example, the background image 191 and the dimensional information 192, the coordinate history 193, the depth image 194, and the like.

The input device 2004 is, for example, a keyboard device, a mouse device, a touch panel device, and the like. The input device 2004 can be used, for example, for inputting the dimension information 192 and setting the background image 191. Further, the input device 2004 may include a distance measuring device such as a depth sensor 2.

The output device 2005 is, for example, a display device such as a liquid crystal display device, a printer, or the like. The output device 2005 can be used, for example, for displaying or printing the coordinate history 193.

The input / output interface 2006 connects the computer 20 to other electronic devices. The input / output interface 2006 includes, for example, a Universal Serial Bus (USB) standard connector or the like. The input / output interface 2006 can be used, for example, to connect the computer 20 to a distance measuring device such as a depth sensor 2.

The communication control device 2007 is a device that connects the computer 20 to a network such as the Internet and controls various communications between the computer 20 and other electronic devices via the network. The communication control device 2007 can be used, for example, for communication between the computer 20 operated as the hand position detection device 1 and the depth sensor 2 via the network 4 (see FIGS. 7 and 24). Further, the communication control device 2007 can be used for communication between the computer 20 operated as the hand position detection device 1 and the external device 3 via the network 4, for example.

The medium drive device 2008 reads out the programs and data recorded in the portable recording medium 21, and writes the data stored in the auxiliary storage device 2003 to the portable recording medium 21. As the medium drive device 2008, for example, a reader / writer for a memory card corresponding to one or more types of standards can be used. When a memory card reader / writer is used as the medium drive device 2008, the portable recording medium 21 is a memory card (flash memory) of a standard supported by the memory card reader / writer, for example, a Secure Digital (SD) standard. ) Etc. can be used. Further, as the portable recording medium 21, for example, a flash memory provided with a USB standard connector can be used. Further, when the computer 20 is equipped with an optical disk drive that can be used as the medium drive device 2008, various optical disks that can be recognized by the optical disk drive can be used as the portable recording medium 21. Optical discs that can be used as the portable recording medium 21 include, for example, Compact Disc (CD), Digital Versatile Disc (DVD), Blu-ray Disc (registered trademark), and the like. The portable recording medium 21 can be used, for example, for storing the above-mentioned hand position detection program. Further, the portable recording medium 21 can be used for storing, for example, a background image 191 and dimensional information 192, a coordinate history 193, and a depth image 194.

When a start command for processing to calculate the three-dimensional coordinates of the center of the hand and the joint is input to the computer 20, the processor 2001 reads out a hand position detection program stored in a non-temporary recording medium such as an auxiliary storage device 2003. And execute. While executing the hand position detection program, the processor 2001 functions (operates) as a depth image acquisition unit 110, a joint position calculation unit 120, a correction unit 130, and an output unit 140 in the hand position detection device 1. A part of the functions of the output unit 140 in the hand position detection device 1 is carried out by the input / output interface 2006 of the computer 20, the communication control device 2007, and the medium drive device 2008. Further, while the hand position detection program is being executed, the RAM of the main storage device 2002 and the storage devices such as the auxiliary storage device 2003 function as the storage unit 190 of the hand position detection device 1.

The computer 20 operated as the hand position detection device 1 does not need to include all the elements 2001 to 2008 shown in FIG. 32, and some elements may be omitted depending on the application and conditions. For example, the computer 20 may be the one in which the medium driving device 2008 is omitted.

The following additional notes will be further disclosed with respect to each of the above-described embodiments.
(Appendix 1)
The computer
The upper limb area corresponding to the upper limb part of the subject is detected from the image containing the information of the distance to the object existing within the predetermined measurement range.
Based on the shape and position of the upper limb region in the image, the positions of the hand center and the upper limb joint in the image are detected, and the detected tertiary of the hand center and the joint within the measurement range. Calculate the original coordinates,
It is determined whether or not each of the calculated three-dimensional coordinates of the hand center and the three-dimensional coordinates of the joint is consistent with the predetermined positional relationship between the hand center and each joint in the upper limb of the subject.
If the calculated three-dimensional coordinates do not match the predetermined positional relationship between the hand center and each joint, the position of the hand center in the depth image is changed to the tertiary of the hand center. Correct the original coordinates and the three-dimensional coordinates of the joint,
When the calculated three-dimensional coordinates match the predetermined positional relationship between the hand center and each joint, the current three-dimensional coordinates of the hand center and the three-dimensional coordinates of the joint are set in the measurement range. Determine the three-dimensional coordinates within,
A hand position detection method characterized by executing a process.
(Appendix 2)
In the process of detecting the positions of the center of the hand and the joints of the upper limbs in the image and calculating the three-dimensional coordinates of the detected center of the hand and the joints within the measurement range, the computer uses the computer.
After detecting the position of the center of the hand,
Based on the detected position of the center of the hand, the position of the joint is detected in order from the joint of the upper limbs closest to the center of the hand.
The hand position detection method according to Appendix 1, wherein the hand position is detected.
(Appendix 3)
In the process of detecting the position of the joint of the upper limb based on the position of the center of the hand, the computer allows the position of the joint of the upper limb to be outside the region of the upper limb.
The hand position detection method according to Appendix 2, wherein the hand position is detected.
(Appendix 4)
After the computer corrects the three-dimensional coordinates of the hand center and the three-dimensional coordinates of the joint, the corrected three-dimensional coordinates and the predetermined positional relationship between the hand center and each joint are determined. Determine if they are consistent,
The hand position detection method according to Appendix 1, wherein the hand position is detected.
(Appendix 5)
When the number of corrections of the three-dimensional coordinates of the hand center and the three-dimensional coordinates of the joint exceeds the threshold value, the computer uses the corrected three-dimensional coordinates and the predetermined hand center and each joint. Regardless of whether the positional relationship is consistent, the current 3D coordinates of the hand center and the 3D coordinates of the joint are determined to be the 3D coordinates within the measurement range.
The hand position detection method according to Appendix 4, wherein the hand position is detected.
(Appendix 6)
In the process of correcting the three-dimensional coordinates of the center of the hand and the three-dimensional coordinates of the joint, the computer is used.
When the position of the center of the hand after the change is outside the upper limb region, information on the distance of the adjacent region including the position of the center of the hand after the change, which is outside the upper limb region, is read out.
When the distance of the adjacent region is shorter than the distance of the upper limb region, the three-dimensional coordinates of the hand center and the three-dimensional coordinates of the joint are corrected based on the changed position of the hand center.
The hand position detection method according to Appendix 1, wherein the hand position is detected.
(Appendix 7)
In the process of correcting the three-dimensional coordinates of the center of the hand and the three-dimensional coordinates of the joint, the computer is used.
When the position of the center of the hand after the change is outside the upper limb region and the distance of the adjacent region including the position of the center of the hand after the change is shorter than the distance of the upper limb region, the distance of the upper limb region. The distance corresponding to the hand center at the changed position of the hand center is calculated based on the information of the above, and the three-dimensional coordinates of the hand center are calculated based on the calculated distance.
The hand position detection method according to Appendix 1, wherein the hand position is detected.
(Appendix 8)
In the process of correcting the three-dimensional coordinates of the center of the hand and the three-dimensional coordinates of the joint, the computer is used.
When the difference between the distance of the adjacent region and the distance of the upper limb region is equal to or greater than the threshold value, it is determined that the distance of the adjacent region is shorter than the distance of the upper limb region.
The hand position detection method according to any one of Supplementary note 6 and 7, wherein the hand position is detected.
(Appendix 9)
In the process of determining whether or not the calculated three-dimensional coordinates and the predetermined positional relationship between the hand center and each joint are in agreement, the computer uses the computer.
For each of the hand center and the joint, the position and the distance of the hand center or the joint in the image are calculated based on the three-dimensional coordinates.
When the calculated distance is shorter than the distance between the center of the hand or the position of the joint in the image, the three-dimensional coordinates do not match the predetermined positional relationship between the center of the hand and each joint. judge,
The hand position detection method according to Appendix 1, wherein the hand position is detected.
(Appendix 10)
In the process of determining whether or not the calculated three-dimensional coordinates and the predetermined positional relationship between the hand center and each joint are in agreement, the computer uses the computer.
For each pair of hand center and joint detected from the upper limb region, the three-dimensional distance between the adjacent hand center and the joint and the three-dimensional distance between the adjacent joints are calculated.
When the difference between the calculated three-dimensional distance and the dimension specified based on the positional relationship between the predetermined hand center and each joint is equal to or greater than the threshold value, the three-dimensional coordinates and the predetermined hand center And it is judged that the positional relationship of each joint is not consistent.
The hand position detection method according to Appendix 1, wherein the hand position is detected.
(Appendix 11)
In the process of determining whether or not the calculated three-dimensional coordinates and the predetermined positional relationship between the hand center and each joint are in agreement, the computer uses the computer.
The movement speed of the hand center and the joint is calculated from the calculated three-dimensional coordinates and the three-dimensional coordinates specified based on the predetermined positional relationship between the hand center and each joint.
When the moving speed of either the center of the hand or the joint is equal to or higher than the threshold value, it is determined that the three-dimensional coordinates and the predetermined positional relationship between the center of the hand and each joint do not match.
The hand position detection method according to Appendix 1, wherein the hand position is detected.
(Appendix 12)
The process performed by the computer further includes a process of detecting a body region corresponding to the body portion of the subject from the image and estimating the position of the shoulder from the body region.
In the process of detecting the positions of the center of the hand and the joints of the upper limbs in the image and calculating the three-dimensional coordinates of the detected center of the hand and the joints within the measurement range, the computer uses the computer.
After detecting the position of the center of the hand, the position of the joint of the upper limb is detected based on the detected position of the center of the hand and the estimated position of the shoulder.
The hand position detection method according to Appendix 1, wherein the hand position is detected.
(Appendix 13)
The predetermined positional relationship between the center of the hand and each joint is information calculated based on the information indicating the dimensions of the upper limbs measured from the subject.
The hand position detection method according to Appendix 1, wherein the hand position is detected.
(Appendix 14)
The predetermined positional relationship between the hand center and each joint is the hand center calculated from the image captured at a time predetermined time before the imaging time of the image currently being processed. Information calculated based on the three-dimensional coordinates of the joint and the three-dimensional coordinates of the joint.
The hand position detection method according to Appendix 1, wherein the hand position is detected.
(Appendix 15)
The upper limb region corresponding to the upper limb portion of the subject is detected from the image including the information of the distance to the object existing in the predetermined measurement range, and the inside of the image is based on the shape and position of the upper limb region in the image. A joint position calculation unit that detects the positions of the joints of the center of the hand and the joints of the upper limbs and calculates the three-dimensional coordinates of the detected center of the hand and the joints within the measurement range.
It is determined and calculated whether each of the calculated three-dimensional coordinates of the hand center and the three-dimensional coordinates of the joint is consistent with the predetermined positional relationship between the hand center and each joint in the upper limb of the subject. If the three-dimensional coordinates are not aligned with the predetermined positional relationship between the hand center and each joint, the position of the hand center in the depth image is changed to make the three-dimensional center of the hand. If the coordinates and the three-dimensional coordinates of the joint are corrected and the calculated three-dimensional coordinates match the predetermined positional relationship between the hand center and each joint, the current three-dimensional of the hand center. A correction unit that determines the coordinates and the three-dimensional coordinates of the joint to the three-dimensional coordinates within the measurement range, and
A hand position detection device characterized by being provided with.
(Appendix 16)
The upper limb area corresponding to the upper limb part of the subject is detected from the image containing the information of the distance to the object existing within the predetermined measurement range.
Based on the shape and position of the upper limb region in the image, the positions of the hand center and the upper limb joint in the image are detected, and the detected tertiary of the hand center and the joint within the measurement range. Calculate the original coordinates,
It is determined whether or not each of the calculated three-dimensional coordinates of the hand center and the three-dimensional coordinates of the joint is consistent with the predetermined positional relationship between the hand center and each joint in the upper limb of the subject.
If the calculated three-dimensional coordinates do not match the predetermined positional relationship between the hand center and each joint, the position of the hand center in the depth image is changed to the tertiary of the hand center. Correct the original coordinates and the three-dimensional coordinates of the joint,
When the calculated three-dimensional coordinates match the predetermined positional relationship between the hand center and each joint, the current three-dimensional coordinates of the hand center and the three-dimensional coordinates of the joint are set in the measurement range. Determine the three-dimensional coordinates within,
A hand position detection program that causes a computer to perform processing.

1 Hand position detection device 110 Depth image acquisition unit 120 Joint position calculation unit 121 Upper limb area detection unit 122 In-image position detection unit 123 Three-dimensional coordinate calculation unit 124 Body region detection unit 130 Correction unit 131 Consistency determination unit 132 Recalculation unit 140 Output unit 190 Storage unit 191 Background image 192 Dimensional information 193 Coordinate history 194 Depth image 2 Depth sensor 3 External device 3A Management device 4 Network 5 Management system 6 Workspace 7 Worker 710 Upper left limb 720 Upper right limb 711,721 Hand 712,722 Wrist 713 Elbow 714,724 Shoulder 750 Head 760 Body part 1101 Depth image 1102 Binary image 1101A, 1101B, 1102A, 1102B Upper limb area 20 Computer 2001 Processor 2002 Main memory 2003 Auxiliary storage 2004 Input device 2005 Output device 2006 Input / output Interface 2007 Communication control device 2008 Media drive device 2010 Bus 21 Portable recording medium

Claims (12)

The computer
The upper limb area corresponding to the upper limb part of the subject is detected from the image containing the information of the distance to the object existing within the predetermined measurement range.
Based on the shape and position of the upper limb region in the image, the positions of the hand center and the upper limb joint in the image are detected, and the detected tertiary of the hand center and the joint within the measurement range. Calculate the original coordinates,
It is determined whether or not each of the calculated three-dimensional coordinates of the hand center and the three-dimensional coordinates of the joint is consistent with the predetermined positional relationship between the hand center and each joint in the upper limb of the subject.
If the calculated three-dimensional coordinates do not match the predetermined positional relationship between the hand center and each joint, the position of the hand center in the image is changed to make the three-dimensional center of the hand. Correct the coordinates and the three-dimensional coordinates of the joint,
When the calculated three-dimensional coordinates match the predetermined positional relationship between the hand center and each joint, the current three-dimensional coordinates of the hand center and the three-dimensional coordinates of the joint are set in the measurement range. Determine the three-dimensional coordinates within,
A hand position detection method characterized by executing a process. In the process of detecting the positions of the center of the hand and the joints of the upper limbs in the image and calculating the three-dimensional coordinates of the detected center of the hand and the joints within the measurement range, the computer uses the computer.
After detecting the position of the center of the hand,
Based on the detected position of the center of the hand, the position of the joint is detected in order from the joint of the upper limbs closest to the center of the hand.
The hand position detection method according to claim 1, wherein the hand position is detected. In the process of detecting the position of the joint of the upper limb based on the position of the center of the hand, the computer allows the position of the joint of the upper limb to be outside the region of the upper limb.
The hand position detection method according to claim 2, wherein the hand position is detected. After the computer corrects the three-dimensional coordinates of the hand center and the three-dimensional coordinates of the joint, the corrected three-dimensional coordinates and the predetermined positional relationship between the hand center and each joint are determined. Determine if they are consistent,
The hand position detection method according to claim 1, wherein the hand position is detected. In the process of correcting the three-dimensional coordinates of the center of the hand and the three-dimensional coordinates of the joint, the computer is used.
When the position of the center of the hand after the change is outside the upper limb region, information on the distance of the adjacent region including the position of the center of the hand after the change, which is outside the upper limb region, is read out.
When the distance of the adjacent region is shorter than the distance of the upper limb region, the three-dimensional coordinates of the hand center and the three-dimensional coordinates of the joint are corrected based on the changed position of the hand center.
The hand position detection method according to claim 1, wherein the hand position is detected. In the process of correcting the three-dimensional coordinates of the center of the hand and the three-dimensional coordinates of the joint, the computer is used.
When the position of the center of the hand after the change is outside the upper limb region and the distance of the adjacent region including the position of the center of the hand after the change is shorter than the distance of the upper limb region, the distance of the upper limb region. The distance corresponding to the hand center at the changed position of the hand center is calculated based on the information of the above, and the three-dimensional coordinates of the hand center are calculated based on the calculated distance.
The hand position detection method according to claim 1, wherein the hand position is detected. In the process of determining whether or not the calculated three-dimensional coordinates and the predetermined positional relationship between the hand center and each joint are in agreement, the computer uses the computer.
For each of the hand center and the joint, the position and the distance of the hand center or the joint in the image are calculated based on the three-dimensional coordinates.
When the calculated distance is shorter than the distance between the center of the hand or the position of the joint in the image, the three-dimensional coordinates do not match the predetermined positional relationship between the center of the hand and each joint. judge,
The hand position detection method according to claim 1, wherein the hand position is detected. In the process of determining whether or not the calculated three-dimensional coordinates and the predetermined positional relationship between the hand center and each joint are in agreement, the computer uses the computer.
For each pair of hand center and joint detected from the upper limb region, the three-dimensional distance between the adjacent hand center and the joint and the three-dimensional distance between the adjacent joints are calculated.
When the difference between the calculated three-dimensional distance and the dimension specified based on the positional relationship between the predetermined hand center and each joint is equal to or greater than the threshold value, the three-dimensional coordinates and the predetermined hand center And it is judged that the positional relationship of each joint is not consistent.
The hand position detection method according to claim 1, wherein the hand position is detected. In the process of determining whether or not the calculated three-dimensional coordinates and the predetermined positional relationship between the hand center and each joint are in agreement, the computer uses the computer.
The movement speed of the hand center and the joint is calculated from the calculated three-dimensional coordinates and the three-dimensional coordinates specified based on the predetermined positional relationship between the hand center and each joint.
When the moving speed of either the center of the hand or the joint is equal to or higher than the threshold value, it is determined that the three-dimensional coordinates and the predetermined positional relationship between the center of the hand and each joint do not match.
The hand position detection method according to claim 1, wherein the hand position is detected. The process performed by the computer further includes a process of detecting a body region corresponding to the body portion of the subject from the image and estimating the position of the shoulder from the body region.
In the process of detecting the positions of the center of the hand and the joints of the upper limbs in the image and calculating the three-dimensional coordinates of the detected center of the hand and the joints within the measurement range, the computer uses the computer.
After detecting the position of the center of the hand, the position of the joint of the upper limb is detected based on the detected position of the center of the hand and the estimated position of the shoulder.
The hand position detection method according to claim 1, wherein the hand position is detected. The upper limb region corresponding to the upper limb portion of the subject is detected from the image including the information of the distance to the object existing in the predetermined measurement range, and the inside of the image is based on the shape and position of the upper limb region in the image. A joint position calculation unit that detects the positions of the joints of the center of the hand and the joints of the upper limbs and calculates the three-dimensional coordinates of the detected center of the hand and the joints within the measurement range.
It is determined and calculated whether each of the calculated three-dimensional coordinates of the hand center and the three-dimensional coordinates of the joint is consistent with the predetermined positional relationship between the hand center and each joint in the upper limb of the subject. If the three-dimensional coordinates obtained do not match the predetermined positional relationship between the hand center and each joint, the position of the hand center in the image is changed and the three-dimensional coordinates of the hand center are changed. And if the calculated 3D coordinates by correcting the 3D coordinates of the joint and the predetermined positional relationship between the hand center and each joint match, the current 3D coordinates of the hand center. And a correction unit that determines the three-dimensional coordinates of the joint to the three-dimensional coordinates within the measurement range.
A hand position detection device characterized by being provided with. The upper limb area corresponding to the upper limb part of the subject is detected from the image containing the information of the distance to the object existing within the predetermined measurement range.
Based on the shape and position of the upper limb region in the image, the positions of the hand center and the upper limb joint in the image are detected, and the detected tertiary of the hand center and the joint within the measurement range. Calculate the original coordinates,
It is determined whether or not each of the calculated three-dimensional coordinates of the hand center and the three-dimensional coordinates of the joint is consistent with the predetermined positional relationship between the hand center and each joint in the upper limb of the subject.
If the calculated three-dimensional coordinates do not match the predetermined positional relationship between the hand center and each joint, the position of the hand center in the image is changed to make the three-dimensional center of the hand. Correct the coordinates and the three-dimensional coordinates of the joint,
When the calculated three-dimensional coordinates match the predetermined positional relationship between the hand center and each joint, the current three-dimensional coordinates of the hand center and the three-dimensional coordinates of the joint are set in the measurement range. Determine the three-dimensional coordinates within,
A hand position detection program that causes a computer to perform processing.

Images