JP7120690B2 - Joint position estimation device, joint position estimation method and program - Google Patents

Description

The present invention relates to a joint position estimating device and a joint position estimating method for estimating joint positions from image data, and further to a program for realizing these.

2. Description of the Related Art Conventionally, various techniques have been proposed for capturing an image of a person with an imaging device and estimating the posture of the person and the motions (gestures, etc.) of the person. For example, Non-Patent Document 1 discloses a technique for estimating the joint positions of a person and estimating the posture of the person. In Non-Patent Document 1, human body parts are estimated from a depth image, and joint positions are estimated by a joint position estimator using the estimation results.

Ryuhei Takahashi, “Special Research Report: Human Posture Estimation with Invariant Viewpoint Using Depth Image by CNN”, February 8, 2018, Internet <URL: http://www.mm.media.kyoto-u.ac .jp/wp-content/uploads/2018/02/2017-b-takahashi.pdf>

However, in Non-Patent Document 1, there is a problem that the accuracy of estimating joint positions decreases unless human body parts are estimated accurately.

An example of an object of the present invention is to provide a joint position estimating device, a joint position estimating method, and a program capable of accurately estimating joint positions.

In order to achieve the above object, a joint position estimation device according to one aspect of the present invention includes:
an image data acquisition unit that acquires image data;
an estimation unit that estimates a human body part from the image data acquired by the image data acquisition unit;
a marking unit that, when the estimating unit estimates a plurality of human body parts, marks one or more pixels in each of the human body part area estimated by the estimating unit and the area other than the human body part area; ,
a dividing unit that applies a watershed algorithm to the image data after marking by the marking unit and divides the area in the image data into a plurality;
a calculation unit that calculates a joint position from two regions corresponding to human body parts adjacent to the joint, among the regions divided by the dividing unit;
Prepare.

Further, in order to achieve the above object, a joint position estimation method according to one aspect of the present invention includes:
obtaining image data;
estimating a human body part for the image data;
when a plurality of human body parts are estimated for the image data, marking one or more pixels in each of the estimated human body part area and the area other than the human body part area;
applying a watershed algorithm to the image data after marking to divide a region in the image data into a plurality;
a step of calculating a joint position from two regions corresponding to human body parts adjacent to the joint among the divided regions;
Prepare.

Furthermore, in order to achieve the above object, a program according to one aspect of the present invention provides a computer that acquires image data,
estimating a human body part for the image data;
when a plurality of human body parts are estimated, marking one or more pixels in each of the estimated human body part area and the area other than the human body part area;
applying a watershed algorithm to the image data after marking to divide a region in the image data into a plurality;
a step of calculating a joint position from two regions corresponding to human body parts adjacent to the joint among the divided regions;
contains instructions to execute

As described above, according to the present invention, joint positions can be accurately estimated.

FIG. 1 is a block diagram showing a schematic configuration of a joint position estimation device according to Embodiment 1. FIG. FIG. 2 is a block diagram specifically showing the configuration of the joint position estimation device. FIG. 3 is a diagram showing an example of image data. FIG. 4 is a diagram showing feature amounts obtained by the feature vector calculator. 5 is a diagram showing regions set for the estimation result by the estimation unit shown in FIG. 4. FIG. FIG. 6 is a diagram showing a marking result by the marking unit. FIG. 7 is a diagram showing a result of segmentation by the segmentation unit. FIG. 8 is a diagram for explaining a calculation method by a calculation unit; FIG. 9 is a flow chart showing operations by the joint position estimation device. FIG. 10 is a block diagram showing an example of a computer that implements the joint position estimation device according to the embodiment.

A joint position estimation device and a joint position estimation method according to an embodiment of the present invention will be described below with reference to FIGS. 1 to 10. FIG.

[Device configuration]
FIG. 1 is a block diagram showing a schematic configuration of a joint position estimation device 1 according to Embodiment 1. As shown in FIG.

The joint position estimation device 1 is a device that estimates parts of a human body (for example, palms and forearms) from image data and calculates joint positions from the estimated parts of the human body. This joint position estimation device 1 includes an image data acquisition unit 2 , an estimation unit 3 , a marking unit 4 , a division unit 5 and a calculation unit 6 .

The image data acquisition unit 2 acquires image data.

The estimation unit 3 estimates a human body part from the image data acquired by the image data acquisition unit 2 .

When the estimating unit 3 estimates a plurality of human body parts, the marking unit 4 adds one or more pixels to each of the human body part area estimated by the estimating unit 3 and the area other than the human body part area. to mark.

The dividing unit 5 applies a watershed algorithm to the image data after marking by the marking unit 4 to divide the image data into a plurality of regions.

The calculation unit 6 calculates the joint position from two regions corresponding to human body parts adjacent to the joint among the regions divided by the division unit 5 .

As described above, in the present embodiment, marking is applied to the results of estimating human body parts, and the watershed algorithm is applied. Therefore, marking can be performed with high accuracy, and by using the result as an input and applying the watershed algorithm, the image data can be segmented with high accuracy. As a result, the region of the human body part adjacent to the joint can also be estimated with high accuracy, so that a highly reliable calculation result of the joint (wrist) position can be obtained.

Next, with reference to FIG. 2, the configuration of the joint position estimation device 1 according to this embodiment will be described more specifically. FIG. 2 is a block diagram specifically showing the configuration of the joint position estimation device 1. As shown in FIG.

The joint position estimating device 1 includes an area setting unit 7 in addition to the image data acquiring unit 2, the estimating unit 3, the marking unit 4, the dividing unit 5, and the calculating unit 6 described above.

The image data acquisition unit 2 acquires image data 50 . The image data acquisition unit 2 may acquire the image data 50 via a network such as a wired or wireless LAN (Local Area Network), or may acquire the image data 50 by short-range wireless communication or the like. An imaging device may be directly connected to the joint position estimating device 1, or the joint position estimating device 1 may include an imaging device, and the image data acquisition unit 2 may acquire the image data 50 from the imaging device.

FIG. 3 is a diagram showing an example of the image data 50. As shown in FIG. The image data 50 is an image captured from the forearm to the finger of the human body. The image data 50 in this embodiment is image data to which the depth of each pixel is added (hereinafter referred to as TOF image). A TOF image is an image in which information (depth information) indicating the distance from an imaging device to an imaging target is associated with each pixel. Note that the image data 50 is not limited to a TOF image, and may be a two-dimensional image.

Also, the image data 50 may be an image in which a full image of a person is captured. In this case, for example, the image data acquisition unit 2 may estimate a part including joints (for example, the entire hand) from an image of the whole person and extract the image data of that part. Then, subsequent processing may be performed on the extracted image data.

The estimating unit 3 estimates the type of the human body part (forearm, finger, etc.) shown in the image data 50 . The estimation unit 3 has a feature vector calculation unit 31 and an estimation execution unit 32 .

The feature vector calculator 31 obtains a feature quantity (feature vector) for each pixel of the image data 50 . FIG. 4 is a diagram showing feature amounts obtained by the feature vector calculator 31. As shown in FIG. As shown in FIG. 4, the feature vector calculation unit 31 obtains feature amounts 51, 52, 53, and 54, respectively. The estimation execution unit 32 performs machine learning using a neural network or the like, and uses the learning model 60 to estimate the type of the object appearing in the image data 50 from the feature amount obtained by the feature vector calculation unit 31 . In the case of FIG. 4, the estimation execution unit 32 estimates that the feature amount 51 is the feature amount of the forearm, the feature amount 52 is the feature amount of the palm, and the feature amount 53 is the feature amount of the finger. In addition, the estimation execution unit 32 estimates that the feature amount 54 is a feature amount of an object other than a human body part and captured in the background. In FIG. 4, the human body parts shown in FIG. 3 are indicated by dashed lines for understanding of the feature amount obtained by the feature vector calculator 31. As shown in FIG.

Since the image data 50 is a TOF image, the estimation accuracy of the human body part in the estimating section 3 is improved. For example, when the image data 50 is an image including joints, the feature vector calculation unit 31 uses the depth information for each pixel to obtain the feature quantity with high accuracy even if the joints are bent. can be done. As a result, the estimation accuracy of the human body part by the estimation execution unit 32 is improved.

The area setting unit 7 sets an area including the forearm and the palm, which are human body parts adjacent to joints, among the human body parts estimated by the estimating part 3 . FIG. 5 is a diagram showing a region 55 set for the estimation result by the estimation unit 3 shown in FIG. In FIG. 5, the area 55 includes the feature amount 53 of the finger, but it should include at least the feature amount 51 of the forearm adjacent to the joint (wrist) and the feature amount 52 of the palm. By setting the area 55, it is possible to prevent unnecessary areas from being processed in subsequent processes.

The marking unit 4 marks one or more pixels within the area 55 set by the area setting unit 7 . 6A and 6B are diagrams showing marking results by the marking unit 4. FIG. The marking unit 4 initially marks the human body parts (forearm, palm) adjacent to the joint (wrist) and the area (background) other than the human body parts in the area 55 . Markings 56 shown in FIG. 6 are initial markings set for the feature amount 51 of the forearm. A marking 57 is an initial marking set for the palm feature amount 52 . Markings 58 indicate the initial markings set for the background area.

When initial marking is performed on the feature amount 51 of the forearm and the feature amount 52 of the palm, the marking unit 4 may be affected by noise if it is close to the boundary with other areas. Mark the center of the volume area. When the marking unit 4 performs initial marking on the background area, the marking unit 4 marks a position outside the predetermined distance range from the markings 56 and 57 . Note that the marking unit 4 may mark only one pixel, but in the watershed algorithm described later, if the change in the depth value of the TOF image is not smooth, the marked area may not expand. Therefore, it is preferable that the marking section 4 marks a plurality of pixels.

The dividing unit 5 applies a watershed algorithm to which the markings 56, 57, and 58 set by the marking unit 4 are input, and divides the region 55 set by the region setting unit 7 into a plurality of regions. In the watershed algorithm, the markings 56, 57, and 58 are the minimum values (or maximum values) of the pixel values (depth values), and the point that becomes the maximum value (or minimum value) of the pixel values is sought while expanding the area. A region having a close pixel value (depth value) is defined as one region with the point of the maximum value as a boundary. 7A and 7B are diagrams showing the results of region division by the division unit 5. FIG. In FIG. 7, region 55 is divided into forearm region 59A, palm region 59B, and background region 59C.

In addition, in the watershed algorithm, the area is not always divided into the same number of areas as the number of markings, and in FIG. 7, the area may be divided into four or more areas. Moreover, although the dividing unit 5 widens the marked area using the depth value, the brightness value or the like may be used as long as it is information of each pixel of the image data 50 .

The calculation unit 6 calculates the joint positions from the two regions 59A and 59B corresponding to the human body parts among the regions divided by the division unit 5 . FIG. 8 is a diagram for explaining a calculation method by the calculation unit 6. FIG. The calculation unit 6 performs labeling processing on each of the regions 59A and 59B, and acquires the contour of each region. In FIG. 8 the contour is shown schematically. As shown in FIG. 8, when the area 59A and the area 59B are separated, the calculation unit 6 obtains the coordinate positions 59A1 and 59B1 that are closest to each other in the two areas 59A and 59B. The calculator 6 calculates a region 59D that includes the shortest straight line L connecting the coordinate positions 59A1 and 59B1, and sets it as the joint position. In addition, when the contours of the two regions 59A and 59B overlap, the calculation unit 6 may set the overlapping portion as the joint position.

[Description of operation]
Next, the operation of the joint position determination device according to this embodiment will be described with reference to FIG. FIG. 9 is a flow chart showing the operation of the joint position estimation device 1. As shown in FIG. In the following description, FIGS. 1 to 8 will be referred to as appropriate. Further, in this embodiment, the joint position estimation method is implemented by operating the joint position estimation device. Therefore, the description of the joint position estimation method in this embodiment is replaced with the description of the operation of the joint position estimation device below.

First, as a premise, it is assumed that an image of a human body part including joints is imaged by an imaging device, and that image data is input to the joint position estimation device 1 .

On the premise described above, the image data acquisition unit 2 acquires input image data (S1). Next, the feature vector calculator 31 of the estimator 3 obtains feature quantities 51 to 54 (see FIG. 4) for each pixel of the image data 50 (S2). Then, the estimation execution unit 32 of the estimation unit 3 recognizes the type of the object having the feature amount from the feature amount obtained by the feature vector calculation unit 31, and estimates the human body part in the image data 50 (S3).

Subsequently, the region setting unit 7 sets a region 55 (see FIG. 5) adjacent to the joint whose position is to be estimated and which includes the human body part estimated by the estimation unit 3 (S4). The marking unit 4 performs initial marking within the area 55 set by the area setting unit 7 (S5). In S5, the marking unit 4 creates a forearm marking 56 (see FIG. 6) adjacent to the joint (wrist) whose position is to be estimated, a palm marking 57, and a region (background) marking 58 other than the human body part. set.

The division unit 5 applies the watershed algorithm with the markings 56, 57, and 58 as inputs to divide the area 55 set in S4 into a plurality of areas 59A, 59B, and 59C (see FIG. 7) (S6). Then, the calculation unit 6 calculates the joint positions from the two regions 59A and 59B corresponding to the human body parts among the regions divided by the division unit 5 (S7).

[Effects of this embodiment]
As described above, according to the present embodiment, the human body part is estimated from the image data 50, and initial marking is performed on the estimation result. Therefore, in the image data 50, even if the target area to be marked, for example, the shape of the forearm or palm has a complicated shape, the initial marking can be performed with high accuracy. Using the initial markings as input and using the watershed algorithm, the two regions adjacent to the joints, the forearm and the palm, are more accurately estimated. If one region of the forearm or palm cannot be determined clearly, the position of the joint (wrist) between the two regions cannot be calculated with high accuracy. (Wrist) position can be calculated.

Further, once the positions of the joints, for example, the wrists are determined, it becomes easier to estimate the positions of the fingers at the tip of the wrist, and it becomes easier to estimate the motions and postures of the hands. As a result, it can be applied to a device that recognizes a gesture of moving a hand.

[program]
The program in the embodiment of the present invention may be any program that causes a computer to execute steps S1 to S7 shown in FIG. By installing this program in a computer and executing it, the joint position estimation device and the joint position estimation method in this embodiment can be realized. In this case, the processor of the computer functions as an image data acquisition section 2, an estimation section 3, a marking section 4, a division section 5 and a calculation section 6, and performs processing.

Further, in this embodiment, the image data acquisition unit 2, the estimation unit 3, the marking unit 4, the division unit 5, and the calculation unit 6 store data files constituting them in a storage device such as a hard disk provided in the computer. or by loading a recording medium storing this data file into a reading device connected to a computer.

Also, the program in this embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the image data acquiring unit 2, the estimating unit 3, the marking unit 4, the dividing unit 5, and the calculating unit 6, respectively.

[Physical configuration]
Here, a computer that implements the joint position estimation device 1 by executing the program in this embodiment will be described with reference to FIG. 10 . FIG. 10 is a block diagram showing an example of a computer that implements the joint position estimation device 1 according to this embodiment.

As shown in FIG. 10 , computer 110 includes CPU 111 , main memory 112 , storage device 113 , input interface 114 , display controller 115 , data reader/writer 116 and communication interface 117 . These units are connected to each other via a bus 121 so as to be able to communicate with each other. Note that the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to the CPU 111 or instead of the CPU 111 .

The CPU 111 expands the programs (codes) in the present embodiment stored in the storage device 113 into the main memory 112 and executes them in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). Also, the program in this embodiment is provided in a state stored in a computer-readable recording medium 120 . Note that the program in this embodiment may be distributed on the Internet connected via the communication interface 117 .

Further, as a specific example of the storage device 113, in addition to a hard disk drive, a semiconductor storage device such as a flash memory can be cited. Input interface 114 mediates data transmission between CPU 111 and input devices 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls display on the display device 119 .

Data reader/writer 116 mediates data transmission between CPU 111 and recording medium 120 , reads programs from recording medium 120 , and writes processing results in computer 110 to recording medium 120 . Communication interface 117 mediates data transmission between CPU 111 and other computers.

Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital); magnetic recording media such as flexible disks; An optical recording medium such as a ROM (Compact Disk Read Only Memory) can be used.

Note that the joint position estimation device 1 according to the present embodiment can also be realized by using hardware corresponding to each part instead of a computer in which a program is installed. Furthermore, the joint position estimation device 1 may be partially realized by a program and the rest by hardware.

Some or all of the above-described embodiments can be expressed by (Appendix 1) to (Appendix 15) described below, but are not limited to the following description.

(Appendix 1)
an image data acquisition unit that acquires image data;
an estimation unit that estimates a human body part from the image data acquired by the image data acquisition unit;
a marking unit that, when the estimating unit estimates a plurality of human body parts, marks one or more pixels in each of the human body part area estimated by the estimating unit and the area other than the human body part area; ,
a dividing unit that applies a watershed algorithm to the image data after marking by the marking unit and divides the area in the image data into a plurality;
a calculation unit that calculates a joint position from two regions corresponding to human body parts adjacent to the joint, among the regions divided by the dividing unit;
A joint position estimation device comprising:

(Appendix 2)
The joint position estimation device according to appendix 1,
an area setting unit that sets an area adjacent to the joint and containing the human body part estimated by the estimation unit;
further comprising
The marking unit marks within the area set by the area setting unit,
The division unit divides the area set by the area setting unit into a plurality of areas.
Joint position estimator.

(Appendix 3)
The joint position estimation device according to Appendix 1 or Appendix 2,
wherein the image data is image data to which a depth for each pixel is added;
Joint position estimator.

(Appendix 4)
The joint position estimation device according to any one of appendices 1 to 3,
The dividing unit divides the area into a plurality of areas, with the pixels marked by the marking unit and the pixels whose pixel values are within a predetermined range of the pixels being the same area.
Joint position estimator.

(Appendix 5)
The joint position estimation device according to any one of appendices 1 to 4,
The calculation unit calculates the joint position based on the position in each of the two regions where the distance between the two regions is the minimum,
Joint position estimator.

(Appendix 6)
obtaining image data;
a step of estimating a human body part for the acquired image data;
when a plurality of human body parts are estimated, marking one or more pixels in each of the estimated human body part area and the area other than the human body part area;
applying a watershed algorithm to the image data after marking to divide a region in the image data into a plurality;
a step of calculating a joint position from two regions corresponding to human body parts adjacent to the joint among the divided regions;
A joint position estimation method comprising:

(Appendix 7)
The joint position estimation method according to appendix 6,
setting a region adjacent to the joint and containing the estimated body part;
further comprising
In the step of marking one or more pixels, marking is performed within a set area;
In the step of dividing the region in the image data into a plurality of regions, the set region is divided into a plurality of regions.
Joint position estimation method.

(Appendix 8)
The joint position estimation method according to appendix 6 or appendix 7,
wherein the image data is image data to which a depth for each pixel is added;
Joint position estimation method.

(Appendix 9)
The joint position estimation method according to any one of appendices 6 to 8,
In the step of dividing the region in the image data into a plurality of regions, the marked pixels and the pixels whose pixel values are within a predetermined range of the pixels are defined as the same region, and the region is divided into a plurality of regions.
Joint position estimation method.

(Appendix 10)
The joint position estimation method according to any one of appendices 6 to 9,
In the step of calculating the joint position, the joint position is calculated based on the position in each of the two regions where the distance between the two regions is the minimum,
Joint position estimation method.

(Appendix 11)
In the computer that acquires the image data,
estimating a human body part for the image data;
when a plurality of human body parts are estimated, marking one or more pixels in each of the estimated human body part area and the area other than the human body part area;
applying a watershed algorithm to the image data after marking to divide a region in the image data into a plurality;
a step of calculating a joint position from two regions corresponding to human body parts adjacent to the joint among the divided regions;
A program containing instructions to cause a

(Appendix 12)
The program according to Supplementary Note 11,
to the computer;
setting a region adjacent to the joint that encompasses the estimated body part;
further comprising an instruction to cause the
In the step of marking the one or more pixels, marking within a set area;
In the step of dividing the region in the image data into a plurality of regions, the set region is divided into a plurality of regions.
program .

(Appendix 13)
The program according to Supplementary Note 11 or Supplementary Note 12,
wherein the image data is image data to which a depth for each pixel is added;
program .

(Appendix 14)
The program according to any one of appendices 11 to 13,
In the step of dividing the region in the image data into a plurality of regions, the marked pixels and the pixels whose pixel values are within a predetermined range of the pixels are defined as the same region, and the region is divided into a plurality of regions.
program .

(Appendix 15)
The program according to any one of appendices 11 to 14,
In the step of calculating the joint position, the joint position is calculated based on the position in each of the two regions where the distance between the two regions is the minimum,
program .

Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

This application claims priority based on Japanese Patent Application No. 2019-46390 filed on March 13, 2019, and the entire disclosure thereof is incorporated herein.

The present invention can accurately estimate joint positions.

1: joint position estimation device 2: image data acquisition unit 3: estimation unit 4: marking unit 5: division unit 6: calculation unit 7: region setting unit 31: feature vector calculation unit 32: estimation execution unit 50: image data 60: Learning models 51, 52, 53, 54: Feature amount 55: Regions 56, 57, 58: Marking

Claims (7)

an image data acquisition unit that acquires image data;
an estimation unit that estimates a human body part from the image data acquired by the image data acquisition unit ;
a marking unit that, when the estimating unit estimates a plurality of human body parts, marks one or more pixels in each of the human body part area estimated by the estimating unit and the area other than the human body part area; ,
a dividing unit that applies a watershed algorithm to the image data after marking by the marking unit and divides the area in the image data into a plurality;
a calculation unit that calculates a joint position from two regions corresponding to human body parts adjacent to the joint, among the regions divided by the dividing unit ;
A joint position estimation device comprising: The joint position estimation device according to claim 1,
an area setting unit that sets an area adjacent to the joint and containing the human body part estimated by the estimation unit ;
further comprising
The marking unit marks within the area set by the area setting unit ,
The division unit divides the area set by the area setting unit into a plurality of areas.
Joint position estimator. The joint position estimation device according to claim 1 or claim 2,
wherein the image data is image data to which a depth for each pixel is added;
Joint position estimator. The joint position estimation device according to any one of claims 1 to 3,
The dividing unit divides the area into a plurality of areas, with the pixels marked by the marking unit and the pixels whose pixel values are within a predetermined range of the pixels being the same area.
Joint position estimator. The joint position estimation device according to any one of claims 1 to 4,
The calculation unit calculates the joint position based on the position in each of the two regions where the distance between the two regions is the minimum,
Joint position estimator. get the image data,
estimating a human body part for the image data;
When a plurality of human body parts are estimated for the image data, marking one or more pixels in each of the estimated human body part area and the area other than the human body part area,
Applying a watershed algorithm to the image data after marking to divide the area in the image data into a plurality,
Calculating joint positions from two regions corresponding to human body parts adjacent to joints among the plurality of divided regions;
Joint position estimation method. In the computer that acquires the image data,
estimating a human body part for the image data;
When a plurality of human body parts are estimated, one or more pixels are marked for each of the estimated human body part area and the area other than the human body part area,
Applying a watershed algorithm to the image data after marking to divide the area in the image data into a plurality,
calculating joint positions from two regions corresponding to human body parts adjacent to joints among the divided regions;
A program that executes instructions.

Images