JP2021144358A - Learning apparatus, estimation apparatus, learning method, and program - Google Patents

Abstract

To provide a technology of generating an identification model for estimating a position of a hidden section of a human body.SOLUTION: A learning apparatus 10 includes: a three-dimensional model generation unit 1 which generates a three-dimensional model of a human body; a depth image generation unit 2 which generates a depth image of the three-dimensional model; a first selection unit 3 which selects a depth image of a first specific section area including a specific section from the depth image of the three-dimensional model of the human body in a posture with the specific section hidden by another section; a second selection unit 4 which selects a depth image of a second specific section area including the specific section from a depth image of the three-dimensional model of the human body in a posture with the specific section not hidden by the other section; and a first generation unit 5 which generates a first identification model for identifying the specific section of the human body from a depth image captured by an imaging apparatus on the basis of the depth image of the first specific section area and the depth image of the second specific section area.SELECTED DRAWING: Figure 1

Description

The present invention relates to a learning device, an estimation device, a learning method, and a program.

So far, research has been conducted on recognition processing that estimates the human skeleton such as the head, shoulders, neck, and elbows of human body parts from depth images. In this recognition process, after generating a three-dimensional model of the human body, machine learning is performed to learn what kind of depth distribution each part (head, shoulder, etc.) in the three-dimensional model has, and each part of the human body. The skeletal position is estimated. In this estimation, if only the visible part of the three-dimensional model is machine-learned, for example, if the shoulder cannot be seen by putting the arm in front, the shoulder joint position may not be estimated.

Patent Document 1 discloses a technique that can recognize not only a visible portion that can be seen by an analysis target but also a hidden portion that cannot be seen because it is hidden by another portion by using a depth image. In Patent Document 1, the classification tree is used to recognize whether the analysis target is a visible part or a hidden part. From the result, the depth value of the analysis target part is restored and each part is estimated. This classification tree is trained to improve recognition performance.

Further, Non-Patent Document 1 discloses a technique for estimating the coordinates of the joint position of the human body from a depth image. In Non-Patent Document 1, the joint position is estimated by regression analysis from the position of the similar part classified into each final node in the classification tree for classifying the human body part of the visible part, and the hidden part can also be estimated.

JP-A-2017-208126

Jamie Shotton, Ross Girshick, Andrew Fitzgibbon, Toby Sharp, Mat Cook, Mark Finocchio, Richard Moore, Pushmeet Kohli, Antonio Criminisi, Alex Kipman, Andrew Blake, “Efficient Human Pose Optimization from Single Depth Images” Article in IEEE Transactions on Software Engineering December 2013

However, in Patent Document 1, since both the visible part and the hidden part are input and whether the analysis target is the visible part or the hidden part is recognized from the classification tree, a large amount of data is required for the recognition process. Therefore, there is a risk that the processing will be complicated or the processing time will be prolonged. Further, in Non-Patent Document 1, since the whole body part of the human body is classified into multi-class, even in Non-Patent Document 1, there is a possibility that the processing becomes complicated or the processing time becomes long.

An example of an object of the present invention is to provide a learning device, a learning method, and a program for generating a discriminative model capable of estimating the position of a hidden part of the human body, and an estimation device using the discriminative model.

In order to achieve the above object, the learning device in one aspect of the present invention is
A 3D model generator that generates a 3D model of the human body,
A depth image generator that generates a depth image of the three-dimensional model,
A first selection unit that selects a depth image of a second specific part region including the specific part from a depth image of the three-dimensional model in which the specific part is hidden by another part.
A second selection unit that selects a depth image of a first specific part region including the specific part from a depth image of the three-dimensional model in a posture in which the specific part of the human body is not hidden by another part.
Based on the depth image of the first specific part region and the depth image of the second specific part region, a first identification model for identifying a specific part of the human body is generated from the depth image captured by the imaging device. , 1st generation part,
To be equipped.

In order to achieve the above object, the estimation device in one aspect of the present invention is
A depth image acquisition unit that acquires a depth image from an image pickup device,
A human detection unit that estimates a region including the neck of the human body in the acquired depth image and detects a person from the depth image based on the estimated region.
When the person is detected, a position estimation unit that estimates the position of a specific part of the human body around the neck based on the depth distribution in the region by using the first discriminative model.
With
The first discriminative model is
From the depth image of the three-dimensional model in which the specific part of the human body is hidden by another part, the depth image of the first specific part region including the specific part is selected, and the specific part is hidden by the other part. A depth image of the second specific part region including the specific part is selected from the depth image of the three-dimensional model having no posture, and the depth image of the first specific part region and the depth image of the second specific part region are selected. Is generated based on.

Further, in order to achieve the above object, the learning method in one aspect of the present invention is:
Steps to generate a three-dimensional model of the human body,
Steps to generate a depth image of the 3D model,
A step of selecting a depth image of a first specific part region including the specific part from the depth image of the three-dimensional model in which the specific part of the human body is hidden by another part.
A step of selecting a depth image of a second specific part region including the specific part from the depth image of the three-dimensional model in a posture in which the specific part is not hidden by another part.
Based on the depth image of the first specific part region and the depth image of the second specific part region, a first identification model for identifying a specific part of the human body is generated from the depth image captured by the imaging device. Steps and
To be equipped.

Further, in order to achieve the above object, the program in one aspect of the present invention is:
On the computer
Steps to generate a three-dimensional model of the human body,
Steps to generate a depth image of the 3D model,
A step of selecting a depth image of a first specific part region including the specific part from the depth image of the three-dimensional model in which the specific part of the human body is hidden by another part.
A step of selecting a depth image of a second specific part region including the specific part from the depth image of the three-dimensional model in a posture in which the specific part is not hidden by another part.
Based on the depth image of the first specific part region and the depth image of the second specific part region, a first identification model for identifying a specific part of the human body is generated from the depth image captured by the imaging device. Steps and
Includes instructions to execute.

As described above, according to the present invention, even if a specific part of the human body is hidden by another part in the depth image captured by the imaging device, it is possible to estimate the position of the specific part.

FIG. 1 is a configuration diagram of a learning device. FIG. 2 is a block diagram showing a specific configuration of the learning device. FIG. 3 is a diagram showing a three-dimensional model of a posture in which occlusion occurs at a specific portion. FIG. 4 is a diagram showing a three-dimensional model of a posture in which occlusion does not occur in a specific part. FIG. 5 is a diagram for explaining a depth image of the neck region selected by the third selection unit. FIG. 6 is a diagram for explaining a depth image of the neck region selected by the third selection unit. FIG. 7 is a block diagram showing the configuration of the estimation device. FIG. 8 is a flow chart showing the operation of the learning device. FIG. 9 is a flow chart showing the operation of the learning device. FIG. 10 is a flow chart showing the operation of the estimation device. FIG. 11 is a block diagram showing an example of a computer that realizes a learning device.

Hereinafter, the learning device and the learning method according to the embodiment of the present invention will be described with reference to FIGS. 1 to 11.

[Device configuration]
FIG. 1 is a configuration diagram of the learning device 10.

The learning device 10 is a device for learning an identification model for estimating the position of a specific part of the human body from a depth image (hereinafter referred to as an captured image) captured by the imaging device. The learning device 10 makes it possible to improve the accuracy of estimating the position of a specific part of the human body from the captured image by learning the discriminative model using the generated three-dimensional model of the human body.

The learning device 10 includes a three-dimensional model generation unit 1, a depth image generation unit 2, a first selection unit 3, a second selection unit 4, and a first generation unit 5.

The three-dimensional model generation unit 1 generates a three-dimensional model of the human body.

The depth image generation unit 2 generates a depth image of the generated three-dimensional model.

The first selection unit 3 selects a depth image of a first specific part region including a specific part from a depth image of a three-dimensional model in which a specific part of the human body is hidden by another part. The "specific site" is the head, shoulders or neck of the human body. Further, the "other part" is a part that can be located in front of a specific part when the three-dimensional model of the human body is viewed from one direction. For example, when the "specific part" is the "shoulder", the "other part" corresponds to the "arm", "hand", "elbow" and the like. The posture in which a specific part of the human body is hidden by another part is a state in which the other part is located in front of the specific part when the three-dimensional model of the human body is viewed from one direction.

In the following description, a posture in which a specific part of the human body is hidden by another part may be referred to as a posture in which occlusion occurs in the specific part.

The second selection unit 4 selects a depth image of the second specific part region including the specific part from the depth image of the three-dimensional model in which the specific part of the human body is not hidden by the other part. The posture in which a specific part of the human body is not hidden by another part is a state in which the other part is not located in front of the specific part when the three-dimensional model of the human body is viewed from one direction.

In the following description, a posture in which a specific part of the human body is not hidden by another part may be referred to as a posture in which occlusion does not occur in the specific part.

The first generation unit 5 generates a first identification model for identifying a specific part of the human body from the captured image based on the depth image of the first specific part region and the depth image of the second specific part region.

According to the learning device 10 having this configuration, a first discriminative model is generated from each of a three-dimensional model in which a specific part of the human body is hidden by another part and a three-dimensional model in which the specific part is not hidden. As a result, when a person is imaged with an imaging device and the position of a specific part of the human body is estimated from the captured image, the posture of the person in the captured image is a posture in which the specific part of the human body is hidden by another part. However, even in a posture that is not hidden, the position of the specific portion can be estimated by using the first discriminative model.

Subsequently, the configuration of the learning device 10 will be specifically described with reference to FIGS. 2 to 6.

FIG. 2 is a block diagram showing a specific configuration of the learning device 10. The learning device 10 includes a third selection unit 7 and a second generation unit in addition to the three-dimensional model generation unit 1, the depth image generation unit 2, the first selection unit 3, the second selection unit 4, and the first generation unit 5. 8, a first discriminative model storage unit 6, and a second discriminative model storage unit 9 are provided.

The three-dimensional model generation unit 1 generates a three-dimensional model of the human body. The three-dimensional model may be generated by using motion capture, or may be generated from an image obtained by imaging a human body, and the generation method thereof is not particularly limited.

The depth image generation unit 2 generates a depth image of the three-dimensional model generated by the three-dimensional model generation unit 1. The depth image is an image having distance information indicating the distance from the camera position to each part of the human body.

When the generated three-dimensional model is in a posture in which occlusion occurs in a specific part, the first selection unit 3 selects a depth image of the first specific part region from the depth image of the three-dimensional model.

FIG. 3 is a diagram showing a three-dimensional model of a posture in which occlusion occurs at a specific portion.

The generated three-dimensional model is manually operated, for example, operated by the operator of the learning device 10, or is automatically operated. When the three-dimensional model is moved, a specific part becomes a posture hidden by another part. FIG. 3 shows a posture in which the left shoulder is hidden by the left arm. Since the three-dimensional model is generated, the position of each part of the human body can be specified. When the specific part is the left shoulder, the first selection unit 3 specifies the position of the left shoulder. Then, the first selection unit 3 generates a rectangular first specific site region 31 centered on the position, and selects a depth image of the first specific site region 31.

In the three-dimensional model of FIG. 3, since the left shoulder is hidden by the arm, the depth image of the first specific part region 31 selected by the first selection unit 3 includes the depth image of the "left shoulder" of the human body. Not.

Similarly, the first selection unit 3 identifies the position of the right shoulder, neck, or head hidden in other parts as a result of moving the three-dimensional model for a specific part other than the left shoulder, and determines the specified position. A rectangular first specific site region 31 as a center is generated, and a depth image of the first specific site region 31 is selected. Further, the first selection unit 3 includes not only the three-dimensional model viewed from the front shown in FIG. 3 but also the three-dimensional model viewed from the side or the upper side of the human body, and similarly includes the specific portion. A depth image of the specific site region 31 is selected. In this way, the first selection unit 3 is moved in various ways to generate the first specific part region 31 in the depth image of the three-dimensional model viewed from multiple directions, and the depth image of the first specific part region 31 is generated. Select at any time.

The second selection unit 4 selects the depth image of the second specific part region from the depth image of the three-dimensional model when the generated three-dimensional model is in a posture in which occlusion does not occur in the specific part. ..

FIG. 4 is a diagram showing a three-dimensional model of a posture in which occlusion does not occur in a specific part.

When the three-dimensional model is in a posture in which a specific part is not hidden by another part and the specific part is the left shoulder, the second selection unit 4 specifies the position of the left shoulder and has a rectangular shape centered on that position. A second specific site region 32 is generated, and a depth image of the second specific site region 32 is selected. The size of the second specific site region 32 is preferably the same as the size of the first specific site region 31.

Similarly, the second selection unit 4 specifies the position of the right shoulder, neck, or head in the three-dimensional model of the posture in which occlusion does not occur for a specific part other than the left shoulder, and centers on the specified position. A rectangular second specific site region 32 is generated, and a depth image of the second specific site region 32 is selected. Further, the second selection unit 4 includes not only the three-dimensional model viewed from the front as shown in FIG. 4 but also the third-dimensional model viewed from the side or the upper side of the human body, and similarly includes the specific portion. A depth image of the specific site region 32 is selected. In this way, the second selection unit 4 is moved in various ways to generate a second specific region region 32 in the depth image of the three-dimensional model viewed from multiple directions, and the depth image of the second specific region region 32 is displayed at any time. select.

The first generation unit 5 is the depth distribution of each of the depth image of the first specific site region 31 selected by the first selection unit 3 and the depth image of the second specific site region 32 selected by the second selection unit 4. Is learned as a feature quantity and generated as a first discriminative model.

The first generation unit 5 stores the generated first identification model in the first identification model storage unit 6. The first discriminative model storage unit 6 is a storage device that stores the first discriminative model. The first identification model storage unit 6 may be provided in the learning device 10 or may be provided outside the learning device 10.

The third selection unit 7 selects a depth image of the neck region including the neck of the human body from the depth image of the generated three-dimensional model.

5 and 6 are diagrams for explaining a depth image of the neck region 33 selected by the third selection unit 7. As described above, since the three-dimensional model is generated, the position of each part of the human body is specified. The third selection unit 7 specifies the position of the "neck" in the three-dimensional model, generates a rectangular neck region 33 centered on the position, and selects a depth image of the neck region 33.

The third selection unit 7 selects a depth image of the neck region including the neck of the human body when viewed from all directions of the three-dimensional model, in addition to the state viewed from the front shown in FIG. FIG. 6 shows a three-dimensional model viewed from above the three-dimensional model. In the case of FIG. 6, the head and shoulders of the three-dimensional model can be seen, and the neck is hidden by the head in an occlusion posture. Even in this state, the third selection unit 7 specifies the position of the neck, generates a rectangular neck region 33 centered on the specified position, and selects a depth image of the neck region 33.

The second generation unit 8 generates a second discriminative model based on the depth information of the depth image of the selected neck region 33. Specifically, the second generation unit 8 learns the depth distribution of the depth image of the neck region 33 as a feature amount and generates it as a second discriminative model.

The second generation unit 8 stores the generated second identification model in the second identification model storage unit 9. The second discriminative model storage unit 9 is a storage device that stores the second discriminative model. The second identification model storage unit 9 may be provided in the learning device 10 or may be provided outside the learning device 10.

In this way, the learning device 10 trains the first discriminative model and the second discriminative model by the three-dimensional model. The first discriminative model and the second discriminative model are used in an estimation device that estimates the position of a specific part of the human body from a captured image. The first discriminative model is used when estimating the position of the head, shoulders or neck, which is a specific part of the human body, from the captured image. Further, the second discriminative model is used when detecting a person as to whether or not a person is shown in the captured image. The estimation device will be described below.

FIG. 7 is a block diagram showing the configuration of the estimation device 20.

The estimation device 20 acquires a captured image from the image pickup device 30. The image pickup device 30 is arranged so as to image a person in the space. The image pickup apparatus 30 may be arranged so as to take an image from the front of the human body, or may be arranged so as to take an image from the side or above. The estimation device 20 detects a person from the acquired captured image and estimates the position of the detected specific part of the human body on the two-dimensional coordinates of the head, shoulders, or neck. By estimating the position of the head, shoulders or neck, the joint positions around the head, shoulders or neck can be estimated. By estimating the joint position, it becomes possible to recognize gesture movements such as gestures and to estimate the posture of a person.

The estimation device 20 and the image pickup device 30 may be independent devices and may be connected to each other so as to be capable of data communication, or the estimation device 20 may be provided with the image pickup device 30. When the first identification model storage unit 6 and the second identification model storage unit 9 are provided outside the learning device 10, the estimation device 20 is the first identification model storage unit 6 and the second identification model storage unit. The configuration may be such that data communication is possible only for 9.

The estimation device 20 includes a depth image acquisition unit 21, a human detection unit 22, a position estimation unit 23, and a joint position estimation unit 24.

The depth image acquisition unit 21 acquires an captured image from the imaging device 30.

The human detection unit 22 estimates a region including the neck of the human body with respect to the acquired captured image, and detects a person from the captured image based on the estimated region. First, the person detection unit 22 generates a reduced image of the captured image according to the distance information included in the acquired captured image. For example, the size of a person in a captured image differs depending on whether the image pickup device 30 is located at a short distance or a long distance. Therefore, the person detection unit 22 generates a reduced image with a large reduction ratio from the captured image in the case of a short distance, and generates a reduced image with a small reduction ratio in the case of a long distance from the captured image.

The human detection unit 22 estimates the position of the neck of the human body in the generated reduced image, and generates a region including the estimated position. Here, the person detection unit 22 generates an area by, for example, a process using NMS (Non Maximum Suppression).

The second discriminative model is the depth distribution of the depth image of the neck region 33 including the neck generated from the three-dimensional model as described above. The human detection unit 22 determines whether the estimated region is a region including the neck by comparing the depth distribution of the region estimated in the captured image (specifically, the reduced image) with the second identification model. If it is determined that the area includes the neck, it is specified that a person is shown in the captured image. As a result, the person detection unit 22 detects a person in the captured image.

Here, it is assumed that the horizontal length (cm) of the rectangular size used as the human detection range and the horizontal size (number of pixels) of the neck region 33 are determined, and the angle of view and the resolution of the imaging device are determined. In this case, it is possible to calculate how many horizontal pixels in the captured image correspond to the horizontal length of human detection at the desired distance. When the captured image is reduced so that the number of horizontal pixels matches the horizontal size pixels of the feature amount, the neck region 33 can be fixed and a person can be detected from the reduced image.

When a person is detected in the captured image, the position estimation unit 23 determines the position of the head, shoulders, or neck, which is a specific part of the human body around the neck, based on the depth distribution in the region estimated by the person detection unit 22. Is estimated using the first discriminative model. The first discriminative model is a discriminative model generated from a three-dimensional model of a posture in which occlusion has occurred and a three-dimensional model of a posture in which occlusion has not occurred. By using the first identification model, the position estimation unit 23 can estimate the position of a specific part regardless of whether or not occlusion has occurred in the captured image. For example, even when the posture of the person in the captured image is a posture in which the left shoulder is hidden by the left hand as shown in FIG. 3, the position estimation unit 23 uses the depth distribution of the region estimated by the person detection unit 22. However, it is possible to estimate the position of the left shoulder of a person in the captured image because it is similar to the learned depth distribution of the first discriminative model generated from the three-dimensional model of FIG.

The position estimation unit 23 assigns a circular label to the estimated position in the captured image. For example, the position estimation unit 23 assigns a label indicating the left shoulder to the estimated position of the left shoulder.

The joint position estimation unit 24 estimates the joint position from the label given by the position estimation unit 23. The joint position estimation unit 24 estimates, for example, a region between the position labeled on the left shoulder and the position labeled on the neck as the position of the right shoulder joint.

When occlusion occurs between specific parts on the captured image, for example, when the neck is hidden by the shoulder, the position estimation unit 23 labels the overlapping area as an overlapping area of the two parts. Is given. Then, the joint position estimation unit 24 estimates the joint position by combining the region where the specific parts do not overlap and the region where the specific parts overlap.

[Device operation]
Next, the operation of the learning device 10 in the present embodiment will be described with reference to FIGS. 8 and 9. 8 and 9 are flow charts showing the operation of the learning device 10. In the following description, FIGS. 1 to 7 will be referred to as appropriate. Further, in the present embodiment, the learning method is implemented by operating the learning device 10. Therefore, the description of the learning method in the present embodiment is replaced with the following description of the operation of the learning device 10.

FIG. 8 is a flow chart at the time of generating the first discriminative model. The three-dimensional model generation unit 1 generates a three-dimensional model (S1). The depth image generation unit 2 generates a depth image of the generated three-dimensional model (S2). The three-dimensional model is moved to a posture in which occlusion occurs and a posture in which occlusion does not occur. The first selection unit 3 selects the depth image of the first specific site region 31 from the depth image of the three-dimensional model in which occlusion has occurred (S3). Specifically, the first selection unit 3 specifies the position of a specific part of the human body in the depth image of the three-dimensional model. Then, the first selection unit 3 generates a rectangular first specific site region 31 centered on the position, and selects a depth image of the first specific site region 31. The second selection unit 4 selects the depth image of the second specific part region 32 from the depth image of the three-dimensional model in which occlusion has not occurred (S4). The first generation unit 5 learns the depth distribution of the selected depth image of the first specific site region 31 and the depth image of the depth image of the second specific site region 32 as feature quantities, and obtains the first discriminative model. Generate (S5). The first generation unit 5 stores the generated first identification model in the first identification model storage unit 6 (S6).

FIG. 9 is a flow chart at the time of generating the second discriminative model. The third selection unit 7 selects a depth image of the neck region 33 including the neck of the human body from the depth image of the three-dimensional model generated in S1 of FIG. 8 (S8). The second generation unit 8 learns the depth distribution of the depth image of the neck region 33 as a feature amount, generates a second discrimination model (S9), and stores it in the second discrimination model storage unit 9 (S10).

The learning device 10 repeatedly executes the processes S3 to S6 of FIG. 8 and the processes S8 to S10 of FIG. 9 while moving the three-dimensional model. Further, the learning device 10 performs each process on the three-dimensional model viewed from all directions.

Next, the operation of the estimation device 20 will be described. FIG. 10 is a flow chart showing the operation of the estimation device 20.

The depth image acquisition unit 21 acquires an captured image from the imaging device 30 (S21). The human detection unit 22 generates a reduced image of the acquired captured image (S22). At this time, the person detection unit 22 generates a reduced image of the captured image according to the distance information included in the captured image. The person detection unit 22 generates a reduced image with a large reduction ratio in the case of a short distance from the captured image, and generates a reduced image with a small reduction ratio in the case of a long distance from the captured image.

The person detection unit 22 detects a person from the generated reduced image (S23). The human detection unit 22 estimates a region including the neck of the human body with respect to the generated reduced image, and compares the depth distribution of the depth image of that region with the second discriminative model. Then, the person detection unit 22 determines whether the estimated region is a region including the neck by comparison, and if it is determined that the region includes the neck, the person detection unit 22 identifies that the captured image shows a person. ..

When a person is detected in the captured image, the position estimation unit 23 determines the position of the head, shoulders, or neck, which is a specific part of the human body around the neck, based on the depth distribution of the region estimated in S23. Estimate using the 1 discriminative model and the 2nd discriminative model (S24). Then, the position estimation unit 23 assigns a circular label to the estimated position (S25). The joint position estimation unit 24 estimates the joint position based on the given label (S26).

As described above, the learning device 10 generates the first discriminative model and the second discriminative model, so that the estimation device 20 estimates the position of a specific part of the person in the captured image regardless of the occurrence of occlusion. can do. Further, the learning device 10 generates a second discriminative model, and the estimation device 20 uses the second discriminative model to detect a person centered on the neck, thereby depending on the imaging angle of the imaging device 30. Instead, it is possible to detect a person in the captured image. Then, the learning device 10 generates the first discriminative model and the second discriminative model from the three-dimensional model viewed from all directions, so that the specific part of the human body in the captured image is independent of the imaging angle of the imaging device 30. The position can be estimated.

[program]
The program in this embodiment may be any program that causes a computer to execute each step shown in FIG. By installing this program on a computer and executing it, the learning device and the learning method in the present embodiment can be realized. In this case, the computer processor is a three-dimensional model generation unit 1, a depth image generation unit 2, a first selection unit 3, a second selection unit 4, a first generation unit 5, a third selection unit 7, and a second generation unit. It functions as 8 and performs processing.

Moreover, the program in this embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer has a three-dimensional model generation unit 1, a depth image generation unit 2, a first selection unit 3, a second selection unit 4, a first generation unit 5, a third selection unit 7, and so on. It may function as any of the second generation units 8.

Further, in the present embodiment, the first identification model storage unit 6 and the second identification model storage unit 9 may be realized by a storage device such as a hard disk provided in the computer, or may be stored in another computer. It may be realized by the device.

Further, the estimation device according to the present embodiment can be realized by installing and executing a program for executing each step shown in FIG. 10 on a computer. In this case, the computer processor functions as a depth image acquisition unit 21, a person detection unit 22, a position estimation unit 23, and a joint position estimation unit 24, and performs processing. This program may be executed by a computer system built by a plurality of computers. In this case, for example, each computer may function as any of the depth image acquisition unit 21, the person detection unit 22, the position estimation unit 23, and the joint position estimation unit 24, respectively.

Examples of computers include smartphones and tablet terminal devices in addition to general-purpose PCs.

[Physical configuration]
Here, a computer that realizes the learning device 10 by executing the program in the present embodiment will be described with reference to FIG. FIG. 11 is a block diagram showing an example of a computer that realizes the learning device 10.

As shown in FIG. 11, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. And. Each of these parts is connected to each other via a bus 121 so as to be capable of data communication.

Further, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to the CPU 111 or in place of the CPU 111. In this aspect, the GPU or FPGA can execute the program in the embodiment.

The CPU 111 executes various operations by expanding the program in the embodiment composed of the code group stored in the storage device 113 into the main memory 112 and executing each code in a predetermined order. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

Further, the program in the embodiment is provided in a state of being stored in a computer-readable recording medium 120. The program in the present embodiment may be distributed on the Internet connected via the communication interface 117.

Further, specific examples of the storage device 113 include a semiconductor storage device such as a flash memory in addition to a hard disk drive. The input interface 114 mediates data transmission between the CPU 111 and an input device 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads a program from the recording medium 120, and writes a processing result in the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and another computer.

Specific examples of the recording medium 120 include a general-purpose semiconductor storage device such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), a magnetic recording medium such as a flexible disk, or a CD-. Examples include optical recording media such as ROM (Compact Disk Read Only Memory).

The learning device 10 in the present embodiment can also be realized by using hardware corresponding to each part instead of the computer in which the program is installed. Further, the learning device 10 may be partially realized by a program and the rest may be realized by hardware.

Since the computer that realizes the estimation device 20 is the same as that in FIG. 11, the description thereof will be omitted.

A part or all of the above-described embodiments can be expressed by the following descriptions (Appendix 1) to (Appendix 10), but the present invention is not limited to the following description.

(Appendix 1)
A 3D model generator that generates a 3D model of the human body,
A depth image generator that generates a depth image of the three-dimensional model,
A first selection unit that selects a depth image of a second specific part region including the specific part from a depth image of the three-dimensional model in which the specific part is hidden by another part.
A second selection unit that selects a depth image of a first specific part region including the specific part from a depth image of the three-dimensional model in a posture in which the specific part of the human body is not hidden by another part.
Based on the depth image of the first specific part region and the depth image of the second specific part region, a first identification model for identifying a specific part of the human body is generated from the depth image captured by the imaging device. , 1st generation part,
A learning device equipped with.

(Appendix 2)
The learning device according to Appendix 1,
The specific site is the head, shoulders or neck of the human body.
The first generation unit learns the depth distribution of the depth image as a feature amount and generates the first discriminative model.
Learning device.

(Appendix 3)
The learning device according to Appendix 1 or Appendix 2.
A third selection unit that selects a depth image of the neck region including the neck of the human body from the depth image of the three-dimensional model.
A second generation unit that generates a second identification model for identifying the presence or absence of a human body from a depth image captured by an imaging device based on the depth information of the selected depth image of the neck region.
A learning device equipped with.

(Appendix 4)
The learning device according to Appendix 3,
The second generation unit learns the depth distribution of the depth image of the neck region as a feature amount to generate the second discriminative model.
Learning device.

(Appendix 5)
A depth image acquisition unit that acquires a depth image from an image pickup device,
A human detection unit that estimates a region including the neck of the human body in the acquired depth image and detects a person from the depth image based on the estimated region.
When the person is detected, a position estimation unit that estimates the position of a specific part of the human body around the neck using a discriminative model based on the depth distribution in the region.
With
The discriminative model is
From the depth image of the three-dimensional model in which the specific part of the human body is hidden by another part, the depth image of the first specific part region including the specific part is selected, and the specific part is hidden by the other part. A depth image of the second specific part region including the specific part is selected from the depth image of the three-dimensional model having no posture, and the depth image of the first specific part region and the depth image of the second specific part region are selected. Generated based on,
Estimator.

(Appendix 6)
The estimation device according to Appendix 5, which is the estimation device.
The human body detection unit
A depth image of a region including the neck of the human body is selected from the depth image of the three-dimensional model, and the person is detected using the second identification model generated from the selected depth image.
Estimator.

(Appendix 7)
Steps to generate a three-dimensional model of the human body,
Steps to generate a depth image of the 3D model,
A step of selecting a depth image of a first specific part region including the specific part from the depth image of the three-dimensional model in which the specific part of the human body is hidden by another part.
A step of selecting a depth image of a second specific part region including the specific part from the depth image of the three-dimensional model in a posture in which the specific part is not hidden by another part.
Based on the depth image of the first specific part region and the depth image of the second specific part region, a first identification model for identifying a specific part of the human body is generated from the depth image captured by the imaging device. Steps and
A learning method that includes.

(Appendix 8)
The learning method described in Appendix 7
The specific site is the head, shoulders or neck of the human body.
In the step of generating the first discriminative model, the depth distribution of the depth image is learned as a feature amount to generate the first discriminative model.
Learning method.

(Appendix 9)
On the computer
Steps to generate a three-dimensional model of the human body,
Steps to generate a depth image of the 3D model,
A step of selecting a depth image of a first specific part region including the specific part from the depth image of the three-dimensional model in which the specific part of the human body is hidden by another part.
A step of selecting a depth image of a second specific part region including the specific part from the depth image of the three-dimensional model in a posture in which the specific part is not hidden by another part.
Based on the depth image of the first specific part region and the depth image of the second specific part region, a first identification model for identifying a specific part of the human body is generated from the depth image captured by the imaging device. Steps and
A program that contains instructions to execute.

(Appendix 10)
The program described in Appendix 9
The specific site is the head, shoulders or neck of the human body.
In the step of generating the first discriminative model, the depth distribution of the depth image is learned as a feature amount to generate the first discriminative model.
program.

1 3D model generation unit 2 Depth image generation unit 3 1st selection unit 4 2nd selection unit 5 1st generation unit 6 1st identification model storage unit 7 3rd selection unit 8 2nd generation unit 9 2nd identification model storage unit 10 Learning device 20 Estimator 21 Depth image acquisition unit 22 Person detection unit 23 Position estimation unit 24 Joint position estimation unit 30 Imaging device 110 Computer 111 CPU
112 Main memory 113 Storage device 114 Input interface 115 Display controller 116 Data reader / writer 117 Communication interface 118 Input device 119 Display device 120 Recording medium 121 Bus

Claims (10)

A 3D model generator that generates a 3D model of the human body,
A depth image generator that generates a depth image of the three-dimensional model,
A first selection unit that selects a depth image of a second specific part region including the specific part from a depth image of the three-dimensional model in which the specific part is hidden by another part.
A second selection unit that selects a depth image of a first specific part region including the specific part from a depth image of the three-dimensional model in a posture in which the specific part of the human body is not hidden by another part.
Based on the depth image of the first specific part region and the depth image of the second specific part region, a first identification model for identifying a specific part of the human body is generated from the depth image captured by the imaging device. , 1st generation part,
A learning device equipped with. The learning device according to claim 1.
The specific site is the head, shoulders or neck of the human body.
The first generation unit learns the depth distribution of the depth image as a feature amount and generates the first discriminative model.
Learning device. The learning device according to claim 1 or 2.
A third selection unit that selects a depth image of the neck region including the neck of the human body from the depth image of the three-dimensional model.
A second generation unit that generates a second identification model for identifying the presence or absence of a human body from a depth image captured by an imaging device based on the depth information of the selected depth image of the neck region.
A learning device equipped with. The learning device according to claim 3.
The second generation unit learns the depth distribution of the depth image of the neck region as a feature amount to generate the second discriminative model.
Learning device. A depth image acquisition unit that acquires a depth image from an image pickup device,
A human detection unit that estimates a region including the neck of the human body in the acquired depth image and detects a person from the depth image based on the estimated region.
When the person is detected, a position estimation unit that estimates the position of a specific part of the human body around the neck using a discriminative model based on the depth distribution in the region.
With
The discriminative model is
From the depth image of the three-dimensional model in which the specific part of the human body is hidden by another part, the depth image of the first specific part region including the specific part is selected, and the specific part is hidden by the other part. A depth image of the second specific part region including the specific part is selected from the depth image of the three-dimensional model having no posture, and the depth image of the first specific part region and the depth image of the second specific part region are selected. Generated based on,
Estimator. The estimation device according to claim 5.
The human body detection unit
A depth image of a region including the neck of the human body is selected from the depth image of the three-dimensional model, and the person is detected using the second identification model generated from the selected depth image.
Estimator. Steps to generate a three-dimensional model of the human body,
Steps to generate a depth image of the 3D model,
A step of selecting a depth image of a first specific part region including the specific part from the depth image of the three-dimensional model in which the specific part of the human body is hidden by another part.
A step of selecting a depth image of a second specific part region including the specific part from the depth image of the three-dimensional model in a posture in which the specific part is not hidden by another part.
Based on the depth image of the first specific part region and the depth image of the second specific part region, a first identification model for identifying a specific part of the human body is generated from the depth image captured by the imaging device. Steps and
A learning method that includes. The learning method according to claim 7.
The specific site is the head, shoulders or neck of the human body.
In the step of generating the first discriminative model, the depth distribution of the depth image is learned as a feature amount to generate the first discriminative model.
Learning method. On the computer
Steps to generate a three-dimensional model of the human body,
Steps to generate a depth image of the 3D model,
A step of selecting a depth image of a first specific part region including the specific part from the depth image of the three-dimensional model in which the specific part of the human body is hidden by another part.
A step of selecting a depth image of a second specific part region including the specific part from the depth image of the three-dimensional model in a posture in which the specific part is not hidden by another part.
Based on the depth image of the first specific part region and the depth image of the second specific part region, a first identification model for identifying a specific part of the human body is generated from the depth image captured by the imaging device. Steps and
A program that contains instructions to execute. The program according to claim 9.
The specific site is the head, shoulders or neck of the human body.
In the step of generating the first discriminative model, the depth distribution of the depth image is learned as a feature amount to generate the first discriminative model.
program.

Images