JP2021092891A - Information processor, information processing method, image processing system, and program - Google Patents

Abstract

To provide an information processor that can accurately estimate positions of human body parts for multiple subjects even in games played on large fields such as soccer and rugby.SOLUTION: A server 270 as an information processor comprises: an estimating image generation part 273 that acquires shape data indicating a three-dimensional shape of a person object generated from a plurality of captured images obtained by imaging the person object from a plurality of directions, and generates a distance image indicating a distance to the person object in accordance with a predetermined viewpoint condition based on the acquired shape data; and a human body part estimation part 274 that estimates a position of the human body part in the person object using the distance image and a trained model for estimating the position of the human body part obtained through prior training.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to a technique for estimating the position of a part of the human body from a distance image.

In recent years, the movement of a player's body has been acquired and analyzed for the purpose of improving sports skills. In team competitions such as soccer and rugby, it is required to analyze the overall formation in addition to the movement of each player. As a technique for analyzing the movement of a player, a distance image of a subject (player) is acquired by a so-called depth camera, and a trained model for estimating a joint position learned in advance is used to estimate the joint position of the player. A method has been proposed (Patent Document 1).

Japanese Unexamined Patent Publication No. 2015-228188 WO2017 / 187641 Japanese Unexamined Patent Publication No. 2017-21128

Shotton, Jamie. "Real-time human pose recognition in parts from single depth images", IEEE Computer Vision and Pattern Recognition (CVPR), 2011. Page (s): 1297 --1304 Alireza Shafaei, James J. Little “Real-Time Human Motion Capture with Multiple Depth Cameras”, IEEE 2016 13th Conference on Computer and Robot Vision (CRV), Page (s): 24-31

In the case of the method shown in Patent Document 1, the estimation accuracy of the joint position and the like is greatly affected by the performance of the image sensor mounted on the depth camera, the imageable distance, and the resolution. In addition, the effective imaging range of a general depth camera is several cm to a dozen meters, but if you try to estimate the joint position of each player for field competitions such as soccer and rugby, which have a wider imaging range. , It is necessary to install a depth camera in the field as well. However, it is actually difficult to install a large number of depth cameras in a vast field.

An object of the present invention is to make it possible to accurately estimate a part of the human body that is a subject with a simple configuration.

The information processing apparatus according to the present disclosure is an acquisition means for acquiring shape data indicating a three-dimensional shape of the person object, which is generated by using a plurality of captured images obtained by imaging the person object from a plurality of directions. , A generation means for generating a distance image showing a distance to the person object when a predetermined viewpoint condition is followed based on the shape data acquired by the acquisition means, and a part of the human body obtained by performing learning in advance. It is characterized in that it includes a part estimation means for estimating the position of a human body part in the person object based on the distance image generated by the generation means using a trained model for estimating the position of. ..

According to the present invention, it is possible to accurately estimate a part of the human body that is a subject with a simple configuration.

The figure which shows an example of the structure of an image processing system The figure which shows the arrangement example of the image pickup module Block diagram showing the functional configuration of the server according to the first embodiment (A) is a diagram showing an example of shape data, and (b) is a diagram showing an example of individual shape data. The figure explaining the separation process for the shape data (A) is a diagram showing an example of an estimation distance image, and (b) is a diagram showing an example of human body part information. (A) and (b) are diagrams showing an example of a color-coded image as teacher data. (A) is an explanatory diagram of input / output in the learning phase, and (b) is an explanatory diagram of input / output in the estimation phase. (A) is a diagram for explaining how a distance image is acquired by a conventional depth camera, and (b) is a diagram for explaining an outline at the time of creating a learning distance image according to the present disclosure. Block diagram showing the functional configuration of the server according to the second embodiment Diagram illustrating how to identify the direction in which a person object is facing A block diagram showing a functional configuration of a server according to a modified example of the second embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the following embodiments do not limit the present invention, and not all combinations of features described in the present embodiment are essential for the means for solving the present invention. The same configuration will be described with the same reference numerals.

[Embodiment 1]
(Basic system configuration and operation)
FIG. 1 is a diagram showing an example of a configuration of an image processing system that estimates the joint position of a subject and the like. The image processing system 100 includes imaging modules 110a to 110p, a database (DB) 250, a server 270, a control device 300, a switching hub 180, and an end user terminal 190. That is, the image processing system 100 has three functional domains, that is, a video acquisition domain, a data storage domain, and a video generation domain. The video acquisition domain includes the imaging modules 110a to 110p, the data storage domain includes the DB 250 and the server 270, and the video generation domain includes the control device 300 and the end user terminal 190.

The control device 300 manages the operating state and controls the parameter setting for each block constituting the image processing system 100 through the network.

First, an operation of transmitting 16 sets of captured images of the imaging modules 110a to 110p from the imaging module 110p to the server 270 will be described. The image pickup modules 110a to 110p each have one camera 112a to 112p. In the following, 16 sets of systems from the imaging modules 110a to 110p may not be distinguished and may be simply referred to as “imaging module 110”. Similarly, the devices in each imaging module 110 may be described as "camera 112" and "camera adapter 120". The cameras 112a to 112p perform imaging in synchronization with each other with high accuracy based on the synchronization signal from the control device 300. The imaging modules 110a to 110p are installed so as to surround the ground or the like, for example, as shown in FIG. The number of imaging modules 110 is 16 sets, but this is just an example and is not limited to this.

The imaging modules 110a to 110p are connected by a daisy chain. The connection form is arbitrary. For example, a star-type network configuration may be obtained in which the imaging modules 110a to 110p are connected to the switching hub 180 and data is transmitted and received between the imaging modules 110 via the switching hub 180.

In the present embodiment, the camera 112 and the camera adapter 120 are separated from each other, but they may be integrated in the same housing. The captured image obtained by the camera 112a in the image pickup module 110a is transmitted to the camera adapter 120b of the image pickup module 110b after being subjected to predetermined image processing such as foreground background separation in the camera adapter 120a. Similarly, the image pickup module 110b transmits the captured image obtained by the camera 112b to the image pickup module 110c together with the captured image acquired from the image pickup module 110a. By continuing such an operation, 16 sets of captured images (including foreground images) are transmitted from the imaging module 110p to the switching hub 180, and then transmitted to the server 270. In the following description, the subject will be referred to as a "person object".

(Functional configuration of server)
FIG. 3 is a block diagram showing a functional configuration of the server 270 as an information processing device for estimating joint positions and the like of a person object according to the present embodiment. The server 270 has a three-dimensional shape derivation unit 271, an object separation unit 272, an image generation unit for estimation 273, a human body part estimation unit 274, and a learning unit 275. Further, the server 270 has various hardware included in a general computer, that is, a CPU, RAM, ROM, an external I / F, a large-capacity storage area, and the like. Then, the CPU executes a predetermined program stored in the RAM or ROM to realize the functions of each part shown in FIG. In the present embodiment, the functions of the above parts are realized by one server 270, but the functions of the above parts may be distributed and realized by a plurality of servers. For example, it may be divided into a server that has two functions of a three-dimensional shape derivation unit 271 and an object separation unit 272, and a server that has three functions of an image generation unit 273 for estimation, a human body part estimation unit 274, and a learning unit 275. Good.

The data of the plurality of captured images (captured images corresponding to a plurality of viewpoints by synchronous shooting) input to the server 270 are first input to the three-dimensional shape derivation unit 271. The three-dimensional shape derivation unit 271 extracts the silhouette of the person object for each captured image having a different viewpoint, and generates shape data representing the three-dimensional shape of the person object by the visual volume crossing method or the like using the obtained silhouette image. To do. Such shape data is also generally referred to as a three-dimensional model. In the present embodiment, it is assumed that a soccer or rugby game in which a plurality of players move around on a vast field as a shooting scene, and each of the captured images includes a plurality of players. That is, the shape data generated by the three-dimensional shape derivation unit 271 includes a portion representing the three-dimensional shape of each of the plurality of person objects. FIG. 4A shows an example of shape data generated by the three-dimensional shape derivation unit 271. As shown in FIG. 4A, in the shape data, the three-dimensional shape of each person object is represented as a set (lump) of unit cubes called, for example, voxels. That is, the shape data generated by the three-dimensional shape derivation unit 271 includes a plurality of voxel groups corresponding to a plurality of athletes. As the three-dimensional shape expression format, another format such as a point cloud format or a polygon format may be used instead of the voxel format.

The object separation unit 272 performs separation processing for cutting out an area corresponding to each player from the shape data generated by the three-dimensional shape derivation unit 271, and individual shape data representing the three-dimensional shape of each person object (hereinafter referred to as , Called "individual shape data"). FIG. 5 is a diagram for explaining the separation process for the shape data. In FIG. 5, the alternate long and short dash line rectangle 501 shows the shape data before the separation process generated by the three-dimensional shape derivation unit 271. Now, the shape data 501 has voxel groups 502 and 503 corresponding to two athletes. At this time, there is no association between voxels, and there is no distinction between which voxel is which voxel of which person object. Therefore, the distance between each voxel and another voxel is obtained, and the voxels that are in contact with each other (that is, the distance is zero) or are within a predetermined distance are identified as voxels belonging to the same person object. I do. For example, in the example of FIG. 5, when the voxel of interest is voxel 504, it is determined that each voxel constituting the voxel group 503 belongs to the same person object. On the other hand, it is determined that each voxel constituting the voxel group 502, which is a certain distance or more from any voxel constituting the voxel group 503, belongs to a different person object. By performing such separation processing, individual shape data corresponding to each person object is generated. The separation method is not limited to the above example. For example, a method may be used in which the distance between regions corresponding to each voxel group is obtained on a two-dimensional image obtained by projecting shape data onto a virtual viewpoint, and the distance is separated when the distance is a certain distance. FIG. 4B shows an example of individual shape data generated by separating the portion of the voxel group 401 in the shape data shown in FIG. 4A. The individual shape data for each person object obtained in this way is sent to the estimation image generation unit 273.

The estimation image generation unit 273 is a distance image for estimating the position of a human body part, which indicates the distance to a human object when a predetermined viewpoint condition is followed, based on the individual shape data received from the object separation unit 272. Generate a distance image for estimation). FIG. 6A shows an example of an estimation distance image generated from the individual shape data of FIG. 4B. In the case of the distance image of FIG. 6A, the closer to white, the closer the distance (front side), and the closer to black, the farther the distance (back side). The larger the number of estimation distance images generated (that is, the larger the number of viewpoints), the higher the estimation accuracy in the later estimation phase of the human body part. The predetermined "viewpoint condition" will be described later. The data of one or more estimation distance images generated for each person object is sent to the human body part estimation unit 274.

The human body part estimation unit 274 receives one or a plurality of estimation distance images received from the estimation image generation unit 273 as input, and uses a learned model for estimating the position of each part of the human body provided by the learning unit 275. It is used to estimate the joint position and the like for each human object. The estimation result by the human body part estimation unit 274 is information representing the positions of each part of the human body such as the shoulder, elbow, knee, waist, wrist, and ankle described above by points on the three-dimensional coordinates (hereinafter, "human body part"). It is called "information"). FIG. 6B shows an example of human body part information output as an estimation result based on the estimation distance image of FIG. 6A. It should be noted that a plurality of human body part estimation units 274 may be provided, and a plurality of individual shape data cut out for each person object may be processed in parallel to speed up the processing.

The learning unit 275 uses a learning distance image showing the distance to each person object created in advance by CG or the like as input data, and color-codes each part of the human body as shown in FIGS. 7A and 7B. Machine learning is performed using images (color-coded images) as teacher data. The algorithm used for machine learning is not particularly limited, and the random forest described in Non-Patent Document 1 and the CNN described in Non-Patent Document 2 can be applied. By this machine learning, a trained model used for estimating the joint position of a person object is generated. The generated trained model is provided to the human body part estimation unit 274.

In the present embodiment, the learning unit 275 is provided in the server 270, but machine learning is performed by an external device outside the system, and the learned model as the learning result is held by the human body part estimation unit 274. It may be configured to be stored.

(Learning and estimation)
FIG. 8A is an explanatory diagram of input / output in the learning phase of the present embodiment. The input data X_t is the data of the learning distance image generated by the learning unit 275. Then, in the learning model, each part of the human body is predicted by paying attention to information that can be extracted from the learning distance image, for example, the distribution of pixel values and the shape of a human object. Specifically, as described in Patent Document 2, first, learning is performed to discriminate a part of the human body in the human object reflected in the input learning distance image by using the color-coded image of the human body part. .. Next, learning is performed to predict the joint positions such as elbows and shoulders by obtaining the three-dimensional center of gravity of the pixels for each of the identified parts. Then, repeated learning is performed so that the amount of deviation L between the predicted joint position and the like and each part of the human body specified by the color-coded image as the teacher data T is minimized.

FIG. 8B is an explanatory diagram of input / output in the estimation phase of the present embodiment. The input data X_i is the data of the estimation distance image generated by the estimation image generation unit 273. The output data Y_i obtained as a result of estimation using the trained model provided by the learning unit 275 is the above-mentioned human body part information.

(About "viewpoint conditions" when generating a distance image for estimation)
The predetermined "viewpoint condition" applied when the distance image generation unit 273 generates the estimation distance image is set by user input via an operation unit (not shown) or by reading the viewpoint information prepared in advance. Will be done. Then, it is desirable that the viewpoint condition at this time matches the viewpoint adopted at the time of creating the learning distance image used in the learning phase as much as possible. The reason will be described below.

Before going into a specific explanation, it is confirmed that a distance image is acquired using a conventional depth camera and the joint position and the like are estimated from the obtained distance image. FIG. 9A is a diagram illustrating how a distance image is obtained by taking a picture with a conventional depth camera. FIG. 9A shows a state in which the player 901 is photographed by two depth cameras 902 and 903 installed at different positions. Then, in the distance image 904 obtained by the depth camera 902 installed at a position close to the player 901, only the upper body portion of the player 901 appears. On the other hand, in the distance image 905 obtained by the depth camera 903 installed at a high position away from the player 901, the whole body of the player 901 appears small. As described above, in the distance image using the conventional depth camera, the distance image obtained by various factors such as the distance to the person object, the installation height of the camera, the direction in which the person object is facing, and the optical characteristics of the camera. The contents of are very different. In other words, the size and appearance of the person object that moves to the distance image greatly depends on the performance of the depth camera, the installation environment, and the like. Then, if it is attempted to obtain a highly reliable trained model by learning to estimate the joint position etc. based on the distance images of various contents obtained under different installation environments, it is necessary for that purpose. The amount of learning data is enormous.

Based on the above problems, in the present embodiment, certain restrictions are set on the viewpoint conditions when creating a distance image as input data in the learning phase by CG or the like. FIG. 9B is a diagram illustrating an outline at the time of creating a learning distance image according to the present embodiment. In the case of FIG. 9B, it is assumed that there is a pseudo camera 911 at a position 5 m away from the person object 910 and at a horizontal position from a height of 1.5 m above the ground, and the image is viewed from there at an angle of view of 45 degrees. It is a figure in the case of creating a distance image in the case of CG. Furthermore, optical characteristics such as resolution are also determined in advance. Then, a learning distance image is created from a plurality of directions in which 360 degrees are equally divided (for example, in the case of 12 equal divisions, 12 directions in increments of 30 degrees) centered on the person object 910. In this way, after fixing the pseudo viewpoint when creating with CG or the like, for example, a learning distance image is created by an external device (PC or the like) and provided to the learning unit 275 as input data. By performing learning using the learning distance image created according to the viewpoint condition fixed in this way, a trained model with high accuracy can be obtained.

In the present embodiment, by fixing the viewpoint condition at the time of creating the distance image for learning, it takes time and effort to create a variation of the trained model when viewed from all viewpoints with different distances and heights in the learning phase. I'm omitting it. That is, learning needs to be performed only with the learning distance image when the person object is viewed under a certain viewpoint condition as shown in FIG. 9B described above. As a result, it is possible to suppress learning costs such as preparation of learning data and time required for learning. Further, since the viewpoint condition is fixed, the trained model finally obtained can be a highly accurate model with less noise. Therefore, it also leads to improvement of accuracy in the estimation phase using the trained model.

<Modification example>
In the present embodiment, the separation process of cutting out the individual shape data corresponding to each person object from the shape data including the voxel group of a plurality of person objects is performed, but it is not essential to perform the separation process independently. The three-dimensional shape derivation unit 271 may be configured to collectively perform processing from shape data including three-dimensional shapes of a plurality of person objects to individual shape data for each person object.

Further, it is not always necessary to separate the individual shape data for each person object. For example, in the learning phase, machine learning may be performed using a learning distance image including a plurality of person objects. In this case, the position of the human body part is estimated at the same time for each of the plurality of person objects by using the estimation distance image including the plurality of person objects and the trained model corresponding to the plurality of person objects.

In the present embodiment, a highly accurate distance image is generated based on shape data representing a three-dimensional shape of a person object generated from a plurality of captured images acquired by using a normal imaging means. Then, the joint position of the human object shown in the captured image is estimated by using the learned model for estimating the position of the human body part obtained by performing machine learning in advance. As a result, even in a shooting scene where it is difficult to properly install a dedicated depth camera and multiple athletes move freely over a wide range, the position of each part of the human body such as athletes can be estimated with high accuracy. It becomes possible.

[Embodiment 2]
Next, a mode for improving the position estimation accuracy of the human body part by first specifying the direction in which the person object is facing and generating an estimation distance image according to the specified direction will be described as the second embodiment. .. The contents common to the first embodiment such as the basic configuration of the image processing system will be omitted, and the functional configuration of the server 270, which is a difference, will be described below.

(Functional configuration of server)
FIG. 10 is a block diagram showing a functional configuration of the server 270'as an information processing device for estimating joint positions and the like of a person object according to the present embodiment. The major difference from the first embodiment is that it has an object orientation estimation unit 1001 that estimates the orientation of a person object, and a second learning unit 1002 that performs machine learning for that purpose and generates a trained model. Is. Further, it is different from the first embodiment in that it has the second estimation image generation unit 1002. Since the blocks having the same name as the server 270 of the first embodiment have the same functions and operations, the description thereof will be omitted. Further, since the functions and operations of the first estimation image generation unit 273 and the first learning unit 275 in FIG. 10 are the same as those of the estimation image generation unit 273 and the learning unit 275 of the first embodiment, the same reference numerals are given. There is.

The object orientation estimation unit 1001 uses the estimation distance image generated by the first estimation image generation unit 273 and the learned model provided by the second learning unit 1002, and the body of each person object is oriented. Estimate the direction. FIG. 11 is a diagram illustrating a method of identifying a direction in which a player is facing when the imaging space is a soccer field. In the example of FIG. 11, the orientation of the athlete's body is identified by the rotation angle when the center of the field (z-axis) shown in the two-dimensional plane of xy is the central axis. Now, the x direction is the reference angle (0 degree). For example, if the player is facing the direction of the goal on the right side, it is "facing 0 degree", and the player is facing the direction of the goal on the left side. In the case, it is "facing 180 degrees".

Although the example of identifying the orientation on the two-dimensional plane has been described here, the orientation in the three-dimensional space including the z-axis may be identified. Such rotation angle information is output to the second estimation image generation unit 1003 as information indicating the direction in which the body of the person object is facing (hereinafter, referred to as "body direction information"). The method of estimating the orientation is not limited to the method based on the estimation distance image generated from the individual shape data. For example, the moving direction of the person object may be obtained from the difference between the current individual shape data and the individual shape data at a certain time in the past, and the moving direction may be estimated to be the direction in which the human body object is facing. Further, instead of using the estimation distance image generated from the individual shape data, the orientation may be estimated using a trained model that estimates the direction in which the person object is facing from the captured image data.

The second learning unit 1002 performs machine learning using the distance images of the person objects facing different postures and directions as input data and the information of the direction of the body of the person object as the teacher data, and estimates the direction of the human body. Generate a trained model for. Similar to the first learning unit 275 of the first embodiment, the distance image as input data when performing this machine learning is only a distance image according to a predetermined viewpoint condition in which the direction in which the human body is facing can be easily estimated. It is desirable to use.

The second estimation image generation unit 1003 uses the individual shape data input from the object separation unit 272 based on the body direction information received from the object orientation estimation unit 1001, and the estimation distance image when viewed from a specific direction ( Hereinafter, it is referred to as a “specific distance image”). Specifically, a specific distance image when viewed from a predetermined specific direction, such as when the person object to be processed is viewed from the front or from the right side, is shown by body direction information. Generated based on the body orientation of the person object. For example, when the direction of the person object is 0 degrees based on the body direction information, when a specific distance image in which the person object is captured from the front is generated, the line-of-sight direction of the pseudo camera is 180 degrees. Then, the human body part estimation unit 274 executes the position estimation of the human body part of each person object based on the specific distance image generated based on the orientation reference of the body of the person object and the above-mentioned learned model.

In the case of the present embodiment, the input data for learning by the first learning unit 275 is only the distance image captured from the desired direction described above. That is, in the case of the first embodiment, although the viewpoint condition is fixed, a learning distance image in which a person object is captured from various directions by 360 degrees is required in order to obtain a highly accurate trained model. On the other hand, in the method of the present embodiment, it is sufficient to prepare a learning distance image captured only from a predetermined specific direction, and the cost required for the learning phase of the position estimation of the human body part can be reduced. In addition, the learning cost for creating a trained model for estimating the direction in which the person object faces is added, but since learning is not performed in a direction other than the above specific direction, the position of the human body part in the estimation phase. Improvement of estimation accuracy can be expected.

<Modification example>
In the above example, the specific direction is one direction, but it may be a plurality of directions. FIG. 12 is a block diagram showing a functional configuration of the server 270 "when two specific directions are provided. Two second estimation image generation units 1002a and 1002b corresponding to different directions and two human body part estimations. Parts 274a and 274b are present. Under such a configuration, for example, the second estimation image generation unit 1002a and the human body part estimation unit 274a are used for the front surface, and the second estimation image generation unit 1002b and the human body part estimation unit 274b are present. For the right side surface, etc., the specific direction desired by the user and each functional unit may be associated with each other. In this case, the first learning unit 275'is subjected to two types of machine learning, one for the front surface and the other for the right side surface. Each of these is performed to provide a trained model for the front surface to the human body part estimation unit 274a and a trained model for the right side surface to the human body part estimation unit 274b.

In the above configuration, the second estimation image generation unit 1002a generates a specific distance image when the person object is viewed from the front based on the body direction information received from the object orientation estimation unit 1001. Similarly, the second estimation image generation unit 1002b generates a specific distance image when the person object is viewed from the right side surface based on the input body direction information.

Then, the human body part estimation units 274a and 274b use the estimation distance images corresponding to the respective specific directions obtained as described above and the trained models provided by the first learning unit 275', respectively. Estimates the position of the human body part of the human object. Then, the estimation result in each specific direction is output to the complement processing unit 1004.

The complement processing unit 1004 uses the received plurality of estimation results to perform complement processing to complement each part that cannot be seen from each specific direction, and outputs human body part information as the final estimation result. As a result, more reliable information on the human body part can be obtained. Although the complementary correction process is additionally performed, the learning cost is lower than that of learning from various directions of 360 degrees. Further, since each trained model is a model trained from one direction, improvement in estimation accuracy can be expected as well.

[Application to virtual viewpoint video]
The contents described in each embodiment can also be applied to a so-called virtual viewpoint video generation system (see Patent Document 3). For example, three-dimensional shape data for each person object is created in advance with polygons or the like, and the movement of the player is reproduced by deforming the shape of the person object based on the human body part information obtained by the above-described embodiment. Can be done. In this case, it is possible to generate a virtual viewpoint video in which the amount of data is significantly reduced as compared with the conventional generation method. In the case of this applied method, it is difficult to reproduce detailed expressions such as facial expressions of athletes, but sufficient information can be obtained to check formations between athletes and individual movements of athletes, and mobile phones such as tablet PCs can be obtained. It can be said that it is suitable for viewing and checking virtual viewpoint images on a tablet terminal.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

270 Server 271 Three-dimensional shape derivation unit 272 Object separation unit 273 Image generation unit for estimation 274 Human body part estimation unit 275 Learning unit

Claims (13)

An acquisition means for acquiring shape data indicating the three-dimensional shape of the person object, which is generated by using a plurality of captured images obtained by imaging the person object from a plurality of directions.
Based on the shape data acquired by the acquisition means, a generation means for generating a distance image showing a distance to the person object when a predetermined viewpoint condition is followed, and a generation means.
Part estimation that estimates the position of the human body part in the person object based on the distance image generated by the generation means using a trained model for estimating the position of the human body part obtained by pre-learning. Means and
An information processing device characterized by being equipped with. When the shape data acquired by the acquisition means includes three-dimensional shapes of a plurality of person objects, it further has a separation means for separating the individual shape data for each person object.
The generation means generates the distance image based on the individual shape data, and generates the distance image.
The part estimation means performs the estimation using a trained model obtained by training for each person object.
The information processing apparatus according to claim 1. The shape data is data in which the three-dimensional shape of the person object is represented by voxels.
The separation means obtains the distance between each voxel and another voxel, and performs a discrimination process for identifying voxels that are in contact with each other or within a predetermined distance as voxels belonging to the same person object. Separate into the individual shape data,
The information processing apparatus according to claim 2. The information processing apparatus according to any one of claims 1 to 3, wherein the predetermined viewpoint condition is a viewpoint condition when a learning distance image used as input data in the learning is created. The information processing apparatus according to claim 4, wherein the predetermined viewpoint condition includes at least elements of a distance to the person object, a height from the ground, and an angle of view. A direction estimating means for estimating the direction in which the person object is facing, and
The generation means generates a specific distance image that captures the person object from a specific direction based on the direction estimated by the direction estimation means.
The site estimation means uses the specific distance image to perform the estimation.
The information processing apparatus according to any one of claims 1 to 5, wherein the information processing device is characterized by the above. The claim is characterized in that the direction estimation means estimates the direction in which the person object is facing by using a trained model for estimating the direction in which the human body is facing, which is obtained by performing learning in advance. The information processing apparatus according to 6. The information processing apparatus according to claim 7, wherein the direction estimation means estimates the direction in which the person object is facing by using the distance image as input data. The information processing apparatus according to claim 7, wherein the direction estimation means estimates the direction in which the person object is facing by using the plurality of captured images as input data. The sixth aspect of the present invention is characterized in that the direction estimating means obtains a moving direction from the difference between the current individual shape data and the past individual shape data, and estimates the moving direction as the direction in which the person object is facing. The information processing device described. A plurality of the site estimation means corresponding to different specific directions, and
A complement processing unit that generates an estimation result that complements a part that cannot be seen from one specific direction by using the estimation results by the plurality of the site estimation means.
The information processing apparatus according to any one of claims 6 to 10, further comprising. A step of acquiring shape data showing a three-dimensional shape of the person object, which is generated by using a plurality of captured images obtained by imaging the person object from a plurality of directions, and
Based on the acquired shape data, a step of generating a distance image showing a distance to the person object when a predetermined viewpoint condition is followed, and a step of generating a distance image.
Using a trained model for estimating the position of a human body part obtained by pre-learning, a step of estimating the position of a human body part in the person object based on the generated distance image, and a step of estimating the position of the human body part in the person object.
An information processing method characterized by including. A program for causing a computer to function as the information processing device according to any one of claims 1 to 11.

Images