JP2015028702A - Information processor, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately estimate the attitude of an object with use of the limited number of pieces of known attitude information.SOLUTION: An object region extraction section 102 extracts person area information from a distance image input. An attitude selection section 103 selects a plurality of pieces of known attitude information stored in a storage section 106, for comparing the person area information extracted with the attitude information. A part attitude selection section 104 determines the conformity between a point group of the person area extracted and a cylindrical model of each part in the attitude information selected, and selects a part attitude representing the conformity more than a prescribed value. An attitude integration section 105 integrates the part attitude information and outputs the information integrated as person attitude information.

Description

  The present invention particularly relates to an information processing apparatus, an information processing method, and a program suitable for use in estimating the posture of an object.

  As a method for estimating the posture of a person in an image, for example, a method described in Non-Patent Document 1 is known. In the method described in Non-Patent Document 1, the posture of the person in the image is estimated by selecting the posture closest to the person in the input image from the known postures. More specifically, the person area silhouette of the input image is classified by the decision tree, and the known posture recorded at the end of the decision tree is output. The method of selecting an output posture from a known posture as described in Non-Patent Document 1 has an advantage that a correct posture can be output as a person because a human posture is selected from an existing posture.

Okada, Ryuzo, Stenger, Bjorn, "A Single Camera Motion Capture System for Human-Computer Interaction", IEICE Transactions on Information and Systems, Volume E91.D, Issue 7, pp. 1855-1862 (2010). Mao Ye, Xianwang Wang, Ruigang Yang, Liu Ren, and Marc Pollefeys, "Accurate 3D pose estimation from a single depth image," IEEE International Conference on Computer Vision, 2011, pp. 731-738

  However, in the method of outputting a known posture as the posture estimation result as in the method described in Non-Patent Document 1, there is a problem that the posture to be output is limited. With such a method, a posture different from the known posture cannot be output, and the closest posture among the known postures is output. Therefore, a posture in which a part of the estimated posture (for example, an arm or a leg) is clearly out of the person area may be output.

  Therefore, as a simple method for solving this problem, it is conceivable to increase the number of known postures prepared in advance. However, if all the possible postures are prepared as known postures, there is a problem that the amount of memory used by the dictionary becomes enormous. In addition, the work itself of preparing information on all possible human postures is very difficult.

  An object of the present invention is to make it possible to estimate the posture of an object more accurately by using a limited number of known posture information.

  An information processing apparatus according to the present invention includes an extraction unit that extracts object region information from an input distance image, a storage unit that stores a plurality of posture information about the object, and an object region extracted by the extraction unit. Selection means for selecting a plurality of posture information for comparison with information from the posture information stored in the storage means, and the posture information selected by the selection means for each part constituting the region of the object An evaluation unit that evaluates consistency with the region information of the object and acquires partial posture information relating to a posture of a part where the consistency is equal to or greater than a predetermined value from the selected posture information; and acquired by the evaluation unit Integrating means for integrating partial posture information to estimate the posture of the object.

  According to the present invention, the posture of an object can be estimated more accurately without increasing the types of known posture information.

It is a block diagram showing an example of composition of an information processor concerning a 1st embodiment. It is a flowchart which shows an example of the basic processing procedure by the information processing apparatus which concerns on 1st Embodiment. It is a figure for demonstrating the process which selects the attitude | position information corresponding to the extracted human body area | region. It is a figure which shows an example of the three-dimensional model of the person created from the joint position of known posture information. It is a figure which shows the example which applied the three-dimensional model of the person created from attitude | position information to the three-dimensional point cloud showing the person area | region of a distance image. It is a figure which shows the example which overlapped the joint position or projection person model of attitude | position information on the person area of the distance image. 5 is a flowchart illustrating an example of a detailed processing procedure by a partial posture selection unit in the first embodiment. It is a block diagram which shows the structural example of the information processing apparatus which concerns on 2nd and 3rd embodiment. It is a flowchart which shows an example of the basic process sequence by the information processing apparatus which concerns on 2nd Embodiment. It is a figure for demonstrating the method to select reference | standard part information. It is a figure which shows the example which integrated reference | standard part information and partial orientation information.

(First embodiment)
In the following embodiments, the object (object) is described as a person. The input image will be described as a distance image. However, the object and image of the present invention are not limited to these.

  In the following description, the posture of the person, the part of the person, and the partial posture are defined as follows. First, it is assumed that the posture of a person is expressed by the position of each joint in a three-dimensional space. This joint position information is called posture information. Next, the part of the person is a part that can be regarded as a rigid body in the person, for example, the head, the trunk, the upper arm, the forearm, the upper leg, the lower leg, and the like. Also, part of the posture information indicating the position and orientation of a part of a certain person is referred to as part information. Next, the partial posture indicates a partial posture of the human posture, for example, an arm posture including an upper arm and a forearm. The partial posture includes the position and posture of a single person's part (for example, only the forearm). Further, information on joint position coordinates representing the position and orientation of the partial posture is referred to as partial posture information.

  The method of expressing the posture information is not limited to the joint position coordinates, and may be expressed by, for example, a method of expressing other joint positions using the three-dimensional position coordinates of the reference joint and the angle from the reference joint. Also, information indicating the position and orientation of the rigid model of each part of the person (for example, the position and orientation may be expressed as center coordinates and rotation information) may be used. As long as the processing of this embodiment can be executed, any information may be used. Also good.

<Overall configuration>
FIG. 1 is a block diagram illustrating a configuration example of the information processing apparatus 100 according to the present embodiment. The information processing apparatus 100 according to the present embodiment executes software (program) acquired via a network or various recording media by a computer including a CPU, a memory, a storage device, an input / output device, a bus, a display device, and the like. Can be realized. As the control computer, a general-purpose computer may be used, or hardware optimally designed for the software of the present invention may be used.

  FIG. 2 is a flowchart illustrating an example of a basic processing procedure performed by the information processing apparatus 100 according to the present embodiment. Hereinafter, basic processing of this embodiment will be described with reference to FIGS. 1 and 2. Detailed processing and contents will be described later.

  The image input unit 101 illustrated in FIG. 1 is configured to input a distance image in which a person whose posture is to be estimated is captured, and executes, for example, the process of step S201 in FIG. The object area extraction unit 102 is configured to extract person area information from the input distance image, and executes, for example, the process of step S202 in FIG. The recording unit 106 stores a plurality of known posture information, and the posture selection unit 103 determines the posture estimated to be close to the human posture in the person area from the known posture information stored in the recording unit 106. Select multiple. Thus, for example, the process of step S203 in FIG. 2 is executed.

  The partial posture selection unit 104 is configured to compare a plurality of selected known posture information and person region information, and select a portion where the known posture information and the person region information match as partial posture information, For example, the process of step S204 in FIG. 2 is executed. The posture integration unit 105 is configured to integrate the partial posture information selected by the partial posture selection unit 104 to generate person posture information and output it as a posture estimation result. For example, the processing of step S205 in FIG. 2 is executed.

  Hereinafter, processing by each configuration will be described in detail.

<Image input unit 101>
The distance image input to the image input unit 101 is an image in which distance information in the depth direction is added to each pixel in the image. The distance image can be acquired by a method of calculating the distance based on the arrival time of the irradiated light, a method of calculating the distance from the deformation of the pattern by projecting pattern light, and the like. The image input unit 101 acquires a distance image from a camera that captures a distance image using these methods. Alternatively, the captured distance images stored in a recording device (not shown) may be sequentially input to the image input unit 101.

<Object region extraction unit 102>
Next, the object area extraction unit 102 extracts a person image area from the input distance image, and uses information such as coordinates representing the image area as person area information. The process of extracting the person area information from the distance image can be performed by the following method.

  First, only a candidate pixel is extracted by performing background difference processing and moving object region extraction on the distance image. Next, the distance value of the pixel that is a foreground candidate is converted into a point group in the camera coordinate system (three-dimensional coordinate system). The center position of the cluster of the three-dimensional point group is obtained, and a point existing in a range that falls within the person size from the point group around the center position is defined as a person region. By projecting the point group labeled as the person area onto the image plane again, a distance image from which the person area is extracted can be acquired. In the following embodiment, the distance image of the extracted person area becomes person area information. The person area information extraction method is not limited to this, and a known method may be used. For example, in addition to the above, a process of estimating the floor surface may be further added so that the person region is more accurately separated.

<Recording unit 106>
Next, the posture information recorded in the recording unit 106 will be described. As illustrated in FIG. 3, m pieces of posture information 301 to 30 m are stored in the recording unit 106. The posture information in the present embodiment may hold the posture information in any form as long as it can represent the positional relationship between human parts such as a person's limbs and head. The form of posture information is, for example, the coordinates of the main part of the person, and the main part of the person is the position of each joint or the terminal part such as the head or limbs. Hereinafter, these are collectively referred to as a joint position.

Posture information 301 to 30m shown in FIG. 3 is information related to joint positions such as the head, neck, shoulder, chest, waist, elbow, wrist, hip joint, knee, and ankle. The information related to the joint position recorded in the recording unit 106 is not limited to this, and may further include a hand tip or a foot tip. In the present embodiment, as a known position information, information of the joint positions are assumed to be recorded as a three-dimensional information P _H relative to the waist position as shown in the following equation (1).

Here, (ξ _xj , ξ _yj , ξ _zj ) (j = 0, 1,..., J) represents the coordinates of the three-dimensional position of the j-th joint. For example, if the coordinates (ξ _x0 , ξ _y0 , ξ _z0 ) are the coordinates of the waist position, ξ _x0 = 0, ξ _y0 = 0, ξ _z0 = 0, and other joint positions are coordinates with the waist position as the origin. It is represented by the position in the system. The joint as the origin may be another joint without being limited to the waist position, but in the following description, it is assumed that three-dimensional information with the waist position as the origin is recorded in the recording unit 106.

  Ideally, the posture information corresponding to any posture is held as the known posture information recorded in the recording unit 106, but in reality, the number of postures is discrete or limited. However, it is desirable that the recorded posture information is posture information that can express many postures by a combination of the partial postures. For example, it is desirable that a plurality of types of known posture information are recorded so that a certain posture can be expressed by a combination of the upper and lower bodies of the known posture information.

<Attitude selection unit 103>
The posture selection unit 103 selects a plurality of pieces of known posture information close to the person region information extracted by the target region extraction unit 102 from the recording unit 106. As illustrated in FIG. 3, n (n <m) pieces of posture information 321 to 32n are selected from the recording unit 106 by the processing of the posture selection unit 103 for the person region 310.

  First, it is assumed that the recording unit 106 records the person area information of the distance image that matches the known posture information together with the known posture information 301 to 30m. The posture selection unit 103 compares the input human region information with the human region information related to the known posture information, and sorts the known posture information in ascending order of error. At this time, the person area information may be converted into a feature amount for comparison. As the feature amount, a feature amount for gray scale such as a HOG (Histograms of Oriented Gradient) feature can be changed and used for a distance image. In the case of performing comparison using the feature amount, the information to be recorded in the recording unit 106 is the image feature amount of the human region having a known posture.

  The posture selection unit 103 selects a predetermined number of known posture information with small errors from the top. In addition, it is not limited to a predetermined number, You may select and output the some attitude | position information which becomes below a predetermined error. In addition, when it is possible to determine that the nearest posture information that minimizes the error matches the input person area information (for example, when the posture information error is very small compared to other posture information). May be an output result of the information processing apparatus 100, which will be omitted below.

  The method for selecting posture information similar to the person area information from the recording unit 106 is not limited to this, and other methods may be used. For example, a decision tree is used as in the method described in Non-Patent Document 1. A method may be used. The method using a decision tree is an effective method in terms of processing speed because it can efficiently search for similarities between a large number of known posture information and person area information. When using a decision tree, the probabilities of a plurality of known posture information are stored in the terminal node, or a plurality of decision trees are used to search for similar known posture information. Attitude information can be selected.

  As another method for efficiently searching for known posture information, not only a decision tree but also other approximate neighborhood search methods can be used. As an approximate neighborhood search method, for example, there is a method using a hash. Furthermore, as another example of the technique, there is a technique of comparing a point group of a person area as described in Non-Patent Document 2 with a point group of each posture stored in advance. Any method may be used as long as a plurality of pieces of known posture information that are highly likely to match the person in the input image can be selected.

  In this embodiment, as a method for selecting a plurality of known posture information, for example, the following is performed. When an error or distance value between each posture information and the person area information is obtained, a plurality of posture information indicating an error or distance value equal to or less than a predetermined value may be selected. On the other hand, if the relationship between each piece of posture information and person area information can be obtained with probability, posture information having a predetermined probability or higher may be selected. Alternatively, a predetermined number of upper known posture information that is highly likely to match the posture shown in the person area information may be always selected. The posture selection unit 103 outputs a plurality of pieces of known posture information that are highly likely to match the posture of the person area.

  In addition, in order to obtain a weighted average of partial posture information, which will be described later, it is necessary to output a value indicating the similarity between the posture information and the person area information together with the selected known posture information. In the above example, errors, distances, probabilities, etc. can be used when calculating the similarity. Hereinafter, these are collectively referred to as a known posture information score, and the higher the score, the higher the similarity between the person area information and the known posture information.

In the present embodiment, the known posture information recorded in the recording unit 106 is joint position information with the waist position as the origin. Therefore, it is necessary to translate or rotate the coordinates of the joint position in the posture information as necessary in accordance with the person position of the input distance image. For example, when the joint coordinates of the selected known posture information are moved in parallel to the camera coordinate system with the camera position as the origin, the joint position coordinates P _C are obtained by the following equation (2).

Here, the coordinates (ξ _xc , ξ _yc , ξ _zc ) are the coordinates of the waist position in the camera coordinate system, and j = 0, 1,. As a result, it is possible to obtain a joint position coordinates P _C of the posture information in the camera coordinate system. The waist position in the camera coordinate system can be obtained by various methods, such as a method of obtaining from the center of gravity of the person area information, or a method of obtaining a position corresponding to the waist position by voting at the same time as selecting the posture. Also, depending on the method of calculating the waist position, a plurality of waist positions in the camera coordinate system may be obtained for one piece of posture information, such as when a plurality of waist position peaks are obtained by voting. In such a case, based on the three-dimensional information P _H of one orientation information, it will get a joint position coordinates P _C of a plurality of the camera coordinate system. The plurality of joint position coordinate P _C may also as a variation of the plurality of orientation information output from the posture selection unit 103. In the following description, it is assumed that the plurality of known posture information output by the posture selection unit 103 represents the joint position coordinates P _C of the camera coordinate system.

<Partial posture selection unit 104>
The partial posture selection unit 104 compares the plurality of posture information selected by the posture selection unit 103 with the person area information, and selects partial posture information that matches the person area information. It is difficult to prepare posture information corresponding to all possible postures in the recording unit 106 as known posture information. Therefore, the posture selection unit 103 selects posture information estimated to be similar to the person area information, but often does not completely match the person area information. However, even in such a case, there are many cases where it partially matches the human posture of the inputted distance image.

  For example, in the nearest known posture information that is most similar to the person region information, the upper body including the arm is consistent with the human posture, but the lower body may be different from the human posture. Even in such a case, it can be expected that the other known posture information includes posture information in which the posture of the lower body matches the person area. The partial posture selection unit 104 collects posture information that partially matches the person area.

Hereinafter, an example of selecting partial posture information will be described with reference to the drawings.
FIG. 4 is a diagram illustrating an example of a three-dimensional model 402 of a person created from the joint positions of known posture information 401. A three-dimensional model 402 shown in FIG. 4 is a model in which each part is approximated by a cylinder, and a cylindrical model of each part is arranged based on each joint coordinate of the joint position. Hereinafter, a method for confirming the correspondence between the person area information and the posture information in the three-dimensional space using the three-dimensional model will be described with reference to FIG.

  FIG. 5A is a diagram illustrating the distance value of the person region 310 as a three-dimensional point cloud 510. Here, the point group 510 of the person area is displayed from a direction different from the line of sight of the camera. FIG. 5B is a diagram in which the joint position of the posture information 321 selected by the posture selection unit 103 with respect to the person region 310 is represented by a three-dimensional model 520. A plurality of known posture information is output from the posture selection unit 103, and the example shown in FIG. 5B is one of them.

  FIG. 5C is a diagram illustrating an example in which the point group 510 illustrated in FIG. 5A and the three-dimensional model 520 illustrated in FIG. In this state, by examining the correspondence between each part of the person and the point group, partial posture information having high consistency with the person region information is selected.

  For example, in the upper left arm 531 shown in FIG. 5C, it can be determined that there is a sufficient point cloud in the vicinity of the cylindrical model, so that it matches well with the person area information. On the other hand, in the left upper thigh 532, since there is no point cloud near the cylindrical model, it can be determined that it does not match the person area information. In this way, the correspondence status between each part and the point group is determined. In the example shown in FIG. 5C, the head, torso, left and right upper arms, and forearm portions can be determined as partial posture information that matches the person area information.

  A more specific process for evaluating the consistency of each part of a person will be described. First, in order to evaluate whether or not a point group corresponds to the cylindrical model of each part, a point existing within a predetermined range from the surface of the cylindrical model is associated with the cylindrical model. When a predetermined number or more points are associated with a cylindrical model of a certain part, it is evaluated that the consistency between the part and the person area (point group) is high. The number of corresponding points used as a criterion for judging the consistency between the cylindrical model and the point cloud may be a number that assumes that the cylindrical model and the point cloud completely match, and even a part of the cylindrical model matches the point cloud. It is good also as the number which should just do. Thus, when the number of corresponding points is equal to or greater than a predetermined value, it is evaluated that the consistency is high.

  Further, since the point group 510 shown in FIG. 5A is generated based only on the portion of the distance image that can be seen from the camera direction, the point group 510 is a part that shows the best consistency with the cylindrical model created from the known posture information. In some cases, the number of corresponding point groups is small. Therefore, the criteria for determining the consistency between the cylindrical model and the point group (threshold value threshold) may be changed depending on the part. In the examples shown in FIGS. 4 and 5, the cylindrical model is used as the three-dimensional model. However, the method of expressing the person model is not limited to this. Any shape model may be used as long as the correspondence with the person area information can be confirmed in the three-dimensional space.

  Next, another method for selecting partial posture information will be described with reference to FIG. In the example shown in FIG. 5, the consistency between each part of the person and the person region is evaluated in the three-dimensional space, but the evaluation may be simply performed on the plane (two-dimensional) of the distance image. FIG. 6A is a diagram illustrating an example in which the person region 310 of the distance image and the joint position of the known posture information 321 are superimposed and displayed. In this method, the consistency between each part and the person area is determined by checking whether or not each joint position or a part between the joints exists on the coordinates of the person area.

  In the example shown in FIG. 6A, the joint positions relating to the head, torso, left and right upper arms, and forearms are present on the person area, so the coordinates of these parts are partial posture information that matches the person area information. Judge. The consistency of each part was evaluated based on whether or not the coordinates of the joint position exist on the person area. However, when the joint position exists near the person area, it is evaluated that the consistency is high. May be. As a result, partial posture information can be selected while allowing some posture deviation.

  FIG. 6B is a diagram showing an example in which a projected person model 322 obtained by projecting the cylindrical model shown in FIG. The consistency of each part may be determined based on the projected person model 322 and the person region 310, and the partial posture information to be selected may be determined. In this case, the consistency can be evaluated by calculating the degree of overlap between each part of the projected person model 322 and the person region 310. For example, when the ratio of the area of a certain part in the projected person model 322 and the area of the person area in the projected person model 322 (number of pixels of the person area) is included, the person area is included in a predetermined ratio or more. Evaluates that the coordinates of the part and the person area information are highly consistent.

  By determining the plurality of posture information obtained by the posture selection unit 103 by the method as described above, partial posture information can be obtained from each piece of posture information. However, if there is posture information that does not match the person area information at all, partial posture information may not be selected from the posture information. The partial posture selection unit 104 outputs the posture information of the partial posture selected as described above (in this embodiment, information on the joint position of the partial posture) as partial posture information. In the example shown in FIGS. 5 and 6, the coordinates of the joint positions related to the head, neck, shoulder, chest, waist, elbow, wrist, and hip joint are output as partial posture information.

<Attitude integration unit 105>
The posture integration unit 105 integrates the partial posture information obtained by the above processing and generates person posture information output from the information processing apparatus 100. In the partial posture information integration process, the coordinates of each joint position in each partial posture information are averaged to generate human posture information. For example, for the head coordinates, the head coordinates are referred to from the partial posture information including the head, and all the head coordinates are averaged. Further, when averaging, weighting based on a score attached to each posture information may be performed. As described above, the posture selection unit 103 outputs a score indicating the similarity to the human region information together with the posture information, and based on this, weighting is applied to the coordinates of the joint position indicated by each partial posture. Person posture information may be obtained by averaging.

  Through the above processing, the human posture information can be output from the information processing apparatus 100. It should be noted that the person position information output from the information processing apparatus 100 may be used as an initial value to perform detailed position alignment on the person area information. The position alignment can be performed by a known technique such as an ICP (Iterative Closest Point) between the cylindrical model of each part and the point group of the person region.

FIG. 7 is a flowchart illustrating an example of a processing procedure performed by the partial posture selection unit 104 corresponding to step S204 of FIG.
First, in step S <b> 701, the partial posture selection unit 104 selects one piece of posture information from a plurality of known posture information selected by the posture selection unit 103. In step S702, the consistency between the selected posture information and the person area information is evaluated. For example, the consistency is evaluated by comparing a three-dimensional point cloud of a person region with a three-dimensional model of a person created based on joint position information of posture information.

  In step S <b> 703, if there is a highly consistent part, the part is selected as a partial posture, and the partial posture information is output to the posture integrating unit 105. In step S704, it is determined whether there is posture information that has not yet been evaluated for consistency. As a result of this determination, if there is posture information whose consistency has not been evaluated, the process returns to step S701. On the other hand, when consistency is evaluated in all posture information and partial posture information is acquired, the process ends.

  As described above, according to the present embodiment, based on a plurality of posture information estimated to be similar to the human region information of the input distance image, partial posture information that partially matches the human region information is selected. The person posture information that should be output is generated by integrating them. As a result, it is possible to output the posture with high accuracy even for an unknown posture that cannot be handled by the posture information recorded in the recording unit 106 in advance.

(Second Embodiment)
In the first embodiment, the method of generating the human posture information by selecting the partial region information that is partially consistent with the human region information from the known posture information and averaging the selected partial region information. In the first embodiment, by averaging the partial posture information, there is a possibility that the human posture information in which the posture that seems to be a person is slightly broken is generated. Therefore, in this embodiment, an example of generating person posture information that is more like a person will be described.

  In this embodiment, the posture information that most closely matches the person area information is set as reference posture information, and a part thereof is set as reference portion information. Then, the human posture information is created by combining the reference part information and the partial posture information acquired from the other posture information. Hereinafter, specific embodiments will be described.

  FIG. 8 is a block diagram illustrating a configuration example of the information processing apparatus 800 according to the present embodiment. FIG. 9 is a flowchart illustrating an example of a basic processing procedure performed by the information processing apparatus 800 according to the present embodiment. Compared to the configuration shown in FIG. 1, the present embodiment further includes a reference site selection unit 107. The reference part selection unit 107 selects, as reference posture information, the posture information that most closely matches the person area information from the plurality of posture information selected by the posture selection unit 103, and selects a part thereof as reference part information.

  The partial posture selection unit 104 of the present embodiment selects a partial posture (part) other than the reference part from the posture information. For example, when the upper body part is selected as the reference part, the optimum lower body partial posture information is searched from a plurality of posture information. The posture integration unit 105 combines the reference part information and the partial posture information to generate person posture information.

  Moreover, since the process of step S901-S903 of FIG. 9 is the same as the process of step S201-S203 of FIG. 2, respectively, description is abbreviate | omitted. Hereinafter, the processing after step S904 will be described in detail.

<Recording unit 106>
In the first embodiment, joint position information of various postures is recorded in the recording unit 106 as known posture information. In the present embodiment, for each posture information, information on the posture is stored together. The information on the posture is, for example, information such as an action category such as walking, running, squatting, the size of a person such as height and physique, and the direction and angle of the person relative to the camera. Hereinafter, these are collectively referred to as posture parameters. The posture parameter is information indicating the posture state and attributes in addition to the information on the joint position coordinates (or joint angle information), and may include information other than the above. In the recording unit 106, each posture information is stored in association with the posture parameter.

<Reference site selection unit 107>
The reference part selection unit 107 according to the present embodiment selects, as reference posture information, posture information most consistent with the person area information from among a plurality of posture information selected by the posture selection unit 103. Further, a portion having high consistency with the person area information is selected as the reference portion from the reference posture information. That is, the process of step S904 in FIG. 9 is performed.

  First, a method for selecting reference posture information will be described. The reference posture information can be obtained by selecting posture information presumed to best indicate the posture of the person area. Therefore, when the calculation result of the posture selection unit 103 is reflected as it is, the posture information having the maximum score is set as the reference posture information. Alternatively, the degree of matching between the posture information and the person area information may be calculated, and the posture information with the highest degree of matching may be used as the reference posture information. The degree of matching is calculated as follows, for example.

  First, as shown in FIG. 5C, a three-dimensional model of a human body created from known posture information and a point group of a person region are superimposed. As in the first embodiment, each part is associated with a point group, and the degree of matching C is obtained by the following equation (3).

  Here, Ω represents the total number of points in the person area, and ω represents the number of points corresponding to the part (number of points existing in a predetermined neighborhood range of the cylindrical model) in the point group of the person area. . T represents the number of sites (10 in the examples shown in FIGS. 4 and 5), and τ represents the number of sites that correspond to the point group (determined to have high consistency). Further, as shown in FIG. 6B, the matching degree C may be obtained in association with a person region using a two-dimensional projection model. In this case, Ω in Equation (3) is the number of pixels in the person area of the distance image, and ω is the number of pixels in the person area included in the portion of the projected person model 322.

  The degree of matching between the posture information and the person area information is calculated by the method as described above. The reference part selection unit 107 sequentially calculates the degree of matching for a plurality of pieces of posture information selected by the posture selection unit 103, and uses the posture information indicating the maximum degree of matching as reference posture information. Alternatively, posture information that most represents the posture of the person region may be selected based on information on the sum or product of the posture score and the degree of matching. Further, as another method for selecting the reference posture information, the reference posture information may be selected based on the consistency between the specific part and the person region. For example, the body that determines the general posture of the person, or posture information in which the body and the head are most consistent with the person region may be used as the reference posture information.

  Next, the reference part selection unit 107 selects a reference part from the reference posture information. The reference part information is obtained by selecting a portion that matches the person area from the reference posture information. This can be performed by a process similar to the process of selecting partial posture information in the first embodiment.

  FIG. 10 is a diagram for explaining a method of selecting the reference part information 1001. In the example illustrated in FIG. 10, the reference posture information 1000 is selected as known posture information having the highest consistency with the person region 310. Next, the head, the torso, and the left and right arms are selected as reference parts as parts that match the person region 310 among the parts in the reference posture information. As a result, the remaining part, that is, the part 1002 corresponding to the left and right legs is searched from the other posture information 1003. Hereinafter, a part that is not included in the reference part and is searched from other posture information is referred to as a search part.

  In the above example, the reference part is determined based on the consistency with the person area information. For this reason, for example, only a terminal part such as the forearm or the lower leg may be a search part. For example, if you want to maintain the relationship between the arms (upper arm and forearm) and legs (upper leg and lower leg) with known posture information, that is, if you want to more strongly guarantee that you are in the correct posture as a person May be inconvenient. In such a case, when the reference part is selected from the reference posture information so that the search part is always the partial posture of the arm or leg, the reference part may be separated only by the shoulder joint and the hip joint.

  As another method for selecting the reference site, the site used as the reference site may always be a specific site. Specifically, the trunk or the trunk and the head may be necessarily used as the reference part. In this case, the extremity is always a search part, and after determining the body posture based on the reference posture information, the optimum posture of the limb is searched from the remaining posture information. In this case, the reference posture information itself may be included in a partial posture search target described later. As a result, the limb part of the reference posture information may be selected as the limb of the human posture information that is finally output.

  The reference part selection unit 107 extracts the joint position information of the reference part determined as described above from the known posture information and outputs it as reference part information. If the reference posture information includes all parts (joint coordinates), or if it can be determined that the reference posture information sufficiently matches the human region, the reference posture information is directly used as the human posture without executing the following processing. It may be information.

<Partial posture selection unit 104>
The partial posture selection unit 104 according to the present embodiment acquires partial posture information related to a search site optimum for a person region from known posture information. That is, the process of step S905 in FIG. 9 is performed. In the example shown in FIG. 10, a portion 1002 corresponding to the left and right legs is a target to be searched. From the plurality of posture information 1003 selected by the posture selection unit 103, the left and right legs having high consistency with the person area information are searched. Thus, the left and right leg partial posture information 1004 is acquired.

  As basic processing of the partial posture selection unit 104, the consistency between all search parts of the posture information 1003 and the person region is evaluated, and the partial posture information having the highest consistency is selected. In the evaluation of consistency, the consistency between the partial posture and the person area can be evaluated by the same procedure as in the first embodiment. In the example shown in FIG. 10, as a result of evaluating the consistency of all search parts (left and right legs) of the posture information 1003, the search part of the posture information 32n is selected as having the highest consistency.

  In the example shown in FIG. 10, the left and right legs are selected together, but partial posture information may be selected from posture information for each part. For example, the upper thigh and the lower thigh that are highly consistent with the person region may be selected independently. Or you may make it select partial attitude | position information from different attitude | position information for every attitude | position of the right leg and left leg which the upper thigh part and lower leg part continued.

  The partial posture information corresponding to the search part may be selected by the method described above, but the partial posture information is selected by evaluating the affinity between the reference part and the partial posture (search part). May be. In the above method, the reference part and the partial posture may be selected in an unfavorable combination depending on the posture variation included in the posture information.

  For example, when the posture information of a person with different height is included in a plurality of posture information, the height of the person selected as the reference part may be significantly different from the height of the person selected as the partial posture. . When this reference part and the partial posture are combined, there is a possibility that personal posture information of a physique that is impossible for a person is generated. The above method may be used when the type of posture information recorded in the recording unit 106 is limited, or when accuracy is not required for human posture information to be output so much, but preferably, the reference part is as follows: Evaluate the affinity between and the partial posture. The affinity between the reference part and the partial posture is evaluated based on whether or not a possible posture is obtained by combining the reference part and the partial posture, and is performed as follows, for example.

  One method for evaluating affinity is a method using posture parameters. As described above, each posture information is recorded in association with the above-described posture parameters. When evaluating the affinity between the reference portion and the partial posture, the affinity between the posture parameter of the original posture information of the reference portion and the posture parameter of the original posture information of the partial posture is evaluated.

  For example, if the posture parameter of the posture information selected as the reference posture information and the posture parameter of the posture information for which partial posture information is to be acquired do not match, the partial posture information is not selected from the posture information To. In this way, only the case where the posture parameters completely match may be allowed, and if the height difference, the camera angle difference, etc. are slightly different, the mismatch of posture parameters is allowed. Also good.

  As another method for evaluating the affinity, the affinity may be evaluated based on the arrangement of the reference portion and the partial posture. Here, the joint connecting the reference part and the partial posture is called a connection point. For example, in the example shown in FIG. 10, the connection point is a hip joint. The reference part information and the partial posture information each indicate coordinates in a three-dimensional space, and the coordinates of each connection point can be obtained. When the distance between the connection points is too far, it can be determined that the partial posture at that time has low affinity with the reference portion. Therefore, the partial posture selection unit 104 may select only partial posture information such that the distance between the connection point coordinates in the reference part information and the connection point coordinates in the partial posture information is a predetermined value or less. .

  When evaluating the affinity, a plurality of items such as the above-described plurality of posture parameters and the distance between connection points may be evaluated together. In the present embodiment, the affinity is evaluated as described above, and the partial posture having the highest consistency with the human region is selected from the partial postures determined to have affinity with the reference region. Further, the affinity between the reference posture information and other posture information may be evaluated in advance, and the evaluation of the consistency between the posture information having low affinity and the person area information may be omitted.

<Attitude integration unit 105>
The posture integration unit 105 generates the human posture information by combining the reference part information and the partial posture information obtained by the above procedure. That is, the process of step S906 in FIG. 9 is performed. FIG. 11A is a diagram showing an example in which the reference part information 1001 and the selected partial posture information 1004 are arranged on the same coordinate plane. As shown in FIG. 11A, the position of the joint (the hip joint in the example shown in FIG. 11) connecting the reference part information 1001 and the partial posture information 1004 may not match. Therefore, the posture information may be generated by correcting the joint position to be connected.

  For example, as shown in FIG. 11B, the human posture information is generated by moving the coordinates of the partial posture information 1004 so that the coordinates of the hip joint of the partial posture information 1004 and the hip joint of the reference part information 1001 coincide. May be. As another method, an intermediate position between the hip joint position in the reference part information and the hip joint position in the partial posture information is set as the hip joint position, and the other joint positions are the coordinates of the original reference part information and the partial posture information. Human posture information may be generated.

  As described above, according to the present embodiment, since the human body posture information is generated by connecting the reference portion obtained by extracting a part of the selected posture information and the partial posture, the shape of the known posture information is set as much as possible. The held person posture information can be output. Accordingly, it is possible to output a human-like posture. In addition, by setting the reference posture information and the reference part information, the number of search for combinations can be reduced.

(Third embodiment)
In the second embodiment, the posture information that most closely matches the person region is set as the reference posture information, and a part thereof is set as the reference part information. In this embodiment, an example using a plurality of reference parts will be described. In this case, an example of searching for the best combination by calculating a combination cost described later will be described. Note that the configuration of the information processing apparatus according to the present embodiment is the same as that shown in FIG.

  In the present embodiment, all the posture information selected by the posture selection unit 103 or posture information in which the degree of matching C between the posture information and the person area information is higher than a predetermined value is set as the reference posture information. Similarly to the second embodiment, in the reference posture information, a part having high consistency with the person region or a specific part (for example, a body part) is set as a reference part. By determining the reference part for each of the plurality of reference posture information, parts other than the reference part are search parts.

  Next, the partial posture selection unit 104 of this embodiment will be described. In the second embodiment, the partial posture selection unit 104 selects one search part that most matches the human region as partial posture information. In contrast, in the present embodiment, a plurality of pieces of partial posture information are selected. The plurality of selected partial posture information is evaluated for consistency with the person area information for each of the plurality of posture information, and among the partial posture information obtained from each posture information, a plurality of higher-level partial posture information with high consistency is obtained. Selected.

  As a result, a plurality of pieces of reference part information and partial posture information are selected. At this stage, there are as many candidates of the human posture information to be output as the number of combinations. The posture integration unit 105 calculates a combination cost for each of the plurality of combinations as follows, and selects the lowest-cost combination as human posture information to be output. The combination cost P between a certain reference part i and a certain partial posture j is calculated by the following equation (4), for example.

Here, S _i represents the score of the reference posture information that is the source of the reference part, and S _j represents the score of the reference posture information that is the source of the partial posture. C _i represents the degree of matching of the reference posture information obtained by the above-described equation (3), and C _j represents the degree of matching between the partial posture information and the person area information. H _ij is an evaluation value calculated based on the affinity between the reference part and the partial posture, and w _{1 to} w ₅ are coefficients for the respective inputs.

The degree of matching C _j between the partial posture information and the person area information can be calculated from the number of correspondences between the three-dimensional model of the human body in the partial area and the point group. Further, the evaluation value H _ij can be calculated based on the relationship between the posture parameters relating to the reference part and the partial posture and the distance between the joint positions to be connected.

When calculating the evaluation value H _ij based on numerical information of posture parameters such as height and orientation with respect to the camera, the calculation is performed as follows. That is, for example, the evaluation value H _ij may be set to be higher as having similar posture parameters, for example, based on the reciprocal of the square error between the posture parameter of the reference part and the posture parameter of the partial posture. On the other hand, when calculating the evaluation value H _ij based on the posture parameters recorded as labels and categories, such as behavior categories such as walking and running, and physique parameters such as skinny and thick, the calculation is performed as follows. . That is, for example, a binary evaluation value is calculated based on whether a label matches or does not match, or a distance between labels is defined in advance, and an evaluation value H _ij is calculated based on a distance corresponding to the combination of labels.

  In the present embodiment, the combination of the reference part i and the partial posture j that minimizes the combination cost P calculated as described above is determined. As in the second embodiment, the posture integration unit 105 outputs the result of combining the reference portion i and the partial posture j as person posture information.

  As described above, according to the present embodiment, a plurality of reference parts are used, and the combination of the reference part and the partial posture that minimizes the combination cost P is used as the person posture information to be output. As a result, an optimum combination of partial postures can be selected from the plurality of posture information output by the posture selection unit 103.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

DESCRIPTION OF SYMBOLS 101 Image input part 102 Object area extraction part 103 Posture selection part 104 Partial posture selection part 105 Posture integration part 106 Recording part

Claims (14)

Extraction means for extracting object region information from the input distance image;
Storage means for storing a plurality of posture information about the object;
Selection means for selecting a plurality of posture information from the posture information stored in the storage means for comparison with the region information of the object extracted by the extraction means;
For the posture information selected by the selection means, the consistency with the region information of the object is evaluated for each portion constituting the region of the object, and the partial posture related to the posture of the portion where the consistency is a predetermined value or more Evaluation means for obtaining information from the selected posture information;
Integration means for estimating the posture of the object by integrating the partial posture information acquired by the evaluation means;
An information processing apparatus comprising:   The evaluation means evaluates consistency by associating each part of a model of an object based on the posture information with a three-dimensional point group based on area information of the object of the distance image. Item 4. The information processing apparatus according to Item 1.   The information processing apparatus according to claim 1, wherein the evaluation unit evaluates consistency by associating a projection model obtained by projecting a model based on the posture information onto a plane and region information of the object. . Extraction means for extracting object region information from the input distance image;
Storage means for storing a plurality of posture information about the object;
Selection means for selecting a plurality of posture information from the posture information stored in the storage means for comparison with the region information of the object extracted by the extraction means;
Setting reference posture information from the posture information selected by the selection means, setting means for setting a reference part from the parts constituting the region of the object in the reference posture information;
The reference part information related to the posture of the reference part set by the setting unit is acquired, and the posture information selected by the selection unit is matched with the region information of the object with respect to the part excluding the reference part Evaluation means for evaluating the position, and acquiring partial posture information related to the posture of the part where the consistency is equal to or greater than a predetermined value;
Integration means for estimating the posture of the object by integrating reference part information and partial posture information acquired by the evaluation means;
An information processing apparatus comprising:   The information processing apparatus according to claim 4, wherein the evaluation unit acquires partial posture information whose affinity with the reference part is a predetermined value or more. The storage means stores posture parameters associated with the plurality of posture information,
The information processing apparatus according to claim 5, wherein the evaluation unit evaluates the affinity between the posture parameter related to the reference part information and the posture parameter related to the partial posture information.   The information processing apparatus according to claim 6, wherein the posture parameter includes information on an object size of the posture information, a category of the posture information, or an orientation of the posture information with respect to the camera.   The information processing apparatus according to claim 5, wherein the evaluation unit evaluates affinity based on a connection point between the reference part and a part related to the partial posture information.   The information processing apparatus according to any one of claims 4 to 8, wherein the setting unit sets the reference part as a specific part of the object. The setting means sets a plurality of reference posture information from the posture information selected by the selection means,
The information processing apparatus according to claim 4, wherein the evaluation unit evaluates a combination of reference part information and partial posture information integrated by the integration unit and determines a combination having the highest evaluation value. An information processing method of an information processing apparatus for storing a plurality of posture information about an object,
An extraction step of extracting region information of the object from the input distance image;
A selection step of selecting a plurality of posture information from the posture information stored in the information processing device for comparison with the posture information for comparison with the region information of the object extracted in the extraction step;
The posture information selected in the selection step is evaluated for consistency with the region information of the object for each portion constituting the region of the object, and the partial posture related to the posture of the portion where the consistency is a predetermined value or more An evaluation step of obtaining information from the selected posture information;
An integration step of estimating the posture of the object by integrating the partial posture information acquired in the evaluation step;
An information processing method characterized by comprising: An information processing method of an information processing apparatus for storing a plurality of posture information about an object,
An extraction step of extracting region information of the object from the input distance image;
A selection step of selecting a plurality of posture information from the posture information stored in the information processing device for comparison with the posture information for comparison with the region information of the object extracted in the extraction step;
Setting reference posture information from the posture information selected in the selection step, and setting a reference portion from the portions constituting the region of the object in the reference posture information,
The reference part information related to the posture of the reference part set in the setting step is acquired, and the posture information selected in the selection step is matched with the region information of the object with respect to the part excluding the reference part An evaluation step for evaluating partiality and acquiring partial posture information related to the posture of a part where the consistency is equal to or greater than a predetermined value;
An integration step of estimating the posture of the object by integrating the reference part information and the partial posture information acquired in the evaluation step;
An information processing method characterized by comprising: A program for controlling an information processing apparatus that stores a plurality of posture information about an object,
An extraction step of extracting region information of the object from the input distance image;
A selection step of selecting a plurality of posture information from the posture information stored in the information processing device for comparison with the posture information for comparison with the region information of the object extracted in the extraction step;
The posture information selected in the selection step is evaluated for consistency with the region information of the object for each portion constituting the region of the object, and the partial posture related to the posture of the portion where the consistency is a predetermined value or more An evaluation step of obtaining information from the selected posture information;
An integration step of estimating the posture of the object by integrating the partial posture information acquired in the evaluation step;
A program that causes a computer to execute. A program for controlling an information processing apparatus that stores a plurality of posture information about an object,
An extraction step of extracting region information of the object from the input distance image;
A selection step of selecting a plurality of posture information from the posture information stored in the information processing device for comparison with the posture information for comparison with the region information of the object extracted in the extraction step;
Setting reference posture information from the posture information selected in the selection step, and setting a reference portion from the portions constituting the region of the object in the reference posture information,
The reference part information related to the posture of the reference part set in the setting step is acquired, and the posture information selected in the selection step is matched with the region information of the object with respect to the part excluding the reference part An evaluation step for evaluating partiality and acquiring partial posture information related to the posture of a part where the consistency is equal to or greater than a predetermined value;
An integration step of estimating the posture of the object by integrating the reference part information and the partial posture information acquired in the evaluation step;
A program that causes a computer to execute.

Images