JP2017097578A - Information processing apparatus and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To calculate position and attitude of an object accurately.SOLUTION: An image input unit 101 inputs an image including an object for which an attitude is estimated. A partial area identifying unit 102 identifies a partial area of the object. An attitude estimation unit 103 collects results of identifying partial areas, and classifies them into known attitude classes, to estimate an attitude of the object. A partial area selection unit 104 selects a partial area related to the estimated attitude, from the partial area identification results of an input image. In the partial area, positional relationship between a representative position of the object and an identification result position of the partial area is recorded in advance. A representative position calculation unit 105 uses information indicating the representative position of the selected partial area, to calculate a representative position of the object in the input image. A position posture determination unit 106 aligns the estimated attitude position in the calculated representative position, to calculate position and attitude of the object.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus that calculates the posture and position of an object in an image.

  As a method of estimating the posture of the human body in the image, the image information of the partial area of each posture is learned in advance, and when estimating the posture, the partial region is identified with respect to the human body region of the input image, and the result is integrated. There is a method for estimating the whole human body posture.

  In Patent Document 1, first, learning images in which human bodies of various postures are captured are prepared, and partial features (shape contexts) are learned for the human body regions of the respective learning images. At the time of posture estimation, a partial feature is calculated for the human body region of the input image, and it is determined which posture of the learning image most matches from the feature set. Information on the joint position of each posture is recorded in advance in each posture of the learning image, and information on the joint position relationship (relative position relationship of each joint) of the estimated posture is output. In Patent Document 1, the above-described processing calculates the joint position relationship of the human body in the image, but at which position the estimated joint position exists in the image or in real space (absolute coordinate system / world coordinate system). It is not clear what to do.

  Therefore, in Patent Document 1, as post-processing, fitting processing for matching the estimated joint position relationship with the input image is performed after the above-described posture estimation processing. In the fitting process in Patent Document 1, a method of obtaining a transformation matrix for fitting joint coordinates to an input image in a two-dimensional image by a morphing method using a feature acquisition position is performed. Specifically, the fitting process is performed by calculating a transformation matrix between the feature position of the learned image of the estimated posture and the feature position of the input image, and applying it to the joint position coordinates. Through this post-processing, the joint position in the input image is determined, and a slight error between the estimated posture and the human body in the image is corrected. Note that the minute error is an error that occurs when the posture that is output as a result of the estimated posture is selected from a known discrete posture (learning posture) and does not match the actual input image posture.

JP 2010-176380 A

  In the fitting process in Patent Document 1, it is premised that a feature is acquired at a correct position in an input image. However, it is difficult to correctly obtain the positions of all features in the input image. In particular, in the embodiment of Patent Document 1, the feature point is a coarse discrete point on the contour of the human body region. Therefore, if even one feature acquisition position deviates greatly, the fitting process is greatly affected.

  As another form of the fitting process, there is a method of moving and deforming a model so that a known model (a multi-joint model in the case of a human body) is adapted to a human body region of an input image. In particular, when estimating a human body posture from a distance image, a technique for optimally placing a known articulated model (eg, ICP; Iterative Closest Point) into 3D point cloud data generated from distance image information Applied method). However, these methods require a separate initial position of the model. In the case of a human body, the model initial position is represented by the human body posture (joint position relationship) and the position of the posture in the three-dimensional space. Furthermore, since the fitting performance varies greatly depending on the initial position of the model, it is important to accurately calculate the initial position of the model.

  However, Patent Document 1 does not disclose a method for obtaining an appropriate initial position after posture estimation.

  In view of this, an object of the present invention is to accurately obtain a place where an estimated posture is located in a posture estimation process for estimating the posture of an object by integrating the identification results of partial areas. In particular, the object is to calculate the position of an object by effectively using the partial region identification result used when estimating the posture and the estimated posture result, and to obtain the position and posture with high accuracy.

In order to achieve the above object, an information processing apparatus according to the present invention provides:
A partial region identifying unit that identifies different partial regions of the object, a posture estimation unit that estimates the posture of the target by integrating the identification results of the partial regions, and identification of the partial region based on the estimated posture From the partial region selection unit that selects the result, the representative position calculation unit that calculates the representative position of the object from the information indicating the representative position associated with the selected partial region, the representative position and the estimated posture, A position / orientation determination unit for calculating the position / orientation of the object is provided.

  With the information processing apparatus according to the present invention, the position and orientation of an object can be estimated with high accuracy.

It is a figure which shows the structure of the information processing apparatus of this invention. It is a figure which shows the processing flow of the information processing apparatus of this invention. It is a figure explaining the information (learning result) previously recorded on the recording part in Example 1. FIG. It is a figure explaining the process of the partial region identification part and attitude | position estimation part in Example 1. FIG. It is a figure explaining the process of a partial region selection part and a representative position calculation part. It is a figure which shows the processing flow of the partial area | region selection part in Example 1. FIG. It is a figure explaining the process of a position and orientation determination part. It is a figure explaining the information (learning result) previously recorded on the recording part in Example 2. FIG. It is a figure explaining the process of the partial region identification part and attitude | position estimation part in Example 2. FIG. It is a figure which shows the processing flow of the partial area | region selection part in Example 2. FIG.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  In the following examples, an embodiment will be described in which the object is a human body and the posture and position of the human body in the image are estimated. Note that the posture of the human body in the present embodiment includes not only the angle of each joint of the human body (relative positional relationship of each part of the human body) but also the direction and angle of the human body with respect to the camera. That is, when human bodies having the same joint position relationship are photographed from different directions, the human bodies in the respective images have different postures.

<Overall configuration>
FIG. 1 is a diagram illustrating the configuration of an information processing apparatus 100 according to the present invention. FIG. 2 is a diagram for explaining the basic processing flow of the present invention.

  The information processing apparatus 100 according to the present invention executes software (program) acquired via a network or various recording media on a computer including a CPU, a memory, a storage device, an input / output device, a bus, a display device, and the like. This can be achieved. As a computer (not shown), a general-purpose computer may be used, or hardware optimally designed for the software of the present invention may be used.

  First, basic processing of the present invention will be described with reference to FIGS. 1 and 2. Detailed processing will be described later.

  The image input unit 101 in FIG. 1 is a part for inputting an image to be processed, and executes steps S201 and S202 in FIG. In this embodiment, the input image is a distance image. A distance image is an image in which distance information in the depth direction is recorded with a predetermined scaling on each pixel in the image, and a method for calculating a distance based on the arrival time of irradiated light, or projecting pattern light, It can be obtained by a method of calculating the distance from the deformation of the pattern. The image input unit 101 acquires a distance image from a camera device that captures a distance image by these methods. Further, a distance image to be processed may be sequentially input to the image input unit 101 from a recording apparatus that stores a previously captured distance image. In order to distinguish from a learning image, which will be described later, hereinafter, the distance image input to the information processing apparatus 100 in order to estimate the posture and position of the human body is referred to as an input image. The image input unit 101 also performs processing for extracting a human body region from the input image.

  The partial area identifying unit 102 identifies which of the learned partial areas each part of the human body area of the input image uses a partial area classifier that has previously learned a partial image of the human body (step S203). .

  The posture estimation unit 103 integrates the partial region identification results and determines which of the known postures the human body of the input image is closest to (step S204). In the partial area selection unit 104, a part used for calculating the representative position based on the posture determined by the posture estimation unit 103 from the identification result of the partial region identified by the partial region identification unit 102 (result of step S203). A region identification result is selected (step S205). The representative position calculation unit 105 calculates the representative position of the human body based on the information associated with the selected partial area (step S206). The position and orientation determination unit 106 performs alignment by translating and rotating information on the human body posture (joint position relationship) to the obtained representative position (step S207), and outputs the result as a human body position and orientation estimation result (step S208). . Information that has been learned in advance is stored in the recording unit 107 and is called up as necessary.

  In this embodiment, the representative position of the human body obtained by the representative position calculation unit 105 is the center position of the human body, more specifically, the hip joint position. The representative position of the human body is not limited to this. For example, the head position may be obtained as the representative position.

<Preparation>
The outline of the process at the time of estimating the position and orientation of the human body has been described above. In order to perform these processes, necessary information is learned in advance from a large number of human body distance images, and the information is stored in the recording unit 107. There is a need. Hereinafter, learning in the first embodiment and information stored in the recording unit 107 will be described with reference to FIG.

  3, 301, 301, and 30m are distance images in which m human bodies are captured. Hereinafter, these distance images used for learning are referred to as learning images. In order to facilitate learning, it is assumed that the human body in the learning image is shown in a predetermined size in the center of the image. Moreover, the coordinate which shows the main site | part of a human body is provided to the learning image according to each attitude | position. In FIG. 3, lines that connect the main parts and the main parts are superimposed and displayed on the learning images 301 to 30m, but actually, the information on the distance image and the main part coordinates are individually stored. For example, the distance image is stored in an image format, and the coordinates of the main part are stored as text data or the like. The main part coordinates are information on a human body part indicating a posture change of the human body, such as the position of a main joint of the human body. The main part in Example 1 is positional information including joint positions and head positions of the limbs as indicated by the circles shown on 301 to 30 m in FIG. 3, and hereinafter collectively referred to as joint positions.

  In the first embodiment, since the human body representative position is the hip joint position, the position / posture determination unit 106 described below requires the hip joint position of each posture. Accordingly, the joint position includes the hip joint position. Further, it is assumed that the coordinates of the joint position are described in a three-dimensional coordinate system (camera coordinate system) with the camera position where the distance image is captured as the origin.

  The recording unit 107 stores partial area information 311 and posture information 321 as information learned from the learning image. Below, learning of each information is demonstrated.

First, a process for learning the partial region information 311 using a learning image will be described. The partial area information 311 includes a partial area identifier 312 and a vector from the partial area to the human body representative position (hereinafter, representative position vector) 313 for each of the partial areas s1 to sn. In FIG. 3, symbols v _s1 and v _{s2 to} v _sn corresponding to each partial region classifier are assigned to the representative position vector 313. The partial region classifier prepares by collecting image groups having similar partial poses for all learning images and learning the image groups having each partial pose. Such learning of the partial region classifier can be performed with reference to Non-Patent Document 1, for example.

[Non-Patent Document 1]
Lubomir Bourdev and Jitendra Malik, “Poselets: Body Part Detectors Trained Using 3D Human Pose Annotations” Information is given and human body partial images with similar joint positions are collected. HOG features of collected human body partial images are learned by SVM, and partial region classifiers with various postures are created. What is necessary is just to produce the partial area | region discriminator of this invention similarly to the nonpatent literature 1. FIG. Specifically, based on information on joint positions accompanying each learning image, a distance image having a partially similar posture is cut out and collected. Then, the collected partial distance images are learned with appropriate image features and classifiers, respectively. As examples of image features and classifiers, HOG features and SVMs created from distance images according to Non-Patent Document 1 can be used. The feature quantity and the discriminator are not limited to this, and other known learning / identification methods may be used. As a result of learning, a partial area discriminator 312 in FIG. 3 is obtained for the partial areas s1 to sn. In FIG. 3, for the sake of illustration, the partial area classifier is shown as a partial image of the human body, but actually, the partial area classifier is composed of image features and a classifier.

  The form of the partial region identifier may be other forms as long as the posture of the part can be identified. Since identification of partial areas is multi-class identification, a method such as a decision tree suitable for multi-class identification is suitable. In addition, an approximate nearest neighbor search method (for example, a method using a hash) is also suitable as a method for performing multi-class partial region class identification.

Further, when learning partial areas, the representative position vector 313 is also learned from each partial area. As described above, in this embodiment, the human body representative position is the hip joint position, and the joint position information accompanying the learning image includes the hip joint position. FIG. 3 shows a state in which the vectors v _s1 to v _sn from the appearance positions of the partial areas to the hip joint positions are learned together for the partial areas s1 to _sn . Here, the start point (black circle) of the vector v indicates the center position of the partial region, and the end point (white circle) indicates the hip joint position. The vector v is assumed to be represented by a three-dimensional vector. That is, the starting point of the vector v is a position in the three-dimensional space calculated from the distance image value of the center position of the partial region, and the end point of the vector v indicates the three-dimensional position of the hip joint position.

  The representative position vector v associated with each partial region is obtained using a range of distance images having similar partial postures used when learning each partial region classifier. Specifically, the representative position vector is obtained by the following procedure. First, the center position of each partial area image is obtained for a partial area image group used for learning a certain partial area. The center position is calculated as a three-dimensional position in the camera coordinate system. Next, for each partial region image, a vector in the hip joint position direction is calculated with the center position as the origin. An average vector of the vectors of these partial area images is set as a representative position vector of a partial area. The method of obtaining the representative position vector is not limited to the average, and may be calculated by a method of obtaining a median value, a method of obtaining an extreme value by mean shift, or the like. When there are a plurality of vector distribution peaks, a plurality of representative position vectors may be associated with one partial region classifier.

As described above, for the partial areas (partial postures) s1 to sn, the partial area identifier 312 and the representative position information (representative position vectors) v _{s1 to} v _sn indicated by the partial areas are combined as the partial area information 311. Store in the recording unit 107.

  Next, posture information 321 will be described as other information stored in the recording unit 107 in advance.

  FIG. 3 shows a state in which each joint coordinate (joint position relationship) 322 in each posture is stored for the postures p1 to pm of the human body. In this embodiment, the number of postures recorded in the recording unit 107, that is, the estimated number of postures, is the same as the number of postures m of the learning image. Therefore, the joint position relationship 322 of each posture is obtained by storing the joint position relationship associated with each of the learning images 301 to 30m. As described above, the coordinates of the joint position relationship associated with the learning image are stored in the camera coordinate system. Although the joint position relationship 322 stored in the recording unit 107 is illustrated in FIG. 8, each joint position coordinate in the camera coordinate system is actually stored. Note that the number of postures to be estimated (the number of postures stored in the recording unit 107) m may be decreased by deleting a similar posture from the learning image. Alternatively, the learning image may be prepared by deleting an image with a similar posture. Note that the posture estimation unit estimates which of the postures p1 to pm the human body of the input image is most similar to. Hereinafter, in the description of posture estimation, the postures p1 to pm may be referred to as posture classes.

  Next, the partial region histogram 323 will be described as information included in the posture information 321. The partial area histogram 323 in the present embodiment is a process for obtaining a posture from the partial region identification result of the input image (processing of the posture estimation unit 103) and a process of selecting a partial region according to the estimated posture (partial region selection unit) 104). In this embodiment, the partial area histogram is used in common for these two processes. However, depending on the embodiment of the partial area selection unit 104, another information is created from the partial area histogram and stored in the recording unit 107. You may do it. Details of this will be described later together with the description at the time of execution.

The partial region histogram 323 (h _{p1 to} h _pm ) is prepared for each posture _{p1 to pm} , as shown in FIG. In the partial region histogram, the horizontal axis represents the partial region class numbers s1 to sn (the partial region identifier numbers s1 to sn described above), and the vertical axis represents the appearance probability of the partial region in each posture. As in the case of the posture estimation execution described later, the partial region histogram is obtained by executing all the partial region classifiers for each learning image and obtaining the result of each partial region classifier for each posture class (each learning image). Create by calculating the probability of being. At the time of posture estimation execution, a partial region classifier is executed on the input image, a partial region histogram of the input image is created, and then compared with the partial region histograms h _{p1 to} h _{pm of} each posture, the most similar posture To decide.

  The joint position relationship 322 and the partial region histogram 323 of each posture described above are stored in the recording unit 107 as posture information 321.

  The information stored in the recording unit 107 in advance has been described above with reference to FIG.

<Pretreatment>
Hereinafter, each process at the time of executing posture estimation will be described in detail.

  As described above, the image input unit 101 reads an input image for performing posture estimation. Further, a human body region is extracted from the input image as preprocessing. In the human body region extraction processing, first, only foreground candidate pixels are extracted by performing background difference processing on the distance image. Next, the distance value of the foreground candidate pixel is converted into a point group in the camera coordinate system (three-dimensional coordinate system). The center position of the cluster of the three-dimensional point group is obtained, and a point existing in a range within the human body size in the point group around the center position is set as a human body region. By projecting the point cloud labeled as the human body region onto the image plane again, a distance image from which the human body region is extracted can be acquired. The human body region extraction method is not limited to this, and a known method may be used.

<Partial area identification unit>
FIG. 4 is a diagram for explaining the processing of the partial region identification unit 102 and the posture estimation unit 103.

The partial area identifying unit 102 identifies a partial area for the human body area of the input image. In FIG. 4, 401 is a human body region extracted from the input image, and 402 is a partial region identification result. The partial area identification unit 102 identifies the partial area using the partial area identifier of the partial area information 311 stored in the recording unit 107. In addition, the partial area identification unit 102 creates a partial area histogram 405 (h _input ) of the input image.

Hereinafter, specific processing of the partial region identification unit 102 will be described. First, a partial area of a predetermined size centered on each pixel on the human body area is cut out, and identification processing is performed by each of the partial area classifiers s1 to sn stored in the recording unit 107. If it is determined that the identification result of each pixel corresponds to a certain partial region class sx, the histogram is added to the bin of the class sx of the partial region histogram h _input . In FIG. 4, the partial area 402 is identified as corresponding to the partial area class s4, and a histogram is added to the s4 bin of the partial area histogram h _input . The partial area histogram h _input is normalized after searching all human areas, and is set as the appearance probability of the partial area class. Also, depending on the embodiment of the partial area classifier, the possibility of a plurality of partial areas may be obtained from each pixel (each partial area). For example, when a partial region class is identified by a plurality of decision trees as a method for identifying a partial region classifier, the result of a plurality of partial region classes that are different in each decision tree may be acquired in identifying a partial region. is there. In such a case, histograms may be added to bins of a plurality of partial area classes for a partial area identification result. Further, depending on the embodiment of the partial area classifier, the partial area classifier can output the reliability of the identification result simultaneously with the identification of the partial area class. For example, when SVM is used as the discriminator, the distance from the discrimination boundary (hyperplane) may be used as the reliability. In this way, when the reliability of the identification result is obtained, weighted addition may be performed according to the identification result reliability of the partial region when adding to the histogram. Furthermore, in the above description, a method for executing partial region identification for a partial region centered on each pixel for all pixels on the human body region has been described. However, from the viewpoint of processing time, a partial region to be processed is selected. It may be reduced. For example, the number of partial area identifications may be reduced by processing such as setting the center pixel in predetermined step increments, or setting the center pixel randomly from the human body area.

<Attitude estimation unit>
Next, processing of the posture estimation unit 103 will be described. The posture estimation unit 103 compares the partial region histogram h _input of the input image, which is the output of the partial region identification unit 102, with the partial region histograms h _{p1 to} h _pm of each posture of the posture information 321 recorded in the recording unit 107. And determine the closest posture class (body posture). Comparison between histograms and posture determination can be performed by calculating the similarity between h _input and h _p1 to h _pm and determining the posture with the highest similarity. As a similarity calculation method, KL divergence, Batachariya distance, Earth Mover's Distance, or the like can be used. Comparison between histograms is not limited to this processing, and may be performed by other nearest neighbor search processing. The posture estimation unit 103 acquires information on the joint position relationship of the determined nearest neighbor posture from the recording unit 107 and outputs the information to the position / posture determination unit 106. Here, the information on the joint position is the coordinates in the camera coordinate system of each joint J1 to J16 (when there are 16 joint positions) as indicated by 404 in FIG. Output as is.

<Partial area selection unit / representative position determination unit>
Through the above processing, the posture of the human body in the input image is estimated, but at this point in time, the position of the posture is not determined. Since the human body region is extracted from the input distance image by the image input unit 101, the rough position of the human body is known. However, it is not clear at which coordinates the coordinates 404 of the joint position obtained by the posture estimation unit 103 exist on the image or in the camera coordinate system of the input image. By calculating the center position of the extracted human body area and matching the center positions of the joint position relationship, it is possible to determine the position where each joint of the human body exists, but due to the influence of distance image noise etc., the human body area is always accurate. It is difficult to obtain the center position of. Therefore, in the present invention, the human body posture is more accurately calculated by calculating the representative position based on the information (representative position vector) indicating the representative position stored in advance for each partial region and aligning the joint position relationship with the representative position. Find the position of. Processing for this is performed by the following partial region selection unit 104 and representative position calculation unit 105.

  The partial region selection unit 104 and the representative position calculation unit 105 calculate the representative position of the human body in the input image, and calculate the coordinates at which the joint positional relationship (posture) obtained by the posture estimation unit 103 is located in the target space. A schematic diagram of these processes is shown in FIG.

  FIG. 5A shows the identification result of the partial area and the hip joint position indicated by the representative position vector of the identified partial area. In FIG. 5A, four partial region identification results are shown for the sake of illustration, but in actuality, partial region identification results centered on each pixel (or pixel in a predetermined step) on the human body region 501 are shown. Exists.

  The recording unit 107 stores a representative position vector v for each partial area. Therefore, when a partial region identification is performed with a certain pixel on the human body region as the center and a result that the partial region is a partial region class is obtained, at the same time, the representative position vector (from the hip joint position to the hip joint position) Vector). For example, a direction vector 505 indicating the hip joint position is obtained from the partial region identification result 502. More specifically, the direction vector 505 is a three-dimensional vector indicating a hip joint position from three-dimensional space coordinates obtained from the distance value of the pixel position.

  Here, the identification result of the partial area is not always obtained correctly. The partial area identification result 503 in FIG. 5A shows a case where an incorrect identification result is obtained. As a result, the representative position vector 504 is also in the wrong direction. If the representative position is obtained using such a representative position vector related to the erroneous partial region identification result, the representative position (the hip joint position) cannot be calculated correctly. An object of the present invention is to accurately extract only a vector that represents a human body representative position by deleting an erroneous partial region identification result based on the posture obtained by the posture estimation unit 103, and to obtain a representative position with high accuracy. It is said. That is, the gist of the present invention is to select a partial region identification result that matches the estimated posture and to obtain a representative position from the selected partial region. By selecting only a partial area originally included in each posture and using it for representative position calculation, the representative position can be calculated with high accuracy.

As compatible partial region selection method on the estimated posture, in this embodiment, utilizes a partial region histogram h _p of the posture stored in the recording unit 107. The partial region selection unit 104 calls the partial region histogram of the posture class estimated by the posture estimation unit 103 from the recording unit 107. The partial region histogram of each posture is information on the partial region included in the posture class, and the bin of the histogram is the appearance probability of each partial region (323 in FIG. 3). Therefore, the partial region histogram of the estimated posture can be selected based on the estimated posture by selecting a partial region of the bin indicating a probability value equal to or higher than a predetermined value. The partial area selection unit 104 selects a partial area identification result in the person area based on the partial area selection criterion based on the partial area histogram.

  In the above description, the method of using the partial region histogram as it is as a partial region selection criterion has been described. However, a partial region included in each posture may be recorded in the recording unit 107 in advance as a partial region selection criterion. good. In order to perform the same processing as the method using the partial area histogram described above, based on the partial area histogram, prepare a correspondence table of each posture class and the partial region class included in the posture as a partial region selection criterion, Each posture class is stored in the recording unit 107. The partial area selection unit may acquire information on the partial area to be selected from the correspondence table based on the posture estimation result.

  FIG. 6 shows a processing flow of the partial area selection unit 104.

  First, in step S601, a partial region selection criterion is determined based on the posture estimated by the posture estimation unit 103. In the embodiment in which the partial region selection criterion is determined from the partial region histogram, first, the partial region histogram of the posture estimated from the recording unit 107 is called. Next, a partial region class having a predetermined probability or higher is determined and used as a partial region selection criterion. When the partial region selection criterion is already stored in the recording unit 107, the partial region selection criterion of the simply estimated posture is called from the recording unit 107.

  Next, in step S602, one is selected from the partial region identification results of the human body region of the input image, and the partial region class of the identification result is compared with the partial region class included in the partial region selection criterion. If the partial region class of the partial region identification result is included in the partial region selection criterion, it is determined in the next step S603 that the criterion is met, and the process proceeds to step S604 for selecting the partial region. On the other hand, if the partial region class of the identification result is not included in the partial region selection criterion, it is determined in step S603 that the criterion is not met, and the process proceeds to step S605 where the partial region is discarded.

  In step S606, it is confirmed whether all the partial region identification results of the human body region have been checked. If not completed, whether the partial region is selected by performing the processing of steps S602 to S605 for the next partial region identification result. Check whether to discard.

  FIG. 5B is a diagram illustrating a processing result of the partial region selection unit 104, and illustrates a result of selecting only a partial region identification result related to the estimated posture. Compared to FIG. 5A, the partial region identification result 503 that was not included in the partial region selection criterion of the estimated posture is excluded from the identification result.

  The representative position calculation unit 105 calculates a representative position from the representative position vector related to the partial region identification result selected by the partial region selection unit 104. FIG. 5C shows a representative position 510 obtained by aggregating vectors indicating waist center positions obtained from the partial region results of FIG. The representative position 510 can be obtained by calculating the average, median, extreme value of the distribution, etc. indicated by the end points of the representative position vectors in FIG. In the present embodiment, each representative position direction vector is a three-dimensional vector based on the human body surface (distance value of the distance image) at the center of the partial region, and the representative position 510 is obtained as a three-dimensional coordinate.

<Position and orientation determination unit>
Through the above processing, the posture estimation unit 103 obtains the human body posture (relative positional relationship between the joint positions of the human body), and the representative position calculation unit 105 obtains the representative position (coordinates of the hip joint position of the human body). It was. The position and orientation determination unit 106 integrates these pieces of information and generates a final output. Specifically, the hip joint position is moved to the hip joint position in the camera coordinate system at the time of posture estimation obtained by the representative position calculation unit 105 while maintaining the joint position relationship of the posture obtained by the posture estimation unit 103. .

  FIG. 7 is a diagram illustrating the processing of the position / orientation determination unit 106. FIG. 7A shows the representative position calculated by the representative position calculation unit 105. The representative position 710 is represented by three-dimensional coordinates in the camera coordinate system at the time of posture estimation. FIG. 7B shows the human body posture as a result of the posture estimation unit 103.

  The human body posture is represented as each joint position coordinate 712 of the human body in the learning camera coordinate system. Here, the joint position coordinates 712 include the hip joint position coordinates 711. The representative position determination unit 106 moves the hip joint position 711 of the posture estimation result to the representative position 710 of the posture estimation camera coordinate system by translation / rotation processing while maintaining the positional relationship of the joint position coordinates 712. Translation / rotation is performed by calculating a translation / rotation amount based on a difference (translation and rotation angle of the camera origin) between the hip joint position 711 and the representative position 710. As a result, as shown in FIG. 7C, information of each joint position coordinate in which the hip joint position is located at the representative position 710 is generated. The representative position calculation unit 105 outputs each joint position coordinate in the camera coordinate system at the time of posture estimation as a position / posture estimation result of the human body.

  The human body posture output from the position / posture determination unit 106 according to the present embodiment is a posture selected from known postures stored in the recording unit 107. Therefore, even if the rough postures coincide with each other, a terminal portion such as a limb may be out of the human body region of the input distance image. If the rough posture can be determined, the result of the position / orientation determination unit 106 may be used as it is. However, when the accuracy of the posture such as the joint position is further required, the accuracy of the position / orientation determination unit 106 may be used. Sometimes it is not enough.

  In order to solve this problem, the fitting process may be further performed using the output of the position / orientation determination unit 106 as an initial value. For the fitting process, for example, a process of fitting a human body model having a plurality of parts to an input distance image by a technique such as ICP can be used. An existing technique may be used for the fitting process, and details thereof are different from the gist of the present invention, and thus detailed description thereof is omitted here. Even when fitting processing is performed as post-processing, the accuracy of the initial position / posture is very important. Therefore, the overall performance can be improved by calculating the position / posture by the method of the present invention.

  In the second embodiment, an embodiment different from the first embodiment will be described regarding the relationship between the partial region class and the posture class and the representative position vector. Note that the description of the parts common to the first embodiment in the configuration is omitted.

  In the first embodiment, the embodiment in which one representative position vector is basically associated with one partial region has been described. The case where a plurality of representative position vectors are associated with a partial area is also briefly described. In this case, when a certain partial area identification result is selected, all representative position vectors associated with the partial area are calculated as representative positions. It was used for. In the second embodiment, a plurality of representative position vectors are associated with the partial area, and an attitude class is associated with each representative position vector. An embodiment in which a partial region related to the estimated posture is selected based on the estimated posture and a representative position vector related to the estimated posture is selected at the same time will be described.

  FIG. 8 is a diagram for explaining learning and information in the recording unit 107 according to the second embodiment.

  First, learning images 801 to 80m in FIG. 8 are images in which information on joint positions is associated with each learning image, similarly to the learning images in the first embodiment. The partial area information 811 and the posture information 821 are learned from these learning images.

  As for learning of the partial area information 811, first, learning of the partial area identifier 812 is performed. In the learning of the partial area classifier 813, as in the first embodiment, the partial area classifier is created by learning the image characteristics and the identification function of the partial area.

  Next, a histogram 813 indicating the degree of association with each posture class is learned for each partial region. This histogram 813 is hereinafter referred to as a posture histogram. The posture histogram 813 shows the probability of which posture class the overall posture is when a certain partial region is obtained. The posture histogram 813 is created as follows, for example.

  First, it is assumed here that the learning images 801 to 80m coincide with the posture classes p1 to pm. That is, the postures of all learning images are assumed to be posture classes that are estimated as they are. In order to create the posture histogram 813, first, a partial region classifier is executed for each learning image. As a result, the appearance frequency of the partial area in each learning image (posture class) is obtained. A posture histogram can be created by counting the appearance frequencies for each partial region. As another method, a posture histogram may be created by the following method. As described in the first embodiment, when creating a partial region classifier, partial images having similar local postures are collected from learning images. Since it is clear from which posture (learning image) the partial image for learning the partial region classifier is obtained, create a histogram of the posture class to which the partial image used for learning belongs for each partial region classifier It may be a posture histogram.

  Next, learning of representative position vectors in the second embodiment will be described. The representative position vector is information indicating the position of the hip joint of the human body from the position of the partial region identification result as in the first embodiment. In the second embodiment, a plurality of representative position vectors are associated with the partial area, and an attitude class is associated with each representative position vector. Hereinafter, this specific example will be described.

FIG. 8 shows that representative position vectors 814 are stored for the postures p1 to pm for each of the partial areas s1 to sn (v _s1p1 , v _s1p2 ,..., V _{snp4 in} FIG. 8). ). For example, the representative position vector V _s1p2 is a vector indicating the representative position for the posture class p2 when the partial region s1 is obtained.

  The representative position vector in the second embodiment is a total of the relative position relationship (representative vector) between the appearance position of the partial region identification result in the learning image and the representative position (waist joint position) for each posture (each learning image). Calculate with For example, first, a certain partial region classifier is identified in a certain learning image. Then, a representative position vector of one partial region in the learning image is calculated by obtaining an average of vectors from the identification result position to the representative position (lumbar joint position) of the learning image. This process is performed for all partial area classifiers in a certain learning image. Furthermore, representative vectors for each partial region are calculated for all learning images (posture classes). As a result, the result of associating the representative position vector of each partial area with each posture class can be acquired.

  As another method of creating the representative position vector, there is a method of creating based on a partial image for learning the partial region classifier. In the first embodiment, at the time of learning of a partial area classifier, collecting similar partial area images from all original learning images (clustering such as collecting partial area images having similar joint position relationships) has been described. . Each of these partial area images is originally known from which position of which learning image (posture class) is cut out, and at the same time, it is clustered in a similar posture, so it belongs to which partial area class It is also known. Therefore, for each partial region class, for a posture class with a certain partial region, the vectors to the hip joint positions are aggregated based on the learning image (posture class) from which the partial image used for learning is extracted and the position. The representative position vector is calculated.

Also, a representative position vector relating to a posture in which the partial area does not appear is not created. For example, in the partial region sn in FIG. 8, it can be seen from the posture histogram h _sn that the partial region sn does not appear in the postures p1 and p3. In this case, a representative position vector related to the postures p1 and p3 is not created for the partial region sn.

  The posture information 821 stores the joint position relationship 822 of the learning image in association with the numbers p1 to pm of each posture class. This is the human body posture (joint position relationship) output by the information processing apparatus 100 as in the first embodiment.

  By storing the above information in the recording unit 107, the learning of the second embodiment is completed.

  FIG. 9 is a diagram for explaining the processing of the partial region identification unit 102 and the posture estimation unit 103 in the second embodiment. The processing for estimating the posture of the human body in the input image using the information stored in the recording unit 107 will be described. FIG.

First, as in the first embodiment, the partial area is identified for the human body area 901 of the input image using the partial area information 811 of the recording unit 107, and a partial area identification result 902 is obtained. In the partial area in the second embodiment, the appearance frequency of the posture class when the partial area is identified is stored as a posture histogram. A histogram 905 (h _input ) of the entire human body region is created by adding up the posture histograms of the partial region identification results of the human body region 901. Since the histogram 905 of the whole human body region is a probability distribution of the posture indicated by the human body region, the posture histogram total is selected by selecting a bin of a posture class that indicates the most likely possibility. Posture can be estimated. In FIG. 9, since the bin of the posture class p1 shows the highest value, the posture of the input image human body is determined as p1. As a result, the posture estimation unit refers to the joint position relationship of the posture p1 from the posture information 821 recorded in the recording unit 107, and outputs it as the estimated posture result 903. The posture information to be output is the coordinate information 904 of each joint position in the learning camera coordinate system of each posture, as in the first embodiment.

  Next, the processing of the partial region selection unit 104 in the second embodiment will be described with reference to the flowchart of FIG. First, in step S <b> 1001, one partial region identification result of the human body region is selected, and it is verified whether it is related to the posture estimated by the posture estimation unit 103. The collation is performed based on the posture class information associated with the partial area, and is performed as follows, for example.

In FIG. 8, it has been described that histograms h _{s1 to} h _sn indicating which posture class is associated with each partial region when the partial region is identified are stored in the recording unit 107 as the partial region information. In the collation processing in step S1001, the posture class histogram of the selected partial region is referred to and the relationship between the selected partial region and the estimated posture is determined. For example, when the estimated posture bin of the posture histogram satisfies a predetermined appearance frequency, it is assumed that the estimated posture is related to the selected partial region. In the next step S1002, the process branches to step S1003 and step S1004 depending on whether or not the partial region identification result is related to the estimated posture. For example, when the partial region identification result indicates the partial region class s1, the posture class histogram h _s1 of the partial region s1 in FIG. 8 is referred to. If the estimated posture class is any one of p1, p2, and p3, the process proceeds to step S1003 on the assumption that the estimated posture and the partial region identification result are related. On the other hand, if the estimated posture class is p4, since it is learned from the learning image that the partial region s1 does not appear in the posture p4, the process proceeds to step S1004 on the assumption that there is no estimated posture and partial region identification result.

If it is determined that the partial region identification result is not related to the estimated posture, the partial region identification result is discarded in step S1004. On the other hand, if it is determined that the estimated posture and the partial region identification result are related, in step S1003, the partial region identification result is selected as a partial region used for representative position calculation, and the process proceeds to the next step S1005. In step S1005, the representative position vector related to the estimated posture is stored for the representative position vector associated with the partial region. For example, for the partial region s1 in FIG. 8, when the posture estimated by the posture estimation unit is p1, the representative position vector v _s1p1 of the posture p1 is stored.

  In the next step S1006, it is confirmed whether all the partial region identification results of the human body region have been checked. If not completed, the processing of steps S1001 to S1005 is performed on the next partial region identification result, and the partial region is adopted. Check whether to discard or discard.

  In the above description of the partial area selection unit, the flow of performing the processing step by step has been described in order to explain the gist of the present invention, but in practice the processing may be simplified. For example, in the above description, the relationship between the estimated posture and the partial region identification result is collated using the posture histogram. However, it is clear which posture has a representative position vector for each partial area. Therefore, the representative position vector related to the estimated posture may be referred to and stored from the partial region without using the posture histogram.

  This completes the collection of representative position vectors from the partial region identification results. The subsequent processing is the same as that of the representative position calculation unit 105 and the position / orientation determination unit 106 of the first embodiment, and thus description thereof is omitted.

In the first embodiment, the partial region for which the representative position is obtained is selected from one posture estimated by the posture estimation unit. In Example 3, an embodiment in which a partial region is selected from a plurality of postures estimated by the posture estimation unit will be described.
There are cases where the number of partial areas is reduced as a result of performing partial area selection according to the posture, for example, when the number of parameters of the partial area identification result is small. If the number of partial areas is extremely small, the number of representative position vectors used when obtaining the representative position decreases, and the representative position may not be obtained correctly. In such a case, the number of representative vectors for obtaining the representative position can be increased by selecting a partial region from a plurality of postures.

  The posture estimation unit according to the first embodiment calculates the similarity between the partial region histogram of the input image and the partial region histogram of the recording unit, and selects one posture having the highest similarity. In the third embodiment, a plurality of postures with high similarity are selected here. As a method for selecting a plurality of postures having a high degree of similarity, there is a method for selecting postures having a higher degree of similarity by a predetermined number. Alternatively, an attitude that indicates a degree of similarity equal to or greater than a predetermined threshold value may be selected. In addition, in order to secure the number of partial areas necessary for obtaining the representative position with a predetermined accuracy, a minimum number of partial areas is determined, and a plurality of postures are selected so as to satisfy the number of partial areas. Anyway.

  When a plurality of different postures are selected in selecting a plurality of postures, the partial position included in them and the representative position vectors associated with the partial regions are also greatly different, resulting in a decrease in representative position calculation accuracy. Therefore, it is desirable that the plurality of postures are similar postures.

  Therefore, a plurality of postures may be selected in consideration of the similarity between the postures to be selected. Hereinafter, the similarity between postures is referred to as an inter-posture distance in order to avoid confusion with the histogram similarity. For example, a method for considering the distance between postures is performed as follows. First, as described above, a plurality of postures are selected based on the histogram similarity. Here, the posture to be finally output is the posture having the highest similarity between the partial region histogram of the input image and the partial region histogram of the recording unit. Hereinafter, this posture is referred to as an output posture. Next, for a plurality of postures other than the output posture selected based on the degree of similarity, a distance between postures with the output posture is calculated. What is necessary is just to calculate the distance between attitude | positions based on the joint positional relationship preserve | saved linked | related with each attitude | position, for example, it can implement as follows. First, each joint position relationship is converted into a coordinate system with each waist joint position as the origin. Then, the distance between postures is calculated based on the sum of squares of the distance difference between the same joints (for example, between right wrists and between left ankles). The distance between postures of the output posture and other postures is calculated by the calculation method as described above. The posture determined to be similar to the output posture with the distance between postures is equal to or smaller than a predetermined threshold is set as a posture to be used for partial region selection together with the output posture.

  With the above processing, a plurality of postures for selecting the partial region are determined. The processing after the selection of the partial area may be performed in the same manner as in the first embodiment except that there are a plurality of posture references for selecting the partial area.

  Further, as a modification of the third embodiment, a configuration including a plurality of output postures may be employed. By providing a plurality of output postures, the partial region selection unit and the representative position calculation unit calculate a representative position for each output posture. The output of the position / orientation estimation unit is a plurality of positions / orientations. This configuration is suitable when there is no need to determine a single position / posture output by the information processing apparatus of the present invention and it is sufficient to output a plurality of possibilities. For example, when performing the fitting process based on the output of the information processing apparatus, the fitting process may be performed with a plurality of positions and orientations as initial values, and the most reliable result may be used as the final result.

<Effect of Example>
In the above, the example of embodiment of this invention was demonstrated by Examples 1-3.

  In the first embodiment, the partial area identification result is selected based on the estimated posture, and the representative position is obtained using information (representative position vector) indicating the representative position associated with the selected partial area with high accuracy. The representative position can be obtained. As a result, the position and orientation of the final output object can be obtained with high accuracy.

  In the second embodiment, compared to the first embodiment, the representative position can be calculated with higher accuracy by using a representative position vector limited to the posture.

  In the third embodiment, the amount of information for obtaining the representative position can be increased by outputting a plurality of estimated postures from the posture estimation unit and selecting a partial region based on the plurality of postures.

<Definition>
As mentioned above, although the Example demonstrated the case where a target object was a human body, the target object of this invention is not limited to a human body. In addition, the state in which the joint angles of the human body are different and the orientation with respect to the camera are referred to as a posture. However, when a rigid body having no joint is used as an object, only the orientation with respect to the camera may be handled as the posture.

  In addition, the partial region identification unit in the present invention includes a partial region identifier that identifies portions of various postures of the target object, and may be anything as long as the posture and the representative position are associated with each other. . Further, the information holding method is not particularly limited. As an example, in the first embodiment, the configuration in which the representative position vector indicating the representative position direction is associated with each partial area classifier has been described. In addition, the partial region classifier is associated with the posture by holding a partial region histogram for each posture. The partial region selection unit selects a partial region identification result of the target region based on the posture estimated by the posture estimation unit based on the relationship between the partial region and the posture.

  In addition, the representative position of the object in the invention may be any position as long as it is uniquely determined regardless of the posture of the object. Since the representative position is obtained from a plurality of representative position vectors, it is preferable to adopt a position near the center of the object. In the embodiment, the representative position is the hip joint position of the human body.

100 information processing apparatus, 102 partial region identification unit, 103 posture estimation unit,
104 partial region selection unit, 105 representative position calculation unit, 106 position and orientation determination unit,
107 recording unit, 301 to 30 m, 801 to 80 m learning image,
312,812 Partial region identifier, 322,822 posture (relative positional relationship of joints),
502,503 Partial region identification result,
504 representative position vector (information indicating the representative position),
510,710 Representative position (lumbar joint position)

Claims (8)

A partial area identifying unit for identifying different partial areas of the object;
A posture estimation unit that integrates the identification results of the partial areas and estimates the posture of the object;
A partial region selection unit that selects an identification result of the partial region based on the estimated posture;
A representative position calculation unit that calculates a representative position of the target object from information indicating the representative position associated with the selected partial region;
An information processing apparatus comprising: a position / orientation determination unit that calculates a position / orientation of the object from the representative position and the estimated attitude.   The information processing apparatus according to claim 1, wherein the posture estimation method of the posture estimation unit estimates a posture based on an appearance frequency of a partial region identification result in the region of the target object.   The partial region selection unit determines a partial region selection criterion based on the posture estimated by the posture estimation unit, and selects an identification result of the partial region based on the partial region selection criterion. Item 4. The information processing apparatus according to Item 1.   The partial area identification unit is associated with the probability of the object posture when the partial area is identified, and is associated with information indicating the representative position of the object posture. The information processing apparatus according to claim 1, wherein information indicating an identification result and a representative position of the partial area is selected based on the posture estimated by the posture estimation unit.   The posture estimation unit outputs a plurality of highly likely postures as posture estimation results, and the partial region selection unit selects a partial region identification result based on the plurality of postures. The information processing apparatus according to 1.   The information processing apparatus according to claim 5, wherein the plurality of postures output by the posture estimation unit are postures determined to be similar postures based on a predetermined reference indicating similarity between postures.   The object is a human body, and the posture is described by a relative positional relationship of joint positions of the human body, and the position / posture determination unit translates the joint position while maintaining the relative positional relationship of the joint position. The information processing apparatus according to claim 1, wherein the information processing apparatus matches the coordinates of the representative position by rotating. A partial area identification step for identifying different partial areas of the object;
A posture estimation step of integrating the identification results of the partial areas and estimating the posture of the object;
A partial region selection step of selecting an identification result of the partial region based on the estimated posture;
A representative position calculating step of calculating a representative position of the object from information indicating the representative position associated with the selected partial region;
An information processing method comprising a position and orientation determination step of calculating a position and orientation of the object from the representative position and the estimated orientation.