JP2015219868A - Information processor, information processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To create a model unique to an object without requesting the specific attitude of the object when creating an object shape model for estimating the position attitude of a deformation object in an image.SOLUTION: An information processor inputs an image, detects an object on the basis of the input image, evaluates relation between portions of the detected object on the basis of an object shape model for estimating the position attitude of the detected object, determines whether to calculate a model parameter on the basis of the evaluated result, calculates the model parameter on the basis of the determination result, and updates the object shape model held by holding means on the basis of the calculated model parameter.

Description

  The present invention relates to a technique for creating an object shape model for estimating the position and orientation of a deformed object in an image.

  2. Description of the Related Art Conventionally, a method for creating a model unique to a target object is known in order to estimate the position and orientation of a deformed object without using a marker using image information. For example, in Patent Document 1, in order to estimate the posture of a human body using a depth image, each part of the body of a subject taking a specific posture, such as a pose that stands or stands on both arms in a distance image A model specific to the subject is created by estimating the size of the subject.

US Patent Application Publication No. 2011/0052006

N. Dalal and B.M. Triggs “Histograms of Oriented Gradients for Human Detection,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, San Diego US. Vol. II, pp. 886-893. P. Felzenszwalb and D.W. Huttenlocher “Pictial Structures for Object Recognition International Journal of Computer Vision,” Vol. 61, no. 1, January 2005 M.M. Andriluka, S.M. Roth and B.M. Schiele, “Pictial Structures Revisited: People Detection and Articulated Pose Estimate,” IEEE Conference on Computer Vision and Pattern Revenue 9 Lubomir Bourdev and Jitendra Malik, “Poselets: Body Part Detectors Trained 3D Human Pose Annotations,” International Conference on Computer 9 V Computer R. Okada and S.M. Soatto, “Relevant Feature Selection for Human Pose Estimate and Localization in Clustered Images,” In Proceedings of the European Conference on Computer V8.

  However, in the method described in Patent Document 1, there is a problem that the target person needs to take a specific posture and the convenience is low.

  The present invention has been made in view of such problems, and an object of the present invention is to create a model specific to a target object without requiring the target object to take a specific posture.

  The information processing apparatus according to the present invention includes, for example, an image input unit that inputs an image, an object detection unit that detects the object based on the image input by the image input unit, and a position and orientation of the object. Obtained from holding means for holding an object shape model for estimation, part relation evaluation means for evaluating a relation between parts of the detected object based on the object shape model, and part relation evaluation means Based on the evaluation result, a determination unit that determines whether to calculate a model parameter, a model parameter calculation unit that calculates the model parameter based on a determination result by the determination unit, and a calculation by the model parameter calculation unit Updating means for updating the object shape model held in the holding means based on the model parameter.

  According to the present invention, it is possible to create a model specific to a target object without requiring the target object to take a specific posture.

It is a figure which shows an example of the input image which concerns on the 1st Embodiment of this invention. It is a figure which shows the structure of the information processing apparatus which concerns on 1st, 2nd and 3rd embodiment of this invention. It is a figure which shows an example of the target object of the information processing apparatus which concerns on the 1st Embodiment of this invention, and its model. It is a figure which shows an example of the structure of the model of the information processing apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of the model of the information processing apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of the process of the information processing apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows the processing flow of the information processing apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of the model of the information processing apparatus which concerns on the 2nd Embodiment of this invention. It is a figure which shows the processing flow of the information processing apparatus which concerns on the 2nd Embodiment of this invention. It is a figure which shows the processing flow of the information processing apparatus which concerns on the 3rd Embodiment of this invention. It is a figure which shows the example of the hardware constitutions of the information processing apparatus of this invention.

  Prior to describing each embodiment according to the present invention, a hardware configuration in which the information processing apparatus shown in each embodiment is mounted will be described with reference to FIG.

  FIG. 11 is a hardware configuration diagram of the information device according to the present embodiment. In the figure, a CPU 1110 comprehensively controls each device connected via a bus 1100. The CPU 1110 reads and executes processing steps and programs stored in a read-only memory (ROM) 1120. In addition to the operating system (OS), each processing program, device driver, and the like according to this embodiment are stored in the ROM 1120, temporarily stored in a random access memory (RAM) 1130, and appropriately executed by the CPU 1110. The input I / F 1140 is input as an input signal in a format that can be processed by the information processing apparatus 1 from an external device (display device, operation device, or the like). The output I / F 1150 is output as an output signal in a format that can be processed by the display device to an external device (display device).

  Each of these functional units is realized by the CPU 1110 developing a program stored in the ROM 1120 in the RAM 1130 and executing processing according to each flowchart described later. Further, for example, when hardware is configured as an alternative to software processing using the CPU 1110, arithmetic units and circuits corresponding to the processing of each functional unit described here may be configured.

(First embodiment)
The information processing apparatus according to the present embodiment determines whether or not the human body model for calculating the position and orientation on the image of each part of the pedestrian in the 2D image is suitable for the calculation of the model parameter. If it is, save the model. The “model” in the present proposal may be anything as long as it represents the shape and posture of an object composed of a plurality of parts using a plurality of parameters, for example, a joint model in which cylinders are connected, For example, a three-dimensional CAD model.

  Hereinafter, an example is shown using figures.

  The person 110 in FIG. 1 is walking sideways. The person 120 faces the front. A center line passing through the right lower leg of the person 110 is a dotted line 111, and a center line passing through the upper right leg is 112. The intersection of the dotted line 111 and the dotted line 112 corresponds to the joint point of the knee. The length of the lower leg of the person 110 can be calculated as the distance from this intersection point to the bottom surface of the right foot. Further, it is also possible to calculate the distance from the corner point to the bottom surface of the right foot by regarding the corner point as a knee joint point like the point 113. On the other hand, the center line of the upper right thigh and the center line of the right lower thigh of the person 120 coincide with the dotted line 121, and the point corresponding to the knee joint point cannot be localized. Further, since there is no image feature such as a corner point, the knee joint point cannot be localized using the corner point. That is, the estimation of the size of the right leg upper leg and the right leg lower leg of the person 120 is not accurate. Therefore, it is inappropriate to update the models of the upper right thigh and right lower leg of the person 120. The information processing apparatus according to the present embodiment determines whether the situation is appropriate for model update based on the estimation result of the relative positional relationship between the upper right thigh and the right lower thigh, and updates the model. In the present embodiment, the upper right thigh and the right lower thigh will be described as an example. Needless to say, the present invention can also be applied to other parts. Therefore, with respect to each part, it is possible to determine whether the model parameter calculation is appropriate by the method described in this embodiment, and it is possible to store only the model parameter of the conforming part.

  As illustrated in FIG. 2, the information processing apparatus 200 according to the present embodiment includes an image input unit 201, an object detection unit 202, a posture estimation unit 203, a part relationship evaluation unit 204, a model parameter calculation unit 205, a model And a parameter storage unit 206.

  The image input unit 201 receives each frame image (real space image) sequentially transmitted from the imaging apparatus 100 and transfers it to the subsequent image feature detection unit 202. The image input unit 201 is realized by an analog video capture board if the output of the imaging apparatus 100 is an analog output such as NTSC. Further, when the output of the imaging apparatus 100 is a digital output such as IEEE1394, it is realized by, for example, an IEEE1394 interface board. The image input from the image input unit 201 may be an RGB image or a gray scale image. Moreover, the image image | photographed in real time may be sufficient and the image image | photographed beforehand may be sufficient. Further, it may be an RGB image or a gray scale image.

  The object detection unit 202 detects a target object from the image input unit 201 from the image.

  The posture estimation unit 203 estimates the position and posture on the two-dimensional image of each part of the person in the image obtained from the image input unit.

  The part relationship evaluation unit 204 evaluates the relative relationship between parts from the information obtained from the posture estimation unit 203. Here, the evaluation value refers to parts likelihood, relative angle, cornerness of image features, similarity to a specific pattern, and the like, which will be described in detail later.

  The model parameter calculation unit 205 selects a part for calculating the model parameter based on the evaluation value obtained by the part relationship evaluation unit 204, and calculates each parameter.

  The model data storage unit 206 holds a human body model (object shape model) used by the posture estimation unit 203.

  Below, the process of the information processing apparatus 200 in this embodiment is demonstrated using the flowchart of FIG.

(Step S101)
First, in step S101, one frame of image is input from the image input unit 201.

(Step S102)
Subsequently, in step S102, the object detection unit 202 detects whether or not there is a person in the image input in step S101. If detected, the process proceeds to step S103. If not detected, the process returns to step S101. As a detection method, since a person is an object in the present embodiment, for example, the method described in Non-Patent Document 1 can be used. Of course, it is not limited to this method.

(Step S103)
Subsequently, in step S103, the posture estimation unit 203 estimates the position and posture on the image of each part of the human body detected in step S102. Hereinafter, the estimation method will be described in detail.

  In the present embodiment, the model for estimating the position and orientation is the Pictorial Structure model described in Non-Patent Document 2 or Non-Patent Document 3. FIG. 3 is a diagram in which the position and orientation of each part of a person who walks sideways is estimated using the Pictorial Structures model. The image 301 is an image in which a bounding box corresponding to each part of the person 110 walking sideways is superimposed with a dotted line. The image 302 is an image that is separately drawn for only the bounding box of the image 301. 311 is the trunk, 312 is the head, 313 is the upper right thigh, 314 is the right lower thigh, 315 is the left upper thigh, and 316 is a bounding box corresponding to the left lower thigh. The parts corresponding to the right arm and the left arm are omitted for the sake of simplicity, but it goes without saying that it can be extended to a model including them.

  Each bounding box of the image 302 has width and length parameters. When the posture estimation is started, a predetermined initial value is given to this parameter. The information processing apparatus evaluates the relative positional relationship between the lower leg and the upper leg while the subject is walking, and corresponds to the lower leg and the upper leg when the target person has a predetermined range of positional relationship. Update the width and length of the bounding box.

In the Pictorial Structures model, the posterior probability of the position and orientation L = {l _i } of each part when the input image I is given is expressed by the following equation (1).

Here, Ψ is a part connection score determined by the positions and orientations l _i and l _j of the parts _i and _j , and Φ is a part likelihood for evaluating the part-likeness of the part i. E is a set of edges connecting the parts. As shown in FIG. 4, the connection between the parts in the present embodiment connects the trunk, the head, the upper right thigh, and the left upper thigh, and the upper right thigh and the right lower thigh, and the left upper thigh and the left lower thigh. It has a structure. l _i is composed of the center position (x _i , y _i ) and the rotation angle θ _{i of} the bounding box corresponding to the part i. L = {l _i } that maximizes the posterior probability P (L | I) expressed in the equation (1) is the part position and orientation to be obtained.

FIG. 5 shows an example of the relative position and orientation of two adjacent bounding boxes. A point 501 belongs to a bounding box 313 corresponding to the upper right thigh and is a joint point connected to the right lower leg, and a point 502 belongs to the bounding box 314 corresponding to the right lower leg 314 and is a joint point connected to the upper right thigh. At this time, the coordinates of the points 501 and 502 are expressed as (x ₃₄ , y ₃₄ ) and (x ₄₃ , y ₄₃ ), respectively. An angle 503 formed by the bounding boxes 313 and 314 is expressed as θ ₃₄ .

  The part connection score Ψ is calculated by the following formula, for example.

In Expression (2), (x _ij , y _ij ) is the image coordinates of the joint point of the bounding box corresponding to the part i and connected to the part j, and θ _ij is the bounding box corresponding to the part i and the part j. It is an angle to make. In the third term, κ and μ are constants.

Further, the part likelihood Φ (l _i ) of the part _i is calculated as follows, for example. A bounding box having a width w _i and a length h _i is arranged at the center position (x _i , y _i ) with an angle θ _i rotated. After the image in the bounding box is cut out and resized to the width w _i ′ and the length h _i ′, a feature vector in which the Shape Context feature amount described in Non-Patent Document 3 is extracted is prepared. Using a discriminator that has previously learned a sample image group having a plurality of widths w _i ′ and length h _i ′ corresponding to the part i using the Support Vector Machine, the score of the feature vector is calculated, and the part likelihood of the part i Let Φ (l _i ).

The width w _i and the length h _i of the bounding box corresponding to the part i are given appropriate initial values at the start of the process. As will be described later, while the subject is walking, Updated.

(Step S104)
Subsequently, in step S104, the part relationship evaluation unit 204 evaluates whether the relative positional relationship between the upper right thigh and the right lower thigh is appropriate. Hereinafter, the evaluation method will be described in detail.

It is assumed that L = {l _i } that maximizes the posterior probability of Expression (1) is calculated by the posture estimation unit 203. First, the part relationship evaluation unit determines whether or not the part likelihoods Φ (l ₃ ) and Φ (l ₄ ) of the target part exceed predetermined threshold values. When both Φ (l ₃ ) and Φ (l ₄ ) exceed the threshold, the part relationship evaluation unit 204 further evaluates the relative angle θ ₃₄ between the target parts. For example, when the relative angle θ ₃₄ is not less than 45 degrees and less than 135 degrees, it is determined in step S105 described later that the relative angle is appropriate. If it is determined that the relative angle is appropriate, the model parameters are calculated in step S106, and the width and height of the bounding boxes of the upper right thigh and the right lower thigh are stored in step 107.

In addition, the part relation evaluation unit 204 can also evaluate the part relation based on the image feature including the target part. An example is shown using FIG. 601 in FIG. 6 is a partial image around the right knee. The center coordinates of the frame from which the partial image is cut out are, for example, the position coordinates of the joint point connected to the right lower leg of the bounding box corresponding to the upper right thigh estimated by the posture estimation unit 203, and the bounding box corresponding to the right lower thigh is the upper right It can be set to the midpoint of the position coordinates of the joint point connected to the thigh. Also, the size of cutting out the partial image can be set to a size proportional to the size of the bounding box of the upper right thigh, for example. The partial image 601 is evaluated with an evaluation value Λ expressed by the following equation (3).
Λ = det (A) −k trace ² (A) (3)
Here, A is a second moment matrix represented by the following formula (4).

In equation (4), w (u, v) is a weight in (u, v), and I _x and I _y are values of luminance gradients in the x and y directions in (u, v), respectively. When the value of Λ is within the predetermined range, the width w _RU and the length h _RU of the bounding box of the upper right thigh are stored. Similarly, the bounding box width w _RL and length h _RL of the right lower leg are also stored. The evaluation value Λ in Expression (3) is generally an image feature amount for evaluating the luminance gradient direction in the partial image. As another image feature amount instead of Λ, for example, an image moment value in a partial image, a determinant value of a Hessian matrix, or the like may be used, and the present invention is not limited to this embodiment.

  Further, the part relationship evaluation unit 204 can also perform evaluation by comparing the partial image 601 in FIG. 6 with a specific pattern. As described in Non-Patent Document 4, it is possible to learn a pattern of a partial image including a part having a specific arrangement relationship in advance, calculate the similarity between the pattern and the partial image 601, and use it as an evaluation value. In that case, the model is updated when the similarity is equal to or higher than a predetermined level in step S106.

(Step S105)
Subsequently, in step S105, the evaluation result in step S104 is received, and if calculation of the model parameter is necessary, the process proceeds to step S106. If it is not necessary to calculate the model parameter, the process returns to step S101.

  A method for determining whether to calculate a model parameter in response to the evaluation result in step S104 will be described. As a method for determining whether to calculate a model parameter, there is a method of calculating a model parameter when a predetermined threshold value is exceeded. As described above, various evaluation values such as a relative angle, a cornerness of a partial image, and an image pattern are conceivable as described above, and are not limited to those described here. Therefore, the threshold value in step S105 also varies depending on the evaluation value. In addition, the model update unit 205 compares the current evaluation value of the evaluation value of the part relationship evaluation unit 204 with the past evaluation value, and calculates the model parameter when the current evaluation value is higher. Conceivable. As a result, the model can be updated when the length of the part can be calculated more accurately than in the past history. Furthermore, when the similarity with a specific pattern is used as the evaluation value, the past history of the similarity with the specific pattern is saved, and the model parameter is used when the similarity is higher than the past history. Can also be calculated.

(Step S106)
Subsequently, in step S106, the model update unit 205 calculates a model parameter.

In the present embodiment, the width and length of the bounding box corresponding to the right upper leg and the width and length of the bounding box corresponding to the right lower leg are calculated. For example, when updating the model of the upper right thigh, the following is performed. While fixing the position and orientation _{l RU} of bounding box corresponding to the upper right thigh, the width _{w RU} and the length _{h RU} changing a part likelihood Φ _{(l RU)} width that is maximum _{w RU} and the length h _RU is calculated. The calculated width w _RU and length h _RU are set as the new width and length of the upper right thigh.

(Step S107)
Subsequently, in step S107, the model update unit 205 stores the model parameter using the model parameter calculated in step S106. After storing the model parameters, the process returns to step S101.

More specifically, for example, a process of replacing the parameter value newly calculated in step S106 with the value stored in the model storage unit 206 is performed. The model update unit 205 can also use model parameter values corresponding to past evaluation value histories. For example, it is assumed that the evaluation values (pattern similarity) in the past four updates are α _t−1 , α _t−2 , α _t−3 , and α _{t− 4} , and a new evaluation value α _t is obtained. . In addition, model parameters that maximize the part likelihood Φ (l _RU ) at each update are p _t , p _t−1 , p _t−2 , p _t−3 , p _t−4 , where p = (w _RU , H _RU ). At this time, the parameter value is updated to p in the following expression. As a result, there is an effect that only information with higher accuracy can be used for updating without depending on the past history. Effect that model updating unit 205 assigns a weight according to the evaluation value of part relationship evaluation unit 204 and updates the model, thereby enabling updating of the model according to the reliability of the calculated part parameter. Is shown.

  As used herein, the “parameter relating to the part” and the “part parameter” may be any parameters that define the shape of the part of the object, for example, the length of a line segment, the length or diameter of a cylinder, It is the length of each axis of the ellipsoid, the length of each side of the rectangular parallelepiped, and the like.

  As described above, in the present embodiment, it is possible to determine whether or not the timing is suitable for accurately calculating the length of the region by evaluating the relative position and posture of the upper right thigh and the right lower thigh.

(Second Embodiment)
The information processing apparatus according to the present embodiment updates a model for estimating the posture of a person in an image that receives a distance image and an RGB image while the posture of the target person is being estimated. .

  The information processing apparatus according to the present embodiment has the same configuration as that of the information processing apparatus according to the first embodiment.

  The image input from the image input unit 201 is assumed to be distance image data and RGB data. The distance image data may be data obtained by any distance image acquisition method such as stereo, pattern light projection, and range scan. These distance images may be images taken in real time or images taken in advance. The input distance image has a size of 320 × 240 pixels, for example, and the distance from the imaging device is stored in each pixel. It is assumed that the RGB data has a resolution similar to or higher than that of the distance image data and has been corrected so as to be compatible with the distance image.

  The object detection unit 202 extracts human candidates in the distance image obtained from the image input unit 201.

  The posture estimation unit 203 sequentially estimates the posture of the human body in the human body candidate region in the distance image obtained by the object detection unit using the distance image and the human body model of FIG.

  The model update unit 205 updates parameters of a model obtained by approximating a human body used in the posture estimation unit 203 with a cylinder according to the evaluation in the part relationship evaluation unit 204. The model storage unit 206 stores the human body model used by the posture estimation unit 203.

  Below, the process of the information processing apparatus 200 in this embodiment is demonstrated using the flowchart of FIG.

(Step S201)
First, in step S201, the image input unit 201 inputs one frame of an image.

(Step S202)
Subsequently, in step S202, the object detection unit 202 detects whether there is a person in the image input in step S201. If detected, the process proceeds to step S203. If not detected, the process returns to step S201.

  First, the object detection unit 202 detects a moving object in the distance image input from the image input unit 201. The moving object is detected by the background difference and the inter-frame difference. Next, the orientation of the human body is specified for the region in the RGB image corresponding to the detected moving body region by the method described in Non-Patent Document 5, etc. Estimate the 3D position and orientation. The detected position and the estimated posture are used as the initial position and initial posture of the target human body. Further, when the position and orientation of the human body are calculated in the previous frame, they can be set as the initial position and the initial orientation.

(Step S203)
Subsequently, in step S203, it is confirmed whether there is a result of estimating the position and orientation in the previous frame. If there is a result, the process proceeds to step S205. If there is no result, the process proceeds to step S204.

(Step S204)
In step S204, the posture estimation unit 203 estimates the initial posture of the human body detected in step S202. After execution of step S204, the process returns to step S201.

(Step S205)
In step S205, the posture estimation unit 203 estimates the current human body position and posture using the current human body model and the position and posture estimation result obtained in the previous frame.

As the human body model, a three-dimensional extension of the Pictorial Structures model is used. The parts include a torso 711, a head 712, an upper right arm 713, a right forearm 714, a left upper arm 715, a left forearm 716, an upper right thigh 717, a right lower leg 718, a left upper leg 719, and a left lower leg 720. The model of each part is composed of a cylindrical mesh, and has a radius and a height as parameters. Each cylinder can be rotated and translated. Elements of the position and orientation l _i of the part i are the position (x _i , y _i , z _i ) and the rotation (α _i , β _i , γ _i ).

  First, the posture estimation unit 203 obtains a model having an initial posture obtained by the processing of the object detection unit 202, a central axis of point cloud data that is a human body candidate region in the depth image, and a central axis of a torso part of the model. The models are arranged in the coordinate space of the point cloud data so that

The position and orientation of each part can be obtained by obtaining L = {l _i } that maximizes the posterior probability of Expression (1). However, the part connection score Ψ is expanded in three dimensions as in the following formula (6).

Here, φ _ij is an angle formed by the central axis of the cylinder corresponding to the part i and the central axis of the cylinder corresponding to the part j. Further, the part likelihood Φ (l _i ) is calculated using the following equation (7).

Here, m _i (k) is the k-th point of the cylindrical mesh of the model corresponding to the part i, and _pk is the point coordinates in the three-dimensional space obtained from the distance image. The translation vector t _i and the rotation matrix R _i correspond to the position (x _i , y _i , z _i ) and posture (α _i , β _i , γ _i ) of the part _i .

  The part relationship evaluation unit 204 evaluates the relative position and posture of each part obtained by the posture estimation unit 203 in the same manner as in the first embodiment. For example, when the evaluation values regarding the upper right thigh and the right lower thigh are appropriate, the model parameter calculation unit 205 described later calculates the model parameters of the upper right thigh and the right lower thigh.

(Step S206)
Subsequently, in step S206, the part relationship evaluation unit 204 evaluates whether the relative positional relationship between the upper right thigh and the right lower thigh is appropriate.

  In addition, the part relationship evaluation unit 204 may evaluate the image feature of a local distance image including the upper right thigh and the right lower thigh. For example, a distance image of a local area corresponding to the right knee is cut out, the distance value is regarded as a luminance value, and the cornerness is evaluated by Expression (3). If the cornerness is a certain value or more, models of the upper right thigh and the right lower thigh Calculate the parameters. At this time, when the knee is fully extended, the object in the local image is linear, and the cornerness index based on the second moment matrix of the gradient of the distance image is low.

  As another method, a method of learning a pattern of a distance image of a local region corresponding to the right knee in advance and calculating model parameters of the upper right thigh and the right lower thigh when the patterns match can be considered. First, a three-dimensional curved surface of the knee when the knee joint angle is 90 ° is created in advance. For example, a three-dimensional curved surface can be created from a CG character, or can be obtained by taking a distance image of a corresponding part of an actual human body and then calculating a curved surface that approximates a point group of the distance image. In the area including the lower end of the upper right thigh and the upper end of the right lower thigh, when fitting the 3D curved surface of the knee when the knee joint angle is 90 ° and the error after fitting is lower than the threshold, the upper right thigh and the right lower thigh Save model parameters of. Similarly, for the upper right arm and the right forearm, fitting the 3D curved surface of the elbow when the elbow joint angle is 90 °, and storing the model parameters of the upper right thigh and right lower leg when the error after fitting is lower than the threshold Can also be done.

(Step S207)
Subsequently, in step S207, the evaluation result in step S206 is received, and if calculation of the model parameter is necessary, the process proceeds to step S208. If calculation of the model parameter is not necessary, the process returns to step S201.

(Step S208)
Subsequently, in step S208, the model parameter calculation unit 205 calculates the radius and length of the cylindrical model corresponding to the upper right thigh and the radius and length of the cylindrical model corresponding to the right lower leg.

  For example, in the case of the model of the upper right thigh, the part likelihood represented by the equation (7) is calculated for a plurality of cylindrical models having different radii and lengths with the position and orientation parameters being fixed, A cylinder with a radius and length that increases the likelihood of parts is adopted as a new model.

(Step S209)
Subsequently, in step S209, the model parameter calculation unit 205 updates the model stored in the model parameter storage unit 206 using the model parameter calculated in step S208. After updating the model, the process returns to step S201.

  As described above, in the second embodiment, even when a three-dimensional human body position / posture is estimated using a distance image and an RGB image (two-dimensional image), the same effect as described in the first embodiment is used. It shows that an effect can be obtained.

(Third embodiment)
Although the first embodiment and the second embodiment are directed to the human body, the information processing apparatus according to the present invention can also target objects other than the human body. In the present embodiment, a quadruped walking animal such as a horse or a dog is targeted. Although these animals have the same part configuration and connection relationship between the parts, parameters such as the length of each part vary greatly depending on the type. Therefore, in the object detection unit, the type of the target object is determined, and the posture of the target object is estimated and the model parameters are stored using a model corresponding to the determination result. Also in this embodiment, it is assumed that the model parameters of the upper right thigh and the right lower thigh are stored as in the first and second embodiments.

  The present embodiment is realized by the same configuration as the second embodiment. Hereinafter, only the modules different from those of the second embodiment will be described.

  The object detection unit 202 in the present embodiment detects a target animal. The target animals are horses, dogs, and cats, for example. Also, each animal has a different model for each type.

  The model storage unit 206 stores a model corresponding to the target animal and its type. Each part of the model is represented by a cylinder whose parameters are radius and length.

  Below, the process of the information processing apparatus 200 in this embodiment is demonstrated using the flowchart of FIG.

(Step S301)
First, in step S301, the image input unit 201 inputs one frame of an image.

(Step S302)
Subsequently, in step S302, the object detection unit 202 detects whether a target object exists in the image input in step S301. If detected, the process proceeds to step S303. If not detected, the process returns to step S301.

  The object detection unit 202 first performs moving object extraction in the same manner as in the second embodiment, and extracts a candidate region of the target animal from the distance image input from the image input unit 201. Next, a partial image of the RGB image corresponding to the extracted candidate area is extracted. Next, it is determined which type of which animal the extracted partial image matches. In the determination, the target type is determined in advance by a multi-class detector in which each class is assigned to a different type. Further, as in the second embodiment, the orientation of the target and the position of each part on the image coordinates are calculated, and the initial position and orientation of the target are calculated based on them.

(Step S303)
Subsequently, in step S303, it is confirmed whether there is a result of estimating the position and orientation in the previous frame. If there is a result, the process proceeds to step S305. If there is no result, the process proceeds to step S304.

(Step S304)
In step S304, the posture estimation unit 203 estimates the initial posture of the object detected in step S302, and selects a model for three-dimensional posture estimation corresponding to the object from the model storage unit 206. After execution of step S304, the process returns to step S301.

  The posture estimation unit 203 estimates the posture of the target in the same manner as in the second embodiment. As a model used for target posture estimation, a model corresponding to the type determination by the object detection unit 202 is selected from the model storage unit 206 and used.

(Step S305)
In step S305, the posture estimation unit 203 estimates the position and posture of the current object using the current object model and the position and posture estimation result obtained in the previous frame.

(Step S306)
Subsequently, in step S306, the part relationship evaluation unit 204 evaluates whether the relative positional relationship between the upper right thigh and the right lower thigh is appropriate.

(Step S307)
Subsequently, in step S307, the evaluation result in step S306 is received, and if calculation of the model parameter is necessary, the process proceeds to step S308. If calculation of the model parameter is not necessary, the process returns to step S301.

(Step S308)
Subsequently, in step S308, the model parameter calculation unit 205 calculates the radius and length of the cylinder model corresponding to the upper right leg and the radius and length of the cylinder model corresponding to the right lower leg.

(Step S309)
Subsequently, in step S309, the model parameter calculation unit 205 stores the model parameter of the corresponding object stored in the model parameter storage unit 206 using the model parameter obtained in step S308. After storing the model parameters, the process returns to step S301.

  As described above, in the third embodiment, the present invention is applied to objects other than the human body such as horses, dogs, and cats. Thus, if it is a deformed object consisting of one or more parts, the same method as shown in the first embodiment or the second embodiment can be applied.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

DESCRIPTION OF SYMBOLS 200 Information processing apparatus 201 Image input part 202 Object detection part 203 Posture estimation part 204 Part relation evaluation part 205 Model update part 206 Model storage part

Claims (11)

An image input means for inputting an image;
Object detection means for detecting the object based on the image input by the image input means;
Holding means for holding an object shape model for estimating the position and orientation of the object;
A part relation evaluation unit that evaluates a relation between parts of the detected object based on the object shape model;
Based on the evaluation result obtained from the part relationship evaluation means, a determination means for determining whether to calculate a model parameter;
Model parameter calculation means for calculating the model parameter based on the result of the determination by the determination means;
An information processing apparatus comprising: an updating unit that updates an object shape model held by the holding unit based on the model parameter calculated by the model parameter calculating unit.   The information processing apparatus according to claim 1, wherein the image input to the image unit is at least one of a distance image and a two-dimensional image.   The information processing apparatus according to claim 1, wherein the part relationship evaluation unit evaluates a relative position and orientation of the first part and the second part of the object.   The information processing apparatus according to claim 3, wherein the part relationship evaluation unit evaluates an angle formed by the first part and the second part.   The information processing apparatus according to claim 1, wherein the part relationship evaluation unit evaluates image features of an image including a first part and an image including a second part of the object.   The said part relationship evaluation means evaluates the similarity degree of the image learned in advance and the image containing the 1st site | part of the said object, the image containing the 2nd site | part, and the said 1st site | part The information processing apparatus described.   The information processing apparatus according to claim 1, wherein the updating unit compares the evaluation of the part relationship evaluation unit with a past evaluation.   The information processing apparatus according to claim 1, wherein the updating unit compares the evaluation of the part relationship evaluation unit with a predetermined level.   The said update means gives the weight according to the evaluation of the said part relation evaluation means to the value computed by the said part relation evaluation means, The said object model is updated, The said Claim 1 characterized by the above-mentioned. Information processing device. An image input process for inputting an image;
An object detection step of detecting the object based on the input image;
A part relation evaluation step for evaluating a relation between parts of the detected object based on an object shape model for estimating a position and orientation of the object held by a holding unit;
A determination step of determining whether to calculate a model parameter based on the evaluation result obtained from the part relationship evaluation means;
Based on the determination result in the determination step, a model parameter calculation step for calculating the model parameter;
An information processing method comprising: an updating step of updating an object shape model held in the holding unit based on the model parameter calculated in the model parameter calculating step.   A program for causing a computer to execute the information processing method according to claim 10.

Images