JP3855050B2 - Clothing state estimation method and program - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a method and a program for calculating a state of a flexible object that deforms greatly, such as clothing, from observation image information. This is an indispensable technology for handling a soft object by the robot, and when this is realized, the robot can be used in a more general situation.
[0002]
[Prior art]
Research on linear objects such as visual recognition for rope manipulation has been done, but for objects that are further spread like clothing, the degree of freedom of deformation greatly increases, and complex self-shielding also occurs, The problem is even more difficult. Although a method has been proposed in which all possible appearances are actually input in advance and this appearance is used as a model, a lot of labor is required for each target clothing.
[0003]
[Problems to be solved by the invention]
The present invention solves such problems and develops a technique for estimating the state of a subject that is complicated and greatly deformed like clothing using a simple target model and a simple image input device. It is aimed.
[0004]
[Means for Solving the Problems]
The clothing state estimation method and program of the present invention estimate the state of the target clothing from the input image. A model representing the rough basic shape of clothing is created, and this model is used to predict all the shapes that can occur when gripping at one point. In accordance with the analysis of the predicted shape, the feature amount obtained from the input image obtained by photographing the target clothing held at one point is processed, and the predicted shape is calculated by calculating which predicted shape is closest to the target shape.
[0005]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, the present invention will be described based on examples. FIG. 1 shows a flow chart when the present invention is applied to shape estimation of a trainer.
step 1:
Create a model that represents the basic shape of clothing.
[0006]
FIG. 2 shows an example of the case where the target is a trainer, which is represented by a planar model composed of nodes capable of connecting up to four, not assuming that the front body and the back body are separated. In the figure, N1 to N13 represent node points, and the left and right sleeves and the body are treated as different parts. The distance between nodes is calculated from rough dimensions such as body width, length, and sleeve length.
[0007]
Step 2:
The model is used to predict the shape that can occur when gripping at a single point.
Various methods can be used for this prediction method according to the required prediction accuracy. Simple prediction of the shape when grasped in the air at a point close to the edge of the clothing can be realized as follows.
Hereinafter, this prediction method is referred to as prediction method A. Assume that the position of a certain node is already fixed. If the current node is the first fixed node by gripping, this direction is the vertical upward direction (0, 1).
[0008]
The distance between the nodes is fixed, and the direction of the adjacent node that has not been fixed is determined at an angle θ determined according to the state of the adjacent node, as shown in FIG. 3a. In (1) to (6) of FIG. 3a, the white circle is the current node, the double circle is already determined when the position of the current node is determined, and the parent node, the black circle is still The position is not determined and represents a child node. (2) to (6) in FIG. 3a show the possible states of adjacent nodes, and the angles θ1, θ2, and θ3 in these drawings are changed according to the rigidity of the object.
[0009]
FIG. 3b shows an example determined by this rule when the node 10 is gripped. Consider the influence of gravity as follows. The joint part where the different parts are joined corresponds to the two shoulder parts in the model of FIG. 2, but with respect to the lower part supported by the connection, the center of gravity is directly below the joint part. Rotate around the joint. FIG. 3c shows the shape after this operation.
FIG. 4 shows the result when the gripping node is changed. States 1 to 13 indicate predicted shapes corresponding to the case where each node in FIG. 2 is held.
[0010]
Step 3:
Analyze the predicted shape according to the required task.
First, the predicted shapes are classified into the states to be identified necessary for performing the task. When a task of “gripping with both shoulders” is given, the state of the predicted shape is classified into the following three classes.
Situation A: Already gripped by one shoulder. State 1, 3. => Indicates the remaining shoulder position.
Situation B: Not gripped by shoulder, but at least one shoulder is convex (easy to grip). State 2, 10, 11, 12, 13. => Indicates the position of the first shoulder.
Situation C: Otherwise (situations where it is difficult to grip with both shoulders concave). State 4, 6, 7, 8,
9. => Indicate the position of situation B.
[0011]
In this example, the tip of the sleeve is always located at the lowest point in all predicted shapes, and it is easy to identify, and if this point is grasped, it can be brought to the state 11 or 13 state B. Therefore, in situation C, specifically, => indicates the lowest point.
Based on the predicted shape, a feature that can be extracted relatively easily and robustly from the detected clothing region is used for state determination. In this example,
I) The position of the lowest point.
II) The center position of the body part.
It is.
[0012]
Step 4:
From the camera, an image obtained by photographing the target clothing held at one point of the edge is input.
Step 5:
Extract clothing regions from the image.
Step 6:
The state is estimated by comparing the extracted area with each predicted shape model as follows.
First, candidates are selected in ascending order of the difference in distance from the actual gripping point to the lowest point. Among the extracted areas, a horizontal area having a width equal to or larger than a threshold is set as a body area candidate. If the vertical position of the body center (node 5) of the predicted shape model is not included in this region, it is deleted from the candidate.
[0013]
Step 7:
The position of each part in the clothing area is obtained as follows, and the point to be gripped next is indicated.
When the obtained candidate states are situations A and B, these shape models are superimposed on the image based on the position of the gripping point. For a node not included in the clothing area, the node is moved toward the horizontal center until it is included in the area. By this processing, the part corresponding to each node on the image is estimated, and the point inside the region boundary closest to the nodes 1 and 3 corresponding to the shoulder is indicated as the next gripping position.
If the first candidate is situation C, the lowermost point of the clothing area is designated as the next gripping position.
[0014]
Step 8:
It is verified whether or not the previous state estimation is correct by predicting the shape when the calculated indication point is gripped and checking whether it matches the state actually observed after gripping.
Here, in the two-point gripping state, the area shape indeterminate element is reduced as compared with the one-point gripping state, and therefore the comparison between the predicted shape and the observation region can be performed by relatively simple template matching. Basically, it is only necessary to match the gripping position on the model with the actual gripping position and examine the overlap.
`` Instruct the next gripping point based on state estimation '' (steps 6 and 7), `` grip '', and `` verify '' (step 8) can be taken to the `` grip with both shoulders '' situation it can. However, in the second and subsequent state estimation, the state can be limited by the previous prediction, and the processing is made fast and robust.
[0015]
Next, a description will be given of performing clothing deformation prediction with higher accuracy by representing the target clothing by a more complicated model as shown in FIG. Hereinafter, this prediction method is referred to as prediction method B.
For the target trainer, based on the three rough sizes of the body width, the body height, and the sleeve length, a simple planar model that does not assume that the front body and the back body are separated is created as shown in FIG. It consists of nodes representing 20 representative points, and each node is connected by a spring as shown by a straight line in the figure. There are three types of springs:
[0016]
The K_1 type spring connects each node and one point of its four neighboring nodes. The K_2 type spring connects each node and one point obtained by removing the four neighboring nodes from the eight neighboring nodes. The K_3 type spring connects each node to the adjacent node with one space between them. However, it is introduced only in the same part (body, right sleeve, left sleeve). For example, between N1-N3, N1-N7, N3-N19 nodes. The K_1 and K_2 types express the elasticity of clothes, and the K_3 type expresses the ease of bending. Here, when a quadrilateral element composed of four nodes is considered, a node adjacent on the side is referred to as 4 neighborhoods, and a node adjacent on the diagonal is referred to as 8 neighborhoods.
[0017]
This model is used to predict what kind of state can exist when this trainer is held at one point by simulating as follows. Here, a left-handed three-dimensional coordinate system in which the Y axis coincides with the direction of gravity is defined as a reference coordinate system. First, the target model is spread on a floor surface horizontal to the XZ plane as shown in FIG. 9a. Gravity works on all nodes. One node assumed to be a holding point is lifted in the vertical direction (Y-axis negative direction), and its deformation is simulated (FIG. 9b).
[0018]
FIG. 9c shows the resulting state held in the air. However, the holding at one point changes greatly depending on the direction on the floor surface where the rotating element around the Y axis is first placed with respect to the holding point. In the actual holding assumed by the present inventor, this is not a single point, but is held by a grip that holds the object with two flat parallel planes having a small area. It is considered that the orientation of the grip surface is determined. Therefore, assuming that the image is always taken from the direction perpendicular to the grip surface, the predicted appearance shape is calculated under the condition that the normal direction of the surface near the holding point of the model predicted shape matches the line of sight. To do. FIG. 9d shows this result.
[0019]
FIG. 10 shows the results of calculating all predicted appearance shapes from State 1 to State 20 held at each node point. In this calculation, the spring constant of the garment model is determined manually so that the resulting garment tension is close to the actual state, and specifically the spring coefficients of K_1, K_2, and K_3 are set to 20000, 2000, and 200, respectively. Set to.
In the classification described in paragraph [0010], the state correspondence when the prediction method B is used is as follows.
Situation A: State 1,3
Situation B: State 2,7,9,13,15,17,19
Situation C: State 4, 5, 6, 8, 10, 11, 12, 14, 16, 18, 20,
[0020]
If prediction method B can be used to predict the observation clothing region in a more realistic state, the overlap between the prediction region and the observation region can be directly used to determine the accuracy of prediction, and the state estimation process is also described below. The procedure is as follows.
The following two features are extracted from the predicted appearance shape of each state in advance.
I) Position of the lowest point of the predicted appearance shape: L_m (L_x, L_y)
Two-dimensional coordinate value relative to the position of the holding point.
II) Number of sleeves that are not directly held but are hanging: N_m
Number of sleeves hanging near the lowest point. In State 2, 5, 8, and 11, N = 2.
[0021]
When the observation image is input, the position of the grip point on the observation image is detected based on the grasped three-dimensional position of the grip of the holding manipulator. The clothing region is extracted as a continuous region with the region feature directly below it as a hint using this holding point as a hint. Let L_o be the lowest coordinate in this area. The average value in the horizontal direction of the image in the region near the upper side of L_o is calculated. If this value is equal to or greater than one sleeve width, N_o = 2, and other values are set to N_o = 1.
[0022]
First, select only models that have the same observation state as the number of hanging sleeves and the closest positional relationship of the lowest point. Specifically, a state satisfying the following condition is selected from State m_i (m_i = 1-20).
N_ {m_i} = N_o
| Ly_o-Ly_ {m_i} | <C_1
| L_o-L_ {m_i} | <C_2
C_1 and C_2 vary depending on how dense the nodes are set, that is, according to the distance between the nodes. In this experiment, C_1 and C_2 were set to 60% and 100% of the longest distance between nodes, respectively.
[0023]
The appearance model that satisfies the condition superimposes these shape models in the image based on the position of the holding point.
Actually, the shape of the body part greatly fluctuates due to a fold that is difficult to predict, whereas the hanging sleeve part has a small variation because of its narrow width. Considering this, partial modification of the appearance model is performed as follows so that this portion more closely matches the actual observation position. However, the hanging sleeve portion refers to a portion of the sleeve that is not directly held. Further, the outer contour refers to a shoulder-side contour (not N1-N13-N15 as an example) (not an axilla side).
[0024]
1) As shown by the thick gray line in FIG. 11a, the outline of the hanging sleeve of the viewing model is moved vertically to a position equal to L_o of the observation image.
[0025]
2) Search for the observation edge closest to the horizontal direction (boundary pixel between the background and the clothing area) from the two node points excluding the lowest point on the outer contour. The horizontal signed distances from the original position to the point as indicated by the arrows in FIG. If the inclination of the line does not change greatly within the error range where the amount of movement of the two points is predicted, this correction is incorporated. Specifically, if all the following three conditions are satisfied, all six node points belonging to the sleeve are moved (da + db) / 2 as shown in FIG. 11b.
| da-db | <C3
| da | <C2
| db | <C2
Here, C3 is determined according to how many times the difference in the inclination of the sleeve between the prediction and the observation state is allowed.
[0026]
3) In this state, calculate the area overlap ratio, R. The area overlap ratio, R, is the sum of (overlapping area) / (observation area area) and (overlapping area) / (predicted shape area) (0 to 2.0). If the value of the appearance model that maximizes this value exceeds the threshold value C_4, this is set as the estimated state. If this condition is not satisfied, it is determined that there is no candidate.
[0027]
If the selected state is class A or B, the edge point of the observation clothing area closest to node 1 or 3 corresponding to the shoulder in the final state after partial correction is set as the next holding point. If a class C state is selected, the lowest point of the area is set as the next holding point. If no state is elected, declare it and indicate the lowest point. As described above, this instruction always changes to State 15 or 19 by the next holding.
[0028]
【Example】
First, the results of Examples using the prediction method A are shown in FIGS. As a result of carrying out using 12 images captured by grasping the trainer at positions corresponding to 12 nodes excluding node 5, a correct state was selected as the first candidate in 6 of these images. FIG. 5 shows an example when State 10 is correctly selected.
FIG. 5a is an input image, FIG. 5b is a display of the selected predicted shape (State 10) superimposed on the result of binarizing the original image, and FIG. The deformed predicted shape is displayed superimposed on the original image. The cross in FIG. 5c indicates the position to be gripped next that is automatically calculated.
[0029]
FIG. 6 shows an example of calculated indication points (crosses in the figure) in situations A, B, and C, respectively. From left to right, (a), (b) is State 1, (c), (d) is State 2, (e), (f) is State 9 and (a), (c) , (e) are input images, and (b), (d), (f) show final results. In State 9, as described in 2.2, the lowest point is selected as the next indication point.
[0030]
FIG. 7 shows an example of the verification process. FIG. 7a shows a situation after the pointing point calculated in FIG. 5c is actually gripped. The shapes predicted by this grip are displayed in an overlapping manner, but they are in good agreement except for some deviation.
On the other hand, FIG. 7c shows a case where the indication point of FIG. 7b selected as the second candidate is grasped. The predicted shape at this time is displayed in an overlapped manner, but the area of the sleeve portion hanging below is greatly deviated from the predicted shape, and it can be known that the initial estimation was wrong.
[0031]
Using Prediction Method B increases the estimated accuracy rate. An example of the result of processing using 12 images similar to the above is shown in FIG. In this embodiment, the threshold values C_1 and C_2 are automatically determined from the predetermined height, width, and sleeve size, and C_3 is set so that the deviation between prediction and observation of the sleeve region is allowed up to 15 degrees, C_4 was set to 1.4 to make it correct with an overlap of about 70%.
[0032]
FIG. 12 shows an example of the result. The position of the holding point on the image was manually given, and the largest bright connected region obtained with a fixed threshold value under that point was automatically extracted as the observation clothing region. 12a and 12b show the original image and the extracted clothing region, respectively. FIGS. 12c, d, and e show candidate states in the order of the first, second, and third largest area overlap ratios before correcting the sleeve portion. FIG. 12 f shows the appearance model state after the first state is partially corrected. In this state, R was 1.64, which was the largest among all the appearance models, so it was determined as an estimation result, and the position indicated by a cross in the figure was calculated as the next point to be retained.
[0033]
FIG. 13 shows another example. FIG. 13A is an original image, and FIGS. 13B, 13C, and 13D show candidate states in order of the first, second, and third largest area overlap ratios before correcting the sleeve portion. In this example, the first and second correction processes were partial corrections, which were not successful, and the third correction state shown in FIG.
In total, the correct state was selected in 9 out of 12 cases and the correct indication point was indicated. FIG. 14 shows each successful example of situation A, situation B, and situation C. For one of the remaining three cases, no state was selected, resulting in “state undecided”. The correct state was selected as the first candidate, but no final decision was made due to an unexpected sleeve break. In this case, it is determined as indeterminate, the lowest point is designated as the next holding point, and the situation class B state is aimed.
[0034]
In the remaining 2 out of 12 cases, the wrong state was estimated. An example of this is shown in FIG. State 4 is correct, but State 7 is selected. The main factor is that there is a big fold of the body part that is not considered in the current simulation.
Furthermore, an experiment was performed in the same manner using 16 images that were captured in a state most different from the predicted state while holding the midpoint of the position corresponding to the node. Again, the correct state was selected in 12 cases and the correct indication point was shown. In one example, “state indeterminate” was determined, and in the remaining three cases, an incorrect state was estimated. Thus, although the target is represented by a fairly rough representative point, even when the node position used for predictive appearance calculation and the actual holding point are separated from each other, a phenomenon in which the performance is not greatly deteriorated was not seen.
[0035]
A trial example of the verification process is shown in FIG. FIG. 16a shows a simulation process in the case of holding the indicated point shown in FIG. 12f. FIG. 16 b shows the simulation result superimposed on the input image in a state where the point is actually held. The predicted shapes are quite overlapping. On the other hand, FIG. 16d is obtained by superimposing the observed image and the predicted image when the wrong state is estimated as shown in FIG. The percentage of non-overlapping parts is very large, and it is possible to recognize that the estimation is incorrect. Further, unlike the simple prediction method described above, if the prediction method B is used, it is also possible to use a state in the middle of deformation as shown in the center of FIG. 16a for verification.
[0036]
【The invention's effect】
Conventionally, it has been impossible to easily estimate the state of a flexible object that deforms greatly, such as clothing. This time, by just giving rough dimensions such as the type of target clothing (trainer, etc.), body width, length, and sleeve length, the possible states can be roughly calculated and used to estimate the state. This became possible. This technology greatly contributes to the realization of robots autonomously handling flexible objects such as clothing.
[Brief description of the drawings]
FIG. 1 is a flow chart when the state of a trainer is estimated.
FIG. 2 is a diagram showing an example of a simple clothing model.
FIG. 3 is a diagram showing a planar model composed of nodes that can be connected to a maximum of four without assuming that the front body and the back body are separated from each other.
FIG. 4 is a diagram showing a predicted shape when the trainer is gripped at one point close to the edge.
5 is a diagram showing Example 1. FIG.
FIG. 6 is a diagram showing an instruction example for “gripping both shoulders” after state estimation;
FIG. 7 is a diagram showing a verification processing embodiment.
FIG. 8 is a diagram showing a target garment represented by a more complicated model.
FIG. 9 is a diagram showing a target model.
FIG. 10 is a diagram illustrating a result of calculating a predicted appearance shape held at each node point.
FIG. 11 is a diagram for explaining partial correction of the appearance model.
FIG. 12 is a diagram showing an example of a result.
FIG. 13 is a diagram showing another example of the result.
FIG. 14 is a diagram showing each successful example of situation A, situation B, and situation C.
FIG. 15 is a diagram illustrating an example in which an incorrect state is estimated.
FIG. 16 is a diagram illustrating a trial example of a verification process.

Claims (6)

In the clothing state estimation method for estimating the state of the target clothing from the input image,
Represents clothing rough basic shape, and to create a planar model consisting of a plurality of nodes connectable,
Using this model, the possible shape when held by a node of a point of said plurality of nodes, by changing the gripping node predicted for all of the plurality of nodes,
This predictive shape is calculated by calculating a feature amount indicating the degree of overlap when the observation area of the target clothing obtained from the input image obtained by capturing the target clothing held at one point is overlapped with respect to the gripping point. A clothing state estimation method for estimating the state of the target clothing by calculating whether the feature amount of the item is the largest .   The clothing state estimation method according to claim 1, wherein the shape prediction is performed based on a given clothing type and a rough size.   The clothing state estimation method according to claim 1, wherein the analysis of the predicted shape classifies the predicted shape according to a required task, and calculates a correspondence rule after the state estimation.   The prediction of the shape is performed by performing a simulation of grasping and lifting one of the elastic models of the target using the spring on the floor, calculating the deformation, and then near the holding point of the predicted shape of the model. The clothing state estimation method according to claim 1, wherein the clothing state estimation method is performed by calculating a predicted appearance shape under a condition in which a normal direction of the surface of the face matches a line of sight. The state is estimated by superimposing the predicted appearance shape on the model and partially deforming based on the input image to determine whether the state can be closer to the observation state. The clothing state estimation method according to claim 4 , wherein the clothing state estimation method is performed by setting an object close to an input image in a correct state. In a clothing state estimation program for estimating the state of a target clothing from an input image,
Represents clothing rough basic shape, and a step of creating a planar model consisting of linkable plurality of nodes,
Using this model, the procedure for predicting the possible shape when held by a node of a point of said plurality of nodes, by changing the gripping nodes for all of the plurality of nodes,
A procedure for processing a feature amount indicating a degree of overlap when the observation area of the target clothing obtained from the input image obtained by capturing the target clothing grasped at one point with the predicted shape is superimposed on the gripping point ;
A procedure for estimating the state of the target clothing by calculating which feature of the predicted shape is the largest ,
Clothing state estimation program for causing a computer to execute.

Images