JP2012008657A - Apparatus and method for generating object shape - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically generate a three-dimensional model from distance images to be acquired as input data.SOLUTION: The object shape generation apparatus includes an image input apparatus, a shape extraction unit, a connection information calculation unit and an image processing unit. The image input apparatus acquires a plurality of distance images of a target object. The shape extraction unit extracts shape information of parts constituting the target object, using the distance images. The connection information calculation unit acquires connection information including coordinate information indicative of a relative position of the parts and angle. The image processing unit generates a three-dimensional model of the target object on the basis of the shape information and connection information.

Description

  Embodiments described herein relate generally to a technique for generating three-dimensional object shape information.

  Conventionally, as a method of detecting a three-dimensional object shape by image processing, there is a method of using a distance image and a three-dimensional model. In this method, a distance image and a three-dimensional model created in advance are collated by selecting a region suitable for collation from the distance image according to a standard set based on error evaluation and processing speed prediction. As a method for generating a three-dimensional model, there are a method for extracting edges and planes from point cloud data, and a method for semi-automatically creating a three-dimensional model of an object from a two-dimensional image set.

JP 2005-293350 A Japanese Patent Laid-Open No. 2004-272459

Silvio Savarese, Li Fei-Fei, “3D generic object categorization, localization and pose estimation”, IEEE 11th International Conference on Computer Vision, ICCV 2007, Rio de Janeiro, Brazil, October 14-20, 2007.

  A method using a three-dimensional model is effective for extracting a three-dimensional object shape by image processing and performing object recognition. Therefore, there is a problem of realizing a method of automatically generating a three-dimensional model from a distance image that can be acquired as input data.

  According to the present embodiment, the object shape generation apparatus includes an input unit, a shape extraction unit, a connection information acquisition unit, and a generation unit. The input means acquires a plurality of distance images of the target object. The shape extracting means extracts shape information of each part constituting the target object using each distance image. The connection information acquisition means acquires connection information including coordinate information representing the relative position and angle of each part. The generation unit generates a three-dimensional model of the target object based on the shape information and the connection information.

The block diagram for demonstrating the structure of the object shape generation apparatus regarding embodiment. The figure which shows the specific example of the three-dimensional model regarding embodiment. The figure which shows the specific example of the three-dimensional model regarding embodiment. The figure which shows the specific example of the three-dimensional model regarding embodiment. The flowchart for demonstrating operation | movement of the object shape generation apparatus regarding embodiment. The figure which shows an example of the distance image regarding embodiment. The figure which shows an example of the mesh data regarding embodiment. The flowchart for demonstrating the procedure of the extraction process of the object part shape regarding embodiment. The flowchart for demonstrating the procedure of the calculation method of the conversion parameter regarding embodiment. The figure which shows the specific example of the division corresponding | compatible area | region of the shape extraction process regarding embodiment. The flowchart for demonstrating the procedure of the connection information calculation regarding embodiment. The figure which shows the specific example of the image recognition regarding embodiment.

  Embodiments will be described below with reference to the drawings.

[Configuration of object shape generator]
FIG. 1 is a block diagram for explaining a configuration of an object shape generation apparatus according to the embodiment.

  As shown in FIG. 1, the object shape generation device includes an image input device (distance image acquisition unit) 10, an input interface 11, a display output device 12, a 3D model database (3D model DB) 13, an image And a processing unit 14.

  The image input device 10 includes, for example, a stereo camera or a range finder, and acquires a distance image of the target object 100. A distance image is image data that can detect a specific shape of a target object measured three-dimensionally by a camera or a range finder.

  The input interface 11 transfers the distance image acquired by the image input device 10 to the image processing unit 14. The image input device 10 acquires a plurality of distance images (a plurality of images) corresponding to the relative positions and angles of the components 101 and 102 of the target object 100 as distance images for creating a three-dimensional model. That is, the image input device 10 captures images while changing the relative positions and angles between the components 101 and 102, and inputs a plurality of distance images obtained by the respective images.

  The image processing unit 14 includes an input / output interface 15, a shape extraction unit 16, a connection information calculation unit 17, and an image recognition unit 18. The input / output interface 15 provides the distance image transferred from the input interface 11 to each of the shape extraction unit 16, the connection information calculation unit 17, and the image recognition unit 18.

  The shape extraction unit 16 extracts shape information of the parts 101 and 102 of the target object 100 based on the distance image. Here, the shape extraction unit 16 inputs a plurality of distance images and processes them as point cloud data or mesh data representing the subject surface of the target object 100. The connection information calculation unit 17 calculates connection information based on the plurality of distance images and the shape information extracted by the shape extraction unit 16. The connection information is information defining a coordinate system constraint defined by the relative position and angle between the components 101 and 102, and a state in each distance image.

  The image processing unit 14 generates a three-dimensional model defined by the shape information and connection information of the components 101 and 102 constituting the target object 100. The image processing unit 14 stores the generated three-dimensional model in the three-dimensional model DB 13 via the input / output interface 15. The image processing unit 14 performs control so that the three-dimensional model generated as necessary is displayed on the screen of the display output device 12.

  The image recognition unit 18 performs image recognition processing of the target object 100 using the acquired distance image and the three-dimensional model stored in the three-dimensional model DB 13. The image processing unit 14 performs control so that the recognition result of the image recognition unit 18 is displayed on the screen of the display output device 12.

[Operation of object shape generator]
Hereinafter, the operation of the object shape detection apparatus of the embodiment will be described.

  First, a specific example of a three-dimensional model according to the embodiment will be described with reference to FIGS.

  As illustrated in FIG. 2, a three-dimensional model including a first component 101, a second component 102, and a connection unit 105 is assumed as a target object model. In the three-dimensional model, the first component 101 and the second component 102 are connected by a connection unit 105.

  The relative positions and angles of the first component 101 and the second component 102 are restricted by the connecting portion 105. Accordingly, the three-dimensional model of the target object includes the shape information of the first component 101 and the coordinate system (coordinate information) 103 representing the position / angle thereof, and the shape information of the second component 102 and the coordinates representing the position / angle thereof. It is defined by constraints and states of the relative position and angle of the system (coordinate information) 104 and the two coordinate systems (103, 104).

  Further, as shown in FIG. 3, as a model of the target object, a three-dimensional model having a structure in which the first component 201 rotates (207) around the rotation axis 206 of the connection unit 205 with respect to the second component 202. Suppose. This three-dimensional model of the target object has a restriction that the coordinate system (coordinate information) 203 of the first component 201 rotates around the rotation axis 206 with respect to the coordinate system (coordinate information) 204 of the second component. . In this case, the state with respect to the coordinate system 204 of the second part in the coordinate system 203 of the first part 201 defines a reference position / angle in advance and is represented by a rotation angle therefrom.

  Furthermore, as shown in FIG. 4, a three-dimensional model having a structure in which the first component 301 can move in the linear direction (305) with respect to the second component 302 is assumed as a target object model. In the three-dimensional model of the target object, the coordinate system (coordinate information) 303 of the first part 301 moves in a linear direction (305) with respect to the coordinate system (coordinate information) 304 of the second part 302. And the state is represented by a movement amount from a preset reference position.

  In addition, for example, when the first part moves on a plane with respect to the second part, a structure that rotates around one point, a structure that moves along a curved line or curved surface, or a bolt and a screw, for example A three-dimensional model of a structure that rotates while moving in parallel can be assumed. Such a three-dimensional model is also represented by the shape (shape information) of each component, the coordinate system (coordinate information) of each component, and the constraints and states of the component coordinate systems. The same applies to a three-dimensional model having a structure that moves under a combination of constraints of two or more parts.

  Next, the operation of the object shape generation device of the embodiment will be described with reference to the flowchart of FIG.

  First, the object shape generation device inputs a plurality of distance images obtained by photographing with the image input device (distance image acquisition unit) 10 while changing the position and angle of the component of the target object (step S1). The image processing unit 14 handles a plurality of input distance images as point cloud data or mesh data composed of vertices representing the three-dimensional position coordinates of the subject surface of the target object. Specifically, a distance image 601 as shown in FIG. 6 is acquired by the image input device (distance image acquisition unit) 10. The distance image 601 is image information with a brighter brightness as the distance is shorter. The image processing unit 14 handles the mesh data 602 corresponding to the distance image 601 as shown in FIG.

  In the image processing unit 14, the shape extraction unit 16 selects two images from a plurality of distance images, and extracts a partial shape (shape information) of the object based on the images (steps S2 and S3). The shape extraction unit 16 executes this extraction process for all or some of the distance image pairs, and integrates the extracted partial shapes (steps S4 and S5). The shape extraction unit 16 outputs the final integrated object shape (shape information) (step S6). Note that when there are two distance images, the shape extraction unit 16 only needs to execute the object part shape extraction process (step S3).

  Here, when extracting a plurality of object or part shapes, the shape extraction unit 16 uses the point cloud data after extracting one object or part and then removing vertices constituting the part. Similarly, the shape information of other objects or parts is calculated.

  Next, the procedure of the object part shape extraction process (step 3) of the shape extraction unit 16 will be described with reference to the flowchart of FIG.

  The shape extraction unit 16 inputs a pair of distance images (distance image 1, distance image 2) from among a plurality of distance images, and executes an object shape extraction process (steps S11 and S12). The shape extraction unit 16 handles two distance images (distance image 1 and distance image 2) as input as point cloud data or mesh data.

  The shape extraction unit 16 calculates the conversion parameter of the three-dimensional position coordinate so that the vertex representing the surface of the target object of the distance image 1 that is input overlaps the vertex representing the surface of the target object of the distance image 2 in the three-dimensional space. (Step S13). The conversion parameter is, for example, a 4 × 4 matrix parameter composed of the origin position in the three-dimensional space of the object coordinate system and the direction of each axis. The conversion parameter is, for example, a quaternion representing the angle of the object and a three-dimensional coordinate value representing the origin position, or a set of three Euler angles representing the angle of the object and a three-dimensional coordinate value representing the origin position. . Further, conversion parameters include those obtained by adding an enlargement / reduction component to these parameters.

  The shape extraction unit 16 converts the distance image 1 using the calculated conversion parameter, and extracts an overlapping region between the two distance images compared to the distance image 2 (steps S14 and S15). The shape extraction unit 16 outputs an object shape (shape information) based on the extracted overlapping area (step S16). Here, as a specific example of the overlapping region, for example, among the vertices of one distance image 1, the vertex of the other distance image 2 is determined as a set of vertices existing within a certain distance.

  Next, the procedure of the conversion parameter calculation method will be described with reference to the flowchart of FIG.

  When the pair of distance images (distance image 1, distance image 2) is input, the shape extraction unit 16 executes region division processing for each of the distance images 1 and 2 (steps S21 to S24). Furthermore, the shape extraction unit 16 calculates a feature amount for each of the divided regions (steps S25 and S26).

  Here, the area dividing process is a process in which at least individual objects or parts are separated from other parts. As a method of area division processing, for example, when distance images 1 and 2 are handled as mesh data, for each edge, a cost is calculated in which the local shape takes a high value for a flat or convex shape and takes a low value for a concave shape. Then, as an initial state, there is a method in which all the vertices are set as another divided region, the region pairs are combined so as to minimize the cost, and this is repeated until the designated number of regions is reached. The size of the region may be used as a selection criterion for the region to be combined. In addition, as a method of area division processing, for example, a method of detecting and using a portion having a convex shape or a concave shape by second-order differentiation of a distance image may be used.

  Further, as the feature quantity to be calculated, for example, an EGI (Extended Gaussian Image) obtained by plotting the normal direction on the surface of a polyhedron approximating a sphere and forming a histogram, and the surrounding shape from the normal direction and normal of the target point The spin image plotted according to the distance is used.

  Next, the shape extraction unit 16 calculates the similarity of the feature amount between the divided areas in the two distance images (step S27). The shape extraction unit 16 calculates the similarity based on, for example, the inner product or the distance of the feature amounts. In addition, when using EGI as the feature quantity, the shape extraction unit 16 uses the feature quantity between all the corresponding faces when the polyhedron in which the normal direction is plotted is rotated in order to absorb the difference due to the shooting angle of the object. And the maximum similarity value is output as the similarity calculation result.

  Here, as a method for calculating the similarity between the divided regions, a normal direction histogram created by plotting the normal direction of the vertices in the region to a polygon approximating a sphere is used as the feature amount of the divided region. Furthermore, the similarity between the feature amounts is calculated for all the correspondences of the faces when the polygon is rotated, and the similarity is calculated as the similarity calculation result. Note that the degree of similarity may be weighted according to the feature value of the divided area.

  Next, the shape extraction unit 16 selects a candidate for a combination of corresponding regions having a high total similarity under constraints such as a positional relationship between divided regions, a distance, and a difference in normal direction (step S28). Here, in addition to the similarity of the corresponding region, a process of weighting the restriction or similarity on the consistency of the positional relationship, distance relationship, and angular relationship of each region may be executed. FIG. 10 is a diagram showing a specific example of a combination of corresponding areas, and is an example in which three sets of corresponding areas (903 and 906, 904 and 907, and 905 and 908) are selected for the distance image pairs 901 and 902.

  The shape extraction unit 16 converts the conversion parameter of the distance image 1 such that the distance between the reference positions (909, 912, 910, 913, 911, and 914) of these corresponding regions is the minimum by a method such as a least square method. Then, the distance image 1 is converted (step S29). Finally, as post-processing, the shape extraction unit 16 adjusts the conversion parameters so that the vertices in the corresponding regions (903, 904, 905, 906, 907, 908) overlap by a method such as the ICP (Iterative Closest Point) method. Then, the alignment process based on the shape is executed (step S30). Based on this alignment, the shape extraction unit 16 outputs a conversion parameter that minimizes the distance between the reference positions of the corresponding regions (step S31). Here, the reference position of the corresponding area is the center position of the circumscribed rectangle of the area or the barycentric position of the vertex in the area.

  FIG. 11 is a flowchart for explaining the procedure of the calculation process of the connection information calculation unit 17. As shown in FIG. 11, the connection information calculation unit 17 inputs a conversion parameter group and a component shape group (shape information) calculated based on the distance image pairs in the plurality of distance images from the shape extraction unit 16 ( Steps S41 and S42).

  The connection information calculation unit 17 sets a coordinate system (coordinate information) for each input component shape (step S43). Further, the connection information calculation unit 17 calculates the relative coordinate system parameters between the component shapes in the distance image based on the input conversion parameters and the coordinate system set for each component shape (step S44). Based on a plurality of relative coordinate system parameters of the same component pair, the connection information calculation unit 17 calculates the constraint on the relative position / angle of the component pair and the state in each distance image, and outputs this as connection information (step). S45, S46).

  Here, when the relative position changes on a straight line as a parameter representing the restriction and state between the components, the restriction is a movement amount defined by a certain reference position on the straight line and the movement direction. When the relative position between components changes on the plane, the constraint is a coordinate value representing one point on the plane defined by a certain reference position on the plane and the normal direction of the plane. When the relative position and angle between parts change about one axis, the constraint is the rotation amount defined by the rotation center position, the rotation center axis direction, and the reference angle. When the relative position and angle between parts change around one point, the constraint is the rotation angle from the reference position defined by the rotation center position, the reference position, and the rotation direction. In addition, when the relative position and angle between parts change along a curve or curved surface, the constraint is a control point that defines the curve or curved surface and a parameter that represents one point on the curve or curved surface defined by the reference position. is there.

  As described above, according to the present embodiment, an object composed of a plurality of parts (partial shapes) is defined by shape information and connection information using a plurality of distance images obtained by photographing a target object. The generated three-dimensional model can be automatically generated. That is, for an object composed of a plurality of parts whose positions and angles are not fixed, a three-dimensional model, which is a shape / structure model including information such as the shape of each part, and movable parts and movable ranges between parts, is automatically generated. Can be stored in the three-dimensional model DB 13. It should be noted that a three-dimensional model in which the operating range between the parts in the connection information is constrained may be generated by performing a collision determination between the parts based on the shape information and the connection information.

  Here, the image recognition unit 18 uses the three-dimensional model generated by the above-described method and stored in the three-dimensional model DB 13 and the distance image acquired from the image input device 10 to perform image recognition processing of the target object. Can be performed. Specifically, as shown in FIGS. 12A and 12B, image recognition of a target object such as a mobile phone can be performed. Here, the image recognition unit 18 can identify a target object such as a cellular phone based on the acquired distance image. Further, the image recognition unit 18 obtains an object coordinate system 1200, link information (relative distance, angle) 1201, and shape information 1202 based on a three-dimensional model (shape / structure model) stored in the three-dimensional model DB 13. By doing so, it becomes possible to automatically recognize the constraints and states between the components of the target object (such as the display and keyboard of the mobile phone).

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

10 ... image input device (distance image acquisition unit), 11 ... input interface,
12 ... Display output device, 13 ... 3D model database (3D model DB),
14 ... image processing unit, 15 ... input / output interface, 16 ... shape extraction unit,
17 ... Connection information calculation unit, 18 ... Image recognition unit, 100 ... Target object, 101, 102 ... Parts.

Claims (14)

Input means for acquiring a plurality of distance images of the target object;
Shape extraction means for extracting shape information of each part constituting the target object using the distance images;
Connection information acquisition means for acquiring connection information including coordinate information representing the relative position and angle of each part;
An object shape generation apparatus comprising: generation means for generating a three-dimensional model of the target object based on the shape information and the connection information. The shape extraction means includes
The shape information is calculated by treating each distance image as point cloud data composed of vertices representing the three-dimensional position coordinates of the surface of the target object or mesh data composed of the vertices and their adjacent information. The object shape generation apparatus according to claim 1, wherein the object shape generation apparatus is configured as described above. The shape extraction means includes
Parameter calculating means for calculating a conversion parameter of three-dimensional position coordinates in which a vertex representing the target object surface of both of the point cloud data overlaps in a pair of distance images among a plurality of distance images;
An overlapping region of the point cloud data coordinate-converted by the conversion parameter is extracted as a surface shape of the target object or a part thereof, and the shape information is calculated by combining the extraction results. The object shape generation device according to claim 2. The connection information acquisition means
A coordinate system representing the position and angle for each part is defined, and the connection information is defined to define the relative position and angle for each part with parameters representing the constraints and states between the parts. The object shape generation device according to any one of claims 1 to 3, wherein the object shape generation device is provided. Database means for storing the three-dimensional model generated by the generating means;
The object shape generation apparatus according to claim 1, further comprising: an image recognition unit that performs image recognition processing of a target object using the distance image and the three-dimensional model. The parameter calculation means includes
4x4 matrix parameters composed of the origin position and the direction of each axis in the three-dimensional space of the object coordinate system, a quaternion representing the angle of the object, a three-dimensional coordinate value representing the origin position, and the angle of the object 4. The conversion parameter is calculated as any one of a set of three Euler angles and a three-dimensional coordinate value representing an origin position, or a parameter obtained by adding an enlargement / reduction component to these parameters. Object shape generator. The parameter calculation means includes
Performing a region dividing process for each distance image;
Calculate the feature amount of each area divided by the area dividing process,
Calculating the similarity of the feature amount between the divided regions;
Under the restriction due to the positional relationship between the previous divided areas, a candidate for a combination of corresponding areas with high similarity is selected,
The object shape generation apparatus according to claim 3, wherein the conversion parameter is configured to calculate the conversion parameter that minimizes the distance between the reference positions of the corresponding areas. The parameter calculation means includes
As a means for calculating the similarity, when the normal direction histogram created by plotting the normal direction of vertices in a region to a polygon approximating a sphere is used as a feature value of the divided region, and the polygon is rotated 8. The object shape generation according to claim 7, wherein among the correspondences of all the planes, the similarity between the feature quantities having the maximum similarity is output as a similarity calculation result. apparatus. The parameter calculation means includes
As a means for selecting candidates for the combination of the corresponding regions,
The configuration according to claim 7, wherein, in addition to the similarity of the corresponding region, a restriction or weighting is performed on the consistency of the positional relationship, the distance relationship, and the angular relationship of each region. Object shape generation device. The parameter calculation means includes
8. The object shape generation apparatus according to claim 7, further comprising means for executing post-processing for adjusting the conversion parameter so that the vertices in the corresponding region overlap each other. The connection information acquisition means
As a parameter representing the constraints and states between the parts,
When the relative position between the parts changes on a straight line, the constraint is a movement amount defined by a certain reference position on the straight line and a movement direction,
When the relative position between the parts changes on the plane, the constraint is a coordinate value representing one point on the plane defined by a certain reference position on the plane and the normal direction of the plane;
When the relative position and angle between the portions change around one axis, the constraint is the rotation amount defined by the rotation center position, the rotation center axis direction, and the reference angle,
When the relative position and angle between the parts change around one point, the constraint is the rotation angle from the reference position defined by the rotation center position, the reference position and the rotation direction,
When the relative position and angle between the parts change along a curve or curved surface, the constraint is a parameter that represents a control point that defines the curve or curved surface and one point on the curve or curved surface that is defined by the reference position. The object shape generation device according to claim 4, wherein the object shape generation device is provided. The shape extraction means includes
In the case of extracting a plurality of object or part shapes, after extracting one object or part and using the point cloud data after removing the vertices constituting the part, another object or part is similarly used. The object shape generation apparatus according to claim 2, wherein the shape information is calculated. The generating means includes
Based on the shape information and the connection information, a collision determination between the parts is performed to generate a three-dimensional model in which the operating range between the parts in the connection information is restricted. The object shape generation apparatus according to claim 1. A process of acquiring a plurality of distance images of the target object;
A process of extracting shape information of each part constituting the target object using each distance image;
Processing for obtaining connection information including coordinate information representing the relative position and angle of each part;
And a process of generating a three-dimensional model of the target object based on the shape information and the connection information.

Images