KR101093793B1 - Method for measuring 3d pose information using virtual plane information - Google Patents

Abstract

The present invention provides a 3D attitude information acquisition method using virtual plane information. The 3D posture information obtaining method using the virtual plane information extracts and registers feature information of a target object based on preset information, and extracts feature information of a 2D image of the target object. The feature points matched with the feature information are used to configure virtual plane information, and based on the virtual plane information, three-dimensional posture information of a target object with respect to a reference coordinate system is obtained. Accordingly, the present invention extracts feature information from an image obtained from a single camera, and then constructs a virtual plane using arbitrary feature points obtained to obtain camera pose information for each virtual plane to obtain 3D pose information of the object to be worked on. Can be measured quickly and accurately.

Description

{METHOD FOR MEASURING 3D POSE INFORMATION USING VIRTUAL PLANE INFORMATION}

The present invention relates to a method for acquiring three-dimensional attitude information using virtual plane information. More particularly, after extracting feature information from an image obtained from a single camera, a virtual plane using arbitrary feature points obtained is configured to configure each virtual. The present invention relates to a three-dimensional attitude information acquisition method using virtual plane information capable of quickly and accurately measuring three-dimensional attitude information of a target object by acquiring camera attitude information about a plane.

Recently, interest and development of intelligent service robots for service support through mutual coexistence with human beings in homes or public places have been actively promoted. In addition, research and application demands for carrying out tasks such as handling or assembling unaligned workpieces under robots in a vehicle manufacturing process are expanding.

Object recognition in automated systems is a technology that increases the intelligence of the system. However, when the object is recognized, the 3D posture information of the object is required for a process such as assembly using the object. This demand is the opportunity for the development of 3D attitude information determination technology, and the application of 3D robot vision in the automobile manufacturing process is expanding along with the development of 3D vision technology.

As an example, there is a robot vision system that precisely performs sealing work of a vehicle by measuring a three-dimensional position and posture of the vehicle body using a 3D vision system in an automobile painting factory.

A typical three-dimensional attitude determination technique is stereo vision technology based on the human visual system. As much as the long research period of stereo vision technology, the research results are shown. However, systems that use multiple cameras, such as stereo vision technology, must suffer a lot in terms of system footprint and cost efficiency.

In particular, it imposes strict constraints on the geometric relationships between the cameras, which directly affect the performance of the system. In addition, the problem of solving the correspondence between the feature points acquired from each camera is included as an important process.

This also has been studied by many researchers because it has a big impact on the performance of the system. Due to the limited performance of these systems, industrial sites are using three-dimensional attitude determination techniques with a single camera.

Among the techniques using a single camera, the one that shows good performance is the active vision technique. The sensor module used in this technique is called a laser vision sensor because it uses a camera and a laser sensor.

The problem with the general single-camera technique arises when the shape of a three-dimensional object is concave. This problem is emphasized when there is a lack of characteristic information of the target object, which emphasizes the importance of the active vision system.

However, the conventional technique as described above has a problem of using a fixed FOV and an expensive laser vision sensor.

In particular, the laser position sensor is widely used in the method of measuring the position of the car body. If the car body length varies according to the car type by fixedly installing the sensor on the floor, the installation and maintenance cost increase due to the installation of a plurality of laser sensors. Has

Therefore, the present invention was created to solve the above-mentioned problems, and an object of the present invention is to solve the three-dimensional relationship more easily when the feature information on the object to be worked on exists in a single plane, which can be obtained from an image. Compose a virtual plane based on the information and determine the exact three-dimensional pose information of the work object by using the camera pose information of each virtual plane, and it is a complex object that cannot express the characteristic information of the object in a single plane. The virtual plane information is constructed by using any three feature points acquired by a single camera, so that not only the three-dimensional position and posture of the workpiece can be easily measured, but also an example of a large workpiece such as a vehicle body. Even when multiple single cameras are placed in the main local position of the object being worked on. After positioning, a plurality of virtual planes are constructed based on the main feature information obtained from the image information acquired from each single camera, and the virtual plane information is used to quickly and accurately measure the three-dimensional position and posture of a large target object. It is to provide a three-dimensional attitude information acquisition method.

In a preferred aspect, the present invention provides a method for obtaining three-dimensional attitude information using virtual plane information.

The 3D posture information acquisition method using the virtual plane information of the present invention extracts and registers feature information of a target object based on preset information, and extracts feature information of a 2D image of the target object. It consists of virtual plane information using feature points that match the registered feature information, and obtains 3D posture information of a target object with respect to a reference coordinate system based on the virtual plane information.

Here, it is preferable to arrange a single image capturing device at a set characteristic position of the work object, and to acquire three-dimensional attitude information of the work object with respect to the reference coordinate system at the specific position.

In addition, the plurality of image capturing apparatuses may be disposed at a set local feature position of the work object, calculate posture information on a local area at the local specific position, and collect the calculated posture information to collect the reference coordinate system. Three-dimensional attitude information of the object to be worked on can be obtained.

In addition, a plurality of image capturing devices are installed in a movable mobile device to be moved and arranged at a set local feature position of the object to be worked on, to calculate posture information for a local area at the local specific position, and to calculate the calculated posture. The information may be collected to obtain three-dimensional attitude information of the object to be worked with respect to the reference coordinate system.

Since the present invention is based on a single plane-based attitude determination technique, it is possible to derive an intuitive geometric relationship.

In addition, the present invention is that each virtual plane is composed of any three points of the extracted feature points, the virtual plane coordinate system that can be configured by these points represent the relationship with the camera coordinate system in a simple plane-based relationship Has the effect to be.

In addition, the geometric relations expressed as described above have a simpler shape than the general three-dimensional attitude determination technique, and thus more efficient results can be derived. By using arbitrary three-point information acquired by a single camera, By constructing virtual plane information, not only can you easily measure the three-dimensional position and posture of the work object, but also, for example, in the case of a large work object such as a car body, a plurality of single cameras are installed at the main local positions of the work object. Afterwards, by constructing a plurality of virtual planes based on the main feature information obtainable from the image information obtained from each single camera, it has the effect of quickly and accurately measuring the three-dimensional position and posture of the large work object.

1 is a block diagram illustrating an embodiment of an operation process of a three-dimensional magnetic world-side method using a virtual plane technique according to the present invention.
2 is a block diagram showing a model registration process according to the present invention.
3 is a block diagram showing an image-based attitude determination step by a single camera according to the present invention.
4 is a block diagram showing a local 3D information collection step according to the present invention.
5 shows a feature vector for a reference coordinate system.
6 is a diagram illustrating a relationship between a camera and a work object coordinate system with respect to a real world reference coordinate system.
7 is a view illustrating a relationship with a virtual plane coordinate system with respect to a work object coordinate system.
8 is a diagram illustrating a relationship with a virtual plane coordinate system for a large object coordinate system.

Hereinafter, the 3D attitude information acquisition method using the virtual plane information of the present invention will be described with reference to the accompanying drawings.

Referring to FIG. 1, the method of acquiring a three-dimensional posture of the present invention includes a model registration step of extracting and registering feature information of a work object based on preset information, and feature information in a two-dimensional image of the work object. A virtual plane information constructing step of configuring virtual plane information using feature points corresponding to the registered feature information, and obtaining 3D posture information of a target object with respect to a reference coordinate system based on the virtual plane information. It includes a three-dimensional attitude information acquisition step.

[Model Registration Step]

1 and 2, the feature components of the target object are extracted and registered based on previously known or preset CAD information of the target object.

Referring to FIG. 2, first, a single image, which is a two-dimensional image of a target object, is acquired through a single camera that is an image acquisition device. Preferably, the acquired image includes feature components including the preset CAD information.

Then, feature components existing in the acquired image are extracted through a feature extractor. Here, the feature extractor used may be a three-dimensional feature extraction module such as SIFT or SURF.

Here, the feature extraction modules as described above may have a problem in that it is difficult to determine an accurate correspondence relationship when a large displacement of an object occurs. Therefore, in the present invention, the Harris corner detection technique or the morphological shape of a general image may be used. By applying techniques such as holes, intersection detectors, etc., the exact correspondence can be determined.

Subsequently, CAD information corresponding to the extracted feature components is registered. Since the performance of the three-dimensional attitude determination module is greatly affected by the performance of the feature extraction module, the components having the highest detection rate among the detected feature information are first selected and applied.

In addition, since the number of feature information must satisfy the unknown number of the proposed three-dimensional attitude determination technique, it is composed of six or more feature components that can constitute at least two virtual planes.

In the model registration step, feature points that can reflect system characteristics are selected and accommodated.

Finally, in order to register such a feature model, it is necessary to register the reference coordinate system of the CAD information appearing in the registration model, that is, the information about the coordinate system of the local area. The local coordinate system refers to a target object coordinate system for each single camera model of the reference object system of a large object, and a homogeneous transformation matrix may be constructed using the geometric relationship therebetween.

[Steps for Constructing Virtual Plane Information]

As shown in FIG. 3, the first two steps may be the same as the model registration step shown in FIG. 2.

In addition, features acquired from the acquired image are compared with feature information used in the model registration step to determine respective correspondences.

Here, the virtual plane information is configured by using feature information satisfying a coarse degree among the extracted feature information. In the feature information-based virtual plane determination method, when feature points of a target object exist in a single plane, an intuitive and simple linear equation may be constructed to obtain 3D geometric information.

In addition, since the feature points of the target object existing in the three-dimensional space cannot always exist in a single plane, the present invention constructs a virtual plane by using any three points among the extracted feature points, and each virtual plane It is good practice to construct a linear system that can determine the attitude information of the camera for.

The geometric relationship between the virtual plane and the reference coordinate system of the target object is represented by a known constant term, and the relationship between the object reference coordinate system and the camera reference coordinate system is unknown because it is unknown in the present invention. Can be configured as a determinant. At this time, the feature points constituting the virtual plane are selected as those registered with the corresponding CAD information among the detected feature points.

5 shows a relationship between the virtual plane reference coordinate system and the target object reference coordinate system.

Any three points among the determined feature points may be configured to form a virtual plane, and the virtual plane reference coordinate system for the configured virtual plane may be represented by a geometric relationship such as a target object coordinate system and Equation (1).

[Equation 1]

If any one of the selected feature points is first determined as a reference point, the elements of the displacement vector with respect to the reference point may be replaced with the element values of the translational relationship in equation (x). Using a plane equation for the other two points, including the reference point, a normal vector perpendicular to the plane and matching it with the z-axis of the virtual plane coordinate system places all the feature points on the x-y plane in the virtual plane coordinate system. Therefore, the element values of the rotational relationship in Equation (1) can be replaced by the rotational displacement relationship matching the normal vector mentioned above. As such, by forming a rotation transformation matrix in a general homogeneous transformation matrix, a final homogeneous transformation relationship can be formed.

로 표현할 수 있다. 이들 관계는 수식 2로 표현되어진다.In other words, a plane formed by F1 and F2 with respect to F0 is called an xy plane, and a homogeneous transformation matrix with respect to the reference coordinate system {O} when the coordinate system for this plane is {P}.

. These relationships are represented by equation (2).

는 병진관계(translation)와 회전관계(rotation)를 나타내는 인자들로 구성된다. 임의의 특징점 F0를 원점으로 하는 좌표계 {P}에 대한 병진관계는 원점벡터의 값이 병진관계의 요소들이기 때문에 별도의 계산 과정이 필요 없다. 하지만, 회전관계를 표현하기 위해서는 나머지 특징 벡터들과 원점으로 결정된 특징 벡터들과의 관계를 이용하여 결정할 수 있다.Homogeneous transformation matrix

Is composed of factors representing translation and rotation. The translational relationship of the coordinate system {P} with the origin of the arbitrary feature point F0 does not need a separate calculation process because the values of the origin vector are elements of the translation relationship. However, in order to express the rotation relationship, it may be determined using the relationship between the remaining feature vectors and the feature vectors determined as the origin.

In this case, the positions of the feature points with respect to the virtual plane coordinate system may be obtained by constructing Equation 2.

[Equation 2]

When the feature points in the virtual plane coordinate system are to be expressed with respect to the object reference coordinate system, the inverse matrix of the homogeneous transformation matrix in Equation 1 may be expressed as Equation 2.

In addition, when the feature points for the object are limited on a single plane, the value corresponding to the z-axis is calculated as 0, which reduces the number of external factors to be determined.

In the present invention, it is possible to apply a simple equation derivation process configured for a single plane by applying the equation to the virtual plane as it is, the angle generated because it is not possible to limit the feature points extracted from the object in space to a single plane There is no need to derive complex equations for feature components.

When defining the relationship between the target object reference coordinate system and the virtual plane coordinate system as described above, the relationship between each virtual plane coordinate system and the camera reference coordinate system which is an image capturing apparatus may be defined through the relationship with the target object reference coordinate system. .

6 illustrates a relationship between a target object reference coordinate system and a camera reference coordinate system with respect to the real world reference coordinate system.

Referring to FIG. 6, the attitude information of the camera coordinate system with respect to the real world reference coordinate system may be represented by a known geometric relationship, and when the geometric relationship of the target object coordinate system with respect to the camera coordinate system is obtained, the geometry of the target object coordinate system with respect to the world reference coordinate system You can determine the relationship. The relationship of Equation 3 can be derived from FIG. 6.

[Equation 3]

Since it is assumed that the feature points of the target object exist in a single plane, the following simultaneous equations for the x and y axes can be obtained by arranging Z _i as 0 as shown in Equation 4.

[Equation 4]

A system of equations such as Equation 4 exists for a plurality of feature points existing on a single plane, and when the least square method is applied, the relation of the object coordinate system with respect to the world reference coordinate system may be used.

Relationship to the camera coordinate system {C} of the world reference coordinate system {W}, such as the ^W H _C are known relationship is due to the camera coordinate system {C} to the target was considered for a single plane object coordinate system {O} geometric relationship ^O H _C must be determined.

At this time, the relationship of the object coordinate system to the camera coordinate system is determined through the relationship to each virtual plane coordinate system {P _n }.

Through this, Equation 3 may be reconfigured and expressed as Equation 5.

[Equation 5]

The geometric relationship shown in Equation 5 may be illustrated as a relationship between the virtual plane coordinate system and the target object coordinate system as shown in FIG. 7.

FIG. 7 illustrates the application of a virtual plane technique based on FIG. 6 showing a geometric relationship to a single plane.

First, except for the known geometric relationship between the camera and the world reference coordinate system, the relationship between the object reference coordinate system and the camera reference coordinate system is expressed through the virtual plane coordinate system.

As shown in Equation 5, the relationship of the camera coordinate system to the object reference coordinate system is expressed through the relationship of the virtual plane coordinate system.

This may be expressed as in Equation 4, as shown in Equation 6 below.

[Equation 6]

When Equation 6 is expressed as a determinant, the unknowns may be classified and expressed as Equation 7.

[Formula 7]

Finally, in Equation 7 obtained, the unknown vector H is a vector constituted by element values of the homogeneous transformation matrix of the object reference coordinate system with respect to the camera reference coordinate system.

Where j is the index for the virtual plane and i is the index for the feature point in the plane.

Through this equation, we construct a variable-size matrix configured to calculate i feature information for each of j planes.

The inverse transformation matrix for the variable size matrix can be obtained by applying a well-known pseudo inverse matrix, and finally, the information of the H matrix containing attitude information can be calculated through Equation (8).

At this time, in order to extract each factor from each H, T _z can be calculated as shown in Equation 8 by using the orthogonality of the rotation transformation matrix.

[Equation 8]

The remaining elements of the rotation transformation matrix can also be obtained using the orthogonality of the rotation transformation matrix.

Subsequently, single camera-based three-dimensional pose information is determined using the finally configured virtual plane information.

[3D posture information acquisition step]

Next, in the case of a large object such as a vehicle body, the local three-dimensional pose information collecting step in FIG. 1 is a single camera-based three-dimensional pose information determined through the same steps as in FIGS. 2 and 3 at the local position of the large object. Since it is the partial three-dimensional information of the acquired object, it undergoes a three-dimensional pose collection process to collect it.

Therefore, the local three-dimensional pose information collecting step is performed by the local three-dimensional pose information of each camera registered in step 2 based on the single camera-based three-dimensional pose information in FIG. 3 as shown in FIG. The geometric relationship is determined as shown in Equation 9. Since the determined local three-dimensional attitude information each includes local observational errors, the three-dimensional attitude information of the final large object can be determined by expressing them as a closed loop equation.

In order to measure accurate three-dimensional position and attitude of a large object such as a vehicle body by using a virtual plane-based three-dimensional positioning method, the problem of collecting partial attitude information obtained from a single camera must be solved. When the attitude of a large object is determined by using a single camera, the distance between the camera and the large object becomes far and the error range for accuracy is widened in order to include the entire area of the large object in the FOV of the single camera. Therefore, as shown in FIG. 8, the partial attitude information of the large object can be determined, and the attitude of the large object can be determined using the known geometric relationship therebetween. First, determine the attitude information ^{C0 ~ 2} H _{L0 ~ 2} about each part of the object with respect to the camera while knowing ^W H _{C0 ~ 2} which is the camera's attitude information about the world reference coordinate system. through the ^{L0 ~} H ² _O, known position information of the reference coordinate system of the object can determine the ^W H _O attitude information of the object reference coordinate system on the world reference frame. The attitude information thus determined may be arranged as shown in Equation (9) based on the attitude of the robot.

[Equation 9]

The first term of Equation 9 shows the relationship of determining the pose of the object through the camera, and the result is expressed in the second term. The last term expresses the posture information of the object based on the posture of the robot. At this time, it is easy to determine the attitude information of the object about the robot by multiplying the second term by the inverse of ^W H _M , which is the attitude of the robot with respect to the world reference coordinate system, without knowing ^M H _O, which is the attitude information of the object. Can be. In expressing these relationships, the only final solution can be determined by applying the least-squares method, which is commonly used.

Hereinafter, the present invention is not limited to the above specific preferred embodiments, and various modifications can be made by those skilled in the art without departing from the gist of the present invention as claimed in the claims. Of course, such changes are within the scope of the claims.

Claims (4)

In order to construct a homogeneous transformation matrix as a reference coordinate system of the CAD information for the target object, the coordinate system information of the local region, which is the reference coordinate system of the target object, for each single camera model of the reference coordinate system of the large target object is obtained. A model registration step of extracting and registering feature information of the object to be worked on, based on the coordinate system information of the local area after registration;
Extracting feature information from the 2D image of the object to be worked on, and comparing the extracted feature information with the registered feature information extracted in the model registration step to detect matching feature points, wherein the detected feature points are single In the case of the plane, intuitive and simple linear equations are constructed to obtain the 3D geometric information to construct the virtual plane information, or when the detected feature points do not exist in the single plane, determine the attitude information of the camera with respect to the virtual plane. After constructing the virtual plane information by using any three points among the detected feature points to form a linear system, the feature of each of the feature points for the virtual plane information through the homogeneous transformation relationship Virtual plane information sphere to determine position and determine single camera based 3D pose information Step; And,
3D attitude information obtaining step of obtaining 3D attitude information of the object to be worked on a reference coordinate system based on the virtual plane information; 3D attitude information acquisition method using the virtual plane information, characterized in that the progress. The method of claim 1,
Considering that the feature points of the target object exist in a single plane, a single image acquisition device is disposed at a specific position with respect to the single plane, and three-dimensional posture information of the target object with respect to the reference coordinate system at the specific position. 3D attitude information acquisition method using virtual plane information, characterized in that to obtain a. The method of claim 1,
In consideration of the fact that the feature points of the object to be worked on do not exist in a single plane, a plurality of image capturing apparatuses are respectively disposed at a local specific position based on any three point feature points among a plurality of feature points,
Calculating posture information for a local area at the local specific location;
3. The method of claim 3, wherein the calculated attitude information is collected to obtain 3D attitude information of the object to be worked on the reference coordinate system. The method of claim 1,
In a state where a plurality of image capturing devices are installed in a movable mobile device, the feature points of the object to be worked on are not present in a single plane, so that the local points are located at a local specific position based on any three point feature points among the plurality of feature points. Move and arrange a plurality of image acquisition devices installed in the mobile device,
Calculating attitude information of a local area at the local specific location,
3. The method of claim 3, wherein the calculated attitude information is collected to obtain 3D attitude information of the object to be worked on the reference coordinate system.

Images