KR20210020279A - Position detecting system for an object using attached markers and method for position detecting using the same - Google Patents

Abstract

Disclosed are an object-position calculating system using an attachable marker and an object-position calculating method using the same, wherein the object-position calculating system comprises a plurality of markers, a photographing part, a position calculation part, a storage part, a matching part, and a detection part. The plurality of markers are attached to the surface of an object. The photographing part acquires an image by photographing the object and the markers. The calculation part calculates the three-dimensional positions of markers from the photographed image. The storage part stores the calculated three-dimensional positions as a key point group. The matching part matches the stored key point group with a three-dimensional model of the object. The detection part calculates a coordinate system of the key point group based on the matches. The present invention can minimize interference by a marker.

Description

A system for calculating the position of an object using an adhesive marker and a method for calculating the position of the object using the same {POSITION DETECTING SYSTEM FOR AN OBJECT USING ATTACHED MARKERS AND METHOD FOR POSITION DETECTING USING THE SAME}

The present invention relates to a system for calculating the position of a target object and a method for calculating the position of the target object using the same, and more particularly, the target object is positioned in a three-dimensional space using an adhesive marker attached to the target object of a three-dimensional shape. The present invention relates to a system for calculating a position of a target object using an adhesive marker capable of accurately calculating the position of the target object and a method for calculating the position of the target object using the same.

Recently, in the medical field, there is an increasing demand for monitoring whether or not a surgical tool used is moving in accordance with a pre-planned, and for this purpose, a technology for tracking a location of a surgical tool or the like is being developed.

In general, in order to track the location of a moving object, a technology for attaching a three-dimensional marker that can be traced to the object and tracking it in real time is applied. In the case of such a three-dimensional marker attachment and tracking technology, as in Korean Patent Registration No. 10-1246515, a three-dimensional marker having a predetermined size and shape is attached to an object to be tracked, and the attached marker cannot be recognized or identified. For ease of use, if the object is a patient, it is generally attached to the patient's head.

However, since the marker is relatively bulky and has a complex shape, when the marker is attached to a patient's head or the like, there is a problem in that surgery on a patient is inconvenient due to the marker in an actual complex surgical environment.

In addition, in an environment in which various surgical tools or surgical equipment are installed, the marker may often be covered by a doctor, a patient, or surgical equipment, and thus, there is a problem that tracking of a target object is delayed or an error occurs.

Korean Patent Registration No. 10-1246515

Accordingly, the technical problem of the present invention is conceived in this respect, and an object of the present invention is to minimize interference by markers in handling or moving a target object, and to track a location even when some markers are covered. It is to provide a system for calculating the location of a target object using an attachable marker that improves the accuracy of tracking the location of and enables location tracking even for a target object having an arbitrary three-dimensional shape.

In addition, another object of the present invention is to provide a method for calculating the position of an object using the system for calculating the position of the object.

A system for calculating a position of an object according to an embodiment for realizing the object of the present invention includes a plurality of markers, a photographing unit, a position calculating unit, a storage unit, a matching unit, and a detection unit. The plurality of markers are attached to the surface of the object. The photographing unit acquires an image by photographing the object and the markers. The calculation unit calculates the three-dimensional positions of the markers from the photographed image. The storage unit stores the calculated 3D position as a key point group. The matching unit matches the stored keypoint group with the 3D model of the target object. The detection unit calculates a coordinate system of the key point group based on the matching.

In one embodiment, a plurality of the markers are attached to the surface of the object as an infrared reflective sticker at predetermined intervals, and the photographing unit may be an infrared stereo camera.

In one embodiment, the object has an arbitrary three-dimensional shape, and the three-dimensional model of the object may be formed of point group data or a three-dimensional mesh model.

In an embodiment, the markers are divided into a plurality of marker groups having different shapes or colors, and the storage unit may distinguish and store different marker groups into different keypoint groups.

In an embodiment, the position calculator may calculate the central coordinates of each of the markers as a three-dimensional position of the markers.

In an embodiment, the matching unit may rotate or translate the 3D model to match the keypoint group while the keypoint group is fixed.

In the method for calculating the position of an object according to an embodiment for realizing the object of the present invention, a plurality of markers are attached to the surface of the object. An image is obtained by photographing the target object and the markers. The three-dimensional positions of the markers are calculated from the captured image. The calculated three-dimensional position is stored as a key point group. The stored keypoint group is matched with the 3D model of the target object. Based on the matching, the coordinate system of the key point group is calculated.

In one embodiment, the calculating of the three-dimensional position includes obtaining a two-dimensional position of the markers in a photographed image of the object and the markers, and using a stereo relationship from the two-dimensional position of the markers. Thus, it may include calculating a three-dimensional position.

In one embodiment, in the step of calculating the 3D position, only identifiable markers among the markers attached to the object may be selectively calculated for the 3D position.

In one embodiment, in the matching step, while the keypoint group is fixed, the 3D model may be rotated or translated to match the keypoint group.

In one embodiment, in the matching step, when the three-dimensional model of the target object is point group data, it may be matched using an iterative closest point (ICP) algorithm.

In an embodiment, in the matching step, when the 3D model of the target object is a 3D mesh model, registration may be performed using a point-to-plane registration algorithm.

In an embodiment, in the matching step, when the matching error is smaller than a preset reference value, the matching may be terminated.

In an embodiment, after calculating the coordinate system of the key point group, the step of verifying the calculated coordinate system based on the coordinate system of the key point group calculated in a previous frame may be further included.

In one embodiment, when the markers are divided into a plurality of marker groups having different shapes or colors, in the step of storing as the keypoint group, the different marker groups are distinguished and stored into different keypoint groups. , In the matching step, the key point groups may be matched with the 3D model.

According to the embodiments of the present invention, it is possible to track an object while omitting a conventional marker having a relatively large volume or a complex structure in tracking the object.

In particular, since it is possible to track an object by attaching a plurality of markers to the surface and tracking it regardless of the shape of the object, it is possible to track objects having various arbitrary shapes or structures. It is possible to solve the problem of having to form an additional structure such as a holder or a hole in the object to attach it.

In addition, in the case of a conventional marker, at least three or more extraction points constitute a plane, and when the configured plane is perpendicular to the photographing direction of the photographing unit, location tracking is most accurate, but in the case of this embodiment, the extracted key points form a plane with each other. Since there is no need and detection in all directions is possible, the accuracy and speed of position tracking can be improved regardless of the posture or position of the object.

In addition, by adjusting the number of markers, tracking accuracy can be improved, and a keypoint group can be formed through a certain number of markers even when some of the markers are not identified due to the location of the object, the surrounding environment, or the surrounding structure. Since location tracking is possible if possible, usability in various environments can be improved.

In this case, by dividing the markers into a plurality of groups and distinguishing them so that they can be distinguished from each other, it is possible to improve the matching speed through selective matching in the formation of keypoint groups and matching with the 3D model.

In addition, in matching, it is possible to select an appropriate algorithm according to the type of the 3D model generated from the target object, so that the position tracking of the target object can be performed by adapting to various tracking environments or programs.

Further, by calculating the coordinate system of the key point group, in tracking the position of the target object, the calculated coordinate system is verified by receiving feedback on the position of the target object in the previous frame, so that the position of the target object Tracking speed as well as accuracy can be improved.

1 is a block diagram showing a system for calculating a position of an object according to an embodiment of the present invention.
FIG. 2 is a flowchart illustrating a method of calculating a position of an object using the position calculating system of FIG. 1.
3A is a perspective view illustrating an example of markers attached to an object in the position calculation method of FIG. 2, and FIG. 3B is an image illustrating an example of a three-dimensional model of the object in the position calculation method of FIG. 2.
4A to 4D are schematic diagrams illustrating a case in which a three-dimensional model of an object is a group of points in the position calculation method of FIG. 2.
5A to 5F are schematic diagrams illustrating a case in which a 3D model of an object is a 3D mesh in the position calculation method of FIG. 2.
6A is a perspective view illustrating another example of markers attached to an object in the method of calculating the position of FIG. 2, and FIG. 6B is an image illustrating an example of a key point group extracted from the object of FIG. 6A.

The present invention will be described in detail in the text, since various modifications can be made and various forms can be obtained. However, this is not intended to limit the present invention to a specific form of disclosure, it is to be understood as including all changes, equivalents, or substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals have been used for similar elements. Terms such as first and second may be used to describe various components, but the components should not be limited by the terms.

In the present application, terms such as "comprise" or "consist of" are intended to designate the presence of features, numbers, steps, actions, elements, parts, or a combination thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance the possibility of being added.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this application. Does not.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings.

1 is a block diagram showing a system for calculating a position of an object according to an embodiment of the present invention.

Referring to FIG. 1, the system 1 for calculating the position of an object according to the present embodiment includes a target object 100 and a tracked object 10 including a plurality of markers 200, a photographing unit 20, and a model. A forming unit 30, a position calculating unit 40, a storage unit 50, a matching unit 60, a determination unit 70, and a detection unit 80 are included.

Meanwhile, for convenience of explanation, the system 1 for calculating the position of the object according to the present embodiment and the method for calculating the position of the object using the same will be simultaneously described with reference to FIG. 1 and the following drawings.

FIG. 2 is a flowchart illustrating a method of calculating a position of an object using the position calculating system of FIG. 1. 3A is a perspective view illustrating an example of markers attached to an object in the position calculation method of FIG. 2, and FIG. 3B is an image illustrating an example of a three-dimensional model of the object in the position calculation method of FIG. 2. 4A to 4D are schematic diagrams illustrating a case in which a three-dimensional model of an object is a group of points in the position calculation method of FIG. 2. 5A to 5F are schematic diagrams illustrating a case where a 3D model of an object is a 3D mesh in the position calculation method of FIG. 2.

First, referring to FIGS. 1, 2, 3A and 3B, in the method of calculating the location of the target object, the tracked object 10 is defined and information on the target object 100 is obtained (step S10).

In this case, the tracked object 10 means the target object 100 to be tracked for convenience of description, and a plurality of markers 200 attached to the target object 100.

Accordingly, in the method of calculating the position of the target object, first, the plurality of markers 200 are attached to the target object 100 (step S11).

As shown in FIG. 3A, a plurality of the markers 200 are attached to the surface of the object 100 in the form of a sticker with a predetermined distance, and the number of the markers 200 The interval may be set in various ways in consideration of the shape or structure of the object 100 or the accuracy of location tracking.

In addition, the markers 200 are, for example, infrared reflective stickers, and can be easily identified by the photographing unit 20 to be described later. Furthermore, each of the markers 200 may have a circular shape, and thus may be attached to the target object 100 with a predetermined area.

As described above, the markers 200 are attached to the surface of the object 100 in the form of stickers, and accordingly, the markers 200 can be easily attached to any shape of the object 100, Therefore, the limitation of the structure or shape of the object that can be tracked through this can be minimized.

Meanwhile, in FIG. 5A, an example in which a plurality of markers 200 are attached on the object 100 is shown in a schematic diagram.

As described above, apart from attaching the plurality of markers 200 on the target object 100, in the model forming unit 30, a three-dimensional model for the target object 100 to be tracked Is generated (step S10).

The 3D model thus formed may be implemented as an image as shown in FIG. 3B, and the image illustrated in FIG. 3B is a 3D mesh model for the object 100.

In this way, the 3D model may be generated as a 3D mesh model for the target object 100, and in contrast, the 3D model is a group of points for the target object 100. It can also be created with data. In this case, the point group data means that the external shape of the object 100 is implemented with a plurality of points spaced apart by a predetermined interval.

The 3D model may be implemented through 3D scanning of the target object 100, and may be extracted from the 3D drawing of the target object 100 if it is generated.

Meanwhile, as an example of a three-dimensional model for the object 100 in FIG. 5D, a three-dimensional mesh model, that is, an external shape model of the object 100 is illustrated, and in FIG. 4B, the object 100 The point group data of is illustrated.

Thereafter, referring to FIGS. 1, 2 and 5B, the photographing unit 20 photographs the object 100 and the markers 200 attached to the object 100 (step S20). ).

In this case, the photographing unit 20 may be an infrared stereo camera, and thus, easy photographing and identification of the markers 200, which are the infrared reflective stickers, are possible.

That is, the photographing unit 20 captures and identifies the markers 200 by recognizing infrared rays reflected from the markers 200.

On the other hand, in the case of the photographing unit 20, the photographing is performed simultaneously with respect to the target object 100, but in the end, the necessary information is position information of the markers 200, and the position calculation unit ( 40).

That is, after that, referring to FIGS. 1, 2, 4A, and 5C, the position calculating unit 40 calculates the positions of the markers 200 from the image captured by the photographing unit 20. (Step S30).

More specifically, the position calculation unit 40 calculates a three-dimensional position of the markers 200, that is, a position in space. For this purpose, first, the position calculation unit 40 calculates the target object 100 ) And the two-dimensional positions of the markers 200 in the photographed images of the markers 200.

Thereafter, from the two-dimensional positions of the markers 200, a three-dimensional position of the markers 200, that is, a position in space, is calculated using a stereo relationship.

Since the photographing unit 20 is a stereo camera as described above, it is possible to calculate the three-dimensional position of the markers 200.

In this way, the locations of the markers 200 calculated by the location calculation unit 40 in the 3D space are extracted as points, as shown in FIGS. 4A and 5C, and the extracted points The so-called are stored in a key point group by the storage unit 50 (step S40).

In this case, the key point group is defined by grouping a plurality of points as shown in FIG. 4A or 5C extracted as positions of the markers 200 into one, and a marker photographed in the frame by the photographing unit 20 It means a group of information on the three-dimensional position of the markers extracted from the fields.

In general, the position or posture of the object 100 is changed every frame, and the surrounding environment of the object 100 is also changed every frame.

Accordingly, some of the markers 200 attached to the object 100 may not be identified as being covered by the posture or position of the object 100 or by the surrounding environment.

Accordingly, the extracted points, that is, points defined as the key point group in the corresponding frame may be different. As such, even if the points are different, in tracking the position of the target object 100 in the frame There is no problem.

That is, if the target object 100 is not completely covered, since a plurality of the markers 200 are attached to the entire surface of the target object 100 at predetermined intervals, at least some of the markers 200 Is because is necessarily photographed, and its location can be calculated.

On the other hand, each of the markers 200 attached to the object 100 has a predetermined area, for example, a circular sticker, and if the photographing unit 20 is photographed from a relatively long distance, Since the circular sticker is recognized as a point, the location of the markers 200 is sufficient to extract the point.

However, if the photographing unit 20 is photographed at a relatively close distance, the circular sticker may be recognized to have a predetermined area. In this case, in this embodiment, each of the markers 200 The location is extracted as a point corresponding to the center coordinate of the recognized predetermined area.

As described above, when the three-dimensional positions of the markers 200 in the corresponding frame are calculated and stored as a so-called keypoint group, as shown in Figs. 1, 2, 4C and 5E, the matching unit 60 ) Performs matching between the stored keypoint group and the 3D model for the target object 100 (step S50).

In this case, the matching unit 60 moves the 3D model for the object 100 while the keypoint group is fixed at its position, that is, rotates the 3D model or translates Matching is performed by moving (step S51).

For example, when the 3D model is point group data, referring to FIG. 4C, the key point group 210 stored as in FIG. 4A is in a state where the position is fixed, and FIG. 4B As shown in FIG. 3, by rotating or translating the 3D model 310 extracted from the object 100 together with coordinates, registration is performed on the key point group.

In this case, since the 3D model 310 is point group data, matching may be performed through point matching between the point groups and the key point group.

Such a matching method may be implemented using, for example, an iterative closest point (ICP) algorithm.

Unlike this, for example, when the 3D model is a 3D mesh model, referring to FIG. 5E, the keypoint group 210 stored as in FIG. 5C is As in 5d, by rotating or translating the 3D model 300 extracted from the object 100 together with the coordinates, registration is performed on the keypoint group.

In this case, since the 3D model 300 is a mesh model, matching between the keypoint group and the mesh model may be implemented using, for example, a point-to-plane registration algorithm.

As described above, according to the type of the 3D model 300, the registration may be performed in various ways.

Meanwhile, in performing the matching step, the determination unit 70 determines whether a matching error is less than a preset threshold, and if the matching error is less than the reference value, the matching step ends ( Step S60).

Thereafter, referring to FIGS. 1, 2, 4D and 5F, the detection unit 80 determines the key point group based on the result of matching the key point group 210 and the 3D model of the target object 100. The position of 210 is derived (step S70).

That is, in the matching step, while the key point group 210 is fixed, the 3D model of the target object 100 is moved. First, movement and rotation of the 3D model of the target object 100 Information is calculated (step S71).

Thereafter, based on the movement and rotation information of the 3D model, the coordinate system of the key point group 210 is calculated as shown in FIGS. 4D and 5F (step S72).

Thus, the position of the target object 100 is calculated through the calculated coordinate system of the key point group 210.

However, whether or not the calculated coordinate system of the keypoint group 210 is correct may be verified again by the detection unit 80, and may be verified by comparing the calculated coordinate system with the coordinate system derived from the previous frame (step S73).

That is, based on the position of the target object 100 in the previous frame, it can be determined whether the position of the target object 100 in the current frame has changed excessively, which is based on the results of the previous frames. The verification may be performed by simultaneously considering the movement trajectory of the object 100.

Thereafter, as shown in FIG. 2, it is determined whether or not the location tracking for the target object 100 is ended (step S80), and if location tracking is still required, the above-described steps may be repeated again.

6A is a perspective view illustrating another example of markers attached to an object in the method of calculating the position of FIG. 2, and FIG. 6B is an image illustrating an example of a key point group extracted from the object of FIG. 6A.

With reference to FIGS. 1 to 5F, it has been described that markers of the same shape or color are attached to the object, but markers having different shapes or colors may be divided into different groups and attached to the object.

That is, referring to FIG. 6A, a plurality of first marker groups 201 representing a first shape or a first color on the object 101, and a second marker group 201 different from the first shape or the first color A plurality of second marker groups 202 representing two shapes or a second color may be attached as a plurality of markers 205.

In this case, the markers of the first marker groups 201 may be attached to each other with a predetermined distance or interval. Likewise, the markers of the second marker groups 202 may also have a predetermined distance or interval from each other. Has and can be attached.

In this way, when the markers have different shapes or colors and are divided into a plurality of groups and attached to the target object 101, the respective marker groups 201 and 202 are separate key point groups. They may be divided into 215 and stored.

That is, the first marker groups 201 may be stored as a first keypoint group 211, and the second marker groups 202 may be stored as a second keypoint group 212.

Accordingly, in the step of matching the stored keypoint groups 215 and the 3D model of the target object, the first keypoint group 211 and the 3D model of the target object may be matched with each other. The two keypoint group 212 and the 3D model of the target object may be matched with each other.

As described above, by dividing and attaching the marker groups into a plurality of groups, creating a keypoint group by dividing it, and performing matching through this, a relatively large number of marker groups are used for registration, thus eliminating the need for registration time. Since the increase can be minimized, there is an advantage that it can be selectively used in a tracking situation that does not require high accuracy.

In addition, in consideration of the shape or structure of the target object, or the posture according to the movement, if a specific group of markers cannot be identified, position tracking can be performed by selectively using only the other group of markers. In consideration of various influences upon tracking, optimal tracking can be performed.

According to the embodiments of the present invention as described above, it is possible to track an object while omitting a conventional marker having a relatively large volume or a complex structure in tracking the object.

In particular, since it is possible to track an object by attaching a plurality of markers to the surface and tracking it regardless of the shape of the object, it is possible to track objects having various arbitrary shapes or structures. It is possible to solve the problem of having to form an additional structure such as a holder or a hole in the object to attach it.

In addition, in the case of a conventional marker, at least three or more extraction points constitute a plane, and when the configured plane is perpendicular to the photographing direction of the photographing unit, location tracking is most accurate. Since there is no need and detection in all directions is possible, the accuracy and speed of position tracking can be improved regardless of the posture or position of the object.

In addition, by adjusting the number of markers, tracking accuracy can be improved, and a keypoint group can be formed through a certain number of markers even when some of the markers are not identified due to the location of the object, the surrounding environment, or the surrounding structure. Since location tracking is possible if possible, usability in various environments can be improved.

In this case, by dividing the markers into a plurality of groups and distinguishing them so that they can be distinguished from each other, it is possible to improve the matching speed through selective matching in the formation of keypoint groups and matching with the 3D model.

In addition, in matching, it is possible to select an appropriate algorithm according to the type of the 3D model generated from the target object, so that the position tracking of the target object can be performed by adapting to various tracking environments or programs.

Further, by calculating the coordinate system of the key point group, in tracking the position of the target object, the calculated coordinate system is verified by receiving feedback on the position of the target object in the previous frame, so that the position of the target object Tracking speed as well as accuracy can be improved.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will be able to variously modify and change the present invention without departing from the spirit and scope of the present invention described in the following claims. You will understand that you can.

1: Target object location calculation system
10: trace 20: photographing unit
30: model forming unit 40: position calculating unit
50: storage unit 60: matching unit
70: determination unit 80: detection unit
100, 101: object 200, 205: markers
210, 215: key point 300, 310: 3D model

Claims (15)

A plurality of markers attached to the surface of the object;
A photographing unit that photographs the target object and the markers to obtain an image;
A position calculator that calculates the three-dimensional positions of the markers from the captured image;
A storage unit for storing the calculated three-dimensional position as a key point group;
A matching unit for matching the stored keypoint group with the 3D model of the target object; And
A system for calculating a position of an object including a detection unit for calculating a coordinate system of the key point group based on the matching. The method of claim 1,
A plurality of the markers are attached to the surface of the target object at predetermined intervals as an infrared reflective sticker,
The photographing unit is an infrared stereo camera. The method of claim 2,
The object has an arbitrary three-dimensional shape,
The three-dimensional model of the object, characterized in that the point (群) (群) data or a three-dimensional mesh (mesh) model is formed of the object position calculation system. The method of claim 1,
The markers are divided into a plurality of marker groups having different shapes or colors,
And the storage unit separates and stores different marker groups into different keypoint groups. The method of claim 1, wherein the position calculation unit,
A system for calculating a position of an object, characterized in that the center coordinates of each of the markers are calculated as the three-dimensional positions of the markers. The method of claim 1, wherein the matching portion,
In a state in which the key point group is fixed, the three-dimensional model is rotated or translated to match the key point group. Attaching a plurality of markers to the surface of the object;
Photographing the object and the markers to obtain an image;
Calculating three-dimensional positions of the markers from the captured image;
Storing the calculated three-dimensional position as a key point group;
Matching the stored keypoint group with the 3D model of the target object; And
And calculating a coordinate system of the key point group based on the matching. The method of claim 7, wherein calculating the three-dimensional position,
Acquiring the two-dimensional positions of the markers in the photographed image of the object and the markers; And
And calculating a three-dimensional position using a stereo relationship from the two-dimensional positions of the markers. The method of claim 8, wherein in the step of calculating the three-dimensional position,
A method for calculating a position of an object, characterized in that, among the markers attached to the object, the three-dimensional position is selectively calculated for only identifiable markers. The method of claim 7, wherein in the matching step,
And rotating or translating the three-dimensional model while the key point group is fixed to match the key point group. The method of claim 10, wherein in the matching step,
When the three-dimensional model of the target object is point group data, matching is performed using an iterative closest point (ICP) algorithm. The method of claim 10, wherein in the matching step,
When the 3D model of the target object is a 3D mesh model, registration is performed using a point-to-plane registration algorithm. The method of claim 7, wherein in the matching step,
When the matching error is less than a preset reference value, the matching is terminated. The method of claim 7, after the step of calculating the coordinate system of the key point group,
And verifying the calculated coordinate system based on the coordinate system of the key point group calculated in a previous frame. The method of claim 7,
When the markers are divided into a plurality of marker groups having different shapes or colors,
In the step of storing as the keypoint group, the different marker groups are separated into different keypoint groups and stored,
In the matching step, the method of calculating a position of an object, characterized in that the key point groups are matched with the 3D model.

Images